akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Graph Isomorphism network with edge weights

Roua_Rouatbi: <p>I am trying to build a GIN model that takes in edge weights with torch_geometric GINEConv, for graph classifcation. But I keep running into issues about how to feed my weights. My data is shaped like so:<br>
Data(x=[7, 4], edge_index=[2, 42], edge_attr=[42, 1], y=[1], num_nodes=7)<br>
the graphs are undirected and there are 4 node features and 1 edge feature.<br>
I tried to make the edge_attr and x input match, and I tried to add edge_dim and to manipulate the edge_attr before feeding it to every convolution but the model either doesn’t work or performs badly.<br>
Here is my code:</p>
<p>class GINE(torch.nn.Module):<br>
“”“GINE”“”<br>
def <strong>init</strong>(self, num_node_features,dim_h,num_classes,edge_dim=1, epsilone = 1e-4):<br>
super(GINE, self).<strong>init</strong>()<br>
self.conv1 = (<br>
Sequential(Linear(num_node_features, dim_h),<br>
BatchNorm(dim_h), ReLU(),<br>
Linear(dim_h, dim_h), ReLU()),edge_dim=edge_dim, train_eps=True)</p>
<pre><code>    self.conv2 = GINEConv(
        Sequential(Linear(dim_h, dim_h), BatchNorm(dim_h), ReLU(),
                   Linear(dim_h, dim_h), ReLU()),edge_dim=edge_dim, train_eps=True)
    self.conv3 = GINEConv(
        Sequential(Linear(dim_h, dim_h), BatchNorm(dim_h), ReLU(),
                   Linear(dim_h, dim_h), ReLU()),edge_dim=edge_dim, train_eps=True
    self.lin1 = Linear(dim_h*3, dim_h*3)
    self.lin2 = Linear(dim_h*3, num_classes)
    self.epsilone = epsilone

def forward(self, x, edge_index, batch, edge_attr):
    
    # Node embeddings 
    #lin = Linear(1, 4)
    #edge_attr=lin(edge_attr)
    h1 = self.conv1(x, edge_index,edge_attr)
    
    
    
    #lin1 = Linear(4, 64)
    
    #edge_attr=lin1(edge_attr)
    
    h2 = self.conv2(h1, edge_index,edge_attr)
    
    
    h3 = self.conv3(h2, edge_index,edge_attr)

    # Graph-level readout
    h1 = global_add_pool(h1, batch)
    h2 = global_add_pool(h2, batch)
    h3 = global_add_pool(h3, batch)

    # Concatenate graph embeddings
    h = torch.cat((h1, h2, h3), dim=1)

    # Classifier
    h = self.lin1(h)
    h = h.relu()
    h = F.dropout(h, p=self.epsilone)
    h = self.lin2(h)
   
    return h, F.log_softmax(h, dim=1)
</code></pre>

gany_alan: <p>Hello Roua,</p>
<p>I have exactly the same issue with the same convolution layer: GINEconv,<br>
I was then wondering if you came up with a solution.</p>
<p>Since it was more than one year ago, I hope you still remember this topic.</p>
How to make the following sort codes differentiable?

Shin_Hayate: <pre><code class="lang-auto">            scores, indices = torch.sort(scores_final[:, 0], descending=True)
            keypoints_scores=scores[:self.num_keypoints].unsqueeze(1)
            # post processing
            object_points=object_points.permute(0,2,1).squeeze(0)
            sorted_points = object_points[indices]
            keypoints= sorted_points[:self.num_keypoints, :3]
</code></pre>
<p>Some messages about the codes above:<br>
<strong>varibales:</strong><br>
scores_final: an output of a neural network, has gradients.<br>
object_points: data from point cloud, has no gradinets.<br>
I want to make the codes differentiable. Thanks a lot.</p>

ptrblck: <p>Indices are not usefully differentiable as their gradient will be zero everywhere and Inf or undefined at the rounding points. Since the cloud points are not attached to any computation graph and are only sorted by constant indices, their output also won’t be attached to the network output.</p>
Designing a Neural Network for Sequential Data Prediction with a global output

basilique: <p>I’m working on designing a neural network in PyTorch that takes a matrix of sequential data with shape <code>(N, L)</code> (where <code>N</code> is the number of samples and <code>L</code> is the number of elements in each sequence) and predicts a matrix of shape <code>(K, L)</code>, where <code>K</code> is a fixed value and different from <code>N</code>.</p>
<p>The challenge I’m facing is that the output matrix is not a per-sample prediction but rather a single matrix for the entire batch. Additionally, the model should handle different values of <code>N</code> during training and testing. For example, I might use a batch size of 128 during training but have an input <code>X</code> with 1000 samples during testing.</p>
<p>Here’s a simplified version of my initial approach using an LSTM model:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(MyModel, self).__init__()
        self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True)
    
    def forward(self, x):
        # x: (N, L)
        lstm_out, _ = self.lstm(x)  # lstm_out: (N, L, hidden_dim)
        # How to reshape or process lstm_out to get (K, L)?
        return output
</code></pre>
<p>However, this model’s output shape depends on the input batch size, which is not what I need. I don’t want to perform reduction operations like <code>mean</code> or <code>max</code> on the batch dimension.</p>
<p>How can I structure the model to extract and compress the information along the batch dimension into a single matrix output without explicitly reducing the size using operations like <code>mean</code> or <code>max</code>?</p>
<p>Thank you for your help!</p>
Load pretrained Detectron2 model as a Pytorch model

negreanu: <p>Any idea of how can I load a detectron2 pretrained model to use it as if it were a normal pytorch model?</p>

ptrblck: <p>Based on <a href="https://github.com/facebookresearch/detectron2/blob/master/docs/tutorials/models.md">this doc</a> it seems that you should be able to create the PyTorch model using:</p>
<pre><code class="lang-python">from detectron2.modeling import build_model
model = build_model(cfg)  # returns a torch.nn.Module
</code></pre>
<p>Loading a checkpoint and using this model is also explained in the same doc.</p>

cyn: <p>I tryed to save the enrire detectron2 model.  I want to run this model only installing pytorch. But fiid that it cant run without detectron2.</p>

Bartosz_Pietrzak: <p>Any luck with using detectron2 model in pure pytorch?</p>
Model outputs same prediction vector graph for different inputs (Time series), N beats

torch.optim: <p>Why does the graph when plotted of the output predicted vector of model looks same for different inputs.<br>
I’m trying to make a N_beats model work for predicting next 7 days electricity usage by giving it past info about electricity usage, the model’s training and testing loss go down similarly to very small values but after giving it the same test data and plotting the output vector it always shows the same graph, why so?<br>
I’m using Adam optimizer, loss_fn is L1Loss</p>
RVV support in pytorch

zhangfeiv0: <p>We plan to add support for the RISC-V Vector ISA extension in PyTorch, and we have already submitted an issue to the PyTorch team at <a href="https://github.com/pytorch/pytorch/issues/129553" rel="noopener nofollow ugc">#129553</a>. On this topic, we are eager to receive guidance from the PyTorch team. Similarly, we also hope that other interested individuals will join in the discussion.</p>
Torchtext AG_NEWS dataset return (int, tensor)

EAly: <p>Hello,</p>
<p>I’m new to torchtext and doing some tutorials with datasets. When I import the AG_NEWS dataset using the following:</p>
<pre><code class="lang-auto">import torchtext

train_set, test_set = torchtext.datasets.AG_NEWS(root='./data')

list(train_set)[0]
</code></pre>
<p>I get a tuple of an int and tensor:</p>
<pre><code class="lang-auto">(2,
 tensor([  432,   426,     2,  1606, 14839,   114,    67,     3,   849,    14,
            28,    15,    28,    16, 50726,     4,   432,   375,    17,    10,
         67508,     7, 52259,     4,    43,  4010,   784,   326,     2]))
</code></pre>
<p>I’m supposed to get a tuple of (int, str), so I’m not sure what am I doing wrong here? or if there is a more proper method to import datasets?</p>
<p>Appreciate any help</p>

ptrblck: <p>I cannot reproduce the issue using the last supported <code>torchtext==0.18</code> release and see:</p>
<pre data-code-wrap="python"><code class="lang-python">(3, "Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.")
</code></pre>
<p>as the output.</p>

EAly: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the reply.<br>
I running <code>torchtext==0.6.0</code> on Macbook, which is installed by default when through <code>conda install pytoruch::torchtext</code><br>
I’ve tried to force install <code>conda install  pytorch::torchtext =0.18.0</code>, but then when I run the above code I get this error <code>AttributeError: module 'torchtext' has no attribute 'datasets' </code></p>

ptrblck: <p>I got the same error and needed to <code>import torchtext.datasets</code> first. After the import your code executed and returned the posted output.</p>

EAly: <p>I did <code>import trochtext.datasets</code> but then I got an error that torchdata is not installed. I did install it, to get the following error:<br>
<code>ImportError: cannot import name 'DILL_AVAILABLE' from 'torch.utils.data.datapipes.utils.common'</code></p>
Hook which can modify gradients during backward

David_Tweedle: <p>Is there a way to achieve the following in pytorch:<br>
As each gradient with require_grad=True is calculated, we apply some function to the gradient. Then these changes are propogated to the rest of the gradient calculations.</p>
<p>For example, if A and B are matrices both with require_grad=True and my model is as follows<br>
output = B @ (A @ x)<br>
and when I put<br>
output.backward()<br>
I want to be able to apply my function to the gradient of B before it is propagated backwards to the gradient of A.<br>
I have tried B.register_hook() and this lets me adjust the gradient of B, but only after the gradient of A is calculated.<br>
Thank you.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/david_tweedle">@David_Tweedle</a>,</p>
<p>Have a look at the different types of hooks here: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_pre_hook" class="inline-onebox" rel="noopener nofollow ugc">Module — PyTorch 2.3 documentation</a></p>

soulitzer: <aside class="quote no-group" data-username="David_Tweedle" data-post="1" data-topic="205310">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_tweedle/48/61901_2.png" class="avatar"> David_Tweedle:</div>
<blockquote>
<p>but only after the gradient of A is calculated.</p>
</blockquote>
</aside>
<p>I’d actually expect applying a hook on B via .register_hook() to do what you want. Do you have a snippet to reproduce the issue?</p>

David_Tweedle: <p>Hi there,</p>
<p><a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> Yes, this is what I was looking for, thank you very much!</p>
<p><a class="mention" href="/u/soulitzer">@soulitzer</a> I went back and tried the following code, and you are right, torch.Tensor.register_hook does what I want too. Here is a snippet to verify.</p>
<pre><code class="lang-auto">import torch
A = torch.tensor([[1.,0],[1.,2.]], requires_grad=True)
B = torch.tensor([1.,2.], requires_grad=True)
x = torch.tensor([1.,2.])
def output(input):
    return B @ (A @ input)
A.grad, B.grad = None, None
loss = output(x)
loss.register_hook(lambda grad: 0 * grad) # gradients of A and B should be zero
loss.backward()
A.grad, B.grad
# (tensor([[0., 0.],
#         [0., 0.]]),
# tensor([0., 0.]))
</code></pre>
<p>Thank you both for your replies.</p>
Concurrent Multi GPU batch inferencing

Alex_Chung: <p>Hello!<br>
I would like to do inference (with gpu) by batches of audio data while concurently saving the outputs to memory with the cpu. For this, I’ve read many chats here and in other pages, and so far I came up with this:</p>
<p>Dataset:</p>
<pre><code class="lang-auto">def create_dataset(path, seed=123):
  paths = rglob_audio_files(path)    # This function iterates over the directories and get the lists of paths from the data (e.g. ../../...wav, .mp3, .flac, etc)
  random.Random(seed).shuffle(paths)
  ds = Dataset(paths, training=False)
  return ds
</code></pre>
<p>Dataloader:</p>
<pre><code class="lang-auto">def create_dataloader(path, batch_size, world_size):
    ds = create_dataset(path=path)
    
    dl = DataLoader(
        ds,
        batch_size=batch_size,
        shuffle=True,
        num_workers=world_size,
        collate_fn=ds.collate_fn
    )
    return dl
</code></pre>
<p>Parallel Inference Function:</p>
<pre><code class="lang-auto">def inference(model, path, out_path, batch_size, device, world_size):
    dist.init_process_group("nccl", rank=device, world_size=world_size)
    pool = concurrent.futures.ThreadPoolExecutor(max_workers=world_size) 
    loader = create_dataloader(path,batch_size,world_size)
    def save(out_path,data):
            for d in data:
                torchaudio.save(out_path, d[None], sampling_rate)
    for data in loader:
        output_data = model(data)[0].cpu()
        out_path = out_path / path.relative_to(path) # I don't know how to get the original path here 
        for i in range(0,world_size,batch_size/world_size):
            if i+batch_size/world_size&lt;batch_size:
                pool.submit(save,out_path,output_data[i:i+batch_size/world_size])
            else:
                pool.submit(save,out_path,output_data[i:])
</code></pre>
<p>Main:</p>
<pre><code class="lang-auto">def main(in_path,out_path,batch_size,world_size):
  def _inference(rank,world_size):
        return inference(in_path,out_path,batch_size,rank,world_size)
  mp.spawn(_inference,
                    args=(args.world_size, ),
                    nprocs = args.world_size)
</code></pre>
<p>Can somebody tell me if this is the correct form of parallelizing and doing this inference concurrently while having the cpu saving the output files? Or does the dataloader already do this?<br>
I also need someone to advice me how to get the original path in my inference function, since to get this path the only thing that occurs to me is that I would need another “for” iterating over a tqdm of the path and that would also need to be in parallel (I guess). If something is not clear about my questions, please ask.</p>
Why does the MPI backend require compiling from source?

nathanbrown-Arm: <p>Hello,</p>
<p>PyTorch has a built-in MPI backend, which is only available if PyTorch is compiled from source on a system which has an MPI implementation already installed <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">link1</a>, <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends" rel="noopener nofollow ugc">link2</a>. Why can’t PyTorch ship with a default MPI implementation and enable the MPI backend without requiring compiling from source?</p>
<p>link1 states “Unfortunately, PyTorch’s binaries cannot include an MPI implementation”, but does not expand as to why.</p>

trusira: <aside class="quote no-group" data-username="nathanbrown-Arm" data-post="1" data-topic="205363">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/9dc877/48.png" class="avatar"> nathanbrown-Arm:</div>
<blockquote>
<p>Unfortunately, PyTorch’s binaries cannot include an MPI implementation</p>
</blockquote>
</aside>
<p>Likely due to licensing reasons with respect to software distribution.</p>

Honnappa: <p>May be due to there being several implementations of MPI?</p>
Understanding Memory usage for the DiT model

Lakshya_LNU: <p>I’m having hard time understanding memory allocation during backward pass of a LLM. As I increase the depth of the LLM, Im seeing huge increment in the backward pass as compared to what estimated using <a href="https://www.latent.space/p/transformers-math" class="inline-onebox" rel="noopener nofollow ugc">The Mathematics of Training LLMs — with Quentin Anthony of Eleuther AI</a> I’m running on a single GPU for now, but have the FSDP(FULL shard) enabled though. which I use to scale the model. Im using pytorch lightning to run the model</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Depth</th>
<th>mem before DiT blocks</th>
<th>mem after DiT blocks</th>
<th>mem after backward</th>
</tr>
</thead>
<tbody>
<tr>
<td>2</td>
<td>6081</td>
<td>8445</td>
<td>27527</td>
</tr>
<tr>
<td>10</td>
<td>13201</td>
<td>16525</td>
<td>64533</td>
</tr>
<tr>
<td>20</td>
<td>22101</td>
<td>26865</td>
<td>OOM</td>
</tr>
</tbody>
</table>
</div><p>I’ve tried to see if its just cache or something, but running torch.cuda.empty_cache() doesn’t really make the numbers very different as well, eventually the model ends up allocating the same memory, to run efficiently I believe.</p>
<p>Does anyone know why this might be the case or what tools I can use to debug this</p>
How do I calculate the mean and standard deviation from training images

JonathanSum: <p>I have training images with different size.  I will resize if needed in the dataloader. However, how do I calculate it for dataloader?</p>

beaupreda: <p>Hello,</p>
<p>You will have to make a script that passes every image in your dataset beforehand. You can use <code>torch.mean(img, dim=(1, 2))</code> and <code>torch.std(img, dim=(1, 2))</code> to compute the mean and standard deviation for one image across the 3 channels.</p>
<pre><code class="lang-auto">    x = torch.rand(size=(3, 64, 64), dtype=torch.float32)
    mean = torch.mean(x, dim=(1, 2))
    std = torch.std(x, dim=(1, 2))
    print(mean)
    # tensor([0.4923, 0.5065, 0.5018])
    print(std)
    # tensor([0.2874, 0.2872, 0.2864])
</code></pre>
<p>Add the mean and std of every image from your dataset, then divide by the total number of images and save the final mean and std values for future uses!</p>
<p>Hope this helps!</p>
Pytorch cannot access all 4 GPUs

Tridib_dutta: <p>Hi,<br>
I am new to running model training on a pytorch on a Azure VM (NC16as_T4_v3). According to the <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/nct4-v3-series" rel="noopener nofollow ugc">documentation</a>, this instance has 16GB for each GPU (x 4 = 64 GB). I am using pytorch (version: ‘2.3.1+cu121’). So I believe the installed torch is the correct version to use GPUs. However, when i am running:<br>
<code>torch.cuda.device_count()</code>, the result shows <code>1</code>, although it should show <code>4</code>. How do I access all 4 GPUs and enable distributed training ?<br>
Any help would be greatly appreciated.</p>

eqy: <p>This table seems to indicate that <code>NC16as_T4_v3</code> only has one GPU:<br>
<a href="https://learn.microsoft.com/en-us/azure/virtual-machines/nct4-v3-series" rel="noopener nofollow ugc">NCas T4 v3-series - Azure Virtual Machines | Microsoft Learn</a><br>
while <code>Standard_NC64as_T4_v3</code> would have 4 GPUs.</p>

Tridib_dutta: <p>oopss!!! How did I miss that? Thanks for pointing that out.</p>
Memory spike Inplace sum of two torch.tensor vector

RAHUL_ROY: <p>I am trying to do an InPlace addition of two torch.tensor vectors.<br>
If the individual sizes are in order X MB after addition it is reaching to in GBs.<br>
I am using python’s  “+=” operator doing this also tried with add_().</p>
<p>If someone can help me with the memory efficient way to do this it will be great help for me.</p>
<p>This is the code snippet:-<br>
for x in range(10000):<br>
x1+=func(z) <span class="hashtag-raw">#func</span>(z) returns a tensor of same shape as x1</p>
<p>resulting x1 seeing a memory spike.</p>

soulitzer: <p>Doing either <code>+=</code> or <code>add_</code> seem fine to me.</p>
<p>What does <code>f</code> do? Do any of your tensors require gradients? It could mean you accumulating a larger and larger autograd graph over time.</p>

RAHUL_ROY: <p>Hi,<br>
Thanks for your reply. Yes I was doing that. That was the reason for accumulating large memory.</p>

soulitzer: <p>If this is still a problem for you, one thing you could do in this case is to apply activation checkpointing <a href="https://pytorch.org/docs/stable/checkpoint.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.checkpoint — PyTorch 2.3 documentation</a>. Instead of saving the activations during forward, accumulating them over time. They will be computed as needed during backward in chunks, where the size of the chunks is as small as you need to suppress the memory spike. (the size of the chunk is determined by how much of the forward you wrap in a given checkpoint)</p>

RAHUL_ROY: <p>First of all Thank you so much. It was so helpful.<br>
What if I extract or remove the grad post propagate function?</p>

soulitzer: <p>What do you mean by the grad post propagate function?</p>

RAHUL_ROY: <p>I mean from the propagate function of MessagePassing we get back the tensor with the grad_function(grad_fn=) attached to it. That increases the memory utilisation.</p>
<p>Instead of that, we just take the “data” part. It brings down the memory utilisation.<br>
Is this a good practice of it can introduce some conflicts/anomalies in the result?</p>

soulitzer: <p>I see, if you don’t need to backward through that graph, you could do torch.no_grad to avoid creating the graph in the first place.</p>
Optimizers not support foreach & fused in LibTorch

Alex_Wan: <p>Hi, I am using Libtorch 2.3.1 + cu118 to train my model. I have massive number of parameters and I divide them into groups for fine-grained optimzation. For example, I may use Adam and each param have different  beta or lr, they may not have the same step, neither. After I read the source code of Libtorch, I found the realization of Adam is a single for-loop and there is no parallel accleration (foreach or fused based on CUDA). Is it not planned to develop? Is it possible for a single person to write the CUDA acceleration code?</p>
Splitting model layers across GPUs

LeonBinder: <p>Hey Guys,<br>
I want to run an llm model in the most efficient and fast way possible.</p>
<p>I already tried splitting up the layers across my 2 GPUs (Tesla M40) but I actually want to split each layer in half. So that per layer one GPU gets 1/2 Layer and the other GPU gets another Half and then sync after every layer.</p>
<p>From my research FSDP sounds good but does someone have experience to use it without a backward_pass? Because I dont want to train the model yet, I only want to run it and answer questions.</p>
<p>Thanks!</p>
Why type _modules as "Module"?

kevin-vitro: <p>I have been working on tracking down an <a href="https://github.com/fastai/fastai/issues/3901" rel="noopener nofollow ugc">issue in fastai’s library</a> that involves not being able to export a learner.  I have partially tracked down the cause of the issue which is what brings me here.</p>
<p>This is the line of code that is partially causing the object to be unpicklable: <a href="https://github.com/pytorch/pytorch/blob/53f462c506286edae673ef76f7b0bedc6279ff2b/torch/nn/modules/module.py#L468" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/nn/modules/module.py at 53f462c506286edae673ef76f7b0bedc6279ff2b · pytorch/pytorch · GitHub</a></p>
<p>The cause of this is that Module is wrapped in double quotes <code>Optional["Module"]</code> and my question is whether this is intentional and if so, what does it mean to have the class in double quotes like this?</p>
<p>I have tried looking through the PEPs that are related to typing, but didn’t really find anything that explained this but basically since torch v1.13.x, this has cause exporting to break in fastai because this is being added to <code>__annotations__</code> and then fastai is using <code>functools.update_wrapper</code> to copy certain hidden object attributes to the instance of the class so instead of <code>__annotations__</code> being hidden in the background, it gets added to <code>__dict__</code> where pickle tries to export it, but isn’t able to because the type gets converted to <code>typing.Optional[ForwardRef('Module')]</code> which is non-pickleable once you dig down into it.</p>
<p>My thought is that maybe this is unintentially a string and it should actually be <code>Module</code> instead of <code>"Module"</code> in the type hints which would be ideal for me, but only a good suggestion if the current double-quotes weren’t intentional.</p>
<p>Thank you for your time,<br>
KevinB</p>

soulitzer: <p>Using “Module” in quotes when doing type annotations within the class definition may be necessary order to avoid circular reference.</p>

kevin-vitro: <p>Ok, that makes sense to me. So it was definitely valid and intentional then. I guess that means that the problem is with how they get converted into ForwardRefs in <code>__annotations__</code> then.  This is helpful and I think it gives me a good clue as to where I need to go explore next. I need to figure out if functools.update_wrapper should be resolving the ForwardRefs before copying annotations or something else.</p>
Getting infinite loss with my simple Linear Regression model

Hyunggon_Yu: <p>hello below is my first linear regression model with PyTorch.<br>
if I increase num_data to 1000, the loss become infinite…<br>
I don’t know why<br>
any help would be appreciated so much!!</p>
<p>My Colab Link :<br>
<a href="https://colab.research.google.com/drive/1oi9jZyb5TucwUERMSaVw1ZEVOt9uxrT4" rel="nofollow noopener">https://colab.research.google.com/drive/1oi9jZyb5TucwUERMSaVw1ZEVOt9uxrT4</a>)</p>

Hyunggon_Yu: <p>Solved it by changing learning_rate. but I don’t know what’s going on <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

Vishma_Pratim_Das: <p>Brother, I don’t know if it’s any help to you, but you hadn’t used an activation function. You must use one, preferrably sigmoid (its simpler) or RELU (a little complexity for the system but still okay)</p>
<p>Hope it helps some new students out there</p>
How to apply threshold on model weights while training?

Mayank_Kumar: <p>Let’s say I have a model with a bunch of linear layers (nn.Linear) . Let’s say a layer ‘fc1’ has weight parameter of shape (m, n). For each row, I want to keep only top-k values (magnitude wise).</p>
<p>What is the correct way to implement this while training the model ? I looked at torch.prune which looks similar but I am not sure how to use it or if there is another way to do it correctly ?</p>

KFrank: <p>Hi Mayank!</p>
<aside class="quote no-group" data-username="Mayank_Kumar" data-post="1" data-topic="205228" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mayank_kumar/48/62230_2.png" class="avatar"> Mayank_Kumar:</div>
<blockquote>
<p>For each row, I want to keep only top-k values (magnitude wise).</p>
</blockquote>
</aside>
<p>Use <code>topk()</code> to get the indices of the <em>smallest</em> <code>n - k</code> values in your<br>
weight and then use <code>scatter()</code> to zero them out:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.3.1
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt;
&gt;&gt;&gt; m = 3
&gt;&gt;&gt; n = 5
&gt;&gt;&gt; k = 2
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.randn (3, 5)
&gt;&gt;&gt;
&gt;&gt;&gt; t
tensor([[-0.0404,  1.7260, -0.8140,  1.3722,  0.5060],
        [-0.4823, -0.7853,  0.6681, -0.4439,  0.1888],
        [ 0.5986,  0.6458,  0.6306, -1.4668, -0.6798]])
&gt;&gt;&gt;
&gt;&gt;&gt; ind = t.abs().topk (n - k, dim = 1, largest = False).indices
&gt;&gt;&gt; t_topk = t.scatter(1, ind, torch.zeros (ind.size()))
&gt;&gt;&gt;
&gt;&gt;&gt; t_topk
tensor([[ 0.0000,  1.7260,  0.0000,  1.3722,  0.0000],
        [ 0.0000, -0.7853,  0.6681,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000, -1.4668, -0.6798]])
</code></pre>
<p>(Note, I believe you want to zero out the smaller values because removing<br>
them from the weight (and changing the shape of the tensor) will likely<br>
break the connections between the various layers in your model.)</p>
<aside class="quote no-group">
<blockquote>
<p>What is the correct way to implement this while training the model ?</p>
</blockquote>
</aside>
<p>Rather than apply this logic to your layers after every optimization step<br>
(which would likely un-zero your zero values), you would probably want<br>
to package this logic as a <a href="https://pytorch.org/tutorials/intermediate/parametrizations.html" rel="noopener nofollow ugc">pytorch parametrization</a> and register it for each<br>
layer that you want to treat this way.</p>
<p>Best.</p>
<p>K. Frank</p>
find_package(Torch REQUIRED) fails - 2.3.1 and nightly

artyom-beilis: <p>I’m working on pytorch opencl backend and I’m failing to use find_package on latest pytorch versions (I use CPU version)</p>
<p>I installed pytorch in virtual environment (checked for both 2.3.1 and nightly)</p>
<p>I have CMakeLists with line</p>
<pre><code>find_package(Torch REQUIRED)
</code></pre>
<p>I run cmake</p>
<pre><code>cmake -DCMAKE_PREFIX_PATH=$VIRTUAL_ENV/lib/python3.12/site-packages/torch/share/cmake/Torch ..
</code></pre>
<p>And get an error</p>
<pre><code class="lang-auto">CMake Error at /home/artik/Projects/venv/pt_3.12_nightly/lib/python3.12/site-packages/torch/share/cmake/Caffe2/public/mkl.cmake:8 (target_include_directories):
  Cannot specify include directories for imported target "caffe2::mkl".
Call Stack (most recent call first):
  /home/artik/Projects/venv/pt_3.12_nightly/lib/python3.12/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:106 (include)
  /home/artik/Projects/venv/pt_3.12_nightly/lib/python3.12/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:4 (find_package)
</code></pre>
<p>It works with older pytorch versions 2.1.1 or 1.1.13</p>
<p>Does anybody know how to solve it?</p>

artyom-beilis: <p>I updated to cmake 3.26 and it solved</p>