akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
NN.Linear latency report?

kovek: <p>Hi PyTorch!</p>
<p>I was wondering if there’s anywhere that has explored different statistics about nn.Linear? For example, how much slower is it in CPU vs GPU, how much time it takes to run depending on dimensions size, how much time does it take to load a matrix for the input to the operating, what about for loading a matrix for the weights, how long does back propagation take?</p>
<p>I think I could do the analysis myself, but I want to see what is out there first.</p>
<p>Thank you,</p>
<p>Kovek</p>
Unable to learn on merged branches

Victoic: <p>Hello.<br>
I’m trying to train a model in which a merge outputs from three different branches, however after one iteration every classification loss is the same, as if the gradients are not being updated.</p>
<p>Below is a general idea of what I’ve made:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bab68c12846dfa197c936a8614cb32b23c70c81.png" data-download-href="https://discuss.pytorch.org/uploads/default/7bab68c12846dfa197c936a8614cb32b23c70c81" title="model"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bab68c12846dfa197c936a8614cb32b23c70c81.png" alt="model" data-base62-sha1="hE1YdYiE3bgTNogdLIMSS2WJbwt" width="690" height="212" data-dominant-color="F0F1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model</span><span class="informations">868×267 16.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Blue branches are frozen, the non-frozen branch is a <a href="https://github.com/yhenon/pytorch-retinanet/tree/master/retinanet" rel="noopener nofollow ugc">retinanet</a>. The outputs from each branch is processed in a Linear layer to standardize the number of channels, then I concatenate each output and pass them through a linear head to get my desired output.</p>
<p>When I say “concatenate”, I mean Tensor.repeat() and Tensor.reshape where necessary so I get same shape for each output. The regression output from the retinanet is learning correctly, only the classification loss is never changed. Any help will be appreciated.</p>
<p>P.S.: BN is highlighted because I’ve tried with and without it, with no change to loss.</p>
InceptionV3 activation function for testing?

niloleart: <p>I’m doing an image classification problem (binary for the moment but it will scale to 3 or 4 classes later on) with InceptionV3.<br>
When training I use the raw output (logits) of the classifier, which is a FC with 2 neurons output. My loss function is a Focal Loss, based on BCE with Logits Loss, due to high class imbalance. It seems to work pretty well, although there is some overfitting.</p>
<p>When testing the saved model with non-seen data, my metrics varies a lot depending on the batch size used for testing. There is clearly something wrong but I cannot find it. I have two ideas:</p>
<ol>
<li>I’m doing something wrong when testing the network, as it probably should have an activation function. A sigmoid or a softmax and then thresholding?</li>
<li>Normalizing/Data augmentation: I’m normalizing my data based on the mean and std deviation of the whole dataset, both for training and testing. And then I perform data augmentation on the training data but not on the testing. I think this is correct. I’ve seen in this forum that the batch normalization layers may be affecting the results when testing, but it should not as I’m setting the model on eval mode.</li>
</ol>

ptrblck: <aside class="quote no-group" data-username="niloleart" data-post="1" data-topic="171901">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/niloleart/48/46263_2.png" class="avatar"> niloleart:</div>
<blockquote>
<p>When testing the saved model with non-seen data, my metrics varies a lot depending on the batch size used for testing.</p>
</blockquote>
</aside>
<p>Did you forget to call <code>model.eval()</code> before starting the evaluation?<br>
This could indicate that e.g. batchnorm layers are still in their <code>training</code> mode and are normalizing the input activation using the batch stats.</p>

niloleart: <p>No. I was calling model.eval() from the beggining.<br>
I’ve found the bug. I was computing the AUROC per batch and then averaging it on every epoch. Due to high class imbalance some of the batches had only negative values.</p>
<p>Is it necessary to set track_running_stats = False for every BatchNorm2 layer?</p>

ptrblck: <p>No, it’s not necessary since this would disable running stats and your inference outputs would then also depend on the batch size.</p>

alihamza: <p>I have the same issue<br>
I trained a dataset on inception net v3 pytorch<br>
I used both losses. Training accueacy reached upto 98% and validation accuracy  89%<br>
But when i test this model i got only 2% testing accuracy dont why it is happening<br>
Testing data is similar like training and validation</p>
Worse performance using model.eval() and dropout layers for regression task

insomnia: <p>I am training a model to predict eigenfrequencies and quality factors for a membrane resonator, given certain design parameters. The architecture is a deep neural network with multiple layers interspersed with dropout layers. Batch normalisation is not used.</p>
<p>The problem occurs when I load a trained model and set model.eval(). This causes the performance to drop significantly. From roughly 0.1 percent error on the training set to 20-30 percent on the test set. If I refrain from setting model.eval(), the performance on the test set is comparable to the performance on the training set.</p>
<p>I have experimented a bit, and it seems to have something to do with how the layers are normalised during dropout. If I shift the p-value of the dropout layer just a tiny bit for the trained model (without setting model.eval()), the increase in error is small. If I turn it all the way to zero the error is similar to when I set model.eval(), roughly 20-30 percent.</p>
<p>Anyone have an idea of how to force the model to normalise properly when using model.eval()? Since my metrics are good without setting model.eval() the model is clearly well-trained. Is it even an acceptable strategy to predict without first setting model.eval()?</p>
<p>At this stage, the dropout layers are not cruicial, but my data is continuously generated and at some stage I suspect dropouts layers may actually be beneficial or even necessary in order to succeed.</p>
<p>Thanks in advance</p>
Intermediate results in the forward method of my neural net have requires_grad=False

mihai145: <p>I have defined a small neural net and tried to train it. However, the loss did not have requires_grad set - that was curious. I set a breakpoint in the forward method of my neural net, and none of the intermediate variables created (embs, means, sim, out) had requires_grad set.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">class Cbow(nn.Module):
    def __init__(self, vocab_size, hctx_len, emb_size):
        super().__init__()
        self.proj = nn.Linear(in_features=vocab_size, out_features=emb_size, bias=False)
        self.hidden = nn.Linear(in_features=emb_size, out_features=vocab_size, bias=False)

    def forward(self, x):
        # x: (num_batches, 2*hctx_len, vocab_size)

        embs = self.proj(x)               # (num_batches, 2*hctx_len, emb_size)
        means = embs.mean(dim=1)          # (num_batches, emb_size)

        sim = self.hidden(means)            # (num_batches, vocab_size)
        out = torch.softmax(sim, dim=1)     # (num_batches, vocab_size)
        # breakpoint()

        return out


def fit_model_layers(model, train_loader, epochs, lr=0.01):
    model.train()

    loss_fn = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=lr)
    optimizer.zero_grad()

    for e in range(epochs):
      running_loss, num_batches = 0, 0
      for x, y in train_loader:
          out = model(x)

          # calculate loss
          loss = loss_fn(out, y)                # loss.requires_grad = False !!!!!!
          running_loss += loss.item()
          num_batches += 1

          # backprop + optimization step
          loss.backward()
          optimizer.step()
          optimizer.zero_grad()

      print(f'Epoch {e+1} loss: {running_loss / num_batches}')
</code></pre>
<p>I don’t quite understand what I do wrong. In the optimization example from PyTorch (<a href="https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Optimizing Model Parameters — PyTorch Tutorials 2.2.1+cu121 documentation</a>) intermediate results in forward() all have requires_grad set.</p>
<p>What could be wrong?</p>

ptrblck: <p>Did you try to add debug <code>print</code> statements into the <code>forward</code> to check this attribute?</p>

mihai145: <p>Sorry for the bother. I ran the model in a different environment (a new Jupyter notebook) and it worked as expected, with the loss and all intermediate values in forward() having requires_grad=True.</p>
<p>Is there any way that some jupyter notebook state (possibly as a result of the other code I’ve already run) could produce this? I can’t explain my previous situation otherwise.</p>

ptrblck: <p>Yes, this could happen e.g. if you’ve previously disabled gradient computation globally.</p>
nn.Conv2 and nn.Linear in 3d points operation

ElijahZh: <p>In PointNet++, suppose the input point data shape is [1, 3, 1024, 32] where the 1024 the centroids, and 32 is the neighbors and 3 is xyz coordinate with batch size as 1.</p>
<p>The input data is passed into nn.Conv2d(in_channels=3, out_channels=64, kernel_size=(1, 1))</p>
<p>Can we use nn.Linear instead of nn.Conv2d:<br>
Change the input point data shape as [1 * 1024 * 32, 3], then pass the data into nn.Linear(3, 64)</p>
<p>I am wondering if these two have the same effect, since the conv2d kernel size is (1, 1)?</p>

KFrank: <p>Hi Elijah!</p>
<aside class="quote no-group" data-username="ElijahZh" data-post="1" data-topic="199401" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elijahzh/48/59766_2.png" class="avatar"> ElijahZh:</div>
<blockquote>
<p>Can we use nn.Linear instead of nn.Conv2d:</p>
</blockquote>
</aside>
<p>Yes (although I’m not sure what the point would be).</p>
<p>Consider the following example (with fewer centroids and neighbors for<br>
convenience):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.2.1
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt;
&gt;&gt;&gt; conv = torch. nn.Conv2d (in_channels = 3, out_channels = 5, kernel_size = (1, 1))
&gt;&gt;&gt;
&gt;&gt;&gt; x = torch.randn (1, 3, 2, 4)
&gt;&gt;&gt;
&gt;&gt;&gt; resConv = conv (x)
&gt;&gt;&gt;
&gt;&gt;&gt; lin = torch.nn.Linear (3, 5)
&gt;&gt;&gt; with torch.no_grad():
...     _ = lin.weight.copy_ (conv.weight.squeeze())
...     _ = lin.bias.copy_ (conv.bias)
...
&gt;&gt;&gt; resLin = lin (x.permute (0, 2, 3, 1).flatten (0, 2)).reshape (1, 2, 4, 5).permute (0, 3, 1, 2)
&gt;&gt;&gt;
&gt;&gt;&gt; torch.allclose (resConv, resLin)
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

ElijahZh: <p>Thank you. This is what I need.</p>
Multi-Class Multi-ouptut classification

Ifyanene: <p>I am working on a multi-output (i.e &gt; 1 output target) multi-class (i.e &gt; 1 class) (I believe this is also called a multi-task problem).<br>
For example, my train_features_data is of shape (4, 6) (i.e three rows/examples and 6 columns/features), and my train_target_data is of shape (4, 3) (i.e 4 rows/examples and 3 columns/targets). For each target I have three different classes (-1, 0, 1).</p>
<p>I define an example model architecture (and data) for this problem like so:</p>
<pre><code class="lang-auto">import pandas as pd
from torch import nn 
from logging import log
import torch
feature_data = {
    'A': [1, 2, 3, 4],
    'B': [5, 6, 7, 8],
    'C': [9, 10, 11, 12],
    'D': [13, 14, 15, 16],
    'E': [17, 18, 19, 20],
    'F': [21, 22, 23, 24]
}

target_data = {
    'Col1': [1, -1, 0, 1],
    'Col2': [-1, 0, 1, -1],
    'Col3': [-1, 0, 1, 1]
}

# Create the DataFrame
train_feature_data = pd.DataFrame(feature_data) 
train_target_data = pd.DataFrame(target_data)
device = "cuda" if torch.cuda.is_available() else "cpu"

# create the model
class MyModel(nn.Module):
  def __init__(self, inputs=6, l1=12, outputs=3):
      super().__init__()
      self.sequence = nn.Sequential(
        nn.Linear(inputs, l1),
        nn.Linear(l1, outputs),
        nn.Softmax(dim=1)
    )
      
  def forward(self, x):
      x = self.sequence(x)
      return x
    
x_train = torch.tensor(train_feature_data.to_numpy()).type(torch.float)
model = MyModel(inputs = 6, l1 = 12, outputs = 3).to(device)
model(x_train.to(device=device))
</code></pre>
<p>When I pass my train data into the model (i.e when i call model(x_train.to(device=device))), I get back an array of shape (4, 3).</p>
<p>By following this resource <a href="https://www.learnpytorch.io/02_pytorch_classification/#83-creating-a-loss-function-and-optimizer-for-a-multi-class-pytorch-model" rel="noopener nofollow ugc">resource</a> My expectation was that I would get something like (4, 3, 3) whereby the 4 is the number of examples in my features and targets, the middle 3 (i.e the second axis) represents the logits (or in this case because I have a softmax function, this will be the predicted probabilities) of each example (and this would be 3 because I have three classes), while the third axis (or rightmost 3 value in the shape) represents the number of outputs/columns I have in my train_target_data.</p>
<p>Can someone please provide some guidance on what I’m doing incorrectly here (if my approach is wrong) and how to go about fixing it. Thanks.</p>
For 4bit quantization

jaeheon: <p>Hi I have a question in the process of quantizing to 4 bits. I created a new backend of get_default_qat_qconfig and set quant_min and quant_max from 0 to 15 to proceed with learning, but an error stating ‘zero_point’ must be between ‘quant_min’ and ‘quant_max’ appeared.<br>
What else do I need to do to resolve this error?<br>
Please refer to the picture below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4829a531b9c21c67b93a95195477ef959d5bb87.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e4829a531b9c21c67b93a95195477ef959d5bb87" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4829a531b9c21c67b93a95195477ef959d5bb87_2_690x193.jpeg" alt="image" data-base62-sha1="wBuG4ifnhEBPNvLZByGXvX9FhBB" width="690" height="193" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4829a531b9c21c67b93a95195477ef959d5bb87_2_690x193.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4829a531b9c21c67b93a95195477ef959d5bb87_2_1035x289.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4829a531b9c21c67b93a95195477ef959d5bb87_2_1380x386.jpeg 2x" data-dominant-color="343738"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1530×429 145 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Dhruven_Rathod: <p>I am having similar issue. can someone help with this?</p>

jerryzh168: <p>can you print the zero point, quant_min and quant_max to see what i sthe problem?</p>
Quantized onnx model run slower

minhhotboy9x: <p>Hi, I’m trying to quantize a simple model with several conv2d layers. When I compared the quantized onnx model with the original model on cpu, the quantized model run slower.<br>
Here is my code:</p><aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1-2bfKGPXTQP1FItFYRBE82MVFv1hsaEj/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1-2bfKGPXTQP1FItFYRBE82MVFv1hsaEj/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1-2bfKGPXTQP1FItFYRBE82MVFv1hsaEj/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1-2bfKGPXTQP1FItFYRBE82MVFv1hsaEj/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">quantize_model.ipynb</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Can anyone help me with this, please? Thank you very much?</p>

jerryzh168: <aside class="quote no-group" data-username="minhhotboy9x" data-post="1" data-topic="199350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minhhotboy9x/48/61753_2.png" class="avatar"> minhhotboy9x:</div>
<blockquote>
<p>Can anyone help me with this, please? Thank you very much?</p>
</blockquote>
</aside>
<p>this is asking about onnx perf? maybe open an issue in onnx repo? <a href="https://github.com/onnx/onnx/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · onnx/onnx · GitHub</a></p>
Trainable sorting order?

bartley: <p>Is it possible to have a trainable sorting layer in Pytorch?</p>
<p>The idea is that given 8 vectors, I want the layer to learn the best order before applying <code>torch.cat()</code>. See the minimal example I created below.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

# Current implementation
x= [torch.rand(1,32) for i in range(8)]
x_out= torch.cat(x, dim=0)
display(x_out.shape)

# Something like this?
sort_layer = nn.Parameter(torch.rand(8))
sort_idxs= torch.argsort(sort_layer)
x_out = torch.cat([x[i] for i in sort_idxs])
display(x_out.shape)
</code></pre>
<p>In my current implementation, the order is fixed. I tried using the <code>nn.Parameter()</code> with <code>torch.argsort()</code> but the parameter values are not updated during training. Any ideas how this could be done?</p>

KFrank: <p>Hi Brendan!</p>
<aside class="quote no-group" data-username="bartley" data-post="1" data-topic="199341" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bartley/48/60569_2.png" class="avatar"> bartley:</div>
<blockquote>
<p>Is it possible to have a trainable sorting layer in Pytorch?</p>
</blockquote>
</aside>
<p>Not really (not in the context of using gradient-descent optimization as is<br>
typical with pytorch).</p>
<p>The problem is that “order” isn’t (usefully) differentiable.  Let’s say that you<br>
start out with order 1, 2, 3 …, and then want to move to order 2, 1, 3 …<br>
This would lead to a discontinuous jump in your loss function, which would<br>
therefore be piece-wise constant, which is not (usefully) differentiable.</p>
<aside class="quote no-group">
<blockquote>
<p>I tried using the <code>nn.Parameter()</code> with <code>torch.argsort()</code> but the parameter values are not updated during training.</p>
</blockquote>
</aside>
<p>You can’t backpropagate through <code>argsort()</code> (because, as discussed above,<br>
“order” is not differentiable).  Note that <code>argsort()</code> returns a tensor with<br>
<code>dtype = torch.int64</code> and such tensors never carry <code>requires_grad = True</code><br>
(because integers are not usefully differentiable).</p>
<p>(Hypothetically, you could have trainable weights that somehow “blend”<br>
together your various orderings, but in your example you would have<br>
<code>factorial (8) = 40320</code> possible orderings which would be impractically<br>
large.)</p>
<p>Best.</p>
<p>K. Frank</p>
Following ops cannot be found: [aten::scaled_dot_product_attention]

tanghao: <p>Hello，I want to deploy the clip model to an Android phone and encountered a problem. Can anyone help me to solve this problem?</p>
<p>first save the model into ‘clip_model.pt’ file;</p>
<pre><code class="lang-python">import torch
import clip
from PIL import Image
from tensorboardX import SummaryWriter
from torch.utils.mobile_optimizer import optimize_for_mobile

model, preprocess = clip.load("ViT-B/32")

model = list(model.children())[0]

image_path = "./image.jpg";
image = preprocess(Image.open(image_path)).unsqueeze(0)

model.eval()
traced_script_model = torch.jit.trace(model, image)
optimized_script_model = optimize_for_mobile(traced_script_model)
optimized_script_model._save_for_lite_interpreter('./clip_model.pt')
</code></pre>
<p>secondly, load the model in android app:</p>
<pre><code class="lang-java">module = LiteModuleLoader.load(assetFilePath(this, "clip_model.pt"));
</code></pre>
<p>the build.gradle file of the android project:</p>
<pre><code class="lang-plaintext">dependencies {
    implementation 'androidx.appcompat:appcompat:1.1.0'
    implementation 'org.pytorch:pytorch_android_lite:1.13.0'
    implementation 'org.pytorch:pytorch_android_torchvision:1.9.0'
}
</code></pre>
<p>the runtime exception:</p>
<pre><code class="lang-plaintext">com.facebook.jni.CppException: Following ops cannot be found: [aten::scaled_dot_product_attention]. Please check if the operator library is included in the build. If built with selected ops, check if these ops are in the list. If you are a Meta employee, please see fburl.com/missing_ops for a fix. Or post it in https://discuss.pytorch.org/c/mobile/ ()
Exception raised from initialize_operators at /home/agunapal/pytorch/torch/csrc/jit/mobile/function.cpp:90 (most recent call first):
(no backtrace available)
</code></pre>

Th_nh_Vu: <p>Maybe it is a little late, but for anyone encounter this error, updating “pytorch_android_lite” to 2.1.0 worked.</p>
Which formula is used to calculate the derivative of torch.where/torch.nonzero

Max_Unhold: <p>Hey,<br>
I would like to know the formula behind the <code>torch.where</code> and <code>torch.nonzero</code> functions<br>
that is used for backpropagation.</p>
<p>Maybe, you even have a general formula for similiar discontinous operations.</p>

ptrblck: <p>You can find the derivative for <code>torch.where</code> <a href="https://github.com/pytorch/pytorch/blob/ed15370aabf951eec2ba0140de5ff71634868791/tools/autograd/derivatives.yaml#L1890-L1894">here</a> and for <code>torch.nonzero</code> <a href="https://github.com/pytorch/pytorch/blob/ed15370aabf951eec2ba0140de5ff71634868791/tools/autograd/derivatives.yaml#L2907-L2908">here</a>, which is not differentiable as its returning indices.</p>
Testing of Inception v3

alihamza: <p>I used inception v3 pretained model for a dataset. in training i used<br>
outputs, secondOut = model(images)<br>
loss1 = criterion(outputs, labels)<br>
loss2 = criterion(secondOut,labels)<br>
loss = loss1 + loss2 * 0.4<br>
training accuracy reached upto 98% and validation is reached upto 90%<br>
but when i test the saved model which i got after training, testing accuracy is just 2% can anyone tell me the reason</p>
<p>This is testing code<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
model = model.to(device)</p>
<h1><a name="set-the-model-to-evaluation-mode-1" class="anchor" href="#set-the-model-to-evaluation-mode-1"></a>Set the model to evaluation mode</h1>
<p>model.eval()</p>
<h1><a name="initialize-variables-for-tp-tn-fp-fn-2" class="anchor" href="#initialize-variables-for-tp-tn-fp-fn-2"></a>Initialize variables for TP, TN, FP, FN</h1>
<p>TP = TN = FP = FN = 0<br>
FP_sum = FN_sum = 0</p>
<h1><a name="initialize-the-prediction-and-target-arrays-3" class="anchor" href="#initialize-the-prediction-and-target-arrays-3"></a>Initialize the prediction and target arrays</h1>
<p>all_predictions = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
all_targets = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<h1><a name="iterate-over-the-test-dataset-and-calculate-tp-tn-fp-fn-4" class="anchor" href="#iterate-over-the-test-dataset-and-calculate-tp-tn-fp-fn-4"></a>Iterate over the test dataset and calculate TP, TN, FP, FN</h1>
<p>with torch.no_grad():<br>
for images, labels in test_loader:<br>
images = images.to(device)<br>
labels = labels.to(device)</p>
<pre><code>    outputs = model(images)
    _, predicted = torch.max(outputs, 1)
    # Convert labels and predictions to CPU
    labels_cpu = labels.cpu().numpy()
    predicted_cpu = predicted.cpu().numpy()

    cm = confusion_matrix(labels_cpu, predicted_cpu, labels=range(num_classes_test))
    TP += np.sum(np.diag(cm))
    # Append predictions and targets to the arrays
    all_predictions.extend(predicted_cpu)
    all_targets.extend(labels_cpu)
</code></pre>
<p>print(f’True Positives: {TP}')</p>
Copy model weights between processes

dtsaras: <p>Hello,</p>
<p>I have a few processes running on the CPU that generate data and one process (potentially more in the future) where I consume the produced data to train my model. The issue is that when all the processes are on the CPU (including the training process) the loss drops, but when the model is on the GPU (MPS specifically), the loss does not drop. What am I doing wrong? Below is the logic of my code:</p>
<pre data-code-wrap="python"><code class="lang-python">model = MyModel()
model.share_memory() # So that all processes share the same weights
start_processes_collect_data(model)
train_model = copy.deepcopy(model)
train_model = accelerator.prepare(train_model) # I use the hugging face accelerator to set the device
start_training(train_agent)
model.load_state_dict(train_model.state_dict())
</code></pre>
<p>Thanks in advance</p>
Affine_grid and grid_sample with scale factor biggern than 1 makes image smaller

MatrixNoob: <p>hi guys,<br>
this is probably a stupid question, but I cannot figure out following problem:<br>
I’m currently trying to understand why my written method returns a smaller image when the scale factor is bigger than 1. Usually a larger scale factor makes the output bigger. But with grid_sample it the way around. I’m not sure if I understand the affine_grid function correctly.</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms.functional as TF
import torch.nn.functional as F

from PIL import Image

def transform_2D(img, angle: float, translate_x: float = 0, translate_y: float = 0, scale_x: float = 1, scale_y: float = 1):
    img_tensor = TF.to_tensor(img).unsqueeze(0)
    _, channels, height, width = img_tensor.size()

    angle = torch.tensor(angle)
    # in pixels
    tx = 2*(translate_x/width)
    ty = 2*(translate_y/height)
    
    theta = torch.deg2rad(angle)   # Rotation angle in radians
    sx = torch.tensor(scale_x)     # Scaling factor along x-axis
    sy = torch.tensor(scale_y)     # Scaling factor along y-axis
    dx = torch.tensor(tx)          # Translation along x-axis
    dy = torch.tensor(ty)          # Translation along y-axis
    
    # Define affine matrix
    affine_matrix = torch.tensor([
            [sx * torch.cos(theta), -torch.sin(theta), dx],
            [torch.sin(theta),  sy * torch.cos(theta), dy]])

    # Generate grid
    grid = F.affine_grid(affine_matrix.unsqueeze(0), img_tensor.size(), align_corners=True)
    # Apply affine transformation using grid_sample with border control
    transformed_image = F.grid_sample(img_tensor, grid, padding_mode='border')
    return transformed_image
</code></pre>
<p>usage:</p>
<pre><code class="lang-auto">transformed_image = transform_2D(img=image,
                                 angle=0,
                                 translate_x=0,
                                 translate_y=0,
                                 scale_x=1,
                                 scale_y=1
                                )
pil_image = TF.to_pil_image(transformed_image.squeeze(0))
display(pil_image)
</code></pre>
Torch.matmul gives runtime error

CSK: <p>I  have pasted my code snippet here:<br>
B, L, H, E = queries.shape<br>
_, S, _, D = values.shape<br>
scale = self.scale or 1. / sqrt(E)</p>
<pre><code>    # scores = torch.einsum("blhe,bshe-&gt;bhls", queries, keys)
    scores = torch.matmul(queries, keys)
</code></pre>
<p>I am trying to use matmul instead of einsum, but I get the following error<br>
scores = torch.matmul(keys,queries)<br>
RuntimeError: Expected batch2_sizes[0] == bs &amp;&amp; batch2_sizes[1] == contraction_size to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
<p>Shapes of the arguments are as follows:<br>
shape of queries is torch.Size([1024, 6, 8, 4])<br>
shape of keys is torch.Size([1024, 6, 8, 4])</p>
<p>Could you please help me with this error. I know the my matrix sizes are incompatible for matrix multiplication, but could you please help me how do I use permute or transpose so that my final shape of scores is ([1024, 8, 6, 6])</p>
How Do I Obtain GROUP_RANK and LOCAL_WORLD_SIZE in Code?

yang_xiaorun: <p>When I start DDP with torchrun, I can get this information through environment variables.</p>
<pre><code class="lang-auto">print("env GROUP_RANK", os.environ["GROUP_RANK"])
print("env LOCAL_WORLD_SIZE", os.environ["LOCAL_WORLD_SIZE"])
</code></pre>
<p>But I can’t assume that my code always works in the torchrun scenario, such as the torch.multiprocessing.spawn scenario where I can’t get these two environment variables. So I wonder if there’s an API like this <code>torch.distributed.get_global_rank </code> that can help me get both of these information steadily. If not, should we provide a new one?</p>

yang_xiaorun: <p>I found that my question had been raised before <code>https://stackoverflow.com/questions/61021029/get-local-world-size-in-torch-distributed-training</code>.But it seems that the problem has not been solved head-on.</p>

Ankur_Mahesh: <p>One option could be to use:</p>
<p><code>len(torch.distributed.get_process_group_ranks(group))</code></p>
<p>Would there be something wrong with that solution?</p>
In torch.nn.functional.embedding, why does padding_idx exist?

bdusell: <p>I have a question about using padding indexes with input embedding layers. For context, suppose I am training a causally masked transformer language model, where sequences are always left-aligned in a batch, with padding on the right.</p>
<p>In <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.embedding.html#torch-nn-functional-embedding" rel="noopener nofollow ugc">torch.nn.functional.embedding</a>, <strong>why does the argument <code>padding_idx</code> exist</strong>? As best I can tell, it’s unnecessary. Its functionality is to (1) always map <code>padding_idx</code> to the 0 vector and (2) ensure that its embedding always has a gradient of 0. But if the output is properly masked (e.g., with <code>ignore_index</code> in <code>torch.nn.functional.cross_entropy</code> ), then the output of the model at that timestep will be ignored, and the parameters used in computation at that timestep, including the embedding, will receive 0 gradient. So you could just use an arbitrary index for padding, say 0, and avoid having spurious embedding parameters for the padding symbol. I wrote a unit test to verify that the behavior is the same whether you (1) set <code>padding_idx</code> to a unique value, or (2) set it to <code>None</code> and use 0 in padded positions, and the results are the same after optimizing for a few steps with Adam.</p>
<p>Is there any benefit to using <code>padding_idx</code> vs. the alternative I described? A downside of using <code>padding_idx</code> is that it forces you to have a spurious embedding vector for the padding symbol, which is not used for useful computation and not truly a parameter of the model.</p>

vdw: <p>Firstly, you might have a dataset from “somewhere else” where the index of the padding token is not 0.</p>
<p>Secondly, not all training setups support the masking of the loss. For example, say you want to train an RNN/LSTM/GRU-based sentiment classifier. If you have batches with sequences of different lengths that need padding – and you don’t use packing – you want to treat the padding as a special word the classifier will hopefully learn to ignore.</p>
<p>In any case, since you train a classifier, i.e., a many-to-one model, you can not mask the passing tokens when calculating the loss between the output and the ground-truth class labels.</p>
<p>That being said, if your setup allows for an alternative solution, you can naturally go for it :).</p>
LSTM input dimensions for batch size padding

Youwumbo: <p>I’m doing a sentiment analysis project with a large dataset of tweets. I didn’t want to pad my data to the largest tweet in the entire dataset, so I padded my data to the largest in each batch instead. But the dimensions across batches are obviously not the same so I can’t figure out what to do with my model’s input dimensions.</p>
<p>Are dynamic input dimensions even possible, or do I need to do either padding to the largest data point in the dataset or some other kind of padding?</p>

vdw: <p>Different sequence lengths <strong>across</strong> different batches are not a problem.</p>
<p>It only matters that the sequences in the <strong>same</strong> batch have the same length.</p>
<p>The model dimension is completely independent from the sequence length. Your LSTM will always go through the whole sequences in a batch, no matter the length. You should have noticed that <code>nn.LSTM</code> offers no parameter to set a sequence length.</p>
How is explicit `pin_memory` different from just calling `.to` and let CUDA handle it?

yaolr: <p>My question is which of these two is faster and why?</p>
<ol>
<li><code>t.to("cuda")</code></li>
<li><code>t.pin_memory().to("cuda")</code></li>
</ol>
<p>I have this question because:</p>
<p>This <a href="https://developer.nvidia.com/blog/how-optimize-data-transfers-cuda-cc/" rel="noopener nofollow ugc">CUDA blog</a> is referenced a lot in related discussion and it says:</p>
<blockquote>
<p>the CUDA driver must first allocate a temporary page-locked, or “pinned”, host array, copy the host data to the pinned array, and then transfer the data from the pinned array to device memory</p>
</blockquote>
<p>It reads to me, CUDA will do a <code>pin_memory</code> operation itself if the data is not already pinned. I understand <code>pin_memory</code> enables us to set the async flag <code>non_blocking=True</code>. However, if instead of using <code>.to("cuda", non_blocking=True)</code>, we use <code>.to("cuda")</code>, why does explicitly calling <code>torch.tensor.pin_memory()</code> make the data transfer faster than without such an explicit call?</p>

ptrblck: <p>Using pinned memory allows you to asynchronously transfer the data. This might be beneficial if you are running other workloads between the transfer and the actual data usage. However, if the next call would need to consume the data you wouldn’t expect to see any benefits from the async copy.<br>
You can also let the <code>DataLoader</code> pin the memory and could move it inside the training loop, which is a common use case.</p>

yaolr: <p>Thank you for your prompt reply, Piotr!</p>
<p>Is it then correct to conclude these two commands are equal in speed</p>
<ol>
<li><code>t.to("cuda", non_blocking=False)</code></li>
<li><code>t.pin_memory().to("cuda", non_blocking=False)</code></li>
</ol>
<p>and <code>pin_memory</code> should always be used with <code>non_blocking=True</code>?</p>

ptrblck: <p>You should profile the code, but the second one might be a bit faster (while still not async) as the staging buffer copy would be avoided.</p>
Memory management in libtorch

sroyyors: <p>hi,<br>
I was wondering how memory allocation is working in libtorch. I think with clone one is allocating new memory, but when one is doing say</p>
<pre><code class="lang-auto">torch::Tensor y=torch::randn({4,5});
torch::Tensor x=y;
</code></pre>
<p>Is this setting x to y’s address or making a copy? I guess I can try some code to check this, but I was wondering in general if there is any documentation of how libtorch manages memory.</p>

ptrblck: <p><code>x</code> should be a reference to <code>y</code> and you can verify it by manipulating <code>x</code> inplace and see the same changes in <code>y</code>.</p>

sroyyors: <p>thank you, I tried out some example codes as well also with function calls. It seems in most cases it is passed by reference.</p>
How to conduct graph classification where nodes belong to multiple graphs

evantianzg: <p>Hi everyone,</p>
<p>I am using GNNs to perform graph classification tasks. I first used the public TUDatasets for graph classification, and it works well. Then I tried to conduct experiments with my own datasets. However, I found that there is a “graph_indicator” file in the public TUDatasets (like MUTAG). After carefully reading the description and content of this “graph_indicator” file, I suddenly realized the difference between my dataset and the public dataset: <strong>Each node on public datasets belongs to a single graph, while in my dataset, some nodes belong to multiple graphs.</strong> Therefore, I cannot create a similar “graph_indicator” file as in the public datasets.</p>
<p><strong>So my question is: how should I do for graph classification where nodes belong to multiple graphs?</strong></p>
<p>Any help or suggestions are appreciated.</p>
DataLoader returns the batch as a list

zqma: <p>When applying the trained model to the actual data, my dataset only has X but labels. I created a dataset class and then feed the dataset to a dataloader. However, the iter on the dataloader returns a list with the batch tensor as the only entry. I expect the return to be a tensor. Where did I do wrong? Below is the sample code.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, dataset):
        self.dataset = dataset
    def __getitem__(self,index):
        data = self.dataset[index]
        return data
    def __len__(self):
        return len(self.dataset)
data_tensor = torch.tensor([[1, 2, 3], [1, 2, 3], [2, 3, 1], [1, 2, 3]])
data_set = MyDataset(TensorDataset(data_tensor))
data_loader = torch.utils.data.DataLoader(data_set, batch_size=2, shuffle=False)
next(iter(data_loader))
</code></pre>
<p>Here is the result</p>
<pre><code class="lang-auto">[tensor([[1, 2, 3],
         [1, 2, 3]])]
</code></pre>
<p>I tested adding a label seq to the dataset class with the X by using <code>TensorDataset(X, y)</code> and then the return is a batch tensor as expected. Wondering what’s the best practice to create the dataloader when the data don’t contain labels.</p>
<p>I found someone has the same question on <a href="https://stackoverflow.com/questions/58612401/pytorch-dataloader-returns-the-batch-as-a-list-with-the-batch-as-the-only-entry" rel="nofollow noopener">stackoverflow</a></p>

JuanFMontesinos: <p>Check sourcecode</p>
<pre><code class="lang-auto">[docs]class TensorDataset(Dataset):
    r"""Dataset wrapping tensors.

    Each sample will be retrieved by indexing tensors along the first dimension.

    Arguments:
        *tensors (Tensor): tensors that have the same size of the first dimension.
    """

    def __init__(self, *tensors):
        assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)
        self.tensors = tensors

    def __getitem__(self, index):
        return tuple(tensor[index] for tensor in self.tensors)

    def __len__(self):
        return self.tensors[0].size(0)
</code></pre>
<p>Why do you wrap tensors with a tensordataset?<br>
I would say this is expected as getitem returns tuples</p>

zqma: <p>thank you!<br>
i followed some tutorial where the tensor was wrapped in a dataset. Using the tensor directly in the dataloader works.</p>

Nils_Smitham86: <p>use your collect_fn in dataloader to return Tensor(of tensor), instead or list of tensor<br>
look at <a href="https://discuss.pytorch.org/t/looping-over-dataloader-returns-list/172407/3" class="inline-onebox">Looping over DataLoader Returns List? - #3 by Imahn</a></p>
Problems with num_workers in dataloader

NeedS0meRest: <p>Hello! I am currently working on an image segmentation problem. I have an issue with dataloaders. When I set num_workers=12, dataloader utilizes 70-90% of cpu for a little while, and then utilization drops, with only one thread working (as I can see in htop). What may cause this issue?</p>

ptrblck: <p>Maybe the workers preloaded enough batches and filled the queue. You can check this by adding debug <code>print</code> statements to the data loading pipeline or also by changing the <code>prefetch_factor</code>.</p>
Is CTC loss badly defined?

Peter_Featherstone: <p>According to CTC loss documentation, the target sequence length must be &lt;= to input sequence length. This allows the case where target sequence length == input length.<br>
But what if your target is constant? Then CTC needs to predict interleaved blanks. In the pessimistic case where target is constant and let’s say equal to N, then model needs to predict N blanks. So really, target sequence length must be &lt;= half input sequence length right?</p>
<p>Also, is there a better alternative to unaligned, unsegmented sequence prediction?</p>

Peter_Featherstone: <p>In fact, if the number of consecutive items within a target, which equals to the number of “blanks” the model needs to predict, plus the size of the target, exceed the model output size, then you get either INF or NAN in CTC. So a sufficient condition for CTC to work always is to to have input_length &gt;= 2 * target_length</p>

kuraga: <p><a class="mention" href="/u/peter_featherstone">@Peter_Featherstone</a> , have you figured out with it?<br>
An issue has been created?</p>
Does pooling always raise the accuracy of a CNN?

NZ1: <p>Pooling is alway presented an an accepted part of a CNN architecture. However, are there any cases where pooling might be detrimental to the performance? The obvious advantages to pooling are usually presented as minimising data, but in cases where memory is no problem, might there be advantages not to pool?</p>

ptrblck: <p>I don’t think the memory and compute reductions are the only advantage of using pooling layers and would claim reducing the dimensionality of the intermediate activations should also help the classifier generalize. However, my point of view might come from a more “classical” ML view, so I would be interested to hear from others, too.</p>

NZ1: <p>Well many thanks for your view, but not much else coming through on this one. I guess people just follow the tutorials without really asking too many questions. I’ll continue to pursue and will post when i have more information. Cheers, NZ1</p>
A bug in torch.rand() with parallel random generators

hammoudeh: <p>Hi,</p>
<p>The last 10 minutes of this video demonstrates a problem in torch.rand(), I thought it could be helpful to share it with the developers of torch. Plase let me know if the problem was already fixed. it seems wrapping torch.rand() using “random” instead of “numpy” could solve the bug.</p>
<div class="youtube-onebox lazy-video-container" data-video-id="6StU6UtZEbU" data-video-title="Lesson 10: Deep Learning Foundations to Stable Diffusion, 2022" data-video-start-time="6238" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=6StU6UtZEbU&amp;t=6238" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/6StU6UtZEbU/maxresdefault.jpg" title="Lesson 10: Deep Learning Foundations to Stable Diffusion, 2022" width="690" height="388">
  </a>
</div>
nn.Parameter() and register_buffer moving to GPU

akt42: <p>In this simplified example, I want <code>alpha</code> to be learnable and <code>pe</code> to be constant. Below is my model:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Model(nn.Module):
    
    def __init__(self):
        super().__init__()
        
        self.alpha = nn.Parameter(torch.ones([10]))

        pe = torch.arange(10)
        self.register_buffer("pe", pe)
            
    def forward(self, x):
        x = x + self.alpha * self.pe
        return x

model = Model()
</code></pre>
<p>The below operation on cpu works fine:</p>
<pre><code class="lang-auto">ip = torch.rand(10)
model(ip)
</code></pre>
<p>When I run this on GPU, I get the following error:</p>
<pre><code class="lang-auto">ip.to('cuda')
model.to('cuda')
model(ip)
</code></pre>
<pre><code class="lang-auto">/tmp/ipykernel_1284080/2457928180.py in forward(self, x)
     10 
     11     def forward(self, x):
---&gt; 12         x = x + self.alpha * self.pe
     13         return x

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
</code></pre>
<p>What am I doing wrong here?</p>

ptrblck: <p>You need to reassign the tensor if you move it to another device:</p>
<pre data-code-wrap="python"><code class="lang-python">ip = ip.to("cuda")
</code></pre>

akt42: <p>For anyone that stumbles on a similar question in future: I was sleepy while writing the question and the original problem I was working was a bit complicated so I wasn’t able to simplify that to a simple example as expected.</p>
<p>What I wanted was to make both <code>alpha</code> and <code>pe</code>, learnable. And this was giving me the above error:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    
    def __init__(self):
        super().__init__()
        
        self.alpha = nn.Parameter(torch.ones([10]))

        self.pe = torch.arange(10)
   
    def forward(self, x):
        x = x + self.alpha * self.pe
        return x
</code></pre>
<p>The issue in this model is that <code>self.pe</code> is not defined as a learnable tensor.</p>
<p>The fix is:</p>
<pre><code class="lang-auto">self.pe = nn.Parameter(torch.arange(10).to(torch.float32))
</code></pre>
Hierarchial Neural Network

Govind_Hrishikesh: <p>I need to build a reccomendation system does it need hierarchial neural network or not?<br>
Like:<br>
Segement is the main node it should have product and user link and also user have link to product.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12f6f26f7a072a6f664599889ba6c9045d0a3789.png" data-download-href="https://discuss.pytorch.org/uploads/default/12f6f26f7a072a6f664599889ba6c9045d0a3789" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12f6f26f7a072a6f664599889ba6c9045d0a3789_2_690x353.png" alt="image" data-base62-sha1="2HLFUyMfAWkVu1Wva7CLwzfZZmp" width="690" height="353" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12f6f26f7a072a6f664599889ba6c9045d0a3789_2_690x353.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12f6f26f7a072a6f664599889ba6c9045d0a3789_2_1035x529.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/2/12f6f26f7a072a6f664599889ba6c9045d0a3789.png 2x" data-dominant-color="DFE8EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1235×632 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>