akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why does ".backward(retain_graph=True)" gives different values each time is called?

username: pablorf
cooked: <p>Starting with a simple example from <a href="https://medium.com/@zhang_yang/the-gradient-argument-in-pytorchs-backward-function-explained-by-examples-68f266950c29" rel="nofollow noopener">here</a>.</p>
<pre><code class="lang-auto">from torch import tensor,empty,zeros
x = tensor([1., 2.], requires_grad=True)
y = empty(3)
y[0] = 3*x[0]**2
y[1] = x[0]**2 + 2*x[1]**3
y[2] = 10*x[1]
</code></pre>
<p>This is a 2 input, 3 outputs model. I’m interested in getting the full Jacobian matrix. To do that, I was thinking:</p>
<pre><code class="lang-auto">J = zeros((y.shape[0],x.shape[0]))
for i in range(y.shape[0]):
	xU = zeros(y.shape[0])
	xU[i] = 1.0
	y.backward(xU,retain_graph=True)
	J[i,:] = x.grad
</code></pre>
<p>However, the output of this is:</p>
<pre><code class="lang-auto">tensor([[ 6.,  0.],
        [ 8., 24.],
        [ 8., 34.]])
</code></pre>
<p>while it should be:</p>
<pre><code class="lang-auto">tensor([[ 6.,  0.],
        [ 2., 24.],
        [ 0, 10.]])
</code></pre>
<p>Trying to debug what’s going on, I found out the following:</p>
<pre><code class="lang-auto">y[0].backward(retain_graph=True); print(x.grad)
y[0].backward(retain_graph=True); print(x.grad)
</code></pre>
<p>gives:</p>
<pre><code class="lang-auto">tensor([6., 0.])
tensor([12.,  0.])
</code></pre>
<p>Can someone explain what is going on under the hood with the call to “.backward(retain_graph=True)”? Why does it give different values when called twice?</p>
<p>I think that’s the key to why my calculation of the Jacobian matrix is not correct.<br>
Thanks</p>

username: ttoosi
cooked: <p>Hi,</p>
<p>Whenever you call backward, it accumulates gradients on parameters. That’s why you call optimizer.zero_grad() before calling loss.backward(). Here, it’s the same thing happening. By the second call to backward you add more gradients on top of the previously added gradients by the first call.</p>

username: albanD
cooked: <p>Another way to solve this is to use <code>autograd.grad()</code> that returns the gradient instead of accumulating them.</p>

username: slobodaapl
cooked: <p>How would one use said returned gradient from <code>autograd.grad()</code> for the backward function?</p>

username: albanD
cooked: <p>I’m not sure to understand your question? This function returns the gradients instead of populating the .grad field on Tensors as is done by .backward().</p>

username: slobodaapl
cooked: <p>Ah I understand now, I was misreading and thinking incorrectly about your explanation, that is my bad. It is clear now.</p>
Clear GPU memory between successive runs (inference)

username: vsant
cooked: <p>Hello,</p>
<p>Let us assume the following runs:</p>
<p><code>python inference.py -testset A</code></p>
<p><code>python inference.py -testset B</code></p>
<p><code>python inference.py -testset C</code></p>
<p>…</p>
<p>In other words, it is the inference phase, the best model has already been saved and I am just evaluating with different test sets. Is it advisable to clear the GPU memory between two runs? Or every new run is a “fresh” start in terms of GPU memory?</p>

username: JuanFMontesinos
cooked: <p>Not an expert about this but in principle pytorch manages the memory efficiently.<br>
Moreover if you run different python scripts, the memory should be 100% released after each process finishes. Maybe if you run daemon subprocesses in c++ libraries from python, i.e., openpose or DALI, these processes can get stuck and not release the memory. But that’s more an exception.</p>

username: vsant
cooked: <p>Thanks Juan. I added all calls/runs into a single shell script.</p>
The expanded size of the tensor (8) must match the existing size (9) at non-singleton dimension 3.  Target sizes: [2, 8, 8, 8].  Tensor sizes: [2, 1, 9, 9]

username: teddylee7561
cooked: <p>help me plz ~! i was trying to train swintransformer/OCR but i I got an error</p>
<p>in train dataset image folder has 10 of .“png” photos and 8 of train.txt   and 2 of val.txt</p>
<p>"The expanded size of the tensor (8) must match the existing size (9) at non-singleton dimension 3.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95a865700e5aa48031440fcdb5c79e21a0c4cbab.png" data-download-href="https://discuss.pytorch.org/uploads/default/95a865700e5aa48031440fcdb5c79e21a0c4cbab" title="화면 캡처 2022-11-10 193025"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95a865700e5aa48031440fcdb5c79e21a0c4cbab_2_690x337.png" alt="화면 캡처 2022-11-10 193025" data-base62-sha1="llVVGnugeQJnC6PPzptHSix42Jd" width="690" height="337" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95a865700e5aa48031440fcdb5c79e21a0c4cbab_2_690x337.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95a865700e5aa48031440fcdb5c79e21a0c4cbab_2_1035x505.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95a865700e5aa48031440fcdb5c79e21a0c4cbab_2_1380x674.png 2x" data-dominant-color="272727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">화면 캡처 2022-11-10 193025</span><span class="informations">1757×859 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/517a73d82a4ca7fb12b535c6dd7ad6487eed2ab3.png" data-download-href="https://discuss.pytorch.org/uploads/default/517a73d82a4ca7fb12b535c6dd7ad6487eed2ab3" title="화면 캡처 2022-11-10 193149"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/517a73d82a4ca7fb12b535c6dd7ad6487eed2ab3_2_690x374.png" alt="화면 캡처 2022-11-10 193149" data-base62-sha1="bCN1LdXruGwxfetDAREK6IAOXcL" width="690" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/517a73d82a4ca7fb12b535c6dd7ad6487eed2ab3_2_690x374.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/517a73d82a4ca7fb12b535c6dd7ad6487eed2ab3_2_1035x561.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/517a73d82a4ca7fb12b535c6dd7ad6487eed2ab3_2_1380x748.png 2x" data-dominant-color="545656"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">화면 캡처 2022-11-10 193149</span><span class="informations">1590×862 156 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Torch AMP autocast and recompute

username: ralzq01
cooked: <p>Hi, I would like to understand how <code>torch.autocast()</code> combines with recompute (torch.utils.checkpoint).<br>
In official documents, <code>loss.backward()</code> is outside regions of <code>with torch.autocast(xxx):</code>, where the re-forward will happen during backward without auto-casting. So actually, it will still use all fp32 instead of fp16 as well as gradient computation, is my understanding correct?</p>
Modify torchvision.models resnet50 layer using hooks

username: Ashish_Gupta1
cooked: <p>I am using resnet50. I have to introduce an additional model to resnet50, and the new model uses 3d Conv. Hence I have to first modify the intermediate layer input by reshaping and then pass it to the new model and reshape its output and pass it to further layers of resnet50. Hence I decided to use <strong>hooks</strong>. Should I modify it like below code stamp? Does the PyTorch backprop work fine as usual? It will automatically compute the gradient for <code>NEWMODEL</code> by considering it as a part of the older model.</p>
<pre><code class="lang-auto">def new_model_insertion(name):
    def hook(model, input, output):     
        return  NEWMODEL(output)
    return hook
</code></pre>
<p>Thanks a lot.</p>
Expected 3D (unbatched) or 4D (batched) input to conv_transpose2d, but got input of size: [64, 128]

username: jurisdiction0080
cooked: <p>Dear all<br>
I got an error message<br>
<strong>‘Expected 3D (unbatched) or 4D (batched) input to conv_transpose2d, but got input of size: [64, 128]’</strong><br>
while I coded the following autoencoder model:<br>
I know there are many similar question that had been answered, but I am sorry that I still could find out the error key point</p>
<p>def <strong>init</strong>(self, latent_dim_value=128):<br>
super(Net, self).<strong>init</strong>()<br>
self.latent_value = latent_value<br>
self.img_size = 32</p>
<pre><code>    self.encoder = nn.Sequential(
        nn.Conv2d(3 ,32,  3, stride=2, padding=1),
        nn.ReLU(True),
        nn.MaxPool2d(2),

        nn.Conv2d(32, 64, 3, stride=2, padding=1),
        nn.ReLU(True),
        nn.MaxPool2d(2),

        nn.Conv2d(64, 128, 3, stride=2, padding=1),
        nn.ReLU(True),
    )
    
    self.fc_layer1 = nn.Sequential(
        nn.Linear(128, self.latent_dim),
        nn.PReLU(),)
    self.fc_layer2 = nn.Sequential(
        nn.Linear(self.latent_dim,128),
        nn.PReLU(),)

    self.decoder = nn.Sequential(
        nn.ConvTranspose2d(128, 64, 5 , 2, 1),
        nn.ReLU(True),
        nn.ConvTranspose2d(64, 32, 9, 2 , 1),
        nn.ReLU(True),
        nn.ConvTranspose2d(32, 256 ,17 , 2, 1),
        nn.Tanh()
    )
            
def forward(self, x):
    feature_map = self.encoder(x)
    y = self.fc_layer1(feature_map.reshape(feature_map.shape[0], -1))
    z = self.fc_layer2(y)
   Residual_x= self.decoder(z)
    
    return y, Residual_x
</code></pre>

username: ptrblck
cooked: <p>The <code>nn.ConvTranspose2d</code> layer expects a 4-dimensional tensor by default in the shape <code>[batch_size, channels, height, width]</code> or a 3-dimensional tensor if you skip the batch dimension.<br>
In your code it seems the input tensor <code>z</code> has 2 dimensions only, which is incompatible.</p>
Multi node, multi gpu system

username: shrutishrestha
cooked: <p>Hi, I am trying to use multi gpu while running my code. I tried with the code provided in pytorch documentation, but its not working.<br>
Could anyone please look at this once?<br>
The thing is I was able to run program in multiple gpu multiple node, using distributed data parallel. But the gradients were not collected as the accuracy and loss are all zero after the first epoch.</p>
<p>Full code is presented here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/shrutishrestha/DeepLearning_with_Multinode_Multigpu/blob/main/DistributedDataParallel.py">
  <header class="source">

      <a href="https://github.com/shrutishrestha/DeepLearning_with_Multinode_Multigpu/blob/main/DistributedDataParallel.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/shrutishrestha/DeepLearning_with_Multinode_Multigpu/blob/main/DistributedDataParallel.py" target="_blank" rel="noopener nofollow ugc">shrutishrestha/DeepLearning_with_Multinode_Multigpu/blob/main/DistributedDataParallel.py</a></h4>


      <pre><code class="lang-py">import os
import csv
import cv2
import torch
from datetime import datetime
import numpy as np
import pandas as pd
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader

import cProfile, pstats
from sklearn.metrics import accuracy_score

import warnings
warnings.filterwarnings("ignore")

#for neural network
from torchvision import datasets, transforms

</code></pre>



  This file has been truncated. <a href="https://github.com/shrutishrestha/DeepLearning_with_Multinode_Multigpu/blob/main/DistributedDataParallel.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: wanchaol
cooked: <p>HI <a class="mention" href="/u/shrutishrestha">@shrutishrestha</a> I looked into your code, if you want to use nccl pg, better set up the cuda device of each process to local rank via <code>torch.cuda.set_device</code></p>

username: shrutishrestha
cooked: <p>Thanks for your reply <a class="mention" href="/u/wanchaol">@wanchaol</a>, but when I tried to use nccl, it was giving me error, so I used gloo for this</p>

username: fduwjj
cooked: <aside class="quote no-group" data-username="shrutishrestha" data-post="1" data-topic="165204">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shrutishrestha/48/51287_2.png" class="avatar"> shrutishrestha:</div>
<blockquote>
<p>able to run program in multiple gpu multiple node, using distributed data parallel. But the gradients were not collected as the accuracy and loss are all zero after the first epoch.</p>
<p>Full code is presen</p>
</blockquote>
</aside>
<p>What environment are you running your code? Because Gloo is for CPU training.</p>

username: shrutishrestha
cooked: <p>I am running my code in GPU. I will check if nccl works for GPU.</p>
Deploying models to Google App Engine Flexible

username: JeffO
cooked: <p>I’m trying to deploy a Torch/Transformers model on Google App Engine Flexible.  During deploy, I get this error:</p>
<pre><code class="lang-auto">Updating service [default] (this may take several minutes)...failed.                                                                      
ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting the 'app_start_timeout_sec' setting in the 'readiness_check' section.
</code></pre>
<p>I’ve done the recommended change and also cranked up the instance memory to 10GB but I still get the same error.  I’m copying my <code>app.yaml</code> below.</p>
<p>Has anyone successfully deployed a model like this to GAE flex?</p>
<p>I’m able to deploy to AWS but need it on GAE flex…</p>
<p>=====</p>
<pre><code class="lang-auto">runtime: python
env: flex

entrypoint: gunicorn -b :$PORT main:app

runtime_config:
  python_version: 3

manual_scaling:
  instances: 2

resources:
  cpu: 1
  memory_gb: 10
  disk_size_gb: 10

network:
  name: default

readiness_check:
  app_start_timeout_sec: 1800
</code></pre>

username: JeffO
cooked: <p>Increasing the disk size to 20gb fixed this for me.  There was actually an error message in the deploy logs but it is easy to miss because of the sheer volume of log messages.</p>

username: Ahmad_Islam
cooked: <p>Hey Jeff I am glad to see you this. I am having same trouble installing pytorch into my app engine application. Can you please share your cloudbuild as well. Whenever I use runtime: Python the build error out.</p>
How to run with "python3 -m x.y"?

username: sev777
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0cf40fd59214d1a01c642c529248c47da62fa827.png" data-download-href="https://discuss.pytorch.org/uploads/default/0cf40fd59214d1a01c642c529248c47da62fa827" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0cf40fd59214d1a01c642c529248c47da62fa827_2_690x18.png" alt="image" data-base62-sha1="1QAD2LLbFZGurCUVvHQvHRXbYOz" width="690" height="18" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0cf40fd59214d1a01c642c529248c47da62fa827_2_690x18.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0cf40fd59214d1a01c642c529248c47da62fa827_2_1035x27.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0cf40fd59214d1a01c642c529248c47da62fa827_2_1380x36.png 2x" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1720×46 10.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There are evalute.py under experiments/, But python3 -m can’t find them.<br>
“/root/data/miniconda3/envs/prompt/bin/python3: No module named experiments.evalute”<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7b3f1b09671bd2d93ec9412b57b20027cc24177.png" data-download-href="https://discuss.pytorch.org/uploads/default/a7b3f1b09671bd2d93ec9412b57b20027cc24177" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7b3f1b09671bd2d93ec9412b57b20027cc24177_2_689x68.png" alt="image" data-base62-sha1="nVzguAGSnrum4rnbOX6yscG3mth" width="689" height="68" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7b3f1b09671bd2d93ec9412b57b20027cc24177_2_689x68.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7b3f1b09671bd2d93ec9412b57b20027cc24177_2_1033x102.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7b3f1b09671bd2d93ec9412b57b20027cc24177.png 2x" data-dominant-color="101112"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1237×122 22.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: huahuanZ
cooked: <p>It seems like a typo error “evalute” → “evaluate”</p>
I am trying to extract features for a dataset and save them in 2D tensor with gradient information. Getting torch.cuda.OutOfMemoryError: CUDA out of memory

username: Debojyoti_Biswas
cooked: <p>Hi,<br>
I am training a CNN model for extracting features for a dataset. I want to save all extracted features in a 2D tensor with the gradient information. The below code block is throwing <strong>torch.cuda.OutOfMemoryError: CUDA out of memory</strong>. The batch size for the dataloader is 4 and the available GPU memory is 49GB.</p>
<pre><code class="lang-auto">a= []

for data , iteration in zip(dataloader, range(0, 10)):

    images = self.preprocess_image(data)

    features = self.backbone(images.tensor)

    for k in features.keys():
        if k == 'p3':
  
            neg_f= features[k].reshape(features[k].shape[0],-1).to(self._cpu_device)
            print(f"Check device neg_f: {neg_local_query_f.is_cuda}") 

            a.append(neg_f[0])
            
a = torch.cat(a, 0)
a = a.reshape(10,640000)
</code></pre>
<p>The neg_f tensor is stored in cpu memory. Although, if I stop gradients tracking I do not get any <strong>torch.cuda.OutOfMemoryError: CUDA out of memory</strong> error. The below code does not give error but I am not getting gradient information.</p>
<pre><code class="lang-auto">a= []

for data , iteration in zip(dataloader, range(0, 10)):

   images = self.preprocess_image(data)

   features = self.backbone(images.tensor)

   with torch.no_grad():
       for k in features.keys():
           print("This is the Backbone Return :",k,"shape",features[k].shape)
           if k == 'p3':

               neg_f= features[k].reshape(features[k].shape[0],-1).to(self._cpu_device)

               a.append(neg_f[0])
               print(f"Check device: {neg_f.is_cuda}")
           

a = torch.cat(a, 0)
a = a.reshape(10,640000)
</code></pre>
<p>Is there any way I can store the features with gradient information and not causing <strong>torch.cuda.OutOfMemoryError: CUDA out of memory</strong> error? And my final goal is to pass the stored 2D tensor features in <strong>Contrastive Loss</strong> as negative examples.</p>
<p>Thanks in advance!!</p>

username: ptrblck
cooked: <p>I assume you want to store all computation graphs to store the “gradient information”?<br>
If so, then the large increase in memory would be expected. Detaching the tensor would reduce the memory usage, but won’t allow you to compute the gradients w.r.t. the previously used parameters anymore.<br>
You could reduce the batch size and compute the gradients using your custom loss for a smaller number of samples while accumulating the gradients (if this is possible using your loss).</p>
AttributeError: 'NoneType' object has no attribute '__module__'

username: Abel
cooked: <p>Hi,</p>
<p>I am following the second example of the official tutorial on the link: <a href="https://pytorch.org/docs/stable/generated/torch.jit.trace.html" rel="noopener nofollow ugc">torch.jit.trace.html</a>.<br>
However, I get the error:</p>
<pre><code class="lang-auto">AttributeError: 'NoneType' object has no attribute '__module__'
</code></pre>
<p>for the line:</p>
<pre><code class="lang-auto">torch.jit.trace()
</code></pre>
<p>tested torch versions:   1.12.1+cu116, 1.13.0+cu116 and 1.13.0+cu117<br>
fastai: 2.7.10</p>
<p>Any suggestions on how to solve this issue?</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue using <code>torch==1.13.0</code> and both example code snippet work without raising an error.</p>
Is there a way to combine classification and regression in single model?

username: Mahdi_Amrollahi
cooked: <p>Is that possible to predict both classification and regression in single model?<br>
Suppose that we have target shape like <code> [b1, b2, c1, c2, c3]</code> which bs are for regression target and cs are for classification target and they are from one-hot encoding.</p>

username: KFrank
cooked: <p>Hi Mahdi!</p>
<aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="165549" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>Is that possible to predict both classification and regression in single model?</p>
</blockquote>
</aside>
<p>Yes, and it can be a reasonable thing to do.</p>
<aside class="quote no-group">
<blockquote>
<p>Suppose that we have target shape like <code> [b1, b2, c1, c2, c3]</code> which bs are for regression target and cs are for classification target and they are from one-hot encoding.</p>
</blockquote>
</aside>
<p>I recommend against using one-hot encoding.  There’s no reason to do<br>
so and it introduces at least a little inefficiency.</p>
<p>(I assume that you mean that you have three classes and that exactly<br>
one of <code>c1</code>, <code>c2</code>, and <code>c3</code> is one, with the other two being zero.)</p>
<p>Instead, let your classification <code>target</code> be a single integer class label that<br>
ranges from 0 to 2.</p>
<p>The simplest approach will be to have your final layer be a <code>Linear</code> with<br>
<code>out_features = 5</code>.  The first two outputs will be understood as the<br>
continuous-valued   predictions for <code>b1</code> and <code>b2</code>.  The remaining three will<br>
be understood as raw-score logits for the three classes.  Then, e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">loss_regression = torch.nn.MSELoss() (output[:, 0:2], reg_target)
loss_classification = torch.nn.CrossEntropyLoss() (output[:, 2:5], class_target)
loss_total = reg_weight * loss_regression + loss_classification
</code></pre>
<p>In this example, <code>reg_target</code> will be a floating-point tensor of shape<br>
<code>[nBatch, 2]</code> and <code>class_target</code> will be a <code>long</code> tensor of shape <code>[nBatch]</code>.<br>
<code>reg_weight</code> is a hyperparameter (not trained) that specifies the relative<br>
weight between your two losses.</p>
<p>(You could also split your model into regression and classification branches<br>
prior to the very last layer.  Whether doing so would offer any advantage<br>
would depend on the details of your use case.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Mahdi_Amrollahi
cooked: <p>And another question: May we expect that both problems go with the same speed? I mean, is that possible that the regression part still is under-fited but the classification  part goes over-fitted? Or they will have the same situation?<br>
Thanks</p>
RuntimeError: Could not infer dtype of ImplicitA

username: Ragnor
cooked: <p>hello, i am trying to add implicit in my code<br>
i want to convert imA into tensor but i got this error.</p>
<pre><code>def forward(self, x):
    _, _, H, W = x.size()
    imA = torch.tensor(ImplicitA(channel=2048))
    backbone_out = self.backbone(x) + imA()
    segmentation_head_out = self.segmentation_head(backbone_out)
    y = segmentation_head_out
    return y
</code></pre>
<p>please help!</p>

username: ptrblck
cooked: <p>Could you describe what <code>ImplicitA</code> is? Are you creating a new object and are trying to convert it to a tensor e.g. via:</p>
<pre><code class="lang-python">class ImplicitA(object):
    def __init__(self):
        pass

torch.tensor(ImplicitA())
# &gt; RuntimeError: Could not infer dtype of ImplicitA
</code></pre>

username: Ragnor
cooked: <p>yes.<br>
i am using Implicit knowledge.<br>
i make a class like</p>
<p>class ImplicitA(nn.Module):<br>
def <strong>init</strong>(self, channel):<br>
super(ImplicitA, self).<strong>init</strong>()<br>
self.channel = channel<br>
self.implicit = nn.Parameter(torch.zeros(1, channel, 1, 1))<br>
nn.init.normal_(self.implicit, std=.02)</p>
<p>now i want to add my backbone output and implicit but I got that error.</p>

username: Ragnor
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/5479738dabe4da6777f9cf8b6ef9da3e9bed1bef.png" data-download-href="https://discuss.pytorch.org/uploads/default/5479738dabe4da6777f9cf8b6ef9da3e9bed1bef" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5479738dabe4da6777f9cf8b6ef9da3e9bed1bef_2_690x411.png" alt="image" data-base62-sha1="c3ijAjTAf9T0QeuN2xZXDWfnb6n" width="690" height="411" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/5479738dabe4da6777f9cf8b6ef9da3e9bed1bef_2_690x411.png, https://discuss.pytorch.org/uploads/default/original/3X/5/4/5479738dabe4da6777f9cf8b6ef9da3e9bed1bef.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/4/5479738dabe4da6777f9cf8b6ef9da3e9bed1bef.png 2x" data-dominant-color="2F3030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">916×546 51.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I’m unsure what exactly you are trying to achieve, but you cannot transform an <code>nn.Module</code> to a tensor via:</p>
<pre><code class="lang-python">imA = torch.tensor(ImplicitA(channel=2048))
</code></pre>

username: Ragnor
cooked: <p>actually I solve this error.</p>
LLVM Error on MPS

username: thainamhoang
cooked: <p>Hi,</p>
<p>I’ve been tried to build SimSiam from scratch for rectangular input. However, I have the following error:</p>
<pre><code class="lang-auto">loc("mps_matmul"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/b6051351-c030-11ec-96e9-3e7866fcf3a1/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":28:0)): error: inner dimensions differ 512 &amp; 2048
loc("mps_matmul"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/b6051351-c030-11ec-96e9-3e7866fcf3a1/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":28:0)): error: invalid shape
LLVM ERROR: Failed to infer result type(s).
</code></pre>
<p>I’m using Mac M1 with MacOS 12.4, <code>torch.__version__ = 1.13.0.dev20220614</code><br>
I have never encountered this error before. Can you guys help me with it? Thank you!</p>

username: Dolan
cooked: <p>Hi,</p>
<p>I also got this error. Have you solved it?</p>

username: thecho7
cooked: <p><a class="mention" href="/u/thainamhoang">@thainamhoang</a> <a class="mention" href="/u/dolan">@Dolan</a><br>
<code>error: inner dimensions differ 512 &amp; 2048</code> seems to be occurred in matrix multiplication.<br>
I don’t use Mac OS but you need to check the dimension of each tensor.<br>
Especially <code>nn.Linear</code>, <code>torch.mul</code> and so on.</p>

username: karbapi
cooked: <p>I too got it. It does not show where it is occuring.</p>
How to load PTL models to make inference?

username: yakupakkaya
cooked: <p>I trained a yolov5 model and concerted to the PyTorch Lite version. How can I load it in python to test performance?</p>
<p>I tried to load the same way with the pt model as below,</p>
<p>model = torch.hub.load(“…/yolov5”, ‘custom’, path=‘…/weights/best.ptl’, source=‘local’)</p>
<p>it did not work. Is there a way to achieve it?</p>
Gradient accumulation with non-linear reduction of loss terms (not sum/mean)

username: ArmanMaesumi
cooked: <p>When doing gradient accumulation, we typically reduce the loss terms via a sum or mean:</p>
<pre><code class="lang-auto">for _ in range(N):
    loss_i = my_loss(x, y) / batch_size
    loss_i.backward()
optimizer.step()
</code></pre>
<p>Here we have a mean over the loss terms, so the “final” loss is implicitly<br>
<code>(1/batch_size) * sum(loss_i)</code><br>
Similarly a sum reduction would just require removing the division by batch size.</p>
<p>What I’m wondering is, is it possible to have a less trivial reduction with gradient accumulation? I would like to, say, have a final loss of <code>sqrt(sum(loss_i ** 2))</code> i.e. an L2 reduction of the losses.</p>
<p>Thanks in advance.</p>
TPU train;how to fix this bug;runtime error: Numpy is not available

username: holly9715
cooked: <pre><code class="lang-auto">import numpy as np
import torch
print(torch.__version__)
print(np.__version__)
a = np.array([1, 2, 3])
t = torch.from_numpy(a)
</code></pre>
<p>1.14.0a0+git8cb5c55<br>
1.19.5</p>
<blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
/tmp/ipykernel_2383/3624488259.py in <br>
4 print(np.<strong>version</strong>)<br>
5 a = np.array([1, 2, 3])<br>
----&gt; 6 t = torch.from_numpy(a)<br>
7</p>
<p>RuntimeError: Numpy is not available<br>
I tried several version :1.12,1.13  ,but it didn’t work</p>
</blockquote>

username: Jack_Cao
cooked: <p>Thanks for reporting! Do you mind open a github issue under <a href="https://github.com/pytorch/xla/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/xla · GitHub</a>? We now handles all TPU and pytorch/xla issue there.</p>
RuntimeError: add does not support non-contiguous NestedTensor inputs

username: Vadim
cooked: <p>Hi.<br>
I am trying to implement a more memory-efficient multi-head attention that uses NestedTensors instead of key_padding_mask.  I am running into this error on backward pass, however:  “RuntimeError: add does not support non-contiguous NestedTensor inputs”.</p>
<p>Does anyone know what this means?</p>
<pre><code class="lang-python">import torch
device = torch.device('cuda')
q = torch.nested.nested_tensor([torch.randn(5, 64, device=device) for _ in range(3)]).contiguous()
kv = torch.nested.nested_tensor([torch.randn([torch.randint(3, 11, [1]), 64], device=device) for _ in range(3)]).contiguous()
q.requires_grad = True
kv.requires_grad = True

# Lifted from _scaled_dot_product_attention_math
attn = torch.matmul(q, kv.transpose(-2, -1))
attn = attn.softmax(-1)
attn = torch.nn.functional.dropout(attn, 0.1, True)
out = torch.matmul(attn, kv)

sum([t.mean() for t in out]).backward()
</code></pre>
Custom static PTQ patterns

username: hh1234
cooked: <p>Hello,</p>
<p>I am new to PyTorch. How do I bring in custom quantization pattern in pytorch?<br>
For example: all the tensors going into concat needs to have the same scale and zp and the concat op itself. How do I bring this about with qnnpack backend?</p>

username: jcaip
cooked: <p>Hi hh1234,</p>
<p>Have you already taken a look at our tutorials for quantization in <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">eager</a> mode and <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html" rel="noopener nofollow ugc">fx</a>? Also FYI I think that quantized concat is already supported.</p>
Memory usage management (how to deal with small batches)

username: JordiB
cooked: <p>Hi, I’m trying to train a convolutional network for 3d data and have some trouble with memory usage.</p>
<p>I’ve searched the topic and found that one solution is to use batches. However, I’ve found that I have to use really small batches for them to start fitting the memory (I use batches of 5 in the “not very new” computer I usually use to work and 32 in a more suited computer).</p>
<p>Given so I wanted to ask two questions:</p>
<ol>
<li>
<p>Is it realistic to try to train an NN using batches with very little representation (I have to classify 20 classes. Also, I expect to have significant variance within each of those classes). If so do I need to adapt my training process in some way? For example, I’m guessing I should probably go for a small learning_rate, would that be correct?</p>
</li>
<li>
<p>My current training loop looks as follows:<br>
for epoch in range(num_epoch):<br>
for i,(inputs,labels) in enumerate(dataloader):<br>
y_pred = model(inputs)<br>
l = loss(y_pred, labels)<br>
optimizer.zero_grad()<br>
l.backward()<br>
optimizer.step()<br>
Where I take a step for every batch of 5 or 32 data points. Could I instead do something like this?:<br>
batch_size = 5 # The one used in the dataloader to not overload memory<br>
desired_batch_size = 5000<br>
labels_ = Tensor()<br>
outs = Tensor()<br>
for epoch in range(num_epoch):<br>
for i,(inputs,labels) in enumerate(dataloader):<br>
y_pred = model(inputs)<br>
labels_ = torch.cat((labels_,labels),0)<br>
outs =  torch.cat((out,y_pred),0)<br>
if i*batch_size &gt;= desired_batch_size:<br>
l = loss (outs, labels_)<br>
optimizer.zero_grad()<br>
l.backward()<br>
optimizer.step()<br>
Where I’m trying to take one step once enough data points have been processed for them to be somewhat representative. Or would this just make it impossible to compute the needed partial derivatives?</p>
</li>
</ol>
<p>Thanks for reading me</p>
How to train a variable just contain 0 or 1?

username: sev777
cooked: <p>The gumbel_softmax may be feasible, but it needs input, and I just want to train a variable <code>delta = torch.zeros(100, requires_grad=True, device="cuda") </code>, each element in delta is 0 or 1?</p>
<p>How to achieve this by gumbel_softmax or other methods?</p>

username: mvalente
cooked: <p>What do you mean? Do you wish your model to output just ones and zeros?</p>

username: sev777
cooked: <p>Yes, I want to get a [0 or 1] matrix W, and use W to choose some features.</p>

username: mvalente
cooked: <p>Why can’t you use an activation function?</p>

username: sev777
cooked: <p>The activation functions (e.g. Sigmoid) do not get strictly 0 or 1, or these activation functions are not differentiable.</p>

username: mvalente
cooked: <p>I need more context, like the type of problem you’re dealing with. Classification? Multi-class? Regression?. Also examples of actual target label.</p>

username: sev777
cooked: <p>Like I have a feature list: F=[f1,f2,…,fn], I want to find which is important, So, I need a [0,1] matrix W to select them. I want to select them not just get the important score, So, the W need to be 0 or 1 strictly.</p>

username: JordiB
cooked: <p>Couldn’t you use floats for the training and then just use a threshold? So all the nums in F that are higher than for example 0.5 get assigned to 1/important.</p>
When does it make sense to use multiple instances of a loss function?

username: Mole_m7b5
cooked: <p>I’m currently using a Unet style network to predict 3 target time-frequency features from a stereo recording, two are directional, and one is the diffuseness. So the output shape of my network is something like <code>(features, frequency, timestep)</code></p>
<p>At present I’m just using a single loss metric (MSE) over the output of the network. Apart from getting a more detailed view of the networks progress on each of the individual target features, are there any benefits to using a separate loss function for each feature and then summing them to get the overall loss for backprop?</p>
<p>My intuition is if all target features are “equally important” so the loss are all equally weighted, it won’t affect the weight updates and therefore won’t affect the networks training. But it may provide insight into which features the network is more easily able to learn the desired mapping for.</p>
Bralow twins embeddings

username: Richa
cooked: <p>Hi I am using Barlow Twins to train a Liver MRI data. I am having a difficulty to get the embeddings from the original code from <a href="https://github.com/facebookresearch/barlowtwins" rel="noopener nofollow ugc">facebook</a>. I tried two methods to get the embeddings farward_hook() and create_feature_extractor() from torchvision.models.feature_extraction. from the hook i am getting two embedding as Za and Zb. and from the  create_feature_extractor i am getting only one embedding. How to get the single embedding for a image not the transformed version. during the training or the inference mode. Please help. I am a beginner  with Pytorch.</p>
Efficient spatio-temporal ROI extraction from videos

username: ghazalehtrb
cooked: <p>Hello,<br>
I’m doing multiperson spatiotemporal action recognition; the action recognition model receives sequences of ROIs (detected per frame with an object detector) for each person and outputs the probabilities.<br>
I’m trying to make the whole pipeline work in real-time (30 fps), and I noticed that the speed gets significantly lower with more people in the video.<br>
The action recognition model receives the input for all people in a batch, but the preprocessing of its input data is done sequentially per person. Basically, I can crop all the frames of the video at the same time, but only for one person.<br>
Is there a way to crop the video at multiple locations at the same time, so I can remove the per person loop?<br>
Thank you.</p>
How to cast a pre-trained PyTorch model in JAX.numpy?

username: lizasemenova
cooked: <p>How can I cast a pre-trained PyTorch model in Numpy, or, even better, JAX.Numpy?</p>
<p>(Context: I need to use a pre-trained PyTorch model as a deterministic transformation within Numpyro, which is based on JAX)</p>
<p>Here is a minimal example which I would appreciate help with:</p>
<pre><code class="lang-auto">import torch.nn as nn
import jax.numpy as jnp

model = nn.Linear(4, 4)
z =  jnp.array([5.,4.,3.,2.])
model(z)
</code></pre>
<p>It produces the error <code>TypeError: linear(): argument 'input' (position 1) must be Tensor, not DeviceArray</code>.</p>
<p>Converting <code>z</code> to a Torch Tensor is not an option since it causes issues with Numpyro. The only option (if I understand correctly) is to port the model’s architecture to JAX.numpy.</p>
<p>The description of my full problem is here: <a href="https://forum.pyro.ai/t/how-to-use-a-pre-trained-pytoch-model-within-numpyro/4749" class="inline-onebox" rel="noopener nofollow ugc">How to use a pre-trained PyToch model within Numpyro? - numpyro - Pyro Discussion Forum</a></p>