akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch gradient computing of multidim tensor

ArkadyBogdanov: <p>Hello, this question has probably already been asked, I just couldn’t find it. If so, could you kindly redirect me to that thread? Thank you.</p>
<p>Is there an easier way of computing gradients of a matrix in PyTorch?</p>
<pre><code class="lang-auto">import torch

x = torch.linspace(0, 1, 50, requires_grad=True).view(-1, 1)
inputs = torch.hstack([torch.cos(w*x) for w in range(5)])


d_inputs = torch.hstack([grad(f.sum(), x, create_graph=True, retain_graph=True)[0] for f in inputs.T])
</code></pre>
<p>Thank you.</p>
How to add a new parameter which is derived from output of the previous layer of the neural network model

Hritik_Gopal_Shah: <p>How to add a new parameter which is derived from output of the previous layer, apart from weights and baises in Custom layer of the neural network model.<br>
I want the ouput of previous layer rolled(torch.roll) by 1 position and use this new parameter in the Custom Layer after it.</p>

ptrblck: <p>Note that the output of a layer would not be a parameter, as it’s not a leaf tensor because it was calculated from another parameter.<br>
In any case, you could try to assign the output of the previous layer to the parameter attributes of the next layer but should note that this custom layer will not be trained by the optimizer anymore.</p>

Hritik_Gopal_Shah: <p>How can i use CVXPY(Convex Optimization) layer as cutsom layer inside Deep Neural networks in Pytorch?</p>

ptrblck: <p>I don’t know how this layer is defined, but assuming it’s derived from <code>nn.Module</code> you should be able to directly use it as any other layer.</p>
Stride 2 conv tip

blackbirdbarber: <p>I have the stide2 conv defined:</p>
<pre><code class="lang-auto">conv = nn.Conv2d(in_channels=256, out_channels=256,kernel_size=3, padding=1,stride=2)
</code></pre>
<p>Out:</p>
<pre><code class="lang-auto">torch.Size([1, 256, 25, 25])
torch.Size([1, 256, 13, 13])
</code></pre>
<p>but I would like to have it like this:</p>
<pre><code class="lang-auto">torch.Size([1, 256, 25, 25])
torch.Size([1, 256, 12, 12])
</code></pre>
<p>I tried altering the dilation, padding, but didn’t work. How to make conv2d use the floor operation ( ⌊x⌋) for dimensions?</p>

ptrblck: <p>You could play around with other settings, e.g. <code>conv = nn.Conv2d(in_channels=256, out_channels=256,kernel_size=5, padding=1,stride=2)</code> would create the desired output shape. However, if you don’t want to change the <code>kernel_size</code> it might be easier to just slice the output.</p>

blackbirdbarber: <p>I just wanted to confirm I am not missing something since I would keep these 3x3 kernels. The only thing I like better are 1x1 kernels.</p>
Why does Resnet34 performs better than Resnet50 in coordinate regression task?

Ayush_Saini: <p>Hey, I’m doing a coordinate regression task for an object.<br>
something like <a href="https://i.ibb.co/Rzzy0hd/dot2.jpg" rel="noopener nofollow ugc">this</a>(not exactly this ), where the label would be (17,40).<br>
So, when I’m using Resnet50 I’m getting poorer results as compared to Resnet34.<br>
Can someone explain me why it could be happening?<br>
Resnet50 despite having more parameters has 1x1 convolutions, does this hurt in my case?</p>
Compute Loss from External Precomputed Loss Value

San: <p>I am trying to backpropagate the error using a value computed externally by a heuristic. The custom loss function is the following.</p>
<p>The problem is, when lb (lambda) equals to zero, all the gradients are zero after the backward( ) loss call.</p>
<p>When lambda is a value &gt; 0 (such as 0.001), the weight gradients will be different from zero.</p>
<p>Any idea ?</p>
<p>Best regards.</p>
<pre><code class="lang-auto">class FCLayerCustomLoss(torch.nn.Module):
    def __init__(self, model, lb = 0.001):
        super(FCLayerCustomLoss, self).__init__()
        self.model = model
        self.lb = lb 

    def forward(self, score):
        
        dense0_params = torch.cat(tuple([x.view(-1) for x in self.model.dense0.parameters()]))
        
        l0_regularization = self.lb * torch.norm(dense0_params, p=2)      
        return score + l0_regularization
</code></pre>
Optuna unfreeze layers

Emmanuel_OM: <p>Hello, I face the next issue when freezing layers in the objective function. The layers remain with requires_grad = True:</p>
<pre><code class="lang-python">def freeze_layer(model, layer_instance, layer_name, layer_numbers, freeze=True):
    for name, layer in model_ft.named_modules():
        for layer_number in layer_numbers:
            layer_id = '%s%d' %(layer_name, layer_number)
            if isinstance(layer, layer_instance):
                if name.startswith(layer_id):
                    for param in layer.parameters():
                        param.requires_grad_(not freeze)

    return model


def objective(trial):

    # Generate the model.
    base_model = models.resnet18(weights="DEFAULT")
    num_ftrs = base_model.fc.in_features
    base_model.fc = nn.Linear(num_ftrs, 10)

    layers2freeze = trial.suggest_categorical("freeze_layers", ['A', 'B', 'C'])
    map_layers2freeze = dict(A=[4], B=[4, 3], C=[4, 3, 2], D=[4, 3, 2, 1])
    layersF = map_layers2freeze[layers2freeze]

    model = freeze_layer(base_model, torch.nn.Conv2d, 'layer', layersF, True)

    for name, param in model.named_parameters():
        print('Name: ', name, 'Requires_Grad:', param.requires_grad)
</code></pre>
'int' object has no attribute 'backward' in tutorial

Sivar: <p>I hope this is correct place to ask a very beginner question.<br>
I’m trying to create some names with this tutorial:<br>
<a href="https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html</a></p>
<p>When I run code copy pasted from it, I get an error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/sivarion/PycharmProjects/pythonProject/tut.py", line 167, in &lt;module&gt;
    output, loss = train(*randomTrainingExample())
  File "/home/sivarion/PycharmProjects/pythonProject/tut.py", line 150, in train
    loss.backward()
AttributeError: 'int' object has no attribute 'backward'
</code></pre>
<p>It looks correct, as in train() function loss is declared as:</p>
<pre><code class="lang-auto">loss = 0
</code></pre>
<p>It is indeed an integer and it is hard to expect for it to have .backward() method.<br>
But in tutorial it states explicit, that:</p>
<blockquote>
<p>The magic of autograd allows you to simply sum these losses at each step and call backward at the end.</p>
</blockquote>
<p>I’m not sure what should I do here and how to correctly define loss in this case.<br>
Thank you in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

sudomaze: <p>Hi,</p>
<p>To utilize the <code>.backward()</code> method, you will need to have your <code>loss</code> be <code>PyTorch Tensor</code>. It is possible that the <code>for loop</code> was skipped (<code>for i in range(input_line_tensor.size(0)):</code> in the tutorial that you shared) which didn’t update <code>loss</code> to be a <code>PyTorch Tensor</code> object.</p>
<p>Can you check what is the value for <code>input_line_tensor.size(0)</code>?</p>

Sivar: <p>OK, so as usual, the problem was between the keyboard and the back of the chair <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
I added a file with my own names to the supplied datasets - it seemed identical to the others and was loaded correctly, so I thought it wouldn’t cause problems.</p>
<p>After your comment, I decided to check if and how many times the loop is called. It turns out that it is called correctly until it hits the name from my file. After removing it, everything works flawlessly <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Now I still need to figure out how the data is different and correct it accordingly, but I suppose that’s no longer a PyTorch-related problem. Thanks for your support!</p>

sudomaze: <p>Awesome, happy to help!</p>
Why put mean and std values ourselves?

AjeelAhmed1998: <pre><code class="lang-auto">transform = transforms.Compose([transforms.ToTensor(), 
                                transforms.Normalize((0.5,0.5,0.5)
                                , (0.5,0.5,0.5))])
</code></pre>
<p>I have seen people normlize the pictures this way, I know we do it because after using transforms.ToTensor() the return values range from [0,1], my question is why are we defining mean and std values ourselves? Shouldn’t we actually calculate what the mean and std would be in each case?</p>

sudomaze: <p>Hi Ajeel,</p>
<p>This isn’t the optimal way to normalize your input because you will be shifting the data to a distribution that most likely doesn’t represent the data distribution. The correct way to do this is to measure the <code>mean</code> and the <code>std</code> of your training dataset, then utilize that information to normalize your input.</p>
Benchmarking dataloading

itamar_kanter: <p>I’m trying to benchmark different <em>datasets</em> object with <code>torch.utils.benchmark</code>.<br>
I have noticed that  I can control the size of threadpool through the <code>num_threads</code>  param.<br>
In <code>torch.utils.data.DataLoader</code> there exists is the <code>num_workers</code> param.<br>
What is the diiffernce between thus two params; <code>num_workers</code> and  <code>num_threads</code>?<br>
I have noticed that changing <code>num_threads</code> didn’t affect the benchmarking results</p>

calwritescode: <p>Maybe this thread helps!</p>
<aside class="quote" data-post="1" data-topic="47045">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/53a042/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-num-workers-vs-torch-set-num-threads/47045">DataLoader num_workers vs torch.set_num_threads</a> 
  </div>
  <blockquote>
    Is there a difference between the parallelization that takes place between these two options? I’m assuming num_workers is solely concerned with the parallelizing the data loading. But is setting torch.set_num_threads for training in general? Trying to understand the difference between these options. Thanks!
  </blockquote>
</aside>
Implementation multiagent learing

TKO: <p>Hello, can someone help me with implementation of coordinated multiagent imitation learning? I use Gym Stag Hunt enviroment.</p>

vmoens: <p>hi <a class="mention" href="/u/tko">@TKO</a><br>
Thanks for this. This is a broad question and I think your chances of getting a detailed answer will increase substantially if you narrow down the scope a little bit. Can you detail a bit what you would like to do what blocks you?</p>

TKO: <p>I have a lot of problems :D. Firstly i need convert this state to 3d array so that my neural network can learn.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c863eb2ce07e1b13f4652562e5d670ca22f2ea06.png" alt="state" data-base62-sha1="sAJzgxb0JyUSvVQuK6ooQcjOlU2" width="365" height="471"></p>

vmoens: <p>You should map it to a float tensor and devide by 255. Have a look at <a href="https://github.com/pytorch/vision/tree/main/torchvision/transforms" rel="noopener nofollow ugc">torchvision transforms</a> for this.<br>
I wonder what those dimensions are though: is it that you have 36 agents, each returning an image as observation?<br>
A better way to represent that would be a tensor of shape [36, 212, 158, 3] so that you keep the channels separated for the transforms.</p>

TKO: <p>It is not 36 agents it is only 2 agents. This screenshot is not complete. It is state of 8 played games</p>

TKO: <p>I use this enviroment <a href="https://github.com/NullDefault/Gym-Stag-Hunt" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NullDefault/Gym-Stag-Hunt: A custom reinfrocement learning environment for OpenAI Gym &amp; PettingZoo that implements various Stag Hunt-like social dilemma games.</a></p>
Is it possible to make `Conv1d` accept `PackedSequence` as input?

GF-Huang: <p>We know <code>Conv1d</code> input shape is (batch_size, channels, length), the <code>channels</code> act as <code>input features</code> of input sequence in my case.</p>
<p>My input sequences have different lengths. I know I can use <code>nn.utils.rnn.pack_sequence</code> for padding sequences to same lengths, but how to make <code>Conv1d</code> compatible with it like RNN?</p>
<p>Because I don’t want to the padding zeros which may effect training.</p>
About torchvision.transforms.functional.adjust_brightness

YuyXiang: <p>Hi, when i use <em>torchvision.transforms.functional.adjust_brightness</em> to change the brightness of tensor image, the generated image is not normal, like adding a gray mask on it.<br>
But when i use pil image, the result is right.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.transforms.functional as F

class Br(nn.Module):
    
    def __init__(self, change):
        super(Br, self).__init__()
        self.change = change
        
    def forward(self, image_and_cover):
        image, cover_image = image_and_cover

        output = F.adjust_brightness(image, self.change)
        return output
</code></pre>
<p>The picture below is the result when brightness_factor is setting to 1.0.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7668f25dc9abdebeab82b94a385ca1c89d227399.png" alt="epoch-test" data-base62-sha1="gTvbI1ZLdabfvT742brUNKT1t9f" width="512" height="256"></p>

ptrblck: <p>I cannot reproduce the issue and get the original image back for <code>PIL.Image</code> and tensor inputs:</p>
<pre><code class="lang-python">pil_image = PIL.Image.open("./tmp03.jpeg")
pil_image

factor = 1.0
output = F.adjust_brightness(pil_image, factor)
output

tensor_image = torch.from_numpy(np.array(pil_image))
tensor_image = tensor_image.permute(2, 0, 1).contiguous()
tensor_output = F.adjust_brightness(tensor_image, factor)

PIL.Image.fromarray(tensor_output.permute(1, 2, 0).contiguous().numpy())
</code></pre>

YuyXiang: <p>Thank you! I find the reason, when I load the dataset, I performed a normalization operation. So before adjusting the brightness, I should perform the inverse.</p>
<pre><code class="lang-auto">image = (image + 1) / 2
</code></pre>
Inference Images look different than training images

mahmood: <p>Hello, I have trained a GAN model on a training set it produces correct images (the target output styled image) but when I run inference it does not produce such images that look distorted and different than how it produces during training. What could be the possible reasons?</p>
<p>Output during training<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc14fb3b19e37dd8ac2bf7b26ba6be3ccf1d853a.jpeg" alt="image" data-base62-sha1="qPQIrt7mvo08dX53JzuZn93hfKi" width="343" height="340"></p>
<p>Output during inference<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/5057bad2bb6385c7850b1c1687c836b2fda66995.png" data-download-href="https://discuss.pytorch.org/uploads/default/5057bad2bb6385c7850b1c1687c836b2fda66995" title="aachen_000015_000019_leftImg8bit"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5057bad2bb6385c7850b1c1687c836b2fda66995_2_500x500.png" alt="aachen_000015_000019_leftImg8bit" data-base62-sha1="bsK9MEyz2vbziieBteXQ8j0HvMN" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5057bad2bb6385c7850b1c1687c836b2fda66995_2_500x500.png, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5057bad2bb6385c7850b1c1687c836b2fda66995.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5057bad2bb6385c7850b1c1687c836b2fda66995.png 2x" data-dominant-color="70486D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">aachen_000015_000019_leftImg8bit</span><span class="informations">512×512 380 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mahmood: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Please help me with this. What could be possible reasons?</p>
How to correctly skip some training steps which dissatisfies expected conditions?

HomerNee: <p>Hello, everyone! I am trying to write code to achieve the following functions:<br>
Due to some reasons, during training, my loss can be NONE. So when the loss is NONE, I skip this training step and directly continue the next iteration step. But it seems that I fail to free up the GPU memory. Thus in the next iteration step, I will encounter “out of memory” error. Some part of my code is as follows, and I use two networks in my code. I have tried to delete the <strong>output</strong>,<strong>input</strong>,<strong>target</strong>, but my GPUs are still out of memory.</p>
<p>I am very grateful if anyone is nice to help me!</p>
<pre><code>    for i, (input, target) in enumerate(train_loader):
        actual_step = int(args.start_step + cnt)
        adjust_learning_rate(optimizer1, actual_step)
        adjust_learning_rate(optimizer2, actual_step)

        input = input.cuda()
        target = target.cuda()

        # compute output
        output1 = model1(input)
        output2 = model2(input)

        loss1_1, loss2_1 = criterion(output1[0], output2[0], target)
        loss1_2, loss2_2 = criterion(output1[1], output2[1], target)
        if loss1_1 is None:
            print("loss1_1 is None!")
            # TODO release GPU memory
            continue # in the next step, I encounter Out of Memory Error.
        if loss1_2 is None:
            print("loss1_2 is None!")
            # TODO release GPU memory
            continue

        loss1 = loss1_1 + 0.4 * loss1_2
        loss2 = loss2_1 + 0.4 * loss2_2

        # compute gradient and do SGD step
        optimizer1.zero_grad()
        loss1.backward()
        optimizer1.step()

        optimizer2.zero_grad()
        loss2.backward()
        optimizer2.step()</code></pre>

MariosOreo: <p>Hello,</p>
<p>You could just call <code>.backward()</code> without <code>optimizer.step()</code> if the step is unexpected condition.</p>
<pre><code class="lang-auto">optimizer.zero_grad()
if condition:
    loss.backward()
else:
    loss.backward()
    optimizer.step()
</code></pre>

HomerNee: <p>Hello,Thanks very much for your reply! But I think this way may accumulate the backward gradient. I have tried another similar way. When loss is none, I change to use a normal way to calculate the loss and everything is fine now.</p>

zzibc: <p>Hi <a class="mention" href="/u/homernee">@HomerNee</a> , I met the same issue. Would you mind tell me how to solve this problem more specific?</p>
How to add gradient noise which is not normal distribution?

BobKim: <p>Hello! everyone!<br>
I have a few questions about optimizer.</p>
<p>I want to add the gradient noise which is not normal distribution. (i want to add the alpha stable distribution noise!!)<br>
I know that a function (torch.randn_like) generates random numbers from a normal distribution.</p>
<p>How do i generate random numbers from a alpha stable distribution?</p>
Generating a bool mask by labels

Godspeed: <p>Hi,</p>
<p>I am trying to generate a mask to get the positive logits.</p>
<p>I get label_1 [1,2,3,4,7]   and  label_2   [1,2,7,2]</p>
<p>the goal is to get a 01 mask M, when i and j are the same class, Mij=1, others = 0</p>
<p>the above label_1 and label_2 could generate such mask with size[5,4]<br>
[[1,0,0,0],<br>
[0,1,0,1],<br>
[0,0,0,0],<br>
[0,0,0,0],<br>
[0,0,1,0]]</p>
<p>Well, I do generate the mask by iteration, but I it is not efficient when labels get larger.</p>

ptrblck: <p>A comparison with broadcasting should work:</p>
<pre><code class="lang-python">label_1 = torch.tensor([1,2,3,4,7])
label_2 = torch.tensor([1,2,7,2])

res = label_1.unsqueeze(1) == label_2
print(res)
# tensor([[ True, False, False, False],
#         [False,  True, False,  True],
#         [False, False, False, False],
#         [False, False, False, False],
#         [False, False,  True, False]])

print(res.long())
# tensor([[1, 0, 0, 0],
#         [0, 1, 0, 1],
#         [0, 0, 0, 0],
#         [0, 0, 0, 0],
#         [0, 0, 1, 0]])
</code></pre>

Godspeed: <p>Really cool!  Thanks a lot.</p>
RuntimeError: shape '[10, 25]' is invalid for input of size 182

Lemuria: <p>I am having an issue here i am not sure how to solve i am thinking that it might be how the input layer of the model is but i don’t see how it relates to input size of 182. I am building an a RL agent using open ai for lunar lander which inputs an observation space of 8 values and action space of 4 hence the input of 8 in the first layer</p>
<pre><code class="lang-auto">Input In [5], in test_model(agent)
      8 score = 0
     10 while not done:
---&gt; 11     params = unpack(agent['params'])
     12     probs = model(state,params)
     13     action = torch.distributions.Categorical(probs=probs).sample()

Input In [1], in unpack(params, layers)
     21 s,e = e,e+np.prod(l)
     22 print("s,e",s,e)
---&gt; 23 weights = params[s:e].view(l)
     24 s,e= e,e+l[0]
     25 bias = params[s:e]
</code></pre>
<p>Here is the code with problem</p>
<pre><code class="lang-auto">def model(x,params):
    l1,b1,l2,b2,l3,b3 = params
    y = torch.nn.functional.linear(x,l1,b1)
    print("YY",y)
    y=torch.relu(y)
    y = torch.nn.functional.linear(y,l2,b2)
    y=torch.relu(y)
    y = torch.nn.functional.linear(y,l3,b3)
    y=torch.log_softmax(y,dim=0)
    print("Y",y)
    return y

def unpack(params,layers=[(25,8),(10,25),(4,10)]):
    unpacked = []
    
    e=0
    for i,l in enumerate(layers):
        print("il",i,l)
        s,e = e,e+np.prod(l)
        print("s,e",s,e)
        weights = params[s:e].view(l)
        s,e= e,e+l[0]
        bias = params[s:e]
        unpacked.extend([weights,bias])
    print("unpacked",unpacked)
    return unpacked
</code></pre>

ptrblck: <p>I don’t know where <code>params</code> is coming from but it seems slicing it via <code>params[s:e]</code> returns a tensor in the shape <code>[182]</code> which causes the <code>view(l)</code> operation with <code>l=[10, 25]</code> to fail:</p>
<pre><code class="lang-python">x = torch.randn(182)
x.view([10, 25])
# RuntimeError: shape '[10, 25]' is invalid for input of size 182
</code></pre>
<p>Make sure the slicing and <code>view</code> operation correspond to each other as it seems the error is caused by a mismatch there.</p>
Training loss reduce to zero in first epoch

Vidyashree: <p>I am a beginner in deep learning and in coding also, kindly help in solving the issue that I am facing. I am training the SketchANet model using the shoe2V dataset, including the sketch and photo images. I am preparing the triplet dataset that is anchor, positive image and negative image.  while training the model I am using the nn.TripletMarginLoss(margin=0.3) loss model. while I am training the model the training loss starts from 0.3 and reduces to 0 in the first epoch itself. Kindly someone helps in solve this issue… Thanks in advance</p>
Initialization Failure of nn.DataParallel on NVIDIA A6000

Alex2: <p>PyTorch version: 1.10.0<br>
GPU: NVIDIA RTX A6000</p>
<p>I have recently come across an issue when using DataParallel on A6000 GPUs. When wrapping a module with DataParallel, the weights on the auxiliary GPUs (outside of cuda:0) are all set to zero and the correct initialization is not copied over. This creates inaccurate results for the splits of the input batch evaluated on each of these models.</p>
<p>Strangely, these weights remain at zero, although sometimes the output from the layer is unpredictable (not zero, despite the output being zero just before being returned in the layer). I’ve attached a script which reproduces the issue, and prints the incorrect weights.</p>
<pre><code class="lang-auto">import os
import torch
import torch.nn as nn

os.environ['NCCL_P2P_DISABLE'] = '1'
os.environ['NCCL_IB_DISABLE'] = '1'
os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'
os.environ['CUDA_VISIBLE_DEVICES'] = '1,2,3'

class Test(nn.Module):
    def __init__(self):
        super().__init__()
        self.a = torch.nn.Linear(10,10)
        self.a.weight.data *= 100.
    def forward(self, dict):
        print(dict['tensor'])     # Tensor is distributed properly
        print(self.a.weight)      # Weight is not distributed properly

        out = self.a(dict['tensor'])

        print(out)                # Output has all zeros for GPUs beyond 0
        print(out.sum())          # Sum is all zeros for GPUs beyond 0
        return out


t = Test().cuda()
opt = torch.optim.Adam(t.parameters())
tt = nn.DataParallel(t)

tensor = {"tensor":torch.randn(2,10)}

for i in range(10000):
    print('Iteration: '+str(i))
    opt.zero_grad()
    a = tt(tensor)

    # Output has the incorrect result for tensor distributed onto GPU 1, etc..
    # However, for iterations after zero, this is not always 0 (strange)
    # Unsure where this value comes from
    print(a)

    print(a.sum())                 # Sum is not equivalent to sum of individual outputs on each GPU
    a.sum().backward()
    opt.step()
</code></pre>
<p>I have tested the same code on Quadro RTX 6000 GPUs with the same version of PyTorch, and it behaves properly, so this issue seems to be related to A6000s.</p>

ptrblck: <p>It sounds as if your system cannot properly communicate between the GPUs so I would recommend to run the <a href="https://github.com/NVIDIA/nccl-tests">NCCL tests</a> and check the numerical errors as well as the bandwidth.</p>
Matrix multiplication for large sparse matrices which does not fit into GPU

nasim_ahmed: <p>I am trying to do matrix multiplication from a large dataframe, and cannot create the matrix, for the following statement.</p>
<pre><code class="lang-auto">scores = torch.sparse.mm(diagnoses * freq_adjustment.unsqueeze(0), diagnoses.permute(1, 0))
</code></pre>
<p>The output in the console is “RuntimeError: CUDA out of memory. Tried to allocate 266.31 GiB (GPU 0; 7.79 GiB total capacity; 2.48 GiB already allocated; 3.52 GiB free; 2.49 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF”</p>
<p>I have 8GB of GPU. How can I do it efficiently in my GPU?</p>

sudomaze: <p>Hi Nasim,</p>
<p>What you can do is reduce the data type of the tensor from your current type to a lower dataset (like from float64 to int8 if your sparse matrices can be repeated by int8 without any issues) and split the tensor into multiple small pieces that can fit into memory.</p>

nasim_ahmed: <p>I already reduced the datatype to int8, and I split the tensor to chunks. It returns a tuple, then how do I pass it to sparse.mm?</p>
<pre><code class="lang-auto">splitted_diagnoses = torch.split(diagnoses, 1000)
</code></pre>
<p>Thanks</p>

sudomaze: <p>You will split based on some columns and some rows as matrix multiplication are done, could be done in series, then you will concatenate the matrices together.</p>
What does out_channels in Conv2d represent?

AjeelAhmed1998: <pre data-code-wrap="import"><code class="lang-nohighlight">import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


net = Net()
</code></pre>
<p>I am taking a look at PyTorch Blitz and in the <code>conv1</code> layer we can see the <code>input_channels=3</code> because it’s the first image so it just has its 3 RGB channels and <code>out_channels=6</code>.</p>
<p>Does that mean the number of filters I have are 6? In which case it’d mean the total number of feature maps I would get are <code>6*3==18</code>? But if that is the case why in <code>conv2</code> am I plugging in <code>input_channels=6</code>, shouldn’t I be plugging in  18 because that was the output from the previous Convolutional layer?</p>

ptrblck: <aside class="quote no-group" data-username="AjeelAhmed1998" data-post="1" data-topic="158355">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajeelahmed1998/48/52089_2.png" class="avatar"> AjeelAhmed1998:</div>
<blockquote>
<p>Does that mean the number of filters I have are 6?</p>
</blockquote>
</aside>
<p>Yes, the <code>out_channels</code> represent the number of output channels and thus the number of filters in the default setup.</p>
<aside class="quote no-group" data-username="AjeelAhmed1998" data-post="1" data-topic="158355">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajeelahmed1998/48/52089_2.png" class="avatar"> AjeelAhmed1998:</div>
<blockquote>
<p>In which case it’d mean the total number of feature maps I would get are <code>6*3==18</code>?</p>
</blockquote>
</aside>
<p>No, you would get <code>out_channels</code> activation maps, as each filter uses <em>all</em> input channels to create one output activation map (in the default setup; grouped convs would work in a different way).</p>

AbdulsalamBande: <p><a class="mention" href="/u/ajeelahmed1998">@AjeelAhmed1998</a><br>
You can watch <a href="https://www.youtube.com/watch?v=Osx4gfa2e5A&amp;ab_channel=PytorchModulesExplained" rel="noopener nofollow ugc">torch.nn.Conv2d Module Explained</a></p>

AjeelAhmed1998: <p>Thank you so much for your help, I understand it now.</p>

AjeelAhmed1998: <p>Yes this helped me understand it as well, thank you.</p>
Multiple input model - input and output features

Anna_Inberg: <p>Good afternoon!<br>
I’m building a multiple-input model with 2 types of inputs:<br>
Images (torch.Size([1, 3, 224, 224])) and landmark features (torch.Size([1, 96])).<br>
Here’s the model itself:</p>
<pre><code class="lang-auto">class MixedNetwork(nn.Module):
    def __init__(self):
        super(MixedNetwork, self).__init__()
        
        image_modules = list(models.resnet50().children())[:-1]
        self.image_features = nn.Sequential(*image_modules)

        self.landmark_features = nn.Sequential(
            nn.Linear(in_features=96, out_features=192,bias=False), 
            nn.ReLU(inplace=True), 
            nn.Dropout(p=0.25),
            nn.Linear(in_features=192,out_features=1000,bias=False), 
            nn.ReLU(inplace=True), 
            nn.Dropout(p=0.25))
        
        self.combined_features = nn.Sequential(
            nn.Linear(1000, 512),
            nn.ReLU(),
            nn.Linear(512, 32),
            nn.ReLU(),
            nn.Linear(32,1))
        
    def forward(self, image, landmarks):
        a = self.image_features(image)
        print(a.shape)
        b = self.landmark_features(landmarks)
        x = torch.cat((a.view(a.size(0), -1), b.view(b.size(0), -1)), dim=1)
        x = self.combined_features(x)
        x = F.sigmoid(x)
        return x
</code></pre>
<p>I’m getting confused when it comes to defining input-output features for Linear layers and combined layers. The last FC layer of resnet50 Linear(in_features=2048, out_features=1000). Does it mean that the last output of self.landmark_features layers also has to be 1000 and the first linear layer of self.combined_features should also be 1000?</p>
<p>Is it correct to assume that if the landmark input size is [1, 96] then the in_features for the first layer of self.landmark_features has to be 96?</p>
<p>With the current dimensions I’m getting the error message:<br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x3048 and 1000x512)<br>
(why 3048 and not 2048?)</p>
Problem with back propogration

zia_badar: <p>Can someone point out the mistake in my code, loss is not updating optimizing parameter:b</p>
<pre><code class="lang-auto">import torch
from torch.nn import Parameter

if __name__ == '__main__':
    a = torch.arange(start=1, end=5, dtype=torch.float32)
    b = torch.arange(start=4, end=0, step=-1, dtype=torch.float32)

    optim = torch.optim.Adam([Parameter(b, requires_grad=True)], lr=1e-3)
    b.requires_grad = True
    b.retain_grad()

    for i in range(100):
        optim.zero_grad()
        loss = a @ b.t()
        loss.backward()
        optim.step()
        print(loss)

</code></pre>

ptrblck: <p>Create <code>b</code> diredctly as an <code>nn.Parameter</code> object and it should work.<br>
Currently you are passing a temporary object to the optimizer, which will not update the <code>b</code> tensor.</p>

zia_badar: <p>It works now</p>
<pre><code class="lang-auto">import torch
from torch import nn

if __name__ == '__main__':
    a = torch.arange(start=1, end=5, dtype=torch.float32)
    b = torch.arange(start=4, end=0, step=-1, dtype=torch.float32)
    b_p = nn.Parameter(b)


    optim = torch.optim.Adam([b_p], lr=1e-3)
    b.requires_grad = True
    b.retain_grad()

    for i in range(100):
        optim.zero_grad()
        loss = a @ b_p
        loss.backward()
        optim.step()
        print(loss, b)
</code></pre>
Weighted Random Sampler in PyTorch

simines: <p>I want to use weighted random sampler for train data in PyTorch for image segmentation. However, I am not sure how I should compute weights since each image contains multiple labels and whether the weights should be based on pixels or number of samples per each class?<br>
Thank you in advance for your help.</p>

sudomaze: <p>You can weight based on the frequency that a single class exists across all of the training dataset. This means, that if a class exists three times in an image and once in another image, whereas another class exists twice in the first image, you will assign the weights based on the class distribution. <code>class_rep = [4,2]</code>. This is a simple but reasonable method to weight samples.</p>
<p>Another method to say if the object exists at least once in an image, then count this as a single sampling instance rather than X number of times it was repeated. It depends on your dataset.</p>

simines: <p>Thank you. Can it be more reasonable if we count the number of pixels per class?</p>

sudomaze: <p>Count the frequency of the label based on the area of the mask (counting the pixels) might give more accurate representation of frequency. However, it might create some bias toward object sizes.</p>
<p>This means an elephant in a single image might have the same frequency of appearing as an apple in 10 images.</p>
<p>It depends on your training procedure. You might try both of them and see what fits your task.</p>
Built-in mini-batch vs. concatenation of each output

HOJUN_LEE: <p>Hi folks,</p>
<p>I happened to not use mini-batch and dataloader in PyTorch as I do some work on multi-instance learning where I need to pass a bag of images with a single label with limited hardware specs. For normal case, I would definitely use the built-in dataloader because it’s probably more stable and efficient. Currently, I’m passing each image in a bag one by one, concatenating features from a backbone, and passing them to a separate binary classifier model.</p>
<p>The question rises when I try to process a “batch of outputs and labels” at the same time to stabilize the learning. So, my model basically processes 4 bags of 32 images sequentially using for loops, combine them as 1x4 tensor. The 1x4 tensors of output and label are then passed to an optimizer function (I know, it is super inefficient) to calculate gradients for the disjointed backbone and binary classifier model.</p>
<p>Would this approach (concatenation of sequential outputs) still mimic the mini-batching?</p>

sudomaze: <p>Hi Hojun,</p>
<p>This would be more like <a href="https://kozodoi.me/python/deep%20learning/pytorch/tutorial/2021/02/19/gradient-accumulation.html" rel="noopener nofollow ugc">accumulating a gradient</a> of 4 bags of 32 images unless you thought a single batch size would be 4*32. Here is an example of how you would need to do this.</p>
<pre><code class="lang-python">for bag in bags:
    accum_iter = len(bag)
    for inputs, labels in bag:
        # extract inputs and labels
        inputs = inputs.to(device)
        labels = labels.to(device)
            
        # forward pass 
        preds = model(inputs)
        loss  = criterion(preds, labels)
    
        # normalize loss to account for batch accumulation
        loss = loss / accum_iter 
    
        # backward pass
        loss.backward()

    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>This example is a refactoring of <a href="https://kozodoi.me/python/deep%20learning/pytorch/tutorial/2021/02/19/gradient-accumulation.html#3.-How-to-make-it-work" rel="noopener nofollow ugc">this post</a> to reflect your problem statement.</p>

HOJUN_LEE: <p>Thanks for the comment and the reference!</p>
Inference time for .ptl is more than .pt

vjayd: <p>IThere are two scenarios<br>
1)<br>
script_cell = torch.jit.script(self.model, [torch.rand(1, 3, 960, 640)])<br>
script_cell.save(“model1.pt”)</p>
<ol start="2">
<li>
</li>
</ol>
<p>traced_cell = torch.jit.trace(self.model, [torch.rand(1, 3, 960, 640)])<br>
traced_script_module = optimize_for_mobile(traced_cell)<br>
traced_script_module._save_for_lite_interpreter(“model2.ptl”)</p>
<p>Here I am observing that inference time of model2.ptl is slower than model1.pt. Could you please help me here? I believe it should be faster because we use save_for_lite_interpreter in case of model2.ptl</p>

ptrblck: <p>Are you comparing both models on a smartphone and if so how large is the difference in their execution time?</p>

vjayd: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> there is a difference of 2.5 seconds. Yes I am comparing the time on a Mobile Phone - MI note 7 pro 6GB RAM</p>
Use vgg19 to train my custom mnist

Ssangwong: <p>Hello I am a newbie and beginner <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"> i would like to modify vgg19 model to train mnist. how can i modify the input size of vgg19 model (224,224) to (28,28)?<br>
i want to modify this code <a href="https://pytorch.org/vision/0.12/_modules/torchvision/models/vgg.html#vgg19" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models.vgg — Torchvision 0.12 documentation</a>  …<br>
thanks for reading really thanks… <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You could start removing layers which would raise an error if they would create an empty output tensor and maybe relax the spatial pooling by e.g. also removing pooling layers more aggressively.<br>
Alternatively, you could increase the spatial size of your MNIST dataset and just use this model for your experiments.</p>

Ssangwong: <p>i appreciate your support really thanks for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
Default anchor sizes for faster r-cnn

SU801T: <p>I can’t seem to find this information anywhere, but wanted to know, if you do not define anchor sizes for your faster r-cnn model, then what is the default number/sizes in the PyTorch implementation?</p>

ptrblck: <p>By default <a href="https://github.com/pytorch/vision/blob/96aa3d928c6faaf39defd846966421679244412d/torchvision/models/detection/faster_rcnn.py#L37-L40">this <code>AnchorGenerator</code></a> would be used unless you pass another custom <code>AnchorGenerator</code> to <code>FasterRCNN</code>.</p>
Inconsistent tensor size, expected tensor [8] and src [1] to have the same number of elements, but got 8 and 1 elements respectively while running model

Lemuria: <p>I am trying to run an evolutionary model for lunar lander but i get this error when running the model<br>
it takes observation space of 8 and action space of 4<br>
HERE IS THE ERROR`Input In [7], in test_model(agent)<br>
10 while not done:<br>
11     params = unpack(agent[‘params’])<br>
—&gt; 12     probs = model(state,params)<br>
13     action = torch.distributions.Categorical(probs=probs).sample()<br>
14     state_,reward,done,info = env.step(action.item())</p>
<p>Input In [1], in model(x, params)<br>
3 def model(x,params):<br>
4     l1,b1,l2,b2,l3,b3 = params<br>
----&gt; 5     y = torch.nn.functional.linear(x,l1,b1)<br>
6     y=torch.relu(y)<br>
7     y = torch.nn.functional.linear(y,l2,b2)</p>
<p>RuntimeError: inconsistent tensor size, expected tensor [8] and src [1] to have the same number of elements, but got 8 and 1 elements respectively`</p>
<p>here is code generating the error<br>
`<br>
def model(x,params):<br>
l1,b1,l2,b2,l3,b3 = params<br>
y = torch.nn.functional.linear(x,l1,b1)<br>
y=torch.relu(y)<br>
y = torch.nn.functional.linear(y,l2,b2)<br>
y=torch.relu(y)<br>
y = torch.nn.functional.linear(y,l2,b3)<br>
y=torch.log_softmax(y,dim=0)<br>
return y</p>
<p>def unpack(params,layers=[(25,8),(10,25),(4,10)]):<br>
`</p>

ptrblck: <p>Check the sahpes of <code>x</code>, <code>l1</code>, and <code>b1</code> and make sure they are compatible with the <code>linear</code> layer as the shape mismatch is raised in: <code>y = torch.nn.functional.linear(x,l1,b1)</code>.</p>
Loss term using gradients

MattyMatt: <p>Hi,</p>
<p>I am working on a classification problem using a typical L2 loss. Is there a proper way of adding a loss term which is a function of gradients? For example, lets say I want the gradients of certain layer to sum up to 1:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8e0f8ebd620df5db7670c83e2aa1f8c54fbe38b.png" alt="losses" data-base62-sha1="o5YdozmECIETs2LVWGpRKXpz3GX" width="514" height="214"></p>
<p>What would be the proper way of implementing that in pytorch?</p>
<pre><code class="lang-auto">x1 = layer1(inputs)
x2 = layer2(x1)
x2.retain_grads()
preds = layer3(x2)

loss_l2 = l2(preds, labels)
loss_l2.backward() # get the gradients

loss_grads = (1-x2.grads.sum())**2

loss = loss_l2 + loss_grads

loss.backward()
optimizer.step()
</code></pre>
<p>Would that be ok? Or it wouldn’t because we are taking second derivative here and one should rather provide gradients of loss_grads manually?</p>
Improving the running time of the libtorch snippet

Mircea_Paul_Muresan: <p>The issue which I am trying to solve, is to improve the running time of the object extraction. I would like to know if there is a more efficient way to do the portion of code bellow(since I observed it takes the a lot of time ).  The variable yolo_output is the output of the yolov5 neural net.</p>
<pre><code>  int confidenceIndex=4;
  int labelStart=5;
   torch::Tensor conf = yolo_output.squeeze(0).
        index({Slice(None),Slice(confidenceIndex,confidenceIndex+1)})
        .squeeze(1);

torch::Tensor keep = torch::gt(conf,0.3);
yolo_output = yolo_output.index({Slice(None),keep,Slice(None)}); 
torch::Tensor scores =  yolo_output.squeeze(0).      
        index({Slice(None),Slice(labelStart,None)});
torch::Tensor score_pos = scores.argmax(1);
scores = std::get&lt;0&gt;(torch::max(scores,1,false));

torch::Tensor final_keep = scores * yolo_output.squeeze(0).
        index({Slice(None),Slice(confidenceIndex,confidenceIndex+1)})
        .squeeze(1);

scores=final_keep;                              
 if(final_keep.numel() == 0)
  {
      std::vector&lt;yolo_element&gt; yolo_elements;
      return yolo_elements;
  }
final_keep = torch::gt(final_keep,0.3);
yolo_output = yolo_output.index({Slice(None),final_keep,Slice(None)}); 
scores = scores.index({final_keep});

if(scores.numel() == 0)
{
    std::vector&lt;yolo_element&gt; yolo_elements;
    return yolo_elements;
}
score_pos = score_pos.index({final_keep});
yolo_output=yolo_output.to(torch::kCPU);
float *out = yolo_output.data_ptr&lt;float&gt;();
int v1,v2;
v1 = yolo_output.size(1);
v2 = yolo_output.size(2);
std::vector&lt;yolo_element&gt; yolo_elements(v1);

for(int i = 0;i &lt; v1;i++)
{
    int index=i*v2;
    int tx = (out[index]-out[index+2]/2);
    int ty = (out[index+1]-out[index+3]/2)* 0.9375;
    int bx = (out[index]+out[index+2]/2);
    int by = (out[index+1]+out[index+3]/2) * 0.9375;

    yolo_element el = {Point(tx,ty),Point(bx,by),scores[i].item&lt;float&gt;(),score_pos[i].item&lt;float&gt;()};
    yolo_elements[i] = el;

}
return yolo_elements;
</code></pre>

ptrblck: <p>I would recommend to debug the issue e.g. via <code>gdb</code> and check the backtrace to see which operation exactly fails and what the reason is. The model should not randomly fail because some samples are “difficult” than others.</p>

Mircea_Paul_Muresan: <p>Thanks for the suggestion. The issue was because the GPU was overloaded(because after the program crashed I did not clean what was loaded on the GPU and ran the program again…) and result was not verified to see if it was empty or not in the next function.</p>