akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there a masking method for CNN input of variable size within a batch?

weedwind: <p>I am using a resnet to do phone recognition from spectrograms. My input is a tensor of shape N C H W, where N is the batch size. But since the speech has different length in time, this causes H to be different in a batch, and I cannot crop the input because they are speech data… Right now, I have to pad zeros within a batch. I am wondering is there a similar way as packed padded sequence in RNN to let all modules in the network know that these  zeros are useless, do not compute the gradient of any parts resulted from them?</p>

jdhao: <p>Hi, if you are not familiar with PyTorch, I suggest you reading this <a href="http://pytorch.org/tutorials/beginner/data_loading_tutorial.html">data loading tutorial</a> first.</p>
<p>I am assuming each of your speech data is just a vector, right? But the vector length is not all the same. PyTorch will by default try to pack the samples in a batch to form a tensor. But if your samples in a batch have variable size, the packing of samples will fail.</p>
<p>The <code>Dataloader</code> class has a parameter called <code>collate_fn</code> which controls how samples in a batch should be packed together. For example you can just store the samples in a batch in a list, with each element a speech sample. Store the corresponding label in a Tensor,  For more info, refer to this <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/3">post</a>.</p>

weedwind: <p>Hi, jdhao,</p>
<p>Thank you for your reply. But my real question is: suppose I already formed a batch of N, C, H, W, and in order to make this batch, I padded some zeros along the H dimension to some of the images, how do I let the CNN know that these zeros are useless, do not compute gradients resulted from these zeros in all the subsequent modules?</p>
<p>Thank you so much</p>

jdhao: <p>I think it is not possible, at least for now. You can record the padding information for each image and load the padding info along with the image batch. Then you can recover the original image from the padding info. In this way, the CNN will not waste time and computation with the padded zeros in the image.</p>
<p>But if you recover the original image, the image samples in a batch may become variable size. You can not feed them to the CNN in a whole and you have to feed the image on by one to the CNN. Since you haven’t given any specific information about the input to the network. I can only give the above suggestions.</p>

wizardk: <p>Hi weedwind,</p>
<p>I didn’t see any masking work in your description. I am doing like this : <a href="https://discuss.pytorch.org/t/padding-and-masking-in-convolution/25564">Padding and masking in convolution</a></p>
<p>Currently, my model seems working well but I have some questions about gradient too.</p>

suke: <p>I have a question, collate_fn seems can control batch, but can not control input shape size, if I want to get variable input feature map size(h&amp;w), how should I do?</p>
Shared Randomness when using SubsetRandomSampler

JayanthRR: <p>My simplified use case is as follows. I have two training instances. In the first instance, given a dataset, I create a mask to choose a subset of the data</p>
<pre><code class="lang-auto">loader1 = DataLoader(dataset, sampler=SubsetRandomSampler(mask1), batch_size=64)
</code></pre>
<p>I create another training instance with the same dataset (but one data point changed/corrupted), and follow the same procedure by creating the loader. How do I ensure that in these two training instances, I generate the same mini batches for each call. I essentially need to measure the divergence between the two model weights when the order in which the data processed remains the same (just one data point is different).</p>
Modifying an existing Architecture for Multiple Outputs

AmardeepGanguly: <p>While implementing a model from scratch I had no trouble with setting multiple outputs(2 in my case and backpropagating the combined loss) . The last fully connected layer has 10  neurons and I need to have two multiple outputs of 10 neurons. This is my code.</p>
<pre><code class="lang-auto">class LeNet(nn.Module):
    def __init__(self):
            super(LeNet,self).__init__()
            self.cnn_model=nn.Sequential(nn.Conv2d(1,6,5),
            nn.ReLU(),
            nn.AvgPool2d(2,stride=2),
            nn.Conv2d(6,16,5),
            nn.ReLU(),
            nn.AvgPool2d(2,stride=2))
            self.fc_model=nn.Sequential(
            nn.Linear(400,120),
            nn.LeakyReLU(),
            nn.Linear(120,84),
            nn.LeakyReLU(),
            nn.Linear(84,10))
    def forward(self,x):
        x1=self.cnn_model(x)
        x1=x1.view(x1.size(0),-1)
        x1=self.fc_model(x1)
        x2=self.cnn_model(x)
        x2=x2.view(x2.size(0),-1)
        x2=self.fc_model(x2)
        
        
        return x1,x2
</code></pre>
<p>My question is how should I change any existing architecture like resnet18,vgg16 to have multiple outputs.(Note that the number of classes are 10).Kindly provide code snippets.</p>
Concatenating two tensors to a 3-D dimensional tensor

AmardeepGanguly: <p>I have two Tensors of size [60,10] how do i concatenate them to get a Tensor of size[60,2,10]</p>

acobobby: <p>Call <code>x</code> and <code>y</code> your tensors, you can use <code>stack</code>:</p>
<pre><code class="lang-auto">torch.stack( (x,y), dim=1 )
</code></pre>

AmardeepGanguly: <p>Thanks Andrea:smile:</p>
Why is the validation loss and IoU jumping?

Christian_Ses: <p>After training a network for a lot of epochs the loss and IoU is constantly changing over time (by jumping). Do you have any idea for fixing it ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9059aeb01729810b328c0f0c0947a8a1c51a4079.png" data-download-href="https://discuss.pytorch.org/uploads/default/9059aeb01729810b328c0f0c0947a8a1c51a4079" title="IoU"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9059aeb01729810b328c0f0c0947a8a1c51a4079_2_666x500.png" alt="IoU" data-base62-sha1="kAYTL5gvf8ZMg6311iD3Z1iJBTX" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9059aeb01729810b328c0f0c0947a8a1c51a4079_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9059aeb01729810b328c0f0c0947a8a1c51a4079_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/9/9059aeb01729810b328c0f0c0947a8a1c51a4079.png 2x" data-dominant-color="FBEFE4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IoU</span><span class="informations">1280×960 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Christian_Ses: <p>Loss: <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c01bff078733dc1ef6ee8cd9afdc838ca58d2ca2.png" data-download-href="https://discuss.pytorch.org/uploads/default/c01bff078733dc1ef6ee8cd9afdc838ca58d2ca2" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c01bff078733dc1ef6ee8cd9afdc838ca58d2ca2_2_666x500.png" alt="loss" data-base62-sha1="rptEFnFaeY5Nz5LXgc3pqEefyzU" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c01bff078733dc1ef6ee8cd9afdc838ca58d2ca2_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c01bff078733dc1ef6ee8cd9afdc838ca58d2ca2_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/c01bff078733dc1ef6ee8cd9afdc838ca58d2ca2.png 2x" data-dominant-color="FBEFE4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1280×960 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

alex.veuthey: <p>How many samples do you have in total (train and valid separately)? What is your batch size (for both train and valid if they are different)?</p>
<p>It looks like some batches are easier than others, it can happen more easily if you have small batch sizes and datasets where difficulty is very varying.</p>
<p>You could try to average those values a bit, see if there’s an increasing (IoU) or decreasing (loss) trend, but at first sight it looks like it’s not learning much…</p>
How to visualize mask and bboxes from object detection tutorial

Diyago: <p>I have been following this official guide <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>  and would like to visualise bboxes and ground truth mask. How can I do that?</p>
<p>Target looks this way:<br>
{‘boxes’: tensor([[849., 525., 959., 635.],<br>
[581., 659., 737., 867.],<br>
[890., 734., 959., 813.],<br>
[632., 839., 855., 945.]]),<br>
‘labels’: tensor([1, 1, 1, 1]),<br>
‘masks’: tensor([[[0, 0, 0,  …, 0, 0, 0],<br>
[0, 0, 0,  …, 0, 0, 0],<br>
[0, 0, 0,  …, 0, 0, 0],<br>
…,<br>
[0, 0, 0,  …, 0, 0, 0],<br>
[0, 0, 0,  …, 0, 0, 0],<br>
[0, 0, 0,  …, 0, 0, 0]],</p>
<pre><code>     [[0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      ...,
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0]],

     [[0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      ...,
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0]],

     [[0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      ...,
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0],
      [0, 0, 0,  ..., 0, 0, 0]]], dtype=torch.uint8),
</code></pre>
<p>‘image_id’: tensor([1]),<br>
‘area’: tensor([12100., 32448.,  5451., 23638.]),<br>
‘iscrowd’: tensor([0, 0, 0, 0])}</p>
<p>I would like to get something like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a36445793b7e1deb79dabd02490256812e0578db.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a36445793b7e1deb79dabd02490256812e0578db" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a36445793b7e1deb79dabd02490256812e0578db_2_690x333.jpeg" alt="image" data-base62-sha1="njqENPXl3CyLQkvg48P2f5W7Kj1" width="690" height="333" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a36445793b7e1deb79dabd02490256812e0578db_2_690x333.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/a/a36445793b7e1deb79dabd02490256812e0578db.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/a36445793b7e1deb79dabd02490256812e0578db.jpeg 2x" data-dominant-color="67665D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">906×438 168 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Why not modify forward hook output?

kaladin: <p>In the PyTorch documentation under <code>register_forward_hook</code>, it says “The hook should not modify the input or output.” (<a href="https://pytorch.org/docs/master/nn.html#torch.nn.Module.register_forward_hook" rel="nofollow noopener">https://pytorch.org/docs/master/nn.html#torch.nn.Module.register_forward_hook</a>)</p>
<p>What would be the danger of modifying the output? This could be done in-place, and I was wondering what would be the consequences of doing this? (i.e. what would break?)</p>
Grad_fn object is not callable with custom loss function

ljj7975: <p>I had to create a custom loss function and it had to be <code>torch.autograd.Function</code> wrapped around <code>torch.nn.module</code></p>
<p>I wanted to verify that my code is programmatically correct so I called grad_fn manually for testing purpose as following<br>
<code>loss.grad_fn(torch.tensor(1).float())</code></p>
<p>When I run above line of code using <code>torch.nn.L1Loss</code>, it correctly returns gradients</p>
<p>However, when I execute the same line with my custom function. I am getting</p>
<pre><code class="lang-auto">loss
 tensor(0.2361, grad_fn=&lt;CustomLossBackward&gt;)
&lt;torch.autograd.function.CustomLossBackward object at 0x8211d9240&gt;
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-511-a9c2beed2c03&gt; in &lt;module&gt;
     32 print(loss.grad_fn)
     33 
---&gt; 34 grad = loss.grad_fn(torch.tensor(1).float())
     35 print('grad\n', grad)

TypeError: 'CustomLossBackward' object is not callable
</code></pre>
<p>Does anyone know why this error is happening??<br>
I see that CustomLossBackward has type of <code>torch.autograd.function.CustomLossBackward</code><br>
(<code>&lt;torch.autograd.function.CustomLossBackward object at 0x8211d9240&gt;</code>)</p>
<p>while L1LossBackward is on its own as <code>L1LossBackward</code><br>
(<code>&lt;L1LossBackward object at 0x82112b470&gt;</code>).</p>
<p>Not sure if this means something</p>