akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What's wrong with output layer and loss function?

Sahil_malik: <p>I have multi-class dataset and labels values range is 5 to 9. I am using SoftMax function, and sparse_categorical_crossentropy loss function.</p>
<p>When I give 10 units in the dense layer it works fine, although data has only 5 classes.<br>
Why it only works with 10 but not with 5 units in output layer?<br>
How to get it done for 5 units in output layer?</p>
<p>Thank you!</p>

Dwight_Foster: <p>It only works with ten because your labels are 5 to 9 so the 9 requires the dense layer to have 10 outputs. If you wanted to change this you would have to lower your labels to a range of 0 to 4.</p>

Sahil_malik: <p>Thank you… How to do it?</p>

Dwight_Foster: <p>Where do you define your labels? You could just subtract 5 from each label.</p>
Bad object classification with faster-rcnn

Chame_call: <p>Hey.</p>
<p>I’m using faster-rcnn for custom object detection.<br>
If the network results in bad classification (the bounding box is classified correctly with a right category but has a low probability or the bounding box is not classified correctly)  but not so bad detection (it finds most of the correct bounding boxes plus some incorrect ones) can I freeze all the network except for last classifier stage and train only it or it’s more complex than that?</p>
<p>Thanks.</p>

Dwight_Foster: <p>Yes when you define the optimizer you can do this</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.roi_heads.box_predictor.cls_score.parameters())
</code></pre>
<p>and it will only train the cls_score layer.</p>

Chame_call: <p>Thanks for your answer.</p>
<p>I have another question.<br>
I see the following lines of code of faster-rcnn in torchvision:</p>
<blockquote>
<p>anchor_sizes = ((32,), (64,), (128,), (256,), (512,))<br>
aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)</p>
</blockquote>
<p>I’m trying to detect manufacter logos/text labels.<br>
They always have small enough size.<br>
Does it mean that I should try to decrease my the rpn anchor_sizes for that case ( since at the moment I also have a problem finding small size bounding boxes) or this settings shouldn’t be changed?</p>

Dwight_Foster: <p>It should still be able to learn smaller boxes but you could mess around with them just to see.</p>
How to get middle layer gradient for batch input?

AllenD1: <p>Hi, I am trying to acquire the gradient in the last hidden layer for batch inputs.</p>
<p>Let says the last layer is a linear layer (512*100) and 100 is the number of classes.</p>
<p>For each sample, the gradient would be 100*512</p>
<p>Currently, my code is as follows.</p>
<pre><code class="lang-auto">losses = self.criterion_without_reduction(outputs, targets)
gradient_batch = []
for loss in losses:
    loss.backward(retain_graph=True)
    gradient_list = reversed(list(model.net.named_parameters()))
    # only output the gradient for the last layer
    for name, parameter in gradient_list:
        if 'weight' in name:
            gradient = parameter.grad.clone() # [column[:, None], row].resize_(100,100)
            gradient = gradient.unsqueeze_(0)
            gradient_batch.append(gradient.unsqueeze_(0))

            break
gradient_batch = torch.cat(gradient_batch, dim=0)
</code></pre>
<p>The above code will generate a tensor ([batch_size, 1, 100, 512])</p>
<p>However, the code runs relatively slow and I am wondering whether there is a more efficient way to do it?</p>

vaibhavballoli: <p>You could access the gradient directly: <code>model.gradient = model.net.fc.weight.grad.clone()</code> (replace fc with the name of your classification layer) The speed is probably an issue with the <code>loss.backward()</code> stage, but this step is probably not the bottleneck.</p>

AllenD1: <p>Hi,</p>
<p>Forgive me if I make stupid mistakes…</p>
<p>I am running the code in the eval() mode and trying to get the gradient matrix for each input x, respectively.</p>
<p>For example, if we have 128 inputs (in a batch), we will get 128 different gradient matrixes.</p>
<p>I try to use model.net.fc.weight.grad.clone() to get the gradient, but the model.net.fc.weight.grad is a NoneType object…</p>
<p>Is there any solution for that?</p>
<p>Thank you in advance!</p>

vaibhavballoli: <p>Ideally you should be running it after <code>model.train()</code>, but in this case there shouldn’t be any problem. I suppose the <code>model.net.fc</code> call should exist, if it does, I suspect the <code>backward()</code> call isn’t affecting the fc layer. Since your first code does work, I suspect you’re trying to access the classification module with the wrong name(fc in this case).</p>
RuntimeError: The size of tensor a (1000) must match the size of tensor b (10) at non-singleton dimension 1

Vipul_Mehra1: <p>Hi Guys,</p>
<p>I have a pre-trained resnet-50 model, which upon training gives this error: The size of tensor a (1000) must match the size of tensor b (10) at non-singleton dimension 1.<br>
I am using MSELoss.</p>
<pre><code class="lang-auto">
for epoch in range(n_epochs):
    N = len(train_loader)
    epoch_train_loss, epoch_test_loss = 0, 0
    val_age_mse, ctr = 0, 0
    _n = len(train_loader)
    for ix, data in enumerate(train_loader):
        # if ix == 100: break
        **loss = train_batch(data, model, optimizer, criteria)** (This is where error shows up)
        epoch_train_loss += loss.item()
</code></pre>
<p>This is where error crops up:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-103-01cf2dc1c2bf&gt; in &lt;module&gt;()
     30     for ix, data in enumerate(train_loader):
     31         # if ix == 100: break
---&gt; 32         loss = train_batch(data, model, optimizer, criteria)
     33         epoch_train_loss += loss.item()
     34 

4 frames
/usr/local/lib/python3.7/dist-packages/torch/functional.py in broadcast_tensors(*tensors)
     69         if any(type(t) is not Tensor for t in tensors) and has_torch_function(tensors):
     70             return handle_torch_function(broadcast_tensors, tensors, *tensors)
---&gt; 71     return _VF.broadcast_tensors(tensors)  # type: ignore
</code></pre>

Dwight_Foster: <p>Can you show your train_batch function and the entire error. Can you also print out the shapes of your inputs and labels.</p>
Issue comparing two tensors with isclose

Miguel: <p>Hi,<br>
I am working on a classification network that should determine whether two graphs are equal or not. In order to do so, I forward pairs of graphs that are exactly equal (1) or different (0) and generate a vector embedding (of length 64) for each input. The two vectors are then concatenated and forwarded through a classification sub network that makes the final prediction. Note that the embedding generation is carried out in the same way for both inputs.</p>
<p>My issue comes from the fact that I am getting false negative (equal input graphs classified as 0) samples when assessing the accuracy of the model. After some examination, I have seen that when two equal graphs are forwarded to the network, torch.isclose sometimes finds differences between the vector embeddings at some positions, but they should be identical as the vector generation process is exactly the same for both inputs (they use the same layers declared in the network). If I print the embeddings I cannot see differences between them and I made sure to call detach in order to get rid of computational graphs beforehand. Does someone know what may be going on?</p>
<p>To illustrate:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f212f58266c7555c54d7c2b41d45610bbeff918f.png" data-download-href="https://discuss.pytorch.org/uploads/default/f212f58266c7555c54d7c2b41d45610bbeff918f" title="Screenshot from 2021-02-23 22-13-37"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f212f58266c7555c54d7c2b41d45610bbeff918f_2_690x499.png" alt="Screenshot from 2021-02-23 22-13-37" data-base62-sha1="yxuanluKroDxzvUbRYTQ1D7QZ9Z" width="690" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f212f58266c7555c54d7c2b41d45610bbeff918f_2_690x499.png, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f212f58266c7555c54d7c2b41d45610bbeff918f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f212f58266c7555c54d7c2b41d45610bbeff918f.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-23 22-13-37</span><span class="informations">958×693 136 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see above, two positions are marked with False in the output of torch.isclose, but the values are the same (-0.0237 at position 15 and 0.0252 at position 27). Obviously, torch.allclose returns False and the euclidean distance between the two vectors is 1.7302e-06. Even though the L2 distance is very low and could be considered 0, the network classifies the sample as 0 (0.49 probability) when it should be 1.</p>
<p>Thanks!</p>
How to disactivate the require_grad for a specific element in the tensor?

HaFred: <p>I am trying to fix some specific elements in the grad_output tensor as below:</p>
<pre><code class="lang-auto">_IDX_LIST = [2, 4, 6, 9]

class GradZero(InplaceFunction):

    @staticmethod
    def forward(ctx, input):
        ctx.inplace = False
        return input

    @staticmethod
    def backward(ctx, grad_output):

        with torch.no_grad():
            grad_input = grad_output.detach().clone()
            grad_input = fix_into_zero(grad_input)

        return grad_input, None, None

def fix_into_zero(x, idx_list=_IDX_LIST):
    x[idx_list] = 0.0
    x[idx_list].requires_grad = False

    return x
</code></pre>
<p>And then I found that for <code>for param in test_model.parameters():        print("{}'s requires_grad is {}".format(param.shape, param.requires_grad))</code>, all the param stay as <code>requires_grad = True</code>. Maybe it is as the element itself has no <code>require_grad</code> attribute?<br>
In such case how could I turn off the autograd for some specific elements as above snippet? I also referred to this post (<a href="https://discuss.pytorch.org/t/how-can-i-disable-all-layers-gradient-expect-the-last-layer-in-pytorch/53043" class="inline-onebox">How can I disable all layers gradient expect the last layer in Pytorch?</a>) but cannot figure out.</p>

KFrank: <p>Hi Fred!</p>
<aside class="quote no-group" data-username="HaFred" data-post="1" data-topic="107689" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hafred/48/27581_2.png" class="avatar"> HaFred:</div>
<blockquote>
<p>In such case how could I turn off the autograd for some specific elements as above snippet?</p>
</blockquote>
</aside>
<p>You can’t.  The whole <code>Tensor</code> is the <code>Parameter</code>, so its <code>requires_grad</code><br>
property applies to the whole <code>Tensor</code>, not just to individual elements.</p>
<p>The safest way to accomplish what you want is to store the values of<br>
the tensor elements you want to freeze, call <code>opt.step()</code>, and then<br>
rewrite the stored values back into the tensor.</p>
<p>(Some optimization algorithms will change elements even if their<br>
gradients are zero.  If you <em>know</em> that your optimizer will <em>not</em> change<br>
zero-gradient elements, you can zero the gradients of the elements<br>
you want to freeze before calling <code>opt.step()</code>.  But you risk breaking<br>
your freezing if you change the details of your optimizer.)</p>
<p>Best.</p>
<p>K. Frank</p>

localh: <p>I am wondering if you can post a quick dummy example to showcase this? I am interested in this method!</p>
Best way to run unique_consecutive() on certain dimension?

galactica147: <p>What’s the best way to call <code>unique_consecutive()</code> on certain dimension of a tensor, and pad the left-out with specified values?<br>
For simplicity, we could use 2D tensor as an example:</p>
<pre><code class="lang-auto">input = tensor([[3, 3, 5, 5, 5],
                [3, 3, 2, 2, 3]])
</code></pre>
<p>if specifying padding value -1, what i hope to get is:</p>
<pre><code class="lang-auto">output = tensor([[3, 5, -1, -1, -1],
                 [3, 2, 3, -1, -1]])
</code></pre>
<p>Also, to save memory, i’d prefer to do so in-place if possible.<br>
Thanks!</p>

Eta_C: <p>Try</p>
<pre><code class="lang-python">x = tensor([[3, 3, 5, 5, 5],
            [3, 3, 2, 2, 3]])
unique_x, indices = torch.unique_consecutive(x, return_inverse=True)
indices -= indices.min(dim=1, keepdims=True)[0]
result = -torch.ones_like(x)
result = result.scatter_(1, indices, x)
</code></pre>

galactica147: <p>Thanks! This is working well!</p>
<p>Could you elaborate a bit what <code>indices -= indices.min(dim=1, keepdims=True)[0]</code> is doing?</p>
<p>This solution creates another tensor for the updated result, which is fine. I wonder if there is any in-place way that can save the extra memory cost.</p>

Eta_C: <aside class="quote no-group" data-username="galactica147" data-post="3" data-topic="112662">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/galactica147/48/19682_2.png" class="avatar"> galactica147:</div>
<blockquote>
<p>Could you elaborate a bit what <code>indices -= indices.min(dim=1, keepdims=True)[0]</code> is doing?</p>
</blockquote>
</aside>
<p>See,</p>
<pre data-code-wrap="python"><code class="lang-python">indices = [[0, 0, 1, 1, 1],
           [2, 2, 3, 3, 4]]
</code></pre>
<p>But here <code>scatter</code> needs <strong>col index</strong>.</p>
<pre data-code-wrap="python"><code class="lang-python">indices = [[0, 0, 1, 1, 1],
           [0, 0, 1, 1, 2]]
</code></pre>
<aside class="quote no-group" data-username="galactica147" data-post="3" data-topic="112662">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/galactica147/48/19682_2.png" class="avatar"> galactica147:</div>
<blockquote>
<p>I wonder if there is any in-place way that can save the extra memory cost.</p>
</blockquote>
</aside>
<p>I have no idea now.</p>

galactica147: <p>This is helpful! Thanks a lot!</p>
Multi-label classification using OneVersusOne (or OneVersusAll) technique

arhouati: <p>Hello All,</p>
<p>To avoid overlapping in multilabel text classification, I need to implement the OneVersusAll technique into a Bert finetuned Model.</p>
<p>This technique exists in SCkLearn lib as follow :</p>
<pre><code class="lang-auto">from sklearn.multiclass import OneVsOneClassifier
class UNERLinearModel(BertPreTrainedModel):
     def __init__(self):
               ...
               self.entity_classifier = OneVsOneClassifier(LinearSVC(random_state=0))
               ...
     
    def _forward_train(self):
             ... 
              entity_clf = self.entity_classifier.fit(entity_repr.detach(), entity_types.detach()).predict(entity_repr.detach())
             ...
</code></pre>
<p>In this case, every time the method “fit” will be called, it erases all learned params and learns new ones using a small part of data. So, I cannot use it in my model.</p>
<p>I didn’t found anything similar in Pytorch.  Any suggestion on how to implement the OneVersusOne (or OneVersusAll) technique?</p>
<p>Thanks in advance</p>
Get tensor value at some coordinate along batch with loader giving out dicts

enterthevoidf22: <p>my dataloader produces dictionaries as outputs, which are input for a neural net.</p>
<p>let’s say:</p>
<pre><code class="lang-auto">def dummy_get_item():
image = torch.rand((1,64,64))
inpainted_image = image.clone()
mask_locs = torch.tensor([5,10,15,22]) #randomly chosen
inpainted_image[0,mask_locs[0]:mask_locs[1],mask_locs[2]:mask_locs[3]] = 0
return {'inpainted_image':inpainted_image,'truth_image':image,'locs':mask_locs}
</code></pre>
<p>i want to be able to measure loss for only the values i silenced, but it’s tricky because from each sample in the batch i’m trying to collect different coordinates<br>
say batch_size = 4</p>
<pre><code class="lang-auto">input_dict['inpainted_image'].shape = [4,1,64,64]
input_dict['mask_locs'].shape = [4,4]
</code></pre>
<p>i try:</p>
<pre><code class="lang-auto">ind = input_dict['mask_loss']
loss = criterion(input_dict['truth_image'][:,ind[:,0]:ind[:,1],ind[:,2]:ind[:,3]],input_dict['inpainted_image'][:,ind[:,0]:ind[:,1],ind[:,2]:ind[:,3]])
</code></pre>
<p>what i’m trying to do is compare the values of “truth image” and “inpainted image” only at the locations where it was masked.</p>
<p>i’m getting the following error:<br>
TypeError: only integer tensors of a single element can be converted to an index</p>
<hr>
<p>update - i figured out i can solve this by doing<br>
locs = (inpainted_image==0) * truth_image!=0)<br>
and then solve the batch issue.</p>
<p>but i still would love to know how can i solve it in the logic i presented before, meaning how to overcome the error i got.</p>
Export PyTorch model to onnx object without the intermediate saving and loading of an .onnx file

vaibhavballoli: <p>While <code>torch.onnx.export(model, input, 'model.onnx')</code> saves to a file, is there a way I can get the model instead of loading it back again as <code>onnx_model = onnx.load('model.onnx')</code> ? TIA</p>
Question about soloV2

Sylvain_Ard: <p>Hi all !</p>
<p>I have trained my dataset with the code :</p>
<aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/WXinlong/SOLO" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/22233070?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/WXinlong/SOLO" target="_blank" rel="noopener nofollow ugc">WXinlong/SOLO</a></h3>


  <p><span class="label1">SOLO and SOLOv2 for instance segmentation, ECCV 2020 &amp;amp; NeurIPS 2020.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>but I have always only one instance where I should have several instances.</p>
<p>I join my JSON annotations files so you could verify if it is correct.</p>
<p>Please help me !</p>
<p>My JSON files :</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://cfl.dropboxstatic.com/static/images/favicon-vflUeLeeY.ico" class="site-icon" width="16" height="16">
      <a href="https://www.dropbox.com/sh/ezmtwn6x5kn8dgj/AABr5f6aL-bs5evIX_KvjgWga?dl=0" target="_blank" rel="noopener nofollow ugc">Dropbox</a>
  </header>
  <article class="onebox-body">
    <img src="https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-folder_dropbox-large.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://www.dropbox.com/sh/ezmtwn6x5kn8dgj/AABr5f6aL-bs5evIX_KvjgWga?dl=0" target="_blank" rel="noopener nofollow ugc">annotations</a></h3>

<p>Shared with Dropbox</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Best regards</p>
How to use MNIST model

bolt25: <p>After writing the code for MNIST handwritten digits, how can I use it to predict new images?<br>
I read the image using PIL and now I’m stuck.<br>
I don’t know how to convert it into 28*28 and also to gray scale in the script.<br>
Is the any way to do this conversion it in the script or do I have to do it manually for every image?</p>

ptrblck: <p>Are you using the <code>torchvision.datasets.MNIST</code> dataset or your “own” handwritten images?<br>
In the former case you could stick to the <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" rel="nofollow noopener">MNIST example</a>. In the latter case, you could load your images as grayscale images using <code>Image.open(PATH).convert('L')</code> and apply your transformations on this image directly.</p>

bolt25: <p>Oh okay… But what about converting it into 28*28</p>

ptrblck: <p>If you’re loading the images using <code>PIL</code>, you can use <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Resize" rel="nofollow noopener">torchvision.transforms.Resize</a>.<br>
As you can see in the example, you can compose several transformations using <code>torchvision.transforms.Compose</code>.</p>

bolt25: <p>Is it advisable to just resize or to crop and then resize? Because on just resizing the digit becomes too small, and if I crop and then resize,the model is not able to correctly predict the outcome.<br>
Any comments?</p>

zwdnet: <p>This is my code to transform my handwriting picutures:</p>
<pre><code class="lang-auto">def changeData(path = "./mynum/num/"):
    # path = "./mynum/num/"
    dir_or_files = os.listdir(path)
    files = []
    for dir_file in os.listdir(path):
        files.append(dir_file)
        
    MNIST_SIZE = 28
    datas = []
    labels = []
    for file in files:
        # transform pictures
        # print(path+files[0])
        # read the image and trun to gray
        img = io.imread(path+file, as_gray=True)
        # resize to 28*28
        translated_img = transform.resize(img, (MNIST_SIZE, MNIST_SIZE))
        # change to 1*784 array
        flatten_img = np.reshape(translated_img, 784)
        # 1 is black，0 is whtie
        result = np.array([1 - flatten_img])
        # get the label
        labels.append([int(file[-5])])
        datas.append(result)
        print(file)
    datas = np.array(datas)
    labels = np.array(labels)
    return datas, labels
</code></pre>
<p>The picture’s name is set like “03_3.png”.The 03 before _ is ID, the 3 after _ 3 is the real number of the picture.<br>
<a href="https://github.com/zwdnet/mnist/blob/main/mnist.py" rel="noopener nofollow ugc">The full code link.</a></p>
Does Dataloader support dynamic BatchSize?

PistonY: <p>How can I adjust batch size while training?</p>

Eta_C: <p>Hey</p><aside class="quote" data-post="1" data-topic="13840">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hij/48/14113_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-for-variable-batch-size/13840">Dataloader for variable batch size</a> 
  </div>
  <blockquote>
    Hi I am new to this and for most application I have been using the dataloader in utils.data to load in batches of images. However I am now trying to load images in different batch size. For example my first iteration loads in batch of 10, second loads in batch of 20. 
Is there a way to do this easily? Thank you.
  </blockquote>
</aside>
Different training loss while training and while evaluating

saul_santos: <p>HI. I realized my training loss is slightly different from the one that is calculated during training. Even though I added a full batch to make sure it’s not an average of the losses in each mini-batch. However, the validation loss gives the same value. I’m not using any fancy stuff such as batch normalization or so.</p>

CedricLy: <p>I had the same issue.<br>
check this thread.<br>
<a href="https://discuss.pytorch.org/t/loaded-model-is-not-the-same/107339/3">Loaded model is not the same - PyTorch Forums</a></p>

saul_santos: <p>Thank you. However, I’m not using fancy stuff like dropout layers. The only thing I’m using is to include gradients via torch.autograd in the cost function which are calculate inside the nn.module and they are used both in eval and train.</p>
How to implement this tricky scatter?

zcajiayin: <p>Suppose I have a tensor <code>A = [row, col]</code>, for example:</p>
<pre><code class="lang-auto"> tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1],
         [1, 1, 1, 1, 1, 1, 1, 1, 1],
         [1, 1, 1, 1, 1, 1, 1, 1, 1]])
</code></pre>
<p>Now, the values in some of rows of <code>A</code> will be modified to value 0.</p>
<p>I have a index tensor <code>B</code> specifying which column the value should be modified and a <em>‘indicator’</em> tensor <code>C</code> indicating which row (the row in <code>C</code> with value 1) should be modified. For example, <code>C = [[1], [0], [1]]</code> and <code>B = [[1], [7]]</code> are saying that the first row and the last row of  and the corresponding column 1 and 7 of <code>A</code> will be changed to value 0:</p>
<pre><code class="lang-auto"> tensor([[1, 0, 1, 1, 1, 1, 1, 1, 1],
         [1, 1, 1, 1, 1, 1, 1, 1, 1],
         [1, 1, 1, 1, 1, 1, 1, 0, 1]])
</code></pre>
<p>How should I do this using pytorch?</p>
<p>Since slicing will yield in-place result, I would prefer inplement it using <code>scatter</code>.</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">x = torch.tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1],
                  [1, 1, 1, 1, 1, 1, 1, 1, 1],
                  [1, 1, 1, 1, 1, 1, 1, 1, 1]])

C = torch.tensor([[1], [0], [1]]).squeeze(1).nonzero().squeeze(1)
B = torch.tensor([[1], [7]]).squeeze(1)
x[C, B] = 0
print(x)
</code></pre>
<p>Note that I kept your original shapes and had to use the <code>squeeze</code> operations. If you could create the indices in the expected shape, you could of course remove them.<br>
Also, I’ve converted the bool index mask <code>C</code> to a value index in order to be able to use <code>C</code> and <code>B</code> together.</p>
How to deal with multiple dataloaders?

ysleer: <p>Hi.</p>
<p>I am trying to train a multiple dataloader using multiprocessing.</p>
<p>I am trying to train by creating multiple dataloaders set to the input_size and batch_size I want.</p>
<p>I have two questions.</p>
<ol>
<li>
<p>If it is assumed that 3 dataloaders are created, is memory allocated to each dataloader immediately after creation?</p>
</li>
<li>
<p>Are there examples of controlling multiple dataloaders using multiprocessing?</p>
</li>
</ol>
<p>Thank you in advance.</p>

ptrblck: <p>For 1: Have a look at <a href="https://discuss.pytorch.org/t/do-dataloader-block-memory/112588/2">this post</a>, which explains the memory usage depending on the implementation of the <code>Dataset</code>.</p>
<p>2: I don’t know how multiprocessing would work on top of the <code>DataLoader</code>, which itself uses multiprocessing for the workers.</p>
Do Dataloader block memory?

mrmoody: <p>Hi everyone,</p>
<p>I have a current setup where I create a few data loader with different batch sizes and number of workers which I use for different purposes. I have one for the normal training, and then another one for visualization, etc; I need different batch sizes.</p>
<p><strong>Question</strong>: I was wondering if a dataloader, e.g. created with 8 workers, block memory, although they are not currently used? Or is it fine to generate a few data loaders, although some are only rarely used?</p>
<p>Thanks!</p>

ptrblck: <p>If you “execute” the <code>DataLoader</code>, i.e. start an iteration, the workers will be spawned, which will create copies of the underlying <code>Dataset</code> and will start creating the batches and adding them to the queue.<br>
The batches will use memory and in case you are loading data in the <code>Dataset.__init__</code> method, these object would also allocate memory.<br>
If you are creating the <code>DataLoader</code> object without iterating it, the memory usage should be minimal.</p>
Is there a built in way to ensure image size match per batch?

Alexander_Soare: <p>I have a <code>torch.utills.data.Dataset</code> that returns images which slightly vary in size.</p>
<p>I produce my training batches using <code>torch.utils.data.DataLoader</code>. Is there an inbuilt way to resize the images such that they can be stacked into a batch on the data loader end?</p>
<p>If there’s not, I’ll probably write a <code>collate_fn</code> to do it. Or is there a better way?</p>
<p>PS: I don’t want to resize my whole dataset to one size. That’s why I’m specifically asking about doing it at a batch level.</p>

ptrblck: <p>If you want to resize the data in the batch only, one approach would be to apply the <code>Resize</code> operation in the <code>collate_fn</code> or return a <code>list</code> in a custom <code>collate_fn</code> and resize the data in the training loop.</p>
<p>Alternatively, you could also use a <code>BatchSampler</code>, which will pass a batch of indices to <code>Dataset.__getitem__</code> and resize all images inside the <code>Dataset</code>.</p>
PyTorch equivalent of K.clear_session()

SD1713: <p>I am training models iteratively and would like to make sure that the session is cleared and the computational graph does not start from the last model’s update. I am deleting the models with <code>del model</code> but is there an equivalent of <code>K.clear_session()</code> to ensure destroying the computational graph and clearing session?</p>
<p>Thanks in advance!</p>

soulitzer: <p>There shouldn’t be a need for something like K.clear_session() since the graph should be automatically freed as long as there are no more references to it. So if <code>model</code> is the last variable referencing your graph, something like <code>del model</code> should do the trick. You might not even need <code>del</code> if <code>model</code> goes out of scope or you rebind your model variable to a new object.</p>
<p>If you find your memory usage still going up, you can add a call to <code>gc.collect()</code> to manually trigger python’s garbage collector. This might be necessary if your code isn’t CPU memory intensive.</p>
CUDNN_STATUS_EXECUTION_FAILED error runtime

Maicol_Polvere: <p>Traceback (most recent call last):<br>
File “main.py”, line 46, in <br>
main(config, args)<br>
File “main.py”, line 19, in main<br>
Mission.train()<br>
File “/opt/notebooks/TextZoom/src/interfaces/super_resolution.py”, line 33, in train<br>
model_dict = self.generator_init()<br>
File “/opt/notebooks/TextZoom/src/interfaces/base.py”, line 149, in generator_init<br>
model = model.to(self.device)<br>
File “/root/.conda/envs/textZoom/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 381, in to<br>
return self._apply(convert)<br>
File “/root/.conda/envs/textZoom/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 187, in _apply<br>
module._apply(fn)<br>
File “/root/.conda/envs/textZoom/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 187, in _apply<br>
module._apply(fn)<br>
File “/root/.conda/envs/textZoom/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 187, in _apply<br>
module._apply(fn)<br>
File “/root/.conda/envs/textZoom/lib/python3.6/site-packages/torch/nn/modules/rnn.py”, line 117, in _apply<br>
self.flatten_parameters()<br>
File “/root/.conda/envs/textZoom/lib/python3.6/site-packages/torch/nn/modules/rnn.py”, line 113, in flatten_parameters<br>
self.batch_first, bool(self.bidirectional))<br>
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</p>
<p>I’m facing this problem. Do you have any suggestion?</p>

J_Johnson: <p>See if this helps.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html</a></p>
<blockquote>
<p>WARNING</p>
<p>There are known non-determinism issues for RNN functions on some versions of cuDNN and CUDA. You can enforce deterministic behavior by setting the following environment variables:</p>
<p>On CUDA 10.1, set environment variable <code>CUDA_LAUNCH_BLOCKING=1</code>. This may affect performance.</p>
<p>On CUDA 10.2 or later, set environment variable (note the leading colon symbol) <code>CUBLAS_WORKSPACE_CONFIG=:16:8</code> or <code>CUBLAS_WORKSPACE_CONFIG=:4096:2</code>.</p>
<p>See the <a href="https://docs.nvidia.com/deeplearning/sdk/cudnn-release-notes/rel_8.html" rel="noopener nofollow ugc">cuDNN 8 Release Notes</a> for more information.</p>
</blockquote>

ptrblck: <p>Could you post an executable code snippet to reproduce this issue (model definition and input shape information might be enough) as well as information about your setup (PyTorch, CUDA, cudnn version, used GPU)?</p>

Maicol_Polvere: <p>I think that I have probably found the error because I am currently using CUDA  version 11.2 with PyTorch version 1.7.1</p>

ptrblck: <p>Did you build PyTorch from source using this setup? If so, which cudnn version were you using?<br>
A code snippet to reproduce this issue would still be needed in order to debug and fix it.</p>
How to handle convolution backward bias in cuDNN backend

Stonepia: <p>From the source code, I found that PyTorch removed handling bias inside cudnn_convolution_backward because of <a href="https://github.com/pytorch/pytorch/pull/31524" rel="noopener nofollow ugc">31524</a> , It just states that bias should be handled outside cudnn_convolution_backward.</p>
<p>The thing is , I could not find a place telling me how to handle the bias (or should I handle it manually?)<br>
Could someone please tell me about how could I handle it? If I just handle it the same as the implementation <a href="https://github.com/pytorch/pytorch/commit/34561dadcddf6ce3c76daf16f09a09adc9c7b73b#diff-e45460d4ddb7877fedddc7b30fcfd310a8273f980839b4a7d753b90a408b19dcL731" rel="noopener nofollow ugc">before</a> , what is the point of moving it out of cuDNN implementation?</p>

ptrblck: <aside class="quote no-group" data-username="Stonepia" data-post="1" data-topic="112559">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stonepia/48/30537_2.png" class="avatar"> Stonepia:</div>
<blockquote>
<p>what is the point of moving it out of cuDNN implementation?</p>
</blockquote>
</aside>
<p>The advantages are mentioned in the PR:</p>
<ul>
<li>faster, especially for backward (see: <a href="https://github.com/zasdfgbnm/things/blob/master/2019/conv-backward-profile.md">https://github.com/zasdfgbnm/things/blob/master/2019/conv-backward-profile.md</a>)</li>
<li>handles 64bit indexing automatically</li>
<li>has less code, less maintenance effort</li>
</ul>
<aside class="quote no-group" data-username="Stonepia" data-post="1" data-topic="112559">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stonepia/48/30537_2.png" class="avatar"> Stonepia:</div>
<blockquote>
<p>The thing is , I could not find a place telling me how to handle the bias (or should I handle it manually?)</p>
</blockquote>
</aside>
<p>If you want to copy the current approach used in PyTorch, you could add the bias manually as seen <a href="https://github.com/pytorch/pytorch/blob/d491fc6d48168bfde4976f486b7df72bb9d38be1/aten/src/ATen/native/Convolution.cpp#L656-L657">here</a>.</p>

Stonepia: <p>Thanks for your help! I want to ask one more thing: how about the backward of bias?</p>
<p>The bias should also be updated in the backward pass. In former implementation, PyTorch used cudnn_convolution_backward_bias , but now they deleted.</p>

ptrblck: <p>The manual <code>bias</code> addition should be tracked by Autograd and the backward operation would thus call into the captured bias backward function.</p>
Torch.where too weird

FruitVinegar: <p>Is this predicted?<br>
It’s not intuitive, at least for me.</p>
<pre><code class="lang-auto">x = torch.normal(0.0, 5, size=(4, 16, 32, 32))

try:
    # trial 1
    y = torch.where(x &gt; 0, x, 0)
except:
    print('trial 1 failed')
else:
    print('trial 1 succeed, the size of y: ', y.size())
    
try:
    # trial 2
    y = torch.where(x &gt; 0, x, 0.0)
except:
    print('trial 2 failed')
else:
    print('trial 2 succeed, the size of y: ', y.size())
    
    
try:
    # trial 3
    y = torch.where(x &gt; 0, x, torch.tensor(0.0))
except:
    print('trial 3 failed')
else:
    print('trial 3 succeed, the size of y: ', y.size())
</code></pre>
<blockquote>
<p>trial 1 failed<br>
trial 2 failed<br>
trial 3 succeed, the size of y:  torch.Size([4, 16, 32, 32])</p>
</blockquote>

soulitzer: <p>Yeah agree. Absolutely not intuitive. But to be fair, its actually documented <a href="https://pytorch.org/docs/stable/generated/torch.where.html" class="inline-onebox" rel="noopener nofollow ugc">torch.where — PyTorch 1.7.1 documentation</a>.</p>
Is torch.quantile differentiable?

SG-391: <p>Hi,</p>
<p>Is the function torch.quantile differentiable? Can I use it in a custom loss function?</p>
<p>Thanks!</p>

soulitzer: <p>Yep. If you want to check if something is differentiable you can just call it with a tensor that requires grad and it’ll spit out a warning if the function does not support autograd</p>

SG-391: <p>Very helpful, many thanks!</p>
Low Validation Score on Pretrained Alexnet from Pytorch models for ImageNet 2012 dataset

gakadam: <p>I am using pre-trained AlexNet network to validate some prior work.</p>
<p>The code is as follows:</p>
<pre><code class="lang-auto">import os
import torch

import torchvision
import torchvision.datasets as datasets
import torchvision.models as models
import torchvision.transforms as transforms


model = torch.hub.load('pytorch/vision:v0.6.0', 'alexnet', pretrained=True)
model.eval()

batchsize = 50000
workers = 1
dataset_path = 'data/imagenet_2012/'
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
val_data = datasets.ImageFolder(root=os.path.join(dataset_path, 'val'), transform=transforms.Compose( [transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), normalize,]))
val_loader = torch.utils.data.DataLoader(val_data, batch_size=batchsize, num_workers=workers)


batch = next(iter(val_loader))
images, labels = batch

with torch.no_grad():
    output = model(images)


for i in output:
    out_soft = torch.nn.functional.softmax(i, dim=0)
    print(int(torch.argmax(out_soft)))

</code></pre>
<p>When I execute this and compare with <code>ILSVRC2012_validation_ground_truth.txt</code> , I get top 1% accuracy of 5% only.</p>
<p>What am I doing wrong here?</p>
<p>Thank you.</p>

ptrblck: <p>Instead of the fop loop you could directly use <code>torch.argmax(output, dim=1)</code>, but your code should also work.<br>
I guess the correspondence between the data and labels might be broken. Could you check some random examples from your ground truth file and the input image manually?</p>

gakadam: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your reply.<br>
And you are right!</p>
<p>I checked the images in the validation folder manually against the class names listed <a href="https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a" rel="nofollow noopener">here</a>. I noticed that the ILSVRC2012_validation_ground_truth.txt file has INCORRECT class names listed. For example, the first image in the folder is of a snake, but groundtruth lists it as a goldfish. Also, the second image was correctly classified when I checked manually, but groundtruth said otherwise.</p>
<p>In this case, I found out I should use the groundtruth files from <a href="https://gist.github.com/ksimonyan/fd8800eeb36e276cd6f9#note" rel="nofollow noopener">here</a>:</p>
<h1>Thank you again.</h1>
<p>update:<br>
New validation accuracy based on the groundtruth in the zip file in the link:<br>
Top_1 = 56.522%<br>
Top_5 = 79.066%</p>

seyeeet: <p>can you share the label file, i cannot find it and have the same issue</p>

gakadam: <p>Hi, did you read my message above? I shared the link in there.<br>
Here, is the direct one:  <a href="http://dl.caffe.berkeleyvision.org/caffe_ilsvrc12.tar.gz" rel="noopener nofollow ugc">http://dl.caffe.berkeleyvision.org/caffe_ilsvrc12.tar.gz</a><br>
Get this and untar it. Look up the *.txt files for labels.</p>
Different tensors, same data and same grad

skzck320: <p>I would like to create bunch of tensors, that share the same data and same gradient accumulation, but are viewed as different tensors by autograd when calculating gradients.<br>
So when I call backward() on the function I try to optimize, pytorch will calculate separate gradient for each of the tensors. But instead of keeping every gradient separely, it will sum them all together. And when I call optimization step(), it will used this shared accumulated gradient to update shared data.<br>
So all tensors always have exactly the same value and exactly the same grad, but are viewed as different tensors by the rest of my calculation graph.</p>

soulitzer: <p>Do you mean to create a view?</p>
<p><a href="https://pytorch.org/docs/stable/tensor_view.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/tensor_view.html</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.ones(2, 2, requires_grad=True)
&gt;&gt;&gt; a.grad
&gt;&gt;&gt; b = a[:]
&gt;&gt;&gt; b._is_view()
True
&gt;&gt;&gt; b._base is a
True
&gt;&gt;&gt; torch.autograd.grad(b * 2 + a * 2, (a,), torch.ones(2, 2))
(tensor([[4., 4.],
        [4., 4.]]),)
&gt;&gt;&gt; torch.autograd.grad(a * 2, (a,), torch.ones(2, 2))
(tensor([[2., 2.],
        [2., 2.]]),)
&gt;&gt;&gt; 

</code></pre>
Explicit resize range in RandomResizedCrop

CDhere: <p>I am trying to follow the data augmentation practice in the original ResNet paper <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="noopener nofollow ugc">Deep Residual Learning for Image Recognition</a>, which includes:</p>
<blockquote>
<p>The image is resized with its shorter side randomly sampled in [256, 480] for scale augmentation [41]. A 224×224 crop is randomly sampled from an image or its horizontal flip, with the per-pixel mean subtracted [21]. The standard color augmentation in [21] is used.</p>
</blockquote>
<p>Here, the random resize is explicitly defined to fall in the range of [256, 480], whereas in the Pytorch implementation of <code>RandomResizedCrop</code>, we can only control the resize ratio, i.e., a range of scaling the images no matter what the resulting size is. While it seems reasonable to do so to keep the resolution consistent, I wonder:</p>
<ol>
<li>
<p>are there ways to make <code>RandomResizedCrop</code> behave in the “explicit resizing” way? (if not I’ll consider implement my own)</p>
</li>
<li>
<p>what’s the essential reason to ditch the “explicit resizing” from the original paper? are papers nowadays all preferring the “ratio resizing” version?</p>
</li>
</ol>

Dwight_Foster: <p>random resized crop does have a shape parameter</p>
<pre><code class="lang-auto">torchvision.transforms.RandomResizedCrop(size, scale=(0.08, 1.0), ratio=(0.75, 1.3333333333333333), interpolation=2)
</code></pre>
<p>The size parameter is the same thing so you can just put the values in there. So like this</p>
<pre><code class="lang-auto">torchvision.transforms.RandomResizedCrop((256,480))
</code></pre>

CDhere: <p>I don’t think the first parameter refers to the intermediate size before the crop; instead I think it means the final output size.</p>
<p>What I want is if we can control the intermediate size it scales to before the random crop.</p>

Dwight_Foster: <p>So you want to change the shape of the image that goes into the crop? Couldn’t you just do a random resize before the crop then.</p>
Why do most of the DDP examples not use the main process?

Brando_Miranda: <p>I was looking some of the DDP examples and noticed that most of them spawn 8 processes but then never use the main process again. e.g. <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.7.1 documentation</a></p>
<pre><code class="lang-auto">def demo_model_parallel(rank, world_size):
    print(f"Running DDP with model parallel example on rank {rank}.")
    setup(rank, world_size)

    # setup mp_model and devices for this process
    dev0 = rank * 2
    dev1 = rank * 2 + 1
    mp_model = ToyMpModel(dev0, dev1)
    ddp_mp_model = DDP(mp_model)

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_mp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    # outputs will be on dev1
    outputs = ddp_mp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(dev1)
    loss_fn(outputs, labels).backward()
    optimizer.step()

    cleanup()


if __name__ == "__main__":
    n_gpus = torch.cuda.device_count()
    if n_gpus &lt; 8:
      print(f"Requires at least 8 GPUs to run, but got {n_gpus}.")
    else:
      run_demo(demo_basic, 8)
      run_demo(demo_checkpoint, 8)
      run_demo(demo_model_parallel, 4)
</code></pre>
<p>why isn’t the main process used? isn’t this wasting 1 entire processor?</p>
<hr>
<p>related: <a href="https://stackoverflow.com/questions/66323651/why-does-my-pytorch-rpc-workers-deadlock-is-it-because-i-am-using-main-as-my-ma" class="inline-onebox" rel="noopener nofollow ugc">python - Why does my pytorch rpc workers deadlock, is it because I am using main as my master? - Stack Overflow</a></p>
Named_parameters: Missing parameter layer because .to(device)

56293658742896592925: <p>Hi,</p>
<p>I am encountering a for me very strange issue with the function <strong>self.named_parameters()</strong>.<br>
Long story short:</p>
<p>I am trying to create the following layer:<br>
<code>self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd)).to(self.device)</code></p>
<p>After creation I generate a param_dict while creating a optimizer with this function:</p>
<pre><code class="lang-auto">def get_param_dict(self):
        return {pn: p for pn, p in self.named_parameters()}
</code></pre>
<p>The strange behavior is that because of the device movement to self.device the pos_emb layer <strong>no longer shows up</strong> within the named_parameters dictionary:</p>
<pre><code class="lang-auto">self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd)).to(self.device)
print(self.get_param_dict())
--&gt; dict_keys([])
</code></pre>
<p>If I remove the <code>.to(self.device)</code> part named_parameters behaves as expected:</p>
<pre><code class="lang-auto">self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd))
print(self.get_param_dict())
--&gt; dict_keys(['pos_emb'])
</code></pre>
<p>Why is this the case and how to fix it?<br>
<strong>Somehow it affects only this nn.Parameter layer, the other layers are listed correctly with device movement or without…</strong><br>
I need a working get_param_dict function for optimizer configuration.</p>
<p>I already tried to train my model without the movement to self.device for the pos_emb layer but then the training fails because the tensor is on the wrong device (obviously).</p>
<p>Thanks a lot for any kind of hint or solution! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

tom: <aside class="quote no-group quote-modified" data-username="56293658742896592925" data-post="1" data-topic="112641">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/5/f07891/48.png" class="avatar"> 56293658742896592925:</div>
<blockquote>
<p>self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd, device=self.device))</p>
</blockquote>
</aside>
<p>Background:<br>
To autograd, <code>.to</code> is computation, so you didn’t assign a parameter to self.pos_emb but an intermediate tensor.</p>