akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Chained numerical comparisons in PyTorch

username: BramVanroy
cooked: <p>In PyTorch we can do comparisons between elements in tensors like so:</p>
<pre><code class="lang-python">import torch

a = torch.tensor([[1,2], [2,3], [3,4]])
b = torch.tensor([[3,4], [1,2], [2,3]])

print(a.size())
# torch.Size([3, 2])
print(b.size())
# torch.Size([3, 2])

c = a[:, 0] &lt; b[:, 0]

print(c)
# tensor([ True, False, False])
</code></pre>
<p>However, when we try to add a condition, the snippet fails:</p>
<pre><code class="lang-python">c = a[:, 0] &lt; b[:, 1] &lt; b[:, 0]
</code></pre>
<p>The expected output is</p>
<pre><code class="lang-python"> tensor([ False, False,  False])
</code></pre>
<p>So, for each element in a, compare its first element with the second element of the corresponding item in b, and compare that element with the first element of the same item in b.</p>
<p>Traceback (most recent call last):<br>
File “scratch_12.py”, line 9, in <br>
c = a[:, 0] &lt; b[:, 1] &lt; b[:, 0]<br>
RuntimeError: bool value of Tensor with more than one value is ambiguous</p>
<p>Why is that, and how can we solve it?</p>

username: stihl1210
cooked: <aside class="quote no-group" data-username="BramVanroy" data-post="1" data-topic="102573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bramvanroy/48/28628_2.png" class="avatar"> BramVanroy:</div>
<blockquote>
<p><code>c = a[:, 0] &lt; b[:, 1] &lt; b[:, 0]</code></p>
</blockquote>
</aside>
<blockquote>
<p>c = (a[:, 0] &lt; b[:, 1])&lt;  b[:, 0]</p>
</blockquote>
<p>I don’t know if the parentheses will solve the problem, it depends what you compare and when</p>

username: BramVanroy
cooked: <p>That seems an odd construct. Aren’t you then doing implicit casting?</p>
<pre><code class="lang-auto">c = (floattensor(true or false)) &lt; other_tensor
</code></pre>
<p>In “regular Python” we can do x &lt; y &lt; z, which is the same as “x &lt; y and y &lt; z”.</p>
The effect of epsilon on Adam optimizer

username: Scott_Hoang
cooked: <p>To train my model, I must use AMP, but AMP causes Div/0 in Adam.<br>
To fix this, I have to increase the default value of eps from 1e-8 to 1e-4. However, I noticed that by doing so, my model progress much slower.<br>
My default LR rate is 0.001.<br>
What is the effect of eps have on Adam??</p>
Longer LSTM Prediction

username: Vitsuji_Aura
cooked: <p>Hello, I am using an LSTM to take 5 sequences as input to predict another 5. I want to know how to predict more than 5 timesteps. I assume its got something to do with the hidden_dim but I can’t figure it out.</p>
<p>Here is my code</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, seq_len=5, n_features=256, n_hidden=256, n_layers=1, output_size=1):
        super().__init__()
        self.n_features = n_features
        self.seq_len = seq_len
        self.n_hidden = n_hidden
        self.n_layers = n_layers
        
        self.l_lstm = nn.LSTM(input_size=self.n_features, hidden_size=self.n_hidden, num_layers=self.n_layers, batch_first=True)

        

    def init_hidden(self, batch_size):
        hidden_state = torch.zeros(self.n_layers,batch_size,self.n_hidden).to(device)
        cell_state = torch.zeros(self.n_layers,batch_size,self.n_hidden).to(device)
        self.hidden = (hidden_state, cell_state)
    
    def forward(self, x):        
        
        lstm_out, self.hidden = self.l_lstm(x,self.hidden)

        return lstm_out
</code></pre>
<p>If anyone knows how to extend the prediction range or could suggest a better way of writing LSTM, I would really appreciate it.</p>

username: googlebot
cooked: <p>Generally speaking, RNNs work one step at a time, so as you may have noticed, seq_len param is never used, that’s by design. So step count is arbitrary, so long as you have per-step inputs. To do multi-step forecasting, Seq2Seq approach can be used, or step_forecast -&gt; next_input assignments in a loop.</p>

username: Vitsuji_Aura
cooked: <p>Oh yes. Seq2Seq. I’ll look into that. Thnx.</p>
Deploying a Facial Recognition model using TorchServe

username: vishak_bharadwaj
cooked: <p>I just finished finetuning a pretrained facial recognition model that spits out an encoding for every image you send into it. I would now like to deploy it in an EC2 instance</p>
<div class="youtube-onebox lazy-video-container" data-video-id="jdE4hPf9juk" data-video-title="Deploying your ML Model with TorchServe" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=jdE4hPf9juk" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/jdE4hPf9juk/maxresdefault.jpg" title="Deploying your ML Model with TorchServe" width="690" height="388">
  </a>
</div>

<p>This tutorial tells you how to set Torchserve up to get a reply as a simple class. How do I set something similar for my model. It needs to take an image, pass it through the model to get the encodings, and then compare them to every image encoding in the database and output the closest one.</p>
<p>Which is the most efficient way to achieve this?<br>
Thanks!</p>
Dice-Score on imbalanced Validation Set

username: maaft1
cooked: <p>Hi,</p>
<p>I have a dataset with positive and negative samples for a segmentation task where:<br>
1. “positive” means, that the image contains at least one object<br>
2. “negative” means, that the image contains no object of interest.</p>
<p>For every positive sample, there are roughly 3 negative samples in the dataset.</p>
<p>Now when I evaluate my model on my validation set, the dice-score will be very high in the beginning as the model just predicts 0 everywhere. After some iterations, the dice score will start to increase again but fails to reach the high scores from the start of the training. This makes the automatic saving of the best performing model very hard, if not impossible.</p>
<p>Any suggestions how to balance the dice-score such that it doesn’t favor negative examples during validation?</p>

username: SoucheChapich
cooked: <p>Hello,</p>
<p>In my opinion you can try to put a weight to your dice score / loss function that is proportional to your dataset samples. If you have roughly 3 negative samples in the dataset, you should reflect it on the loss by setting a weight of 3.</p>

username: maaft
cooked: <p>Sorry for the very late answer, but I currently reiterate this issue.</p>
<p>Although your suggestion is possible, this will still lead to a dice-score of at least 0.5 when the output of the network is constant 0. What if the real working best performance of that model is only 0.4?</p>
<p>Meanwhile I searched the net a bit and am a little bit confused that this issue does not arise more often. Maybe someone else can step in?</p>

username: maaft
cooked: <p>Okay, I guess the solution is to aggregate all predictions and to calculate the dice-score over all results simultaniously.</p>
<p>I.e. instead of:<br>
<code>result = (dice_score(pred1,gt1) + dice_score(pred2,gt2)) / 2</code></p>
<p>do</p>
<pre><code class="lang-python">result = dice_score(torch.cat([pred1, pred2, ...], dim=1), torch.cat([gt1, gt2, ...], dim=1))
</code></pre>
<p><strong>EDIT</strong><br>
This of course could need huge amounts of RAM and compute, therefore better aggregate the intersection and union and calculate the dice-score in the end.</p>
Iou computation code

username: promach
cooked: <p>Could anyone explain the last four lines (<em>lt, rb, inter,</em> and the return expression) of the following code ? I do not quite understand how : works inside [] for pytorch</p>
<pre><code class="lang-auto">def box_iou(boxes1, boxes2):
    # https://github.com/pytorch/vision/blob/master/torchvision/ops/boxes.py
    """
    Return intersection-over-union (Jaccard index) of boxes.
    Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
    Arguments:
        boxes1 (Tensor[N, 4])
        boxes2 (Tensor[M, 4])
    Returns:
        iou (Tensor[N, M]): the NxM matrix containing the pairwise
            IoU values for every element in boxes1 and boxes2
    """

    def box_area(box):
        # box = 4xn
        return (box[2] - box[0]) * (box[3] - box[1])

    area1 = box_area(boxes1.t())
    area2 = box_area(boxes2.t())

    lt = torch.max(boxes1[:, None, :2], boxes2[:, :2])  # [N,M,2]
    rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:])  # [N,M,2]

    inter = (rb - lt).clamp(min=0).prod(2)  # [N,M]
    return inter / (area1[:, None] + area2 - inter)  # iou = inter / (area1 + area2 - inter)
</code></pre>
Handle out of memory issue when using gradient accumulation

username: HuangZiliAndy
cooked: <p>Hi, I am trying to handle the out of memory issue. I see someone point to this example <a href="https://github.com/pytorch/fairseq/blob/50a671f78d0c8de0392f924180db72ac9b41b801/fairseq/trainer.py#L283" rel="noopener nofollow ugc">fairseq</a>. But my situation is that I am also using some gradient accumulation. It would be a waste of data if I directly del p.grad. So my question is (1) when I catch an OOM during forward/backward, is the gradient already added/ not added to p.grad? (2) Or do you have some suggestions on how to handle the OOM exception when using the gradient accumulation? Thanks!</p>
Backpropagation Over Summed Hidden States in BiRNN

username: raraz15
cooked: <p>Hello, I am trying to implement my own Bidirectional RNN Model where the <em>output</em> is defined as the <em>sum of hidden states from both directions</em>. I want to know if autograd can backprop through this summation.</p>
<pre><code class="lang-auto">def forward(self,x,state=None):
        
        hidden_f = []  # forward direction hidden states
        hidden_b = []  # backward direction hidden states
        
        dimensions = x.size()
        batch_sz, sequence_len = dimensions[0],dimensions[1]
        
        hidden_sz = self.cell.hidden_size
        
        if state == None:
            h_n_f = torch.zeros(batch_sz,hidden_sz) 
            h_n_b = torch.zeros(batch_sz,hidden_sz)
        else:
            h_n_f = state[:,:,0] 
            h_n_b = state[:,:,1] 
            
        for i in range(sequence_len):
            
            h_n_f = self.cell(x[:,i,:],h_n_f) # Calculate next hidden state in forward direction    
            hidden_f.append(h_n_f)

            h_n_b = self.cell(x[:,-i-1,:],h_n_b) # Calculate next hidden state in backward direction  
            hidden_b.append(h_n_b)

        # MY PROBLEMS START HERE
        # ,requires_grad=True ==&gt; leaf node moved to interior
        output = torch.zeros(batch_sz,sequence_len,hidden_sz) # Recording the outputs
        
        #output = variable(torch.zeros(batch_sz,sequence_len,hidden_sz)) 
        
        # COULD BE PROBLEMATIC FOR BACKPROP
        for i in range(sequence_len):
            output[:,i,:] = hidden_f[i]+hidden_b[-i-1] # Outputs are defined as the sum of both direction
            
        return output
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Do you actually see an error?<br>
From your comments, you should not set <code>requires_grad=True</code> on the <code>output</code> if you don’t need to access its <code>.grad</code> field later.<br>
And you don’t need to wrap things in Variables<br>
So it should work just fine yes.</p>

username: raraz15
cooked: <p>Thank you! The code executes just fine and I am not seeing any errors. I just  wanted to be sure that the backpropagation can take into account the sum of the two hidden states from both directions inside the final for loop, which is a little strange operation. I am guessing that for loops and list summations are taken care in autograd and I don’t need to do <code>.unbind()</code> or any other kind of operation. Also I was confused as I am initializing the root variable <code>output</code> as zeros and than I am setting each entry using other varaibles.</p>

username: albanD
cooked: <p>Yes that will work just fine.<br>
In general, if the autograd computes something, it is right. We (should) always raise an error if something would give a wrong result!</p>
The change on name definition...Help

username: rui_zhou
cooked: <p>I have original defined name for VOC dataset , but I want to use folder structure as this:<br>
––plant<br>
——Images<br>
———plant001_rgb.png<br>
———plant002_rgb.png<br>
——Masks<br>
———plant001_label.png<br>
———plant002_label.png</p>
<p>How should I make the change…I am so confusing, any help would be so glad</p>
<p>def <strong>getitem</strong>(self, index):<br>
if index == 0:<br>
shuffle(self.train_lines)</p>
<pre><code>    annotation_line = self.train_lines[index]
    name = annotation_line.split()[0]

    jpg = Image.open(r"./VOCdevkit/VOC2007/JPEGImages" + '/' + name + ".jpg")
    png = Image.open(r"./VOCdevkit/VOC2007/SegmentationClass" + '/' + name + ".png")

    if self.random_data:
        jpg, png = self.get_random_data(jpg,png,(int(self.image_size[1]),int(self.image_size[0])))
    else:
        jpg, png = letterbox_image(jpg, png, (int(self.image_size[1]),int(self.image_size[0])))


    png = np.array(png)
    png[png &gt;= self.num_classes] = self.num_classes
    
    seg_labels = np.eye(self.num_classes+1)[png.reshape([-1])]
    seg_labels = seg_labels.reshape((int(self.image_size[1]),int(self.image_size[0]),self.num_classes+1))
    jpg = np.transpose(np.array(jpg),[2,0,1])/255

    return jpg, png, seg_labels
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f988b520fcdc820fbdb8504c1f2a254ee299f9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e1f988b520fcdc820fbdb8504c1f2a254ee299f9" title="51"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1f988b520fcdc820fbdb8504c1f2a254ee299f9_2_427x500.jpeg" alt="51" data-base62-sha1="wf43pspCUcb3W0K7DIOzRhyeg9X" width="427" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1f988b520fcdc820fbdb8504c1f2a254ee299f9_2_427x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f988b520fcdc820fbdb8504c1f2a254ee299f9.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f988b520fcdc820fbdb8504c1f2a254ee299f9.jpeg 2x" data-dominant-color="F0F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">51</span><span class="informations">543×635 54.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Understanding fractionally strided convolutions

username: Bruno1
cooked: <p>Hello,</p>
<p>I have a problem in understanding convtranspose2d with stride in pytorch.<br>
According to this paper <a href="https://arxiv.org/pdf/1603.07285.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1603.07285.pdf</a>, convtranspose2d with stride is equivalent to a convolution on a modified input by insterting 0s between the entries (that’s why it is sometimes called fractionnally strided convolution).<br>
I run a quick test but it doesnt work and I don’t understand why. Could someone explain what is wrong here ?</p>
<blockquote>
<p>x = torch.rand(1,1,20,20)<br>
k = torch.rand(1,1,3,3)</p>
<p>w = torch.zeros(3, 3)<br>
w[1, 1] = 1<br>
zerofill = F.conv_transpose2d(x, w.expand(1, 1, 3, 3),stride=2) <span class="hashtag-raw">#interleave</span> tensor x with 0s</p>
<p>out1 = F.conv_transpose2d(x,k,stride=2)<br>
out2 = F.conv2d(zerofill,k,padding=1)</p>
<p>err = (out1-out2).pow(2)<br>
print(err.sum().item()) # large error</p>
</blockquote>
<p>Thanks</p>
Implementing the non-negative logistic regression in pytorch for single output classification with with all positive weights?

username: Richard_S
cooked: <p>I read some tutorials for logistic regression in pytorch  but they didn’t explain how can i implement the non-negative logistic regression model?</p>
<p>basically i have a linear classifier that i need to output a single probability number (i assume using sigmoid), <strong>and i need to have all the weights of the network to be non-negative</strong></p>
<p>it was suggested that i should use non-negative variant of logistic regression to have non-negative weights but I’m not sure how to implement this? my model currently is a simple 1 layer linear classifier.</p>

username: Richard_S
cooked: <p>any help is appreciated</p>
Back prorogation from fc layer to conv layer

username: flipflop
cooked: <p>I am trying to manually derive and update the gradient for a simple nn .<br>
input size is <code>12*12</code> ,apply <code>10*3*3</code> kernel  ,so i have a conv layer with output features  <code>10*10*10</code>  ,and by flattening it to <code>1*1000</code> to create one dimension vector to connect to a fully connected layer with weight dimension <code>1000*2</code>.  I can get the fc layers weight to be updated , but how do I update weight value from <code>1000*2</code> matrix to <code>10*3*3</code> kernel matrix</p>

username: Kushaj
cooked: <p>Your loss function contains average. When you backprop from loss to fc layers you will get a scaler value (a vector in case of batching) due the the average. During calculation of gradients of conv layer you will multiply the previous scaler with the grads of conv layer.</p>
<p>If you derive the equations then it will become clearer. If something needs more clarification, feel free to ask.</p>

username: flipflop
cooked: <p>Hi kushaj ,thank you for your answer , but i am still confused , if back propagate from loss to fc layer weight ,which here is 1000<em>2 , how do we get a scalar ? shouldnt be still 1000</em>2 weight matrix with all the weight value updated ?</p>

username: Kushaj
cooked: <p>I think the confusion is due to the limitation of the text. Check the derivation below and I am happy to help with any more questions.</p>
<p>Note:- To implement in pytorch, you need to respect the dimensions (like adding squeeze and unsqueeze where necessary).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea895c90842cf8061d55f76f8993f93a6efc3123.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ea895c90842cf8061d55f76f8993f93a6efc3123" title="WhatsApp Image 2020-11-12 at 4.30.09 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea895c90842cf8061d55f76f8993f93a6efc3123_2_641x500.jpeg" alt="WhatsApp Image 2020-11-12 at 4.30.09 PM" data-base62-sha1="xsO1tVclB32mTzuPU4kcvAs7dpp" width="641" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea895c90842cf8061d55f76f8993f93a6efc3123_2_641x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea895c90842cf8061d55f76f8993f93a6efc3123_2_961x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea895c90842cf8061d55f76f8993f93a6efc3123.jpeg 2x" data-dominant-color="D0D0CD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WhatsApp Image 2020-11-12 at 4.30.09 PM</span><span class="informations">1280×997 46.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Batchnorm mean and variance calcuate for each dimension?

username: Sunnydreamrain
cooked: <p>Hi,</p>
<p>For batchnorm, it says in the doc "The mean and standard-deviation are calculated per-dimension over the mini-batches ". But for batchnorm1d, when input is of size (N，C，L), it seems N and L is merged together and the mean/var are calculated together for C. I checked the dimension of the running mean/var, it is of size C.<br>
I was wondering is there any built-in way to implement mean/var for each C and L, but the weight/bias is only for C (sharing over L).</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>You could use <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html?highlight=layernorm#torch.nn.LayerNorm"><code>nn.LayerNorm</code></a> and specify the <code>normalized_shape</code> which should be used to calculate the mean and standard deviation. However, I think you would need to set <code>elementwise_affine=False</code> and could apply a linear layer instead on the output using your desired shape.</p>

username: Sunnydreamrain
cooked: <p>Thanks. But LayerNorm computes the mean/var over the neruons and they are computed for both training and test. I still want to compute the mean/var for each neuron over the batches only in training and fixed for test.</p>

username: ptrblck
cooked: <p>In that case, your best bet might be to implement this type of normalization layer manually. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: Sunnydreamrain
cooked: <p>Okay. Thanks anyway.</p>
Error on pytorch install

username: ziywang50
cooked: <p>ERROR: torch-1.7.0+cpu-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.</p>

username: ptrblck
cooked: <p>Are you using a 32bit Python installation in your current environment? If so, you would need to install a 64bit package and retry the install steps.<br>
If that’s not the case, could you give more information about your system and setup?</p>

username: parth15041995
cooked: <p>environment?</p>
<p>you can print your environment with:<br>
wget <a href="https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py" rel="noopener nofollow ugc">https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py</a></p>
<p>then,<br>
python collect_env.py</p>

username: ziywang50
cooked: <p>my computer is 64-bit。</p>

username: ziywang50
cooked: <p>‘wget’ is not recognized as an internal or external command,<br>
operable program or batch file.</p>

username: ziywang50
cooked: <p>and I install a 64-bit package</p>

username: ptrblck
cooked: <p>Your Python installation could be a 32bit version regardless which CPU/OS you are using.</p>

username: ziywang50
cooked: <p>ok, you are right. The problem is 32-bit python. I accidentally did that.</p>
THFloatBlas_gemm is gone, what should I use instead?

username: Red-Eyed
cooked: <p>Hello all!</p>
<p>I’ve updated pytorch to <code>v1.7</code> (from <code>1.6</code>) on my <code>Ubuntu 20.04</code><br>
previously, I had CUDA and CPU code that was linked to <code>libtorch_cpu.so</code><br>
that code uses <code>THFloatBlas_gemm</code> function, but that symbol is missing in the latest version <code>1.7</code></p>
<p>I didn’t see any news about dropping the <code>THFloatBlas_gemm</code> symbol in the release note.</p>
<ol>
<li>Is there any plans to revert back <code>THFloatBlas_gemm</code> ?</li>
<li>Is there any kind of replacement for <code>THFloatBlas_gemm</code> ?</li>
</ol>

username: ptrblck
cooked: <p>Double post from <a href="https://github.com/pytorch/pytorch/issues/47750">here</a> with follow-up.</p>
The expanded size of the tensor (1) must match the existing size (3) at non-singleton dimension 1

username: ky_Pa
cooked: <p>Hi,<br>
I encountered the following error. How should I modify the code?</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "E:\Wx\pixelda\pixelda2.py", line 261, in &lt;module&gt;
    imgs_A = Variable(imgs_A.type(FloatTensor).expand(batch_size, 1, opt.img_size, opt.img_size))
RuntimeError: The expanded size of the tensor (1) must match the existing size (3) at non-singleton dimension 1
[Finished in 4.3s with exit code 1]
</code></pre>
<p>This is a code snippet:</p>
<pre><code class="lang-auto"> # Configure input
        imgs_A = Variable(imgs_A.type(FloatTensor).expand(batch_size, 1, opt.img_size, opt.img_size))
        labels_A = Variable(labels_A.type(LongTensor))
        imgs_B = Variable(imgs_B.type(FloatTensor))
</code></pre>

username: ptrblck
cooked: <p><code>tensor.expand</code> returns a new view of the tensor with singleton dimensions expanded to a larger size.<br>
Based on your code snippet I assume you would like to expand the batch dimension.<br>
If that’s the case, you would need to keep the channel, height, and width dimension equal, which is not the case in your code snippet.<br>
<code>imgs_A</code> seems to have 3 channels, while you are passing <code>1</code> to the <code>expand</code> operation in <code>dim1</code>.</p>
<p>PS: <code>Variable</code>s are deprecated since PyTorch <code>0.4</code>, so you can use tensors in newer versions. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Question about model ensembling performance

username: coincheung
cooked: <p>Hi,</p>
<p>Suppose I want to ensemble two models, I find two method, one is like this:</p>
<pre><code class="lang-auto">model1 = torchvision.models.resnet50().cuda()
model2 = torchvision.models.resnet101().cuda()

inten = torch.randn(16, 3, 224, 224).cuda()
n_iter = 1000

for i in range(n_iter):
    out = model1(inten).softmax(1)
    out += model2(inten).softmax(1)
</code></pre>
<p>And another method is to define these two models as children of a <code>nn.Module</code>:</p>
<pre><code class="lang-auto">class Model(torch.nn.Module):

    def __init__(self):
        super().__init__()
        self.model1 = torchvision.models.resnet50().cuda()
        self.model2 = torchvision.models.resnet101().cuda()

    def forward(self, x):
        out = self.model1(x).softmax(1)
        out += self.model2(x).softmax(1)
        return out

model = Model().cuda()

for i in range(n_iter):
    out = model(inten)
</code></pre>
<p>Which one is faster and better optimized during inference ? Will there be difference in terms of speed and memory usage between these two methods ?</p>

username: ptrblck
cooked: <p>Both methods should yield the same performance.<br>
The two code snippets would take slightly different Python paths, but the overhead shouldn’t be visible in your profiling.</p>
Wrong tensor size in GRU

username: erikt516
cooked: <p>I am trying to create a GRU but I am running into a tensor size problem. The shape of the input tensor is (16, 3, 256, 256). But when I try to run the GRU I get an error on self.GRU(out) that says <strong>RuntimeError</strong> : input must have 3 dimensions, got 4 and I believe this is because my out shape is (16, 3, 127, 127) but I don’t know I how would go about changing this to the right shape.</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net, self).__init__()
    self.pool = nn.MaxPool2d(4, 2)
    self.gru = nn.GRU(input_size = 128, hidden_size=hidden_size, bidirectional=True)

def forward(self, x):
    out = self.pool(x)
    out = torch.squeeze(out, -1)
    out = self.gru(out)</code></pre>

username: ptrblck
cooked: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html?highlight=gru#torch.nn.GRU">docs</a>:</p>
<blockquote>
<p><strong>input</strong> of shape (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence. See <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_padded_sequence.html#torch.nn.utils.rnn.pack_padded_sequence"><code>torch.nn.utils.rnn.pack_padded_sequence()</code></a> for details.</p>
</blockquote>
<p>So indeed a 3-dimensional input tensor is expected in the mentioned shape.</p>
Do I need to both set `requires_grad=True` and only pass those parameters I want to update to the optimizer to perform layerwise training?

username: soham96
cooked: <p>Hi, I am trying to train a model with transfer learning. When I set <code>requires_grad=True</code> for a few layers, but I send all my model parameters to the optimizer (by using <code>model.parameters()</code>), I see that even layers with <code>requires_grad=False</code> gets its weights updated. However, if instead I use <code>filter(lambda p: p.requires_grad, net.parameters())</code> to set my model parameters in the optimizer, the correct weights get updated.</p>
<p>Is this the correct behavior, or is there some error in my code?<br>
If this is the correct behavior, then how does my optimizer update weights when <code>requires_grad=False</code>?</p>

username: dbp.pat94
cooked: <p>Can you share the code snippet?</p>

username: ptrblck
cooked: <p>If these layers were previously updated, i.e. they got a valid gradient in previous iterations, are now frozen via <code>.requires_grad=True</code>, optimizers with running stats (such as Adam) can still update these parameters.<br>
To avoid this, you could try to set their <code>.grad</code> attributes to <code>None</code>, which should skip the updates.</p>

username: soham96
cooked: <p>Thanks for the clarification. However, I am not using Adam. I am using vanilla SGD with no momentum.</p>

username: ptrblck
cooked: <p>In that case, could you post an executable code snippet to show this behavior, as asked by <a class="mention" href="/u/dbp.pat94">@dbp.pat94</a>?</p>
The result of layer-wise execution of pytorch code is strange

username: kh_c
cooked: <p>Currently I am trying to use mobilenet-v2 like the code below.</p>
<pre><code>input_data = np.random.uniform(-1,1,size=(1,3,224,224))
layer = models.mobilenet_v2(pretrained=True).features[0:4]
layer.eval()
out_original = layer(input_data)
</code></pre>
<p>In addition, to check the intermediate result value,i split the layer and inferenced as follows.</p>
<pre><code>L_zero_from_two = layer[0:3]
output_middle = L_zero_from_two(input_data)

L3 = layer[3]

output = L3.conv[0](output_middle)
output = L3.conv[1](output)
output = L3.conv[2](output)
output = L3.conv[3](output)

output_test = L3(output_middle)
print(output[0][0][0][0], " : ", output_test[0][0][0][0] )
</code></pre>
<p>The output result is as follows.<br>
<strong>tensor(0.8066, grad_fn=)  :  tensor(0.9868, grad_fn=)</strong></p>
<p>the results are not the same. I think it should be the same, but I don’t understand why they are not.<br>
some anyone tell me which part is the problem?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>The <code>L3</code> module is not a pure <code>nn.Sequential</code> module and uses a skip connection, is <code>use_res_connect</code> is defined as seen <a href="https://github.com/pytorch/vision/blob/1f4e3a46609df5550789ea681820cd77b186b515/torchvision/models/mobilenet.py#L88-L89">here</a>.</p>
<p>Adding this skip connection to your code yields the same outputs:</p>
<pre><code class="lang-python">input_data = torch.randn(1, 3, 224, 224)
layer = models.mobilenet_v2(pretrained=True).features[0:4]
layer.eval()
out_original = layer(input_data)

L_zero_from_two = layer[0:3]
output_middle = L_zero_from_two(input_data)

L3 = layer[3]

output = L3.conv[0](output_middle)
output = L3.conv[1](output)
output = L3.conv[2](output)
output = L3.conv[3](output)
if L3.use_res_connect:
    output = output + output_middle

output_test = L3(output_middle)
print((output - output_test).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
print((out_original - output_test).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

username: kh_c
cooked: <p>Thank you!! it solve my problem!<br>
Thank you ptrblck!</p>
BERT model consuming too much memory

username: Hamza_Benyamina
cooked: <p>I am trying to train a BERT model, But one pass consumes about 11Gb. I am using the transformers library. Is the library not optimized enough or what?<br>
Here’s the code :</p>
<pre><code class="lang-auto">from transformers import BertConfig, BertForSequenceClassification
import torch

config = BertConfig()
model =  BertForSequenceClassification(config)

model.cuda(5)
model.forward(input_ids=torch.ones([15,512],dtype=torch.int64).cuda(5),
              attention_mask=torch.ones([15,512],dtype=torch.int64).cuda(5),
              labels=torch.ones([15],dtype=torch.int64).cuda(5))
</code></pre>
<p>I just mentioned the relevant code here, but I can provide other snippets if needed</p>

username: ptrblck
cooked: <p>What memory usage do you expect and how are you measuring it?<br>
Calculating the tensor shapes of all parameters and buffers as well as the intermediate activations and gradients would give you an approx. estimate of the memory footprint.<br>
Also, you could use the memory profiler to check where memory is allocated.</p>

username: Hamza_Benyamina
cooked: <p>I am measuring the GPU memory consumption. I am using the command nvidia-smi to check how much memory is allocated for the particular PID. Whenever I execute the code attached above, 11Gb is added to the memory allocated for the process (literally every time I do inference). That memory does not get unallocated until I restart the kernel.</p>

username: ptrblck
cooked: <p>I haven’t profiled the memory consumption of this model, but do you have an estimation what memory usage would be expected or why do you think it’s too much?<br>
Also note that <code>nvidia-smi</code> shows the memory used by the CUDA context, the allocated memory, and the cached memory (which can be reused by PyTorch without new allocations).</p>
Hyperparameter using skorch - How to set some optimizer as parameter?

username: dedeco
cooked: <p>Hi Guys,</p>
<p>I’m trying to set a dict with some parameters, like optimizers and others. How I should do?</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self):
        '''
        A feedForward neural network.
        Argurmets:
            n_feature: How many of features in your data
            n_hidden:  How many of neurons in the hidden layer
            n_output:  How many of neuros in the output leyar (defaut=1)
        '''
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(D_in, H, bias=True)   # hidden layer
        self.predict = torch.nn.Linear(H, D_out, bias=True)   # output layer
        self.n_feature, self.n_hidden, self.n_output = D_in, H, D_out
    def forward(self, x,**kwargs):
        '''
        Argurmets:
            x: Features to predict
        '''
        torch.nn.init.constant_(self.hidden.bias.data,1)
        torch.nn.init.constant_(self.predict.bias.data,1)
        x = torch.sigmoid(self.hidden(x))      # activation function for hidden layer
        x = torch.sigmoid(self.predict(x))     # linear output
        return x
</code></pre>
<pre><code class="lang-auto">from skorch import NeuralNetRegressor
net = NeuralNetRegressor(Net
                         , max_epochs=100
                         , lr=0.001
                         , verbose=1)
X_trf = X
y_trf = y.reshape(-1, 1)
</code></pre>
<pre><code class="lang-auto">from sklearn.model_selection import GridSearchCV
#from sklearn.metrics import SCORERS
#SCORERS.keys()

params = {
    'lr': [0.001,0.005, 0.01, 0.05, 0.1, 0.2, 0.3],
    'max_epochs': list(range(500,5500, 500))
}

gs = GridSearchCV(net, params, refit=False, scoring='r2', verbose=1, cv=10)

gs.fit(X_trf, y_trf)
</code></pre>

username: wqsgsutd
cooked: <p>Hi, dedeco,<br>
Have you solve this issue. I met same question.</p>
<p>Thanks and best regards<br>
Qiang</p>
Pytorch2onnx support return of List[Dict[str, Tensor]]

username: RunningLeon
cooked: <p>Hi, how to export models to onnx with output of List[Dict[str, tensor]], just like detection models in torchvision.</p>
<p>Following code mocking <a href="https://github.com/pytorch/vision/blob/1f4e3a46609df5550789ea681820cd77b186b515/torchvision/models/detection/roi_heads.py#L756" rel="noopener nofollow ugc">here</a>, but does not work.<br>
Any ideas?<br>
Thanks in advance.</p>
<pre><code class="lang-auto">### test return with dict
import torch
import onnx
import io
import onnx.helper as helper
from torch.jit.annotations import Tuple, List, Dict, Optional
from torch import Tensor

class DummyModel(torch.nn.Module):
    def __init__(self):
        super(DummyModel, self).__init__()

    def forward(self, x: Tensor, y: Tensor, z: int)-&gt;List[Dict[str, Tensor]]:
        res = torch.jit.annotate(List[Dict[str, torch.Tensor]], [])
        xy = x + y
        xz = x + z
        yz = y + z
        res.append({'xy' : xy, 'xz' : xz, 'yz' : yz})
        return res

model = DummyModel()
input_data = (torch.arange(4).reshape(2,2) for _ in range(3))

desired = model(*input_data)
print(f'Expect:\n{desired}')

# onnx_io = "sample.onnx"
onnx_io = io.BytesIO()
torch.onnx.export(
    model,
    input_data,
    onnx_io,
    verbose=False,
    input_names=None,
    output_names=None,
)

model_onnx = onnx.load(onnx_io)

# print(helper.printable_graph(model_onnx.graph))
sess = rt.InferenceSession(out_path)
input_names = [_.name for _ in sess.get_inputs()]
print(input_names)
#forward model
input_data_npy = [_.detach().cpu().numpy() for _ in input_data]
actual = sess.run(None, {name : data for name, data in zip(input_names, input_data_npy)})
actual = actual[0]
print(f'Actual:\n{actual}')
for k, v in desired.items():
    np.testing.assert_allclose(desired[k], actual[k])

</code></pre>

username: chi
cooked: <p>Hi, It seems that there is something wrong with the format of the input data.</p>
<p>I use <code>input_data = [torch.arange(4).reshape(2,2) for _ in range(3)]</code> and</p>
<pre><code class="lang-python">torch.onnx.export(
    model,
    tuple(input_data),
    onnx_io,
    verbose=False,
    input_names=None,
    output_names=None,
)
</code></pre>
<p>instead, and it works fine.</p>

username: RunningLeon
cooked: <p>Thanks a lot.<br>
the output are not dict[str, tensor], but list[tensor] and the names are not right.<br>
Do you have any suggestion with regard to this ?</p>
<pre><code class="lang-auto">Expect:
[{'xy': tensor([[0, 2],
        [4, 6]]), 'xz': tensor([[0, 2],
        [4, 6]]), 'yz': tensor([[0, 2],
        [4, 6]])}]
input_names=['0', '1', '2']
Actual:
[array([[0, 2],
       [4, 6]], dtype=int64), array([[0, 2],
       [4, 6]], dtype=int64), array([[0, 2],
       [4, 6]], dtype=int64)]
</code></pre>

username: chi
cooked: <p>As discussed in github, one option is using <code>desired_list, _ = torch.jit._flatten(desired)</code> to set the data type of the outputs in <code>pytorch</code> backend to a <code>List</code></p>
Gaussian Filter/Kernel how to apply?

username: Ripley
cooked: <p>Hi,</p>
<p>I am currently working on a semantic segmentation project based on satellite images,<br>
I was doing some research and I chance upon this thing called gaussian filter where the weights of the<br>
filter are concentrated in the middle. However I am confused as to what it really is and where do I apply this in my model. Is a gaussian filter/kernel a data augmentation? Is it the same as gaussian blur or gaussian noise? or issit something I have to imlpement on my conv layers when they have kernels inside. Hope someone can explain this to me thank you!</p>

username: ptrblck
cooked: <p>A Gaussian filter in image processing is also called Gaussian blur and is a low-pass filter.<br>
You can apply it on your images to blur them, if you think it might be beneficial for the training.<br>
It’s not the same as Gaussian noise, which is usually additive noise to the input signal.</p>
<p>You can also implement the Gaussian filters in a conv layer, but depending on your use case you might want to use <code>PIL</code> methods to add the blur to the images.</p>
<p>Where did you read about these filters and did the source explain where and how these filters are applied?</p>

username: Ripley
cooked: <p>I can’t really recall from where I read it, but yea was a little confused when doing my research on this. So like you mentioned, some of them were applying it as a data augmentation while some are applying it in their model when calling nn.Conv2D(…dilation=2). But I sort of understand the difference between these two already. Thank you for the explanation!</p>
<p>The case I am working with is a sen12ms dataset, which has 13 bands in a image. So I have to do my data augmenting on tensors instead of PIL image. As PIL can only work on a 3 channel image.</p>
Bug on torchvision.transforms.functional.resize?

username: Ripley
cooked: <p>Hi,</p>
<p>I encountered a strange problem where my input of a torch.tensor on torchvision.transforms.functional.resize works with visual studio code but when i try to run torchvision.transforms.functional.resize on jupyter notebook with the same input, it gives an error of this</p>
<p>‘’‘TypeError: img should be PIL Image. Got &lt;class ‘torch.Tensor’&gt;’’’</p>
<p>Both notebook running the same version of torchvision 0.7.0.</p>

username: BaruchG
cooked: <p>It’s hard to say without looking at your code, but check the type of whatever is being passed into <code>resize</code> to ensure that it’s a PIL image as expected (you mentioned that you are putting in a tensor and it expects a PIL image).  When weird issues come up in a jupyter notebook it might help to refresh the kernel to ensure that you are not using old variables as well.</p>

username: Ripley
cooked: <p>yea, the resize accepts either a PIL image or a torch.tensor as stated in the documentation, thats why I find it quite weird</p>

username: ptrblck
cooked: <p>Transformations accepting tensors were added in <code>0.8.0</code> as stated in the <a href="https://github.com/pytorch/vision/releases/tag/v0.8.0">release notes</a>, so I guess your VS environment might use the latest <code>torchvision</code> release, while your Jupyter env might use an older one.</p>

username: Ripley
cooked: <p>Thanks for this info! I updated it!</p>
QAT Achieving 99.999975% similarity, please help with last mile error

username: 111407
cooked: <p>Dear all,<br>
I have a Pruned Quantized MobileNet v2 model,<br>
and is now trying to simulate its inference from scratch.</p>
<p>Here is a brief sample of my code:<br>
(not the actual code, but very similar)<br>
(m is the layer)</p>
<pre><code class="lang-auto">           # scan through feature map _xin
           for _bgn_y in range(_xin_h):       
               for _bgn_x in range(_xin_w):   

                 all_channels = 0.0

                 # scan through kernel                   
                 for _kin in range(_w_input_ch): 
                         _ftmp = 0.0                                                                                                                                                                        
                        for _x in range(_w_kernel):                                                                                  
                            for _y in range(_w_kernel):                                                                              
                                fx = torch.dequantize(xin).numpy()[args.img_in_batch][_kin][_bgn_y+_y]_bgn_x+_x] 
                                fw = torch.dequantize(m.weight()).numpy()[_kout][_kin][_y][_x]                                       
                                _ftmp  += (fw*fx) 
                        all_channel += _ftmp
                 out[_kout][_bgn_y][_bgn_x] = (_fall_channel + m.bias()[_kout])/m.scale + m.zero_point
</code></pre>
<p>I assumed the results would be exactly the same as PyTorch’s Quantized Model.<br>
However, it’s 99.999975% the same:<br>
Out of the 40,million Feature Map parameters, only 1 or 2 points is off by 1.</p>
<p>After some investigation, I’ve found the points off were  .5 values,<br>
and are all randomly distributed.</p>
<p>Ex:<br>
My Calculation:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/cea59e4ad264e839e429ca8e8b6eedbdd72aba4a.png" alt="image" data-base62-sha1="tu5c3PPFHqJ3S0VWtKTFxehahKy" width="558" height="88"></p>
<p>pyTorch’s Output:<br>
72</p>
<p>It seems like its a rounding issue, so I have tried different Rounding methods, but all in vain.<br>
(I’m currently using the Bankers Rounding, the one python3 uses)</p>
<p>Any help would be appreciated!<br>
Thanks</p>
<p>Best wishes,<br>
James</p>
Torchscript made in PyTorch 1.7 not loading on 1.4

username: IsCoelacanth
cooked: <p>converted a model (VGG19 + with just 1 FC layer) using PyTorch 1.7, when I load it on a container with 1.4, it fails to load with a “version too old” error.<br>
Is this the intended behavior?</p>
<p><code>RuntimeError: version_ &lt;= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /pytorch/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 4, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /pytorch/caffe2/serialize/inline_container.cc:132)</code></p>

username: Kushaj
cooked: <p>torchscript maintains backward compatibility, not forward compatibility.</p>
Using torch.nn.DataParallel with torch.distributions.Laplace throws TypeError: 'Laplace' object is not iterable

username: Jimmy2027
cooked: <p>I am trying to train my model on multiple GPUs but I have some trouble with <code>torch.distributions.Laplace</code> that I call in the forward pass.<br>
I have uploaded a <a href="https://gist.github.com/Jimmy2027/a8b460180bc8bca757da4cc647df6595" rel="noopener nofollow ugc">minimal working example</a> that runs fine without torch.nn.DataParallel but fails when using it.</p>
<p>Is there any way to make this code run on multiple GPUs?</p>

username: Kushaj
cooked: <p>I wasn’t able to reproduce the error. On which pytorch version are you? I tested on 1.6.</p>

username: Jimmy2027
cooked: <p>hmm interesting, me too I am on 1.6.0. I checked again and found out that at least two GPUs need to be available to reproduce the error</p>

username: Kushaj
cooked: <p>That was dumb of me. I forgot on single GPU the code behaves identical.</p>

username: Kushaj
cooked: <p>Is there any specific reason for using <code>DataParallel</code> instead of <code>DistributedDataParallel</code>? I have experience with only single GPU machines so I don’t know about the details here.</p>

username: Jimmy2027
cooked: <p>no particular reason, I have just seen more examples using <code>DataParallel</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
But could be worth trying out if things look differently with <code>DistributedDataParallel</code></p>

username: Jimmy2027
cooked: <p><code>DistributedDataParallel</code> seems to work without problems, thanks for the hint <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I have uploaded a <a href="https://gist.github.com/Jimmy2027/26e18935b4da98f74478ead67f5141e4" rel="noopener nofollow ugc">gist</a> showing how the  code now runs with <code>world_size=2</code></p>