akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Select certain indices from a tensor

supervan: <p>I have two tensors A and B. A is a float tensor with shape (batch size, hidden dim). B is a Long tensor with shape (batch size, data len). What I want is somewhat like A[:, B], a float tensor still with shape (batch size, data len), the elements are certain indices from A which depends on B.</p>
<p>An example would be A=[[5, 2, 6], [7, 3, 4]] and B=[[0, 2, 1, 1], [2, 2, 1, 0]]. Then what I want is a tensor [[5, 6, 2, 2], [4, 4, 3, 7]]. Is there any way to achieve this?</p>
<p>I have tried A[:, B], but what I achieve is a tensor with shape (batch size, batch size, data len), which is a large tensor. And I only want the “diagonal value” from this tensor.</p>

albanD: <p>Hi,</p>
<p>I think you want: <code>A.gather(1, B)</code>.</p>

supervan: <p>Thanks for the reply! How about when A has another dimension, now the shape of A is (batch size, hidden dim, data dim). And B is still the same, a long tensor with shape (batch size, data len). I actually want to have the resulting tensor with shape (batch size, data len, data dim).</p>
<p>For example, A=[[[5], [2], [6]], [[7], [3], [4]]] and B=[[0, 2, 1, 1], [2, 2, 1, 0]]. Then what I want is a tensor [[[5], [6], [2], [2]], [[4], [4], [3], [7]]]. Basically now data dim is 1 as an example.</p>
<p>It seems that A.gather(1, B) requires A and B to have the same shapes.</p>

albanD: <p>If you add extra dimensions, you can simply expand B in these dimensions:</p>
<pre><code class="lang-auto">A.gather(1, B.unsqueeze(-1).expand(batch, data_len, data_dim)
</code></pre>
Something strange with torch.load

ShadowVolumeGuy: <p>Hi all,</p>
<p>I currently try to make prediction using this pretrained network <a href="https://github.com/meteorshowers/StereoNet-ActiveStereoNet" rel="nofollow noopener">StereoNet</a>.</p>
<p>After some effort (there must be some push errors on this repositery), I manage to load the pre-trained network. And unfortunately, a nice message warns me that my GPU is too low in RAM. So I removed everything that made an explicit call to cuda and specified :</p>
<p>device = torch.device(“cpu”)</p>
<p>and</p>
<p>model.to(device)</p>
<p>Then I ran the prediction on an image pair while having a terminal with nvidia-smi running in loop mode. The machine became unusable and I had the time to notice that python was consuming 500Mb of VRAM, obviously endless exchanges between CPU and GPU. After a reboot, I searched for which line of my python script the GPU was being used.<br>
It turns out that it was at the time of loading the network weights <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>data = torch.load(“checkpoint_pretrain_secneflow.pth”)[‘state_dict’]</p>
<p>I followed all the other lines avoiding the prediction step and at no other time did the use of VRAM move.</p>
<p>Does anyone have an idea to avoid this GPU usage and why the simple reading of a 2.6 MB file generates the use of 500 MB of VRAM before even attaching it to a network?</p>
<p>Thank you in advance for your help!</p>

albanD: <p>Hi,</p>
<p>The cuda state on the GPU is quite big. Taking few hundreds of MB.</p>
<p>To avoid loading on the GPU a model saved on the gpu, you can do: <code>torch.load('checkpoint_pretrain_secneflow.pth', map_location=torch.device('cpu'))</code>.</p>
How to initialize weight/bias data to specific vector without making it a leaf variable?

cruzas: <p>Hello everyone!</p>
<p>This is my first post here.</p>
<p>I have been trying to initialize bias data to a specific vector but have not found a way to do so without creating a leaf variable. I am coding in C++.</p>
<p>In the layer “affine2”, I can initialize the entries as follows:</p>
<pre><code class="lang-auto">affine2-&gt;bias[0] = 0.1;
affine2-&gt;bias[1] = 0.2;
affine2-&gt;bias[2] = 0.3;
</code></pre>
<p>However, later on, when trying to do backpropagation , this leads to the error “leaf variable has been moved into the graph interior”.</p>
<p>Does anyone know how to do exactly what I displayed without introducing leaf variables?</p>
<p>Thank you for your time.</p>

albanD: <p>Hi,</p>
<p>You want to make changes that are not tracked by the autograd here.<br>
So you should wrap these into a <code>NoGradGuard</code>.</p>

cruzas: <p>Thank you very much, albanD! It worked! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>In case anyone comes across this post in the future, this is how albanD’s solution ended up looking as for me:</p>
<pre><code class="lang-auto">{
   torch::NoGradGuard no_grad;
   affine2-&gt;bias[0] = 0.5;
   affine2-&gt;bias[1] = 1.0;
   affine2-&gt;bias[2] = 0.5;
}
</code></pre>
PyTorch 1.4 Release

ludwigwinkler: <p>Hi there,</p>
<p>I’m just curious how far along the amazing devs are on the PyTorch 1.4 release.</p>
<p>I’m really excited about the named tensor capabilities.</p>
<p>Is there a rough road map like March/April/May 2020?</p>
<p>Best and thanks for this amazing library!</p>

albanD: <p>The branch has been cut, binaries are being finalized. So, in a few days <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ludwigwinkler: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd410160250656ae0134b1a43186b9504ee8678f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/bd410160250656ae0134b1a43186b9504ee8678f" title="iu"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bd410160250656ae0134b1a43186b9504ee8678f_2_333x249.jpeg" alt="iu" data-base62-sha1="r0dvWKO0ULiux1j4xaPqt8gug47" width="333" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bd410160250656ae0134b1a43186b9504ee8678f_2_333x249.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bd410160250656ae0134b1a43186b9504ee8678f_2_499x373.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bd410160250656ae0134b1a43186b9504ee8678f_2_666x498.jpeg 2x" data-dominant-color="B79D5A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">iu</span><span class="informations">1160×870 294 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue_winking_eye.png?v=12" title=":stuck_out_tongue_winking_eye:" class="emoji" alt=":stuck_out_tongue_winking_eye:" loading="lazy" width="20" height="20"> … looking forward to it!</p>
Why does Transforms.ToTensor convert the range to 0..1?

asif_31_iqbal: <p>What’s the reason behind Transforms.ToTensor converting the value range from 0 to 1? Let’s say If the input image is in 0 - 255 scale (or any other scale for that matter), why not just leave it like that, and allow the user to later transform the data in whatever way he wants?</p>

SimonW: <p>Because in 99% of the cases and in convention when training networks we use [0, 1] image range, with possible normalization following to make it 0 mean and 1 std. You can write you own transform that does other things if you want to.</p>

Gedamu_Alemu: <p>Neural networks process inputs using small weight values, and inputs with large integer values can disrupt or slow down the learning process. As such it is good practice to normalize the pixel values so that each pixel value has a value between 0 and 1.</p>
Usage of drop_last on data_loader

oasjd7: <p>dataset_a : 60,000 images<br>
dataset_b: 100,000 images<br>
If I use <code>drop_last=True</code> in dataloader, Can I use whole data of dataset_b ?</p>
<pre><code class="lang-auto"> for step, (data_a, data_b) in enumerate(zip(data_loader_a, data_loader_b)):
...
</code></pre>

beaupreda: <p>Hello,</p>
<p>The <code>drop_last=True</code> parameter ignores the last batch (when the number of examples in your dataset is not divisible by your <code>batch_size</code>) while <code>drop_last=False</code> will make the last batch smaller than your <code>batch_size</code> (see <a href="https://pytorch.org/docs/1.3.0/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">docs</a>). This is not related to your issue of seeing or not the whole <code>dataset_b</code>.</p>
<p>In your case, you will not use the whole <code>dataset_b</code> because your <code>for</code> loop will only iterate over the smallest dataloader in your <code>zip</code> function <em>i.e.</em> <code>dataloader_a</code>. In other words, you will have <code>n</code> iterations (<code>step)</code> where <code>n = 60000 / batch_size</code>, which means that <code>40000</code> examples of <code>dataset_b</code> will not be seen.</p>
<p>Hope it clarifies a bit!</p>
Changing and using a pretrained output linear projection layer in LSTM

nilinykh: <p>I have a question specific to using some pretrained output linear projection layer of one LSTM network for another.</p>
<p><strong>What I have:</strong></p>
<ol>
<li>A pretrained fully connected layer which was used to output word probabilities into the actual words for some language generation model, something like this:</li>
</ol>
<p><code>self.output_linear_projection = nn.Linear(self.wordRNN_dim, self.vocab_size)</code></p>
<ul>
<li>Here, <code>self.wordRNN_dim</code> is 512 (hidden size of LSTM), and <code>self.vocab_size</code> is the number of words I have. For this pretrained model, vocabulary size is 10509, where it has 10508 words and the last element is projection of the <code>&lt;end&gt;</code> and <code>&lt;start&gt;</code> token (same projection is used for these tokens)</li>
</ul>
<ol start="2">
<li>My language generation model, for which I want to use the output linear projection layer from the pretrained model. Vocabulary of my model also includes <code>&lt;pad&gt;</code> token, which is absent in the pretrained model. So my vocabulary’s size is 10510 then.</li>
</ol>
<p><strong>The question:</strong></p>
<ul>
<li>if I want to use this pretrained output linear projection layer in my model, what should/could I do? Is it somehow conventional to not project <code>&lt;pad&gt;</code> tokens? If so, should I somehow ignore it in my projection and how? And if there are any steps I should take, any hints on how to do them in PyTorch?</li>
</ul>
Move image to frequency domain and calculate the gradient wrt to input image

Sam3: <p>Hi,</p>
<p>i want to move image to frequency domain by fft, how can i implemnt fft in Pytorch?</p>

Eta_C: <p><a href="https://pytorch.org/docs/stable/torch.html?highlight=fft#torch.fft" rel="nofollow noopener">torch.fft</a></p>
Save a dataset and use in another machine

YinYang_Untalan: <p>I want to take a dataset  i created from ImageFolder and save it into a file. I then will use the file in another computer.</p>
<p>I tried torch.save which was not good for datasets. It did save a file but it doesn’t bring the images with it, only the info it needs to build the dataset - so when I used it on another machine, it was looking for a directory from my other computer.</p>
<p>Thank you. I am new pytorch and would really appreciate any help.</p>

ptrblck: <p>Since <code>ImageFolder</code> loads the images lazily, i.e. in each call to its <code>__getitem__</code> method a single sample will be loaded from one of the folders, I don’t think you can simply store the <code>Dataset</code> in this way.<br>
A possible workaround would be to zip the parent folder containing all image folders and move it to your new machine.</p>

YinYang_Untalan: <p>I rescaled the pics along with a few transforms. I thought I could just save the dataset so that it would probably be smaller in storage. Thank you for the response.</p>
How to use saved dataloader?

oasjd7: <p>After save the data_loader with torch.save(), how to use this data_loader?<br>
<code>torch.save(data_loader, 'path/data_loader.pt')</code></p>

ptrblck: <p>What is your use case that you would like to save the <code>DataLoader</code>?<br>
Usually you would lazily load the data by calling into your <code>Dataset</code>'s <code>__getitem__</code>, which would mean that your <code>DataLoader</code> instance wouldn’t save anything.</p>

oasjd7: <p>Thanks for you answer.<br>
I have some list which have [features,labels] in training process.<br>
and I want to use like this.<br>
<code>loader = torchdata.DataLoader(list, batch_size=64, shuffle=True, drop_last=True, num_workers=2)</code></p>
<p>According to you answer, I have to save the list not the dataloader.<br>
Then how can I save this list?</p>

ptrblck: <p>You could store the list by writing to a file directly in Python or alternatively you could transform the list to a numpy array or PyTorch tensor and use their <code>save</code> methods. There are a lot of options how to store a list and the best approach depends on your actual use case.</p>
<p>I’m not sure, if you can pass the list directly to your <code>DataLoader</code>. The usual work flow is to create a <code>Dataset</code> and pass it to the <code>DataLoader</code>.<br>
Have a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">the data loading tutorial</a> for more information.</p>
Dataloader from numpy array without labels?

kavorka: <p>I have 100 images read as numpy array of shape (100,100,100,3). I want to create a dataloader from this array but without labels( For GAN). Please help me to create a dataloader for pytorch</p>

Eta_C: <pre><code class="lang-auto"># Dataset with labels
def __getitem__(self, item):
    x, y = get_image(item), get_label(item)
    return x, y

# Dataset without labels
def __getitem__(self, item):
    x = get_image(item)
    return x
</code></pre>

kavorka: <p>What is this get_image function?</p>

Eta_C: <p>It depends on you.<br>
If you do not know how to build a <code>Dataset</code> and <code>DataLoader</code>, here’s the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">tutorial</a>.</p>
Ensemble method

WanQianSucceed: <p>I am reading a paper named " IEA: Inner ensemble average within a convolutional neural network"<a href="https://arxiv.org/abs/1808.10350" rel="noopener nofollow ugc">https://arxiv.org/abs/1808.10350</a>  and the author proposed a structure as followed<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef3763abd7e421b8ffd6699aa5806b8ff96bc879.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef3763abd7e421b8ffd6699aa5806b8ff96bc879" title="QQ图片20200114223519"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef3763abd7e421b8ffd6699aa5806b8ff96bc879.png" alt="QQ图片20200114223519" data-base62-sha1="y8cMLvyXOQyViPfnaKc0j1VUsDD" width="690" height="312" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ图片20200114223519</span><span class="informations">712×322 51 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f0768fb929b4a4f265641a3a60e5a7fbc751b62.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f0768fb929b4a4f265641a3a60e5a7fbc751b62" title="微信图片_20200114222747"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f0768fb929b4a4f265641a3a60e5a7fbc751b62_2_690x465.png" alt="微信图片_20200114222747" data-base62-sha1="kpi9zbPp9OKbQCGBS0jRHJVAd0u" width="690" height="465" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f0768fb929b4a4f265641a3a60e5a7fbc751b62_2_690x465.png, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f0768fb929b4a4f265641a3a60e5a7fbc751b62.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f0768fb929b4a4f265641a3a60e5a7fbc751b62.png 2x" data-dominant-color="F0EEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20200114222747</span><span class="informations">793×535 44.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And  I am unable to understand the IEA method, because <strong>I think</strong> the average of the three conv, bn, and relu operations is <strong>the same result</strong> of one conv, bn, and relu operation.<br>
Can anyone explain it for me? Or give me some sample code to make me understand<br>
<strong>THANK YOU!!!</strong></p>
How to boradcast a mask?

John_Deterious: <p>I have a tensor <code>100 x 50 x 60</code>, and I have a mask acting on images  <code>50 x 60</code> and selects some 1000 pixels of interst (out of 30,000 in that image).</p>
<p>When I apply the mask to the tensor mentioned in the beginning, it throws an error and doesn’t broadcast to the 100 images. I want an output like this : <code>100 x 1000</code> i.e. the mask applied to 100 images (batch size).</p>
<p>If there’s no way to broadcast mask in Pytorch, how can I do it manually?</p>

John_Deterious: <p>I think I got it.</p>
<blockquote>
<p><strong>tensor[: , mask]</strong></p>
</blockquote>
<p>Just that. It works at least, but I don’t know if it is correct.</p>
Maping a list to new list

Majid_Shirazi: <p>Hi everyone,</p>
<p>I have ground_truth which is numpy list and is hot vector list of actions.</p>
<pre><code class="lang-auto">array([[0. , 1. , 0. ],
       [0. , 1. , 0. ],
       [0. , 1. , 0. ],
       ...,
       [0. , 0. , 0.8],
       [0. , 0. , 0.8],
       [0. , 0. , 0.8]], dtype=float32)
</code></pre>
<p>I want to map it like</p>
<pre><code class="lang-auto">{  '[-1.0, 0.0, 0.0]' :  '[1.0, 0.0, 0.0, 0.0]',
   '[+1.0, 0.0, 0.0]' : '[0.0, 1.0, 0.0, 0.0]',
   '[0.0, +1.0, 0.0]' : '[0.0, 0.0, 1.0, 0.0]',
   '[0.0, 0.0, 0.8]' :   '[0.0, 0.0, 0.0, 1.0]'  }
</code></pre>
<p>I did this loop below to make it but in some cases although my ground truth is sth else, but it gives me like [0.0, 0.0, 0.0, 0.0]. Would you please help me about it.</p>
<p>my already code is:</p>
<pre><code class="lang-auto">new_ground_truth=[]
z = [0 , 0 , 0]
for g in range (len(ground_truth)):
    blank = np.zeros(4)
    if(not(ground_truth[g] == z).all()):
        if(ground_truth[g][0]== -1): blank[0] = 1  
        if(ground_truth[g][0]== 1) : blank[1] = 1 
        if(ground_truth[g][1]== 1) : blank[2] = 1  
        if(ground_truth[g][2]==0.8): blank[3] = 1 
    else:
        None
    new_ground_truth.append(np.asarray(blank))
</code></pre>
<p>one of test cases which gives me wrong value:</p>
<pre><code class="lang-auto">ground_truth[4989] = array([0. , 0. , 0.8], dtype=float32)
but
new_ground_truth[4989]= array([0., 0., 0., 0.])
</code></pre>

Eta_C: <p>Hey, I have a faster solution.</p>
<pre><code class="lang-python">import numpy as np

gt = np.array([
    [-1.0, 0.0, 0.0],
    [1.0, 0.0, 0.0],
    [0.0, 1.0, 0.0],
    [0.0, 0.0, 0.8],
    [1.0, 0.0, 0.0]
])

new_gt = np.zeros((gt.shape[0], 4), dtype=gt.dtype)
new_gt[:, 1:] = gt &gt; 0
new_gt[:, 0] = gt[:, 0] == -1
print(new_gt)
</code></pre>
<p>Addition:<br>
<code>new_gt = new_gt[gt.sum(axis=1) != 0, :]</code> will filter out <code>[0., 0., 0.]</code> in <code>gt</code></p>
.to(device) not working properly

hyperactve: <p>I’m trying to run the following code and run a network on Cuda. However, it seems I’m running into problem as the error: <code>RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same</code> on the line <code>out = net(p)</code>.</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)
class Net(nn.Module):
    def __init__(self, input_channel=1, output_channel=1, num_filters=64, net_length=17):
        super(Net, self).__init__()
        self.net_length = net_length
        self.convs = []
        self.batch_norms = []
        self.convs.append(nn.Conv2d(in_channels=input_channel, out_channels=num_filters, kernel_size=3, padding=1))
        self.batch_norms.append(0)
        for i in range(net_length-2):
            self.convs.append(nn.Conv2d(num_filters, num_filters, 3, padding=1))
            self.batch_norms.append(torch.nn.BatchNorm2d(num_features = num_filters))
        self.convs.append(nn.Conv2d(num_filters, output_channel, 3, padding=1))
        #print(self.convs)
        
    def forward(self, x):
        #print(x.size())
        x = F.relu(self.convs[0](x))
        #print(x.size())
        for i in range(1,self.net_length-2):
            x = F.relu(self.batch_norms[i](self.convs[i](x)))
            #print(x.size())
        x = self.convs[self.net_length-1](x)
        #print(x.size())
        
        return x
net = Net()
net.to(device)

print(net)

p = torch.as_tensor(np.random.rand(1,1,32,32).astype(np.float32)).to(device)
out = net(p)
</code></pre>
<p>I’m not sure, what is wrong with my code. I think declaring all the weights in a list caused the problem. Any remedy to this issue?</p>

ptrblck: <p>Your modules won’t be properly registered, if you append them in a plain Python list.<br>
Use <code>nn.ModuleList</code> instead and the <code>model.to()</code> call should push all parameters to the device.</p>
KFAC impelementation for batchnorm

gebrahimi: <p>Is there an implementation for kfac that can handle batchnorm? I found this implementation of kfac but this does not support batchnorm. I need some hint on how to extend it to batchnorm. I greatly appreciate it.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/alecwangcq/KFAC-Pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/13307310?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/alecwangcq/KFAC-Pytorch" target="_blank" rel="nofollow noopener">alecwangcq/KFAC-Pytorch</a></h3>

<p>Pytorch implementation of KFAC and E-KFAC (Natural Gradient). - alecwangcq/KFAC-Pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Error visualizing my Unet output

kishanbala: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/neda">@Neda</a> I referred to your discussion to map my multiclass segmentation problem. The output shape of my network matches that of my input i.e [8, 4, 572, 572] 8 is my batch size and 4 is  the number of classes in my image including the background. I tried this code snippet to visualize my network output<br>
output = model(images)<br>
print(output.size()) #*[8,4,572,572]<br>
prob = F.softmax(output, 1)<br>
print(prob.size())# [8,4,572,572] outputs match<br>
prob_imgs = torchvision.utils.make_grid(output.permute(1, 0, 2, 3))<br>
print(prob_imgs.size()) <span class="hashtag">#RuntimeError:</span> The size of tensor a (3) must match the size of tensor b (8) at non-singleton dimension 0<br>
I dont know where I am wrong. Could someone help me out.</p>

kishanbala: <p>update: output = model(images)<br>
print(output.size())<br>
pred = torch.argmax(output, dim=0).detach().cpu()<br>
plt.imshow(pred.permute(1,2,0))<br>
plt.show()</p>
<p>This code snippet solved this issue</p>
Can I set the batch_size of lstm model to be None like tf in pytorch

cmcai0104: <p>I want to create a nn model by pytorch to implement reinforcement.</p>
<p>I need to use this model to interact with the environment, so I want to set the batch_size=1.</p>
<p>But the model cannot perform well, so I want to train the model firstly by supervised learning in order to enhance the performance of my model, so I want to set the batch_size&gt;1(like 64) to accelerate the training process.</p>

albanD: <p>Hi,</p>
<p>In pytorch, the batch size is not a property of the model.<br>
You can give an input of any batch size as input and it will give you the corresponding output with the same batch size.</p>
Should I one_hot the category feature before send the feature to Embedding layer?

kailing_ye: <p>For example, I have the category feature named fruit that contains apple, banana, peach, pear. Should I do one hot encoding before sending to the Embedding layer?</p>

albanD: <p>Hi,</p>
<p>You usually associate an index to each word. The Embedding layer takes the indices as input, not one-hot encoding of these indices.</p>
Transferring constant member `Tensors` to correct GPUs with `DataParallel`

gspr: <p>My network (an <code>nn.Module</code> subclass) has a <em>constant</em> member tensor <code>A</code> involved in some of the computation in its <code>forward</code> method. If I wrap the network in <code>nn.DataParallel</code> to run it across multiple GPUs, I get complaints that <code>A</code> resides on a different GPU than the incoming data (correctly split across the batch dimension by <code>DataParallel</code>). What do I need to do for <code>DataParallel</code> to copy <code>A</code> to each separate GPU when it duplicates the network to each GPU?</p>

albanD: <p>Hi,</p>
<p>You can register your constant Tensors with <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_buffer"><code>self.register_buffer()</code></a> on the <code>nn.Module</code> so that it gets moved around with the Module.</p>
About using two optimizers

oasjd7: <p><code>Overall architecture: [model_1] →  [model_2]</code></p>
<pre><code class="lang-auto">optimizer_1 = SGD(model_1.parameters())
optimizer_2 = SGD(model_2.parameters())

out_1 = model_1(input)
loss_1 = criterion(out_1)
optimizer_1.zero_grad()
loss_1.backward()
optimizer_1.step()

out_2 = model_2(out_1.detach())
loss_2 = criterion(out_2)
optimizer_2.zero_grad()
loss_2.backward()
optimizer_2.step()
</code></pre>
<p>When I train only model_1 , I can get 90% accuracy,<br>
but If I add train model_2, model_1 accuracy decreased significantly.<br>
Why model_2 affect to model_1 even if I use different optimizer and detach()</p>

albanD: <p>Hi,</p>
<p>If you do this exactly, it should not change anything.<br>
You do compute the accuracy based only on <code>out_1</code>?</p>
About masked or filtered output layers

Hb121234: <p>Hello,<br>
I’m trying to implement the A3C reinforcement learning model with pytorch.<br>
I am currently planning to construct a model in which the number of valid (legal) action depends on the state.<br>
I have studied and searched a lot, but have concluded that there is no single answer, and I am using a mask on the output layer in the way I think is most logically correct.<br>
We decided to use a soft max function as the output layer and implemented it.<br>
But in detail, there are some problems. First, my mask is not fixed but a dynamically changing mask. As a result, it is not possible to conclude that the backpropagation process works normally.<br>
I think it should be backpropagated only for units that make valid action in the soft max layer. Is this right? Also, how does Pytorch implement this?</p>
<p>*I would also appreciate comments and tips on how to filter out valid actions in reinforcement learning.</p>
Create heatmap over image

Carsten_Ditzel: <p>I would like to create a heatmap that is overlaid on RGB imagery.</p>
<p>The map is the result of some attention mapping and has the same height and width as the camera image but only one channel consisting of values between 0 and 1, corresponding to probabilities. Is there a natural way to use this map to highlight areas on the camera images in a heatmap-like fasion prior to saving the image via torch.save_image() ?</p>
<p>thanks in advance</p>

ptrblck: <p>You could try to use <code>PIL.Image.blend</code> to blend the heatmap with your image.<br>
Here is a small dummy example:</p>
<pre><code class="lang-python"># Create heatmap image in red channel
heatmap = torch.empty(1, 252, 271).uniform_(0, 1)
heatmap = torch.cat((heatmap, torch.zeros(2, 252, 271)))

import torchvision.transforms.functional as TF
img = TF.to_pil_image(x)  # assuming your image in x
h_img = TF.to_pil_image(heatmap)

res = Image.blend(img, h_img, 0.5)
</code></pre>

Carsten_Ditzel: <p>thank you patrick!</p>
<p>I suppose this only works for batch sizes of 1 and cpu-based tensors, am I right?</p>

ptrblck: <p>Yes, the current code uses a single image, since <code>Image.blend</code> and the transformations won’t take a batch of images. Usually the number of images you would like to visualize isn’t that large, so that it shouldn’t be a problem.</p>
<p>If you are working with CUDA tensors, just push them to the CPU first using <code>tensor = tensor.to('cpu')</code>.</p>

Carsten_Ditzel: <p>thank you! I am a little confused as to why the displayed image has its coloring all messed up…</p>
<p>edit: Ok the method does no scaling by default and so I have to rescale the 8bit image to the range (0,1)</p>

edgarriba: <p>you can perform such operation in batch using <code>kornia.add_weighted</code></p>
<p>See: <a href="https://kornia.readthedocs.io/en/latest/color.html#kornia.color.AddWeighted" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/color.html#kornia.color.AddWeighted</a></p>
Product of two tensors of size 4

abd: <p>Hi,<br>
u= torch.randn(3, 1, 40, 64)<br>
v= torch.randn(1, 3, 40, 64)<br>
hawo can I compute the product of u and v.torch.bmm not work for this example.</p>
<p>Thanks,</p>

albanD: <p>What do you mean by product here? matrix-matrix multiplication? Or broadcasting + element-wise?</p>

abd: <p><a class="mention" href="/u/alband">@albanD</a><br>
Multiplication</p>

albanD: <p>What is the output size you expect?? [3, 40, 64]? [1, 40 64]? [3, 3, 40]?</p>

Eta_C: <p>Let me guess…<br>
Do you mean</p>
<pre><code class="lang-auto">u -&gt; (A, B, batch_1, batch_2)
b -&gt; (B, A, batch_1, batch_2)
result -&gt; (A, A, batch_1, batch_2)
</code></pre>
<p>which <code>result[:, :, i, j]</code> equals to <code>u[:, :, i, j] @ v[:, :, i, j]</code>?<br>
Just like a 4-dim bmm?</p>

abd: <p>Hi@Eta_C<br>
Exactly the result will be :<br>
(A, A, batch_1, batch_2)<br>
What do you mean by i and j?</p>

Eta_C: <p>If I understand correctly…<br>
Try this, you should build a small example to test its correctness.</p>
<pre><code class="lang-python">import torch

u = torch.randn(3, 1, 40, 64)
v = torch.randn(1, 3, 40, 64)
_u = u.reshape(3, 1, -1)
_v = v.reshape(1, 3, -1)
_result = torch.bmm(_u.permute(2, 0, 1), _v.permute(2, 0, 1))
result = _result.permute(1, 2, 0).reshape(3, 3, 40, 64)
</code></pre>
Using Conv3d with Dataset that has different depths between volumes

Faraz: <p>I am working with Medical Images, where I have 130 Patient Volumes, each volume consists of  <code>N</code>  number of DICOM Images/slices.</p>
<p>The problem is that between the volumes the the number of slices  <code>N</code> , varies.</p>
<p>Majority, 50% of volumes have 20 Slices, rest varies by 3 or 4 slices, some even more than 10 slices (so much so that interpolation to make number of slices equal between volumes is not possible)</p>
<p>I am able to use Conv3d for volumes where the depth  <code>N</code>  (number of slices) is same between volumes, but I have to make use of entire data set for the classification task. So how do I incorporate entire dataset and feed it to my network model ?</p>

ptrblck: <p>Similar to varying spatial sizes in images, you could try to resize the DICOM volume.<br>
However, since you are dealing with medical images, the resampling might not be straightforward.<br>
<a class="mention" href="/u/christianperone">@christianperone</a> might have a good idea as the author of <a href="https://github.com/perone/medicaltorch">MedicalTorch</a>. I couldn’t find a resampling method for the z-dimension, but I might have missed it while skimming through the code.</p>
<p>The last time I had to resample a DICOM image I used ITK for it, but I assume there should be easier methods by now. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

christianperone: <p>So, there is not a single answer to that as there are many approaches. But you’ll probably want first to resample (offline, not during mini-batch sampling) to a common voxel space, which is a very important step to make sure you have the same physical meaning across axes for different volumes. Now, what people usually do when resampling is not an option, where you have very different volume sizes, is the same that is done on 2D, you do inference and training using patches, which when extended to 3D are cubes. That is also the same reason why not a lot of people use 3D, because it is very cumbersome to work with. Everything I said is usually done for segmentation, but as it seems, you’re doing a classification task, and for that you’ll have more options, such as using final layers that are independent of the spatial size, such as global average pooling, etc. You can also add padding in the other images to avoid issues with the batching, etc. Anyway, there are many approaches and all these approaches will highly depend on your application domain because differently than training with natural images, medical imaging have a lot of peculiarities depending if it is CT, MRI, etc. I would consider converting the DICOM to NIfTI as well as they are much easier to work with. Hope it helped a little, but again, you should consider your application domain when taking these decisions.</p>
JIT android issue

vishalagarwal: <p>I have created JIT model of the object detection SSD network and it runs fine when I run the traced model on Python or C++. I am following PyTorch mobile’s tutorial to load the network on phone. But when I do forward pass of the same traced model in Java environment of Android Studio, it is waits on that line for hours and the execution is completed. Anyone with any pointers what might be the issue?<br>
Help appreciated.</p>

IvanKobzarev: <p>Hello <a class="mention" href="/u/vishalagarwal">@vishalagarwal</a> ,<br>
Which version of pytorch android api do you use 1.3, 1.3.1 or nightly builds?</p>
<p>We had a deadlock issue that was fixed in  <a href="https://github.com/pytorch/pytorch/pull/29885" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/29885</a></p>
<p>If you are not using nightly builds, could you please check if it reproduces on nightlies?(<a href="https://github.com/pytorch/pytorch/tree/master/android#nightly" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/android#nightly</a>)</p>

vishalagarwal: <p>I am using pytorch 1.3.1<br>
Let me try with nightly build</p>

vishalagarwal: <p>It resolved with the nightly build. Thanks!</p>
How to split Autoencoder to two NNs?

John_Deterious: <p>I have an autoencoder with 2 submodules (trained and ready). How to split it to two separate NNs each with its own weights and forward method?</p>

ptrblck: <p>How did you define your current model?<br>
If both submodules are instances of <code>nn.Sequential</code>, you could recreate them separately outside of your current model and just load the <code>state_dicts</code> into them.</p>
<p>If your modules and forward is more complicated, you could still redefine the submodules, but would most likely have to filter out unwanted keys from the <code>state_dict</code> of the complete model before loading it into the submodels.</p>

John_Deterious: <p>Thank you <strong>ptrblck</strong> . I figured if my ultimate goal is to split the AE, then the best thing is build two different NNs for encoder and decoder, and then combine them in a third one for training purpose.</p>

ptrblck: <p>That sounds like the best idea! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Question about learning rate

chunchun: <p>if the gradient  range from 1e^(-5) to 1e^(-4),what orders of magnitude should the learning rate be?<br>
Thanks a lot.</p>

crowsonkb: <p>I don’t know—I think the optimal learning rate has to do with other aspects of the problem besides gradient scale. That said, SGD and other optimizers like Adam use the learning rate in different ways. Adam automatically adapts the step size to the gradient scale on a per-parameter basis; its ‘learning rate’ hyperparameter sets the maximum any parameter may change per step. Often Adam’s default learning rate of 1e-3 is ‘good enough’ for most problems and people don’t tune it further. But try and see!</p>

chunchun: <p>Thank you very much.I have got the idea.</p>
Backprop Through Sparse Tensor Is Not Memory Efficient?

Kevin96: <p>Hi,</p>
<p>When I computed the gradient of a sparse tensor on GPU, I got out of memory issue.</p>
<p>I need a tensor of size <code>[1, 3, 224 * 224, 224 * 224]</code> which only has 1 * 3 * 224 * 224 * 25 nonzero entries. So I store it in sparse_coo format.</p>
<p>I take the summation over the last dim, then I get a tensor of size <code>[1, 3, 224 * 224]</code>:</p>
<pre><code class="lang-auto">b = torch.sparse.sum(a, dim=3).to_dense()
</code></pre>
<p>I reshape the tensor into size <code>[1, 3, 224, 224]</code>:</p>
<pre><code class="lang-auto">b = b.view(1, 3, 224, 224)
</code></pre>
<p>and then feed it into a neural network.</p>
<p>The memory that required to excute this computation should be very small. The forward pass is fine. However, I encouter an error during backward</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 28.14 GiB
</code></pre>
<p>It seems like pytorch creates some tensor of size <code>[1, 3, 224 * 224, 224 * 224]</code> during backward, which exactly consume that amount of memory.</p>
<p>I am wondering whether I misuse the sparse tensor functionality such that autodiff construct the full dense matrix explictly during backprop?</p>

albanD: <p>Hi,</p>
<p>I’m afraid that the backward of the <code>to_dense()</code> does not create a sparse Tensor. And so it tried to create a full size Tensor <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
The support for sparse Tensors is quite limited so we don’t usually create them in the backward to reduce the number of errors like “XXX op is not implemented for sparse Tensors”.</p>

Kevin96: <p>Thanks for clarification!</p>
<p>Hopefully, in my case, the backprop operation is quite simple. I have implemented an customized backprop operation to solve the memory issue.</p>
Parameter definition

sinaabdollahi: <p>Do I need to define/register Parameter for each of the Variables in my PyTorch code? or if I just define Linear/etc. it automatically assigns weights and updates the weights during training phase?</p>

ptrblck: <p>A layer derived from <code>nn.Module</code> or an <code>nn.Parameter</code> assigned as an attribute inside a custom model, will be registered as a model parameter and thus will be returned by calling <code>model.parameters()</code>.</p>
<p>To optimize these parameters, you would have to pass them to the optimizer.<br>
If you want to train all registered parameters, you could simply use:</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
</code></pre>
<p>On the other hand, you can pass a list with whatever parameters you would like to optimizer.<br>
E.g. you could pass only the parameters of a specific layer as:</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD(model.my_layer.parameters(), lr=1e-3)
</code></pre>
<p>Have a look at e.g. <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py">this tutorial</a> to learn more about the PyTorch work flow.</p>
Problems with requires_grad

smu226: <p>Hello! I have this simple (part of a) code:</p>
<pre><code class="lang-auto">a = torch.tensor(0.5)
a.requires_grad=True
lr = 1e-3
for i in range(20):
    yy = torch.cos(a*x) 
    loss = torch.mean((yy-y)**2)
    loss.backward()
    a = a-a.grad.data
</code></pre>
<p>Where x and y are fixed numbers, generated using y=(0.538*x). Basically I want to make a go from 0.5 to 0.538 using gradient descent.</p>
<p>The first loop works fine and a goes in the right direction, but at the second loop I am getting this error at the <code>a = a-a.grad.data</code> line: <code>AttributeError: 'NoneType' object has no attribute 'data'</code>. What am I doing wrong? What changes from the first to second iteration? Thank you!</p>

albanD: <p>Hi,</p>
<p>Few things:</p>
<ul>
<li>Do not use <code>.data</code>, it’s not a thing anymore and causes a lot of bugs. If you want to make a weight update here, do :</li>
</ul>
<pre><code class="lang-auto">with torch.no_grad():
  a -=  a.grad
</code></pre>
<ul>
<li>And your real issue is that when you do <code>a = xxx</code>. Then you re-assign <code>a</code> in a differentiable manner. Meaning that your new <code>a</code> is not a leaf anymore. Using the code in the point above will change <code>a</code> inplace and it will thus remain a leaf.</li>
<li>(EDIT) Also <code>a.grad.zero_()</code> inside the no_grad block as <a class="mention" href="/u/crowsonkb">@crowsonkb</a> mentioned below</li>
</ul>

crowsonkb: <p>Try this instead:</p>
<pre><code class="lang-python">a = torch.tensor(0.5)
a.requires_grad = True
lr = 1e-3
for i in range(20):
    yy = torch.cos(a*x)
    loss = torch.mean((yy-y)**2)
    loss.backward()
    with torch.no_grad():
        a -= lr * a.grad
        a.grad.zero_()
</code></pre>
<p>The following also works (but may be less efficient?) :</p>
<pre><code class="lang-python">    with torch.no_grad():
        a = a - lr * a.grad
        a.requires_grad = True
</code></pre>
<p>The problem you are running into is that when you do <code>a = a - a.grad</code> you are getting a new <code>a</code> which does not have <code>requires_grad</code> set. So either you need to do in-place operations on <code>a</code> (preserving the flag) or manually re-enable it. Also, the gradient descent step has to be done inside a <code>no_grad()</code> context.</p>

kai-tub: <p>I was too slow…<br>
But if you are interested in a nice introductory tutorial, which helped me understand the basics, take a look at this medium article:<br>
<a href="https://towardsdatascience.com/understanding-pytorch-with-an-example-a-step-by-step-tutorial-81fc5f8c4e8e" rel="nofollow noopener">https://towardsdatascience.com/understanding-pytorch-with-an-example-a-step-by-step-tutorial-81fc5f8c4e8e</a><br>
I covers exactly the problem you were having and goes a bit more into detail about what is happening. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Multiple parallel networks

jondapper: <p>Hello. I have a network that has a few separate encoding branches, the latent features produced by which are combined towards the end. Something like:</p>
<p>self.encode_1 = Encode()<br>
self.encode_2 = Encode()<br>
self.encode_3 = Encode()</p>
<p>def forward(self, x):<br>
y1 = self.encode_1(x)<br>
y2 = self.encode_2(x)<br>
y3 = self.encode_3(x)<br>
y_final = y1 + y2 + y3</p>
<p>This works fine, but I don’t like the fact that I have hard-coded 3 branches. What can I do to allow for a variable number of branches?</p>
<p>Thanks!</p>

albanD: <p>You can do:</p>
<pre><code class="lang-auto">nb_branches = 3

# Assuming you are in a nn.Module
self.encoders = nn.ModuleList()
for i in range(nb_branches):
  self.encoders.append(Encode())

def forward(self, x):
  y_final = 0
  for encoder in self.encoders:
    y_final += encoder(x)
</code></pre>
Help or Suggestion Needed in Designing CNN

jmandivarapu1: <p>I am designing an network in which I am passing an vector and Image as Input.</p>
<p>I wonder how can I design such network and loss.</p>
<p>Approach 1 (Not preffered): Flatten the Image then concat with that vector and send it to linear layer. But this won’t capture any features of the input image.</p>
<p>Approach 2: Can I design something combination of Convolution and linear at the input level itself? If so How can I do that.</p>
<p>Approach 3 : Have Two different networks one for convolution and other for the latent vector and combine them at later. But I fear this might lead to loss not getting decreased</p>
<p>Any Help or Opinion or Suggestion would be great.</p>
<p>I am referring to this below post for designing it</p>
<aside class="quote" data-post="3" data-topic="66551">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmandivarapu1/48/3544_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-i-run-2-parallel-cnn-on-same-data-then-combine-outputs-at-fc-layer/66551/3">How can I run 2 parallel CNN on same data then combine outputs at FC layer?</a> 
  </div>
  <blockquote>
    How to back propagate the loss?
  </blockquote>
</aside>
Questions about changing input size when fine tuning fcn_resnet101 segmentation model

alex_d: <p>I am fine tuning a pre-trained fcn resnet101 segmentation model with my own data set and would like to change the input size to something like (750, 1000) instead of (224, 224).</p>
<p>My understanding is that this model was originally trained on images that are 224 by 224, but I would like to keep the size of my images and its mask 750 by 1000 because I would like to get a more accurate, higher level of detail from the segmentation. If I simply shrink my images to 224 by 224 the images will become too pixelated, and the segmented part will most likely lose some information that I care about such as the shape/contour of the segment.</p>
<p>I am following this <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">tutorial</a> and this is the part where I see input size is set:</p>
<pre><code class="lang-auto">def initialize_model(model_name, num_classes, feature_extract, use_pretrained=True):
    # Initialize these variables which will be set in this if statement. Each of these
    #   variables is model specific.
    model_ft = None
    input_size = 0

    if model_name == "resnet":
        """ Resnet18
        """
        model_ft = models.resnet18(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.fc.in_features
        model_ft.fc = nn.Linear(num_ftrs, num_classes)
        input_size = 224
</code></pre>
<p>So if I want to make the input size 750 by 1000, would I simply change the value of the input_size variable to 750, 100 (so input_size = 750,1000)?</p>
<p>Would I have to change any other parameters besides input_size so I don’t get errors when fine tuning?</p>
<p>And finally are there any potential negative consequences of fine tuning a model with a different, bigger input size than the original pretrained model? I think a potential one might be that it would take more time and GPU power to fine tune on bigger size images because when I used the the pretrained model out of the box to evaluate images that were about 750 by 1000 it took significantly more time and GPU resources to predict the segments, but overall I got decent results on those larger images. I am more concerned about negative effects like loosing accuracy, not so much loosing speed.</p>