akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Floating point addition

cbd: <p>I found strange behaviour while adding two tensor. Here “mat” tensor is output of mathematical equation and its precision value is 0 after decimal point ( I checked by placing torch.set_printoptions(precision=10) before print). After that if i do addition for below code it gives me output “98310.<strong>1016</strong>” instead of “98310.<strong>1</strong>”. Its very simple but i don’t know from where <strong>“.1016”</strong> comes in answer instead of <strong>“.1”</strong>.</p>
<pre><code class="lang-auto">##    mat tensor([[98310., 98310., 98310.],
        [98310., 98310., 98310.],
        [98310., 98310., 98310.]], device='cuda:0')  


        e1=(torch.eye(3,3)*0.1).to(0)
        print("**e1 ",e1)
        mat=torch.add(mat,e1)</code></pre>

ptrblck: <p>The small abs. error is most likely created due to the limited floating point precision and a different order of operations between your reference the the current code.</p>

cbd: <p>I agreed but if “mat” has some floating point value why its not showing even after using “torch.set_printoptions(precision=10)” before print of “mat” variable? Just forgot to mention “mat” is of size [1,15876,3,3]</p>
<p>Even i tried to subtract “0.0016” from final answer using below code but output remains same as input. Both datatype is FloatTensor.</p>
<pre><code class="lang-auto">e2=(torch.eye(3,3)*0.0016).to(0)
mat=torch.sub(mat,e2)
</code></pre>
<p>I change the code as below then its subtracting integer value but not floating point value. So output is  “98309.<strong>1016</strong> ” instead of “98309.<strong>1</strong> ”</p>
<pre><code class="lang-auto">e2=(torch.eye(3,3)*1.0016).to(0)
mat=torch.sub(mat,e2)
</code></pre>
Errors related to torch.nn.Upsample

nikiguo93: <p>Hi, I have an image with size of 3 * 500 * 500 (channel *width *height). I want to use torch.nn.Upsample to resize this image into the one with size of 3 * 256 * 256. but when I use the folowing code:<br>
upsample = torch.nn.Upsample(256)<br>
image = upsample(image)<br>
it returned an image with size of 3 * 500 * 256<br>
can anybody tell me how to fix this problem?</p>

ptrblck: <p>Your input is most likely missing the batch dimension so that <code>[3, 500, 500]</code> would be treated as a temporal signal with <code>[batch_size=3, channels=500, seq_len=500]</code>. <code>unsqueeze</code> the batch dimension and it should work:</p>
<pre><code class="lang-python">image = torch.randn(3, 500, 500)
upsample = torch.nn.Upsample(256)
image = upsample(image)
print(image.shape)
# &gt; torch.Size([3, 500, 256])

image = torch.randn(1, 3, 500, 500)
upsample = torch.nn.Upsample(256)
image = upsample(image)
print(image.shape)
# &gt; torch.Size([1, 3, 256, 256])
</code></pre>
Custom dataset for time-series data for an LSTM model

errezeta: <p>Hi,<br>
I have a *.csv file with time-series data that I want to load in a custom dataset and then use dataloader to get batches of data for an LSTM model.<br>
I’m struggling to get the batches together with the sequence size.<br>
This is the code that I have so far. I’m not even sure if I suppose to do it this way:</p>
<pre><code class="lang-auto">class CMAPSSDataset(Dataset):
    def __init__(self, csv_file, sep=' ', sequence_length=40):
        self.df_cmapss = pd.read_csv(csv_file, sep=sep)
        self.df_data = self.df_cmapss.iloc[:, 3:27]
        self.targets = self.df_cmapss['RUL']
        self.sequence_length = sequence_length

    def __len__(self):
        return len(self.df_data)

    def __getitem__(self, idx):
        if (idx+self.sequence_length) &gt; len(self.df_data):
            indexes = list(range(idx, len(self.df_data)))
        else:
            indexes = list(range(idx, idx + self.sequence_length))
        data = self.df_data.iloc[indexes, :].values
        target = self.targets.iloc[indexes].values
        return torch.tensor(data), torch.tensor(target)

cmapss_dataset = {x: CMAPSSDataset(csv_file='data/CMAPSSData/'+x+'_FD001.csv', sep=' ')
                  for x in ['train', 'test']}

batch_size = 32
dataloaders = {x: DataLoader(cmapss_dataset[x], batch_size=batch_size,
                             num_workers=0, pin_memory=True)
               for x in ['train', 'test']}
</code></pre>
<p>I get the following error when I iterate through the dataloader:<br>
<code>RuntimeError: stack expects each tensor to be equal size, but got [40, 24] at entry 0 and [39, 24] at entry 16</code><br>
Any help would be appreciated.</p>

ptrblck: <p>Based on the error message it seems you are dealing with variable sequence lengths.<br>
The <code>collate_fn</code> of the <code>DataLoader</code> tries to <code>stack</code> the samples into a single batch, which would fail if the dimensions (besides the batch dim) don’t have the same shape. You could e.g. pad the smaller sequences to the largest sequence length in the batch to be able to create a full batch.</p>

errezeta: <p>Hi, Thank you very much for your answer. I used the <code>collate_fn</code> functions in this way:</p>
<pre><code class="lang-auto">def collate_batch(batch):
    data = [item[0] for item in batch]
    data = pad_sequence(data, batch_first=True)
    targets = [item[1].unsqueeze_(0) for item in batch]
    targets = pad_sequence(targets, batch_first=True)
    return data, targets
</code></pre>
<p>is this correct?</p>

ptrblck: <p>Yes, your code looks good. I would check the created batch inside the <code>DataLoader</code> loop and see, if all values are expected as a quick sanity check.</p>
Pytorch summary - parameters count

Davide_Murari: <p>I am experimenting with various deep networks and I always want to know how may parameters are involved. I was using pytorch summary however I noticed that if I use many times the same module in the forward pass, its associated parameters are counted multiple times.</p>
<p>An example is this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self,):
        super(Net, self).__init__()
        self.lin = nn.Linear(3,3)

    def forward(self,x):
      x = self.lin(x)
      x = self.lin(x)
      x = self.lin(x)
      x = self.lin(x)
      return x
net = Net()
from torchsummary import summary
summary(net.to(device),(1,3))
</code></pre>
<p>where you get 48 Total params, which is 12*4. 12, in this case, is actually the number of trainable parameters of the network.</p>
<p>Thus, my question is, is there a way to make Pytorch summary print out the number of “single” trainable parameters of the model?</p>
<p>Otherwise, I know to use a script like this</p>
<pre><code class="lang-auto">model_parameters = filter(lambda p: p.requires_grad, net.parameters())
params = sum([np.prod(p.size()) for p in model_parameters])
print(f"The network has {params} trainable parameters")
</code></pre>
<p>to get the desired result, but I like how pytorch summary works.</p>
Semantic segmentation codes for large images

Sylvain_Ard: <p>Hi,<br>
I am looking for semantic segmentation codes for example on github working with large images (about 4000 pixels width) for segment fine details.<br>
Could you give me some codes URL please ?<br>
Thank you<br>
Best regards</p>

Sylvain_Ard: <p>nobodyt has an answer ?</p>
Integrated Gradients - Convergence Delta as quality measure?

brix: <p>Hi,<br>
I would like to use integrated gradients from captum to analyze a neuronal net:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://captum.ai/api/integrated_gradients.html">
  <header class="source">
      <img src="https://captum.ai/img/captum.ico" class="site-icon" width="256" height="256">

      <a href="https://captum.ai/api/integrated_gradients.html" target="_blank" rel="noopener nofollow ugc">captum.ai</a>
  </header>

  <article class="onebox-body">
    <img src="https://captum.ai/img/captum-icon.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://captum.ai/api/integrated_gradients.html" target="_blank" rel="noopener nofollow ugc">Captum · Model Interpretability for PyTorch</a></h3>

  <p>Model Interpretability for PyTorch</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>What can the return parameter “delta” be used for?<br>
What does a high delta actually mean for practical usage?</p>
<p>Can or should I use it as a quality parameter to remove samples having a high absolute convergence delta before doing further processing of obtained gradients?</p>
<p>Thanks</p>
Questions for train and validation loss

Chin_GM: <p>Dear all,</p>
<p>I got a question regarding the train and validation loss.</p>
<p>What is the normal practice to check?</p>
<ol>
<li>loss per epoch?</li>
<li>loss per batch?</li>
</ol>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/438a59ac7d423e48b5924f9267bd6148f3401e51.png" alt="image" data-base62-sha1="9Dup5smRoOCglOk2DLsswrfSiFX" width="370" height="287"><br>
blue refers to train loss per epoch<br>
orange refers to validation loss per epoch</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c80e6ae56fd51a748eb561a3a5687ec261ee99e3.png" alt="image" data-base62-sha1="sxMnMt8yTxWMxPnsvCeMyaYobir" width="380" height="283"><br>
training loss per batch</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/1/4184eb869dd90b54c7481a30c50d2c1d784d8c20.png" alt="image" data-base62-sha1="9lBOys9WNlRnB42ZjI7ua2sdUQg" width="364" height="300"><br>
validation loss per epoch</p>
<p>If we check on loss per epoch, it is a good fit.<br>
But if we compare the loss per batch, it is overfit since val loss higher than train loss.</p>
<p>Anyone can please explain which should we observe in normal practice. loss per epoch or per batch?</p>
<p>Thanks</p>
Binning tensor values

erap129: <p>Hello,<br>
I am looking for a way to bin tensor values.<br>
Say my tensor has values between -0.255 and +0.256, I am looking for a way to represent this will “less numbers”.<br>
For example I would like to bin this range into 32 values so I will only have the values -0.255, -0.239, -0.223 and so on… (the bins are in jumps of 16 because 512/32=16)<br>
I found torch.histc but it gives me only a histogram, this way I lose “which index was mapped to each bin”.<br>
Thanks!</p>

nemz: <p>I’m having the same issue. Have you found a clean solution for this?</p>

Christoph_Heindl: <p><a class="mention" href="/u/erap129">@erap129</a> , <a class="mention" href="/u/nemz">@nemz</a> I guess you are looking for quantization.</p>
<p>The following snippet ‘bins’ values to the nearest multiple of <code>0.05</code></p>
<pre><code class="lang-auto">In [17]: a = torch.rand(10)

In [18]: a
Out[18]:
tensor([0.1523, 0.5210, 0.9864, 0.3272, 0.3473, 0.5408, 0.3397, 0.4553, 0.0236,
        0.7338])

In [19]: b=0.05

In [20]: torch.round(a/b)*b
Out[20]:
tensor([0.1500, 0.5000, 1.0000, 0.3500, 0.3500, 0.5500, 0.3500, 0.4500, 0.0000,
        0.7500])
</code></pre>
Boolean value of Tensor with more than one value is ambiguous when idexing

RadAlienware: <p>Hello advance sorry for being noob. I am currently doing a cnn scratch project. Here in maxpooling class in backpropagation the condition for the indexing is giving error of boolean ambiguous. I am giving the screenshots below!</p>
<p><a href="/uploads/short-url/7dw8fQ0XBVBdNarp8WMlyYr3aFA.jpeg">Screenshot - 10_15_2021 , 9_41_37 PM|690x322</a></p>
<p>here is the error</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83118a0cd1c9c04702bf14c3fd1c8e1224a292d9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/83118a0cd1c9c04702bf14c3fd1c8e1224a292d9" title="Screenshot - 10_15_2021 , 9_40_12 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83118a0cd1c9c04702bf14c3fd1c8e1224a292d9_2_690x217.jpeg" alt="Screenshot - 10_15_2021 , 9_40_12 PM" data-base62-sha1="iHu7PyDWUL4y92c3PRxbwp96n33" width="690" height="217" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83118a0cd1c9c04702bf14c3fd1c8e1224a292d9_2_690x217.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83118a0cd1c9c04702bf14c3fd1c8e1224a292d9_2_1035x325.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/83118a0cd1c9c04702bf14c3fd1c8e1224a292d9_2_1380x434.jpeg 2x" data-dominant-color="3C3C3D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot - 10_15_2021 , 9_40_12 PM</span><span class="informations">1759×555 89.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The error is raised if you are trying to use more than a single <code>bool</code> value in an if condition:</p>
<pre><code class="lang-python">x = torch.tensor([True, False])
if x:
    print('lala')
# &gt; RuntimeError: Boolean value of Tensor with more than one value is ambiguous
</code></pre>
<p>Depending on your use case, you might want to use <code>.any()</code>, <code>.all()</code>, or split the tensor to take different code paths.</p>

RadAlienware: <p>Thanks. I have tried without using if condition. Now it works fine !</p>
[need urgent help] Chatbot tutorial is not working

nayany31: <p>Hi, I am making <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Chatbot Tutorial — PyTorch Tutorials 1.9.1+cu102 documentation</a> This Pytorch tutorial</p>
<p>If loadFilename= None its working but</p>
<p>I am training , loadFilename = “4000_checkpoint.tar” it says</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "torch_chatbot.py", line 617, in &lt;module&gt;
    print_every, save_every, clip, corpus_name, loadFilename)
  File "torch_chatbot.py", line 438, in trainIters
    for _ in range(n_iteration)]
  File "torch_chatbot.py", line 438, in &lt;listcomp&gt;
    for _ in range(n_iteration)]
  File "torch_chatbot.py", line 232, in batch2TrainData
    inp, lengths = inputVar(input_batch, voc)
  File "torch_chatbot.py", line 209, in inputVar
    indexes_batch = [indexesFromSentence(voc, sentence) for sentence in l]
  File "torch_chatbot.py", line 209, in &lt;listcomp&gt;
    indexes_batch = [indexesFromSentence(voc, sentence) for sentence in l]
  File "torch_chatbot.py", line 190, in indexesFromSentence
    return [voc.word2index[word] for word in sentence.split(' ')] + [EOS_token]
  File "torch_chatbot.py", line 190, in &lt;listcomp&gt;
    return [voc.word2index[word] for word in sentence.split(' ')] + [EOS_token]
KeyError: 'can'
</code></pre>
<p>I need help please</p>

ptrblck: <p>The vocabulary is missing the <code>can</code> work. Check how the vocabulary was created and why the word is missing.</p>
Learning rate new object difference

H3LL0FR13ND: <p>Hi,</p>
<p>Is there any difference between changing the learning rate via param group vs creating a new object itself with a reduced learning rate as parameter?</p>

ptrblck: <p>Assuming “object” refers to an <code>optimizer</code> here: yes, it could make a difference especially if the optimizer track internal states, as they would be kept in the former and reset in the latter case.</p>
ToTensor float vs uint8

mreister: <p>I have a PIL image i want to convert to a tensor, but when i do this it converts the data from [0 -255] to [1.0 - 0.0]. How do i get the ToTensor() function to convert to a tensor of uint8?</p>

ptrblck: <p>You could use <code>from_numpy</code> to transform the type from a numpy array to a PyTorch tensor without any normalization:</p>
<pre><code class="lang-python"># create or load PIL.Image
tmp = np.random.randint(0, 255, (224, 224, 3)).astype(np.uint8)
img = PIL.Image.fromarray(tmp)

# get numpy array
arr = np.array(img)
tensor = torch.from_numpy(arr)
print(tensor)
</code></pre>
<p>You might additionally want to <code>permute</code> the axes to make the tensor channels-first, which is the expected default format.</p>
How do I stack columns of a 2D tensor sequentially

minesh_mathew: <p>If I have a 2D tensor (matrix)  of size say (5,6), how do I transform it into a (10,3) matrix  where each column in the resultant matrix is formed by stacking two consecutive columns in the initial matrix, with a step size of 2. That is, first two columns in the initial matrix are stacked to form the first column in the resultant matrix , 3rd and 4th columns in the initial matrix stacked to obtain the second column in the new matrix and likewise</p>
<p>In a more general sense I want to slide a window, width-wise where all columns in a window are stacked to form single column</p>

ptrblck: <p>This should work for your shapes:</p>
<pre><code class="lang-python">x = torch.arange(5*6).view(5, 6)
print(x)

y = x.permute(1, 0).contiguous().view(3, 10).permute(1, 0)
print(y)
</code></pre>
<p>I’m unsure what the most efficient way would be so let’s wait if others can come up with a better approach.</p>

minesh_mathew: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="134319">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>x.permute(1, 0)</code></p>
</blockquote>
</aside>
<p>Thank you for the quick response. For a more general case, suppose If I want to move a sliding window with overlapping columns, is there any approach without writing any loops.</p>
<p>For example in <code>y</code>  I want first column to be first and second columns of <code>x</code> stacked, and second column of <code>y</code> to be second and third columns of <code>x</code> stacked together.</p>

ptrblck: <p>To create windows you could use <code>tensor.unfold</code>, which is also known as the <code>im2col</code> method.</p>
Pytorch + pytest + multigpu

sh0416: <p>Hi, I want to test the equality of the gradient on multi-gpus and single-gpu.</p>
<p>The problem is that I don’t have multi-gpu in my local machine.</p>
<p>The question is that could I test the behavior of my code in multi-gpu without multi-gpu environment?</p>
<p>When reminding of my undergrads experience, there is a mockup object that mimics the real behavior to resolve this issue.</p>
<p>Thanks,</p>
Using for loop to increase number of hidden layers

Rjacob: <p>Hi Folks,<br>
I am trying to increase the number of hidden layers in my existing code-</p>
<pre><code class="lang-auto">D = 32*32
n = 64 
C = 1
classes = 10

fc_model = nn.Sequential(

    nn.Flatten(),

    nn.Linear(D,  n), # I/P layer

    nn.Tanh(),
 # Hidden layers
    nn.Linear(n,  n),
    nn.Tanh(),
    nn.Linear(n,  n),
    nn.Tanh(),

    nn.Linear(n, classes),

)
</code></pre>
<p>How would be able to write say a for loop which will repeat the hidden layers multiple times ? If i try to write a for loop within sequential, it throws a syntax error.</p>

KFrank: <p>Hi Ronish!</p>
<aside class="quote no-group" data-username="Rjacob" data-post="1" data-topic="134343" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/958977/48.png" class="avatar"> Rjacob:</div>
<blockquote>
<p>How would be able to write say a for loop which will repeat the hidden layers multiple times ?</p>
</blockquote>
</aside>
<p>You can use a loop to insert your desired layers into a python<br>
<code>OrderedDict</code> and then construct your <code>Sequential</code> by passing<br>
in that <code>OrderedDict</code> as a single constructor argument (rather than<br>
passing in all of the layers as individual constructor arguments).</p>
<p>Best.</p>
<p>K. Frank</p>
Is it possible use torch.argsort output to index a tensor

AgentDS007: <p>Assume I have a model that takes as input a sequence of vectors i.e. <code>[B,N,C] = input.shape</code> with B as batch dimension, N as sequence length and C as latent dim.</p>
<p>Additionaly at some point in my model I get a tensor <code>w</code>, that is of<br>
shape <code>[B,N]</code> that weights each element of my sequence i.e. <code>w.sum(dim=1) == [1., 1., 1., ...]</code> of size <code>[B,]</code></p>
<p>What the follwing code showes is my attempt to get the top k sequence elements of my input <code>X</code> that have the highest values in <code>w</code>:</p>
<pre><code class="lang-auto">X = torch.rand(4,7,128)
w = torch.rand(4,7)
w_sorted = torch.argsort(w, dim = 1, descending = True)
w_trunc = w_sorted[:,:5]

X_topk = torch.empty(X.shape[0], 5, 128)
for i in range(X.shape[0]):
    X_topk[i,:,:] = X[i,w_trunc[i,:],:]  
</code></pre>
<p>Is there is a better way to do this without the loop and directly indexing multiple dimensions of <code>X</code> using <code>w_trunc</code>?</p>

KFrank: <p>Hi Agent!</p>
<aside class="quote no-group" data-username="AgentDS007" data-post="1" data-topic="134327" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/agentds007/48/43032_2.png" class="avatar"> AgentDS007:</div>
<blockquote>
<p>Is there is a better way to do this without the loop and directly indexing multiple dimensions of <code>X</code> using <code>w_trunc</code>?</p>
</blockquote>
</aside>
<p>Yes, you can create appropriate indices and use pytorch tensor<br>
indexing to avoid the loop.  (You can also gain a bit of efficiency<br>
by using <code>torch.topk()</code> to perform the typically cheaper <em>partial<br>
sort</em> rather than the full sort used by <code>torch.argsort()</code>.)</p>
<p>Here is an illustrative script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2021)

B = 4
N = 7
C = 128
k = 5

X = torch.rand (B, N, C)
w = torch.rand (B, N)
w_sorted = torch.argsort (w, dim = 1, descending = True)
w_trunc = w_sorted[:, :k]

# method with loop
X_topk = torch.empty(X.shape[0], k, C)
for i in range(X.shape[0]):
    X_topk[i, :, :] = X[i, w_trunc[i, :], :]

# method with no-loop indexing and partial instead of full sort
i0 = torch.arange (B).unsqueeze (-1).unsqueeze (-1).expand (B, k, C)
i1 = torch.topk (w, k, dim = 1).indices.unsqueeze (-1).expand (B, k, C)
i2 = torch.arange (C).expand (B, k, C)

X_topk_B = X[i0, i1, i2]

print ('X_topk_B.equal (X_topk):', X_topk_B.equal (X_topk))
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.9.0
X_topk_B.equal (X_topk): True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
What's the most efficient way to get the Hessian of the loss with respect to all parameters for all input samples?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I was wondering what’s the most efficient way of getting the Hessian of the loss with respect to all parameters for all samples? I’ve done something like this for the Jacobian of my network and I did it via using hooks and just manually defining the gradients as the terms from the hooks (within an einsum op).</p>
<p>Any help would be greatly appreicated!</p>
<p>Thank you!</p>
The derivative for 'running_mean' is not implemented

avalon1511: <p>Trying to update the <code>batchnorm.running_mean</code> to a new value during training, where <code>new_mean = old_mean * x + b</code></p>
<p>Seems like batchnorm parameters are not trainable and cannot be manipulated - is there a workaround to this?</p>

AlphaBetaGamma96: <p>Because it’s a <code>running_mean</code> it’s updated as an moving average between the current estimate of the mean and the old value rather than being done via 2 tunable parmeters. It’s explained within the docs <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html?highlight=batchnorm#torch.nn.BatchNorm1d" class="inline-onebox" rel="noopener nofollow ugc">BatchNorm1d — PyTorch 1.9.1 documentation</a></p>

avalon1511: <p>Yeah, I am trying to get it to update with tunable parameters though, wondering if this is possible at all</p>

AlphaBetaGamma96: <p>Quite possibly, you could create a <code>nn.Module</code> class with learnable parameters and just defined the forward pass to be the same BatchNorm? That might work?</p>
Custom optimizer

sandeep1: <p>I am not able to understand the basics why the model.parameter is not given as parameter to the optimizer. And how a new parameter optimizer_grouped_parameters is created ?</p>
<pre><code class="lang-auto">text_model=AADP()
text_model.to(device)
criterion = nn.BCELoss()

no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
    {'params': [p for n, p in text_model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
    {'params': [p for n, p in text_model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.01}
]
optimizer = AdamW(optimizer_grouped_parameters, lr=3e-5)
</code></pre>

ptrblck: <p><code>optimizer_grouped_parameters</code> is creating two parameter groups by checking the <code>no_decay</code> names in each parameter. It seems that no <code>weight_decay</code> should be used in one of the cases. However, given that both param groups are using <code>'weight_decay': 0.01</code>, you could also remove this filtering.</p>
Creating the dataset kills the kernal of jupyter notebook

ElQadasi: <p>I am trying to read the images (CIFAR 10 = 50000 training sample) from the local machine and creating the dataset. The problem is that the kernel of Jupyter is killed abruptly. I tried to use the kernal of anaconda by running  file_name.py in the terminal but I have the same issue. When I debugged the problem, the kernel was killed sometimes after loading 25000 images and sometimes after 45000 images. I would mention that I resized the images from 32<em>32 to 224</em>224 to fit the input of ResNet-18.</p>
<p>Do you think it is resources-based problem? Moreover, I did not use GPU while loading images to create a dataset because I will use GPU in the training step.</p>
<p>The kernel dies before it finishes this code (inside Dataset Class):</p>
<pre><code class="lang-auto"> for sub_dir in os.listdir(self.dir_path):
     files_lst = os.listdir(os.path.join(self.dir_path,sub_dir))
     for name in files_lst:
         directory = os.path.join(self.dir_path,sub_dir,name)
         image = Image.open(directory)
         image = transform(image)
         data_lst.append(image)
         target_lst.append(sub_dir)

  self.dataset = torch.stack(data_lst, 0)
  self.labels = torch.tensor(target_lst,dtype=torch.long)
</code></pre>
<p>The specifications of my laptop are: 11th Gen intel core vPro i9, RTX A4000 Laptop GPU, 32Gb RAM.</p>
<p>Thanks for any help.</p>

ptrblck: <p>I think you might be running out of host memory.<br>
Check the memory usage via <code>htop</code> or run the script in a terminal to get a better error message.</p>
Possible to install PyTorch 1.2 with CUDA 11?

truGeneral7: <p>Hi, I wonder would that be possible to install an early version of Pytorch, such as 1.2 with CUDA 11?</p>
<p>I want to run an opensource code which was compiled under PyTorch 1.2 and CUDA 10.0. I set up an anaconda environment with the same config. However, I keep getting the error “RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED”, despite Re-installing Pytorch and CUDA of the same versions.</p>
<p>I have a A6000 GPU, and I did not get such errors with Pytorch 1.9 with CUDA 11, but the opensource code did not seem to be compatible with Pytorch 1.9, I’ve got a lot of computability issues with numpy.</p>
<p>I appreciate any kind of help! Thank you!</p>

ptrblck: <p>You would have to check the CUDA11 PRs in <a href="https://github.com/pytorch/pytorch/pulls">GitHub</a>, checkout the <code>1.2</code> branch, cherry-pick these PRs into this old branch, solve merge conflicts, and rebuild this version locally.<br>
It depend how much time you would like to spend on this workflow, but I would guess that updating the old PyTorch script to be compatible with <code>1.9.1</code> might be easier and the better approach.</p>
Help with dimensions for LSTM

jekasm: <p>First time on the forum, glad to be here! I intend on making torch my go-to for deep learning.</p>
<p>So I am trying to run a tensor with dim:<br>
[# texts, # sequential words, len of 1-hot for each word]<br>
[2500,13,100]</p>
<p>target is a single string per text: [2500,1]</p>
<p><strong>so I’m trying to map a sequence of 13 one-hot vectors of len 100 to 1 string for 2500 instances.</strong></p>
<p>heres my code:</p>
<pre><code class="lang-auto">D_in, D_out = ([torch_x.shape[1], torch_x.shape[2]]), torch_y.shape[1]
print(D_in, D_out)
[13, 100] 1

H1, H2, H3 = 500, 800, 300


class RNN(nn.Module):
    def __init__(self, D_in, H1, H2, H3, D_out):
        super(RNN, self).__init__()
        
        self.lstm = nn.LSTM(D_in, H1)
        self.linear1 = nn.Linear(H1, H2)
        self.linear2 = nn.Linear(H2, D_out)
        
        
   
</code></pre>
<pre><code class="lang-auto">def forward(self, x):
        y_pred, states = self.lstm(x)
        y_pred         = self.linear1(y_pred).clamp(min=0)
        y_pred         = self.linear2(y_pred)
        return y_pred
model = RNN(D_in, H1, H2, H3, D_out)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4 * 2)
criterion = nn.MSELoss(reduction='sum')
</code></pre>
<pre><code class="lang-auto">
losses = []
for t in range(50):
    y_pred = model(torch_c)
</code></pre>
<pre><code class="lang-auto">    
    loss = criterion(y_pred, torch_y)
    print(t, loss.item())
    losses.append(loss.item())
    
    if torch.isnan(loss):
        break
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>and the error for this is:</p>
<p>—&gt; 14         y_pred, states = self.lstm(x)<br>
RuntimeError: input.size(-1) must be equal to input_size. Expected 13, got 100</p>
<p>if I change  D_in  to a 1-D of 100 , it’s:</p>
<p>----&gt; 5     loss = criterion(y_pred, torch_y)<br>
RuntimeError: The size of tensor a (13) must match the size of tensor b (2500) at non-singleton dimension 1</p>
<p>any thoughts on how I can do this?<br>
thanks so much!</p>

AbdulsalamBande: <p><a class="mention" href="/u/jekasm">@jekasm</a><br>
I think the dimension of [# texts, # sequential words, len of 1-hot for each word] is okay, <strong>BUT</strong> make sure the <code>batch_first argument</code> of the lstm is <strong>True</strong> just as the documentation suggested .<a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="inline-onebox" rel="noopener nofollow ugc">LSTM — PyTorch 1.9.1 documentation</a></p>
How to pass word embeddings and inputs to torch.nn.MultiheadAttention?

ADONAI_TZEVAOT: <p>In the  <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">documentation</a> it says that the inputs to <code>Multiheadattention</code> are <code>key, query and value</code>.<br>
I have the following 2 doubts :</p>
<ol>
<li>How do I specify the inputs (word embeddings) on which the multiheadattention has to be performed ?</li>
<li>What should be the key, query and value inputs to the function? Aren’t they the weights that the network will learn ?</li>
</ol>

AbdulsalamBande: <p><a class="mention" href="/u/adonai_tzevaot">@ADONAI_TZEVAOT</a><br>
So for  self attention, the key , query and value are all the same. So if your input is for example ‘how are you’ and you have embedding dim of 300. The Key , Query and Value will all be <code>[3,300]</code> , since you have 3 words.<br>
For the second doubt. Key, Query and Value are not the weights, they are your inputs. Pytorch creates random weights behind the scenes. You can  watch this video for a detailed explanation <a href="https://www.youtube.com/watch?v=_iDanMWVj98&amp;ab_channel=PytorchModulesExplained" class="inline-onebox" rel="noopener nofollow ugc">Self Attention with torch.nn.MultiheadAttention Module - YouTube</a></p>
GRU overfit problems

Chin_GM: <p>Dear all,</p>
<p>I faced some problems with my overfitting model.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/6301523d7b8e8a899490e08d2bcc726f4e10daa5.png" alt="image" data-base62-sha1="e7Q6dI39bVlbpF5Y2f8bhELLF3f" width="473" height="354"><br>
grey refers to validation loss<br>
orange refers to train loss.</p>
<p>batch_size = 16</p>
<p>My model as per below:<br>
class SpeedClassificationPredictionModel(nn.Module):</p>
<pre><code>def __init__(self, n_features, n_hidden=12, n_layers=2):

    super().__init__()

    

    self.n_hidden = n_hidden

    

    self.gru = nn.GRU(

        input_size=n_features,

        hidden_size=n_hidden,

        batch_first=True,

        num_layers=n_layers,

        dropout=0.8

    )



    self.regressor = nn.Linear(n_hidden, 1)   # output single number as prediction



def forward(self, x):

    self.gru.flatten_parameters()    # sort out gpu memory during distributed training

    _, hidden = self.gru(x)           # for gru

    out = hidden[-1]                 # get the output from last layer

    

    return self.regressor(out)
</code></pre>
<p>loss function: MSE loss<br>
optimizer: AdamW(self.parameters(), lr=0.0001, weight_decay=0.1)</p>
<p>I tried everything to reduce the overfit, but the outcome still the same.</p>
<p>Is anyone can give me some advice?</p>
<p>Thanks</p>

AbdulsalamBande: <aside class="quote no-group" data-username="Chin_GM" data-post="1" data-topic="134350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chin_gm/48/30412_2.png" class="avatar"> Chin_GM:</div>
<blockquote>
<p>alidation</p>
</blockquote>
</aside>
<p>First of all I will advise you remove the dropout as it is usually used for large networks and try using an Adam Optimizer <code>optimizer = torch.optim.Adam(net.parameters(), lr=0.001)</code></p>
Memory efficiency filter with arbitrarily connected kernel

YutongZheng: <p>Hi, I’m trying to implement a filter layer with arbitrary kernel connections. For example:</p>
<pre><code class="lang-auto">def ArbitrayConnectFilter(x, kernel, connection):
    '''
       x: input tensor with shape (spatial_in, channels),
       kernel: filter kernel with shape (spatial_out, K),
       connection: indices of x that connected to each kernel with shape (spatial_out, K)
    '''
        x = x[connection.flatten()].view(*connection.shape, -1) # (spatial_out, K, channels)
        x = (x.permute(0,2,1) @ kernel.unsqueeze(2)).squeeze(2) # (spatial_out, channels)
    return x
</code></pre>
<p>But I figured that the intermediate indexing step might take too much memory if the kernel size K is too large. Is there any way to save memory and do the operation without caching the indexed intermediate tensor? Thanks!</p>
Learning the learning rate via meta-learning

blade: <p>I’m trying to use a meta-learning approach to learn the learning rate of my optimization algorithm. An inner-loop optimization (SGD) learns the model parameters and an outer-loop optim learns the learning rate.  A regular SGD has the form:</p>
<pre><code>for group in self.param_groups:
    grad = torch.autograd.grad(loss, group['params'], create_graph=True)
    for idx, p in enumerate(group['params']):
        with torch.no_grad():
            p.add_(grad[idx], alpha=-group['lr'])
</code></pre>
<p>Lets define parameter <code>lr</code> as my learning rate. The way I understand, for the outer-loop optimization to compute the gradient of this parameter, I need to remove <code>torch.no_grad()</code> from my inner-loop optimization, so that backprop sees the parameter <code>lr</code> in my computational graph. However, doing so results in the following error:</p>
<blockquote>
<p>RuntimeError: a leaf Variable that requires grad is being used in an<br>
in-place operation.</p>
</blockquote>
<p>How should I approach this?</p>
Docker Template for Universal PyTorch Source-Builds that gives (hopefully) x10 Speed Boost

veritas: <aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/veritas9872/PyTorch-Universal-Docker-Template">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/veritas9872/PyTorch-Universal-Docker-Template" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/2806abdb9bc33cab61c32d8beb0334c42479043e5ccfb4a8e8ce91fdcf77469e/veritas9872/PyTorch-Universal-Docker-Template" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/veritas9872/PyTorch-Universal-Docker-Template" target="_blank" rel="noopener nofollow ugc">GitHub - veritas9872/PyTorch-Universal-Docker-Template: Template repository...</a></h3>

  <p>Template repository to build PyTorch projects from source on any version of PyTorch/CUDA/cuDNN. - GitHub - veritas9872/PyTorch-Universal-Docker-Template: Template repository to build PyTorch projec...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Dear fellow PyTorch users, I would like to propose a project that may increase your run speeds by x10.</p>
<p>A tremendous academic effort has gone into the design and implementation of efficient neural networks in recent years to cope with the ever-increasing amount of data on ever-smaller and more efficient devices. Yet, as of the time of writing, most researchers are unaware of even the most basic acceleration techniques for deep learning on GPUs.</p>
<p>Especially in academia, many do not even use Automatic Mixed Precision (AMP), which can reduce memory requirements to 1/4 and increase speeds by x4~5. This is the case even though AMP can be enabled without much hassle using the PyTorch Lightning or HuggingFace Accelerate libraries.</p>
<p>Even the novice who has only just dipped their toes into the murky depths of deep learning knows that more compute is a key ingredient for success. No matter how brilliant the scientist, outperforming a rival with x10 more compute is no mean feat.</p>
<p>I have created a repository with the aim of enabling researchers and engineers without much knowledge of GPUs, CUDA, Docker, etc. to squeeze every last drop of performance from their GPUs <strong><em>using the same hardware and neural networks</em>.</strong></p>
<p>If you are among those who before could only long for a quicker end to the hours and days spent staring at Tensorboard as your models inched past the epochs, this project may be just the thing for you. When using a source build of PyTorch with the latest version of CUDA, combined with AMP, one may achieve training/inference times x10 faster than a naïve PyTorch environment.</p>
<p>I sincerely hope that my project will be of service to practitioners in both academia and industry.</p>
<p><strong>Please show your appreciation by starring my repository.</strong></p>
Moving a Tensor to another GPU modifies the data

cwitkowitz: <p>The following code snippet reproduces the error:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.__version__)
1.9.0+cu102
&gt;&gt;&gt; test = torch.tensor([0.5]).to('cuda:0')
&gt;&gt;&gt; test
tensor([0.5000], device='cuda:0')
&gt;&gt;&gt; test.to('cuda:1')
tensor([0.], device='cuda:1')
</code></pre>
<p>Oddly enough, this only happens on one of my machines.</p>

ptrblck: <p>This could indicate an issue in your node setup and a broken communication.<br>
Could you run the <a href="https://github.com/NVIDIA/nccl-tests">NCCL-tests</a> and see, if any issues are detected while communicating between the devices?</p>

cwitkowitz: <p>Is NCCL required? I didn’t have it installed previously. I’m not using both of the GPUs for a single task. I’m simply trying to transfer data from ‘cuda:0’ to ‘cuda:1’ offline.</p>
<p>It is interesting to note that when running the nccl-tests, I am seeing a lot of lines with “IO_PAGE_FAULT”.</p>

ptrblck: <p>Yes, NCCL would be needed to run the tests.<br>
The <code>IO_PAGE_FAULT</code> issues are also most likely shown in <code>dmesg</code> when you run your previous code snippet, which could point towards a faulty communication between these devices.<br>
Try to disable <code>IOMMU</code> as described <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs">here</a>.</p>

cwitkowitz: <p>Thanks for pointing me in the right direction. You were right about the <code>IO_PAGE_FAULT</code> messages being shown in <code>dmesg</code>. I was able to fix the problem by following <strong><a href="https://github.com/ngimel" rel="noopener nofollow ugc">ngimel</a></strong>’s comment in <a href="https://github.com/pytorch/pytorch/issues/1637" class="inline-onebox" rel="noopener nofollow ugc">Multi-GPU K80s · Issue #1637 · pytorch/pytorch · GitHub</a>.</p>
<p>In my previous comment, I was wondering whether NCCL is required in general (not only for the tests), since everything seemed to be working without it (besides my issue in this post). Could you please elaborate on how/when NCCL would be beneficial?</p>

ptrblck: <p>NCCL is one backend for device communications between NVIDIA GPUs. PyTorch provides multiple backends (gloo, MPI) so you could pick one of them.</p>
<p>Good to hear you were able to solve the issue!</p>
Torchvision pretrained models perform bad on very simple images classification

Fnjn: <p>I tried torchvision pretrained models vgg16_bn and resnet50 to classify simple images, but they perform really bad.</p>
<p>Here are their performance examples:<br>
vgg16_bn<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9ec6222315ec1bc39ce1c34f298018afb592af24.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/9ec6222315ec1bc39ce1c34f298018afb592af24" title="Screenshot%20from%202018-03-31%2000-25-23"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9ec6222315ec1bc39ce1c34f298018afb592af24_2_284x500.jpg" alt="Screenshot%20from%202018-03-31%2000-25-23" data-base62-sha1="mEzWdMUmz00yQqtmAVyflztbEry" width="284" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9ec6222315ec1bc39ce1c34f298018afb592af24_2_284x500.jpg, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9ec6222315ec1bc39ce1c34f298018afb592af24_2_426x750.jpg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/9/9ec6222315ec1bc39ce1c34f298018afb592af24.jpg 2x" data-dominant-color="E8E5ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202018-03-31%2000-25-23</span><span class="informations">460×809 86.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>resnet50<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e6462279f3fba1646c02e770984bb31ed21160f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/e6462279f3fba1646c02e770984bb31ed21160f3" title="resnet50"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e6462279f3fba1646c02e770984bb31ed21160f3_2_300x499.png" alt="resnet50" data-base62-sha1="wR651xZD9xsJMpAfA1aNBV6fRHJ" width="300" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e6462279f3fba1646c02e770984bb31ed21160f3_2_300x499.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e6462279f3fba1646c02e770984bb31ed21160f3_2_450x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/e/e6462279f3fba1646c02e770984bb31ed21160f3.png 2x" data-dominant-color="E9E6ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resnet50</span><span class="informations">483×804 97.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I cant do transfer learning, because I dont have any label for these images. Anyone know how to improve the performance? Thanks</p>

bobey: <p>Normalize the images…probably do some more valid transformations like …standard deviation</p>
Choosing the correct design for a cnn model

Hadi_Nayebi: <p>I am having a problem designing and choosing the correct model for my project. I will appreciate any insight or suggestions.</p>
<p>I am building an autoencoder for a proteins sequence. I want to represent my protein sequence as a sequence of one-hot vectors of size 20.<br>
For a sequence of length N, I will start with an Nx20 matrix.<br>
[[1, 0, 0, 0, …, 0],<br>
[0, 0, 1, 0, …, 0],<br>
[0, 0, 0, 0, …, 1],<br>
…<br>
[0, 1, 0, 0, …, 0]]<br>
then I want each one-hot vector to pass through a small single-layer NN (20x4). This NN will be learning how to map each amino acid to a 4D space (given my problem space).<br>
It results in an Nx4 output matrix.<br>
The Nx4 input will then be flattened to enter a CNN layer (or a fully connected NN layer), shrink into a 1x10 encoding, and continue to the decoder and end up with the Nx20 one-hot vectors.<br>
I can’t visualize the process to understand how I would design the network. Is there any online source to visually build a custom network and receive the architecture design details?<br>
I have problems coming up with network design details on the fly, such as the matching dimensions, and many visual guides I find are incomplete or confusing.</p>
Tf.reverse equivalent operation in Pytorch?

ntomita: <p>Hi, I was looking for a tensor operation in PyTorch to reverse the order of values on specific axis. Suppose there is a tensor X of size n x m x k. After the reverse operation on the second axis, the value of X_reversed[0, -1, 0] must be the same as X[0, 0, 0].</p>
<p>I found the exact method in TensorFlow, i.e., tf.reverse (<a href="https://www.tensorflow.org/api_docs/python/tf/reverse" class="inline-onebox" rel="noopener nofollow ugc">tf.reverse  |  TensorFlow Core v2.6.0</a>), however, I could not found any equivalent method in PyTorch.</p>
<p>I achieved what I need to do by converting a PT tensor to numpy, apply tf.reverse, covert back to numpy, then copied to PT tensor. Though it’s working, I felt it’s really patchworking where it should not be like this mess and I believe it would have been easier if PyTorch had an equivalent implementation already.</p>
<p>If anyone know why the operation is not available in PyTorch (maybe suggested before and rejected?) or if there exists a PyTorch method I just missed, please let me know.</p>

eqy: <p>Would something like <code>flip</code> work for you: <a href="https://pytorch.org/docs/stable/generated/torch.flip.html" rel="noopener nofollow ugc">torch.flip — PyTorch 1.9.1 documentation</a> ?</p>

ntomita: <p>Oh, I totally missed it! Thanks <a class="mention" href="/u/eqy">@eqy</a> !</p>