akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dataset: multiple patches from the same CT + shuffle

username: asere
cooked: <p>Hello,</p>
<p>I have 3D CT images (and the corresponding segmentation masks), from each of which I would like to retrieve a variable number of 3D patches, centered around a ROI (based on the labels of the segmentation masks).</p>
<p>I would like to train on these patches at random, independently of which CT they came from (i.e. <code>shuffle</code> in the <code>DataLoader</code> should be <code>True</code>).</p>
<p>For this purpose, the <code>Dataset</code>'s <code>__getitem__</code> should give me a single <code>patch</code> + <code>patch_mask</code> pair.</p>
<p>I can do this by loading the <em>same</em> full_size CT image <em>every</em> time I want to crop a patch from it, like so:</p>
<pre><code class="lang-auto">class myDataset(Dataset):

    def __init__(self, df, size):
        # df is a dataframe with the paths to the full-size CT's and masks, like so:
        #  index |    ID        |     ID_mask       |   label
        #    0    'path_to_CT1'   'path_to_CT1_mask'      1
        #    1    'path_to_CT1'   'path_to_CT1_mask'      2
        #    2    'path_to_CT2'   'path_to_CT2_mask'      1
        #    3    'path_to_CT2'   'path_to_CT2_mask'      2
        #    4    'path_to_CT2'   'path_to_CT2_mask'      3
        #    5    'path_to_CT2'   'path_to_CT2_mask'      4
        #    6    'path_to_CT3'   'path_to_CT3_mask'      1
        #   ...       ....              ....            ....
        self.df = df 
        self.size = size # size of the patches

    def __len__(self): 
        return len(self.df)

    def __getitem__(self, idx): 
        path_to_CT = self.df.at[idx,'ID']
        path_to_CT_mask = self.df.at[idx,'ID_mask']
        CT = sitk.ReadImage(path_to_CT) # loads entire CT (512x512x512)
        CT_mask = sitk.ReadImage(path_to_CT_mask) # loads entire CT_mask  (512x512x512)
        label = self.df.at[idx,'label']  

        # crop a single patch + patch mask around label (ROI) of size e.g. 128x128x128
        patch, patch_mask = crop_roi(CT, CT_mask , label, self.size) 

        return {'imgs': patch, 'masks': patch_mask}
</code></pre>
<p>I can also retrieve all of the patches from the same CT at once and store them for example in a dictionary in the <code>__init__</code> and then access them in the next <code>__getitem__</code>. However, this would either force me to go through all of the patches in order (with no shuffle) or store all of the patches of all of the CT’s in a dictionary which doesn’t work well in terms of memory.</p>
<p>I’ve read in a <a href="https://discuss.pytorch.org/t/manually-generate-training-batch/17082">previous post</a> that I can “stack the crops into the batch dimension”. I’ve tried it, but because I have a variable number of patches per image, I don’t think it works.</p>
<p>My question: is there a way to avoid loading the same CT every time I need a patch from it (considering that the number of patches of each CT can be up to 150)?</p>
Backward() and weight update for two linked nets

username: zelazny
cooked: <p>I have gone through the Introduction to Pytorch and searched information on backward() on the forum but an aspect of autograd remains unclear to me. Suppose we have two nets, <code>A</code> and <code>B</code>:</p>
<pre><code class="lang-auto">A(input) = a
output = B(a)

B_solver = optim.Adam(B.parameters(), lr=learning_rate_B)
B_loss = loss(output, target)
B_loss.backward()
B_solver.step()
</code></pre>
<p>The way <code>backward()</code> is described, this would calculate gradients all the way back through net <code>A</code>. Am I right in thinking that <code>B_solver.step()</code> will not update the weights for <code>A</code>? If so, then what would be a natural way of updating the weights of <code>A</code>? We do not have an explicit loss function for <code>A</code> alone. Doing the obvious, we quickly run into the need to retain the computational graph (which is discouraged in the documentation):</p>
<pre><code class="lang-auto">A_solver = optim.Adam(A.parameters(), lr=learning_rate_A)
A_solver.step()
</code></pre>

username: JuanFMontesinos
cooked: <pre><code class="lang-auto">A(input) = a
B(a) = output
</code></pre>
<p>This is not a very clear code. You cannot assign a returning to a variable</p>

username: zelazny
cooked: <p><a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>  Sorry about that, corrected in the original post.</p>

username: JuanFMontesinos
cooked: <p>Still what you are doing is a bit strange.</p>
<pre><code class="lang-auto">class Dummy():
    def __init__(self):
        self.x = {'dummy':None}
    def __call__(self, *args, **kwargs):
        return self.x['dummy']
        
b=Dummy()
b()=a
  File "&lt;ipython-input-8-e815e88d0509&gt;", line 1
    b()=a
         ^
SyntaxError: can't assign to function call
</code></pre>
<p>I would say it’s not possible to return something from an object call and assign it to a variable<br>
as you are doing there. My IDE agrees.</p>
<pre><code class="lang-auto">A(input) = a
</code></pre>

username: JuanFMontesinos
cooked: <p>Assuming that you meant this:</p>
<pre><code class="lang-auto">a=A(input)
output = B(a)

B_solver = optim.Adam(B.parameters(), lr=learning_rate_B)
B_loss = loss(output, target)
B_loss.backward()
B_solver.step()
</code></pre>
<p>A weights won’t be updated as the haven’t been passed to the optimizer. You can update them just by passing also A to the optimizer</p>
Multiple GPUs vs one with huge memory

username: brix
cooked: <p>Hi,<br>
sorry that this question is more general and not only related to PyTorch, but I think here I find the experts who can give me an advice <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>I would like to build a new server for DeepLearning.<br>
This shall be used mainly for CNNs, Mask RCNNs and GANs.</p>
<p>Initially I thought of buying 4x RTX 2080TI. As I know I could combine two of these with a NVLink-Bridge which would give me two times 22 GB. But I could not get 44 GB.</p>
<p>For nearly the same price as for 4x RTX 2080TI I could buy e.g. only one Quadro RTX 8000 with 48 GB.</p>
<p>I just read that 11 GB should be enough for the cases I described whereas for example for NLP a huge amount of GPU memory might be necessary.</p>
<p>Is this correct?</p>
<p>What is your experience?<br>
How much memory would you like to have today for the described cases?</p>
<p>Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Problem training gamma correction

username: dudy_karl
cooked: <p>hi everybody,<br>
i’m trying to learn parameters for forward and inverse gamma correction:<br>
input-&gt;gamma_correction-&gt;network-&gt;another_gamma_correction-&gt;loss</p>
<p>where: gamma_correction(x,gamma_parameter)=x**gamma_parameter</p>
<p>what i do is simply define a nn.Parameter() for the initial correction and another nn.Parameter() for the final correction.</p>
<p>when i only try to learn the initial gamma correction there’s no problem but when i try to learn both initial and final gamma corrections i get nan values for the parameters after the first backpropagation.</p>
<p>any thought?<br>
thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: notacode
cooked: <p>I had a similar problem, you might be getting <code>nan</code>'s because some of the inputs to the second gamma correction layer are very close to 0 and you raise them to a power which leads to problems with gradients. If I am not mistaken about how it works in PyTorch (roughly), the derivative of <code>x^a</code> is being evaluated with respect to <code>a</code> and with respect to <code>x</code>, either one could cause problems (say, <code>a&lt;1</code>).</p>
<p>Now, I hope this small bit info helps you, because in my case I was unable to resolve my issue.</p>

username: dudy_karl
cooked: <p>actually it appears that even in the forward direction there is a problem</p>

username: crowsonkb
cooked: <p>Try the following <code>Gamma</code> layer:</p>
<pre><code class="lang-python">class Gamma(nn.Module):
    def __init__(self, eps=1e-8):
        super().__init__()
        self.eps = eps
        self.gamma = nn.Parameter(torch.tensor(1.))
    
    def forward(self, input):
        return input.sign() * (input.abs() + self.eps)**self.gamma
</code></pre>
<p>There are two tricks here. The first is to handle negative inputs by <a href="https://en.wikipedia.org/wiki/Even_and_odd_functions#Odd_functions" rel="nofollow noopener">odd symmetry</a>; f(-x) is computed as -f(x). The second is to handle zero or close-to-zero inputs by adding a small ‘epsilon’ value to ensure numerical stability.</p>
<p>I tested this layer and was able to get it to train whereas without these tricks the <code>self.gamma</code> parameter would end up NaN.</p>
Infer Torch model via gunicorn (wsgi)

username: gvskalyan
cooked: <p>gunicorn is used as <em>gunicorn  app:app  --preload --workers 3</em><br>
Preload is used to share the resources among the workers.<br>
Set the OMP_NUM_THREADS to 2.</p>
<p>app.py contains the following code</p>
<pre><code class="lang-auto">from flask import Flask, jsonify
import torch  
from create_model import testtype or paste code here

app = Flask(__name__) 
model = torch.load('model.pt')

@app.route('/predict',methods = ['POST', 'GET']) 
def prediction(): 
    constant_input = torch.randn(20, 16, 50, 100)
    prediction = model(constant_input)
    return jsonify(prediction)
</code></pre>
<p>model.pt is created using create_model.py containing</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class test(nn.Module):
    def __init__(self):
        super(test, self).__init__()
        self.conv1 = nn.Conv2d(16, 33, 3, stride=2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return x

m = test()
input = torch.randn(20, 16, 50, 100)
print(m(input))
torch.save(m, 'model.pt')
</code></pre>
<p>But I am not able to infer it .<br>
Instead of using a torch model if I use some numpy operation and just return its output, it is able to.</p>
<p>Though using <em>gunicorn  app:app  --preload --workers 3 --threads 2</em> I am able to infer. But anyone please tell me why does it differ only when threads are used and that too in case of pytorch.<br>
Thanks.</p>

username: gvskalyan
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any idea?    .</p>

username: miguelvr
cooked: <p>Do not use <code>torch.save()</code> on your the class object. Either save the <code>state_dict</code> or use the a traced version of the model.</p>
<p>Saving the model with JIT:</p>
<pre><code class="lang-python">traced_model = torch.jit.trace(model, torch.randn(1, 16, 50, 100))
torch.jit.save(traced_model, output_path)
</code></pre>
<p>Loading the model in  the server code:</p>
<pre><code class="lang-python">model = torch.jit.load(model_path, map_location='cpu')
</code></pre>
<p>The rest should be the same.</p>
<p>Just a tip on your code style: do not start class names with lowercase, it’s not pythonic</p>
Compatibility of tensor types

username: GlebBrykin
cooked: <p>Good afternoon! I have a question why when executing the following code</p>
<pre><code class="lang-auto">import torch
a = torch.tensor([1, 2, 3], dtype = torch.float32)
b = torch.tensor([1, 2, 3], dtype = torch.int)
print(a + b)
</code></pre>
<p>I get an error, but when I run other code</p>
<pre><code class="lang-auto">import torch
a = torch.tensor([1, 2, 3], dtype = torch.float32)
print(a + int(1))
</code></pre>
<p>there is no error and the code works correctly?<br>
I noticed that you can only add tensors of one type. Why is this necessary?</p>

username: tom
cooked: <p>It isn’t and indeed PyTorch 1.3 introduced type promotion and make the first work, too.</p>

username: GlebBrykin
cooked: <p>Thank you very much! I’m using PyTorch 0.4 and it’s not there. I will be updated.</p>
About CrossEntropyLoss

username: yuyaya
cooked: <p>During the VAE training,<br>
I was comparing loss between (Input pic - Reconstructed pic)<br>
and loss between (Input pic - Input pic).</p>
<p>Even though picture reconstruction is not done, I found out Loss between (Input pic - Reconstructed pic) will be smaller than the loss between (Input pic - Input pic)  during the training.</p>
<p>Does anyone know why this happens?<br>
As the title shows, I’m using ‘only’ CrossEntropyLoss as a loss function.<br>
(I’m not using KLdivergence )</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>I would assume the reconstructed image to have the same shape and type as the input image.<br>
However, if you are using <code>nn.CrossEntropyLoss</code>, the target should be a <code>LongTensor</code> containing some class indices.<br>
Could you explain your use case a bit or post a (small) executable code snippet?</p>

username: yuyaya
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thank you for your reply.<br>
I think I solved this problem by checking out the document.</p>
<p>What I was doing was, giving softmax activation to the output of the VAE model even though<br>
<code>nn.CrossEntropyLoss</code> consist of <code>nn.LogSoftmax()</code>  and  <code>nn.NLLLoss()</code>.</p>
<p>Thank you anyway</p>
Where I can download previous libtorch?

username: dancedpipi
cooked: <p>I can’t find anywhere to download libtorch previous.<br>
Could you provide one?</p>

username: ptrblck
cooked: <p>I’m not sure, if there is a better way, but you could try to change the URL from the latest release:</p>
<pre><code class="lang-auto">https://download.pytorch.org/libtorch/cu101/libtorch-shared-with-deps-1.3.1.zip
</code></pre>
<p>to e.g.</p>
<pre><code class="lang-auto">https://download.pytorch.org/libtorch/cu100/libtorch-shared-with-deps-1.2.0.zip
</code></pre>
<p>for <code>1.2.0</code> with CUDA10.0</p>

username: dancedpipi
cooked: <p>Nice, it works! thanks!!!</p>
Associate a priority to each input

username: vainaijr
cooked: <p>Suppose I have total 10 images, of 2 classes, lets say 6 cats and 4 dogs, now I want my neural network to associate a priority to each of these images, so if it predicts one of the 6 cats with a higher priority correctly, then it will be rewarded, and if it predicts one of the 6 cats with a lower priority correctly, then it does not matter much.</p>
<p>Furthermore, this priority is not something that I hard code and tell my neural network, that give high priority to this image, and low to this image, instead it is a parameter, so neural network will update these priorities by gradient descent, and figure out which image it needs to give higher priority, and which image it needs to give lower priority.</p>
<p>What kind of reward should I give to my neural network, if I set loss to zero, for predicting correctly images, then it does not consider priority, should I write a custom loss function, that takes priority into consideration?</p>
<pre><code class="lang-auto">class Net(nn.Module):
  def __init__(self):
    super().__init__()
    self.priority_with_images = nn.Parameter(number of images * size of priority vector) 
    # each image will have a one dimensional priority, 
    # (I could set priority for each image as a number also, but I think 1d vector priority would be better)
 def forward(self, input_images):
   # suppose I get a vector representing probability image 
   # is one of the classes, after passing image through neural network
   # do I compute loss in forward itself, and return it?
   # do I return something like, loss * priority_with_images, 
   # will this ensure that parameters of priority_with_images gets updated?
</code></pre>
How to count number of 1's in column axis

username: ati
cooked: <p>Hi<br>
I have a tensor and i need to count number of 1’s  in each column</p>
<pre><code class="lang-auto">B=torch.Tensor([[[0., 0., 0., 0., 1.]],
                [[0., 0., 0., 0., 1.]],
                [[0., 1., 0., 0., 1.]]])
</code></pre>
<p>and i need an output like this:</p>
<pre><code class="lang-auto"> output=tensor([[0.],
                [1.],
                [0.],
                [0.],
                [3.]])
</code></pre>
<p>I will be grateful if someone gives me a help</p>

username: seongmin
cooked: <p>For your example specifically,</p>
<blockquote>
<p>output = B.sum(0).view(-1,1)</p>
</blockquote>
<p>should work.</p>

username: ati
cooked: <p>Yeah it worked…thank you very much</p>
Add two optimizers in single model for two different path

username: ankitjha16
cooked: <p>I want to add two optimizers for my network, the network have two different paths one for loss1  and another for loss2.<br>
first i want to update the loss1 with optimizer1 and after that update, i want to update loss2 (that is a function of loss1) with optimizer2.</p>
<p>please help me how to do that.  <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/smth">@smth</a></p>

username: tom
cooked: <p>So I’m neither of the two, but if you allow me to give a shot:</p>
<ol>
<li>there is no inherent need to define two optimizers for this, but nothing keeps you from doing so if you want.</li>
<li>if you do this and have optimizers with state derived from gradient statistics (plain SGD hasn’t, SGD with momentum, Adam, … have), the optimizer will only see the gradient statistics of those used with its step. This could be desired (because it implicitly equilibrates the loss gradients when used with adaptive optimizers) or non-desired.</li>
<li>The more common method is to make it a joint update.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: It is generally preferred to not tag people. While they invariably are have the best answers, they don’t scale arbitrarily and it’s best to have them as jokers for the hardest questions.</p>

username: ankitjha16
cooked: <p>thanks for replying, i will take your valuable suggestions.<br>
As model.parameters() will give all the parameters. But along with it I need one more optimizer which will update the these parameters along with the new one.</p>
<p>opt1 = optim.Adam(model.parameters(), lr=1e-4)<br>
opt2= optim.Adam(<strong>other_los</strong>s, lr=1e-5)</p>

username: tom
cooked: <p>Oh, you use the same parameters (vor a list oft filtered parameters).<br>
You would geht the separate gradients by</p>
<pre><code class="lang-auto">opt1.zero_grads()
loss1.backward()
opt1.step()
opt2.zero_grads()
other_loss.backward()
opt2.step()
</code></pre>
<p>so that opt2 is ignorant of loss1’s gradient and opt1 of <code>other_loss</code>'s.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ankitjha16
cooked: <p>thanks, it’s working.</p>
Best practices to train RNN using GPU

username: MrRobot
cooked: <h2>Problem</h2>
<p>I tried to train my RNN on Google colab (which has free GPU to use). But it seems that the runtime does not seem to decrease much, which makes me doubt whether I have done correct things when processing data. Specifically</p>
<ul>
<li>Is it true <strong>only</strong> when data is sent in batches could the runtime be reduced? Otherwise, when data is sent into network one after another, there is no hope there could be speedup.</li>
<li>If the first point is true. Does this mean that the data fed in has to have <strong>same input dimension</strong>.<br>
For example. say I would like to pass words {“my”, “name”, “is”} into network for some classification task and I decide to use one-hot encoding. Then “my” and “is” should be padded to the same length as “name”.</li>
</ul>
<p>Thank you for any input.</p>
Should we avoid using one-hot encoding in neural networks?

username: MrRobot
cooked: <h2>Problem</h2>
<p>I am trying to reproduce the results shown in this <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="nofollow noopener">tutorial</a>. However, using <code>torch.nn.RNN</code> (unlike in the tutorial, the author used the RNN he wrote by himself) outputs <strong>score and loss are both constantly 0</strong>.</p>
<h2>Goal</h2>
<p>Classify names into their origins (languages) using RNN.</p>
<h2>Data Preprocessing</h2>
<p>I have a bunch of text files where each text file includes many names corresponding to the same language (label).  I one-hot encoded all the names and form a dataset and each entry looks like <code>((L, D), 1)</code>, where <code>L</code> is the number of characters in the name, <code>D</code> is the dimension of one-hot representation and 1 corresponds to class label.</p>
<p>In my case, <code>D</code> is 57 and there are 18 classes. So for name like “Mona”, the corresponding data is of shape <code>((4, 57), 1)</code>.</p>
<h2>Model and Training Loop</h2>
<pre><code class="lang-python">class RNNNameClassifier(nn.Module):
  def __init__(self, input_size, hidden_size, output_size, **kwargs):
    super(RNNNameClassifier, self).__init__()
    self.hidden_size = hidden_size

    self.hidden = self.init_hidden()
    self.rnn = nn.RNN(input_size=input_size, hidden_size=hidden_size, **kwargs)
    self.classifier = nn.Linear(hidden_size, output_size)

  def init_hidden(self):
    return torch.zeros(1, 1, self.hidden_size)
  
  def forward(self, embedding):
    output, self.hidden = self.rnn(embedding, self.hidden)
    output = self.classifier(output)
    output = F.log_softmax(output, dim=1)

    return output
</code></pre>
<pre><code class="lang-python">model = RNNNameClassifier(input_size=EMBEDDING_DIM, hidden_size=HIDDEN_DIM, output_size=OUTPUT_DIM)
optimizer = optim.Adam(model.parameters(), lr=LR)
criterion = nn.NLLLoss()
</code></pre>
<pre><code class="lang-python">loss_list = list()
for epoch in range(MAX_EPOCH):
  print("epoch: %d / %d" % (epoch + 1, MAX_EPOCH))
  for i, (X_train, y_train) in enumerate(train_dataset):
    X_train = torch.tensor(X_train, dtype=torch.float32)
    y_train = torch.tensor(y_train, dtype=torch.int64)

    optimizer.zero_grad()
    model.hidden = model.init_hidden()

    score = model(X_train.view(X_train.size(0), 1, -1))
    loss = criterion(input=score[-1], target=torch.tensor([y_train]))
    loss.backward()
    
    optimizer.step()

    loss_list.append(loss.detach().cpu().numpy())
    if (i + 1) % PRINT_FREQ == 0:
      print("\tloss: %.5f" % loss_list[-1])
</code></pre>
<h2>Question</h2>
<p>I am not sure the issue arises from some errors in my implementation or something else. Specifically, one potential problem might be I <strong>should not</strong> use one-hot encoding. Maybe some fine-tuning of word2vec is required.</p>
<p>Do I understand it correctly? Thank you in advance!</p>
How to make a random prediction?

username: alx
cooked: <p>I’m asked to make a random prediction to evaluate my model.</p>
<p>I have 5 classes so the easy way would be to simply do :</p>
<p><code>pred_label = random.randint(0, 4)</code></p>
<p>However, I need the randomness to be in a tensor format since I’m doing some tensor manipulation later on it the code.</p>
<p>my batch size 128, so I tried:</p>
<p><code>pred_label = torch.randn(128, 1000, dtype=torch.float)</code></p>
<p>However it does not seem to work later on (and gives me an accuracy of 2%).</p>
<pre><code class="lang-auto">image, label = image.to(device), label.to(device)

          pred_label = torch.randn(128, 1000, dtype=torch.float)

          prediction_cpu = pred_label.cpu().argmax(dim=1, keepdim=True)

          ground_truths = label.view_as(prediction_cpu)
</code></pre>
<p>What’s the best way to predict randomly?</p>
Packing in RNNs

username: Nitesh
cooked: <p>Hi,<br>
For the following data:</p>
<blockquote>
<p>Input(names) - [‘Griffiths’, ‘Jemaldinov’, ‘Naifeh’]<br>
True Output(Language) - [‘English’, ‘Russian’, ‘Arabic’]</p>
</blockquote>
<p>When used pack_padded_sequence for the above names the output generated is</p>
<blockquote>
<p>J G N e r a m i i a f f l f e d i h i t n h o s v</p>
</blockquote>
<p>which is clearly not in the order in which it were sent because of their varying lengths.</p>
<p>The question is, in what order should I arrange the true outputs so that they are compared to their corresponding predicted output for the computation of loss.</p>
<p>I hope I’m able to explain my doubt <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

username: yonti
cooked: <p>short answer:<br>
before using pack_padded_sequence you need to sort the inputs by length and pass that to the function.<br>
sorting will return the permutation index and you can use that to permute your output</p>
<pre><code class="lang-auto">x = ['aaa', 'bbbb', 'cc']
lengths = torch.tensor([3,  4, 2])
lengths_permuted, perm_index = length.sort(0, descending=True)
x_permuted = x[perm_index]
x_packed = torch.nn.utils.pack_padded_sequence(x_permuted, lengths)
lstm_out_packed, h = LSTM(x_packed, h)
output_permuted = output[perm_index]
</code></pre>
<p>longer answer:<br>
when using packed sequence the strings are rearranged by length<br>
then they are stacked vertically<br>
in your example it will be<br>
Jemaldinov<br>
Griffiths<br>
Naifeh<br>
each iteration it will process the i’th column across all rows if exists and pass it to the next rnn cell<br>
that’s why it seems to you like the letters got scrambled.</p>
<ul>
<li>check this blog for a good example of how to use padded seq:<br>
<a href="https://towardsdatascience.com/taming-lstms-variable-sized-mini-batches-and-why-pytorch-is-good-for-your-health-61d35642972e" rel="nofollow noopener">https://towardsdatascience.com/taming-lstms-variable-sized-mini-batches-and-why-pytorch-is-good-for-your-health-61d35642972e</a>
</li>
<li>check this repo for understanding the inside of a packed seq:<br>
<a href="https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Sequence-Labeling#language-models" rel="nofollow noopener">https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Sequence-Labeling#language-models</a>
</li>
</ul>

username: Nitesh
cooked: <p>Thanks, I liked the shorter answer <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>
Error Message: /pytorch/aten/src/THC/THCBlas.cu:206

username: WeiQin_Chuah
cooked: <p>Hi, I am getting the following error message while performing eval testing but my code still runs without stopping. My CUDA version is already 9.2. Just wondering why prompted this error? My pytorch version is 1.3.0 cu92</p>
<blockquote>
<p>/pytorch/aten/src/THC/THCBlas.cu:206: UserWarning: Matrix multiplication for dimensions larger than 2^21 has known bugs on your combination of CUDA version and device type. Please consider upgrading to CUDA 9.2 or later.</p>
</blockquote>

username: ptrblck
cooked: <p>What does <code>torch.cuda.version()</code> print?</p>

username: WeiQin_Chuah
cooked: <p>Hi,<br>
I can’t run <code>torch.cuda.version()</code> as it throws this error <code>AttributeError: module 'torch.cuda' has no attribute 'version'</code>. Not sure if <code>torch.version.cuda</code> is the one you after but this is the output: <code>'9.2.148'</code></p>

username: ptrblck
cooked: <p>Yes, sorry for the wrong command.<br>
OK, in that case you could probably ignore this warning.<br>
I’ll have a look, why the warning is thrown even though your CUDA version shouldn’t trigger it.</p>
<p>Thanks for raising this issue <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Which will be faster, retrieving each word vector from the word embedding dictionary or from the nn.Embedding()?

username: Mainul
cooked: <p>For getting embeddings of a sequence, if I use python dictionary, I have to hit the dictionary for each word but if I copy the vectors in nn.Embedding(), for each sequence I have to send the indices of the word and even I can get word embeddings of several sequences just by one call, for example:</p>
<blockquote>
<blockquote>
<blockquote>
<p>input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])<br>
embedding(input)`</p>
</blockquote>
</blockquote>
</blockquote>
<p>But, pytorch embedding uses numpy matrix as lookup table.<br>
So, my question is, isn’t for executing the previous code, there will be 7 hits on the numpy matrix? Or it will be retrieved parallelly? Even it runs parallelly, there should be another dictionary to convert word to indices. That also needs 7 hits on the word2indices dictionary.</p>
<p>These questions and issues I am asking because of my original question,<br>
Which will be a faster process ?:<br>
For a sequence:</p>
<ol>
<li>
<p>getting each word vector from the word vector dictionary</p>
</li>
<li>
<p>Get the indices from the word2indices dictionary and the run like the mentioned code.</p>
</li>
</ol>

username: albanD
cooked: <p>Hi,</p>
<p>We don’t use numpy objects to store things, only pytorch’s Tensors.<br>
The good think about indexing is that a single indexing operation will return all the concatenated results, while you would have to do it by hand if you use a python dictionnary.</p>
How to feed a 3D tensor into Neural Network Forward() pass

username: ElleryL
cooked: <p>Consider I have a input data</p>
<p><code>X = torch.Tensor(sample_size,dim1,dim2)</code></p>
<p>For each data point, its a matrix form with dim1xdim2</p>
<p>Now when I feed into neural network with</p>
<p><code>nn.Linear(input_Dim,hidden_dim)</code></p>
<p>I have</p>
<pre><code class="lang-auto">output = input.matmul(weight.t())
RuntimeError: size mismatch,
</code></pre>
<p>This makes sense because forward only does operation of matrix multiply with a vector</p>
<p>How should I modify my neural net work?</p>

username: ptrblck
cooked: <p>You can add additional dimensions like this:</p>
<pre><code class="lang-auto">batch_size = 10
add_features = 5
in_features = 7
out_features = 12

x = torch.randn(batch_size, add_features, in_features)
lin = nn.Linear(in_features, out_features)
output = lin(x)
</code></pre>
<p>Is this what you are looking for?</p>

username: ElleryL
cooked: <p>Hi; wondering if I can do the same thing for <code>nn.Conv2D</code> and <code>nn.ConvTranspose2d</code> ?</p>
<p>Right now; I did <code>x = torch.randn(batch_size, add_features, channels, in_features, in_features)</code> but it raises error states that <code>RuntimeError: Expected 4-dimensional input for 4-dimensional weight</code></p>

username: ptrblck
cooked: <p>If you would like to apply the same kernels on <code>add_features</code>, you could push the <code>add_features</code> dimension to the batch dim and apply the standard <code>nn.Conv2d</code>.<br>
Here is a small dummy example with the comparison of the manual approach:</p>
<pre><code class="lang-python">batch_size, add_features, c, h, w = 2, 4, 3, 24, 24

x = torch.randn(batch_size, add_features, c, h, w)
x_ = x.view(batch_size*add_features, c, h, w) # push add_features into batch dim

conv = nn.Conv2d(c, 1, 3, 1, 1, bias=False)
output = conv(x_)
output = output.view(batch_size, add_features, 1, h, w)

# manual approach to verify output
outputs = []
for idx in range(add_features):
    x_part = x[:, idx]
    # get kernel
    kernel = conv.weight
    out = F.conv2d(x_part, kernel, padding=1)
    outputs.append(out)

outputs = torch.stack(outputs)
outputs = outputs.permute(1, 0, 2, 3, 4)

print((outputs - output).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>Let me know, if that would work for you.</p>
Pytorch much slower than numpy for simple arithmetics

username: jtaets
cooked: <p>I was wondering if there is a way to limit the overhead of arithmetics in pytorch as this lets us code everything in pytorch as compared to only using pytorch when the gradient of the calculation is needed.</p>
<p>This gives much cleaner code in reinforcement learning as often small calculations are needed of arrays which don’t need to be backpropagated.</p>
<pre><code class="lang-auto">import numpy as np
import torch
a = np.array([1.,2.,3.,4.,5.])
%timeit 2*a    #703 ns ± 34.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
a = torch.from_numpy(a)
%timeit 2*a    #7.99 µs ± 78.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
</code></pre>

username: crowsonkb
cooked: <p>First, a baseline on my system:</p>
<pre><code class="lang-python">an = np.linspace(1, 5, 5, dtype=np.float32)
%timeit 2*an  # 1.27 µs ± 14.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

a = torch.linspace(1, 5, 5)
%timeit 2*a  # 9.25 µs ± 49.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
</code></pre>
<p>Maybe the problem is that broadcasting is slow in pytorch. If I do instead:</p>
<pre><code class="lang-python">b = torch.ones_like(a) * 2
%timeit b*a  # 3.74 µs ± 147 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
</code></pre>
<p>But then if I do:</p>
<pre><code class="lang-python">bn = np.ones_like(an) * 2
%timeit bn*an  # 609 ns ± 5.38 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
</code></pre>
<p>So… numpy benefits from getting rid of broadcasting too. I’m not sure what to conclude here.</p>

username: crowsonkb
cooked: <p>What if we don’t allocate new ndarrays/tensors in the inner loop?</p>
<pre><code class="lang-python">c = torch.zeros_like(a)
%timeit torch.mul(b, a, out=c)  # 2.23 µs ± 224 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

cn = np.zeros_like(an)
%timeit np.multiply(bn, an, out=cn)  # 866 ns ± 36.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
</code></pre>
<p>Allocation can’t be the main source of PyTorch overhead either. It still isn’t as fast as naive NumPy.</p>
<p>For reference, my benchmarks were performed on a 2-core MacBook Pro, with Python 3.7.4 installed via Homebrew, numpy 1.17.2 installed via pip (and thus built vs openblas), and torch 1.3.1 installed via pip.</p>

username: albanD
cooked: <p>Hi,</p>
<p>These benchmark are a very good example for what is the overhead of our framework. All these operations require no compute at all. So here you just measure how long it takes to go all the way down the stack and back up.<br>
Unfortunately, because we support more hardware, layouts, and feature such as autograd, it is very hard to be as lean as numpy.<br>
Hopefully for large enough operations, this overhead should be small enough to not be a problem in practical neural network operations.</p>
GPU Device Ordering

username: Novak
cooked: <p>I have a system with a GTX 1070 and a Titan RTX installed in it.</p>
<p>Nvidia-smi yields:</p>
<blockquote>
<p>|   0  GeForce GTX 1070    Off  | 00000000:03:00.0 Off |                  N/A |<br>
|   1  TITAN RTX                 Off  | 00000000:04:00.0 Off |                  N/A |</p>
</blockquote>
<p>However,</p>
<blockquote>
<p>print(torch.cuda.get_device_name(“cuda:0”))<br>
print(torch.cuda.get_device_name(“cuda:1”))</p>
</blockquote>
<p>yields:</p>
<blockquote>
<p>TITAN RTX<br>
GeForce GTX 1070</p>
</blockquote>
<p>Clearly, the results are out of order with each other.  This presents no real difficulty, since I can always identify them at least up to the card type directly, so this is mostly curiosity:</p>
<p>Is this typical/intended behavior?  Are there intended guarantees (or no guarantees) for card order of nvidia-smi vs the internal ordering seen through PyTorch?</p>

username: ptrblck
cooked: <p><code>nvidia-smi</code> shows the devices by PCI order, while the CUDA driver and Runtime API should sort the devices based on their “performance”.<br>
To get the same order in your PyTorch script, you could try to set</p>
<pre><code class="lang-auto">export CUDA_DEVICE_ORDER=PCI_BUS_ID
</code></pre>
<p>before running your script.</p>

username: Novak
cooked: <p>Yes, that looks like the “solution” to this not-quite-a-problem.<br>
Previously, all my systems were either homogeneous, or sorted in the correct order by luck.</p>
<p>Thanks!</p>
Indexing argument function passed to bakward hooks

username: Kamil_Adamczewski
cooked: <p>Is there a way to pass an index with the function given as the parameter of the backward hook. That is I would like not to make a separate statement for every layer</p>
<pre><code> self.activation1.register_backward_hook(self._func1)
 self.activation2.register_backward_hook(self._func2)
 self.activation3.register_backward_hook(self._func3)
 self.activation4.register_backward_hook(self._func4)
etc.
</code></pre>
<p>I can make an array of self.activation[i]. But is there way also to do it with the argument function to the backward hook?</p>

username: vainaijr
cooked: <p>do you mean something like this,</p>
<pre><code class="lang-auto">def func(name):
  def hook(module, grad_input, grad_output):
    # hook implementation
  return hook
for name, layer in model.named_modules():
  layer.register_backward_hook(func(name))
</code></pre>
Multiprocessing/dataloader performance - file descriptor vs file system

username: yonti
cooked: <p>Hi,</p>
<p>been trying to speed up training time by optimizing the dataloader configuration while running inside a docker.<br>
at first, I was limited by permission issues and the number of open file descriptors to using the ‘file system’ option. without it, my code was crashing on the first batch once using more than one worker.<br>
later, I was able to raise the docker shm_size and also the number of open file descriptors so it was feasible to use the default ‘file descriptor’ option.<br>
I was expecting a speed up in performance as it is the default and recommended option, but was getting and 2x slowdown.<br>
any clues/ideas?</p>

username: ptrblck
cooked: <p>How many workers are you using in your current <code>DataLoader</code>?<br>
I assume you are comparing it to a <code>DataLoader</code> with <code>num_workers=0</code> and see the 2x slowdown?</p>

username: yonti
cooked: <p>your assumption is wrong <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>been trying many configurations to check how they affect the run time.<br>
compared different batch sizes in comparison to different num of workers.<br>
then compared the same configuration(batch size 256 / 10 workers) with file descriptor vs file system</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="yonti" data-post="3" data-topic="60648">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/ea5d25/48.png" class="avatar"> yonti:</div>
<blockquote>
<p>then compared the same configuration(batch size 256 / 10 workers) with file descriptor vs file system</p>
</blockquote>
</aside>
<p>So you see a 2x slowdown using a <code>DataLoader</code> with <code>batch_size=256</code> and <code>num_workers=10</code> in comparison to “file system”?<br>
Could you post some more information on your use cases as well as your system and when you are seeing a slowdown (how large etc.)?</p>
How to use NVIDIA Nsight Systems?

username: flyerjia
cooked: <p>I want to analyze my mixed c++ cuda code. How to use NVIDIA Nsight Systems with python and pytorch?</p>

username: albanD
cooked: <p>The same way you use it for c++ but by changing the executable to be <code>python your_test_script.py</code> IIRC</p>

username: flyerjia
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/7801ae0430d99352dc61f5be5e870a2647216d02.png" data-download-href="https://discuss.pytorch.org/uploads/default/7801ae0430d99352dc61f5be5e870a2647216d02" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/7801ae0430d99352dc61f5be5e870a2647216d02.png" alt="image" data-base62-sha1="h7CTvyc0sABu4W4nH2fdjNYqUxk" width="690" height="295" data-dominant-color="E0E1E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">698×299 11.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Is it right? This can’t work.</p>

username: tom
cooked: <p>At least with nvvp that always worked for me…<br>
What is your concern that you say it can not work?</p>
<p>Best regards</p>
<p>Thomas</p>

username: flyerjia
cooked: <p>My nsight systems can’t work.  Perhaps the command line and working directory are wrong. What i wrote in that two settings is right?</p>

username: albanD
cooked: <p>When I used these settings for nvvp, it always worked for me.</p>
<p>Can you use nsight properly with other cuda programs?</p>

username: flyerjia
cooked: <p>Unfortunately, though my code can run successfully for nvvp, my server always crashes after running the code. I have to restart my server times by times. And some functions of nvvp can’t support my server, whose compute capability is bigger than 7.2. Thus, i want to use nsight system as a substitute. For nsight systems, it shows that “found no nvidia driver”. However, the same code can run successfully for pycharm and windows’ cmd.  I have been confused for a long time.</p>

username: ptrblck
cooked: <p>How did you install Nsight Systems and is the CLI available via <code>nsys</code> or Nsight Compute via <code>nv-nsight-cu-cli</code>?</p>
Function to limit memory cached

username: mahendrasnilagiri
cooked: <p>Hello,</p>
<p>Is there any function available to limit the memory that is cached?</p>
<p>Thanks,<br>
Mahendra S</p>

username: albanD
cooked: <p>Hi,</p>
<p>You mean GPU memory?<br>
If so no there is not unfortunately.<br>
Why would you need such function?</p>

username: mahendrasnilagiri
cooked: <p>I’m running a resized Unet inference. The max input size it would accept is 1280x1280, for this pytorch caches around 4.5 GB of memory. At some edge cases like 505x960 (not a good aspect ratio)input is given the cache memory used is 7.8 GB. This doesnt happen often.</p>
<p>I’m running 2 workers, GPU has 16 GB of memory. I get OOM for some edge cases as mentioned above.</p>

username: albanD
cooked: <p>We do not have such feature at the moment.</p>
<p>The main limitation I recall for this is that the CUDA driver uses a significant amount of memory that we cannot track. Thus the limit set by the user my not be respected because of this extra memory. This makes the feature much weaker as it has some unexpected behavior for the end user.<br>
If you have a proposal to overcome this issues, please feel free to open a feature request on github !</p>
Torch.cuda.empty_cache() replacement in case of CPU only enviroment

username: geniebilal
cooked: <p>Currently, I am using PyTorch built with CPU only support. When I run inference, somehow information for that input file is stored in cache and memory keeps on increasing for every new unique file used for inference. On the other hand, memory usage does not increase if i use the same file again and again.</p>
<p>Is there a way to clear cache like cuda.empty_cache() in case of CPUs only.</p>

username: ptrblck
cooked: <p>Are you closing the files after reading the data?<br>
There is no <code>empty_cache</code> for the RAM and I think you might unnecessarily store the data manually.</p>

username: geniebilal
cooked: <p>I am serving the model as a flask app and inputs are images. i am loading data using torch dataloader function. How do I free that Dataloader? I thinks its done automatically.</p>
How to properly Register a Buffer

username: zetyquickly
cooked: <p>Hello,<br>
have recently worked with detectron2 and found that developers use <code>nn.Module._buffers</code> explicitly (<a href="https://github.com/facebookresearch/detectron2/blob/ef096f9b2fbedca335f7476b715426594673f463/detectron2/modeling/anchor_generator.py#L18" rel="nofollow noopener">https://github.com/facebookresearch/detectron2/blob/ef096f9b2fbedca335f7476b715426594673f463/detectron2/modeling/anchor_generator.py#L18</a>). So I wanted to check will it work similar way if we use <code>buffers()</code> method instead.</p>
<p>So here’s the experiment:</p>
<pre><code class="lang-auto">import torch

class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.batch_norm = nn.BatchNorm1d(num_features=1)
        self.register_buffer(str('new_buf'), torch.rand(1))

    def forward(self, x):
        return x + 10

m = MyModule()
print(m._buffers.values(), m._buffers.keys())
print([el for el in m.buffers()])
</code></pre>
<p>This code produces the following:</p>
<pre><code class="lang-auto">odict_values([tensor([0.0600])]) odict_keys(['new_buf'])
[tensor([0.0600]), tensor([0.]), tensor([1.]), tensor(0)]
</code></pre>
<p>Why there is no mention of BatchNorm parameters in first case, we see only buffers added by registration? But registered is visible in second print.<br>
Correct me if I am wrong: “One should use <code>nn.Module.buffers()</code> to see all buffers, and if it is needed to see just user defined use <code>._buffers.values()</code> instead (user defined == registered by <code>.register_buffer()</code>)”</p>

username: ptrblck
cooked: <p>The <code>nn.BatchNorm1d</code> buffers are registered in this particular module and you can access them via:</p>
<pre><code class="lang-python">m.batch_norm._buffers
</code></pre>
<p>However, note that <code>_buffers</code> is an internal method (exactly for this reason of confusion <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ) and you should stick to <code>buffers()</code> to get all buffers recursively.</p>
Does share_memory model is lock free?

username: SrujithPoondla
cooked: <p>Hi all, I am trying to implement the vanilla hogwild using pytorch and found an example in pytorch examples. But I am wondering, is share_memory() a lock_free to write and read?<br>
Because vanilla hogwild is lock-free asynchronous weight update method.</p>
<pre><code class="lang-auto">torch.manual_seed(args.seed)

    model = Net()
    model.share_memory() # gradients are allocated lazily, so they are not shared here

    processes = []
    for rank in range(args.num_processes):
        p = mp.Process(target=train, args=(rank, args, model))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>
<p><a class="mention" href="/u/soumith_chintala">@Soumith_Chintala</a> Could you please provide some info about this?</p>

username: GIOVANNI_COLUCCI
cooked: <p>hello, i have the same problem right now, i don’t understand the right way ‘share_memory()’  work;<br>
to now i have splitted my dataset and given every single part to a different process with the same model, at the end i want to call the optimizer (assuming that every single process have called the backpropagation on its data), so having the updated weight w.t.r. to the sum of the gradient (every process calling backward may sum the current grad with one’s).</p>