akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Passing a list of indices instead of single index to Dataset returns error

BadTimeManagement: <p>I tried to pass a list of indices to the dataset however the <code>__getitem__</code> function in my custom dataset The list of indices is from <code>BatchSampler</code>. But I’m getting a Type Error <code>list indices must be integers or slices, not list</code> when running the code.<br>
How do I solve this issue?</p>
<pre><code class="lang-auto">class MyData(Dataset):
    def __init__(self, data, transform = None):
        self.data = data
        self.transform = transform

    def __getitem__(self, index):

        img, label = self.data[index]  #&lt;------ The error occurs here

        if self.transform is not None:
            img = self.transform(img)
        return img, label


    def __len__(self):
        return len(self.data)


training_set = MyData(trainset,transform = train_aug)
sampler = torch.utils.data.sampler.BatchSampler(
    torch.utils.data.sampler.RandomSampler(training_set),
    batch_size=10,
    drop_last=False)

trainloader = DataLoader(
    training_set,
    sampler=sampler)
</code></pre>

mariosasko: <p>Instead of <code>trainloader = DataLoader(training_set, sampler=sampler)</code>, try <code>trainloader = DataLoader(training_set, batch_sampler=sampler)</code>.</p>

BadTimeManagement: <p>If i change it to sampler, I’m requesting index instead of a batch. or I’m misunderstanding things here?</p>

mariosasko: <p><code>batch_sampler</code> yields a list of lists where each list represents batch indices and <code>sampler</code> yields an index of next item (these items are then combined to form a batch according to the <code>batch_size</code>).</p>
Help in C++ syntax

theAayushbajaj: <p>I trained my model in PyTorch python API and wanted to do the inference on C++. Completed most of the part but couldnt figure out the equivalent C++ syntax of this python code snippet:</p>
<pre><code class="lang-auto">transformations = transforms.Compose([
			transforms.ToPILImage(),
			transforms.Resize(256),
			transforms.CenterCrop(224),
			transforms.ToTensor(),
			transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
		])
</code></pre>

glaringlee: <p><a class="mention" href="/u/theaayushbajaj">@theAayushbajaj</a><br>
Can you create an issue at <a href="https://github.com/pytorch/vision/issues" rel="nofollow noopener">https://github.com/pytorch/vision/issues</a><br>
To my knowledge, there is no c++ equivalent for transforms, but you can ask there.</p>
Partition datasets.ImageFolder to have equal number of images per class

alet: <p>Hi,</p>
<p>I have Imagenet in a <code>datasets.ImageFolder</code>, I want to partition the training in two datasets, but using <code>torch.utils.data.random_split</code> results in literal random split so the number of images per class follows a binomial instead of all being the same (up to rounding). What’s the best way of achieving this uniform-number-per-class split?</p>
<p>I’ve seen <a href="https://discuss.pytorch.org/t/how-to-split-test-and-train-data-keeping-equal-proportions-of-each-class/21063">this similar question</a> but ImageNet is huge so the default answer won’t do. There’s an answer involving indices, but I was hoping there would be something simpler exploiting the nice structure of <code>datasets.ImageFolder</code>.</p>
<p>Thanks!</p>
Pix2pix model tracing

sepiiii: <p>Hi all,<br>
I am trying to deploy the pix2pix model in an android app, I need to trace the model with torch.jit.script() but it doesnt seem to work.<br>
Here is my code:</p>
<pre><code class="lang-auto">    model = create_model(opt)     
    model.setup(opt)             
    model.eval()
    a = torch.jit.script(model)
    torch.jit.save(a,'optmodel.pt')
</code></pre>
<p>and the full stacktrace is as follows:</p>
<pre><code class="lang-auto">  File "toScript.py", line 58, in &lt;module&gt;
    a = torch.jit.script(model)
  File "/local-scratch/Anaconda3/envs/pix2pix/lib/python3.7/site-packages/torch/jit/__init__.py", line 1263, in script
    qualified_name = _qualified_name(obj)
  File "/local-scratch/Anaconda3/envs/pix2pix/lib/python3.7/site-packages/torch/_jit_internal.py", line 760, in _qualified_name
    name = obj.__name__
AttributeError: 'Pix2PixModel' object has no attribute '__name__'
</code></pre>
<p>What’s wrong?<br>
Any help would really be really appreciated.</p>

sepiiii: <p>I found the solution, the generator should only be scripted.</p>
Extract feature representation of each layer during training

GMBarra: <p>Hi, I am working on the implementation of a certain loss that requires the extraction of the feature maps of each layer in a Fully ConvNet to calculate the new loss, it is possible to do that? I know that is possible to extract the feature map after the training, but, it’s possible to do it during training without stopping it?</p>

mariosasko: <p>It can be done easily with the forward hooks. Check this <a href="https://discuss.pytorch.org/t/extracting-intermediate-layer-ouputs-in-resnet/92940">answer</a> of mine for more details.</p>

GMBarra: <p>This is what i was looking for, thank you!</p>
What is the purpose of 10 in official example?

blackbirdbarber: <pre><code class="lang-auto">emb = nn.Embedding(10, 3)
print(emb.weight.shape)
print(emb.weight)
input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
print('input:', input.shape)
embo = emb(input)
print(embo.shape)
print('embo:',embo)
</code></pre>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html" rel="nofollow noopener">Here</a> is a demo code that works but I don’t understand what is 10.</p>
<p>Since our input is torch.Size([2, 4]), output is torch.Size([2, 4, 3]) because of the embedding <code>emb.weight.shape[1]</code> is 3.</p>
<p>Can someone explain me in simple words what is the meaning of 10 = <code>emb.weight.shape[1]</code>?</p>
<p>My current assumption one additional hidden layer will be created of 10x10, but I am not sure.</p>

albanD: <p>Hi,</p>
<p>You can check the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html?highlight=embedding#torch.nn.Embedding">doc</a>.<br>
But this is how many entries there are in the embedding table. And so the maximum index that can be contained in your input.</p>

blackbirdbarber: <p>I cannot really see “the maximum index that can be contained in my input”. If or example:</p>
<pre><code class="lang-auto">emb = nn.Embedding(10, 3)
print(emb.weight.shape)
print(emb.weight)
input = torch.LongTensor([[1,2,4,5],[4,3,2,9],[2,3,4,5]])
print('input:', input.shape)
embo = emb(input)
print(embo.shape)
print('embo:',embo)
</code></pre>
<p>If I provide the input that has <code>torch.Size([3,4])</code> elements the embedding output will have <code>torch.Size([3, 4, 3])</code>. So there is no reflection of 10 in the output.</p>
<p>I copied the example from the official document you pointed <a class="mention" href="/u/alband">@albanD</a> so we are on the same page.</p>

albanD: <p>Hi,</p>
<p>Not the maximum number of elements. The maximum value in there <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
As you can see, everything in there is in [0, 9]. If you try to set one value to <code>42</code>, you will get an error (anything &gt;= 10 will as it is 0-indexed).</p>
A newbie question about .pth files

shirishr: <p>A have an orphan .pth file. Do not know where it has come from and whether it contains the whole model or state_dictionary of some kind. Can I find out more details from the pth file?</p>

albanD: <p>Hi,</p>
<p>I think the best is just to load it and see what is the python object that you get.<br>
If it contains nn.Module then it’s a whole model (but you might get into trouble if you don’t have all the classes locally and you most certainly won’t even be able to load it).<br>
If it contains a dictionary with keys like <code>module.0.conv1.weight</code>, then this is most likely a state dict.</p>

shirishr: <p>how should I load?<br>
torch.load(path) resulted in an error <strong>_pickle.UnpicklingError: A load persistent id instruction was encountered,</strong><br>
<strong>but no persistent_load function was specified.</strong></p>

albanD: <p>That most certainly means that this file was not saved using <code>torch.save()</code>.<br>
Using pickle (that torch.save/load uses), you can only reload a file the same way it was saved. In particular, this seems to indicate that your file was not saved using <code>torch.save()</code>.</p>
<p>The thing is that <code>.pth</code> file is not a specific thing. It is just a python pickled object. Just that people usually use special extension like <code>.pth</code> or <code>.pt</code> to mean that it contains pytorch objects.</p>

shirishr: <p>The file was saved as the final step in the tutorial <strong>deploy_seq2seq_hybrid_frontend_tutorial.ipynb</strong></p>
<p>scripted_searcher.save(“scripted_chatbot.pth”)</p>
<p>where scripted searcher is instance of GreedySearchDecoder which in turn is a torch.jit.ScriptMeta</p>

albanD: <p>Then as stated in the doc about torch.jit <a href="https://pytorch.org/docs/stable/jit.html#torch.jit.ScriptModule.save">here</a> you might want to use <code>torch.jit.load()</code> ?</p>

shirishr: <p>Thank you sir,<br>
It’s a little difficult to absorb the tutorial and associated docs in parallel.<br>
It worked. Going forward I will use torch.load() and if it fails try torch.jit.load()</p>

YiqunChen: <p>Hi, I encountered the same error. I saved a model by using torch.save() in Pytorch 1.6.0, but I can not load the model by using torch.load() in Pytorch 1.0.0, is there any solution? Thanks!</p>

albanD: <p>Hi,</p>
<p>This is unfortunately expected.<br>
Allowing very old versions of pytorch to load file saved from recent version would be very hard. And while it usually work from one version to the other. I wouldn’t expect it to work for such a big version change.</p>

YiqunChen: <p>Thanks, I will try some other solutions.</p>
1 titan RTX or 2 2080TI NVlinked?

chaslie: <p>morning,</p>
<p>I have the opportunity to either upgrade the GPU to a titanRTX or buy a 2nd 2080TI and nvlink together and improve the cooling system (it works out at about the same price).</p>
<p>The question is which is better? a single titan RTX does give the benefit of improving the speed of models by about 10-20% over a single 2080Ti, however this is alot slower than 2 2080TI’s. The next benefit is the significant increase in GPU memort (24GB vs 11GB), however can you use the NVlink and pytorch to combine the 2x2080ti into behaving as a single 22GB 2080Ti, as all the multi GPU benchmarks keep the batch sizes the same for 2x2080ti?</p>
<p>I keep hitting the upper limit of the CUDA memory and the only way to get over this is to keep reducing the batch size, but there will come a point where this is no longer viable.</p>
<p>This leads me to the next question, is the increase in speed on the titan just down to the increased memory size?</p>

albanD: <p>Hi,</p>
<p>I think the difference will greatly depend on your workload indeed.<br>
If you need the memory, it is a simple choice as there is no way that I’m aware of to merge two cards together.<br>
The biggest benefit I would see from having two cards though is that when running cross validation (or any parameter search). It is much simpler to run two job at the same time. And this will most likely be faster than the single card (as long as you have enough memory on each).</p>
<blockquote>
<p>This leads me to the next question, is the increase in speed on the titan just down to the increased memory size?</p>
</blockquote>
<p>No I think it’s unrelated. The die is just bigger with more compute cores.</p>

chaslie: <p>Thanks, thats what i understood the situation to be.</p>
Confused by this implementation of Linear from the docs

pwasoutside: <p>The class definition of <code>LinearFunction</code> <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">here</a> confuses me in one way –</p>
<p>is the bias here a vector of size <code>output.shape</code> where all the entries are the same?<br>
Wouldn’t the proper definition (where all of entries of the bias are free) have <code>grad_bias = grad_output</code> ?</p>
<p>Or is there a misunderstanding here on my part … thanks.</p>

albanD: <p>Hi,</p>
<p>You need to keep in mind that this module expects a batch of inputs.<br>
And so in the forward pass, the bias is actually automatically broadcasted over all the elements in the batch. In the backward pass, we need to do the backward of this broadcast on the 0th dimension which is a sum over this 0th dimension.</p>
Invalid literal for int() with base 10: 'import os'

chunchun: <p>I wirte the data to *.txt and use pandas.read_csv() to get the data.<br>
But it shows errors as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/2146962baffeba6525dd8a224bee236c9669ecc1.png" data-download-href="https://discuss.pytorch.org/uploads/default/2146962baffeba6525dd8a224bee236c9669ecc1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/2146962baffeba6525dd8a224bee236c9669ecc1.png" alt="image" data-base62-sha1="4KmZer5xlzazTUKBSCiS1HLPbdT" width="690" height="260" data-dominant-color="5F6F89"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">695×262 9.87 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can I do?</p>

G.M: <p>Can u double check ur .txt file, it seems that there is an “<code>import os</code>” in it.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; int('import os')
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
ValueError: invalid literal for int() with base 10: 'import os'
</code></pre>

bennetcole: <p>The error message invalid literal for int() with base 10 would seem to indicate that you are passing a string that’s not an integer to the int() function . In other words it’s either empty, or has a character in it other than a digit.</p>
<p>You can solve this error by using Python isdigit() method to check whether the value is number or not. The returns True if all the characters are digits, otherwise False .</p>
<p><code>if val.isdigit():</code></p>
<p>The other way to overcome this issue is to wrap your code inside a Python try…except block to handle this error.</p>
<p><strong>Python2.x and Python3.x</strong></p>
<p>Sometimes the difference between Python2.x and Python3.x that leads to this ValueError: <a href="http://net-informations.com/python/err/int.htm" rel="nofollow noopener">invalid literal for int()</a> with base 10 .</p>
<p>With Python2.x , int(str(3/2)) gives you “1”. With Python3.x , the same gives you (“1.5”): ValueError: invalid literal for int() with base 10: “1.5”.</p>

KrunalLathiya: <p>You can solve this error by using  <strong><a href="https://appdividend.com/2019/11/18/python-string-isdigit-function-example/" rel="nofollow noopener">Python isdigit() method</a></strong>  to check whether the value is number or not. The returns  <strong>True</strong>  if all the characters are digits, otherwise  <strong>False</strong></p>
<p>The error message invalid literal for int() with base 10 would seem to indicate that you are passing a string that’s not an integer to the int() function. In other words, it’s either empty or has a character in it other than a digit.</p>
Pytorch version of Tensorflow XLA

chaslie: <p>morning,</p>
<p>I was wondering there is a pytorh equivlant of the TensorFlow XLA?</p>
<p>thanks,</p>
<p>chaslie</p>

albanD: <p>Hi,</p>
<p>Yes, you can check  <a href="https://github.com/pytorch/xla/">https://github.com/pytorch/xla/</a></p>
Why the annotation is integer, i wanna add my list as decimal number

shangguan91: <pre><code class="lang-auto">import matplotlib.pyplot as plt

import numpy as np

plt.rcdefaults()

fig, ax = plt.subplots()

# Example data

y_pos = np.arange(len(label_list))

performance = y3

error = np.random.rand(len(label_list))

rects3 = ax.barh(y_pos, performance, xerr=error, align='center')

ax.set_yticks(y_pos)

ax.set_yticklabels(label_list)

ax.invert_yaxis()  # labels read top-to-bottom

ax.set_xlabel('Average test accuracy over 100 epochs%')

ax.set_title('Average performance for each model over 100 epochs')

def autolabel(rects):

    for rect in rects:

        width = rect.get_width()

        plt.text(1.05*rect.get_width(), rect.get_y()+0.5*rect.get_height(),

                 '%d' % int(width),

                 ha='center', va='center')

autolabel(rects3)

plt.show()
</code></pre>

albanD: <p>Hi,</p>
<p>What is your question?  Could you give more context on where you got this code from? What you’re trying to do and what is the issue?</p>
ValueError: Stratified CV requires explicitely passing a suitable y

omarabdelaziz: <p>I’n trying to use skorch for crosss after I convert my Y into one host encoded vector but I get this error<br>
this is the dataset class</p>
<pre><code class="lang-auto">
class AmericanSignLanguage(Dataset):
  
  def __init__(self, batch_size):
    'initilization'
    #loading features
    self.x = np.load('drive/My Drive/x_features_v2_t.npy')
    self.shape  = self.x.shape
    print(self.shape)
    self.x = np.reshape(self.x,(self.shape[1], self.shape[2],self.shape[0]))
    
    #loaading labels (target)
    self.y = np.load('drive/My Drive/y_labels_v2_t.npy') 
    
    #loading sequnce list
    self.seq_lengths = np.load('drive/My Drive/seq_lengths_t.npy')
    
    #conveting the labels into one host encoding 
    self.encoder = LabelBinarizer()
    
    #one hot encoding
    self.one_hot_labels = self.encoder.fit_transform(self.y)
    
    #splitting the data into train and test
    self.num_classes = len(np.unique(self.y))
    self.validation_split = .2
    self.shuffle_dataset = True
    self.random_seed= 42
    
    # create feature and targets tensor for train set. As you remember we need variable to accumulate gradients. Therefore first we create tensor, then we will create variable
    self.featuresTrain = torch.tensor(self.x)
    self.targetsTrain = torch.tensor(self.one_hot_labels)# data type is long
    self.seq_lengths = torch.tensor(self.seq_lengths)
    
    # Pytorch train and test sets
    self.dataset = torch.utils.data.TensorDataset(self.featuresTrain,self.targetsTrain,self.seq_lengths)
    
    # Creating data indices for training and validation splits:
    self.dataset_size = len(self.dataset)
    self.indices = list(range(self.dataset_size))
    

   
    
    #cleaning memory stuff

    gc.collect()
 

  def __len__(self):
    return dataset_size
  
  def __getitem__(self, idx):
    return self.dataset[idx]
  
  def getDataset(self):
    return self.dataset
  def getX(self):
    return self.x
  def getY(self):
    return self.one_hot_labels
  def getShape(self):
    return self.shape
  def getNumClasses(self):
   return self.num_classes
</code></pre>
<p>and this is skorch part</p>
<pre><code class="lang-auto">net = NeuralNetClassifier(
    model,
    max_epochs=20,
    lr=0.01,
    device='cuda',  # uncomment this to train with CUDA
)
net.fit(X, y)
</code></pre>

aatrey: <p>I have a similar problem. Did you get around to solving it?</p>
How to pass additional parameters to forward function during training and validation?

odats: <p>I use Ignite to train LSTM model based on <a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb" rel="nofollow noopener">this tutorial</a>.</p>
<p><code>def forward(self, source, target, teacher_force_ratio=0.5):</code></p>
<p>How to pass <code>target</code> and <code>teacher_force_ratio</code> during training (<code>teacher_force_ratio=0.5</code>) and validation process (<code>teacher_force_ratio=0)</code>?</p>
<p>I have checked <a href="https://pytorch.org/ignite/_modules/ignite/engine.html#create_supervised_trainer" rel="nofollow noopener">create_supervised_trainer</a> and could not find such possibility.</p>
<p>Should I extend Dataset class:<br>
<code>return (source, target) -&gt; return ((source, target, teacher_force_ratio), target)</code></p>

vfdev-5: <p>Hi <a class="mention" href="/u/odats">@odats</a></p>
<p>You can create a trainer with a custom <code>train_step</code> (or <code>update_fn</code>) as here:<br>
<a href="https://pytorch.org/ignite/concepts.html#engine" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/ignite/concepts.html#engine</a></p>
<p>HTH</p>
Different values after loading tensor with torch.load

morrohype: <p>Hey,</p>
<p>I’m trying to save a tensor and then load it using torch.save and torch.load. But I’m losing information doing so.</p>
<p>How I’ve done it:<br>
<code>torch.save(data, path)</code><br>
<code> ld_data = torch.load(path)</code></p>
<p>When comparing <code>data</code> with <code>ld_data</code> one can see, that the outputs are different:</p>
<p><code>data = tensor([[ 1.7173e-01, -2.5369e-02,  4.3857e-02,  ..., -1.7239e-02,          -3.3998e-11, -2.7491e-02],         [-2.5369e-02,  7.7908e-02,  3.1315e-02,  ..., -3.0796e-02,           1.4849e-11, -6.0712e-02],         [ 4.3857e-02,  3.1315e-02,  8.2180e-02,  ..., -3.8967e-02,          -1.1373e-11, -8.8518e-02],         ...,         [-1.7239e-02, -3.0796e-02, -3.8967e-02,  ...,  5.5503e-02,          -2.0781e-11,  7.9386e-02],         [-3.3998e-11,  1.4849e-11, -1.1373e-11,  ..., -2.0781e-11,           1.0000e-04, -1.7010e-11],         [-2.7491e-02, -6.0712e-02, -8.8518e-02,  ...,  7.9386e-02,          -1.7010e-11,  2.0927e-01]])</code></p>
<p><code>ld_data = tensor([[ 1.2479e+00, -1.9974e-02,  7.4556e-02,  ..., -4.1448e-02,           3.9807e-13, -7.2844e-02],         [-1.9974e-02,  1.0799e+00,  3.2171e-02,  ..., -2.7830e-02,           9.3514e-15, -7.1534e-02],         [ 7.4556e-02,  3.2171e-02,  1.1049e+00,  ..., -4.7784e-02,           3.3500e-14, -1.2831e-01],         ...,         [-4.1448e-02, -2.7830e-02, -4.7784e-02,  ...,  1.0771e+00,           3.3965e-14,  1.2947e-01],         [ 3.9807e-13,  9.3514e-15,  3.3500e-14,  ...,  3.3965e-14,           1.0000e+00, -1.6675e-12],         [-7.2844e-02, -7.1534e-02, -1.2831e-01,  ...,  1.2947e-01, -1.6675e-12,  1.3670e+00]])</code></p>
<p>How can I save and load such data without losing information?</p>
<p>Thank you!</p>

ptrblck: <p>I haven’t seen this issue before, so could you post an executable code snippet, which would reproduce this behavior, please?</p>

morrohype: <p>SOLVED: there was an error in the datapath. This led to a different tensor to be loaded…</p>
Batch efficient random sampling with replacement given mask

DomainFlag: <p>I have a tensor of size BxNx2, where B is the batch size, N is the number of bounding boxes, and 2 is the two floats min, max scalars for ray directions that once scaled and translated by the ray origin position are the 2 intersections within those bounding boxes.</p>
<p>The tensor itself is sparse, thus there is a mask of size BxN that filters boxes with no intersections.</p>
<p>The objective is to perform batch sampling for efficient ray casting, by that I need to sample e.g. 64 relevant voxels / boxes where intersection occurs with replacement (as there can be less) so that resulting tensor has size Bx64x2.</p>
<p>Is there any efficient way of doing it? My idea is to create a tensor of ones_like BxN and based on the mask assign 1e-10 weights for non-intersections, subsequently, use torch multinational distribution to sample accordingly, but I feel it’s too much inefficient?</p>
What does the output of nn.lstm mean?

juhyung: <p>Hi,</p>
<p>in pytoch docs, lstm outputs <code>output, (h_n, c_n)</code><br>
which</p>
<ul>
<li>
<strong>output</strong>  of shape (seq_len, batch, num_directions * hidden_size): …</li>
<li>
<strong>h_n</strong>  of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len.Like  <em>output</em> , the layers can be separated using…</li>
<li>
<strong>c_n</strong>  (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t = seq_len</li>
</ul>
<p>then in <strong>output</strong>, what is batch in (seq_len, batch, num_directions * hidden_size)<br>
is that mean my model’s batch_size?<br>
if so, output[:, 0, :] is the first batch of the lstm output?</p>

tom: <aside class="quote no-group" data-username="juhyung" data-post="1" data-topic="26901">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juhyung/48/6984_2.png" class="avatar"> juhyung:</div>
<blockquote>
<p>is that mean my model’s batch_size?<br>
if so, output[:, 0, :] is the first batch of the lstm output?</p>
</blockquote>
</aside>
<p>Yes exactly.</p>
<p>Best regards</p>
<p>Thomas</p>

abhigenie92: <p><a class="mention" href="/u/tom">@tom</a> how is output(hidden state) different from h_n(hidden state). Both seem to be the hidden state, thus I am getting here.</p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Specific Aten object causes pytorch build to hang and then fail on raspberry pi zero

Brittany: <p>Hi,</p>
<p>I am building Pytorch 1.7 from source on a raspberry pi zero (raspbian-lite: kernel version 4.19) and the build is hanging at the same spot every single time nearly indefinitely. Here are my steps:</p>
<pre><code class="lang-auto">sudo apt install libopenblas-dev libblas-dev m4 cmake cython python3-dev python3-yaml python3-setuptools python3-wheel python3-pillow python3-numpy git

git clone --recursive https://github.com/pytorch/pytorch
cd pytorch

export USE_CUDA=0
export USE_CUDNN=0
export BUILD_TEST=0
export USE_MKLDNN=0
export USE_DISTRIBUTED=0
export USE_NNPACK=0

</code></pre>
<p>(I have tried not setting these as well and I still get stuck at the same spot so I don’t think it’s this.I have also tried to run python3 setup.py clean prior and still no luck)</p>
<p><code>python3 setup.py install --verbose</code></p>
<p><strong>It always hangs indefinitely (12+ hours) at this line Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/Functions.cpp.o</strong></p>
<p>or it hangs there and produces this error:</p>
<p>c++: fatal error: Killed signal terminated program cc1plus<br>
compilation terminated.<br>
make[2]: *** [caffe2/CMakeFiles/torch_cpu.dir/build.make:3632: caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/Functions.cpp.o] Error 1<br>
make[1]: *** [CMakeFiles/Makefile2:1558: caffe2/CMakeFiles/torch_cpu.dir/all] Error 2<br>
make: *** [Makefile:141: all] Error 2<br>
Traceback (most recent call last):<br>
File “setup.py”, line 737, in <br>
build_deps()<br>
File “setup.py”, line 321, in build_deps<br>
cmake=cmake)<br>
File “/home/pi/pytorch/tools/build_pytorch_libs.py”, line 62, in build_caffe2<br>
cmake.build(my_env)<br>
File “/home/pi/pytorch/tools/setup_helpers/cmake.py”, line 345, in build<br>
self.run(build_args, my_env)<br>
File “/home/pi/pytorch/tools/setup_helpers/cmake.py”, line 141, in run<br>
check_call(command, cwd=self.build_dir, env=env)<br>
File “/usr/lib/python3.7/subprocess.py”, line 347, in check_call<br>
raise CalledProcessError(retcode, cmd)<br>
subprocess.CalledProcessError: Command ‘[‘cmake’, ‘–build’, ‘.’, ‘–target’, ‘install’, ‘–config’, ‘Release’, ‘–’, ‘-j’, ‘1’]’ returned non-zero exit status 2.</p>
<p>Does anyone have an idea what could be causing this and what I could do to resolve?</p>
<p><strong>Any help would be greatly appreciated!</strong></p>

ptrblck: <p>Could you check the memory usage on your RPi and check, if you might be running out of memory while compiling on the device?<br>
If that’s the case, you might need to add a larger <code>swap</code> and rerun the build.</p>
<p>The last time I’ve compiled PyTorch on my RPi ver1, I needed to increase the <code>swap</code> by quite a bit and I think it took approx. a day to finish the compilation on the device (it was the first RPi with a single core).</p>
How to change weights and bias nn.Module layers?

Stefano_Setti: <p>Hi, so i have this network:</p>
<pre><code class="lang-auto">
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Sequential( 
                        nn.Conv2d(1, 20, 5),
                        nn.ReLU())
        self.conv2 = nn.Sequential(
                       nn.Conv2d(20, 20, 5),
                       nn.Sigmoid())

    def forward(self, x):
        x = self.conv1(x)
        return self.conv2(x)
</code></pre>
<p>how can i access the weights an bias of each layer and modify them?</p>

user_123454321: <pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Sequential( 
                        nn.Conv2d(1, 20, 5), # layer1
                        nn.ReLU())
        self.conv2 = nn.Sequential(
                       nn.Conv2d(20, 20, 5), # layer2
                       nn.Sigmoid())

    def forward(self, x):
        x = self.conv1(x)
        return self.conv2(x)
</code></pre>
<p>One can get the weights and biases of layer1 and layer2 in the above code using,</p>
<pre><code class="lang-auto">model = Model()
weights_layer1 = model.conv1[0].weight.data # gets weights
bias_layer1 = model.conv1[0].bias.data # gets bias
weights_layer2 = model.conv2[0].weight.data
bias_layer2 = model.conv2[0].bias.data
</code></pre>
<p>Similarly you can modify the weights/bias using,</p>
<pre><code class="lang-auto">model.conv1[0].weight.data = my_weights # make sure the weights are of same shape
</code></pre>
AMP out of memory during validation

T_Qri: <p>Hi, I’m using AMP in my training process and validate the model with torch.no_grad periodically, say every 1000 updates. For the first 1000 updates, the program works correctly, showing fewer GPU memory consumption and faster speed compared to regular FP32 training. However, as soon as I started validation, the memory quickly blows up and throws an CUDA OOM error. On the other hand, FP32 training with exactly the same hyperparameters and training data does not OOM throught the whole training process.</p>
<pre><code class="lang-python">@torch.no_grad()
def validate(args, model, dev_itr):
    ...
def train():
    for i, batch in enumerate(train_itr):
        with torch.cuda.amp.autocast(True):
            loss=model(batch) # batch and model both fp32
            loss.backward()
            optimizer.step()
        if i%val_every==0:
            with torch.cuda.amp.autocast(True):
               validate(args, model, dev_itr) # OOM
</code></pre>
<p>If I disable the <code>autocast</code> right outside <code>validate()</code>, the training proceeds successfully.<br>
Any clue on the source of the problem would be helpful.</p>

ptrblck: <p>Could you post an executable code snippet to reproduce this issue?<br>
Also, which GPU, CUDA, and cudnn versions are you using?</p>
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/THCGeneral.cpp:50

ftamur: <p>Hello Everyone,</p>
<p>I have encountered the error:</p>
<p>RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/THCGeneral.cpp:50</p>
<p>I tried many things I have seen on the internet but I could not solve my problem.<br>
I am working on a high-performance cluster and here its attributes.</p>
<blockquote>
<blockquote>
<p>nvidia-smi</p>
</blockquote>
</blockquote>
<p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|===============================+======================+======================|<br>
|   0  Quadro M2000        Off  | 00000000:08:00.0 Off |                  N/A |<br>
| 56%   41C    P0    23W /  75W |     41MiB /  4043MiB |      0%      Default |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  Tesla P4            Off  | 00000000:88:00.0 Off |                    0 |<br>
| N/A   50C    P0    24W /  75W |   7346MiB /  7611MiB |      0%      Default |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID   Type   Process name                             Usage      |<br>
|=============================================================================|<br>
|    1     43048      C   julia                                        109MiB |<br>
±----------------------------------------------------------------------------+</p>
<blockquote>
<blockquote>
<p>print(torch.<strong>version</strong>)<br>
1.2.0<br>
print(torch.cuda.device_count())<br>
2<br>
print(torch.cuda.is_available())<br>
True<br>
print(torch.cuda.current_device())<br>
0<br>
print(torch.cuda.get_device_name(0))<br>
Tesla P4</p>
</blockquote>
</blockquote>

ptrblck: <p>Could you update to the latest stable PyTorch release and retry the code?<br>
The error message seems fishy, as apparently PyTorch detects the device in the posted script.</p>
Train few neurons in last fully connected layer

awais980: <p>Is it possible to train few neurons in last fully connected layers of any mode in PyTorch? For example, if the last two layers are 256,128 in size and output has 10 nodes. Then, is it possible to train last 128 neurons of 256-size layer, last 64 neurons in 128-size layer and all 10 neurons of the output?</p>

ptrblck: <p>You could manually zero out the weights which you do not want to train.<br>
Note however, that weight decay (or other regularization terms) could still update parameters with a zero gradient.</p>
Best way to get an activation/layer output BasicBlock from pertained ResNet50

watermelon: <p>Hi there!</p>
<p>What is the best way to get ReLu/any other layer output from a BasickBlock from ResNet50? It’s easy to call for layers form Bottleneck like <code>model.layer4[0].conv3 </code>and use, for example, <code>forward_hook</code> to get values from this layer.<br>
How to change the classes structure to obtained the required information?</p>

ptrblck: <p>You could also use a forward hook and specify the <a href="https://github.com/pytorch/vision/blob/27278ec8887a511bd7d6f1202d50b0da7537fc3d/torchvision/models/resnet.py#L48-L52">module</a> you want, e.g. <code>conv1</code>, <code>bn</code>, <code>relu</code>, etc.</p>
Why is there this warning message?

coincheung: <p>Hi,</p>
<p>I am trying to implement an qkv attention, and my attention module is like this:</p>
<pre><code class="lang-auto">class QKVAttention(nn.Module):

    def __init__(self, n_chan_q=112, n_chan_kv=1280, mid_chan=256):
        super(QKVAttention, self).__init__()
        self.mid_chan = mid_chan
        self.conv_q = ConvBlock(n_chan_q, mid_chan, 1, 1, 0)
        self.conv_k = ConvBlock(n_chan_kv, mid_chan, 1, 1, 0)
        self.conv_v = ConvBlock(n_chan_kv, mid_chan, 1, 1, 0)

    def forward(self, feat, featq):
        n, c, h, w = featq.size()
        q = self.conv_q(featq).view(n, self.mid_chan, -1)
        k = self.conv_k(feat).view(n, self.mid_chan, -1)
        v = self.conv_v(feat).view(n, self.mid_chan, -1)
        #  qkv = q.permute(0, 2, 1).bmm(k).softmax(dim=2).bmm(v.permute(0, 2, 1))
        #  qkv = v.bmm(k.permute(0, 2, 1).bmm(q).softmax(dim=1))
        qkv = v.bmm(k.transpose(1, 2).bmm(q).softmax(dim=1))
        qkv = qkv.view(n, self.mid_chan, h, w)
        return qkv
</code></pre>
<p>The <code>ConvBlk</code> is a <code>nn.Conv</code> followed by a <code>nn.BatchNorm2d</code> and a <code>nn.ReLU</code>, my problem is that when I run this code there is a warning message like this:</p>
<blockquote>
<p>[W TensorIterator.cpp:918] Warning: Mixed memory format inputs detected while calling the operator. The operator will output contiguous tensor even if some of the inputs are in channels_last format. (function operator())</p>
</blockquote>
<p>Would  you tell me which operator broughts this warning, and what is wrong with my implementation ?</p>

ptrblck: <p>This warning can be raised, if the <code>TensorIterator</code> detects a <code>channels_last</code> tensor and another one in the <code>contiguous</code> memory format.<br>
I guess <code>k.transpose(1, 2).bmm(q)</code> might be raising it.<br>
Could you print the <code>.size()</code> and <code>.stride()</code> of both involved tensors?</p>

coincheung: <p>Hi,</p>
<p>I printed the two involved tensors, their size and stride are like this:</p>
<pre><code class="lang-auto">q.stride():  (147456, 576, 1)
q.size():  torch.Size([8, 256, 576])
k.transpose(1, 2).stride():  (147456, 1, 576)
k.transpose(1, 2).size():  torch.Size([8, 576, 256])
</code></pre>
<p>Will this warning have negative impact what I want to do, or is there any better way to implement this ?</p>
<p>Since the kernel size of my <code>nn.conv2d</code> (1, 1), is it better for me to use <code>nn.linear</code> ?</p>

ptrblck: <p>The warning shouldn’t have any negative effects, if you are not changing the memory format manually.<br>
You could replace the conv layer with a linear layer, but this should not be necessary and I would recommend to stick to the approach, which makes the code easier to understand.</p>
<p>I think the memory format cannot be automatically detected due to the currently used (ambiguous) kernel shape.</p>
Evaluation loop per class, and transformation

azer: <p>Hi,</p>
<p>Urgent help!</p>
<p>I would appreciate it if someone could help me. I’m trying to implement mask rcnn on Pascal VOC 2012. My main concerned are the transformations and AP. I resized the boxes, images and mask in the dataset class and used transforms separately. After 15 epochs I got AP  Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.65 for the boxes and 0.53 for the mask. However, the AP for small objects is not higher than 0.15.  I am not really sure if I am applying the transformation correctly if I need to rotate and them scale. I run another version of the code with transforms built in the class for the mask and image and separately rescale the box. I did not use the horizontal flip and I got the results below in one epoch.</p>
<p>Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.756<br>
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.659 ,</p>
<p>The issue with small objects remained.</p>
<p>Also, I am using the coco metrics . Does anyone know if it is possible to get the AP per class using COCO Average Precision  (AP) @[ IoU=0.50?<br>
I 've been stuck on this, any piece of advice will be greatly appreciated.</p>
<p>def resize_img(self, imag, img_max_size):<br>
“”“Resizes images and masks from PIL Images”""</p>
<pre><code>    h_, w_ = self.img_max_size[0], self.img_max_size[1]
    
    
    img_resized= imag.resize((h_,w_))
    # returns resized image
    return img_resized

def resize_box(self, box, img_max_size):
    """resizes bounding boxes"""
    h_ratio =  img_max_size[0]/h
    w_ratio =  img_max_size[1]/w

    box[0] = round(box[0] * w_ratio)
    box[2] = round(box[2] * w_ratio)
    box[1] = round(box[1] * h_ratio)
    box[3] = round(box[3] * h_ratio)
  
    return box   
</code></pre>
<p>def get_transform(train):<br>
transforms = []<br>
transforms.append(T.ToTensor())<br>
if train:<br>
transforms.append(T.RandomHorizontalFlip(0.5))<br>
return T.Compose(transforms)</p>
Understanding backward in WGAN-GP

pnavarre: <p>Hi, I’m having trouble understanding the code of WGAN with gradient penalty <a href="https://github.com/caogang/wgan-gp/blob/master/gan_toy.py" rel="noopener nofollow ugc">here</a>.</p>
<p>The code defines two tensors:</p>
<blockquote>
<p>one = torch.FloatTensor([1])<br>
mone = one * -1</p>
</blockquote>
<p>Later, it defines the loss and takes gradient like this:</p>
<blockquote>
<p>D_real = netD(real_data_v).mean()<br>
D_real.backward(mone)</p>
</blockquote>
<p>D_real is an autograd.Variable, and backward is a method associated to D_fake. What is the argument “mone” for?</p>
<p>Same for the argument “one” in:</p>
<blockquote>
<p>D_fake = netD(inputv).mean()<br>
D_fake.backward(one)</p>
</blockquote>
<p>Does it have something to do with the labels for fake and real? Wouldn’t we just need to call .backward() with no arguments? This seems to be the standard source for WGAN-GP. I see a similar code in <a href="https://github.com/AlexiaJM/Deep-learning-with-cats/blob/master/Generating%20cats/WGAN-GP.py" rel="noopener nofollow ugc">here</a>, but I guess it was modified from the previous.</p>

tom: <p>The -1 is multiplied to the gradient, so the loss term contains it negatively. The 1 is probably not needed, but we all copied it from the DCGAN in the pytorch examples or the WGAN code.<br>
For DCGAN and plain WGAN I can see the advantage over adding loss components (memory consumption), fir WGAN-GP its probably just copy-paste without thinking, at least that is the case for me.</p>
<p>Best regards</p>
<p>Thomas</p>

hbxalwayshappy: <p>In order to Update D network:<br>
lossD = Expectation of D(fake data)  - Expectation of  D(real data) + gradient penalty<br>
lossD ↓，D(real data) ↑</p>
<p>so you need to add minus one to the gradient process</p>

saba: <p>Hi Tom,<br>
I hope you are well. Sorry in this lines when we update the discriminator which cost is used for updating ?</p>
<pre><code class="lang-auto">errD_penalty = errD_fake - errD_real + grad_penalty
errD = errD_fake - errD_real
optimizerD.step()
</code></pre>
<p>or here</p>
<pre><code class="lang-auto">D_cost = D_fake - D_real + gradient_penalty
 Wasserstein_D = D_real - D_fake
optimizerD.step()
```</code></pre>
How to split a dataset into multiple databases and create a new dataset

Blueberry70: <p>Hi all,</p>
<p>I have an image dataset loaded from ImageFolder:</p>
<pre><code class="lang-auto">import torchvision.datasets as datasets

train_dataset = datasets.ImageFolder(root=train_dir, transform=transform)

I want to split image into multiple chunk and create a new train_dataset, something like: 
new_train_dataset = torch.utils.data.TensorDataset(img_chunk1, img_chunk2, img_chunk3, y_train)

then create train_loader from the new dataset with multiple inputs to train model as: 

for i, (input1, input2, input3, labels) in enumerate(train_load):
     .....
     output = model(input1, input2, input3) 
</code></pre>
<p>How to split the first train_dataset to small chunk of images and feed into new_train_dataset?</p>
<p>Thank you very much for help. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
MPI Backend with GPU support

alam4545: <p>I want to train my model using 2 nodes in a HPC system. Each node contains 4 Nvidia V100 GPUs. It requires MPI if more than 1 node is used. However, I don’t have enough expertise on MPI. Besides, I have found a paucity of information about Pytorch MPI Backend with GPU support. During the preparation of my model, I intended to train it to a single machine with 8 GPUs. Unfortunately, I don’t have access to that sort of machine. The HPC as mentioned is the only option for me. I have already gone through Open MPI documents and successfully compiled pytorch 1.5.1(from source) with cuda (10.1) and Open MPI (3.0.4) (CUDA-aware). I would highly appreciate if someone provide me a snippet of code to make specific changes to my source code so that I can train the model in the HPC.<br>
Thank you.</p>

mrshenli: <p>Did you hit any error when using CUDA-aware MPI backend? Based on <a href="https://discuss.pytorch.org/t/mpi-cuda-stream/80702">past discussion</a>, you might need to synchronize CUDA streams in the application code when using CUDA-aware MPI. BTW, is MPI the only option for you, or would Gloo backend work?</p>

alam4545: <p>MPI is the only option for me.</p>

Pedro_Lima: <p>maybe this will work <a href="https://github.com/horovod/horovod" rel="nofollow noopener">https://github.com/horovod/horovod</a></p>
Custom weight initiatlization in conv1d

Alhpa_Delta: <p>I am trying to create custom weights in conv1d as follows:</p>
<p>import torch<br>
from torch import nn<br>
conv = nn.Conv1d(1,1,kernel_size=2)<br>
K = torch.Tensor([[[0.5, 0.5]]])<br>
with torch.no_grad():<br>
conv.weight = K</p>
<p>But I am getting the error<br>
"File “D:\ProgramData\Miniconda3\envs\pytorchcuda102\lib\site-packages\torch\nn\modules\module.py”, line 611, in <strong>setattr</strong><br>
raise TypeError(“cannot assign ‘{}’ as parameter ‘{}’ "<br>
TypeError: cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected)”</p>
<p>What am I doing wrong?</p>

Mughees: <p>Welcome to Pytorch Discussion Forum. I answered your question at <a href="https://stackoverflow.com/a/63442061/13073020" rel="nofollow noopener">Stackoverflow</a> as well.</p>
<p>As per the discussion <a href="https://discuss.pytorch.org/t/custom-weight-initialization/20544/7">here</a>, update your code to include <code>torch.nn.Parameter()</code>, which basically makes the weight recognizable as a parameter in optimizer.</p>
<pre><code>import torch
from torch import nn
conv = nn.Conv1d(1,1,kernel_size=2)
K = torch.tensor([[[0.5, 0.5]]]) #use one dimensional as per your conv layer
conv.weight = nn.Parameter(K) #use nn.parameters
</code></pre>
<p>for Bigger and complex models you can see this toy example which uses pytorch <code>state_dict()</code> method.</p>
<pre><code>import torch
import torch.nn as nn
import torchvision

net = torchvision.models.resnet18(pretrained=True)

pretrained_dict = net.state_dict()
conv_weights = pretrained_dict['conv1.weight'] #64,3,7,7

new = torch.tensor((), dtype=torch.int32)
new = new.new_ones(conv_weights.shape) #assigning all ones

pretrained_dict['conv1.weight'] = new


net.load_state_dict(pretrained_dict)
param = list(net.parameters())

print(param[0])
</code></pre>
Colab Pytorch Version

n4tman: <p>Hi All,</p>
<p>I have recently started using torch in Google Colab.</p>
<p>When I print out <code>torch.__version__ </code> I get 1.6.0+cu101.</p>
<p>I assume this means pytorch version 1.6.0 is installed?</p>
<p>Thanks!</p>

RupertP: <p>yes it uses pytorch 1.6</p>

n4tman: <p>So colab is using Pytorch version 1.6.0?</p>

Mughees: <p>yes along with Cuda10.1</p>