akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Load only some layers pytorch

Mathieu_Grosso: <p>Hello,<br>
i have model that i have trained on rgb images, now i want to train it on raw data but the training is really long (it took me a week to train the model on rgb data with a good gpu).<br>
I would like to load the weights some of the layers from the model that i have trained on rgb data and reuse them in my new model that works with raw.<br>
Basically only the first layers change, and i would like to load the weights in every other layer. does someone know how to do ?<br>
Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I think loading the entire trained <code>state_dict</code> and re-initializing the first layers would be the easiest and cleanest approach. Alternatively, you could also try to load the parameters using a per-layer approach, but this sounds unnecessary complicated.</p>
Failed to save weights into a local file

44_Pashikanti_Pranit: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97f258b844a4b44c1b9f83e27142db51129fe827.png" data-download-href="https://discuss.pytorch.org/uploads/default/97f258b844a4b44c1b9f83e27142db51129fe827" title="Screenshot (373)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97f258b844a4b44c1b9f83e27142db51129fe827_2_690x388.png" alt="Screenshot (373)" data-base62-sha1="lGbk2c51qFYe7CnSO0z4p2Ohcur" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97f258b844a4b44c1b9f83e27142db51129fe827_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97f258b844a4b44c1b9f83e27142db51129fe827_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97f258b844a4b44c1b9f83e27142db51129fe827_2_1380x776.png 2x" data-dominant-color="332527"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (373)</span><span class="informations">1920×1080 251 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on the error message <code>./models/weights.pt</code> doesn’t exist and it’s unfortunately not possible to give any better advice than to check the current working directory and to make sure this file and folder are indeed there without any more information.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```.</p>
What is the official way to use miniImagenet in Pytorch?

pinocchio: <p>I wanted to make sure I was using the best and standard way to use miniImagent for pytorch. Is this the recommended way:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/markdtw/meta-learning-lstm-pytorch/blob/master/dataloader.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/markdtw/meta-learning-lstm-pytorch/blob/master/dataloader.py" target="_blank" rel="nofollow noopener">markdtw/meta-learning-lstm-pytorch/blob/master/dataloader.py</a></h4>
<pre><code class="lang-py">from __future__ import division, print_function, absolute_import

import os
import re
import pdb
import glob
import pickle

import torch
import torch.utils.data as data
import torchvision.datasets as datasets
import torchvision.transforms as transforms
import PIL.Image as PILI
import numpy as np

from tqdm import tqdm


class EpisodeDataset(data.Dataset):

</code></pre>

  This file has been truncated. <a href="https://github.com/markdtw/meta-learning-lstm-pytorch/blob/master/dataloader.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>related:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-does-one-download-a-data-set-from-a-file-automatically-with-pytorch/74262" class="inline-onebox">How does one download a data set from a file automatically with Pytorch?</a></li>
<li>I am having errors with the other solutions suggested: <a href="https://discuss.pytorch.org/t/how-does-one-download-dataset-from-gdrive-using-pytorch/74636" class="inline-onebox">How does one download dataset from gdrive using pytorch?</a>
</li>
</ul>

Konstantin_Burlachen: <p>The original paper with dataset does not make any specific recommendation: <a href="https://proceedings.neurips.cc/paper/2016/file/90e1357833654983612fb05e3ec9148c-Paper.pdf" rel="noopener nofollow ugc">https://proceedings.neurips.cc/paper/2016/file/90e1357833654983612fb05e3ec9148c-Paper.pdf</a></p>
How to write numpy array to checkpoint in PyTorch

rajeshpiryani: <p>To write checkpoint in TensorFlow, first it will build the computational graph, the nodes and operations and how they are connected to each other and then it will use session.</p>
<p>Following codes storing the checkpoint in TensorFlow:</p>
<pre><code class="lang-auto">import os
import tensorflow.compat.v1 as tf
import numpy as np

def savecheckpoint(var_name, db, dtype, ckpt_path):
  with tf.Graph().as_default():
    init_value = tf.py_func(lambda: db, [], dtype, stateful=False)
    init_value.set_shape(db.shape)
    tf_db = tf.get_variable(var_name, initializer=init_value)
    saver = tf.train.Saver([tf_db])
    with tf.Session() as session:
      session.run(tf.global_variables_initializer())
      saver.save(session, ckpt_path)

path = 'home/Desktop/checkpoints/'
temp = ['1','2','3']
savecheckpoint("coref", np.array([m[0] for m in temp], dtype=np.int32), tf.int32, os.path.join(path, "var.npz"))
</code></pre>
<p>I am new to PyTorch, I have tried to convert above mentioed code in PyTorch:</p>
<pre><code class="lang-auto">import os
import torch
import numpy as np

def write_to_checkpoint(var_name, np_db, dtype, checkpoint_path):
    """Write np array to checkpoint."""
    t_db = torch.from_numpy(np_db)
    checkpoint = {
        var_name:t_db
    }
    torch.save(checkpoint,checkpoint_path+var_name+'.pth')
</code></pre>
<p>Are these codes equivalent? If not then please suggest correct equivalent codes.</p>

ptrblck: <p>I don’t know what the TF code is exactly doing, but in PyTorch you can save and load numpy arrays as well and adding them to a <code>dict</code> is also a valid idea:</p>
<pre><code class="lang-python">a= np.random.randn(10)
checkpoint = {
    'np_arr': a
}
torch.save(checkpoint, 'tmp.pth')

b = torch.load('tmp.pth')['np_arr']

print(all(a == b))
&gt; True
</code></pre>
Not seeing any performance improvement on resnet50 with torch.jit.script

tumble-weed: <p>I tried to trace a modified resnet50 model ( taken from the <a href="https://github.com/facebookresearch/TorchRay" rel="noopener nofollow ugc">torchray repo</a>) . i havent observed any average improvements in forward pass upon doing this. my code is here: <a href="https://colab.research.google.com/drive/1AnCBRe1_4mUsDnIAGE24EFLRDjDSx4gg?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>can anyone help with what i might be doing wrong?</p>
Are fx.fuse_modules and torch.jit.script complementary?

tumble-weed: <p>I had a look a fx.fuse_modules <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide.html" rel="noopener nofollow ugc">write up</a>, and it suggests that modules to be fused need to be tracable. So i wanted to know if say i do scripting on a model, does it do fusing automatically, or are the 2 operations complementary and i should do them separately to get added performance gains?</p>
Get weight matrix for convolution network layer

Rajarshi_Banerjee: <p>I need to know the actual weight matrix that is obtained from the weights learnt by the convolution neural network. For a linear layer this is straight forward:</p>
<pre><code class="lang-auto">weights = linear_layer.weight 
</code></pre>
<p>This gives you the actual weight matrix with (also one can concat the bias if needed) which the intermediate input is multiplied with and so on and so forth. Unfortunately this is not as straight forward for the convultion layers where the weights are mostly shared and zero. Is there any function or method by which I can obtain a similar weight matrix for the convultion layers.</p>
<p>I need this as I want to perform some theoretical experiments involving the comutation of spectral norms for weight matrices for each layer and I can’t seem to find a straightforward way to do that for the convolution layers.</p>

ayalaa2: <p>Getting the convolution weights should be as easy as your example.</p>
<p>Are you instead trying to find a matrix M that represents your convolution such that if you multiply your input by this matrix you’ll get the corresponding output? Something like this? <a href="https://stackoverflow.com/questions/16798888/2-d-convolution-as-a-matrix-matrix-multiplication" class="inline-onebox" rel="noopener nofollow ugc">neural network - 2-D convolution as a matrix-matrix multiplication - Stack Overflow</a></p>
RuntimeError: 1D target tensor expected, multi-target not supported

NeuralFoX: <p>I am a beginner, and I am experimenting in PyTorch.<br>
I am trying to learn Custom Datasets with a toy example.<br>
Here is my script:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.utils.data as data
import torchvision.datasets as datasets
device=torch.device('cuda:0');
class CustomDataset(data.Dataset):
 def __init__(self,high):
  self.src=list(range(high));
  self.trgt=list(iter*1 for iter in range(high));
  self.allItems=self.src;
 def inttoTensor(self,integer):
  tensor=torch.zeros(len(self.src));
  index=0;
  for item in self.allItems:
   if item==integer:
    index=self.allItems.index(item);
  print(index);
  tensor[index]=1;
  return tensor;
 def __len__(self):
  return len(self.src);
 def __getitem__(self,idx):
  number=self.src[idx];
  numbert=self.trgt[idx];
  numtensor=self.inttoTensor(number);
  numberttensor=self.inttoTensor(numbert);
  return numtensor,numberttensor;
dset=CustomDataset(50);
dataloader=data.DataLoader(dset,batch_size=1);
model=nn.Linear(50,50);
loss=nn.CrossEntropyLoss();
optimizer=torch.optim.SGD(model.parameters(),lr=0.05);
for batch,(X,y) in enumerate(dataloader):
 pred=model(X);
 lossv=loss(pred,y);
 optimizer.zero_grad();
 loss.backward();
 optimizer.step();

</code></pre>
<p>I have looked at a lot of tutorials and some problems like this on this forum, though I am not able to fix it. When I run the script: This is the output I get:</p>
<pre><code class="lang-auto">0
0
Traceback (most recent call last):
  File "cust.py", line 35, in &lt;module&gt;
    lossv=loss(pred,y);
  File "/home/samarth/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/samarth/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 1121, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/samarth/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2824, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1D target tensor expected, multi-target not supported
</code></pre>
<p>Can anyone help me?<br>
Thank you.</p>

zetyquickly: <p>Hi <a class="mention" href="/u/neuralfox">@NeuralFoX</a></p>
<p>Seems like you have the sequence of ascending numbers in</p>
<p><code>self.trgt=list(iter*1 for iter in range(high));</code></p>
<p>The <code>cross_entropy_loss</code> treats it like multi-class labels. Probably you have a buggy code in the snippet above</p>

NeuralFoX: <p>Thank you. I will try this out. So you are saying that there is something wrong with the list comprehension? Thanks again.</p>

NeuralFoX: <p>Sorry. I figured out my problem. I had written</p>
<pre><code class="lang-auto">
self.trgt=list(iter*1 for iter in range(high));

</code></pre>
<p>I meant it to be <code>iter*2</code> instead of <code>iter*1</code>. Thanks a lot for helping me.</p>
Conditional GAN loss issue

ArshadIram: <p>I want to calculate the loss between two images that are<br>
``</p>
<p><code>But the problem is that: </code><br>
x shape is [128, 11, 28, 28] and x’ shape is [128,1,28,28].<br>
``<br>
How can I calculate the loss between these two images?</p>
<p>Any help would be appriciated.</p>

ptrblck: <p>You could broadcast the second tensor to the same shape as the first one and could then apply e.g. <code>nn.MSELoss</code>. The right approach depends on the actual use case and the desired loss function, i.e. since both tensors have a different shape, what <em>should</em> the loss represent?</p>

ArshadIram: <p>Thanks I was thinking the same</p>
Link to doc issue: borken link due to change

sbelharbi: <p>hi,<br>
a month ago i used a link to a torchvision doc page in a report.<br>
now, the link does not work. it seems that the structure of the site has changed.<br>
what is the best way to reference a page in the doc?<br>
or we shouldnt reference the doc as it is subject to change? thanks</p>
<p>the initial <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ColorJitter" rel="noopener nofollow ugc">link</a>  which worked fine a month ago but now it does not exist:</p>
<p><code>https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ColorJitter</code></p>
<p><a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ColorJitter" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ColorJitter</a></p>
<p>now, it is replaced with this new <a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ColorJitter" rel="noopener nofollow ugc">link</a> :<br>
<code>https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ColorJitter</code></p>
<p><a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ColorJitter" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ColorJitter</a></p>
<p>thanks</p>

KFrank: <p>Hi Soufiane!</p>
<aside class="quote no-group" data-username="sbelharbi" data-post="1" data-topic="131632" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f475e1/48.png" class="avatar"> sbelharbi:</div>
<blockquote>
<p>the initial <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ColorJitter" rel="noopener nofollow ugc">link</a>  which worked fine a month ago but now it does not exist:</p>
</blockquote>
</aside>
<p>A link that points to documentation for the “stable” version of pytorch<br>
does change out from under you as new stable versions are released.</p>
<p>It may be safer to link to a specific version, such as:</p>
<p><code>https://pytorch.org/docs/1.6.0/torchvision/transforms.html#torchvision.transforms.ColorJitter</code></p>
<p><code>https://pytorch.org/vision/0.10/transforms.html#torchvision.transforms.ColorJitter</code></p>
<p>(Of course the documentation elves can do whatever they want,<br>
including breaking links for specific versions, but, in my experience,<br>
specific-version links are less likely to undergo link-rot.)</p>
<p>Best.</p>
<p>K. Frank</p>

sbelharbi: <p>hi Frank,<br>
thanks.<br>
i thought it is the other way around where <code>stable</code> version is actually ‘stable’ one and does not change.<br>
but, i think you are right. stable version is the one that is more likely to change while previous versions are more likely to stay the way they are since they wont receive any updates.<br>
thanks!</p>
Resetting or changing weights for specific layers, to simulate bitflips

Jonas_ar: <p>Hello everyone,</p>
<p>I want to change or reset the weights of specific layers to see the effect on the object classification accuracy of some models.<br>
I basically want to test the fault resillience of certain object classification models and simulate a bitflip that changes the weights in one or multiple layers.<br>
Right now I implemented that the weights of all layers get randomized after lets say 25 out of 50 epochs. Obviously the object classification accuracy goes down significantly from 95% to 30-35 % (basically 1/3, which makes sense, because its the “guessing” accuracy), since I am resetting all weights of a pretrained model (AlexNet in my case).</p>
<p>def init_params(m):</p>
<pre><code>if type(m)==nn.Linear or type(m)==nn.Conv2d:

   m.weight.data=(torch.randn(m.weight.size())*.01).to(device) #Random weight initialisation
</code></pre>
<p>Thats how I randomize the weights and how I apply them to the model after 25 epochs:</p>
<p>setting the random weights after specific epoch</p>
<pre><code>    if epoch == 25:

        alexnet.apply(init_params) # torch randn weight initalisation
</code></pre>
<p>From my understanding you can use <em>state_dict</em> to see the models changeable parameters. For my model for example that is:</p>
<p>Print model’s state_dict</p>
<p>print(“Model’s state_dict:”)</p>
<p>for param_tensor in alexnet.state_dict():</p>
<pre><code>print(param_tensor, "\t", alexnet.state_dict()[param_tensor].size())
</code></pre>
<p>Model’s state_dict:<br>
features.0.weight 	 torch.Size([64, 3, 11, 11])<br>
features.0.bias 	 torch.Size([64])<br>
features.3.weight 	 torch.Size([192, 64, 5, 5])<br>
features.3.bias 	 torch.Size([192])<br>
features.6.weight 	 torch.Size([384, 192, 3, 3])<br>
features.6.bias 	 torch.Size([384])<br>
features.8.weight 	 torch.Size([256, 384, 3, 3])<br>
features.8.bias 	 torch.Size([256])<br>
features.10.weight 	 torch.Size([256, 256, 3, 3])<br>
features.10.bias 	 torch.Size([256])<br>
classifier.1.weight 	 torch.Size([4096, 9216])<br>
classifier.1.bias 	 torch.Size([4096])<br>
classifier.4.weight 	 torch.Size([4096, 4096])<br>
classifier.4.bias 	 torch.Size([4096])<br>
classifier.6.weight 	 torch.Size([3, 4096])<br>
classifier.6.bias 	 torch.Size([3])</p>
<p>And now I want to change the weights of “features.0.weight”  resp. conv2d layer 1 or any other layer. The rest should remain the same though.</p>
<p>Any help would be appreciated.</p>

Jonas_ar: <p>A small update, I found a way to manipulate the stat_dict from the pretrained AlexNet.</p>
<p>I did it with</p>
<p>sd = model.state_dict()<br>
sd[‘features.0.weight’][0:2].zero_()</p>
<p>My question is, if there is a simular way to override the tensor within the state_dict with random numbers instead of just zeros</p>
<p>So basically<br>
“sd[‘features.0.weight’][0:2].randn_()”</p>

ptrblck: <p><code>tensor.normal_</code> should work</p>
Update network with multiple losses and branches

YoYoYo: <p>I have an autoencoder network with one shared encoder and two decoders for two different objects. For instance, I have the shared encoder and decoder_A and decoder_B. I am confused that when updating the weights, which one of the following scheme is correct:</p>
<p>(1) Run object A first: model(input_A, ‘A’), then compute loss_A, update encoder and decoder_A. Then run object B, model(input_B, ‘B’), compute loss_B, update encoder and decoder_B<br>
(2) Run object A and B: model(input_A, ‘A’) model(input_B, ‘B’), compute loss = loss_A + loss_B, then update encoder, decoder_A and decoder_B.</p>
<p>Are these two methods completely equal? In method (2), will the gradients in the encoder will accumulate when model(input_A, ‘A’) and model(input_B, ‘B’) are processed sequentially.</p>
<p>Thanks for the help!</p>
Creating a torchscript wrapper?

Michael_Moran: <p>I’m aware that torchscript does not support mixed serialization but is there anyway to easily save a torchscript file with a preprocessing wrapper? The actual input to my models are vectorized features derived from a text file and the features chosen are model hyperparameters, as such I wanted to hide that process under the hood allowing users to pass a raw text file and have the preprocessing steps wrapped around torchscript object but I can’t see how to do this without mixed serialization?</p>
<p>Is there a work around?</p>

vferrer: <p>Just make a class wrapper. First, make all preprocessing functions torchscript friendly, then you can create a class like:</p>
<pre><code class="lang-python">class Wrapper(nn.Module):
    def __init__(self, preprocess_fn, model):
       self.preprocess_fn = preprocess_fn
       self.model = model

    def forward(self, x): # remember, by default, x is expected to be a torch.Tensor. 
        return self.model(self.preprocess_fn(x))

wrapper = Wrapper(your_preprocess_function, your_model)
wrapper_ts = torch.jit.script(wrapper)
</code></pre>
<p>And that’s all!<br>
I did the same approach but to integrate some post processing steps without needing to deploy extra python files.</p>

Sele: <p><a class="mention" href="/u/vferrer">@vferrer</a> Hello! Your reply is very helpful, but I need some clarifications. What do you mean by “functions torchscript friendly”? What if I need to have an audio file as input and the preprocessing function is the one that extracts the feature tensor?</p>

vferrer: <p><a class="mention" href="/u/sele">@Sele</a> With “functions torchscript friendly” I mean python functions that can be scripted: <code>ts_fn = torch.jit.script(fn)</code> as they will compiled to torchscript (they are inside the forward pass).</p>
Where is the latency coming from post gpu kernel execution?

Crispin: <p>I am profiling my pytorch module with nvvp; it seems to have very low gpu usage.</p>
<p>Looking at one section of the profiler output - in this case this is from a line of code which raises each element of a pytorch cuda tensor to a constant power e.g. tensor**x (with tensor.requires_grad=True) - I see that compared to the CUDA compute time, large time periods are taken by</p>
<p>(i) the CPU  - is this really expected to be longer than the CUDA time for a tensor of 500,000 elements?</p>
<p>and (ii) by a mysterious latency between the end of the CUDA operation and the start of the next CPU op. Given execution should be asynchronous, why is this here?</p>
<p>Note there is no memcpy going on so I’m assuming my tensor is on the gpu as it should be.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f39b96bc2b1a10e2ecd64665e19e6e5a6c1dd12d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f39b96bc2b1a10e2ecd64665e19e6e5a6c1dd12d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f39b96bc2b1a10e2ecd64665e19e6e5a6c1dd12d_2_690x253.jpeg" alt="image" data-base62-sha1="yL3n73Jqa2OhQOIc9HKM1lLejEh" width="690" height="253" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f39b96bc2b1a10e2ecd64665e19e6e5a6c1dd12d_2_690x253.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f39b96bc2b1a10e2ecd64665e19e6e5a6c1dd12d_2_1035x379.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f39b96bc2b1a10e2ecd64665e19e6e5a6c1dd12d_2_1380x506.jpeg 2x" data-dominant-color="E2ECE4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1824×670 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Convolutional architecture to pick up this visual artefact

Omroth: <p>How many layers and what kernel size and stride would you suggest to pick up the <em>size</em> of the two dark areas here?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6559eb3ea19a8eef38e83acbece980facb164da.png" alt="image" data-base62-sha1="z9aTEkVIwnEuOuXYztbXwHW6Rmy" width="517" height="488"></p>
Can you store a cummulative sum of weight updates in a memory efficient way without applying them?

sad_robot: <p>Hi I’m doing a meta-learning AI project (using MAML), which requires you making multiple copies of one model then computing the loss, adapting (without backward()), then computing the loss again for each task.</p>
<p>The 3rd step requires aggregating all the losses (and by default: their compute graphs) into one loss for the meta-learner to optimize on. This apparently dramatically increases memory requirements such that it causes the program to crash almost automatically.</p>
<p>What I want to know is if it there is a memory efficient way to keep a cumulative sum of weight updates across meta-learning tasks so that I don’t end up with a massive compute graph by doing just loss1+loss2+… etc… In particular would using backward() multiple times achieve something like this?</p>
Pytorch and AMD GPU

Flock1: <p>Hi,</p>
<p>I have an RX 6700 AMD GPU and I have installed ROCm, which is getting recognized by the driver. Then I installed pytorch for ROCm and when I ran <code>import torch</code>, I got the following:</p>
<pre><code class="lang-auto">"hipErrorNoBinaryForGpu: Unable to find code object for all current devices!"
</code></pre>
<p>Can someone help me with the issue?</p>
3D Convolution Data mis-shaped

Omroth: <p>Hi guys.</p>
<p>I’m trying to train a 3D conv on an [32,32,32] image with 3 channels.</p>
<p>My batch (of size 1 at the moment) has shape (the channels are the last dimension):</p>
<p><code>torch.Size([1, 32, 32, 32, 3])</code></p>
<p>I get this error when doing layer.forward:</p>
<pre><code class="lang-auto">RuntimeError: Given groups=1, weight of size [16, 3, 5, 5, 5], expected input[1, 32, 32, 32, 3] to have 3 channels, but got 32 channels instead
</code></pre>
<p>My layer is defined by:</p>
<pre><code class="lang-auto">
nn.Conv3d(in_channels=3,out_channels=16,kernel_size=5, padding=2)
</code></pre>
<p>Where am I going wrong?</p>

Omroth: <p>Oh it looks like my input should be 3, 32, 32, 32</p>
Problem image dimensions fcn_resnet50 segmentation

Cedric_Baron: <p>Hello,</p>
<p>I am working with multi band input (for remote sensing) and I want to use a pre trained model.<br>
I modified the first convolution of my model to handle 7 channels image.<br>
Everything is working fine, except for an unknown reason the output dimensions are changed from 38x38 to 19x19.</p>
<p>When I run a simple fcn_resnet50 on my RGB image original dimensions are not changed. So I do not understand why it does know, since I kept the same conv parameters (stride and padding).</p>
<p>Here is the code for my model:</p>
<pre><code class="lang-auto">class TransferLearningModel(nn.Module):
  
  def __init__(self, args):

    super(TransferLearningModel, self).__init__() 
    
    self = self.float()
    
    # creating first layer
    self.convRGB = nn.Sequential(nn.Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False),
                                 nn.BatchNorm2d(64,eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                 nn.ReLU(inplace=True))
    self.convbands = nn.Sequential(nn.Conv2d(args.nb_channels-3, 64,kernel_size=(7,7), stride=(2,2), padding=(3, 3), bias=False),
                                   nn.BatchNorm2d(64,eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
                                   nn.ReLU(inplace=True))
    
    self.main = models.segmentation.fcn_resnet50(True)
    self.main.requires_grad  = False
    self.convRGB[0].weight.data = torch.clone(self.main.backbone.conv1.weight.data)
    self.convRGB[1].weight.data = torch.clone(self.main.backbone.bn1.weight.data)
    
    # emptying the first resnet conv
    self.main.backbone.conv1 = torch.nn.Identity()
    self.main.backbone.bn1 = torch.nn.Identity()
    self.main.backbone.relu = torch.nn.Identity()
    
    self.convRGB.requires_grad = False
    
        
    self.main.classifier[4] = nn.Conv2d(512, 1, kernel_size=(1, 1), stride=(1, 1))
    self.main.classifier[4].requires_grad  = True
    
    #weight initialization
    self.convbands[0].apply(self.init_weights)
    self.main.classifier[4].apply(self.init_weights)
    
    self.opti =  optim.Adam(self.parameters())
    
  def init_weights(self,layer): 
  
    nn.init.kaiming_normal_(layer.weight, mode='fan_out', nonlinearity='relu')
    
  def forward(self, input):

    # passing through our layers
    x1bands = self.convbands(input[:,3:,:,:])
    x1RGB = self.convRGB(input[:,0:3,:,:])
    
    x2 = self.main(x1bands+x1RGB)
    out = x2["out"]
    
    return out

</code></pre>
How do I sample images only from a specific set of classes among all the available classes without using torchvision.datasets.ImageFolder?

nikhil6041: <p>I came across this answer <a href="https://how-to-get-only-specific-classes-from-pytorchs-fashionmnist-dataset" rel="noopener nofollow ugc">https://stackoverflow.com/questions/63975130/how-to-get-only-specific-classes-from-pytorchs-fashionmnist-dataset</a> which samples images from a subset of classes using ImageFolder but I want to achieve the same thing using a custom Dataset class, any suggestions on how can i do it?<br>
I have attached my Custom Dataset class for the references.</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
	def __init__(self, root_dir,  transforms=None):
	        self.root_dir = root_dir
		self.transforms = transforms
		self.img_list = sorted(glob.glob(root_dir + "/*/*"))

	def __len__(self):
		return len(self.img_list)

	def __getitem__(self, idx):
		if torch.is_tensor(idx):
		    idx = idx.tolist()	
		img_name = self.img_list[idx]
		image = io.imread(img_name)
		if self.transforms:
		    sample = self.transforms(image)	
		return image
</code></pre>

ptrblck: <p>You won’t be able to use the same approach, as your custom <code>Dataset</code> doesn’t return any targets and you thus cannot use them to filter out the samples.</p>

nikhil6041: <aside class="quote no-group" data-username="nikhil6041" data-post="1" data-topic="131578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikhil6041/48/33142_2.png" class="avatar"> nikhil6041:</div>
<blockquote>
<pre><code class="lang-auto">class CustomDataset(Dataset):
	def __init__(self, root_dir,  transforms=None):
	        self.root_dir = root_dir
		self.transforms = transforms
		self.img_list = sorted(glob.glob(root_dir + "/*/*"))

	def __len__(self):
		return len(self.img_list)

	def __getitem__(self, idx):
		if torch.is_tensor(idx):
		    idx = idx.tolist()	
		img_name = self.img_list[idx]
		image = io.imread(img_name)
		if self.transforms:
		    sample = self.transforms(image)	
		return image
</code></pre>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
what if i change my custom dataset to return the labels in that case?</p>

ptrblck: <p>In case you are creating the targets in the <code>__init__</code> method or are passing them to the <code>Datset</code>, i.e. they are pre-calculated and not lazily computed, you could apply the same approach as in your posted link by filtering out the <code>.targets</code> as well as the corresponding <code>.img_list</code> or by using a <code>Subset</code>.</p>

nikhil6041: <p>Okay got it . Thanks!!</p>
Transformer: reinitialising embeddings breaks them?

st-vincent1: <p>I’m implementing adding new tokens to a pretrained Transformer model. For that I need to reinitialise the embedding vectors (expand them to include new tokens). However, I noticed that my <code>expand_embeddings</code> method, when run, breaks something in the model (loss goes up, predictions go rubbish).</p>
<p>Here’s what it looks like:</p>
<pre><code class="lang-auto">    def expand_embeddings(self, new_vocab_size, device):
        # Clone weights from embedding layer
        old_embeddings = self.embed.token_embed.weight.data.clone().transpose(0,1).to(device)

        # TEST: Don't include new tokens, instead simply reinitialise the layers as you would
        # if new tokens were to be added
        new_vocab_size = 32004
        
        # Initialise the embedding layer to the old weights
        self.embed = EmbeddingLayer(new_vocab_size, self.d_model, self.max_length, old_embeddings).to(device)
        # Tie weights with the output layer
        self.decoder.generator = Generator(self.d_model, new_vocab_size, self.embed.token_embed.weight.data).to(device)
</code></pre>
<p>Here are the Generator and EmbeddingLayer init’s for reference:</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self, d_model, vocab_size, tied_weights):
        super(Generator, self).__init__()
        self.proj = nn.Linear(d_model, vocab_size)
        self.proj.weight.data = tied_weights
    
    def forward(self, x):
        (...)

class EmbeddingLayer(nn.Module):
    def __init__(self, vocab_size, d_model, max_length, tied_weights=None):
        super(EmbeddingLayer, self).__init__()
        self.token_embed = Embedding(vocab_size, d_model)
        if tied_weights is not None:
            # Must transpose weights since Linear transposes by default but Embedding does not
            self.token_embed.weight.data = tied_weights.clone().transpose(0, 1)
        self.pos_embed = Embedding(max_length, d_model)
        self.vocab_size = vocab_size
    def forward(self, x, pos):
        (...)
</code></pre>
<p>When <code>expand_embeddings</code> is not run, the training loss starts at <strong>15.12</strong>, when it is run it starts at <strong>93.06</strong>. However, I’d expect these values to be identical.</p>
Conditional GAN concatenation of real image and label

ArshadIram: <p>I am concatenating real images with labels and the output shape is [128, 11,28, 28]. How can I visualize this image?</p>
<pre><code class="lang-auto">encoded_labels = F.one_hot(reallabels, num_classes)
encoded_labels = encoded_labels [:, :, None, None]
encoded_labels = encoded_labels .repeat(1, 1, mnist_shape[1], mnist_shape[2]) 
</code></pre>
<p>Here the encoded_labels size is torch.Size([128, 10, 28, 28])</p>
<p>Now I want to concatenate it with images<br>
<code>cat_real_labels = torch.cat((realImages.float(), encoded_labels .float()),1)</code><br>
output: torch.Size([128, 11, 28, 28])</p>
<p>How can I visualize this image?</p>

ptrblck: <p>You cannot visualize it directly, as the tensor doesn’t have a valid image shape.<br>
However, you could visualize the 11 channels separately of all 128 samples (so 1408 grayscale images).</p>

ArshadIram: <p>Can you just give me a hint of code? How can I visualize it separately for all 128 samples?</p>

ptrblck: <p>Something like <a href="https://discuss.pytorch.org/t/activation-values-in-resnet/128029/19">this approach</a> would work. The difference would be that you would have to flatten the first two dimensions in case you want to visualize all <code>11*128</code> grayscale images.</p>
TensorRT 8.0: Compiling pytorch 1.8 with CUDA11.3

Michael_Hsi: <p>Hello,</p>
<p>I’m trying to setup TensorRT 8.0, the Jetson TX2 I’m using uses ARM, hence I need to run TensorRT8.0+CUDA11.3. To make sure training and deployment have similar environment, I followed the documentation for <a href="https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing" rel="noopener nofollow ugc">TensorRT setup</a> and downloaded CUDA11.3.</p>
<p>Now I’m trying to get pytorch 1.8, given pytorch 1.8 is compiled with CUDA 11.1, I’m trying to compile myself. I got the pytorch 1.8 from: pytorch/releases/tag/v1.8.1 and is trying to follow <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">this guide</a> to compile. However, I can’t get pass the ‘git submodule update --init --recursive’, it says ‘fatal: not a git repository’</p>
<p>Hence:</p>
<ol>
<li>Should i be following this guide at all? Given im compiling version 1.8</li>
<li>If ‘Yes’ to 1, how do I circumvent the ‘fatal: not a git repository’ error?</li>
<li>if ‘No’ to 1, what should I be doing to compile pytorch 1.8?</li>
</ol>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Michael_Hsi: <p>I can’t put more than 2 links</p>
<p>I got pytorch 1.8 from: <a href="https://github.com/pytorch/pytorch/releases/tag/v1.8.1" class="inline-onebox" rel="noopener nofollow ugc">Release Small bug fix release · pytorch/pytorch · GitHub</a></p>

ptrblck: <ol>
<li>Yes, but I would recommend to use the latest release (<code>1.9.0</code>) or the master instead of <code>1.8.1</code> (or do you have a specific reason to stick to this older version?)</li>
<li>It seems you are not in the <code>pytorch</code> folder so <code>cd pytorch</code> after executing <code>git clone</code>.</li>
</ol>
@autocast drammatically decrease perfomance

andreys42: <p>In searching of ways to speed up training of my CNN U-Net base model I found that there is some autocast features in torch in torch.cuda.amp like ‘autocast’. If I understood right the conception is that Torch can automatically select appropriate tensor dtype in a runtime. So, I tried to decorate my ‘forward’ method with <span class="mention">@autocast</span>():</p>
<pre><code class="lang-auto">       ...
    @autocast()
    def forward(self, x):
     ...
</code></pre>
<p>Unfortinatelly this led to unpredicted behavior of model in training time : train loss decrease from 0.7 to 0.4 and then stagnate. All weights in model became 0 (i guess) because there is black picture as prediction… Without <span class="mention">@autocast</span> loss decreases from 0.7 to 0.01 in several epochs and perform well (but training is slow)</p>
<p>I’m wonder what can cause such behavior?</p>
<p>ps: My model got FloatTensor (X) and LongTensor (y in my case basically 0-1 mask). Previously I tried to cast input to HalfTensor but this gave no boost in speed (strange)<br>
pps: Time per step of training ~40 millions params take 5-6 seconds when using one gtx3070ti. I guess it is not really good result so I think there is some weakness in my code</p>

gphilip: <p>I don’t understand AMP very well, but I think you have to do gradient scaling as well, as described <a href="https://pytorch.org/docs/stable/amp.html#gradient-scaling" rel="noopener nofollow ugc">here</a>.</p>

andreys42: <p>I will test it , thank you!</p>
Loss isn't converging when my loss function contains more than one neural net weights

Harsh_Choudhary: <p>I am quite new to Pytorch so apologies beforehand if the readability of my post is not so good, I am designing  a network using pytorch which solves coupled differential equations. For that, I am designing a neural net for each variable i.e.<br>
<strong>NN_x(t), NN_y(t)</strong>,  Now, for this system I am using my custom loss function which is some function of NN_x and NN_y and the differentials of the neural nets.<br>
i.e. <strong>loss(NN_x, NN_y, dNN_x, d_NN_y)</strong><br>
I am initializing weights of both the neural nets with <strong>requires_grad = true</strong> So my loss function is essentially a function of all the weights of both the networks</p>
<p>I am using loss.backward() to get the gradients and then update the weights using gradient desend with momentum but when I see the loss, it never converges, I have tried different learning rates but i got no luck</p>
<p>I wanted to know is it the right approach i.e. will my loss.backward() function take the gradient  with respect to all the weights which my loss is a function of? i.e. weights of both the networks. If yes, why is my loss not converging and what else can I try so that my loss converges.</p>
<p>I am putting Some part of code where i am initializing the weights:</p>
<h1>
<a name="weights-initialization-for-1-hidden-layered-network-for-neural-net-nn_x-and-nn_y-1" class="anchor" href="#weights-initialization-for-1-hidden-layered-network-for-neural-net-nn_x-and-nn_y-1"></a>Weights initialization for 1 hidden layered network for Neural net NN_x and NN_y</h1>
<p>W_x = [Variable(torch.from_numpy(np.random.normal(loc=0.0, scale = np.sqrt(2/(1+20)), size = (1,20))), requires_grad=True), Variable(torch.from_numpy(np.random.normal(loc=0.0, scale = np.sqrt(2/(20+1)), size = (20,1))), requires_grad=True)]</p>
<p>W_y = [Variable(torch.from_numpy(np.random.normal(loc=0.0, scale = np.sqrt(2/(1+20)), size = (1,20))), requires_grad=True), Variable(torch.from_numpy(np.random.normal(loc=0.0, scale = np.sqrt(2/(20+1)), size = (20,1))), requires_grad=True)]</p>
<p>P.s. I am putting grad.data =0 after each step.</p>
Wav2Vec2: Forward() takes 2 positional arguments but 3 were given

dersu: <p>I am trying to use wav2vec2_base model for audio feature extraction. I am getting the error in the title for this code:</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    @overrides
    def forward(self, input_):
        return input_
</code></pre>
<pre><code class="lang-auto">            model = wav2vec2_base(num_out=32) 
            model.load_state_dict(torch.load("wav2vec2-base-960h.pt"))
            model.encoder = Identity()
</code></pre>
<pre><code class="lang-auto">            sample_rate, samples = wavfile.read(wav_file)
            samples = samples.reshape((1,samples.shape[0]))
            samples = torch.from_numpy(samples)
            #samples = samples.type(torch.short)
            print("Start")
            samples = samples.float()

            output = model(samples.float())
</code></pre>
<p>I have tried to select the related parts in the code but there is not much left besides these. What is wrong with my forward function? Thanks for any help!</p>

ptrblck: <p>Could you post the entire model definition as well as the error message including the stack trace?</p>