akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Same model after loading state dict is slow. Is it possible?

591ece3a2a04495180b7: <p>I tried some benchmarks on CPU.<br>
I tested inference speed of one CNN model which can compute optical flow.<br>
The inference speed I checked was 0.4 s with random init, but when I loaded the state dict then the speed became 2s. I don’t know why. This situation doesn’t occur if I used other models.<br>
Does anyone know the reason?</p>

albanD: <p>Hi,</p>
<p>I would try to set flush denormal to True, see <a href="https://pytorch.org/docs/stable/torch.html#torch.set_flush_denormal" rel="nofollow noopener">doc</a>.<br>
If that solves the issue, then the problem comes from the fact that your trained model contains very very small numbers that are badly handled by your cpu.</p>

591ece3a2a04495180b7: <p>Great! It works. Thank you very much</p>

KamWoh_Ng: <p>It works for me as well! Thanks! Debug for a whole day, I thought my code was ghosted.</p>
Jit.trace pass custom objects

tjoseph: <p>Hi everyone,<br>
I am wondering whether there is a way to pass custom objects to <code>forward()</code> when using jit.trace. I use custom objects to isolate some behaviour from the modules I pass my object to. For example some modules in my model can work with either Tensors or dictionaries of tensors as long as <code>view()</code> is supported. I implemented a wrapper that provides element-wise <code>view()</code> for dictionaries of tensors.</p>
<p>Here are some examples that show what I want to achieve and what the problem is:</p>
<h2>
<a name="example-1-1" class="anchor" href="#example-1-1"></a>Example 1:</h2>
<pre><code class="lang-auto">def foo(x, y):
    return 2 * x["a"] + y


inputs_bar = ({"a": torch.rand(3)}, torch.rand(3))
traced_foo = torch.jit.trace(foo, inputs_bar)
</code></pre>
<p>This works fine as <code>dict</code> is supported.</p>
<h2>
<a name="example-2-2" class="anchor" href="#example-2-2"></a>Example 2</h2>
<pre><code class="lang-auto">class Bar:
    def __init__(self, a):
        self.a = a

    def return_modified_content(self):
        return self.a + 10


def bar(x: Bar, y):
    return 2 * x.return_modified_content() + y

inputs_bar = (Bar(torch.rand(3)), torch.rand(3))
traced_bar = torch.jit.trace(bar, inputs_bar)
</code></pre>
<p>Results in</p>
<pre><code class="lang-auto">traced = torch._C._create_function_from_trace(
RuntimeError: Type 'Tuple[__torch__.Bar, Tensor]' cannot be traced. Only Tensors and (possibly nested) Lists, Dicts, and Tuples of Tensors can be traced
</code></pre>
<p>From the error I conclude that for some reason arguments must be of some default container types.</p>
<h2>
<a name="example-3-3" class="anchor" href="#example-3-3"></a>Example 3</h2>
<pre><code class="lang-auto">class Bar2(dict):
    def __init__(self, a):
        super().__init__({"a": a})
        self.a = a

    def return_modified_content(self):
        return self.a + 10


def bar2(x: Bar2, y):
    print(type(x))
    return 2 * x.return_modified_content() + y


inputs_bar2 = (Bar2(torch.rand(3)), torch.rand(3))

traced_bar2 = torch.jit.trace(bar2, inputs_bar2)
</code></pre>
<p>outputs <code>&lt;class 'dict'&gt;</code> and throws error <code>AttributeError: 'dict' object has no attribute 'return_modified_content'</code>. It seems like Bar2 is converted to dict.</p>
<p>Is there a way to make this work?</p>

tjoseph: <p>Additional question: Anyone know where to find a roadmap for the JIT?</p>
Segmentation methods for time-frames dataset

D_Assaad: <p>I am learning image segmentation meanwhile, and I can say that I am just a beginner, and I know some models used for segmentation (e.g. Unet). But lastly, I was working with a dataset where images are time-correlated, so the dataset is a time-frames taken from a video. So my question is: is it because we have this time correlation, some architecture models are better than others for this type of image? what do you recommend in such a case?<br>
Thank you</p>
GRU: many-to-one / one-to-many

papyrus: <p>Hi everyone,</p>
<p>I would like to implement a GRU able to encode a sequence of vectors to one vector (many-to-one), and then another GRU able to decode a vector to a sequence of vector (one-to-many). The size of the vectors wouldn’t be changed. I would like to have an opinion about what I implemented.</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">class GRU(nn.Module):
    def __init__(self, opt):
        super(GRU, self).__init__()

        self.input_size = 512*3
        self.length_sequence = 30
        self.hidden_size = self.input_size
        self.num_layers = 1

        self.GRU_enc = nn.GRU(input_size=self.input_size, hidden_size=self.hidden_size, num_layers=self.num_layers, batch_first=True)

        self.GRU_dec = nn.GRU(input_size=self.input_size, hidden_size=self.hidden_size, num_layers=self.num_layers, batch_first=True)
    
    # many-to-one
    def enc(self, x):
        # x is of shape BxLx512*3

        h0 = torch.zeros(self.num_layers, x.shape[0], self.hidden_size).cuda()

        out, _ = self.GRU_enc(x, h0)

        out_last = out[:, -1, :]

        return out_last

    # one-to-many
    def dec(self, x):
        x = x[:, None, ...]

        h = torch.zeros(self.num_layers, x.shape[0], self.hidden_size).cuda()

        outputs = []

        for i in range(self.length_sequence):
            out, h = self.GRU_dec(x, h)
            outputs.append(out)

        output = torch.cat(outputs, dim=1)

        output = output.view(-1, self.length_sequence, self.input_size)

        return output

    def forward(self, x):
        one = self.enc(x)

        many = self.dec(one)

        return many
</code></pre>
<p>I am not sure whether this is the good way to do a one-to-many GRU. Could I have some opinions about this?</p>
<p>Also, do you think I need to add some Linear layers after the encoding / decoding, or some activation function?</p>
<p>Finally, the number of parameters is quite big (28M with one layer, 55M with two layers). How many samples would I need in my dataset in order to be able to learn something?</p>
<p>Thanks for reading!</p>
Build error ' in build_caffe2'

xingchen0120: <p>Hi all! I am facing this error with my build from source, anyone with any clues? My command is: python setup.py install</p>
<p>[ 62%] Linking C static library …/…/lib/libnnpack.a<br>
[ 62%] Built target nnpack<br>
[ 62%] Built target generate-torch-sources<br>
[ 62%] Built target dnnl_cpu_x64<br>
[ 62%] Built target dnnl_cpu<br>
[ 62%] Built target fbgemm_avx2<br>
gmake: *** [all] error 2<br>
Traceback (most recent call last):<br>
File “setup.py”, line 818, in <br>
build_deps()<br>
File “setup.py”, line 320, in build_deps<br>
cmake=cmake)<br>
File “/home/username/work_libtorch/pytorch/tools/build_pytorch_libs.py”, line 58, in build_caffe2<br>
cmake.build(my_env)<br>
File “/home/username/work_libtorch/pytorch/tools/setup_helpers/cmake.py”, line 345, in build<br>
self.run(build_args, my_env)<br>
File “/home/username/work_libtorch/pytorch/tools/setup_helpers/cmake.py”, line 140, in run<br>
check_call(command, cwd=self.build_dir, env=env)<br>
File “/home/username/miniconda3/envs/libtorch/lib/python3.6/subprocess.py”, line 311, in check_call<br>
raise CalledProcessError(retcode, cmd)<br>
subprocess.CalledProcessError: Command ‘[‘cmake’, ‘–build’, ‘.’, ‘–target’, ‘install’, ‘–config’, ‘Release’, ‘–’, ‘-j’, ‘104’]’ returned non-zero exit status 2.</p>
Multiply a tensor to a list of list of tensors of different sizes efficiently

soulless: <p>I am looking for a way to perform operations on the following two structures efficiently:</p>
<p>I have a tensor of size (32, 40, 7). I also have a list of list of tensors with the outer list of length 32 and each inner list of length 40. Within the inner list, it has a tensor of (n, 7) where n can be different for each inner list.</p>
<p>For each (1, 1,  7) vector in the (32, 40, 7), I would like to perform certain operations with its corresponding (n, 7) tensor (e.g. let’s say dot product to each row).</p>
<p>What would be an efficient way to do that? Right now I am simply looping over the tensor which is highly inefficient. Sorry if I miss any obvious solution to this and thank you very much for your help in advance</p>

Sentient07: <p>Hi. I believe you’re looking for <code>torch.bmm</code> or the batch matrix multiplication. An example is,</p>
<pre><code class="lang-auto">In [1]: import torch

In [2]: t1 = torch.rand((32, 40, 7))

In [3]: t2 = torch.rand((32, 80, 7))

In [4]: bm = torch.bmm(t1, t2.transpose(2,1))

In [5]: bm.shape
Out[5]: torch.Size([32, 40, 80])

In [6]: 
</code></pre>
<p>Hope that helps</p>
Using PyTorch for professional usage

jeanbon: <p>Hi,</p>
<p>I’m currently investigating the different options for a professionnal usage of IA libs. So, concerning PyTorch I have the following questions:</p>
<p>1- What is the strategy for the numbering of the versions? Which level of compatibility is guaranteed for a same major version?</p>
<p>2- What is your definition of Long-Term Support (how many years of support is guaranteed)?</p>
<p>3- Is it mandatory to subcribe to the “Pytorch Enterprise Support Program” to use an LTS version?</p>
<p>Thanks for your answers.</p>

ptrblck: <p>CC <a class="mention" href="/u/jspisak">@jspisak</a> and <a class="mention" href="/u/suraj.pt">@suraj.pt</a> as they might know the details <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Data augmentation in Dataloader batch

han-yeol: <p>I want to implement dataloader below<br>
how can I do this?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8a076cb188f176994d6aec5774494b03c219380.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f8a076cb188f176994d6aec5774494b03c219380" title="research note-6"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8a076cb188f176994d6aec5774494b03c219380_2_386x500.jpeg" alt="research note-6" data-base62-sha1="ztscJ6VtdWWHFm5iZJ84zCLQZWM" width="386" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8a076cb188f176994d6aec5774494b03c219380_2_386x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8a076cb188f176994d6aec5774494b03c219380_2_579x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f8a076cb188f176994d6aec5774494b03c219380_2_772x1000.jpeg 2x" data-dominant-color="F5F3E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">research note-6</span><span class="informations">2244×2904 315 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you explain the use case a bit more, please?<br>
Would you like to randomly augment the sample/batch and concatenate these transformed versions to the current sample/batch?<br>
If so, I assume the batch size would be variable, since these transformations are applied randomly.<br>
In that case, I think the easiest way would be to apply the transformations inside the <code>DataLoader</code> loop and <code>torch.cat</code> these transformed sample with the original batch to the new input.</p>

han-yeol: <p>Oh. such a good solution,<br>
I can transform both image and label(Ground Truth)<br>
and stack them and feed for network</p>
<p>Thank you so much!</p>
torch.nn.DataParallel(): Why GPUs use different amount of VRAM?

petasis: <p>Hi all,</p>
<p>I am trying to fine-tune a GPT2 model, following the tutorial here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/15wa925dj7jvdvrz8_z3vU7btqAFQLVlG">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210629-060140-RC00_382053243" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/15wa925dj7jvdvrz8_z3vU7btqAFQLVlG" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/15wa925dj7jvdvrz8_z3vU7btqAFQLVlG" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>At some point, the code uses <code>torch.nn.DataParallel()</code>, which uses both my GPUs. However, the usage of GPUs is imbalanced, for example the training occupies 22GB on the first GPU, while only 9GB in the second card.<br>
Why is this happening? Why the first GPU uses 13GB VRAM more? What is stored there, that is not stored in the second GPU?</p>

ptrblck: <p>This shortcoming is expected in <code>nn.DataParallel</code> and is one reason I generally recommend to use <code>DistributedDataParallel</code> as well as for performance reason. The underlying mechanics and reason of it are explained in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a>.</p>
Tensor product 1d, 3d

namgyu: <p>batch_size = 3</p>
<p>x = [1,2,3]<br>
y = [[[1, 1, 1, 1],<br>
[2, 2, 2, 2]],<br>
[[1, 1, 1, 1],<br>
[2, 2, 2, 2]],<br>
[[1, 1, 1, 1],<br>
[2, 2, 2, 2]]]</p>
<p>x*y = [[[1, 1, 1, 1],<br>
[2, 2, 2, 2]],<br>
[[2, 2, 2, 2],<br>
[4, 4, 4, 4]],<br>
[[3, 3, 3, 3],<br>
[6, 6, 6, 6]]]</p>
<p>Pytorch is too difficult…<br>
What do you do with this method of multiplication?</p>
<p>Help me, please.</p>

ptrblck: <p>Could you explain the question a bit more?<br>
Are you wondering when or why broadcasting is used?<br>
If so, I think the <a href="https://numpy.org/doc/stable/user/basics.broadcasting.html">numpy broadcasting docs</a> might be helpful to see a more detailed explanation and some examples.</p>
How to train two identical models in pytorch?

dgjung: <p>Hi,</p>
<p>I have a question about training in pytorch.<br>
There is a pseudo code that I wanna ask.</p>
<pre><code class="lang-auto">model1 = Net()
model2 = copy.deepcopy(model1)

optimizer1 = torch.optim(model1.parameters(), lr)
optimizer2 = torch.optim(model2.parameters(), lr)

for i in range(dataset):

    x = random(dataset[i])
    y = dataset[i]

    update_step1(model1, optimizer1, x, y)
    update_step2(model2, optimizer2, x, y)
</code></pre>
<p>I thought this would be updated separately in each function(update_step1, 2), but the two values were the same.</p>
<p>How should I do to train model at the same time?<br>
Note that the reason for doing this at the same time is to give the same random X value both models.</p>
<p>Thank you.</p>

ptrblck: <p>Your code should work and both models as well as optimizers would be independent.<br>
However, if you don’t have any randomness in the models and are also feeding the same data as well as target into them, the training might be equal (unless non-deterministic operations are used internally with an undefined order of ops).<br>
Here is a small example:</p>
<pre><code class="lang-python">def check_models(model1, model2):
    for (name1, param1), (name2, param2) in zip(model1.named_parameters(),
                                                model2.named_parameters()):
        print('name {}, is equal {}'.format(
            name1, torch.allclose(param1, param2)))

def update_step(model, optimizer, x, y):
    optimizer.zero_grad()
    output = model(x)
    loss = F.mse_loss(output, y)
    loss.backward()
    optimizer.step()


model1 = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    #nn.Dropout(),
    nn.Linear(10, 10)
)
model2 = copy.deepcopy(model1)

check_models(model1, model2)

lr = 1e-3
optimizer1 = torch.optim.SGD(model1.parameters(), lr)
optimizer2 = torch.optim.SGD(model2.parameters(), lr)


for i in range(2):
    x = torch.randn(10, 10)
    y = torch.randn(10, 10)

    update_step(model1, optimizer1, x, y)
    update_step(model2, optimizer2, x, y)
    check_models(model1, model2)
</code></pre>
<p>As you can see, the updated parameters will be the same unless I enable the dropout layer in the model.</p>
怎么创建一个跟随模型变更设备的张量？How to creat a Tensor with the same device as the model？

ChenzhouWeiYu: <p>怎么在nn.Module里面，创建一个张量，他的设备跟随这一个模型改动？</p>
<p>比如说，我需要一个卷积核K，他参数是多少我给定的，不需要参与训练。但是我forward的时候需要他进行几次卷积。除了这一个K以外还有其他的卷积层。</p>
<p>如果model().cuda()，这个张量也是在GPU上，如果.cpu()，这个张量也在CPU上。</p>
<p>I wanna a Tensor created in nn.Module, it has the same device as the Module.</p>
<p>If model().cuda(), the Tensor is also moved to GPU, and if model.cpu(), it is moved to CPU</p>

ptrblck: <p>From Google translate:<br>
Title:</p>
<blockquote>
<p>How to create a tensor that follows the model change device?</p>
</blockquote>
<p>Post:</p>
<blockquote>
<p>How to create a tensor in nn.Module, and his equipment changes with this model?<br>
For example, I need a convolution kernel K, the parameters of which are given by me, and I don’t need to participate in training. But when I forward, I need him to perform several convolutions. In addition to this K, there are other convolutional layers.<br>
If model().cuda(), this tensor is also on the GPU, if .cpu(), this tensor is also on the CPU.</p>
</blockquote>
<p>Please use an online translation service before posting the question here, so that all users could help you. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

111541: <p>He just posted a dual-language question.</p>

ptrblck: <p>I don’t think that’s entirely true, as my posted translation contains more details than the initial post, which might be helpful to answer the question. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p><a class="mention" href="/u/chenzhouweiyu">@ChenzhouWeiYu</a> to register parameters (such as conv weights) in the <code>forward</code> method of your model, you can either use <code>self.register_parameter</code> or assign the <code>nn.Parameter</code> to an internal attribute via <code>self.param = nn.Parameter(...)</code>.</p>
Dice Coefficient from Integer Labels

conv8d: <p>For a segmentation task, I have the predictions and the target as tensors of  shape <code>[batch_size, h, w]</code> consisting of values in <code>[0, num_classes - 1]</code>.<br>
How can I compute the dice coefficient from these?<br>
One way that I can think of is to one-hot encode the tensors and then compute the dice. Is there a better way?</p>
Performance degrades with DataParallel

barakb: <p>Hi guys, when I’m training with my model with one gpu I’m getting x% performance (accuracy).<br>
Now I’m trying to make my model a bit bigger, I don’t have enough memory so I used 2 GPUs with dataparallel of course.<br>
My result wasn’t good, so I took again my small model and ran it on 2 GPUs with dataparallel, the same model that got on 1 GPU x% is now getting (x-3)%, and tried to run multiple times.</p>
<p>Does anyone have an idea?<br>
Notice I’m talking about DataParallel and not DistributedDataParallel</p>
<p>Thanks!</p>

ptrblck: <p>Did you double the batch size using <code>nn.DataParallel</code> or are you using a batch size &lt;2x?<br>
The batch will be chunked in dim0 and each chunk will be send to a specific GPU.<br>
If you don’t increase the batch size, each GPU will use a smaller batch, which might decrease the performance e.g. due to noisy running estimates in batch norm layers.</p>

dakeoffer: <p>I got 4GPUs so I let the (batch size)<em>4 together with the LR</em>3.69, but the accuracy of the model seems to get decreased. Must the LR multi the same times? And what if I used batch size with 3 times of the original but train on 4 GPUs, will this influence the performance?</p>
<p>Final question, what would be the best data-parallel solution regarding the model’s maintaining the same performance or even better compared with training on one GPU?<br>
nn.DataParallel() vs DistributedDataParallel vs PyTorch Lightning Horovod</p>

ptrblck: <p>I usually recommend to use <code>DistributedDataParallel</code>, as it’s the faster than <code>nn.DataParallel</code>, and also allows you to use <code>SyncBatchNorm</code> layers to share the running stats. I’m not familiar with Horovod, so cannot comment on it.</p>
Loss function for image classification considering similarity

Hyeonuk_Woo: <p>Hi, I’m not sure that it is appropriate to ask this kind of question here.</p>
<p>If my question is inappropriate for this forum and you know of an appropriate forum for my question, please recommend!</p>
<p>I have a general question about loss function for image classification.</p>
<p>Why my loss function isn’t work well?</p>
<p>I have a CNN model that classify image, and my model was trained well with cross entropy loss. ( answer probability distribution <code>p</code> is given one-hot vector  for  <code>loss= sum of pi*log(qi)</code>.</p>
<p>I want to give  a partial score even if model predict incorrectly. In other words, I want to consider  similarity between categories for loss calculation.</p>
<p>For example, I want to classify a given image into one of the 4 categories:  <code>c1, c2, c3, c4</code>. I have a score table like below.</p>
<pre><code class="lang-auto">             c1         c2           c3             c4
 c1         2.0        1.0          0.0           -1.0
 c2         1.0        2.0         -1.0            0.0 
 c3         0.0       -1.0          0.0            1.0
 c4        -1.0        0.0          1.0            2.0
</code></pre>
<p>The output of my model is probability vector that belongs to each category, noramlized by softmax function. <code> output : [q1,q2,q3,q4]  # q1+q2+q3+q4=1.0</code></p>
<p>I hope my model return to maximize expectation score, which is calculated with similarity matrix.</p>
<p>When given image is belong to c1, i.e ground truth probability distribution <code>p : [1.0, 0.0, 0.0, 0.0]</code>,<br>
I hope to calculate loss for this problem as follows,  <code>loss = ( S_11*q1 + S_12*q2 + S_13*q3 + S_14*q4 ) * ( -1)</code> , where <code>S_ij </code>is element of similarity matrix described above.<br>
However, training is really slow. Loss value decreases very slowly. And other observable measure related to performance of model said that training is really slow.</p>
<p>I think that my loss function has same objective with cross entropy loss function : when the image of <code>ci</code> is given, maximize <code>qi</code>.<br>
Why my loss function doesn’t work well??</p>
<p>I already consider loss function form of cross entropy considering similarity, <code>sum of pi*log(qi)</code>, where <code>[p1,p2,p3,p4]</code> is not one hot vector anymore. But, it is difficult to obtain a probability distribution (p) from the similarity matrix. And this is not what I want.</p>
<p>Actually, I’m working on physical chemistry and bioinformatics, not classifying image.</p>
<p>Thank you for reading.</p>
Argmax for semantic segmentation

drydenwiebe: <p>Hello,</p>
<p>I have a tensor representing multi class semantic segmentation that is the output of my network. It is of the shape [B, C, H, W]</p>
<p>Where B is the batch size, C is the number of classes, H is the image height and W is the image width. I want to get a one hot vector for each class for each pixel (for each image in the batch).</p>
<p>How can I accomplish this with torch.argmax?</p>
<p>Thank you so much!</p>

Ramchandra_Cheke: <p>In case your current target shape is <code>[batch_size, c, h, w]</code>, try to convert it using:</p>
<pre><code class="lang-auto">target = torch.argmax(target, 1)
</code></pre>
<p>Please find link: <a href="https://discuss.pytorch.org/t/semantic-segmentation-loss-function-shape-of-prediction-and-target/44800" class="inline-onebox">Semantic segmentation loss function / shape of prediction and target</a></p>

drydenwiebe: <p>When I try this the shape becomes [batch_size, h, w], when what I want is [batch_size, c, h, w] but c is one hot encoded. Any ideas there?</p>
<p>Thanks</p>
Mypy error error: Name 'torch.cuda.DoubleTensor' is not defined

jlcv: <p>I am trying to create a python dataclass that supports both cpu and gpu <code>DoubleTensor</code>:</p>
<pre><code class="lang-auto">@dataclass
class SomeDC:
    element: Union[torch.DoubleTensor, torch.cuda.DoubleTensor]
</code></pre>
<p>However, this results in the <code>mypy</code> error <code>error: Name 'torch.cuda.DoubleTensor' is not defined</code>. I tried <code>import torch.cuda</code> but the same error occurs. Anyone know if there is a workaround for this or do I just ignore this <code>mypy</code> error?</p>
<p>Note, the code runs fine this way, the only problem is the <code>mypy</code> error.</p>
How to load load_state_dict for ResNet model

Ramchandra_Cheke: <p>Hi,<br>
I have two models. ModelA and ModelB both are resnet50 model. I have pre-trained weights from modelA and they looks like.</p>
<pre><code class="lang-python">|module.encoder.conv1.weight | torch.Size([64, 3, 7, 7])|
|module.encoder.bn1.weight | torch.Size([64])|
|module.encoder.bn1.bias | torch.Size([64])|
|module.encoder.bn1.running_mean | torch.Size([64])|
|module.encoder.bn1.running_var | torch.Size([64])|
</code></pre>
<p>However, when I define modelB it has model weight in</p>
<pre><code class="lang-python">encoder.conv1.weight   torch.Size([64, 3, 7, 7])
encoder.bn1.weight   torch.Size([64])
encoder.bn1.bias   torch.Size([64])
encoder.bn1.running_mean   torch.Size([64])
encoder.bn1.running_var    torch.Size([64]). 

RuntimeError: Error(s) in loading state_dict for ResNetEncoder:
	Unexpected key(s) in state_dict: "module.encoder.conv1.weight", "module.encoder.bn1.weight", "module.encoder.bn1.bias", "module.encoder.bn1.running_mean", "module.encoder.bn1.running_var", "module.encoder.bn1.num_batches_tracked", 
</code></pre>
<p>I am trying to load modelA weights into modelB. However, I am getting error. Please help me.</p>

Ramchandra_Cheke: <pre data-code-wrap="python"><code class="lang-python"># create new OrderedDict that does not contain `module.`
from collections import OrderedDict

# original saved file with DataParallel
ckpt = torch.load("pretraied_model.pth")
state_dict = ckpt['model']    # incase there are extra parameters in the model

new_state_dict = OrderedDict()

for k, v in state_dict.items():
    name =  k.replace("module.encoder.","encoder.")
    new_state_dict[name] = v

# load params
model.encoder.load_state_dict(new_state_dict,strict=False)
</code></pre>
<p>Thanks to post</p><aside class="quote" data-post="4" data-topic="1686">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/48/58_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/4">[solved] KeyError: 'unexpected key "module.encoder.embedding.weight" in state_dict'</a> 
  </div>
  <blockquote>
    I was thinking about something like the following: 
# original saved file with DataParallel
state_dict = torch.load('myfile.pth.tar')
# create new OrderedDict that does not contain `module.`
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove `module.`
    new_state_dict[name] = v
# load params
model.load_state_dict(new_state_dict)
  </blockquote>
</aside>
Loss computation in multi branch neural network in PyTorch

Anshul_Pundhir: <p>Hi,<br>
I am new to PyTorch so have difficulty in understanding Gradient computation on multi-branch neural network. Someone, please explain how the loss will be backpropagated from output to back? Consider any arbitrary case that would make the explanation easier.</p>
<p>Thanks</p>

soulitzer: <p>If by multi-branch network you mean something like a inception module. Consider the following example:</p>
<pre><code class="lang-python">import torch
x = torch.ones(1, 2, requires_grad=True)  # 1
y = torch.cat([x, x * 2])  # 2
y.sum().backward()  # 3
x.grad  # torch.tensor([3, 3])  # 4
</code></pre>
<p>(4) By convention, the gradient wrt to the loss itself (a scalar) is 1 (and also a scalar).<br>
(3) When doing something like SumBackward, this value is expanded to the size of y. Because for each term y_i in y will have dloss/dy_i = dloss/dloss.<br>
(2) Cat takes two inputs. For now lets call them a and b. Since the gradients wrt cat’s output has the same shape as cat’s output, and the gradients wrt a, b are the same as a and b, the way CatBackward is defined here is also very natural.<br>
(1) Finally since x is used twice in (2), by the chain rule, the gradient wrt x should be summed. To see what gives rise to the chain rule, for this step, just consider a, b functions of x, and then define y = cat(a(x), b(x)).</p>
Cuda error - cnn

Geri: <p>I have used this network architecture for MNIST and CIFAR10 and runs successfully but when I try to train the CIFAR100 data i get the following error. Of course I changed various settings in the network class for MNIST and CIFAR10 but cannot understand what is wrong with CIFAR100.</p>
<p>Code and Error message are attached.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1abe9cb57bca57ad414b2f3b0b702ed3ab158fae.png" data-download-href="https://discuss.pytorch.org/uploads/default/1abe9cb57bca57ad414b2f3b0b702ed3ab158fae" title="error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1abe9cb57bca57ad414b2f3b0b702ed3ab158fae.png" alt="error" data-base62-sha1="3OANwQqox19YWKTUFEvAaMywuYe" width="583" height="500" data-dominant-color="1E2B31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">798×684 33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Geri: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/291c9fbaedbda4e1b9b017bf1e79e7861143bb3d.png" data-download-href="https://discuss.pytorch.org/uploads/default/291c9fbaedbda4e1b9b017bf1e79e7861143bb3d" title="code"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/291c9fbaedbda4e1b9b017bf1e79e7861143bb3d.png" alt="code" data-base62-sha1="5RGTYoTTGGRXYK3JpsZTfU4RbpH" width="467" height="500" data-dominant-color="212A34"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">code</span><span class="informations">682×729 24.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

RaLo4: <blockquote>
<p>You can debug these CUDA Runtime Errors best if you run your code on CPU. This will give you a more useful traceback error.</p>
<p>But Most of the time CUDA Runtime Errors can be the cause of some index mismatching so like you tried to train a network with 10 output nodes on a dataset with 100 labels.</p>
</blockquote>
<p>(Taken from <a href="https://stackoverflow.com/questions/68166721/cuda-error-device-side-assert-triggered-on-colab" rel="noopener nofollow ugc">here</a>.)</p>
<p>Which is exactly what your screenshot is showing.<br>
Keep in mind that CIFAR-10 and MNIST have 10 classes but CIFAR-100 has 100 classes.<br>
Did you maybe forget to change the</p>
<pre><code class="lang-auto">self.fc2 = nn.Linear(64, 10)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">self.fc2 = nn.Linear(64, 100)
</code></pre>
<p>?</p>

Kapil_Rana: <p>Yes agreed with <a class="mention" href="/u/ralo4">@RaLo4</a><br>
you have forgotten to change the size at the last layer from 10 to 100, that’s why it’s giving an error during loss computation.</p>
Where is libtorch 1.7.0 cuda 10.1?

tejal567: <p>I can’t find libtorch 1.7.0 cuda 10.1 version anywhere. Is it no longer officially supported and therefore removed? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>I’m not sure, if <code>libtorch==1.7.0</code> was built with CUDA10.1 (I think it was), and assume you should be able to change the download link to the desired config, e.g. from <a href="https://pytorch.org/get-started/locally/">here</a>:</p>
<pre><code class="lang-python"># change 
https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.8.1.zip
# to cu101 and the corresponding libtorch version:
https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.7.0.zip
</code></pre>
<p>However, I also get an <code>access denied</code> error while trying to download the CUDA10.2 <code>libtorch==1.8.1</code> build, so I guess something might create issues with the download links.<br>
CC <a class="mention" href="/u/seemethere">@seemethere</a></p>

tejal567: <p>Yes, this link is also not working (access denied error):<br>
<a href="https://download.pytorch.org/libtorch/cu101/libtorch-win-shared-with-deps-1.7.0.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu101/libtorch-win-shared-with-deps-1.7.0.zip</a></p>

jdsalaro: <p>I found it at <a href="https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.7.0%2Bcu101.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.7.0%2Bcu101.zip</a></p>
How to implement the Reinforcement Learning Agent to act at different time steps using Pytorch

uBoscuBo: <p>I was wondering how would you change how often the agent interacts with the environment. Looking at previous custom environments and gym environment it looks like the agent interacts with the environment per time-step. For my purposes I want my agent to interact with the environment once per episode.</p>
<p>It looks like <code>TensorFlow</code> has a build-in function to accommodate how frequently the agent interacts with the environment. Please see the following code as an example.</p>
<blockquote>
<pre><code class="lang-auto">env = suite_gym.load('CartPole-v0')
tf_env = tf_py_environment.TFPyEnvironment(env)

time_step = tf_env.reset()
rewards = []
steps = []
num_episodes = 5

for _ in range(num_episodes):
  episode_reward = 0
  episode_steps = 0
  while not time_step.is_last():
    action = tf.random.uniform([1], 0, 2, dtype=tf.int32)
    time_step = tf_env.step(action)
    episode_steps += 1
    episode_reward += time_step.reward.numpy()
  rewards.append(episode_reward)
  steps.append(episode_steps)
  time_step = tf_env.reset()

num_steps = np.sum(steps)
avg_length = np.mean(steps)
avg_reward = np.mean(rewards)

print('num_episodes:', num_episodes, 'num_steps:', num_steps)
print('avg_length', avg_length, 'avg_reward:', avg_reward)
</code></pre>
</blockquote>
<p>So I was wondering if PyTorch has a similar functionality? If not, I guess I would need to redo my custom environment to accommodate different time steps?</p>
Training metrics on GPU: good habits

AleTL: <p>Hello!</p>
<p>This is a curious post more than a asking one. I just want to know how do you record the metrics of your training when you’re using a GPU. I’m use to print the metrics of each epoch is a separated file. I open the file at each epoch and print train loss, test loss and so on. I know this is highly unefficient, because I’m opening a file and printing at every epoch, but with small datasets and models I really don’t care, the difference is not that high.</p>
<p>But when working with GPU and huge models and datasets, there’s a huge difference because the computations are so fast the the performance bottleneck is precisely on the metrics printing and so on. I’m just curious to know how do you solve this problem, do you just plot at the end the curves? Do you just write everything at the end?</p>
<p>Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How does NoGradGuard works in Cpp?

dfalbel: <p>This is more a conceptual question… Consider:</p>
<pre><code class="lang-auto">NoGradGuard guard;
b -= lr * b.grad();
</code></pre>
<p>I know that <code>NoGradGuard</code> will disable action recording for the next gradient calculation.<br>
But how does it really works?</p>

albanD: <p>Hi,</p>
<p>It actually sets a global flag that is checked by the backend whenever an op is done on a variable.<br>
The guard itself saved the current status and set it to false in the constructor. And restore the saved status in it’s destructor. That way it is similar to a <code>with torch.no_grad():</code> block in python.</p>
Warning: Attempted to insert a call_module Node with no underlying reference in the owning GraphModule!

nscotto: <p>Hi,<br>
I am getting this warning when inserting a <code>call_module</code> node even though I called <code>GraphModule.add_submodule</code> (or <code>.add_module</code>, or even both):</p>
<pre><code class="lang-auto">/(...)/anaconda3/envs/test/lib/python3.7/site-packages/torch/fx/graph.py:606: UserWarning: Attempted to insert a call_module Node with no underlying reference in the owning GraphModule! Call GraphModule.add_submodule to add the necessary submodule
  warnings.warn("Attempted to insert a call_module Node with "
</code></pre>
<p>Here is the minimal example that produces the warning:</p>
<pre><code class="lang-auto">import torch
import torch.fx as fx
import torch.nn as nn


class MyModule(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x


class MyOtherModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(24, 32, 3)

    def forward(self, x):
        x = nn.functional.relu(self.conv(x))


def insert_after(mod: fx.GraphModule, from_idx: int, insertme: fx.GraphModule):
    mod.add_submodule('inserted', insertme)
    cutoff_node: fx.Node = list(mod.graph.nodes)[from_idx]
    next_node: fx.Node = list(mod.graph.nodes)[from_idx+1]

    with mod.graph.inserting_after(cutoff_node):
        new_node = model.graph.call_module('inserted', (cutoff_node,), {})

    next_node.replace_input_with(cutoff_node, new_node)

    mod.delete_all_unused_submodules()
    mod.graph.eliminate_dead_code()
    mod.recompile()
    mod.graph.lint()

    return mod


mod = fx.symbolic_trace(MyModule())
other = fx.symbolic_trace(MyOtherModule())
new_mod = insert_after(mod, 0, other)
</code></pre>
<p>As you can see I am inserting a <code>call_module</code> node on a module I just added with <code>GraphModule.add_submodule</code>, but the warning seems not to see it.<br>
Have I missed something?</p>
How can I obtain the gradient of a gradient?

afps: <p>Say I have a function f_w(x) with input x and parameters w.</p>
<p>For optimizing it I obtain the gradients of a custom loss function g_q(y) parametrized by q with respect to w. Let’s call it w.grad .</p>
<p>Next I want to obtain the gradients of w.grad with respect to the parameters q of the loss function.</p>
<p>How can I do it? See some code below.</p>
<pre><code class="lang-auto">class MyFun:
    def __init__(self, d_input, d_output, constant):
        self.W = constant * torch.ones((d_input, d_output), requires_grad=True)
        
    def fp(self, x):
        self.W.retain_grad()
        return x @ self.W     

f = MyFun(1, 1, torch.tensor([2.]))
g = MyFun(1, 1, torch.tensor([5.]))

x = torch.tensor([8.])

out = f.fp(x)
loss = g.fp(out)

loss.backward(retain_graph=True)

print(x.grad, f.W.grad, g.W.grad)

gradient = f.W.grad

gradient.backward()
</code></pre>
<p>The print statement returns “None tensor([[40.]]) tensor([[16.]])” as it should.</p>
<p>gradient.backward() gives me the error “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”.</p>

SebGruber1996: <p>I don’t know if this question is still relevant, but I included some recursive gradient examples in my post  <a href="https://discuss.pytorch.org/t/error-by-recursively-calling-jacobian-in-a-for-loop/125924">error-by-recursively-calling-jacobian-in-a-for-loop</a></p>
Error loading caffe2_detectron_ops_gpu.dll or one of its dependencies

r_khanna: <p>Hello, I keep getting this error when I run my file from cmd-</p>
<p><code>OSError: [WinError 193] &lt;no description&gt; Error loading "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll" or one of its dependencies.</code></p>
<p>Below is the command that I run.</p>
<pre><code class="lang-auto">C:\Users\Lenovo\Documents\Data science\stylegan2-ada-pytorch&gt;python psnr.py -d0 dataset2Oscar_all/Input -d1 dataset2Oscar_all/Peihao_result -o psnr_peihao.txt --use_gpu
Traceback (most recent call last):
  File "psnr.py", line 3, in &lt;module&gt;
    import torch
  File "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\__init__.py", line 123, in &lt;module&gt;
    raise err
OSError: [WinError 193] &lt;no description&gt; Error loading "C:\Users\Lenovo\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll" or one of its dependencies.
</code></pre>
Train a network by updating only part of the weights

Bixqu: <p>Let’s say we have a network made of 5 layers:</p>
<pre><code>input = torch.nn.Linear()
layer2 = torch.nn.Linear()
layer3 = ...
layer4 = ...
output = torch.nn.Linear(.., 2)
</code></pre>
<p>I want to train the network with all 5 layers, as usual, for about 10 epochs, then I want to train just the last 3 layers (layer 3, layer 4 and output). How can I tell pytorch to stop updating the weights of input and layer 1? Could someone provide a very simple, but working, example?</p>

smth: <pre><code class="lang-python">for p in input.parameters():
    p.requires_grad=False
for p in layer1.parameters():
    p.requires_grad=False
</code></pre>

Bixqu: <p>Thanks for the reply., should I do this for every layer? For example if the above architecture is part of a model class.</p>
<pre><code>for epoch in number_of_epochs:
    if epoch &lt; 10 : 
         for p in model.input.parameters():
             p.requires_grad = True
         for q in model.layer1.parameters():
             q.requires_grad = True
   else: 
         for p in model.input.parameters():
             p.requires_grad = False
         for q in model.layer1.parameters():
             q.requires_grad = False
</code></pre>
<p>would this thing work?</p>

backpropper: <p>You just need to make <code>requires_grad=False</code> for the both the input and the layer1 layers once at the 10th epoch.</p>
<pre>
for p in model.parameters():
    p.requires_grad = True
for epoch in number_of_epochs:
    if epoch == 10 : 
         for p in model.input.parameters():
             p.requires_grad = False
         for q in model.layer1.parameters():
             q.requires_grad = False
</pre>
<p>This should work.</p>

Bixqu: <p>The above method seems to have no effect whatsoever on my network. Are you sure this is the way to prevent a layer’s weights from updating?</p>

smth: <p><a class="mention" href="/u/bixqu">@bixqu</a> yes if you declare that your weights dont need grad (with requires_grad=False) then no gradient will be computed wrt the weights.</p>
How to find number of features in .pth file

i_m_a_q: <p>Hi, I am new to pytorch.<br>
I have <strong>123.pth</strong> features file.<br>
I have determined its size and it output “torch.Size([1, 2048, 25, 19])”<br>
I want to know how many features(region/grid) this file has. Do I need to multiply <strong>25*19</strong> and answer would be the number of features?</p>

ptrblck: <p>It depends on the tensor and how you would define “features”.<br>
Based on the tensor shape it could be an intermediate activation in the shape <code>[batch_size, channels, height, width]</code>.<br>
If you are flattening it and passing to e.g. a linear layers, the usual approach would be to use: <code>x = x.view(x.size(0), -1)</code>, which would result in <code>2048*25*19=972800</code> “features”.</p>

i_m_a_q: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your response. The paper that these features belongs to, says the number of features are from <strong>(100-608)</strong> depending upon the size of image. It means that they are counting <strong>height*width</strong>? I’d like to minimize the number of features to <strong>36</strong> (i-e using clustering). That’s why I am asking if I could make these features to 36 using h*w?</p>
How to Extend Torch via it's C++ Frontend

bosskwei: <p>Hi guys,<br>
Is there any method to extend LibTorch using C++?</p>
<p>For PyTorch, there is a <code>__torch_function__</code>, then I can do something like this:</p>
<pre><code class="lang-auto">def CustomTensor:
    def __torch_function__(self, func, types, args=(), kwargs=None):
        if func is xxx: ...

x = CustomTensor()
linear = nn.Linear(1024, 32)
y = linear(x)
</code></pre>
<p>But seems there’s no way to create a <code>CustomTensor</code> by overiding or extending <code>torch.Tensor</code> in C++ frontend</p>
<pre><code class="lang-auto">// torch modules are defined in pimpl.h
namespace torch::nn {
TORCH_MODULE(Linear);
} // namespace torch::nn
</code></pre>
<p>Any suggestion? Thanks a lot</p>
Long(real long) training time on bert finde-tuning

ygl: <p>Hi, I am training a dataset with 700,000 samples. Basically they are just text with a binary label. What I am doing is</p>
<p>model = transformer.BertModel.from_pretrained(“uncased_bert”)<br>
outputs = model(ids=ids, mask=mask, token_type_ids=token_type_ids)<br>
loss = loss_fun(outputs, targets).</p>
<p>So for each piece of text,  I used encode_plus(text) to get ids, mask, etc .<br>
I am using a batch size of 32 and learning rate is lr=3e-5(smaller is better by some research) find this is taking really really long. Like a few days if I calculate correctly. I also found that using 2 gpus, each of them will be cost like 6G memory and using 1 gpu, it will take 11G. Is there anything that I can do to speed this up? Thanks!</p>
AttributeError: 'Tensor' object has no attribute 'RandomCrop'

desert_ranger: <p>I am trying to perform random cropping on an image. Here is my code. I tried following the examples given over <a href="https://pytorch.org/vision/stable/transforms.html" rel="noopener nofollow ugc">here</a> but got stuck.</p>
<pre><code class="lang-auto">import numpy as np
from sklearn.datasets import load_sample_images
import matplotlib.pyplot as plt
import torch
import torch.nn as nn

def random_crop(imgs, out=84):


    imgs = torch.tensor(84)
    transforms = torch.nn.Sequential(
        imgs.RandomCrop(84),
    )
    imgs = transforms.numpy()
    return imgs
    
dataset = load_sample_images()
first_img_data = dataset.images[0]
first_img_data  = first_img_data.reshape(-1, 427, 640)
first_img_data = first_img_data[1, :, :]
foo = random_crop(first_img_data.reshape(-1,1,427, 640), out=84)
</code></pre>

desert_ranger: <p>Figured it out -</p>
<pre><code class="lang-auto">import numpy as np
from sklearn.datasets import load_sample_images
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torchvision.transforms as transforms

def random_crop(imgs, out=84):


    imgs = torch.tensor(imgs)
    change = torch.nn.Sequential(
        transforms.RandomCrop(84)
    )
    imgs = change(imgs).numpy()
    return imgs

dataset = load_sample_images()
first_img_data = dataset.images[0]
first_img_data  = first_img_data.reshape(-1, 427, 640)
first_img_data = first_img_data[1, :, :]
foo = random_crop(first_img_data.reshape(-1,1,427, 640), out=84)
plt.imshow(np.squeeze(foo))
plt.show()
</code></pre>