akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why is ReLU6 capped at 6?

YuhskeHujisaki: <p>ReLU6 is capped at 6, but why 6, and why not 5 or 4 or 3? And is there any paper comparing them?</p>

ptrblck: <p>I believe Alex Krizhevsky used it in <a href="http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf">Convolutional Deep Belief Networks on CIFAR-10</a> and describes it at:</p>
<blockquote>
<p>Our ReLU units dieer from those of [8] in two respects. First, we cap the units at 6, so our ReLU activation function is<br>
y = min(max(x, 0), 6).<br>
In our tests, this encourages the model to learn sparse features earlier. In the formulation of [8], this is<br>
equivalent to imagining that each ReLU unit consists of only 6 replicated bias-shifted Bernoulli units, rather than an innute amount. We will refer to ReLU units capped at n as ReLU-n units.</p>
</blockquote>
Torch.equal does not return bool

tsugumi-sys: <pre><code class="lang-python">from unittest import mock
print(torch.equal(mock._Call(), torch.zeros((2, 2))))
</code></pre>
<p>and this output is like this.</p>
<pre><code class="lang-auto">call.__torch_function__(&lt;built-in method equal of type object at 0x10cc80770&gt;, (&lt;class 'unittest.mock._Call'&gt;,), (call(), tensor([[0., 0.],
        [0., 0.]])))
</code></pre>
<p>Is <code>torch.equal</code> correctly working in this case? I found this by my typo in unit testing like this.</p>
<pre><code class="lang-python">import unittest
from unittest.mock import MagicMock, patch
import torch


def calc_mean(input_tensor):
    means = torch.mean(input_tensor)
    return means


class Test(unittest.TestCase):
    @patch("torch.mean")
    def test_sample(self, mock_torch_mean: MagicMock):
        mock_torch_mean.return_value = 1.0
        input_tensor = torch.zeros((2, 2))
        result = calc_mean(input_tensor=input_tensor)
        # My typo is here and mock_torch_mean.call_args.args[0] is correct.
        # However, self.assertTrue passes in this case.
        self.assertTrue(torch.equal(mock_torch_mean.call_args, input_tensor))


if __name__ == "__main__":
    unittest.main()

</code></pre>
Help to identify this layers in EfficientNet_b0

Miguel_Campos: <p>I am implementing a paper that says:</p>
<p><code>we extract vectors from layers 7 (level 2), 20 (level 4), and 26 (level 5) in an EfficientNet-B5.</code></p>
<p>I am trying the <code>torchvision.model.efficientnet_b0</code> implementation but, I am not sure what layers are.</p>
<p>Could you help me? I can not catch the exactly feature maps. Thanks</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1cfddf9dfca8046720f265e8cb8864de79dcb6d.png" alt="image" data-base62-sha1="wdCMBS7MyynppzNS7woXBMMZkSN" width="483" height="300"></p>
<p>This is my best try</p>
<pre><code class="lang-auto">import torchvision

torch_model = torchvision.models.efficientnet_b0(False)
return_nodes = {
    'features.1': 'level2',
    'features.3': 'level4',
    'features.4': 'level5',
}
eff_model = create_feature_extractor(torch_model, return_nodes)
o = eff_model(torch.rand(1, 3, 224, 224))
for k in o:
  print(k, o[k].shape)
</code></pre>
<p>Output</p>
<pre><code class="lang-auto">level2 torch.Size([1, 16, 112, 112])
level4 torch.Size([1, 40, 28, 28])
level5 torch.Size([1, 80, 14, 14])
</code></pre>
GPU memory consumption and inference time is higher for first inference

farah: <p>It is noticed that gpu memory consumption and inference time is much higher for first inference than subsequent inferences. I have set batch size = 1. E.g. Performing inference on 9 samples is giving this inference time and memory consumption. For memory consumption I reset peak memory stats at every iteration of dataloader and use max memory allocated after the inference to get the gpu memory consumption for each inference. For inference time, I simply use time library and get the difference of start and end time.</p>
<p>Here is the code snippet:</p>
<pre><code class="lang-auto">
    with torch.no_grad():
        for shape_index, (file_path) in enumerate(tqdm(dataset.file_paths, desc='eval', ncols=0)):

            torch.cuda.empty_cache()
            torch.cuda.reset_peak_memory_stats()


            data = np.loadtxt(file_path).astype(np.float32)
            start_time = time.time()

            # model inference
            inputs = torch.from_numpy(data).float().to(configs.device)
            vote_confidences = F.softmax(model(inputs), dim=1)

            time_consumption = (time.time() - start_time)
            memory_consumption = (torch.cuda.max_memory_allocated())
			
</code></pre>
<p>Here are the  inference time and memory consumption.</p>
<p>memory_consumption:  [10448984064, 1206454272, 1206454272, 1206454272, 1206454272, 1206454272, 1206454272, 1206454272, 1206454272]</p>
<p>time_consumption:  [4.7059266567230225, 0.8842918872833252, 0.9028427600860596, 0.8580131530761719, 0.862271785736084, 0.8550527095794678, 0.8648409843444824, 0.8601820468902588, 0.8670477867126465]</p>
<p>Is there any specific reason for this observation?</p>
<p>Many thanks</p>
What happens when the stride is larger than kernel_size?

Vedant_Roy: <p>In Pytorch’s <code>nn.ConvTranspose2d</code> module, we can set the stride to be larger than the <code>kernel_size</code>.<br>
For example:</p>
<ul>
<li>Input matrix of size 2x2</li>
<li>Kernel of size 2x2 (<code>kernel_size=2</code>)</li>
<li>Stride of 4 (<code>stride=4</code>).</li>
</ul>
<p>The output image will have dimensions 6x6.<br>
What is going on here?</p>
Class weighting for FasterRCNN using Resnet50

einsteinxx: <p>I am using the pytorch Resnet50 backbone for a FasterRCNN model as a basis for fine-tuning an object detector. I have done some transfer training on an ultrasound dataset (800 images) with decent results for a two class system. When I do further training with a much larger dataset (18000 images) with 80% belonging to one class, I get a lot of false negatives. Is there a way to weight the smaller label set higher than the larger label set during training for FasterRCNN?</p>

oke-aditya: <p>For imbalanced dataset can you try using RetinaNet? It uses focal loss and might handle such weight imbalances better.</p>
<p><a href="https://pytorch.org/vision/stable/models/retinanet.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/models/retinanet.html</a></p>
<p>Also see paper</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1708.02002">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon-32x32.png" class="site-icon" width="" height="">

      <a href="https://arxiv.org/abs/1708.02002" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/402;"><img src="https://static.arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png" class="thumbnail" width="" height=""></div>

<h3><a href="https://arxiv.org/abs/1708.02002" target="_blank" rel="noopener nofollow ugc">Focal Loss for Dense Object Detection</a></h3>

  <p>The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>As paper states</p>
<blockquote>
<p>“Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design and train a simple dense detector we call RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors.”</p>
</blockquote>
<p>This is just a try, cannot guarantee it would work better, but might just help.</p>
X1, y1, x2, y2 in iou loss stand for TopLeft, Bottom Right of bbox?

bryanbocao: <p>Hello,</p>
<p>I was trying to use iou loss in PyTorch, but the document is not clear in terms of <code>x1, y1, x2, y2</code>. Do <code>x1, y1</code> refer to one bbox’s top left corner while <code>x2, y2</code> stand for the bottom right corner? Assume the coordinate system is as follows: <code>x</code> is in horizontal axis and <code>y</code> is in vertical axis and <code>(0, 0)</code> is in the top left corner of an image.</p>
<p><a href="https://pytorch.org/vision/main/generated/torchvision.ops.distance_box_iou_loss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.ops.distance_box_iou_loss.html</a><br>
<a href="https://pytorch.org/vision/main/generated/torchvision.ops.generalized_box_iou_loss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.ops.generalized_box_iou_loss.html</a></p>
<p>Thanks!</p>

bryanbocao: <p>The document mentioned that</p>
<pre><code class="lang-auto">Both sets of boxes are expected to be in ``(x1, y1, x2, y2)`` format with
    ``0 &lt;= x1 &lt; x2`` and ``0 &lt;= y1 &lt; y2``, and The two boxes should have the
    same dimensions.
</code></pre>
<p>which seems to imply that and answers my question, without explicitly specifying the top-left and bottom-right corners.</p>

oke-aditya: <p>Yes. You are right. They may also be better understood as <code>(xmin, ymin, xmax, ymax)</code></p>
Is lowering the number of parameters in MultiHeadAttention possible?

edenlum: <p>In MultiHeadAttention, we perform linear projection to q,k,v and after that perform the attention operation.<br>
The term inside the softmax (QK^T) can be decomposed as follows<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/227dc416566d86ff905b23e2fc5565289722a9e7.png" alt="image" data-base62-sha1="4V7GygHy0jVCW3CgagnSgW0qxin" width="639" height="210"></p>
<p>This means we can remove the projection for the key (and by doing so save parameters) and instead change the query projection to fit with key D_in (so instead of Wq being D_q_in * D_embed it will be D_q_in * D_k_in).</p>
<p>We can also use this to reduce the number of parameters for an already trained model, by swapping Wq by WqWk^T and Bq by BqWk^T.</p>
<p>the reason we don’t need the last 2 terms is because softmax(x+c) = softmax(x) and we do the softmax on the query dimension.</p>
<p>Am I missing something? Any input will be appreciated.<br>
Thanks</p>
<p>*Images taken from <a href="https://arxiv.org/pdf/2006.16362.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2006.16362.pdf</a> - section 2.2</p>
Train a tokenizer like XLM Roberta tokenizer

huung: <p>Hi, How can I train a tokenizer like XLM Roberta tokenizer from scratch with <code>sentencepiece.bpe.model</code>?</p>
<p>I tried to use load their tokenizer and use <code>tokenizer.train_new_from_iterator</code> but it throw <code>PanicException: likelihood is NAN. Input sentence may be too long.</code></p>
<p>So what sentence’s length does <code>train_new_from_iterator</code> allow? And I see in their repo has a <code>sentencepiece.bpe.model</code> but I am sure about the correct way to get a file like that.</p>
<p>And in XLMR’s paper, the author said that they use Sentenpiece with Unigram model. But in <code>sentencepiece.bpe.model</code> is <code>bpe</code>. So does Huggingface traind a new one with bpe model instead?</p>
<p>Thank you for reading!</p>
In LSTM why h_t output twice?

laro: <p>According the LSTM design:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b79fd1708a0781325befddbc89c65334c85d7c01.jpeg" alt="image" data-base62-sha1="qcpMIMh88Yblm0w8FDKDRsRsGQh" width="579" height="347"></p>
<p>The hidden state (ht) is output twice (1 and 2 in the picture).</p>
<ol>
<li>If they are the same, why we need them twice ?</li>
<li>Is there a different use for each one of them ?</li>
<li>According to</li>
</ol>
<blockquote>
<p>nn.lstm</p>
</blockquote>
<p>there are 3 outputs (output, h_n, c_n).<br>
I didnt understand what is the different between output and h_n ? (Doesn’t they need to be the same) ?</p>

omid_khalafbeigi: <p>Hi.<br>
LSTM is type of Recurrent Neural Networks, so current output (output at time <strong>t</strong>) will affect on processing of input at time <strong>t+1</strong> (each input at specific time known as <strong>time-step</strong>)<br>
In according to above explanation, we need two <strong>ht</strong> (one for next layer and one for next time-step).<br>
I think this diagram can help you:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79a32af34142ceadaef4c036625f02c295efb4dd.png" data-download-href="https://discuss.pytorch.org/uploads/default/79a32af34142ceadaef4c036625f02c295efb4dd" title="13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79a32af34142ceadaef4c036625f02c295efb4dd_2_690x264.png" alt="13" data-base62-sha1="hm3mh1Jl3Pq3F17DThU3hGDMFjL" width="690" height="264" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79a32af34142ceadaef4c036625f02c295efb4dd_2_690x264.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79a32af34142ceadaef4c036625f02c295efb4dd_2_1035x396.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/9/79a32af34142ceadaef4c036625f02c295efb4dd.png 2x" data-dominant-color="EDEFC1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">13</span><span class="informations">1090×418 82.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>in pytorch, lstm has three outputs:<br>
output: what we use it for next or output layer.<br>
h_n: same output (when we need it to use for next time-step), known as hidden-state<br>
c_n: cell-state (upper horizontal line with two operators on it _ multiplication and sum)</p>
Loss function not converging

Nima_Emami: <p>Hi everyone</p>
<p>I am a beginner here and just ran into a problem. I have this model but I can not figure out why the loss function is not converging. the loss just fluctuates around the same value.</p>
<p>here is the code:</p>
<pre><code class="lang-auto">
import torch
import torch.nn as nn


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

class ConvBlock(nn.Module):
    def __init__(self,in_channel, out_channel,):
        super().__init__()
        self.conv_block = nn.Sequential(
            nn.Conv2d(in_channel,out_channel,kernel_size=3,padding='same'),
            nn.BatchNorm2d(out_channel),
            nn.Conv2d(out_channel,out_channel,kernel_size=3,padding='same'),
            nn.BatchNorm2d(out_channel),
            nn.ReLU()
            )
        self.conv_block.cuda()

    def forward(self,x):
        return self.conv_block(x)

class FFNN(nn.Module):
    def __init__(self,in_channel , out_channel=10,hidden_layers=[4096,256,64]) -&gt; None:
        super().__init__()
        self.block = nn.Sequential()
        self.in_channel = in_channel
        self.out_channel  = out_channel
        self.hidden_layers = hidden_layers
        for i,layer in enumerate(self.hidden_layers):
            self.block.add_module(f'Linear_{i}',nn.Linear(self.in_channel,layer))
            self.block.add_module(f'Relu_{i}',nn.ReLU())
            self.in_channel = layer

        self.block.add_module('Linear2',nn.Linear(self.hidden_layers[-1],self.out_channel))

        self.block.cuda()

    def forward(self,x):
        return self.block(x)

class UnetEncoder(nn.Module):
    def __init__(self,in_channel=1,features=[64,128,256,512], hidden_layers=None):
        super().__init__()
        self.pool = nn.AvgPool2d(kernel_size=(2,2),stride=2)
        self.encoders = nn.ModuleList()
        self.hidden_layers = hidden_layers


        for feature in features:
            self.encoders.append(ConvBlock(in_channel=in_channel,out_channel=feature))
            in_channel = feature
     

    def forward(self,x):

        for encoder in self.encoders:
            x = encoder(x)
            x = self.pool(x)
        x = x.reshape(x.size(0),-1)
        if self.hidden_layers:
            final = FFNN(in_channel=x.size(-1),hidden_layers=self.hidden_layers)
        
        else: 
            final = FFNN(in_channel=x.size(-1))
        
        return final(x)

</code></pre>
Efficiently backpropagate subset of minibatch

JannisB: <p>Hi, I have a batch size of 2048 and based on the results from the forward pass I only want to backpropagate half of the minibatch (I have 1024 indices). If I do this naively, this is just as slow as backpropagating the whole minibatch but is there a way to do this efficiently in PyTorch?<br>
Best, Jannis</p>
Is there a way to load a single big (97.2 gb) file using torch.load?

Vadim_Bulatov: <p>I’ve only 16gb CPU, and I cant find a way to load a 97.2gb dataset. Batching is made down the road, as I understand, with the 1st step of data loading using torch.load</p>
<p>Is there a way to somehow split the file into pieces ?</p>
Error in first gan

MILI: <p>Hi everyone<br>
when I run the code from Coursera week 1 “your new GAN” this error appears?</p>
<p>PS D:\python\all.py&gt; &amp; “C:/Program Files/Python39/python.exe” d:/python/all.py/me.py<br>
Traceback (most recent call last):<br>
File “d:\python\all.py\me.py”, line 63, in <br>
test_gen_block(25,12)<br>
File “d:\python\all.py\me.py”, line 48, in test_gen_block<br>
block = get_generator_block(in_features, out_features)<br>
File “d:\python\all.py\me.py”, line 40, in get_generator_block<br>
nn.Linear(None, None),<br>
File “C:\Users\Active\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\linear.py”, line 96, in <strong>init</strong><br>
self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))<br>
TypeError: empty() received an invalid combination of arguments - got (tuple, dtype=NoneType, device=NoneType), but expected one of:</p>
<ul>
<li>(tuple of ints size, *, tuple of names names, torch.memory_format memory_format, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)</li>
<li>(tuple of ints size, *, torch.memory_format memory_format, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool<br>
pin_memory, bool requires_grad)<br>
should I change any thing or not?</li>
</ul>

ptrblck: <p>This initialization:</p>
<pre><code class="lang-python">nn.Linear(None, None)
</code></pre>
<p>is wrong as you are supposed to set values for <code>in_features</code> and <code>out_features</code> and cannot pass <code>None</code> as arguments.<br>
Are you sure you should just execute the code or are you supposed to fill in the missing pieces?</p>

MILI: <p>yes, thank you<br>
I got it</p>
Using Pytorch with f2py Python module during training?

IZugec: <p>Does anyone know whether I could use f2py during training?. Idea is to use model’s output as an input in the fortran subroutine which I can import in python via numpy’s f2py. The question is whether the computation graph would continue once I do operations within the subroutine?</p>

KFrank: <p>Hi Ivan!</p>
<aside class="quote no-group" data-username="IZugec" data-post="1" data-topic="155530" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/izugec/48/51049_2.png" class="avatar"> IZugec:</div>
<blockquote>
<p>The question is whether the computation graph would continue once I do operations within the subroutine?</p>
</blockquote>
</aside>
<p>I don’t know anything about f2py, but if I understand your proposal, no,<br>
the computation graph won’t continue through the call into numpy / f2py,<br>
which is to say, you won’t be able to backpropagate gradients through<br>
the numpy / f2py call.</p>
<p>To be able to use pytorch’s autograd for backpropagation, you have two<br>
choices:</p>
<p>You could translate the fortran code into pytorch tensor operations, after<br>
which you will get autograd and backpropagation “for free.”</p>
<p>Or you could write a <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="noopener nofollow ugc">custom autograd function.</a>  In this case, you would<br>
implement a <code>forward()</code> function that computes the desired function and<br>
could be implemented using numpy / f2py.  But you also have to implement<br>
the companion <code>backward()</code> function that (roughly speaking) computes<br>
the gradient of your function.</p>
<p>To write <code>backward()</code>, you would typically differentiate your function<br>
analytically, and then implement the numerical evaluation of that analytic<br>
derivative.  (You could also differentiate it numerically.)  But it would be<br>
perfectly fine to implement the derivative evaluation that forms the guts of<br>
<code>backward()</code> in fortran via numpy / f2py (with a wrapper that converts its<br>
inputs and outputs from and to pytorch tensors).</p>
<p>Best.</p>
<p>K. Frank</p>
Manually set weights for nn.Conv2d?

Vedant_Roy: <p>I’m trying to manually set the weights for a<code>nn.Conv2d</code>. I’ve seen this post: <a href="https://discuss.pytorch.org/t/how-to-set-nn-conv2d-weights/67407" class="inline-onebox">How to set nn.conv2d weights</a>, however I’m trying to use a numpy array as the weights. Why? Because <code>np.asarray(x)</code> allows me to easily construct my weights array using a nested Python list (<code>x</code> is Python list).</p>
<p>Example code:</p>
<pre><code class="lang-auto">IN = 2
OUT = 3
KERNEL_SIZE = 2
proof_conv = nn.Conv2d(IN, OUT, kernel_size=KERNEL_SIZE, padding=0, bias=False)
assert proof_conv.weight.shape == (OUT, IN, KERNEL_SIZE, KERNEL_SIZE)

ZERO_FILTER = [[0., 0.], [0., 0.]]
ONE_FILTER =  [[1., 1.], [1., 1.]]
weights = [
           [ZERO_FILTER, ZERO_FILTER],
           [ONE_FILTER, ONE_FILTER],
           [ONE_FILTER, ZERO_FILTER],
]
weights = np.asarray(weights).astype(np.float64)
proof_conv.weight = nn.Parameter(torch.from_numpy(weights))

ones = torch.ones((IN, 4, 4))
out_img = proof_conv(ones)
</code></pre>
<p>However, this code fails with the error:</p>
<blockquote>
<p>RuntimeError: expected scalar type Float but found Double</p>
</blockquote>
<p>Does anyone know what’s going on here?</p>

ptrblck: <p>Transform the tensor to <code>float</code> and it should work:</p>
<pre><code class="lang-python">with torch.no_grad():
    proof_conv.weight.copy_(torch.from_numpy(weights).float())
</code></pre>
Apply a if operation pointwise on a torch tensor

DHOUIBM: <p>Hello,<br>
So there is something that’s I can’t figure out, is it possible to do the operation (A==0 + A==1)&gt;0 on the same operation when A is a pytorch tensor ?<br>
So the operation result is simply a matrix equal to true were  simple A[x] in [0,1] that does the or operation pointwise.<br>
Same for the operation  (A!=0 * A!=1) to do the and operation<br>
Thanks in advance !</p>

ptrblck: <p>I don’t know if I understand your use case correctly, but would this work?</p>
<pre><code class="lang-python">a = torch.randint(0, 3, (10, 10))
res = (a==0) | (a==1)
</code></pre>
Pytorch error when i try to train my data

joel2: <hr>
<h2>
<a name="h-2-frames-1" class="anchor" href="#h-2-frames-1"></a>2 frames</h2>
<p><a href="https://localhost:8080/#" rel="noopener nofollow ugc">/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py</a> in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing) 2994 if size_average is not None or reduce is not None: 2995 reduction = _Reduction.legacy_get_string(size_average, reduce) → 2996 return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) 2997 2998</p>
<p>ValueError: Expected input batch_size (150000) to match target batch_size (50000).</p>

Loki_K: <aside class="quote no-group" data-username="joel2" data-post="1" data-topic="155509">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c6cbf5/48.png" class="avatar"> joel2:</div>
<blockquote>
<p>ValueError: Expected input batch_size (150000) to match target batch_size (50000).</p>
</blockquote>
</aside>
<p>Apparently there is batch size mismatch in the tensors you gave to loss function.<br>
You gave 150k predictions to compare with 50k targets in your loss function.<br>
You should post give n predictions to compare with n targets.</p>
<p>Have a good day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

joel2: <p>Thank you it worked (-1,32<em>32</em>3)</p>
How to use pretrained embeddings in machine learning project?

mark_eu: <p>Hi,</p>
<p>I have an NLP project where I would like to compare the results of the NN (torchtext) approach to the classic ML approach in a way to use pre-trained glove/fasttext embeddings for ML, for example, Linear Regression in sklearn.<br>
Is it possible? This approach is because I want to use the same dataset split and the same vectors.</p>
<p>Thank you for any help, especially code snippets or links with examples</p>
How to access all the labels at once from dataloader?

ADONAI_TZEVAOT: <p>I have the dataloaders as such</p>
<pre><code class="lang-auto">train_dl= torch.utils.data.DataLoader(dataset, batch_size=32, sampler=train_sampler)
val_dl = torch.utils.data.DataLoader(dataset, batch_size=32, sampler=valid_sampler)
</code></pre>
<p>I finished training the model and now I want to get the entire list of labels from val_dl to calculate<br>
confusion_matrix(val_labels, y_pred_tag) . How do I do that ?<br>
Please help !</p>

ptrblck: <p>You could store the labels with the predictions while iterating the <code>val_dl</code>.</p>
Fuse ModuleList for quantization aware training

Shen: <p>Hi, I’m trying to perform quantization aware training. Currently, I have a customized Conv module which consists of a <code>ModuleList</code> of several <code>Conv2d</code> modules. This is then followed by a <code>ModuleList</code> of several <code>BatchNorm2d</code> modules, e.g., <code>ModuleList(Conv2d(), Conv2d()) - ModuleList(BatchNorm2d(), BatchNorm2d)</code>.</p>
<p>According to <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training" rel="noopener nofollow ugc">this</a>, a fuse operation is required. However, it looks like this operation only supports a limited number of combinations (e.g., conv-bn-relu, conv-bn, etc.). I’m wondering,</p>
<ul>
<li>How to fuse <code>ModuleLists</code> in order to use PyTorch quantization machanism?</li>
<li>Can PyTorch quantization aware training work without fusing these modules?</li>
</ul>
<p>Please help! Thanks!</p>
What is src and tgt for a transformer model?

voyo: <p>For nlp, I have understood that it might for instance be sentences in english and french, but what about other types of problems?</p>
<p>I have binary time series data, and want to do classification. What would be the input to a transformer (src and tgt)?</p>

JuanFMontesinos: <p>Transformers work with sequences of any kind.<br>
So src is nothing but the sequence you know and tgt is the sequence you want to predict. Anyway there are many ways of using transformers so it’d better if you find a nice tutorial or a bunch of papers that give you some ideas.</p>
Fastest implementation for indexing into multi-dimensional tensors

hakzc: <p>Hello,<br>
I have an application where I keep in GPU memory a 4-dimensional tensor that I use as a lookup table, where I store some pre-computed values in order to avoid to compute them each time in the forward pass of a GCNN layer saving on inference time.<br>
From several experiments I’ve noticed that the more the GPU is performant, the smaller the savings are. The bottleneck is actually on the indexing in that lookup table, indeed it is 10 times slower than the other computations I do after the indexing. Seems that with high performing GPUs such as Nvidia RTX8000, Nvidia RTX A6000 the computation from scratch of these pre-computed values is comparable to just take the pre-computed values stored in the tensor.<br>
My question is: are there in PyTorch faster ways for indexing into multi-dimensional tensors other than just doing <code>tensor[list_indexes1, list_indexes2]</code> ?</p>
Plot a 1d array from a 3d torch tensor

miraboreasu: <pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt

def get_number(swnew):
    temp = torch.div(1, swnew)
    temp[temp == float("Inf")] = 200
    return temp

if __name__ == '__main__':

    swnew     = torch.zeros(10, 1, 1, requires_grad=True, dtype=torch.float64)    
    y= get_number(swnew)
    plt.plot(swnew[:,1,1], y[:,1,1])


</code></pre>
<p>Hey, sorry this one is pretty easy, but I don’t know why it shows<br>
IndexError: index 1 is out of bounds for dimension 1 with size 1</p>
<p>Thank you</p>

Loki_K: <p>You intialized swnew with shape (10, 1, 1) and while plotting you are trying to access an axis which is not there. .i.e. you only have one axis and you are accessing 2nd axis.<br>
remember in python indexing starts from 0.</p>
<p>you should try something like swnew[: ,0, 0]</p>
Error while building libtorch for ARM64 platform

MeenalNarkhede: <p>Hi,</p>
<p>I am trying to build libtorch by cross-compiling for ARM64 platform. I am foolowing the steps given here for the same: <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" class="inline-onebox" rel="noopener nofollow ugc">pytorch/libtorch.rst at master · pytorch/pytorch · GitHub</a>.</p>
<p>I am getting the following error when the process reaches compiling the abi-check.cpp:<br>
– Configuring build for SLEEF-v3.6.0<br>
Target system: Linux<br>
Target processor: aarch64<br>
Host system: Linux-5.4.0-96-generic<br>
Host processor: x86_64<br>
Detected C compiler: GNU @ /home/meenal/x86_64-petalinux-linux/usr/bin/aarch64-xilinx-linux/aarch64-xilinx-linux-gcc<br>
CMake: 3.15.3<br>
Make program: /home/meenal/x86_64-petalinux-linux/usr/bin/ninja<br>
Crosscompiling SLEEF.<br>
Native build dir:<br>
– Using option <code>-Wall -Wno-unused -Wno-attributes -Wno-unused-result -Wno-psabi -ffp-contract=off -fno-math-errno -fno-trapping-math</code> to compile libsleef<br>
– Building shared libs : OFF<br>
– Building static test bins: OFF<br>
– MPFR : /home/meenal/aarch64-xilinx-linux/usr/lib/libmpfr.so<br>
– MPFR header file in /home/meenal/aarch64-xilinx-linux/usr/include<br>
– GMP : /home/meenal/aarch64-xilinx-linux/usr/lib/libgmp.so<br>
– RT : /home/meenal/aarch64-xilinx-linux/usr/lib/librt.so<br>
– FFTW3 : LIBFFTW3-NOTFOUND<br>
– OPENSSL :<br>
– SDE : SDE_COMMAND-NOTFOUND<br>
– RUNNING_ON_TRAVIS :<br>
– COMPILER_SUPPORTS_OPENMP : 1<br>
AT_INSTALL_INCLUDE_DIR include/ATen/core<br>
core header install: /home/meenal/code/pytorch/build_libtorch/build/aten/src/ATen/core/TensorBody.h<br>
core header install: /home/meenal/code/pytorch/build_libtorch/build/aten/src/ATen/core/aten_interned_strings.h<br>
core header install: /home/meenal/code/pytorch/build_libtorch/build/aten/src/ATen/core/enum_tag.h<br>
– /home/meenal/x86_64-petalinux-linux/usr/bin/aarch64-xilinx-linux/aarch64-xilinx-linux-g++ /home/meenal/code/pytorch/torch/abi-check.cpp -o /home/meenal/code/pytorch/build_libtorch/build/abi-check<br>
/home/meenal/code/pytorch/torch/abi-check.cpp:1:10: fatal error: iostream: No such file or directory<br>
1 | <span class="hashtag">#include</span> <br>
|          ^~~~~~~~~~<br>
compilation terminated.<br>
CMake Error at caffe2/CMakeLists.txt:1213 (message):<br>
Could not compile ABI Check: 1</p>
<p>Not sure why iostream is not found.<br>
Any help is appreciated.</p>
<p>Thanks and regards,<br>
Meenal</p>
subprocess.CalledProcessError: Command '['/share/software/user/open/python/3.9.0/bin/python3', '-u', 'main_pretrain.py', '--local_rank=3']' returned non-zero exit status 1

cyril: <p>I’m trying to train my network on my university cluster on 4 A100 GPUs with 80 GBs RAM but it seems to be crashing mid-epoch with the following uninformative error:</p>
<pre><code class="lang-auto">[03:00:26.169445] Start training for 800 epochs
[03:00:29.881443] Epoch: [0]  [  0/157]  eta: 0:09:42  lr: 0.000000  loss: 2.5381 (2.5381)  time: 3.7102  data: 2.6209  max mem: 7930
[03:01:20.581232] Epoch: [0]  [ 20/157]  eta: 0:05:54  lr: 0.000000  loss: 2.2845 (2.3390)  time: 2.5349  data: 0.4167  max mem: 9074
[03:02:11.337795] Epoch: [0]  [ 40/157]  eta: 0:05:00  lr: 0.000000  loss: 2.3415 (2.3727)  time: 2.5378  data: 0.8262  max mem: 9074
[03:02:59.493182] Epoch: [0]  [ 60/157]  eta: 0:04:03  lr: 0.000001  loss: 2.2510 (2.3554)  time: 2.4077  data: 0.0760  max mem: 9074
[03:03:48.747358] Epoch: [0]  [ 80/157]  eta: 0:03:12  lr: 0.000001  loss: 2.3077 (2.3214)  time: 2.4627  data: 0.7480  max mem: 9074
[03:04:40.015038] Epoch: [0]  [100/157]  eta: 0:02:23  lr: 0.000001  loss: 1.8802 (2.2680)  time: 2.5633  data: 0.1273  max mem: 9074
[03:05:27.755995] Epoch: [0]  [120/157]  eta: 0:01:32  lr: 0.000001  loss: 1.9293 (2.2229)  time: 2.3870  data: 0.0006  max mem: 9074
Killing subprocess 57832
Killing subprocess 57833
Killing subprocess 57834
Killing subprocess 57835
Traceback (most recent call last):
  File "/share/software/user/open/python/3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/share/software/user/open/python/3.9.0/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/share/software/user/open/py-pytorch/1.8.1_py39/lib/python3.9/site-packages/torch/distributed/launch.py", line 340, in &lt;module&gt;
    main()
  File "/share/software/user/open/py-pytorch/1.8.1_py39/lib/python3.9/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/share/software/user/open/py-pytorch/1.8.1_py39/lib/python3.9/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/share/software/user/open/python/3.9.0/bin/python3', '-u', 'main_pretrain.py', '--local_rank=3']' returned non-zero exit status 1.
</code></pre>
<p>I use the following to train:</p>
<pre><code class="lang-auto">LOGLEVEL=INFO python3 -u -m torch.distributed.launch --nproc_per_node=4 main_pretrain.py
</code></pre>
<p>What’s weird is that the same exact code runs on my local setup of 4 V100 GPUs with 40 GBs RAM.</p>
<p>Both setups are running on Pytorch 11.1 with CUDA 11. Any advice is greatly appreciated!</p>

cyril: <p>For future reference, setting<br>
<code>from torch.distributed.elastic.multiprocessing.errors import record</code><br>
and the decorator <code>@record</code> should help with debugging! In my case it was a missing image file that was causing the crash!</p>
VGG16-LSTM number of inputs

python_Dev: <p>I am trying to replicate this <a href="https://www.palermo.edu/ingenieria/pdf2020/CyT_20_03.pdf" rel="noopener nofollow ugc">paper</a>.<br>
Since I am using cross-entropy, I won’t add the sigmoid function when training.<br>
The input size of the lstm is 25088 but I am not sure what hidden_size and lstm_layers to use in order to be able to use cross-entropy.</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects a model output in the shape <code>[batch_size, nb_classes]</code> so you would have to make sure your model creates such outputs containing logits.<br>
Depending on the model architecture (and if the <code>LSTM</code> is the last layer), the <code>hidden_size</code> could define the number of features and could thus match the number of classes. However, if the LSTM output is passed to another layer (e.g. <code>nn.Linear</code>) then this layer would have to output the class logits.</p>

python_Dev: <p>Thank you, I’ve used another nn.Linear layer and fixed that.</p>
Batch size=6 running in 4*3090, I got different batch size between input and output

JimmyMa99: <p>My code can be runned in batch size=8、15, but it cannot be runned in batch size=6.<br>
I didnot change the batch size in my model, it is so weird.</p>
<p>Here is the code, when the batch size is 6, it will be crops into 4 pieces, so the input bs of the module is 6*4=24, but I get output bs=18.</p>
<pre><code class="lang-auto">class Qtransfer_sp5(nn.Module):
    def __init__(self):
        super().__init__()
        ch_f = 56
        self.b1 = ResBlock(9, 128, 128, stride=2)
        self.b1_1 = ResBlock(128, 128, 128)
        self.b1_2 = ResBlock(128, 128, 128)

        self.b2 = ResBlock(128, 256, 256, stride=2)
        self.b2_1 = ResBlock(256, 256, 256)
        self.b2_2 = ResBlock(256, 256, 256)
        
        self.b3 = ResBlock(256+4096, 512, 512, stride=2)
        self.b3_1 = ResBlock(512, 512, 512)
        self.b3_2 = ResBlock(512, 512, 18)
        
        self.getQ1 = nn.Sequential(self.b1,
                                   self.b1_1,
                                   self.b1_2)
        
        self.getQ2 = nn.Sequential(self.b2,
                                   self.b2_1,
                                   self.b2_2)
        
        self.getQ3 = nn.Sequential(self.b3,
                                   self.b3_1,
                                   self.b3_2)

    
    def get_Qfeats(self,x,probs):
        b,c,h,w=probs.shape#bs,9,448*448
        x1=self.getQ1(probs)

        x2=self.getQ2(x1)
        feat = F.interpolate(
                x, (int(h/4), int(w/4)), mode='bilinear', align_corners=False)

        cat=torch.cat([feat.detach(),x2],dim=1)
        x3=self.getQ3(cat)
        return x3

    
    def get_sp_cam(self,logits,deconv_para):
        bg= upfeat(logits[:,0:1],deconv_para[:,:9],1,1)
        fg= upfeat(logits[:,1:],deconv_para[:,9:],1,1)
        logits =torch.cat([bg,fg],dim=1)
        return logits

    def forward(self, feat, logits, probs,cam_map):
        batch_size, c_x, h_x, w_x = feat.shape
        b,c,w,h=probs.shape#bs,9,448*448
        

        
        feats_parameters= self.get_Qfeats(feat,probs)#bs,18,56*56
        bg_para=get_noliner(F.softmax(feats_parameters[:,:9],dim=1))#torch.sum(fg_aff).max()# fg_aff[0,:,10:20,10:20].detach().cpu().numpy()
        fg_para=get_noliner(F.softmax(feats_parameters[:,9:],dim=1))
        deconv_parameters= torch.cat([bg_para,fg_para],dim=1)
        feats = self.get_sp_cam(cam_map,deconv_parameters)
        feats = F.interpolate(feats,(h_x, w_x)).type_as(feats)
        
        return feats
</code></pre>
<p>My torch version is ‘1.8.1+cu111’</p>
Changing layers of a pretrained model

maryma: <p>Hi, I am new to the concept of transfer learning. I have one pretrained model with pth extension. This model trained with a lot of medical images and its purpose was segmentation. But I wanted to use the model for prediction. My purpose is to detect survival times for different patients from their MRI images. I thought that the model that can make segmentation can extract good features for survival times. So I wanted to load the pretrainedmodel.pth and delete some layers of this model and add some fully connected layers to it. First I don’t know how is it possible to get the layers of that pretrained model second I do not know how can I delete some of that pretrained model layers and add new layers to it. I am familiar with the freezing topic.<br>
If you know the answer can you please explain with codes?</p>

ptrblck: <p>Assuming the stored file contains the trained <code>state_dict</code> of the model, you could create a new object of the model, load the <code>state_dict</code>, and use parts of your pretrained model in a new classification model.<br>
Something like this should work:</p>
<pre><code class="lang-python"># create object of pretrained model
pretrained_model = SegmentationModel(args)
# load state_dict
pretrained_model.load_state_dict(torch.load(path_to_state_dict))

# create object of new classification model
model = ClassificationModel(pretrained_model)
# in the class definition of Classification model pretrained_model's layers will be reused
</code></pre>

maryma: <p>I am a little confused so excuse me if I ask basic questions. What is SegmentaionModel and args in front of it?<br>
I just could find a file on github it was segmentation.pth so I do not have any idea what is this model architecture. first I wanted to load this model second I wanted to understand what is architecture for example with something like model.summary we can find that it has 1 cnn 1 relu 1 pool 1 cnn 1 relu 1 global avg pool 1 flatten 1 fc. Then I wanted to delete 1 cnn 1 relu 1 global avg pool 1 flatten 1 fc from last layers of model and add some arbitrary layers to it. Now I have these problems:</p>
<ol>
<li>How can understand the architecture of model(Code)</li>
<li>How can delete last layers(Code)</li>
<li>How can add new layers(Code)<br>
When you explained the answer you made two objects of two classes : SegmentationModel and<br>
ClassificationModel. But what are these classes?</li>
</ol>
Problem using han_window()

Afekst: <p>Hi everyone,<br>
first of all I’d like to ask if anybody know where libtorch’s documentation has disappeared?</p>
<p>and for my main question…<br>
when I’m trying to use the function i get an exception. Example:</p>
<pre><code class="lang-auto">torch::Tensor win = torch::hann_window(512, true, torch::TensorOptions().requires_grad(false));

</code></pre>
<p>implies the following exception:</p>
<pre><code class="lang-auto">Unhandled exception at 0x00007FF9C386474C in project.exe: Microsoft C++ exception: c10::Error at memory location 0x00000026850FBC80.
</code></pre>
<p>I’m using visual studio and running everything on the cpu.</p>
<p>please help, thank you.</p>

ptrblck: <aside class="quote no-group" data-username="Afekst" data-post="1" data-topic="155475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e19adc/48.png" class="avatar"> Afekst:</div>
<blockquote>
<p>first of all I’d like to ask if anybody know where libtorch’s documentation has disappeared?</p>
</blockquote>
</aside>
<p>The docs can still be found <a href="https://pytorch.org/cppdocs/">here</a> and I’m unaware of any removal. Could you describe which parts were removed from there?</p>
<aside class="quote no-group" data-username="Afekst" data-post="1" data-topic="155475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e19adc/48.png" class="avatar"> Afekst:</div>
<blockquote>
<p>implies the following exception:</p>
</blockquote>
</aside>
<p>This error is quite cryptic, but I get a <code>dtype</code> mismatch when executing your code on Linux, which is fixed via:</p>
<pre data-code-wrap="python"><code class="lang-python">#include &lt;torch/script.h&gt; // One-stop header.
#include &lt;iostream&gt;

int main(int argc, const char* argv[]) {
  torch::Tensor win = torch::hann_window(512, true, at::TensorOptions().dtype(at::kFloat).requires_grad(false));
  std::cout &lt;&lt; win &lt;&lt; std::endl;
  return 0;
}
</code></pre>

Afekst: <p>thank you very much, it indeed solved the problem.<br>
in documentation i ment method level documentation, this is a link that’s no longer active for example:<br>
<a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1a2121746050a71d97d90349d944b53391.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/function_namespacetorch_1a2121746050a71d97d90349d944b53391.html</a></p>
To Tensor transform is messing my images?

Loki_K: <pre><code class="lang-auto"># create image dataset
f_ds = torchvision.datasets.ImageFolder(data_path)
# transform image to tensor.
to_tensor = torchvision.transforms.ToTensor()


for idx, (img, label) in enumerate(f_ds):
    if idx == 23:
        # random pil image 
        plt.imshow(img)
        plt.show()
 
        # image to np array
        n_arr = np.asarray(img)
        print("np array shape :", n_arr.shape)

        h, w, c = n_arr.shape
         
        # reshaping the numpy array has no problem
        re_n_arr = n_arr.reshape(c, h, w)
        plt.imshow(re_n_arr.reshape(h, w, c))
        plt.show()

        # torch tensor
        torch_ten = to_tensor(img)
        print("torch_tensor shape:", torch_ten.shape)
        print(torch_ten.numpy().shape)
        plt.imshow(torch_ten.numpy().reshape(h, w, c))

        break
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/546470ec741addae35826ffb18cba75b0ae6f852.png" data-download-href="https://discuss.pytorch.org/uploads/default/546470ec741addae35826ffb18cba75b0ae6f852" title="pytorch forum"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/546470ec741addae35826ffb18cba75b0ae6f852_2_236x500.png" alt="pytorch forum" data-base62-sha1="c2ziHlw9aqYsLgotW9Gi5pvXWtI" width="236" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/546470ec741addae35826ffb18cba75b0ae6f852_2_236x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/546470ec741addae35826ffb18cba75b0ae6f852_2_354x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/4/546470ec741addae35826ffb18cba75b0ae6f852.png 2x" data-dominant-color="989A8D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch forum</span><span class="informations">379×801 365 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please help me with this. I’m stuck here.</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p><code>ToTensor</code> permutes the array and returns a tensor in the shape <code>[C, H, W]</code>. Using <code>reshape</code> to swap the order of dimensions is wrong and will interleave the image as is  seen in your last output:</p>
<pre><code class="lang-python">        # torch tensor
        torch_ten = to_tensor(img)
        print("torch_tensor shape:", torch_ten.shape)
        print(torch_ten.numpy().shape)
        plt.imshow(torch_ten.numpy().reshape(h, w, c))
</code></pre>
<p>use <code>plt.imshow(torch_ten.permute(1, 2, 0).numpy())</code> and it should work.</p>
Gradident slightly change even not added to Optimizier

Erica_Zheng: <p>Hi, I encounter a strange effect in Pytorch.</p>
<pre><code class="lang-auto">model_1 = Net()
model_2 = Net()

optimizer = optim.Adam([{'params': model_1.parameters(), 'lr': 0.001},  # only update model
                     ])
</code></pre>
<p>However, I found that <code>model_2</code> grad has slightly changed.</p>
<pre><code class="lang-auto">model_1.conv1.weight.grad =  tensor([0.1468, 0.1595, 0.1519, 0.1456, 0.1540], device='cuda:1')
model_2.conv1.weight.grad =  tensor([ 0.0480,  0.0073, -0.0298, -0.0482, -0.0545], device='cuda:1')

model_1.conv1.weight.grad =  tensor([0.0545, 0.0608, 0.0546, 0.0486, 0.0517], device='cuda:1')
model_2.conv1.weight.grad =  tensor([ 0.0503,  0.0087, -0.0272, -0.0466, -0.0535], device='cuda:1')
</code></pre>

ptrblck: <p>Could you post a minimal, executable code snippet reproducing this issue, please?</p>