akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The gradient backpropagation in feature residual learning

PaTrickWwW: <p>I have a pretrained network and two images. Two separate features are extracted from the middle layer of the pretrained network, named F_l and F_h. Afterward, I used two convolution layers to learn the residual between them, but the residual learning seems not working, is there something wrong with gradient backpropagation?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a82ffdd63873c300789ec272691979375ada3c79.png" alt="image" data-base62-sha1="nZR2g8KlD05R7W38NkU10tdQ9WV" width="589" height="96"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/d/adb474a48420bffa8f745e48e27d8aada5399c61.png" alt="image" data-base62-sha1="oMFe3PzaIJkCLMsrElsGS0m2mxX" width="528" height="57"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/531d4f24dfc468c9f14916c1e97e1df52b588b92.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/531d4f24dfc468c9f14916c1e97e1df52b588b92" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/531d4f24dfc468c9f14916c1e97e1df52b588b92_2_690x138.jpeg" alt="image" data-base62-sha1="bRgqkg5xh7o21c4i6ypWKvHw0WS" width="690" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/531d4f24dfc468c9f14916c1e97e1df52b588b92_2_690x138.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/3/531d4f24dfc468c9f14916c1e97e1df52b588b92.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/3/531d4f24dfc468c9f14916c1e97e1df52b588b92.jpeg 2x" data-dominant-color="272729"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">874×176 44.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab6268c85b40ef3138226d16ecb36df307f0cac1.png" alt="image" data-base62-sha1="os8uhsZaIPVlNkrN20SIjMrGERH" width="664" height="87"></p>

soulitzer: <p>The call to <code>hr_feature = torch.tensor(hr_feature)</code> would prevent gradients from flowing through. Is that intended?</p>

PaTrickWwW: <p>It is not intended, it is a suggestion from ChatGPT.</p>
Const tensor vs input in AOT FX graph

jasmine_n: <p>Is there a way to discern <code>self.consttensor</code> from <code>input</code> in AOT FX graph? For the test example, is there any attribute or method that tells that arg3_1 is the <code>input</code>, and arg2_1 is the <code>self.consttensor</code>. The input always goes in the end of the arg list, but the model could have variable number of inputs</p>
<pre><code class="lang-auto">class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = torch.nn.Linear(20, 30)
        self.consttensor = torch.randn(1, 30)
    
    def forward(self, input):
        out = self.linear(input)
        out = out + self.consttensor
        return out
</code></pre>
<pre><code class="lang-auto">opcode         name    target              args                 kwargs
-------------  ------  ------------------  -------------------  --------
placeholder    arg0_1  arg0_1              ()                   {}
placeholder    arg1_1  arg1_1              ()                   {}
placeholder    arg2_1  arg2_1              ()                   {}
placeholder    arg3_1  arg3_1              ()                   {}
call_function  t       aten.t.default      (arg0_1,)            {}
call_function  addmm   aten.addmm.default  (arg1_1, arg3_1, t)  {}
call_function  add     aten.add.Tensor     (addmm, arg2_1)      {}
output         output  output              ((add,),)            {}
</code></pre>
Problem with image transformation

Ahmed_Sallam: <p>I have a problem transforming my images of dimensions of 320*160 I used this transformation and it trains but when trying to actually run the model it fails after the flattening layer which after testing a lot of solutions it came to my mind that the transformation is probably the problem specially that I don’t really know how to transform using pytorch correctly this my transformation<br>
<code>transform_car = transforms.Compose(         [transforms.Resize([66, 200]),          transforms.RandomCrop([62,194]),          transforms.RandomHorizontalFlip(),          transforms.ToTensor()])</code><br>
is there a way to fix it? or like how should I resize,crop and randomly flip the images</p>

ptrblck: <p>The transformation looks alright and will create an output tensor in the shape <code>[channels,  62, 194]</code>. What’s the actual issue? Are you running into a shape mismatch? If so, check the flattened activation shape and make sure the next layer expects the same number of features (e.g. a linear layer).</p>

Ahmed_Sallam: <p>It trains fine but when I try to run the model it says that mat 1 and mat2 shapes cannot be multiplied (64x429 and 1088x100)</p>

ptrblck: <p>What does “run” mean here? If you are properly able to train the model you are already executing the forward and backward pass.<br>
Assuming the failure is raised during inference check the input shapes and make sure they are equal to the training case.</p>

Ahmed_Sallam: <p>The model is plugged in a simulator from which I collected the data so when I try to run it on the sim it just gives this error</p>

ptrblck: <p>Did you make sure to use the same transformation or at least guarantee the same spatial size is used?</p>

Ahmed_Sallam: <p>Nope I think that’s it, this is really is my first big deep learning project and my first pytorch one so I think I made alot of mistakes I will sure try that</p>
Ceil_mode in avg_pool2d seems to output wrong result

sashasasha: <p>It seems that <code>torch.nn.functional.avg_pool2d()</code> doesn’t take <code>ceil_mode=True</code> into account when “doing math”.</p>
<pre><code class="lang-auto">import torch
t = torch.ones(1, 1, 4, 4)
out = torch.nn.functional.avg_pool2d(input=t, kernel_size=3, stride=2, padding=1, ceil_mode=True)
print(out)

#out:
#tensor([[[[0.4444, 0.6667, 0.3333],
#          [0.6667, 1.0000, 0.5000],
#          [0.3333, 0.5000, 0.2500]]]])
</code></pre>
<p>The whole input, when padded with <code>padding=1</code> AND <code>ceil_mode=True</code> should look like this:</p>
<pre><code class="lang-auto">0 0 0 0 0 0 0
0 1 1 1 1 0 0
0 1 1 1 1 0 0
0 1 1 1 1 0 0
0 1 1 1 1 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
</code></pre>
<p>in order to get the proper output dim.</p>
<p>Isolating the top-right corner of the input that the kernel sees for the top-right output value:</p>
<pre><code class="lang-auto">0 0 0
1 0 0
1 0 0
</code></pre>
<p>The output pixel here should be <code>2 / 9 = 0.2222</code>, but instead we get <code>2 / 6 = 0.3333</code>, as if the extra padding from <code>ceil_mode=True</code> is not accounted for when “doing math”.</p>
<p>Similarly, all the other values along the right and bottom padded areas have, to me, unexpected values.</p>
<p>Is this a bug, or is it by design?</p>
Should all nodes be running on the same pytorch version for multi-node training?

hoanchar: <p>Should all nodes be running on the same pytorch version for multi-node training? and/or do all nodes that I’m going to be running for multi-node training need to have the same library versions?</p>
<p>For example, if one of my nodes is running on torch==1.10.1, do all of my other nodes need to be running on torch==1.10.1 also or can it run on another torch version such as torch==1.13.1?<br>
Also, if one of my nodes is running on transformers==4.20.1, do all of my other nodes need to be running on transformers==4.20.1 also or can it run on another huggingface version such as transformers==4.26.1?</p>

ptrblck: <p>I would stick to the same versions in all libraries as I doubt anyone is testing any diverged configuration and it could thus easily break.</p>

hoanchar: <p>I see. I was wondering if you know anyone who has accidentally run different library versions for different nodes?</p>
<p>Because I accidentally run one of my experiments with different nodes with different library versions and I am not sure if that result is reliable or not.</p>
Non-text sequence classification with nn.transformer

arjsto: <p>This seems trivial given the otherwise clear tutorial, but I cannot figure out how to adjust the sequence-to-sequence modeling code (at <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Sequence-to-Sequence Modeling with nn.Transformer and TorchText — PyTorch Tutorials 1.7.1 documentation</a>) to</p>
<ol>
<li>perform sequence classification instead, and</li>
<li>to do just that for not text but multi-feature sequences.</li>
</ol>
<p>For instance, where should one make adjustments to output class predictions? And how to determine the “vocabulary” size of a series of non-text sequences? Each sequence is 5 features by 200 timepoints large, and has one label. Any pointers would be much appreciated.</p>

ali_hamidi: <p>Always, everywhere, everyone talks about text! not digits. There is also sequential numerical data, but no comments or tutorials about it.</p>
Constrained weights in MLP are not that constrained

Mystify2823: <p>Hello,</p>
<p>I have those lines of codes:</p>
<pre><code class="lang-auto">class weightConstraint(object):
    def __init__(self):
        pass
    
    def __call__(self,module):
        if hasattr(module,'weight'):
            print("Entered")
            w=module.weight.data
            w=w.clamp(0.1001,1.0)
            module.weight.data=w

model = pm.ModelCustomLayerTimeWindow(input_size, 1024 , 1)
model.window.register_forward_hook(get_activation('window'))
constraints=weightConstraint()
model._modules['window'].apply(constraints)
</code></pre>
<p>I want to constrain the value of my window layer between 0.1001 and 1.0, but during training, I see that they can take some values below 0.1001 (for example, 0.0287). The problem is when the weight of this layer take the value of 0.0287, they are stuck and doesn’t update anymore, which is the reason i am training to put the constraint</p>
<p>Is there a way to have a more constrained interval?</p>

KFrank: <p>Hi Mystify!</p>
<aside class="quote no-group" data-username="Mystify2823" data-post="1" data-topic="189263" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mystify2823/48/61564_2.png" class="avatar"> Mystify2823:</div>
<blockquote>
<p>I want to constrain the value of my window layer between 0.1001 and 1.0, but during training, I see that they can take some values below 0.1001</p>
</blockquote>
</aside>
<p>The short story is consider using pytorch’s <a href="https://pytorch.org/tutorials/intermediate/parametrizations.html" rel="noopener nofollow ugc">parametrize</a> functionality to bound<br>
your parameter.</p>
<p>I have over time come to lean against the idea of constraining parameters by<br>
modifying them by hand.  The problem is that doing things that way doesn’t<br>
really cooperate with backpropagation and optimization.  <code>opt.step()</code> can modify<br>
a parameter so that it no longer satisfies some constraint and then you reimpose<br>
that constraint by hand (and then the next <code>opt.step()</code> can move it outside of<br>
the constraint again).</p>
<p>I would suggest that you work with an unconstrained parameters that runs from<br>
<code>-inf</code> to <code>inf</code> and pass it through a <code>parametrization</code> that maps it to <code>(0.1001, 1.0)</code>.<br>
(You can use <code>sigmoid()</code> to map <code>(-inf, inf)</code> to (<code>0.0, 1.0)</code> and then use simple<br>
arithmetic to map that to <code>(0.1001, 1.0)</code>.)</p>
<p>(As an aside, <code>.data</code> has been deprecated for some time and can cause errors.<br>
I don’t know whether it is likely to cause problems in your specific code, but, in<br>
general, you shouldn’t use it.)</p>
<p>Best.</p>
<p>K. Frank</p>

Mystify2823: <p>Hi KFrank,</p>
<p>Thanks for the answer! I will try in the way you suggested to me then.</p>
Extract layer parameters

marksaroufim: <p>Hi folks I’m building a proof of concept where I detect large convolutions and break them apart into smaller ones for devices with small on device memory</p>
<p>As a first step I wanted to just detect where a large convolution may occur</p>
<p>To do so I’m using <code>torch.fx</code> to extract out <code>in_channels</code>, <code>out_channels</code> and the <code>stride</code> and if they’re larger than a certain threshold then I know I have a big convolution</p>
<p>Unfortunately I can’t seem to find where the parameters of a convolution operation are stored on the node <a href="https://github.com/pytorch/pytorch/blob/master/torch/fx/node.py" rel="noopener nofollow ugc">https://github.com/…/pytorch/blob/master/torch/fx/node.py</a></p>
<p>I know the trace is aware of the parameters if I just run <code>symbolic_trace(model)</code></p>
<p>Any suggestions? Should I be working with the <code>code</code> object instead of the <code>graph</code> object?</p>
<p>More specifically</p>
<pre><code class="lang-auto">class SplittableCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels= 32, out_channels= 64, kernel_size= 1, stride=1)
        self.conv2 = nn.Conv2d(in_channels= 64, out_channels= 128, kernel_size= 1, stride=1)
        self.conv3 = nn.Conv2d(in_channels= 128, out_channels= 32, kernel_size= 1, stride=2)
    def forward(self, x):
        output = self.conv3(self.conv2(self.conv1(x)))
        return output
</code></pre>
<pre><code class="lang-auto">model = SplittableCNN() 
symbolic_trace(model)

# outputs
SplittableCNN(
  (conv1): Conv2d(32, 64, kernel_size=(1, 1), stride=(1, 1))
  (conv2): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1))
  (conv3): Conv2d(128, 32, kernel_size=(1, 1), stride=(2, 2))
)
</code></pre>
<p>I want to extract the <code>in_channels</code> and <code>out_channels</code> and at a later stage change them<br>
`</p>

Chillee: <p>The module isn’t stored on the node itself, but the node contains the qualified name of the module. So you can use the qualified name to query the module itself to obtain the actual module, with which you can just query the parameters.</p>
<p>See <a href="https://github.com/pytorch/pytorch/blob/master/torch/fx/experimental/optimization.py#L69" class="inline-onebox" rel="noopener nofollow ugc">pytorch/optimization.py at master · pytorch/pytorch · GitHub</a></p>

marksaroufim: <p>This was really cool</p>
<pre><code>trace = symbolic_trace(model)
modules = dict(trace.named_modules())
</code></pre>
<p><code>modules['conv1'].in_channels</code> → would correctly output 32</p>

parikshit: <p>Is there a way to extract the associated parameters like weights and biases from a node/graph using the above approach?</p>
Problem to install pytorch cuda12.2 with conda

Mahmoud_Ghorbel: <p>Hello<br>
i’m trying to install pytorch-gpu for cuda12.2 but i can’t do it.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c77e596b3a5b5a7a3c3ffef650a4593a206553ec.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c77e596b3a5b5a7a3c3ffef650a4593a206553ec" title="nvidia-smi"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c77e596b3a5b5a7a3c3ffef650a4593a206553ec_2_690x399.jpeg" alt="nvidia-smi" data-base62-sha1="ssNIyNsyof3gc3OvvwK4YBYDcEA" width="690" height="399" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c77e596b3a5b5a7a3c3ffef650a4593a206553ec_2_690x399.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c77e596b3a5b5a7a3c3ffef650a4593a206553ec_2_1035x598.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c77e596b3a5b5a7a3c3ffef650a4593a206553ec.jpeg 2x" data-dominant-color="1F1F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nvidia-smi</span><span class="informations">1103×638 98.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
i did try with:<br>
<code>conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch-nightly -c nvidia</code><br>
but when i check the device with :</p>
<pre><code class="lang-auto">device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")
</code></pre>
<p>i get cpu</p>
Pytorch equivalent of this tensorflow focal loss

granth_jain: <p>This implementation of focal loss in tensorflow is giving good results with my dataset.</p>
<pre><code class="lang-auto">from keras import backend as K

def focal_loss(gamma=2., alpha=.25):
	def focal_loss_fixed(y_true, y_pred):
		pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))
		pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))
		return -K.mean(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1+K.epsilon())) - K.mean((1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0 + K.epsilon()))
	return focal_loss_fixed
</code></pre>
<p>I want to test in pytorch, so I created custom loss function in pytorch as below, but seems getting error.</p>
<pre><code class="lang-auto">def focal_loss(y_true, y_pred, gamma=2.0, alpha=0.25):
    pt_1 = torch.where(y_true == 1, y_pred, torch.ones_like(y_pred))
    pt_0 = torch.where(y_true == 0, y_pred, torch.zeros_like(y_pred))
    epsilon = 1e-7

    loss = -torch.mean(alpha * (1.0 - pt_1).pow(gamma) * torch.log(pt_1 + epsilon))
    loss -= torch.mean((1 - alpha) * pt_0.pow(gamma) * torch.log(1.0 - pt_0 + epsilon))

    return loss
</code></pre>
<p><code>clf = ts_learner(dls, loss_func=focal_loss, arch="InceptionTimePlus", metrics=[precision, recall], cbs=[save_best_model])</code></p>
<p>can somebody please let me know how to use this focal loss in above code that uses pytorch.</p>
How to train non quantized layers of quantized model on GPU

parikshit: <p>Hi,</p>
<p>I am trying to use resnet50 quantized model for transfer learning and training the last fully connected layer. From the documentation what I figured was quantized model can only be trained on CPUs but the non-quantized layer can be trained on GPU but how to transfer this layer to GPU? Cannot find much related to this anywhere.</p>
<p>Model looks like:</p>
<pre><code class="lang-auto">def create_combined_model(model_fe):
  # Step 1. Isolate the feature extractor.
  model_fe_features = nn.Sequential(
    model_fe.quant,  # Quantize the input
    model_fe.conv1,
    model_fe.bn1,
    model_fe.relu,
    model_fe.maxpool,
    model_fe.layer1,
    model_fe.layer2,
    model_fe.layer3,
    model_fe.layer4,
    model_fe.avgpool,
    model_fe.dequant,  # Dequantize the output
  )

  # Step 2. Create a new "head"
  new_head = nn.Sequential(
    nn.Dropout(p=0.5),
    nn.Linear(num_ftrs, num_classes),
  )

  new_model = nn.Sequential(
    model_fe_features,
    nn.Flatten(1),
    new_head,
  )
  return new_model
</code></pre>

jerryzh168: <p>you can move a layer to GPU by: <code>model.layer.cuda()</code> I think? it probably won’t work if you have mixed cpu/cuda layers I feel</p>
Number of epochs in DistributedDataParallel

haisamrafid: <p>Hi,</p>
<p>I wrote a program recently to train an AlexNet model on ImageNet dataset. I don’t want to share the code for now. I just want to make sure I understand something correctly. I didn’t find the answer anywhere. I understand that if my original batch size is for example 128, for parallel training I need to set the batch size to 256 given I use 2 GPUs. But, I am confused about the number of epochs. If I want to train the model for 90 epochs, do I set the number of epochs to 45 for 2 GPUs?</p>

ptrblck: <p>Your script should still have a main training loop where the number of epochs are defined.<br>
E.g. take a look at the <a href="https://github.com/pytorch/examples/blob/cead596caa90600188e1055cd9166ab4e7dfd303/imagenet/main.py#L273">ImageNet example</a> which also uses <code>DDP</code> if enabled.</p>
Conv2d backward pass using unfold+mm

Karthik_Ganesan: <p>Hi,<br>
I am trying to implement the backward pass for Conv2d using unfold+mm. My first implementation used <code>torch.nn.grad.conv2d_input</code>, which works correctly. That function internally calls <code>torch.conv_transpose2d</code>. For this, I am using the code here to implement <code>conv2d_transpose</code>: <a href="https://discuss.pytorch.org/t/convtranspose2d-using-unfold/120350/4" class="inline-onebox">ConvTranspose2d using unfold - #4 by santacml</a>.</p>
<p>However, I am running into an issue and I’m not sure how to proceed. I have provided my code below, including the forward pass for comparison.</p>
<pre><code class="lang-auto">def forward(ctx, inp, weight, bias=None, stride=(1, 1), padding=(0, 0), dilation=(1, 1), groups=1):
	ctx.save_for_backward(inp, weight, bias)
	# Removed code showing stide, padding, dilation and groups being saved to ctx.

	(b, n_C_prev, n_H_prev, n_W_prev) = inp.shape
	(n_oC, n_iC, f, f) = weight.shape

	n_H = ((n_H_prev - f + (2 * padding[0])) // stride[0]) + 1
	n_W = ((n_W_prev - f + (2 * padding[1])) // stride[1]) + 1

	inp_padded = torch.nn.functional.pad(inp, (padding[0], padding[0], padding[1], padding[1]))
	inp_unf = torch.nn.functional.unfold(inp_padded, (f, f), padding=padding, stride=stride)

	inp_unf = inp_unf.transpose(1, 2)
	inp_unf = inp_unf.reshape(-1, inp_unf.shape[-1])
	weight_t = weight.view(weight.size(0), -1).t()

	out_unf = torch.matmul(inp_unf, weight_t)
	out_unf = out_unf.view(inp.shape[0], -1, out_unf.shape[-1])
	out_unf = out_unf.transpose(1, 2)
	output = out_unf.view(b, n_oC, n_H, n_W) + bias[:, None, None]
	return output

def backward(ctx, grad_output):  # grad_output.shape = [100,20,8,8]
	inp, weight, bias = ctx.saved_tensors # weight.shape = [20, 10, 5, 5]
	# Removed code which restores stride, padding, dilation &amp; groups from forward.
	grad_input = None	

	if ctx.needs_input_grad[0]:
		# This code works correctly and trains the CNN. I am using it as a comparison for my code.
		# grad_input_check.shape = [100, 10, 12, 12]
		grad_input_check = torch.nn.grad.conv2d_input(inp.shape, weight, grad_output, stride, padding, dilation, groups)

		# Does the same thing as `_grad_input_padding` in `torch/nn/grad.py`		
		gi_padding = grad_input_padding(grad_output, inp.shape, stride, padding, ks, dilation)

		go_unf = torch.nn.functional.unfold(grad_output, ks, padding=gi_padding) # [100, 500, 16]
		go_unf = go_unf.transpose(1, 2)	# [100, 16, 500]

		w_rot = torch.rot90(weight, 2, [2, 3]) # [20, 10, 5, 5]
		w_rot = w_rot.view(w_rot.size(0), -1).t()	# [250, 20]

		grad_input = go_unf.mm(w_rot)
</code></pre>
<p>Ofcourse, I cannot multiply <code>go_unf</code> and <code>w_rot</code> with those shapes. I think the problem stems from the number of channels in <code>grad_output</code>. Working backwards, it seems I need a size of <code>[100,250,72]</code> for the output of <code>unfold</code> to get the right dimensions. Any help would be much appreciated! thank you!</p>
Ninja: build stopped: subcommand failed, FbgemmFP16 error

dangonnadan: <p>Hi,<br>
I am facing a problem when installing pytorch 1.5.0 from source on Windows 10. At the stage of building files, the following error occurs:  " FAILED: third_party/fbgemm/CMakeFiles/fbgemm_generic.dir/src/FbgemmFP16.cc.obj" .  My gcc version is 6.3.0. Can you guess what is the problem?</p>
MyModel() object has no attribute 'load_state_dict'

ZG12: <p>Hi. I am trying to load a model to test it with:</p>
<pre><code class="lang-auto">model = MyModel()
model.load_state_dict(torch.load(PATH, map_location=torch.device('cpu')))
</code></pre>
<p>But it says :</p>
<p><code> 'MyModel' object has no attribute 'load_state_dict'</code></p>
<p>What should I do ?</p>

ptrblck: <p>Did you derive <code>MyModel</code> from <code>nn.Module</code> or from another base class?<br>
Make sure to derive it as:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        ...
</code></pre>
Torch.cuda.memory_allocated() returns 0 if PYTORCH_NO_CUDA_MEMORY_CACHING = 1

harsanyidani: <p>There are clearly tensors allocated in my GPU memory. When I turn PYTORCH_NO_CUDA_MEMORY_CACHING enviroment variable back to 0 it works seemingly fine. Is this a bug?</p>
<p>I’ve read <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management" rel="noopener nofollow ugc">pytorch documentation on memory management</a> but I still don’t understand.</p>

ptrblck: <p>Disabling the caching allocator is a debugging feature and some utils won’t work, such as CUDA Graphs. You could suggest a fix in case you are interested to see the used memory stats.</p>

harsanyidani: <p>Should I use <a href="https://pytorch.org/docs/stable/generated/torch.cuda.mem_get_info.html" rel="noopener nofollow ugc">torch.cuda.mem_get_info()</a> in this case? What’s the difference ? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p><code>torch.cuda.mem_get_info()</code> uses the CUDA runtime API and reports the free and total memory of the specified device, which will include other processes besides the PyTorch script as well. You can of course use it, but have to make sure to read the results right and to subtract the memory usage from other processes. Also, no caching information will be given, but you are already disabling it explicitly.</p>
LOADING A TORCHSCRIPT CMakeLists build error!

DonghunP: <p>Hi guys. I follow the tutorial below.</p>
<p><a href="https://pytorch.org/tutorials/advanced/cpp_export.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_export.html</a></p>
<p>I download libtorch/ and below is my <strong>CMakeLists.txt</strong></p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(custom_ops)

find_package(Torch REQUIRED)

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)
</code></pre>
<p>during cmake build I got an error below</p>
<pre><code class="lang-auto">pytorch_TorchScript/
 ㄴlibtorch/
 ㄴ build/
 ㄴexample-app.cpp

$ cd build
$ cmake -DCMAKE_PREFIX_PATH='/home/dhpark/pytorch_TorchScript/libtorch' ..
</code></pre>
<p>Error)</p>
<pre><code class="lang-auto">pytorch_TorchScript/build# cmake -DCMAKE_PREFIX_PATH='/home/dhpark/pytorch_TorchScript/libtorch' ..
-- Caffe2: CUDA detected: 11.6
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.6
-- Found cuDNN: v8.5.0  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 4dd39364
-- Autodetected CUDA architecture(s):  8.6 8.6 8.6 8.6
-- Added CUDA NVCC flags for: -gencode;arch=compute_86,code=sm_86
CMake Warning at libtorch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library c10_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  libtorch/share/cmake/Torch/TorchConfig.cmake:70 (append_torchlib_if_found)
  CMakeLists.txt:8 (find_package)


CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find Torch (missing: TORCH_LIBRARY)
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  libtorch/share/cmake/Torch/TorchConfig.cmake:190 (find_package_handle_standard_args)
  CMakeLists.txt:8 (find_package)
</code></pre>
<p>I think the reason of the problem is<br>
’ static library c10_LIBRARY-NOTFOUND not found.’</p>
<p>I don’t know how to solve it.</p>
<p>Can anyone help ??</p>
<p>Thanks alot.</p>

ptrblck: <p>It seems that <code>cmake</code> has trouble finding <code>libtorch</code> in your setup. Make sure the <code>libtorch</code> folder contains all expected files and maybe move it out of your project dir.</p>

li-2132: <p>I met the same problem, did you solve this problem?</p>
Spatial CNN architecture Loss Oscillation

DrDumbenstein: <p>Hi Guy… I’m having this problem when I use a slice by slice convolution I Get the Losses so Oscillating. What may cause this?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3f092b7055c3c22a143136ebf2d8aff03c52070.png" alt="download" data-base62-sha1="ueUaEgwoN2xN0azIQQtNVFIO3aE" width="576" height="455"></p>
Classification linear model from scratch

JustQerq: <p>Short intro: I don’t know a lot about the math behind machine learning, but I think I understand the basics and the general idea of it. At the moment I am following along the fastai’s course/book and decided that I’d like to write a multi-class linear classification model from scratch as a learning excercise.</p>
<p>I have a working version of this simple model which classifies 28*28 pictures of hand-written digits from the MNIST dataset. The highest accuracy I’ve been able to achieve is ~58%. Here’s a link to it on Kaggle: <a href="https://www.kaggle.com/code/justqerq/digits-recognizer?scriptVersionId=144893616" class="inline-onebox" rel="noopener nofollow ugc">Digits recognizer | Kaggle</a></p>
<p>I would like to ask a few questions in order to understand what I could improve on:</p>
<ol>
<li>
<p>As I understand it, a model with random weights should have an accuracy of about 10%, so an accuracy of 58% should indicate that my model is doing something right and isn’t just blindly guessing, correct?</p>
</li>
<li>
<p>If my model is indeed working, how good is the accuracy of 58%? What is the reasonable accuracy one could expect from a model with a singular layer and no non-linearities?</p>
</li>
<li>
<p>I have picked learning rates at random and chose the ones that gave good accuracy. What better way of picking learning rates would you suggest?</p>
</li>
</ol>
<p>And finally, I would really appreciate it if someone were to look at my model and give some advice on how to improve it or my coding style. I am more than willing to provide some comments/explanations about my model in case my code is confusing.</p>

ptrblck: <p>Based on your code it seems you are using a few manual implementations, which might suffer from numerical stability issues. E.g. replace the <code>softmax().log()</code> with <code>F.log_softmax</code> for a better numerical stability. Also, could you explain why <code>argmin</code> is used to compute the predictions instead of <code>argmax</code>?</p>

JustQerq: <p>Thank you for the suggestion.<br>
I am aware of the built-in <strong>log_softmax</strong> function, but decided to write a manual implementation to see how well it would work. I had to introduce some correction terms such as <strong>b</strong> in <strong>softmax()</strong> to deal with <em>exp(x) = inf</em> overflow and <strong>epsilon</strong> in <strong>nll()</strong> for <em>log(0) = -inf</em>. Are there any other numerical stability issues I am not aware of?</p>
<p>As for the usage of <strong>argmin()</strong>: <strong>nll()</strong> function, the negative log likelihood, returns only positive values, where 0 corresponds to 100% confidence/probability and <em>+inf</em> to 0 confidence, so I use <strong>argmin()</strong> during the validation phase to pick the lowest value, which is the model’s “best guess” and see if it was correct. That’s how I understood it based on chapter 5 in fastai’s book (<a href="https://github.com/fastai/fastbook/blob/master/05_pet_breeds.ipynb" rel="noopener nofollow ugc">https://github.com/fastai/fastbook/blob/master/05_pet_breeds.ipynb</a>), please correct me if I’m wrong.</p>
Cudf for python 3.11

eugenio_b: <p>Hi everybody,<br>
I am using python 3.11<br>
I am trying to speed up csv loading.<br>
Up to now I do the following operations</p>
<ul>
<li>load csv via csv_read (pandas)</li>
<li>copy from dataframe to torch.tensor</li>
<li>manage data.<br>
Of course via gpu processing is considerable faster, but loading and copy is the most time-wasting operation.<br>
I found there is cudf, a cuda package to load csv directly to GPU, but it seems to be unsupported for my python version.<br>
Did some of you have some good news for me?<br>
Cheers</li>
</ul>
<p>Eugenio</p>

ptrblck: <aside class="quote no-group" data-username="eugenio_b" data-post="1" data-topic="189225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eugenio_b/48/63658_2.png" class="avatar"> eugenio_b:</div>
<blockquote>
<p>I found there is cudf, a cuda package to load csv directly to GPU, but it seems to be unsupported for my python version.</p>
</blockquote>
</aside>
<p>You might want to contact the maintainers of this package directly in their repository to see if this Python version is supported or if the support is planned.</p>
Exception with opacus library

sam435: <p>I am using integrated gradients for feature attributions on a model trained using DP-SGD with opacus library. However, calling attribute() throws the exception below. Calling the same on a non-private model works without any issues. Any hints on why captum throws this error on a private model will be much appreciated.</p>
<pre><code class="lang-auto">attributions_train, delta_train = ig.attribute(input_train, baseline_train, target=0, return_convergence_delta=True)
                         
  File "/lib/python3.11/site-packages/captum/log/__init__.py", line 42, in wrapper
    return func(*args, **kwargs)
       
  File "/lib/python3.11/site-packages/captum/attr/_core/integrated_gradients.py", line 286, in attribute
    attributions = self._attribute(
        
  File "/lib/python3.11/site-packages/captum/attr/_core/integrated_gradients.py", line 351, in _attribute
    grads = self.gradient_func(
        
  File "/lib/python3.11/site-packages/captum/_utils/gradient.py", line 119, in compute_gradients
    grads = torch.autograd.grad(torch.unbind(outputs), inputs)
        
  File "/lib/python3.11/site-packages/torch/autograd/__init__.py", line 303, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  
  File "/lib/python3.11/site-packages/torch/nn/modules/module.py", line 69, in __call__
    return self.hook(module, *args, **kwargs)
      
  File "/lib/python3.11/site-packages/opacus/grad_sample/grad_sample_module.py", line 326, in capture_backprops_hook
    activations, backprops = self.rearrange_grad_samples(
        
  File "/python3.11/site-packages/opacus/grad_sample/grad_sample_module.py", line 388, in rearrange_grad_samples
    activations = module.activations.pop()
        
IndexError: pop from empty list
</code></pre>
Split Cifar10(train part) dataset of torchvision to train and validation without shuffling

Azadeh_Ashouri: <p>Hello everyone.<br>
How can I split Cifar10 dataset of torch-vision without shuffling?<br>
I appreciate your help.</p>

ptrblck: <p>You could create the training and validation indices (without shuffling) and pass them to <code>Subset</code>s to create the dataset splits.</p>
Not match the results of the official DQN examples

ssn: <p>Hello, everyone. I’m new to the RL, so I used the official examples to test the RL codes. For the “CartPole-v1” problem, I learn the codes from this page <a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" rel="noopener nofollow ugc">DQN example</a>, we can see it shows the output with small duration at the end.<br>
But when I test it on the Colab or in my own computer, I found it show the wrong big duration at the end whether by cpu or gpu.<br>
How to solve this problem, I’m so confused for the code example.</p>

ssn: <p>this is the colab and my computer’s output<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/314effce52a24be8da1785b8609299487ba4c183.png" alt="new-dqn" data-base62-sha1="72cEn0Ed42ojQz5OriE5XkY6lsn" width="571" height="455"></p>
Debugging PyTorch CUDA Extensions

Jerome_Ku: <p>Are there tools / best practices for mixed <code>python</code> / <code>cpp</code> / <code>cuda</code> debugging for torch CUDA Extensions (extensions built with <code>CUDAExtension</code>)?  I.e., I’d like to run / debug a PyTorch program from Python and then step into a debugger such as <code>cuda-gdb</code> when the extension is called.</p>
<p>Any recommendations greatly appreciated!</p>

ptrblck: <p>You could launch your script with <code>cuda-gdb</code> and set a breakpoint in your extension.</p>
Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead

Bond_Girl_007: <p>Hi I am trying to work through this tutorial: <a href="https://pytorch.org/tutorials/intermediate/reinforcement_ppo.html" class="inline-onebox" rel="noopener nofollow ugc">Reinforcement Learning (PPO) with TorchRL Tutorial — PyTorch Tutorials 2.0.1+cu117 documentation</a> on an M2 MacBook.</p>
<p>I have installed the following:</p>
<pre><code class="lang-auto">conda create --name gpu-torch-rl python=3.10 spyder=5.4.3
    conda activate gpu-torch-rl
    
    git clone https://github.com/pytorch/rl /Users/bondgirl007/anaconda3/repos/rl
    
    cd /Users/bondgirl007/anaconda3/repos/rl/
    pip install -e .
    
    ImportError: cannot import name 'unravel_key' from 'tensordict' (/Users/myname/anaconda3/envs/gpu-torch-rl/lib/python3.10/site-packages/tensordict/__init__.py) 
    pip uninstall tensordict
    pip install git+https://github.com/pytorch-labs/tensordict
    
    Supported version of '_set_gym_args' has not been found.
    pip install gym --upgrade
    
    DependencyNotInstalled: No module named 'mujoco'. (HINT: you need to install mujoco)
    pip install mujoco
</code></pre>
<p>I have an issue running the following line of code:<br>
<code>base_env = GymEnv("InvertedDoublePendulum-v4", device="mps", frame_skip=frame_skip)</code><br>
I saw there was a suggestion by the library developers in <a href="https://github.com/pytorch/rl/issues/1198:" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/rl · GitHub</a><br>
Add a <code>DoubleToFloat</code> transform after creating the env on cpu and then map the data with <code>DeviceCastTransform</code> to the MPS device but I don’t understand what syntax to use for that. So I am trying:</p>
<pre><code class="lang-auto">base_env = GymEnv("InvertedDoublePendulum-v4", device="cpu", frame_skip=frame_skip)
base_env = DoubleToFloat(base_env)
base_env = DeviceCastTransform(base_env, device="mps", orig_device="cpu")
</code></pre>
<p>But I have an error:<br>
<code>TypeError: DeviceCastTransform.__init__() got multiple values for argument 'device'</code></p>
<p>What is the correct syntax to be able to use mps?</p>
<p>Is there anything someone could recommend please?</p>

soulitzer: <p>You might be looking for<br>
<a href="https://pytorch.org/rl/reference/generated/torchrl.envs.transforms.DeviceCastTransform.html" class="inline-onebox" rel="noopener nofollow ugc">DeviceCastTransform — torchrl main documentation</a>?</p>
Freeze few blocks in convolution layer

just_started_coding: <p>For training resnet, I want to freeze some convolution blocks in the first conv layer. So for:<br>
<code>self.conv1 = nn.Conv2d(3,64, kernel_size=3, stride=1, padding=1, bias=False)</code> - This layer has about 64 filters in total and I want to freeze say half of the filters (say first 32) and let the other half train. How can I do this?</p>

soulitzer: <p>You can register a backward hook to the weight that zeros out the gradient corresponding to the parts of the weights that you don’t want to train.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html?highlight=register_hook#torch.Tensor.register_hook" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html?highlight=register_hook#torch.Tensor.register_hook</a></p>

KFrank: <p>Hi Just (and Soulitzer)!</p>
<aside class="quote no-group" data-username="just_started_coding" data-post="1" data-topic="189191" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/fbc32d/48.png" class="avatar"> just_started_coding:</div>
<blockquote>
<p>This layer has about 64 filters in total and I want to freeze say half of the filters (say first 32) and let the other half train.</p>
</blockquote>
</aside>
<p>I would caution <em>against</em> zeroing out gradients (whether with a hook or by some<br>
other method) as a general technique for freezing weights.  The reason is that<br>
many optimizers (for example, <code>SGD</code> with weight decay) will modify parameters<br>
even if their gradients are zero.</p>
<p>(If you are using <em>plain-vanilla</em> <code>SGD</code>, with no momentum and no weight decay,<br>
zeroing the gradients will freeze the corresponding parameters.)</p>
<p>For the specific use case you describe, you could either:</p>
<ol>
<li>
<p>Split <code>conv1</code> into two <code>Conv2d (3, 32)</code>'s (and <code>cat</code> their outputs back<br>
together to pass into a subsequent <code>Conv2d (64, out channels)</code>)<br>
and leave the second <code>Conv2d (3, 32)</code> out of the optimizer.</p>
</li>
<li>
<p>Use a single <code>Conv2d (3, 64)</code>, save copies of the parameter values<br>
you want frozen, call <code>optimizer.step()</code>, and copy back the frozen<br>
parameter values.</p>
</li>
</ol>
<p>Best.</p>
<p>K. Frank</p>

soulitzer: <p>Good point, however, if you zeroed out your weights wrt those parameters from the very beginning, I wouldn’t expect any momentum to build up because the gradients have always been zero.</p>
TypeError: linear(): argument 'input' (position 1) must be Tensor, not NoneType

anthonysirico: <p>Here is the model:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin0 = torch.nn.Linear(1, dim, dtype=torch.float64)

        nn = Sequential(Linear(5, 128, dtype=torch.float64), ReLU(), Linear(128, dim * dim, dtype=torch.float64))
        self.conv = NNConv(dim, dim, nn, aggr='mean')
        self.gru = GRU(dim, dim)

        self.set2set = Set2Set(dim, processing_steps=3)
        self.lin1 = torch.nn.Linear(2 * dim, dim, dtype=torch.float64)
        self.lin2 = torch.nn.Linear(dim, 1, dtype=torch.float64)

    def forward(self, data):
        out = F.relu(self.lin0(data.x))
        h = out.unsqueeze(0)

        for i in range(3):
            m = F.relu(self.conv(out, data.edge_index))
            out, h = self.gru(m.unsqueeze(0), h)
            out = out.squeeze(0)

        out = self.set2set(out, data.batch)
        out = F.relu(self.lin1(out))
        out = self.lin2(out)
        return out.view(-1)
</code></pre>
<p>Here is the dimensions of the data: <code>Data(edge_index=[2, 16], weight=[16], x=[14, 1], catalog=[1], compiled=[1], simulated=[1], Utility=[1], Objectives=[5])</code></p>
<p><code>out</code> is type <code>torch.float64</code> and <code>data.edge_index</code> is <code>torch.int64</code> as its supposed to be, but why am I still getting <code>TypeError: linear(): argument 'input' (position 1) must be Tensor, not NoneType</code>?</p>
<p>Lastly here is the run code:</p>
<pre><code class="lang-auto">def train(epoch):
    model.train()
    loss_all = 0

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        loss = F.mse_loss(model(data), data.Utility)
        loss.backward()
        loss_all += loss.item() * data.num_graphs
        optimizer.step()
    return loss_all / len(train_loader.dataset)


def test(loader):
    model.eval()
    error = 0

    for data in loader:
        data = data.to(device)
        error += (model(data) * std - data.Utility * std).abs().sum().item()  # MAE
    return error / len(loader.dataset)


best_val_error = None
for epoch in range(1, 301):
    lr = scheduler.optimizer.param_groups[0]['lr']
    loss = train(epoch)
    val_error = test(val_loader)
    scheduler.step(val_error)

    if best_val_error is None or val_error &lt;= best_val_error:
        test_error = test(test_loader)
        best_val_error = val_error

    print(f'Epoch: {epoch:03d}, LR: {lr:7f}, Loss: {loss:.7f}, '
          f'Val MAE: {val_error:.7f}, Test MAE: {test_error:.7f}')
</code></pre>
Training with freeze weights

Trevor_Yap: <p>Hi all,</p>
<p>I have a model with freeze weights. I would like to train without updating them.<br>
From what i saw in this post: <a href="https://discuss.pytorch.org/t/correct-way-to-freeze-layers/26714/2">how to freeze weight correctly</a>,  I can simply change the <code>weight.require_grad</code> to <code>False</code> and set the optimizer in the following ways:<br>
<code>optimizer = optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)</code></p>
<p>May I ask if I uses <code>model.train()</code><br>
Will this be affect the training and cause the freeze weights to update?? or i can safely run this?</p>

soulitzer: <p>Setting eval/train for a module will not make it so that <code>requires_grad=False</code> weights will update. <code>model.train()</code>/ <code>model.eval()</code> actually toggles the behavior of certain modules like batch norm and drop out have different behavior, e.g. whether to update running mean/var or drop activations, so if you think you have any of those types of modules you will see a behavioral difference.</p>
<p>For more information, you can see <a href="https://pytorch.org/docs/stable/notes/autograd.html#evaluation-mode-nn-module-eval" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 2.0 documentation</a></p>

Trevor_Yap: <p>Thank you for your reply. Just to clarify so if i set <code>optimizer = optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)</code><br>
the Batch norm will not update right?</p>

ptrblck: <p>If you’ve set the <code>.requires_grad</code> attribute of all batchnorm layers to <code>False</code>, you are correct. Otherwise refer to <a class="mention" href="/u/soulitzer">@soulitzer</a>’s post explaining that the <code>self.training</code> attribute, toggled via <code>model.train()/.eval()</code> will not (un)freeze the trainable parameters.</p>
What Param.requires_grad_ do

Ema: <p>Hi,<br>
I will appreciate your help in clarifying my question.<br>
What is the difference between Para.requires_grad_ =false<br>
and Param.reruires_grad=false (without under score).<br>
Each one  gives me different results. Requires_grad reduces the training time while the other one doesn’t.</p>
<pre><code class="lang-auto">for param in model.parameters():
    param.requires_grad_=True
</code></pre>

J_Johnson: <p>See here: <a href="https://github.com/pytorch/pytorch/blob/d0ad848aa55aae7afdbde22d7cbbf55045f4b26b/torch/nn/modules/module.py#L2420" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/d0ad848aa55aae7afdbde22d7cbbf55045f4b26b/torch/nn/modules/module.py#L2420</a></p>
<p>From the looks of it, that just deals with modules that contain multiple separate parameters. It only gets called under certain circumstances and just sets .requires_grad for all of the individual parameters.</p>
<p>Here is some sample code to demonstrate how you might use .requires_grad_:</p>
<pre><code class="lang-auto">import torch.nn as nn

class TestModel(nn.Module):
    def __init__(self):
        super().__init__()

        self.layer1 = nn.Linear(5, 10)
        self.layer2 = nn.Linear(10, 20)
        self.layer3 = nn.Linear(20, 1)

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return self.layer3(x)


model = TestModel()

model.requires_grad_(False)

for param in model.parameters():
    print(param.requires_grad)
</code></pre>
<p>Note, that one call set all of the layers to <code>requires_grad = False</code>.</p>

Ema: <p>Thank you for your reply .</p>