Now I just want to train the two Weight of linear layer, w(w1, w2, w3,…… W100), now just want to train W1, w2, how to choose these two weights to train?

xiaojin: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aa15ae7f624914b5f2e86504b97b71af0b1a213.png" data-download-href="https://discuss.pytorch.org/uploads/default/6aa15ae7f624914b5f2e86504b97b71af0b1a213" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aa15ae7f624914b5f2e86504b97b71af0b1a213_2_690x282.png" alt="image" data-base62-sha1="fdijroHcvzS1CyckJqzqbWjpyrp" width="690" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aa15ae7f624914b5f2e86504b97b71af0b1a213_2_690x282.png, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aa15ae7f624914b5f2e86504b97b71af0b1a213.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aa15ae7f624914b5f2e86504b97b71af0b1a213.png 2x" data-dominant-color="EFEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">848×347 42.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Pranavan_Theivendira: <p>Hi ,</p>
<p>The linear layer is actually a matrix(n_feature x n_hidden1), is not a vector. I show below how I can stop training of W[0,0].</p>
<pre><code class="lang-auto">model.hidden1.weight[0, 0].requires_grad = False
</code></pre>
<p>Likewise, you can make other weights to False if you do not want to learn them.</p>
<p><strong>EDIT:</strong> Frank pointed a <strong>mistake</strong> in this solution. <code>requires_grad</code> can only be applied for a whole tensor, not on an individual element.</p>
<p>Thanks</p>

KFrank: <p>Hi Pranavan (and Jin)!</p>
<aside class="quote no-group" data-username="Pranavan_Theivendira" data-post="2" data-topic="136250" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pranavan_theivendira/48/12366_2.png" class="avatar"> Pranavan_Theivendira:</div>
<blockquote>
<pre><code class="lang-auto">model.hidden1.weight[0, 0].requires_grad = False
</code></pre>
</blockquote>
</aside>
<p>No, this won’t do what you want.  <code>requires_grad</code> applies to entire<br>
tensors and not individual elements.  And to prevent unexpected<br>
errors, pytorch won’t let you do this.</p>
<p>Probably the easiest way to freeze part of a tensor is to store the<br>
value of the element you want frozen, let the optimizer modify the<br>
entire tensor, and then restore the original value of the element<br>
you want frozen.</p>
<p>Thus:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt;
&gt;&gt;&gt; lin = torch.nn.Linear (2, 3)
&gt;&gt;&gt; lin.weight[0, 0].requires_grad = False
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
&gt;&gt;&gt;
&gt;&gt;&gt; opt = torch.optim.SGD (lin.parameters(), lr = 0.1)
&gt;&gt;&gt;
&gt;&gt;&gt; lin.weight
Parameter containing:
tensor([[-0.5226,  0.0189],
        [ 0.3430,  0.3053],
        [ 0.0997, -0.4734]], requires_grad=True)
&gt;&gt;&gt; w00 = lin.weight[0, 0].item()
&gt;&gt;&gt; lin (torch.ones (2)).sum().backward()
&gt;&gt;&gt; opt.step()
&gt;&gt;&gt; lin.weight
Parameter containing:
tensor([[-6.2264e-01, -8.1068e-02],
        [ 2.4303e-01,  2.0531e-01],
        [-3.3811e-04, -5.7337e-01]], requires_grad=True)
&gt;&gt;&gt; with torch.no_grad():
...     lin.weight[0, 0] = w00
...
&gt;&gt;&gt; lin.weight
Parameter containing:
tensor([[-5.2264e-01, -8.1068e-02],
        [ 2.4303e-01,  2.0531e-01],
        [-3.3811e-04, -5.7337e-01]], requires_grad=True)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

xiaojin: <p>The idea provided is very useful, the problem has been solved, thank you very much.</p>

xiaojin: <p>*The method provided is very useful, the problem has been solved, thank you very much.</p>

Pranavan_Theivendira: <p>Hi Frank,</p>
<p>It is a mistake. Thanks for pointing out.</p>
<p>But just have a quick question about storing the weights and retrieving them back after training. Weights w3-w100 are used for training the whole network ie, these weights influence the training of some other weights in the network. This is not intended from the original question.</p>
<p>The gradients should not flow through the weights from w3-w100 in an ideal training.</p>
<p>Thanks</p>