Query: Confusion regarding requires_grad default state when defining a layer

rao: <p>Hi,<br>
New here. This may be silly but I am unsure about the default value of requires_grad for parameters of a nn object.</p>
<p>For example:</p>
<pre><code class="lang-auto">linear = nn.Linear(10,10)
linear.train()
for params in linear.parameters():
    print('requires_grad of parameters -',params.data.requires_grad)
    print('requires_grad of output of layer - ',linear(torch.tensor([0.]*10)).requires_grad)
    break
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">requires_grad of parameters - False
requires_grad of output of layer -  True
</code></pre>
<p>Why do the parameters of a layer in train mode have requires_grad as false? Why does the output of the same layer have requires_grad true? I am confused. Just started pytorch so I am unsure if I am making some instantiation mistake.</p>
<p>Also, nothing changes when the layer is being evaluated:</p>
<pre><code class="lang-auto">linear = nn.Linear(10,10)
linear.eval()
for params in linear.parameters():
    print('requires_grad of parameters -',params.data.requires_grad)
    print('requires_grad of output of layer - ',linear(torch.tensor([0.]*10)).requires_grad)
    break
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">requires_grad of parameters - False
requires_grad of output of layer -  True
</code></pre>
<p>Thanks</p>

KFrank: <p>Hi Rao!</p>
<aside class="quote no-group" data-username="rao" data-post="1" data-topic="183369" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rao/48/61834_2.png" class="avatar"> rao:</div>
<blockquote>
<pre><code class="lang-auto">    print('requires_grad of parameters -',params.data.requires_grad)
</code></pre>
</blockquote>
</aside>
<p>Get rid of <code>.data</code> and try <code>params.requires_grad</code>.</p>
<p><code>.data</code> has been deprecated for quite some time and can break things<br>
so you shouldn’t be using it.</p>
<p>(<code>.data</code>, in effect, digs, the raw – without <code>requires_grad = True</code> –  tensor<br>
out of its wrapper tensor that carries <code>requires_grad = True</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

rao: <p>Hi Frank,</p>
<p>Thanks for the suggestion. It all makes sense now!</p>
<p>Also, if I wanted to manually set the values of the weights, is .data still not the way to go? I have been doing something like this in my code to manually update the weights.</p>
<pre><code class="lang-auto">for param in DNN1.parameters():
    param.data = theta_t[start:end].view(param.shape)
</code></pre>
<p>theta_t is just a tensor I generate from another process.</p>
<p>Rao</p>

KFrank: <p>Hi Rao!</p>
<aside class="quote no-group" data-username="rao" data-post="3" data-topic="183369" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rao/48/61834_2.png" class="avatar"> rao:</div>
<blockquote>
<pre><code class="lang-auto">for param in DNN1.parameters():
    param.data = theta_t[start:end].view(param.shape)
</code></pre>
</blockquote>
</aside>
<p>Again, <code>.data</code> is deprecated.  It could, potentially, be removed from pytorch.<br>
There is also no guarantee that it will do what you want.</p>
<p>The approved idiom for this would be to use a <code>no_grad():</code> block and an<br>
inplace <code>copy_()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    for param in DNN1.parameters():
        param.copy_ (theta_t[start:end].view (param.shape))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>