Requires_grad and leaf nodes

username: Mahmoud_Abdelkhalek
cooked: <p><span data-date="2020-05-14" class="discourse-local-date" data-timezone="America/New_York" data-email-preview="2020-05-14T04:00:00Z UTC">2020-05-14T04:00:00Z</span></p>
<p>In this <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">tutorial</a>, the following code is used for forward propagation:</p>
<pre><code class="lang-auto"># zero the parameter gradients
optimizer.zero_grad()

# forward
# track history if only in train
with torch.set_grad_enabled(phase == 'train'):
    outputs = model(inputs)
    _, preds = torch.max(outputs, 1)
    loss = criterion(outputs, labels)

    # backward + optimize only if in training phase
    if phase == 'train':
        loss.backward()
        optimizer.step()

# statistics
running_loss += loss.item() * inputs.size(0)
running_corrects += torch.sum(preds == labels.data)
</code></pre>
<p>However, could they have not done this instead?</p>
<pre><code class="lang-auto"># zero the parameter gradients
optimizer.zero_grad()

inputs.requires_grad = True
outputs = model(inputs)
_, preds = torch.max(outputs, 1)
loss = criterion(outputs, labels)

# backward + optimize only if in training phase
if phase == 'train':
    loss.backward()
    optimizer.step()

# statistics
running_loss += loss.item() * inputs.size(0)
running_corrects += torch.sum(preds == labels.data)
</code></pre>
<p>Since <code>inputs</code> is a leaf node and if a leaf node has <code>requires_grad=True</code><br>
then all subsequent tensors will also have <code>requires_grad=True</code> ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You don’t actually need gradients for inputs here. All the parameters inside the model already require gradients and you don’t need to set it on <code>inputs</code>.</p>
<p>The goal is to disable the autograd when you’re not training to reduce the memory consumption (because it won’t build the computational graph and won’t save the necessary intermediary results).<br>
So yes, you can remove it if you want, it will work as well, but it will use more memory.</p>

username: Mahmoud_Abdelkhalek
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Thanks a lot for your quick response.</p>
<p>So just to be clear, if <strong>any</strong> one of the following:</p>
<ul>
<li>The input to the model</li>
<li>All model parameters</li>
</ul>
<p>Has <code>requires_grad=True</code>, then gradients will be tracked, correct? This is because all operations involve one of these two components right?</p>
<p>Also, does <code>model.train()</code> set <code>requires_grad=True</code> for all the parameters? If not, what <code>model.train()</code> actually do internally? Similarly, does <code>model.eval()</code> set <code>requires_grad=False</code> for all parameters? If not, what does it do internally?</p>

username: Mahmoud_Abdelkhalek
cooked: <p>In addition to my last response, I do not see <code>requires_grad</code> being set to <code>True</code> for the parameters of the <code>resnet18</code> model anywhere in the code, so when exactly did gradients start to be tracked for the parameters?</p>
<p>Thanks again for your help,<br>
Mahmoud</p>

username: albanD
cooked: <p>Hi,</p>
<blockquote>
<p>Has <code>requires_grad=True</code> , then gradients will be tracked, correct?</p>
</blockquote>
<p>Basically when you set requires_grad=True on a Tensor, you ask for gradients for that Tensor (assuming a leaf Tensor here).<br>
So if you set requires_grad on the input, it will compute the gradient wrt to the input.<br>
If you set it on the parameters, it will compute the gradients for the parameters.<br>
And if you set both, it will compute both.<br>
So you should only set it for the Tensor for which you actually want gradients. Otherwise you might increase memory/runtime for no reason.</p>
<blockquote>
<p>Also, does <code>model.train()</code> set <code>requires_grad=True</code> for all the parameters?</p>
</blockquote>
<p>.train() changes the behavior of the <code>nn.Module</code>. requires_grad informs the autograd whether gradient should be computed for this Tensor or not.<br>
Same for eval.<br>
Train and eval change the behavior of nn.Modules for training or evaluation. For example dropout is deactivated during evaluation. And batchnorm uses saved statistics during training instead of the ones computed on the current batch.</p>
<blockquote>
<p>when exactly did gradients start to be tracked for the parameters?</p>
</blockquote>
<p>nn.Parameter have require_grad=True by default. So when you create an nn.Parameter, it is always a leaf Tensor that requires grad (unless you pass requires_grad=False to the constructor).</p>

username: Mahmoud_Abdelkhalek
cooked: <p>Thanks a lot for the help!</p>