Confused about "set_grad_enabled"

username: Carsten_Ditzel
cooked: <p>I am confused about the following <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">snippet</a> taken from the tutorials about transfer learning.</p>
<pre><code class="lang-auto"> for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]
</code></pre>
<ol>
<li>What does the part</li>
</ol>
<pre><code class="lang-auto">  with torch.set_grad_enabled(phase == 'train'):
</code></pre>
<p>do? I thought that model.eval() and model.train() is enough to put the model into states in which to save and evaluate backprob information.</p>
<ol start="2">
<li>
<p>How does it compare to requires_grad() ? do the fulfill the same purpose?</p>
</li>
<li>
<p>Why is the epoch_loss divided by the size of the training set? According to the docs the loss is by default averaged over the batch size (reduction=‘mean’), so why would I divide by the entire data set size in the end?</p>
</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p><code>model.train()</code> and <code>model.eval()</code> change the behavior of some layers. E.g. <code>nn.Dropout</code> won’t drop anymore and <code>nn.BatchNorm</code> layers will use the running estimates instead of the batch statistics. The <code>torch.set_grad_enabled</code> line of code makes sure to clear the intermediate values for evaluation, which are needed to backpropagate during training, thus saving memory. It’s comparable to the <code>with torch.no_grad()</code> statement but takes a bool value.</p>
</li>
<li>
<p>All new operations in the <code>torch.set_grad_enabled(False)</code> block won’t require gradients. However, the model parameters will still require gradients.</p>
</li>
<li>
<p>The <code>running_loss</code> will be “de-averaged” by multiplying it with <code>inputs.size(0)</code>. Therefore you should divide by the whole dataset length, not the number of batches.</p>
</li>
</ol>

username: Carsten_Ditzel
cooked: <p>thank you for your reply, I see clearer now.</p>
<p>Recarding question 3:</p>
<ol>
<li>
<p>Why would one de-average the loss only to divide later by the entire data set size? for smoothing reasons of the results?</p>
</li>
<li>
<p>Can the de-averaging also be done by omitting the multiplication with</p>
</li>
</ol>
<pre><code class="lang-auto">inputs.size(0)
</code></pre>
<p>and setting the reduction parameter to ‘none’ in the loss function initialization?</p>

username: ptrblck
cooked: <ol>
<li>
<p>If your dataset length is not divisible by the batch size, the last batch will contain less samples than all others. Thus taking the averaged loss and divide it by the number of batches will give a a slightly biased result.</p>
</li>
<li>
<p>You could use <code>sum</code> to get the accumulated loss for debugging purposes. However, I would still recommend to take the mean for the loss for backpropagation, since otherwise e.g. the learning rate will depend on the batch size.</p>
</li>
</ol>

username: Carsten_Ditzel
cooked: <p>thank you very much!</p>

username: Upgrade_Yourself
cooked: <p>Another thing, scheduler.step() must be called after optimizer.step() according to the docs:<br>
<a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate</a></p>

username: hoangle_tttm
cooked: <p>In your second statement :" All new operations in the <code>torch.set_grad_enabled(False)</code> block won’t require gradients. However, the model parameters will still require gradients".<br>
For example, my pretrained model is an “Encoder” and all parameters of it have been set to “False” in my <strong>init</strong> function of my “Feed Forward Classifier”. In my train loop,  “with torch.set_grad_enabled(phase == ‘train’):…” is used and phase==“train” is True.<br>
Will my classifier model update my freeze parameter of my Encoder?</p>

username: ptrblck
cooked: <p>No, it won’t.<br>
Setting <code>set_grad_enabled</code> to <code>True</code> allows the gradient calculation, but will not (somehow) force Autograd to track all operations:</p>
<pre><code class="lang-python">encoder = nn.Linear(1, 1, bias=False)
encoder.weight.requires_grad_(False)

classifier = nn.Linear(1, 1, bias=False)

x = torch.randn(1, 1)

with torch.set_grad_enabled(True):
    out = encoder(x)
    out = classifier(out)
    
out.backward()
print(encoder.weight.grad)
&gt; None

print(classifier.weight.grad)
&gt; tensor([[0.3575]])
</code></pre>

username: hoangle_tttm
cooked: <p>Thank you!! So “torch.set_grad_enabled” (a context manager) only decides whether or not to perform the gradient calculation within it and not changes any parameters’ “requires_grad” values.<br>
If torch.set_grad_enabled(True) only the gradient calculation of “requires_grad=True” parameters within block will be done.</p>

username: Carsten_Ditzel
cooked: <p>exactly, it does not switch on the calculation of gradients for parameters that were previously disabled</p>