Requires_grad is not working

Aryan_Asadian: <p>Hi everyone. I have created a modular model via this class:</p>
<pre><code class="lang-auto">class VGG8StepBySTepModelModular(nn.Module):
    def __init__(self):
        super(VGG8StepBySTepModelModular, self).__init__()
        #empty_model.add_module("stage_1", vgg_8_sub_student)

        #empty_model.add_module("stage_2", nn.Sequential(*list(vgg_8_model.features)[7:]))
        #empty_model.add_module("classifier", nn.Sequential(vgg_8_model.classifier))
        vgg_8_model = get_vgg8()

        self.add_module("stage_1",vgg_8_model.features[:8])
        self.add_module("stage_2",vgg_8_model.features[8:])
        self.add_module("classifier",vgg_8_model.classifier)

    def forward(self, x):
        x = self.stage_1(x)
        x = self.stage_2(x)
        x = x.view(-1,512 * 7 * 7)
        x = self.classifier(x)
        return x

combined_vgg8_student = VGG8StepBySTepModelModular()
</code></pre>
<p>Since I have trained a smaller model which contains only the stage_1 of my model, I want to load the trained model for stage_1, and train the rest of the model for the next step. I have loaded the trained stage_1 module successfully. But when I try to set the <code>requires_grad = False</code> for the stage_1 module, it does not working. However, I found that <code>combined_vgg8_student.stage_1.eval()</code> will change the <code>training = False</code>. What is the problem? any help would be appreciated.</p>
<pre><code class="lang-auto"> for params in combined_vgg8_student.stage_1.parameters():
        params.requires_grad = False
    
    print("STAGE 1 ",combined_vgg8_student.stage_1.training) # Prints True
    print("STAGE 2 ",combined_vgg8_student.stage_2.training) # Prints True
    print("Classifier ",combined_vgg8_student.classifier.training) # Prints True

    combined_vgg8_student.stage_1.eval()
    print("STAGE 1 ",combined_vgg8_student.stage_1.training) # Prints False
    print("STAGE 2 ",combined_vgg8_student.stage_2.training) # Prints True
    print("Classifier ",combined_vgg8_student.classifier.training) # Prints True


</code></pre>

albanD: <p>Hi,</p>
<p>The two are not related.<br>
The <code>mod.train()</code> and <code>mod.eval()</code> change the <code>mod.training</code> flag on the module (and its children). This is used to know how modules like batchnorm or dropout should behave.<br>
The requires_grad field on the Parameters tells the autograd if it should track gradients for these Tensors or not.</p>

Aryan_Asadian: <p>Hi,</p>
<p>Thank you for your reply. It helps me a lot. So as far as I understood, calling <code>eval()</code> for a part of a model is meaningless. Am I right?<br>
For instance, in my code snippet, <code>combined_vgg8_student.stage_1.eval()</code> does not prevent autograd in the forward loop from computing the gradients.</p>

albanD: <blockquote>
<p>calling <code>eval()</code> for a part of a model is meaningless. Am I right?</p>
</blockquote>
<p>It depends what you want to do.<br>
If you want to prevent gradient computation, then it is not what you want indeed. You need to set the requires_grad field, or <code>combined_vgg8_student.stage_1.requires_grad_(False)</code> that will set the field on all the parameters.</p>
<p>If you wanted to disable dropout and put batchnorm in evaluation mode (that uses the saved statistics), then this was the right thing to do <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>