Is it necessary to use with torch.no_grad() for feature extraction?

SU801T: <p>I’m attempting feature extraction in an unorthodox way. I extract features in eval() mode to switch off the batch norm and dropout layers and use the running means and std provided by ImageNet.</p>
<p>I use a feature extractor to extract features from two related images and concatenate the two tensors stackwise before passing through a linear dense classifier model for training. I’m wondering whether I can avoid using <code> with torch.no_grad()</code> as the two models are unrelated.</p>
<p>Here is a simplified version:</p>
<pre><code class="lang-auto">num_classes = 2 
num_epochs = 10
criterion = nn.CrossEntropyLoss().to(device)
optimizer = torch.optim.Adam(classifier.parameters(), lr=0.001)

densenet= DenseNetConv()
densenet.eval() # set densenet to eval to switch off batch norm and dropout layers
densenet.to(device)

classifier = nn.Linear(4416, num_classes)
classifier.to(device)

for epoch in range(num_epochs):
  
  classifier.train()

  for i, (inputs_1, inputs_2,  labels) in enumerate(dataloaders_dict['train']):
       inputs_1= inputs_1.to(device)
       inputs_2 = inputs_2.to(device)
       labels = labels.to(device)

       features_1 = densenet(inputs_1) # extract features 1
       features_2 = densenet(inputs_2) # extract features 2
       
       combined = torch.cat([features_1, features_2], dim=1) # combine features 
       combined = combined(-1, 4416) # reshape 
       
       optimizer.zero_grad()

       # Forward pass to get output/logits
       outputs =  classifier(combined)
           
       # Calculate Loss: softmax --&gt; cross entropy loss
       loss = criterion(outputs, labels)
         
       _, pred = torch.max(outputs, 1)
       equality_check = (labels.data == pred)

       # Getting gradients w.r.t. parameters
       loss.backward()
       optimizer.step()
       
</code></pre>
<p>As you can see, I do not call <code> with torch.no_grad()</code>, despite having <code>densenet.eval()</code> as my separate feature extractor. Is there an issue with the way this is implemented or can I assume that this will not interfere with the <code>classifier</code> model?</p>

thecho7: <p>You don’t need to calculate the gradient during the feature extraction.<br>
It means</p>
<pre><code class="lang-auto">with torch.no_grad():
    features_1 = densenet(inputs_1) # extract features 1
    features_2 = densenet(inputs_2) # extract features 2
</code></pre>
<p>is okay.</p>

SU801T: <p>Would this not interfere with the <code>classifier</code> model I am training? Or can I get away with not applying <code>with torch.no_grad():</code>. ?</p>
<p>My only concern is whether the gradients can leak to the other models in the same batch loop.</p>

thecho7: <p>I don’t get what interfere means but if it means <strong>I want to train the classifier only</strong> then,<br>
there are at least 2 ways to do that.</p>
<ol>
<li>Freeze the feature extraction model</li>
<li>Put only classifier parameters to the optimizer and get the features under <code>torch.no_grad()</code>
</li>
</ol>
<p>These are the best ways to what you want to do without wasting memory.</p>

SU801T: <p>ah ok, so the only issue is wasting memory,… from what you are saying, the feature extractor will have no impact on the classifier model parameters even if I don’t set <code>torch.no_grad()</code>?</p>

thecho7: <p>(modify the reply)<br>
<code>.eval()</code> does not prevent making gradient graph.<br>
The entire model will be updated and this is not what you want.</p>
<p>The methods I suggested will help you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>For more details, please check the belows,<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=eval#torch.nn.Module.eval" class="inline-onebox" rel="noopener nofollow ugc">Module — PyTorch 1.12 documentation</a><br>
<a href="https://pytorch.org/docs/stable/notes/autograd.html#locally-disable-grad-doc" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 1.12 documentation</a></p>

SU801T: <p>I’m confused, I know it won’t prevent the <code>densenet</code> graph from being created but does it impact the other model who’s parameters are set to be updated (<code>classifier</code>) as there are essentially two models in the loop…</p>

thecho7: <p>Oh I missed the optimizer contains the classifier parameters only.</p>
<p>Your code would be work as you designed.</p>