Pretraining with MobileNetv3

username: darkdante
cooked: <p>Hi, I am trying to use mobilenetv3 in transfer learning to classify images in 4 categories. At the first, I try to freeze the weights so I can use them for feature extraction and I add a custom layer at the end. However, I get the following error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “trial13MobileNetxray.py”, line 431, in <br>
train(mobilenetv3, args)<br>
File “trial13MobileNetxray.py”, line 230, in train<br>
loss.backward()               # backward pass (compute parameter updates)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/tensor.py”, line 245, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 147, in backward<br>
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn<br>
File “/opt/conda/lib/python3.6/site-packages/torch/tensor.py”, line 245, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 147, in backward<br>
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
</blockquote>
<p>My code is as follows:</p>
<pre><code>#Otain pretrained mobilenet from pytorch models
mobilenetv3 = torchvision.models.mobilenet_v3_large(pretrained=True)

#Freeze the pretrained weights for use 
for param in mobilenetv3.parameters():
  param.requires_grad = False

# add custom layers to prevent overfitting and for finetuning 
mobilenetv3.fc = nn.Sequential(nn.Dropout(0.2),
                              nn.BatchNorm1d(1280), #320
                              nn.ReLU(),
                              nn.Dropout(0.3),      
                              nn.Linear(320, 4),
                              nn.LogSoftmax(dim=1)
                              )
</code></pre>
<p>Please help</p>

username: Dwight_Foster
cooked: <p>When you set the parameters of the mobilenet to have no gradient the backward function cannot be calculated because there is nothing to calculate it with. Are you trying to freeze the mobilenet layers? If so there is a better way to do that. When you define your optimizer only put in the parameters you want like this:</p>
<pre><code class="lang-auto">optimizer  = optim.Adam(mobilenetv3.fc.parameters())
</code></pre>
<p>that will only update the fc parameters.</p>

username: omarfoq
cooked: <p>Hello <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a>?</p>
<p>I think the part of code he shows is not incorrect. In fact, I think this the standard way of freezing gradient: if he don’t set  <code>param.requires_grad = False</code>, all gradients well be computed during the backward iteration, which is not memory efficient if he don’t was to use all of them to update the model.</p>
<p>He probably did a mistake when creating the optimizer, and need to create it as you said</p>
<aside class="quote no-group" data-username="Dwight_Foster" data-post="2" data-topic="116992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dwight_foster/48/18182_2.png" class="avatar"> Dwight_Foster:</div>
<blockquote>
<p><code>optimizer  = optim.Adam(mobilenetv3.fc.parameters())</code></p>
</blockquote>
</aside>

username: Dwight_Foster
cooked: <p>Yes thank you, you are correct. Thank you for pointing that out.</p>