What is the requires_grad state when we load a pretrained model?

username: Gal_Elias
cooked: <p>Hello,<br>
if i write this line of code:<br>
model = torchvision.models.googlenet(pretrained=True)<br>
and i train the model as it is without any modifications, what layers will be trained? what is the default requires_grad state? and as i saw, the output size (# of classes) of the pretrained model is 1000 and i trained it on 2 classes dataset without modifying the last layer. How can it be possible?</p>
<p>Thanks!</p>

username: SumanthRH
cooked: <p>Hey. You can simply run the following block of code:</p>
<pre><code class="lang-auto">for parameter in model.parameters(): 
     print(parameter.requires_grad)
</code></pre>
<p>To check what the default state is for the model parameters. By default, <code>requires_grad</code> is True and the model is in train mode. (<code>model.training</code> is True)</p>

username: Gal_Elias
cooked: <p>Oh nice, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> and you know maybe why i can train the model with a dataset with different number of classes even if i don’t change the last layer to fit this number of classes?</p>

username: SumanthRH
cooked: <p>What’s the shape of the <code>label</code> tensor you’re passing? And the predictions are of shape <code>(batch_size, 1000)</code> itself? Would be better if you can provide the code block you’re running.</p>

username: Gal_Elias
cooked: <p>This is the training part:</p>
<pre><code class="lang-auto">### Set Device ###
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"Device used: {device}")

### Load pretrain model ###
model = torchvision.models.googlenet(pretrained=True)
model.to(device)

### Loss and Optimizer ###
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

### Train Network ###
for epoch in range(num_epochs):
    losses = []

    for batch_idx, (data, targets) in enumerate(train_loader): # data - images. targets - correct labels
        # Get data to cuda if possible
        data = data.to(device=device)
        targets = targets.to(device=device)

        # Forward
        scores = model(data)
        loss = criterion(scores, targets)
        losses.append(loss.item())

        # Backward
        optimizer.zero_grad() # Reset the gradients for each batch before the calculations so it won't use previous batch's gradients
        loss.backward()

        # Gradient descent or adam step
        optimizer.step() # Update the weights

    print(f'Loss at epoch #{epoch+1}: {sum(losses)/len(losses):.4f}')
</code></pre>
<p>and the dataset is the dogs_cats dataset with the labels 1 for dog and 0 for cat. so the  size of the label tensor is (batch) and the predictions tensor is of shape (batch, 1000).</p>

username: SumanthRH
cooked: <p>The PyTorch docs mention the following about the input arguments for <code>nn.CrossEntropyLoss</code> :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4abc4a03f9530be48b3119090196d8212c184728.png" data-download-href="https://discuss.pytorch.org/uploads/default/4abc4a03f9530be48b3119090196d8212c184728" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4abc4a03f9530be48b3119090196d8212c184728_2_690x208.png" alt="image" data-base62-sha1="aF8JU5PdGgA7f4n7kERw4s5WX9e" width="690" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4abc4a03f9530be48b3119090196d8212c184728_2_690x208.png, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4abc4a03f9530be48b3119090196d8212c184728.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4abc4a03f9530be48b3119090196d8212c184728.png 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">866×262 23.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If you have only two classes in your dataset, the loss will still accept it because it expects all <code>target</code> tensor values to be in the range [0, 1000-1]. So there won’t be an error raised. What you’re essentially doing is training a model for 1000-class classification and only showing it examples from the first two classes.</p>

username: Gal_Elias
cooked: <p>Oh i see… thank you very much!</p>