Updating model with more classes

shomerj: <p>I have trained resnet34 on my own data that has 53 classes. Every so often new classes come out and I need to retrain the model with the additional classes. For example, this week 9 new classes came out and I have to retrain my model with the additional 9 classes.</p>
<p>My approach is to load current model and state_dict for that model and then replace the fully connect layer with one that outputs 62 classes (in this case). Here is my code for resuming training based on my explanation:</p>
<pre><code class="lang-auto">        if os.path.isfile(args.resume):
            print(f"Loading Checkpoint from {args.resume}")
            checkpoint = torch.load(args.resume, map_location=device)
            args.start_epoch = checkpoint['epoch']
            best_acc = checkpoint['best_acc']
            arch = args.arch
            state_dict = checkpoint['state_dict']
            optim = checkpoint['optimizer']

            if args.num_classes != checkpoint['num_classes']:
                model = ItemDetectorRes(checkpoint['num_classes'], arch)
                model.load_state_dict(state_dict)
                in_features = model.resnet.fc.in_features
                model.resnet.fc = nn.Linear(in_features, args.num_classes)
                args.start_epoch = 0
                best_acc = 0.0
                model.to(device)

</code></pre>
<p>The issue I am having is that it is not training. I cannot figure out why this would be the case. Any suggestions?</p>

ptrblck: <p>Do you see a constant loss or the same accuracy or what do you mean by “not training”?<br>
The code looks alright, but could you make sure that the new linear layer is being updated by printing the gradients after calling <code>loss.backward()</code>:</p>
<pre><code class="lang-python">print(model.resnet.fc.weight.grad)
</code></pre>

shomerj: <p>It does seem to be updating.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0a31b628a67eb4b0b4c8550a0b5435750fcf2d36.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a31b628a67eb4b0b4c8550a0b5435750fcf2d36" title="27%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0a31b628a67eb4b0b4c8550a0b5435750fcf2d36_2_544x500.png" alt="27%20PM" data-base62-sha1="1sbhr1ZemJS41tMW6iuNSJzQGQ6" width="544" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0a31b628a67eb4b0b4c8550a0b5435750fcf2d36_2_544x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0a31b628a67eb4b0b4c8550a0b5435750fcf2d36_2_816x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0a31b628a67eb4b0b4c8550a0b5435750fcf2d36_2_1088x1000.png 2x" data-dominant-color="101010"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">27%20PM</span><span class="informations">1200×1102 74.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is an image of the training for a couple epochs:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b106e6aeec9a06c4ea5e9e9c6e79c1882b640e53.png" data-download-href="https://discuss.pytorch.org/uploads/default/b106e6aeec9a06c4ea5e9e9c6e79c1882b640e53" title="48%20PM"><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b106e6aeec9a06c4ea5e9e9c6e79c1882b640e53.png" alt="48%20PM" data-base62-sha1="pg3isPnE7topWx0QQrOyku8PrQD" width="645" height="500" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">48%20PM</span><span class="informations">930×720 37.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I retrain from scratch it trains just fine.</p>

ptrblck: <p>You might need to play around with the hyperparameters (e.g. lower the learning rate), which might help fine tuning a pretrained model.</p>

shomerj: <p>I’ll give that a shot. It is still weird that it is not learning with a higher learning rate. Currently it is at 3e-3</p>

ptrblck: <p>Depending on the model, <code>3e-3</code> is not very low.<br>
What was the learning rate you’ve used to train from scratch?</p>

shomerj: <p>Same with it decaying by a factor of 10 every 10 epochs. I am training resnet34</p>

ptrblck: <p>If you are trying to finetune a model, I would generally recommend to lower your learning rate relative to the initial learning rate, since the majority of your parameters should already be in a good state.</p>

shomerj: <p>Good to know. Although I think there must be an issue with my code since even at a very low learning rate it is still not learning. I will keep debugging and post a solution when I get to it</p>