MNIST Incremental Learning

username: flamingice
cooked: <p>I want to train iteratively, first training on 0 and 1, but then pause training, modify the output layer to include a third output and then train on 0,1 AND 2 by adding 2s back in my dataset. I do not have to train from scratch. Train 10 epochs every time I add in a new digit. Just Be sure to train using the same model that was already trained with one less digit with the augmented output layer (the weights should not be reset only the output layer is increased by 1). After training for 10 epochs, add in 3 digits and expand the output layer of the network again. This has to be done all the way up until I have eventually done training on the entire dataset.<br>
How to achieve this on the below code?</p>
<p><a href="https://colab.research.google.com/drive/1KeRoh1DqtGbHIoZJK372i_zDxS_G1R8W" rel="nofollow noopener">Code</a><br>
<span class="mention">@ptrblck_de</span></p>

username: ptrblck
cooked: <p>For the MNIST dataset, you could try to either manipulate the underlying <code>data</code> and <code>target</code> attributes or write a custom <code>Dataset</code> using the MNIST dataset as the parent class and implement your sampling logic e.g. in the <code>forward</code> method by passing a specific class flag.<br>
I would prefer the latter case, as it sounds cleaner.</p>
<p>To add a new neuron to the last layer, I would recommend to use the functional API, store the parameters e.g. in a <code>nn.ParameterList</code>, and add the new parameter to this list.<br>
Alternatively, you could also manipulate the last linear’s parameters also by reassigning the new weight parameter and copy the old values in.</p>
<p>I’m not sure how you would like to deal with the optimizer. Would you like to reinitialize it after each change in the model?</p>

username: flamingice
cooked: <p>Thank you for giving a head start to the solution. Is there any source/link which I can look up to?<br>
Also, I don’t want to reinitialize the optimizer after each change in the model.</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/possible-to-add-initialize-new-nodes-to-hidden-layer-partway-through-training/3809/4">Here</a> is an example of directly manipulating the layer.<br>
Note that you shouldn’t use the <code>.data</code> attribute, but instead wrap the manipulation in a <code>with torch.no_grad()</code> block.</p>
<p>Alternatively, here is an example of the <code>nn.ParameterList</code> usage:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.params = nn.ParameterList()
        self.params.append(nn.Parameter(torch.randn(2, 10)))
        
    def forward(self, x):
        w = torch.cat(tuple(self.params.parameters()), 0)
        x = F.linear(x, w)
        return x
    

model = MyModel()
x = torch.randn(1, 10)
out = model(x)
print(out.shape)

# Add param
new_param = nn.Parameter(torch.randn(1, 10))
model.params.append(new_param)

out = model(x)
print(out.shape)
</code></pre>
<aside class="quote no-group" data-username="flamingice" data-post="3" data-topic="74236">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flamingice/48/21990_2.png" class="avatar"> flamingice:</div>
<blockquote>
<p>I don’t want to reinitialize the optimizer after each change in the model.</p>
</blockquote>
</aside>
<p>In that case you could use <code>optimizer.add_param_group</code> with your new parameter.</p>