Add neurons to an existing layer

username: CarlosHernandezP
cooked: <p>I am trying an approach to solve a specific problem that requires me to add <strong>n</strong> neurons to the last layer of a pre-trained model. The tricky part is that it has to be on the same layer. I found an example in the forums here: <a href="https://discuss.pytorch.org/t/possible-to-add-initialize-new-nodes-to-hidden-layer-partway-through-training/3809">older question</a></p>
<p>I want to keep the pre-trained weights intact and add <strong>n</strong> randomly initialized ones.</p>
<p>In said question, they gave a code snippet that I modified to my problem. It looks like this:</p>
<pre><code class="lang-auto">def add_units(self, n_new):
    '''
    n_new : integer variable counting the neurons you want to add
    '''

    # take a copy of the current weights stored in self._fc which is an 
    # ModuleList variable with only one layer
    current = [ix.weight.data for ix in self._fc]

    # randomly initialize a tensor with the size of the wanted layer 
    hl_input = torch.zeros([n_new, current[0].shape[1]])
    nn.init.xavier_uniform_(hl_input, gain=nn.init.calculate_gain('relu'))

    # concatenate the old weights with the new weights
    new_wi = torch.cat([current[0], hl_input], dim=0)

    # reset weight and grad variables to new size
    self._fc[0] = nn.Linear(current[0].shape[1]+n_new, 2)

    # set the weight data to new values
    self._fc[0].weight.data = torch.tensor(new_wi, requires_grad=True)
</code></pre>
<p>This method is inside the model class and can be called by typing “mode.add_units(N)”.<br>
Am I preserving the original waits of the layer “_fc” and adding new, randomly initialize ones or am I missing something?</p>
<p>Running a quick code, adding just two nodes seems to print the right shape:</p>
<pre><code class="lang-auto">[In]   print(model._fc)
[Out]  Linear(in_features=2560, out_features=2)
[In]   model.add_units(2)
[In]   print(model._fc)
[Out]  Linear(in_features=2562, out_features=2)
</code></pre>
<p>I am not sure if by doing this I am preserving the weights and the bias, I assumed that the last line did so but could not know for certain.</p>

username: ptrblck
cooked: <p>To make sure the old weights are present, you could simply print the old and new weight tensor and compare them or use a proper comparison via <code>old_weight == new_weight[:, :old_weight_num]</code>.<br>
I’m not sure how <code>ix</code> etc. is defined, so cannot see, if the creation is correct.</p>
<p>I would recommend to avoid using the <code>.data</code> attribute, as it might have unwanted side effects but instead assign a new <code>nn.Parameter</code> after the manipulation.</p>

username: CarlosHernandezP
cooked: <p>Thank you for your answer. Regarding the last part, if I understand correctly, the code should be changed to:</p>
<pre><code class="lang-auto">self._fc[0].weight = torch.nn.Parameter(new_wi)
</code></pre>
<p>Will this update the parameters of the assigned layer with the values of “new_wi”?</p>

username: ptrblck
cooked: <p>Yes, this should work.</p>

username: CarlosHernandezP
cooked: <p>In order to make the problem more comprehensible, I changed from ModuleList to a simple layer, thus the code has the next form:</p>
<p>want to keep the pre-trained weights intact and add <strong>n</strong> randomly initialized ones.</p>
<p>In said question, they gave a code snippet that I modified to my problem. It looks like this:</p>
<pre><code class="lang-auto">def add_units(self, n_new):
    '''
    n_new : integer variable counting the neurons you want to add
    '''

    # take a copy of the current weights stored in self._fc
    current = self._fc.weight.data
    current_bias = self._fc.bias.data #Only used at the end of the post

    # randomly initialize a tensor with the size of the wanted layer 
    hl_input = torch.zeros([n_new, current.shape[1]])
    nn.init.xavier_uniform_(hl_input, gain=nn.init.calculate_gain('relu'))

    # concatenate the old weights with the new weights
    new_wi = torch.cat([current, hl_input], dim=0)

    # reset weight and grad variables to new size
    self._fc = nn.Linear(current.shape[1]+n_new, 2) #2 is the size of my output layer

    # set the weight data to new values
    self._fc.weight = torch.nn.Parameter(new_wi)
</code></pre>
<p>My question is, by doing this am I also copying the bias or only the weights? If I try to do a:</p>
<pre><code class="lang-auto">current_bias  = self._fc.bias.data
</code></pre>
<p>the dimension that I get equals the output dimension, not the input dimension. What changes should I make to correctly add the old biases into this new layer?</p>
<pre><code class="lang-auto">self._fc.bias = current_bias #Does not throw an error but doesn't look right.
</code></pre>
<p>Diving into the nn.Linear documentation I found this:</p>
<pre><code class="lang-auto">'''
Attributes:
        weight: the learnable weights of the module of shape
            :math:`(\text{out\_features}, \text{in\_features})`. The values are
            initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})`, where
            :math:`k = \frac{1}{\text{in\_features}}`
        bias:   the learnable bias of the module of shape :math:`(\text{out\_features})`.
                If :attr:`bias` is ``True``, the values are initialized from
                :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where
                :math:`k = \frac{1}{\text{in\_features}}`
'''
</code></pre>
<p>So it does make sense that the size equals my output size but I’m still left unsure as to whether my method updates the bias correctly or not.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="CarlosHernandezP" data-post="5" data-topic="79583">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carloshernandezp/48/48526_2.png" class="avatar"> CarlosHernandezP:</div>
<blockquote>
<p>So it does make sense that the size equals my output size but I’m still left unsure as to whether my method updates the bias correctly or not.</p>
</blockquote>
</aside>
<p>If you are not changing the number of output features, you could just reuse the bias.<br>
Your approach should work, besides the usage of <code>.data</code> which should be avoided. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: Mr_Citizen
cooked: <p>So updating the weights will update the bias automatically? And what do you mean by avoiding .data? How would you do that?</p>
<p>Also, do you know why I my weight.grad is None when I do this? Is there something I have to do to initialize them?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Mr_Citizen" data-post="7" data-topic="79583">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mr_citizen/48/49642_2.png" class="avatar"> Mr_Citizen:</div>
<blockquote>
<p>So updating the weights will update the bias automatically?</p>
</blockquote>
</aside>
<p>No, that’s not the case. Since the <code>bias</code> is defined by the number of <code>out_features</code> it wouldn’t have to be changed if only the <code>in_features</code> differ.</p>
<aside class="quote no-group" data-username="Mr_Citizen" data-post="7" data-topic="79583">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mr_citizen/48/49642_2.png" class="avatar"> Mr_Citizen:</div>
<blockquote>
<p>And what do you mean by avoiding .data? How would you do that?</p>
</blockquote>
</aside>
<p>Don’t use the <code>.data</code> attribute, but reassign a new <code>nn.Parameter</code> to the <code>.weight</code> attribute.</p>
<aside class="quote no-group" data-username="Mr_Citizen" data-post="7" data-topic="79583">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mr_citizen/48/49642_2.png" class="avatar"> Mr_Citizen:</div>
<blockquote>
<p>Also, do you know why I my weight.grad is None when I do this?</p>
</blockquote>
</aside>
<p>Maybe you are detaching the computation graph, so could you post a minimal, executable code snippet showing this issue?</p>

username: Mr_Citizen
cooked: <p>This is the function I’ve been trying. I based it mostly off a similar post to this one, but the code used there and here are mostly similar.</p>
<p>Coords tells the function what layer we’re adding neurons to, num_neurons is how many neurons to add, and duplication says whether to leave them as zero or copy the value of other neurons into these neurons.</p>
<pre><code class="lang-auto">def add_neurons(self, coords, num_neurons, duplicate=False):
        '''Add new neurons to a layer in the model'''
        # Copy the current weights
        current = [layer.weight.data for layer in self.layers]

        # Make the new weights you'll be adding for the I/O layers
        # Give them values, if we're duplicating
        hl_input = torch.zeros([num_neurons, current[coords].shape[1]])
        if duplicate: hl_input = current[coords][1, (current[coords].shape[1]-num_neurons):current[coords].shape[1]]
        nn.init.xavier_uniform_(hl_input, gain=nn.init.calculate_gain('relu'))

        hl_output = torch.zeros([current[1].shape[0], num_neurons])
        if duplicate: hl_output = current[coords][0, (current[coords+1].shape[1]-num_neurons):current[coords].shape[1]]
        nn.init.xavier_uniform_(hl_input, gain=nn.init.calculate_gain('relu'))

        # Concatenate the old I/O weights with the new I/O weights
        new_wi = torch.cat([current[coords], hl_input], dim=0)
        new_wo = torch.cat([current[coords+1], hl_output], dim=1)

        # Reset weight and grad variables to new size
        self.layers[coords] = nn.Linear(current[coords].shape[1], self.layer_sizes[coords])
        self.layers[coords+1] = nn.Linear(self.layer_sizes[coords], current[coords+1].shape[0])

        # Set the weight data to new values
        self.layers[coords].weight = torch.nn.Parameter(new_wi)
        self.layers[coords+1].weight = torch.nn.Parameter(new_wo)
</code></pre>

username: ptrblck
cooked: <p>Your code is unfortunately not executable so I cannot reproduce the issue and debug it and also don’t see any obvious mistakes.</p>

username: Mr_Citizen
cooked: <p>Here’s my entire model class, if that helps.</p>
<pre><code class="lang-auto">class QNet(nn.Module):
    def __init__(self, input_size, output_size):
        super().__init__()

        # 
        # Build the model
        #
        self.layers = nn.ModuleList()
        self.layer_sizes = []

        # Add hidden layers
        self.num_layers = 0
        # Randomized hidden layers
        for i in range(randrange(1, 3)):
            size = randrange(32, 256)
            self.layers.append(nn.Linear(input_size, size))
            input_size = size  # For the next layer
            self.layer_sizes.append(input_size)
            self.num_layers += 1
        
        # Output layer
        self.layers.append(nn.Linear(input_size, output_size))
        self.layer_sizes.append(input_size)
        self.num_layers += 1

        # 
        # Set whether or not to use gpu
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        
        #
        # Set optimizer and loss
        self.optimizer = optim.Adam(self.parameters(), lr=LR)
        self.criterion = nn.MSELoss()


    def forward(self, x):
        '''Get output from model.'''
        #for layer in self.layers: x = layer(x)
        for i, layer in enumerate(self.layers): x = layer(x) if i==self.num_layers else layer(F.relu(x))
        return x


    def add_neurons(self, coords, num_neurons, duplicate=False):
        '''Add new neurons to a layer in the model'''
        # Copy the current weights
        current = [layer.weight.data for layer in self.layers]

        # Make the new weights you'll be adding for the I/O layers
        # Give them values, if we're duplicating
        hl_input = torch.zeros([num_neurons, current[coords].shape[1]])
        if duplicate: hl_input = current[coords][1, (current[coords].shape[1]-num_neurons):current[coords].shape[1]]
        nn.init.xavier_uniform_(hl_input, gain=nn.init.calculate_gain('relu'))

        hl_output = torch.zeros([current[1].shape[0], num_neurons])
        if duplicate: hl_output = current[coords][0, (current[coords+1].shape[1]-num_neurons):current[coords].shape[1]]
        nn.init.xavier_uniform_(hl_input, gain=nn.init.calculate_gain('relu'))

        # Concatenate the old I/O weights with the new I/O weights
        new_wi = torch.cat([current[coords], hl_input], dim=0)
        new_wo = torch.cat([current[coords+1], hl_output], dim=1)

        # Reset weight and grad variables to new size
        self.layers[coords] = nn.Linear(current[coords].shape[1], self.layer_sizes[coords])
        self.layers[coords+1] = nn.Linear(self.layer_sizes[coords], current[coords+1].shape[0])

        # Set the weight data to new values
        self.layers[coords].weight = torch.nn.Parameter(new_wi)
        self.layers[coords+1].weight = torch.nn.Parameter(new_wo)
</code></pre>

username: crizCraig
cooked: <p>One workaround to this may be to add a new head to your network since you just want to add to the last layer. The advantage of this vs the above approach would be that optimizer state (e.g. Adam momentum/decay params) will be preserved. Just make sure to calculate the loss for your new head.</p>