How to dynamically change the size of nn.Linear

username: rmehta1987
cooked: <p>I’m trying to find a way to change the nn.Linear size dynamically.  For example lets say I have the following layers:</p>
<pre><code class="lang-auto">self.fc1 = nn.Linear(z_dim, h_dim)
self.fcmean = nn.Linear(h_dim, z_dim)

</code></pre>
<p>Now lets say for simplicity I want to change z_dim dynamically by increasing it’s size based on a coin flip.  In every epoch z_dim will increase in size by 1 or remain the same with probability .5.</p>
<p>I found this example:</p>
<aside class="quote quote-modified" data-post="1" data-topic="7957">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/quoniammm/48/1761_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-i-implement-a-dynamic-parameter-in-linear-layer-by-using-pytorch/7957">How can I implement a dynamic parameter in Linear Layer by using pytorch?</a> 
  </div>
  <blockquote>
    class LinearText(nn.Module): 
    def __init__(self):
        super(CNNText, self).__init__()
        self.encoder_tit = nn.Embedding(3281, 64)
        self.encoder_con = nn.Embedding(496037, 512)
        
        self.title_fc = nn.Linear(64, 32) # 1 * len(tit) * 32
        self.con_fc = nn.Linear(512, 256) # 1 * len(con) * 256
# t.cat((content_out_3,content_out_4,content_out_5),dim=1)
# input_size = len(tit) + len(con)
        self.fc = nn.Linear(input_size, 9)

    def forward(self, title, co…
  </blockquote>
</aside>

<p>But it is unclear how to exactly to use the Functional and nn.Parameter to incorporate a dynamic size of nn.linear</p>

username: ptrblck
cooked: <p>Well, based on the link you’ve posted, you could use the functional API and change the weight tensor manually.<br>
However, if you increase the size of your weight, what new values should be there?<br>
Should it be initialized randomly, while the rest of the weight tensor was already learned?<br>
If so, here is a small example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.weight = nn.Parameter(torch.randn(10, 2))
        
    def forward(self, x):
        x = F.linear(x, self.weight)
        return x

model = MyModel()
x = torch.randn(1, 2)
output = model(x)
output.mean().backward()
print(model.weight.grad)
model.zero_grad()

# Add another input feature
with torch.no_grad():
    model.weight = nn.Parameter(torch.cat((model.weight, torch.randn(10, 1)), 1))

x = torch.randn(1, 3)
output = model(x)
output.mean().backward()
print(model.weight.grad)
model.zero_grad()
</code></pre>

username: rmehta1987
cooked: <p>Thanks for the simple example @ <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> can I also do this in the forward function itself such that it yields the same results?</p>
<p>That is:</p>
<pre><code class="lang-auto">if rand &gt; 0.5:
      self.weight = nn.Parameter(torch.cat((self.weight, torch.randn(10, 1)), 1))
</code></pre>

username: ptrblck
cooked: <p>That might be causing some trouble, as your optimizer won’t be able to update the parameter.<br>
I’ve missed it in the other post, but after updating your weight parameter, you should also re-create the optimizer with the new parameter.<br>
You could probably try to create a new optimizer after each forward call, but that would restrict you to using optimizers without running estimates.</p>

username: rmehta1987
cooked: <aside class="quote no-group" data-username="rmehta1987" data-post="3" data-topic="29634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rmehta1987/48/7612_2.png" class="avatar"> rmehta1987:</div>
<blockquote>
<p>@ <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a></p>
</blockquote>
</aside>
<p>It seems there is a way to dynamically change the learning rate as in:</p>
<p><a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate</a></p>
<p>Can I follow a similar step to change the parameters without losing the running averages?  I know param_groups keeps the learning rate, is there a similar access to all the weights?</p>

username: ptrblck
cooked: <p>There could be a way, but it would be very hacky, since you would have to work a lot in the internals of the optimizer implementation.<br>
I’ve created a small <a href="https://gist.github.com/ptrblck/4a32a16b1ddacc9ef35690b5d948654d">Gist</a> to show, how it would be done, but again I would like to point out that this code should better be not used in any form.<br>
Besides the hacking into internals (which might break at any time), I’m not sure, if using old internal states for the parameter would give any advantages methodically.</p>

username: Surekha_Gaikwad
cooked: <p>I tried using the same approach that you mentioned for dynamically changing weight matrix.<br>
But I am getting error as<br>
RuntimeError: size mismatch, m1: [771 x 20], m2: [3 x 20] at /opt/conda/conda-bld/pytorch-cpu_1532576596369/work/aten/src/TH/generic/THTensorMath.cpp:2070</p>
<p>In the code I have to actually add atleast 12 hidden neurons dynamically but initially I tried using only one neuron. (trying cascade architecture with RPROP)<br>
Code:-<br>
class MultiLayerNet(torch.nn.Module):<br>
def <strong>init</strong>(self, n_input, n_hidden, n_output):<br>
super(MultiLayerNet, self).<strong>init</strong>()<br>
self.sigmoid = torch.nn.Sigmoid()<br>
self.relu = torch.nn.ReLU()<br>
self.weight1 = nn.Parameter(torch.randn(n_input, 3)) # for all hidden layers<br>
self.weight2 = nn.Parameter(torch.randn(2, 2)) # for output layer</p>
<pre><code>def forward(self, x,count):
    print(count)
    print(x.shape, self.weight1)   
    if count == 1:
        h_pred = self.relu(F.linear(x,self.weight1))
        
    y_pred = self.sigmoid(F.linear(h_pred,self.weight2))
        
    return y_pred
</code></pre>
<p>Note:-<br>
input as x is of shape 771 x 20 and weight1 matrix is of size 20 x 3 which should actually get easily multiplied but I observed that in linear function instead of directly using the passed weight1 matrix  it is using weight.t1() (transpose of the matrix) not sure why?</p>
<p>Further logs:-</p>
<p> in forward(self, x, count)<br>
19         print(x.shape, self.weight1)<br>
20         if count == 1:<br>
—&gt; 21             h_pred = self.relu(F.linear(x,self.weight1))<br>
22<br>
23         y_pred = self.sigmoid(F.linear(h_pred,self.weight2))</p>
<p>/usr/local/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)<br>
1024         return torch.addmm(bias, input, weight.t())<br>
1025<br>
-&gt; 1026     output = input.matmul(weight.t())<br>
1027     if bias is not None:<br>
1028         output += bias</p>

username: rmehta1987
cooked: <p>Hi,</p>
<p>Just to note the weight matrix is transposed because it is the formula for the linear layer:<br>
<code>y = xA^T + b</code></p>
<p>So you must pass weight matrix that is transposed.</p>

username: JosueCom
cooked: <p>I can see that you are changing the input size of linear here. How would one go about dynamically changing the output size instead? Would it just be changing the last 1 for 0?</p>
<pre><code class="lang-auto">model.weight = nn.Parameter(torch.cat((model.weight, torch.randn(10, 1)), 1))
</code></pre>
<p>to</p>
<pre><code class="lang-auto">model.weight = nn.Parameter(torch.cat((model.weight, torch.randn(10, 1)), 0))
</code></pre>

username: ptrblck
cooked: <p>No, you would also need to change the newly added tensor:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.weight = nn.Parameter(torch.cat((model.weight, torch.randn(1, 2)), 0))
</code></pre>

username: JosueCom
cooked: <p>I see what you mean. Thank you</p>