Access all weights of a model

TGI_Fernando: <p>Hi,</p>
<p>Is there a way to access all weights of a neural network model?</p>
<p>E.g. If we have the following class</p>
<p>import torch<br>
import torch.nn as nn</p>
<p>class mlp_new(nn.Module):<br>
def <strong>init</strong>(self, n_in, n_hid, n_out):<br>
super(mlp_new, self).<strong>init</strong>()<br>
self.layers = nn.Sequential(<br>
nn.Linear(n_in, n_hid),<br>
nn.ReLU(),<br>
nn.Linear(n_hid, n_out)<br>
)</p>
<pre><code>def forward(self, x):
    return self.layers(x)
</code></pre>
<p>my_mlp = mlp_new(10, 5, 1)</p>
<p>Can I access all weights of my_mlp (e.g. my_mlp.layers.weight - not working)?</p>
<p>Actually I want to update all weights of the model using my own method with a single statement like optimizer.step().</p>
<p>Please note that, I know that weights can be accessed layer-wise ( my_mlp.layers[0].weight, my_mlp.layers[2].weight).</p>
<p>Thanks with best regards.</p>

ptrblck: <p>You could iterate the parameters to get all <code>weight</code> and <code>bias</code> params via:</p>
<pre><code class="lang-python">for param in model.parameters():
    ....

# or
for name, param in model.named_parameters():
    ...
</code></pre>
<p>You cannot access all parameters with a single call.<br>
Each parameter might have (and most likely has) a different shape, can be pushed to a different device etc.</p>

TGI_Fernando: <p>Thanks in deed ptrblck.</p>
<p>With best regards.</p>
<p>TGI Fernando</p>

hpf: <p>Hi，I have a question about this.Here is the code for my training phase.<br>
<strong>I want to get the weight parameters of the first level convolution</strong>，</p>
<pre><code class="lang-auto">for epoch in range(EPOCH): 
    NET.train()
    torch.manual_seed(3)
    print('Epoch:', epoch + 1, 'Training...') 
    for i, (batch_x, batch_y) in enumerate(loader): 
        batch_x=Variable(batch_x)
        output = NET(batch_x)
        loss = lossfunc(output, batch_y)  
        loss.backward()
        optimizer.step()
        wc_loss.append(loss.data)   
        print('Epoch: ', epoch, '| Step: ', i, '| loss: ',wc_loss[i])        
        for name, param in NET.named_parameters():
            if 'network.0.conv1.weight_v' in name:
                parm[name]=param.detach().numpy()  
                b.append(parm['network.0.conv1.weight_v'])  
</code></pre>
<p>but the result of 【b】is always repeated,same result every updata，like this.（<strong>note：The first convolution layer is conv1d</strong>,<br>
<code>conv1d(1,3,kernel=3)</code>）</p>
<pre><code class="lang-auto">[array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
 array([[[-11.6726265 ,  12.326811  ,  12.306071  ]],
 
        [[  0.4224267 ,  -0.21204475,  -0.07533604]],
 
        [[ 11.559458  ,  11.859747  ,  12.272797  ]]], dtype=float32),
......
......
</code></pre>
<p>However, the value of 【loss】 decreased normally.I don’t know why the value of the weight 【b】 is not updated.</p>

ptrblck: <p>I cannot reproduce the constant output using this small code snippet:</p>
<pre><code class="lang-python">model = models.resnet18()
optimizer = torch.optim.SGD(model.parameters(), lr=1.)
criterion = nn.CrossEntropyLoss()

data = torch.randn(1, 3, 224, 224)
target = torch.randint(0, 1000, (1,))

for _ in range(3):
    out = model(data)
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()
    print(model.fc.weight.detach().numpy())
</code></pre>
<p>Try to <code>.clone()</code> the parameter before storing them in the list and let me know, if that helps.<br>
If not, your computation graph might be detached at one point and the parameter might not get any gradients.</p>

hpf: <p>Thank you for your reply！<br>
um…<br>
I don’t know how to use clone properly (this time I’m not going to delve into this one), so my choice is: <strong>clear all variables, restart the program, and restart training</strong>. This time I set 【epoch = 1】, manual training. I didn’t change the original code, I just ran it again. I get the change of the weight parameter value in each epoch.</p>
<p><strong>Note: for each epoch, the parameter is updated 1180 times.</strong><br>
I only select a certain weight parameter(I call it weight B) in the model and observe the change of its value in the process of updating.<br>
After the end of each time model training, I will draw the change of weight into a graph. Then, without any changes, retrain.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/300034e353e3c28ddc6bb8f85bccb00daf83da28.png" alt="图片1" data-base62-sha1="6QDmotEgT9hxyyE7dAUQdrLO8jK" width="506" height="322"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83e527ae7229899cd3532ef8934698cffc3b9bf9.png" alt="图片2" data-base62-sha1="iONvG91GbFxMrh9pmHt7fNI1h7H" width="526" height="330"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c21655e003a72bfbe3b1b9829be78f44cb8dd20.png" alt="图片3" data-base62-sha1="fqz7h2ae5BlF2HUIIcrwGMDNiLe" width="483" height="316"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea417c2b25d1262ed3e195f91d27d7209228155f.png" alt="图片4" data-base62-sha1="xqk1NWB11d81JSOMpc1SJMpS3tJ" width="497" height="318"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/4/149b0e61afb581b090621bfac80beab5f6b26e4c.png" alt="图片5" data-base62-sha1="2WhKR3CmtOCnXJOgQ7VF9wa64JC" width="491" height="308"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66bcdb62138744050129d7f768128fedfd41bc7e.png" alt="图片6" data-base62-sha1="eERkdMUZrAeKoxcbIsO7sHDUbHo" width="496" height="299"></p>
<p>The model was trained 12 times (manual training), and the above 6 images were obtained. Each graph shows the update of weight B.<br>
It can be seen that in the first five training, the value of weight B has been changing. But in the sixth training, the weight B did not change. From the 6th to the 12th training, the weight B still did not change and remained at -0.5233551.<br>
The following is the 【loss curve】 from the 7th to the 12th. (in order to reduce the space, I merged these 6 images)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30b9a5f6d56ca8ed06c09089510547f7a9c7f256.png" alt="图片7" data-base62-sha1="6X2FqEGDKqZH2bQUKTNugwUsx0O" width="528" height="326"></p>
<p><strong>Does this mean that since the sixth training, the whole model has not learned anything new from the data, and the model has reached its best state? If so, why are the losses still changing?</strong></p>

ptrblck: <p>If the 6 images correspond to epochs, then this particular parameter might not get (large) gradients anymore and is thus constant.<br>
If the loss is still decreasing, other parts of the model seem to be trained further.</p>

hpf: <p>thanks，I think maybe you are right. I can’t find any other reasonable explanation.</p>

Ohm: <p>Hi,<br>
Is there any way to simply convert all wights of the PyTorch’s model into a single vector? (the model has conv, pool, and … each of which has their own weights)<br>
(For sure the dimension of a resulted vector will be 1 * n in which the n represents all number of weights in PyTorch’s model).</p>

ptrblck: <p>You could flatten each parameter, append it in a <code>list</code>, and create a tensor:</p>
<pre><code class="lang-python">model = models.resnet18()

params = []
for param in model.parameters():
    params.append(param.view(-1))
params = torch.cat(params)
print(params.shape)
&gt; torch.Size([11689512])
</code></pre>

Ohm: <aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="77672">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">params = []
for param in model.parameters():
    params.append(param.view(-1))
params = torch.cat(params)
print(params.shape)
</code></pre>
</blockquote>
</aside>
<p>AttributeError: ‘Tensor’ object has no attribute ‘append’</p>

ptrblck: <p>Based on the error I guess you might have redefined <code>params</code> as a tensor in your code.<br>
Could you check the <code>type</code> of <code>params</code> and make sure it stays a <code>list</code> until you use <code>torch.cat</code> to create the tensor?</p>

Ohm: <p>It works. Thank you!!</p>

Ohm: <p>Is there any simple way to rearrange it again to the initial shape?(after we get params = torch.cat(params))</p>
<p>I am using “params[0].view(param.data.shape)” and … to do this.</p>

ptrblck: <p>You would have to store the original shapes with the parameters in order to reshape them, as this information is lost if you flatten all parameters to a single flat tensor.</p>

SungmanHong: <p>Hi, could share the way that how to save weight and bias into graph?<br>
Because I want to check every weight cuz i want to freeze some layers and want to check it is done well<br>
Thanks you.</p>

hpf: <p>graph? you mean  picture or computation Graph?<br>
I only know the general way to view it:<br>
View the weight and bias parameters for all networks:</p>
<pre><code class="lang-auto">model.state_dict()
</code></pre>
<p>Save model parameters:<br>
<code> torch.save(model.state_dict(), '\model.pth')</code></p>

Sourabh: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you for the answer. I have been trying to use your answer and try to implement something like weight clustering for some model optimization experiments for my project. I have posed a question for the same at <a href="https://discuss.pytorch.org/t/weight-clustering/142155">Weight clustering question</a>.As this is the most relevant answer that I have found I was wondering if you could confirm if the way that I am trying actually makes sense or is there a better way or an alternate way to achieve the same. It would be great if you could help me out, please.</p>

nivedita_shrivastava: <p>Hi,<br>
I wish to store all the model parameters and biases in a NumPy array. I have used the following approach.</p>
<p>for param_tensor in model.state_dict():<br>
double_x = model.state_dict()[param_tensor].numpy()</p>
<p>Can someone verify if this is the correct way? Additionally, what is the datatype of the single value of the bias?</p>

ptrblck: <p>The code looks generally correct, but note that the <code>dtype</code> would be the one you’ve used in your PyTorch model, so <code>float32</code> by default unless you’ve explicitly used another format.</p>