How to get all the weight (not all paramerters)

111179: <p>Hi, I would like to know the max(abs(weight)) of every epoch. As show in figure.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e571bd9b14904bc575a21b8dc8c0d2c23b44fee6.png" alt="bit位数" data-base62-sha1="wJL1NgJULM0FbYIdm0UZcxUPKke" width="506" height="356">.<br>
I think I can get max(abs(weight)) in epoch loop, for example:</p>
<pre><code>for epoch in range(1, epochs + 1):
    train_loss,t_accuracy = train(model, train_loader, optimizer)
   test_loss, accuracy = test(model, test_loader)
   epoch_nums.append(epoch)
   training_loss.append(train_loss)
   validation_loss.append(test_loss)
   test_accuracy.append(accuracy)
   train_accuracy.append(t_accuracy)
  **output_1=model.fc1.weight.data**
  if (epoch) % 1 == 0:
      print('Epoch {:d}: Training loss= {:.4f}, Validation loss= {:.4f},train_accuracy={:.4%},    Accuracy={:.4%}'.format(epoch, train_loss,test_loss,t_accuracy,accuracy))
    *print(max(abs(output_1)))*
</code></pre>
<p><strong>But, I can only get the one layer’s max(abs(weight)),  can you give me advice to get the all weights of the model, not the parameters!</strong></p>
<p>Thank you very much for your comment and your time!!!</p>

yash1994: <p>Hi,</p>
<p>You can iterate through model state_dict and get max of all weight matrix. Something like this:</p>
<pre><code class="lang-python">for k, v in model.state_dict().items():
    if "weight" in k:
        print(max(abs(v)))
</code></pre>

111179: <p>Thank you for your reply. <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:"><br>
This is indeed a good idea…but when I try this comment in weight, I get the error: “bool value of Tensor with more than one value is ambiguous”…<br>
I can get the max(abs(bias))…maybe I know the problem, but I don’t know how to solve it <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>
<p>Thank you again…<br>
Best regards</p>

yash1994: <p>Hi,</p>
<p>Sorry for the late reply. You need to determine dimension along which you need the max value from the weight tensor becuase different layers will have different multiple dimensions, for example linear layers will have <code>(out_features,in_features)</code> shape with 2 dimensions.</p>

111179: <p>Oh, no, no…don’t need to say sorry, your help is my pleasure.<br>
Maybe I can understand your meaning, but I don’t know how to specify one dimension…<br>
I am so sorry I am a new beginner <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><br>
But I try…</p>
<p>Thank you!</p>

yash1994: <p>No worries, consider following example.</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randint(10, (3,3))
tensor([[8, 4, 5],
        [3, 0, 3],
        [9, 4, 6]])
</code></pre>
<p>consider x is your weight matrix. And the shape of x is <code>torch.Size([3, 3])</code>.<br>
when you try to find max value with <code>x.max(dim=0)</code> where <code>dim</code> is the argument along which you are asking for max value, which in our case is <code>0</code> means we are asking for max values from all the columns so it will return</p>
<pre><code class="lang-auto">tensor([9, 4, 6])
</code></pre>
<p>And when you try to find max value with <code>x.max(dim=1)</code>, in this case, <code>dim</code> is <code>1</code> means we are asking for max values across all the rows so it will return</p>
<pre><code class="lang-auto">tensor([8, 3, 9])
</code></pre>
<p>So this difference i was trying to convay by</p>
<aside class="quote no-group" data-username="yash1994" data-post="4" data-topic="74897">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yash1994/48/21611_2.png" class="avatar"> yash1994:</div>
<blockquote>
<p>You need to determine dimension along which you need the max value</p>
</blockquote>
</aside>

111179: <p>I get it!!!<br>
Thank you so much!!</p>

jmandivarapu1: <aside class="quote no-group" data-username="111179" data-post="3" data-topic="74897">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111179/48/22195_2.png" class="avatar"> 111179:</div>
<blockquote>
<p>is comment in weight, I get the error: “bool value of Tensor with more than one value is ambiguous”…<br>
I can get the max(abs(bias))…maybe I know</p>
</blockquote>
</aside>
<p>You can flatten the model then the max of the numpy aray. Just pass your net to below function it will give you fattened numpy array of model and al it’s shapes. Then you can apply all numpy operations on that.</p>
<pre><code class="lang-auto">#############################################################################
# Flattening the NET
#############################################################################
def flattenNetwork(net):
    flatNet = []
    shapes = []
    for param in net.parameters():
        #if its WEIGHTS
        curr_shape = param.cpu().data.numpy().shape
        shapes.append(curr_shape)
        if len(curr_shape) == 2:
            param = param.cpu().data.numpy().reshape(curr_shape[0]*curr_shape[1])
            flatNet.append(param)
        elif len(curr_shape) == 4:
            param = param.cpu().data.numpy().reshape(curr_shape[0]*curr_shape[1]*curr_shape[2]*curr_shape[3])
            flatNet.append(param)
        else:
            param = param.cpu().data.numpy().reshape(curr_shape[0])
            flatNet.append(param)
    finalNet = []
    for obj in flatNet:
        for x in obj:
            finalNet.append(x)
    finalNet = np.array(finalNet)
    return finalNet,shapes
</code></pre>

111179: <p>I have try it. That’s great! Thank you very much!</p>

Ajinkya_Bankar: <p>Dear Sir, I am using your function flattenNetwork() to get the network parameters/weights and shape. I manipulate the weights and I don’t change the shape of the network. How can I get back the original network with manipulated weights? Please help. Thank you.</p>

jmandivarapu1: <aside class="quote no-group" data-username="Ajinkya_Bankar" data-post="10" data-topic="74897">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/d9b06d/48.png" class="avatar"> Ajinkya_Bankar:</div>
<blockquote>
<p>How can I get back the original network with manipulated weights? Please help. Thank you.</p>
</blockquote>
</aside>
<p>Not sure what you mean exactly. Do you mean you want to place updated weights in the network back?</p>