How to print the computed gradient values for a network

isalirezag: <p>I want to print the gradient values before and after doing back propagation, but i have no idea how to do it.</p>
<p>if i do <code>loss.grad</code> it gives me None.</p>
<p>can i get the gradient for each weight in the model (with respect to that weight)?</p>
<p>sample code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.conv11 = nn.Conv2d(3, 64, 3, padding = 1 )
        self.pool1 = nn.AvgPool2d(2, 2)

        self.conv21 = nn.Conv2d(64, 64*2, 3, padding = 1 )
        self.pool2 = nn.AvgPool2d(2, 2)
        
        self.conv52 = nn.Conv2d(64*2, 10, 1)
        self.pool5 = nn.AvgPool2d(8, 8)
        
    def forward(self, x):
        
        x = F.relu(self.conv11(x))
        x = self.pool1(x)

        x = F.relu(self.conv21(x))
        x = self.pool2(x)
        
        x = self.conv52(x)
        x = self.pool5(x)
        
        x = x.view(-1, 10)
        return x
    

net = Net()
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)
net.to(device)
inputs = torch.rand(4,3,32,32)
labels = torch.rand(4)*10//5
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)
inputs = inputs.to(device)
labels = labels.to(device)

outputs = net(inputs)

loss = criterion(outputs, labels.long() )

print(loss.grad)
loss.backward()
print(loss.grad)

optimizer.step()
 

</code></pre>

ptrblck: <p>Before the first backward call, all <code>grad</code> attributes are set to <code>None</code>. After the first <code>backward</code> you should see some gradient values. Thereafter the gradients will be either zero (after <code>optimizer.zero_grad()</code>) or valid values.</p>

isalirezag: <p>I understand, but why it is not showing the gradient values <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:"><br>
am i doing something wrong:</p>
<pre><code class="lang-auto">
# Initialization
net = Net()
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net.to(device)
# defining loss
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)

#some random input and lables
inputs = torch.rand(4,3,32,32)
labels = torch.rand(4)*10//5
inputs, labels= inputs.to(device), labels.to(device)

# zero_grad
net.zero_grad()
optimizer.zero_grad()

outputs = net(inputs)
loss = criterion(outputs, labels.long() )
print(loss.data)
print(loss.grad)
loss.backward()
print(loss.grad)
optimizer.step()
print(loss.grad)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor(2.3276, device='cuda:0')
None
None
None
</code></pre>

zhl515: <p>Yes, you can get the gradient for each weight in the model w.r.t that weight. Just like this:</p>
<pre><code class="lang-auto">print(net.conv11.weight.grad) 
print(net.conv21.bias.grad)
</code></pre>
<p>The reason you do loss.grad it gives you None is that “loss” is not in optimizer, however, the “net.parameters()” in optimizer.</p>
<pre><code class="lang-auto">optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)
</code></pre>
<p>And “loss” is not leaf node in computation graph, so you  can’t add it to optimizer directly</p>

ptrblck: <p>Sorry for the misunderstanding. I haven’t realized you would like to see the gradient of your <code>loss</code>.<br>
In that case, <a class="mention" href="/u/zhl515">@zhl515</a> is right, and you would need to use hooks to get the gradients w.r.t. some intermediate values (i.e. calculated from leaf variables).<br>
Could you try to add <code>loss.register_hook(lambda grad: print(grad))</code> before the backward call?</p>

isalirezag: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> when i put <code>loss.register_hook(lambda grad: print(grad))  </code> before <code>loss.backward()</code> it gives me <code>tensor(1., device='cuda:0')</code>, is it what it is supposed to show? what intermediate values it is computing the gradient wrt?</p>
<p><a class="mention" href="/u/zhl515">@zhl515</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have a follow up question:</p>
<pre><code class="lang-auto">print(net.conv11.weight.grad) 
</code></pre>
<p>let me to print the grad values for conv11.weights, if i want to set these weights value to zero i thought i can do this:</p>
<pre><code class="lang-auto">Temp = net.conv11.weight.grad =net.conv11.weight.grad.clone()
net.conv11.weight.grad = torch.zeros(Temp.size())
</code></pre>
<p>but it is throwing</p>
<blockquote>
<p>RuntimeError: assigned grad has data of a different type</p>
</blockquote>
<p>can you please let me know your suggestion on that?</p>
<p>thanks</p>
<p>update:</p>
<p>I noticed that the second question is solved when i do the following <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">net.conv11.weight.grad = torch.zeros(Temp.size()).to(device)
</code></pre>

ptrblck: <aside class="quote no-group" data-username="isalirezag" data-post="6" data-topic="34179">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/fbc32d/48.png" class="avatar"> isalirezag:</div>
<blockquote>
<p>when i put <code>loss.register_hook(lambda grad: print(grad))</code> before <code>loss.backward()</code> it gives me <code>tensor(1., device='cuda:0')</code> , is it what it is supposed to show? what intermediate values it is computing the gradient wrt?</p>
</blockquote>
</aside>
<p>Yes, as you are calculating <code>dLoss/dLoss = 1</code>. Note that you can also pass this gradient directly to your <code>backward</code> call.</p>

isalirezag: <p>wow <img src="https://discuss.pytorch.org/images/emoji/apple/sunglasses.png?v=12" title=":sunglasses:" class="emoji" alt=":sunglasses:" loading="lazy" width="20" height="20"><br>
so cool!<br>
thanks</p>
<p>the following code making it more clear for myself, maybe it helps others too</p>
<pre><code class="lang-auto">from torch import FloatTensor
from torch.autograd import Variable


# Define the leaf nodes
a = Variable(FloatTensor([4]))

weights = [Variable(FloatTensor([i]), requires_grad=True) for i in (2, 5, 9, 7)]

# unpack the weights for nicer assignment
w1, w2, w3, w4 = weights

b = w1 * a
c = w2 * a
d = w3 * b + w4 * c
L = (10 - d)

L.register_hook(lambda grad: print(grad)) 
d.register_hook(lambda grad: print(grad)) 
b.register_hook(lambda grad: print(grad)) 
c.register_hook(lambda grad: print(grad)) 
b.register_hook(lambda grad: print(grad)) 

L.backward()


for index, weight in enumerate(weights, start=1):
    gradient, *_ = weight.grad.data
    print(f"Gradient of w{index} w.r.t to L: {gradient}")
</code></pre>
<blockquote>
<p>tensor([1.])<br>
tensor([-1.])<br>
tensor([-7.])<br>
tensor([-9.])<br>
tensor([-9.])<br>
Gradient of w1 w.r.t to L: -36.0<br>
Gradient of w2 w.r.t to L: -28.0<br>
Gradient of w3 w.r.t to L: -8.0<br>
Gradient of w4 w.r.t to L: -20.0</p>
</blockquote>

Paul_Gureghian: <p>Print the ‘state_dict_keys’ for the model, then print the specific key and get the values.</p>

K_Kk: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. Before training we zero the optimizer values. What is the need for it? I understand why we do it after updating the weights. But don’t know why we do that before training.</p>

ptrblck: <p>I would argue it depends a bit on your coding style. We recently had a discussion about it <a href="https://discuss.pytorch.org/t/the-most-common-question-on-optimizer-zero-grad-just-re-confirming-my-understanding/45062/6">here</a>.</p>

nonoc: <p>This is not gradient value, in fact it is parameter value.</p>

RoseFun: <p>Here, you print the grad after <code>register_hook</code>, so how to keep the grad of <code>d,b,c</code> as a variable？</p>

Spencerfonbuena: <p>hey <a class="mention" href="/u/ptrblck">@ptrblck</a> I know this is an old thread, but I am wondering if there is a way to print gradients, even if their nested away from the training script. So far I have only been able to get the weights if they are immediately available. (sorry for the vague language, I’m not too familiar with the syntax I should be using) but hopefully this will make it clearer.<br>
In my program, this instantiates the model, and is part of my run.py script:<br>
net = Transformer(window_size=window_size,<br>
timestep_in=d_input,<br>
channel_in=d_channel,<br>
heads=heads,<br>
d_model=d_model,<br>
device=DEVICE,<br>
dropout = dropout,<br>
class_num=d_output,<br>
stack=stack,<br>
p=p,<br>
).to(DEVICE)</p>
<p>The transformer model is part of my transformer.py script. This is the call to the model during training:<br>
for i, (x, y) in enumerate(train_dataloader):<br>
x, y = x.to(DEVICE), y.to(DEVICE)<br>
optimizer.zero_grad()<br>
y_pre = net(x)<br>
loss = loss_function(y_pre, y)<br>
loss.backward()</p>
<p>This is the code inside my transformer.py script, that calls to pytorch’s implementation of the encoder layer</p>
<p>self.channel_tower = ModuleList([<br>
TransformerEncoderLayer(<br>
d_model=d_model,<br>
nhead=heads,<br>
dim_feedforward=4 * d_model,<br>
dropout=dropout,<br>
activation=F.gelu,<br>
batch_first=True,<br>
norm_first=True,<br>
device=device<br>
) for _ in range(stack)<br>
])</p>
<pre><code>    #Timestep Init
    self.timestep_tower = ModuleList([
        TransformerEncoderLayer(
             d_model=d_model,
             nhead=heads,
             dim_feedforward=4 * d_model,
             dropout=dropout,
             activation=F.gelu,
             batch_first=True,
             norm_first=True,
             device=device
        ) for _ in range(stack)
    ])
</code></pre>
<p>and then, the actual weight layers themselves are hidden in the pytorch.nn.transformerencoderlayer package. in this way the actual layers are sort of abstracted away, and seemingly hard to get a hold of and peek at. Do you know of a way to peek at them?</p>

ptrblck: <p>You can still directly access all parameters of this module. Here is a small example accessing some of them:</p>
<pre><code class="lang-python">encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
encoder_layer.self_attn.in_proj_weight
encoder_layer.linear2.weight
encoder_layer.linear1.weight
</code></pre>

Megh_Bhalerao: <p>wandb.watch can help one with monitoring these parameters, that could also be an alternative.</p>