Noob questions where is the gradient stored during backpropagation?

GGPYTORCH000: <p>I know gradients are used to update weights during next fwd pass. But looking at two examples I have, I am confused at where the gradients are stored during backpropagation in typical training model in pytorch?<br>
From example1 below which sort of simulates very simple backpropagation, this gives me an impression that that tensor a/b should have stored it. I can see after Q.backward is called (I guess that is possible when operands of Q function has gradient), the print(a.grad, g.brad) returns non-null number before that it returned null. So that gives me an impression it is stored in Q’s operands.</p>
<blockquote>
<h1><a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">A Gentle Introduction to torch.autograd — PyTorch Tutorials 2.3.0+cu121 documentation</a></h1>
<p>import torch<br>
a = torch.tensor([2, 3, 12], dtype=torch.float32, requires_grad=True)<br>
b = torch.tensor([6, 4, 5], dtype=torch.float32, requires_grad=True)<br>
Q = 3*a<strong>3 - b</strong>2<br>
external_grad = torch.tensor([1, 1, 1], dtype=torch.float32)<br>
Q.backward(gradient=external_grad)</p>
</blockquote>
<p>But in example2 here with very simple resnet model with input data, it is less clear where the gradients are stored whereas in example1, I can know where it is stored and can see its value before and after backward() is called:</p>
<blockquote>
<p>import torch, torchvision<br>
model = torchvision.models.resnet18(pretrained=True)<br>
data = torch.rand(1, 3, 64, 64)<br>
labels = torch.rand(1, 1000)<br>
prediction = model(data)<br>
loss = (prediction - labels).sum()<br>
loss.backward()</p>
</blockquote>
<p>Are gradients stored somewhere in the model along with weights? If so, how to print out?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/ggpytorch000">@GGPYTORCH000</a>,</p>
<p>The gradient values are stored within the <code>.grad</code> attribute of a Tensor, and are initialized as <code>None</code> when you create the <code>a</code> and <code>b</code> Tensors. When you call <code>loss.backward()</code> you populate the <code>.grad</code> attribute with the gradient of the loss with respect to the variable.</p>
<p>In the case of example 2, the gradients are found in the <code>.grad</code> attribute of the <code>model.parameters()</code> generator. If you want to see these gradients you can do something like,</p>
<pre><code class="lang-auto">for param in model.parameters():
  print(param.grad)
</code></pre>

GGPYTORCH000: <p>Thx, that put out lot of data, so i converted to numpy and printed shapes, and datas are filled after calling backward(), as expected. This is what i got below. Apparently there are various shapes, i am looking to see if i can print out which gradients are assoc-d with certain layers etc.,</p>
<p>def printModelGrad(pModel):<br>
print(“Grads:”)<br>
for param in pModel.parameters():<br>
print(np.array(param.grad).shape)</p>
<p>…</p>
<p>Grads:<br>
(64, 3, 7, 7)<br>
(64,)<br>
(64,)<br>
(64, 64, 3, 3)<br>
(64,)<br>
(64,)<br>
(64, 64, 3, 3)<br>
(64,)<br>
(64,)<br>
(64, 64, 3, 3)<br>
(64,)<br>
(64,)<br>
(64, 64, 3, 3)<br>
(64,)<br>
(64,)<br>
(128, 64, 3, 3)<br>
(128,)<br>
(128,)<br>
(128, 128, 3, 3)<br>
(128,)<br>
(128,)<br>
(128, 64, 1, 1)<br>
(128,)<br>
(128,)<br>
(128, 128, 3, 3)<br>
(128,)<br>
(128,)<br>
(128, 128, 3, 3)<br>
(128,)<br>
(128,)<br>
(256, 128, 3, 3)<br>
(256,)<br>
(256,)<br>
(256, 256, 3, 3)<br>
(256,)<br>
(256,)<br>
(256, 128, 1, 1)<br>
(256,)<br>
(256,)<br>
(256, 256, 3, 3)<br>
(256,)<br>
(256,)<br>
(256, 256, 3, 3)<br>
(256,)<br>
(256,)<br>
(512, 256, 3, 3)<br>
(512,)<br>
(512,)<br>
(512, 512, 3, 3)<br>
(512,)<br>
(512,)<br>
(512, 256, 1, 1)<br>
(512,)<br>
(512,)<br>
(512, 512, 3, 3)<br>
(512,)<br>
(512,)<br>
(512, 512, 3, 3)<br>
(512,)<br>
(512,)<br>
(1000, 512)<br>
(1000,)<br>
optim:  SGD (<br>
Parameter Group 0<br>
dampening: 0<br>
differentiable: False<br>
foreach: None<br>
fused: None<br>
lr: 0.01<br>
maximize: False<br>
momentum: 0.9<br>
nesterov: False<br>
weight_decay: 0<br>
)</p>

AlphaBetaGamma96: <p>If you want to look at certain layers you can save the gradients within a <code>dict</code> and just use the layer’s id as a key to get the gradients for that layer, i.e.,</p>
<pre><code class="lang-auto">grads = {name: param.grad for name, param in model.named_parameters()}
</code></pre>
<p>Then pick a layer via its key, i.e.,</p>
<pre><code class="lang-auto">first_key = list(grads.keys())[0] #change this to any key
print(grads[first_key])
</code></pre>