Error with manually traversing computational graph

k.chen: <p>I have a simple linear regression model and I am running one iteration of training on it.</p>
<pre><code class="lang-auto"># model definition
model = torch.nn.Linear(W_target.size(0), 1)
print(model.state_dict())

# zero gradient
model.zero_grad()

# forward pass and compute loss
y_preds = model(batch_x)  # computes y = xA^T + b
output = F.smooth_l1_loss(y_preds, batch_y)
loss = output.item()
</code></pre>
<p>Then I attempt to manually perform <code>loss.backward()</code> as such:</p>
<pre><code class="lang-auto">    dloss = torch.tensor(1.)
    print("output2.grad_fn:", output.grad_fn)
    smooth_l1_loss_backward = output.grad_fn
    dz = smooth_l1_loss_backward(dloss)[0]
    print(dz)
    print("smooth_l1_loss_backward.next_functions:", smooth_l1_loss_backward.next_functions)
    print(id(smooth_l1_loss_backward.next_functions[0][0]))
    assert smooth_l1_loss_backward.next_functions[0][0] == y_preds.grad_fn
    addmm_backward = smooth_l1_loss_backward.next_functions[0][0]
    print(addmm_backward)
    print("addmm_backward.next_functions:", addmm_backward.next_functions)
    d_ypreds = addmm_backward(dz)
    print(d_ypreds)
    d_bias, d_sample, d_weight_t = d_ypreds
    back_bias = addmm_backward.next_functions[0][0]
    transpose_backward = addmm_backward.next_functions[2][0]
    print(transpose_backward.next_functions)
    d_weight = transpose_backward(d_weight_t)
    back_weight = transpose_backward.next_functions[0][0]
    back_bias(d_bias)
    back_weight(d_weight)
    print(model.bias.grad)
    print(model.weight.grad)
</code></pre>
<p>However back_bias(d_bias) is not working due to a size error. Also, for some reason <code>dz = smooth_l1_loss_backward(dloss)[0]</code> is a tensor with <code>grad_fn</code>. Should that be happening?</p>
<p>In a tutorial that I found</p>
<pre><code class="lang-auto">x = torch.randn(4, 4, requires_grad=True)
y = torch.randn(4, 4, requires_grad=True)
z = x * y
l = z.sum()
dl = torch.tensor(1.)
back_sum = l.grad_fn
dz = back_sum(dl)
back_mul = back_sum.next_functions[0][0]
dx, dy = back_mul(dz)
back_x = back_mul.next_functions[0][0]
back_x(dx)
back_y = back_mul.next_functions[1][0]
back_y(dy)
print(x.grad)
print(y.grad)
</code></pre>
<p><code>dz = back_sum(dl)</code> is Tensor with <code>grad_fn=None</code>.</p>
<p>Please advise. Thank you! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

soulitzer: <p>Yes, that is expected. When you run backward normally it runs it in a no_grad context, and that is why there is no grad normally.</p>

k.chen: <p>So when I run <code>dz = smooth_l1_loss_backward(dloss)[0]</code>, it should be a tensor with <code>grad_fn=None</code> right? But like I said above, I am getting a grad_fn.</p>

soulitzer: <p>No getting a grad_fn is fine. The reason the tutorial is getting grad_fn=None is because they may be doing something that breaks the graph in some way.</p>

k.chen: <p>can you please explain why <code>back_bias(d_bias)</code> is causing the error <code>RuntimeError: output with shape [1] doesn't match the broadcast shape [32, 1]</code>. I don’t understand why <code>d_bias</code> is at a shape <code>[32, 1]</code>.</p>

soulitzer: <p>d_bias should have the same shape as bias</p>

k.chen: <p>yep I understand that that’s the issue, but I don’t understand the cause of the issue and how to fix it. my gut feeling tells me that it theres an issue with <code>dz = smooth_l1_loss_backward(dloss)[0]</code> because my <code>d_bias</code> is equal to <code>dz</code>.</p>