When is the grad attribute of a tensor updated

ayl: <p>The code is as follows:</p>
<pre><code class="lang-auto">import torch
tensor_to_name = {}

def make_tensor_hook(tensor):
    def tensor_hook(grad):
        if tensor.requires_grad:
            print("\ntensor_hook", tensor_to_name[tensor])
            print("\tgrad", grad)
            for t, name in tensor_to_name.items():
                print("\t", name, t, t.grad)
        else:
            print("tensor doesn't require grad", tensor_to_name[tensor])
    return tensor_hook

a = torch.tensor([1.0, 2.0], requires_grad=True)
b = a + a 
c = b.mean()
b.retain_grad()
c.retain_grad()
tensor_to_name[a] = "a"
tensor_to_name[b] = "b"
tensor_to_name[c] = "c"
a.register_hook(make_tensor_hook(a)) 
b.register_hook(make_tensor_hook(b)) 
c.register_hook(make_tensor_hook(c)) 
c.backward()
</code></pre>
<p>After we executing the code, we get:</p>
<pre><code class="lang-auto">tensor_hook c
         grad tensor(1.)
         a tensor([1., 2.], requires_grad=True) None
         b tensor([2., 4.], grad_fn=&lt;AddBackward0&gt;) None
         c tensor(3., grad_fn=&lt;MeanBackward0&gt;) None

tensor_hook b
         grad tensor([0.5000, 0.5000])
         a tensor([1., 2.], requires_grad=True) None
         b tensor([2., 4.], grad_fn=&lt;AddBackward0&gt;) None
         c tensor(3., grad_fn=&lt;MeanBackward0&gt;) tensor(1.)

tensor_hook a
         grad tensor([1., 1.])
         a tensor([1., 2.], requires_grad=True) None
         b tensor([2., 4.], grad_fn=&lt;AddBackward0&gt;) tensor([0.5000, 0.5000])
         c tensor(3., grad_fn=&lt;MeanBackward0&gt;) tensor(1.)
</code></pre>
<p>According to the above output, we can see the <code>grad</code> attribute of a tensor is <code>None</code> during its corresponding hook. For example, <code>c.grad</code> is <code>None</code> during the registered hook while gets its value during the hook of <code>b</code>. The doc <a href="https://pytorch.org/docs/stable/notes/autograd.html#the-order-in-which-the-different-hooks-are-fired" rel="noopener nofollow ugc">Backward Hooks execution</a> says the <code>grad</code> field is updated after the prehook of the node, but it’s too vague for me. <strong>So the question is that when the grad attribute of a tensor is updated and where the corresponding source code</strong>.</p>

ptrblck: <p>Gradient hooks allow you to manipulate the gradient before it’s being accumulated so it’s expected to see a <code>None</code> gradient attribute inside the hook.</p>

ayl: <p>Can you help me locate the specific codes for updating the <code>grad</code> field?</p>
<p>From the following source code (<a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/autograd/engine.cpp#L912" rel="noopener nofollow ugc">torch/csrc/autograd/engine.cpp</a>) I found, I can identify the ordering of <code>tensor_pre_hooks</code>, <code>pre_hooks</code> and <code>post_hooks</code>. But I cannot find when and where the engine updates the <code>grad</code> field.</p>
<pre><code class="lang-auto">static variable_list call_function(
    std::shared_ptr&lt;GraphTask&gt;&amp; graph_task,
    Node* func,
    InputBuffer&amp; inputBuffer) {
  CheckpointValidGuard cpvguard(graph_task);
  auto&amp; fn = *func;
  auto inputs =
      call_tensor_pre_hooks(fn, InputBuffer::variables(std::move(inputBuffer)));
  inputs = call_pre_hooks(fn, std::move(inputs));
  if (!graph_task-&gt;keep_graph_) {
    fn.will_release_variables();
  }

  const auto has_post_hooks = !fn.post_hooks().empty();
  variable_list outputs;

  if (has_post_hooks) {
    // In functions/accumulate_grad.cpp, there is some logic to check the
    // conditions under which the incoming gradient can be stolen directly
    // (which elides a deep copy) instead of cloned. One of these conditions
    // is that the incoming gradient's refcount must be 1 (nothing else is
    // referencing the same data).  Stashing inputs_copy here bumps the
    // refcount, so if post hooks are employed, it's actually still ok for
    // accumulate_grad.cpp to steal the gradient if the refcount is 2.
    //
    // "new_grad.use_count() &lt;= 1 + !post_hooks().empty()" in
    // accumulate_grad.cpp accounts for this, but also creates a silent
    // dependency between engine.cpp (ie, this particular engine
    // implementation) and accumulate_grad.cpp.
    //
    // If you change the logic here, make sure it's compatible with
    // accumulate_grad.cpp.
    auto inputs_copy = inputs;
    outputs = fn(std::move(inputs_copy));
  } else {
    outputs = fn(std::move(inputs));
  }

  validate_outputs(fn.next_edges(), outputs, [&amp;](const std::string&amp; msg) {
    std::ostringstream ss;
    ss &lt;&lt; "Function " &lt;&lt; fn.name() &lt;&lt; " returned an " &lt;&lt; msg;
    return ss.str();
  });

  if (has_post_hooks) {
    // NOLINTNEXTLINE(bugprone-use-after-move)
    return call_post_hooks(fn, std::move(outputs), inputs);
  }
  return outputs;
}
</code></pre>

ptrblck: <p>The <code>accumulateGrad</code> function should perform the actual accumulation from <a href="https://github.com/pytorch/pytorch/blob/d0fe8fa5db6dd06adfe1246a72b6d3a5215ff86e/torch/csrc/autograd/functions/accumulate_grad.h#L62-L116">here</a>.</p>

ayl: <p>Thanks. I also find that  <code>retains_grad_hooks()</code> which are called in <code>call_tensor_pre_hooks</code> actually update the <code>grad</code> field.</p>