Where does the ctx variable come from?

username: Andrew_Holmes
cooked: <p>I want to understand how autograd works a little bit on the backend, but I have no idea how the <code>ctx</code> variable gets passed for custom <code>Functions</code>. I understand that it’s the backward object that correlates to a Function, but where in the source code are you guys passing the ctx to the <code>forward()</code> and <code>backward()</code> methods so that it’s accessible to users?</p>
<p><strong>Example:</strong></p>
<pre><code class="lang-python">import torch
from torch.autograd import Function


class MulConstant(Function):
    @staticmethod
    def forward(tensor, constant):
        return tensor * constant

    @staticmethod
    def setup_context(ctx, inputs, output):
        # ctx is a context object that can be used to stash information
        # for backward computation
        tensor, constant = inputs
        print(ctx, type(ctx), ctx.__class__.__name__, sep="\n")
        ctx.constant = constant

    @staticmethod
    def backward(ctx, grad_output):
        # We return as many input gradients as there were arguments.
        # Gradients of non-Tensor arguments to forward must be None.
        return grad_output * ctx.constant, None


def mul_constant(tensor, c=1):
    return MulConstant.apply(tensor, c)


tensor = torch.ones((5, 1), requires_grad=True)
result = mul_constant(tensor, c=10)
</code></pre>
<p><strong>Output:</strong></p>
<pre><code class="lang-auto">&lt;torch.autograd.function.MulConstantBackward object at 0x114199300&gt;
&lt;class 'torch.autograd.function.MulConstantBackward'&gt;
MulConstantBackward
</code></pre>

username: tom
cooked: <p>Hi Andrew,</p>
<p><code>Function.apply</code> creates the ctx as an instance of the backward node class, this is relatively deep in the C++ guts of the autograd engine, below is the C++ implementation of <code>Function.apply</code>.<br>
I used to offer an “All about autograd” course, but sadly, I have not updated it to PT2 yet, so it is missing AOTAutograd other things that came after 2021.</p>
<p>Best regards</p>
<p>Thomas</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/0276d5621ad7cec93a54929ab7221b3bba050dcb/torch/csrc/autograd/python_function.cpp#L1043">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/0276d5621ad7cec93a54929ab7221b3bba050dcb/torch/csrc/autograd/python_function.cpp#L1043" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/0276d5621ad7cec93a54929ab7221b3bba050dcb/torch/csrc/autograd/python_function.cpp#L1043" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/0276d5621ad7cec93a54929ab7221b3bba050dcb/torch/csrc/autograd/python_function.cpp#L1043</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="1033" style="counter-reset: li-counter 1032 ;">
          <li></li>
          <li>  // setup_context gets "leaked" - we return a new reference and hold onto it</li>
          <li>  // forever.</li>
          <li>  auto setup_context = PyObject_GetAttrString(function, "setup_context");</li>
          <li>  if (!setup_context)</li>
          <li>    return nullptr;</li>
          <li>  THPFunction_setup_context = setup_context;</li>
          <li>  return THPFunction_setup_context;</li>
          <li>}</li>
          <li></li>
          <li class="selected">PyObject* THPFunction_apply(PyObject* cls, PyObject* inputs) {</li>
          <li>  HANDLE_TH_ERRORS</li>
          <li></li>
          <li>  // save a local copy of seq_id before it gets incremented</li>
          <li>  auto seq_id = at::sequence_number::peek();</li>
          <li>  auto info_pair = unpack_input&lt;false&gt;(inputs);</li>
          <li>  UnpackedInput&amp; unpacked_input = info_pair.first;</li>
          <li>  InputFlags&amp; input_info = info_pair.second;</li>
          <li></li>
          <li>  // Call record function after all the inputs have been decoded, but</li>
          <li>  // before context has been allocated.</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Andrew_Holmes
cooked: <p>Hey <a class="mention" href="/u/tom">@tom</a>,</p>
<p>This was pretty helpful. It just peaked my curiosity because I just started learning about autograd and was just really confused on where exactly it originated from. Honestly, I’d still like to check out some parts of your course since I want to learn about eager mode autograd. Do you have a link to it so I can check out some of it?</p>
<p>Thanks,<br>
Andrew</p>

username: tom
cooked: <p>I sent you an invite link via PM.</p>
<p>Best Regards</p>
<p>Thomas</p>

username: luciaquirke
cooked: <p>Hey <a class="mention" href="/u/tom">@tom</a>, could I please get the link too? Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>