Question about the Extending PyTorch tutorial

Aakira: <p>Hi, I’m reading the “Extending PyTorch” tutorial and have the following questions:</p>
<ol>
<li>In the document, it says:</li>
</ol>
<blockquote>
<p><code>Tensor</code> arguments that track history (i.e., with <code>requires_grad=True</code>) will be converted to ones that don’t track history before the call, and their use will be registered in the graph.</p>
</blockquote>
<p>Does this happen implicitly, or do I need to do this manually? I see in the examples below when passing parameters into <code>forward()</code>, there is no line to disable the gradient tracking. However, further down, it says:</p>
<blockquote>
<p>Inputs to <code>backward</code>, i.e., <code>grad_output</code>, can also be tensors that track history. So if <code>backward</code> is implemented with differentiable operations, (e.g., invocation of another custom <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" rel="noopener nofollow ugc"><code>Function</code></a>), higher-order derivatives will work.</p>
</blockquote>
<p>This is confusing because I don’t understand if I need to disable tracking manually, why isn’t it done in the example? And if this is done implicitly, how can I enable tracking for a tensor?</p>
<ol start="2">
<li>There’s an example code below that I don’t fully understand the intention of:</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">class MyCube(torch.autograd.Function):
    @staticmethod
    def forward(x):
        # We wish to save dx for backward. In order to do so, it must
        # be returned as an output.
        dx = 3 * x ** 2
        result = x ** 3
        return result, dx

    @staticmethod
    def setup_context(ctx, inputs, output):
        x, = inputs
        result, dx = output
        ctx.save_for_backward(x, dx)

    @staticmethod
    def backward(ctx, grad_output, grad_dx):
        x, dx = ctx.saved_tensors
        # In order for the autograd.Function to work with higher-order
        # gradients, we must add the gradient contribution of `dx`,
        # which is grad_dx * 6 * x.
        result = grad_output * dx + grad_dx * 6 * x
        return result

# Wrap MyCube in a function so that it is clearer what the output is
def my_cube(x):
    result, dx = MyCube.apply(x)
    return result
</code></pre>
<p>Why do we need to return the sum of the first and second-order derivatives? What does this mean?</p>
<ol start="3">
<li>It seems the documentation isn’t written in a very detailed or easy-to-understand way (or maybe I missed something, if so, please explain and forgive me). Even though I’ve referred to many other sources, I still feel there are many components that aren’t explained but are included as examples (or explained as if they are obvious), for example:  <a href="https://pytorch.org/docs/stable/notes/extending.html#:~:text=error.%0A%20%20%20%20%20%20%20%20if%20ctx.-,needs_input_grad,-%5B0%5D%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20grad_input" rel="noopener nofollow ugc">In this line</a>, why is <code>needs_input_grad</code> used even though it hasn’t been initialized? Is it automatically assigned a value based on something else?<br>
Which additional sources should I read to better understand these points?</li>
</ol>
<p>Thanks</p>

soulitzer: <ol>
<li></li>
</ol>
<blockquote>
<p>Does this happen implicitly, or do I need to do this manually?</p>
</blockquote>
<p>The disabling of gradient tracking is done implicitly for you in the scope of forward</p>
<blockquote>
<p>puts to <code>backward</code> , i.e., <code>grad_output</code> , can also be tensors that track history.</p>
</blockquote>
<p>The above refers to forward, but this refers to the backward pass. It is possible for computation  to be enabled, if you are trying to compute higher-order gradients</p>
<ol start="2">
<li></li>
</ol>
<p>You can think of dx as just some random output to that function. If someone ended up using dx in the computation of the final loss, the backward function should properly backprop the gradient computation back to x.</p>
<ol start="3">
<li></li>
</ol>
<blockquote>
<p>Is it automatically assigned a value based on something else?</p>
</blockquote>
<p>That is correct. needs_input_grad and other things on ctx are provided for you and have been preinitialized to have some quantities that may be useful during backward.</p>