Why are my tensor's gradients unexpectedly None or not None?

username: suraj.pt
cooked: <h3>tl;dr</h3>
<p>Ensure that</p>
<ul>
<li><code>tensor.is_leaf == True</code></li>
<li>
<code>tensor.requires_grad == True</code>.</li>
<li>tensor.grad_fn is None; if it is not None, you need to <code>retain_grad()</code>.</li>
<li>gradient computation is not disabled using
<ul>
<li>
<code>torch.no_grad()</code> context manager</li>
<li><code>torch.autograd.set_grad_enabled(False)</code></li>
</ul>
</li>
<li>you are not running any non-differentiable operation.</li>
</ul>
<hr>
<p>By default, Autograd populates gradients for a tensor <code>t</code> in <code>t.grad</code> only when <code>t.is_leaf == True</code> and <code>t.requires_grad == True</code>.</p>
<h3>What is a leaf tensor?</h3>
<p>Leaf tensors are tensors at the beginning of the computational graph, which means they are not the outputs of any differentiable operation. A model’s weights and biases, as well as any inputs to it, are all leaf tensors.</p>
<p>Outputs of hidden layers (<em>activations</em>) are <strong>not</strong> leaf tensors, because they are the result of a differentiable op (eg: <code>matmul()</code>). You can see the operation that generated a tensor in <code>tensor.grad_fn</code></p>
<p>Read more about leaf tensors: <a href="https://discuss.pytorch.org/t/what-is-the-purpose-of-is-leaf/87000" class="inline-onebox">What is the purpose of `is_leaf`?</a></p>
<h3>But I need the gradients of intermediate outputs!</h3>
<p>If a tensor is created from an operation that’s <a href="https://discuss.pytorch.org/t/are-all-operations-defined-in-torch-tensor-differentiable/4318">“differentiable” by Autograd</a> - including operations like <code>.to()</code> which don’t look differentiable - it is not a leaf tensor and will not have gradients accumulated by default.</p>
<p>You can explicitly instruct Autograd to accumulate gradients for tensors by calling <code>tensor.retain_grad()</code> before calling <code>.backward()</code>. See this thread for an example:  <a href="https://discuss.pytorch.org/t/method-grad-returns-none-for-a-tensor/91627" class="inline-onebox">Method grad returns None for a tensor</a></p>
<hr>
<h5>!! Gotcha - Avoid using <code>to()</code> for <code>nn.Parameters</code> as they will be deregistered from the model</h5>
<p>You will end up overwriting your leaf <code>nn.Parameter</code> with a non-leaf tensor.</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.param = nn.Parameter(torch.randn(1)).to(torch.float64)
model = MyModel()
print(dict(model.named_parameters())) # empty
</code></pre>
<hr>
<h3>Non-Differentiable operations</h3>
<p>The output of non-differentiable operations will have <code>requires_grad=False</code> even if the inputs have <code>requires_grad=True</code>. Gradients cannot be computed for this operation, and you will see the error <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>. See this thread for an example: <a href="https://discuss.pytorch.org/t/custom-loss-function-gradients-are-none/57181" class="inline-onebox">Custom loss function: gradients are None</a></p>