Gradients contaminated by unused inputs

Leander_Thiele: <p>I have the following minimal example:</p>
<pre><code class="lang-auto">import torch

torch.manual_seed(137)

L = torch.nn.Linear(8, 1)
x = torch.cat((torch.rand((4, 8)), torch.full((4,8), float('nan'))), dim=0)
x = L(x)[:4].sum()
x.backward()

print(L.weight.grad)
</code></pre>
<p>The gradients are NaN, although the output does not depend on the NaN entries.<br>
Is this behaviour expected?<br>
(In more complex architectures the same behaviour can be achieved by setting the unused entries to values that are close to overflowing the 32bit floats.)</p>

KFrank: <p>Hi Leander!</p>
<aside class="quote no-group" data-username="Leander_Thiele" data-post="1" data-topic="135053" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leander_thiele/48/30837_2.png" class="avatar"> Leander_Thiele:</div>
<blockquote>
<p>The gradients are NaN, although the output does not depend on the NaN entries.<br>
Is this behaviour expected?</p>
</blockquote>
</aside>
<p>I would say that it’s expected, although in some cases, such as yours,<br>
maybe not desirable.</p>
<p>The cause is that pytorch computes gradients during the backward<br>
pass by applying the chain rule <em>numerically.</em></p>
<p>Let’s look at the gradient of the slicing operation:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; a = torch.ones (8, requires_grad = True)
&gt;&gt;&gt; a
tensor([1., 1., 1., 1., 1., 1., 1., 1.], requires_grad=True)
&gt;&gt;&gt; a[:4].sum().backward()
&gt;&gt;&gt; a.grad
tensor([1., 1., 1., 1., 0., 0., 0., 0.])
</code></pre>
<p>That’s reasonable – the elements that were not included in the slice<br>
have zero gradient.</p>
<p>We now backpropagate the gradient of the slicing operation through<br>
the application of the <code>Linear</code> layer, <code>L</code>.  Simplifying things a bit, you<br>
are calculation <code>L.weight @ x</code>.  The gradient (more precisely, the<br>
Hessian) of this expression (with respect to <code>L.weight</code>) is, in essence,<br>
<code>x</code>.  Letting <code>g</code> be the gradient of the slicing operation, the chain rule<br>
gives us, more or less, <code>x @ g</code>.</p>
<p>The problem is by the time <code>g</code> gets to this step in the backpropagation,<br>
autograd no longer knows that some elements were ignored by the<br>
slicing operation – all it knows is that some elements of <code>g</code> happen<br>
to be <code>0.0</code>.  (Perhaps those zeros were calculated numerically, e.g.,<br>
<code>2.0 - 6.0 / 3.0</code>, rather than coming from an “ignore” operation.)</p>
<p>According to the highly desirable rules of floating-point arithmetic,<br>
<code>nan * 0.0 = nan</code>.  Autograd doesn’t know that it’s supposed to<br>
ignore those <code>nan</code>s in <code>x</code>, rather, it’s supposed to multiply them by <code>0.0</code><br>
and thus will get <code>nan</code>s as the results.</p>
<p>In a hypothetical world, floating-point numbers could have a special<br>
<code>ignore</code> value, in addition to things like <code>nan</code> and <code>inf</code>.  In such a<br>
world, the gradient of the slicing operation could have <code>ignore</code> for<br>
the gradient of the elements not included in the slice.  Then, perhaps,<br>
autograd could say that <code>nan * ignore = 0.0</code> (or something), and<br>
you would get the result you were hoping for.  But we don’t live in<br>
that hypothetical world.</p>
<p>Best.</p>
<p>K. Frank</p>

Leander_Thiele: <p>Thank you for the detailed explanation! It makes perfect sense, the result is just a bit counter-intuitive.</p>