Gradients are different on single and double precision

aonurdasdemir: <p>I am implementing my own grad check function as autograd’s built-in one is slow on large tensor input and outputs. What I do is choose some random points on the input tensor and perturbe them to calculate finite differences, then I compare the results with automatic gradient. Here is my implementation,</p>
<pre><code class="lang-auto">def grad_check(model, data, target, loss_fn, eps=1e-3, control_num=3, x_list = None, y_list = None):

    num_grads = []
    orig_grads = []
    output_original = model(data)
    loss_original = loss_fn(torch.squeeze(output_original), torch.squeeze(target))
    print('Loss original: ', loss_original)
    loss_original.backward()
    grad_original = torch.squeeze(data.grad)
    (shape_x, shape_y) = torch.squeeze(data).shape[0], torch.squeeze(data).shape[1]
    if x_list is None and y_list is None:
        x_list = np.random.randint(0, shape_x, size=control_num)
        y_list = np.random.randint(0, shape_y, size=control_num)
    with torch.no_grad():
        for i in range(control_num):
            data_copy = data.clone()
            data_copy[x_list[i],y_list[i]] = data_copy[x_list[i],y_list[i]] + eps
            output_ptb = model(data_copy)
            loss_ptb = loss_fn(torch.squeeze(output_ptb), torch.squeeze(target))
            print('Loss numeric: ', loss_ptb)
            grad_num = (loss_ptb-loss_original)/eps
            num_grads.append(grad_num)
            orig_grads.append(grad_original[x_list[i],y_list[i]])
    print('Numerical grad: ', num_grads)
    print('Original grad: ', orig_grads)
    print('Ratio: ', np.array(num_grads)/np.array(orig_grads))
    return x_list, y_list
</code></pre>
<p>Then I run this function with the following simple model:</p>
<pre><code class="lang-auto">class TestModel(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        out = (x+1)/2
        return out
</code></pre>
<p>When I check the gradients of the above simple model (actually a function without learnable parameters just for demonstration), I observe that while using single precision, gradients don’t match but they match if I use double precision:</p>
<pre><code class="lang-auto"># Double calculations
loss_function =nn.MSELoss(reduction="sum")
mymodel = TestModel().double()
input_tensor = torch.randn(140, 120, dtype=torch.double,  requires_grad=True)
target_tensor = torch.randn(140, 120, dtype=torch.double,  requires_grad=True)

x_list, y_list = grad_check(mymodel, input_tensor, target_tensor, loss_function, eps=1e-3, control_num=5)
</code></pre>
<p>Results with double precision:</p>
<pre><code class="lang-auto">Numerical grad:  [tensor(-0.4685, dtype=torch.float64), tensor(0.6474, dtype=torch.float64), tensor(-0.5744, dtype=torch.float64), tensor(-0.3790, dtype=torch.float64)]
Original grad:  [tensor(-0.4687, dtype=torch.float64), tensor(0.6472, dtype=torch.float64), tensor(-0.5746, dtype=torch.float64), tensor(-0.3792, dtype=torch.float64)]
Ratio:  [0.99946664 1.00038628 0.99956495 0.9993408 ]
</code></pre>
<p>Single precision model:</p>
<pre><code class="lang-auto"># Single precision calculations
mymodel = TestModel()
input_tensor = torch.randn(140, 120, requires_grad=True)
target_tensor = torch.randn(140, 120, requires_grad=True)

grad_check(mymodel, input_tensor, target_tensor, loss_function, eps=1e-3, control_num=5, x_list=x_list,
           y_list=y_list)
</code></pre>
<p>Results with single precision:</p>
<pre><code class="lang-auto">Numerical grad:  [tensor(-1.9531), tensor(0.), tensor(1.9531), tensor(1.9531)]
Original grad:  [tensor(-0.4842), tensor(1.2964), tensor(2.9638), tensor(1.1156)]
Ratio:  [4.033863  0.        0.6590007 1.7507753]
</code></pre>
<p>What is the reason of this difference? Should I not consider the finite difference grad check to verify if my gradients are true or would that difference create a problem when training my model?</p>

KFrank: <p>Hi Ahmet!</p>
<p>The lesson here is that calculating gradients numerically can be nuanced and<br>
requires thoughtful attention to floating-point arithmetic.</p>
<aside class="quote no-group quote-modified" data-username="aonurdasdemir" data-post="1" data-topic="154957" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/50afbb/48.png" class="avatar"> aonurdasdemir:</div>
<blockquote>
<p>What I do is choose some random points on the input tensor and perturbe them to calculate finite differences</p>
<pre><code class="lang-auto">def grad_check(model, data, target, loss_fn, eps=1e-3, control_num=3, x_list = None, y_list = None): 
</code></pre>
<pre><code class="lang-auto"># Single precision calculations
input_tensor = torch.randn(140, 120, requires_grad=True)
target_tensor = torch.randn(140, 120, requires_grad=True)
</code></pre>
</blockquote>
</aside>
<p>Two things to pay attention to:</p>
<p>Your <code>data</code> (and <code>target</code> and the <code>output</code> of your <code>model</code>) are of order 1, and<br>
your individual squared-error values are of order 1.  However, the number<br>
of elements in <code>data</code> is of order <code>1.e4</code>, so your summed loss function<br>
(<code>MSELoss(reduction="sum")</code>) is also of order <code>1.e4</code>.</p>
<p>Your <code>eps</code> is <code>1.e-3</code> and it the change it produces in an individual contribution<br>
to your loss is of order <code>1.e-3</code>, so it produces a <em>relative</em> change in your loss<br>
function of <code>1.e-3 / 1.e4 = 1.e-7</code>.  This is the precision of a single-precision<br>
floating-point number, so floating-point round-off error could easily introduce<br>
a 100% error into the finite difference you calculate.</p>
<p>So, using single-precision arithmetic, you get numerical gradients that are<br>
basically completely wrong.</p>
<p>On the other hand, the precision of  double-precision arithmetic is about<br>
<code>1.e-15</code>, which is precise enough to do a solid job of computing your<br>
numerical gradients with acceptable accuracy.</p>
<p>The short answer – for the use case you posted – is to use double-precision<br>
for your numerical gradients.</p>
<p>If you want to use single-precision, you will need to increase <code>eps</code> (which will<br>
introduce larger non-round-off errors into your gradient computation), or<br>
reduce the number of elements in <code>data</code> (so that the perturbation from any<br>
one element doesn’t get swamped by the total summed value of all the others).</p>
<p>Best.</p>
<p>K. Frank</p>

aonurdasdemir: <p>Got it, thanks for your detailed answer! I have an additional question:</p>
<p>I see that because of the precision of the single-precision floating numbers, numerical errors can be wrong. Would it also affect the autograd’s gradient calculations? For such kind of data and loss function, should I use double-precision always?</p>

KFrank: <p>Hi Ahmet!</p>
<aside class="quote no-group" data-username="aonurdasdemir" data-post="3" data-topic="154957" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/50afbb/48.png" class="avatar"> aonurdasdemir:</div>
<blockquote>
<p>I see that because of the precision of the single-precision floating numbers, numerical errors can be wrong. Would it also affect the autograd’s gradient calculations?</p>
</blockquote>
</aside>
<p>No*, because autograd does not use numerical <em>differentiation</em> to calculate<br>
gradients.</p>
<p>In more detail, when an autograd-supporting function is written, it’s derivative<br>
is calculated analytically, and programmed as a companion “backward”<br>
function.  When <code>.backward()</code> is called to compute gradients, that analytic<br>
derivative is evaluated numerically, and then numerically chained together<br>
with the rest of the computation graph by autograd using calculus’s chain<br>
rule.</p>
<p>By way of example:</p>
<p>The analytic derivative of <code>x**p</code> is <code>p * x**(p - 1)</code>.  When <code>torch.pow()</code> was<br>
written, not only was the “forward” function that computes <code>x**p</code> programmed,<br>
but the companion “backward” function that computes <code>p * x**(p - 1)</code> was<br>
programmed, as well.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; x = torch.tensor ([2.0], requires_grad = True)
&gt;&gt;&gt; y = torch.pow (x, 3)
&gt;&gt;&gt; y.backward()
&gt;&gt;&gt; x.grad
tensor([12.])
</code></pre>
<p>When <code>y.backward()</code> is called, autograd uses <code>pow()</code>'s backward function<br>
to evaluate the analytic derivative, <code>3 * 2.0**(3 - 1)</code>, numerically, but the<br>
differentiation itself is not performed numerically.</p>
<p>*)  For any given use case you may need more or less precision to perform<br>
the forward pass adequately, and you may need more or less precision to<br>
perform the backward pass – autograd’s gradient computation – adequately.</p>
<p>For many use cases, single-precision is fine.  Some use cases might require<br>
double-precision (but not because autograd performs <em>numerical</em> differentiation,<br>
because it doesn’t).  Some use cases might require only half-precision, and<br>
for some use cases, Nvidia’s <strong>reduced-precision</strong> (and <strong>misleading-named</strong>)<br>
TensorFloat-32 might suffice.</p>
<p>Best.</p>
<p>K. Frank</p>