How to debug with floating point differences

username: pytorcher
cooked: <p>I am trying to convert code from an old pytorch version on an old computer to a new one.  I have it very close, but it is still not getting quite the same results.  Im having trouble determining where actual changes are happening and where it is just tiny floating point differences.  I already have converted my entire system to .double() but that actually seems to yield larger problems than without it.  Any suggestions?</p>

username: Kushaj
cooked: <p>To check for tiny floating point errors</p>
<pre><code class="lang-python">def test_close(a, b, eps=1e-5):
    # Now depending on what a, b are you can add code here
    return abs(a-b)&lt;eps # If scaler
    return (abs(a-b)&lt;eps).all() # If array
</code></pre>

username: pytorcher
cooked: <p>That finds the differences, the problem is I don’t know when the differences are because of floating point rounding and when they are because there is actual functionality differences.</p>

username: KFrank
cooked: <p>Hi Py!</p>
<aside class="quote no-group" data-username="pytorcher" data-post="3" data-topic="82397" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>I don’t know when the differences are because of floating point rounding and when they are because there is actual functionality differences.</p>
</blockquote>
</aside>
<p>First, don’t look at differences in your results after training for a<br>
number of (batch) iterations.  It is perfectly possible for your<br>
model parameters to wander off in distinctly different, although<br>
content-wise, broadly-equivalent directions, after accumulated<br>
round-off errors start them down different paths.</p>
<p>You should look at whether your differences are compatible with<br>
round-off error after passing a single batch through your model,<br>
checking your output, your loss, and your gradients after<br>
backpropagating once.  If you still think these differences exceed<br>
reasonable, somewhat accumulated round-off error, trace<br>
intermediate steps through the layers of your model.  Unless you<br>
have an ill-conditioned <code>Linear</code> in there somewhere, no single<br>
layer should give results that differ by more than a few times<br>
round-off error.</p>
<p>The next general check you can make is to move your model and<br>
all of your data to double-precision.  Round-off error in your results,<br>
even after accumulating somewhat, should be reduced such that<br>
you have about eight or nine additional decimal digits of precision.</p>
<p>You comment that converting your “entire system to .double() …<br>
seems to yield larger problems than without it.”  Is this after<br>
training multiple iterations after your parameters may have started<br>
wandering down different paths?  Or is this still the case after only<br>
a single forward and/or backward pass through your system?</p>
<p>Best.</p>
<p>K. Frank</p>

username: pytorcher
cooked: <p>Hi K. Frank!</p>
<p>I am looking at values after single batches.  To try to fix this for the more complicated project I am using a model is very simple, its just a conv layer, a sigmoid, and another conv.  I am printing the exact weights and bias as well as the grad for every weight after each batch as well as ensuring they initialize to the same values.  It gives the same up to 16 decimal places for the first few batches but then drift starts to accumulate.  With double precision the drift started immediately.  Maybe I am not using .double() in the correct places?  I basically just made everywhere I was doing .cuda() to now be .cuda().double().</p>

username: KFrank
cooked: <p>Hi Py!</p>
<aside class="quote no-group" data-username="pytorcher" data-post="5" data-topic="82397" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>It gives the same up to 16 decimal places for the first few batches but then drift starts to accumulate.</p>
</blockquote>
</aside>
<p>This is strong evidence that your two versions are doing the same<br>
thing, up to floating-point round-off.  You can’t expect any more<br>
than this.</p>
<p>(As an aside, 16 decimal places is what you would expect for<br>
double precision.  So you’re actually doing better than you would<br>
expect for single precision.  This behavior is actually a bit odd,<br>
but I can cheerfully concoct some plausible explanations for it.)</p>
<aside class="quote no-group">
<blockquote>
<p>With double precision the drift started immediately.</p>
</blockquote>
</aside>
<p>Whether this is a concern depends on the details of what you<br>
mean by this.  After a <em>single</em> forward or backward pass, do your<br>
double-precision results agree to significantly less than 16 decimal<br>
digits?  If not, then you’re getting as much as you have an right<br>
to get.</p>
<p>Best.</p>
<p>K. Frank</p>

username: pytorcher
cooked: <p>Yes, after a single forward pass I get different results to less than 16 digits.  I am not initializing to different random weights or passing in data in a different order.  So I print out every weight in my network and and running them in the diff tool meld, and get exactly the same values up to 16 digits.  But then on the very first forward pass I get losses of 1.9986734118816107 and 1.9986734009655582.  When I was only using float precision things did always stay exactly the same for a few batches before the drift started.  I get what you are saying in the other thread that this will happen and they will converge to equally correct minima.  But I am only debugging with this very simple model, the one I am actually trying to correct once I figure this out is very complicated and uses lots of funky custom functions.  The evidence something is wrong is that with the real model everything works reliably in pytorch 0.3 and does not work at all in 1.3.1.</p>

username: KFrank
cooked: <p>Hi Py!</p>
<aside class="quote no-group" data-username="pytorcher" data-post="7" data-topic="82397" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>But then on the very first forward pass I get losses of 1.9986734118816107 and 1.9986734009655582.</p>
</blockquote>
</aside>
<p>If I understand correctly, this happens when you are running everything<br>
(as far as you know) in double precision.</p>
<p>This is odd.  Your two results agree up to single precision, so it’s not<br>
as if one version is <em>completely</em> broken.  It’s as if one version does<br>
something only in single precision, even though it should be in double<br>
precision.</p>
<aside class="quote no-group">
<blockquote>
<p>When I was only using float precision things did always stay exactly the same for a few batches before the drift started.</p>
</blockquote>
</aside>
<p>Do I understand correctly (based on your earlier post) that this means<br>
that the two versions agree (for a few batches) up to <em>double</em> precision,<br>
even though you are running in single precision?</p>
<p>This is also odd (although not “wrong” or impossible).  Am I right<br>
that you are saying that when running in single precision, the two<br>
versions agree up to double precision, but when running in double<br>
precision, the two versions only agree up to single precision (at least<br>
for the first batch)?</p>
<aside class="quote no-group">
<blockquote>
<p>But I am only debugging with this very simple model,</p>
</blockquote>
</aside>
<p>Could you post a simple, complete, runnable script that illustrates<br>
this issue, together with its output?</p>
<aside class="quote no-group">
<blockquote>
<p>The evidence something is wrong is that with the real model everything works reliably in pytorch 0.3 and does not work at all in 1.3.1.</p>
</blockquote>
</aside>
<p>What does “does not work at all in 1.3.1” mean?  It sounds like (in<br>
single precision) 1.3.1 agrees with 0.3 up to double precision.  So<br>
that sounds like it’s working.  (And in double precision, they agree<br>
up to single precision, so it’s more-or-less working.)</p>
<p>Also, could you tell us the exact version of your 0.3.0 version?<br>
(You can run <code>print (torch.__version__)</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: pytorcher
cooked: <p>That is the correct understanding of the two numbers.</p>
<p>That is also correct, with float precision the two versions agreed for a few batches up to 32 decimal places when I was not using double precision.  and yes when I switch to double it seems that they agree to 32 decimal places when I specify exact values, but then as soon as any operations are done, forward pass, backward grad or loss calculations, they disagree immediately.</p>
<p>Does not work at all in 1.3.1 means that the built in pytorch functions work properly and and it will converge to a slightly different local minima without using my custom functions, but when it uses my custom functions it makes no difference at all while it used to in 0.3.  The old version is 0.3.0.post4.</p>
<p>It looks like in 1.3.1 it actually is not giving consistent values to begin with.  When I run the same code initialized with the same values I can print the network weights, network output, loss value all to be exactly the same to 16 decimal points but then actual grads that get calculated only agree up to 7.  I can try to make a minimal example to show code if that would help.  My understanding is that floats might lose some precision, but the same math on the same system should always do the same rounding and give the same values.</p>
<p>Here is what I’m doing for debugging now, the second grad value changes every time past the first 7 digits while the first manually calculated grad is identical every time to all digits printed:</p>
<pre><code class="lang-auto">(Pdb) X = outputs
(Pdb) y = targets
(Pdb) m = y.shape[0]
(Pdb) grad = F.softmax(X,1)
(Pdb) grad[range(m),y] -= 1
(Pdb) grad = grad/m
(Pdb) ((net.layer1.module.layer1(inputs)).sigmoid()[:,0,0,0] * grad[:,0]).sum().tolist()
0.03555078059434891
(Pdb) (net.layer2.module.layer1.weight.grad[0][0][0][0]).tolist()
0.03555076941847801
</code></pre>
<p>Edit:  I noticed a pattern while running this many times and decided to see if there was one.  It turns out after 225 trials this second value only ever takes on 8 different values.  It is possible somewhere in the pytorch code for back propagating cross entropy loss there are 3 bits that are not being set to any particular values so they get randomly assigned?</p>
<p>Eidt2: This seems to be based on batch size.  8 Values was with batch size of 128, with batch size 64 there are only 7 values, with batch size 16 down to 3, batch sizes 4, 2 and 1, consistently return the same correct value</p>

username: KFrank
cooked: <p>Hi Py!</p>
<aside class="quote no-group quote-modified" data-username="pytorcher" data-post="9" data-topic="82397" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>Does not work at all in 1.3.1 means that the built in pytorch functions work properly and and it will converge to a slightly different local minima without using my custom functions, but when it uses my custom functions it makes no difference at all while it used to in 0.3.</p>
<p>It looks like in 1.3.1 it actually is not giving consistent values to begin with.  When I run the same code initialized with the same values I can print the network weights, network output, loss value all to be exactly the same to 16 decimal points but then actual grads that get calculated only agree up to 7.</p>
</blockquote>
</aside>
<p>If I understand properly what you are saying, it sounds like you<br>
have this issue only when you are using custom functions and<br>
you are running on 1.3.1.</p>
<p>This suggests that there is a bug in 1.3.1 that is uncovered in<br>
by your custom functions.  Or, I suppose, there could be a bug<br>
in your custom functions that 0.3.0 is letting you get away with.</p>
<p>Could you post the code for your custom functions?</p>
<aside class="quote no-group">
<blockquote>
<p>I can try to make a minimal example to show code if that would help.</p>
</blockquote>
</aside>
<p>A minimal example would be the best thing.  A complete, runnable<br>
script (with all necessary imports, hard-coded or randomly generated<br>
sample, and anything else needed so that it can be run just by<br>
copy-pasting it from the forum) would be very helpful.</p>
<aside class="quote no-group">
<blockquote>
<p>Here is what I’m doing for debugging now, the second grad value changes every time past the first 7 digits while the first manually calculated grad is identical every time to all digits printed:</p>
</blockquote>
</aside>
<p>This makes it sound like the bug – whether yours, or in 1.3.1 – is<br>
associated with the <code>.backward()</code> of one of your custom functions.</p>
<p>A minimal example – that will presumably contain the code for<br>
your custom functions – would be the best next step.  Please<br>
also post the output produced by your minimal example, when<br>
run on both 0.3.0 and 1.3.1.</p>
<p>Also, I’m not saying that you should necessarily do this, but have<br>
you tried upgrading to pytorch 1.5 (or maybe 1.4) to see if this<br>
issue still persists?</p>
<p>[Edit:]  One possibility:  Are you using <code>.data()</code> in your custom<br>
functions.  <code>.data()</code> is deprecated since 0.4.0, and, although<br>
still present, doesn’t work correctly for all use cases.  Please<br>
see the <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/#what-about-data" rel="noopener nofollow ugc">Pytorch 0.4.0 Migration Guide</a> for details.</p>
<p>Best.</p>
<p>K. Frank</p>

username: pytorcher
cooked: <p>I gave the Migration Guide a read.  Thanks for sending me that!  I do suspect once I get to the bottom of this something from that guide will be what I did wrong since my custom functions were using .data in 0.3 and 1.3 versions.  However, when making the minimal example my custom functions were not needed to replicate this problem.  The following code also shows that the hand calculated grad does not match up with the autograd calculated one in 1.3.1.  I made similar code in 0.3.0 and it also does not match there.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

import torchvision.transforms as transforms
from torchvision import datasets

class Small_Net(nn.Module):
    def __init__(self, device):
        super(Small_Net, self).__init__()
        self.layer1 = nn.Conv2d(1, 40, kernel_size=28, stride=1, padding=0, bias=True)
        self.layer2 = nn.Conv2d(40, 10, kernel_size=1, stride=1, padding=0, bias=True)
        self.layer1.to(device)
        self.layer2.to(device)

    def forward(self, x):        
        out2 = self.layer1(x).sigmoid()
        self.midOut = out2.detach().clone()
        out3 = self.layer2(out2)            
        out6 = out3.view(out3.size(0), -1)
        return out6

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net = Small_Net(device)  
net.to(device)
net = torch.nn.DataParallel(net, device_ids=range(torch.cuda.device_count())).to(device)
start_epoch = 0
trainloader = torch.utils.data.DataLoader(datasets.MNIST('../mnist_data', 
                                                          download=True, 
                                                          train=True,
                                                          transform=transforms.Compose([
                                                              transforms.ToTensor(), # first, convert image to PyTorch tensor
                                                              transforms.Normalize((0.1307,), (0.3081,)) # normalize inputs
                                                          ])), 
                                           batch_size=128, 
                                           shuffle=True)
criterion = nn.CrossEntropyLoss()             
def train(epoch, mode):
    net.train()
    optimizer = optim.SGD(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1, momentum=0, weight_decay=0)  # 5e-4)
    for batch_idx, (inputs, targets) in enumerate(trainloader):
        batchSize, outputMaps, x, y = inputs.size()
        inputs, targets = inputs.to(device), targets.to(device)  # GPU settings        
        optimizer.zero_grad()
        output = net(inputs)
        loss = criterion(output, targets)
        loss.backward()    
        X = output
        y = targets
        m = y.shape[0]
        grad = F.softmax(X,1)
        grad[range(m),y] -= 1
        grad = grad/m
        print('is same?')        
        print(str((net.module.layer1(inputs).sigmoid()[:,0,0,0] * grad[:,0]).sum().tolist()))
        print((net.module.midOut[:,0,0,0] * grad[:,0]).sum().tolist())
        print(str((net.module.layer2.weight.grad[0][0][0][0]).tolist()))
        del inputs, targets, output
        optimizer.step() 
    
train(0, 'n')
</code></pre>
<p>Example output:</p>
<blockquote>
<p>is same?<br>
-0.0011598279234021902<br>
-0.0011598279234021902<br>
-0.0011598295532166958</p>
</blockquote>
<p>Edit: with double precision it does agree much more but still not the same.  Is this just the best that can be expected because of what autograd is doing under the hood?</p>
<blockquote>
<p>is same?<br>
-0.00020605623917282047<br>
-0.00020605623917282047<br>
-0.00020605623917282427</p>
</blockquote>

username: KFrank
cooked: <p>Hi Py!</p>
<aside class="quote no-group" data-username="pytorcher" data-post="11" data-topic="82397" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>my custom functions were using .data in 0.3 and 1.3 versions.</p>
</blockquote>
</aside>
<p>My conclusion is that your use of <code>.data()</code> is the cause of “does<br>
not work at all in 1.3.1.”</p>
<aside class="quote no-group">
<blockquote>
<p>my custom functions were not needed to replicate this problem.</p>
</blockquote>
</aside>
<p>Note, what you show below does not replicate the "does not work at<br>
all " problem – it replicates the “agrees up to expected floating-point<br>
round-off” so-called problem.</p>
<aside class="quote no-group">
<blockquote>
<p>Example output:</p>
<blockquote>
<p>is same?<br>
-0.0011598279234021902<br>
-0.0011598279234021902<br>
-0.0011598295532166958</p>
</blockquote>
</blockquote>
</aside>
<p>I assume that this is running on 1.3.1 with (the default) single<br>
precision (<code>.float()</code>).  Note, that your third result (from autograd)<br>
agrees up to single-precision round-off with the first two.  This is<br>
really the best you can expect.</p>
<aside class="quote no-group">
<blockquote>
<p>Edit: with double precision it does agree much more but still not the same.</p>
<blockquote>
<p>is same?<br>
-0.00020605623917282047<br>
-0.00020605623917282047<br>
-0.00020605623917282427</p>
</blockquote>
</blockquote>
</aside>
<p>Now I assume this is 1.3.1 with double precision (<code>.double()</code>).<br>
Your results still agree up round-off error, but this time up to the<br>
more accurate double precision.</p>
<aside class="quote no-group">
<blockquote>
<p>Is this just the best that can be expected</p>
</blockquote>
</aside>
<p>Yes, this is the best you can expect.  It should not be viewed as<br>
something being wrong, or any kind of issue (other than that’s<br>
how floating-point arithmetic works).</p>
<aside class="quote no-group">
<blockquote>
<p>because of what autograd is doing under the hood?</p>
</blockquote>
</aside>
<p>I wouldn’t say that this is “because of … autograd.”  You do the<br>
gradient calculation twice – once your way, and once autograd’s<br>
way.  Neither one is more right or wrong than the other.  They<br>
just happen to perform certain calculations in different orders.<br>
The different orders are <em>mathematically</em> equivalent, but differ<br>
when using floating-point arithmetic.  So you get differences at<br>
the level of floating-point round-off.</p>
<p>But, again, it’s not autograd, per se.  You could, yourself, calculate<br>
the gradient twice, but purposely reorder some of the calculations<br>
in a <em>mathematically</em> equivalent way, and you would see similar<br>
differences.</p>
<p>Best.</p>
<p>K. Frank</p>

username: pytorcher
cooked: <p>That all makes sense.  I just would have thought that at least doing the same calculations, e.g. two runs of autograd, with the same numbers would always do the same rounding.  Now that I am using double() correctly I was able to create a test that is repeatable to 8 decimal places over my training epochs.  I will use the migration guide to redo the changes I made with the custom functions and be especially careful about .data().  Thanks so much for all the help!</p>