F.l1_loss does not match with NumPy

username: vmirly1
cooked: <p>I tried to compare PyTorch <code>F.l1_loss</code> with NumPy computation and found that when the input and targets have <code>dtype=float64</code>, the results match but when they are <code>flat32</code>, they do not match.</p>
<pre><code class="lang-python">import unittest
import numpy as np
import torchvision
import torch.nn.functional as F
from PIL import Image

np.random.seed(1)


class TestL1loss(unittest.TestCase):

    @staticmethod
    def load_images():
        img = Image.open('1.jpg')

        img_width, img_height = img.size
        img_a = img.crop([0, 0, img_width//2, img_height])
        img_b = img.crop([img_width//2, 0, img_width, img_height])
        img_a.save('img_a.png')
        img_b.save('img_b.png')

        tsfm = torchvision.transforms.ToTensor()
        return tsfm(img_a), tsfm(img_b)

    def test_l1_loss_float32(self):
        t_a, t_b = TestL1loss.load_images()

        # both float32
        self.assertAlmostEqual(
            F.l1_loss(t_a, t_b, reduction='mean').item(),
            np.mean(np.abs(t_a.numpy() - t_b.numpy())))

        self.assertAlmostEqual(
            F.l1_loss(t_a, t_b, reduction='mean').item(),
            np.mean(np.abs(t_a.numpy() - t_b.numpy())),
            places=3)

    def test_l1_loss_float64(self):
        t_a, t_b = TestL1loss.load_images()

        # both double/float64
        t_a_arr = t_a.numpy().astype(np.float64)
        t_b_arr = t_b.numpy().astype(np.float64)
        self.assertAlmostEqual(
            F.l1_loss(t_a.double(), t_b.double(), reduction='mean').item(),
            np.mean(np.abs(t_a_arr - t_b_arr)))

    def test_l1_loss_mixed(self):
        t_a, t_b = TestL1loss.load_images()

        # double tensors vs. float32 arrays
        self.assertAlmostEqual(
            F.l1_loss(t_a.double(), t_b.double(), reduction='mean').item(),
            np.mean(np.abs(t_a.numpy() - t_b.numpy())))
</code></pre>
<p>I got the example image from the facades dataset (<a href="http://cmp.felk.cvut.cz/~tylecr1/facade/" rel="nofollow noopener">http://cmp.felk.cvut.cz/~tylecr1/facade/</a>).</p>
<p>and the following shows that two of these tests fails:</p>
<pre><code class="lang-auto">F.F
======================================================================
FAIL: test_l1_loss_float32 (test_l1loss.TestL1loss)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vahid/github/test_l1loss.py", line 31, in test_l1_loss_float32
    np.mean(np.abs(t_a.numpy() - t_b.numpy())))
AssertionError: 0.41515448689460754 != 0.41536176 within 7 places (0.0002072751522064209 difference)

======================================================================
FAIL: test_l1_loss_mixed (test_l1loss.TestL1loss)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vahid/github/test_l1loss.py", line 54, in test_l1_loss_mixed
    np.mean(np.abs(t_a.numpy() - t_b.numpy())))
AssertionError: 0.41536183419634415 != 0.41536176 within 7 places (7.214953018364056e-08 difference)

----------------------------------------------------------------------
Ran 3 tests in 0.151s

FAILED (failures=2)
</code></pre>
<p>In case of float32, they only match upto 3 decimal places. The test case with mixed dtype shows that when the tensors are in double format, but NumPy arrays are in float32 type, the result is very close (error ~7e-8). So that tells me that NumPy’s float32 dtype is as accurate as PyTorch double format.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Can you give more information about the images you use to check this?</p>
<p>I tried with random Tensors but I can’t see the same thing:</p>
<pre><code class="lang-auto">a = torch.rand(250, 250)
b = torch.rand(250, 250)

F.l1_loss(a, b, reduction='mean').item() - np.mean(np.abs(a.numpy() - b.numpy()))
&gt; 2.9802322387695312e-08
</code></pre>
<p>What can happen is that the reduction is imprecise if the values in your image are large.<br>
Can you try comparing to <code>(a - b).abs().mean()</code> for torch as well?</p>

username: vmirly1
cooked: <p>Thanks <a class="mention" href="/u/alband">@albanD</a> for the suggestion.</p>
<p>I confirmed that the error with <code>(a - b).abs().mean()</code> is smaller than <code>F.l1_loss</code>.</p>
<p>Error with <code>(a - b).abs().mean()</code> =&gt; <code>2.086162567138672e-07</code><br>
Error with <code>F.l1_loss</code> =&gt; <code>0.0002072751522064209</code></p>

username: albanD
cooked: <p>I would guess that this is because l1_loss is still implemented with the old THNN library which does the <code>mean()</code> reduction differently and is usually less precise.</p>