Different results for forward pass of two equal tensors

username: dozed
cooked: <p>Hi,</p>
<p>forwarding a tensor <code>img</code> through a PyTorch model produces a different result than forwarding <code>img + torch.zeros_like(img)</code>?</p>
<p>Here is a minimal example: <a href="https://github.com/dozed/pytorch-issue-1/blob/main/test_issue.py" rel="noopener nofollow ugc">https://github.com/dozed/pytorch-issue-1/blob/main/test_issue.py</a></p>
<p>The relevant code is:</p>
<pre><code class="lang-python">def do_forward_pass(
        model: nn.Module,
        add_zeros: bool,
) -&gt; Tensor:
    # prepare input
    img = torchvision.io.image.read_image('sky1024px.jpg')
    img = FT.convert_image_dtype(img, torch.float32)
    img = img.unsqueeze(dim=0)
    img = img.to(device)

    # maybe add zero tensor
    if add_zeros:
        zeros = torch.zeros_like(img)
        img_updated = img + zeros
    else:
        img_updated = img

    assert torch.allclose(img, img_updated)
    assert torch.equal(img, img_updated)

    # forward pass
    result = model(img_updated)

    return result


result1 = do_forward_pass(model, add_zeros=False)
result2 = do_forward_pass(model, add_zeros=True)

assert torch.allclose(result1, result2)
assert torch.equal(result1, result2)
</code></pre>
<p><code>result1</code> and <code>result2</code> should be equal, but they are different.</p>
<p>Any ideas what causes this divergence?</p>
<p>Regards,<br>
Stefan</p>

username: ptrblck
cooked: <p>Are you using any random layers, such as dropout? Also, how large is the difference?</p>

username: dozed
cooked: <p>The model does not use random layers:</p>
<ul>
<li><a href="https://github.com/dozed/pytorch-issue-1/blob/main/test_issue.py#L39-L42" rel="noopener nofollow ugc">https://github.com/dozed/pytorch-issue-1/blob/main/test_issue.py#L39-L42</a></li>
<li><a href="https://github.com/dozed/pytorch-issue-1/blob/main/Inception.py" rel="noopener nofollow ugc">https://github.com/dozed/pytorch-issue-1/blob/main/Inception.py</a></li>
</ul>
<p>The difference is as follows:</p>
<pre><code class="lang-python">torch.linalg.norm(result1 - result2)
# tensor(6.4035e-06)
</code></pre>

username: ptrblck
cooked: <p>Are you seeing the same difference in two different runs without adding the zeros to the tensor?</p>

username: dozed
cooked: <p>Without adding zeros the result tensors of two runs are equal, i.e. the following assertions hold:</p>
<pre><code class="lang-python">result1 = do_forward_pass(model, add_zeros=False)
result2 = do_forward_pass(model, add_zeros=False)

assert torch.allclose(result1, result2)
assert torch.equal(result1, result2)
</code></pre>

username: dozed
cooked: <p>I could further reduce the example case to a model with two <code>Conv2d</code> layers:</p>
<pre><code class="lang-python"># prepare input
img = torchvision.io.image.read_image('sky1024px.jpg')
img = FT.convert_image_dtype(img, torch.float32)
img = img.unsqueeze(dim=0)

# prepare input + zero
zeros = torch.zeros_like(img)
img_updated = img + zeros

# input tensors are identical
assert torch.allclose(img, img_updated)
assert torch.equal(img, img_updated)

# prepare model
conv1 = nn.Conv2d(in_channels=3, out_channels=129, kernel_size=3, padding=1)
conv2 = nn.Conv2d(in_channels=129, out_channels=4, kernel_size=1)

# forward 1
x = conv1(img)
result1 = conv2(x)

# forward 2
y = conv1(img_updated)
result2 = conv2(y)

# ISSUE: the results are not equal but should be, since only zeros are added
print(torch.linalg.norm(result1 - result2))
assert torch.allclose(result1, result2)
assert torch.equal(result1, result2)
</code></pre>
<p>Example: <a href="https://github.com/dozed/pytorch-issue-1/blob/main/test_issue.py#L39-L42" rel="noopener nofollow ugc">https://github.com/dozed/pytorch-issue-1/blob/main/test_issue.py#L39-L42</a></p>

username: ptrblck
cooked: <p>I cannot reproduce the issue and get a zero error using a fixed random input:</p>
<pre><code class="lang-python">import torch
import torchvision
from torch import nn
from torchvision.transforms import functional as FT


def test_add_zero_different_result():
    # prepare input
    torch.manual_seed(2809)
    img = torch.randn(1, 3, 224, 224)

    # prepare input + zero
    zeros = torch.zeros_like(img)
    img_updated = img + zeros

    # input tensors are identical
    assert torch.allclose(img, img_updated)
    assert torch.equal(img, img_updated)

    # prepare model
    conv1 = nn.Conv2d(in_channels=3, out_channels=129, kernel_size=3, padding=1)
    conv2 = nn.Conv2d(in_channels=129, out_channels=4, kernel_size=1)
    conv1.requires_grad_(False)
    conv2.requires_grad_(False)

    # forward 1
    x = conv1(img)
    result1 = conv2(x)

    # forward 2
    y = conv1(img_updated)
    result2 = conv2(y)

    # tensors after conv1 are equal
    assert torch.allclose(x, y)
    assert torch.equal(x, y)

    # ISSUE: the results are not equal but should be, since only zeros are added
    print(torch.linalg.norm(result1 - result2))
    assert torch.allclose(result1, result2)
    assert torch.equal(result1, result2)

for _ in range(100):
    test_add_zero_different_result()

# tensor(0.)
# tensor(0.)
# tensor(0.)
# tensor(0.)
...
</code></pre>

username: dozed
cooked: <p>Please try the exact code from the example repository. It works for random input, but not for the image input.</p>
<p>Also when you change the number of channels from 129 to 128, the test passes.</p>

username: dozed
cooked: <p>Another finding: without the <code>img.unsqueeze(dim=0)</code> the test passes</p>

username: dozed
cooked: <p>After some further debugging I could pinpoint why this happens.</p>
<p><code>torchvision.io.image.read_image</code> produces a tensor which is the CHW permuted version of a HWC tensor. That means for a <code>[3, 1, 2]</code> sized CHW tensor the strides are <code>[1, 6, 3]</code>.</p>
<pre><code class="lang-python">def print_tensor_info(x: torch.Tensor) -&gt; None:
    print('size:', x.size())
    print('stride:', x.stride())

orig = torch.tensor([
    [
        [1, 2, 3],
        [4, 5, 6],
    ],
])

print_tensor_info(orig)
# size: torch.Size([1, 2, 3])
# stride: (6, 3, 1)

permuted = orig.permute(2, 0, 1)

print_tensor_info(permuted)
# size: torch.Size([3, 1, 2])
# stride: (1, 6, 3)
</code></pre>
<p><code>unsqueeze</code> of this <code>permuted</code> CHW tensor to a NCHW tensor sets an <em>invalid</em> batch stride of <code>3</code>:</p>
<pre><code class="lang-python">permuted_unsqueezed = permuted.unsqueeze(dim=0)

print_tensor_info(permuted_unsqueezed)
# size: torch.Size([1, 3, 1, 2])
# stride: (3, 1, 6, 3)
</code></pre>
<p>The correct value would be <code>6</code>. There is only a simple heuristic which chooses that value in <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/TensorShape.cpp#L3191-L3199" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/TensorShape.cpp#L3191-L3199</a>:</p>
<pre><code class="lang-cpp">InferUnsqueezeGeometryResult
inferUnsqueezeGeometry(const Tensor&amp; tensor, int64_t dim) {
  InferUnsqueezeGeometryResult result(tensor.sizes(), tensor.strides());
  int64_t new_stride = dim &gt;= tensor.dim() ? 1 : result.sizes[dim] * result.strides[dim];
  result.sizes.insert(result.sizes.begin() + dim, 1);
  result.strides.insert(result.strides.begin() + dim, new_stride);

  return result;
}
</code></pre>
<p>This <code>permuted_unsqueezed</code> tensor with sizes <code>[1, 3, 1, 2]</code> and strides <code>[3, 1, 6, 3]</code> is treated as having a <code>Contiguous</code> memory format (and not a <code>ChannelsLast</code> memory format): <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/core/TensorBase.h#L270-L289" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/core/TensorBase.h#L270-L289</a>.</p>
<p>Adding <code>zeros_like</code> yields the following tensor:</p>
<pre><code class="lang-python">zeros_like = torch.zeros_like(permuted_unsqueezed)

permuted_unsqueezed_added_zeros = permuted_unsqueezed.add(zeros_like)

print_tensor_info(permuted_unsqueezed_added_zeros)
# size: torch.Size([1, 3, 1, 2])
# stride: (6, 1, 6, 3)
</code></pre>
<p>Note that the stride <code>[6, 1, 6, 3]</code> is correct this time. Therefore the tensor is handled as having <code>ChannelsLast</code> memory format.</p>
<p>When both tensors are put through the same convolution layer, they will produce slightly different results since there are different cases for <code>Contiguous</code> and <code>ChannelsLast</code> tensors: <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/ConvolutionMM2d.cpp#L247-L283" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/ConvolutionMM2d.cpp#L247-L283</a>:</p>
<pre><code class="lang-cpp">if (is_channels_last) {
    // ...
  
    at::native::cpublas::gemm(...);
  } else {
    // ...

    at::native::cpublas::gemm(...);
}
</code></pre>
<p>The difference is in 10^-8, so in practice it should not be a problem.</p>
<p>Also this seems to be a known issue: <a href="https://github.com/pytorch/pytorch/issues/68430#issuecomment-970895522" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/68430#issuecomment-970895522</a></p>