The output of Tensor.mean() are different according to the position

hideinshadow: <p>When I was checking the feature maps of my model, I found something bizarre:</p>
<p>Although each channel of the input tensor X (the size of x is 256 * 9 * 9, C * H * W) is a constant value. The result of X.mean(0) is not in accord with expectations. The following is the result of plt.imshow(X.mean(0).numpy()):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e55d39f315f32f24a6556b54fd3d4afdc2380fca.png" alt="下载 (1)" data-base62-sha1="wJ34N8Fbtjc2ZRR9lgCqM7TGrpg" width="408" height="413"></p>
<p>What this visualized result tells us is that the value in X.mean(0) are different according to the position although their inputs are the same. Actually the value of X.mean(0)[0, 0] is 0.07183486223220825, while the value of X.mean(0)[-1, -1] is 0.07183484733104706 (their difference is 1.4901161e-08).</p>
<p>I know this difference is very very tiny, but I am curious why there is a difference.<br>
The torch version is 1.8.2+cu102 and the numpy version is 1.20.3.</p>

hideinshadow: <p>In addition, the result of plt.imshow(X.numpy().mean(0)) is as follows:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/403e0fcc184cc4a26088023444a4b7933a500346.png" alt="2" data-base62-sha1="9ajwwXRxyqglNcRD00HJGpdHUFw" width="408" height="413"></p>

hideinshadow: <p>Here is a binary file in NumPy <code>.npy</code> format of the input X. You can use X = np.load(filepath) to load it and X = torch.FloatTensor(X) to transform X from ndarray to tensor.</p>
<p><a href="https://drive.google.com/file/u/1/d/1rpLIUyT2avfpSy4W6fUAbdsMnrV7hnay/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/u/1/d/1rpLIUyT2avfpSy4W6fUAbdsMnrV7hnay/view?usp=sharing</a></p>

hideinshadow: <p>Each channel of the input tensor X is a constant value like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9f3d7cdf38648f0c719fcacb05def5077abaa4d.png" data-download-href="https://discuss.pytorch.org/uploads/default/b9f3d7cdf38648f0c719fcacb05def5077abaa4d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9f3d7cdf38648f0c719fcacb05def5077abaa4d.png" alt="image" data-base62-sha1="qx0LjP8es5dNBPqKGRnSTM6VWxf" width="313" height="499" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">466×744 978 Bytes</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

hideinshadow: <p>And I found that when H and W of the input tensor are 2 to the power of N (N is nonnegative integer), the result of X.mean(0) will be correct.</p>

ptrblck: <p>The small absolute errors are most likely created by the limited floating point precision and due to a different order of operations.</p>

hideinshadow: <p>Thanks for your reply. But there should be no precision loss between torch.float32 and numpy.float32. And the result of</p>
<pre><code class="lang-auto">import numpy as np
import torch

X = np.load('x.npy')
plt.imshow(torch.FloatTensor(X).numpy().mean(0))
plt.show()
</code></pre>
<p>is as follows:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/403e0fcc184cc4a26088023444a4b7933a500346.png" alt="2" data-base62-sha1="9ajwwXRxyqglNcRD00HJGpdHUFw" width="408" height="413"><br>
So I think there is no precision loss whether it is from torch.float32 to numpy.float32 or vice versa. I think the real problem is the difference between the algorithm of torch.mean() and numpy.mean().</p>

ptrblck: <p>Yes, there won’t be a precision loss while transforming between numpy as PyTorch, since <code>from_numpy</code> and <code>tensor.numpy()</code> would share the underlying data. The difference would come from potentially different implementations (and order of operations) in the <code>mean</code> op.</p>

hideinshadow: <p>So how could I know the implementation of torch.mean()?</p>

ptrblck: <p>I think it’s definition can be found <a href="https://github.com/pytorch/pytorch/blob/2f099c7555ab41e55ddfd12ba3ed804e52894820/aten/src/ATen/native/ReduceOps.cpp#L1122-L1154">here</a>.</p>

hideinshadow: <p>Many thanks for your detailed answer! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=10" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"></p>