Torch and Numpy have different norm

username: jozhang97
cooked: <p>It seems to me that np.linalg.norm and torch.norm have different outputs.</p>
<p>torch version: 1.0.0</p>
<pre><code class="lang-auto">import torch
import numpy as np

var_torch = torch.randn((3,333,333))
var_numpy = var_torch.detach().numpy()
print(torch.norm(var_torch))
&gt;&gt;&gt; tensor(1884.0406)
print(np.linalg.norm(var_numpy))
&gt;&gt;&gt; 577.75214
</code></pre>
<p>Is this expected?</p>

username: vmirly1
cooked: <p>I think they handle the dimensions differently. If I specify the order of dimensions, then I can get the same results using NumPy and PyTorch, so first taking the norm of dimensions 1, 2 and then the norm of the resulting tensor/array:</p>
<pre><code class="lang-python">&gt;&gt;&gt; np.linalg.norm(np.linalg.norm(var_numpy, axis=(1,2)))
577.3682
&gt;&gt;&gt; torch.norm(torch.norm(var_torch, dim=(1,2)))
tensor(577.3682)
</code></pre>
<p>This result is consistent with what NumPy gives without specifying dimensions. However, I don’t know how PyTorch calculates its result <code>tensor(1883.6027)</code>:</p>
<pre><code class="lang-python">torch.norm(var_torch)
tensor(1883.6027)
</code></pre>
<p><a class="mention" href="/u/smth">@smth</a>  <a class="mention" href="/u/ptrblck">@ptrblck</a>  Any comment?</p>

username: ptrblck
cooked: <p>This issue might be related to <a href="https://discuss.pytorch.org/t/output-of-torch-norm-x-depends-on-size-of-x-not-in-a-good-way/33299/5?u=ptrblck">this bug</a>.<br>
Soumith opened an issue <a href="https://github.com/pytorch/pytorch/issues/15602" rel="nofollow noopener">here</a> to track it.</p>
<p>Just to double check if these issues are related: Do you get the correct result on GPU?</p>

username: vmirly1
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. I get correct results on GPU:</p>
<pre><code class="lang-python">&gt;&gt;&gt; torch.norm(var_torch.to(device))
tensor(577.3682, device='cuda:0')
</code></pre>

username: ptrblck
cooked: <p>Thanks for testing it <a class="mention" href="/u/vmirly1">@vmirly1</a> (as I currently don’t have access to a GPU <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"> ).<br>
In that case it looks like they are related.</p>
<p>As Soumith said, it’s a high priority bug and should be fixed before the next minor release.</p>

username: jozhang97
cooked: <p>Ahh I see the duality with the previous thread.<br>
Thanks so much.</p>

username: Junhao_Wen
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Sorry to reopen this issue, I found that np.linalg.norm() and torch.norm give similar (I say similar is because the results have different decimal points) results for Frobenius norm, but for 2-norm, the results are more different:</p>
<p>Here is the code for reproduce:</p>
<blockquote>
<p>import torch<br>
from scipy.linalg import norm<br>
import numpy as np<br>
a = np.arange(9) - 4.0<br>
a = a.reshape((3, 3))<br>
test1 = np.linalg.norm(a)<br>
7.745966692414834<br>
test2 = torch.norm(torch.from_numpy(a).cuda())<br>
tensor(7.7460, device=‘cuda:0’, dtype=torch.float64)<br>
test1 = np.linalg.norm(a, ord=2)<br>
7.3484692283495345<br>
test2 = torch.norm(torch.from_numpy(a).cuda(), p=2)<br>
tensor(7.7460, device=‘cuda:0’, dtype=torch.float64)</p>
</blockquote>
<p>Do you think this is normal? I’m using Pytorch 1.2.0 on Ubuntu 18.04. I need to obtain exactly the same results for 2-norm.</p>
<p>Hao</p>