CPU and GPU tensor norm different

caiwenpu: <pre><code class="lang-auto">torch.manual_seed(0)
W = torch.FloatTensor(10, 5000).normal_(0,1)

print(W.norm(2) ** 2, W.sum())
W = W.cuda()

print(W.norm(2) ** 2, W.sum())

W = W.cpu()
print(W.norm(2) ** 2, W.sum())
</code></pre>
<p>When I run this code, the results is</p>
<pre><code class="lang-auto">tensor(97479.4609) tensor(-358.1119)
tensor(50004.2148, device='cuda:0') tensor(-358.1118, device='cuda:0')
tensor(97479.4609) tensor(-358.1119)
</code></pre>
<p>The results of norm is significantly different, but the results of sum is almost the same.<br>
When I change the size of tensor W from (10,5000) to (10,1000). I get the results:</p>
<pre><code class="lang-auto">tensor(10038.8496) tensor(-106.9817)
tensor(10038.8516, device='cuda:0') tensor(-106.9818, device='cuda:0')
tensor(10038.8496) tensor(-106.9817)
</code></pre>
<p>The results of norm is almost the same.<br>
I want to figure out how to explain these results?</p>

sakaia: <p>Interesting results! But I try to do it on Google Colab, the problem occured on large tensor.</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(0)
W = torch.FloatTensor(100, 5000000).normal_(0,1)

print(W.norm(2) ** 2, W.sum())
W = W.cuda()

print(W.norm(2) ** 2, W.sum())

W = W.cpu()
print(W.norm(2) ** 2, W.sum())
!nvidia-smi
</code></pre>
<pre><code class="lang-auto">tensor(1.5166e+08) tensor(-34600.6719)
tensor(5.0001e+08, device='cuda:0') tensor(-34600.1719, device='cuda:0')
tensor(1.5166e+08) tensor(-34600.6719)
Mon May 27 07:17:41 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   71C    P0    31W /  70W |   3244MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
</code></pre>

sakaia: <p>It seems CPU side problem.</p>
<pre><code class="lang-auto">import torch
HIGHT = 100
WIDTH = 5000000
torch.manual_seed(0)
W = torch.FloatTensor(HIGHT, WIDTH).normal_(0,1)
Wv = W.view(HIGHT*WIDTH,1)
WTW = Wv * Wv
print(WTW.sum())
print(W.norm(2) ** 2, W.sum())

W = W.cuda()
Wv = W.view(HIGHT*WIDTH,1)
WTW = Wv * Wv
print(WTW.sum())
print(W.norm(2) ** 2, W.sum())

W = W.cpu()
print(W.norm(2) ** 2, W.sum())
!nvidia-smi
</code></pre>
<pre><code class="lang-auto">tensor(4.8664e+08)
tensor(1.5166e+08) tensor(-34600.6719)
tensor(5.0001e+08, device='cuda:0')
tensor(5.0001e+08, device='cuda:0') tensor(-34600.1719, device='cuda:0')
tensor(1.5166e+08) tensor(-34600.6719)
Mon May 27 07:41:05 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   70C    P0    31W /  70W |   5152MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
</code></pre>

caiwenpu: <p>Oh, thanks, I understand. It is a bug on CPU side and GPU side has no problem.</p>