[nonissue] Autograd fails when using half-precision - overflow on matrix size

dimitry12: <p>I got Titan V and have been experimenting with half-precision.</p>
<p>In half-precision mode I can’t backpropagate through matmul of two all-zeros matrices, because the <em>number of elements</em> in the resulting matrix is outside of half-precision range.</p>
<p>I am getting the same error if I use Conv1d or Conv2d or bmm.</p>
<p>This minimal computation graph replicates the problem:</p>
<pre><code>import torch, torch.autograd, torch.nn,numpy
with torch.cuda.device(0):

    test_input = torch.autograd.Variable(torch.zeros(257, 509)).cuda().half()
    test_w = torch.nn.Parameter(torch.zeros(509,263)).cuda().half()

    matmul_result = torch.matmul(test_input, test_w)
    print(matmul_result.size())
    print(numpy.prod(matmul_result.size()))

    test_output = matmul_result.abs().mean()
    test_output.backward()
</code></pre>
<p>And the result is:</p>
<pre><code>torch.Size([257, 263])
67591
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 11, in &lt;module&gt;
  File "/home/dzmitry/miniconda3/lib/python3.6/site-packages/torch/autograd/variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/home/dzmitry/miniconda3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)
RuntimeError: value cannot be converted to type Half without overflow: 67591
</code></pre>
<p>I am using PyTorch 0.3 and could replicate the issue with CUDA 8 and CUDA 9.</p>

dimitry12: <p>I found my mistake. It actually fails because I use <code>.mean()</code>, which of course has the total number of elements in it.</p>

rahulbhalley: <p>So what is the solution to this problem? I’m still stuck! Would you recommend any changes to <code>mean()</code> code?</p>

ptrblck: <p>Reductions are sensitive to overflow if you are using FP16.<br>
You should perform all reductions in FP32 just to make sure to get a valid result.<br>
Just call <code>.float()</code> on your tensor before passing it to <code>torch.mean()</code>.<br>
Autograd will rewind this operation in the backward pass so that your model will still be in <code>half</code> precision.</p>

rahulbhalley: <p>Thanks. It works but the loss goes to NaN. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=6" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> I think I gotta read some low-precision training papers before trying to do my own experiments with the handling of these NaNs. I find that it’s super unstable to train with low precision. The value get washed out easily.</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/training-partially-with-fp16/31873/2?u=ptrblck">this post</a> for some info on FP16 training.</p>

suke: <p>Hello, I have a question about FP16, I found if I apply FP 16 in forward process, it will very slow.<br>
e.g. FP32 forward time 3.27s<br>
FP16 forward time 13s</p>
<p>there are nn.Linear and torch.matmul in my nework. I want to ask does FP16 optimized for special ops not for all ops?</p>

ptrblck: <p>FP16 operations can be accelerated using Tensor cores (or Volta cores), which were introduced in Volta and Turing GPUs (e.g. GTX 2080).</p>
<p>Linear layers use GEMMs to compute the output. The matrices involved in this computation must have sizes of multiples of 8 to use Tensor cores:</p>
<pre><code class="lang-python">a = torch.randn(8, 16)
b = torch.randn(16, 32)
torch.matmul(a, b)
</code></pre>
<p>This requirement exists for all cublas and cudnn versions, as far as I know.</p>
<p>There were also other requirements for convolution layers for cudnn 7.2 and older.<br>
The current PyTorch binaries ship with cudnn 7.6, so you don’t have to worry about it.</p>
<p>That being said, the posted times look quite bad.<br>
Could you post the code you’ve used to profile your model?<br>
Also, which GPU are you using?</p>

suke: <p>Hi, code is too much, I just try in my code, seems problem occurred in torch.matmul() function.<br>
below is my training process<br>
FP32 each batch<br>
Time: 0.00418s</p>
<p>Time: 0.00555s</p>
<p>Time: 0.00513s</p>
<p>Time: 0.00510s</p>
<p>Time: 0.00475s</p>
<p>Time: 0.00477s</p>
<p>Time: 0.00465s</p>
<p>Time: 0.00469s</p>
<p>in case of FP16 each batch:<br>
Time: 0.46559s</p>
<p>Time: 0.46569s</p>
<p>Time: 0.46576s</p>
<p>Time: 0.46529s</p>
<p>Time: 0.46545s</p>
<p>Time: 0.46568s</p>
<p>Time: 0.46566s</p>
<p>Time: 0.46571s</p>
<p>Time: 0.46584s</p>
<p>my GPU is GTX1080ti</p>

ptrblck: <p>Your GTX 1080Ti does not use Tensor / Volta cores, so you won’t see any speedup.<br>
However, a performance degradation of x100 shouldn’t occur.<br>
Without code, it’s hard to debug the issue. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

suke: <p>test code:</p>
<p>torch.matmul(col, weight)<br>
col shape[10, 22500, 1, 1728]<br>
weight shape[22500, 1728, 3]</p>
<p>that will slow using precision half</p>

ptrblck: <p>Unfortunately, only 1728 is a multiple of 8, so that this operation won’t be sped up in a GEMM using FP16.<br>
On my Titan V, the operation in FP16 is approx. 10x slower than in FP32.<br>
When I adapt the shapes to multiples of 8, I get approx. a 2x speedup.</p>

suke: <p>I improve my code:<br>
start = time.process_time()<br>
a = torch.randn(8, 8000, 1, 1728)<br>
b = torch.randn(8000, 1728, 3)<br>
<span class="hashtag">#a</span> = a.half()<br>
<span class="hashtag">#b</span> = b.half()<br>
if True:<br>
device = torch.device(“cuda”)<br>
else:<br>
device = torch.device(“cpu”)<br>
a = a.to(device)<br>
b = b.to(device)<br>
torch.matmul(a, b)<br>
end = time.process_time()<br>
print(“Elapsed time of {}: {} s”.format(‘main’, np.round(end - start, decimals=5)))</p>
<p>FP16: 4.7057 s<br>
FP32: 3.43783 s</p>

ptrblck: <p>As I said, your GPU does not have any Volta or Tensor cores so the performance won’t be improved.<br>
However, you’ll see a smaller memory usage using FP16.</p>
<p>If you would like to train your model using mixed precision, I would recommend to have a look at <a href="https://github.com/NVIDIA/apex" rel="nofollow noopener">apex/amp</a>.</p>

suke: <p>thank you for your reply, I have another problem.<br>
test code:<br>
a = torch.randn(1, 49920, 512)<br>
c = torch.randn(1, 512, 49920)<br>
<span class="hashtag">#b</span> = torch.randn(153000, 288, 3, 3)<br>
<span class="hashtag">#a</span> = a.half()<br>
<span class="hashtag">#b</span> = b.half()<br>
if True:<br>
device = torch.device(“cuda”)<br>
else:<br>
device = torch.device(“cpu”)<br>
a = a.to(device)<br>
<span class="hashtag">#b</span> = b.to(device)<br>
c = c.to(device)<br>
torch.cuda.synchronize()<br>
start = time.process_time()<br>
<span class="hashtag">#torch</span>.matmul(a, b)<br>
<span class="hashtag">#F</span>.conv1d(a, c)<br>
torch.matmul(a,c)</p>
<p>as I checked the input feature map size should be 49920x512x4 /1024/1024 = 97.5M<br>
but when I run it in GPU (GTX1080ti), the top memory will cost more than 10G.<br>
do you why for that, and how to avoid it.</p>

ptrblck: <p>Since the result of <code>torch.matmul(a, c)</code> will have the shape <code>[1, 49920, 49920]</code>, it will use approx. 9.5GB of memory.<br>
Given that the CUDA context uses a bit of memory as well as <code>a</code> and <code>b</code>, the usage of 10GB seems reasonable.</p>

suke: <p>yes, it is. thank you!</p>