RuntimeError: arange_out not supported on CPUType for Half

catosphere: <p>Hello,</p>
<p>While browsing for half precision or mixed precision training in pytorch, I found these tools <a href="https://nvidia.github.io/apex/index.html" rel="nofollow noopener">https://nvidia.github.io/apex/index.html</a> and particularly the apex.amp that should allow for a rather straight-forward switch between different FP for training.</p>
<p>However, when I run existing FP32 codes to mixed precision or FP16 I get the following:<br>
RuntimeError: arange_out not supported on CPUType for Half<br>
which seems to be caused by torch.stft that I use for computing spectral reconstruction losses.</p>
<p>… Any way to use stft with FP16 please ? (I use pytorch 1.1.0 and cuda 9.1.85)</p>
<p>Thanks ! And if anyone can recommend some other tools/ressources for mixed precision training in pytorch, I would be very thankful for more references as I did not get good results with apex.amp</p>

ptrblck: <p>Some methods are not implemented for CPU tensors in FP16.<br>
Could you post a code snippet showing, how you’ve used <code>amp</code> and where this issue is triggered?</p>
<p>Also, could you elaborate a bit more about your results and what went wrong using <code>amp</code>?</p>

catosphere: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for answering my topic and picking up as well the issue I posted on github ! I put more details regarding amp in the github issue so I will try to answer you more details on this thread.</p>
<p>About the error when calling the stft operation. Since you answered about the difference for CPU and GPU tensors, I manually checked on ipython and indeed calling stft on a FP16 cpu tensor raises an error which is not the case for the same sent on a cuda device.</p>
<p>However, my code runs always on GPU. The line that causes the error in the loss calculation is</p>
<p><em>mag_grains = torch.norm(torch.stft(mb_grains,n_fft,hop_length=hop_size,win_length=win_size,window=torch.hann_window(win_size).to(device,non_blocking=True),center=False),dim=3)</em></p>
<p>mb_grains.type() == ‘torch.cuda.HalfTensor’ and device(type=‘cuda’, index=1) so everything should be in GPU memory … I dont understand what is the CPU data causing the error here, do you see it ?</p>
<p>About amp, are you okay with continuing the discussion on the github issue please ?</p>
<p>ps: upper in the code I put torch.set_default_dtype(torch.float16) to make sure that the window is created with the right dtype (as it is not automatically casted by amp in that case if I understood correct)</p>

ptrblck: <p>Sure! We can debug amp specific problems in <a href="https://github.com/NVIDIA/apex/issues/373">the issue</a>.</p>
<p>Thanks for the information!<br>
It looks like this problem is not amp-specific, but might be caused by setting the default dtype to <code>torch.float16</code>.<br>
E.g. this code should throw the same error:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.set_default_dtype(torch.float16)

device = 'cuda'
input = torch.randn(100, device=device)
n_fft = 100
hop_size = 10
win_size = 10

res = torch.stft(input,n_fft,hop_length=hop_size,win_length=win_size,window=torch.hann_window(win_size).to(device,non_blocking=True),center=False)
print(res)
</code></pre>
<p>Setting the default type to <code>float16</code> might most likely cause other errors, e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">input = torch.randn(100).to(device)
</code></pre>
<p>will also cause an error:</p>
<blockquote>
<p>RuntimeError: <em>th_normal</em> not supported on CPUType for Half</p>
</blockquote>

catosphere: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> I will answer on the other issue about setting up correctly amp for speeding up trainings.</p>
<p>Regarding pytorch, I removed the set_default_dtype and instead pass a dtype argument to functions that process data but that are not automatically casted by amp at the input of the model forward.</p>
<p>I leave dtype=None if running opt_level at O0 (or running my original FP32 code), then data is not manually casted and is all the way created with the implicit FP32 default type.</p>
<p>For the other opt_levels, I set dtype=torch.float16 and inside the model functions, I cast data either:<br>
when creating with argument dtype=dtype eg. for the window in the stft function<br>
or with .type(dtype) eg.  when I sample the model prior <em>self.prior_distrib = distrib.Normal(torch.zeros(z_dim),torch.ones(z_dim))</em> it seems calling the sample function does not automatically cast the prior batch to FP16. So I do that manually before computing regularization.</p>
<p>Is this a better way to go for having a single code that can be run either in FP32, mixed or FP16 please ?</p>
<p>Now about the runtime error<br>
<em>mag_grains = torch.norm(torch.stft(mb_grains,n_fft,hop_length=hop_size,win_length=win_size,window=torch.hann_window(win_size,dtype=dtype).to(device),center=False),dim=3)</em><br>
in this case, the spectral reconstruction error function is called with dtype=torch.float16 so accordingly, the window is created with a manual cast argument dtype=dtype.</p>
<p>I get the <em>RuntimeError: arange_out not supported on CPUType for Half</em> ; if inspecting with ipython I have *mb_grains.type() == ‘torch.cuda.HalfTensor’ *and then when calling <em>torch.hann_window(win_size,dtype=dtype).to(device).type()</em> instead of having as well ‘torch.cuda.HalfTensor’ (which I expected) I got the same RuntimeError …</p>
<p>Which pointed me to modify the line throwing the error as<br>
<em>mag_grains = torch.norm(torch.stft(mb_grains,n_fft,hop_length=hop_size,win_length=win_size,window=torch.hann_window(win_size).to(device).type(dtype),center=False),dim=3)</em><br>
which runs now without error.</p>
<p>Is that an expected behavior ?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5e8fa8d526b44ce3194be50f9acc60bd0abbc1ad.jpeg" alt="36" data-base62-sha1="duwFCqpcmQonQPCnWXh1JINkHlX" width="612" height="407"></p>
<p>Now that the computation seems running all the way with FP16, I will look (with you on the other issue if you’re up for it) at possible speed gains with amp.</p>
<p>Thanks !</p>