Use of FP16 in backward() causes NaN in gradients only when create_graph = True

username: sjscotti
cooked: <p>Hi<br>
I am using pytorch within a chatbot training routine and I would like to get FP16’s advantages in GPU memory/speed.   My routine seems to work fine using FP32.  In switching it to FP16, my problem appears to be caused by the <code>loss.backward()</code>  pytorch routine.  For my optimizer to work, I need to use the argument <code>create_graph = True</code> in  <code>backward</code>.  When I do that with the model I am working with (for any scale factor on the loss that I’ve tried), <code>backward</code> gives <code>NaN</code> for some of the parameter gradients while others appear to have reasonable values.  However, when I use the argument <code>create_graph = False</code> to see how these gradients compare with the former case, there doesn’t appear to be any <code>NaN</code> generated and the gradients that weren’t <code>NaN</code> in the former case agree with what is computed here.  Since the <code>create_graph</code> is supposed to allow computation of higher order derivatives (but not the first derivative), I am confused why the <code>create_graph = True</code> argument is having this effect.  Any suggestions you can provide would be appreciated!</p>

username: ptrblck
cooked: <p>Could you post a minimal, executable code snippet to reproduce the issue, please?<br>
Also, the output of <code>python -m torch.utils.collect_env</code> would be needed.</p>

username: sjscotti
cooked: <p>The code is a fine tuning job for a chatbot with over 400M parameters (BlenderBot2, see <a href="https://github.com/facebookresearch/ParlAI/tree/main/projects/blenderbot2" class="inline-onebox" rel="noopener nofollow ugc">ParlAI/projects/blenderbot2 at main · facebookresearch/ParlAI · GitHub</a>) so it is not amenable to creating a code snippet to show the issue.  Is there another way to provide useful information?</p>
<p>Here is the env info…<br>
(ParlAI) C:\Users\Steve&gt;python -m torch.utils.collect_env<br>
Collecting environment information…<br>
PyTorch version: 1.9.0<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 10 Home<br>
GCC version: Could not collect<br>
Clang version: Could not collect<br>
CMake version: version 3.19.1<br>
Libc version: N/A</p>
<p>Python version: 3.8 (64-bit runtime)<br>
Python platform: Windows-10-10.0.19042-SP0<br>
Is CUDA available: True<br>
CUDA runtime version: 11.2.142<br>
GPU models and configuration: GPU 0: GeForce RTX 2070 SUPER<br>
Nvidia driver version: Could not collect<br>
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin\cudnn_ops_train64_8.dll<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.20.3<br>
[pip3] pytorch-ranger==0.1.1<br>
[pip3] torch==1.9.0<br>
[pip3] torch-optimizer==0.1.0<br>
[pip3] torchtext==0.10.0<br>
[conda] blas                      2.106                       mkl    conda-forge<br>
[conda] cudatoolkit               11.1.1               heb2d755_7    conda-forge<br>
[conda] libblas                   3.9.0                     6_mkl    conda-forge<br>
[conda] libcblas                  3.9.0                     6_mkl    conda-forge<br>
[conda] liblapack                 3.9.0                     6_mkl    conda-forge<br>
[conda] liblapacke                3.9.0                     6_mkl    conda-forge<br>
[conda] mkl                       2020.4             hb70f87d_311    conda-forge<br>
[conda] numpy                     1.21.1           py38h09042cb_0    conda-forge<br>
[conda] pytorch                   1.9.0           py3.8_cuda11.1_cudnn8_0    pytorch<br>
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi<br>
[conda] torch                     1.9.0                    pypi_0    pypi<br>
[conda] torch-optimizer           0.1.0                    pypi_0    pypi<br>
[conda] torchtext                 0.10.0                   pypi_0    pypi</p>

username: ptrblck
cooked: <p>Could you post a code snippet to see how you’ve initialized the model and post random input shapes, which we could use to try to reproduce the issue instead?</p>