Pytorch is giving an error saying that it had found a bug and telling me to report it to pytorch

GVSS_Akhil: <p>i have the latest version of cuda ==12.2 but pytorch doesnt support a stable version if cuda==12.2 so i had downloaded pytorch==2.1.0 with a pytorch-cuda=12.1 but when iam running the torch code it is giving me an error saying that found a bug please report to pytorch.</p>
<p>Traceback (most recent call last):<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/bin/fairseq-train”, line 8, in <br>
sys.exit(cli_main())<br>
^^^^^^^^^^<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 574, in cli_main<br>
distributed_utils.call_main(cfg, main)<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/distributed/utils.py”, line 404, in call_main<br>
main(cfg, **kwargs)<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 205, in main<br>
valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/contextlib.py”, line 81, in inner<br>
return func(*args, **kwds)<br>
^^^^^^^^^^^^^^^^^^^<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 331, in train<br>
log_output = trainer.train_step(samples)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/contextlib.py”, line 81, in inner<br>
return func(*args, **kwds)<br>
^^^^^^^^^^^^^^^^^^^<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/trainer.py”, line 1029, in train_step<br>
raise e<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/trainer.py”, line 974, in train_step<br>
self.task.optimizer_step(<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/tasks/fairseq_task.py”, line 546, in optimizer_step<br>
optimizer.step()<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/fp16_optimizer.py”, line 223, in step<br>
self.fp32_optimizer.step(closure, groups=groups)<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/fairseq_optimizer.py”, line 135, in step<br>
self.optimizer.step(closure)<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/torch/optim/optimizer.py”, line 373, in wrapper<br>
out = func(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/adam.py”, line 195, in step<br>
state[“exp_avg_sq”] = torch.zeros_like(p_data_fp32)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: handle_0 INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1695392035891/work/c10/cuda/driver_api.cpp”:15, please report a bug to PyTorch.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc319c0428742e16e9ed5d16f2e01dbf2bbd9aef.png" data-download-href="https://discuss.pytorch.org/uploads/default/dc319c0428742e16e9ed5d16f2e01dbf2bbd9aef" title="Screenshot 2023-10-19 at 2.59.56 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc319c0428742e16e9ed5d16f2e01dbf2bbd9aef_2_690x84.png" alt="Screenshot 2023-10-19 at 2.59.56 PM" data-base62-sha1="vpVkxfuHQXryhrz9BCLdOX0vSH5" width="690" height="84" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc319c0428742e16e9ed5d16f2e01dbf2bbd9aef_2_690x84.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc319c0428742e16e9ed5d16f2e01dbf2bbd9aef_2_1035x126.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc319c0428742e16e9ed5d16f2e01dbf2bbd9aef_2_1380x168.png 2x" data-dominant-color="282828"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-10-19 at 2.59.56 PM</span><span class="informations">2522×310 33.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the error by wrapping it into three backticks ```, please?</p>

GVSS_Akhil: <p>so basically i am using fairseq model in my code and when iam trying to train it it is giving me this error in this file of the model <code>/fairseq/optim/adam.py</code> (adam optimizer) and this is the line where iam getting the error :</p>
<blockquote>
<pre><code>            if len(state) == 0:
                state["step"] = 0
                # Exponential moving average of gradient values
                state["exp_avg"] = torch.zeros_like(p_data_fp32)
                # Exponential moving average of squared gradient values
                state["exp_avg_sq"] = torch.zeros_like(p_data_fp32)
</code></pre>
</blockquote>
<p>Thank you</p>