[Reporting bug] INTERNAL ASSERT FAILED at "C:/w/b/windows/pytorch/aten/src\\ATen/native/cuda/Reduce.cuh":929, please report a bug to PyTorch

C_J: <p>I have below error message:</p>
<pre><code class="lang-auto">~~~~\lib\site-packages\torch\autograd\__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: sub_iter.strides(0)[0] == 0INTERNAL ASSERT FAILED at "C:/w/b/windows/pytorch/aten/src\\ATen/native/cuda/Reduce.cuh":929, please report a bug to PyTorch.
</code></pre>
<p>I have this issue when I increase my model size (not batch size).<br>
I have checked that this is not a memory issue, it only uses half of the free memory at its maximum.</p>
<p>Thanks for reading!</p>

ptrblck: <p>Are you seeing this issue using the latest master build or nightly binary? If so, could you post a minimal, executable code snippet to reproduce the issue, please?</p>

C_J: <p>Thanks for reply,<br>
I really want to, but my code very complicated (or messy) at this point so I’m not sure I could do that.</p>
<p>But what I can say is, this is happening alternatively with another memory issue.</p>
<p><strong>A (normal case)</strong>. Normal model size, batch size=2 per each GPU (takes &lt;50% of total gpu memory)<br>
<strong>B (CUDA out of memory)</strong>. Normal model size, batch size=3or2 per each GPU(takes &lt;&lt;70% of total gpu memory) link: <a href="https://discuss.pytorch.org/t/cuda-out-of-memory-how-to-reserve-memory-in-gpu/149797" class="inline-onebox">[CUDA out of memory] How to reserve memory in GPU?</a><br>
<strong>C (This case)</strong>. Larger model size, batch size=1 per each GPU (takes &lt;50% of total gpu memory)</p>
<p>So I think this is something related to the memory issue, but I have no clue.</p>
<p>Thanks,</p>

C_J: <p>I’m using pytorch 1.10 at window 10 machine.<br>
I’m sorry but I don’t know what you mean by master build or nightly binary.</p>

ptrblck: <p>By “master build” I was referring to a source build from the current master branch on GitHub and by “nightly binary” I was referring to the pip wheels or conda binaries, which are built each night from the current master branch. You can install the latter binaries by selecting “Preview (Nightly)” <a href="https://pytorch.org/get-started/locally/">here</a>.</p>

H_A: <p>I am having the same issue and using Google Colab’s GPU.<br>
the error is occuring on the line of code is:<br>
import torch</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
2 <span class="hashtag">#If</span> the video is corrupted delete the video.<br>
3 import glob<br>
----&gt; 4 import torch<br>
5 import torchvision<br>
6 from torchvision import transforms</p>
<p>/usr/local/lib/python3.8/dist-packages/torch/<strong>init</strong>.py in <br>
190     if USE_GLOBAL_DEPS:<br>
191         _load_global_deps()<br>
 → 192     from torch._C import *  # noqa: F403<br>
193<br>
194 # Appease the type checker; ordinarily this binding is inserted by the</p>
<p>RuntimeError: THPDtypeType.tp_dict == nullptr INTERNAL ASSERT FAILED at “…/torch/csrc/Dtype.cpp”:135, please report a bug to PyTorch.</p>

Alexander_Goryunov: <p>I am in the same boat. Using Google Colab with torch version 1.13.0+cu116</p>
<pre><code class="lang-auto"> 6 import yaml
      7 from sklearn.model_selection import train_test_split
----&gt; 8 from torch.utils.data import Subset
      9 from torch.nn.modules.container import ModuleList
     10 from transformers import AutoTokenizer, TrainingArguments

/usr/local/lib/python3.8/dist-packages/torch/__init__.py in &lt;module&gt;
    190     if USE_GLOBAL_DEPS:
    191         _load_global_deps()
--&gt; 192     from torch._C import *  # noqa: F403
    193 
    194 # Appease the type checker; ordinarily this binding is inserted by the

RuntimeError: THPDtypeType.tp_dict == nullptr INTERNAL ASSERT FAILED at "../torch/csrc/Dtype.cpp":135, please report a bug to PyTorch. 
</code></pre>
<p>And trying to import torch:</p>
<pre><code class="lang-auto">ValueError                                Traceback (most recent call last)

&lt;ipython-input-6-beb55f24d5ec&gt; in &lt;module&gt;
----&gt; 1 import torch
      2 print(torch.__version__)

/usr/local/lib/python3.8/dist-packages/torch/__init__.py in &lt;module&gt;
    190     if USE_GLOBAL_DEPS:
    191         _load_global_deps()
--&gt; 192     from torch._C import *  # noqa: F403
    193 
    194 # Appease the type checker; ordinarily this binding is inserted by the

ValueError: module functions cannot set METH_CLASS or METH_STATIC
</code></pre>
<p>Restarting the environment made the errors magically disappear.</p>

ptrblck: <p>I cannot reproduce the issue in a new Colab notebook using <code>torch==1.13.0+cu116</code> and <code>sklearn==1.0.2</code>.</p>

Alexander_Goryunov: <p>It probably has something to do with Colab itself, because I don’t see it anymore either.</p>

msf: <p>Hi all,<br>
I have this error too, but every time I restart the runtime in google Colab, the error is fixed, and after running the code two times, I again have this error. Thanks for the help<br>
<strong>false INTERNAL ASSERT FAILED at “…/c10/cuda/CUDAGraphsC10Utils.h”:73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus32740</strong></p>

ptrblck: <p>Could you share a minimal and executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

Ruicky8: <p>Hi!</p>
<p>I was having this error on colab today, and as mary said I fixed it restarting the runtime and changing from CPU to GPU. I have run the command python -m torch.utils.collect_env before and after the runtime restart, maybe it’s useful for you.</p>
<p>Before the runtime restart</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.22.6
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.10.147+-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.13.1+cu116
[pip3] torchaudio==0.13.1+cu116
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.14.1
[pip3] torchvision==0.14.1+cu116
[conda] Could not collect
</code></pre>
<p>After the restart:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.22.6
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.10.147+-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.13.1+cu116
[pip3] torchaudio==0.13.1+cu116
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.14.1
[pip3] torchvision==0.14.1+cu116
[conda] Could not collect
</code></pre>
<p>Also I was encountering the error when running the following:</p>
<pre><code class="lang-auto">import torch
from torch import nn

device = "cuda" if torch.cuda.is_available() else "cpu"
</code></pre>