CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

shamoons: <p>I see this has been posted a few times before, but none of those responses have helped.</p>
<p>From <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-error-cublas-status-alloc-failed-when-calling-cublascreate-handle/78545/2">here</a>, I’m trying:</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 &amp;&amp; export CUDA_VISIBLE_DEVICES=0 &amp;&amp; python train_transformer_encoder.py --batch_size 1
</code></pre>
<p>But I’m getting:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_transformer_encoder.py", line 207, in &lt;module&gt;
    main()
  File "train_transformer_encoder.py", line 129, in main
    train_loss = train(reconstruct_spect_model, optimizer=optimizer, dataset=train_loader)
  File "train_transformer_encoder.py", line 194, in train
    pred = model(x)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shamoon/speech-reconstruction/models/transformer_reconstruct.py", line 45, in forward
    src = self.inp_embedding(src)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`
</code></pre>
<p>So I’ve tried a <code>batch_size</code> of 1 and the <code>CUDA_LAUNCH_BLOCKING</code> as well, but no dice. Any help would be greatly appreciated.</p>
<p>Thank you in advance</p>

shamoons: <p>Also - I want to add that when. I run on CPU, it’s fine.</p>

shamoons: <p>I created a MUCH simpler example:</p>
<pre><code class="lang-auto">import torch

if __name__ == "__main__":
    DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    m = torch.nn.Linear(20, 30).to(DEVICE)
    input = torch.randn(128, 20).to(DEVICE)
    output = m(input)
    print('output', output.size())
    exit()
</code></pre>
<p>and get the same error.</p>

Daniel_Hernandez: <p>I am joining the wagon.</p>
<p>I am getting the same error as <a class="mention" href="/u/shamoons">@shamoons</a> on the same example. The fix of <code>CUDA_LAUNCH_BLOCKING=1</code> and <code>CUDA_VISIBLE_DEVICES=0</code> did nothing. Running on CPU works well.</p>
<p>This operation succeeds:</p>
<pre><code class="lang-python">&gt;&gt;&gt; a = torch.tensor([1]).cuda()
&gt;&gt;&gt; b = torch.rand([1]).cuda()
&gt;&gt;&gt; c = a + b
&gt;&gt;&gt; print(c)
$ tensor([2], device='cuda:0')
</code></pre>
<p>The following throws the original error in this post</p>
<pre><code class="lang-python">&gt;&gt;&gt; l = torch.nn.Linear(1, 1).cuda()
&gt;&gt;&gt; a = torch.tensor([1.]).cuda()
&gt;&gt;&gt; l(a)
</code></pre>
<p>My specs:<br>
PyTorch version: <code>1.8.0</code><br>
CUDA version: <code>11.0</code><br>
Driver version: <code>450.102.04</code><br>
NVIDIA-SMI: <code>450.102.04</code><br>
GPU: <code>NVIDIA GeForce RTX 2080 SUPER</code><br>
OS: <code>Ubuntu 20.04.2</code><br>
Kernel: <code>5.8.0-44</code></p>

Daniel_Hernandez: <p>PARTIAL FIX:</p>
<p>It seems that downgrading to PyTorch version <code>1.7.1</code> fixed the issue for me. This is obviously not idea, but maybe it can help you too <a class="mention" href="/u/shamoons">@shamoons</a>.</p>
<p>It would still be nice to hear from official sources to see if they have something to say <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> (thank you a lot <code>torch</code> devs and admins for making our lives better)</p>

shamoons: <p>It helped with the simple case by downgrading. But now my actual code has the same error. If it matters, I’m using a virtual environment.</p>

Daniel_Hernandez: <p>Can you share the same specs that I shared for my system above? Hopefully we can figure something out</p>

shamoons: <p>Are you on Discord or something perhaps? We can screen share</p>

ptrblck: <p><a class="mention" href="/u/shamoons">@shamoons</a> <a class="mention" href="/u/daniel_hernandez">@Daniel_Hernandez</a>  could you both post the output of <code>python -m torch.utils.collect_env</code>?</p>
<p><a class="mention" href="/u/daniel_hernandez">@Daniel_Hernandez</a> 's env shows CUDA11.0, which is not shipped in the <code>1.8.0</code> binaries (10.2 and 11.1 are used), so I’m unsure where this package comes from. Did you build it from source?</p>
<p>If you are using the 10.2 pip wheels (not conda), note that <code>sm_75</code> was pruned from them and you would have to use the 11.1 pip wheels.<br>
This issue is also tracked <a href="https://github.com/pytorch/pytorch/issues/53336">here</a>.</p>

Daniel_Hernandez: <p>It’s the legendary <a class="mention" href="/u/ptrblck">@ptrblck</a>! Thank you a lot, truly.</p>
<p>I am going to post 2 different calls to <code>python -m torch.utils.collect_env</code>. One from my system environment using version <code>1.7.1</code> and another using torch <code>1.8.0</code> on a virtual environment. Oddly enough it seems that <code>torch</code> is using a different CUDA version than what’s specified in <code>nvidia-smi</code>! (it’s using 10.2, instead of 11.0).</p>
<p>Torch version <code>1.7.1</code> where the original error message DOES NOT appear:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 SUPER
Nvidia driver version: 450.102.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.7.1
[conda] Could not collect
</code></pre>
<p>Now, the virtualenv <code>1.8</code> version where the original error DOES happen:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.8.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 SUPER
Nvidia driver version: 450.102.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.8.0
[conda] Could not collect
</code></pre>
<pre><code>Did you build from source?
</code></pre>
<p>No, I installed via <code>pip install torch</code> with <code>pip</code> version <code>20.0.2</code> on Python version <code>3.8.5</code>. The <code>pip</code> version inside my virtualenv is <code>21.0.1</code></p>
<p>I am sorry for the noob question. How might I install the <code>11.1</code> pip wheels? (assuming that <code>11.1</code> refers to the conda version)</p>
<p>Thanks again!<br>
Dani</p>

ptrblck: <aside class="quote no-group" data-username="Daniel_Hernandez" data-post="10" data-topic="114341">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daniel_hernandez/48/35645_2.png" class="avatar"> Daniel_Hernandez:</div>
<blockquote>
<p>Oddly enough it seems that <code>torch</code> is using a different CUDA version than what’s specified in <code>nvidia-smi</code>! (it’s using 10.2, instead of 11.0).</p>
</blockquote>
</aside>
<p>That’s expected, since the pip wheels and conda binaries ship with their own CUDA runtime.<br>
Your local CUDA toolkit (shown via e.g. <code>nvidia-smi</code>) would be used if you are building a custom CUDA extension or PyTorch from source.</p>
<p>Thanks for the env information. Based on this, you are indeed using the pip wheels with the CUDA10.2 runtime, which are broken on the Turing architecture (see the linked issue).</p>
<aside class="quote no-group" data-username="Daniel_Hernandez" data-post="10" data-topic="114341">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daniel_hernandez/48/35645_2.png" class="avatar"> Daniel_Hernandez:</div>
<blockquote>
<p>I am sorry for the noob question. How might I install the <code>11.1</code> pip wheels? (assuming that <code>11.1</code> refers to the conda version)</p>
</blockquote>
</aside>
<p>11.1 refers to the CUDA runtime version. You can install it by selecting <code>CUDA 11.1</code> <a href="https://pytorch.org/get-started/locally/">here</a>:</p>
<pre><code class="lang-auto">pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>

Daniel_Hernandez: <p>Oh yeah! It worked! And I’ve now learnt that pip wheels ship with their own CUDA runtime!</p>
<p>Installing torch with CUDA <code>11.1</code> with the following command did fix the initial issue with torch <code>1.8</code>:</p>
<pre><code> pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>May you have an amazing day! Good luck <a class="mention" href="/u/shamoons">@shamoons</a> with you issue, do try out <a class="mention" href="/u/ptrblck">@ptrblck</a>’s suggestions and hopefully you’ll be set!</p>

skpig: <p>I’ve encountered the same error but with <code>torch 1.10.2</code> and <code>CUDA 11.3</code>.</p>
<p>the error info as below:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 349, in &lt;module&gt;
    main_train()
  File "train.py", line 315, in main_train
    train(rank, args, train_dataset, valid_dataset, model, collator, tokenizer)
  File "train.py", line 199, in train
    loss += train_iter(model, batch, optimizer, scheduler, device)
  File "train.py", line 99, in train_iter
    labels=labels, return_dict=False)
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/deepspeed/runtime/engine.py", line 1588, in forward
    loss = self.module(*inputs, **kwargs)
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/transformers/models/led/modeling_led.py", line 2370, in forward
    return_dict=return_dict,
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/transformers/models/led/modeling_led.py", line 2238, in forward
    return_dict=return_dict,
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/transformers/models/led/modeling_led.py", line 2102, in forward
    use_cache=use_cache,
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/transformers/models/led/modeling_led.py", line 1024, in forward
    output_attentions=output_attentions,
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/huangbz/.conda/envs/Graph/lib/python3.6/site-packages/transformers/models/led/modeling_led.py", line 819, in forward
    attn_weights = torch.bmm(query_states, key_states.transpose(1, 2))
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasGemmStridedBatchedExFix( handle, opa, opb, m, n, k, (void*)(&amp;falpha), a, CUDA_R_16F, lda, stridea, b, CUDA_R_16F, ldb, strideb, (void*)(&amp;fbeta), c, CUDA_R_16F, ldc, stridec, num_batches, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)`
terminate called after throwing an instance of 'c10::Error'
  what():  NCCL error in: /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/distributed/c10d/NCCLUtils.hpp:181, unhandled cuda error, NCCL version 21.0.3
</code></pre>
<p>the output of <code>python -m torch.utils.collect_env</code>:</p>
<pre><code class="lang-auto">PyTorch version: 1.10.2
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.9

Python version: 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)  [GCC 7.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-65-generic-x86_64-with-debian-bullseye-sid
Is CUDA available: True
CUDA runtime version: 11.3.58
GPU models and configuration:
GPU 0: NVIDIA A40
GPU 1: NVIDIA A40
GPU 2: NVIDIA A40
GPU 3: NVIDIA A40

Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.10.2
[conda] blas                      1.0                         mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] cudatoolkit               11.3.1               h2bc3f7f_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl                       2022.0.1           h06a4308_117    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] numpy                     1.19.5                    &lt;pip&gt;
[conda] pytorch                   1.10.2          py3.6_cuda11.3_cudnn8.2.0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] pytorch-mutex             1.0                        cuda    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
</code></pre>
<p>I installed torch via <code>conda install pytorch cudatoolkit=11.3</code></p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce this issue, please?</p>

Rosa-sinensis: <p>Hey there,<br>
I have encountered the same problem as <a class="mention" href="/u/skpig">@skpig</a> .<br>
I ran the simple code given by shamoons (as shown below) in two different GPUs, which are <strong>Tesla V100-PCIE-32GB</strong>, and <strong>NVIDIA A100-SXM-80GB</strong></p>
<pre><code class="lang-auto">import torch

if __name__ == "__main__":
    DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    m = torch.nn.Linear(20, 30).to(DEVICE)
    input = torch.randn(128, 20).to(DEVICE)
    output = m(input)
    print('output', output.size())
    exit()
</code></pre>
<p>The code above run fine on <strong>Tesla V100</strong>, but when it runs on <strong>NVIDIA A100</strong>, it throws out the following error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
</code></pre>
<p>I then execute <code>python -m torch.utils.collect_env</code> for both GPUs to try find any discrepancies between the two GPUs I’m testing on.<br>
For  <strong>Tesla V100</strong>, it gave:</p>
<pre><code class="lang-auto">PyTorch version: 1.10.2
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: CentOS Stream release 8 (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-8)
Clang version: 13.0.0 (Red Hat 13.0.0-3.module_el8.6.0+1074+380cef3f)
CMake version: version 3.20.2
Libc version: glibc-2.28

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.18.0-358.el8.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 11.2.152
GPU models and configuration: 
GPU 0: Tesla V100-PCIE-32GB
GPU 1: Tesla V100-PCIE-32GB
GPU 2: Tesla V100-PCIE-32GB
GPU 3: Tesla V100-PCIE-32GB
GPU 4: Tesla V100-PCIE-32GB
GPU 5: Tesla V100-PCIE-32GB
GPU 6: Tesla V100-PCIE-32GB
GPU 7: Tesla V100-PCIE-32GB

Nvidia driver version: 470.82.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] numpydoc==1.2
[pip3] pytorch-msssim==0.2.1
[pip3] torch==1.10.2
[pip3] torchaudio==0.10.2
[pip3] torchvision==0.11.3
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] mypy_extensions           0.4.3            py39h06a4308_1  
[conda] numpy                     1.20.3           py39hf144106_0  
[conda] numpy-base                1.20.3           py39h74d4b33_0  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] pytorch                   1.10.2          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-msssim            0.2.1                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.10.2               py39_cu113    pytorch
[conda] torchvision               0.11.3               py39_cu113    pytorch

</code></pre>
<p>And for <strong>NVIDIA A100</strong>, it gave:</p>
<pre><code class="lang-auto">PyTorch version: 1.10.2
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: CentOS Stream release 8 (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10)
Clang version: 13.0.0 (Red Hat 13.0.0-3.module_el8.6.0+1074+380cef3f)
CMake version: version 3.20.2
Libc version: glibc-2.28

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.18.0-348.el8.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 11.2.152
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM-80GB
GPU 1: NVIDIA A100-SXM-80GB
GPU 2: NVIDIA A100-SXM-80GB
GPU 3: NVIDIA A100-SXM-80GB

Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] numpydoc==1.2
[pip3] pytorch-msssim==0.2.1
[pip3] torch==1.10.2
[pip3] torchaudio==0.10.2
[pip3] torchvision==0.11.3
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] mypy_extensions           0.4.3            py39h06a4308_1  
[conda] numpy                     1.20.3           py39hf144106_0  
[conda] numpy-base                1.20.3           py39h74d4b33_0  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] pytorch                   1.10.2          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-msssim            0.2.1                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.10.2               py39_cu113    pytorch
[conda] torchvision               0.11.3               py39_cu113    pytorch
</code></pre>
<p>I’m wondering what causes this issue and how to solve it. Any help is really appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>It seems the <code>CUBLAS_STATUS_INTERNAL_ERROR</code> changed to <code>CUBLAS_STATUS_NOT_INITIALIZED</code> which could point towards a setup issue.<br>
Was this setup working in the past and if so, did you update the drivers without a reboot etc.?</p>

Rosa-sinensis: <p>I’m not really sure as this is my first time using the Nvidia A100 GPU.<br>
If this is a setup issue, I’ll ask my administrator then about this issue. Thanks for the reply</p>

Maddie_Newell: <p>I am using gpu. If I run to many instances (more then one training sessions) on the gpu could this happen?</p>

Maddie_Newell: <p>I updated via ```<br>
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a> and got a <strong>RuntimeError: CUDA error: out of memory.</strong></p>
<p>If this is helpful to anyone in the future.</p>

Owais_Ansari: <p>Reduce the memory consumption by reducing the batch size or using lighter architecture. You can also reduce the image size or the token length if you working with text.</p>