CUDA Runtime Error when running Dreambooth/Stablediffusion

username: caldwecg
cooked: <p>Hello, I am running dreambooth on a ubuntu g4dn.2xlarge instance and am running into the following error when training the model. Below is the error and my configurations. Thanks!</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling <code>cublasGemmStridedBatchedExFix( handle, opa, opb, m, n, k, (void*)(&amp;falpha), a, CUDA_R_16F, lda, stridea, b, CUDA_R_16F, ldb, strideb, (void*)(&amp;fbeta), c, CUDA_R_16F, ldc, stridec, num_batches, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fd7a899caffd63970688b481875e8b302e79ff6.png" data-download-href="https://discuss.pytorch.org/uploads/default/0fd7a899caffd63970688b481875e8b302e79ff6" title="Screen Shot 2023-01-31 at 8.37.10 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fd7a899caffd63970688b481875e8b302e79ff6_2_690x465.png" alt="Screen Shot 2023-01-31 at 8.37.10 AM" data-base62-sha1="2g9cVm3dsE2lUlkscSpeRsiAlAG" width="690" height="465" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fd7a899caffd63970688b481875e8b302e79ff6_2_690x465.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fd7a899caffd63970688b481875e8b302e79ff6_2_1035x697.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fd7a899caffd63970688b481875e8b302e79ff6_2_1380x930.png 2x" data-dominant-color="131313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-01-31 at 8.37.10 AM</span><span class="informations">1446×976 56.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: caldwecg
cooked: <p>More info:</p>
<p>root:/home/ubuntu# python3 -m torch.utils.collect_env<br>
Collecting environment information…<br>
PyTorch version: 1.13.1+cu117<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.7<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 22.04.1 LTS (x86_64)<br>
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0<br>
Clang version: Could not collect<br>
CMake version: version 3.25.0<br>
Libc version: glibc-2.35</p>
<p>Python version: 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] (64-bit runtime)<br>
Python platform: Linux-5.15.0-1028-aws-x86_64-with-glibc2.35<br>
Is CUDA available: True<br>
CUDA runtime version: 11.5.119<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration: GPU 0: Tesla T4<br>
Nvidia driver version: 510.108.03<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] mypy-extensions==0.4.3<br>
[pip3] numpy==1.24.1<br>
[pip3] torch==1.13.1<br>
[pip3] torchvision==0.14.1</p>

username: caldwecg
cooked: <p>Update: installed PyTorch version: 1.12.1+cu116 and still see the error:</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling <code>cublasGemmStridedBatchedExFix(...)</code></p>

username: cardcounter
cooked: <p>I get an same error with:</p>
<pre><code class="lang-auto">torch=1.13.0
cuda 11.5
</code></pre>
<p>Did you fix it?</p>

username: caldwecg
cooked: <p>I was able to fix it but changed a bunch of things. Below is a partially finished article I typed out for training the model, it lists the requirements that ended up working for me: <a href="https://medium.com/@carter2abq/running-dreambooth-with-scalable-aws-backend-b9a075c658b9" rel="noopener nofollow ugc">https://medium.com/@carter2abq/running-dreambooth-with-scalable-aws-backend-b9a075c658b9</a></p>