UserWarning by PyTorch CUBLAS_WORKSPACE_CONFIG can not be parsed

username: Dylan_Prins
cooked: <p>The following errors occur:</p>
<p>python3.8/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at /opt/conda/conda-bld/pytorch_1682343998658/work/aten/src/ATen/cuda/CublasHandlePool.cpp:56.)</p>
<p>lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py:200: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at /opt/conda/conda-bld/pytorch_1682343998658/work/aten/src/ATen/cuda/CublasHandlePool.cpp:56.)</p>
<p>The warning is implemented in Pytorch itself. But i find it impossible to fix it. When setting the CUBLAS_WORKSPACE_CONFIG variable myself the error still occurs.</p>
<p>This seems to not be a problem with the previous major version of pytorch.</p>
<p>The warning is annoying because it gets spammed in our logs so it is difficult to find other warnings.</p>
<p>Any idea why this could occur and how to fix it?</p>
<h3>
<a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<p><strong>Versions</strong></p>
<pre><code class="lang-auto">PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.8.16 (default, Mar  2 2023, 03:21:46)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-1030-ibm-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla V100-PCIE-16GB
Nvidia driver version: 520.61.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   40 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          16
On-line CPU(s) list:             0-15
Vendor ID:                       GenuineIntel
Model name:                      Intel Xeon Processor (Cascadelake)
CPU family:                      6
Model:                           85
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
Stepping:                        6
BogoMIPS:                        4988.13
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat pku ospke avx512_vnni md_clear arch_capabilities
Virtualization:                  VT-x
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       512 KiB (16 instances)
L1i cache:                       512 KiB (16 instances)
L2 cache:                        32 MiB (8 instances)
L3 cache:                        16 MiB (1 instance)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-15
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Mitigation; Enhanced IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] numpy-ext==0.9.8
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchvision==0.15.2
[pip3] triton==2.0.0
[conda] blas                      1.0                         mkl  
[conda] captum                    0.6.0                         0    pytorch
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h7f8727e_0  
[conda] mkl_fft                   1.3.1            py38hd3c417c_0  
[conda] mkl_random                1.2.2            py38h51133e4_0  
[conda] numpy                     1.23.1                   pypi_0    pypi
[conda] numpy-base                1.24.3           py38h31eccc5_0  
[conda] numpy-ext                 0.9.8                    pypi_0    pypi
[conda] pytorch                   2.0.1           py3.8_cuda11.8_cudnn8.7.0_0    pytorch
[conda] pytorch-cuda              11.8                 h7e8668a_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] tensorflow-base           2.11.0          mkl_py38he5f8e37_0  
[conda] torchaudio                2.0.2                py38_cu118    pytorch
[conda] torchtriton               2.0.0                      py38    pytorch
[conda] torchvision               0.15.2               py38_cu118    pytorch
</code></pre>

username: ptrblck
cooked: <p>Which value did you set the variable to?</p>

username: Dylan_Prins
cooked: <p>Hi! Thanks for the response.</p>
<p>Both tried <code>:4096:8</code> and <code>8519680</code> (The default)</p>
<p>I tried it first by setting it in my bashrc (export CUBLAS_WORKSPACE_CONFIG=‘’:4096:8" or “8519680”) then since that didn’t work i tried to fix it by directly giving it in the terminal command (CUBLAS… python executable.py)</p>
<p>It basically seems to not be able to parse the CUBLAS_WORKSPACE_CONFIG variable in general</p>

username: ptrblck
cooked: <p>The env variable works for me so could you post exactly how you are running the script and the error you are seeing?</p>