I have Unrecognized CachingAllocator option: 1024 error

Kureshik: <p>Every time I use AI app such as Stable Diffusion, Language models or Voice changer etc. The following error occurs: <code>Unrecognized CachingAllocator option: 1024</code></p>
<p>I have tried reinstalling apps, changing their configs but it doesn’s help at all. Also I have newest nvidia drivers.</p>
<p>I have Nvidia GeForce RTX 3060 with 12Gb VRAM. My platform is windows. All programmes work on python<br>
I also installed NVIDIA CUDA Tollkit 12.3 as some people recommended to me.</p>
<p>For example Fooocus logs:</p>
<pre><code class="lang-auto">C:\Users\Roman\Downloads\Fooocus_win64_2-1-791&gt;.\python_embeded\python.exe -s Fooocus\entry_with_update.py
Already up-to-date
Update succeeded.
[System ARGV] ['Fooocus\\entry_with_update.py']
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Fooocus version: 2.1.822
Running on local URL:  http://127.0.0.1:7865

To create a public link, set `share=True` in `launch()`.
Exception in thread Thread-2 (worker):
Traceback (most recent call last):
  File "threading.py", line 1016, in _bootstrap_inner
  File "threading.py", line 953, in run
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\modules\async_worker.py", line 25, in worker
    import modules.default_pipeline as pipeline
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\modules\default_pipeline.py", line 1, in &lt;module&gt;
    import modules.core as core
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\modules\core.py", line 1, in &lt;module&gt;
    from modules.patch import patch_all
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\modules\patch.py", line 6, in &lt;module&gt;
    import fcbh.model_base
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\backend\headless\fcbh\model_base.py", line 2, in &lt;module&gt;
    from fcbh.ldm.modules.diffusionmodules.openaimodel import UNetModel
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\backend\headless\fcbh\ldm\modules\diffusionmodules\openaimodel.py", line 16, in &lt;module&gt;
    from ..attention import SpatialTransformer
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\backend\headless\fcbh\ldm\modules\attention.py", line 10, in &lt;module&gt;
    from .sub_quadratic_attention import efficient_dot_product_attention
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\backend\headless\fcbh\ldm\modules\sub_quadratic_attention.py", line 27, in &lt;module&gt;
    from fcbh import model_management
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\backend\headless\fcbh\model_management.py", line 114, in &lt;module&gt;
    total_vram = get_total_memory(get_torch_device()) / (1024 * 1024)
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\Fooocus\backend\headless\fcbh\model_management.py", line 83, in get_torch_device
    return torch.device(torch.cuda.current_device())
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\python_embeded\lib\site-packages\torch\cuda\__init__.py", line 769, in current_device
    _lazy_init()
  File "C:\Users\Roman\Downloads\Fooocus_win64_2-1-791\python_embeded\lib\site-packages\torch\cuda\__init__.py", line 298, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unrecognized CachingAllocator option: 1024
</code></pre>

Blasha: <p>I am also getting this error. It’s weird because I got it to work on another pc with the same environment I have right now which is Ubuntu 20.04, cuda 12.1, torch 2.1.1, and python 3.10.13, but somehow it doesn’t work with my RTX 3060 TI with 8GB of vram. I got it to work on a RTX 2080 super. I uninstalled the previous cuda version I had by purging it and I installed the new one following official nvidia docs (I followed the deb local installation): <a href="https://developer.nvidia.com/cuda-12-1-0-download-archive" rel="noopener nofollow ugc">https://developer.nvidia.com/cuda-12-1-0-download-archive</a></p>
<p>print(torch._C._cuda_getDeviceCount()) <span class="hashtag-raw">#outputs</span> 1<br>
And nvidia-smi says I am using cuda version 12.1 with driver version 530.30.02<br>
Does anyone know what I can do?</p>
<p><a class="mention" href="/u/kureshik">@Kureshik</a> I don’t know if cuda 12.3 is compatible with any torch versions yet since this cuda release is new. I was struggling to get it to work with torch and couldn’t. Maybe try downgrading to cuda 12.1 from the link I provided and reboot your pc and see if that resolves the issue.</p>

ptrblck: <aside class="quote no-group" data-username="Blasha" data-post="2" data-topic="192205">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blasha/48/65972_2.png" class="avatar"> Blasha:</div>
<blockquote>
<p>I don’t know if cuda 12.3 is compatible with any torch versions yet since this cuda release is new. I was struggling to get it to work with torch and couldn’t. Maybe try downgrading to cuda 12.1 from the link I provided and reboot your pc and see if that resolves the issue.</p>
</blockquote>
</aside>
<p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or custom CUDA extensions, since the PyTorch binaries ship with their own CUDA runtime dependencies.<br>
You would only need a properly installed NVIDIA driver.</p>

Farrenzo: <p>Was this ever resolved? I am running the following simple code:</p>
<pre><code class="lang-auto">import torch
print(torch.cuda.get_device_properties(0))
"""
ERROR:
File "C:\Programming\Python\Python311\Lib\site-packages\torch\cuda\__init__.py", line 319, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unrecognized CachingAllocator option: True
"""
</code></pre>

ptrblck: <p>Are you exporting env variables trying to manipulate the caching allocator?</p>

Kureshik: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> In fact, that was the case!<br>
<a class="mention" href="/u/farrenzo">@Farrenzo</a> yes, I was able to fix the error quite accidentally by finding the CachingAllocator (or smth like that) environment variable with a value of 1024 on my computer. I have absolutely no idea how it ended up there) Just deleted it, although I’m sure that’s not your case</p>

Farrenzo: <p>Yes this solved it. At first I didn’t understand what you were saying but then I checked environment variables and I had <code>PYTORCH_CUDA_ALLOCATOR</code> set to <code>True</code>. I simply deleted it and it worked.</p>