CUDA out of memory , empty_cache returns ">"

MarkCRobinson: <p>Trying to train, I get<br>
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1002.00 MiB. GPU 0 has a total capacity of 15.69 GiB of which 788.88 MiB is free.</p>
<p>Recommended answer is<br>
torch.cuda.empty_cache()</p>
<p>All I get when I enter this command is<br>
A greater-than cursor</p>
<p>Ubuntu<br>
Using Meta-Llama-3-8B-Instruct and Oogabooga to train.<br>
16G GPU - nothing signifigant should be running.</p>
<p>NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |<br>
|-----------------------------------------±-----------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |<br>
|=========================================+========================+======================|<br>
|   0  NVIDIA GeForce RTX 4070 …    Off |   00000000:01:00.0  On |                  N/A |<br>
|  0%   35C    P8              7W /  285W |   15273MiB /  16376MiB |      3%      Default |<br>
|                                         |                        |                  N/A |<br>
±----------------------------------------±-----------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------------------+<br>
| Processes:                                                                              |<br>
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |<br>
|        ID   ID                                                               Usage      |<br>
|=========================================================================================|<br>
|    0   N/A  N/A      2501      G   /usr/lib/xorg/Xorg                            195MiB |<br>
|    0   N/A  N/A      2884    C+G   …libexec/gnome-remote-desktop-daemon        211MiB |<br>
|    0   N/A  N/A      2937      G   /usr/bin/gnome-shell                           26MiB |<br>
|    0   N/A  N/A      3807      G   /opt/teamviewer/tv_bin/TeamViewer              16MiB |<br>
|    0   N/A  N/A      4551      G   …AAAAAAAACAAAAAAAAAA= --shared-files         25MiB |<br>
|    0   N/A  N/A      4646      G   /usr/bin/nautilus                              27MiB |<br>
|    0   N/A  N/A      5920      G   …seed-version=20240913-050142.817000         85MiB |<br>
|    0   N/A  N/A     14891      C   python                                      14652MiB |</p>
<p>Suggestions appreciated.</p>

ptrblck: <p>I’m unsure I understand your question correctly. Are you in a Python REPL and are just seeing the <code>&gt;&gt;&gt;</code> markers waiting for your input or is this call printing it?</p>

MarkCRobinson: <p>I typed it on a Linux command line.<br>
I’m just getting started and only have a glimmer of what I’m doing. Should I be inside an environment first?</p>

ptrblck: <p>I’m not sure where exactly you are seeing this output, but in any case calling <code>empty_cache()</code> won’t find the OOM issue since PyTorch will try to release the cache and reallocate the memory internally first before raising the OOM error to the user. You might this need to decrease the overall memory usage e.g. by reducing the batch size of your training.</p>