torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate

antonis14: <p>Hi there.</p>
<p>I’m trying to run a version of the new Llama3 model.<br>
Specifically this one <a href="https://huggingface.co/NurtureAI/Meta-Llama-3-8B-Instruct-64k-GGUF" class="inline-onebox" rel="noopener nofollow ugc">NurtureAI/Meta-Llama-3-8B-Instruct-64k-GGUF · Hugging Face</a></p>
<p>When I run the code in the example, or modify the messages a bit, all works fine.</p>
<p>If I give a very large message ( let’s say some HTML code ) I get this error :</p>
<pre><code class="lang-auto">python ./main.py
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:05&lt;00:00,  1.36s/it]
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Traceback (most recent call last):
  File "/home/antouank/_REPOS_/test-llama3-64k/./main.py", line 1439, in &lt;module&gt;
    outputs = model.generate(
              ^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 1622, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 2791, in _sample
    outputs = self(
              ^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 1208, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 1018, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 756, in forward
    hidden_states = self.mlp(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 240, in forward
    down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/modules/activation.py", line 396, in forward
    return F.silu(input, inplace=self.inplace)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antouank/_REPOS_/test-llama3-64k/venv/lib/python3.11/site-packages/torch/nn/functional.py", line 2102, in silu
    return torch._C._nn.silu(input)
           ^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.68 GiB. GPU
</code></pre>
<p>my GPU is a 4090 and for sure has more free VRAM than 1.68! So what’s the issue here?<br>
How can I make it allow large messages?</p>
<p>I tried googling this but I cannot find any similar reference anywhere.</p>
<p>Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="antonis14" data-post="1" data-topic="201536">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/antonis14/48/69544_2.png" class="avatar"> antonis14:</div>
<blockquote>
<p>4090 and for sure has more free VRAM than 1.68</p>
</blockquote>
</aside>
<p>The error points to a failed allocation trying to allocate 1.68GB in addition to what is already allocated. Your error message is cut but should show the overall memory usage.</p>

antonis14: <p>that is the whole output. there’s nothing “cut”, unless I misunderstood what you say.</p>
<p>so a larger prompt can affect the memory needs in GBs scale??<br>
I didn’t expect that.</p>
<p>I made a video, it seems like it does fill up the VRAM ( look at the nvidia-smi output below )</p><div class="youtube-onebox lazy-video-container" data-video-id="Sua_2SFuuf8" data-video-title="simplescreenrecorder 2024 04 25 11 15 19" data-video-start-time="" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=Sua_2SFuuf8" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/Sua_2SFuuf8/hqdefault.jpg" title="simplescreenrecorder 2024 04 25 11 15 19" width="120" height="90">
  </a>
</div>

<p>Is there a way to offload some to the RAM?<br>
And how can I calculate what’s the max prompt it can handle with ~22GB of VRAM?</p>
<p>I’m baffled it works with a prompt of 500 lines but not with a prompt of 1500 lines. ( basically I’m just giving it an HTML page as input, it’s not that much tokens per line )</p>

ptrblck: <p>I’m not deeply familiar with the model, but you could check how the sequence length is used inside it and estimate the memory using e.g. <a href="https://dev-discuss.pytorch.org/t/how-to-measure-memory-usage-from-your-model-without-running-it/2024">this approach</a>.</p>