Gpu tesla t4 suddenly has slow processing no more that 1 % solved after reboot it

ferscp: <p>I have a vm using vmware software, the gpu are running under a vgpu system,  the os is ubuntu 22.04 and running cuda version 12.4 with nvidia driver 550.90.07, on this machine i have a gpu Tesla t4</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1b44dad82efe4bb65da428d46f13e6a44c2cf92.png" data-download-href="https://discuss.pytorch.org/uploads/default/a1b44dad82efe4bb65da428d46f13e6a44c2cf92" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1b44dad82efe4bb65da428d46f13e6a44c2cf92.png" alt="image" data-base62-sha1="n4vaBxFcXIEuSAs7mXnG2CnPFui" width="690" height="284" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">735×303 6.72 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>when I starting running a script using pytorch using cuda:0 as a device,  it runs normally, I noticed how the gpu works as expected, but after 15-20 minutes suddenly  the gpu start working really slowly and when I said slow is like using les than the 1% of the processing power, if i reboot the vm, and run the script again it start working fast and using all gpu capability, but after that period of time (15-20) it slow dramatically.</p>
<p>This happen if i run a complex model like llm llama2-7b or if i run a really simple pytorch code, next the information when it is running at 98%</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/1818c38df9106dec21d3b12224900beba35c9603.png" data-download-href="https://discuss.pytorch.org/uploads/default/1818c38df9106dec21d3b12224900beba35c9603" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/1818c38df9106dec21d3b12224900beba35c9603.png" alt="image" data-base62-sha1="3ravYeznrLjb0YvgKwuWvJhaCmT" width="690" height="323" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">736×345 11.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i need to mention that in all the cases , when running fast or slow, pytorch recognizes cuda available and is able to locale the info in its memory or the gpu</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4fcf3fb0a0af3b688bfc1bcfaf9ac7ee6e2656ab.png" data-download-href="https://discuss.pytorch.org/uploads/default/4fcf3fb0a0af3b688bfc1bcfaf9ac7ee6e2656ab" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4fcf3fb0a0af3b688bfc1bcfaf9ac7ee6e2656ab.png" alt="image" data-base62-sha1="bo1KpNJsjq8spHWJFsdA4kJluTV" width="690" height="303" data-dominant-color="090909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">740×325 7.47 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>here some data that be helpful</p>
<blockquote>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Sun_Jul_28_19:07:16_PDT_2019<br>
Cuda compilation tools, release 10.1, V10.1.243</p>
</blockquote>
<blockquote>
<p>llm-lnx1:~$ lshw -c display<br>
WARNING: you should run this program as super-user.<br>
*-display<br>
description: VGA compatible controller<br>
product: SVGA II Adapter<br>
vendor: VMware<br>
physical id: f<br>
bus info: pci@0000:00:0f.0<br>
version: 00<br>
width: 32 bits<br>
clock: 33MHz<br>
capabilities: vga_controller bus_master cap_list rom<br>
configuration: driver=vmwgfx latency=64<br>
resources: irq:16 ioport:1070(size=16) memory:e8000000-efffffff memory:fe000000-fe7fffff memory:c0000-dffff<br>
*-display<br>
description: VGA compatible controller<br>
product: TU104GL [Tesla T4]<br>
vendor: NVIDIA Corporation<br>
physical id: 1<br>
bus info: pci@0000:02:01.0<br>
version: a1<br>
width: 64 bits<br>
clock: 66MHz<br>
capabilities: vga_controller bus_master cap_list<br>
configuration: driver=nvidia latency=0<br>
resources: irq:69 memory:fc000000-fcffffff memory:d0000000-dfffffff memory:fa000000-fbffffff</p>
</blockquote>
<p>this is an example code,</p>
<pre><code class="lang-auto">import torch
import math

torch.backends.cudnn.benchmark = True
dtype = torch.float
device = torch.device("cuda:0")
torch.backends.cudnn.benchmark = True
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# Create random input and output data
x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype)
y = torch.sin(x)

# Randomly initialize weights
a = torch.randn((), device=device, dtype=dtype)
b = torch.randn((), device=device, dtype=dtype)
c = torch.randn((), device=device, dtype=dtype)
d = torch.randn((), device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(2000):
    # Forward pass: compute predicted y
    y_pred = a + b * x + c * x ** 2 + d * x ** 3

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    
    print(t, loss)

    # Backprop to compute gradients of a, b, c, d with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_a = grad_y_pred.sum()
    grad_b = (grad_y_pred * x).sum()
    grad_c = (grad_y_pred * x ** 2).sum()
    grad_d = (grad_y_pred * x ** 3).sum()

    # Update weights using gradient descent
    a -= learning_rate * grad_a
    b -= learning_rate * grad_b
    c -= learning_rate * grad_c
    d -= learning_rate * grad_d


print(f'Result: y = {a.item()} + {b.item()} x + {c.item()} x^2 + {d.item()} x^3')
</code></pre>
<p>for the 98% of the consumption i am using another experiment that is running a llm llama2 model, but in practice the problem is not the amount of processing but the slowness that happen every 15-20 minutos, i need to mention when the gpu start to be so slow it never recovery, only a sudo reboot solves the issue.</p>
<p>here my python libraries:</p>
<blockquote>
<p>Package                      Version</p>
<hr>
<p>absl-py                      2.1.0<br>
accelerate                   0.31.0<br>
aiohttp                      3.9.5<br>
aiosignal                    1.3.1<br>
annotated-types              0.7.0<br>
astunparse                   1.6.3<br>
async-timeout                4.0.3<br>
attrs                        23.2.0<br>
bitsandbytes                 0.43.1<br>
certifi                      2024.6.2<br>
charset-normalizer           3.3.2<br>
cmake                        3.29.5.1<br>
cuda-python                  12.5.0<br>
dataclasses-json             0.6.7<br>
datasets                     2.20.0<br>
dill                         0.3.8<br>
exceptiongroup               1.2.1<br>
expecttest                   0.2.1<br>
filelock                     3.15.1<br>
flatbuffers                  24.3.25<br>
frozenlist                   1.4.1<br>
fsspec                       2024.5.0<br>
gast                         0.5.4<br>
google-pasta                 0.2.0<br>
greenlet                     3.0.3<br>
grpcio                       1.64.1<br>
h5py                         3.11.0<br>
huggingface-hub              0.23.4<br>
hypothesis                   6.103.2<br>
idna                         3.7<br>
Jinja2                       3.1.4<br>
jsonpatch                    1.33<br>
jsonpointer                  3.0.0<br>
keras                        3.3.3<br>
langchain                    0.1.11<br>
langchain-community          0.0.38<br>
langchain-core               0.1.52<br>
langchain-text-splitters     0.0.2<br>
langsmith                    0.1.81<br>
libclang                     18.1.1<br>
lintrunner                   0.12.5<br>
Markdown                     3.6<br>
markdown-it-py               3.0.0<br>
MarkupSafe                   2.1.5<br>
marshmallow                  3.21.3<br>
mdurl                        0.1.2<br>
ml-dtypes                    0.3.2<br>
mpmath                       1.3.0<br>
multidict                    6.0.5<br>
multiprocess                 0.70.16<br>
mypy-extensions              1.0.0<br>
namex                        0.0.8<br>
networkx                     3.3<br>
ninja                        1.11.1.1<br>
numpy                        1.26.4<br>
nvidia-cublas-cu12           12.1.3.1<br>
nvidia-cuda-cupti-cu12       12.1.105<br>
nvidia-cuda-nvcc-cu12        12.3.107<br>
nvidia-cuda-nvrtc-cu12       12.1.105<br>
nvidia-cuda-runtime-cu12     12.1.105<br>
nvidia-cudnn-cu12            8.9.2.26<br>
nvidia-cufft-cu12            11.0.2.54<br>
nvidia-curand-cu12           10.3.2.106<br>
nvidia-cusolver-cu12         11.4.5.107<br>
nvidia-cusparse-cu12         12.1.0.106<br>
nvidia-nccl-cu12             2.20.5<br>
nvidia-nvjitlink-cu12        12.3.101<br>
nvidia-nvtx-cu12             12.1.105<br>
opt-einsum                   3.3.0<br>
optree                       0.11.0<br>
orjson                       3.10.5<br>
packaging                    23.2<br>
pandas                       2.2.2<br>
pillow                       10.2.0<br>
pip                          24.0<br>
protobuf                     4.25.3<br>
psutil                       5.9.8<br>
psycopg2-binary              2.9.9<br>
pyarrow                      16.1.0<br>
pyarrow-hotfix               0.6<br>
pydantic                     2.7.4<br>
pydantic_core                2.18.4<br>
Pygments                     2.18.0<br>
pyodbc                       5.1.0<br>
python-dateutil              2.9.0.post0<br>
pytz                         2024.1<br>
PyYAML                       6.0.1<br>
quanto                       0.2.0<br>
redis                        5.0.1<br>
regex                        2024.5.15<br>
requests                     2.32.3<br>
rich                         13.7.1<br>
safetensors                  0.4.3<br>
setuptools                   65.5.0<br>
six                          1.16.0<br>
sortedcontainers             2.4.0<br>
SQLAlchemy                   2.0.31<br>
sympy                        1.12.1<br>
tenacity                     8.4.1<br>
tensorboard                  2.16.2<br>
tensorboard-data-server      0.7.2<br>
tensorflow                   2.16.1<br>
tensorflow-io-gcs-filesystem 0.37.0<br>
termcolor                    2.4.0<br>
tokenizers                   0.19.1<br>
torch                        2.3.1+cu121<br>
torchaudio                   2.3.1+cu121<br>
torchvision                  0.18.1+cu121<br>
tqdm                         4.66.4<br>
transformers                 4.42.0.dev0<br>
triton                       2.3.1<br>
types-dataclasses            0.6.6<br>
typing_extensions            4.12.2<br>
typing-inspect               0.9.0<br>
tzdata                       2024.1<br>
urllib3                      2.2.2<br>
Werkzeug                     3.0.3<br>
wheel                        0.43.0<br>
wrapt                        1.16.0<br>
xxhash                       3.4.1<br>
yarl                         1.9.4</p>
</blockquote>
<p>any help would be really appreciated</p>