Repository: wandb/wandb

Issue:
[CLI]: WARNING: No AMD GPUs specified

![Screenshot 2023-12-21 at 12 10 03](https://github.com/wandb/wandb/assets/149976183/a2154368-24b5-4a55-bb1e-384edc51ba50)
### Describe the bug

<!--- Description of the issue below  -->
I have this weird WARNING: No AMD GPUs specified whenever I log using wandb on my cluster. The cluster has NVIDIA Tesla V100 32GB GPU. This warning keeps printing for every batch  and hence slows the code. Do you know how can I fix this?

<!--- A minimal code snippet between the quotes below  -->
```python

```

<!--- A full traceback of the exception in the quotes below -->
```shell
![image](https://github.com/wandb/wandb/assets/149976183/45d7a14b-1b86-4c80-832b-a2f75beff85b)

```


### Additional Files

_No response_

### Environment

WandB version: wandb==0.16.1

OS:Ubuntu 20

Python version: 3.8.18

Versions of relevant libraries:
absl-py==2.0.0
antlr4-python3-runtime==4.9.3
appdirs==1.4.4
av==11.0.0
cachetools==5.3.2
certifi==2023.11.17
charset-normalizer==3.3.2
click==8.1.7
contourpy==1.1.1
cycler==0.12.1
docker-pycreds==0.4.0
ffmpeg-python==0.2.0
filelock==3.13.1
fonttools==4.46.0
fsspec==2023.12.1
future==0.18.3
fvcore==0.1.5.post20221221
gitdb==4.0.11
gitpython==3.1.40
google-auth-oauthlib==1.0.0
google-auth==2.23.4
grpcio==1.59.3
h5py==3.10.0
huggingface-hub==0.19.4
idna==3.4
imageio==2.33.0
importlib-metadata==6.8.0
importlib-resources==6.1.1
info-nce-pytorch==0.1.4
iopath==0.1.10
jinja2==3.1.2
joblib==1.3.2
kiwisolver==1.4.5
kornia==0.7.0
lightning-utilities==0.10.0
lpips==0.1.4
markdown==3.5.1
markupsafe==2.1.3
matplotlib==3.7.4
mkl-fft==1.3.8
mkl-random==1.2.4
mkl-service==2.4.0
mpmath==1.3.0
networkx==3.1
numpy==1.22.0
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.18.1
nvidia-nvjitlink-cu12==12.3.101
nvidia-nvtx-cu12==12.1.105
oauthlib==3.2.2
omegaconf==2.3.0
opencv-python==4.5.4.58
packaging==21.3
pandas==2.0.3
parameterized==0.9.0
pillow==9.3.0
pims==0.6.1
pip==23.3.1
portalocker==2.8.2
protobuf==4.25.1
psutil==5.9.6
pyasn1-modules==0.3.0
pyasn1==0.5.0
pyparsing==3.1.1
python-dateutil==2.8.2
python-hostlist==1.21
pytorchvideo==0.1.5
pytz==2023.3.post1
pyyaml==6.0.1
regex==2023.10.3
requests-oauthlib==1.3.1
requests==2.31.0
rsa==4.9
sacremoses==0.1.1
scikit-learn==1.3.2
scipy==1.10.1
sentry-sdk==1.35.0
setproctitle==1.3.3
setuptools==68.0.0
six==1.16.0
slicerator==1.1.0
smmap==5.0.1
sympy==1.12
tabulate==0.9.0
tensorboard-data-server==0.7.2
tensorboard==2.14.0
termcolor==2.3.0
threadpoolctl==3.2.0
timm==0.4.12
tokenizers==0.10.3
torch==1.8.1
torchaudio==0.8.0a0+e4e171a
torchmetrics==1.2.1
torchvideo==0.0.0
torchvision==0.9.1
tqdm==4.50.2
transformers==4.10.2
triton==2.1.0
typing-extensions==4.7.1
tzdata==2023.3
urllib3==2.1.0
wandb==0.16.1
werkzeug==3.0.1
wheel==0.41.2
yacs==0.1.8
zipp==3.17.0


### Additional Context

_No response_


Discussion:
Hi @mdivyanshu97, thank you for writing in, are you using any external library for your experiments such as pytorch or tensorflow?
Thank you for the quick response. Yes I am using pytorch for my experiments.
Thank you for your quick response, could you also please let me know what operating system you are using?

Trying to reproduce this behavior on my side. 

In the meantime, could you see if setting `WANDB_SILENT=True`, fixes your issue? We have docs on our env vars [here](https://docs.wandb.ai/guides/track/environment-variables#optional-environment-variables)   
Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

Hi, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

hey @mdivyanshu97, do you have `rocm-smi` installed on your cluster by any chance? we use that to monitor AMD GPU metrics, but since you have NVIDIA GPUs, as a quick workaround could you try uninstalling that tool?
Hey, 

> Thank you for your quick response, could you also please let me know what operating system you are using?
> 
> Trying to reproduce this behavior on my side.
> 
> In the meantime, could you see if setting `WANDB_SILENT=True`, fixes your issue? We have docs on our env vars [here](https://docs.wandb.ai/guides/track/environment-variables#optional-environment-variables)

Hey, sorry for the late reply. I tried WANDB_SILENT but it doesn't fix the issue.
The issue only occurs on the cluster which uses the following OS.
 Operating System: CentOS Linux 7 (Core)
      
         
> hey @mdivyanshu97, do you have `rocm-smi` installed on your cluster by any chance? we use that to monitor AMD GPU metrics, but since you have NVIDIA GPUs, as a quick workaround could you try uninstalling that tool?

Hey thank you for help. I will ask the IT department if they have any idea as I do not have sudo privileges on the cluster to uninstall this.
I also have this issue.
@gil2rok do you have rocm-smi installed on your machine? 
@ArtsiomWB sorry for the late reply. Yes I do have `rocm-smi` installed on my machine.

Any advice on what to do here?

Also, setting `WANDB_SILENT` to `True` does not fix this problem.

Resolving patch:
diff --git a/wandb/sdk/internal/system/assets/gpu_amd.py b/wandb/sdk/internal/system/assets/gpu_amd.py
--- a/wandb/sdk/internal/system/assets/gpu_amd.py
+++ b/wandb/sdk/internal/system/assets/gpu_amd.py
@@ -30,10 +30,10 @@
 
 def get_rocm_smi_stats() -> Dict[str, Any]:
     command = [str(ROCM_SMI_CMD), "-a", "--json"]
-    output = (
-        subprocess.check_output(command, universal_newlines=True).strip().split("\n")
-    )[0]
-    return json.loads(output)  # type: ignore
+    output = subprocess.check_output(command, universal_newlines=True).strip()
+    if "No AMD GPUs specified" in output:
+        return {}
+    return json.loads(output.split("\n")[0])  # type: ignore
 
 
 _StatsKeys = Literal[
@@ -166,13 +166,29 @@ def __init__(
     @classmethod
     def is_available(cls) -> bool:
         rocm_smi_available = shutil.which(ROCM_SMI_CMD) is not None
-        if rocm_smi_available:
-            try:
-                _ = get_rocm_smi_stats()
-                return True
-            except Exception:
-                pass
-        return False
+        if not rocm_smi_available:
+            # If rocm-smi is not available, we can't monitor AMD GPUs
+            return False
+
+        is_driver_initialized = False
+
+        try:
+            # inspired by https://github.com/ROCm/rocm_smi_lib/blob/5d2cd0c2715ae45b8f9cfe1e777c6c2cd52fb601/python_smi_tools/rocm_smi.py#L71C1-L81C17
+            with open("/sys/module/amdgpu/initstate") as file:
+                file_content = file.read()
+                if "live" in file_content:
+                    is_driver_initialized = True
+        except FileNotFoundError:
+            pass
+
+        can_read_rocm_smi = False
+        try:
+            if get_rocm_smi_stats():
+                can_read_rocm_smi = True
+        except Exception:
+            pass
+
+        return is_driver_initialized and can_read_rocm_smi
 
     def start(self) -> None:
         self.metrics_monitor.start()


Test patch:
diff --git a/tests/pytest_tests/unit_tests/test_system_metrics/test_gpu_amd.py b/tests/pytest_tests/unit_tests/test_system_metrics/test_gpu_amd.py
--- a/tests/pytest_tests/unit_tests/test_system_metrics/test_gpu_amd.py
+++ b/tests/pytest_tests/unit_tests/test_system_metrics/test_gpu_amd.py
@@ -115,6 +115,7 @@ def test_gpu_amd(test_settings):
             settings=settings,
             shutdown_event=shutdown_event,
         )
+        gpu.is_available = lambda: True
         assert gpu.is_available()
         gpu.start()
         probe = gpu.probe()
@@ -164,6 +165,7 @@ def test_gpu_amd_missing_keys(test_settings):
             settings=settings,
             shutdown_event=shutdown_event,
         )
+        gpu.is_available = lambda: True
         gpu.start()
         time.sleep(1)
         shutdown_event.set()