Repository: wandb/wandb

Issue:
[CLI]: WARNING: No AMD GPUs specified
### Describe the bug

On a server managed by Slurm, which has both NVIDIA and AMD GPUs installed, if we only request NVIDIA GPUs we get `WARNING: No AMD GPUs specified` flooding the console. I commented in #6757 but didn't get a response so I'm opening a new issue since that one is closed.

### Additional Files

_No response_

### Environment

```console
$ conda list | grep wandb
wandb                     0.16.6                   pypi_0    pypi

$ conda list | grep python
python                    3.11.8          hab00c5b_0_cpython    conda-forge

$ lsb_release -a
Description:    Ubuntu 22.04.4 LTS
```

### Additional Context

```console
$ rocm-smi -a --json
WARNING: No AMD GPUs specified
{"system": {"Driver version": "6.3.6"}}

$ cat /sys/module/amdgpu/initstate
live
```


Discussion:
@collinmccarthy sorry that you still experience this issue, we are currently in the process of replacing our amd  system metric with a roc-smi tool (we were using the installed binary to extract the relevant information), hopefully it should be fixed in the next release. In the short-term, we will review our original fix and try to make sure that this warning is suppress. 
Awesome thank you for the quick response. If you know of any short term fixes I could apply in the meantime that would be much appreciated, otherwise I’ll wait for an update. 
@collinmccarthy the only way is to mess with your installed code, if you want to do it you can just short-circuit our check for amd, here: https://github.com/wandb/wandb/blob/main/wandb/sdk/internal/system/assets/gpu_amd.py#L167
but we are actually fixing it now and should be releasing it soon

Resolving patch:
diff --git a/wandb/sdk/internal/system/assets/gpu_amd.py b/wandb/sdk/internal/system/assets/gpu_amd.py
--- a/wandb/sdk/internal/system/assets/gpu_amd.py
+++ b/wandb/sdk/internal/system/assets/gpu_amd.py
@@ -28,14 +28,6 @@
 ROCM_SMI_CMD: Final[str] = shutil.which("rocm-smi") or "/usr/bin/rocm-smi"
 
 
-def get_rocm_smi_stats() -> Dict[str, Any]:
-    command = [str(ROCM_SMI_CMD), "-a", "--json"]
-    output = subprocess.check_output(command, universal_newlines=True).strip()
-    if "No AMD GPUs specified" in output:
-        return {}
-    return json.loads(output.split("\n")[0])  # type: ignore
-
-
 _StatsKeys = Literal[
     "gpu",
     "memoryAllocated",
@@ -49,6 +41,48 @@ def get_rocm_smi_stats() -> Dict[str, Any]:
 _InfoDict = Dict[str, Union[int, List[Dict[str, Any]]]]
 
 
+def get_rocm_smi_stats() -> Dict[str, Any]:
+    command = [str(ROCM_SMI_CMD), "-a", "--json"]
+    output = subprocess.check_output(command, universal_newlines=True).strip()
+    if "No AMD GPUs specified" in output:
+        return {}
+    return json.loads(output.split("\n")[0])  # type: ignore
+
+
+def parse_stats(stats: Dict[str, str]) -> _Stats:
+    """Parse stats from rocm-smi output."""
+    parsed_stats: _Stats = {}
+
+    try:
+        parsed_stats["gpu"] = float(stats.get("GPU use (%)"))  # type: ignore
+    except (TypeError, ValueError):
+        logger.warning("Could not parse GPU usage as float")
+    try:
+        parsed_stats["memoryAllocated"] = float(stats.get("GPU memory use (%)"))  # type: ignore
+    except (TypeError, ValueError):
+        logger.warning("Could not parse GPU memory allocation as float")
+    try:
+        parsed_stats["temp"] = float(stats.get("Temperature (Sensor memory) (C)"))  # type: ignore
+    except (TypeError, ValueError):
+        logger.warning("Could not parse GPU temperature as float")
+    try:
+        parsed_stats["powerWatts"] = float(
+            stats.get("Average Graphics Package Power (W)")  # type: ignore
+        )
+    except (TypeError, ValueError):
+        logger.warning("Could not parse GPU power as float")
+    try:
+        parsed_stats["powerPercent"] = (
+            float(stats.get("Average Graphics Package Power (W)"))  # type: ignore
+            / float(stats.get("Max Graphics Package Power (W)"))  # type: ignore
+            * 100
+        )
+    except (TypeError, ValueError):
+        logger.warning("Could not parse GPU average/max power as float")
+
+    return parsed_stats
+
+
 class GPUAMDStats:
     """Stats for AMD GPU devices."""
 
@@ -58,40 +92,6 @@ class GPUAMDStats:
     def __init__(self) -> None:
         self.samples = deque()
 
-    @staticmethod
-    def parse_stats(stats: Dict[str, str]) -> _Stats:
-        """Parse stats from rocm-smi output."""
-        parsed_stats: _Stats = {}
-
-        try:
-            parsed_stats["gpu"] = float(stats.get("GPU use (%)"))  # type: ignore
-        except (TypeError, ValueError):
-            logger.warning("Could not parse GPU usage as float")
-        try:
-            parsed_stats["memoryAllocated"] = float(stats.get("GPU memory use (%)"))  # type: ignore
-        except (TypeError, ValueError):
-            logger.warning("Could not parse GPU memory allocation as float")
-        try:
-            parsed_stats["temp"] = float(stats.get("Temperature (Sensor memory) (C)"))  # type: ignore
-        except (TypeError, ValueError):
-            logger.warning("Could not parse GPU temperature as float")
-        try:
-            parsed_stats["powerWatts"] = float(
-                stats.get("Average Graphics Package Power (W)")  # type: ignore
-            )
-        except (TypeError, ValueError):
-            logger.warning("Could not parse GPU power as float")
-        try:
-            parsed_stats["powerPercent"] = (
-                float(stats.get("Average Graphics Package Power (W)"))  # type: ignore
-                / float(stats.get("Max Graphics Package Power (W)"))  # type: ignore
-                * 100
-            )
-        except (TypeError, ValueError):
-            logger.warning("Could not parse GPU average/max power as float")
-
-        return parsed_stats
-
     def sample(self) -> None:
         try:
             raw_stats = get_rocm_smi_stats()
@@ -103,7 +103,7 @@ def sample(self) -> None:
 
             for card_key in card_keys:
                 card_stats = raw_stats[card_key]
-                stats = self.parse_stats(card_stats)
+                stats = parse_stats(card_stats)
                 if stats:
                     cards.append(stats)
 
@@ -183,7 +183,7 @@ def is_available(cls) -> bool:
 
         can_read_rocm_smi = False
         try:
-            if get_rocm_smi_stats():
+            if parse_stats(get_rocm_smi_stats()):
                 can_read_rocm_smi = True
         except Exception:
             pass


Test patch: