Improve GPU monitoring: switch to multiprocessing and use amdsmi for AMD GPUs
Author: Abdennacer-Badaoui
Pull request: yes

**Summary**
This PR improves GPU monitoring performance by switching from threading to multiprocessing and using the amdsmi Python library instead of rocm-smi subprocess calls for AMD GPUs.

**Changes**
- Threading → Multiprocessing: GPU monitoring now runs in a separate process instead of a thread, avoiding Python GIL contention during benchmarks
- amdsmi for AMD GPUs: Replaced `rocm-smi` subprocess calls with direct `amdsmi`Python library calls, eliminating subprocess overhead that caused significant slowdowns on AMD systems

**Tests (Example on AMD MI325)**

**Threading (old)** 

- With GPU Monitoring 

```python
====================================================================================================
Config: 3 warmup, 10 iterations, with GPU monitoring, batch size 32, sequence length 128, 32 generated tokens, flex_attention attention, compiled, not kernelized, regular generate

  E2E Latency.........: avg=1.13s       std=3.81ms    min=1.12s       med=1.13s       max=1.14s       p95=1.13s      
  Time to First Token.: avg=400.00ms    std=200.00ms  min=0.00ns      med=500.00ms    max=500.00ms    p95=500.00ms   
  Inter-Token Latency.: avg=18.75ms     std=6.25ms    min=15.62ms     med=15.62ms     max=31.25ms     p95=31.25ms    
  Throughput..........: avg=906.55tok/s std=3.06tok/s min=902.04tok/s med=906.81tok/s max=911.01tok/s p95=910.78tok/s
====================================================================================================
``` 
- Without GPU Monitoring 

```python
====================================================================================================
Config: 3 warmup, 10 iterations, no GPU monitoring, batch size 32, sequence length 128, 32 generated tokens, flex_attention attention, compiled, not kernelized, regular generate

  E2E Latency.........: avg=396.32ms     std=1.53ms    min=394.65ms     med=395.96ms     max=400.42ms     p95=398.98ms    
  Time to First Token.: avg=250.00ms     std=250.00ms  min=0.00ns       med=250.00ms     max=500.00ms     p95=500.00ms    
  Inter-Token Latency.: avg=4.69ms       std=7.16ms    min=0.00ns       med=0.00ns       max=15.62ms      p95=15.62ms     
  Throughput..........: avg=2583.81tok/s std=9.91tok/s min=2557.32tok/s med=2586.13tok/s max=2594.70tok/s p95=2593.32tok/s
====================================================================================================
```  
GPU monitoring with threading causes ~2 to 3x overhead. 

**Multiprocessing (New)** 

- With GPU Monitoring 

```python
====================================================================================================
Config: 3 warmup, 10 iterations, with GPU monitoring, batch size 32, sequence length 128, 32 generated tokens, flex_attention attention, compiled, not kernelized, regular generate

  E2E Latency.........: avg=399.23ms     std=3.05ms     min=396.52ms     med=398.16ms     max=407.46ms     p95=404.84ms    
  Time to First Token.: avg=200.00ms     std=244.95ms   min=0.00ns       med=0.00ns       max=500.00ms     p95=500.00ms    
  Inter-Token Latency.: avg=9.38ms       std=7.65ms     min=0.00ns       med=15.62ms      max=15.62ms      p95=15.62ms     
  Throughput..........: avg=2565.09tok/s std=19.35tok/s min=2513.16tok/s med=2571.86tok/s max=2582.44tok/s p95=2579.94tok/s
====================================================================================================
```

- Without GPU Monitoring 

```python 
====================================================================================================
Config: 3 warmup, 10 iterations, no GPU monitoring, batch size 32, sequence length 128, 32 generated tokens, flex_attention attention, compiled, not kernelized, regular generate

  E2E Latency.........: avg=394.16ms     std=1.16ms    min=392.65ms     med=394.03ms     max=397.20ms     p95=395.98ms    
  Time to First Token.: avg=200.00ms     std=244.95ms  min=0.00ns       med=0.00ns       max=500.00ms     p95=500.00ms    
  Inter-Token Latency.: avg=7.81ms       std=7.81ms    min=0.00ns       med=7.81ms       max=15.62ms      p95=15.62ms     
  Throughput..........: avg=2597.93tok/s std=7.63tok/s min=2578.08tok/s med=2598.80tok/s max=2607.92tok/s p95=2606.36tok/s
====================================================================================================
```

Pretty much the same :) 

Inspired from @IlyasMoutawwakil 's [optimum-benchmark memory tracker](https://github.com/huggingface/optimum-benchmark/blob/main/optimum_benchmark/trackers/memory.py).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43552). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

You might wanna test if the installation of amdsmi is possible with no issues on nvidia

> LGTM ! You can probably even reduce the sampling interval for finer measuring without impacting the main process.

Reducing it to 50 ms. Tested locally; no impact compared to 100 ms.

> You might wanna test if the installation of amdsmi is possible with no issues on nvidia

Tested it. amdsmi installs successfully on NVIDIA systems, but fails at import time (missing libamd_smi.so). This is fine since we use platform checks (is_rocm_platform()) before importing, so it's never imported on NVIDIA.