NVMLError_NotSupported when creating Trainer() object.
Author: singhal-shagun
Pull request: no

### System Info

No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.41.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.11.3
- Huggingface_hub version: 0.23.1
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

Hello @muellerzr, @SunMarc!

I am trying to finetune the "flax-community/clip-rsicd-v2" model on my local computer for the DLRSD dataset. I am using `CLIPModel()` from HuggingFace's `transformers` library for my purpose. My jupyter notebook [is here](https://github.com/github-user-en/RandomCode/blob/033c1c8ca869a43d8e61bca15498a65109f0df31/CLIP-DLRSD_2_ModelTraining.ipynb).

As you can see, when creating the `Trainer()` object in cell 16, I get the error: `NVMLError_NotSupported: Not Supported`. And this happens irrespective of whether I use `CUDA:1` or `CPU` as my device for my model. What am I doing incorrectly?

The complete error log is as follows:
```py
[codecarbon INFO @ 18:26:07] [setup] RAM Tracking...
[codecarbon INFO @ 18:26:07] [setup] GPU Tracking...
[codecarbon INFO @ 18:26:07] Tracking Nvidia GPU via pynvml
---------------------------------------------------------------------------
NVMLError_NotSupported                    Traceback (most recent call last)
Cell In[17], [line 1](vscode-notebook-cell:?execution_count=17&line=1)
----> [1](vscode-notebook-cell:?execution_count=17&line=1) trainer = Trainer(
      [2](vscode-notebook-cell:?execution_count=17&line=2)     model=model,
      [3](vscode-notebook-cell:?execution_count=17&line=3)     args=training_args,
      [4](vscode-notebook-cell:?execution_count=17&line=4)     train_dataset=trainingDataset,
      [5](vscode-notebook-cell:?execution_count=17&line=5)     eval_dataset=validationDataset,
      [6](vscode-notebook-cell:?execution_count=17&line=6)     # tokenizer=tokenizer,
      [7](vscode-notebook-cell:?execution_count=17&line=7)     # data_collator=data_collator,
      [8](vscode-notebook-cell:?execution_count=17&line=8)     # compute_metrics=compute_metrics,
      [9](vscode-notebook-cell:?execution_count=17&line=9) )

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\transformers\trainer.py:678](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer.py:678), in Trainer.__init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics)
    [676](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer.py:676) self.label_names = default_label_names if self.args.label_names is None else self.args.label_names
    [677](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer.py:677) self.can_return_loss = can_return_loss(self.model.__class__)
--> [678](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer.py:678) self.control = self.callback_handler.on_init_end(self.args, self.state, self.control)
    [680](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer.py:680) # Internal variables to help with automatic batch size reduction
    [681](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer.py:681) self._train_batch_size = args.train_batch_size

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\transformers\trainer_callback.py:450](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:450), in CallbackHandler.on_init_end(self, args, state, control)
    [449](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:449) def on_init_end(self, args: TrainingArguments, state: TrainerState, control: TrainerControl):
--> [450](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:450)     return self.call_event("on_init_end", args, state, control)

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\transformers\trainer_callback.py:498](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:498), in CallbackHandler.call_event(self, event, args, state, control, **kwargs)
    [496](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:496) def call_event(self, event, args, state, control, **kwargs):
    [497](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:497)     for callback in self.callbacks:
--> [498](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:498)         result = getattr(callback, event)(
    [499](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:499)             args,
    [500](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:500)             state,
    [501](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:501)             control,
    [502](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:502)             model=self.model,
    [503](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:503)             tokenizer=self.tokenizer,
    [504](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:504)             optimizer=self.optimizer,
    [505](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:505)             lr_scheduler=self.lr_scheduler,
    [506](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:506)             train_dataloader=self.train_dataloader,
    [507](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:507)             eval_dataloader=self.eval_dataloader,
    [508](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:508)             **kwargs,
    [509](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:509)         )
    [510](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:510)         # A Callback can skip the return of `control` if it doesn't change it.
    [511](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/trainer_callback.py:511)         if result is not None:

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\transformers\integrations\integration_utils.py:1475](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/integrations/integration_utils.py:1475), in CodeCarbonCallback.on_init_end(self, args, state, control, **kwargs)
   [1472](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/integrations/integration_utils.py:1472) def on_init_end(self, args, state, control, **kwargs):
   [1473](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/integrations/integration_utils.py:1473)     if self.tracker is None and state.is_local_process_zero:
   [1474](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/integrations/integration_utils.py:1474)         # CodeCarbon will automatically handle environment variables for configuration
-> [1475](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/transformers/integrations/integration_utils.py:1475)         self.tracker = self._codecarbon.EmissionsTracker(output_dir=args.output_dir)

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\codecarbon\emissions_tracker.py:285](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/emissions_tracker.py:285), in BaseEmissionsTracker.__init__(self, project_name, measure_power_secs, api_call_interval, api_endpoint, api_key, output_dir, output_file, save_to_file, save_to_api, save_to_logger, logging_logger, save_to_prometheus, prometheus_url, gpu_ids, emissions_endpoint, experiment_id, experiment_name, co2_signal_api_token, tracking_mode, log_level, on_csv_write, logger_preamble, default_cpu_power, pue)
    [283](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/emissions_tracker.py:283) if gpu.is_gpu_details_available():
    [284](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/emissions_tracker.py:284)     logger.info("Tracking Nvidia GPU via pynvml")
--> [285](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/emissions_tracker.py:285)     gpu_devices = GPU.from_utils(self._gpu_ids)
    [286](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/emissions_tracker.py:286)     self._hardware.append(gpu_devices)
    [287](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/emissions_tracker.py:287)     gpu_names = [n["name"] for n in gpu_devices.devices.get_gpu_static_info()]

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\codecarbon\external\hardware.py:121](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:121), in GPU.from_utils(cls, gpu_ids)
    [119](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:119) @classmethod
    [120](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:120) def from_utils(cls, gpu_ids: Optional[List] = None) -> "GPU":
--> [121](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:121)     return cls(gpu_ids=gpu_ids)

File <string>:4, in __init__(self, gpu_ids)

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\codecarbon\external\hardware.py:63](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:63), in GPU.__post_init__(self)
     [62](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:62) def __post_init__(self):
---> [63](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:63)     self.devices = AllGPUDevices()
     [64](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:64)     self.num_gpus = self.devices.device_count
     [65](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:65)     self._total_power = Power(
     [66](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:66)         0  # It will be 0 until we call for the first time measure_power_and_energy
     [67](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/external/hardware.py:67)     )

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\codecarbon\core\gpu.py:186](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:186), in AllGPUDevices.__init__(self)
    [184](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:184) for i in range(self.device_count):
    [185](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:185)     handle = pynvml.nvmlDeviceGetHandleByIndex(i)
--> [186](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:186)     gpu_device = GPUDevice(handle=handle, gpu_index=i)
    [187](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:187)     self.devices.append(gpu_device)

File <string>:8, in __init__(self, handle, gpu_index, energy_delta, power, last_energy)

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\codecarbon\core\gpu.py:24](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:24), in GPUDevice.__post_init__(self)
     [23](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:23) def __post_init__(self):
---> [24](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:24)     self.last_energy = self._get_energy_kwh()
     [25](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:25)     self._init_static_details()

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\codecarbon\core\gpu.py:28](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:28), in GPUDevice._get_energy_kwh(self)
     [27](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:27) def _get_energy_kwh(self):
---> [28](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:28)     return Energy.from_millijoules(self._get_total_energy_consumption())

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\codecarbon\core\gpu.py:95](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:95), in GPUDevice._get_total_energy_consumption(self)
     [91](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:91) def _get_total_energy_consumption(self):
     [92](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:92)     """Returns total energy consumption for this GPU in millijoules (mJ) since the driver was last reloaded
     [93](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:93)     https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html#group__nvmlDeviceQueries_1g732ab899b5bd18ac4bfb93c02de4900a
     [94](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:94)     """
---> [95](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/codecarbon/core/gpu.py:95)     return pynvml.nvmlDeviceGetTotalEnergyConsumption(self.handle)

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\pynvml\nvml.py:2411](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:2411), in nvmlDeviceGetTotalEnergyConsumption(handle)
   [2409](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:2409) fn = _nvmlGetFunctionPointer("nvmlDeviceGetTotalEnergyConsumption")
   [2410](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:2410) ret = fn(handle, byref(c_millijoules))
-> [2411](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:2411) _nvmlCheckReturn(ret)
   [2412](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:2412) return c_millijoules.value

File [d:\Tutorials\Python\AI\venv\Lib\site-packages\pynvml\nvml.py:833](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:833), in _nvmlCheckReturn(ret)
    [831](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:831) def _nvmlCheckReturn(ret):
    [832](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:832)     if (ret != NVML_SUCCESS):
--> [833](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:833)         raise NVMLError(ret)
    [834](file:///D:/Tutorials/Python/AI/venv/Lib/site-packages/pynvml/nvml.py:834)     return ret

NVMLError_NotSupported: Not Supported
```

For reference, I run two NVIDIA Quadro K1200 GPUs with CUDA v12 in my local machine. 
```py
PS C:\Users\admin> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:30:10_Pacific_Daylight_Time_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
```
The `nvidia-smi` output is as follows:
```py
PS C:\Windows\System32> nvidia-smi
Thu Jun 20 18:22:50 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 528.89       Driver Version: 528.89       CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro K1200       WDDM  | 00000000:01:00.0  On |                  N/A |
| 52%   65C    P0     3W /  35W |   3819MiB /  4096MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Quadro K1200       WDDM  | 00000000:03:00.0 Off |                  N/A |
| 42%   42C    P8     1W /  35W |     60MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       828    C+G   ...5n1h2txyewy\SearchApp.exe    N/A      |
|    0   N/A  N/A      1336    C+G   C:\Windows\System32\dwm.exe     N/A      |
|    0   N/A  N/A      1928    C+G   ...y\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A      2104    C+G   C:\Windows\explorer.exe         N/A      |
|    0   N/A  N/A      3440    C+G   ...lPanel\SystemSettings.exe    N/A      |
|    0   N/A  N/A      8368    C+G   ...icrosoft VS Code\Code.exe    N/A      |
|    0   N/A  N/A     10732    C+G   ...werToys.ColorPickerUI.exe    N/A      |
|    0   N/A  N/A     10788    C+G   ...\PowerToys.FancyZones.exe    N/A      |
|    0   N/A  N/A     10856    C+G   ...werToys.PowerLauncher.exe    N/A      |
|    0   N/A  N/A     11120    C+G   ...2txyewy\TextInputHost.exe    N/A      |
|    0   N/A  N/A     11188    C+G   ...e\PhoneExperienceHost.exe    N/A      |
|    0   N/A  N/A     11904    C+G   ... Download Manager\fdm.exe    N/A      |
|    0   N/A  N/A     12216    C+G   ...anPlus\MonitorianPlus.exe    N/A      |
|    0   N/A  N/A     12476    C+G   ...fice\Office16\OUTLOOK.EXE    N/A      |
|    0   N/A  N/A     13120      C   ...iles\Python311\python.exe    N/A      |
|    0   N/A  N/A     14352    C+G   ...86)\TickTick\TickTick.exe    N/A      |
|    0   N/A  N/A     14652    C+G   ...gram Desktop\Telegram.exe    N/A      |
|    0   N/A  N/A     15416    C+G   ...ams\obsidian\Obsidian.exe    N/A      |
|    0   N/A  N/A     19052    C+G   ...cw5n1h2txyewy\LockApp.exe    N/A      |
|    0   N/A  N/A     21992    C+G   ...3d8bbwe\CalculatorApp.exe    N/A      |
|    0   N/A  N/A     22620    C+G   ...me\Application\chrome.exe    N/A      |
|    0   N/A  N/A     33620    C+G   ...visor\MuMuVMMHeadless.exe    N/A      |
|    0   N/A  N/A     35956    C+G   ...ram Files\Opera\opera.exe    N/A      |
|    0   N/A  N/A     44400    C+G   ...Office\Office16\EXCEL.EXE    N/A      |
|    0   N/A  N/A     48064    C+G   ...ge\Application\msedge.exe    N/A      |
|    0   N/A  N/A     48840      C   ...iles\Python311\python.exe    N/A      |
|    0   N/A  N/A     49772    C+G   ...8bbwe\WindowsTerminal.exe    N/A      |
|    1   N/A  N/A      1336    C+G   C:\Windows\System32\dwm.exe     N/A      |
|    1   N/A  N/A     16976    C+G   ...12.0\shell\MuMuPlayer.exe    N/A      |
|    1   N/A  N/A     32012    C+G   ...Office\Office16\EXCEL.EXE    N/A      |
+-----------------------------------------------------------------------------+
```
While `Trainer()` results in this error, I am able to use the model in inference mode with my GPU. 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://github.com/github-user-en/RandomCode/blob/033c1c8ca869a43d8e61bca15498a65109f0df31/CLIP-DLRSD_2_ModelTraining.ipynb

### Expected behavior
At some level. I think `Trainer()` API doesn't expect the NVML library to raise this error on Quadro K1200 since it's a fully supported GPU as per [NVML API Documentation](https://docs.nvidia.com/deploy/nvml-api/nvml-api-reference.html#nvml-api-reference). However, even if this error is thrown, I think the `Trainer()` constructor should return an object successfully in cell 16 of the linked notebook. Not being able to get the total energy consumption information should, at max, result in an error when energy consumption information is sought.

Comments:
Looks like it stems from `codecarbon`. do `pip uninstall codecarbon -y` and you can get further along!

Thanks. Uninstalling `codecarbon` helped.