Inconsistency between GPU memory usage in torch.cuda.memory_summary and nvidia-smi
Labels: module: cuda, triaged

### 🐛 Describe the bug

I also encountered a similar problem where PyTorch reports inconsistent memory usage between vGPU memory and actual GPU memory
The value of GPU memory obtained through PyTorch is 5GB, but when checking with nvidia-smi it shows 22GB. I have tried many methods, such as upgrading both Torch and Cuda, but the bug still persists.

https://github.com/pytorch/pytorch/issues/37250

### Versions


## env
- OS:unbantu
- Python:python3.9
- Transformers:transformers==4.27.1
- PyTorch:torch==1.10.1 /  torch==2.0.0 same
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :True


```python
Python 3.9.16 (main, Dec  7 2022, 01:11:58)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
>>> gpu_id = torch.cuda.current_device()
>>> torch.cuda.set_device(0)
>>> batch_size = 1024
>>> data_shape = (3, 224, 224)
>>> tensor = torch.zeros([batch_size] + list(data_shape)).cuda(device=0)
>>> batch_size = 10240
>>> tensor = torch.zeros([batch_size] + list(data_shape)).cuda(device=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: CUDA out of memory. Tried to allocate 5.74 GiB (GPU 0; 5.50 GiB total capacity; 588.00 MiB already allocated; 4.13 GiB free; 588.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
>>>
>>> torch.set_default_tensor_type(torch.FloatTensor)
>>> total_memory = torch.cuda.max_memory_allocated()
>>> free_memory = torch.cuda.max_memory_cached()
/home/admin/langchain-ChatGLM/venv/lib/python3.9/site-packages/torch/cuda/memory.py:392: FutureWarning: torch.cuda.max_memory_cached has been renamed to torch.cuda.max_memory_reserved
  warnings.warn(
>>>
>>> print(f"Total memory: {total_memory / (1024 ** 3):.2f} GB")
Total memory: 0.57 GB
>>> print(f"Free memory: {free_memory / (1024 ** 3):.2f} GB")
Free memory: 0.57 GB

>>> print(f"status：{torch.cuda.memory_summary()}")
初始状态：|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 1            |        cudaMalloc retries: 1         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |  602124 KB |  602124 KB |  602124 KB |       0 B  |
|       from large pool |  602112 KB |  602112 KB |  602112 KB |       0 B  |
|       from small pool |      12 KB |      12 KB |      12 KB |       0 B  |
|---------------------------------------------------------------------------|
| Active memory         |  602124 KB |  602124 KB |  602124 KB |       0 B  |
|       from large pool |  602112 KB |  602112 KB |  602112 KB |       0 B  |
|       from small pool |      12 KB |      12 KB |      12 KB |       0 B  |
|---------------------------------------------------------------------------|
| GPU reserved memory   |  604160 KB |  604160 KB |  604160 KB |       0 B  |
|       from large pool |  602112 KB |  602112 KB |  602112 KB |       0 B  |
|       from small pool |    2048 KB |    2048 KB |    2048 KB |       0 B  |
|---------------------------------------------------------------------------|
| Non-releasable memory |    2035 KB |    2040 KB |    2040 KB |    4608 B  |
|       from large pool |       0 KB |       0 KB |       0 KB |       0 B  |
|       from small pool |    2035 KB |    2040 KB |    2040 KB |    4608 B  |
|---------------------------------------------------------------------------|
| Allocations           |       5    |       5    |       5    |       0    |
|       from large pool |       1    |       1    |       1    |       0    |
|       from small pool |       4    |       4    |       4    |       0    |
|---------------------------------------------------------------------------|
| Active allocs         |       5    |       5    |       5    |       0    |
|       from large pool |       1    |       1    |       1    |       0    |
|       from small pool |       4    |       4    |       4    |       0    |
|---------------------------------------------------------------------------|
| GPU reserved segments |       2    |       2    |       2    |       0    |
|       from large pool |       1    |       1    |       1    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       1    |       1    |       1    |       0    |
|       from large pool |       0    |       0    |       0    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|
```

```shell
# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Thu May 11 02:39:19 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.172.01   Driver Version: 450.172.01   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P40           Off  | 00000000:88:00.0 Off |                    0 |
| N/A   43C    P0    50W / 250W |   2924MiB / 22919MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```

cc @ngimel