out of memory with pytorch version after 1.8.1
Labels: module: cuda, module: memory usage, triaged, module: CUDACachingAllocator

### 🐛 Describe the bug

I realize that my code can only run with pytorch1.8.1
pytorch1.11 pytorch1.13 and pytorch1.8.2 all give OOM errors.

The code is pretty complicated so I cannot provide a minimal example.

On pytorch1.8.1, my code can run on RTX 2080ti (11G) with no problem.
But with pytorch1.11 pytorch1.13 and pytorch1.8.2, even with RTX 3090 (24G) I get OOM errors:

CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 23.70 GiB total capacity; 1.21 GiB already allocated; 6.69 MiB free; 1.22 GiB reserved in total by PyTorch)

The error is also weird since there is around 22G memory missing

pytorch1.11 pytorch1.13 and pytorch1.8.2 trigger the OOM error at different lines

Below is the memory summary right before the line that triggers the OOM errors with pytorch1.8.2 with CUDA_LAUNCH_BLOCKING = 1
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = True
 
![Screenshot from 2022-10-31 23-07-55](https://user-images.githubusercontent.com/16913207/199004477-8107fa28-93e2-4ab4-b481-c2f85d9708a6.png)


### Versions

Collecting environment information...
PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.17.0-rc1
Libc version: glibc-2.10

Python version: 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-131-generic-x86_64-with-debian-bullseye-sid
Is CUDA available: True
CUDA runtime version: 11.1.105
CUDA_MODULE_LOADING set to: 
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 2080 Ti
GPU 1: NVIDIA GeForce RTX 2080 Ti

Nvidia driver version: 470.141.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.8.1
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==2.0.1
[pip3] torch-scatter==2.0.8
[pip3] torch-sparse==0.6.12
[pip3] torch-spline-conv==1.2.1
[pip3] torchaudio==0.8.0a0+e4e171a
[pip3] torchmetrics==0.8.2
[pip3] torchvision==0.9.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              h8f6ccaa_8    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py37h8f50634_2    conda-forge
[conda] mkl_fft                   1.3.0            py37h54f3939_0  
[conda] mkl_random                1.2.0            py37h9fdb41a_1    conda-forge
[conda] numpy                     1.19.2           py37h54aff64_0  
[conda] numpy-base                1.19.2           py37hfa32c7d_0  
[conda] pyg                       2.0.1           py37_torch_1.8.0_cu102    pyg
[conda] pytorch                   1.8.1           py3.7_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-cluster           1.5.9           py37_torch_1.8.0_cu102    pyg
[conda] pytorch-scatter           2.0.8           py37_torch_1.8.0_cu102    pyg
[conda] pytorch-sparse            0.6.12          py37_torch_1.8.0_cu102    pyg
[conda] pytorch-spline-conv       1.2.1           py37_torch_1.8.0_cu102    pyg
[conda] torchaudio                0.8.1                      py37    pytorch
[conda] torchmetrics              0.8.2              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.9.1                py37_cu102    pytorch

---------------------------------------------------------------------------------------

Collecting environment information...
PyTorch version: 1.8.2
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.13 (default, Oct 21 2022, 23:50:54)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.15.0-194-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090

Nvidia driver version: 520.61.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.8.2
[pip3] torchaudio==0.8.2
[pip3] torchvision==0.9.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h7f8727e_0  
[conda] mkl_fft                   1.3.1            py38hd3c417c_0  
[conda] mkl_random                1.2.2            py38h51133e4_0  
[conda] numpy                     1.23.3           py38h14f4228_0  
[conda] numpy-base                1.23.3           py38h31eccc5_0  
[conda] pytorch                   1.8.2           py3.8_cuda11.1_cudnn8.0.5_0    pytorch-lts
[conda] torchaudio                0.8.2                      py38    pytorch-lts
[conda] torchvision               0.9.2                py38_cu111    pytorch-lts

cc @ngimel