OOM during backward() leads to memory leaks
Labels: needs reproduction, module: autograd, module: memory usage, triaged

### 🐛 Describe the bug

Like what is shown in the logs below, I print the currently allocated memory by torch.cuda.memory_allocated() before the forward pass in each step. After the OOM in the backward() call, the intermediate tensors are not freed, which will lead to OOM in the following steps. 

OOM in the forward passes will not lead to the issue.

```
[INFO 2022-07-26 14:40:33,673] [Step 10115] 0.501 sec/step (0.555), lr=1.6427E-04, loss=0.15372 (Ave. 0.32287), grad_norm=0.747
[INFO 2022-07-26 14:40:33,675] Current memory: 6413632512
[INFO 2022-07-26 14:40:34,251] [Step 10116] 0.577 sec/step (0.552), lr=1.6426E-04, loss=0.17586 (Ave. 0.32030), grad_norm=1.508
[INFO 2022-07-26 14:40:34,254] Current memory: 6424706560
[ERROR 2022-07-26 14:40:34,609] Failed due to RuntimeError, input shape: torch.Size([28, 69920]), target shape: torch.Size([28, 82])
Traceback (most recent call last):
  File "train.py", line 210, in main
    losses['loss'].backward()
  File "/idiap/user/mhe/miniconda3/lib/python3.9/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/idiap/user/mhe/miniconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 23.70 GiB total capacity; 20.20 GiB already allocated; 1.69 MiB free; 22.09 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[INFO 2022-07-26 14:40:34,829] Current memory: 21698449408
```

I've checked all the pytorch tensors with gc.get_objects() and confirmed that there are hundreds of tensors alive after the OOM that don't exist prior to the backward pass, indicating that they are created during the backward() call. Calling gc.collect() will not release them. sys.getrefcount(t) on one of these tensors gives like 4, but gc.get_referrers(t) gives nothing but t itself, so there should be 2 references from untracked objects like something in C extensions.

I attempted to reproduce it with some simpler codes, but they will just work fine. Hopefully we can find some logs somewhere to better locate the error.

Hundreds of following warnings are shown when I stop the program, unknown if relevant:
```
[W python_variable.cpp:200] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn)
```


### Versions

PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.10.0-16-amd64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.2.152
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 510.39.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.0
[pip3] torchaudio==0.12.0
[pip3] torchvision==0.13.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] numpy-base                1.21.5           py39hf524024_1
[conda] pytorch                   1.12.0          py3.9_cuda11.3_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.11.0                   pypi_0    pypi
[conda] torchvision               0.13.0               py39_cu113    pytorch

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7