Potential memory leak in Adam optimizer in AMD chips (CPU)
Labels: needs reproduction, module: optimizer, module: rocm, module: memory usage, triaged

### 🐛 Describe the bug

Adam optimizer seems to be leaking memory(?) in AMD chips. 

A minimal example doing some simple training can be found here:
https://gist.github.com/lingchunkai/dbb0c001a2fca1273487d69ac04c2f7e

After 5 million iterations, virtual memory usage rises to 70804240 (kb) and resident memory to 54691068 (kb). When using CUDA on the same machine, no such growth is seen and we use around 2GB of GPU memory and roughly 14617576 and 4707448 memory of virtual and resident memory. Both virtual and resident memory are seen to increase seemingly without bounds. I have done some simple testing and it seems RMSProp suffers from the same issue. 

I am using an AMD Ryzen Threadripper 1950X 16-Core Processor, with python 3.9.7 and pytorch version 1.10.1. Using the same python and pytorch version and running on an Intel machine does not yield the same problem. I have also tested on my Mac which has a M1 chip, and the memory usage seems to be stable.


### Versions

PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-39-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080
Nvidia driver version: 495.29.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.3
[pip3] torch==1.10.1
[pip3] torch-scatter==2.0.9
[pip3] torchaudio==0.10.1
[pip3] torchvision==0.11.2
[conda] _pytorch_select           0.1                       cpu_0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.21.3                   pypi_0    pypi
[conda] numpy-base                1.21.2           py39h79a1101_0
[conda] pytorch                   1.10.1          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-scatter           2.0.9           py39_torch_1.10.0_cu115    pyg
[conda] torchaudio                0.10.1               py39_cu113    pytorch
[conda] torchvision               0.11.2               py39_cu113    pytorch

cc @vincentqb @jbschlosser @albanD @jeffdaily @sunway513 @jithunnair-amd @ROCmSupport @KyleCZH