distributed.scatter memory leak in source rank
Labels: oncall: distributed, module: memory usage

### 🐛 Describe the bug

distributed scatter doesn't free memory on the source rank. The allocated and reserved memory values are correct but leaves less free memory on the source rank device.

```python
shape = (1_000_000_000,)

chunk = torch.empty(shape, device=rank)
if rank == 0:
    torch.distributed.scatter(chunk,  [torch.empty(shape, device=rank) for _ in range(world_size)], src=rank)
else:
    torch.distributed.scatter(chunk, src=0)

torch.cuda.synchronize()
gc.collect()
torch.cuda.empty_cache()

print(f'{rank}:'
      f' Free: {torch.cuda.mem_get_info(device)[0] // 1_000_000 :,}MB,'
      f' Reserved: {torch.cuda.memory_reserved(device) // 1_000_000 :,}MB,'
      f' Alloc: {torch.cuda.memory_allocated() // 1_000_000 :,}MB'
      )
```

### Versions

```
Collecting environment information...
PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.31

Python version: 3.10.11 (main, Apr 20 2023, 19:02:41) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version: 525.85.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          240
On-line CPU(s) list:             0-239
Thread(s) per core:              1
Core(s) per socket:              1
Socket(s):                       240
NUMA node(s):                    2
Vendor ID:                       AuthenticAMD
CPU family:                      25
Model:                           1
Model name:                      AMD EPYC 7J13 64-Core Processor
Stepping:                        1
CPU MHz:                         2449.998
BogoMIPS:                        4899.99
Virtualization:                  AMD-V
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       15 MiB
L1i cache:                       15 MiB
L2 cache:                        120 MiB
L3 cache:                        3.8 GiB
NUMA node0 CPU(s):               0-119
NUMA node1 CPU(s):               120-239
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip pku ospke vaes vpclmulqdq rdpid fsrm arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchdata==0.6.1
[pip3] torchtext==0.15.2
[pip3] torchvision==0.15.2
[pip3] triton==2.0.0
[conda] blas                      1.0                         mkl
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2023.1.0         h6d00ec8_46342
[conda] mkl-service               2.4.0           py310h5eee18b_1
[conda] mkl_fft                   1.3.6           py310h1128e8f_1
[conda] mkl_random                1.2.2           py310h1128e8f_1
[conda] numpy                     1.24.3          py310h5f9d8c6_1
[conda] numpy-base                1.24.3          py310hb5e798b_1
[conda] pytorch-cuda              11.8                 h7e8668a_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torchaudio                2.0.2               py310_cu118    pytorch
[conda] torchdata                 0.6.1                     py310    pytorch
[conda] torchtext                 0.15.2                    py310    pytorch
[conda] torchvision               0.15.2              py310_cu118    pytorch
[conda] triton                    2.0.0                    pypi_0    pypi
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu