PR #89436 looks like it causes or enables a memory leak
Labels: module: memory usage, triaged, module: mps

### 🐛 Describe the bug

Running  InvokeAI using a pytorch built from master with PR #89436 applied uses way more memory than without the PR applied and that memory does not get released after image generation has completed, it looks a lot like a memory leak.

initial Run without the PR the Python executable uses between 8 to 12 Gb which ebbs and flows thoughout the process and releases memory down to ~7Gb afterward image generation.

With the PR, there is a similar pattern during the sampling phase, but post sampling, the memory usage start to consistently increase to just under 20Gb and that memory is not released once image generation is complete.

These are the command I used to test (including typo and errors)

First without the pr

``` 1001  cd Torch/test_venv
 1002  . bin/activate
 1003  ls -lrt
 1004  ls -lrt
 1005  cd ..
 1006  ks
 1007  ls
 1008  cd pytorch
 1009  ls
 1010  cd ..
 1011  rm -rf pytorch
 1012  git clone https://github.com/pytorch/pytorch.git
 1013  cd build_pytorch
 1014  ls
 1015  cd ..
 1016  ls
 1017  pip uninstall torch torchvision
 1018  cd build_pytorch
 1019  xs ../pytorch
 1020  cd ../pytorch
 1021  CMAKE_PREFIX_PATH=/Users/xxx/Torch/build_pytorch USE_DISTRIBUTED=ON USE_GLOO=OFF MACOSX_DEPLOYMENT_TARGET=13.0 CC=clang CXX=clang++ python setup.py install\n
 1022  cd ../InvokeAI
 1023  export PYTORCH_ENABLE_MPS_FALLBACK=1
 1024  python scripts/invoke.py
 1025  vi ~/.invokeai
 1026  python scripts/invoke.py
 1027  pip3 install -U --pre torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cpu
 1028  python scripts/invoke.py
 1029  open /Users/xxx/SSD/Source/Python/invoke_output/000041.1013875009.png
```

And to test with the PR (same terminal session, so PYTORCH_ENABLE_MPS_FALLBACK was still set) 

``` 1030  cd ../pytorch
 1031  pip uninstall torch torchvision
 1032  pip uninstall torch torchvision
 1033  pip uninstall torch torchvision
 1034  cd ..
 1035  rm -rf pytorch
 1036  git clone https://github.com/pytorch/pytorch.git
 1037  cd pytorch
 1038  git fetch origin pull/89436/head:test_89436
 1039  git checkout test_89436
 1040  CMAKE_PREFIX_PATH=/Users/xxx/Torch/build_pytorch USE_DISTRIBUTED=ON USE_GLOO=OFF MACOSX_DEPLOYMENT_TARGET=13.0 CC=clang CXX=clang++ python setup.py install\n
 1041  pip3 install -U --pre torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cpu
 1042  cd ../InvokeAI
 1043  python scripts/invoke.py
```

The same prompt and parameters where used for both runs.

Timing show a big improvement in the sampling speed (hope you can keep that it back to 12.1 speeds)
and the massive slow done for the post sampling phase where the memory issues occur.

Without the PR
```>> Ksampler using karras noise schedule (steps < 30)
>> Sampling with k_lms starting at step 0 of 10 (10 new sampling steps)
100%|█████████████████████████████████████| 10/10 [01:51<00:00, 11.11s/it]
Generating: 100%|██████████████████████████| 1/1 [02:00<00:00, 120.42s/it]
>> Usage stats:
>>   1 image(s) generated in 121.02s```

With the PR
```>> Ksampler using karras noise schedule (steps < 30)
>> Sampling with k_lms starting at step 0 of 10 (10 new sampling steps)
100%|█████████████████████████████████████| 10/10 [00:45<00:00,  4.59s/it]
Generating: 100%|██████████████████████████| 1/1 [05:03<00:00, 303.23s/it]
>> Usage stats:
>>   1 image(s) generated in 303.56s```


### Versions

PyTorch version: 1.14.0a0+gitf6fbbdc
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: version 3.25.0
Libc version: N/A

Python version: 3.10.8 (main, Nov  9 2022, 16:53:27) [Clang 12.0.5 (clang-1205.0.22.9)] (64-bit runtime)
Python platform: macOS-13.0.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] pytorch-lightning==1.7.7
[pip3] torch==1.14.0a0+gitf6fbbdc
[pip3] torch-fidelity==0.3.0
[pip3] torchdiffeq==0.2.3
[pip3] torchmetrics==0.10.0
[pip3] torchsde==0.2.5
[pip3] torchvision==0.15.0.dev20221207
[conda] Could not collect
```

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev