[FSDP] [Checkpointing] Loading optimizer state dict with use_orig_params True causes OOM
Labels: triaged, module: fsdp

### 🐛 Describe the bug

When use_orig_params is set to True in FSDP, `FSDP.optim_state_dict_to_load` call uses much higher memory on GPU than when use_orig_params is False. This causes an OOM later when optimizer.load_state_dict clones the state dict. 

There's also no way to offload this to CPU. 

For a Llama 7B model on a single node, it produced 9GB extra allocation on the GPU after optim_state_dict_to_load when compared to use_orig_params=False.


```
2023-09-27 20:14:45 I [train_lib.py:642] Created optimizer
warm up iteration = 1000.0
2023-09-27 20:14:46 I [checkpoints.py:400] Loading checkpoint from /opt/ml/input/data/train/checkpoints/llama-7b-test/llama_fast-10steps/ ...
NCCL version 2.18.3+cuda11.8
2023-09-27 20:14:57 I [checkpoints.py:336] Loaded model state from disk
2023-09-27 20:14:57 I [checkpoints.py:337] Loaded state from disk: start_train_path_index 0, start_batch_index 10.
2023-09-27 20:15:08 I [learning_rates.py:122] Overriding learning rate value to 0.0003
2023-09-27 20:15:08 I [learning_rates.py:122] Overriding minimum learning rate value to 3e-05
2023-09-27 20:15:08 I [learning_rates.py:122] Overriding warmup iterations value to 1000.0
2023-09-27 20:15:08 I [learning_rates.py:122] Overriding total number of iterations value to 12500
2023-09-27 20:15:08 I [learning_rates.py:122] Overriding decay style value to cosine
2023-09-27 20:15:28 I [checkpoints.py:352] Loaded and sharded optimizer state from disk
2023-09-27 20:15:29 I [checkpoints.py:361] Converted optimizer state dict for FSDP
Traceback (most recent call last):
  File "/opt/ml/code/train.py", line 13, in <module>
main()
  File "/opt/ml/code/train.py", line 9, in main
    train_lib.main(args)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
  File "/opt/ml/code/train_lib.py", line 658, in main
) = load_checkpoint(
  File "/opt/ml/code/checkpoints.py", line 416, in load_checkpoint
loaded = _load_sharded(
  File "/opt/ml/code/checkpoints.py", line 362, in _load_sharded
optimizer.load_state_dict(flattened_osd)
  File "/opt/conda/lib/python3.10/site-packages/torch/optim/optimizer.py", line 379, in load_state_dict
state_dict = deepcopy(state_dict)
  File "/opt/conda/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
  File "/opt/conda/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
  File "/opt/conda/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
  File "/opt/conda/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.10/copy.py", line 153, in deepcopy
y = copier(memo)
  File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 118, in __deepcopy__
new_storage = self._typed_storage()._deepcopy(memo)
  File "/opt/conda/lib/python3.10/site-packages/torch/storage.py", line 684, in _deepcopy
return self._new_wrapped_storage(copy.deepcopy(self._untyped_storage, memo))
  File "/opt/conda/lib/python3.10/copy.py", line 153, in deepcopy
y = copier(memo)
  File "/opt/conda/lib/python3.10/site-packages/torch/storage.py", line 98, in __deepcopy__
new_storage = self.clone()
  File "/opt/conda/lib/python3.10/site-packages/torch/storage.py", line 112, in clone
return type(self)(self.nbytes(), device=self.device).copy_(self)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 4; 39.56 GiB total capacity; 36.98 GiB already allocated; 64.81 MiB free; 37.84 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

### Versions

```
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2 (x86_64)
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15)
Clang version: Could not collect
CMake version: version 3.27.5
Libc version: glibc-2.26

Python version: 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.10.186-179.751.amzn2.x86_64-x86_64-with-glibc2.26
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              96
On-line CPU(s) list: 0-95
Thread(s) per core:  2
Core(s) per socket:  24
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
Stepping:            7
CPU MHz:             1996.805
BogoMIPS:            5999.98
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-23,48-71
NUMA node1 CPU(s):   24-47,72-95
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke avx512_vnni

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] pytorch-lightning==1.9.4
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchmetrics==1.2.0
[pip3] torchvision==0.15.2
[pip3] triton==2.0.0
[conda] blas                      2.116                       mkl    conda-forge
[conda] blas-devel                3.9.0            16_linux64_mkl    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            16_linux64_mkl    conda-forge
[conda] mkl                       2022.1.0           h84fe81f_915    conda-forge
[conda] mkl-devel                 2022.1.0           ha770c72_916    conda-forge
[conda] mkl-include               2022.1.0           h84fe81f_915    conda-forge
[conda] numpy                     1.23.5                   pypi_0    pypi
[conda] pytorch                   2.0.1           sm_py3.10_cuda11.8_cudnn8.7.0_nccl2.18.3_0_smp_2.0.0b1_pt_2.0.1    s3://smdistributed-modelparallel-preview/smp-2.0.0b1-pt-2.0.1/build_artifacts/2023-09-15/smp-v2_preview
[conda] pytorch-cuda              11.8                 h7e8668a_5    pytorch
[conda] pytorch-lightning         1.9.4                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                2.0.2               py310_cu118    pytorch
[conda] torchmetrics              1.2.0                    pypi_0    pypi
[conda] torchtriton               2.0.0+b8b470bc59           py310    pytorch-nightly
[conda] torchvision               0.15.2                   pypi_0    pypi
```

cc @zhaojuanmao @mrshenli @rohan-varma @awgu @fegin @penguinwu