Saving LLAMA 13B checkpoint with FSDP finetuning results in disk full error
Author: ari9dam
Pull request: no
Labels: name: solved

### System Info

transformers - installed from source
accelerate - installed from source
torch 2.0.1


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Finetune LLAMA 13B with accelerate launcher. Saving strategy "epoch". FSDP based training.

### Expected behavior

I would be able to save the checkpoint. Now getting disk full error. Note that the disk initially had space. My code was working with transformers-4.28, accelerate 0.18 and torch 1.13. This error started after I moved to accelerate based launcher and upgraded the packages.

Comments:
It seems saving a checkpoint requires more than 213GB (Free memory on my hard disk). Not sure if it is intended.

```
Filesystem      Size  Used Avail Use% Mounted on
overlay         251G   27G  213G  11% /
tmpfs            64M     0   64M   0% /dev
tmpfs           434G     0  434G   0% /sys/fs/cgroup
shm             2.0G     0  2.0G   0% /dev/shm
/dev/sdb1       251G   27G  213G  11% /tmp
tmpfs           434G   12K  434G   1% /proc/driver/nvidia
/dev/root       124G   23G  102G  18% /usr/bin/nvidia-smi
tmpfs            87G  2.4M   87G   1% /run/nvidia-persistenced/socket
devtmpfs        434G     0  434G   0% /dev/nvidia0
```

Hi @ari9dam, thanks for raising this issue. 

Could you a minimal code snippet we can use to reproduce the error? Specifically how accelerate launcher is being used, training arguments, and FDSP config. 

For the transformers and accelerate source installs, which commit are you running from? 

When you say saving a checkpoint - am I right in saying this is the memory requirement for saving a single checkpoint after 1 epoch of training is > 213 GB? 

Yes, " the memory requirement for saving a single checkpoint after 1 epoch of training is > 213 GB?". 

`accelerate launch --config_file accelerate_config.yaml --num_machines 4 --num_processes 16 --machine_rank $NODE_RANK --main_process_ip $MASTER_ADDR --main_process_port $MASTER_PORT ./trainer.py --model_name_or_path ".." --data_path "..." --per_device_train_batch_size 16 --per_device_eval_batch_size 16 --do_train --evaluation_strategy no --output_dir outputs --learning_rate 2e-5 --num_train_epochs 4 --lr_scheduler_type cosine --warmup_ratio 0.03 --weight_decay 0.0 --logging_steps 1 --save_strategy epoch --bf16 true --tf32 true --load_best_model_at_end false --model_max_length 1024 --gradient_checkpointing true --save_total_limit 1 --model_resume_from_checkpoint false --torch_compile false`

### accelerate_config.yaml
```
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: false
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
  fsdp_use_orig_params: true
main_training_function: main
num_machines: 1
num_processes: 2
mixed_precision: bf16
rdzv_backend: static
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```
transformers==4.31.0.dev0 (https://github.com/huggingface/transformers/commit/45025d92f815675e483f32812caa28cce3a960e7)
 accelerate==0.21.0.dev0 (https://github.com/huggingface/accelerate/commit/7954a28a71d484c4182a6b1074c1b8cc51642fc9)
 
 

@ari9dam Thanks for the additional info

cc @pacman100 @muellerzr 

Hello @ari9dam, please show the contents of the checkpoint along with their sizes

total 201G
  69 Jul 14 10:40 added_tokens.json
 656 Jul 14 10:40 config.json
 137 Jul 14 10:40 generation_config.json
 97G Jul 14 10:46 optimizer.bin
6.1G Jul 14 10:41 optimizer.pt
9.3G Jul 14 10:42 pytorch_model-00001-of-00006.bin
9.3G Jul 14 10:42 pytorch_model-00002-of-00006.bin
9.3G Jul 14 10:42 pytorch_model-00003-of-00006.bin
9.2G Jul 14 10:42 pytorch_model-00004-of-00006.bin
9.2G Jul 14 10:42 pytorch_model-00005-of-00006.bin
2.4G Jul 14 10:41 pytorch_model-00006-of-00006.bin
 49G Jul 14 10:44 pytorch_model.bin
 33K Jul 14 10:40 pytorch_model.bin.index.json
 18K Jul 14 10:40 rng_state_0.pth
 18K Jul 14 10:40 rng_state_10.pth
 18K Jul 14 10:40 rng_state_11.pth
 18K Jul 14 10:40 rng_state_12.pth
 18K Jul 14 10:40 rng_state_13.pth
 18K Jul 14 10:40 rng_state_14.pth
 18K Jul 14 10:40 rng_state_15.pth
 18K Jul 14 10:40 rng_state_1.pth
 18K Jul 14 10:40 rng_state_2.pth
 18K Jul 14 10:40 rng_state_3.pth
 18K Jul 14 10:40 rng_state_4.pth
 18K Jul 14 10:40 rng_state_5.pth
 18K Jul 14 10:40 rng_state_6.pth
 18K Jul 14 10:40 rng_state_7.pth
 18K Jul 14 10:40 rng_state_8.pth
 18K Jul 14 10:40 rng_state_9.pth
 627 Jul 14 10:40 scheduler.pt
 435 Jul 14 10:40 special_tokens_map.json
 745 Jul 14 10:40 tokenizer_config.json
1.8M Jul 14 10:40 tokenizer.json
489K Jul 14 10:40 tokenizer.model
 11K Jul 14 10:40 trainer_state.json
4.1K Jul 14 10:40 training_args.bin

I'm not sure about `49G pytorch_model.bin`. It looks to be a duplicate.

Hello, PR https://github.com/huggingface/transformers/pull/24926 should resolve the duplicate saving issue. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.