Lower Training performance after updating from 4.29.2 to 4.30.0+
Author: waterhorse1
Pull request: no
Labels: name: solved

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.4.0-149-generic-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.2
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The codebase is https://github.com/OFA-Sys/gsm8k-ScRel.

1. Run train.py to train the llama model, the only modification is to modify this line https://github.com/OFA-Sys/gsm8k-ScRel/blob/f4d01761ec03d88a39486399c4617d29ee1dca7f/train.py#L264 to model_args.model_name_or_path. And here is the training scripts I am using:
```
export MODEL_PATH="huggyllama/llama-7b"
export SAVE_PATH="/data/ziyu/rft_model_xidong/llama1-7b-sft-test-430"
export MASTER_ADDR="localhost"
export MASTER_PORT="1231"

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python3 -m torch.distributed.launch --master_addr ${MASTER_ADDR} --master_port ${MASTER_PORT} --nproc_per_node=8 --use_env train.py \
    --model_name_or_path $MODEL_PATH \
    --data_path data/train_use.jsonl \
    --bf16 True \
    --output_dir $SAVE_PATH \
    --num_train_epochs 3 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 200 \
    --save_total_limit 40 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True \
    --cache_dir "/data/ziyu/hf_cache/huggingface/hub"
```
2. after training the model, test it with https://github.com/OFA-Sys/gsm8k-ScRel/blob/main/test_7b_13b.sh.
```
sh test_7b_13b.sh /data/ziyu/rft_model_xidong/llama1-7b-sft-test-430
``` 
3. Run the evaluation code here: https://github.com/OFA-Sys/gsm8k-ScRel/blob/main/eval.py, need to modify this line https://github.com/OFA-Sys/gsm8k-ScRel/blob/f4d01761ec03d88a39486399c4617d29ee1dca7f/eval.py#L146 to your model path, which is /data/ziyu/rft_model_xidong/llama1-7b-sft-test-430 in our case.


### Expected behavior

We observe that with accelerate=0.21.0, torch=2.0.1, transformers=4.29.2, we can get 0.35+ performance on llama1 finetuning on gsm8k, but with  transformers=4.30.0 and beyond, we can only get 0.32 performance using exactly the same code. For llama2 sft it is similar, I can get 0.41 with transformers=4.29.2 but just 0.36 with transformers=4.30.0. Any reasons for that?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @ArthurZucker @pacman100 

@pacman100 and @muellerzr would be nice to check this out

@waterhorse1 this may have been some stuff we had to fix throughout the Accelerate integration with transformers potentially. Regardless, after running through `gsm8k-ScRel` on 4xA100, I was able to get an ending score of `35.78468536770281`, which I believe should solve your issue.

This was ran on `transformers` and `accelerate` main. If this seems acceptable to you, let me know!

Sure, thx for that.