Consume some many memory when setting eval_accumulation_steps
Author: SingL3
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Machine Info: 8xA100(80G), Memory 960GB
Codebase: [stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca)
Model: Pythia-6.9b
Dataset: [c-s-ale/dolly-15k-instruction-alpaca-format](https://huggingface.co/datasets/c-s-ale/dolly-15k-instruction-alpaca-format)
The eval dataset and the training dataset is all the same(the whole dataset).
Leaving `eval_accumilation_steps=None` would cause GPU OOM, so setting `eval_accumilation_steps` to a valid number (I have tried 1 and 8).
And the eval process would stuck at evaluating like about 20 samples and would take all the CPU memory and raise a CPU memory OOM.
I have tried the above process using deepspeed and torchrun on all 8 gpus.

However, when I tried running LoRA tuning on 1 gpu without deepspeed or torchrun under the same setting and datasets, the CPU memory will never OOM.

### Expected behavior

Running without OOM

Comments:
I'm not sure how you want us to help without even telling us which arguments you used to launch this script or which exact model on the Hub you are using.

@sgugger, sorry that I didnt provide enough info
Launch script:
```
torchrun --nproc_per_node=4 --master_port=30500 train.py \
    --model_name_or_path <path_to_pythia_6.9b> \
    --data_path ./alpaca_data.json \
    --bf16 True \
    --output_dir ./output_dir/ \
    --num_train_epochs 3 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "steps" \
    --eval_steps 1 \
    --save_strategy "steps" \
    --save_steps 10000 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'GPTNeoXLayer' \
    --tf32 True \
    --model_max_length 2048 \
    --eval_accumulation_steps 1
```
(Need to set `use_fast=True` at [L196](https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py#L196) and set `eval_dataset=training_dataset` at [L179](https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py#L179))
I have also using a metric func and add it to the trainer by `trainer = Trainer(model=model, tokenizer=tokenizer, args=training_args, compute_metrics=build_acc_metric_fn(), **data_module)` at [L215](https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py#L215)
```
def build_acc_metric_fn(**kwargs) -> Callable[[EvalPrediction], Dict]:

    def acc_metric(pred: EvalPrediction):
        label_ids = pred.label_ids
        logits = pred.predictions

        # Get output tokens
        greedy_tokens = logits.argmax(axis=-1)

        # Shift label and logits to match
        label_ids = label_ids[:, 1:]
        greedy_tokens = greedy_tokens[:, :-1]

        token_match = 0
        valid_len = 0
        seq_match = 0
        for i in range(label_ids.shape[0]):
            # Mask out ignore
            mask = (label_ids[i] != IGNORE_INDEX)
            valid_len += mask.sum()
            token_match += (label_ids[i][mask] == greedy_tokens[i][mask]).sum()
            seq_match += (label_ids[i][mask] == greedy_tokens[i][mask]).all()

        return {
            'seq_acc': float(seq_match / label_ids.shape[0]),
            'token_acc': float(token_match / valid_len)
        }

    return acc_metric
```

You cannot use such a metric function with the Trainer without using a large amount of memory: the logits (once accumulated) have a size of your dataset length x max sequence length x vocab size, which is huge. You should do this evaluation batch by batch outside of the Trainer (using Accelerate for instance).

@sgugger 
I see. So I actually using a small eval_dataset.
However, this didnt explain why doing that using LoRA work on single gpu but dont work using all param finetune on eight gpus.