May be detach the loss when evaluating? in case GPU OOM
Author: ziranyang0
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.15.0-86-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: v0.15.0
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: 4*3090
- Using distributed or parallel set-up in script?: default DDP in Trainer

### Who can help?

@muellerzr  @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

the code below are quite easy, finetuning a gpt2 on wikitext2

If you run this on some 3090, then OOM is expected to be raised when evaluation(i.e. the **results = trainer.evaluate()** line)

```
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel, DataCollatorForLanguageModeling, TrainingArguments, Trainer
from datasets import load_dataset
import wandb
import os
wandb.init(project="gpt2-finetune-wikitext2")

# 1. Preparation
MODEL_NAME = 'gpt2-medium'
tokenizer = GPT2Tokenizer.from_pretrained(MODEL_NAME)
model = GPT2LMHeadModel.from_pretrained(MODEL_NAME)
tokenizer.pad_token = tokenizer.eos_token

# 2. Loading the wikitext-2 dataset
wikitext_dataset = load_dataset('wikitext', 'wikitext-2-raw-v1')

# Tokenization
def tokenize_function(examples):
    return tokenizer(examples['text'], truncation=True, padding='max_length', max_length=128)

train_dataset = wikitext_dataset['train'].map(tokenize_function, batched=True)
valid_dataset = wikitext_dataset['validation'].map(tokenize_function, batched=True)

# Removing unnecessary columns
train_dataset = train_dataset.remove_columns(["text"])
valid_dataset = valid_dataset.remove_columns(["text"])

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False
)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    loss = torch.nn.CrossEntropyLoss()(logits.view(-1, logits.size(-1)), labels.view(-1))
    perplexity = torch.exp(torch.tensor(loss)).item()
    return {"loss": loss, "perplexity": perplexity}

# 3. Fine-tuning

training_args = TrainingArguments(
    output_dir="./gpt2-wikitext2",
    overwrite_output_dir=True,
    num_train_epochs=2,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=4,
    eval_steps=400,
    save_steps=800,
    warmup_steps=500,
    logging_dir='./logs',
    logging_strategy="steps",
    logging_steps=10,
    report_to="wandb",  # Add this line to log to wandb
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=valid_dataset,
    compute_metrics=compute_metrics,
)

trainer.train()

# 4. Evaluation 
results = trainer.evaluate()
print(results)


perplexity = torch.exp(torch.tensor(results["loss"]))

# Log perplexity to wandb
wandb.log({"perplexity": perplexity.item()})

print(f"Perplexity: {perplexity.item()}")


# Save the model
model.save_pretrained("./gpt2-wikitext2")
tokenizer.save_pretrained("./gpt2-wikitext2")

wandb.finish()
```

### Expected behavior

 I'm finetuning a gpt2 on wikitext2. The problem is that during evaluation(which is the line **results = trainer.evaluate()**), an OOM, out of memory error is raised.
 
 I think this should not happen since training is OK with a even bigger batchsize, after checking the source code in Trainer(**/transformers/trainer.py**), I found it useful to add 1 line at line 3011 just after 
```
loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
``` 
in **evaluation_loop()**
 
 the line is:
 ```
            loss = loss.detach().cpu()
```
which detach the loss and the OOM error would not happen any more.



after all, I am not sure whether the OOM during evaluation is a bug or a feature, please excuse me if bothering

Comments:
I'm not sure why exactly that'd be helpful, since we already detach the loss during `prediction_step`: https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3407

And convert it to the CPU later on during the call to `nested_numpify`: https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3249-L3251

If you don't have the memory to handle this, you can use `eval_accumulation_steps` instead to have that conversion to the CPU happen faster

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.