Training failure with `--use_cpu` option when GPU memory is saturated
Author: qgallouedec
Pull request: no

### System Info

Transformers: 4.33.2
Torch 2.0.1
Python 3.9.12

### Who can help?

@muellerzr and @pacman100

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The issue arises in a very specific context but needs to be reported nonetheless:

- The GPU memory is saturated.
- Another training is launched with the `--use_cpu` argument.

in this setting, the second training fails, raising a "CUDA out of memory" error. This outcome is unexpected since the `--use_cpu` option is supposed to enforce the use of the CPU for training.

**Root Cause:**
This issue stems from the fact that, by default, `--dataloader_pin_memory` is set to `True`. At a certain point, the dataloader attempts to pin the batch to the GPU, even if the device is set to CPU. Roughly equivalent to:

```python
import torchvision.transforms as transforms
import torchvision.datasets as datasets
from torch.utils.data import DataLoader


# Download and load the training data
dataset = datasets.MNIST("~/.pytorch/MNIST_data/", download=True, train=True, transform=transforms.ToTensor())

# Define a dataloader with pin_memory=True
dataloader = DataLoader(dataset, pin_memory=True)

# Get one batch of images and labels
images, labels = next(iter(dataloader))
```

### Expected behavior

To resolve the issue, the best solution would be to enforce `pin_memory` to `False` in `TrainingArguments.__post_init__` when `--use_cpu` is set to `True`

```python
if self.use_cpu:
    self.dataloader_pin_memory = False
```

### Alternative

The alternative is of course to write in the docs that `--use_cpu` should be used with `dataloader_pin_memory` set to False, but AFAIK there is never a situation where you want to use pinned memory and a cpu device.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

It still needs to be addressed 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

It's not a big deal, but it's still an issue that needs to be addressed.
I can quickly submit a PR if I have approval.

A PR with the solution would be great! However I think the solution eventually should be on the Accelerate side since we handle the dataloader creation eventually/wrapping and we can do this under the hood when accelerate-ing the dataloaders

In fact, it seems that the proposed solution solves the issue in both cases (using accelerate or not):

(Context: GPU memory saturated)
Without the fix:
```
accelerate launch 26556.py --output_dir tmp --use_cpu  # fails
26556.py --output_dir tmp --use_cpu  # fails
```

With the fix:
```
accelerate launch 26556.py --output_dir tmp --use_cpu  # ok
26556.py --output_dir tmp --use_cpu  # ok
```

`26556.py`:
```python
from transformers import HfArgumentParser, Trainer, TrainingArguments
from torch import nn
from datasets import Dataset
import numpy as np


class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(100_000, 2)

    def forward(self, x, y):
        z = self.linear(x)
        loss = nn.CrossEntropyLoss()(z, y)
        return {"loss": loss}


def main():
    parser = HfArgumentParser(TrainingArguments)
    training_args = parser.parse_args_into_dataclasses()[0]
    model = MyModel()
    dataset = Dataset.from_dict({"x": np.random.rand(100, 100_000), "y": np.random.randint(0, 2, (100,))})
    trainer = Trainer(model=model, args=training_args, train_dataset=dataset)
    trainer.train()


if __name__ == "__main__":
    main()
```


Trainer uses accelerate under the hood, so that’s not surprising. Go ahead and open a PR and cc me @qgallouedec if you’d like, and I’ll look at handling it on the accelerate side!