TrainingArguments does not support `mps` device (Mac M1 GPU)
Author: saattrupdan
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.0.dev0
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.8.9
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```bash
export TASK_NAME=wnli
python run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/
```

### Expected behavior

When running the `Trainer.train` on a machine with an MPS GPU, it still just uses the CPU. I expected it to use the MPS GPU. This is supported by `torch` in the newest version 1.12.0, and we can check if the MPS GPU is available using `torch.backends.mps.is_available()`.

It seems like the issue lies in the [`TrainingArguments._setup_devices` method](https://github.com/huggingface/transformers/blob/49cd736a288a315d741e5c337790effa4c9fa689/src/transformers/training_args.py#L1266), which doesn't appear to allow for the case where `device = "mps"`.

Comments:
A simple hack fixed the issue, by simply overwriting the `device` attribute of `TrainingArguments`:

```python
import torch
from transformers import TrainingArguments


class TrainingArgumentsWithMPSSupport(TrainingArguments):

    @property
    def device(self) -> torch.device:
        if torch.cuda.is_available():
            return torch.device("cuda")
        elif torch.backends.mps.is_available():
            return torch.device("mps")
        else:
            return torch.device("cpu")
```

This at least shows that it might just be the aforementioned `_setup_devices` that needs changing.

Another observation: Some PyTorch operations have not been implemented in `mps` and will throw an error. One way to get around that is to set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1`, which will fallback to CPU for these operations. It still throws a `UserWarning` however.

This is not supported yet, as this has been introduced by PyTorch 1.12, which also breaks all speech models due to a regression there. We will look into the support for Mac M1 GPUs once we officially support PyTorch 1.12 (probably won't be before they do a patch 1.12.1).

@sgugger And it's not possible to add a `use_mps` flag to `TrainingArguments`, which just requires PyTorch 1.12.x, alongside a warning of some kind? Or is that too unstable?

I have no idea, since we haven't tried and tested it out yet. And as I said our whole CI is constrained by PyTorch < 1.12 right now, so until that pin is dropped we can't test the integration :-). You can certainly try it on your own fork in the meantime!

I'm seeing this odd behavior. I'm trying a code from [here](https://github.com/nlp-with-transformers/notebooks/blob/main/02_classification.ipynb), adapted with @saattrupdan solution. It runs but the problem is that I'm getting very different results when using `cpu` and `mps`. With `device_type = "cpu"` I get the expected results (`f1=0.92`) but when using `device_type = "mps"` I'm getting a very low f1 (~0.3), likely as a result a random guess. 


```python
device_type = "mps"
device = torch.device(device_type)
# Tokenizer
from transformers import AutoTokenizer
model_ckpt = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
def tokenize(batch):
    return tokenizer(batch["text"],padding = True,truncation = True)

# Load Data
from datasets import list_datasets
from datasets import load_dataset
emotions = load_dataset("emotion")
emotions_encoded = emotions.map(tokenize,batched = True,batch_size = None)

#Model
from transformers import AutoModelForSequenceClassification
num_labels = 6
model = (AutoModelForSequenceClassification.from_pretrained(model_ckpt,num_labels = num_labels).to(device))

#Metric
from sklearn.metrics import accuracy_score,f1_score
def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(1)
    f1 = f1_score(labels,preds,average = "weighted")
    acc = accuracy_score(labels,preds)
    return {"accuracy":acc,"f1":f1}

#Train
from transformers import Trainer,TrainingArguments
class TrainingArgumentsWithMPSSupport(TrainingArguments):
    @property
    def device(self) -> torch.device:
        if device_type == "mps":
            return torch.device("mps")
        else:
            return torch.device("cpu")
batch_size = 64
loggin_steps = len(emotions_encoded["train"])
model_name = f"{model_ckpt}-finetuned-emotion"
train_args = TrainingArgumentsWithMPSSupport(output_dir = model_name,
                               num_train_epochs = 2,
                               learning_rate = 2e-5,
                               per_device_train_batch_size = batch_size,
                               per_device_eval_batch_size = batch_size,
                               weight_decay = 0.01,
                               evaluation_strategy = "epoch",
                               disable_tqdm = False,
                               logging_steps = loggin_steps,
                               push_to_hub = False,
                               log_level = "error"
                              )
trainer = Trainer(model = model,args = train_args,
                  compute_metrics = compute_metrics,
                  train_dataset = emotions_encoded["train"],
                  eval_dataset = emotions_encoded["validation"],
                  tokenizer = tokenizer)

print("Trainner device:",trainer.args.device)

trainer.train()
```


We've also observed a drop in metrics when training, see [this issue](https://github.com/pytorch/pytorch/issues/82707).

Now that PyTorch `1.12.1` is out I think we should reopen this issue! cc @pacman100 

Note that on the inference side, pipelines now support `device="mps"` since #18494

@julien-c That's great to hear! In my own scripts I've used [this implementation](https://github.com/saattrupdan/ScandEval/blob/main/src/scandeval/training_args_with_mps_support.py), just tweaking the `TrainingArguments._setup_devices` method.

I also guess that the `no_cuda` training argument has to either be changed to `no_gpu`, if the current functionality should be preserved, or otherwise the handling of this keyword needs to be changed in the method (potentially adding a `no_mps` argument as well then, but I'm not sure if that's desirable). 

I can open a PR if needed 🙂 



Hi Team. Thanks for the mac integration. Quick question - Is this not part of the most recent `pip install`? Because I have the latest pip package version (`4.21.2`) but couldn't find `--use_mps_device` function parameter in it.

Here's a simple snippet from the jupyter notebook 
```
from transformers import TrainingArguments
args = TrainingArguments(use_mps_device=False)
```
and the error message:
```
TypeError                                 Traceback (most recent call last)
/var/folders/_p/8spsq7dj5mg51p7kdrqzlmgr0000gn/T/ipykernel_2337/1658168950.py in <module>
----> 1 args = TrainingArguments(use_mps_device=False)

TypeError: __init__() got an unexpected keyword argument 'use_mps_device'

Hello @V-Sher, it is yet to be released. For time being, you can install transformers from the source to use this feature via the below command
```bash
pip install git+https://github.com/huggingface/transformers
```

Hi All:
I am finetuning a BERT model with HuggingFace Trainer API in Mac OS Ventura (Intel),  Python 3.10 and Torch 2.0.0.
It takes 14 min in a simple scenery with CPU, with no problem.
I  changed to GPU with mps. Initially, GPU was not used, but after redefining TrainingArguments in this way, it worked

```
class TrainingArgumentsWithMPSSupport(TrainingArguments):
    @property
    def device(self) -> torch.device:
        return torch.device(device)

training_args = TrainingArgumentsWithMPSSupport(...)
```

But the problem is that improvement over CPU is scarce (barely from 14 min to 10 min). Monitor says %GPU is only 15% peak.

Any idea about why such poor improvement?

Thanks for any help
Alberto

The is the full code

```
from transformers import BertForSequenceClassification, BertTokenizerFast, Trainer, TrainingArguments
import nlp
import torch
from torch.utils.data import Dataset, DataLoader

device = torch.device("mps:0")

_DATASET = '../IMDB.csv'

dataset = nlp.load_dataset('csv', data_files=[_DATASET], split='train[:1%]')

dataset = dataset.train_test_split(test_size=0.3)
train_set = dataset['train']  
test_set = dataset['test']


class CustomDataset(Dataset):

    def __init__(self, dataset, mytokenizer):
        self.tokenizer = mytokenizer
        self.dataset = dataset
        self.texts = dataset["text"] 

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, index):
        theText = self.dataset[index]['text']
        theLabel = self.dataset[index]['label']
        inputs = self.tokenizer(theText, max_length=512, padding='max_length', truncation=True)
        ids = inputs['input_ids']
        mask = inputs['attention_mask']
        token_type_ids = inputs["token_type_ids"]

        ids = torch.tensor(ids, dtype=torch.long).to(device)
        mask = torch.tensor(mask, dtype=torch.long).to(device)
        theLabel = torch.tensor(theLabel, dtype=torch.long).to(device)

        result = {
            'input_ids': ids,
            'attention_mask': mask,
            'label': theLabel
        }

        return result


model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

training_set = CustomDataset(train_set, tokenizer)
testing_set = CustomDataset(test_set, tokenizer)

batch_size = 8
epochs = 2
warmup_steps = 500
weight_decay = 0.01


class TrainingArgumentsWithMPSSupport(TrainingArguments):
    @property
    def device(self) -> torch.device:
        return torch.device(device)



training_args = TrainingArgumentsWithMPSSupport(
	output_dir='./results',
	num_train_epochs=epochs,
	per_device_train_batch_size=batch_size,
	per_device_eval_batch_size=batch_size,
	warmup_steps=warmup_steps,
	weight_decay=weight_decay,
	# evaluate_during_training=True,
	evaluation_strategy='steps',
	logging_dir='./logs',
)

trainer = Trainer(
	model=model.to(device),
	args=training_args,
	train_dataset=training_set,
	eval_dataset=testing_set
)

trainer.train()  # full finetune
trainer.evaluate()
```




After installing `transformers` package from source as suggested by @pacman100 like this:
```bash
pip install git+https://github.com/huggingface/transformers
``` 
the `mps` device is used with the standard `TrainingArguments` class. Does not require the custom `TrainingArgumentsWithMPSSupport` class.

Now the M1 Mac GPU is ~90% utilized.
![Screenshot 2023-06-14 at 16 03 57](https://github.com/huggingface/transformers/assets/98090437/a7583667-64be-4670-b9ba-934a63798468)



> After installing `transformers` package from source as suggested by @pacman100 like this:
> 
> ```shell
> pip install git+https://github.com/huggingface/transformers
> ```
> 
> the `mps` device is used with the standard `TrainingArguments` class. Does not require the custom `TrainingArgumentsWithMPSSupport` class.
> 
> Now the M1 Mac GPU is ~90% utilized. ![Screenshot 2023-06-14 at 16 03 57](https://user-images.githubusercontent.com/98090437/245818183-a7583667-64be-4670-b9ba-934a63798468.png)

Comfirm, it works for me too.

Hi, 

Still getting the same error on apple Macbook Pro M1 
ValueError: BF16 Mixed precision training with AMP (`--bf16`) and BF16 half precision evaluation (`--bf16_full_eval`) can only be used on CUDA or CPU/TPU/NeuronCore devices.
current python libs versions accelerate==0.22.0.dev0  peft==0.4.0 bitsandbytes==00.41.1 transformers==4.31.0 trl==0.4.7 torch==2.1.0.dev20230820
upgrade to the latest transformers start getting errors on the 
model loading
ImportError: Using `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or pip install bitsandbytes` 

any recommendations

I'm also on Macbook Pro M1 and is getting the same error as the post above me by @inaim .

Similar problem here: https://stackoverflow.com/questions/76589840/cant-run-transformer-fine-tuning-with-m1-mac-cpu

My recommendation is to install `bitsandbytes` and `accelerate`

Hello, `load_in_8bit` isn't supported for Mac devices. So, load the model without that param. @younesbelkada, who has better idea about bitsandbytes.

Hi @pacman100 , 

The config used for quantization
# bitsandbytes parameters

use_4bit = True

bnb_4bit_compute_dtype = "float16"

bnb_4bit_quant_type = "nf4"

use_nested_quant = False

Hi @ArthurZucker ,

They are already installed

> bitsandbytes

bitsandbytes does not work for mps.

> I'm seeing this odd behavior. I'm trying a code from [here](https://github.com/nlp-with-transformers/notebooks/blob/main/02_classification.ipynb), adapted with @saattrupdan solution. It runs but the problem is that I'm getting very different results when using `cpu` and `mps`. With `device_type = "cpu"` I get the expected results (`f1=0.92`) but when using `device_type = "mps"` I'm getting a very low f1 (~0.3), likely as a result a random guess.
> 
> ```python
> device_type = "mps"
> device = torch.device(device_type)
> # Tokenizer
> from transformers import AutoTokenizer
> model_ckpt = "distilbert-base-uncased"
> tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
> def tokenize(batch):
>     return tokenizer(batch["text"],padding = True,truncation = True)
> 
> # Load Data
> from datasets import list_datasets
> from datasets import load_dataset
> emotions = load_dataset("emotion")
> emotions_encoded = emotions.map(tokenize,batched = True,batch_size = None)
> 
> #Model
> from transformers import AutoModelForSequenceClassification
> num_labels = 6
> model = (AutoModelForSequenceClassification.from_pretrained(model_ckpt,num_labels = num_labels).to(device))
> 
> #Metric
> from sklearn.metrics import accuracy_score,f1_score
> def compute_metrics(pred):
>     labels = pred.label_ids
>     preds = pred.predictions.argmax(1)
>     f1 = f1_score(labels,preds,average = "weighted")
>     acc = accuracy_score(labels,preds)
>     return {"accuracy":acc,"f1":f1}
> 
> #Train
> from transformers import Trainer,TrainingArguments
> class TrainingArgumentsWithMPSSupport(TrainingArguments):
>     @property
>     def device(self) -> torch.device:
>         if device_type == "mps":
>             return torch.device("mps")
>         else:
>             return torch.device("cpu")
> batch_size = 64
> loggin_steps = len(emotions_encoded["train"])
> model_name = f"{model_ckpt}-finetuned-emotion"
> train_args = TrainingArgumentsWithMPSSupport(output_dir = model_name,
>                                num_train_epochs = 2,
>                                learning_rate = 2e-5,
>                                per_device_train_batch_size = batch_size,
>                                per_device_eval_batch_size = batch_size,
>                                weight_decay = 0.01,
>                                evaluation_strategy = "epoch",
>                                disable_tqdm = False,
>                                logging_steps = loggin_steps,
>                                push_to_hub = False,
>                                log_level = "error"
>                               )
> trainer = Trainer(model = model,args = train_args,
>                   compute_metrics = compute_metrics,
>                   train_dataset = emotions_encoded["train"],
>                   eval_dataset = emotions_encoded["validation"],
>                   tokenizer = tokenizer)
> 
> print("Trainner device:",trainer.args.device)
> 
> trainer.train()
> ```

`TypeError: expected str, bytes or os.PathLike object, not TrainingArgumentsWithMPSSupport`