mac m1 `mps` integration
Author: pacman100
Pull request: yes

# What does this PR do?

1. Enables users to leverage Apple M1 GPUs via mps device type in PyTorch for faster training and inference than CPU. Fixes #17971
2. User has to just pass `--use_mps_device` argument. 
For example, you can run the offical Glue text classififcation task (from the root folder) using Apple Silicon M1 GPU with below command:

```bash
export TASK_NAME=mrpc

python examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --use_mps_device \
  --overwrite_output_dir
```

Below are the output logs:

```bash
python examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --use_mps_device \
  --overwrite_output_dir
NOTE: Redirects are currently not supported in Windows or MacOs.
08/12/2022 15:30:13 - WARNING - __main__ - Process rank: -1, device: mps, n_gpu: -1distributed training: False, 16-bits training: False
08/12/2022 15:30:13 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=-1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_steps=None,
evaluation_strategy=no,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=2e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=-1,
log_level=-1,
log_level_replica=-1,
log_on_each_node=True,
logging_dir=/tmp/mrpc/runs/Aug12_15-30-12_Sourabs-MacBook-Pro.local,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=steps,
lr_scheduler_type=linear,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
no_cuda=False,
num_train_epochs=3.0,
optim=adamw_hf,
output_dir=/tmp/mrpc/,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=32,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=/tmp/mrpc/,
save_on_each_node=False,
save_steps=500,
save_strategy=steps,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
tf32=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=True,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
xpu_backend=None,
)
08/12/2022 15:30:14 - INFO - datasets.info - Loading Dataset Infos from 

...

[INFO|configuration_utils.py:643] 2022-08-12 15:30:17,041 >> loading configuration file config.json from cache at /Users/sourabmangrulkar/.cache/huggingface/hub/models--bert-base-cased/snapshots/a8d257ba9925ef39f3036bfc338acf5283c512d9/config.json
[INFO|configuration_utils.py:695] 2022-08-12 15:30:17,042 >> Model config BertConfig {
  "_name_or_path": "bert-base-cased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.22.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 28996
}

...

08/12/2022 15:30:19 - INFO - __main__ - Sample 2619 of the training set: {'sentence1': 'The proceedings were taken up with prosecutors outlining their case against Amrozi , reading 33 pages of documents outlining allegations against him .', 'sentence2': 'Proceedings were taken up with prosecutors outlining their case against Amrozi , reading a 33-page accusation letter to the court .', 'label': 1, 'idx': 2916, 'input_ids': [101, 1109, 10830, 1127, 1678, 1146, 1114, 24987, 1149, 13260, 1147, 1692, 1222, 7277, 2180, 5303, 117, 3455, 3081, 5097, 1104, 4961, 1149, 13260, 9966, 1222, 1140, 119, 102, 20661, 1127, 1678, 1146, 1114, 24987, 1149, 13260, 1147, 1692, 1222, 7277, 2180, 5303, 117, 3455, 170, 3081, 118, 3674, 21100, 2998, 1106, 1103, 2175, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}.
...

[INFO|trainer.py:1612] 2022-08-12 15:30:22,027 >> ***** Running training *****
[INFO|trainer.py:1613] 2022-08-12 15:30:22,027 >>   Num examples = 3668
[INFO|trainer.py:1614] 2022-08-12 15:30:22,027 >>   Num Epochs = 3
[INFO|trainer.py:1615] 2022-08-12 15:30:22,027 >>   Instantaneous batch size per device = 32
[INFO|trainer.py:1616] 2022-08-12 15:30:22,027 >>   Total train batch size (w. parallel, distributed & accumulation) = 32
[INFO|trainer.py:1617] 2022-08-12 15:30:22,027 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1618] 2022-08-12 15:30:22,027 >>   Total optimization steps = 345
100%|█████████████████████████████████████████████████████████████| 345/345 [09:38<00:00,  2.04s/it][INFO|trainer.py:1857] 2022-08-12 15:40:00,410 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 578.4189, 'train_samples_per_second': 19.024, 'train_steps_per_second': 0.596, 'train_loss': 0.4251004426375679, 'epoch': 3.0}
100%|█████████████████████████████████████████████████████████████| 345/345 [09:38<00:00,  1.68s/it]
[INFO|trainer.py:2647] 2022-08-12 15:40:00,481 >> Saving model checkpoint to /tmp/mrpc/
[INFO|configuration_utils.py:440] 2022-08-12 15:40:00,487 >> Configuration saved in /tmp/mrpc/config.json
[INFO|modeling_utils.py:1569] 2022-08-12 15:40:01,553 >> Model weights saved in /tmp/mrpc/pytorch_model.bin
[INFO|tokenization_utils_base.py:2114] 2022-08-12 15:40:01,561 >> tokenizer config file saved in /tmp/mrpc/tokenizer_config.json
[INFO|tokenization_utils_base.py:2121] 2022-08-12 15:40:01,561 >> Special tokens file saved in /tmp/mrpc/special_tokens_map.json
***** train metrics *****
  epoch                    =        3.0
  train_loss               =     0.4251
  train_runtime            = 0:09:38.41
  train_samples            =       3668
  train_samples_per_second =     19.024
  train_steps_per_second   =      0.596
08/12/2022 15:40:01 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:729] 2022-08-12 15:40:01,619 >> The following columns in the evaluation set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: sentence2, sentence1, idx. If sentence2, sentence1, idx are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.
[INFO|trainer.py:2898] 2022-08-12 15:40:01,637 >> ***** Running Evaluation *****
[INFO|trainer.py:2900] 2022-08-12 15:40:01,637 >>   Num examples = 408
[INFO|trainer.py:2903] 2022-08-12 15:40:01,637 >>   Batch size = 8
100%|███████████████████████████████████████████████████████████████| 51/51 [00:04<00:00, 11.68it/s]
***** eval metrics *****
  epoch                   =        3.0
  eval_accuracy           =     0.8407
  eval_combined_score     =     0.8644
  eval_f1                 =     0.8881
  eval_loss               =     0.3957
  eval_runtime            = 0:00:04.80
  eval_samples            =        408
  eval_samples_per_second =     84.915
  eval_steps_per_second   =     10.614
```

Attaching plots showing GPU usage on M1 pro with 10 CPU and 14 GPU cores:
<img width="393" alt="Screenshot 2022-08-12 at 3 37 37 PM" src="https://user-images.githubusercontent.com/13534540/184333060-85df7fc3-28ab-4a7f-9a61-787df6c19c90.png">

Note: Pre-requisites: Installing torch with `mps` support
```python
# installing torch with m1 support on mac
# install python 3.10.5
# check the platform
import platform
platform.platform()
'macOS-12.5-arm64-arm-64bit' 
# (This is compatible as the macOS version is above 12.3 and it is the ARM64 version) 
# install torch 1.12 via the below command
# pip3 install torch torchvision torchaudio
# test the `mps` device support
>>> import torch
>>> torch.has_mps
True
>>> a = torch.Tensor([10,11])
>>> a.to("mps")
/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
tensor([10.0000, 11.0000], device='mps:0')
```


Comments:
_The documentation is not available anymore as the PR was closed or merged._