Support MUSA (Moore Threads GPU) backend in transformers
Author: fmo-mt
Pull request: yes

# What does this PR do?
Accelerate has merged the supporting of `MUSA` backend (https://github.com/huggingface/accelerate/pull/2917) , and we are just following what the MLU team did (yes, again...thanks MLU team: https://github.com/huggingface/transformers/pull/29627).

So, this PR enables users to leverage the MUSA for training and inference of 🤗 Transformers models, and below is the running example of `glue text-classification` using MUSA backend:

```shell
python -m torch.distributed.run --nproc_per_node 8 run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir ./output

WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
07/11/2024 21:28:30 - WARNING - __main__ - Process rank: 5, device: musa:5, n_gpu: 1, distributed training: True, 16-bits training: False
07/11/2024 21:28:30 - WARNING - __main__ - Process rank: 7, device: musa:7, n_gpu: 1, distributed training: True, 16-bits training: False
07/11/2024 21:28:30 - WARNING - __main__ - Process rank: 6, device: musa:6, n_gpu: 1, distributed training: True, 16-bits training: False                                                                                                                      07/11/2024 21:28:30 - WARNING - __main__ - Process rank: 2, device: musa:2, n_gpu: 1, distributed training: True, 16-bits training: False
07/11/2024 21:28:30 - WARNING - __main__ - Process rank: 3, device: musa:3, n_gpu: 1, distributed training: True, 16-bits training: False
07/11/2024 21:28:30 - WARNING - __main__ - Process rank: 1, device: musa:1, n_gpu: 1, distributed training: True, 16-bits training: False
07/11/2024 21:28:30 - WARNING - __main__ - Process rank: 0, device: musa:0, n_gpu: 1, distributed training: True, 16-bits training: False
07/11/2024 21:28:30 - WARNING - __main__ - Process rank: 4, device: musa:4, n_gpu: 1, distributed training: True, 16-bits training: False
07/11/2024 21:28:30 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,                                                                                                                                                                                                                                                  do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=None,
eval_strategy=no,
eval_use_gather_object=False,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,                                                                                                                                                                                                                              include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=2e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=./output/runs/Jul11_21-28-29_mccx6,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,                                                                                                                                                                                                                                    [274/856]
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=linear,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=3.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=./output,
overwrite_output_dir=False,
past_index=-1,                                                                                                                                                                                                                                                 per_device_eval_batch_size=8,
per_device_train_batch_size=32,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=./output,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=None,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,                                                                                                                                                                                                                                                use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
Overwrite dataset info from restored data version if exists.
07/11/2024 21:28:30 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists.
Loading Dataset info from /root/.cache/huggingface/datasets/glue/mrpc/0.0.0/654741f6b0be315d
07/11/2024 21:28:30 - INFO - datasets.info - Loading Dataset info from /root/.cache/huggingface/datasets/glue/mrpc/0.0.0/654741f6b0be315d
Found cached dataset glue (/root/.cache/huggingface/datasets/glue/mrpc/0.0.0/654741f6b0be315d)
07/11/2024 21:28:30 - INFO - datasets.builder - Found cached dataset glue (/root/.cache/huggingface/datasets/glue/mrpc/0.0.0/654741f6b0be315d)
Loading Dataset info from /root/.cache/huggingface/datasets/glue/mrpc/0.0.0/654741f6b0be315d
07/11/2024 21:28:30 - INFO - datasets.info - Loading Dataset info from /root/.cache/huggingface/datasets/glue/mrpc/0.0.0/654741f6b0be315d
[INFO|configuration_utils.py:731] 2024-07-11 21:28:30,296 >> loading configuration file bert-base-cased/config.json                                                                                                                                   
[INFO|configuration_utils.py:800] 2024-07-11 21:28:30,299 >> Model config BertConfig {
  "_name_or_path": "bert-base-cased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "finetuning_task": "mrpc",
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",                                                                                                                                                                                                                                          "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.43.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 28996
}
[INFO|tokenization_utils_base.py:2159] 2024-07-11 21:28:30,300 >> loading file vocab.txt
[INFO|tokenization_utils_base.py:2159] 2024-07-11 21:28:30,300 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2159] 2024-07-11 21:28:30,300 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2159] 2024-07-11 21:28:30,300 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2159] 2024-07-11 21:28:30,300 >> loading file tokenizer_config.json
[INFO|configuration_utils.py:731] 2024-07-11 21:28:30,300 >> loading configuration file bert-base-cased/config.json
[INFO|modeling_utils.py:3556] 2024-07-11 21:28:30,352 >> loading weights file bert-base-cased/model.safetensors
[INFO|modeling_utils.py:4357] 2024-07-11 21:28:30,601 >> Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predi
ctions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
07/11/2024 21:28:30 - INFO - datasets.arrow_dataset - Loading cached processed dataset at /root/.cache/huggingface/datasets/glue/mrpc/0.0.0/654741f6b0be315d/cache-df65166b4ebe49df.arrow
[WARNING|modeling_utils.py:4369] 2024-07-11 21:28:30,660 >> Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.                                                                                                                                                 07/11/2024 21:28:33 - INFO - __main__ - Sample 2619 of the training set: {'sentence1': 'The proceedings were taken up with prosecutors outlining their case against Amrozi , reading 33 pages of documents outlining allegations against him .', 'sentence2': '
Proceedings were taken up with prosecutors outlining their case against Amrozi , reading a 33-page accusation letter to the court .', 'label': 1, 'idx': 2916, 'input_ids': [101, 1109, 10830, 1127, 1678, 1146, 1114, 24987, 1149, 13260, 1147, 1692, 1222, 72
77, 2180, 5303, 117, 3455, 3081, 5097, 1104, 4961, 1149, 13260, 9966, 1222, 1140, 119, 102, 20661, 1127, 1678, 1146, 1114, 24987, 1149, 13260, 1147, 1692, 1222, 7277, 2180, 5303, 117, 3455, 170, 3081, 118, 3674, 21100, 2998, 1106, 1103, 2175, 119, 102, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
]}.
07/11/2024 21:28:33 - INFO - __main__ - Sample 456 of the training set: {'sentence1': "Chechen officials working for the Moscow-backed government are a frequent target for rebels and tension is running high ahead of next Sunday 's presidential election in
 war-torn Chechnya .", 'sentence2': "Officials in Chechnya 's Moscow-backed government are a frequent target for rebels , and tension is running high ahead of Sunday 's presidential election in the war-ravaged region .", 'label': 1, 'idx': 509, 'input_ids
': [101, 20394, 11252, 1424, 3878, 1684, 1111, 1103, 4116, 118, 5534, 1433, 1132, 170, 6539, 4010, 1111, 9283, 1105, 6646, 1110, 1919, 1344, 3075, 1104, 1397, 3625, 112, 188, 5200, 1728, 1107, 1594, 118, 7820, 20394, 11252, 15449, 119, 102, 9018, 1116, 11
07, 20394, 11252, 15449, 112, 188, 4116, 118, 5534, 1433, 1132, 170, 6539, 4010, 1111, 9283, 117, 1105, 6646, 1110, 1919, 1344, 3075, 1104, 3625, 112, 188, 5200, 1728, 1107, 1103, 1594, 118, 187, 15677, 3660, 1805, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}.
07/11/2024 21:28:33 - INFO - __main__ - Sample 102 of the training set: {'sentence1': "Standard & Poor 's 500 stock index futures declined 4.40 points to 983.50 , while Nasdaq futures fell 6.5 points to 1,206.50 .", 'sentence2': "The Standard & Poor 's 50
0 Index was up 1.75 points , or 0.18 percent , to 977.68 .", 'label': 0, 'idx': 116, 'input_ids': [101, 6433, 111, 11767, 112, 188, 2260, 4482, 7448, 2174, 1116, 5799, 125, 119, 1969, 1827, 1106, 5103, 1495, 119, 1851, 117, 1229, 11896, 1116, 1810, 4426,
2174, 1116, 2204, 127, 119, 126, 1827, 1106, 122, 117, 20278, 119, 1851, 119, 102, 1109, 6433, 111, 11767, 112, 188, 2260, 10146, 1108, 1146, 122, 119, 3453, 1827, 117, 1137, 121, 119, 1407, 3029, 117, 1106, 5311, 1559, 119, 5599, 119, 102, 0, 0, [80/857]
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}.
07/11/2024 21:28:52 - WARNING - accelerate.utils.other - Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
[INFO|trainer.py:813] 2024-07-11 21:28:53,708 >> The following columns in the training set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: sentence1, idx, sentence2. If sentence1, idx, sentence2 are no
t expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.
07/11/2024 21:28:54 - WARNING - evaluate.loading - Using the latest cached version of the module from /root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--glue/05234ba7acc44554edcca0978db5fa3bc600eeee66229abe79ff9887eacaf3ed (last mo
dified on Thu Jul 11 21:18:13 2024) since it couldn't be found locally at evaluate-metric--glue, or remotely on the Hugging Face Hub.
[INFO|trainer.py:2136] 2024-07-11 21:28:56,081 >> ***** Running training *****
[INFO|trainer.py:2137] 2024-07-11 21:28:56,081 >>   Num examples = 3,668
[INFO|trainer.py:2138] 2024-07-11 21:28:56,081 >>   Num Epochs = 3
[INFO|trainer.py:2139] 2024-07-11 21:28:56,081 >>   Instantaneous batch size per device = 32
[INFO|trainer.py:2142] 2024-07-11 21:28:56,081 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:2143] 2024-07-11 21:28:56,081 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2144] 2024-07-11 21:28:56,081 >>   Total optimization steps = 45
[INFO|trainer.py:2145] 2024-07-11 21:28:56,081 >>   Number of trainable parameters = 108,311,810
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 45/45 [00:33<00:00,  1.59it/s]
[INFO|trainer.py:3519] 2024-07-11 21:29:29,251 >> Saving model checkpoint to ./output/checkpoint-45
[INFO|configuration_utils.py:472] 2024-07-11 21:29:29,252 >> Configuration saved in ./output/checkpoint-45/config.json
[INFO|modeling_utils.py:2693] 2024-07-11 21:29:29,791 >> Model weights saved in ./output/checkpoint-45/model.safetensors
[INFO|tokenization_utils_base.py:2574] 2024-07-11 21:29:29,792 >> tokenizer config file saved in ./output/checkpoint-45/tokenizer_config.json
[INFO|tokenization_utils_base.py:2583] 2024-07-11 21:29:29,792 >> Special tokens file saved in ./output/checkpoint-45/special_tokens_map.json
[INFO|trainer.py:2391] 2024-07-11 21:29:30,757 >>

Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 34.6759, 'train_samples_per_second': 317.338, 'train_steps_per_second': 1.298, 'train_loss': 0.6095658196343317, 'epoch': 3.0}
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 45/45 [00:34<00:00,  1.30it/s]
[INFO|trainer.py:3519] 2024-07-11 21:29:30,762 >> Saving model checkpoint to ./output
[INFO|configuration_utils.py:472] 2024-07-11 21:29:30,763 >> Configuration saved in ./output/config.json                                                                                                                                                [0/857]
[INFO|modeling_utils.py:2693] 2024-07-11 21:29:31,249 >> Model weights saved in ./output/model.safetensors
[INFO|tokenization_utils_base.py:2574] 2024-07-11 21:29:31,250 >> tokenizer config file saved in ./output/tokenizer_config.json
[INFO|tokenization_utils_base.py:2583] 2024-07-11 21:29:31,250 >> Special tokens file saved in ./output/special_tokens_map.json
***** train metrics *****
  epoch                    =        3.0
  total_flos               =   705718GF
  train_loss               =     0.6096
  train_runtime            = 0:00:34.67
  train_samples            =       3668
  train_samples_per_second =    317.338
  train_steps_per_second   =      1.298
07/11/2024 21:29:31 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:813] 2024-07-11 21:29:31,260 >> The following columns in the evaluation set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: sentence1, idx, sentence2. If sentence1, idx, sentence2 are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.
[INFO|trainer.py:3835] 2024-07-11 21:29:31,262 >>
***** Running Evaluation *****
[INFO|trainer.py:3837] 2024-07-11 21:29:31,262 >>   Num examples = 408
[INFO|trainer.py:3840] 2024-07-11 21:29:31,262 >>   Batch size = 8
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 12.04it/s]
***** eval metrics *****
  epoch                   =        3.0
  eval_accuracy           =     0.7034
  eval_combined_score     =     0.7607
  eval_f1                 =      0.818
  eval_loss               =      0.552
  eval_runtime            = 0:00:00.65
  eval_samples            =        408
  eval_samples_per_second =    622.818
  eval_steps_per_second   =     10.686
```

Comments:
@SunMarc @muellerzr Hi guys, good day! Mind take a look at this PR, please?

@fmo-mt can you try rebasing from `main`?

> @fmo-mt can you try rebasing from `main`?

Thanks for reviewing! I rebased from main already, and it seems like we have to wait for CI update accelerate to the one MUSA was introduced?

@fmo-mt this is what I meant, we need to include a check for the minimum supported version of accelerate to use anything MUSA related. Since it is on `accelerate` main right now, this is `accelerate>=0.33.0`

> @fmo-mt this is what I meant, we need to include a check for the minimum supported version of accelerate to use anything MUSA related. Since it is on `accelerate` main right now, this is `accelerate>=0.33.0`

Ah, got it! I add a version check on `deps` (see `src/transformers/utils/import_utils.py:656`), looking forward to see the release of accelerate 0.33.0!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31913). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@fmo-mt if you do `pip install -e .[quality] -U; make fixup` it should fix the failing issue! 

> @fmo-mt if you do `pip install -e .[quality] -U; make fixup` it should fix the failing issue!

Okk! I just did that.

@muellerzr Hi! Is there anything that I need to do before this PR get merged?

Hey @fmo-mt, the only thing missing is to fix the comment from amy https://github.com/huggingface/transformers/pull/31913#discussion_r1699148209. Should we even check @muellerzr since we didn't do it for other backends ? 

@SunMarc yes we must, otherwise users running it with lower accelerate versions will get attribute errors and their code will break with no reason. So what we need to do here is return `False` if accelerate is below the minimum version. It'd be better to have a check somewhere (maybe the TrainingArguments) for these backends to ensure their minimum accelerate versions are installed.

> @SunMarc yes we must, otherwise users running it with lower accelerate versions will get attribute errors and their code will break with no reason. So what we need to do here is return `False` if accelerate is below the minimum version. It'd be better to have a check somewhere (maybe the TrainingArguments) for these backends to ensure their minimum accelerate versions are installed.

Ah, I got it, will fix it soon!

@muellerzr @SunMarc Hi, can you please take another look at this PR? Thanks!

Great job @fmo-mt !