Add support for Ascend NPU
Author: ji-huazhong
Pull request: yes

# What does this PR do?

This PR enables users to leverage the Ascend NPU for training and inference of 🤗 Transformers models.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #22600 
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 




Comments:
For example, you can run the official question answering task using Ascend NPU with below command:

```
python examples/pytorch/question-answering/run_qa.py
       --model_name_or_path bert-base-uncased \
       --dataset_name squad \
       --do_train \
       --do_eval \
       --device_id 5  \     // The specific device to be used for single card training on Ascend NPUs.
       --per_device_train_batch_size 24 \
       --num_train_epochs 2 \
       --learning_rate 3e-5 \
       --max_seq_length 384 \
       --doc_stride 128 \
       --save_steps 5000 \
       --fp16_opt_level O2 \
       --half_precision_backend apex \
       --dataloader_drop_last \
       --overwrite_output_dir \
       --output_dir ./output \
```

Below are the output logs:
```
04/07/2023 01:53:22 - WARNING - __main__ - Process rank: -1, device: npu:5, n_gpu: 1distributed training: False, 16-bits training: True
04/07/2023 01:53:22 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=True,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
device_id=5,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_steps=None,
evaluation_strategy=no,
fp16=True,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O2,
fsdp=[],
fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
greater_is_better=None,
group_by_length=False,
half_precision_backend=apex,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=3e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=-1,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=./output/runs/Apr07_01-53-21_localhost,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=steps,
lr_scheduler_type=linear,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
no_cuda=False,
num_train_epochs=2.0,
optim=adamw_hf,
optim_args=None,
output_dir=./output,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=24,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=[],
resume_from_checkpoint=None,
run_name=./output,
save_on_each_node=False,
save_safetensors=False,
save_steps=5000,
save_strategy=steps,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
xpu_backend=None,
)
04/07/2023 01:53:24 - INFO - datasets.builder - No config specified, defaulting to the single config: squad/plain_text
04/07/2023 01:53:24 - INFO - datasets.info - Loading Dataset Infos from /root/.cache/huggingface/modules/datasets_modules/datasets/squad/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453
04/07/2023 01:53:24 - INFO - datasets.builder - Overwrite dataset info from restored data version.
04/07/2023 01:53:24 - INFO - datasets.info - Loading Dataset info from /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453
04/07/2023 01:53:24 - WARNING - datasets.builder - Found cached dataset squad (/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)
04/07/2023 01:53:24 - INFO - datasets.info - Loading Dataset info from /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453

  0%|          | 0/2 [00:00<?, ?it/s]
100%|██████████| 2/2 [00:00<00:00, 196.27it/s]
[INFO|configuration_utils.py:668] 2023-04-07 01:53:24,715 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/0a6aa9128b6194f4f3c4db429b6cb4891cdb421b/config.json
[INFO|configuration_utils.py:720] 2023-04-07 01:53:24,723 >> Model config BertConfig {
  "_name_or_path": "bert-base-uncased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.28.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

[INFO|configuration_utils.py:668] 2023-04-07 01:53:25,029 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/0a6aa9128b6194f4f3c4db429b6cb4891cdb421b/config.json
[INFO|configuration_utils.py:720] 2023-04-07 01:53:25,033 >> Model config BertConfig {
  "_name_or_path": "bert-base-uncased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.28.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

[INFO|tokenization_utils_base.py:1809] 2023-04-07 01:53:25,035 >> loading file vocab.txt from cache at /root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/0a6aa9128b6194f4f3c4db429b6cb4891cdb421b/vocab.txt
[INFO|tokenization_utils_base.py:1809] 2023-04-07 01:53:25,035 >> loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/0a6aa9128b6194f4f3c4db429b6cb4891cdb421b/tokenizer.json
[INFO|tokenization_utils_base.py:1809] 2023-04-07 01:53:25,035 >> loading file added_tokens.json from cache at None
[INFO|tokenization_utils_base.py:1809] 2023-04-07 01:53:25,035 >> loading file special_tokens_map.json from cache at None
[INFO|tokenization_utils_base.py:1809] 2023-04-07 01:53:25,035 >> loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/0a6aa9128b6194f4f3c4db429b6cb4891cdb421b/tokenizer_config.json
[INFO|configuration_utils.py:668] 2023-04-07 01:53:25,036 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/0a6aa9128b6194f4f3c4db429b6cb4891cdb421b/config.json
[INFO|configuration_utils.py:720] 2023-04-07 01:53:25,037 >> Model config BertConfig {
  "_name_or_path": "bert-base-uncased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.28.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

[INFO|modeling_utils.py:2478] 2023-04-07 01:53:25,108 >> loading weights file pytorch_model.bin from cache at /root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/0a6aa9128b6194f4f3c4db429b6cb4891cdb421b/pytorch_model.bin
[WARNING|modeling_utils.py:3118] 2023-04-07 01:53:27,033 >> Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[WARNING|modeling_utils.py:3130] 2023-04-07 01:53:27,034 >> Some weights of BertForQuestionAnswering were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
04/07/2023 01:53:27 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453/cache-d1f3bae3544867f1.arrow
04/07/2023 01:53:27 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453/cache-7bcc744f960ea416.arrow
[INFO|trainer.py:621] 2023-04-07 01:53:28,908 >> Using apex half precision backend
[INFO|trainer.py:1766] 2023-04-07 01:53:30,199 >> ***** Running training *****
[INFO|trainer.py:1767] 2023-04-07 01:53:30,199 >>   Num examples = 88,524
[INFO|trainer.py:1768] 2023-04-07 01:53:30,199 >>   Num Epochs = 2
[INFO|trainer.py:1769] 2023-04-07 01:53:30,200 >>   Instantaneous batch size per device = 24
[INFO|trainer.py:1770] 2023-04-07 01:53:30,200 >>   Total train batch size (w. parallel, distributed & accumulation) = 24
[INFO|trainer.py:1771] 2023-04-07 01:53:30,200 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1772] 2023-04-07 01:53:30,200 >>   Total optimization steps = 7,376
[INFO|trainer.py:1773] 2023-04-07 01:53:30,203 >>   Number of trainable parameters = 108,893,186
Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.

Defaults for this optimization level are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
combine_grad           : None
combine_ddp            : None
ddp_replica_count      : 4
check_combined_tensors : None
user_cast_preferred    : None
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
combine_grad           : None
combine_ddp            : None
ddp_replica_count      : 4
check_combined_tensors : None
user_cast_preferred    : None

100%|██████████| 7376/7376 [37:49<00:00,  3.25it/s]

[INFO|trainer.py:2865] 2023-04-07 02:31:19,790 >> Saving model checkpoint to ./output
[INFO|configuration_utils.py:457] 2023-04-07 02:31:19,793 >> Configuration saved in ./output/config.json
[INFO|modeling_utils.py:1839] 2023-04-07 02:31:21,077 >> Model weights saved in ./output/pytorch_model.bin
[INFO|tokenization_utils_base.py:2170] 2023-04-07 02:31:21,079 >> tokenizer config file saved in ./output/tokenizer_config.json
[INFO|tokenization_utils_base.py:2177] 2023-04-07 02:31:21,080 >> Special tokens file saved in ./output/special_tokens_map.json
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
{'loss': 2.2345, 'learning_rate': 2.7966377440347073e-05, 'epoch': 0.14}
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
{'loss': 1.3912, 'learning_rate': 2.5932754880694143e-05, 'epoch': 0.27}
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
{'loss': 1.2321, 'learning_rate': 2.3899132321041215e-05, 'epoch': 0.41}
{'loss': 1.1749, 'learning_rate': 2.1865509761388288e-05, 'epoch': 0.54}
{'loss': 1.0975, 'learning_rate': 1.983188720173536e-05, 'epoch': 0.68}
{'loss': 1.0988, 'learning_rate': 1.779826464208243e-05, 'epoch': 0.81}
{'loss': 1.0514, 'learning_rate': 1.5764642082429502e-05, 'epoch': 0.95}
{'loss': 0.8971, 'learning_rate': 1.3731019522776571e-05, 'epoch': 1.08}
{'loss': 0.7757, 'learning_rate': 1.1697396963123646e-05, 'epoch': 1.22}
{'loss': 0.7823, 'learning_rate': 9.663774403470717e-06, 'epoch': 1.36}
{'loss': 0.7851, 'learning_rate': 7.630151843817788e-06, 'epoch': 1.49}
{'loss': 0.7617, 'learning_rate': 5.5965292841648585e-06, 'epoch': 1.63}
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
{'loss': 0.7459, 'learning_rate': 3.5629067245119307e-06, 'epoch': 1.76}
{'loss': 0.7529, 'learning_rate': 1.529284164859002e-06, 'epoch': 1.9}
{'train_runtime': 2269.5834, 'train_samples_per_second': 78.009, 'train_steps_per_second': 3.25, 'train_loss': 1.0397178831948635, 'epoch': 2.0}
***** train metrics *****
  epoch                    =        2.0
  train_loss               =     1.0397
  train_runtime            = 0:37:49.58
  train_samples            =      88524
  train_samples_per_second =     78.009
  train_steps_per_second   =       3.25
04/07/2023 02:31:21 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:763] 2023-04-07 02:31:21,152 >> The following columns in the evaluation set don't have a corresponding argument in `BertForQuestionAnswering.forward` and have been ignored: example_id, offset_mapping. If example_id, offset_mapping are not expected by `BertForQuestionAnswering.forward`,  you can safely ignore this message.
[INFO|trainer.py:3126] 2023-04-07 02:31:21,156 >> ***** Running Evaluation *****
[INFO|trainer.py:3128] 2023-04-07 02:31:21,157 >>   Num examples = 10784
[INFO|trainer.py:3131] 2023-04-07 02:31:21,157 >>   Batch size = 8

100%|██████████| 1348/1348 [01:07<00:00, 26.69it/s]04/07/2023 02:32:40 - INFO - utils_qa - Post-processing 10570 example predictions split into 10784 features.

100%|██████████| 10570/10570 [01:09<00:00, 152.04it/s]04/07/2023 02:33:50 - INFO - utils_qa - Saving predictions to ./output/eval_predictions.json.
04/07/2023 02:33:50 - INFO - utils_qa - Saving nbest_preds to ./output/eval_nbest_predictions.json.

100%|██████████| 1348/1348 [02:40<00:00,  8.37it/s]
[INFO|modelcard.py:451] 2023-04-07 02:34:02,898 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Question Answering', 'type': 'question-answering'}, 'dataset': {'name': 'squad', 'type': 'squad', 'config': 'plain_text', 'split': 'validation', 'args': 'plain_text'}}
***** eval metrics *****
  epoch                   =        2.0
  eval_exact_match        =    80.0946
  eval_f1                 =     87.853
  eval_runtime            = 0:00:51.51
  eval_samples            =      10784
  eval_samples_per_second =    209.317
  eval_steps_per_second   =     26.165
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22644). All of your documentation changes will be reflected on that endpoint.

The test case needs to be executed on the Ascend NPU, and the results are shown below:
![image](https://user-images.githubusercontent.com/28150734/230536813-d8457a5d-c73c-4165-9344-9433bc154811.png)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@sgugger now if i want to use Ascend NPU and accelerate to complete distributed training.  how can i start and is there any examples for reference

The PR hasn't been moved there to add support for NPUs, so for now it's not possible.

@sgugger Currently, I can use transformers on npu based on this PR, but I find that accelerate cannot be used. What should I do if I want to use accelerate on npu? Is there any reference?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.