device-agnostic deepspeed testing
Author: ji-huazhong
Pull request: yes

# What does this PR do?

Part of https://github.com/huggingface/transformers/issues/25654#issuecomment-1783704306

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Verified with Ascend NPU via the following `spec.py` file:
```python
import torch
import torch_npu
# !! Further additional imports can be added here !!
# Specify the device name (eg. 'cuda', 'cpu', 'npu')
DEVICE_NAME = 'npu:0'
# Specify device-specific backends to dispatch to.
# If not specified, will fallback to 'default' in 'testing_utils.py`
MANUAL_SEED_FN = torch.npu.manual_seed_all
EMPTY_CACHE_FN = torch.npu.empty_cache
DEVICE_COUNT_FN = torch.npu.device_count
```
test results:
```text
(ds) [root@node-43 transformers]# RUN_SLOW=1 TRANSFORMERS_TEST_BACKEND="torch_npu" TRANSFORMERS_TEST_DEVICE="npu:0" TRANSFORMERS_TEST_DEVICE_SPEC="spec.py" python -m pytest -v -k "not bf16"  tests/deepspeed/test_deepspeed.py
=============================================================================================================================== test session starts ================================================================================================================================
platform linux -- Python 3.8.18, pytest-7.4.3, pluggy-1.3.0 -- /home/miniconda3/envs/ds/bin/python
cachedir: .pytest_cache
rootdir: /home/ds/transformers
configfile: setup.cfg
collected 81 items / 35 deselected / 46 selected

tests/deepspeed/test_deepspeed.py::CoreIntegrationDeepSpeed::test_init_zero3_fp16 PASSED                                                                                                                                                                                     [  2%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_errors_zero2_fp16 PASSED                                                                                                                                                            [  4%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_errors_zero3_fp16 PASSED                                                                                                                                                            [  6%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16_ds_optim_ds_scheduler PASSED                                                                                                                                      [  8%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16_ds_optim_hf_scheduler PASSED                                                                                                                                      [ 10%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16_hf_optim_ds_scheduler PASSED                                                                                                                                      [ 13%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16_hf_optim_hf_scheduler PASSED                                                                                                                                      [ 15%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16_ds_optim_ds_scheduler PASSED                                                                                                                                      [ 17%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16_ds_optim_hf_scheduler PASSED                                                                                                                                      [ 19%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16_hf_optim_ds_scheduler PASSED                                                                                                                                      [ 21%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero3_fp16_hf_optim_hf_scheduler PASSED                                                                                                                                      [ 23%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_config_object PASSED                                                                                                                                                                                    [ 26%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_ds_scheduler_hf_optimizer PASSED                                                                                                                                                                        [ 28%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_early_get_last_lr_zero2_fp16 PASSED                                                                                                                                                                     [ 30%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_early_get_last_lr_zero3_fp16 PASSED                                                                                                                                                                     [ 32%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_fake_notebook_no_launcher_zero2_fp16 PASSED                                                                                                                                                             [ 34%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_fake_notebook_no_launcher_zero3_fp16 PASSED                                                                                                                                                             [ 36%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_gradient_accumulation_zero2_fp16 PASSED                                                                                                                                                                 [ 39%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_gradient_accumulation_zero3_fp16 PASSED                                                                                                                                                                 [ 41%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_hf_ds_config_mismatch PASSED                                                                                                                                                                            [ 43%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_hf_optimizer_with_offload_zero2_fp16 PASSED                                                                                                                                                             [ 45%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_hf_optimizer_with_offload_zero3_fp16 PASSED                                                                                                                                                             [ 47%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_hf_scheduler_ds_optimizer PASSED                                                                                                                                                                        [ 50%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_hf_scheduler_hf_optimizer PASSED                                                                                                                                                                        [ 52%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_hyperparameter_search SKIPPED (test requires optuna)                                                                                                                                                    [ 54%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_load_best_model_zero2_fp16 PASSED                                                                                                                                                                       [ 56%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_load_best_model_zero3_fp16 PASSED                                                                                                                                                                       [ 58%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_load_state_dict_from_zero_checkpoint_zero2_fp16 PASSED                                                                                                                                                  [ 60%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_load_state_dict_from_zero_checkpoint_zero3_fp16 PASSED                                                                                                                                                  [ 63%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_save_checkpoints_zero2_fp16 PASSED                                                                                                                                                                      [ 65%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_save_checkpoints_zero3_fp16 PASSED                                                                                                                                                                      [ 67%]
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_stage3_nvme_offload SKIPPED (test requires deepspeed async-io)                                                                                                                                          [ 69%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_basic_distributed_zero2_fp16 PASSED                                                                                                                                                                       [ 71%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_basic_distributed_zero3_fp16 PASSED                                                                                                                                                                       [ 73%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_clm_from_config_zero3_fp16 PASSED                                                                                                                                                                         [ 76%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_clm_zero2_fp16 PASSED                                                                                                                                                                                     [ 78%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_clm_zero3_fp16 PASSED                                                                                                                                                                                     [ 80%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_do_eval_no_train PASSED                                                                                                                                                                                   [ 82%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_distributed_zero2_fp16 PASSED                                                                                                                                                                        [ 84%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_distributed_zero3_fp16 PASSED                                                                                                                                                                        [ 86%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_non_distributed_zero2_fp16 PASSED                                                                                                                                                                    [ 89%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_fp32_non_distributed_zero3_fp16 PASSED                                                                                                                                                                    [ 91%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_inference_1_fp16 PASSED                                                                                                                                                                                   [ 93%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_inference_2_fp32 PASSED                                                                                                                                                                                   [ 95%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_resume_train_not_from_ds_checkpoint_zero2_fp16 PASSED                                                                                                                                                     [ 97%]
tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_resume_train_not_from_ds_checkpoint_zero3_fp16 PASSED                                                                                                                                                     [100%]

================================================================================================================================= warnings summary =================================================================================================================================
../../../miniconda3/envs/ds/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18
  /home/miniconda3/envs/ds/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
    warnings.warn(

../../../miniconda3/envs/ds/lib/python3.8/site-packages/_pytest/config/__init__.py:1373
  /home/miniconda3/envs/ds/lib/python3.8/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests/deepspeed/test_deepspeed.py::CoreIntegrationDeepSpeed::test_init_zero3_fp16
  /home/miniconda3/envs/ds/lib/python3.8/site-packages/deepspeed-0.12.3+6d33acc2-py3.8.egg/deepspeed/comm/comm.py:163: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    utils.logger.warn("HCCL backend in DeepSpeed not yet implemented")

tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_errors_zero2_fp16
  /home/miniconda3/envs/ds/lib/python3.8/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]

tests/deepspeed/test_deepspeed.py: 12 warnings
  /home/miniconda3/envs/ds/lib/python3.8/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
    warnings.warn(

tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_load_best_model_zero2_fp16
  /home/ds/transformers/src/transformers/tokenization_utils_base.py:2614: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
    warnings.warn(

tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_load_best_model_zero2_fp16
  /home/ds/transformers/src/transformers/models/t5/modeling_t5.py:893: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at torch_npu/csrc/aten/common/TensorFactories.cpp:74.)
    shifted_input_ids[..., 0] = decoder_start_token_id

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================================================================================== 44 passed, 2 skipped, 35 deselected, 21 warnings in 1176.45s (0:19:36) ======================================================================================================
```


@ydshieh please take a look at this commit :-)

Thanks a lot @statelesshz ! Great there is no failure at all 💯 

> Thanks a lot @statelesshz ! Great there is no failure at all 💯

To be honest, I just skipped some bf16-related test cases because npu's support for bf16 is not comparable to that of gpu. 😅 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27342). All of your documentation changes will be reflected on that endpoint.