device agnostic extended testing
Author: ji-huazhong
Pull request: yes

# What does this PR do?




## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
- verified on mac studio m2 with instruction:
  ```
  RUN_SLOW=1 TRANSFORMERS_BACKEND_DEVICE="mps" TRANSFORMERS_BACKEND_DEVICE_SPEC="spec.py" python -m pytest -v tests/extended

  ```
  `spec.py`:
  ```
  import torch
  
  DEVICE_NAME = 'mps'
  
  # Specify device-specific backends to dispatch to.
  # If not specified, will fallback to 'default' in 'testing_utils.py`
  MANUAL_SEED_FN = torch.mps.manual_seed
  EMPTY_CACHE_FN = torch.mps.empty_cache
  ```

- the output:
  ```
  ============================================================================================================================= test session starts ==============================================================================================================================
  platform darwin -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0 -- /opt/homebrew/Caskroom/miniconda/base/envs/hf/bin/python
  cachedir: .pytest_cache
  hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/Users/yun/github/transformers/.hypothesis/examples'))
  rootdir: /Users/yun/github/transformers
  configfile: setup.cfg
  plugins: hypothesis-6.88.1, dash-2.14.1, timeout-2.2.0, xdist-3.3.1
  collected 10 items                                                                                                                                                                                                                                                             
  
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq 
  PASSED                                                                                                                                                                                              [ 10%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_apex SKIPPED (test requires apex)                                                                                                                                                                   [ 20%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_bnb SKIPPED (test requires bnb)                                                                                                                                                                     [ 30%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_ddp SKIPPED (test requires multiple accelerators)                                                                                                                                                   [ 40%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_dp SKIPPED (test requires multiple accelerators)                                                                                                                                                    [ 50%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_no_dist PASSED                                                                                                                                                                                      [ 60%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_trainer_log_level_replica_0_base SKIPPED (test requires multiple accelerators)                                                                                                                                  [ 70%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_trainer_log_level_replica_1_low SKIPPED (test requires multiple accelerators)                                                                                                                                   [ 80%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_trainer_log_level_replica_2_high SKIPPED (test requires multiple accelerators)                                                                                                                                  [ 90%]
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_trainer_log_level_replica_3_mixed SKIPPED (test requires multiple accelerators)                                                                                                                                 [100%]
  
  =============================================================================================================================== warnings summary ===============================================================================================================================
  ../../../../opt/homebrew/Caskroom/miniconda/base/envs/hf/lib/python3.10/site-packages/_pytest/config/__init__.py:1373
    /opt/homebrew/Caskroom/miniconda/base/envs/hf/lib/python3.10/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
    
      self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
  
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_no_dist
    /Users/yun/github/transformers/src/transformers/training_args.py:1392: FutureWarning: `--adafactor` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--optim adafactor` instead
      warnings.warn(
  
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq
    /opt/homebrew/Caskroom/miniconda/base/envs/hf/lib/python3.10/site-packages/codecarbon/input.py:9: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      import pkg_resources
  
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_no_dist
    /Users/yun/github/transformers/src/transformers/generation/utils.py:1473: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use and modify the model generation configuration (see https://huggingface.co/docs/transformers/generation_strategies#default-text-generation-configuration )
      warnings.warn(
  
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_no_dist
  tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_no_dist
    /Users/yun/github/transformers/src/transformers/generation/utils.py:1273: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.
      warnings.warn(
  
  -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
  ============================================================================================================= 2 passed, 8 skipped, 7 warnings in 564.29s (0:09:24) =============================================================================================================

  
  ```