Device agnostic testing
Author: vvvm23
Pull request: yes

# What does this PR do?

Adds extra capabilities to `testing_utils.py` to support testing on devices besides from `cuda` and `cpu` without having to upstream device-specific changes.

This involves introducing some device agnostic functions that dispatch to specific backend functions. Users can specify new backends and backends for device agnostic functions via creating a device specification file and pointing the test suite to it using `TRANSFORMERS_TEST_DEVICE_SPEC`.

An example specification for a hypothetical CUDA device without support for `torch.cuda.empty_cache` could look like this:
```python
import torch

# !! Specify additional imports here !!

# Specify the device name (eg. 'cuda', 'cpu')
DEVICE_NAME = 'cuda2'

# Specify device-specific backends to dispatch to.
# If not specified, will fallback to 'default' in 'testing_utils.py`
MANUAL_SEED_FN = torch.cuda.manual_seed
EMPTY_CACHE_FN = None
DEVICE_COUNT_FN = torch.cuda.device_count
```

By default, we have `cpu` and `cuda` backends available, so not to affect default behaviour.

We also introduce a new decorator `@require_torch_accelerator` which can be used to specify that a test needs an accelerator (but not necessarily a CUDA one).

Crucially, these changes _should_ not change the behaviour of upstream CI runners. They aim to be as non-intrusive as possible and do not break compatibility with tests before these changes are made.

In this PR, only a subset of all tests are updated to support these new features at first. These are:
- `test_modeling_bloom` – demonstrating usage of new `@require_torch_accelerator`
- `test_modeling_codegen` – demonstrating usage of device agnostic function (`accelerator_manual-seed`)
- `test_modeling_opt` – demonstrating another device agnostic function, this time to check whether the current device supports `torch.float16`
- `test_modeling_reformer` – decorator version of the above. 

Related #25654 

TODO:
- [ ] Write some documentation on `TRANSFORMERS_TEST_DEVICE_SPEC` (once we finalise the PR)
- [ ] Additional checks and finding edge cases
- [ ] Verify this PR does indeed have no effect on the Huggingface CI runners.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ydshieh 

Comments:
Thanks, I will  take a look @vvvm23 

@vvvm23 Could you pull the latest main to your local clone and rebase your PR branch on top of your local `main`? Thanks!

I rebased a few days ago, but realised I forgot to ping! Sorry @ydshieh!

No problem, I will take a look this week 🙏 

Hey @ydshieh, just writing to let you know that @arsalanu will be picking up this PR on my behalf as I am changing jobs.

Please let me or him know if you have any further review comments 🤗 

@vvvm23 well noted. Thank you for the contribution, and best wishes for your next adventure!

(I was just back from a break, will take a look and pin @arampacha for necessary changes if any)

Hi, as Alex mentioned, I’ll be taking over this PR. Just wanted to check in on the status of the review, please let me know when you can if there are any comments/further changes you’d like made 🙂

Hi @arsalanu 

Toward the end of `src/transformers/testing_utils.py`: I think it would be less confusing if we use `backend_` instead of `accelerator_` (and similarly for  `ACCELERATOR_`) 

For example, instead of `ACCELERATOR_MANUAL_SEED` and `accelerator_manual_seed`,  use the name `BACKEND_MANUAL_SEED` and `backend_manual_seed`. 

If we check `require_torch_accelerator` excludes `cpu`, so it means we don't consider `cpu` as an accelerator. So it's somehow strange we can use `accelerator_manual_seed` with CPU.

And those methods are actually method for torch backends.

WDYT?

Other than this, it looks good to me. It would be great to see an example run on `npu` backend/device and make sure this PR works with it.

Finally, we should update the documentation here

https://github.com/huggingface/transformers/blob/b5ca8fcd20da3ed6aa562ca926c4c8e2c56fe6a4/docs/source/en/testing.md?plain=1#L514

In this PR, we don't have to apply all the new things like `accelerator_manual_seed` or `require_torch_accelerator` everywhere. We can make it simple, make it works as expected and merge (after approved by a core maintainer). Then we can apply the changes in a follow up PR.



Hi, I've updated the PR to rename the functions, I agree that `backend_*` makes more sense here. Also updated the docs to include an explanation of using the spec file.

On the example run, what is sufficient for showing the changes are working with a different backend?

We have functional tests with these changes running on a custom backend, but we are not able to share details about the backend itself, but we could, for example, provide some CI runner outputs with backend names redacted.

> On the example run, what is sufficient for showing the changes are working with a different backend?

If you have access to a npu machine, maybe just run a test (that is ) with that device/backend , and showing the device name?
Say `test_generate_fp16` in `tests/models/opt/test_modeling_opt.py`? Otherwise, probably just showing the device name doesn't contain `cuda` or `cpu`.


Hi @ydshieh, I have a [Colab notebook](https://colab.research.google.com/drive/1F4dWfy_BCA5koUZwrCGC-xxOnxmymcwR?usp=sharing) to show the changes working with a custom backend, for demo purposes I used XLA with a TPU machine as this is publicly available. The notebook runtime has our fork pointed to this branch, with an XLA-specific device specification file. I run `pytest` for just `test_generate_fp16` in `test_modeling_opt.py`. 

The XLA backend fails with an op support issue, but we just want confirmation that the test is running on the specified device, you can take a look at the pytest output in the notebook for evidence, which shows the backend-specific error:
```
RuntimeError: torch_xla/csrc/convert_ops.cpp:86 : Unsupported XLA type 10
```

Could you share the content of `spec.py` you used in colab? Thank you

Sure! This is what `spec.py` looks like for the example:

```
import torch
import torch_xla
import torch_xla.core.xla_model as xm

DEVICE_NAME=str(xm.xla_device())

MANUAL_SEED_FN = xm.set_rng_state
DEVICE_COUNT_FN = None
EMPTY_CACHE_FN = None
```

I believe the contents should be visible in the notebook outputs as well as I `cat` the `spec.py` file in the cell just before I run `pytest` (second-last cell)

Hi @arsalanu . I haven't been able to try with TPU, as it always complains about no backend available. It doesn't matter much.
I see this PR is on top of a very old commit on the main branch. Could you pull the latest main and rebase this PR on top of the latest main?

I will request a review from a core maintainer tomorrow. Thank you in advance!

Hi @ydshieh I've tested this amazing work on Ascend NPU. Just refer to the Colab notebook https://github.com/huggingface/transformers/pull/25870#issuecomment-1747121445 shared by @arsalanu for testing.  Here is the `spec.py`  I used:
```
import torch
import torch_npu
# !! Further additional imports can be added here !!
# Specify the device name (eg. 'cuda', 'cpu', 'npu')
DEVICE_NAME = 'npu'
# Specify device-specific backends to dispatch to.
# If not specified, will fallback to 'default' in 'testing_utils.py`
MANUAL_SEED_FN = torch.npu.manual_seed_all
EMPTY_CACHE_FN = torch.npu.empty_cache
DEVICE_COUNT_FN = torch.npu.device_count
```

And when I do the following test instruction:
```
TRANSFORMERS_TEST_BACKEND="torch_npu" TRANSFORMERS_TEST_DEVICE="npu"  TRANSFORMERS_TEST_DEVICE_SPEC="spec.py" python3 -m pytest tests/models/opt/test_modeling_opt.py -k "test_generate_fp16"
```

It complains a `Unknown testing device` error
```
ImportError while loading conftest '/data/hf_test/transformers/conftest.py'.
conftest.py:25: in <module>
    from transformers.testing_utils import HfDoctestModule, HfDocTestParser
src/transformers/testing_utils.py:724: in <module>
    raise RuntimeError(
E   RuntimeError: Unknown testing device specified by environment variable `TRANSFORMERS_TEST_DEVICE`: npu
```
This is because we need to register test backend first before using test device of third-party accelerators, so I make a small change, see https://github.com/huggingface/transformers/pull/26708
and everything is fine now :-)

The output log is as following：
```
(hf_test) [root@localhost transformers]# TRANSFORMERS_TEST_BACKEND="torch_npu"  TRANSFORMERS_TEST_DEVICE="npu" TRANSFORMERS_TEST_DEVICE_SPEC="spec.py" python3 -m pytest tests/models/opt/test_modeling_opt.py -k "test_generate_fp16"
=================================================== test session starts ====================================================
platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /data/hf_test/transformers
configfile: setup.cfg
plugins: xdist-3.3.1, timeout-2.2.0
collected 124 items / 123 deselected / 1 selected                                                                          

tests/models/opt/test_modeling_opt.py .                                                                              [100%]

===================================================== warnings summary =====================================================
../../anaconda/envs/hf_test/lib/python3.8/site-packages/_pytest/config/__init__.py:1373
  /data/anaconda/envs/hf_test/lib/python3.8/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests/test_modeling_common.py:2746
  /data/hf_test/transformers/tests/test_modeling_common.py:2746: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2773
  /data/hf_test/transformers/tests/test_modeling_common.py:2773: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2815
  /data/hf_test/transformers/tests/test_modeling_common.py:2815: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2857
  /data/hf_test/transformers/tests/test_modeling_common.py:2857: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2894
  /data/hf_test/transformers/tests/test_modeling_common.py:2894: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

tests/test_modeling_common.py:2931
  /data/hf_test/transformers/tests/test_modeling_common.py:2931: PytestUnknownMarkWarning: Unknown pytest.mark.flash_attn_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.flash_attn_test

../../anaconda/envs/hf_test/lib/python3.8/site-packages/torch/utils/cpp_extension.py:28
  /data/anaconda/envs/hf_test/lib/python3.8/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]

../../anaconda/envs/hf_test/lib/python3.8/site-packages/pkg_resources/__init__.py:2871
  /data/anaconda/envs/hf_test/lib/python3.8/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

tests/models/opt/test_modeling_opt.py::OPTModelTest::test_generate_fp16
  /data/hf_test/transformers/tests/models/opt/test_modeling_opt.py:116: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/_internal/cpython-3.8.17/lib/python3.8/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h:74.)
    input_ids[:, -1] = self.eos_token_id  # Eos Token

tests/models/opt/test_modeling_opt.py::OPTModelTest::test_generate_fp16
  /data/hf_test/transformers/src/transformers/generation/utils.py:1260: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================== 1 passed, 123 deselected, 11 warnings in 15.02s =====================================
```







@statelesshz 

Thanks, LGTM, but would like @arsalanu to double check it. Is it OK if we include your change in this PR once they confirm (we will add you as a contributor).

Hi @arsalanu, please take a look at my result https://github.com/huggingface/transformers/pull/25870#issuecomment-1754523487 thx -:)

Hi, thanks for your comments @statelesshz, @ydshieh  and apologies for the delay, I was on vacation last week.

I have made the requested changes to the docs to reflect the usage of the `npu` backend and rebased on main, and thank you for testing it. It makes sense to register and import the backend before using the device. #26708 LGTM 👍 

I have not yet copied the changes from that PR into this one, if you confirm its OK with you I can make your changes here.

P.S. I can't resolve conversations on this PR as I don't have write access to the library and am not the original author of the PR, please feel free to do so if scrolling down is getting annoying 😄 

Hi  @statelesshz, thank you for explanation. Is it OK if @statelesshz include your work in #26708 in this PR? (Will put you as a coauthors of course).

 @ydshieh  Yes, of course 😄 But I'm not sure how to include https://github.com/huggingface/transformers/pull/26708  into this PR

I mean for @arsalanu to include your work - typo 😅 

I will manually include the change, I believe when this is merged we can add you @statelesshz as a contributer in this PR? @ydshieh I am not sure but are you able to do this?

Yes, I only need to add an entry in the merge message when I click the merge button.

Thanks, @statelesshz - I've made your changes to this PR. 

@ydshieh when you can, could you please change this PR from a 'draft' to ready-for-review if you feel it is ready to be approved? (I don't have the access to do this) Thank you! 

Sure. BTW, could you run `make style` and/or `make quality` to fix the code quality issue.
Or more simply,
```
black --check examples tests src utils
```

Looks like they're all passing now 👍 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25870). All of your documentation changes will be reflected on that endpoint.

Hi @LysandreJik 

This PR is ready for a review from core maintainers 🙏 . It enables the testing on different type of accelerators. 

The only thing I see that might be a bit inconvenience is the file `testing_utils.py` might get (even) larger if we ever need to add more device specific methods: see below

If this is the case, we can move the actual definitions to a new file and import them from there to `testing_utils`.

https://github.com/huggingface/transformers/blob/a1230d45c2a5e9ef89c832121273b780b1be7c3a/src/transformers/testing_utils.py#L2201-L2281

looking forward to this  PR being accepted asap 🤗

Thank you @LysandreJik, @fxmarty for the look-over. I'm just waiting for the tests to pass and then will merge.

@ydshieh (or @vvvm23!) I don't have the access to merge this. It looks ready to go, could one of you click the button? 

Reminder to include @statelesshz as a contributer :) 

I don't have the ability sadly 😅

Thank you again @vvvm23 @arsalanu and @statelesshz 

(and especially reminding me to add @statelesshz as coauthor!)

![image](https://github.com/huggingface/transformers/assets/44398246/f7f29205-367c-4197-b1bc-24a6a41a107e)
Email for @statelesshz seems right but name is wrong in the commit 🤔 

My bad 😭 sorry. But I think GitHub counts with the email, so @statelesshz still appears

<img width="320" alt="Screenshot 2023-10-24 202831" src="https://github.com/huggingface/transformers/assets/2521628/d8d70505-78e9-4abc-842b-97bd70493fd9">