[tests] add more tests to `NOT_DEVICE_TESTS` 
Author: faaany
Pull request: yes

## What does this PR do?
As a guy working on XPU, I found PR [#29213](https://github.com/huggingface/transformers/pull/29213) extremely useful, but I discovered more test cases that will not be tested on accelerators anyway.
The only concern I have is the "repository-skip": currently I find all test files under `repo_utils`, `tools` and `utils` repositories belong to `NOT_DEVICE_TESTS`, because these test files don't have the keyword `torch_device` nor `gpu`. But this doesn't guarantee that people will add DEVICE-relevant tests to these repositories in the future. What are your thoughts on this?
[@amyeroberts](https://github.com/amyeroberts) [@fxmarty](https://github.com/fxmarty) [@ydshieh](https://github.com/ydshieh)

Comments:
Thank you @faaany, happy this is helpful to you.

> The only concern I have is the "repository-skip": currently I find all test files under repo_utils, tools and utils repositories belong to NOT_DEVICE_TESTS, because these test files don't have the keyword torch_device nor gpu. But this doesn't guarantee that people will add DEVICE-relevant tests to these repositories in the future. What are your thoughts on this?

It is a fair point, I think for these three folders it is fine but for others tests listed there it could be that they are modified in the future to use `torch_device`.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29670). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thank you @faaany, happy this is helpful to you.
> 
> > The only concern I have is the "repository-skip": currently I find all test files under repo_utils, tools and utils repositories belong to NOT_DEVICE_TESTS, because these test files don't have the keyword torch_device nor gpu. But this doesn't guarantee that people will add DEVICE-relevant tests to these repositories in the future. What are your thoughts on this?
> 
> It is a fair point, I think for these three folders it is fine but for others tests listed there it could be that they are modified in the future to use `torch_device`.

cool! Thanks for the hint. I removed the 2 tests.  But today I discovered more: `test_trainer_utils.py`, `test_configuration_utils.py`, `test_data_collator.py`, `test_beam_constraints.py`, `test_trainer_callback.py`. I think `test_trainer_utils.py` and `test_configuration_utils.py` should be safe as well. What do you think? 

@faaany looking at these files, I think yes it is fine to add them to the list.

About `test_image_transforms` and `test_optimization`: sorry my message was unclear, these are fine to add too. I just meant that in general some tests that currently do not use `torch_device` may use it in the future, which may be an issue to keep the list up to date.

> @faaany looking at these files, I think yes it is fine to add them to the list.
> 
> About `test_image_transforms` and `test_optimization`: sorry my message was unclear, these are fine to add too. I just meant that in general some tests that currently do not use `torch_device` may use it in the future, which may be an issue to keep the list up to date.

Yeah, I see your point. This is the drawback of a whitelist. But for now, this method is simple and effective. If the need becomes higher in the future, we can consider optimizing it. PR updated, pls have a review. Thanks a lot!  

I am wondering if `NOT_DEVICE_TESTS` and `not_device_test` really matter. They are only 4 occurrence, and all of them are in the file `conftest.py`.

And I don't see any usage of `pytest -v -m not_device_test`.

When does one need to use this marker and why?

> I am wondering if `NOT_DEVICE_TESTS` and `not_device_test` really matter. They are only 4 occurrence, and all of them are in the file `conftest.py`.
> 
> And I don't see any usage of `pytest -v -m not_device_test`.
> 
> When does one need to use this marker and why?

the usage of `pytest -v -m not_device_test` doesn't make a lot of sense, but `pytest -v -m "not not_device_test"` does, because it filters out a lot of CPU-only tests (in other words, DEVICE-irrelevant tests) for guys who work on specific DEVICE. For example, I am working on XPU and I am only interested in those unit tests that will run on my XPU device. Without this maker, I will need to run >50000 tests; With this maker, I only have to run ~20000 tests. 



@ydshieh I introduced it in https://github.com/huggingface/transformers/pull/29213 (explained there) as per AMD request, that also wanted to be able to run only the tests that were on device (in their case: cuda device) so as to make their CI significantly faster.

Thanks for explaining!