Enable tests to run on third-party devcies
Author: ji-huazhong
Pull request: yes

### What does this PR do?
It is crucial to enable unit tests to ensure the smooth integration of third-party devices. This PR enables ~~MPS and~~ NPU to reuse most existing test cases.






Comments:
_The documentation is not available anymore as the PR was closed or merged._

_The documentation is not available anymore as the PR was closed or merged._

Hi @statelesshz, thanks for opening this PR! 

Selecting e.g. "mps" by default if available is definitely something we're moving towards for e.g. [Trainer](https://github.com/huggingface/transformers/blob/b0f23036f16b961e14cf480522fef8581c4bf19c/src/transformers/training_args.py#L1783).  

My main concern with this at the moment is that we don't currently run CI on any other devices that GPU and CPU. Additionally, some operations and kernel are not currently implemented for mps c.f. [this pytorch issue](https://github.com/pytorch/pytorch/issues/77764).

As such, it's entirely possible that many of these tests wouldn't actually pass for these hardware. If we silently default to a different device, becomes difficult for people to debug and figure out why things have started failing. 

Have you managed to run the test suite on an MPS or NPU device? 

cc @ydshieh 




Agree with @amyeroberts !

> Hi @statelesshz, thanks for opening this PR!
> 
> Selecting e.g. "mps" by default if available is definitely something we're moving towards for e.g. [Trainer](https://github.com/huggingface/transformers/blob/b0f23036f16b961e14cf480522fef8581c4bf19c/src/transformers/training_args.py#L1783).
> 
> My main concern with this at the moment is that we don't currently run CI on any other devices that GPU and CPU. Additionally, some operations and kernel are not currently implemented for mps c.f. [this pytorch issue](https://github.com/pytorch/pytorch/issues/77764).
> 
> As such, it's entirely possible that many of these tests wouldn't actually pass for these hardware. If we silently default to a different device, becomes difficult for people to debug and figure out why things have started failing.
> 
> Have you managed to run the test suite on an MPS or NPU device?
> 
> cc @ydshieh

Hi @amyeroberts, thanks for pointing out that. I removed the modification regarding mps on which as some test cases failed. As for NPU, everything is going smoothly. BTW, to avoid breaking the current testing process, an additional environment variable `RUN_THIRD_PARTY_DEVICE_TESTS` is introduced to handle test cases on third-party devices. Could you kindly review this pull request once more? Thx:)



Hi @statelesshz 

My opinion on this 
> It is crucial to enable unit tests to ensure the smooth integration of third-party devices

==> It's only meaningful to us (`transformers`) to add those 3rd party devices in `testing_utils.py` if we do run those CI on our side, which is currently not the case.

But I understand that it might be helpful for people working extensively with those devices and want to make sure `transformers` work with them (well, I also see you added `TestTrainerDistributedNPU` a few weeks ago). So this PR LGTM, thanks.

BTW, could you show us a test run page that ran against NPU? I am curious it works smoothly 💯 !



@sgugger Could you help me merge this PR? thx:)

Thanks again!