xpu: support xpu backend from stock pytorch (>=2.4)
Author: dvrogozh
Pull request: yes

Fixes: https://github.com/huggingface/transformers/issues/31237

XPU backend is available in the stock PyTorch starting from version 2.4, see [1]. This commit extends huggingface transformers to support XPU from both IPEX and the stock pytorch. IPEX is being tried first.

Raising this PR as WIP and Draft to facilitate further discussion around XPU backend enabling in huggingface and be able to communicate observed XPU issues back to PyTorch.

See: https://github.com/pytorch/pytorch/issues/114842
Requires: https://github.com/huggingface/accelerate/pull/2825

cc: @muellerzr, @EikanWang, @jgong5, @kding1, @sywangyi


Comments:
Happy to see this addition 🚀 ! Just wondering if now we should be careful with the names like `Pytorch XPU` and `IPEX XPU`. (and sorry if this doesn't make sense 😅 )

I tried this PR (+ https://github.com/huggingface/accelerate/pull/2825 on which it depends) as much as I could in the IPEX-CPU, IPEX-XPU, Pytorch-XPU, Pytorch-CPU scenarios. Tried to run some tests from accelerate and transformers and some examples from transformers. All seem to work engaging with XPU when expected. I promote these PRs from drafts for the qualified review. Let me know if any concerns or any feedback needs to be addressed.

I added one more commit to enable some tests for xpu backend.

Applied `python utils/check_copies.py --fix_and_overwrite` to propagate change in gpt2 to decision_transformer. This fixes failure noted by ci. Test for the later passes for xpu backend.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31238). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> however: should we limit the accelerate version required for the xpu support to the new accelerate version? (to come out next month)

@muellerzr : By bumping accelerate version to 0.32.0?
https://github.com/huggingface/transformers/blob/c624d5ba0b41542eb3daa0e2fdc985231551d776/setup.py#L99


> By bumping accelerate version to 0.32.0?

We most certainly shouldn't do that :)

Actually, I think we're fairly okay, as accelerate will do a passthrough and IIUC this PR doesn't break old behavior, correct? (Basically, per my understanding if users run an old accelerate version nothing will break, right?)

The question is if we should have a flag for a minimum accelerate version if they are on the xpu branch/logic

> Actually, I think we're fairly okay, as accelerate will do a passthrough and IIUC this PR doesn't break old behavior, correct?

Yes, i think so. Till users are within previous usages (with IPEXes) nothing should change for them and be compatible.

> The question is if we should have a flag for a minimum accelerate version if they are on the xpu branch/logic

New accelerate is indeed required on xpu branch, otherwise there will be runtime error. So check will be useful. **I will add.**

Hi @dvrogozh

Thank you for this support. As mentioned by @faaany and [my comment](https://github.com/huggingface/transformers/pull/31402#issuecomment-2168013905), it would be better not to include things like

> BACKEND_MANUAL_SEED["xpu"]
> BACKEND_EMPTY_CACHE["xpu"]

in this PR (so far) and let the user to have them define when they need to use other device.

Happy to revise that design in a separate PR.



> As mentioned by @faaany and https://github.com/huggingface/transformers/pull/31402#issuecomment-2168013905, it would be better not to include things like

@ydshieh :  removed. Used `TRANSFORMERS_TEST_DEVICE_SPEC=spec.py` on my side:
```
import torch

DEVICE_NAME = 'xpu'

MANUAL_SEED_FN = torch.xpu.manual_seed
EMPTY_CACHE_FN = torch.xpu.empty_cache
DEVICE_COUNT_FN = torch.xpu.device_count
```

fyi, I think CI failure is unrelated to this PR. Needs re-triggering?
```
FAILED examples/tensorflow/test_tensorflow_examples.py::ExamplesTests::test_run_image_classification - ValueError: The repository for hf-internal-testing/cats_vs_dogs_sample contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hf-internal-testing/cats_vs_dogs_sample.
```

For 

> contains custom code which must be executed to correctly load the dataset

not related to this PR.

And you can rebase on main to include #31407 that will make that failure disappear

> And you can rebase on main to include https://github.com/huggingface/transformers/pull/31407 that will make that failure disappear

Hm. Code is already on top of latest master and includes #31407.

> I think everything runs smooth on your side, right? 

For xpu backend w/ spec.py? yes. And I ran non-xpu stuff as much as I locally could.

OK, I will check CI. Thank you again for contributing 💯 !



Thank you for the merge. If all will be good, should land in `transformers==4.42.0` as far as I could tell.