Extend Transformers Trainer Class to Enable XPU 
Author: mingxiaoh
Pull request: yes

Rename GPU utils to CUDA
Add XPU backend
Doc on XPU backend


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22551). All of your documentation changes will be reflected on that endpoint.

Thanks for your PR, but we are not going to rename core utils like this.

> Thanks for your PR, but we are not going to rename core utils like this.

thanks for the review, if so, I just rolled back the rename. 

@sgugger  May you please help review it again? thanks.

Thanks for your PR. We can leave this branch as is for people who want to use XPUs but:
1. We do not have the ability to test them, so cannot commit to maintain this yet.
2. The Trainer will be rewritten to use Accelerate very soon, so the support should be added in Accelerate (I believe there is already a PR open) and then will come in the Trainer for free.

> Thanks for your PR. We can leave this branch as is for people who want to use XPUs but:
> 
> 1. We do not have the ability to test them, so cannot commit to maintain this yet.
> 2. The Trainer will be rewritten to use Accelerate very soon, so the support should be added in Accelerate (I believe there is already a PR open) and then will come in the Trainer for free.

Yes, my colleague is working with me preparing the PR for accelerate now. But, we still would like to merge this change into transformer since intel-extension-for-pytorch cpu backend is already in transformer, and we expect user can use it directly even without accelerate.

The Trainer will require a dependency on Accelerate in roughly a month, so that point is moot.

@sgugger  thanks for the info. May you please help explain how transformer depend on Accelerate?  Does it will look like below?
first, use accelerate to wrap model,optimizer, data
    model, optimizer, data = accelerator.prepare(model, optimizer, input)
second, pass the model/optimizer/data to Trainer of transformer 
   Trainer(model,args, ...)
Besides, accelerate seems only cover training path, how about inference path when Trainer has dependency on Accelerate?



@sgugger sorry for being push, but your reply is quite important to us, thanks in advance.

@sgugger  we are investigating how to provide solution for xpu now, another PR is in https://github.com/huggingface/accelerate/pull/1118, detailed info about how  transformer depend on accelerate is important to us, may you please take time to explain it to us? Thanks in advance.

@mingxiaoh I would appreciate that you stop pinging me on this PR repeatedly. I said migrating the Trainer to use Accelerate for all the boilerplate code is a work in progress that will take roughly a month, so I would appreciate your patience on this and let us do the actual work.
Once the migration is done and the PR on the Accelerate side is merged, there will be no need for this PR or any other kind of PR, XPU will just work out-of-the-box with the Trainer.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

hello,  @pacman100 
May you please take some time to help explain this issue? Thanks in advance.
I found that transormers Trainer is using Accelerate now, is it the final solution? Why I ask so is because I found that, currently, if a model is wrapped by transformers trainer(e.g., IPEX CPU), it won't use accelerate to prepare model(see https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1734-L1737), which means, for IPEX cpu, it won't use accelerate to prepare model, for IPEX xpu, it will use accelerate to prepare model, this might make users confused. Besides, for accelerate, it could not tell whether it is inference mode or training mode, currently, in accelerate, IPEX xpu always assumes it is training mode and wrap model into training mode, in this case, we could not run transformers in inference mode for IPEX xpu, any comments for this issue?


@pacman100  my colleague told me that you are the owner of integration of traniner & accelerate, if it is not too trouble, may you please take time to help explain it a little bit for us? Thanks.

Hello @mingxiaoh, we are still in the process of migration. The next steps would involve shifting to Accelerate for ipex and adding the functionality in Accelerate if it isn't available there yet. Gentle ping @muellerzr who will be looking into this. 

@pacman100  hello, may I know the the process of migration? Intel would like to extend it to xpu asap for customers' usge. Thanks in advance.

@mingxiaoh XPU support is currently happening in Accelerate due to efforts of @abhilash1910 and @sywangyi. It was in the last Accelerate release, so from here we need to look at the specific DDP efforts the Trainer has that need to be handled with XPU specific support that *isn't* already done via the Accelerator object. (aka add an arg to configure the `state` properly if needed, otherwise most of the trainer should work OOTB with the XPU). (v 0.20.0). Related PR here: https://github.com/huggingface/accelerate/pull/1118

As the Trainer is now using Accelerate for all of it's device/compute configuration and specialized code

@muellerzr Thanks, I know https://github.com/huggingface/accelerate/pull/1118 and xpu is supported in accelerate, we were working together for it before. But the issue is, we would like to know how the process of migration, in accelerate currently, inference is not supported on xpu, but for transformers, we must consider inference case. So, may I know the process of migration? we were told there is some problem about one month ago.