Fix wrong xpu device in DistributedType.MULTI_XPU mode
Author: faaany
Pull request: yes

## Problem
When running lora fine-tuning on XPU in single-node&multi-card way,  I noticed that the device is not correctly set up for distributed fine-tuning in the "__setup_devices" function. 
![image](https://github.com/huggingface/transformers/assets/24477841/8f5a0233-86ef-4854-85c4-e0b5f02dc7ce)
As can be seen from the red box, if the distributed_type is `DistributedType.MULTI_XPU`, the device shouldn't be hard-coded to "xpu:0". Otherwise, the GPU process with rank 1 will end up with moving its model to "xpu:0" instead of the correct "xpu:1".

## Solution
We can remove the entire code in the red box because the device has already been correctly set in the above orange box. which is consistent to the device set by accelerator shown as below:
![image](https://github.com/huggingface/transformers/assets/24477841/ded014cd-de65-44c5-b5a6-c59f17ac7823)

## Reproduce
My training script is from [this ](https://huggingface.co/blog/Lora-for-sequence-classification-with-Roberta-Llama-Mistral)blog article and here is the command for launching this script:
```bash
accelerate launch training.py
```

And this is my Environment: 
```bash
- `transformers` version: 4.37.0.dev0
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_XPU
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: 1,2
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - ipex_config: {'use_xpu': True}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.1.0a0+cxx11.abi (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed
```

Pls, have a review, thanks!
@muellerzr and @pacman100


Comments:
@ArthurZucker would you take a look at this PR as well? Not quite sure who is the best reviewer for this case. Many thanks!

Hi folks, any concern or feedback for this PR? 

cc @abhilash1910

Thanks @muellerzr  for highlighting. As per our previous issue : #27716  (cc @ArthurZucker ) , this was causing an issue in ipex . @faaany I will sync with you offline to find a proper solution. Thanks

#28442 might give a solution as well

> #28442 might give a solution as well

Thanks for the review! I will have an offline meeting with @abhilash1910  and @zhuhong61 for an alignment and get back to you soon.


@ArthurZucker we talked with each other offline today and decided to move forward with the solution proposed in this PR. 
Pls give your comments as well, thanks a lot! @abhilash1910 @zhuhong61

Hi @ArthurZucker We can close this PR https://github.com/huggingface/transformers/pull/28442, and use #28386 as the final solution. Thanks

Thanks, does this need a specific version of torch / accelerate to work (which I would suspect as the patch was introduced )

> Thanks, does this need a specific version of torch / accelerate to work (which I would suspect as the patch was introduced )

yes, a patched version for pytorch is still needed for XPU currently.

> Thanks, does this need a specific version of torch / accelerate to work (which I would suspect as the patch was introduced )

@ArthurZucker, yes, for making XPU work, people need to follow this link https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=gpu&version=v2.1.10%2Bxpu to set up env, it includes the installation intel oneCCL, intel patched PyTorch and Intel Extensions for PyTorch, these 3 support xpu.

And Intel PyTorch team is also working with Meta to make xpu a built-in device of stock PyTorch. By that time, we can use XPU with stock PyTorch.

Hi @muellerzr @ArthurZucker, this PR has been there for a while and we are also internally aligned with this solution. Do you guys still have any concerns? 

Yes for sure! 

> LGTM my only concern is that this can only work starting latest version of torch, thus we would need to keep the old solution for older versions. WDYT?

Hi @ArthurZucker , the old solution is no longer needed as the changes discussed here fall perfectly in line with the expectations from the xpu . Conclusively this implies that the code changes can be used as is ( as it was being used before with accelerate and others) . 
For any breaking changes related to core pytorch which would require change of apis on xpu side, we would definitely raise a significant PR to address those. But as of now, this change is perfectly fine with the existing software stack. Thanks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28386). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> LGTM my only concern is that this can only work starting latest version of torch, thus we would need to keep the old solution for older versions. WDYT?

Hmmm, I don't think people using older versions of torch would have problems with our current fix. In contrast, they would get the correct device set once our PR is merged. But anyway, thanks so much for the review and approval!   

Hi @ArthurZucker, could you help merge this PR? Many thanks!

Thanks for the explanations!