DDP for XPU in trainer
Author: yash3056
Pull request: no
Labels: name: Feature request

### Feature request

DDP support for xpu like cuda, trainer automatically take multi cuda devices with the help of accelerate.
Trainer should be able to use detect and use multiple xpu devices by default.

### Motivation

Writing DDP codes with trainer is fast and effective. 
In pytorch writing training loop takes time.

### Your contribution

None

Comments:
cc @muellerzr @SunMarc 

Accelerate should handle multi-xpu distributed training just like cuda. What is the issue that you are facing ? are you receiving cuda specific error when wanting to train on xpu ? 

@SunMarc it is not by default picking up all xpu devices like cuda. 

number of gpu available

```
# Print number of available XPUs
print(f"Number of available XPUs: {torch.xpu.device_count()}")

(aza) sdp@bvcoe:~$ /home/sdp/.conda/envs/aza/bin/python /home/sdp/main.py
[WARNING] Failed to create Level Zero tracer: 2013265921
Number of available XPUs: 16

```

number of device selected automatically

```
(aza) sdp@bvcoe:~$ sudo xpu-smi dump -m 0,18
Timestamp, DeviceId, GPU Utilization (%), GPU Memory Used (MiB)
15:15:11.000,    0, 49.15,  N/A
15:15:11.000,    1, 0.00,  N/A
15:15:11.000,    2, 0.00,  N/A
15:15:11.000,    3, 0.00,  N/A
15:15:11.000,    4, 0.00,  N/A
15:15:11.000,    5, 0.00,  N/A
15:15:11.000,    6, 0.00,  N/A
15:15:11.000,    7, 0.00,  N/A
15:15:12.000,    0, 49.18, 8194.71
15:15:12.000,    1, 0.00, 121.34
15:15:12.000,    2, 0.00, 121.33
15:15:12.000,    3, 0.00, 121.34
15:15:12.000,    4, 0.00, 121.34
15:15:12.000,    5, 0.00, 121.28
15:15:12.000,    6, 0.00, 121.22
15:15:12.000,    7, 0.00, 121.22
15:15:13.000,    0, 49.17, 8194.71
15:15:13.000,    1, 0.00, 121.34
15:15:13.000,    2, 0.00, 121.33
15:15:13.000,    3, 0.00, 121.34
15:15:13.000,    4, 0.00, 121.34
15:15:13.000,    5, 0.00, 121.28
15:15:13.000,    6, 0.00, 121.22
15:15:13.000,    7, 0.00, 121.22
15:15:14.000,    0, 49.20, 8194.71
15:15:14.000,    1, 0.00, 121.34
15:15:14.000,    2, 0.00, 121.33
15:15:14.000,    3, 0.00, 121.34
15:15:14.000,    4, 0.00, 121.34
15:15:14.000,    5, 0.00, 121.28
15:15:14.000,    6, 0.00, 121.22
15:15:14.000,    7, 0.00, 121.22
```


@SunMarc , I'll take this issue, thx.
Hi, @yash3056 , could you share your scripts and reproducing steps, I can dig a bit into this issue.

@yao-matrix 
code for reproducing 
```
import torch
import intel_extension_for_pytorch as ipex
from transformers import GPT2Tokenizer, GPT2ForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset

# Load the IMDB dataset
dataset = load_dataset('imdb')

# Initialize the tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token

# Tokenize the dataset
def tokenize_function(examples):
    return tokenizer(examples['text'], padding='max_length', truncation=True, max_length=512)

tokenized_datasets = dataset.map(tokenize_function, batched=True, num_proc=192)

# Prepare the model
model = GPT2ForSequenceClassification.from_pretrained('gpt2', device_map='auto')
model.config.pad_token_id = model.config.eos_token_id

# Set up training arguments
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=2,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    evaluation_strategy='epoch',
    save_total_limit=1,
    remove_unused_columns=False,
)

# Initialize the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['test'],
)

# Train the model
trainer.train()

# Evaluate the model
trainer.evaluate()

```

@yash3056 ,
For your observation that: `torch.xpu.device_count()` says there are 16 cards in system, but `xpu-smi` says there are 8 cards.
The reason is the card you are using has a 2-compute-tile structure(as below), we call it 1-card-2-tiles, and to make best use of these 2 compute tiles to best efficiency, Intel one API by default set [`ZE_FLAT_DEVICE_HIERARCHY=FLAT`](https://www.intel.com/content/www/us/en/docs/oneapi/optimization-guide-gpu/2024-1/exposing-device-hierarchy.html) , which means 8 physical cards will be viewed as 16 compute cards from PyTorch's perspective.
![image source： https://www.techpowerup.com/gpu-specs/data-center-gpu-max-1550.c4068](https://github.com/user-attachments/assets/63360828-5440-464c-89f0-36f46d6158da)
If you wanna get a compatible view through `xpu-smi`, you can use `xpu-smi dump -t 0,1 -m 0,18` here `-t 0,1` means you'd like to see tile 0 and 1 for each card, then the output will be like:
> Timestamp, DeviceId, TileId, GPU Utilization (%), GPU Memory Used (MiB)
> 01:11:15.848,    0,    0,  N/A, 38.98
> 01:11:15.849,    0,    1,  N/A, 38.91
> 01:11:15.849,    1,    0,  N/A, 38.99
> 01:11:15.849,    1,    1,  N/A, 38.92
> 01:11:15.849,    2,    0,  N/A, 38.86
> 01:11:15.849,    2,    1,  N/A, 38.92
> 01:11:15.849,    3,    0,  N/A, 38.99
> 01:11:15.849,    3,    1,  N/A, 38.92

If you still want to treat 1 physical card as 1 card and don't want to care tile, you can set `export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE`, then your `torch.xpu.device_count()` will return 8. But this is not suggested if you pursue better performance.

Hope it helps.

@yao-matrix 
you have diverted from the current issue.

if you run this code which I slightly modified for testing on kaggle two t4, you can see that it automatically uses both gpus without any further steps like deepspeed, etc. **and I wish for this same feature for multi xpus**

``` python
import torch
from transformers import GPT2Tokenizer, GPT2ForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset

# Load the IMDb dataset
train_dataset = load_dataset('imdb', split='train[:1000]')
test_dataset = load_dataset('imdb', split='test[:1000]')

# Initialize the tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token

# Tokenize the dataset
def tokenize_function(examples):
    # Tokenize the input text and add the labels
    return tokenizer(examples['text'], padding='max_length', truncation=True, max_length=512)

# Tokenize train and test datasets
tokenized_train_dataset = train_dataset.map(tokenize_function, batched=True, num_proc=4)
tokenized_test_dataset = test_dataset.map(tokenize_function, batched=True, num_proc=4)

# Convert datasets to PyTorch tensors
def format_dataset(dataset):
    # Rename 'label' column to 'labels' for compatibility with Trainer
    dataset = dataset.rename_column("label", "labels")
    # Set format to PyTorch tensors
    dataset.set_format(type="torch", columns=["input_ids", "attention_mask", "labels"])
    return dataset

# Apply formatting
tokenized_train_dataset = format_dataset(tokenized_train_dataset)
tokenized_test_dataset = format_dataset(tokenized_test_dataset)

# Prepare the model
model = GPT2ForSequenceClassification.from_pretrained('gpt2', num_labels=2)  # IMDb is binary classification
model.config.pad_token_id = model.config.eos_token_id

# Set up training arguments
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=2,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    eval_strategy='epoch',
    save_strategy='epoch',
    logging_dir='./logs',
    report_to='none'  # Disable wandb 
)

# Initialize the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train_dataset,
    eval_dataset=tokenized_test_dataset,
    tokenizer=tokenizer,
)

# Train the model
trainer.train()

# Evaluate the model
trainer.evaluate()
```



accelerator is only detecting only 1 gpu

## Code:
```
from accelerate import Accelerator

# Initialize the accelerator
accelerator = Accelerator()

# Get the number of devices
device_count = accelerator.num_processes

print(f"Number of devices available: {device_count}")
```

## Output:
```
Number of devices available: 1
[WARNING] Failed to create Level Zero tracer: 2013265921
```

@yash3056, I can reproduce your observation. Here are the things under the hood:
1) When you use `python` to launch your scripts, `Trainer` will use `DataParallel` to run across multi-cards, [ `DataParallel`](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) is a single-process-multi-thread parallelism mechanism from PyTorch and dis-encouraged by PyTorch after it enabled `DistributedDataParallel (DDP)`. Since PyTorch is deprecating DP, XPU doesn't support DP as of now, so this path doesn't work as you see. BTW: since DP is a multi-thread way, so `accelerator.num_processes` doesn't work to check whether parallelism works, in GPU case, it also returns `1`. If you want to check, you can enable `transformers.logging.set_verbosity_info()`, and it will show logs like below:
> Num examples = 1,000
> Num Epochs = 2
> Instantaneous batch size per device = 2
> **Training with DataParallel so batch size has been adjusted to: 4**
> **Total train batch size (w. parallel, distributed & accumulation) = 4**
> Gradient Accumulation steps = 1
> Total optimization steps = 500
> Number of trainable parameters = 124,441,344
2) We recommend you use accelerate to run DDP training, you can use `accelerate config` to config your env, then use `accelerate launch <your_python_script.py>` to run DDP on XPU.

Thanks for your time on this issue.