Trainer defaults to NCCL backend for ddp on windows
Author: btrude
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run the following script natively on windows with >1 nvidia GPU. This example uses jsonl input in the same format as the openai api, eg:
```json
{"prompt": "I wish ddp worked with the trainer in windows", "completion": "it does, you are just doing it wrong!"}
{"prompt": "The moon is made of green cheese", "completion": "ok, but I was asking about the huggingface trainer..."}
```

```py
from datasets import load_dataset
from transformers import AutoConfig
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer
from transformers import DataCollatorForLanguageModeling
from transformers import Trainer
from transformers import TrainingArguments


def train(
    train_file_path,
    eval_file_path=None,
    name=None,
    n_epochs=5,
    model_name="EleutherAI/gpt-neo-125m",
    use_scheduler=False,
):
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token
    tokenizer.padding_side = "left"

    model = AutoModelForCausalLM.from_pretrained(model_name)
    print(model.config)

    if eval_file_path:
        train_dataset = load_dataset("json", data_files=train_file_path)
        eval_dataset = load_dataset("json", data_files=eval_file_path)
    else:
        eval_dataset = load_dataset("json", data_files=train_file_path, split="train[:5%]")
        train_dataset = load_dataset("json", data_files=train_file_path, split="train[5%:]")

    def tokenize_dataset(entry):
        inputs = tokenizer(entry["prompt"] + entry["completion"], return_tensors="pt")
        return {
            "input_ids": inputs["input_ids"],
            "attention_mask": inputs["attention_mask"],
        }

    n_steps_epoch = len(train_dataset)
    train_dataset = train_dataset.map(tokenize_dataset).remove_columns(["prompt", "completion"])
    eval_dataset = eval_dataset.map(tokenize_dataset).remove_columns(["prompt", "completion"])
    data_collator = DataCollatorForLanguageModeling(tokenizer, mlm=False)

    training_args = TrainingArguments(
        output_dir=name,
        per_device_train_batch_size=8,
        per_device_eval_batch_size=8,
        gradient_accumulation_steps=32,
        logging_dir=f"{name}/logs",
        logging_steps=16,
        evaluation_strategy="steps",
        save_steps=n_steps_epoch // 2048,
        eval_steps=n_steps_epoch // 1024,
        save_total_limit=3,
        report_to="tensorboard",
        tf32=True,
        seed=1679815,
    )
    training_args.set_optimizer(
        "adamw_torch_fused",
        learning_rate=1e-4,
        beta1=.9,
        beta2=.95,
        epsilon=1e-8,
        weight_decay=.1
    )
    if use_scheduler:
        training_args.set_lr_scheduler(
            name="linear",
            warmup_steps=250,
            num_epochs=n_epochs,
        )
    print(training_args)

    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
    )
    trainer.train()


if __name__ == "__main__":
    train("path/to/json/dataset", name="no_gloo")
```

### Expected behavior

Outside of the context of the huggingface trainer I am able to use the gloo backend in conjunction with mpi for distributed training with pytorch using the following setup:

```
def setup_ddp(hps):
    if hps.ddp:
        port = "29500"
        rank = MPI.COMM_WORLD.Get_rank()
        world_size = MPI.COMM_WORLD.Get_size()
        os.environ["RANK"] = str(rank)
        os.environ["WORLD_SIZE"] = str(world_size)
        os.environ["MASTER_ADDR"] = "localhost"
        os.environ["MASTER_PORT"] = port
        dist.init_process_group("gloo", rank=rank, world_size=world_size)
        group = dist.new_group(ranks=devices())
    else:
        rank, world_size, group = 0, 1, None

    return rank, world_size, group
```
When running the provided training script in linux (via WSL2) with >1 GPU, everything executes as one would expect - but WSL2 is significantly slower than native sadly.

I would expect that the trainer would expose the backend that is used as a variable, but `xpu_backend` does not affect the behavior I am experiencing nor is it immediately clear if this is meant to be configurable as-is. NCCL is not supported in windows on pytorch currently (https://github.com/pytorch/pytorch/issues/89688) and so the trainer should not attempt to default to NCCL unless it is installed/supported by the OS (ie windows should always default to a different, functional backend).

At the very least I would expect a clean error message that explains that DDP is not supported on windows or whatever the actual state of compatibility is. Instead, the script throws warnings about pytorch not being compiled with NCCL support and fails on the first forward pass with inscrutable cuda errors.

Comments:
On main `xpu_backend` (soon to be renamed to `ddp_backend`) lets you pick up the backend you want.

Excellent, thank you!