TrainingArguments with pytorch on Mac: AttributeError: module 'torch.distributed' has no attribute 'is_initialized'
Author: radkoff
Pull request: no
Labels: name: bug

### System Info

```shell
macOS 12.4 running on Apple silicon

python                    3.9.0           h4b4120c_5_cpython    conda-forge
transformers              4.18.0           py39hca03da5_0  
pytorch                   1.10.2          cpu_py39h23cb94c_0
```


### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm trying to run a minimal example based on [this documentation](https://huggingface.co/docs/transformers/training#finetune-with-trainer): finetuning for text classification.

```
from datasets import load_metric, load_dataset
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer


dataset = load_dataset("yelp_review_full")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

def tokenize_function(examples):
    return tokenizer(examples['text'], padding='max_length', truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))

model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)
training_args = TrainingArguments(output_dir="out")
metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)
trainer.train()
```

I am seeing the following output:
```
Traceback (most recent call last):
  File "/Users/evan/Library/Application Support/JetBrains/PyCharmCE2022.1/scratches/scratch_9.py", line 18, in <module>
    training_args = TrainingArguments(output_dir="out")
  File "<string>", line 91, in __init__
  File "/Users/evan/miniforge3/envs/docbot-server/lib/python3.9/site-packages/transformers/training_args.py", line 865, in __post_init__
    and (self.device.type != "cuda")
  File "/Users/evan/miniforge3/envs/docbot-server/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 781, in wrapper
    return func(*args, **kwargs)
  File "/Users/evan/miniforge3/envs/docbot-server/lib/python3.9/site-packages/transformers/training_args.py", line 1099, in device
    return self._setup_devices
  File "/Users/evan/miniforge3/envs/docbot-server/lib/python3.9/site-packages/transformers/utils/generic.py", line 48, in __get__
    cached = self.fget(obj)
  File "/Users/evan/miniforge3/envs/docbot-server/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 781, in wrapper
    return func(*args, **kwargs)
  File "/Users/evan/miniforge3/envs/docbot-server/lib/python3.9/site-packages/transformers/training_args.py", line 1024, in _setup_devices
    if torch.distributed.is_initialized() and self.local_rank == -1:
AttributeError: module 'torch.distributed' has no attribute 'is_initialized'
```

It looks like torch doesn't expose the `is_initialized` API unless distributed training is supported. Should the `TrainingArguments` class first check `torch.distributed.is_available()` before trying to call that?

Thanks


### Expected behavior

```shell
Fine-tuning on the Yelp dataset using pytorch.
```


Comments:
This has been fixed already, you should try again on the latest version.

Thank you, upgrading to 4.19.2 worked.

Great to hear!