run_mlm is not working with TPU 
Author: Shiro-LK
Pull request: no

### System Info

I am using colab : 
python:3.10

for torch and xla : 
`pip install cloud-tpu-client==0.10 torch==2.0.0 torchvision==0.15.1 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-2.0-cp310-cp310-linux_x86_64.whl`

transformers == 4.30.2


after using this command the training get this error : 

`2023-07-12 21:07:17.770888: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:09.577189: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:09.645535: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:09.848706: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.028873: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.122547: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.322647: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.612495: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-12 21:08:10.867921: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Exception in device=TPU:0: Invalid device format: cpu
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 334, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 328, in _start_fn
    fn(gindex, *args)
  File "/content/run_mlm.py", line 654, in _mp_fn
    main()
  File "/content/run_mlm.py", line 239, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/usr/local/lib/python3.10/dist-packages/transformers/hf_argparser.py", line 346, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 111, in __init__
  File "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py", line 1341, in __post_init__
    and (get_xla_device_type(self.device) != "GPU")
  File "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py", line 127, in get_xla_device_type
    return xm.xla_real_devices([device])[0].split(":")[0]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 271, in xla_real_devices
    return [_xla_real_device(device) for device in devices]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 271, in <listcomp>
    return [_xla_real_device(device) for device in devices]
  File "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py", line 266, in _xla_real_device
    raise RuntimeError('Invalid device format: {}'.format(device_str))
RuntimeError: Invalid device format: cpu
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/xla_spawn.py:83 in <module>                                         │
│                                                                              │
│   80                                                                         │
│   81                                                                         │
│   82 if __name__ == "__main__":                                              │
│ ❱ 83 │   main()                                                              │
│   84                                                                         │
│                                                                              │
│ /content/xla_spawn.py:79 in main                                             │
│                                                                              │
│   76 │   # Patch sys.argv                                                    │
│   77 │   sys.argv = [args.training_script] + args.training_script_args + ["- │
│   78 │                                                                       │
│ ❱ 79 │   xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)               │
│   80                                                                         │
│   81                                                                         │
│   82 if __name__ == "__main__":                                              │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/torch_xla/distributed/xla_multiproce │
│ ssing.py:397 in spawn                                                        │
│                                                                              │
│   394   if pf_cfg.num_devices == 1:                                          │
│   395 │   _start_fn(0, pf_cfg, fn, args)                                     │
│   396   else:                                                                │
│ ❱ 397 │   result = torch.multiprocessing.start_processes(                    │
│   398 │   │   _mp_start_fn,                                                  │
│   399 │   │   args=(pf_cfg, fn, args),                                       │
│   400 │   │   nprocs=pf_cfg.num_devices,                                     │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py:197   │
│ in start_processes                                                           │
│                                                                              │
│   194 │   │   return context                                                 │
│   195 │                                                                      │
│   196 │   # Loop on join until it returns True or raises an exception.       │
│ ❱ 197 │   while not context.join():                                          │
│   198 │   │   pass                                                           │
│   199                                                                        │
│   200                                                                        │
│                                                                              │
│ /usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py:149   │
│ in join                                                                      │
│                                                                              │
│   146 │   │   │   │   │   signal_name=name                                   │
│   147 │   │   │   │   )                                                      │
│   148 │   │   │   else:                                                      │
│ ❱ 149 │   │   │   │   raise ProcessExitedException(                          │
│   150 │   │   │   │   │   "process %d terminated with exit code %d" %        │
│   151 │   │   │   │   │   (error_index, exitcode),                           │
│   152 │   │   │   │   │   error_index=error_index,                           │
╰──────────────────────────────────────────────────────────────────────────────╯
ProcessExitedException: process 0 terminated with exit code 17`


### Who can help?

@sgugger ,  @ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. choose TPU platform on colab 
2. ```
!python xla_spawn.py --num_cores 8 run_mlm.py \
    --model_name_or_path roberta-base \
    --tpu_num_cores 8 \
    --train_file tr.txt \
    --validation_file dev.txt \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --max_seq_len 200 \
    --line_by_line True \
    --pad_to_max_length True \
    --output_dir mlm_tpu-v2
```


### Expected behavior

The training should run.

Comments:
It looks like PyTorch XLA cannot see your TPUs, are you sure you properly set up your instance?

I suppose it is, I am using colab TPU and I have installed thiis package : 
`pip install cloud-tpu-client==0.10 torch==2.0.0 torchvision==0.15.1 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-2.0-cp310-cp310-linux_x86_64.whl`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.