Repository: huggingface/transformers

Issue:
🐛 TPU Training broken due to recent changes
# 🐛 Bug
Looks like due to changes in file_utils.py, the TPU Training has become broken. Reverting transformers to a version before https://github.com/huggingface/transformers/commit/2cfb947f59861d5d910f84eba3be57da200b5599 fixes the problem.

## Information
Seems like file_utils.py is trying to reinitialize the TPU system right after being imported. This fails because xla_spawn.py has already initialized the TPU.
Model I am using (Bert, XLNet ...): roberta (but doesn't matter)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

With a setup capable of training on TPU, replicating the official language modeling example 
```
/transformers/examples$ python xla_spawn.py --num_cores 8 language-modeling/run_language_modeling.py    --output_dir=output     --model_type=roberta     --model_name_or_path=roberta-base     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
The failure stacktrace-
```
 File "/home/saurabh/chat-ai/vendor/transformers/examples/language-modeling/run_language_modeling.py", line 29, in <module>
                                                                                                                               
    self = reduction.pickle.load(from_parent)                      
    from transformers import (          
  File "/home/saurabh/chat-ai/vendor/transformers/examples/language-modeling/run_language_modeling.py", line 29, in <module>
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/__init__.py", line 23, in <module>
    from transformers import (                    
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/__init__.py", line 23, in <module>          
    from transformers import (                                                                                                 
    from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/__init__.py", line 23, in <module>          
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/configuration_albert.py", line 18, in <modul
e>                                                                                                                             
    from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/configuration_albert.py", line 18, in <modul
e>                                                                                                                            
    from .configuration_utils import PretrainedConfig
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/configuration_utils.py", line 25, in <module
>                                                                                                                                  from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/configuration_albert.py", line 18, in <modul
e>                                                                                                                      
    from .configuration_utils import PretrainedConfig
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/configuration_utils.py", line 25, in <module
>                                                                                                                              
    from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_url
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/file_utils.py", line 76, in <module>
    from .configuration_utils import PretrainedConfig
    from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_url
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/configuration_utils.py", line 25, in <module
>
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/file_utils.py", line 76, in <module>
    tpu_device = xm.xla_device()
    from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_url
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 146, in xla_device
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/transformers/file_utils.py", line 76, in <module>
    tpu_device = xm.xla_device()
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 146, in xla_device
    tpu_device = xm.xla_device()
    devkind=[devkind] if devkind is not None else None)
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 146, in xla_device
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 50, in get_xla_support
ed_devices
    devkind=[devkind] if devkind is not None else None)
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 50, in get_xla_support
ed_devices
    xla_devices = torch_xla._XLAC._xla_get_devices()
    devkind=[devkind] if devkind is not None else None)
RuntimeError: tensorflow/compiler/xla/xla_client/xrt_computation_client.cc:1245 : Check failed: session.Run({tensorflow::Output
(result, 0)}, &outputs) == ::tensorflow::Status::OK() (Already exists: From /job:tpu_worker/replica:0/task:0:
2 root error(s) found.
  (0) Already exists: Resource localhost/tpu_mesh_common_state/N10tensorflow3tpu21TpuMeshStateInterfaceE
         [[{{node configure_distributed_tpu/_0}}]]
(1) Already exists: Resource localhost/tpu_mesh_common_state/N10tensorflow3tpu21TpuMeshStateInterfaceE
         [[{{node configure_distributed_tpu/_0}}]]
0 successful operations.
0 derived errors ignored. vs. OK)
```
## Expected behavior
Model trains

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0 (master)
- Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0a0+af05158 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: yes, 8 way parallelism with xla_spawn.py



Discussion:
See https://github.com/huggingface/transformers/issues/4814 as well. Over there the TPU evaluation is broken.
To make the TPU pipeline reliable, a end-to-end test could really help.
Hi! Thank you for raising this issue, I'll take a look.

Of course, having an end-to-end test would really help. Unfortunately, such suites don't exist with TPU right now.

Resolving patch:
diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py
--- a/src/transformers/file_utils.py
+++ b/src/transformers/file_utils.py
@@ -71,9 +71,7 @@
 
 
 try:
-    import torch_xla.core.xla_model as xm
-
-    tpu_device = xm.xla_device()
+    import torch_xla.core.xla_model as xm  # noqa: F401
 
     if _torch_available:
         _torch_tpu_available = True  # pylint: disable=


Test patch: