Repository: huggingface/transformers

Issue:
save_pretrained() does not check if xla is available
## Environment info

- `transformers` version: 3.3.1
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.4
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): Bert


The problem arises when using:
* [ X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. load any model trained on TPU with `BertModel.from_pretrained(tpu_checkpoint_path)`
2. run/train the model - works fine
3. save the model with `model.save_pretrained(save_path)`

```
line 720, in save_pretrained
    import torch_xla.core.xla_model as xm
ModuleNotFoundError: No module named 'torch_xla'
```
## Expected behavior

I am pretraining a LM on TPU, and for the downstream task fine-tuning I load the saved checkpoints on a non-TPU device.
Loading works fine now (#5636), but saving again does not.

`save_pretrained` should check whether the device is still xla - the original config attribute that is used in `save_pretrained` to check for the device persists when loading the xla model on another device:
```
getattr(config, 'xla_device')
True
```
It is easy to fix by changing the config attribute `setattr(config, 'xla_device', False)` in the script, but I would still consider it a bug.

save_pretrained() does not check if xla is available
## Environment info

- `transformers` version: 3.3.1
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.4
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): Bert


The problem arises when using:
* [ X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. load any model trained on TPU with `BertModel.from_pretrained(tpu_checkpoint_path)`
2. run/train the model - works fine
3. save the model with `model.save_pretrained(save_path)`

```
line 720, in save_pretrained
    import torch_xla.core.xla_model as xm
ModuleNotFoundError: No module named 'torch_xla'
```
## Expected behavior

I am pretraining a LM on TPU, and for the downstream task fine-tuning I load the saved checkpoints on a non-TPU device.
Loading works fine now (#5636), but saving again does not.

`save_pretrained` should check whether the device is still xla - the original config attribute that is used in `save_pretrained` to check for the device persists when loading the xla model on another device:
```
getattr(config, 'xla_device')
True
```
It is easy to fix by changing the config attribute `setattr(config, 'xla_device', False)` in the script, but I would still consider it a bug.



Discussion:

Resolving patch:
diff --git a/src/transformers/modeling_utils.py b/src/transformers/modeling_utils.py
--- a/src/transformers/modeling_utils.py
+++ b/src/transformers/modeling_utils.py
@@ -716,7 +716,7 @@ def save_pretrained(self, save_directory):
         # If we save using the predefined names, we can load using `from_pretrained`
         output_model_file = os.path.join(save_directory, WEIGHTS_NAME)
 
-        if getattr(self.config, "xla_device", False):
+        if getattr(self.config, "xla_device", False) and is_torch_tpu_available():
             import torch_xla.core.xla_model as xm
 
             if xm.is_master_ordinal():


Test patch: