cannot import name 'is_torch_xla_available' from 'transformers'
Author: YooSungHyun
Pull request: no

### System Info

transformers==4.38.2
torch==2.2.1

rtx 3090 * 4 and cuda 12.1

### Who can help?

@muellerz @pacman100 @stevhliu 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. run https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling 's run_clm.py
2. use this script
``` bash
python run_clm.py \
    --model_name_or_path openai-community/gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-clm
```
3. raised error like `cannot import name 'is_torch_xla_available' from 'transformers'` at 49 line

### Expected behavior

import is success

Comments:
Hi @YooSungHyun, this would indicate to me that there's an issue with the transformers version installed in your environment e.g. the python path and the pip path are not aligned. 

I'm able to run the following without issue:

```py
from transformers import is_torch_xla_available
```

There is a problem with version: transformers==4.38.2
try version 2.2

> try version 2.2

@borodache Could you clarify? Version 2.2 of what (pytorch?)

of transformers wasn't that clear?

No - this is why I asked. I'm not sure why you'd suggest using v2.2 of transformers, which was released  2 major versions and 5 years ago.  

You have a point. look at this post: https://github.com/huggingface/transformers/issues/1956
it solves the issue with version 2.2. However, it doesn't mean that newer versions do not work with it. You can try and let me know...

I have exactly the same issue, trying to run exactly the same script. This must be a bug in HuggingFace code, as it is LITERALLY in their script (run_clm) and even after running pip install requirements, it still fails.

Ok, I think I figured out the problem, and I think it stems from the instructions that we probably both found (https://github.com/marshmellow77/nlp-model-shakespeare/blob/main/train-nlp-model.ipynb). By cloning out the current version, you mess up the dependencies with transformers, which will *probably* not be the same version as is installed. 

I forced this to the following (you can of course change what you want about the version, I just checked out a recent stable release).

!git clone --single-branch --branch v4.39-release https://github.com/huggingface/transformers
!pip install transformers==4.39.1
print(transformers.__version__)
!pip install -r transformers/examples/pytorch/language-modeling/requirements.txt




> Ok, I think I figured out the problem, and I think it stems from the instructions that we probably both found (https://github.com/marshmellow77/nlp-model-shakespeare/blob/main/train-nlp-model.ipynb). By cloning out the current version, you mess up the dependencies with transformers, which will _probably_ not be the same version as is installed.
> 
> I forced this to the following (you can of course change what you want about the version, I just checked out a recent stable release).
> 
> !git clone --single-branch --branch v4.39-release https://github.com/huggingface/transformers !pip install transformers==4.39.1 print(transformers.**version**) !pip install -r transformers/examples/pytorch/language-modeling/requirements.txt

i did exactly this and im having this error
```
2024-03-26 10:57:33.635984: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-26 10:57:33.636052: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-26 10:57:33.638226: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-26 10:57:36.295064: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
  File "/content/drive/MyDrive/mlm_code/run_mlm.py", line 58, in <module>
    check_min_version("4.40.0.dev0")
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/__init__.py", line 251, in check_min_version
    raise ImportError(
ImportError: This example requires a source install from HuggingFace Transformers (see `[https://huggingface.co/docs/transformers/installation#install-from-source`](https://huggingface.co/docs/transformers/installation#install-from-source%60)), but the version found is 4.39.1.
Check out https://github.com/huggingface/transformers/tree/main/examples#important-note for the examples corresponding to other versions of HuggingFace Transformers.```


hi, I ran into same error with setup of 4060-ti x 2 and cuda 12.1, manage to fix the import error.

BEFORE
transformers==4.38.2
torch==2.2.1

ERROR:
```
CUDA_VISIBLE_DEVICES=0,1 python examples/pytorch/language-modeling/run_clm.py --model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
/miniconda/envs/llama_factory/lib/python3.10/site-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
Traceback (most recent call last):
  File "/home/pop/hosting/colab/bonito_dataset/transformers/examples/pytorch/language-modeling/run_clm.py", line 59, in <module>
    **check_min_version("4.40.0.dev0")**
  File "/miniconda/envs/llama_factory/lib/python3.10/site-packages/transformers/utils/__init__.py", line 251, in check_min_version
    raise ImportError(
ImportError: This example requires a source install from HuggingFace Transformers (see `https://huggingface.co/docs/transformers/installation#install-from-source`), but the version found is 4.39.1.
Check out https://github.com/huggingface/transformers/tree/main/examples#important-note for the examples corresponding to other versions of HuggingFace Transformers.
```

INVESTIGATION:
according to error message, required minimal transformer version: 4.40.0.dev0

WORKAROUND:
install latest main branch
```
git clone https://github.com/huggingface/transformers.git
pip install -e .
```
install the rest requirements
```
pip install -r transformers/examples/pytorch/language-modeling/requirements.txt
```

AFTER:
```
Grouping texts in chunks of 1024: 100%|███████████████████████████████████████████████████████████████████| 3760/3760 [00:00<00:00, 22833.65 examples/s]
[INFO|trainer.py:561] 2024-03-27 02:15:29,602 >> max_steps is given, it will override any value given in num_train_epochs
[INFO|trainer.py:1995] 2024-03-27 02:15:29,727 >> ***** Running training *****
[INFO|trainer.py:1996] 2024-03-27 02:15:29,727 >>   Num examples = 2,318
[INFO|trainer.py:1997] 2024-03-27 02:15:29,727 >>   Num Epochs = 1
[INFO|trainer.py:1998] 2024-03-27 02:15:29,727 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2000] 2024-03-27 02:15:29,727 >>   Training with DataParallel so batch size has been adjusted to: 8
[INFO|trainer.py:2001] 2024-03-27 02:15:29,727 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:2002] 2024-03-27 02:15:29,727 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2003] 2024-03-27 02:15:29,727 >>   Total optimization steps = 200
[INFO|trainer.py:2004] 2024-03-27 02:15:29,727 >>   Number of trainable parameters = 124,439,808
````
hope this helps!

@YooSungHyun you can replace

```python
from transformers import is_torch_xla_available
```

with 

```python
from transformers import is_torch_tpu_available
```

but this has been deprecated and will be removed in 4.41.0. Find more details in the following function

https://github.com/huggingface/transformers/blob/536ea2aca234fb48c5c69769431d643b0d93b233/src/transformers/utils/import_utils.py#L518

> @YooSungHyun you can replace
> 
> ```python
> from transformers import is_torch_xla_available
> ```
> 
> with
> 
> ```python
> from transformers import is_torch_tpu_available
> ```
> 
> but this has been deprecated and will be removed in 4.41.0. Find more details in the following function
> 
> https://github.com/huggingface/transformers/blob/536ea2aca234fb48c5c69769431d643b0d93b233/src/transformers/utils/import_utils.py#L518

Nice finding, thanks for the info.