OSError: Tensor parallel is only supported for `torch>=2.5`
Author: mxjmtxrm
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.53.1
- Platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.33.2
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- DeepSpeed version: 0.17.1
- PyTorch version (accelerator?): 2.5.0a0+e000cf0ad9.nv24.10 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@SunMarc @zach-huggingface

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, I met the following error when I tried to finetune a model using run_clm.py.
```
Traceback (most recent call last):
  File "/workspace/run_clm.py", line 679, in <module>
    main()
  File "/workspace/run_clm.py", line 235, in main
    model = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 593, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 311, in _wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4443, in from_pretrained
    tp_plan, device_map, device_mesh = initialize_tensor_parallelism(tp_plan, tp_size=None)
  File "/usr/local/lib/python3.10/dist-packages/transformers/integrations/tensor_parallel.py", line 49, in initialize_tensor_parallelism
    raise OSError("Tensor parallel is only supported for `torch>=2.5`.")
OSError: Tensor parallel is only supported for `torch>=2.5`.
```
The torch version is `PyTorch version (accelerator?): 2.5.0a0+e000cf0ad9.nv24.10 (CUDA)`.
The startup command is as follows:
```
accelerate launch --config_file "deepspeeds.yaml"  run_clm.py \
    --do_train \
    --do_eval \
    --model_name_or_path model_to_ft \
    --trust_remote_code True \
    --torch_dtype auto \
    --train_file train.json \
    --validation_file val.json \
    --streaming True \
    --block_size 8192 \
    --output_dir ./output \
    --weight_decay 0.01 \
    --learning_rate 5e-6 \
    --lr_scheduler_type "cosine" \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 16 \
    --gradient_checkpointing True \
    --gradient_checkpointing_kwargs '{"use_reentrant": true}' \
    --max_grad_norm 0.5 \
    --warmup_ratio 0.03 \
    --bf16 \
    --num_train_epochs 1 \
    --aux_lambda 1e-5 \
    --eval_strategy "steps" \
    --eval_steps 5000 \
    --save_strategy "steps" \
    --save_steps 1000 \
    --report_to "tensorboard" \
    --save_total_limit 2 \
    --logging_steps 10 \
    --log_level "info" \
    --include_num_input_tokens_seen True \
    --save_only_model False
```

### Expected behavior

I didn't set any parameters related to TP, so why is TP being used by default? How can I resolve this issue?

Comments:
Thanks for the report @mxjmtxrm . Make sure you are indeed running this with the latest transformers. This shouldn't happen if you didn't specify `tp_plan` in `from_pretrained`.