FSDP of Trainer does not work well with Accelerate
Author: gouchangjiang
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.3
- Platform: Linux-6.6.97+-x86_64-with-glibc2.35
- Python version: 3.11.11
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.1+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@3outeille @ArthurZucker @SunMarc 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
"""
Simple example of training BERT with Transformers Trainer and FSDP
Uses random data for quick demonstration
"""

import torch
from transformers import (
    BertForSequenceClassification,
    BertTokenizer,
    Trainer,
    TrainingArguments,
)
from torch.utils.data import Dataset


# Create a simple dataset with random data
class RandomDataset(Dataset):
    def __init__(self, tokenizer, num_samples=1000, max_length=128):
        self.tokenizer = tokenizer
        self.num_samples = num_samples
        self.max_length = max_length
        
    def __len__(self):
        return self.num_samples
    
    def __getitem__(self, idx):
        # Generate random token IDs
        input_ids = torch.randint(
            0, self.tokenizer.vocab_size, (self.max_length,)
        )
        attention_mask = torch.ones(self.max_length)
        labels = torch.randint(0, 2, (1,)).item()  # Binary classification
        
        return {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
            "labels": labels,
        }


def main():
    # Initialize tokenizer and model
    model_name = "bert-base-uncased"
    tokenizer = BertTokenizer.from_pretrained(model_name)
    model = BertForSequenceClassification.from_pretrained(
        model_name, num_labels=2
    )
    
    # Create random datasets
    train_dataset = RandomDataset(tokenizer, num_samples=1000)
    eval_dataset = RandomDataset(tokenizer, num_samples=200)
    
    # Configure FSDP training arguments
    training_args = TrainingArguments(
        output_dir="./bert_fsdp_output",
        num_train_epochs=3,
        per_device_train_batch_size=8,
        per_device_eval_batch_size=8,
        logging_steps=50,
        eval_strategy="steps",
        eval_steps=100,
        save_steps=200,
        save_total_limit=2,
        
        # FSDP Configuration
        fsdp="full_shard auto_wrap",  # Enable FSDP with full sharding
        fsdp_config={
            "fsdp_transformer_layer_cls_to_wrap": ["BertLayer"],  # Wrap BERT layers
            "fsdp_backward_prefetch": "backward_pre",
            "fsdp_forward_prefetch": False,
            "fsdp_use_orig_params": True,
        },
        
        # Additional settings
        learning_rate=5e-5,
        warmup_steps=100,
        weight_decay=0.01,
        logging_dir="./logs",
        report_to="none",  # Disable wandb/tensorboard for simplicity
    )
    
    # Initialize Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
    )
    
    # Train the model
    print("Starting training with FSDP...")
    trainer.train()
    
    # Save the final model
    trainer.save_model("./bert_fsdp_final")
    print("Training completed!")


if __name__ == "__main__":
    # Note: Run this script with torchrun for multi-GPU training
    # Example: torchrun --nproc_per_node=2 train_bert_fsdp.py
    main()
```

torchrun --nproc_per_node=2 train_bert_fsdp.py

### Expected behavior

It will fail silently. The trace stack, 
```bash
W1210 12:49:05.011000 104846 site-packages/torch/distributed/run.py:803] 
W1210 12:49:05.011000 104846 site-packages/torch/distributed/run.py:803] *****************************************
W1210 12:49:05.011000 104846 site-packages/torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W1210 12:49:05.011000 104846 site-packages/torch/distributed/run.py:803] *****************************************
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Starting training with FSDP...
Starting training with FSDP...
W1210 12:49:10.985000 104846 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 104934 closing signal SIGTERM
E1210 12:49:11.016000 104846 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 0 (pid: 104933) of binary: /opt/conda/envs/py311/bin/python3.11
Traceback (most recent call last):
  File "/opt/conda/envs/py311/bin/torchrun", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/torch/distributed/run.py", line 936, in main
    run(args)
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train_bert_fsdp.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-12-10_12:49:10
  host      : full-node-0
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 104933)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
```
I debuged a little, the program crashes at [self.optimizer = self.accelerator.prepare(self.optimizer)](https://github.com/huggingface/transformers/blob/6564633bc87b1f808de5773a74db2ec01f681520/src/transformers/trainer.py#L2342), and it should not go to this line, the judge `use_accelerator_prepare = model is self.model` should be False, but it's True since FSDP seems does not work.

Comments:
You need to run with accelerate launch instead. Try to check trl script for a working fsdp script 

Thank you for the suggestion.  `accelerate config` generated the config file,

```python
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
enable_cpu_affinity: false
fsdp_config:
  fsdp_activation_checkpointing: false
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_reshard_after_forward: true
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_version: 2
machine_rank: 0
main_training_function: main
mixed_precision: 'no'
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
``` 

and I run the training by `accelerate launch train_bert_fsdp.py`, got the same error, 

```bash
Starting training with FSDP...
Starting training with FSDP...
Starting training with FSDP...
Starting training with FSDP...
W1211 07:00:55.037000 43130 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 43287 closing signal SIGTERM
W1211 07:00:55.038000 43130 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 43288 closing signal SIGTERM
W1211 07:00:55.038000 43130 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 43290 closing signal SIGTERM
E1211 07:00:55.215000 43130 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 2 (pid: 43289) of binary: /opt/conda/envs/py311/bin/python3.11
Traceback (most recent call last):
  File "/opt/conda/envs/py311/bin/accelerate", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
    args.func(args)
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1268, in launch_command
    multi_gpu_launcher(args)
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/accelerate/commands/launch.py", line 899, in multi_gpu_launcher
    distrib_run.run(args)
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py311/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train_bert_fsdp.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-12-11_07:00:55
  host      : full-node-0
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 43289)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.