accelerate training confusion
Author: zwhus
Pull request: no

I have some questions about training with accelerate.

When I use transformers==4.28.0.dev0 and configure accelerate, the specific configuration is as follows:
```
- `Accelerate` version: 0.23.0
- Platform: Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- Numpy version: 1.22.4
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 1007.61 GB
- GPU type: NVIDIA A800-SXM4-80GB
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: FSDP
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - fsdp_config: {'fsdp_auto_wrap_policy': 'TRANSFORMER_BASED_WRAP', 'fsdp_backward_prefetch_policy': 'BACKWARD_PRE', 'fsdp_forward_prefetch': False, 'fsdp_offload_params': False, 'fsdp_sharding_strategy': 1, 'fsdp_state_dict_type': 'SHARDED_STATE_DICT', 'fsdp_sync_module_states': True, 'fsdp_transformer_layer_cls_to_wrap': 'LlamaDecoderLayer', 'fsdp_use_orig_params': False}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
```

And I can run the following command :
`
accelerate launch --num_processes 4 --main_process_port 23786 xx.py config/xx.py 
`

But in 4.28.0. Dev0 code not initialize Accelerate（https://github.com/huggingface/transformers/blob/v4.28.0/src/transformers/trainer.py#L213), may I ask why accelerate can be started and whether FSDP in it works?

Because I observed in transformers = = 4.31.1 (https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/trainer.py#L209), It seems reasonable that accelerate and FSDP are initialized in the code.
Looking forward to your reply.

Comments:
Hey, pretty sure you can still launch with `accelerate launch` so what is the issue you are having? 

ALso would be relevant to use the latest version of transformers! 

I would like to know whether the model is using FSDP for training after I start accelerate launch normally and configure FSDP under transformers= 4.28.0.Dev0（https://github.com/huggingface/transformers/blob/v4.28.0/src/transformers/trainer.py#L213), because the code of this version does not initialize this part.


I don't quite understand whether deepspeed or FSDP can be used normally during training after deepspeed or FSDP is set using `accelerate config` in transformers= 4.28.0.Dev0.

Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!
And it does not seem to be a bug or an issue
Thanks!

Ok， Thanks for your reply


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.