TypeError: __init__() got an unexpected keyword argument 'forward_prefetch'
Author: Hiusam
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.13.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@AlexWertheim 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. run stanford-alpaca's training command: https://github.com/tatsu-lab/stanford_alpaca
```
torchrun --nproc_per_node=4 --master_port=<your_random_port> train.py \
    --model_name_or_path <your_path_to_hf_converted_llama_ckpt_and_tokenizer> \
    --data_path ./alpaca_data.json \
    --bf16 True \
    --output_dir <your_output_dir> \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2000 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LLaMADecoderLayer' \
    --tf32 True
```

### Expected behavior

```
Traceback (most recent call last):
  File "train.py", line 231, in <module>
    train()
  File "train.py", line 225, in train
    trainer.train()
  File "/home/projects/transformers/src/transformers/trainer.py", line 1644, in train
    return inner_training_loop(
  File "/home/projects/transformers/src/transformers/trainer.py", line 1731, in _inner_training_loop
    model = self._wrap_model(self.model_wrapped)
  File "/home/projects/transformers/src/transformers/trainer.py", line 1469, in _wrap_model
    self.model = model = FSDP(
TypeError: __init__() got an unexpected keyword argument 'forward_prefetch'
```
The error is raised at the trainer.py:
```
                if type(model) != FSDP:
                    # XXX: Breaking the self.model convention but I see no way around it for now.
                    self.model = model = FSDP(
                        model,
                        sharding_strategy=self.fsdp,
                        cpu_offload=cpu_offload,
                        auto_wrap_policy=auto_wrap_policy,
                        mixed_precision=mixed_precision_policy,
                        device_id=self.args.device,
                        backward_prefetch=self.backward_prefetch,
                        forward_prefetch=self.forword_prefetch,
                        limit_all_gathers=self.limit_all_gathers,
                    )
```
I think forward_prefetch is not supported in PyTorch1.12. Is there a possible solution to enable me to use FSDP with PyTorch 1.12? If not, I suggest adding some version-checking codes.

Comments:
FSDP support in Transformers requires PyTorch 1.12, so no. You should have hit [this error](https://github.com/huggingface/transformers/blob/55dae94c0ccd088003aa46bcecb2e55321a7f00b/src/transformers/trainer.py#L429) before anything else, not sure why you did not.

Hi, thanks for your reply. This is not an issue with FSDP support. It's an issue that FSDP does not support the keyword argument "forward_prefetch" in torch1.12

Hi, I met the same problem with transformers==4.27.1 and the solution is to degrade to transformers==4.26.1 .This may be a version compatibility issues for hugging face transformers .

Oh thanks for clarifying. cc @pacman100