The `Trainer` only save the model parameters when `is_fsdp_enabled` is True
Author: calico-1226
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The bug occurred when I was using `transformers.Trainer` to train a `LlamaForSequenceClassification` model with the FSDP arguments `--fsdp "full_shard auto_wrap --fsdp_transformer_layer_cls_to_wrap "LlamaDecoderLayer"`. 
Specifically, when I used the `Trainer.save_model()` function to save the training results to `output_dir`, it only stored the model weights, without the corresponding model config, tokenizer, and training arguments. This issue only occurred when I trained the model using FSDP, but when not using FSDP, all of these components were saved correctly.

I located the corresponding code section in `Trainer.save_model()`

```python
elif (
    ShardedDDPOption.ZERO_DP_2 in self.args.sharded_ddp
    or ShardedDDPOption.ZERO_DP_3 in self.args.sharded_ddp
    or self.fsdp is not None
    or self.is_fsdp_enabled
):
    if self.is_fsdp_enabled:
        os.makedirs(output_dir, exist_ok=True)
        self.accelerator.state.fsdp_plugin.save_model(self.accelerator, self.model, output_dir)
```

and section in `FullyShardedDataParallelPlugin.save_model()` of `accelerate-0.20.3`

```python
def save_model(self, accelerator, model, output_dir, model_index=0):
    from torch.distributed.fsdp.fully_sharded_data_parallel import FullyShardedDataParallel as FSDP
    from torch.distributed.fsdp.fully_sharded_data_parallel import StateDictType

    if is_torch_version("<=", "1.13.5"):
        with FSDP.state_dict_type(model, self.state_dict_type, self.state_dict_config):
            state_dict = model.state_dict()
    else:
        FSDP.set_state_dict_type(model, self.state_dict_type, self.state_dict_config)
        state_dict = model.state_dict()

    if self.state_dict_type == StateDictType.FULL_STATE_DICT:
        weights_name = f"{MODEL_NAME}.bin" if model_index == 0 else f"{MODEL_NAME}_{model_index}.bin"
        output_model_file = os.path.join(output_dir, weights_name)
        if accelerator.process_index == 0:
            print(f"Saving model to {output_model_file}")
            torch.save(state_dict, output_model_file)
            print(f"Model saved to {output_model_file}")
    else:
        weights_name = (
            f"{MODEL_NAME}_rank{accelerator.process_index}.bin"
            if model_index == 0
            else f"{MODEL_NAME}_{model_index}_rank{accelerator.process_index}.bin"
        )
        output_model_file = os.path.join(output_dir, weights_name)
        print(f"Saving model to {output_model_file}")
        torch.save(state_dict, output_model_file)
        print(f"Model saved to {output_model_file}")
```

`FullyShardedDataParallelPlugin.save_model()` only saves the weights of the model, so we need to manually save other components.

### Expected behavior

Save the corresponding model config, tokenizer, and training arguments together with the trained model when using FSDP.

Comments:
cc @pacman100 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.