remove SharedDDP as it is deprecated
Author: ji-huazhong
Pull request: yes

# What does this PR do?

As mentioned previously([see](https://github.com/huggingface/transformers/pull/24825)), fairscale's ShardedDDP is deprecated,  and PyTorch FSDP is the recommended method for scaling to large NN models. Now it's time to say goodbye to this library👋. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerz Good day. Could you please review this PR? Thanks😄 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerz and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->




Comments:
@sgugger sorry for bothering you, but would you mind taking a look at this PR?

cc @muellerzr and @pacman100 who will take over the Trainer.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25702). All of your documentation changes will be reflected on that endpoint.

Rebase my commits to master HEAD to fix the merge conflict.
BTW, Is this PR still under reviewing? Any review suggestions? Please let me know if there is anything else that needs to be done.

@muellerzr Could you please take a second look at this PR? 
Some modifications were made based on the code review comments.

I've tested this PR using 4xA100-80G on [FastChat](https://github.com/lm-sys/FastChat) with the following scripts
```
CUDA_VISIBLE_DEVICES=4,5,6,7  torchrun --nproc_per_node=4 --master_port=20001 fastchat/train/train_mem.py \
    --model_name_or_path lmsys/vicuna-7b-v1.5  \
    --data_path data/dummy_conversation.json \
    --bf16 True \
    --output_dir output_vicuna \
    --num_train_epochs 1 \
    --max_steps 10 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1200 \
    --save_total_limit 10 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard offload auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing False \
    --lazy_preprocess True

```

@pacman100 Could you please take a look once again :-) I think it's ready to be merged.

Resolve merge conflicts by rebasing to main branch

@pacman100 Thanks for your comments and patience, I've reverted the modifications to the IPEX logic

Rebasing my commits to master HEAD and resolving merge conflicts

Hi there. This PR is approved and the tests are green :D
cc @muellerzr and @pacman100 

This PR is approved and the tests are green. @muellerzr Could you help to merge it?


Thank you @statelesshz!

Let me rebase quickly and merge if tests are green