Add Fill-in-the-middle training objective example - PyTorch
Author: tanaymeh
Pull request: yes

# What does this PR do?
This PR adds a Fill-in-the-middle training objective example to 🤗 transformers.

FIM objective was proposed in [Efficient Training of Language Models to Fill in the Middle](https://arxiv.org/abs/2207.14255). They showed that autoregressive language models can learn to infill text after applying a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end.

As discussed in #27059 

## Who can review?
@sayakpaul @ArthurZucker 

Comments:
This PR is currently in draft mode until I can get a GPU VM (shouldn't take more than a week at max) to test to if a model is training all fine using this script.

@sayakpaul @ArthurZucker This PR is ready for review. I have added two scripts: `run_fim.py` and `run_fim_no_trainer.py`. Both have almost the same code structure as `run_clm.py` and `run_clm_no_trainer.py` respectively.

I have adapted the FIM dataset transformation from the [SantaCoder repository](https://github.com/loubnabnl/santacoder-finetuning/blob/main/fim.py#L22C13-L83) with changes to make it work with dataset's `.map()` function.

Training in both scripts works but I have yet to train a proper model on a GPU to compare the performance. Let me know if I need to make any changes to the script.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27464). All of your documentation changes will be reflected on that endpoint.

@ArthurZucker @pacman100 Are there any issues in this PR? Kindly let me know!

> thanks for this! Could you also add maybe and example log from the run to the readme>?

@ArthurZucker, Since the training log is almost entirely the same as `run_clm.py` (the only difference is the dataset transformation for adding FIM tokens), should I instead add the parts of the log concerned with the training part only?

Sure 🤗 

@ArthurZucker I have added the `pad_to_multiple_of` argument based on the device (for GPU it will be 8 and for TPUs, it will be 128), you can see it [here](https://github.com/tanaymeh/transformers/blob/98a5976cc555696c2ae0e4eb891b9e77d989c291/examples/pytorch/language-modeling/run_fim.py#L521-L533).

I have added `attn_implementation` support in the models ([here](https://github.com/tanaymeh/transformers/blob/98a5976cc555696c2ae0e4eb891b9e77d989c291/examples/pytorch/language-modeling/run_fim.py#L504)) and the model's embedding resizing changes as you requested ([here](https://github.com/tanaymeh/transformers/blob/98a5976cc555696c2ae0e4eb891b9e77d989c291/examples/pytorch/language-modeling/run_fim.py#L529-L560)).

Please let me know if these are the correct changes or do I need to make any amends!

Hey! Sorry for the delay, it's a big PR I'll review it later this week 🤗 

@ArthurZucker Thanks for your review! I have added your suggested changes and also made sure all CIs are now green.

Just waiting for @pacman100's review to get this merged!

@tanaymeh just waiting to make sure the deepspeed thing works and we should be able to merge

@ArthurZucker @pacman100 Thanks for the reviews and sorry for the delay, I was busy with my coursework.

I have added the suggested changes and the deepspeed compatibility to both trainer and non-trainer scripts. Could you please review and merge?

TIA!

I already approved, I am just waiting for @pacman100's approval here! 🤗 


Thanks for approving the PR @ArthurZucker and @pacman100!

I have pushed all the changes and the PR should be ready to be merged 🤗

@tanaymeh Thanks for working on this! I'll merge as I can see the requested changes are added and I'm on repo watch this week :)