Add OLMoE
Author: Muennighoff
Pull request: yes

# What does this PR do?

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

The model will be released in ~1 week - can we already review this so that we can merge right upon release?

Comments:
> Hey! Feel free to ping me for a review once ready!

It's ready! :) I will update the README & double-check the slow tests once the model is released if that's fine!

@ArthurZucker would be great if we could get it reviewed soon 😇

We'll release the model on Tuesday, would be amazing if we could have this approved by then!

Oups sorry reviewing now

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32406). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

COuld you make sure you rebased and CIs are green? Will review again after that!

Great, all passing! 🙌

Thanks for your contribution! 🔥 

Is it a problem on my end if I notice that a higher native batch size during training results in higher losses? (With DeepSpeed)

![image](https://github.com/user-attachments/assets/be67f606-ec29-4b74-8f5d-813145ddf2d9)

Purple line is higher native bs FFT, orange line is gradient accumulation instead of a higher native batch size.

I'm guessing expert routing parallelism is maybe not properly handled for bs>1?

cc @Muennighoff would have a lot more clues than me 😉 

> Is it a problem on my end if I notice that a higher native batch size during training results in higher losses? (With DeepSpeed)
> 
> ![image](https://private-user-images.githubusercontent.com/66376113/364742842-be67f606-ec29-4b74-8f5d-813145ddf2d9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU3NDgwNzcsIm5iZiI6MTcyNTc0Nzc3NywicGF0aCI6Ii82NjM3NjExMy8zNjQ3NDI4NDItYmU2N2Y2MDYtZWMyOS00Yjc0LThmNWQtODEzMTQ1ZGRmMmQ5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA3VDIyMjI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTgyZTU5ZDBjNmU4OTkzYzlhY2EzMjQ4NjllYWExYTIyZmRjOGVkZDI0MmNjY2FhZWUyNWE1ZDJhMTcyYmUzZjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.TXKWoM54cgRqpyQUubZ2LzUaVRUvizzjh1NdZ9V3v_A)
> 
> Purple line is higher native bs FFT, orange line is gradient accumulation instead of a higher native batch size.
> 
> I'm guessing expert routing parallelism is maybe not properly handled for bs>1?

Hm not sure about this - are other models the same for you in both scenarios? How about other MoEs?