Add model files for LLaMA-MoE
Author: Spico197
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR add a new model called LLaMA-MoE which is related to https://github.com/huggingface/transformers/issues/28306 .

LLaMA-MoE is a Mixture-of-Experts model based on LLaMA-2 where FFNs are split into experts.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case. https://github.com/huggingface/transformers/issues/28306
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

LLaMA-MoE is a text-based model on LLaMA-2 with MoE framework. Based on the PR template, I'd be appreciated if @ArthurZucker and @younesbelkada could participate in reviewing this code.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada

Documentation: @stevhliu and @MKhalusova

 -->


Comments:
Hi @Spico197, thanks for opening this PR! 

We have recently been trying to push for `model on the hub` and have as much support as we can there. It will also be easier to integrate it! Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models) if that sound good to you!

> Hi @Spico197, thanks for opening this PR! 
> 
> We have recently been trying to push for `model on the hub` and have as much support as we can there. It will also be easier to integrate it! Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models) if that sound good to you!

Hi there, thanks for your information~
We have already uploaded our model weights and custom model&config code files to the hub: https://huggingface.co/llama-moe

One of the model has been downloaded for 12.1k. In that case, we'd like to integrate the refined code files into the transformers lib for easier loading and developing.

@amyeroberts  Besides the downloading statistics, the llama-moe repo [LLaMA-MoE](https://github.com/pjlab-sys4nlp/llama-moe) has 635 stars, attracting broad interests in the MoE community.

Thanks for your response~ Besides sliding window and GQA, LLaMA-MoE has the following differences with Mixtral that are not compatible:

|Architecture|LLaMA-MoE|Mixtral|
|:-|:-|:-|
|Gate|MLP with non-linear activation|linear|
| Gate Training Noise | learnable noise w/ softplus activation  | No noise |
| Gate Probs | softmax after topk | softmax before topk |
|Load Balancing| gate load & gate importance balancing | Switch balancing|
|Different Expert Size|Configured by `size_experts`|Not Supported|

Thanks, then in that case make sure you isolate these difference in the code! This means using `Copied from Mixtral` wherever possible to facilitate our review and the understanding of users!

Actually the part of the code is adopted from LLaMA rather than Mixtral since our internal development is earlier than the Mixtral release. Can we apply `# Copied from LLaMA` comments into our code instead?

Mixtral has copied from Mistral which has copied from Llama. Let's try to use as many  # Copied from as posssible so please use Mixtral 🤗 This helps everyone understand the diffs with current architecture w.r.t. the release date!

Got it. Thanks for your explanation~

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.