Added model Sigma-MoE
Author: jubueche
Pull request: yes

# What does this PR do?

Added model from "[Approximating Two-Layer Feedforward Networks for Efficient Transformers](https://openreview.net/pdf?id=zM3mlyflTt)" and replicated experiments on WikiText-103.

The Sigma-MoE is different to the conventional Switch-like architecture when it comes to initialisation of the expert-weights, the routing function and the load balancing loss. Using the sigmoid function instead of the softmax avoids competition between the experts and leads to better training behaviour. Furthermore, Sigma-MoE employs a simple load balancing function that simply uses the entropy of the router outputs as a regulariser.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Thrilled to see this! Feel free to ping me for reviews! 

Thanks, I will.

@ArthurZucker in the tests, I am getting
```
RuntimeError: Failed to import transformers.models.sigma_moe.modeling_sigma_moe because of the following error (look up to see its traceback):
'str' object has no attribute 'impl'
```
but I am not sure how to debug that. Also regarding the tests that I have, they are very scarce right now. I also got an error that I don't have `all_model_classes`. What do you recommend for the test structure?

Also, the MoE layer has a special triton and CUDA implementation (CUDA is a fallback in case triton is not installed). Will that create any problems with the tests? How should I address this?


Alright few things here! I realized I am not sure there exists any publicly available checkpoints (checked the original repo and does not seem like they released weights no?), so would actually not be a great addition to `transformers` 😢 That's not a problem! The easiest way to distribute this model is to add it to the hub using [this tutorial ](https://huggingface.co/docs/transformers/custom_models)! You won't face all the issue with the CI and can make it work a lot faster! 
WDYT? 🤗 

I was actually in contact with Robert Csordas (author of the paper) while I was implementing this and he could probably provide some checkpoints. But I can also take a look at distributing it via the hub, no problem.
Let me know what you think: If we have checkpoints would the hub still be better or should we then proceed with a PR?

If we have checkpoints it's a bit up to you! It's easier to first share on the hub and if it is popular  in the community we add support for it here! 

Ok, then let’s do the hub! Thanks
________________________________
From: Arthur ***@***.***>
Sent: Wednesday, January 31, 2024 8:09:35 AM
To: huggingface/transformers ***@***.***>
Cc: Julian Büchel ***@***.***>; Author ***@***.***>
Subject: [EXTERNAL] Re: [huggingface/transformers] Added model Sigma-MoE (PR #28765)

If we have checkpoints it's a bit up to you! It's easier to first share on the hub and if it is popular in the community we add support for it here! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!2G-qyhSVoZJQUO-RsiKtBhwlIPf4lEj8plGSPope49wMTO_79upgdiJE8U-wHDT3HjuN-RRgBD_NiGHnuboAftpDMFAi7G9cAkto5IZ_eMvOUyGX$>
Report Suspicious

ZjQcmQRYFpfptBannerEnd

If we have checkpoints it's a bit up to you! It's easier to first share on the hub and if it is popular in the community we add support for it here!

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/pull/28765#issuecomment-1918513124>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHK2FWM56KKVRC4DRHP6BZTYRHU27AVCNFSM6AAAAABCQEEG5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJYGUYTGMJSGQ>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


> First, make sure your model is fully defined in a .py file. It can rely on relative imports to some other files as long as all the files are in the same directory (we don’t support submodules for this feature yet).

My model has CUDA and triton folders. Will that be a problem? 

Should be alright even on the hub! I would do something similar to what was done for `RWKV`which has custom kernels 🤗 

So I followed [this](https://huggingface.co/docs/transformers/custom_models) tutorial on how to also push the code to the hub, but when I try to load it, it can't find some of the files. When I looked at the code being pushed, it was just the modeling and the config one. This is my code structure:
```
.
- push_to_hub.py
- configuration_sigma_moe.py
- modeling_sigma_moe.py
- modeling_outputs.py
- moe_layer.py
- __init__.py
- triton_src/cvmm.py
- cuda_src/cvmm/cvmm.cu
- cuda_src/cvmm/cvmm.py
```

I saw in the tutorial that pushing packages is not supported, which is why I asked my previous question and it doesn't seem to work.
Putting all the code in the modeling file seems weird. I looked at the RWKV case, but I see that they are integrated into HF directly. They have the kernel in `src/transformers/kernels`, so I don't understand what you mean. Do you want me to put my kernels there?

Ok I am creating a private package for now that I will put on github.

Sorry didn’t have time to come back to this! 
‘I think you can push kernels but not with model.push to hub 

Feel free to ping me for any additional help will be glad to help!