Add MinGRU
Author: jogonba2
Pull request: yes
Labels: name: New model

# What does this PR do?

Add `MinGRU` from the paper [Were RNNs all we needed?](https://arxiv.org/abs/2410.01201) by Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, and Hossein Hajimirsadegh.

MinGRU is a novel recurrent architecture such as S4, Mamba, and Aaren which (i) no longer need to BPTT and can be efficiently trained in parallel and (ii) use significantly fewer parameters than their traditional GRU counterpart. MinGRU is 175x faster than GRU for a sequence length of 512 while matching the empirical performance of recent sequence models.

The authors do not provide the original code nor the checkpoint of the model they trained to carry out the Language Modelling experiments in Section 4.3 of the paper. Still, the results are very promising. So, I also have trained a very small causal language model using the [Shakespeare dataset](https://huggingface.co/datasets/karpathy/tiny_shakespeare) they use in the paper and uploaded the weights to the [hub](https://huggingface.co/symanto/mingru-shakespeare).

Which I have done for `MinGRU` is listed below:

- [X] "Fix" an overflow issue in the parallel scan code from [Appendix B.1 Log-space Implementation](https://arxiv.org/pdf/2410.01201)
- [X] Added the Torch model skeleton to 🤗 Transformers for `Encoder`, `ForSequenceClassification`, `ForTokenClassification` and `ForCausalLM`
- [X] Ran forward() pass in 🤗 Transformers for all the model heads
- [X] Trained `MinGRU` for sequence classification, token classification, and causal language modeling using 🤗 Transformers' Trainer
- [X] Added tokenizer in 🤗 Transformers (~same as reported in the paper -BPE from GPT2-)
- [X] Uploaded model weights to the Hub

Some things that are pending and maybe I will need some help:

- [ ] Finish docstrings (`MinGRUConfig` and `MinGRUModel` docstrings do not match their signature, so the quality test fails)
- [ ] Attention masks are not implemented (especially useful for `SequenceClassification`, e.g., when pooling last then pick the last non-pad token)
- [ ] Cache is not implemented for decoding

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker

Anyone in the community is free to review the PR once the tests have passed.


Comments:
There is an implementation from lucidrains at https://github.com/lucidrains/minGRU-pytorch which follows the paper and suggestions from the authors (see https://github.com/lucidrains/minGRU-pytorch/issues/1); from a quick glance there should be some additions/fixes:
- conv1d
- prenorm instead of postnorm
- expansion factor for minGRU ops

For caching+padding you could also take a look at my repo over here: https://github.com/vasqu/minGRU-torch/tree/main (it's a hybrid cache but it hopefully makes sense what I do in the minGRU block).

I'm also not sure how well it fits into the transformers ecosystem since it usually depends on adapting existing models/code which is not the case here. 

> There is an implementation from lucidrains at https://github.com/lucidrains/minGRU-pytorch which follows the paper and suggestions from the authors (see [lucidrains/minGRU-pytorch#1](https://github.com/lucidrains/minGRU-pytorch/issues/1)); from a quick glance there should be some additions/fixes:
> 
>     * conv1d
> 
>     * prenorm instead of postnorm
> 
>     * expansion factor for minGRU ops
> 
> 
> For caching+padding you could also take a look at my repo over here: https://github.com/vasqu/minGRU-torch/tree/main (it's a hybrid cache but it hopefully makes sense what I do in the minGRU block).
> 
> I'm also not sure how well it fits into the transformers ecosystem since it usually depends on adapting existing models/code which is not the case here.

Thanks for your comment @vasqu!

I did a search trying to find existing implementations and I completely forgot the lucidrains' repos 🤦I will take a look to the Leo Feng's comments in more detail this weekend.

Apart from conv1d and the expansion factor, the prenorm vs postnorm seems to play a crucial role here. Without normalization I observed overflows in the parallel scan code, did you experienced the same?

Regarding the transformer's ecosystem, I have the same doubt. Since I've implemented it in an internal branch I thought it was a good idea to contribute for other people to train the model within the HF ecosystem, but maybe this do not completely fit the philosophy of transformers as you say.

Thanks again @vasqu  :)

@jogonba2 Ye, no worries ;) take your time and I think it's worth a shot even if it's "just" a fork/branch. The more the merrier!

Haven't had the time to train a model yet but that's really interesting that prenorm is as important as it seems to be the case here. I just thought that it followed the current trend where prenorm is considered better than postnorm, but it may have had more reasons. If I take a look at your code, it could be interpreted as another norm to fix the overflow (as you also mention) :eyes: 

Sounds good! Best of luck and ping me if you think this is ready for review! 🤗