Community contribution - `BetterTransformer` integration for more models!
Author: younesbelkada
Pull request: no
Labels: name: Good First Issue

## `BetterTransformer` integration for more models!

`BetterTransformer` API provides faster inference on CPU & GPU through a simple interface! 

Models can benefit from very interesting speedups using a one liner and by making sure to install the latest version of PyTorch. A complete guideline on how to convert a new model has been created on the [BetterTransformer documentation](https://huggingface.co/docs/optimum/bettertransformer/tutorials/contribute)!

Here is a list of models that could be potentially supported, pick one of the architecture below and let's discuss about the conversion! 

Text models 🖊️  :

- [x] FSMT - [FSMTEncoderLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/fsmt/modeling_fsmt.py#L397) / @Sumanth077  https://github.com/huggingface/optimum/pull/494
- [ ] MobileBERT - [MobileBertLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/mobilebert/modeling_mobilebert.py#L498) / @raghavanone https://github.com/huggingface/optimum/pull/506
- [x] MBart - [MBartEncoderLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/mbart/modeling_mbart.py#L296) + [M2M100EncoderLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/m2m_100/modeling_m2m_100.py#L345) / https://github.com/huggingface/optimum/pull/516 @ravenouse 
- [ ] ProphetNet - [ProphetNetEncoderLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/prophetnet/modeling_prophetnet.py#L1130)
- [x] RemBert - [RemBertLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/rembert/modeling_rembert.py#L415) / @hchings  https://github.com/huggingface/optimum/pull/545
- [ ] RocBert - [RocBertLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/roc_bert/modeling_roc_bert.py#LL519C7-L519C19) / @shogohida  https://github.com/huggingface/optimum/pull/542
- [ ] RoFormer - [RoFormerLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/roformer/modeling_roformer.py#L448)
- [x] Tapas - [TapasLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/tapas/modeling_tapas.py#L524) / https://github.com/huggingface/optimum/pull/520


Vision models 📷  :

- [ ] Detr - [DetrLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/detr/modeling_detr.py#L610)
- [ ] Flava - [FlavaLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/flava/modeling_flava.py#L597) / https://github.com/huggingface/optimum/pull/538
- [x] GLPN - [GLPNLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/glpn/modeling_glpn.py#L292) (cannot be supported)
- [x] ViLT - [ViLTLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/vilt/modeling_vilt.py#L472) / https://github.com/huggingface/optimum/pull/508

Audio models 🔉  :

- [ ] Speech2Text - [Speech2TextLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/speech_to_text/modeling_speech_to_text.py#L350)
- [ ] NEW: Audio Speech Transformer - [ASTLayer](https://github.com/huggingface/transformers/blob/f2e7d270ec795be09e6187dd2459edb43bd861c1/src/transformers/models/audio_spectrogram_transformer/modeling_audio_spectrogram_transformer.py#L274) / @ravenouse  https://github.com/huggingface/optimum/pull/548 

Let us also know if you think that some architectures can be supported that we missed. Note that for encoder-decoder based models below, we expect to convert the encoder only.

**Support for decoder-based models coming soon!**

cc @michaelbenayoun @fxmarty 

https://github.com/huggingface/optimum/issues/488

Comments:
> NotImplementedError: The Better Transformers implementation for the model DebertaV2Model has not beenimplemented yet. Please open an issue requesting the addition of this model with its `BetterTransformer`implementation.

It's not on your list, but would you complain if I did this for DebertaV2Model?

It is not in the list because `DebertaV2` does not have a regular attention mechanism, so it is not possible to use it with BetterTransformer.

Yes I second what @michaelbenayoun said, please see related: https://github.com/huggingface/optimum/issues/487

makes a lot of sense - sorry I should have thought about that a bit harder before posting!

I noticed that Better Transformers for the T5 model has not been implemented yet. Will it be implemented in the future (if possible)? Thanks.

Hi @GenVr 
Thanks a lot for your reply! Unfortunately `T5` cannot be supported because of the nature of its attention mechanism. In fact `T5` uses attention bias and this is not supported for `BetterTransformer`
Thanks!

Hi :) I would like to work on the implementation for [RemBertLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/rembert/modeling_rembert.py#L415).

What are the next steps in getting started?

Thank you!

Hey @RJZauner !
Thanks so much for your interest in helping us integrating more models for `BetterTransformer` ! 
RemBert seems to use the same attention mechanism as BERT, the only difference should be on the Embedding layer, which is fine for us! So I would say you can move ahead and start forking [optimum](https://github.com/huggingface/optimum) library, create a new branch and open a draft PR. Feel free to have some inspiration from what has been done by  https://github.com/huggingface/optimum/pull/494 and https://github.com/huggingface/optimum/pull/508 to see what exactly needs to be done ;) Ping us (myself, @michaelbenayoun & @fxmarty) whenever you feel that you need help!

Hi @younesbelkada, I would like to work on the easiest of the models mentioned above. Which one do you recommend? What I said might sound a bit weird but I want to tackle a simple one since I'm not very familiar with these models 🙏 

Hello, I would like to tackle the implementation for [TapasLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/tapas/modeling_tapas.py#L524).

May I ask you how I can start the further steps?

Thank you for your time.

Hi @shogohida and @JuheonChu ,

You can read this [page](https://huggingface.co/docs/optimum/bettertransformer/tutorials/contribute) for learning how to contribute. You can then open a PR with your code, and ask questions there, we will be glad to help!

Also @shogohida, I think they are all similar in terms of difficulty, so do not block on that, maybe choose a model with the modality the most familiar to you.

Seconding what @michaelbenayoun said, feel free to check some example PRs https://github.com/huggingface/optimum/pull/508 or https://github.com/huggingface/optimum/pull/494 for reference! 
@shogohida , you can take RocBERT, actually it copies from Bert so the conversion will be very easy :) 

Thanks guys for your replies! I will take RocBERT then!

Thanks @michaelbenayoun ! I will take TapasLayer !

Hi! Thank you so much for opening this issue. 

1. I was implementing the RemBERT and had some questions. But then I noticed that @RJZauner had already been working on that. I am going to hold my work on that and I am looking forward to see RJZauner's implementations!
2. I will work on the mBART.
3. I also found some dead links and some points unclear on this [page](https://huggingface.co/docs/optimum/bettertransformer/tutorials/contribute). How should I report and help to solve the problems I found? 

Hello @younesbelkada,

I would like to take [DetrLayer](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/detr/modeling_detr.py#L610). Nice tutorial btw 😀

Hi @blakechi !
Sure you can take it ;) let me know if you need help opening a PR!

Hi @ravenouse !
Thanks for your help! Yes you can take MBART ;) 
Regarding the dead link could you open an issue at optimum? 
Thanks!





> Hey @RJZauner !
> Thanks so much for your interest in helping us integrating more models for `BetterTransformer` !
> RemBert seems to use the same attention mechanism as BERT, the only difference should be on the Embedding layer, which is fine for us! So I would say you can move ahead and start forking [optimum](https://github.com/huggingface/optimum) library, create a new branch and open a draft PR. Feel free to have some inspiration from what has been done by [huggingface/optimum#494](https://github.com/huggingface/optimum/pull/494) and [huggingface/optimum#508](https://github.com/huggingface/optimum/pull/508) to see what exactly needs to be done ;) Ping us (myself, @michaelbenayoun & @fxmarty) whenever you feel that you need help!

Thank you for the info!

Hello @michaelbenayoun and @younesbelkada ! 

First time contributing for me :) 

I would like to handle the implementation for [Speech2Text](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/speech_to_text/modeling_speech_to_text.py#L350)

What are the first steps ? Create a PR ?

Thanks in advance.

> Hello @michaelbenayoun and @younesbelkada !
> 
> First time contributing for me :)
> 
> I would like to handle the implementation for [Speech2Text](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/speech_to_text/modeling_speech_to_text.py#L350)
> 
> What are the first steps ? Create a PR ?
> 
> Thanks in advance.

Hello, I am absolutely sure that they will give you a better suggestion than what I have.
I would like to share that it is good to read `CONTRIBUTING.md` in the transformer repository.
I read through every content very carefully and made my first contribution!

> > Hello @michaelbenayoun and @younesbelkada !
> > First time contributing for me :)
> > I would like to handle the implementation for [Speech2Text](https://github.com/huggingface/transformers/blob/95754b47a6d4fbdad3440a45762531e8c471c528/src/transformers/models/speech_to_text/modeling_speech_to_text.py#L350)
> > What are the first steps ? Create a PR ?
> > Thanks in advance.
> 
> Hello, I am absolutely sure that they will give you a better suggestion than what I have. I would like to share that it is good to read `CONTRIBUTING.md` in the transformer repository. I read through every content very carefully and made my first contribution!

Hello @JuheonChu :) 

I am definitely have a look at it ! thanks

Hi @lucaspct,

Yes the first steps would be to read [the guide explaining how to contribute to `optimum.bettertransformer`](https://huggingface.co/docs/optimum/bettertransformer/tutorials/contribute), and then opening a [PR on Optimum](https://github.com/huggingface/optimum/pulls), we will support you there!

Hi @younesbelkada @michaelbenayoun I'd love to take on the RoFormer model if it isn't claimed yet. Will open a PR after I read through the guide!

I would like to take a crack at the ProphetNet encoder if it has not been claimed yet 

Thank you very much @miyu386 & @adit299 !
Of course yes you can give a try on that ;) feel free to start to open a PR on `optimum` and we'll guide you from there 💪 

I would like to work on the `ASTLayer` if no one has taken it!

Hi @younesbelkada I'd like to tackle the `FlavaLayer` if it has not been taken!

Hi @katiele47 
Sure no problem! Feel free to open a PR and tag us there! I will update the table above once the PRs are open ;) 

Hi, @younesbelkada I'd like to take `GLPNLayer` if no one has claimed it. will open the PR soon for this :)

Hi @hazrulakmal !
Sure! Let me know once you open the PR ;) 

Hi! I'd love to contribute wherever I can be useful.

Hi @younesbelkada !
I found the `torch._transformer_encoder_layer_fwd()` function is called to execute forwarding in our `BetterTransformerBaseLayer`.  To better understand what's going on under the hood, I searched this function online but didn't find much information about it. Could you tell me where I can check the source code of it? 
Thank you so much! 

Hello,
I'd like to work on `RocBert` layer. I'll go over the contributing guide and open a PR. Anything else I need to go through as a starting point?

Hi @younesbelkada, I added a [PR](https://github.com/huggingface/optimum/pull/545) for RemBERT. Since RemBERT's primary changes are in embeddings (decoupling in/output embeddings, increasing output embeddings during pre-training to improve downstream performance, etc), the needed changes should be straightforward. But please kindly let me know if I missed anything. 

I also want to apologize to @RJZauner, I realized you've claimed RemBERT after re-reading this thread now. I'll be more careful next time!! And lmk if you want me to withdraw the PR to use your changes instead. If not, feel free to add on top of it and hopefully we can both learn together 🙏🙏.

Hi @M0315G 
Thanks so much! Unfortunately this is already a WIP from @shogohida here: https://github.com/huggingface/optimum/pull/542 
Feel free to take another model that is free ;) also more models will be added in the list as `transformers` is continuing integrating more models

Should I take RoFormer then?

Yes, I think that you can take this one. From my understanding of `RoFormer` it consists of a LM similar as BERT, but uses Rotary positional embedding and the attention mechanism is classic, i.e. similar as BERT. Ideally you can double check that too but for me `BetterTransformer` should be supported on `RoFormer` so I would say that you can take it yes

Understood. Will go through the docs and open a PR in optimum. Any other things I should take care of?

Just take some inspiration from other PRs and everything should go smoothly!

Hi @ravenouse !
From what I got, this function is a C++ binding of the transformer encoder operation that is first defined [here](https://github.com/pytorch/pytorch/blob/226e803ecb483d2488f2381afb86c1e849814714/aten/src/ATen/native/native_functions.yaml#L13272) and fully defined [here](https://github.com/pytorch/pytorch/blob/226e803ecb483d2488f2381afb86c1e849814714/aten/src/ATen/native/transformers/transformer.cpp#L64) as you can see, the whole transformer encoder operations (self attention + ffn) is defined in a single operation

Hi @hchings !
Just reviewed your PR ;) will merge that soon!


Hi @blakechi I saw that you wanted to work on DertLayer a week ago ? How is your progress now? Just asking cus If you are not working on it anymore, I’m more than happy to help you out with the conversion and take over the PR? :)

Hi @hazrulakmal 
I realized that [DPT model ](https://github.com/huggingface/transformers/blame/6a707cf586f37865d1a9fac02c56c43cf8dcf979/src/transformers/models/dpt/modeling_dpt.py#L311)(Depth Estimation model) can be supported too as it uses ViT as a backbone. Would you like to give it a try with this one instead? 🙏 
We are currently looking at speeding up this model so this will be a nice addition

Hi @younesbelkada, is there any model left that I can work on? Seems like the Detr was claimed a while ago but not sure if there is a PR opened for that. 

@younesbelkada yup! definitely. I can take this up! I checked on the encoder layer and it looks possible to integrate with BT, should I name the new class as `DPTViTLayerBetterTransformer`?

```
(encoder): DPTViTEncoder(
    (layer): ModuleList(
      (0): DPTViTLayer(
        (attention): DPTViTAttention(
          (attention): DPTViTSelfAttention(
            (query): Linear(in_features=32, out_features=32, bias=True)
            (key): Linear(in_features=32, out_features=32, bias=True)
            (value): Linear(in_features=32, out_features=32, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): DPTViTSelfOutput(
            (dense): Linear(in_features=32, out_features=32, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): DPTViTIntermediate(
          (dense): Linear(in_features=32, out_features=37, bias=True)
          (intermediate_act_fn): GELUActivation()
        )
        (output): DPTViTOutput(
          (dense): Linear(in_features=37, out_features=32, bias=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (layernorm_before): LayerNorm((32,), eps=1e-12, elementwise_affine=True)
        (layernorm_after): LayerNorm((32,), eps=1e-12, elementwise_affine=True)
      )
```


Very cool! Yes you can name it like this ;) Looking forward to seeing your PR 💪 

Hi @miyu386 
ViT-Hybrid has just been integrated to `transformers` would you like to take this one?
https://github.com/huggingface/transformers/blob/d151a8c55032d5a21800ea0813c4304af8b8e9f7/src/transformers/models/vit_hybrid/modeling_vit_hybrid.py#L362

> Hi @younesbelkada, I added a [PR](https://github.com/huggingface/optimum/pull/545) for RemBERT. Since RemBERT's primary changes are in embeddings (decoupling in/output embeddings, increasing output embeddings during pre-training to improve downstream performance, etc), the needed changes should be straightforward. But please kindly let me know if I missed anything.
> 
> I also want to apologize to @RJZauner, I realized you've claimed RemBERT after re-reading this thread now. I'll be more careful next time!! And lmk if you want me to withdraw the PR to use your changes instead. If not, feel free to add on top of it and hopefully we can both learn together 🙏🙏.

Hey :) don't sweat it - no need to withdraw your PR.

Your implementation looks great - thanks!

@younesbelkada Yes, I'd like to give it a try! Thanks

May I ask you a question if there is anyway that we can see a "BetterTransformer" features for models that already have like Tapas or FSMT?

Hi @JuheonChu , you can see it here: https://github.com/huggingface/optimum/pull/520/files , https://github.com/huggingface/optimum/pull/494/files

Hey @younesbelkada, happy to take a look at any remaining model that needs integration!

Same here @younesbelkada, any model you need

Hi @HVjay @mszsorondo don't hesitate to pick any model you are interested in and is using a classic encoder attention + feed-forward architecture! You can open an PR in Optimum, and if you need help we'll guide you from there.

> Understood. Will go through the docs and open a PR in optimum. Any other things I should take care of?

Hi @younesbelkada ,
Opened a PR for RoFormer integration for BetterTransformer form my other Github Account. But please kindly let me know if I missed anything.

Hi @fxmarty wanted to confirm that Conditional Detr - [`ConditionalDetrEncoderLayer`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/conditional_detr/modeling_conditional_detr.py#L763) can be supported!

Hi @HVjay,
Thanks for your interest! I think Detr can be supported as well as ConditionalDetr as it seems to use classic attention mechanism - this can be also confirmed by [the paper]( https://arxiv.org/pdf/2005.12872.pdf ) that states that the method uses classic transformer-based models. However note that only the encoder part can be converted.

Hi @mszsorondo,
Thank you for your message! Recently `BLIP` has been added, the model should support BetterTransformer integration (Vision + text)

Hi @HVjay ,

Actually there is already someone working on Detr, check: https://github.com/huggingface/optimum/pull/684 

Hi @younesbelkada , could I pick up RoFormer ?

@sushmanthreddy are you doing Detr anymore...? if doing please tell

@younesbelkada Hi 👋 could I take `Speech2Text` 🙂



@younesbelkada Hello, I would love to contribute to this issue. I am new to contributing in transformers. Can you please tell me which of the model layers are vacant I would like to take one up :)

@younesbelkada I would like to work on Detr. 

@mszsorondo Are you still working on it? There has not been any activity on your [PR](https://github.com/huggingface/optimum/pull/684) since Jan 8. I can pull from your PR and fix the failing tests.



@awinml I actaully submitted the pr for Detr Model 
- so i forgot to mention,sorry buddy
- you can look for other model available 
- [here is the pr](https://github.com/huggingface/optimum/pull/1022)

@dewasahu2003 No problem. 

Its always better to inform the original author and pull from their PR so they get due credit. Hence the question was aimed at @mszsorondo.

@younesbelkada Hey  👋

I have submitted the pr for  `BetterTransformer` for detr
I mentioned you there [PR](https://github.com/huggingface/optimum/pull/1022)
From next time i would keep in mind to ask pr authors

Hi, @younesbelkada I'd like to work on `ProphetNet` 😀

> @younesbelkada I would like to work on Detr.
> 
> @mszsorondo Are you still working on it? There has not been any activity on your [PR](https://github.com/huggingface/optimum/pull/684) since Jan 8. I can pull from your PR and fix the failing tests.

Go for it! Sorry for the delay

Hi @younesbelkada, @michaelbenayoun, and @fxmarty,

I would like to work on Speech2TextLayer. 

What are the next steps in getting started?

Thank you!

Hi! @younesbelkada @michaelbenayoun @fxmarty 
I'm interested in adding support for one of the models in the list. Although, I believe that the only model left might be Speech2TextLayer and has been claimed by @Jack-Chuang 

Hello @younesbelkada @fxmarty and @michaelbenayoun 
I would like to work on the `RoFormer` layer since I saw that someone had already worked on `ProphetNet`. Has the model been claimed ?

Hello @younesbelkada @fxmarty and @michaelbenayoun
I would love to help you with the integration of more models for `BetterTransformer`! I'm happy to take what is left since a lot of developers are already contributing to most of the models I think. Let me know if I can still help with something! 


@younesbelkada is there anything I can help with in this issue?

@younesbelkada  please could you update the original list of pending items?
Or has this project been stalled?

Is splade possible?

Hi @younesbelkada ! I'm new to the Open Source community but have good experience with torch, transformers, numpy, etc. can I be assigned the RoFormer task, I'd like to give it a shot!

Hi @younesbelkada,
Can I take up ProphetNet task? I'm new to open source and might take some time but eager to try my hands at this.

Hi everyone,
Sorry for the delay in replying to this issue and community contribution - we had some internal discussion and we decided to migrate the `BetterTransformer` API into transformers core by directly supporting `torch.scaled_dot_product_attention` in the modeling files. Check out this issue: https://github.com/huggingface/transformers/issues/26557 for more details and this PR for the PoC: https://github.com/huggingface/transformers/pull/26572 
We will possibly open a community contribution for that to extend the support for all architectures but not sure. I will keep you all posted!
Thanks again for all your effort and amazing contribution! 🎉 

Hi @younesbelkada, @michaelbenayoun, and @fxmarty
The model type vision-encoder-decoder is not yet supported to be used with BetterTransformer !!!
Hope you to support soon <3