Edgeformer
Author: patrickvonplaten
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

[EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation](https://arxiv.org/abs/2202.07959)

EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation. Tao Ge and Furu Wei

March 2022: release code and pretrained checkpoints.

## Open source status

* [x] the model implementation is available: https://github.com/microsoft/unilm/blob/900f5416c8137a753b1c8f53cd5015d0ceca7061/edgelm/fairseq/models/transformer/transformer_legacy.py#L226
* [x] the model weights are available: https://github.com/microsoft/unilm/tree/900f5416c8137a753b1c8f53cd5015d0ceca7061/edgelm#pretrained-models
* [x] who are the authors: Maybe @gitnlp 

Happy to help with a model contribution here!

Comments:
@patrickvonplaten If I can work on this and contribute, do let me know. Meanwhile, I will proceed to read and understand the paper.

Hey @reichenbch feel free to work on this if you are interested. Patrick is on vacation this week so I would be happy to help with this :) 

@patil-suraj I was thinking first to read the paper once and then look into the available implementation (they are using fairseq library) and checkpoints. Is that the correct approach ? Secondly, what all things would I need for this ? Any model creation guide available ? I know model templates are available in the repo.

For implementing the model I would suggest a code-first approach. My approach is to

- First  setup the original code base
- Load model and be able to do inference and inspect the outputs, intermediate values.
- Add the modeling code for transformers
- Convert the weights
- Do forward pass using both the original model and transformers model, compare if the outputs match, if not then debug and iterate.

Here are some docs that might help when adding a new model :slightly_smiling_face:
- https://huggingface.co/docs/transformers/add_new_model
- https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_model#adding-a-new-model

Most seq2seq models in `fairseq` are similar to bart/mbart, so I would suggest to refer to those models and use the `transformers-cli add-new-model-like` command which can create a bart/mbart like template.

Hope this helps!

Hey @reichenbch how's it going ? Let us know if you need any help :) 

Hey @patil-suraj Work is in progress, I had the misfortune to witness some health issues sometime back. I will update the files and try to get back on track

Hope you are feeling okay now! And no rush, just wanted to check-in. Take your time  🤗 

Hey @patrickvonplaten @patil-suraj @reichenbch i am interested to work on this issue, let me know if I can contribute.

@inderpreetsingh01, feel free to open a PR if you want :-)

@inderpreetsingh01 @patrickvonplaten is anyone actively working on this issue. I was wondering if either I could take it up or shadow someone working on it. I'd like to start learning how to contribute models to huggingface.

@pramodith , I started working on it but got occupied with some personal things, I had gone through the resources shared what I understood from the paper:

EdgeFormer uses:
- Layer adaptation 
- Interleaved transformer decoder layer
- Load balanced and encoder favored parametrization
- Each parameter is used a minimum 4 to a max of 6 times

I am not clear on the layer adaptation part and couldn’t find any parameter related to that in fairseq or edge_architecture function used to define the model. 

Let me know if you want to discuss and work on it.


@inderpreetsingh01 I believe that for the layer adaptation technique new parameters are only required for the LoRA method. The parameters for this are defined in [this](https://github.com/microsoft/unilm/blob/db95173cd050de0a58fce48077026c2f0247f782/edgelm/fairseq/modules/transformer_layer.py#L70) file in the fairseq repository. The file also contains the code for the Interleaved decoder.

If you're busy with other things I can definitely have a go at adding this model to the huggingface repo.

@pramodith thanks for clearing it, i actually looked at the original fairseq repository which is not having the adaptation part. I can contribute on this, we can connect [here](https://join.slack.com/t/slack-9ke1548/shared_invite/zt-1eln74e78-JtH6aQ83yEJCiR6j89~~hw)

Let me know if you need any help :-) 

Hey @patrickvonplaten, I wanted to start porting the edgeformer model into the transformers library so I used the `transformers-cli add-new-model-like` command, however one of the questions that follows is `Please give a checkpoint identifier (on the model Hub) for this new model.` does this mean that I need to upload the pretrained weights file to the Huggingface hub?

Hey @pramodith, 

It means that you should specific the checkpoint name that you intend to you when uploading the weights to the Hub.

Hi @patrickvonplaten, @pramodith Is this issue still open? I would like to contribute but I don't see a related PR.