Add RemBERT model code to huggingface
Author: Iwontbecreative
Pull request: yes
Labels: name: WIP

Add RemBERT model to Huggingface ( https://arxiv.org/abs/2010.12821 ).

This adds code to support the RemBERT model in Huggingface.

In terms of implementation, this is roughly a scaled up version of mBERT with ALBERT-like factorized embeddings and tokenizer.

Still needs to be done:
- [x] Check results validity
- [x] Upload model to model hub
- [x] FastTokenizer version
- [x]  More testing
- [x] TF code

Fixes #9711 


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@LysandreJik seems appropriate here. 

Comments:
@LysandreJik I've been mostly following: https://huggingface.co/transformers/add_new_model.html so far.
Trying to add the tokenizer. I think I am good for the slow one but not sure what to do for the fast one, in particular how to generate the tokenizer.json style files (e.g: https://huggingface.co/t5-small/resolve/main/tokenizer.json ). Do you have any pointers to that?

I also see that the doc mentions that there is no fast version for sentencepiece, which this model uses. Is that the case given T5 seems to have one?

Edit: Seem to have found a way to add a FastTokenizer version, doc still seems out of sync.

For the TF code, I'm struggling a bit to initialize an output embedding layer in `TFRemBertLMPredictionHead()` that interacts well with  `_resize_token_embeddings`. Welcoming any suggestions as what the right approach there is.

@LysandreJik : They are still some things to iron out but I think this is ready for a first look.

What is missing:
- Model hub upload 
- Minor discrepancy between model and original tf implementation


What I'd like some input on:
- I'm having some issue on the `TFRemBertLMPredictionHead` implementation. I'd like to initialize a new projection from hidden_size to vocab_size (embeddings are decoupled) but I'm struggling to find how to make my implementation compatible with all the `get_bias`, `set_bias` details so that it's `resize_embeddings` friendly. Any chance you could help here? This is the culprit for tests failing AFAICT.
- Model hub upload: should this be done on top level (how) or on the Google org model hub?
- I'm finding a discrepancy between this implementation and the original tf one. Results are exactly equal up to the first hidden layer (so embeddings and upprojection). On the first layer it differs but by small amounts (~0.002), difference eventually increases up to 0.007. Any idea what are common culprits here? This is just the standard BERT model and differences are small so maybe numerical stability?


> Model hub upload: should this be done on top level (how) or on the Google org model hub?

Toplevel models were for legacy (historical) integrations and we now namespace all models. If this work was conducted at Google yes google is the right namespace! Do you want us to add you to the `google` org?

> A difference of *e-3 doesn't look too bad, but looking at the integration test you have provided, it seems that the difference is noticeable. Is it possible that a bias is missing, or something to do with attention masks?

Not impossible but given the transformer section is simply Bert I doubt it. Also does seem like the results would change more.
 
> If it proves impossible to get the two implementations closer to each other, then we'll rely on a fine-tuning tests: if we can obtain similar results on a same dataset with the two implementations, then we'll be good to go.

I've tried to do that for a bit, unfortunately hard to fine-tune this model on a colab on XNLI (training gets interrupted too early on). Will try to see if I can get a better finetuning setup.




> > Model hub upload: should this be done on top level (how) or on the Google org model hub?
> 
> Toplevel models were for legacy (historical) integrations and we now namespace all models. If this work was conducted at Google yes google is the right namespace! Do you want us to add you to the `google` org?

That would be helpful, though I'm no longer affiliated with Google so not sure what the usual policy is there. If it is ok that will be easier than having to send the checkpoints to @hwchung so he uploads them.

> That would be helpful, though I'm no longer affiliated with Google so not sure what the usual policy is there.

Ultimately the org admins should decide, but for now I think it's perfectly acceptable if you're a part of the org. I added you manually.

@Iwontbecreative I opened a PR on your branch that should fix all the failing tests here: https://github.com/Iwontbecreative/transformers/pull/1

I've separated each test suite (TF, style, docs) in three commits if you want to have a look at smaller portions at a time.

Thanks Lysandre. Have not forgotten about this issue, just need one more approval from Google to open source the checkpoint so waiting for this.

Sure @Iwontbecreative, let us know when all is good!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Reopen: this is still in progress, just got the open sourcing done from google last Friday. Travelling for a few days and then can tackle this. See: https://github.com/google-research/google-research/tree/master/rembert

That's great news @Iwontbecreative!

(Finally) some updates on this now that the Google open sourcing is done.

* Updated the PR for the lates transformers
* Included @LysandreJik's change (manually instead of merging your PR since I messed up the order, sorry about that)
* Uploaded the model to the model hub under `iwontbecreative/rembert`

Main issue that still exists: the discrepancy between my implementation and the tf one. Welcoming ideas on this one. The code should now be easier for everyone to test now that the tf version is available and mine is uploaded to the model hub.

@LysandreJik think this is ready for another look. Thanks for the patience here!


Welcome back! We'll take a look shortly. Do you have an example of code to run to quickly test this version with the TF one?

Sadly the tensorflow code to run the model is not available externally. 
I tried to replicate for some time this morning but it is hard because it relies on https://github.com/google-research/bert which has dependencies that are not available on pip anymore...

The main change to the modelling code is here:
https://github.com/google-research/bert/blob/master/modeling.py#L813-L823
needing to be replaced with:
https://github.com/google-research/albert/blob/master/modeling.py#L1083-L1088
on the tokenization front, it is mainly replacing `BertTokenizer` with `AlbertTokenizer`

I do however have example inputs and outputs run by my coauthor:

### Model outputs
Example modelling outputs at several layers for different input_ids:
https://pastebin.com/t9bPFmeM 
This is the  `[batch, length, :3]` section of the  `[batch, length, hidden]` outputs.

### Tokenization outputs
https://pastebin.com/j6D6YE1e

Fine-tuning was, here is what I was able to run, comnparing performance on XNLI:

https://docs.google.com/spreadsheets/d/1gWWSLo7XxEZkXpX272tQoZBXTgs96IFvh-fwqVqihM0/edit#gid=0

Performance matches in English but does seem to be lower on other languages. We used more hyperparam tuning at Google but I do not think that explains the whole difference for those languages. I think there might be a subtle difference that is both causing the results to differ slightly and the worse fine-tuning outcomes. The model is still much better than random so most of it should be there.

Performance does look pretty similar, and good enough for me to merge it.

There are a few `# Copied from` statements missing though as said in my previous message, in both the PyTorch and TensorFlow implementations. Do you mind adding them? If you're lacking time let me know and I'll do it myself.

Hi @LysandreJik 

- Added copy statements
- Merged with last master
- Uploaded model to google org

Seems like it is mostly ready, though tests fail at the `utils/check_copies.py` stage of `make quality`.
I am actually not sure what the issue is in this instance, any chance you could help investigate/fix/merge after?

Actually, managed to find the issue

`utils/check_copies.py` crashes without a helpful error message if the "Copied from" statement is before the decorator. I was just overeager with my copied from statements.

Also renamed rembert-large to rembert since this is the only version we are open-sourcing at this time.

Edit: Not sure why the model templates check is failing, but think this should be ready for merge with one last review.  

Fantastic, thanks a lot @Iwontbecreative! I'll take a final look, fix the model templates issue and ping another reviewer.

@patrickvonplaten Thanks for the helpful feedback. Incorporated most of it. See the comment on possible needed changes to the cookiecutter templates to address on of your comments in the future.
For the discrepancy in results, see my answer above.

@sgugger Regarding older template: Yes, this PR ended up being delayed due to slow open-sourcing process at Google, so the templates were a bit out of date. Thanks for catching most of the mistakes.



Hi @LysandreJik, any last remaining steps before this can be merged? Would like to get this in to avoid further rebases if possible. 

I think this can be merged - thanks for your effort @Iwontbecreative, fantastic addition!

I'm not entirely sure why there was 88 authors involved or 250 commits squashed into a single one - but I did verify only your changes were merged.

Could you let me know how you handled the merge/rebasing of this branch so that I may understand what happened w.r.t the number of commits included?

I think I just merged the master's changes into my branch to ensure it was up to date with upstream. Maybe I needed to rebase?

Hi @Iwontbecreative thanks for adding the RemBERT model! Do you have a list of the 110 languages used in the pretraining of the model?

Sure, here's the list:

['af', 'am', 'ar', 'az', 'be', 'bg', 'bg-Latn', 'bn', 'bs', 'ca', 'ceb', 'co', 'cs', 'cy', 'da', 'de', 'el', 'el-Latn', 'en', 'eo', 'es', 'et', 'eu', 'fa', 'fi', 'fil', 'fr', 'fy', 'ga', 'gd', 'gl', 'gu', 'ha', 'haw', 'hi', 'hi-Latn', 'hmn', 'hr', 'ht', 'hu', 'hy', 'id', 'ig', 'is', 'it', 'iw', 'ja', 'ja-Latn', 'jv', 'ka', 'kk', 'km', 'kn', 'ko', 'ku', 'ky', 'la', 'lb', 'lo', 'lt', 'lv', 'mg', 'mi', 'mk', 'ml', 'mn', 'mr', 'ms', 'mt', 'my', 'ne', 'nl', 'no', 'ny', 'pa', 'pl', 'ps', 'pt', 'ro', 'ru', 'ru-Latn', 'sd', 'si', 'sk', 'sl', 'sm', 'sn', 'so', 'sq', 'sr', 'st', 'su', 'sv', 'sw', 'ta', 'te', 'tg', 'th', 'tr', 'uk', 'ur', 'uz', 'vi', 'xh', 'yi', 'yo', 'zh', 'zh-Hans', 'zh-Hant', 'zh-Latn', 'zu']

cf https://github.com/google-research/google-research/tree/master/rembert