Adding the LXMERT pretraining model (MultiModal  languageXvision)  to HuggingFace's suite of models
Author: eltoto1219
Pull request: yes

LXMERT is considered a dual-stream language-vision model. Dual stream meaning that it uses a transformer encoder to perform self-attention between each modality of data and then a cross-modality transformer encoder for a fine-grained cross-attention between each modality. It's achieved tremendous (STOTA results) success across a wide variety of downstream tasks (GQA, VQA2, NLVR2). 

Here is the original link to the paper:  https://arxiv.org/pdf/1908.07490.pdf
Here is the link to the original implementation: https://github.com/airsplay/lxmert
and here is the weight to the original model weights: https://nlp1.cs.unc.edu/data/model_LXRT.pth

Please let me know if there is anything i missed, and I would be very grateful for any help if I end up running into any blockers, but I will do my best to follow the detailed instructions in the templates. 

This is also a work in progress!

Comments:
Thank you so much!!! I really appreciate you offering to help too!

To cover what you mentioned quickly, I can upload the model weights tomorrow! For updating the documentation, that should be no problem either. I think for the model outputs regarding this model, there probably is quite a bit of information to return (pooled output, hidden states, and attentions for the language, vision, and cross-modality encoders). I can see about adding these tomorrow and get your thoughts on that. And lastly, I am glad I can help with adding more model heads! 

I have a bit more testing to do especially with the tensorflow model, but i will see what I can get done and let you know if I run into any blockers or questions. Look forward to getting back to you!

Hi, sorry for the slight delay. I added a new dataclass for Lxmert outputs, added the model card, finished all tests for the torch model, among a couple of other things. 

For the tests, I have forgone the ModelMixin parent as I have found that some of the tests are hard to apply to lxmert. I have also temporarily forgone adding example commands for lxmert. Is this a neccesity, or would it be alright to leave these out?

I think I updated the documentation to the new standard, but if there are still some errors, any help would be appreciated!

I am running a donwstream task with the pretrained weights right now to ensure that the results are still the same! I will get back to you with these in about a day!

Hi thank you so much for a really quick response and review! In my next commit, Ill implement the following changes and suggestions. If it wouldn't be to much to ask,  I think I could actually do with much help adding the mixin tester. Given that there are quite a few tests that seem lxmert seems to be incompatable with (for example I think one of the tests required that the LxmertConfig had the 'num_hidden_layers' attribute, which wouldn't be applicable since I suppose instead, we let the user specify the number of hidden layers for each of the visual, language, and cross-modality encoders. You're judgement on deciding what makes sense with regards to Lxmert test compatibility is probably greater than mine. 

Also just one last implementation detail that I should probably bring up is that for the output of the cross-modality encoder attentions, I only output the attentions when the language-hidden states are used as the input to the cross attention. Since this encoder is used for bidirectional attention, I do not store the attentions when the visual-hidden states are used as the input. Also For every layer of the cross attention encoder, each of the visual and language states are further processed by a separate self-attention layer, and I do not keep track of the attention outputs for these either. The only reason I keep track of the language attentions when used as input to the cross-modality attention layer is because it is those hidden states that are used for downstream pooling. 

I have yet to change the output for the TFLxmertModeling classes, which I will probably add in the commit following the one that addresses your review, but it probably would save me some time if you were able to cover that and would be greatly appreciated aswell!

Commited the above changes, I need to do some testing with the conversion script again, but besides that. I will wait to change the tensorflow code and the modeling tests for your response. If I do make a change in my next commit to modeling_tf_lxmert, it will just be to the documentation

I also just added some functionality to edit the number of question answering labels, simialr to the "resize_output_embeddings" utility in the LxmertPretrainedModel class.  However, what I added seems a lot less complex than the process for resizing embeddings, so if you could take a look at these new functions, that would be awesome! I added some tests, and they do seem to work.

Hi! Thanks for the thorough explanation. I'm adding the common tests to the tests, and will report here. May I push directly on your fork?

Actually, since I'm bound to make changes to some of the files, I'd like you to review the changes I'm making so that we may discuss. I'll open a PR on your branch when it's in a good state.

Okay  sounds good! the output hidden states argument should be added in
about 30 minutes, and then I can see about adding the TF changes in the
next hour.

On Mon, Aug 10, 2020 at 8:12 AM Lysandre Debut <notifications@github.com>
wrote:

> Actually, since I'm bound to make changes to some of the files, I'd like
> you to review the changes I'm making so that we may discuss. I'll open a PR
> on your branch when it's in a good state.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/5793#issuecomment-671318378>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADLBORYS3OXI4RBT4LHTJLLR77P3FANCNFSM4O3KPDOA>
> .
>


Great, I'll take a look at doing the Mixin tomorrow once your changes are up!

the lxmert model in pytorch should be ready to go!

Just pushed the same changes for TF as the PyTorch ones in https://github.com/eltoto1219/transformers/pull/1, alongside docs changes and a few patches to the PyTorch version. 

I'll take care of the merge commit once everything is done, it's due to `isort==5` and `black==20.8b` being released.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5793?src=pr&el=h1) Report
> Merging [#5793](https://codecov.io/gh/huggingface/transformers/pull/5793?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/930153e7d2d658267b7630a047a4bfc85b86042d?el=desc) will **decrease** coverage by `1.74%`.
> The diff coverage is `79.63%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5793/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5793?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5793      +/-   ##
==========================================
- Coverage   79.36%   77.62%   -1.75%     
==========================================
  Files         157      161       +4     
  Lines       28569    29816    +1247     
==========================================
+ Hits        22675    23144     +469     
- Misses       5894     6672     +778     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5793?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/commands/convert.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy9jb252ZXJ0LnB5) | `27.58% <0.00%> (-1.51%)` | :arrow_down: |
| [src/transformers/modeling\_lxmert.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19seG1lcnQucHk=) | `70.01% <70.01%> (ø)` | |
| [src/transformers/modeling\_tf\_lxmert.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9seG1lcnQucHk=) | `88.31% <88.31%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.30% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `93.33% <100.00%> (+0.15%)` | :arrow_up: |
| [src/transformers/configuration\_lxmert.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2x4bWVydC5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `78.85% <100.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `97.82% <100.00%> (+2.27%)` | :arrow_up: |
| [src/transformers/tokenization\_lxmert.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbHhtZXJ0LnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yZWZvcm1lci5weQ==) | `16.87% <0.00%> (-79.30%)` | :arrow_down: |
| ... and [23 more](https://codecov.io/gh/huggingface/transformers/pull/5793/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5793?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5793?src=pr&el=footer). Last update [930153e...4ed21b4](https://codecov.io/gh/huggingface/transformers/pull/5793?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).