Add OPT
Author: younesbelkada
Pull request: yes

# What does this PR do?

A PR to add OPT-350-m model on transformers library :hugs: 
We tested the logits and the generation script and the logits on the DGX and we match the results from [metaseq](https://github.com/patrickvonplaten/metaseq/blob/main/README.md)

## TO DOs

- [ ] code lint + integration tests
- [ ] discuss how to import the logits for the hardcoded tests
- [ ] correct documentation


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc  @LysandreJik  @patrickvonplaten @stas00 @sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger




Comments:
Also I have noticed something here: https://github.com/younesbelkada/transformers/blob/13632212d5bacc83bb9a9b8b6816a70bde865dbc/src/transformers/models/opt/modeling_opt.py#L249 
Initially in our implementation the attention mask has a shape `batch_size, 1, seq_len, seq_len` whereas in the metaseq implementation the masks is identitcal across the whole batch therefore has a shape of `seq_len, seq_len`. Is it a common practice that the attention mask is identical across the whole batch ?

> identitcal

I shouldn't but I think for causal LM it's fine to do this since the attention_mask is usually just the causal mask which is the same for every batch. Later tokens can never attend to earlier tokens when there is a causal mask, so it's often good enough to use that even when tokens are padded

Thank you very much for the explanation regarding the attention mask
Since I took the template from BART I think that it considered this attention strategy by default

I am facing [this CI issue](https://app.circleci.com/pipelines/github/huggingface/transformers/39343/workflows/6c764409-9785-4c68-8a29-e05ac79adb88/jobs/441415) when running `make repo-consistency` and I can't figure out what is happening.. Do you have any idea by change what could cause this issue?
![Screenshot from 2022-05-05 00-00-12](https://user-images.githubusercontent.com/49240599/166832823-e85fc11b-9004-49ef-a15d-e18a010368ff.png)
 

Linking this issue https://github.com/facebookresearch/metaseq/issues/31 
to understand how to load properly sharded models

Crashing the party here (apologies for not having more context on this codebase): is it normal for model imports to require >8k LOC added? Or is there something in particular about OPT models that requires all this additional glue code? 

If this is is OPT's fault - is there anything we can do (on the [metaseq](https://github.com/facebookresearch/metaseq) side) to bridge this gap?

> Crashing the party here (apologies for not having more context on this codebase): is it normal for model imports to require >8k LOC added? Or is there something in particular about OPT models that requires all this additional glue code?
> 
> If this is is OPT's fault - is there anything we can do (on the [metaseq](https://github.com/facebookresearch/metaseq) side) to bridge this gap?

Hey @suchenzang,

Cool that you're taking a look here! There is lot of code that was copied in a first pass from Bart that should be removed. This will make the code much more concise. Also note that we'll add the model both in TF and Flax as well. 

On the other hand, you guys are the authors of OPT, so you should lead the addition of OPT to Transformers if you want to. Would you like to open a PR yourself instead that adds the model instead of us doing it? We took the initiative here because of timing, but if you would like to take over the integration, please let us know :-)

Just cleaned up a bit the code - hopefully more tests should pass now - I have to add and correct some documentation

> On the other hand, you guys are the authors of OPT, so you should lead the addition of OPT to Transformers if you want to. Would you like to open a PR yourself instead that adds the model instead of us doing it? We took the initiative here because of timing, but if you would like to take over the integration, please let us know :-)

@patrickvonplaten Not at all - love seeing this initiative (and we likely don't have bandwidth to take on integration work)!  We have things to clean up on the metaseq side too, so I'm generally looking for feedback on this front as more folks try out our models.

Just out of curiosity, and to prevent duplicate work: Since fairseq models can be converted 1 on 1 to the XGLM model type, would that be possible with this one too or is this model too different to warrant a totally new model type?
Secondly, this is just the 350m model. Are other variants of this model going to be supported too? Hate to see this merged without full OPT model support.

_The documentation is not available anymore as the PR was closed or merged._

Here the checkpoints that should be converted: https://huggingface.co/patrickvonplaten/opt_metaseq_6700m/tree/main/model

Should we merge? 🤗

Actually one sec, I'll update the model docs a bit again