CamembertForCausalLM
Author: patil-suraj
Pull request: yes

This PR adds `CamembertForCausalLM` by subclassing `RobertaForCausalLM`, so that it can be used with the `EncoderDecoderModel`.

@patrickvonplaten 

Comments:
@patrickvonplaten  didn't add any tests since it subclasses `RobertaForCausalLM` which is already tested.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6577?src=pr&el=h1) Report
> Merging [#6577](https://codecov.io/gh/huggingface/transformers/pull/6577?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/12d7624199e727f37bef7f53d527df7fabdb1fd6&el=desc) will **increase** coverage by `1.14%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6577/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6577?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6577      +/-   ##
==========================================
+ Coverage   79.18%   80.33%   +1.14%     
==========================================
  Files         156      156              
  Lines       28129    28132       +3     
==========================================
+ Hits        22275    22599     +324     
+ Misses       5854     5533     -321     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6577?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.28% <ø> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `78.73% <ø> (ø)` | |
| [src/transformers/modeling\_camembert.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jYW1lbWJlcnQucHk=) | `100.00% <100.00%> (ø)` | |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `28.84% <0.00%> (-58.66%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `69.06% <0.00%> (-29.32%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `71.84% <0.00%> (-23.17%)` | :arrow_down: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `88.26% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.59% <0.00%> (+0.68%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.29% <0.00%> (+0.97%)` | :arrow_up: |
| ... and [9 more](https://codecov.io/gh/huggingface/transformers/pull/6577/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6577?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6577?src=pr&el=footer). Last update [12d7624...1e68a94](https://codecov.io/gh/huggingface/transformers/pull/6577?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


hi @patrickvonplaten , could you take a look ? Thanks !

Yes, right. But this intended for the `EncoderDecoder` model class which can use pre-trained encoder for seq2seq tasks.
See PR #6411 and #6538 . Also Patrick shared this awesome paper which explores this idea and has shown some good results on seq2seq tasks.  https://arxiv.org/pdf/1907.12461.pdf

Thanks for the explanation! Yes it makes perfect sense to use these models for seq2seq but I still think maybe we should add a note somewhere in the document to explain this?

Definitely! Will update the docs to explicitly state that this is intended for seq2seq and might not perform well on just causal modelling.  Will also link the paper in `EncoderDecoder` doc page, so people will know what to choose.

Btw, @patil-suraj did you start doing some Roberta2Roberta experiments? Wanted to start running some experiments next week - wanted to check if you already had some interesting results

@patrickvonplaten  Just started one experiment for qg and going to run cnn/dm after that, will let you know the results. 
Since Roberta has `bos`, what's the `decoder_start_token_id` for `Roberta2Roberta`, is it `bos` or `pad` token ?

In my experiments with bert2bert, I used the same token for encoder bos and decoder bos, but it's up to you! `bos` makes more sense to me in the case of Roberta2Roberta.

Okay, Thanks Patrick!