[EncoderDecoderModel] add a `add_cross_attention` boolean to config
Author: patrickvonplaten
Pull request: yes

The `EncoderDecoderModel` uses models from `AUTO_MODEL_FOR_CAUSAL_LM` as their decoder models. The problem is that these models can be used in two ways:

1) As a stand-alone decoder model (GPT2) like **without** cross-attention layers
2) As part of a `EncoderDecoderModel` **with** cross-attention layers.

Currently it is decided via the parameter `config.is_decoder` whether cross-attention layers should be added. The problem is that `config.is_decoder` is `True` for both 1) and 2), which is correct since both 1) and 2) should use a causal mask, but means that for 1) cross-attention layers are added without ever being used. 

This PR solves this problem by introducing a new config param called `add_cross_attention` which is only relevant for models in `AUTO_MODEL_FOR_CAUSAL_LM`.

I also played around with the idea of not having the flag in the config, but just passing it along the `init` function, such as:
```python
        super().__init__(config, add_cross_attention=False)
```
in 

and then calling setting this param to `True` for all encoder-decoder models. I decided to put the param in the config instead because:
a) The init signature does not have to change and 
b) EncoderDecoderModels make extensive use of `AutoModelForCausalLM.from_pretrained(...)` which would have meant that all models that are part of `MODEL_FOR_CAUSAL_LM_MAPPING` have to have this signature. 

Taking all this into account I think the first solution (putting `add_cross_attenion` into the config) is the better way to go here.

# IMPORTANT: This PR introduces a breaking change. All `EncoderDecoderModel` models have to be updated with `add_cross_attention=True`.
=> All "bert2bert" models were updated: https://huggingface.co/models?search=bert2bert

## TODO: 
After this, I think the framework is flexible enough to handle all other models and I can extend `EncoderDecoderModel` to GPT2, Roberta, Longformer and maybe Reformer as well. 
EncoderDecoder is not yet officially released, I think, so this slightly backwards compatibility breaking change is OK. I will updated all Bert2Bert models on the model hub with `add_cross_attention=True` and add a bigger message in this PR when merged.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6377?src=pr&el=h1) Report
> Merging [#6377](https://codecov.io/gh/huggingface/transformers/pull/6377?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1429b920d44d610eaa0a6f48de43853da52e9c03&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `90.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6377/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6377?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6377      +/-   ##
==========================================
- Coverage   78.38%   78.36%   -0.03%     
==========================================
  Files         148      148              
  Lines       27196    27202       +6     
==========================================
- Hits        21317    21316       -1     
- Misses       5879     5886       +7     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6377?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/6377/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `91.02% <66.66%> (-1.19%)` | :arrow_down: |
| [src/transformers/configuration\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/6377/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2VuY29kZXJfZGVjb2Rlci5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6377/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.57% <100.00%> (+0.02%)` | :arrow_up: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6377/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `88.45% <100.00%> (+0.05%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6377/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.21% <0.00%> (-1.26%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6377/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.18% <0.00%> (-0.26%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6377?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6377?src=pr&el=footer). Last update [1429b92...e2fcc0d](https://codecov.io/gh/huggingface/transformers/pull/6377?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


`All EncoderDecoderModel models have to be updated with add_cross_attention=True.`

How do I _exactly_ do this? I got hit by `AttributeError: 'GPT2Config' object has no attribute 'add_cross_attention'` after updating to newest release.

Hey @xxbidiao, 

You have to set `gpt2.config.add_cross_attention = True` and then save this config. Or you can directly add the parameter `add_cross_attention=True` to the gpt2 config json file