Repository: huggingface/transformers

Issue:
`resize_token_embeddings` not taken into account in `save_pretrained` for `EncoderDecoderModel`
## Environment info
- `transformers` version: 4.5.0
- Platform: Darwin-17.7.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten, @patil-suraj


## Information
I am extending the embeddings of the decoder of an `EncoderDecoderModel` model. When I save it, the config does not reflect the new size. However, it works fine when I try doing the same for non `EncoderDecoderModel` models.

## To reproduce
```
In [1]: model = t.EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')

In [2]: model.decoder.bert.embeddings.word_embeddings
Out[2]: Embedding(30522, 768, padding_idx=0)

In [3]: model.decoder.resize_token_embeddings(30522+100)
Out[3]: Embedding(30622, 768)

In [4]: model.save_pretrained('test-bert')
```



## Expected behavior
The updated embedding size should be saved in `config.json`



Discussion:
This is caused by the EncoderDecoderConfig which initializes independent objects ([link](https://github.com/huggingface/transformers/blob/d9c62047a8d75e18d2849d345ab3394875a712ef/src/transformers/models/encoder_decoder/configuration_encoder_decoder.py#L84)) instead of utilizing the already existing ones.

You can fix that for the moment by calling:
```
model.config.decoder = model.decoder.config
model.config.encoder = model.encoder.config
```
PR will follow.

Resolving patch:
diff --git a/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py b/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py
--- a/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py
+++ b/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py
@@ -175,6 +175,21 @@ def __init__(
 
         self.encoder = encoder
         self.decoder = decoder
+
+        if self.encoder.config.to_dict() != self.config.encoder.to_dict():
+            logger.warning(
+                f"Config of the encoder: {self.encoder.__class__} is overwritten by shared encoder config: {self.config.encoder}"
+            )
+        if self.decoder.config.to_dict() != self.config.decoder.to_dict():
+            logger.warning(
+                f"Config of the decoder: {self.decoder.__class__} is overwritten by shared decoder config: {self.config.decoder}"
+            )
+
+        # make sure that the individual model's config refers to the shared config
+        # so that the updates to the config will be synced
+        self.encoder.config = self.config.encoder
+        self.decoder.config = self.config.decoder
+
         assert (
             self.encoder.get_output_embeddings() is None
         ), "The encoder {} should not have a LM Head. Please use a model without LM Head"
@@ -458,6 +473,12 @@ def prepare_inputs_for_generation(
         }
         return input_dict
 
+    def resize_token_embeddings(self, *args, **kwargs):
+        raise NotImplementedError(
+            "Resizing the embedding layers via the EncoderDecoderModel directly is not supported."
+            "Please use the respective methods of the wrapped objects (model.encoder.resize_token_embeddings(...) or model.decoder.resize_token_embeddings(...))"
+        )
+
     def _reorder_cache(self, past, beam_idx):
         # apply decoder cache reordering here
         return self.decoder._reorder_cache(past, beam_idx)


Test patch:
diff --git a/tests/test_modeling_encoder_decoder.py b/tests/test_modeling_encoder_decoder.py
--- a/tests/test_modeling_encoder_decoder.py
+++ b/tests/test_modeling_encoder_decoder.py
@@ -34,6 +34,7 @@
     import torch
 
     from transformers import (
+        AutoConfig,
         AutoTokenizer,
         BartForCausalLM,
         BertGenerationDecoder,
@@ -884,3 +885,38 @@ def get_pretrained_model(self):
 
     def test_encoder_decoder_model_shared_weights(self):
         pass
+
+
+@require_torch
+class EncoderDecoderModelTest(unittest.TestCase):
+    def get_from_encoderdecoder_pretrained_model(self):
+        return EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "bert-base-uncased")
+
+    def get_decoder_config(self):
+        config = AutoConfig.from_pretrained("bert-base-uncased")
+        config.is_decoder = True
+        config.add_cross_attention = True
+        return config
+
+    def get_encoderdecoder_model(self):
+        return EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert-cnn_dailymail-fp16")
+
+    def get_encoder_decoder_models(self):
+        encoder_model = BertModel.from_pretrained("bert-base-uncased")
+        decoder_model = BertLMHeadModel.from_pretrained("bert-base-uncased", config=self.get_decoder_config())
+        return {"encoder": encoder_model, "decoder": decoder_model}
+
+    def _check_configuration_tie(self, model):
+        assert id(model.decoder.config) == id(model.config.decoder)
+        assert id(model.encoder.config) == id(model.config.encoder)
+
+    @slow
+    def test_configuration_tie(self):
+        model = self.get_from_encoderdecoder_pretrained_model()
+        self._check_configuration_tie(model)
+
+        model = EncoderDecoderModel(**self.get_encoder_decoder_models())
+        self._check_configuration_tie(model)
+
+        model = self.get_encoderdecoder_model()
+        self._check_configuration_tie(model)