EncoderDecoderModel not working with DDP
Author: ayubSubhaniya
Pull request: no

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Using Distributed

### Who can help
 Trainer: @sgugger 
 EncoderDecoderModel: @patrickvonplaten 

## Information
I am using a combination of `distilroberta-base` as encoder and `distilgpt2` as a decoder. Used `EncoderDecoderModel` for model initialisation and `Trainer` class for fine-tuning model. For launching distributed processes, used `torch.distributed.launch`.

Below is the model config.
```
{
    "add_cross_attention": false,
    "architectures": null,
    "bad_words_ids": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "decoder": {
        "_num_labels": 1,
        "activation_function": "gelu_new",
        "add_cross_attention": true,
        "architectures": [
            "GPT2LMHeadModel"
        ],
        "attn_pdrop": 0.1,
        "bad_words_ids": null,
        "bos_token_id": 50256,
        "chunk_size_feed_forward": 0,
        "decoder_start_token_id": null,
        "do_sample": false,
        "early_stopping": false,
        "embd_pdrop": 0.1,
        "eos_token_id": 50256,
        "finetuning_task": null,
        "gradient_checkpointing": false,
        "id2label": {
            "0": "LABEL_0"
        },
        "initializer_range": 0.02,
        "is_decoder": true,
        "is_encoder_decoder": false,
        "label2id": {
            "LABEL_0": 0
        },
        "layer_norm_epsilon": 1e-05,
        "length_penalty": 1.0,
        "max_length": 20,
        "min_length": 0,
        "model_type": "gpt2",
        "n_ctx": 1024,
        "n_embd": 768,
        "n_head": 12,
        "n_inner": null,
        "n_layer": 6,
        "n_positions": 1024,
        "no_repeat_ngram_size": 0,
        "num_beams": 1,
        "num_return_sequences": 1,
        "output_attentions": false,
        "output_hidden_states": false,
        "pad_token_id": null,
        "prefix": null,
        "pruned_heads": {},
        "repetition_penalty": 1.0,
        "resid_pdrop": 0.1,
        "return_dict": false,
        "sep_token_id": null,
        "summary_activation": null,
        "summary_first_dropout": 0.1,
        "summary_proj_to_labels": true,
        "summary_type": "cls_index",
        "summary_use_proj": true,
        "task_specific_params": {
            "text-generation": {
                "do_sample": true,
                "max_length": 50
            }
        },
        "temperature": 1.0,
        "tie_encoder_decoder": false,
        "tie_word_embeddings": true,
        "tokenizer_class": null,
        "top_k": 50,
        "top_p": 1.0,
        "torchscript": false,
        "use_bfloat16": false,
        "use_cache": true,
        "vocab_size": 50257,
        "xla_device": null
    },
    "decoder_start_token_id": null,
    "do_sample": false,
    "early_stopping": false,
    "encoder": {
        "add_cross_attention": false,
        "architectures": [
            "RobertaForMaskedLM"
        ],
        "attention_probs_dropout_prob": 0.1,
        "bad_words_ids": null,
        "bos_token_id": 0,
        "chunk_size_feed_forward": 0,
        "decoder_start_token_id": null,
        "do_sample": false,
        "early_stopping": false,
        "eos_token_id": 2,
        "finetuning_task": null,
        "gradient_checkpointing": false,
        "hidden_act": "gelu",
        "hidden_dropout_prob": 0.1,
        "hidden_size": 768,
        "id2label": {
            "0": "LABEL_0",
            "1": "LABEL_1"
        },
        "initializer_range": 0.02,
        "intermediate_size": 3072,
        "is_decoder": false,
        "is_encoder_decoder": false,
        "label2id": {
            "LABEL_0": 0,
            "LABEL_1": 1
        },
        "layer_norm_eps": 1e-05,
        "length_penalty": 1.0,
        "max_length": 20,
        "max_position_embeddings": 514,
        "min_length": 0,
        "model_type": "roberta",
        "no_repeat_ngram_size": 0,
        "num_attention_heads": 12,
        "num_beams": 1,
        "num_hidden_layers": 1,
        "num_return_sequences": 1,
        "output_attentions": false,
        "output_hidden_states": false,
        "pad_token_id": 1,
        "prefix": null,
        "pruned_heads": {},
        "repetition_penalty": 1.0,
        "return_dict": false,
        "sep_token_id": null,
        "task_specific_params": null,
        "temperature": 1.0,
        "tie_encoder_decoder": false,
        "tie_word_embeddings": true,
        "tokenizer_class": null,
        "top_k": 50,
        "top_p": 1.0,
        "torchscript": false,
        "type_vocab_size": 1,
        "use_bfloat16": false,
        "use_cache": true,
        "vocab_size": 50265,
        "xla_device": null
    },
    "eos_token_id": null,
    "finetuning_task": null,
    "id2label": {
        "0": "LABEL_0",
        "1": "LABEL_1"
    },
    "is_decoder": false,
    "is_encoder_decoder": true,
    "label2id": {
        "LABEL_0": 0,
        "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "encoder_decoder",
    "no_repeat_ngram_size": 0,
    "num_beams": 1,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "pad_token_id": null,
    "prefix": null,
    "pruned_heads": {},
    "repetition_penalty": 1.0,
    "return_dict": false,
    "sep_token_id": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torchscript": false,
    "use_bfloat16": false,
    "use_cache": true,
    "xla_device": null
}
```

Got this error when using `Trainer` to fine-tune this model.
```
Traceback (most recent call last):
  File "training/run_training.py", line 200, in <module>
    raise e
  File "training/run_training.py", line 197, in <module>
    main()
  File "training/run_training.py", line 163, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 768, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1116, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1142, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 526, in forward
    self.reducer.prepare_for_backward(list(_find_tensors(output)))
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
```

I checked this variable is `find_unused_parameters` set as true. Also above training works well in a single GPU setting.

Comments:
@sgugger @patrickvonplaten Can someone please help me?

I tried running @patrickvonplaten  `bert-bert` Encoder-Decoder summarization script using DDP but got same error. 
Below is the script. Have modified script a bit to skip some download for fast experimentation.

``` patrick_script.py
#!/usr/bin/env python3
import os
import nlp
import logging
from transformers import BertTokenizer, EncoderDecoderModel, Trainer, TrainingArguments

logging.basicConfig(level=logging.INFO)

local_rank = int(os.environ.get('LOCAL_RANK', -1))
print("local rank", local_rank)

model = EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "bert-base-uncased")
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

# CLS token will work as BOS token
tokenizer.bos_token = tokenizer.cls_token

# SEP token will work as EOS token
tokenizer.eos_token = tokenizer.sep_token

# load train and validation data
# train_dataset = nlp.load_dataset("cnn_dailymail", "3.0.0", split="train")
train_dataset = None
val_dataset = nlp.load_dataset("cnn_dailymail", "3.0.0", split="validation[:1%]")

# # load rouge for validation
# rouge = nlp.load_metric("rouge")


# set decoding params
model.config.decoder_start_token_id = tokenizer.bos_token_id
model.config.eos_token_id = tokenizer.eos_token_id
model.config.max_length = 142
model.config.min_length = 56
model.config.no_repeat_ngram_size = 3
model.early_stopping = True
model.length_penalty = 2.0
model.num_beams = 4


# map data correctly
def map_to_encoder_decoder_inputs(batch):
    # Tokenizer will automatically set [BOS] <text> [EOS]
    # cut off at BERT max length 512
    inputs = tokenizer(batch["article"], padding="max_length", truncation=True, max_length=512)
    # force summarization <= 128
    outputs = tokenizer(batch["highlights"], padding="max_length", truncation=True, max_length=128)

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask

    batch["decoder_input_ids"] = outputs.input_ids
    batch["labels"] = outputs.input_ids.copy()
    # mask loss for padding
    batch["labels"] = [
        [-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]
    ]
    batch["decoder_attention_mask"] = outputs.attention_mask

    assert all([len(x) == 512 for x in inputs.input_ids])
    assert all([len(x) == 128 for x in outputs.input_ids])

    return batch


# def compute_metrics(pred):
#     labels_ids = pred.label_ids
#     pred_ids = pred.predictions
#
#     # all unnecessary tokens are removed
#     pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
#     label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)
#
#     rouge_output = rouge.compute(predictions=pred_str, references=label_str, rouge_types=["rouge2"])["rouge2"].mid
#
#     return {
#         "rouge2_precision": round(rouge_output.precision, 4),
#         "rouge2_recall": round(rouge_output.recall, 4),
#         "rouge2_fmeasure": round(rouge_output.fmeasure, 4),
#     }


# set batch size here
batch_size = 1

# make train dataset ready
# train_dataset = train_dataset.map(
#     map_to_encoder_decoder_inputs, batched=True, batch_size=batch_size, remove_columns=["article", "highlights"],
# )
# train_dataset.set_format(
#     type="torch", columns=["input_ids", "attention_mask", "decoder_input_ids", "decoder_attention_mask", "labels"],
# )

# same for validation dataset
val_dataset = val_dataset.map(
    map_to_encoder_decoder_inputs, batched=True, batch_size=batch_size, remove_columns=["article", "highlights"],
)
val_dataset.set_format(
    type="torch", columns=["input_ids", "attention_mask", "decoder_input_ids", "decoder_attention_mask", "labels"],
)

# set training arguments - these params are not really tuned, feel free to change
training_args = TrainingArguments(
    output_dir="./",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    evaluate_during_training=True,
    do_train=True,
    do_eval=True,
    logging_steps=1000,
    save_steps=1000,
    eval_steps=1000,
    overwrite_output_dir=True,
    warmup_steps=2000,
    save_total_limit=10,
    local_rank=local_rank
)

# instantiate trainer
trainer = Trainer(
    model=model,
    args=training_args,
    # compute_metrics=compute_metrics,
    train_dataset=val_dataset,
    eval_dataset=train_dataset,
)

# start training
trainer.train()
````

ran it using
`python -m torch.distributed.launch --nproc_per_node ${GPUS_ALLOWED} --use_env patrick_script.py`


error stack trace
```
I1021 08:03:09.387925 140506964375360 arrow_dataset.py:905] Loading cached processed dataset at /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8/cache-34961a58ac716d5b0323e755fe4ab272.arrow
I1021 08:03:09.390125 139812404635456 filelock.py:274] Lock 139808428511864 acquired on /root/.cache/huggingface/datasets/720d2e20d8dc6d98f21195a39cc934bb41dd0a40b57ea3d323661a7c5d70522c.4fe1f8a4d3f3c15617ba15dd2d93f559a09627c62d0b04e22f89a5131b7bffb9.py.lock
I1021 08:03:09.390361 139812404635456 load.py:331] Found main folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /usr/local/lib/python3.6/dist-packages/nlp/datasets/cnn_dailymail
I1021 08:03:09.390519 139812404635456 load.py:344] Found specific version folder for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /usr/local/lib/python3.6/dist-packages/nlp/datasets/cnn_dailymail/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8
I1021 08:03:09.390662 139812404635456 load.py:357] Found script file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py to /usr/local/lib/python3.6/dist-packages/nlp/datasets/cnn_dailymail/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8/cnn_dailymail.py
I1021 08:03:09.390966 139812404635456 load.py:371] Found dataset infos file from https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/dataset_infos.json to /usr/local/lib/python3.6/dist-packages/nlp/datasets/cnn_dailymail/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8/dataset_infos.json
I1021 08:03:09.391108 139812404635456 load.py:382] Found metadata file for dataset https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/cnn_dailymail/cnn_dailymail.py at /usr/local/lib/python3.6/dist-packages/nlp/datasets/cnn_dailymail/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8/cnn_dailymail.json
I1021 08:03:09.391246 139812404635456 filelock.py:318] Lock 139808428511864 released on /root/.cache/huggingface/datasets/720d2e20d8dc6d98f21195a39cc934bb41dd0a40b57ea3d323661a7c5d70522c.4fe1f8a4d3f3c15617ba15dd2d93f559a09627c62d0b04e22f89a5131b7bffb9.py.lock
I1021 08:03:09.394302 139812404635456 info.py:236] Loading Dataset Infos from /usr/local/lib/python3.6/dist-packages/nlp/datasets/cnn_dailymail/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8
I1021 08:03:09.395207 140506964375360 arrow_dataset.py:563] Set __getitem__(key) output type to torch for ['input_ids', 'attention_mask', 'decoder_input_ids', 'decoder_attention_mask', 'labels'] columns  (when key is int or slice) and don't output other (un-formated) columns.
/usr/local/lib/python3.6/dist-packages/transformers/training_args.py:332: FutureWarning: The `evaluate_during_training` argument is deprecated in favor of `evaluation_strategy` (which has more options)
  FutureWarning,
I1021 08:03:09.395819 139812404635456 builder.py:169] Overwrite dataset info from restored data version.
I1021 08:03:09.396007 139812404635456 info.py:194] Loading Dataset info from /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8
I1021 08:03:09.396629 139812404635456 builder.py:388] Reusing dataset cnn_dailymail (/root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8)
I1021 08:03:09.396816 139812404635456 builder.py:590] Constructing Dataset for split validation[:1%], from /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8
I1021 08:03:09.400055 139812404635456 info_utils.py:39] All the checksums matched successfully for post processing resources
I1021 08:03:09.434219 139812404635456 arrow_dataset.py:905] Loading cached processed dataset at /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/d8c27f2d603e2864036d92b0ec379f081896f6c28605ffd2e194c42cd04d48d8/cache-34961a58ac716d5b0323e755fe4ab272.arrow
I1021 08:03:09.441347 139812404635456 arrow_dataset.py:563] Set __getitem__(key) output type to torch for ['input_ids', 'attention_mask', 'decoder_input_ids', 'decoder_attention_mask', 'labels'] columns  (when key is int or slice) and don't output other (un-formated) columns.
/usr/local/lib/python3.6/dist-packages/transformers/training_args.py:332: FutureWarning: The `evaluate_during_training` argument is deprecated in favor of `evaluation_strategy` (which has more options)
  FutureWarning,
/usr/local/lib/python3.6/dist-packages/nlp/utils/py_utils.py:191: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return function(data_struct)
Epoch:   0%|                                                                                                                                     | 0/3 [00:00<?, ?it/s/usr/local/lib/python3.6/dist-packages/nlp/utils/py_utils.py:191: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return function(data_struct)
                                                                                                                                                                      Traceback (most recent call last):                                                                                                       | 1/67 [00:00<00:40,  1.63it/s]
  File "patrick_script.py", line 129, in <module>
    trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 763, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1113, in training_step
Traceback (most recent call last):
  File "patrick_script.py", line 129, in <module>
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1137, in compute_loss
    trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 763, in train
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 526, in forward
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1113, in training_step
    self.reducer.prepare_for_backward(list(_find_tensors(output)))
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1137, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 526, in forward
    self.reducer.prepare_for_backward(list(_find_tensors(output)))
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Epoch:   0%|                                                                                                                                     | 0/3 [00:00<?, ?it/s]
Iteration:   1%|█▊                                                                                                                      | 1/67 [00:00<01:02,  1.06it/s]
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-u', 'patrick_script.py']' returned non-zero exit status 1.
```

@patrickvonplaten I will be happy to fix this issue if you can give me some lead as to what might be wrong in code.

The problem is, in the encoder forward method, BERT forward method returns a tuple `(encoder_hidden_state, pooler_output)` [link](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L847).
Now in encoder-decoder model for decoding, we use only `encoder_hidden_state` [link](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_encoder_decoder.py#L406). 
Thus will be no grad computed for the `pooler` layer and the code is breaking.

Solution
pass `encoder_add_pooling_layer=False` in model intialisation.

Model Architecture
```
(encoder): RobertaModel(
    (embeddings): RobertaEmbeddings(
      (word_embeddings): Embedding(50265, 768, padding_idx=1)
      (position_embeddings): Embedding(514, 768, padding_idx=1)
      (token_type_embeddings): Embedding(1, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): RobertaEncoder(
      (layer): ModuleList(
        (0): RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
    (pooler): RobertaPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
```

@patrickvonplaten It is interesting that BERT forward method returns both `encoder output` and `pooler output`. How should we handle this in `EncoderDecoderModel`. Should we change BERT implementation? or we may set `pooler` layer as unused parameter. Not sure how to do that.

Hey @ayubSubhaniya - yeah I see where the bug is coming from! 
Your reasoning is 100% correct, great catch! 

I think the best solution is to actually pass a `encoder_add_pooling_layer=False` variable at initialization so it looks like:

```python
from transformers import EncoderDecoderModel
model = EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "bert-base-uncased", encoder_add_pooling_layer=False)
print(model.encoder.pooler) # should give `None`
```

This is pretty hard to see though, so I think we should add an explicit warning/error message for this case. 

I think one thing we should do is add a warning statement to the `__init__` of  all `BertModel`, `RobertaModel`, ... (all models have this pooling layer structure) that checks a) if the model is in parallel mode - think this can be done via `isinstance(m, nn.DataParallel)` and b) if `add_pooling_layer=True` => If both a) and b) are True => then display a warning `That if model is used within `EncoderDecoderModel` and if errors arrises with unused parameters, use `encoder_add_pooling_layer=False`. 

I think this is the best we can to do help the user. It would be amazing if you want to open a PR for this -> otherwise I'll add it to my ToDo List :-) 

Will create a PR for this thanks 🙂 

```
from transformers import EncoderDecoderModel
model = EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "bert-base-uncased", encoder_add_pooling_layer=False)
print(model.encoder.pooler) # should give `None`
```
this does print None but after 
`model.save_pretrained("model")`
the error returns and `print(model.encoder.pooler)`
print 
```
LongformerPooler(
  (dense): Linear(in_features=768, out_features=768, bias=True)
  (activation): Tanh()
)
```

@alexyalunin yes I too faced same issue, workaround for this is to set pooler None explicitly .
```model.encoder.pooler = None```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.