Getting the same results when evaluating Model2Model with different encoder inputs.
Author: dimi1357
Pull request: no
Labels: name: Core: Encoder-Decoder

name: seq2seq

After fine-tuning the Model2Model with 'bert-base-uncased', I am getting the same losses values, no matter what is the encoder input. On the PreTrainedEncoderDecoder documentation, it's said that "During prediction, we perform one forward pass through the encoder,
          and then perform several forward passes with the encoder's hidden
          state through the decoder to decode a full sequence."
I couldn't find the place on the source code, that does the connection between the encoder and the decoder. If someone could help me to show me where this is happening, it will be a great help, 
thanks!

Comments:
It's in the forward pass of the EncoderDecoder:

https://github.com/huggingface/transformers/blob/d490b5d5003654f104af3abd0556e598335b5650/src/transformers/modeling_encoder_decoder.py#L205-L237

> It's in the forward pass of the EncoderDecoder:
> 
> https://github.com/huggingface/transformers/blob/d490b5d5003654f104af3abd0556e598335b5650/src/transformers/modeling_encoder_decoder.py#L205-L237

Yes but if I do not pass an "encoder_ hidden_states" then the encoder input does not affect the decoder output?

If you pass encoder_hidden_states, the encoder is skipped (not called at all). If you do not explicitly pass encoder_hidden_states, the inputs will go through the encoder and the hidden states will be used as encoder_hidden_states.

https://github.com/huggingface/transformers/blob/d490b5d5003654f104af3abd0556e598335b5650/src/transformers/modeling_encoder_decoder.py#L228-L232

Oh I guess I have an old version, for me the line 

`encoder_hidden_states = encoder_outputs[0]`
does not exists, I'll update and try again.

Thanks

Let me know if you run into other issues.

> Let me know if you run into other issues.

Still, I don't see any difference when I am changing the encoder input, for example:
`>>>model(torch.tensor([[10,20,300,4,500,600]]), torch.tensor([[400,500]]), decoder_lm_labels=torch.tensor([[400,500]]))[0]`
`tensor(17.1395, grad_fn=<NllLossBackward>)`

`>>>model(torch.tensor([[100,200,300,400]]), torch.tensor([[400,500]]), decoder_lm_labels=torch.tensor([[400,500]]))[0]`
`tensor(17.1395, grad_fn=<NllLossBackward>)`

I'm seeing the exact same issue. 
you can reproduce it here: https://colab.research.google.com/drive/1DH07pETO_F0eoxE7HaErooEWwptWf_SQ

What's interesting is that if I train the model, it will output something different, but it will output that same thing regardless of what the input to the trained model is. 

Also the same thing happens if I use `PreTrainedEncoderDecoder`

This is what I have discovered (I think):
when this calculation is evaluated in Model2Model forward:
`decoder_outputs = self.decoder(decoder_input_ids, **kwargs_decoder)`
we are getting to:
`outputs = self.bert(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_attention_mask,
        )`
and then:
`encoder_outputs = self.encoder(
            embedding_output,
            attention_mask=extended_attention_mask,
            head_mask=head_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_extended_attention_mask,
        )`
to:
`layer_outputs = layer_module(
                hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask
            )`
to:
` if self.is_decoder and encoder_hidden_states is not None:
            cross_attention_outputs = self.crossattention(
                attention_output, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask
            )
            attention_output = cross_attention_outputs[0]
            outputs = outputs + cross_attention_outputs[1:]  # add cross attentions if we output attention weights
`

and in this condition `if self.is_decoder and encoder_hidden_states is not None`, is decoder is always 
`False` so we never go into this clause, and never using the `encoder_hidden_states`, so we get always the same results, that does not depends on the encoder input or output.

@dimi1357 It looks like the issue is that the model is getting `is_decoder` set to True **after** it has been initialized to False, but at that point `BertLayer` has `is_decoder` set to False and so it stays like that. 
This seems to be a workaround: 
```
decoder_config = config = AutoConfig.from_pretrained('bert-base-uncased', is_decoder=True)
model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased', 'bert-base-uncased', decoder_config=decoder_config)
```

> @dimi1357 It looks like the issue is that the model is getting `is_decoder` set to True **after** it has been initialized to False, but at that point `BertLayer` has `is_decoder` set to False and so it stays like that.
> This seems to be a workaround:
> 
> ```
> decoder_config = config = AutoConfig.from_pretrained('bert-base-uncased', is_decoder=True)
> model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased', 'bert-base-uncased', decoder_config=decoder_config)
> ```

Yes, that seems to fix the issue,
Thanks a lot!!