Marian Encoder's last hidden states from MarianMT and TFMarianMTModel don't match
Author: harshitadd
Pull request: no

## Environment info
- `transformers` version: 4.10.2
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Additional Environment Info: Tensorflow Version - 2.6.0

### Who can help
@patil-suraj, @Rocketknight1
## Information

Model I am using (Bert, XLNet ...): MarianMT and TFMarianMT Models  ( [pretrained opus-mt en-hi model](https://huggingface.co/Helsinki-NLP/opus-mt-en-hi/tree/main)) 

The problem arises when using:
* [ ] my own modified scripts: [Reproducible Error Script](https://colab.research.google.com/drive/1mIEip64duk6upCerF8J1HOoVyN4Dz7sF?usp=sharing)

The tasks I am working on is:
* [ ] my own task or dataset: MT, Not Applicable

## To reproduce

Steps to reproduce the behavior:

1. Run the provided error script - It can be noticed that the encoder's last hidden state is not the same for MarianMT and TFMarianMT Models. 

## Expected behavior

In my best understanding, The hidden states of the last layer of the encoder should be consistent across both variants ( TFMarianMT and MarianMT ). The variance in the encoder's hidden states will carry forward to introduce difference in the decoded output's generation, potentially leading to different outputs from the same pretrained model.  


Thanks in Advance! 

Comments:
Hi! I ran your script, but I noticed that when I re-ran the code, even with the same framework, I got different outputs. When I set `training=False` for the TF model call and called `model.eval()` for PyTorch this stopped, so I'm guessing it was caused by Dropout layers or something similar.

Even after I did this, though, the output of the two models remained different, so this may indeed be a bug. We're investigating!

> Hi! I ran your script, but I noticed that when I re-ran the code, even with the same framework, I got different outputs. When I set `training=False` for the TF model call and called `model.eval()` for PyTorch this stopped, so I'm guessing it was caused by Dropout layers or something similar.
> 
> Even after I did this, though, the output of the two models remained different, so this may indeed be a bug. We're investigating!

Thanks for the quick response. I debugged a little myself by looking into generation_tf_utils.py and figured that I got the pytorch equivalent outputs by calling:


encoder = model.get_encoder()
encoded_sequence = encoder(input_ids)

where model is an isntance of the TFMarianMTModel. The key inconsistency then appears to be the difference in this output and the encoder.last_hidden_state. It might also be useful to point this out in the documentation since, prima facie, the attribute does not seem to be returning an expected value. 


Edit: Once I replaced the encoded_sequence ( in the file ) with the encoder_sequence returned by the new method I was also able to replicate the performance i.e get a nearly identical output from that of the one generated by the pytorch model. 

We believe this may be linked to another issue: https://github.com/huggingface/transformers/issues/12647

We're now investigating both - hopefully we can resolve both soon.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.