A bug in modeling_tf_marian.py and modeling_tf_pegasus.py SinusoidalPositionalEmbedding _init_weight
Author: JunnYu
Pull request: no


## Information

should  create a new np.array and then store np.sin&np.cos result in it
```python
table = np.zeros_like(position_enc)
# index 0 is all zero
table[:, 0 : dim // 2] = np.sin(position_enc[:, 0::2])
table[:, dim // 2 :] = np.cos(position_enc[:, 1::2])
# convert to tensor
table = tf.convert_to_tensor(table)
tf.stop_gradient(table)
```
https://github.com/huggingface/transformers/blob/bd3b599c12cfcf5ef517c5ffe526afbdbaa92539/src/transformers/models/marian/modeling_tf_marian.py#L147-L157

https://github.com/huggingface/transformers/blob/8d43c71a1ca3ad322cc45008eb66a5611f1e017e/src/transformers/models/pegasus/modeling_tf_pegasus.py#L148-L158




Comments:
Hey @JunnYu,

Could you give a bit more context on what the issue is exactly? And how your solution solves the issue?

@patrickvonplaten 
After this line `position_enc[:, 0 : dim // 2] = np.sin(position_enc[:, 0::2]) ` and
` position_enc[:, 0 : dim // 2] ` value will be overridden in place.
If we compute  `np.cos(position_enc[:, 1::2])`  and the result is inconsistent with the expected result.
So we should  init a np.array `table = np.zeros_like(position_enc)`  to store the sinusoidalposition embeddings.
![1622509362(1)](https://user-images.githubusercontent.com/50394665/120252804-25454600-c2b8-11eb-8dfe-299ac5081fb4.jpg)
![1622509402(1)](https://user-images.githubusercontent.com/50394665/120252834-3b530680-c2b8-11eb-8858-10ca92c2fbdb.jpg)



True, good catch! Do you mind opening a PR to fix it? We should then also run the slow tests to be sure the model performance is not affected

@patrickvonplaten  I have opened a PR  https://github.com/huggingface/transformers/pull/11897 to fix this. 
I think the **pretrained** tf model's performance will **not be affected**.  But when we init a **new** tf model, the model's performance will **be affected**!
Because the pretrained `tf_model.h5` contains the correct `embedding weight` (this embedding weight is converted by `pytorch_model.bin`). When we load a pretrained tf model , the tf model will load this correct `embedding weight` .
 
```python
# old code
from transformers.models.marian import MarianModel, TFMarianModel
import torch
pt_model = MarianModel.from_pretrained(
    "Helsinki-NLP/opus-mt-en-de")
tf_model = TFMarianModel.from_pretrained(
    "Helsinki-NLP/opus-mt-en-de")
pt_emb_weight = pt_model.encoder.embed_positions.weight

tf_emb_weight = torch.from_numpy(
    tf_model.model.encoder.embed_positions.weight.numpy())

print(pt_emb_weight.equal(tf_emb_weight))
# True
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@patrickvonplaten I have opened a PR #11897. Can you take the time to look at this PR？Thanks.

Thanks for pinging me again & super sorry to have this on wait for so long!