Repository: huggingface/transformers

Issue:
A bug in modeling_tf_marian.py and modeling_tf_pegasus.py SinusoidalPositionalEmbedding _init_weight

## Information

should  create a new np.array and then store np.sin&np.cos result in it
```python
table = np.zeros_like(position_enc)
# index 0 is all zero
table[:, 0 : dim // 2] = np.sin(position_enc[:, 0::2])
table[:, dim // 2 :] = np.cos(position_enc[:, 1::2])
# convert to tensor
table = tf.convert_to_tensor(table)
tf.stop_gradient(table)
```
https://github.com/huggingface/transformers/blob/bd3b599c12cfcf5ef517c5ffe526afbdbaa92539/src/transformers/models/marian/modeling_tf_marian.py#L147-L157

https://github.com/huggingface/transformers/blob/8d43c71a1ca3ad322cc45008eb66a5611f1e017e/src/transformers/models/pegasus/modeling_tf_pegasus.py#L148-L158



A bug in modeling_tf_marian.py and modeling_tf_pegasus.py SinusoidalPositionalEmbedding _init_weight

## Information

should  create a new np.array and then store np.sin&np.cos result in it
```python
table = np.zeros_like(position_enc)
# index 0 is all zero
table[:, 0 : dim // 2] = np.sin(position_enc[:, 0::2])
table[:, dim // 2 :] = np.cos(position_enc[:, 1::2])
# convert to tensor
table = tf.convert_to_tensor(table)
tf.stop_gradient(table)
```
https://github.com/huggingface/transformers/blob/bd3b599c12cfcf5ef517c5ffe526afbdbaa92539/src/transformers/models/marian/modeling_tf_marian.py#L147-L157

https://github.com/huggingface/transformers/blob/8d43c71a1ca3ad322cc45008eb66a5611f1e017e/src/transformers/models/pegasus/modeling_tf_pegasus.py#L148-L158





Discussion:

Resolving patch:
diff --git a/src/transformers/models/marian/modeling_tf_marian.py b/src/transformers/models/marian/modeling_tf_marian.py
--- a/src/transformers/models/marian/modeling_tf_marian.py
+++ b/src/transformers/models/marian/modeling_tf_marian.py
@@ -152,11 +152,12 @@ def _init_weight(n_pos: int, dim: int):
         position_enc = np.array(
             [[pos / np.power(10000, 2 * (j // 2) / dim) for j in range(dim)] for pos in range(n_pos)]
         )
+        table = np.zeros_like(position_enc)
         # index 0 is all zero
-        position_enc[:, 0 : dim // 2] = np.sin(position_enc[:, 0::2])
-        position_enc[:, dim // 2 :] = np.cos(position_enc[:, 1::2])
+        table[:, 0 : dim // 2] = np.sin(position_enc[:, 0::2])
+        table[:, dim // 2 :] = np.cos(position_enc[:, 1::2])
         # convert to tensor
-        table = tf.convert_to_tensor(position_enc)
+        table = tf.convert_to_tensor(table)
         tf.stop_gradient(table)
         return table
 
diff --git a/src/transformers/models/pegasus/modeling_tf_pegasus.py b/src/transformers/models/pegasus/modeling_tf_pegasus.py
--- a/src/transformers/models/pegasus/modeling_tf_pegasus.py
+++ b/src/transformers/models/pegasus/modeling_tf_pegasus.py
@@ -153,11 +153,12 @@ def _init_weight(n_pos: int, dim: int):
         position_enc = np.array(
             [[pos / np.power(10000, 2 * (j // 2) / dim) for j in range(dim)] for pos in range(n_pos)]
         )
+        table = np.zeros_like(position_enc)
         # index 0 is all zero
-        position_enc[:, 0 : dim // 2] = np.sin(position_enc[:, 0::2])
-        position_enc[:, dim // 2 :] = np.cos(position_enc[:, 1::2])
+        table[:, 0 : dim // 2] = np.sin(position_enc[:, 0::2])
+        table[:, dim // 2 :] = np.cos(position_enc[:, 1::2])
         # convert to tensor
-        table = tf.convert_to_tensor(position_enc)
+        table = tf.convert_to_tensor(table)
         tf.stop_gradient(table)
         return table
 


Test patch: