Attention Mask for TFXLM Model doesn't work
Author: dakshvar22
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLM

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import *
tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-enfr-1024')
model = TFXLMModel.from_pretrained('xlm-mlm-enfr-1024')
text = "Good evening."
input_ids = tokenizer.encode(text, add_special_tokens=True)
last_hidden_states = model(np.array([input_ids]), attention_mask=np.ones_like(np.array(input_ids)))
```

Error output:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/transformers/modeling_tf_xlm.py", line 589, in call
    outputs = self.transformer(inputs, **kwargs)
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/transformers/modeling_tf_xlm.py", line 348, in call
    mask, attn_mask = get_masks(slen, lengths, self.causal, padding_mask=attention_mask)
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/transformers/modeling_tf_xlm.py", line 88, in get_masks
    tf.debugging.assert_equal(shape_list(mask), [bs, slen])
  File "/Users/daksh/miniconda3/envs/rasa-tf2/lib/python3.6/site-packages/transformers/modeling_tf_utils.py", line 546, in shape_list
    static = x.shape.as_list()
AttributeError: 'tuple' object has no attribute 'as_list'
```

Works fine if I attention mask is removed

## Expected behavior
`last_hidden_states` is a tuple of type `tf.Tensor`

## Environment info
     
- `transformers` version: 2.3.0
- Platform: OSX
- Python version: 3.6.5
- Tensorflow version (GPU?): 2.1.0(CPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Hi! There seems to be an error in the current implementation where it doesn't accept NumPy arrays, only TensorFlow arrays. I'm working on it in [the branch fix-tf-xlm](https://github.com/huggingface/transformers/tree/fix-tf-xlm). In the meantime, you can use a tf.Tensor instead and it should work fine.

Please be aware that your attention mask should be defined as `np.ones_like(np.array([input_ids]))` instead of your current `np.ones_like(np.array(input_ids))` or else it'll be a dimension short.

The following code is your code modified to run:

```py
from transformers import *
import numpy as np
import tensorflow as tf

tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-enfr-1024')
model = TFXLMModel.from_pretrained('xlm-mlm-enfr-1024')
text = "Good evening."
input_ids = tokenizer.encode(text, add_special_tokens=True)
last_hidden_states = model(np.array([input_ids]), attention_mask=tf.constant(np.ones_like(np.array([input_ids]))))

```

Hi @LysandreJik When can we expect your fix to be merged and released in an official release?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, is this bug now fixed? 
Thanks!