Repository: huggingface/transformers

Issue:
TFXLNet int32 to float promotion error
## 🐛 Bug

Using TFXLNet on GLUE datasets results in a TypeError when computing the input_mask because the attention_mask is represented as an int32 and is not automatically cast or promoted to a float.

Model I am using (Bert, XLNet....):

TFXLNet

Language I am using the model on (English, Chinese....):

English 

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Use the attached run_tf_glue_xlnet.py script

```
Traceback (most recent call last):
  File "run_tf_glue_xlnet.py", line 63, in <module>
    validation_data=valid_dataset, validation_steps=valid_steps)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 224, in fit
    distribution_strategy=strategy)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 547, in _process_training_inputs
    use_multiprocessing=use_multiprocessing)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 594, in _process_inputs
    steps=steps)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2419, in _standardize_user_data
    all_inputs, y_input, dict_inputs = self._build_model_with_inputs(x, y)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2622, in _build_model_with_inputs
    self._set_inputs(cast_inputs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2709, in _set_inputs
    outputs = self(inputs, **kwargs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 842, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/data/conda/envs/transformers/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in converted code:
    relative to /data/conda/envs/transformers/lib/python3.6/site-packages:

    transformers/modeling_tf_xlnet.py:907 call  *
        transformer_outputs = self.transformer(inputs, **kwargs)
    tensorflow_core/python/keras/engine/base_layer.py:842 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    transformers/modeling_tf_xlnet.py:542 call  *
        input_mask = 1.0 - attention_mask
    tensorflow_core/python/ops/math_ops.py:924 r_binary_op_wrapper
        x = ops.convert_to_tensor(x, dtype=y.dtype.base_dtype, name="x")
    tensorflow_core/python/framework/ops.py:1184 convert_to_tensor
        return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
    tensorflow_core/python/framework/ops.py:1242 convert_to_tensor_v2
        as_ref=False)
    tensorflow_core/python/framework/ops.py:1296 internal_convert_to_tensor
        ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    tensorflow_core/python/framework/tensor_conversion_registry.py:52 _default_conversion_function
        return constant_op.constant(value, dtype, name=name)
    tensorflow_core/python/framework/constant_op.py:227 constant
        allow_broadcast=True)
    tensorflow_core/python/framework/constant_op.py:265 _constant_impl
        allow_broadcast=allow_broadcast))
    tensorflow_core/python/framework/tensor_util.py:449 make_tensor_proto
        _AssertCompatible(values, dtype)
    tensorflow_core/python/framework/tensor_util.py:331 _AssertCompatible
        (dtype.name, repr(mismatch), type(mismatch).__name__))

    TypeError: Expected int32, got 1.0 of type 'float' instead.
```

## Expected behavior

The script should run the same as tf_run_glue.py

## Environment

* OS: CentOS-7
* Python version: 3.6.9
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 2.1.1 (master from git)
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

run_tf_glue_xlnet.py:
```python
import os
import tensorflow as tf
import tensorflow_datasets
from transformers import XLNetForSequenceClassification, TFXLNetForSequenceClassification, glue_convert_examples_to_features, XLNetTokenizer

# script parameters
BATCH_SIZE = 32
EVAL_BATCH_SIZE = BATCH_SIZE * 2
USE_XLA = False
USE_AMP = False

# tf.config.optimizer.set_jit(USE_XLA)
# tf.config.optimizer.set_experimental_options({"auto_mixed_precision": USE_AMP})

# Load tokenizer and model from pretrained model/vocabulary
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
model = TFXLNetForSequenceClassification.from_pretrained('xlnet-base-cased')

# Load dataset via TensorFlow Datasets
data, info = tensorflow_datasets.load('glue/mrpc', with_info=True)
train_examples = info.splits['train'].num_examples
valid_examples = info.splits['validation'].num_examples

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(
        data['train'],
        tokenizer,
        max_length=512,
        output_mode="classification",
        task='mrpc',
        pad_on_left=True, # pad on the left for xlnet
        pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],
        pad_token_segment_id=4
        )

valid_dataset = glue_convert_examples_to_features(
        data['validation'],
        tokenizer,
        max_length=512,
        output_mode="classification",
        task='mrpc',
        pad_on_left=True, # pad on the left for xlnet
        pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],
        pad_token_segment_id=4
        )
train_dataset = train_dataset.shuffle(128).batch(BATCH_SIZE).repeat(-1)
valid_dataset = valid_dataset.batch(EVAL_BATCH_SIZE)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)
if USE_AMP:
    # loss scaling is currently required when using mixed precision
    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, 'dynamic')
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=opt, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
train_steps = train_examples//BATCH_SIZE
valid_steps = valid_examples//EVAL_BATCH_SIZE

history = model.fit(train_dataset, epochs=2, steps_per_epoch=train_steps,
                    validation_data=valid_dataset, validation_steps=valid_steps)

# Save TF2 model
os.makedirs('./save/', exist_ok=True)
model.save_pretrained('./save/')

# Load the TensorFlow model in PyTorch for inspection
pytorch_model = XLNetForSequenceClassification.from_pretrained('./save/', from_tf=True)

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = 'This research was consistent with his findings.'
sentence_1 = 'His findings were compatible with this research.'
sentence_2 = 'His findings were not compatible with this research.'
inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors='pt')
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors='pt')

pred_1 = pytorch_model(**inputs_1)[0].argmax().item()
pred_2 = pytorch_model(**inputs_2)[0].argmax().item()
print('sentence_1 is', 'a paraphrase' if pred_1 else 'not a paraphrase', 'of sentence_0')
print('sentence_2 is', 'a paraphrase' if pred_2 else 'not a paraphrase', 'of sentence_0')
```

[run_tf_glue_xlnet.zip](https://github.com/huggingface/transformers/files/3798525/run_tf_glue_xlnet.zip)



Discussion:
Problem can be addressed by updating line 542 in transformers/modeling_tf_xlnet.py to :
```python
            input_mask = 1.0 - tf.cast(attention_mask, dtype=dtype_float)
```

Resolving patch:
diff --git a/transformers/modeling_tf_xlnet.py b/transformers/modeling_tf_xlnet.py
--- a/transformers/modeling_tf_xlnet.py
+++ b/transformers/modeling_tf_xlnet.py
@@ -539,7 +539,7 @@ def call(self, inputs, attention_mask=None, mems=None, perm_mask=None, target_ma
         assert input_mask is None or attention_mask is None, "You can only use one of input_mask (uses 1 for padding) " \
             "or attention_mask (uses 0 for padding, added for compatbility with BERT). Please choose one."
         if input_mask is None and attention_mask is not None:
-            input_mask = 1.0 - attention_mask
+            input_mask = 1.0 - tf.cast(attention_mask, dtype=dtype_float)
         if input_mask is not None and perm_mask is not None:
             data_mask = input_mask[None] + perm_mask
         elif input_mask is not None and perm_mask is None:


Test patch: