text: 
  
  TITLE
  Tensorflow pretrained FlauBERT mixed precision error
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  <!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
       Don't forget to fill out the missing fields in that output! -->
  
  - `transformers` version: 4.0.0
  - Python version: 3.7.6
  - Tensorflow version (GPU?):  2.3.0
  - Using GPU in script?: Yes  - GPU Tesla V100-SXM2-16GB, compute capability 7.0
  - Using distributed or parallel set-up in script?: No
  
  ### Who can help
  @jplu 
  
  ## Information
  
  Model I am using (Bert, XLNet ...):  "jplu/tf-flaubert-small-cased"
  
  The problem arises when using:
  * [ x] the official example scripts: (give details below)
  * [ ] my own modified scripts: (give details below)
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: 
  * [x ] my own task or dataset: text classification
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  1. Set the dtype policies to mixed precision "float16" with tensorflow
  2. Load pre-trained tensorflow flaubert model ("jplu/tf-flaubert-small-cased")
  
  <!-- If you have code snippets, error messages, stack traces please provide them here as well.
       Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
       Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
  **After executing the following code:**
  ```python
  from transformers import TFFlaubertModel
  import tensorflow as tf
  from tensorflow.keras.mixed_precision import experimental as mixed_precision
  
  policy = mixed_precision.Policy('mixed_float16')
  mixed_precision.set_policy(policy)
  
  model_name = "jplu/tf-flaubert-small-cased"
  
  model = TFFlaubertModel.from_pretrained(model_name)
  ```
  
  **I got the following error:** 
  ```
  InvalidArgumentError: cannot compute AddV2 as input #1(zero-based) was expected to be a half tensor but is a float tensor [Op:AddV2]
  ```
  
  ## Expected behavior
  There should not be any problem. When I run "bert-base-case" pretrained model it works perfectly (the bottom code does not return any error) 
  ```python
  from transformers import TFBertModel
  import tensorflow as tf
  from tensorflow.keras.mixed_precision import experimental as mixed_precision
  
  policy = mixed_precision.Policy('mixed_float16')
  mixed_precision.set_policy(policy)
  
  model_name = "bert-base-cased"
  
  model = TFBertModel.from_pretrained(model_name)
  ```
  Maybe there is an issue about hard-coded uses of float32 in FlauBERT ? and it is not fixed yet unlike other models ?
  