text: 
  
  TITLE
  run_glue.py with XLNet model on CoLA dataset reaches 0 accuracy
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  - `transformers` version: 4.1.1
  - Platform: Linux
  - Python version: 3.6.10
  - PyTorch version (GPU?): 1.7.0 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: Yes
  - Using distributed or parallel set-up in script?: No
       
  ### Who can help
  @TevenLeScao 
  
  ## Information
  Model I am using: XLNet
  
  The problem arises when using:
  *  The official example scripts of `run_glue.py`
  
  The tasks I am working on is:
  * an official GLUE: CoLA
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  I am using the "run_glue" cmd as described here: https://github.com/huggingface/transformers/tree/master/examples/text-classification
  `python run_glue.py --task_name cola  --model_name_or_path xlnet-base-cased --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --do_predict --overwrite_output_dir`
  
  That's the results I get: 
  ```
  [p]$ cat res_cola_xlnet.txt 
  eval_loss = 0.612945020198822
  eval_matthews_correlation = 0.0
  epoch = 3.0
  ```
  
  ## Expected behavior
  
  Results > 0