text: 
  
  TITLE
  [BUG] Wrong Scores for many SQUAD models
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  @julien-c
  @VictorSanh
  
  ## Information
  
  All models trained with run_squad.py have abstaining threshold of 0.0 and possibly wrong evaluation scores.
  Those model have their wrong score in their model cards many people rely on. 
  
  For example: ahotrod/electra_large_discriminator_squad2_512
  
  The results with current evaluation script:
  
  ```
  "exact": 87.09677419354838,
    "f1": 89.98343832723452,
    "total": 11873,
    "HasAns_exact": 84.66599190283401,
    "HasAns_f1": 90.44759839056285,
    "HasAns_total": 5928,
    "NoAns_exact": 89.52060555088309,
    "NoAns_f1": 89.52060555088309,
    "NoAns_total": 5945,
    "best_exact": 87.09677419354838,
    "best_exact_thresh": 0.0,
    "best_f1": 89.98343832723432,
    "best_f1_thresh": 0.0
  ```
  
  The problem and its fix can be found in: [[Bug Fix] Fix run_squad.py evaluation code doesn't use probabilities](https://github.com/huggingface/transformers/pull/7319) #7319
  The problem arises when using:
  * [ ] run_squad.py
  
  The tasks I am working on is:
  * [ ] SQuAD
  
  ## To reproduce
  
  Steps to reproduce the behaviour:
  
  1. run `run_squad.py`
  
  ## Expected behaviour:
  Resulting 'best_f1_thresh' wont be 0.0 .
  