HANS evaluation
Author: ns-moosavi
Pull request: yes

Adding the evaluation on the HANS dataset in examples

Comments:
Thanks a lot for that Nafise!

I've started to update the readme.

Do you think you would have an example of a command to run the script together with associated results?

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2239?src=pr&el=h1) Report
> Merging [#2239](https://codecov.io/gh/huggingface/transformers/pull/2239?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/dc17f2a1110aed8d1729e77b0619601e3d96b84e?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2239/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2239?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2239      +/-   ##
==========================================
- Coverage   74.67%   74.66%   -0.02%     
==========================================
  Files          87       87              
  Lines       14800    14802       +2     
==========================================
  Hits        11052    11052              
- Misses       3748     3750       +2
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2239?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/metrics/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/2239/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL21ldHJpY3MvX19pbml0X18ucHk=) | `26.66% <0%> (-1.25%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2239?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2239?src=pr&el=footer). Last update [dc17f2a...258ed2e](https://codecov.io/gh/huggingface/transformers/pull/2239?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi Thomas,

This is an example of using test_hans.py:

```
export HANS_DIR=path-to-hans 
export MODEL_TYPE=type-of-the-model-e.g.-bert-roberta-xlnet-etc
export MODEL_PATH=path-to-the-model-directory-that-is-trained-on-NLI-e.g.-by-using-run_glue.py

python examples/test_hans.py \
        --task_name hans \
        --model_type $MODEL_TYPE \
        --do_eval \
        --do_lower_case \
        --data_dir $HANS_DIR \
        --model_name_or_path $MODEL_PATH \
        --max_seq_length 128 \
        -output_dir $MODEL_PATH \
```

This will create the hans_predictions.txt file in MODEL_PATH, which can then be evaluated using hans/evaluate_heur_output.py from the HANS dataset.

The results of the BERT-base model that is trained on MNLI using batch size 8 and the random seed 42 on the HANS dataset is as follows:


```
Heuristic entailed results:
lexical_overlap: 0.9702
subsequence: 0.9942
constituent: 0.9962

Heuristic non-entailed results:
lexical_overlap: 0.199
subsequence: 0.0396
constituent: 0.118
```


Great thanks a lot @ns-moosavi, merging this.
So happy to welcome HANS in the examples!