[Benchmark] Converting a QA distilbert model to onnx - the f1 score plummet
Author: pievalentin
Pull request: no

# 🖥  Benchmarking onnx QA `transformers`

## Issue

Poor benchmark result (squadv2) of converted onnx QA model using run_squad_onnx benchmark.

## Context

As part of my day job, my goal is to convert our QA model to onnx to make them available in production in Java.
My teammate generated a distilbert model trained on SQUADv2. He reproduced SOTA result on the squad benchmark.

I want to evaluate the quality of my converted model.

## Set-up

### Hardware and OS

Training of pytorch model:

- 	OS: Debian
- 	pytorch version: 1.7.0 on GPU

Conversion and benchmarking:

- 	OS: MacOS
- 	onnxruntime: 1.6.0 (CPUExecutioner)
- 	onnx: 1.8.1

### To reproduce

To convert my model I used the convert fucntion as follow:
`convert('pt', <path-to-distilbert-model>, '/tmp/onnx', 12, 'distilbert-base-uncased-distilled-squad')`

This generate an onnx model succesfully. To benchmark the model I modified the legacy run_squad with onnx inference. You can find the source here: https://gist.github.com/pievalentin/c0007be4c2483bb113326fed0b1bddb2

I modified the inputs to make it ort compatible and I update the start_logit and end_logit to match the output generated by the inference session.
I run the script with the following config:

python  run_squad_ort.py --framework=ort --ort_model_path=<path-to-onnx-model> --model_name_or_path=<path-to-original-pt-model> --model_type=question-answering --output_dir=/tmp/qa --max_seq_length=384 --doc_stride=128 --n_best_size=20 --max_answer_length=30 --data_dir=/eai/datasets/squad2 --tokenizer_name=distilbert-base-uncased-distilled-squad

here is the config.json of the pytorch model:

```
{
    "_name_or_path": "distilbert-base-uncased-distilled-squad",
    "activation": "gelu",
    "architectures": [
        "DistilBertForQuestionAnswering"
    ],
    "attention_dropout": 0.1,
    "dim": 768,
    "dropout": 0.1,
    "hidden_dim": 3072,
    "initializer_range": 0.02,
    "max_position_embeddings": 512,
    "model_type": "distilbert",
    "n_heads": 12,
    "n_layers": 6,
    "pad_token_id": 0,
    "qa_dropout": 0.1,
    "seq_classif_dropout": 0.2,
    "sinusoidal_pos_embds": false,
    "tie_weights_": true,
    "vocab_size": 30522,
    "return_dict": false
}
```

## Results 
With the previous config, I get those poor result:

```
{
    "exact": 8.641455403015245,
    "f1": 11.271879679607238,
    "total": 11873,
    "HasAns_exact": 0.1349527665317139,
    "HasAns_f1": 5.403344709172896,
    "HasAns_total": 5928,
    "NoAns_exact": 17.12363330529857,
    "NoAns_f1": 17.12363330529857,
    "NoAns_total": 5945,
    "best_exact": 50.07159100480081,
    "best_exact_thresh": 0,
    "best_f1": 50.07310704960835,
    "best_f1_thresh": 0
}
```

As if the weight of the model were discarded. So I am wondering what I am missing. Is it a poor configuration or benchmarking is not done the right way. I saw this script which is pretty similar to mine: https://github.com/onnx/models/blob/f6779d235046f28c0d3bf4ec25e4456c4689d2ce/text/machine_comprehension/bert-squad/dependencies/run_onnx_squad.py

So I would guess I must be missing something either in the conversion or the benchmarking. 





Comments:
After further investigating, I found this useful repo: https://github.com/airKlizz/benchmark-for-transformers
Working on adding squad to this