DISTILBERT: run_squad.py not working
Author: fdlci
Pull request: no

Hi, 

I am trying to use the transformers/examples/legacy/question-answering/run_squad.py script to train and evaluate DistilBERT on squad2.0. Unfortunately it throws the following error:

**TypeError: forward() got an unexpected keyword argument 'token_type_ids'**

I used the following tokenizer: distilbert-base-uncased

Is there a way for me to fix this issue? Thank you for your reply!

Comments:
Hi! This script is a legacy script that we will not be maintaining anymore. Have you tried with the `run_qa.py` script available [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch/question-answering)? This script should be simpler to understand and more complete.

Let us know if you run in any issues, thanks.

Thank You! I will try this script then!


Hi!

I followed your advice and ran DistilBERT with run_qa.py on squad_v2 with the following arguments:

python transformers/examples/pytorch/question-answering/run_qa.py   --model_name_or_path  distilbert-base-uncased  --dataset_name squad_v2   --do_train   --do_eval   --per_device_train_batch_size 16   --learning_rate 5e-5   --num_train_epochs 3   --max_seq_length 384   --doc_stride 128 --output_dir output_distilbert

Unfortunately, for the evaluation I get the following error:
ValueError: max() arg is an empty sequence

Did I forget to add one parameter? Thank you for your answer


Could you provide the full stack trace? Thank you!

@sgugger 

Yes of course! Thank you

Traceback (most recent call last):
  File "transformers/examples/pytorch/question-answering/run_qa.py", line 613, in <module>
    main()
  File "transformers/examples/pytorch/question-answering/run_qa.py", line 581, in main
    metrics = trainer.evaluate()
  File "/home/ines/Ibert/transformers/examples/pytorch/question-answering/trainer_qa.py", line 56, in evaluate
    metrics = self.compute_metrics(eval_preds)
  File "transformers/examples/pytorch/question-answering/run_qa.py", line 543, in compute_metrics
    return metric.compute(predictions=p.predictions, references=p.label_ids)
  File "/home/ines/Ibert/venv_ibert/lib/python3.7/site-packages/datasets/metric.py", line 402, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/ines/.cache/huggingface/modules/datasets_modules/metrics/squad/513bf9facd7f12b0871a3d74c6999c866ce28196c9cdb151dcf934848655d77e/squad.py", line 109, in _compute
    score = evaluate(dataset=dataset, predictions=pred_dict)
  File "/home/ines/.cache/huggingface/modules/datasets_modules/metrics/squad/513bf9facd7f12b0871a3d74c6999c866ce28196c9cdb151dcf934848655d77e/evaluate.py", line 67, in evaluate
    exact_match += metric_max_over_ground_truths(exact_match_score, prediction, ground_truths)
  File "/home/ines/.cache/huggingface/modules/datasets_modules/metrics/squad/513bf9facd7f12b0871a3d74c6999c866ce28196c9cdb151dcf934848655d77e/evaluate.py", line 52, in metric_max_over_ground_truths
    return max(scores_for_ground_truths)
ValueError: max() arg is an empty sequence

Ah, since you're using the squad V2 dataset I believe you must also tell the script that it should understand examples that don't have an answer. For this, you can add the `--version_2_with_negative` argument when running your script.

Does that help?

Yes it does, I thought it was taken into accound with the name of the dataset, but I was wrong. Thank you!


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.