SlowGuess/ACL_2020_OCR huggingface dataset:
The user can monitor the progress of each submission from the dashboard, and cancel the submission at any point previous to completion of the evaluation run. The results of successful evaluation runs are automatically posted on the leaderboard. A user is prevented from submitting a new system for a week starting from the date of the most recent submission, as it appears on the public leaderboard. The user dashboard provides means for anonymizing and de-anonymizing a successful submission (for example, for paper review purposes). An anonymized submission retains the name of the system provided by the user, but hides the user's affiliation as well as the optional link to a paper.
Systems are required to analyze the 50 documents associated with each question, and produce 5 candidate answers. Each answer consists of a document ID, start and end character offsets from the beginning of the detagged text of the Technote, and a score. The score is compared with a threshold provided by the system for the run. Systems must return scores lower than the threshold
<sup>6</sup>https://developer.ibm.com/answers/questions
<sup>7</sup> https://www.ibm.com/developerworks
Dataset	Split	Question length in tokens	Split	Answer length in tokens
min	mean	max	std	min	mean	max	std
SQuAD 2.0	training	1	9.9	40	3.4	training	1	3.2	43	3.4
devset	3	10.0	31	3.45	devset	1	3.1	29	3.1
HOTPOTQA	training	3	17.8	108	9.5	training	1	2.2	89	1.8
devset	6	15.7	46	5.5	devset	1	2.5	29	1.8
TECHQA	training	8	52.1	259	31.6	training	1	48.1	302	37.8
devset	10	53.1	194	30.4	devset	1	41.2	137	27.7
Table 2: Statistics of the question and answer lengths in white-space-separated tokens for SQuAD 2.0, HOTPOTQA and TECHQA.
Figure 2: Number of white space separated tokens in training questions (title plus body.) and answers (for answerable questions only). The bin at 200 also contains all questions longer than 200 tokens.
Figure 3: Number of white space separated tokens in devtest questions (title plus body) and answers (for answerable questions only).
to indicate that no answer exists in the Technote; however, they also must indicate the best span extracted from the document: this is used to compute the two ancillary metrics described below.
The evaluation score computed for the leaderboard is a zero-one value for a question/document pair with score below the threshold, and character-overlap F1 for a question/document pair with score greater than or equal to the threshold.
The main metric, called F1 on the leaderboard, is the macro average of the evaluation scores computed on the first of the five answers provided by the system in response to each question.
The leaderboard displays three ancillary metrics. HA_F1@1 is the macro average of the evaluation scores computed on the first of the five answers and averaged over the answerable evaluation questions. This metric should be compared to the inter-annotator agreement of 76.3 reported in Section 3. HA_F1@5 consists of computing the evaluation score for each of the 5 answers, selecting the maximum, and computing the macro average over all answerable questions. BEST_F1 is the value of the F1 metric corresponding to the optimal choice of the threshold. The time required for the run will also be made available.
6 Baseline Results
Table 3 show the results of three baseline systems on the development set. These are a model trained on SQuAD 2.0, a model trained on NQ, and the TAP system submitted to HOTPOTQA $^8$ .
Both SQuAD and NQ models consists of a BERTLarge (whole word masking) language model (Devlin et al., 2019) with additional layers. For SQuAD 2.0 these are two fully connected FF layers followed by softmax for answer begin- and end-boundary extraction, like in (Devlin et al., 2019). The NQ model further adds a layer for target type prediction as in (Alberti et al., 2019), tuned as described in (Pan et al., 2019). The table contains entries for both models out-of-the box and after fine-tuning on the TECHQA dataset.
The TAP system consists of a document ranker module followed by an answer span selector, both based on pretrained BERT small. If the largest score produced by the ranker exceeds a threshold, the question is declared answerable and the answer span selection is invoked on the documents.
Table 3 shows that, without domain adaptation, the SQuAD and NQ models fail to produce interesting answers, and their best performance is roughly that of a dumb system that declares all questions unanswerable. Fine-tuning yields a notable improvement for both models. The TAP model has slightly lower performance but yields the highest HA_F1@5.
We have introduced TECHQA, a question-answering dataset for the IT technical support domain. The overall size of the released data (600 training questions) is in line with real-world scenarios, where the high cost of domain expert time limits the amount of quality data that can reasonably be collected. Thus, the dataset is meant to stimulate research in domain adaptation, in addition to developing algorithms for longer questions and answers than the current leaderboards.
We have created a leaderboard to evaluate systems against a blind dataset of 490 questions with a ratio of answerable to unanswerable questions similar to that of the development set. The leaderboard ranks submissions according to a metric consisting of the character overlap F1 measure for answerable questions and the zero-one metric for non-answerable questions. The leaderboard also reports the F1 at the top result and at the top 5 results averaged over the answerable questions.
TECHQA is a challenging dataset for models developed for existing open-domain MRC systems. Their out-of-the-box performance is very low, especially considering that a system that declares every question as unanswerable achieves  $\mathrm{F1} = 48.4\%$  on the development set. The obvious approach of fine-tuning these models using the TECHQA training set yields systems that barely beat the baseline.
The initial version of the dataset was created by selecting questions and answers that are relevant to the IT technical support domain but at the same time do not diverge excessively from the spirit of other existing MRC datasets. We consider TECHQA to be a stepping stone on which to build future data collections and leaderboards.
We plan on releasing questions with answers in a broader and more diverse collection that will include documents with a less formulaic structure than the Technotes. We will also relax the length limitations to include questions rich in details, and
<sup>8</sup>https://hotpotqa.github.io/
Systems	F1	HA_F1@1	HA_F1@5	BEST_F1
SQuAD 2.0 - FT	1.67	3.25	4.51	48.39
SQuAD 2.0 + FT	54.05*	22.01	35.50	54.05
NQ - FT	2.74	5.32	9.07	48.39
NQ + FT	55.31*	34.69	50.52	55.31
TAP_v0.1	51.36	16.39	57.49	52.67
Table 3: Baseline systems performance the dev set. The first 4 systems were pre-trained on the dataset indicated in the first column. In the same column,  $-\mathrm{FT}$  indicates no fine-tuning after pre-training, while  $+ \mathrm{FT}$  indicates further fine-tuning using the TECHQA corpus. Entries marked with  $*$  use a threshold tuned on the development set using the F1 metric; hence, F1 equals BEST_F1.
answers that include complex procedures; in the same spirit, we will allow answers consisting of multiple spans from a single document.
Many answers cannot be obtained by extracting portions of a document based on language alone: in many cases, domain knowledge is needed and often a question cannot be answered from the data collection without reasoning steps. We envision a roadmap where future releases of TECHQA will require synergy between multiple AI disciplines, from deep-learning based MRC to reasoning, knowledge base acquisition, and causality detection.
Our gratitude goes to our annotators: Abraham Mathews (IBM), Kat Harkavy, Irina Paegelow, Daniele Rosso, Chie Ugumori and Eva Maria Wolfe (ManpowerGroup Associates), for their dedication to TECHQA and their relentless effort.
InsuranceQA, a question answering corpus in insurance domain. https://github.com/shuzi/insuranceQA. Last commit: January 16, 2017.
Chris Alberti, Kenton Lee, and Michael Collins. 2019. A BERT baseline for the natural questions. arXiv preprint arXiv:1901.08634, pages 1-4.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language understanding. In NAACL-HLT.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328.
Tomáš Kočisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The Narrative QA reading comprehension challenge. TACL, 6:317-328.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a benchmark for question answering research. TACL.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Lin Pan, Rishav Chakravarti, Anthony Ferritto, Michael Glass, Alfio Gliozzo, Salim Roukos, Radu Florian, and Avirup Sil. 2019. Frustratingly easy natural question answering.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. EMNLP.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq
large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16(1):138.
Georg Wiese, Dirk Weissenborn, and Mariana Neves. 2017. Neural domain adaptation for biomedical question answering. CoNLL.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.