SlowGuess/NAACL_2022_OCR huggingface dataset:
The confidence of a predicted answer span  $a_{i}$  is defined as the maximum confidence of the tokens within  $a_{i}$ . Suppose there are  $k$  spans that been tagged as answers and the predicted number of span is  $n$ , if  $n < k$ , we rank the predicted spans by confidence score, and keep the top- $n$  answer spans as answers.
Method	MultiSpanQA	MultiSpanQA (expand)
Exact Match	Partial Match	Exact Match	Partial Match
P	R	F	P	R	F	P	R	F	P	R	F
Single (v1)	1.07	0.37	0.55	28.04	69.99	40.04	8.98	5.53	6.85	59.83	71.27	65.05
Single (v1) + t	1.32	0.53	0.76	27.89	73.48	40.44	8.81	5.82	7.01	57.63	73.03	64.42
Single (v2)	15.92	5.55	8.23	58.86	48.23	53.02	12.60	7.77	9.61	67.64	58.36	62.66
Single (v2) + t	16.20	12.98	14.41	60.31	76.78	67.56	13.36	12.05	12.66	63.01	73.09	67.73
Tagger	52.45	61.11	56.45	75.91	74.53	75.22	39.43	43.54	41.38	70.79	69.42	70.10
Multi (joint)	54.51	62.55	58.25	77.53	75.49	76.50	40.14	42.88	41.47	73.09	69.68	71.35
Multi (full)	58.12	60.50	59.28	79.56	73.23	76.26	42.74	41.81	42.26	74.05	68.06	70.47
Table 4: Model performance on MultiSpanQA test set. "Single" without "t" means the single-span baseline with single-span prediction. "Single" with "t" means we additionally tune a confidence score threshold to choose multiple spans from the  $n$ -best single-span predictions. "Tagger" means the sequence tagging baseline. "Multi (joint)" represents the proposed tagger model joint training with span number prediction and structure prediction, "Multi (full)" signifies "Multi (joint)" with the proposed span adjustment module.
Answer Type	Exact Match	Structure
P	R	F	Acc
DESCRIPTION	25.56	34.34	29.31	82.50
LOCATION	57.22	67.30	61.85	93.06
HUMAN	70.10	75.55	72.72	84.83
NUMERIC	41.02	44.13	42.52	72.41
OTHER ENTITY	64.89	65.08	64.99	77.55
Table 5: Results on MultiSpanQA (expanded) dev set over different question types.
For all baselines and our model, we use the HuggingFace implementation of  $BERT_{Base}$  (Wolf et al., 2019; Devlin et al., 2019) as our encoder with max_sequence_length = 512 and doc_stride = 128 to deal with long passages. For the proposed span encoder, we use a multi-head self-attention layer with 4 heads followed by a linear layer to encode the spans. The maximum span number is set to 30 for the input of the span encoder. For training, we use the BertAdam optimizer with default hyperparameters and learning rate of 3e-5. All models are trained with a batch size of 4 for 3 epochs. We use a two-layer feed-forward network with a ReLU activation function for all linear layers.
For answer structure prediction, we use accuracy to evaluate the model performance. For answer span prediction, we evaluate in terms of exact match and partial match performance.
Exact match An exact match occurs when a prediction fully matches one of the ground-truth answers, and the F1 score is computed by treating the predicted and ground-truth answer spans as a set of spans. We use micro-averaged precision, recall, and F1 score for evaluation based on the standard formulation of Precision  $= TP / (TP + FP)$ , Recall  $= TP / (TP + FN)$ , and F1  $= 2 *$  Precision * Recall / (Precision + Recall), where  $TP$  (True Positive) is the number of answer spans correctly predicted by the model,  $FP$  (False Positive) is the number of spans incorrectly predicted by the model, and  $FN$  (False Negative) is the number of answer spans not predicted by the model. In the case of an unanswerable question with the expanded dataset, we use a virtual span which indicates no answer.
Partial Match To measure the overlap between the predictions and ground truth answers, we propose the partial match precision, recall, and F1 by treating each predicted span or ground-truth answer span as a string. In detail, for each pair of prediction  $p_i$  and ground truth answer  $t_j$ , we define the partial retrieved score and partial relevant score as the length of the longest common substring (LCS) between  $p_i$  and  $t_j$ , divided by the length of  $p_i$  and  $t_j$ , respectively, as:
$$
s _ {i j} ^ {r e t} = \operatorname {l e n} \left(L C S \left(p _ {i}, t _ {j}\right)\right) / \operatorname {l e n} \left(p _ {i}\right) \tag {8}
$$
$$
s _ {i j} ^ {r e l} = \operatorname {l e n} \left(L C S \left(p _ {i}, t _ {j}\right)\right) / \operatorname {l e n} \left(t _ {j}\right) \tag {9}
$$
Suppose there are  $n$  predictions and  $m$  ground truth answers for a question. Since we do not know the correspondence between predictions and an-
#Span	Exact Match	Structure
P	R	F	Acc
1	34.95	45.09	39.38	-
2-3	54.13	64.08	58.69	81.92
4-7	62.50	63.70	63.09	91.89
>7	82.25	71.83	76.69	81.25
Table 6: Results on MultiSpanQA (expanded) dev set categorised by number of spans.
svers, we compute the partial retrieved score between a prediction and all answers and keep the highest one as the retrieved score of the prediction. Similarly, for each ground truth answer, the relevant score is the highest one between it and all predictions. The precision, recall, and F1 are finally defined as follows:
$$
\text {P r e c i s i o n} = \frac {\sum_ {i = 1} ^ {n} \max  _ {j \in [ 1 , m ]} \left(s _ {i j} ^ {r e t}\right)}{n} \tag {10}
$$
$$
\operatorname {R e c a l l} = \frac {\sum_ {j = 1} ^ {m} \max _ {i \in [ 1 , n ]} \left(s _ {i j} {} ^ {r e l}\right)}{m} \tag {11}
$$
$$
F 1 = \frac {2 * \text {P r e c i s i o n} * \text {R e c a l l}}{\text {P r e c i s i o n} + \text {R e c a l l}} \tag {12}
$$
We use micro-averaged scores for all metrics.
Table 4 shows the dev set results on MultiSpanQA, where the left part is results on multi-span questions only (the basic dataset), and the right part is the results on the expanded dataset (including single-span answers and unanswerable questions).
Single-span model From the table, we see that single-span (v1) gets very low exact match scores but higher partial match scores (compared to exact match), as it is trained to find a single long span that overlaps with all answer spans. By comparison, single-span (v2) improves the exact match scores on the basic dataset because it is trained on independent single-span answers. This is also the reason that: (1) single-span (v1) always gets a lower score in partial match precision and a higher score in partial match recall compared with single model (v2); and (2) after applying the tuned threshold, single-span (v2) gets a clear boost while single-span (v1) does not exhibit a substantial change in results. The overall performance of the single-span baselines is relatively low, simply because the models can only predict a single-span answer, which is incompatible with the MultiSpanQA dataset.
Sequence tagging model Compared to the single-span baselines, the sequence tagging models perform much better. Without changing the encoder, there is an improvement of over 30 absolute points on the exact match metrics, and about 8 for the partial match F1 metric in MultiSpanQA. Performance is boosted using joint training with span number prediction and answer semantics prediction. Our proposed model achieves the best F1 score in most settings.
Another interesting finding is that single-span models usually attain higher precision, while sequence tagging models attain higher recall. This demonstrates that single-span models are more accurate in the single-span answer they predict, while sequence tagging models predictably tend to make more predictions.
Comparing the two datasets Comparing results on the two datasets, we see that single-span baselines are boosted over the expanded dataset (where we add single-span answers and unanswerable questions), as single-span answers are more tractable for these simpler models. The relative improvements for sequence tagging models are more modest, but they still have a clear advantage over the single-span baselines.
Difficulty analysis To explore the difficulty of the MultiSpanQA dataset, we report the dev set results categorised by answer type in Table 5 and categorised by the number of spans in Table 6. From the answer type perspective, the model performs best on HUMAN questions, followed by OTHER ENTITY and LOCATION (largely following the natural distribution of the respective classes in the dataset). There is quite a drop for the NUMERIC class, and a big drop again for the DESCRIPTION class, which was also the class our annotators found most difficulty with.
From the perspective of the number of spans, the model performs best on questions with many  $(>7)$  answers. We think this is because the answers are usually a list of spans with similar semantics, often structured as a simple coordination. The performance drops as the answer number decreases because the syntactic pattern in which answer spans occurs is less predictable.
Answer Semantics From the answer type perspective, LOCATION answers usually have easily predictable structure, while the structure of NUMERIC answers is the most difficult to predict.
Figure 3: Confusion matrix of answer structure prediction, based on log values.
From the perspective of the number of spans, answers consisting of 4-7 spans are relatively easy to predict and there is no significant difference between answers contain few (2 or 3) spans or many  $(>7)$  spans. Figure 3 shows the confusion matrix of the answer structure predictions. We can see that our model tends to predict CONJUNCTION and NON-REDUNDANT, and there are no REDUNDANT or SHARE predictions.
The overall answer structure accuracy is  $84.38\%$ , which is slightly higher than the proportion of CONJUNCTION (the majority class) in the dataset. This suggests that directly applying a simple feedforward network to the pooled encoder output is ineffective for answer semantics prediction, and that this should be an area for future model refinement.
We present MultiSpanQA, a reading comprehension dataset where answers consist of multiple discrete spans. As part of this, we proposed a method for classifying the semantic structure of answers, based on the semantic relation between answer spans. We also provide an expanded version of the dataset which includes unanswerable questions and single-answer questions, to make it both more challenging and more realistic. We additionally presented a number of models for multi-span QA extraction, and found that the best-performing model was sequence tagging-based, augmented by a span number prediction module and span adjustment module.
The authors would like to thank the anonymous reviewers for their constructive reviews. This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE170100200. This research was supported by Australian Research Council grant DP170100109.
Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: BookTest dataset for reading comprehension. CoRR, abs/1610.00956.
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358-2367, Berlin, Germany. Association for Computational Linguistics.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentauu Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174-2184.
Peng Cui, Dongyao Hu, and Le Hu. 2021. ListReader: Extracting list-form answers for opinion questions. CoRR, abs/2110.11692.
Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5924-5931.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, USA, pages 4171-4186.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2368-2378.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. CoRR, abs/1704.05179.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: long form question answering. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 3558-3567.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Annual Conference on Neural Information Processing Systems, pages 1693-1701.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The Goldilocks principle: Reading children's books with explicit memory representations. In 4th International Conference on Learning Representations.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1596-1606.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601-1611.
Tomás Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Xin Li and Dan Roth. 2002. Learning question classifiers. In 19th International Conference on Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings
of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches 2016 collocated with the 30th Annual Conference on Neural Information Processing Systems, volume 1773.
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Lixin Su, and Xueqi Cheng. 2019. HAS-QA: hierarchical answer spans model for open-domain question answering. In The Thirty-Third AAAI Conference on Artificial Intelligence, pages 6875-6882.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 784-789.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Azriel Rosenfeld and Mark Thurston. 1971. Edge and curve detection for visual scene analysis. IEEE Transactions on Computers, 20(5):562-569.
Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, and Jonathan Berant. 2020. A simple and effective model for answering multi-span questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3074-3080.
Pontus Stenetorp, Sampo Pyysalo, Goran Topic, Tomoko Ohta, Sophia Ananiadou, and Jun'ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 102-107, Avignon, France. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013-2018.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference
on Empirical Methods in Natural Language Processing, pages 2369-2380.