SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 4 shows the MRR results on the five datasets with in-domain training. SPARTA can achieve the best performance across all domains with a large margin. In terms of average MRR across the five domains, SPARTA is  $114.3\%$  better than BM25,  $50.6\%$  better than USE-QA and  $26.5\%$  better than Poly-Encoders.
Two additional insights can be drawn from the results. First, BM-25 is a strong baseline and does not require training. It performs particularly well in domains that have a high-rate of word-overlapping between the answer and the questions. For example, SQuAD's questions are generated by crowd workers who look at the ground truth answer, while ques
1 https://tfhub.dev/google/universal-sentence-encoder-multilingual-qa
Data	BM25	USE-QA	Poly Enc	SPARTA (ours)
SQuAD	58.0	62.5	64.6	78.5
News	19.4	26.2	28.3	46.6
Trivia	29.0	41.2	39.5	55.5
NQ	19.7	58.2	69.9	77.1
HotPot	23.9	25.5	51.8	63.8
Avg	30.0	42.7	50.8	64.3
Table 4: MRR comparison for the in-domain settings. The proposed SPARTA consistently outperform all the baseline models with large margin. BM25 and USEQA are unsupervised and pre-trained respectively.
tion data from NQ/News are generated by question makers who do not see the correct answer. BM25 works particularly well in SQuAD while performing the poorest in other datasets. Similar observations are also found in prior research (Ahmad et al., 2019).
Second, the results in Table 4 confirms our hypothesis on the importance of rich interaction between the answer and the questions. Both USE-QA and Poly Encoder use powerful transformers to encode the whole question and model word-order information in the queries. However, their performance is bounded by the simple dot-product interaction between the query and the answer. On the other hand, despite the fact that SPARTA does not model word-order information in the query, it is able to achieve a big performance gain compared to the baselines, confirming the effectiveness of the proposed token-level interaction method in Eq. 4.
5.2 Out-of-domain Generalization
Table 5 summarized the results for out-of-domain performance comparison. SPARTA trained only on SQuAD outperforms the baselines, achieving  $54.1\%$  gain compared to BM25,  $26.7\%$  gain compared to USE-QA and  $25.3\%$  gain compared to Poly-Encoders in terms of average MRR across 11 different datasets. When SPARTA is trained on SQuAD+NQ, an additional 1.7 MRR improvement is gained compared to SPARTA-SQuAD.
We can observe that Poly-Encoder is able to achieve similar in-domain performance for the domains that are included in the training. However, its performance decreases significantly in new domains, a  $25.0\%$  drop compared to its full performance for Poly-Encoder that is trained on SQuAD and  $29.2\%$  drop when it's trained on SQuAD+NQ.
Meanwhile, SPARTA generalizes its knowledge
from the training data much better to new domains. When trained on SQuAD, its performance on News, Trivia, NQ, and HotPot is only  $19.2\%$  lower than the full performance and  $18.3\%$  drop when it's trained on SQuAD+NQ. Also, we note that SPARTA's zero-shot performance on News (MRR=41.2) and Trivia (MRR=45.8) is even better than the full performance of Poly-Encoder (News MRR=28.3 and Trivia MRR=39.5).
6 Model Analysis
6.1 Interpreting Sparse Representations
One common limitation of deep neural network models is poor interpretability. Take dense distributed vector representation for example, one cannot directly make sense of each dimension and has to use dimension reduction and visualization methods, e.g. TSNE (Maaten and Hinton, 2008). On the contrary, the resulting SPARTA index is straightforward to interpret due to its sparse nature. Specifically, we can understand a SPARTA vector by reading the top K words with non-zero  $f(t,(a,c))$ , since these terms have the greatest impact to the final ranking score.
Table 6 shows some example outputs. It is not hard to note that the generated terms for each answer sentence is highly relevant to both  $a$  and  $c$ , and contains not keywords that appeared in the answer, but also include terms that are potentially in the query but never appear in the answer itself. Two experts manually inspect the outputs for  $500(a,c)$  data points from Wikipedia, and we summarize the following four major categories of terms that are predicted by SPARTA.
Conversational search understanding: the third row is an example. "Who" appears to the top term, showing it learns Bill Gates is a person so that it's likely to match with "Who" questions.
Keyword identification: terms such as "gates, google, magnate, yellowstone" have high scores in the generated vector, showing that SPARTA learns which words are important in the answer.
Synonyms and Common Sense: "benefactor, investors" are examples of synonyms. Also even though "Utah" does not appear in the answer, it is predicted as an important term, showing that SPARTA leverages the world-knowledge from a pretrained language model and knows Yellowstone is related to Utah.
Model	SQuAD	News	Trivia	NQ	HotPot	Bio	DROP	DuoRC	RACE	RE	Text	Avg
Unsupervised or pretrained
BM25	58.0	19.4	29.0	19.7	23.9	8.9	32.6	20.1	14.8	87.4	21.6	30.5
USE-QA	62.5	26.2	41.2	58.2	25.5	7.7	31.9	20.8	25.6	84.8	26.4	37.4
Trained on SQuAD
PolyEnc	64.6*	22.2	35.9	57.6	26.5	9.1	32.6	25.4	24.7	88.3	26.0	37.5
SPARTA	78.5*	41.2	45.8	62.0	47.7	14.5	37.2	35.9	29.7	96.0	28.7	47.0
Trained on SQuAD + NQ
PolyEnc	63.9*	19.8	36.9	69.7*	29.6	8.8	30.7	19.6	25.2	72.8	24.6	36.5
SPARTA	79.0*	40.3	47.6	75.8*	47.5	15.0	37.9	36.3	30.0	97.0	29.3	48.7
Table 5: MRR comparison in the out-of-domain settings. The proposed SPARTA is able to achieve the best performance across all tasks, the only learning-based method that is able to consistently outperform BM25 with larger margin in new domains. Results with * are in-domain performance.
Answer (a, c)	Top terms
Google was founded in September 1998 by Larry Page and Sergey Brin while they were Ph.D. students at Stanford University in California.	google, when, founded, page, stanford, sergey, larry, found-ing, established, did, 1998, was, year, formed ...
Yellowstone National Park is an American national park located in the western United States, with parts in Wyoming, Montana and Idaho.	montana, yellowstone, wyoming, idaho, park, where, national, western, american, us, utah ...
William Henry Gates is an American business magnate, software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation.	who, gates, investors, magnate, developer, microsoft, philanthropist, benefactor, investors, ...
Question answering (QA) is a computer science discipline within the fields of information retrieval and natural language processing (NLP).	answering, question, q, computer, information,,re-trieval,language, natural, human, nl, science, ...
Table 6: Top-k terms predicted by SPARTA. The text in bold is the answer sentence and the text surrounded it is encoded as its context. Each answer sentence has around 1600 terms with non-zero scores.
Top-K	SQuAD	NQ
MRR	R@1	MRR	R@1
50	69.5	61.3	63.2	52.5
100	72.3	64.4	65.6	55.7
500	76.9	69.4	74.4	64.3
1000	78.2	70.8	75.5	65.6
1500	78.6	71.2	75.7	65.7
2000	78.9	71.4	75.9	66.0
Full	79.0	71.6	75.8	66.0
Table 7: Performance on ReQA task with varying sparsity. SPARTA outperforms all baselines with top-50 terms on SQuAD, and with top-500 terms on NQ.
6.2 Sparsity vs. Performance
Sparsity not only provides interpretability, but also offers flexibility to balance the trade-off of memory footprint vs. performance. When there are memory constraints on the vector size, the SPARTA vector can be easily reduced by only keeping the top-K important terms. Table 7 shows performance on SQuAD and NQ with varying K. The resulting sparse vector representation is very robust to smaller K. When only keeping the top 50 terms in each answer vector, SPARTA achieves 69.5 MRR, a better score than all baselines with only  $1.6\%$
memory footprint compared to Poly-Encoders (768 x 4 dimension). NQ dataset is more challenging and requires more terms. SPARTA achieves a close to the best performance with top-500 terms.
In short, we propose SPARTA, a novel ranking method, that learns sparse representation for better open-domain QA. Experiments show that the proposed framework achieves the state-of-the-art performance for 4 different open-domain QA tasks in 2 languages and 11 retrieval QA tasks. This confirm our hypothesis that token-level interaction is superior to sequence-level interaction for better evidence ranking. Analyses also show the advantages of sparse representation, including interpretability, generalization and efficiency.
Our findings also suggest promising future research directions. The proposed method does not support multi-hop reasoning, an important attribute that enables QA systems to answer more complex questions that require collecting multiple evidence passages. Also, current method only uses a bag-of-word features for the query. We expect further gain by incorporating word-order information.
Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019. Reqa: An evaluation for end-to-end answer retrieval models. arXiv preprint arXiv:1907.04780.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2019. Learning to retrieve reasoning paths over wikipedia graph for question answering. arXiv preprint arXiv:1911.10470.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533-1544.
Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415-1425.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250.
Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pretraining tasks for embedding-based large-scale retrieval. arXiv preprint arXiv:2002.03932.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL).
Muthuraman Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning cross-lingual sentence representations via a multi-task dual-encoder model. arXiv preprint arXiv:1810.12836.
Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pretrained models for chinese natural language processing. In Findings of EMNLP. Association for Computational Linguistics.
Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2018. A span-extraction dataset for chinese machine reading comprehension. arXiv preprint arXiv:1810.07366.
Zhuyun Dai and Jamie Callan. 2020. Context-aware document term weighting for ad-hoc search. In Proceedings of The Web Conference 2020, pages 1897-1907.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W Cohen. 2020. Differentiable reasoning over a virtual knowledge base. arXiv preprint arXiv:2002.10640.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1156-1165.
David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010. Building watson: An overview of the deepqa project. AI magazine, 31(3):59-79.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. arXiv preprint arXiv:1910.09753.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 55-64.
Matthew Henderson, Inigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Ivan Vulić, et al. 2019. Convert: Efficient and accurate conversational representations from transformers. arXiv preprint arXiv:1911.03688.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019. Retrieve, read, rerank: Towards end-to-end multi-document reading comprehension. arXiv preprint arXiv:1906.04618.
Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2017. Pacrr: A position-aware neural in model for relevance matching. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1049-1058.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. CoRR
abs/1905.01969. External Links: Link Cited by, 2:2-2.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wentau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4999-5007.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683.
Jinhyuk Lee, Minjoon Seo, Hannaneh Hajishirzi, and Jaewoo Kang. 2020. Contextualized sparse representations for real-time open-domain question answering. arXiv: Computation and Language.
Jinhyuk Lee, Seongjun Yun, Hyunjae Kim, Miyoung Ko, and Jaewoo Kang. 2018. Ranking paragraphs for improving answer recall in open-domain question answering. arXiv preprint arXiv:1810.00494.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00300.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115.
Mike Lewis and Angela Fan. 2018. Generative question answering: Learning to answer the whole question.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Expansion via prediction of importance with contextualization. arXiv preprint arXiv:2004.14245.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge university press.
Michael McCandless, Erik Hatcher, Otis Gospodnetic, and O Gospodnetic. 2010. Lucene in action, volume 2. Manning Greenwich.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A discrete hard em approach for weakly supervised question answering. arXiv preprint arXiv:1909.04849.
Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and robust question answering from minimal context over documents. arXiv preprint arXiv:1805.08092.
Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019. From doc2query to doccttttquery.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-389.
Amrita Saha, Rahul Aralikatte, Mitesh M Khapra, and Karthik Sankaranarayanan. 2018. Duorc: Towards complex language understanding with paraphrased reading comprehension. arXiv preprint arXiv:1804.07927.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with
dense-sparse phrase index. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4430-4441.
Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2018. Drcd: a chinese machine reading comprehension dataset. arXiv preprint arXiv:1806.00920.
Anshumali Shrivastava and Ping Li. 2014. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). In Advances in Neural Information Processing Systems, pages 2321-2329.
Adam Trischler, Tong Wang, Xingdi (Eric) Yuan, Justin D. Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16(1):138.
Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. R 3: Reinforced ranker-reader for open-domain question answering. In Thirty-Second AAAI Conference on Artificial Intelligence.
Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage bert: A globally normalized bert model for open-domain question answering. arXiv preprint arXiv:1908.08167.
Yuqing Xie, Wei Yang, Luchen Tan, Kun Xiong, Nicholas Jing Yuan, Baoxing Huai, Ming Li, and Jimmy Lin. 2020. Distant supervision for multi-stage fine-tuning in retrieval-based question answering. In Proceedings of The Web Conference 2020, pages 2934-2940.
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval, pages 55-64.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019a. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01718.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. 2019b. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.04307.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
Size details of multi-domain ReQA task.
Domain	# of Query	# of Candidate Answer
SQuAD	11,426	10,250
News	8,633	38,199
Trivia	8,149	17,845
NQ	1,772	7,020
Hotpot	5,901	38,906
BioASQ	1,562	13,802
DROP	1,513	2,488
DuoRC	1,568	5,241
RACE	674	10,630
RE	2,947	2,201
Textbook	1,503	14,831
Table 8: Size of the evaluation test set for the 11 corpora included in MultiReQA.