SlowGuess/NAACL_2021_OCR huggingface dataset:
chine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. ArXiv, abs/1705.00652.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Herve Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, pages 1-1.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Berant. 2017. Learning recurrent span representations for extractive question answering. In arXiv 1611.01436.
Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2020. Unified open-domain question answering with structured and unstructured knowledge. arXiv preprint arXiv:2012.14610.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Radim Šehúrek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45-50, Valletta, Malta. ELRA.
Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333-389.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242, Brussels, Belgium. Association for Computational Linguistics.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72-77, Minneapolis, Minnesota. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
Vicky Zayats, Kristina Toutanova, and Mari Ostendorf. 2021. Representations for question answering from documents with tables and text. In Proceedings of the 2021 Conference of the European Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota. Association for Computational Linguistics.
The DTR model uses a batch size of 256. We pre-trained the question and table encoders for 1M steps, and fine-tuned them for a maximum of 200,000 steps, with a learning rate of 1.25e-5 using Adam and linear scheduling with warm-up and dropout rate 0.2. The hyper-parameter values were selected based on the values used by Herzig et al. (2020) on the SQA dataset. We evaluate DTR performance using recall@k, and do early stopping according to recall@10 on the dev set. We use only the tables that appear in the dev set as the corpus for the early stopping for efficiency.
For the QA reader, we initialize the model from the public TAPAS checkpoint. We use a batch size of 512, train for 50,000 steps with a learning rate of 1e-6, and dropout rate 0.2. In this setup we do not use early stopping but always train the model for the full number of steps. We limit the maximal answer length to 10 word pieces. The hyper-parameters of the QA model were optimized using a black box Bayesian optimizer similar to Google Vizier (Golovin et al., 2017). We used the hyper-parameter bounds given in Table 3.
parameter	min	max
learning rate	1e-6	1e-2
warm up ratio	0.0	0.2
dropout	0.0	0.2
Table 3: Hyper-parameter ranges for tuning the QA model.
We trained all models on 32 Cloud TPU v3. Pre-Training a retrieval model takes approx. 6 days. Training a retrieval model takes approx. 4-5h. Training a QA model takes approx. 10h.
The number of parameters is the same as for a BERT large model: 340M.
Dev and test results for the retrieval experiments are given in Table 4. Dev and test results for end-to-end QA are given in the appendix in Table 5.
Model	Dev	Test
R@1	R@10	R@50	R@1	R@10	R@50
BM25	17.13	42.13	57.21	16.77	40.06	58.39
DTR-Text	28.68 ± 0.37	67.76 ± 0.56	85.75 ± 0.84	32.90 ± 0.57	72.00 ± 2.14	86.86 ± 1.25
DTR-Schema	29.38 ± 2.11	67.67 ± 0.28	85.29 ± 0.19	34.36 ± 1.09	74.24 ± 1.46	88.37 ± 0.47
DTR-Text +hnbm25	35.30 ± 2.05	74.10 ± 1.50	87.85 ± 0.23	40.88 ± 0.66	78.10 ± 0.74	91.48 ± 0.21
DTR	31.58 ± 0.09	71.79 ± 0.00	88.38 ± 0.37	36.24 ± 0.57	76.02 ± 0.10	90.25 ± 0.89
DTR +hnbm25	37.86 ± 1.77	75.65 ± 0.72	89.58 ± 0.39	42.17 ± 2.91	80.51 ± 0.66	92.31 ± 0.70
DTR +hn	39.14 ± 0.98	76.13 ± 1.30	89.91 ± 0.35	42.42 ± 2.94	81.13 ± 1.61	92.56 ± 0.96
DTR -pt	9.05 ± 1.08	35.73 ± 2.21	60.34 ± 3.20	16.64 ± 1.49	47.80 ± 1.42	68.68 ± 0.91
Table 4: Table retrieval results on NQ-TABLES dev and test sets. hn: hard negatives, hnbm25: hard negatives from BM25 baseline, pt: pre-training.
Retriever	Reader	Dev	Test
EM	F1	Oracle EM	Oracle F1	EM	F1	Oracle EM	Oracle F1
BM25	TAPAS	18.76 ± 0.80	26.32 ± 1.23	25.57 ± 0.77	37.45 ± 0.81	21.46 ± 0.71	28.24 ± 0.78	29.51 ± 0.49	40.79 ± 0.43
DTR-Text	Bert	21.11 ± 1.09	29.17 ± 1.11	30.74 ± 1.07	42.82 ± 0.80	29.58 ± 0.85	37.38 ± 0.87	39.39 ± 0.31	51.48 ± 0.30
DTR-Text	TAPAS	27.67 ± 1.30	37.13 ± 1.74	36.44 ± 1.35	49.17 ± 1.66	33.78 ± 1.12	43.49 ± 1.23	42.83 ± 0.74	56.46 ± 0.55
DTR-Text +hnbm25	TAPAS	27.84 ± 0.95	38.00 ± 1.14	38.97 ± 0.81	52.38 ± 0.97	36.89 ± 0.77	46.67 ± 0.98	46.30 ± 0.65	59.22 ± 0.84
DTR-Schema	TAPAS	27.12 ± 1.04	36.14 ± 1.17	36.19 ± 0.93	48.81 ± 1.29	32.75 ± 0.36	42.19 ± 0.21	42.63 ± 0.68	55.05 ± 0.59
DTR	TAPAS	27.84 ± 1.62	37.77 ± 1.86	38.43 ± 0.75	51.26 ± 0.69	35.50 ± 0.45	45.44 ± 0.53	46.09 ± 0.47	59.01 ± 0.30
DTR +hn	TAPAS	28.67 ± 0.57	39.14 ± 0.46	39.38 ± 0.69	53.08 ± 0.43	37.69 ± 0.87	47.70 ± 1.05	48.20 ± 0.53	61.50 ± 0.34
DTR +hn c=1	TAPAS	23.50 ± 0.12	33.44 ± 0.30	23.50 ± 0.12	33.44 ± 0.30	31.12 ± 0.31	39.44 ± 0.20	31.12 ± 0.31	39.44 ± 0.20
DTR +hn c=50	TAPAS	23.97 ± 2.22	33.72 ± 2.81	42.11 ± 1.95	58.02 ± 2.08	30.73 ± 2.79	40.77 ± 3.26	47.26 ± 1.33	63.04 ± 1.51
DTR +hnbm25	TAPAS	27.67 ± 0.63	37.77 ± 0.93	40.26 ± 0.94	53.77 ± 0.80	36.61 ± 0.76	46.74 ± 0.82	47.46 ± 0.83	60.72 ± 0.91
Table 5: QA results on NQ-TABLES dev and test set. c: Number of candidates (default is 10), hn: With hard negatives, hnbm25: with hard negatives from BM25.