SlowGuess/NAACL_2021_OCR huggingface dataset:
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020. RikiNet: Reading Wikipedia pages for natural question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6762-6771, Online. Association for Computational Linguistics.
Shanshan Liu, Xin Zhang, Sheng Zhang, Hui Wang, and Weiming Zhang. 2019. Neural machine reading comprehension: Methods and trends. Applied Sciences, 9(18):3698.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mtest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193-203.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocttäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2087-2097, Brussels, Belgium. Association for Computational Linguistics.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. 2020. Exploiting structured knowledge in text via graph-guided representation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8980-8994, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc.
Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018. Evidence aggregation for answer re-ranking in open-domain question answering. In International Conference on Learning Representations.
Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu. 2020. Cluster-former: Clustering-based sparse transformer for long-range dependency encoding.
Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized BERT model for open-domain question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5878-5882, Hong Kong, China. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Ming Yan, Jiangnan Xia, Chen Wu, Bin Bi, Zhongzhou Zhao, Ji Zhang, Luo Si, Rui Wang, Wei Wang, and Haiqing Chen. 2019. A deep cascade model for multi-document reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7354-7361.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 5754-5764.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting
of the Association for Computational Linguistics, pages 5059-5069, Florence, Italy. Association for Computational Linguistics.
Bo Zheng, Haoyang Wen, Yaobo Liang, Nan Duan, Wanxiang Che, Daxin Jiang, Ming Zhou, and Ting Liu. 2020. Document modeling with graph attention networks for multi-grained machine reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6708-6718, Online. Association for Computational Linguistics.
NQStruct We derive a subset of 46k examples from about 100k training examples with short answers. From the set of 46K, we set aside a subset of 2.5K examples as our dev set. Similarly, we derive a subset of 2.3K examples from the original NQ dev set, and use that as our test set. In the cases that multiple short answers are available, we use the first one for training and evaluation. The number of examples in training/dev/test split is shown in Table 4. The distribution of document length is shown in Table 5.
D2DStruct The public release of Doc2Dial only provides train and dev sets<sup>3</sup>. For filtering out the non-answer agent turns, we filter out the cases where agent turn is grounded on the (sub)section titles. We combine the two and then create train/dev/test splits as  $70\%$ ,  $15\%$ ,  $15\%$  where  $50\%$  of the dev/test set are from documents unseen in training set. The number of examples in training/dev/test split is shown in Table 4. The distribution of document length is shown in Table 5.
# examples	Training	Dev	Test
NQStruct	43294	2500	2333
D2DStruct	7184	1065	1144
Table 4: Number of examples in datasets.
Dataset	≤ 0.5k	0.5k-1k	1k-2k	>2k
NQStruct	7%	9%	15%	69%
D2DStruct	25%	45%	24%	6%
The deep learning systems are in PyTorch, and use Transformer encoder from HuggingFace Transformers. We use Roberta-base pretrained model and max sequence length of 512 unless otherwise stated. All experiments were done with fp16, on a single V100 GPU. Table 8 presents generation configurations and hyper-parameters that are shared by both datasets. For evaluation, we use the evaluation script 2.0 of SQuAD.
Table 9 presents configuration specifics for D2DStruct. For each dialogue in Doc2Dial, we combine all previous turns in reverse order as a
query. Table 10 presents configuration specifics for NQStruct.
C Results on Dev Sets
Table 6 and 7 present evaluation results on the dev sets of D2DStruct and NQStruct datasets respectively.
Table 5: Statistics of document length (in tokens).
Model	F1	Exact Match
SW	53.0 ± 0.4	36.3 ± 0.4
Longformer	56.6 ± 0.8	37.0 ± 0.6
IR+SW	44.5 ± 0.4	31.3 ± 0.5
LeafJC	56.2 ± 0.3	39.7 ± 0.3
TreeJC	57.3± 0.4	41.0± 0.3
Table 6: Results on D2DStruct dev set.
Model	F1	Exact Match
SW	59.6 ± 0.3	49.3 ± 0.2
Longformer	57.8 ± 0.5	47.3 ± 0.5
IR+SW	50.6 ± 0.3	41.7 ± 0.4
LeafJC	58.5 ± 0.4	48.0 ± 0.5
TreeJC	61.3± 0.5	50.6± 0.7
Table 7: Results on NQStruct dev set.
<sup>3</sup>http://doc2dial.github.io/data.html
SW	Longformer	IR+SW	LeafJC	TreeJC
Total batch size	54	64	54	54	54
Batch size per GPU	27	2	27	27	27
Learning rate	3e-5	3e-5	3e-5	3e-5	3e-5
Weight decay	0.01	0.01	0.01	0.01	0.01
N best size	20	20	20	5	5
Warmup proportion	10%	10%	10%	10%	10%
Checkpoint freq	1/4 epoch	1/4 epoch	1/4 epoch	1/10 epoch	1/10 epoch
Table 8: General parameters
SW	Longformer	IR+SW	LeafJC	TreeJC
# epochs	8	8	8	1	1
# instances per epoch	43,961	7,247	16,441	266,853	266,951
max sequence length	512	4096	512	512	512
Document stride	128	3072	128	128	128
max query length	128	128	128	128	128
max answer length	30	30	30	30	30
Table 9: Dataset-specific parameters for D2DStruct experiments
SW	Longformer	IR+SW	LeafJC	TreeJC
# epochs	8	5	8	1	1
# instances per epoch	879, 249	84, 788	275, 261	3, 232, 359	3, 241, 152
max sequence length	512	4096	512	512	512
Document stride	256	3072	256	256	256
max query length	64	64	64	64	64
max answer length	30	30	30	30	30
Table 10: Dataset-specific parameters for NQStruct experiments