SlowGuess/NAACL_2021_OCR huggingface dataset:
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378.
Xiachong Feng, Xiaocheng Feng, Bing Qin, Xinwei Geng, and Ting Liu. 2020. Dialogue discourse-aware graph convolutional networks for abstractive meeting summarization. arXiv preprint arXiv:2012.03502.
Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq Joty, Steven CH Hoi, Caiming Xiong, Irwin King, and Michael Lyu. 2020. Discern: Discourse-aware entailment reasoning network for conversational machine reading. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2439-2449.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401.
Yinya Huang, Meng Fang, Xunlin Zhan, Qingxing Cao, Xiaodan Liang, and Liang Lin. 2020. Remnet: Recursive erasure memory network for commonsense evidence refinement. arXiv preprint arXiv:2012.13185.
Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Gabriel Murray. 2019. Discourse analysis and its applications. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 12-17, Florence, Italy. Association for Computational Linguistics.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. *IJCAI* 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Aman Madaan, Dheeraj Rajagopal, Yiming Yang, Abhilasha Ravichander, Eduard Hovy, and Shrimai Prabhumoye. 2020. Eigen: Event influence generation using pre-trained language models. arXiv preprint arXiv:2010.11764.
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243-281.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391.
Todor Mihaylov and Anette Frank. 2019. Discourse-aware semantic self-attention for narrative reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2541–2552, Hong Kong, China. Association for Computational Linguistics.
Lis Pereira, Xiaodong Liu, Fei Cheng, Masayuki Asahara, and Ichiro Kobayashi. 2020. Adversarial training for commonsense inference. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 55-60, Online. Association for Computational Linguistics.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In LREC. Citeseer.
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2019. Dynamically fused graph network for multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the
Association for Computational Linguistics, pages 6140-6150, Florence, Italy. Association for Computational Linguistics.
Dheeraj Rajagopal, Niket Tandon, Peter Clark, Bhavana Dalvi, and Eduard Hovy. 2020. What-if i ask you to explain: Explaining the effects of perturbations in procedural text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3345-3355.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019. Numnet: Machine reading comprehension with numerical reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2474-2484.
Amrita Saha, Shafiq Joty, and Steven CH Hoi. 2021. Weakly supervised neuro-symbolic module networks for numerical reasoning. arXiv preprint arXiv:2101.11802.
Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. Wiqa: A dataset for "what if..." reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6078-6087.
Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264-1274.
Wei Wang, Piji Li, and Hai-Tao Zheng. 2020. Consistency and coherency enhanced story generation. arXiv preprint arXiv:2010.08822.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020a. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021-5031, Online. Association for Computational Linguistics.
Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, Joey Tianyi Zhou, and Chengqi Zhang. 2020b. Deep
reinforcement learning with stacked hierarchical attention for text-based games. In Advances in Neural Information Processing Systems, volume 33, pages 16495-16507. Curran Associates, Inc.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. In *ICLR* 2020: Eighth International Conference on Learning Representations.
A Discourse Delimiter Library
Our discourse delimiter library consists of two parts, the "Explicit" connectives annotated in Penn Discourse TreeBank 2.0 (DPTB 2.0) (Prasad et al., 2008), as well as a set of punctuation marks. The overall discourse delimiters used in our method are presented in Table 4.
Explicit Connectives
'once', 'although', 'though', 'but', 'because', 'nevertheless', 'before', 'for example', 'until', 'if', 'previously', 'when', 'and', 'so', 'then', 'while', 'as long as', 'however', 'also', 'after', 'separately', 'still', 'so that', 'or', 'moreover', 'in addition', 'instead', 'on the other hand', 'as', 'for instance', 'nonetheless', 'unless', 'meanwhile', 'yet', 'since', 'rather', 'in fact', 'indeed', 'later', 'ultimately', 'as a result', 'either or', 'therefore', 'in turn', 'thus', 'in particular', 'further', 'afterward', 'next', 'similarly', 'besides', 'if and when', 'nor', 'alternatively', 'whereas', 'overall', 'by comparison', 'till', 'in contrast', finally, 'otherwise', 'as if', 'thereby', 'now that', 'before and after', 'additionally', 'meantime', 'by contrast', 'if then', 'likewise', 'in the end', 'regardless', 'thereafter', 'earlier', 'in other words', 'as soon as', 'except', 'in short', 'neither nor', 'furthermore', 'lest', 'as though', 'specifically', 'conversely', 'consequently', 'as well', 'much as', 'plus', 'and', hence, by then, accordingly, on the contrary, 'simultaneously', 'for', 'in sum', 'when and if', 'insofar as', 'else', 'as an alternative', on the one hand on the other hand
Punctuation Marks
Table 4: The discourse delimiter library in our implementation.
We fine-tune RoBERTa-Large (Liu et al., 2019) as the backbone pre-trained language model for DGAN, which contains 24 hidden layers with hidden size 1024. The overall model is end-to-end trained and updated by Adam (Kingma and Ba, 2015) optimizer with an overall learning rate of 5e-6 and a weight decay of 0.01. The overall dropout rate is 0.1. The maximum sequence length is 256. We tune the model on the dev set to obtain the best iteration steps of graph reasoning, which is 2 for ReClor data, and 3 for LogiQA data. The model is trained for 10 epochs with a batch size of 16 on Nvidia Tesla V100 GPU.
For the answer prediction module, the hidden size of GRU is the same as the token embeddings in the pre-trained language model, which is 1024. The two-layer perceptron first projects the concatenated vectors with a hidden size of  $1024 \times 3$  to 1024, then project 1024 to 1.