SlowGuess/NAACL_2022_OCR huggingface dataset:
national Joint Conference on Artificial Intelligence, IJCAI 2020, pages 1230-1236. ijcai.org.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13042-13054.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics.
Hsin-Yuan Huang, Eunsol Choi, and Wen-tau Yih. 2019. Flowqa: Grasping flow in history for conversational machine comprehension. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. 2019. Technical report on conversational question answering. arXiv preprint arXiv:1909.10772.
Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. 2020. Sequential latent knowledge selection for knowledge-grounded dialogue. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Gangwoo Kim, Hyunjae Kim, Jungsoo Park, and Jaewoo Kang. 2021. Learn to resolve conversational dependency: A consistency training framework for conversational question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6130-6141, Online. Association for Computational Linguistics.
Yasuhito Ohsugi, Itsumi Saito, Kyosuke Nishida, Hisako Asano, and Junji Tomita. 2019. A simple but effective method to incorporate multi-turn context with BERT for conversational machine comprehension. In Proceedings of the First Workshop on NLP for Conversational AI, pages 11-17, Florence, Italy. Association for Computational Linguistics.
Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019a. BERT with history answer embedding for conversational question answering. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1133-1136. ACM.
Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W. Bruce Croft, and Mohit Iyyer. 2019b. Attentive history selection for conversational question answering. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019, pages 1391-1400. ACM.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 355-363.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Mark Yatskar. 2019. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2318-2323, Minneapolis, Minnesota. Association for Computational Linguistics.
Yi-Ting Yeh and Yun-Nung Chen. 2019. FlowDelta: Modeling flow information gain in reasoning for conversational machine comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 86-90, Hong Kong, China. Association for Computational Linguistics.
Jing Zhao, Junwei Bao, Yifan Wang, Yongwei Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2021. RoR: Read-over-read for long document machine reading comprehension. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 1862-1872, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020. Knowledge-grounded dialogue generation with pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3377-3390, Online. Association for Computational Linguistics.
Chujie Zheng, Yunbo Cao, Daxin Jiang, and Minlie Huang. 2020. Difference-aware knowledge selection for knowledge-grounded conversation generation. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 115-125, Online. Association for Computational Linguistics.
Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. Sdnet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812.03593.
To model the history information in conversations, a kind of study focus on how to explicitly select important history question-answer pairs at the input step (Zhu et al., 2018; Reddy et al., 2019; Ju et al., 2019). Besides, a few studies argue that historical answer spans in context are more crucial, so they mark the answers when encoding the context (Choi et al., 2018; Qu et al., 2019a; Ohsugi et al., 2019; Qu et al., 2019b).
Another mainline of studies focus on conversation history reasoning. Huang et al. (2019) proposes a flow operation, feeding models entire hidden representation obtained by reasoning process when answering previous questions. The hidden states of each turn are passed back in turn by a unidirectional GRU. To avoid the changes of the captured representations during multi-turn reasoning, Yeh and Chen (2019) propose a flowdelta mechanism to explicitly capture the information gain in the conversation flow. Moreover, Chen et al. (2020) implement history reasoning by a flow operation on context graphs.
Recently, Vakulenko et al. (2021) propose to rewrite the current question using conversation history, with the goal to seek dependable clues to recover the default contents or resolve the co-references. On the basis, Kim et al. (2021) resolve the conversational dependency via consistency regularization, and jointly use the original and rewritten questions to lead the supervised learning of QA models.
Modeling conversation history is a hot research area. It has also been widely studied in the direction of Knowledge-Grounded Conversation Generation (Kim et al., 2020; Zheng et al., 2020; Zhao et al., 2020). Such methods tend to model the full context and knowledge to generate responses, while GHR tends to model the context that is effective for answering the current question, and simultaneously learn the global history reasoning gain brought by different turns of history and the optimization of answer extraction.
A.2 Experiments on CoQA
CoQA (Reddy et al., 2019) is another typical ConvQA dataset. Table 4 lists the results of GHR model with from 1 to 7 of the GHA layers. It shows that the GHR model only improves  $0.7\%$  F1 over the baseline ELECTRA model on CoQA's dev
Models	F1
ELECTRA	89.0
w/ 1 GHA layer	89.3
w/ 2 GHA layers	89.5
w/ 3 GHA layers	89.7
w/ 4 GHA layers	89.5
w/ 5 GHA layers	89.3
w/ 6 GHA layers	88.8
w/ 7 GHA layers	88.6
Table 4: Effects of the number of GHA layers on the CoQA dev set.
Case #1
Section: Early political career: John Sherman Cooper (id: C_5caef3e3024c4f9294e1dacda1ff09b7_1)
Q0: What was the first job he held?
A0: After being urged into politics by his uncle, Judge Roscoe Tartar, Cooper ran unopposed for a seat in the Kentucky House of Representatives
Q1: What was the first office he ran for?
A1: Kentucky House of Representatives
Q2: How long was he in office?
A2: CANNOTANSWER
Current Question Q3: Did he run for another political office after that?
Ground Truth: In 1929, Cooper declared his candidacy for county judge of Pulaski County.
ELECTRA: In 1939, he sought the Republican gubernatorial nomination.
GHR (ELECTRA): In 1929, Cooper declared his candidacy for county judge of Pulaski County.
Table 5: An example of the results predicted by GHR (ELECTRA) and ELECTRA on the dev set.
set. This is because CoQA's problems are quite straightforward, most of which can be predicted without conversation history (Yatskar, 2019). Thus CoQA is not suitable for our global history reasoning goals. Nevertheless, we still find that the effect of GHA layers on GHR was consistent in CoQA and QuAC, which means that GHR based on the same pretraining model performed relatively stable in different ConvQA tasks. As a result, GHR can play a role in modeling global history information in a variety of ConvQA tasks.
The example in Table 5 compares the predictions between the model with ELECTRA and GHR (ELECTRA) model. On the dev set, we observe that GHR (ELECTRA) outperforms the ELECTRA model without the global history reasoning mechanism in almost all instances that contain similar long-distance referential relations. For example, we can see in Case #1, to correctly answer the current question  $\mathrm{Q}_3$ , the model needs to infer that he refers to Cooper in  $\mathrm{A}_0$ .