SlowGuess/NAACL_2022_OCR huggingface dataset:
Jin Qian $^{1}$ , Bowei Zou $^{2}$ , Mengxing Dong $^{1}$ , Xiao Li $^{1}$ , Ai Ti Aw $^{2}$ , Yu Hong $^{1*}$
$^{1}$ School of Computer Science and Technology, Soochow University, China
$^{2}$ Institute for Infocomm Research, A*STAR
{jaytsien, ayumudong, emilyxiao0512, tianxianer}@gmail.com {zou_bowei, aaiti}@i2r.a-star.edu.sg
Conversational Question Answering (ConvQA) is required to answer the current question, conditioned on the observable paragraph-level context and conversation history. Previous works have intensively studied history-dependent reasoning. They perceive and absorb topic-related information of prior utterances in the interactive encoding stage. It yielded significant improvement compared to history-independent reasoning. This paper further strengthens the ConvQA encoder by establishing long-distance dependency among global utterances in multi-turn conversation. We use multi-layer transformers to resolve long-distance relationships, which potentially contribute to the reweighting of attentive information in historical utterances. Experiments on QuAC show that our method obtains a substantial improvement  $(1\%)$ , yielding the F1 score of  $73.7\%$ . All source codes are available at https://github.com/jaytsien/GHR.
ConvQA is a task of answering questions conditioned on the conversation history as well as referential contexts. It heavily relies on thetraceback of conversation history. For example, the pronoun "it" in  $Q_{2}$  in Table 1 needs to be resolved first. It is indispensable to pursue its coreference "alchemy index" that appeared in the first-turn historical conversation (i.e.,  $Q_{0}$  and  $\mathrm{A_0}$ ). Therefore, the challenge of ConvQA is to detect the relevant evidence hidden in conversation history, and use it to strengthen the current round of question answering.
Recently, utilizing global conversation history for enhancement is increasingly gaining interest, because it potentially contributes to capturing long-distance relevant evidence for answering. Both historical-answer-aware dynamic encoding of context (Qu et al., 2019b) and flow-based interaction
Section: Thrice: The Alchemy Index (2006-2008)
Context: In September 2006, the band announced plans for a new album (later titled The Alchemy Index) on their official website. The album was conceived as a series of 4 EPs, (...) The band maintained a studio blog titled "Alchemy Index" throughout the recording process. (...) The Alchemy Index Vols. I & II was released on October 16, 2007 and sold 28,000 copies in its first week. (...)
$\mathbf{Q}_0$  : What is the alchemy index?
$\mathbf{A}_0$  : In September 2006, the band announced plans for a new album (later titled The Alchemy Index) on their official website.
$\mathbf{Q}_1$  : What is notable about the album?
$\mathbf{A}_1$  : The album was conceived as a series of 4 EPs (...
$\mathbf{Q}_2$  : Was it well received?
$\mathbf{A}_2$  : The Alchemy Index Vols. I & II was released on October 16, 2007 and sold 28,000 copies in its first week.
Table 1: An example from QuAC with the clues in conversation history (in blue) and context (in red).
modeling over shifting topics (Yeh and Chen, 2019) appear as successful solutions, where global conversation history is involved. However, some historical information fails to be maintained due to 1) the omission of historical questions and 2) disconnection from the earliest-stage conversation when topic frequently shifts.
In this paper, we develop a Global History Reasoning (GHR) model. GHR is not only capable of separately encoding different rounds of question-answering (QA) conversations, but sequentially fusing the encoded information of all QA pairs in visible conversations by a multi-layer attention network. It is designed to avoid the omission of available historical information and disconnection. We experiment on QuAC (Choi et al., 2018). The test results show that GHR yields substantial improvements when using BERT and ELECTRA as the baselines, and it achieves competitive performance compared to state-of-the-art methods.
The input of GHR comprises the referential context  $c$ , the current question in the  $t$ -th round, and all his-
* Corresponding author. Email:tianxianer@gmail.com
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2071 - 2078
Figure 1: Architecture of global history reasoning (GHR) model.
torical QA pairs  $\mathcal{H}_t = [(q_1, a_1), \ldots, (q_{t-1}, a_{t-1})]$ , where  $q_i$  and  $a_i$  denote the question and answer in the  $i$ -th round in conversation, respectively. GHR models local history by learning utterance representation for the QA pair in every single round. Then, GHR fuses all visible local history and models their interaction with the referential context by global history attention. Finally, a linear layer with softmax is applied for answer prediction.
Figure 1 shows the overall architecture of GHR, where the sequentially stacked cards (drawn with light green rectangles) denote the encoding stages for the questions issued at different times. For example, the visible structure in the top card illustrates the encoding stage for the first question  $q_{1}$ , and at the time, both the local conversation history  $d_{1}$  and the visible global history are NULL. The global attention mechanism is shown at the right side of the diagram, whose input is the representation encoded by each transformer layer (blue arrows), while the output is the refined representation by the masked global interaction (black arrows). The masking operation is used to temporally disable the subsequent QA conversations when the current question is being dealt with (as required in the task of ConvQA).
2.1 Local History Encoding
We follow the most commonly-used ConvQA scheme (Zhao et al., 2021) to form the input of our encoder. Given the current question  $q_{t}$ , we consider the historical QA pairs in the last two rounds  $d_{t} = (q_{t - 2}, a_{t - 2}, q_{t - 1}, a_{t - 1})$  as the local history of  $q_{t}$ , and concatenate it with  $q_{t}$  and the context  $c$  as the input sequence. For the context whose
length exceeds the maximum input length (usually 512 tokens) of the encoder, we divide them into multiple fragments and put them in a batch in order. Then we use a pre-trained language model (PLM) to encode the input sequence into contextualized representations:
$$
\mathbf {h} _ {t} = P _ {L M} \left(d _ {t}, q _ {t}, c\right), \mathbf {h} _ {t} \in \mathbb {R} ^ {L _ {t} \times d _ {h}} \tag {1}
$$
where  $P_{LM}(\cdot)$  denotes a transformer-based PLM encoder,  $L_{t}$  denotes the maximum length of input sequence and  $d_h$  is the hidden size.
2.2 Global History Reasoning
Most existing ConvQA studies suppose that the latest two-round conversation history has the most direct correlation to the current question. Therefore, they merely encode them into the input representation (Ohsugi et al., 2019; Ju et al., 2019) as mentioned in Eq.(1). This results in the omission of other essential information from the entire conversation, such as that signaling the long-distance reference and topic consistency. Some previous work extended the local encoding by absorbing attentive information from a larger range of QA pairs in history. However, the gradual attenuation for encoding (Yeh and Chen, 2019) causes the failure in giving more prominence to the long-distance related information. Besides, due to the limited and fixed size of the input sequence, the complete conversational interaction process actually has been divided into fragments. This makes it hard to ensure the coherence during encoding a series of conversation units (each unit is a QA pair occurring in the conversation history).
A sufficient flow of information among entire conversation units is warranted to compensate for these defects. Specifically, to enhance the interaction of multi-turn utterances, we design a global history attention mechanism to sequentially fuse contextualized representations of visible historical QA pairs  $\mathcal{D}_C$ . We denote the latent information representation of  $\mathcal{D}_C$  as  $\mathbf{H}_C = [\mathbf{h}_1,\mathbf{h}_2,\dots ,\mathbf{h}_T]$  where  $\mathbf{H}_C\in \mathbb{R}^{T\times L_t\times d_h}$ ,  $T$  is the number of rounds of the conversation. For example,  $\mathbf{h}_i$  is the representation of the  $i$ -th round QA pair.
To obtain deep interactions between different rounds, we transpose the dimension of the input matrices  $\mathbf{H}_C$  as  $L_{t}\times T\times d_{h}$  and obtain  $\mathbf{H}_C^{\mathsf{T}} = [\mathbf{h}_1^{\mathsf{T}},\mathbf{h}_2^{\mathsf{T}},\dots ,\mathbf{h}_T^{\mathsf{T}}]$ . Then we add absolute positional embeddings  $\mathbf{p}_i$  to  $\mathbf{h}_i^{\mathsf{T}}$  to incorporate the position information of each token. The final input embeddings is:
$$
\mathbf {H} _ {C} ^ {\mathsf {T}} = [ (\mathbf {h} _ {1} ^ {\mathsf {T}}; \mathbf {p} _ {1}), (\mathbf {h} _ {2} ^ {\mathsf {T}}; \mathbf {p} _ {2}), \dots , (\mathbf {h} _ {T} ^ {\mathsf {T}}; \mathbf {p} _ {T}) ], (2)
$$
where  $\mathbf{H}_C^\top \in \mathbb{R}^{L_t\times T\times d_h}$
During history reasoning, we apply a Global History Attention (GHA) layer to model the whole history and learn the history-aware representations for each token in the utterance:
$$
\mathbf {R} _ {C} ^ {\top} = \left[ \mathbf {r} _ {1} ^ {\top}, \mathbf {r} _ {2} ^ {\top}, \dots , \mathbf {r} _ {T} ^ {\top} \right] = G H A \left(\mathbf {H} _ {C} ^ {\top}\right), \tag {3}
$$
where  $\mathbf{r}_i^{\mathrm{T}}\in \mathbb{R}^{L_t\times d_h}$  denotes the history-aware representation of the  $i$  -th round,  $GHA(*)$  is a transformer layer with history attention mask. In a real dialogue scene, a speaker is able to see all happened historical utterances before the current round, while subsequent utterances are unseen. Therefore, to avoid leaking the unseen utterances, we leverage the self-attention mask mechanism (Dong et al., 2019) (as shown in the right of Figure 1) to capture the visible historical information associated with token embeddings of each position. We then re-transpose  $\mathbf{R}_C^{\mathrm{T}}\in \mathbb{R}^{L_t\times T\times d_h}$  to obtain  $\mathbf{R}_C = [\mathbf{r}_1,\mathbf{r}_2,\dots ,\mathbf{r}_T]\in \mathbb{R}^{T\times L_t\times d_h}$  for easily fusing the local and global history.
To integrate the contextualized representations  $\mathbf{H}_C$  with the history-aware representations  $\mathbf{R}_C$ , we first adopt  $N$  GHA layers to obtain  $\mathbf{R}_C^l$ . In particular, the  $l$ -th GHA layer (Eq.(5)) is connected behind the  $(l-1)$ -th Transformer layer (Eq.(4)).  $N$  is a hyper-parameter that indicates that each of the last  $N$  Transformer layers is followed by a GHA
layer. Then, we perform layer normalization (Ba et al., 2016) to update the final  $\mathbf{H}_C$  (Eq.(6)).
$$
\mathbf {H} _ {C} ^ {l} = \text {T r a n s f o r m e r} \left(\mathbf {H} _ {C} ^ {l - 1}\right) \tag {4}
$$
$$
\mathbf {R} _ {C} ^ {l} \Leftarrow G H A \left(\mathbf {H} _ {C} ^ {l - 1}\right) \tag {5}
$$
$$
\mathbf {H} _ {C} ^ {l} = \text {L a y e r N o r m} \left(\mathbf {H} _ {C} ^ {l} + \mathbf {R} _ {C} ^ {l}\right) \tag {6}
$$
Finally, we get the fusion representation  $\mathbf{O}_C$  by concatenating  $\mathbf{H}_C^L$  and  $\mathbf{R}_C^{L + 1}$  for the subsequent answer prediction.
$$
\mathbf {O} _ {C} = \left[ \mathbf {H} _ {C} ^ {L}; \mathbf {R} _ {C} ^ {L + 1} \right]. \tag {7}
$$
2.3 Answer Prediction
Given the representations  $\mathbf{O}_C = [\mathbf{o}_1,\mathbf{o}_2,\dots ,\mathbf{o}_T]$  from the global history reasoning module, an answer span is predicted by two linear layers with softmax that calculate the probability of each token being the start and end tokens over  $\mathbf{o}_t$  ..
$$
p _ {t} ^ {s}, p _ {t} ^ {e} = \operatorname {S o f t M a x} (\operatorname {L i n e a r} (\mathbf {o} _ {t})), t \in [ 1, T ] \tag {8}
$$
where  $p_t^s, p_t^e$  are the probabilities of the start and end positions of the answer span in the  $t$ -th round, respectively.
In the training step, we utilize cross entropy to compute the loss of the start and end predictions.
$$
\mathcal {L} _ {s p a n} = - \frac {1}{T} \sum_ {t = 1} ^ {T} \left(y _ {t} ^ {s} \operatorname {l o g p} _ {t} ^ {s} + y _ {t} ^ {e} \operatorname {l o g p} _ {t} ^ {e}\right) \tag {9}
$$
Besides optimizing by the position loss of the answer span (Eq.(9)), we also apply multi-task optimization (Zhao et al., 2021) for training. Specifically, we apply three linear layers with softmax over the "[CLS]" vector of  $\mathbf{o}_t$  to determine the current question's dialog acts, including answerability, affirmation, and continuation (Choi et al., 2018).
$$
p _ {t} ^ {n a}, p _ {t} ^ {a f}, p _ {t} ^ {c t} = \operatorname {S o f t M a x} \left(\operatorname {L i n e a r} \left(\mathbf {o} _ {t} ^ {c l s}\right)\right) \tag {10}
$$
where  $p_t^{na}$  denotes the fractional vector of answerable probability of the question,  $p_t^{af}$  is the probability of affirmation, and  $p_t^{ct}$  is the probability of continuing to ask subsequent questions. We use cross entropy to compute losses of the acts.
$$
\mathcal {L} _ {n a} = - \frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} ^ {n a} \log p _ {t} ^ {n a} \tag {11}
$$
$$
\mathcal {L} _ {a f} = - \frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} ^ {a f} \log p _ {t} ^ {a f} \tag {12}
$$
$$
\mathcal {L} _ {c t} = - \frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} ^ {c t} \log p _ {t} ^ {c t} \tag {13}
$$
For simplicity, we use  $\Leftarrow$  in Eq.(5) to indicate the calculation process with the two transpose steps mentioned above.
where  $y_{t}^{na}, y_{t}^{af}$ , and  $y_{t}^{ct}$  are the ground-truths of answerability, affirmation, continuation, respectively. The final optimization goal is as follow.
$$
\mathcal {L} = \alpha \mathcal {L} _ {s p a n} + \beta (\mathcal {L} _ {c t} + \mathcal {L} _ {a f} + \mathcal {L} _ {n a}) \tag {14}
$$
where  $\alpha$  and  $\beta$  are coefficients for adjusting  $\mathcal{L}_{\text{span}}$  and the combination of  $\{\mathcal{L}_{ct}, \mathcal{L}_{af}, \mathcal{L}_{na}\}$ .
We conduct experiments on QuAC (Choi et al., 2018) consisting of 100K questions obtained from 14K information-seeking dialogues, which proposes unique challenges since these questions are open-ended, descriptive, highly contextual, and probably unanswerable. In particular, many questions require sufficient co-referencing and reasoning through interactions with conversation history.
We employ  $\mathrm{BERT}_{large}^2$  (Devlin et al., 2019) and  $\mathrm{ELECTRA}_{large}^3$  (Clark et al., 2020) as local history encoders. Meanwhile, we apply the voting strategy to implement comparable baseline models. The trade-off coefficients  $\alpha$  and  $\beta$  in the loss function are set to 0.7 and 0.1 respectively (Zhao et al., 2021). The max query length and the stride of sliding window of GHR is set to 128. The batch size is set to 12. The answer length is set to 50 and learning rate is 2e-5. We rely on Pytorch and HuggingFace Transformer libraries (Wolf et al., 2020) for our experiments.
Following Choi et al. (2018), we adopt word-level macro-F1 and human equivalence score (HEQ) as evaluation metrics. HEQ-Q and HEQ-D measure the percentage of answers that the model accurately predicts but the human does not by given questions and dialogues.
We compare our GHR-based ConvQA model with the state-of-the-art models that are reported for performance on the QuAC leaderboard $^{4}$ , including: BiDAF++ (Choi et al., 2018) further augments BiDAF with self-attention and contextualized embeddings.
BiDAF++ w/2-ctx (Choi et al., 2018) additionally models conversation history from the previous two turns of QA pair by encoding their positions in conversation within the question embeddings and
Models	F1	HEQ-Q	HEQ-D
BiDAF++	51.8/50.2	45.3/43.3	2.0/2.2
BiDAF++ w/2-ctx	60.6/60.1	55.7/54.8	5.3/4.0
HAE	63.9/62.4	59.7/57.8	5.9/5.1
FlowQA	64.6/64.1	-/59.6	-/5.8
GraphFlow	-/64.9	-/60.3	-/5.1
FlowDelta	66.1/65.5	-/61.0	-/6.9
HAM	66.7/65.4	63.3/61.8	9.5/6.7
RoR	75.7/74.9	73.4/72.2	17.8/16.4
BERT (ours)	67.7/-	62.9/-	7.8/-
GHR (BERT)	69.0/-	64.6/-	8.0/-
ELECTRA (ours)	73.2/72.7	69.8/68.8	12.2/11.9
GHR (ELECTRA)	74.9/73.7	71.7/69.9	14.6/13.7
Table 2: Results on the dev/test set of QuAC.
concatenating the marker embeddings to the passage embeddings.
HAE (Qu et al., 2019a) introduces a history answer embedding to incorporate the conversation history into BERT.
FlowQA (Huang et al., 2019) feeds the model with the hidden embeddings generated by reasoning in each new round of conversation.
GraphFlow (Chen et al., 2020) encodes conversation history into context graphs for context reasoning and analysis.
FlowDelta (Yeh and Chen, 2019) passes down the information gain between different turns to ensure that the model can focus on more informative cues in context.
HAM (Qu et al., 2019b) adopts position attention embeddings for history selection and optimizes the model from both answer span prediction and dialog act prediction via a multi-task learning framework.
RoR (Zhao et al., 2021) uses chunk reader to obtain chunk answers, which are aggregated for document reader to read again, and votes for the final answer.
Table 2 shows the comparison of the existing published models with relatively high performance on the QuAC leaderboard with our GHR models on both the dev and test set. The test results show that all improvements of GHR are statistically significant (paired t-test (Dror et al., 2018), p-value  $< 0.01$ ). We obtain the following observations.
1) Compared with the baseline models, the performances of the GHR models with BERT and ELECTRA are improved by 1.3 and 1.7 absolute F1 scores on the dev set, respectively. It indicates that GHR is effective for the ConvQA task, even based on the PLM with stronger representation capability (ELECTRA). This suggests that general PLM-dependent ConvQA models may have limita
$^{2}$ https://github.com/google-research/BERT
<sup>3</sup>https://github.com/google-research/electra
4https://quac.ai. Note that we only compare the proposed model to the methods with published papers.
Models	F1	HEQ-Q	HEQ-D
ELECTRA	73.2	69.8	12.2
w/ 1 GHA layer	74.3	71.2	14.5
w/ 2 GHA layers	74.4	71.3	14.1
w/ 3 GHA layers	74.9	71.7	14.6
w/ 4 GHA layers	74.7	71.5	14.4
w/ 5 GHA layers	73.7	70.3	14.8
w/ 6 GHA layers	73.3	69.8	14.0
w/ 7 GHA layers	72.6	69.3	12.7
Table 3: Effects of the number of GHA layers on the QuAC dev set.
tions without a step specifically targeting conversation history interactions.
2) GHR outperforms other models that utilize global conversation history, such as FlowQA, GraphFlow, and FlowDelta. We believe that these "flow"-based models tend to attenuate or ignore earlier conversation histories, which may prioritize recent utterances. Thus, when the current question is correlated to an earlier history or topic drifts, it is disadvantageous for the "flow"-based models. On the contrary, the GHR model still maintains a high focus on the early utterances through the GHA mechanism, so it can effectively utilize the global dialogue history.
3) Compared with the typical ConvQA model HAM, GHR outperforms it by a 2.3 absolute F1 score on the dev set with the same PLM settings (BERT). The reason might be that HAM only employs the answers of at most the first 4 rounds in conversations and pays more attention to the distribution of answer spans in the context, but does not model the question. Thus HAM is also difficult to solve the problem caused by the topic drifting. For GHR, we believe that modeling global history is the most effective factor leading to the benefits of GHR.
4) Compared with the best model on the QuAC leaderboard (RoR), our GHR is 1.2 absolute F1 scores lower than it on the test set. The reason is that RoR employs transfer learning to first finetune itself on the CoQA dataset, but such a method is meaningless for us since the target of this paper is quite different. Moreover, RoR focuses on modeling long contexts, but not modeling conversation history. It only utilizes the question in the current round. Also, GHR can be easily combined with RoR in implementation. Therefore, we suggest fusing the two models to further improve the performance of the ConvQA task.
We conduct an ablation experiment to investigate the effect of the number of the global history atten
tion (GHA) layers on performance. Table 3 shows the results of the GHR (ELECTRA) models with from 1 to 7 of GHA layers. We can observe that the performance is improved when introducing the GHA layers, which verifies the effectiveness of our proposed approach. Moreover, GHR (ELECTRA) achieves the best performance when the number of GHA layers is 3. We also notice that when the number of layers is 1-4, the performance gaps between the models are not large, and all of them are close to the best performance. When the number of layers is greater than 4, the model performance begins to decline. We believe the reason is that the average round number of QuAC is 7, so a deeper structure may lead to overfitting.
Finally, we analyze the different results of the GHR model with its corresponding baseline, to directly observe the improvement brought by the global history attention mechanism (See Appendix A.3 for details).
In this paper, we propose a global history reasoning (GHR) approach to capture interactions between all utterances for conversational question answering. Experimental results conducted on QuAC show the effectiveness of the proposed model. In the future, we will explore how to implement interactions between conversation history and context by the position features of history answers. In addition, we will extend our conversation history modeling approach to the knowledge-grounded conversation generation task (see more related work in Appendix A.1).
The research is supported by National Key R&D Program of China (2020YFB1313601), National Science Foundation of China (62076174) and Institute of Infocomm Research of A*STAR (CR-2021-001).
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020. Graphflow: Exploiting conversation flow with graph neural networks for conversational machine comprehension. In Proceedings of the Twenty-Ninth Inter-