SlowGuess/NAACL_2022_OCR huggingface dataset:
Hao Huang $^{1*}$ , Xiubo Geng $^{2\dagger}$ , Guodong Long $^{1}$ , Daxin Jiang $^{2\dagger}$
<sup>1</sup>Australian AI Institute, School of CS, FEIT, University of Technology Sydney
$^{2}$ STCA NLP Group, Microsoft
{hao.huang-4,guodong.long}@{student.uts,uts}.edu.au
{xigeng,djiang}@microsoft.com
This work studies temporal reading comprehension (TRC), which reads a free-text passage and answers temporal ordering questions. Precise question understanding is critical for temporal reading comprehension. For example, the question "What happened before the victory" and "What happened after the victory" share almost all words except one, while their answers are totally different. Moreover, even if two questions query about similar temporal relations, different varieties might also lead to various answers. For example, although both the question "What usually happened during the press release?" and "What might happen during the press release" query events which happen after the press release, they convey divergent semantics. To this end, we propose a novel reading comprehension approach with precise question understanding. Specifically, a temporal ordering question is embedded into two vectors to capture the referred event and the temporal relation. Then we evaluate the temporal relation between candidate events and the referred event based on that. Such fine-grained representations offer two benefits. First, it enables a better understanding of the question by focusing on different elements of a question. Second, it provides good interpretability when evaluating temporal relations. Furthermore, we also harness an auxiliary contrastive loss for representation learning of temporal relations, which aims to distinguish relations with subtle but critical changes. The proposed approach outperforms strong baselines and achieves state-of-the-art performance on the TORQUE dataset. It also increases the accuracy of four pre-trained language models (BERT base, BERT large, RoBERTa base, and RoBETRa large), demonstrating its generic effectiveness on divergent models.
Paragraph 1: The European Union and the United States have frozen aid to the Palestinian Authority ever since the Hamas-led government took power in March, two months after its upset parliamentary election victory. Abbas's Fatah faction and Hamas labored for months to reach a power sharing agreement that would meet international conditions to lift the siege and end the spiraling crisis, but those talks failed late last month.
Q1: What happened after the victory? (Basic)
A1: frozen took labored failed (Basic)
Q2: What failed to happen after the victory? (Negated)
A2: agreement meet lift end (Negated)
Q3: What happened right after the victory? A3: frozen took labored (Constrained)
Paragraph 2: "The agreement, which was signed by the two governments Thursday, demonstrates Nepal's continuing commitment to development activities in agriculture and forestry," it said in a press release. "These two sectors provide income and employment opportunities for almost 80 percent of Nepal's population," it added.
Q4: What usually happened during the press release? (Common)
A4: said (Common)
Q5: What might happen during the press release? (Uncertain)
A5: said added (Uncertain)
Figure 1: Examples of temporal reading comprehension. Temporal relations are diverse: Q1-Q5 list examples of possible varieties of temporal relations. Small changes in the question might lead to substantially divergent semantics: replacing usually in Q4 with might in Q5 leads to different answers. Related events are underlined in the passage.
Understanding temporal relationships between events in a passage is essential for natural language understanding (Wang et al., 2019; Dong et al., 2019). Temporal reading comprehension (TRC) (Ning et al., 2020) is a natural way to study temporal relations since natural language questions are flexible to capture divergent temporal relations (Zhou et al., 2021). Figure 1 shows several examples of temporal reading comprehension, where given a free-text passage, a system is required to answer temporal questions like "What usually happened during the press release?"
A natural solution for temporal ordering understanding is to compare each candidate answer and the referred event in the question and classify their temporal relation into several pre-defined cate
*Work is done during internship at Microsoft.
Human Language Technologies, pages 385 - 394
gories, e.g., UzZaman et al. (2013) defines 13 possible relations such as after, ends, equal to. Nonetheless, since temporal relationships vary greatly, it is almost impossible to enumerate all possible relationships. Figure 1 shows several divergent varieties of temporal relations: one might query about plain after in Q1, negated after in Q2, constrained after in Q3, etc. Similarly, a question might query about usually happen in Q4, might happen, or other relations. Moreover, creating sufficient labels for all such relations is costly and poses great challenges for real-world applications. Therefore, the classification-based approach is incompetent to handle the flexible relations in temporal reading comprehension.
Another paradigm is to formulate it as a reading comprehension problem and directly predict the answer to a question. With the help of large pre-trained language models (e.g., BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019)), such approaches have achieved relatively good performance. However, they still struggle for the temporal reading comprehension task due to the lack of precise question understanding. For example, given the same passage, the BERT model fine-tuned on SQuAD (Rajpurkar et al., 2016) predicts the same answer to the two questions (Ning et al., 2020), "What happened before a woman was trapped" and "What happened after a woman was trapped". In this case, although the two questions share almost the same words, the only different one between before and after leads to completely opposite intentions. Moreover, even if two questions query about similar relations, different varieties might also lead to various answers. Take the question Q4 "What usually happened during the press release?" and "What might happen during the press release?" in Figure 1 as an example. Although they both query about events occurring after the press release, the slight difference conveys divergent semantics and leads to different answers.
To tackle these challenges, we propose a novel question answering approach with precise question understanding. Intuitively, temporal ordering questions consist of two elements, referred events, and concerned temporal relations. For example, the question "What usually happened during the press release?" can be decomposed into the referred event the press release and the concerned relation usually happen during. Inspired by this observation, we first encode such questions into two
representations, the event vector  $\mathbf{h}_{\mathbf{c}}$  and the relation vector  $\mathbf{h}_{\mathbf{r}}$ . Then we evaluate how well each candidate answer matches the relation  $\mathbf{h}_{\mathbf{r}}$  compared to  $\mathbf{h}_{\mathbf{c}}$  with a separate MLP module. Such fine-grained representations enable a better understanding of questions by focusing on different elements with different vectors and further provides good interpretability about the reasoning process. More importantly, it empowers the model to capture the semantics of divergent variants of temporal relations. Specifically, we harness an auxiliary contrastive loss that aims to distinguish relations with subtle but critical changes.
We evaluate the proposed approach on the TORQUE dataset and achieve state-of-the-art performance compared to strong baselines. We further testify its effectiveness based on four different models (i.e., BERT base, BERT large, RoBERTa base, RoBERTa large) and demonstrate that precise question understanding can improve the QA accuracy for all models. Ablation study shows that both question representation learning and contrastive loss play a critical role in the approach.
Temporal machine comprehension is closely related to two areas of works, i.e., machine reading comprehension and temporal ordering reasoning.
2.1 Machine Reading Comprehension
Machine reading comprehension (MRC) (Rajpurkar et al., 2016, 2018) has attracted much attention in recent years. Traditional solutions to MRC tasks focus on utilizing the interaction information between questions and passages via attention-based structures (Kadlec et al., 2016; Dhingra et al., 2017). Later on, pre-trained language models (PLMs), e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019), have been widely used for MRC tasks. With the sheer scale of parameters and the pretraining strategies, PLMs capture more knowledge from the context and have shown outstanding performance on traditional MRC benchmarks.
For more challenging MRC tasks which introduce multi-hop reasoning (Yang et al., 2018), numerical reasoning (Dua et al., 2019), etc., the generic PLMs become not applicable. Recent efforts use graph-based reasoning approaches (Chen et al., 2020) or define specific pretraining training techniques (Raffel et al., 2020) to solve the above challenges. However, existing MRC approaches
still struggle for the temporal reading comprehension task due to the lack of temporal relation understanding (Ning et al., 2020). Hence, we propose a novel question answering approach with precise question understanding to tackle this challenge.
2.2 Temporal Ordering Reasoning
Traditional temporal order reasoning tasks (UzZaman et al., 2013; Cassidy et al., 2014; Ning et al., 2018), are often formulated as relation extraction tasks. Given the context passage, the target is to classify the relation between every two events from a predefined relation set, e.g., UzZaman et al. (2013) defines 13 possible relations such as after, ends, equal to. Existing solutions can be roughly classified into two categories. The first category focuses on developing the structure of the encoder to capture more temporal information. For example, Cheng et al. (2020) add up a GRU-based dynamically updating structure upon the outputs of the common BERT sentence encoder. The second category focuses on joint learning with external knowledge or some specific constraints. For instance, Ning et al. (2019) significantly improve the extraction performance by joint training temporal and causal relations.
However, the success of the existing approaches is limited to the formulation of the traditional temporal order reasoning tasks, where the events and the candidate temporal relation set are fixed. However, the fixed candidate relation set cannot cover all temporal relations in our daily uses. The most recent released dataset, TORQUE (Ning et al., 2020), formulates temporal ordering reasoning as a machine reading comprehension task. Given a context passage, we need to answer a free-text question about the temporal relations in the context passage. The task is much analogous to our real-world tasks and is more challenging – we need to automatically identify the events and the relations in the free-text question to retrieve the answers from the context passage. To the best of our knowledge, we are the very first to address this challenge.
We first introduce the definition of temporal reading comprehension (TRC) and then describe the model architecture consisting of contextual encoder, question understanding, and event relation assessment. Finally, we provide details for the learning and inference process.
The Temporal Reading Comprehension (TRC) task is defined as follows. Given a passage  $P$  which describes a set of events, a system is required to answer a temporal ordering question  $Q$ . Here events refer to verbs or nouns which define actions or states. A temporal ordering question usually queries events satisfying some concerned temporal relations considering one or more referred events. For example, the first passage in Figure 1 describes events about Hamas government, and question  $Q1$  queries which events have the temporal relation happen after with the referred event the victory. The answer set  $A$  to a question  $Q$  could be empty when no events meet the requirement.
Figure 2: An overview of the proposed model.
Figure 2 depicts the proposed model architecture. Specifically, the passage  $P$  and question  $Q$  are first encoded by a contextual-aware encoder, after which the representations of the question are passed to a question understanding module. Finally, each candidate answer is evaluated considering whether it satisfies the concerned relation to the referred event by an event relation assessment module.
Contextual Encoder We first encode the passage-question pairs with a pre-trained language model encoder, and here we take BERT as an example. Given a question  $Q = [q_i]_{i=1}^m$  and a passage  $P = [p_i]_{i=1}^n$ , where  $m$  and  $n$  are token numbers, we concatenate them into a sequence with the format of [cls] question [sep] passage [sep], which is then fed into the contextual encoder to generate the embeddings,
$$
\begin{array}{l} \left[ \mathbf {h} _ {\mathbf {1}} ^ {\mathbf {q}}, \dots , \mathbf {h} _ {\mathbf {m}} ^ {\mathbf {q}}, \mathbf {h} _ {\mathbf {1}} ^ {\mathbf {p}}, \dots , \mathbf {h} _ {\mathbf {n}} ^ {\mathbf {p}} \right] = \\ \operatorname {B E R T} ([ q _ {1}, \dots , q _ {m}, p _ {1}, \dots , p _ {n} ]), \\ \end{array}
$$
where  $\mathbf{h}_{\mathbf{i}}^{\mathbf{q}},\mathbf{h}_{\mathbf{i}}^{\mathbf{p}}\in \mathcal{R}^{d}$  are embeddings for question token  $q_{i}$  and passage token  $p_i$  , and  $d$  is the embedding size.
Figure 3: The structure of attention-based event/relation extractor, with attention loss for it.
Question Understanding As discussed in Section 1, precise question understanding plays an essential role in TRC task. Therefore, we propose a question understanding module to achieve that. Intuitively, a temporal ordering question consists of two elements, referred events, and concerned temporal relation. For example, the question "What usually happened during the press release" queries the temporal relation usually happen to the event the press release. A straightforward solution is to decompose the question into two segments directly. However, natural language questions vary a lot, and hard decomposition is risky and might propagate errors to downstream modules, which is verified by experimental analysis in Section 4.5,
Therefore, we design an attention-based extractor to decompose the question implicitly, and obtain two hidden representations,  $\mathbf{h}_{\mathrm{c}}$  for the referred event and  $\mathbf{h}_{\mathrm{r}}$  for the concerned temporal relation as follows,
$$
\mathbf {s} _ {\mathbf {i}} ^ {(\mathbf {z})} = \tanh  (\mathbf {W} _ {\mathbf {1}} ^ {(\mathbf {z})} \mathbf {h} _ {\mathbf {i}} ^ {\mathbf {q}} + b _ {1} ^ {(z)}), \quad z \in \{c, r \} \quad (2)
$$
$$
\alpha_ {i} ^ {(z)} = \operatorname {S o f t m a x} \left(\mathbf {W} _ {\mathbf {2}} ^ {(\mathbf {z})} \mathbf {s} _ {\mathbf {i}} ^ {(\mathbf {z})} + b _ {2} ^ {(z)}\right), \quad z \in \{c, r \} \tag {3}
$$
$$
\mathbf {h} _ {\mathbf {z}} = \sum_ {i = 1} ^ {m} \alpha_ {i} ^ {(z)} \mathbf {h} _ {\mathbf {i}} ^ {\mathbf {q}}, \quad z \in \{c, r \} \tag {4}
$$
where  $\mathbf{W}^{(\mathbf{c})},\mathbf{W}^{(\mathbf{r})}\in \mathcal{R}^d$  , and  $b^{(c)},b^{(r)}\in \mathcal{R}$  are learn-able weights for the extractor,  $\mathbf{h}_{\mathbf{i}}^{\mathbf{q}}\in \mathcal{R}^{d}$  is the embedding for the  $i$  -th question token. To effectively learn  $\mathbf{h}_{\mathbf{r}}$  and  $\mathbf{h}_{\mathbf{c}}$  , we employ several auxiliary losses in the training phase, which will be described in section 3.3.
Figure 4: The structure of the event relation assessment, with answer prediction loss for it.
Event Relation Assessment Given the question representations  $\mathbf{h}_{\mathbf{r}}$  and  $\mathbf{h_c}$ , the event relation assessment module evaluates how a candidate answer satisfy the relation  $\mathbf{h_r}$  with respect to  $\mathbf{h_c}$ . Let  $e = p_i\dots p_{i + l}$  denotes the candidate answer, which consists of  $l$  tokens in the passage  $P$ . We first get the representation of  $e$  by pooling over according token vectors,
$$
\mathbf {h} _ {\mathbf {e}} = \operatorname {P o o l} \left(\mathbf {h} _ {\mathbf {i}} ^ {\mathbf {p}}, \dots , \mathbf {h} _ {\mathbf {i + 1}} ^ {\mathbf {p}}\right). \tag {5}
$$
Then we concatenate the representations of the candidate event  $\mathbf{h}_{\mathrm{e}}$ , question relation  $\mathbf{h}_{\mathbf{r}}$ , and the question event  $\mathbf{h}_{\mathbf{c}}$ , and feed it into a two-layer MLP, followed by a softmax function to get the final probability,
$$
\mathbf {o _ {e}} = \tanh (\mathbf {W _ {1} ^ {o} [ h _ {e} ; h _ {c} ; h _ {r} ] + b _ {1} ^ {o}}), \qquad (6)
$$
$$
\mathbf {p} _ {\mathbf {e}} = \operatorname {S o f t m a x} \left(\mathbf {W} _ {2} ^ {\mathbf {o}} \mathbf {o} _ {\mathbf {e}} + \mathbf {b} _ {2} ^ {\mathbf {o}}\right), \tag {7}
$$
where  $\mathbf{W}_1^0\in \mathcal{R}^{3d\times d'}$ $\mathbf{W}_2^0\in \mathcal{R}^{d'\times 2}$ $\mathbf{b}_1^0\in \mathcal{R}^{d'}$ $\mathbf{b}_2^0\in \mathcal{R}^2$  are model parameters, and ; indicates concatenation.  $\mathbf{p_e}\in \mathcal{R}^2$  is the probability whether the candidate  $e$  satisfies the temporal relation  $h_r$  with respect to event  $h_c$
3.3 Learning Objectives
We employ three learning objectives for model training, including a classification loss  $L_{qa}$  function for final answer prediction, and an attention loss  $L_{att}$  and a contrastive loss  $L_{con}$  for precise question understanding. The overall loss is a weighted combination of all the objectives,
$$
\mathcal {L} = w _ {q a} L _ {q a} + w _ {a t t} L _ {a t t} + w _ {c o n} L _ {c o n}. \tag {8}
$$
Answer Prediction Loss The training objective for final answer prediction is defined as,
$$
L _ {q a} = - \sum_ {e \in \mathcal {C}} w _ {e} \hat {\mathbf {p}} _ {\mathbf {e}} ^ {T} \log \mathbf {p} _ {\mathbf {e}}, \tag {9}
$$
where  $\mathcal{C}$  is the candidate event set,  $w_{e}$  is the weight for candidate  $e$ ,  $\mathbf{p}_{\mathbf{e}} \in \mathcal{R}^{2}$  is the predicted probability from Eq. (7), and  $\hat{\mathbf{p}}_{\mathbf{e}} \in \{0,1\}^{2}$  is the golden label indicating whether the candidate  $e$  belongs to the final answer of the question.
Usually, the candidate set  $\mathcal{C}$  is derived by preliminary filtering all unigrams in the passage  $P$ . However, some candidates are easy to be classified while others are not. For example, it is easy to classify the word government in Figure 1 as a negative answer since it is not an event. In contrast, predicting whether the word frozen is the answer for Q1 in Figure 1 is more challenging. Inspired by this observation, we assign weights  $w_{e}$  for candidates in the learning objective,  $w_{e} = 1.5$  if  $e$  is an event, and otherwise  $w_{e} = 1.0$ . The label of whether a word is an event can be derived when labeling the final answer with little effort, so we can safely assume that we always have such annotation<sup>1</sup>.
Attention Loss Besides the answer prediction loss, we also leverage an auxiliary loss to guide the learning of the attention score  $\alpha_{i}^{(c)}$  and  $\alpha_{i}^{(r)}$  defined in Eq. (3). We first derive silver annotation for referred events and concerned relation in a passage using a rule-based approach, which will be detailed in Section 4.2. Let  $Q_{c}, Q_{r}$  be the set of event and relation tokens according to the silver annotation. Then we have  $\hat{\alpha}_{i}^{(z)}(z \in \{c, r\})$  as the derived attention label,
$$
\hat {\alpha} _ {i} ^ {(z)} = \left\{ \begin{array}{l l} \frac {1}{| Q _ {z} |}, & \text {i f} q _ {i} \in Q _ {z}, \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {10}
$$
The attention loss is defined as,
$$
L _ {a t t} = L _ {c} + L _ {r}, \tag {11}
$$
$$
L _ {z} = - \sum_ {i} \hat {\alpha} _ {i} ^ {(z)} \log \alpha_ {i} ^ {(z)}, \quad z \in \{c, r \}. \tag {12}
$$
Contrastive Loss As shown in Figure 1, a small change of a question might lead to substantially divergent temporal relations. To this end, we propose
Figure 5: Illustration of the contrastive loss for question understanding.
to leverage a contrastive loss for precise learning of question relation representations.
For the relation representation  $\mathbf{h}_{\mathbf{r}}$  of a question  $Q$ , we derive a positive vector  $\mathbf{h}_{\mathbf{r}}^{\mathbf{p}}$  and a set of negative ones  $\{\mathbf{h}_{\mathbf{r},\mathbf{i}}^{\mathbf{n}}\}_{i = 1}^{N}$ . The positive sample  $\mathbf{h}_{\mathbf{r}}^{\mathbf{p}}$  is obtained in two ways. First, we search questions with the same temporal relations but different events, from which we randomly sample one and take its relation representation as  $\mathbf{h}_{\mathbf{r}}^{\mathbf{p}}$ . Note we can get the silver annotation of events and relations in a question by a rule-based approach. Please refer to section 4.2 for more details. Second, if no such questions can be found, we take the similar approach as in SimCSE (Gao et al., 2021), which applies a different dropout on  $\mathbf{h}_{\mathbf{r}}$  and gets a variant of  $\mathbf{h}_{\mathbf{r}}$  as  $\mathbf{h}_{\mathbf{r}}^{\mathbf{p}}$ . We search questions that contain the same events by different temporal relations with respect to  $Q$ , and take their relation representations as the negative set  $\{\mathbf{h}_{\mathbf{r},\mathbf{i}}^{\mathbf{n}}\}_{i = 1}^{N}$ .
Given the triple  $(\mathbf{h_r},\mathbf{h_r^p},\{\mathbf{h_r^n}\})$  for the question  $Q$ , its loss is defined as,
$$
\begin{array}{l} L _ {c o n} (Q) = \\ - \log \frac {e ^ {\cos \left(\mathbf {h} _ {\mathbf {r}} , \mathbf {h} _ {\mathbf {r}} ^ {\mathbf {p}}\right)}}{e ^ {\cos \left(\mathbf {h} _ {\mathbf {r}} , \mathbf {h} _ {\mathbf {r}} ^ {\mathbf {p}}\right)} + \frac {1}{N} \sum_ {i = 1} ^ {N} e ^ {\cos \left(\mathbf {h} _ {\mathbf {r}}, \mathbf {h} _ {\mathbf {r} , \mathbf {i}} ^ {\mathbf {n}}\right)}}, \tag {13} \\ \end{array}
$$
where  $\cos()$  indicates cosine similarity.
The inference phase takes three steps. First, we generate a candidate set  $\mathcal{C}_p$  for each passage  $P$ . Generally speaking, one can take any n-gram in  $P$  as a candidate. In temporal relation understanding, we usually take a triggering word as an event candidate. Therefore,  $\mathcal{C}_p$  is the set of all unigrams in  $P$ . Then, we filter  $\mathcal{C}_p$  according to part-of-speech (POS) tagging. Specifically, we use an off-the-shelf POS tagger to tag all words in  $P$ , and then keep only verbs and nouns in  $\mathcal{C}_p$ . Finally, each candidate  $e \in \mathcal{C}_p$
<sup>1</sup>The TORQUE dataset in our experiment also contains such annotation, and we use it directly in our approach
together with the passage  $P$  and the question  $Q$  is fed into our proposed model, and  $e$  is evaluated according to Eq. (7) and gets its score  $\mathbf{p_e}$ , where  $p_{e,0}$  represents the probability that the candidate matches the question  $Q$ . Then we can get the final answer set  $A$  as  $A = \{e : e \in \mathcal{C}_p \text{ and } p_{e,0} > \tau\}$ , where  $\tau$  is a predefined threshold.
This section describes an empirical evaluation of our proposed approach. We also provide analysis, ablation studies, and case analysis to demonstrate its effectiveness.
Dataset We evaluate the proposed approach on the TORQUE dataset. TORQUE is a temporal reading comprehension benchmark. Each training sample contains a passage and a question requiring understanding temporal relation between events in the passage. Figure 1 shows several examples of training data. The answer to a question consists of an event set  $A$ , and  $A$  could be empty if no event in the passage satisfies the requirement of the question. In TORQUE, events are defined as event triggers, usually verbs or nouns describing actions or states. There are 3.16k passages with 30.7k questions in total and 2 events for an answer on average. We follow the official split<sup>2</sup> with 80%/5%/15% of data in training/validation/test.