SlowGuess/NAACL_2022_OCR huggingface dataset:
Yongwei Zhou $^{1}$ , Junwei Bao $^{2*}$ , Chaoqun Duan $^{2}$ , Haipeng Sun $^{2}$ , Jiahui Liang $^{2}$ , Yifan Wang $^{2}$ , Jing Zhao $^{2}$ , Youzheng Wu $^{2}$ , Xiaodong He $^{2}$ , Tiejun Zhao $^{1\dagger}$
$^{1}$ Harbin Institute of Technology  $^{2}$ JD AI Research ywzhou@hit-mtlab.net baojunwei@jd.com tjzhao@hit.edu.cn
Machine reading comprehension (MRC) that requires discrete reasoning involving symbolic operations, e.g., addition, sorting, and counting, is a challenging task. According to this nature, semantic parsing-based methods predict interpretable but complex logical forms. However, logical form generation is nontrivial and even a little perturbation in a logical form will lead to wrong answers. To alleviate this issue, multi-predictor-based methods are proposed to directly predict different types of answers and achieve improvements. However, they ignore the utilization of symbolic operations and encounter a lack of reasoning ability and interpretability. To inherit the advantages of these two types of methods, we propose OPERA, an operation-pivoted discrete reasoning framework, where lightweight symbolic operations (compared with logical forms) as neural modules are utilized to facilitate the reasoning ability and interpretability. Specifically, operations are first selected and then softly executed to simulate the answer reasoning procedure. Extensive experiments on both DROP<sup>1</sup> and RACENum datasets show the reasoning ability of OPERA. Moreover, further analysis verifies its interpretability.<sup>2</sup>
Machine reading comprehension (MRC) that requires discrete reasoning is a valuable and challenging task (Dua et al., 2019), especially involving symbolic operations such as addition, sorting, and counting. The examples in Table 1 illustrate the task. To answer the question "Who threw the longest pass", it requires a model to choose the person with the longest pass from all the people and
Passage: Houston would tie the game in the second quarter with kicker Kris Brown getting a 53-yard and a 24-yard field goal. Oakland would take the lead in the third quarter with wide receiver Johnnie Lee Higgins catching a 29-yard touchdown pass from Russell, followed up by an 80-yard punt return...
Question: Who threw the longest pass?
Answer: Russell
Table 1: An example of question-answer pair along with a passage from DROP dataset (Dua et al., 2019). Question words in color indicate the potential operations for reasoning, i.e., ARGMAX and KEY_VALUE.
corresponding distance pairs based on the given passage. This task has various application scenarios in the real world, such as analyzing financial reports and sports news.
Existing approaches for this task can be roughly divided into two categories: the semantic-parsing-based (Chen et al., 2020b; Gupta et al., 2020) and the multi-predictor-based methods (Dua et al., 2019; Ran et al., 2019; Hu et al., 2019; Chen et al., 2020a; Zhou et al., 2021). The former maps the natural language utterances into logical forms and then executes them to derive the answers. For example, Chen et al. (2020b) propose NeRd. It includes a reader to encode the passage and question, and a programmer to generate a logical form for multi-step reasoning. Intuitively, this method has an advantage in interpretability. However, semantic parsing over text is nontrivial and even a little perturbation will result in wrong answers, which hinders the MRC performance.
To alleviate the heavy dependence on logical forms in the first category, the latter directly employs multiple predictors to derive different types of answers. For example, Dua et al. (2019) and Chen et al. (2020a) divide instances of the DROP dataset into several types and design a model with multi-predictors to deal with different answer types, i.e., question/passage span(s), count, and arithmetic
*Work was done during the first author's internship at JD AI mentored by Junwei Bao: baojunwei001@gmail.com.
1https://leaderboard.allenai.org/drop/submissions/public
$^{2}$ Codes is released at https://github.com/JD-AI-Research-NLP/OPERA.
Human Language Technologies, pages 1655 - 1666
expression. It is the capability of deriving different types of answers that improves the performance of models. However, such methods are lack of the necessary components to imitate discrete reasoning, which leads to inadequacy in reasoning ability and interpretability.
To alleviate the shortcomings of the above methods and preserve their advantages, we attempt to summarize reasoning steps into a set of operations and adopt them as the pivot to connect the question and the answer, which makes it possible to perform discrete reasoning. For example, to answer the question in Table 1, it needs two steps: (1) finding all persons and the corresponding distance of touchdown pass, and (2) choosing the one with the longest pass among them. We attempt to convert them into two operations, KEY_VALUE and ARGMAX, respectively. We then use them to produce the answer. Specifically, we design a set of lightweight symbolic operations (compared with logical forms) to cover all of the questions in the datasets and utilize them as neural modules to facilitate reasoning ability and interpretability. We denote this method as OPERA, an operation-pivoted discrete reasoning MRC framework. To utilize the operations, we propose an operation-pivoted reasoning mechanism composed of an operation selector and an operation executor. Specifically, the operation selector automatically identifies relevant operations based on the input. To enhance the performance of this sub-mechanism, we further design an auxiliary task to learn the alignment from a question to operations according to a set of heuristics rules. The operation executor softly integrates the selected operations to perform discrete reasoning over text via an attention mechanism (Vaswani et al., 2017).
To verify the effectiveness of the proposed method, comprehensive experiments are conducted on both the DROP and RACENUM datasets, where RACENUM used in this paper is a subset of the RACE dataset following (Chen et al., 2020a). Experimental results indicate that our method outperforms strong baselines and achieves the state-of-the-art on both datasets under the single model setting. We further analyze the interpretability of OPERA. Overall, this paper primarily makes the following contributions:
(1) We propose OPERA, an operation-pivoted discrete reasoning MRC framework, improving both the reasoning ability and interpretability.
(2) Extensive experiments on DROP and RACENUM dataset demonstrate the reasoning ability of OPERA. Moreover, statistic analysis and visualization indicate the interpretability of OPERA.
(3) We systematically design operations and heuristic rules to map questions to operations, aiming to facilitate research on symbolic reasoning.
Recently, machine reading comprehension (MRC) methods tend to deal with more practical problems (Yang et al., 2018; Dua et al., 2019; Zhao et al., 2021), for example, answering complex questions that require discrete reasoning (Dua et al., 2019) such as arithmetic computing, sorting, and counting. Intuitively, semantic parsing-based methods, which are well explored to deal with discrete reasoning in question answering with structured knowledge graphs (Bao et al., 2016) or tables, have potential to address the discrete reasoning MRC problem. Therefore, semantic parsing-based methods for discrete reasoning over text are proposed to firstly convert the unstructured text into a table, and then answer questions over the structured table with a grammar-constrained semantic parser (Krishnamurthy et al., 2017). NeRd (Chen et al., 2020b) is a generative model that consists of a reader and a programmer, which are responsible for encoding the context into vector representation and generating grammar-constrained logical forms, respectively. NMNs (Gupta et al., 2020) learned to parse compositional questions as executable logical forms. However, it only adapts to limited question types matched with a few pre-defined templates.
Multi-predictor-based methods employ multiple predictors to derive different types of answers. NAQANET (Dua et al., 2019), a number-aware framework, employed multiple predictors to produce corresponding answer types, including a span, count, and arithmetic expression. Based on NAQANET, MTMSN(Hu et al., 2019) added a negation predictor to solve the negative question and re-ranked arithmetic expression candidates. To aggregate the relative magnitude relation between two numbers, NumNet (Ran et al., 2019) and NumNet+ leveraged a graph convolution network to perform multi-step reasoning over a number graph. QDGAT (Chen et al., 2020a) proposed a question-directed graph attention network for reasoning over a heterogeneous graph composed of entity and number nodes. EviDR (Zhou et al., 2021), an evidence
Operations	Description	Examples
ADDITION/DIFF	Addition or subtraction	How many more yards was Kris Browns's first field goal over his second?
MAX/MIN	Select the maximum/minimum one from given numbers	How many yards was the longest field goal in the game?
ARGMAX/ARGMIN	Select key with highest/lowest value from key-value pairs	Which player had the longest touchdown reception?
ARGMORE/ARGLESS	Select key with higher/lower value from two key-value pairs	Who scored more field goals, David Akers or John Potter?
COUNT	Count the number of spans	How many field goals did Kris Brown kick?
KEY_VALUE	Extract key-value pairs	How many percent of Forth Worth households owned a car?
SPAN	Select spans from input sequence	Which team scored the final TD of the game?
Table 2: All the operations, descriptions and the corresponding examples.
emphasized MRC method, performed reasoning over a multi-grained evidence graph. Compared with these existing methods, our proposed OPERA focuses on bridging the gap from questions to answers with operations and integrating them to simulate discrete reasoning.
3.1 Task and Model Overview
Given a question  $Q$  and a passage  $P$ , MRC that requires discrete reasoning aims to predict an answer  $\hat{A}$  with the maximum probability over the candidate space  $\Omega$  as follows:
$$
\hat {A} = \underset {A \in \Omega} {\arg \max } p (A | Q, P) \tag {1}
$$
where the answer  $\hat{A}$  in this task could not only be span(s) extracted from context but also a number calculated with some numbers in context. To handle this task, it generally requires not only natural language understanding but also performing discrete reasoning over text, such as comparison, sorting and arithmetic computing.
To address the aforementioned challenges in this task, we propose OPERA, an operation-pivoted discrete reasoning MRC framework and it is briefly illustrated in Figure 1. In our framework, a set of operations  $\mathcal{OP}$ , defined in Table 2, are introduced to support the modeling of answer probability  $p(A|Q,P)$  as follows:
$$
p (A | Q, P) = \sum_ {O \in \mathcal {O P}} p (A | Q, P, O) p (O | Q, P), \tag {2}
$$
where  $O \in \mathcal{OP}$  represents one of the operations. Concretely, in our framework, we first design an operation selector  $p(O|P, Q)$  for choosing the correct question-related operations. These selected operations are then softly executed over the given context. Eventually, answer predictor  $p(A|Q, P, O)$  utilizes the execution result to predict the final answer.
3.2 Definition of Operations
To imitate discrete reasoning, we design a set of operations  $\mathcal{OP}$  as shown in Table 2. The set contains 11 operations and each one represents a reasoning unit. Specifically, for questions that need to be answered by calculation, we design operations ADDITION/DIFF to represent addition and subtraction. For questions which need to be answered by counting or sorting, we also design operations COUNT, MAX/MIN, ARGMAX/ARGMIN, and ARGMORE/ARGLESS. The rest operations KEY_VALUE and SPAN are used to extract spans from the question and the passage. To incorporate them into OPERA, each operation is denoted as a tuple. Formally,  $i$ -th operation  $\mathcal{OP}_i$  is  $\langle \mathbf{E}_i^{OP}, f_i(\cdot) \rangle$ , where  $i \in \{1, 2, \dots, n\}$  and  $n$  is the numbers of operations.  $\mathbf{E}_i^{OP} \in \mathbb{R}^{d_h}$  represents the learnable embedding of the  $i$ -th operation.  $f_i(\cdot)$  is a neural executor parameterized with trainable matrices  $\mathbf{W}_{q,i}^{OP}$ ,  $\mathbf{W}_{k,i}^{OP}$  and  $\mathbf{W}_{v,i}^{OP} \in \mathbb{R}^{d_h \times d_h}$ . The neural executor  $f_i(\cdot)$  is capable of performing execution of  $\mathcal{OP}_i$  on the given context. Specifically, it takes the representation of context as input and outputs the executed representation as  $\mathbf{m}_i^{OP}$  (§ 3.3.2).
3.3 Architecture of OPERA
3.3.1 Context Encoder
The context encoder aims to learn the contextual representation of the input. Formally, given a question  $Q$  and a passage  $P$ , we concatenate them into a sequence and feed it into a pre-trained language model (Liu et al., 2019; Clark et al., 2020; Lan et al., 2020) to obtain their whole representation  $\mathbf{H} \in \mathbb{R}^{l \times d_h}$ . After that, we split  $\mathbf{H}$  into the question and passage representations, which are respectively denoted as  $\mathbf{H}^Q \in \mathbb{R}^{l_q \times d_h}$  and  $\mathbf{H}^P \in \mathbb{R}^{l_p \times d_h}$ .  $l_q$ ,  $l_p$ , and  $l$  are the number of tokens in question, passage and concatenation of them.  $d_h$  is the dimension of the representations.
Figure 1: The architecture of OPERA. It consists of a context encoder, an operation-pivoted reasoning module, and a prediction module. The prediction module supports five types of answers, including question span, passage span, arithmetic expression, count, and multi-spans. MHA means a multi-head attention mechanism.
3.3.2 Operation-pivoted Discrete Reasoning
The operation-pivoted reasoning module is composed of an operation selector and an operation executor. The operation selector is adopted to select operations related to the given question. The operation executor is responsible for imitating the execution of the selected operations with an attention mechanism.
Operation Selector To imitate discrete reasoning, existing methods usually adopt a logical form generated by a semantic parser to address this task. However, these methods suffer severely from the cascade error, where a little perturbation in the logical form may result in wrong answers. Therefore, we propose to map each question into an operation set, instead of logical forms. Namely, we intend to select relevant operations from the  $\mathcal{OP}$ . To this end, we adopt a bilinear function to compute the similarity between each operation and the question and normalize them with a softmax as follows:
$$
p (O | Q, P) = \operatorname {s o f t m a x} (\mathbf {E} ^ {O P} \mathbf {W h} ^ {Q}), \quad (3)
$$
where  $\mathbf{E}^{OP}\in \mathbb{R}^{n\times d_h}$  is a learnable parameter, which denotes the operation embedding matrix.  $\mathbf{h}^Q\in \mathbb{R}^{d_h}$  is the representation of the question, which is obtained by executing weighted pooling on the  $\mathbf{H}^Q$  .  $\mathbf{W}\in \mathbb{R}^{d_h\times d_h}$  is a parameter matrix and  $p(O|Q,P)$  is the distribution over operations.
Operation Executor The operation executor is responsible for performing the execution of the selected operations over the given context. Inspired by previous studies (Andreas et al., 2016; Gupta et al., 2020), we implement the operation executor
based on the neural module network, which takes advantage of neural network in fitting and generalization, and the composition characteristics of symbolic processing. Specifically, for each operation  $\mathcal{OP}_i = \langle \mathbf{E}_i^{OP},f_i(\cdot)\rangle$ ,  $i = \{1,2,\dots,n\}$ , we use a multi-head cross attention mechanism (Vaswani et al., 2017) to implement  $f_{i}(\cdot)$ . In detail, we leverage the embedding of each operation  $\mathbf{E}_i^{OP}$  as query and the representations of the whole input sequence  $\mathbf{H}$  as keys and values, respectively, to model  $f_{i}(\cdot)$  as follows:
$$
\alpha_ {i} ^ {O P} = \operatorname {s o f t m a x} \left(\frac {\left(\mathbf {E} _ {i} ^ {O P} \mathbf {W} _ {q , i} ^ {O P}\right) \left(\mathbf {H} \mathbf {W} _ {k , i} ^ {O P}\right) ^ {T}}{\sqrt {d _ {h}}}\right), \tag {4}
$$
$$
\mathbf {m} _ {i} ^ {O P} = \alpha_ {i} ^ {O P} (\mathbf {H W} _ {v, i} ^ {O P}), \tag {5}
$$
where  $\mathbf{W}_{q,i}^{OP},\mathbf{W}_{k,i}^{OP},\mathbf{W}_{v,i}^{OP}\in \mathbb{R}^{d_h\times d_h}$  are the parameter matrices in executor of operation  $\mathcal{O}\mathcal{P}_i$ $\mathbf{m}_i^{OP}\in \mathbb{R}^{d_h}$  is the representation of the execution result of the  $i$  -th operation.
Finally, we softly integrate all of the execution results as the final output  $\mathbf{h}^{OP}\in \mathbb{R}^{d_h}$  with the distribution  $p(O|Q,P)$  as follows:
$$
\mathbf {h} ^ {O P} = \sum_ {i = 1} ^ {n} p (O = \mathcal {O P} _ {i} | Q, P) \mathbf {m} _ {i} ^ {O P}. \tag {6}
$$
The operation-aware semantic representation  $\mathbf{h}^{OP}$  is further fed into the prediction module to reason the final answer (§ 3.3.3).
As described above, OPERA introduces operations that assist in understanding questions and integrates them into the model to perform discrete reasoning. Therefore, it achieves an advantage in
the reasoning capability and interpretability over the previous multi-predictor-based methods (Hu et al., 2019; Chen et al., 2020a; Zhou et al., 2021). Moreover, soft execution and composition of operations in OPERA alleviate the cascaded error that the semantic parsing methods (Ran et al., 2019; Chen et al., 2020b) suffer from. More experiments and analyses about reasoning ability and interpretability are illustrated in § 4.4 and § 4.5.
3.3.3 Prediction Module
In this section, we introduce the prediction module to derive different types of answers via multi-predictors. Each predictor first reasons out a derivation and then performs execution to obtain the final answer. This answer prediction procedure is formalized as follows:
$$
p (A | Q, P, O) = \sum_ {D \in \mathcal {D}} \mathbb {I} (g (D) = A) p (D | Q, P, O), \tag {7}
$$
where  $\mathbb{I}(g(D) = A)$  is an indicator function with value 1 if the answer  $A$  can be derived from a derivation executor  $g(\cdot)$  based on  $D$ , and 0 otherwise.  $p(D|Q,P,O)$  models the derivation prediction. Specifically, a derivation  $D = \langle T,L\rangle$  includes an answer type  $T$  and a corresponding label  $L$ . For example, in Table 3, the textual answer  $A$  of the question "how many yards was the longest field goals in the game" is "80". The possible derivations  $\mathcal{D}$  to this answer include a span  $\langle \mathrm{Span},(100,102)\rangle$ , and an arithmetic expression  $\langle \mathrm{AE},(0*29) + (1*80)\rangle$ . Inspired by previous studies (Chen et al., 2020a; Zhou et al., 2021), the derivation predictor
$$
p (D | Q, P, O) = \sum_ {T \in \mathcal {T}} p _ {T} (L | Q, P, O) p (T | Q, P, O) \tag {8}
$$
is decomposed into an answer type predictor  $p(T|Q, P, O)$  and corresponding label predictors  $p_{T \in \mathcal{T}}(L|Q, P, O)$  where  $\mathcal{T} = \{ \text{Question Span}, \text{Passage Span}, \text{Count}, \text{Arithmetic Expression}, \text{Multi-spans} \}$  includes all the answer types defined in this paper. Each label predictor takes question-passage representation  $\mathbf{H}$  and the operation-pivo representation  $\mathbf{h}^{OP}$  as input and calculates the probability of label  $L$ . Specifically, these label predictors are specified as follows and more details are shown in Appendix A.3.
Question / Passage Span The probability of a question/passage span is the product of the probabilities of the start index and the end index. Following
MTMSN (Hu et al., 2019), we use a question-aware decoding strategy to predict the start and end index across the input sequence, respectively.
Count As indicated in QDGAT (Chen et al., 2020a), questions with 0-9 as answers account for  $97\%$  in all the count questions. Hence, such questions are modeled as a 10-class (0-9) classification problem.
Arithmetic Expression Similar to NAQANet (Dua et al., 2019), we first assign a sign (positive, negative, or zero) to each number in the context and then compute the answer by summing them.
Multi-spans Inspired by Segal et al. (2020), the multi-span answer (a set of non-contiguous spans) is derived with a sequence labeling method, in which each token of the input is tagged with BIO labels. Finally, each span which is tagged with continuous B and I is taken as a candidate span.
3.4 Learning with Weak Supervision
3.4.1 Training Instance Construction
Each training instance is originally composed of a passage  $P$ , a question  $Q$ , and answer text  $A$ . Since the derivations (i.e., labels for the spans, arithmetic expressions, and count) are not provided, weak supervision is adopted in OPERA. Specifically, for each training instance, given the golden textual answer  $A$ , we heuristically search all the possible derivations  $\mathcal{D}$  as supervision signals, each of which can derive the correct answer  $A$ . Table 3 shows an example of  $\mathcal{D}$ .
In addition, we propose heuristic rules to map a question to its related operations denoted as  $\mathcal{O} \subseteq \mathcal{OP}$ . For example, to detect the operations intimated by the question  $Q$  in Table 3, we design a question template "how many yards [Slot] longest [Slot]" which maps matched questions to the operation MAX. Overall, a training instance can be constructed as a tuple  $\langle P, Q, A, \mathcal{O}, \mathcal{D} \rangle$ . The one-shot heuristic rules to obtain operation labels reduce the cost of human annotations. Moreover, when applying OPERA to other discrete reasoning MRC tasks, both the operations  $\mathcal{OP}$  and the heuristic rules can be extended and adjusted if necessary. Fortunately, there is no need to construct strict logical forms in our architecture, but only the set of lightweight operations involved in the question. It tremendously reduces the difficulty of adapting OPERA to other discrete reasoning MRC tasks.
Meanwhile, we analyze the distribution of operations in the training set. More details about the
P	...Oakland would take the lead in the third quarter with wide receiver Johnnie Lee Higgins catching a 29-yard touchdown pass from Russell, followed up by an 80-yard punt return for a touchdown ...
Q	How many yards was the longest field goals
A	80
O	MAX ⇌ How many yards [Slot] longest [Slot]
D	<Span, (100, 102)>; <AE, (0 * 29) + (1 * 80)
Table 3: An example of building training instances.
heuristic rules for mapping questions to operations and the operation distribution in the dataset are respectively given in the Appendix A.1 and A.2.
3.4.2 Joint Training
The training objective consists of two parts, including the loss for answer prediction and operation selection. The loss for answer prediction  $\mathcal{L}_a$  is
$$
\mathcal {L} _ {a} = - \log p (A | Q, P). \tag {9}
$$
Note that the calculation of loss  $\mathcal{L}_a$  takes all possible derivations that can obtain the correct answer  $A$  into account, which means that OPERA does not require labeling answer types for training. In addition, to learn better alignment from a question to operations, we introduce auxiliary supervision for the operation selector and calculate the loss
$$
\mathcal {L} _ {o p} = - \sum_ {O \in \mathcal {O}} \log p (O | Q, P), \tag {10}
$$
where  $\mathcal{O}$  indicates the operations provided by the heuristic rules. Finally, OPERA is optimized by minimizing the loss  $\mathcal{L} = \mathcal{L}_a + \lambda \mathcal{L}_{op}$  where  $\lambda$  is a hyperparameter as a trade-off of the two objectives.
We conduct experiments on the following two MRC datasets to examine the discrete reasoning capability of our model. We employ Exact Match (EM) and F1 score as the evaluation metrics.