SlowGuess/NAACL_2022_OCR huggingface dataset:
Wanjun Zhong $^{1*}$ , Siyuan Wang $^{3*}$ , Duyu Tang $^{2}$ , Zenan Xu $^{1*}$ , Daya Guo $^{1*}$ , Yining Chen $^{2}$ , Jiahai Wang $^{1}$ , Jian Yin $^{1}$ , Ming Zhou $^{4}$  and Nan Duan $^{2}$
<sup>1</sup> The School of Data and Computer Science, Sun Yat-sen University.
$^{2}$  Microsoft Research  $^{3}$  Fudan University, China  $^{4}$ Langboat Technology
{zhongwj25, xuzn, guody5}@mail2.sysu.edu.cn
{wangjiah@mail,issjyin@mail}.sysu.edu.cn
{dutang,nanduan,yining.chen}@microsoft.com
wangsyl8@fudan.edu.cn; zhouming@chuangxin.com
Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to make conclusions. However, current neural models with implicit reasoning ability struggle to solve this task. In this paper, we study the challenge of analytical reasoning of text and collect a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016. We analyze what knowledge understanding and reasoning abilities are required to do well on this task, and present an approach dubbed ARM. It extracts knowledge such as participants and facts from the context. Such knowledge are applied to an inference engine to deduce legitimate solutions for drawing conclusions. In our experiments, we find that ubiquitous pretrained models struggle to deal with this task as their performance is close to random guess. Results show that ARM outperforms pre-trained models significantly. Moreover, we demonstrate that ARM has better explicit interpretable reasoning ability.
Transformer-based pre-trained language models including BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019) and RoBERTa (Liu et al., 2019) have achieved state-of-the-art performance on a variety of NLP tasks (Zhong et al., 2020b; Li et al., 2020; Sun et al., 2022; Li et al., 2022). However, they still struggle to perform deep reasoning beyond shallow-level semantic understanding of literal clues. For example, Talmor et al. (2020) show that pre-trained models fail completely on half of eight reasoning tasks that require symbolic operations. We hope to challenge current systems and take a step further towards analytical reasoning.
[Grouping Game] Passage:
Seven directors -A, B, C, D, E, F, and G- serves on
the X committee or the Y committee.
If A serves on X, then B serves on Y. R-1
If C serves on X, then D and E serve on Y. R-2
F serves on a different committee with G. R-3
E serves on a different committee with A. R-4
If G serves on X, so does B. R-5 Rules
If  $D$  and  $F$  both serve on the  $X$  committee, Fact
then which one of the following could be true?
A. A and C both serve on the X committee. (C on  $X$ )&(D on  $X$ ) conflict with R-2
B. A and E both serve on the Y committee.
$(A$  on  $Y)\& (E$  on  $Y)$  conflict with R-4
C. B and G both serve on the X committee.
$(G$  on  $X)\& (F$  on  $X)$  conflict with R-3
D. C and E both serve on the Y committee.
E. G and E both serve on the X committee.
Participants Positions Fact
$(A,B,C,D,E,F,G)$ $(X,Y)$ $(D$  on  $X)\& (F$  on  $X)$
Rules to Logical Expressions
R-1:  $A$  on  $X \to B$  on  $Y$
R-2:  $C$  on  $X \to (D$  on  $Y)\&(E$  on  $Y)$
R-3: Position of  $F \neq$  Position of  $G$
R-4: Position of  $E \neq$  Position of  $A$
R-5:  $G$  on  $X \to B$  on  $X$
Figure 1: An example of the required reasoning process to do well on the AR task. The input is a passage, a question and multiple options, and the output is the most plausible answer.
Analytical reasoning assesses the ability of systems to understand the knowledge, including participants, facts and literal rules mentioned in the context, perform reasoning over the extracted knowledge, and make conclusions. In this paper, we study the challenge of analytical reasoning (AR). We collect a new dataset AR-LSAT from the Law School Admission Test $^{2}$  (LSAT) from 1991 to 2016 to facilitate research on analytical reasoning. An example of analytical reasoning in LSAT is given in Figure 1, whose task is to separate participants (i.e.,  $A, B, \text{etc.}$ ) into two positions (i.e.,  $X$  committee and  $Y$  committee) under certain constraints. We can see that solving the problem requires a system to understand the knowledge in the context including participants, positions, rules expressed in natural
1The data and code are provided in https://github. com/zhongwanjun/AR-LSAT.
$^{2}$ https://en.wikipedia.org/wiki/Law_School_Admission_Test
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2306 - 2319
language (e.g., "If  $G$  serves on  $X$ , so does  $B$ ) and facts (e.g., "D and  $F$  both serve on the  $X$  committee"). Then, it needs to deduct logical expressions (e.g., "G on  $X \rightarrow B$  on  $X$ ) from the rules, and draw inference before making conclusions.
In this paper, we analyze the knowledge understanding and reasoning ability required for solving this task and present Analytical Reasoning Machine (ARM), a framework that can comprehend the context and perform reasoning for making a conclusion. It extracts participants, rules and facts described in the context of text. Each literal rule is mapped into an executable logical constraint function, which assesses whether a solution satisfies a particular rule. With such logical-level understanding, ARM is capable of deducing a group of legitimate solutions for the question and select the most plausible option as the answer.
Experiments show that pre-trained models struggle to learn this task, which indicates that this task is very challenging for current models as it requires the complex reasoning ability far beyond implicit reasoning over the literal clues. Our system outperforms pre-trained models significantly. Further analysis demonstrates that our system has better interpretability. The contributions are threefold.
- We collect a new dataset AR-LSAT to facilitate research on analytical reasoning.
- We present a reasoning framework that can comprehend the context and perform explicit interpretable reasoning to draw conclusion.
- Experiments indicate that this task is challenging and our system outperforms pre-trained models significantly.
There is an increasing trend on machine reasoning research in recent years. The reasoning ability investigated are partitioned into several major aspects, including (1) logical reasoning; (2) commonsense reasoning; (3) mathematical reasoning and (4) multi-hop reasoning.
Logical Reasoning The task of Natural Language Inference (NLI) (Dagan et al., 2005; Bowman et al., 2015; Wang et al., 2019; Williams et al., 2018; Welleck et al., 2019; Khot et al., 2018; Nie et al., 2020; Bhagavatula et al., 2020; Liu et al., 2020a) requires the models to detect the logical entailment relationship of two sentences. There have been Machine Reading Comprehension (MRC)
works (Gao et al., 2021; Rajpurkar et al., 2016; Welbl et al., 2018a; Yang et al., 2018a; Huang et al., 2019a; Wang et al., 2021) that examine the ability of logical reasoning. LogiQA (Liu et al., 2020b) and ReClor (Yu et al., 2020) are sourced from examination in realistic scenario and examine a range of logical reasoning skills.
Commonsense Reasoning There are many recent benchmarks that assess the commonsense reasoning capabilities from different aspects, like social (Rashkin et al., 2018), physics (Talmor et al., 2019; Zellers et al., 2019; Zhong et al., 2019), or temporal (Zhou et al., 2019; Zhong et al., 2020a) aspects. There exist several MRC datasets that require commonsense knowledge (Ostermann et al., 2018; Zhang et al., 2018; Huang et al., 2019b).
Mathematical Reasoning There are many existing datasets (Kushman et al., 2014; Hosseini et al., 2014; Koncel-Kedziorski et al., 2015; Clark et al., 2016; Ling et al., 2017) that focus on mathematical word problems. Ling et al. (2017) builds a dataset that encourages generating answer rationales beyond simply selecting the correct answer. DROP (Dua et al., 2019) is a benchmark MRC dataset requiring mathematical reasoning. Saxton et al. (2019) focuses on algebraic generalization.
Multi-hop Reasoning Multi-hop reasoning over textual data (Talmor and Berant, 2018; Welbl et al., 2018b; Yang et al., 2018b; Inoue et al., 2020; Zhong et al., 2022) requires a model to reason over multiple paragraphs before making prediction.
To the best of our knowledge, there has not an existing benchmark dataset that completely focuses on the analytical reasoning over textual data. We introduce a new dataset to fill this gap and to foster research on this area.
3 Task and Dataset
In this section, we describe the task of analytical reasoning and introduce the dataset AR-LSAT we collected from the Law School Admission Test.
3.1 Task: Analytical Reasoning of Text
Taking a passage, a question, and multiple options as the input, a system is required to select the most plausible answer as the output. Each passage describes a reasoning game belonging to various types, including three dominant types: ordering games, grouping games, and assignment games,
[Ordering Game] Passage
A professor must determine the order in which five of her students - Fernando, Ginny, Hakim, Juanita, and Kevin- will perform in a recital.
Ginny perform earlier than Fernando. R-1
Kevin perform earlier than Hakim and Juanita. R-2
Hakim perform either immediately before or immediately after Fernando. R-3
Question
Which one of the following could be the order the students perform?	Options
A. Ginny, Fernando, Hakim, Kevin, Juanita ×R-2
B. Ginny, Juanita, Kevin, Hakim, Fernando ×R-2
C. Ginny, Kevin, Hakim, Juanita, Fernando ×R-3
D. Kevin, Ginny, Juanita, Fernando, Hakim
E. Kevin, Juanita, Fernando, Hakim, Ginny ×R-1
Fact
Uncertain	Positions
(1st, 2nd, 3rd, 4th, 5th)	Participants
(Fernando, Ginny, Hakim, Juanita, Kevin)
Rules to Logical Expressions
R-1: Pos. of Ginny < Pos. of Fernando
R-2: (Pos. of Kevin < Pos. of Hakim) & (Pos. of Kevin < Pos. of Juanita)
R-3: (Pos. of Hakim = Pos. of Fernando + 1) | (Pos. of Hakim = Pos. of Fernando - 1)
[Assignment Game] Passage
Five cashiers-Adams, Bates, Cox, Drake, and Edwards-each of whom works alone on exactly one day, Monday through Friday
Adams will work only on Tuesday or Thursday. R-1
Bates will not work on Monday or Wednesday. R-2
Cox works on Friday. F-1
Edwards don't work next to Drake R-3
Question
Which one of the following is a possible work schedule?	Options
A. Edwards, Bates, Adams, Drake, Cox ×R-1
B. Drake, Adams, Bates, Edwards, Cox ×R-2
C. Edwards, Adams, Cox, Bates, Drake ×F-1
D. Edwards, Adams, Drake, Bates, Cox √
E. Drake, Edwards, Bates, Adams, Cox ×R-3
Fact
Cox on Fri.	Participants
(Adams, Bates, Cox, Drake, Edwards)
Positions
(Mon., Tues., Wed., Thur., Fri.)
Rules to Logical Expressions
R-1: Adams on Tues. | Adams on Thur.
R-2: ¬(Bates on Mon. | Bates on Wed.)
R-3: Pos. of Edwards ≠ Pos. of Drake + 1
Figure 2: Examples of ordering game and assignment game in AR task. Facts and Rules are highlighted in orange and blue, respectively. Example of grouping game is shown in Figure 1.  $\times$  indicates conflict.
which are described as follows and examples are given in Figures 1 and 2:
- Ordering games are to order participants based on given facts and rules.
- Grouping games are to separate participants into groups with given facts and rules.
- Assignment games are to assign characteristics to the participants with given rules, like assigning schedules for people.
3.2 Dataset: AR-LSAT
We collect data from nearly 90 LSAT exams from 1991 to 2016 and select questions from the analytical reasoning part to construct the dataset, dubbed AR-LSAT. Each exam in LSAT consists of 101 questions, 24 of which are AR questions. We finally leave up the questions with 5 answer options. The statistics are shown in Table 1. We manually categorize and analyze question types in AR-LSAT according to different reasoning types, and describe the detailed descriptions and corresponding examples in the Appendix D.
Number of questions	2,046
Average length of passages	99.3
Average length of questions	19.1
Average length of answers	6
Number of options	5
Ratio of ordering game	42.5%
Ratio of grouping game	38.75%
Ratio of assignment game	18.75%
Table 1: Data statistics of AR-LSAT dataset.
3.3 Baseline: Pre-trained Model
Pre-trained Transformer (Vaswani et al., 2017) based language models achieved impressive performance on a wide variety of tasks. There
are several representative pre-trained models, like BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), and ALBERT (Lan et al., 2020). We employ these powerful pre-trained models as our baselines after being fine-tuned on our dataset. Specifically, we take the concatenated sequence  $X = \{[CLS], \text{passage}, [SEP], \text{question}, \text{option}\}$  as the input, where  $[CLS]$  is the ending special token and  $[SEP]$  is used to split two types of input. The final hidden vector at [CLS] is taken for classification. However, we find that these models struggle to deal with this task as their performances are close to random guess. For example, RoBERTa achieves  $23.1\%$  accuracy on the test set.
3.4 Challenges
In this part, we point out the reasoning ability required for solving AR questions, and put forward the challenges that systems should face.
As we can observe from the examples in Figure 1 and Figure 2, AR questions test a range of reasoning skills:
1) Comprehending the knowledge including participants of events, facts, and rules described in the context.
2) Extracting machine-understandable logical functions (expressions) from the rules. For example, the rule "If  $A$  serves on  $X$ , then  $B$  serves on  $Y$ ." needs to be transferred as logical expression "A on  $X \rightarrow B$  on  $Y$ ",
3) Making deductions to derive legitimate solutions that satisfy extracted logical functions.
4) Selecting the answer that satisfies all the rules with the deducted legitimate solutions. In the
Figure 3: An overview of our approach. The original example is given in Figure 1. It extracts arguments from the context ( $\S 4.1$ ). Then it extracts constraint functions based on rules ( $\S 4.3$ ). Afterwards, it conducts deduction to find legitimate assignments ( $\S 4.4$ ). Lastly, it matches the options and legitimate assignments for prediction ( $\S 4.5$ ).
examples, a system should eliminate options that conflict with rules and select the option that accords with legitimate solutions.
Therefore, this task requires the machine to perform explicit complex reasoning, far beyond just understanding the literal clues presented in the text.
We describe how our system, the Analytical Reasoning Machine (ARM), comprehends the knowledge, performs reasoning over the knowledge, and makes conclusions. Figure 3 gives an overview of our approach. Our system operates in four steps: (1) extracting the participants, positions, facts and rules from the passage and the hypothesis of the question ( $\S$  4.1); (2) interpreting rules into a set of logical constraint functions defined in  $\S$  4.2, whose arguments are selected from participants and positions ( $\S$  4.3); (3) reasoning with the logical functions and finally generating a group of legitimate assignments (solutions) that satisfy all the rules ( $\S$  4.4); (4) selecting the most plausible option by matching the legitimate assignments and options ( $\S$  4.5). ARM sheds a light on the logical-level reasoning procedure for analytical reasoning and each procedure can be further developed for both performance and expandability.
4.1 Arguments Extraction
In order to understand the context and formalize the problem, the first step is to extract the participants, positions, facts and rules expressed in natural language from the passage and hypothesis of the question. An assignment represents a solution that assigns participants to positions. An assignment of participants is represented as a table, whose rows and columns represent participants and positions, respectively. Each grid represents whether a par
participant is assigned to a position, and has the value of three possible states: (True, False, Unknown). The rules describe the constraints of assignments while the facts describe certain assignments. Therefore, we take the sentences that mention specific assignments (e.g.,  $A$  on  $X$ ) as facts and the other sentences as rules. Facts represent initial assignments to the grids of the assignment table and the default state is noted with Unknown. We take the example in Figure 1 as a running example to show the extracted participants, positions, facts and rules from the context.
Specifically, we extract the entities from the leading sentence of the passage with a neural Named Entity Recognition (NER) model (Peters et al., 2017) and group the extracted entities into participants or positions. We parse groups of entities that appear together in the leading sentence of the passage as groups of participants or positions, where participants always appear before positions. For the ordering game, positions can not be directly extracted, so we take them as the order (e.g., first, second) of participants.
4.2 Constraint Function Definition
We introduce a set of predefined logical functions, which encode constraints expressed in the literal rules and check if an assignment satisfies these constraints. These functions are the foundation of the reasoning process.
The logical functions include three basic types: (1) relational function; (2) compositional function; (3) counting function. A fragment of the predefined functions is shown in Table 2. A function consists of arguments and a executor to check whether an assignment satisfies the constraint function. The detailed definition of each function is listed in Appendix B.
Relational Function The relational functions represent the constraints of the relationship between two participants or a participant and a position. The arguments of relational function involve participant or position. For example, the function Before(Ginny, Fernando) indicates that Ginny should be in the position before Fernando in the ordering game. To  $(A, X)$  indicates that participant  $A$  should be assigned to position  $X$ .
Compositional Function A compositional function expresses the relationship between two sets of functions, like the conditional rule (if-then rule) and the if-and-only-if rule. The arguments of compositional functions involve two sets of subfunctions. For example, the rule "If  $A$  serves on the  $X$ , then  $B$  serves on the  $Y$ " should be expressed as IfThen(\{To(A, X)\}, \{To(B, Y)\}).
Counting Function The counting functions focus on the calculation problem of participants under specific constraints. The arguments of counting functions involve a participant and a number. For example,  $\text{LastPos}(A, 3)$  checks whether the participant A is assigned to the last 3 positions.
The input of a function executor is an assignment and the output is a Bool value indicates whether the assignment satisfies the constraint.
4.3 Function Extraction
Based on the extracted arguments, we parse the rules expressed in natural language into a set of constraint logical functions that can check whether an assignment satisfy the rules.
One straightforward way is to design a symbolic parsing method. We define an API set to include roughly 20 types of functions like Before, After, To, IfThen and realize their executors. For each function, we follow NSM (Liang et al., 2017) that uses trigger words to match a potential function. For example, the function Before can be triggered by words "before" and "earlier". All the functions and trigger words are listed in Appendix B. To extract potential arguments from a given rule, we match the participants, positions, and number from the text. If a function is recognized by a trigger word, we select its arguments from all the potential arguments according to their relative positions to the trigger word. The relational and counting functions can be constituted into compositional functions based on grammar patterns. For example, for the grammar pattern "If  $P$ , then  $Q$ ", Each function is grouped into the function set  $F_{1}$  if it occurs in  $P$ ,
or the function set  $F_{2}$  if it occurs in  $Q$ .  $F_{1}$  and  $F_{2}$  are taken as the arguments of the function IfThen.
Furthermore, to handle the uncertain cases and improve the coverage of extracted functions, we build a neural semantic parsing model based on a pre-trained language model RoBERTa (Liu et al., 2019). It takes the sentence and two parsed arguments in the sentence as the input and predicts their potential function type ("Null" if no function exists). Specifically, following Xu et al. (2021), we modify the sentence by adding a special token "@" before and after the first argument, and a special token "#" before and after the second argument. Then, we encode the modified sentence  $X$  with RoBERTa to obtain contextual representations  $H = \text{RoBERTa}(X)$ . for tokens. Afterwards, we take the representation of the first "@" and "#" for classification.
$$
f = \operatorname {a r g m a x} (\text {c l a s s i f i e r} ([ H ^ {\circ}; H ^ {\#} ])) \quad (1)
$$
where  $[\cdot ]$  denotes concatenation, and the classifier is a linear layer followed by a softmax function. Since there is no annotated data of corresponding logical functions, we need to construct the training data automatically. The training data consist of (1) positive instances: all the  $\{input$ : (rule, arguments); label: function\} pairs that extracted by the symbolic parsing method from the training set; (2) negative instances: the same number of instances that have arguments with no function related.
Afterwards, we extract a set of constraint functions with the combination of symbolic and neural parsing methods. These functions are utilized for reasoning process introduced in the following part.
4.4 Legitimate Assignments Deduction
Given the extracted logical constraint functions and the initial assignment table, we conduct reasoning to find the legitimate assignments that satisfy all the constraints. The process is formulated into a tree-based reasoning algorithm. As shown in Figure 4, each node in a tree corresponds to a table assignment and each edge indicates a constraint function. A node  $v$  with path  $\{e_0, e_1, \dots, e_i\}$  from the root indicates that its assignment satisfies constraint functions  $\{f_0, f_1, \dots, f_i\}$ . Suppose we have  $n$  constraint functions, we need to find all the leaf nodes with depth  $n$ . These leaf nodes satisfy all the functions and thus become legitimate assignments.
Therefore, we introduce how to construct the complete reasoning tree by the following steps:
Type	Function	Args	Description
Relational Functions	Before/After	participant1 participant2	Whether participant1 is in the position before/after participant2.
Same/Different	Whether participant1 is in the same/different position with participant2.
To	participant1 position1	Whether participant1 is assigned to position1.
Compositional Functions	IfThen	function set F1 function set F2	If functions in F1 satisfied, then functions in F2 satisfied.
Counting Functions	FirstPos/LastPos	participant1, number m	Whether participant1 is assigned to the first/last m positions.
Table 2: A fragment of the logical constraint function definition.
Function  $f_{0}$
$$
f _ {0} = I f T h e n \left(\left\{T o (A, X) \right\}, \left\{T o (B, Y) \right\}\right)
$$
Assignment Generation
Initial assignment  $a_0$
(2) Function Execution
to find conflict
Conflict with  $f_{0}$
Reasoning Tree Extension
Figure 4: An example of the reasoning process. Newly added participants in  $f_{0}$  are highlighted. (1) and (2) conducted recursively until depth  $= n$ .  $(T / F / - ) =$  (True/False/Unknown)