SlowGuess/NAACL_2022_OCR huggingface dataset:
1) Firstly, we start with the root, which is the certain initial assignment decided by facts. For the function  $f_{0}$ , we generate all possible assignments related to newly added arguments in  $f_{0}$ . As shown in the example in Figure 4, for the function IfThen  $(\text{To}(A, X), \text{To}(B, Y))$ , we generate all possible assignments related to the new participants  $A$  and  $B$ .
2) We execute  $f_0$  to find all the legitimate assignments that satisfy  $f_0$  as a group of children of the root. In the same example, we keep the assignments that meets IfThen  $(\operatorname{To}(A, X), \operatorname{To}(B, Y))$ .
3) Then we select each child as a new root and select function  $f_{1}$  for further extension of the reasoning tree.
These processes are recursively conducted until depth  $n$ , which means that all the functions are
used to construct the reasoning tree. The procedure is summarized into pseudo-code in Appendix A.
It is worth mentioning that although both our algorithm and forward-chaining algorithm deduce new facts based on rules. However, forward-chaining algorithm struggles to do this task because it assumes that all the assignments are already known to the systems while the assignments are always unknown before the deduction steps.
Therefore, this algorithm has advantages of performing explicit interpretable reasoning over the extracted functions and handling uncertain assignments. Moreover, the tree-based manner reduces the computational complexity.
4.5 Answer Selection
Previous steps understand the passage and the question. In this part, we introduce how to analyze the options, and match the options with the deducted legitimate assignments beyond word-level for making a final prediction. Specifically, we can derive two types of information from an option:
1) Assignment-based option indicates a table assignment. For example, "A and C both serve on the X committee" can be interpreted as a assignment in the table:  $\{(A,X) = \text{True}; (C,X) = \text{True}\}$ . For this type, we match the parsed option assignment with all the legitimate assignments and calculate an assignment-based matching score.
2) Function-based option indicates an option representing a constraint function, like "The sedan is serviced earlier in the week than the roadster", which can be parsed into the function "Before(sedan, roadster)". We execute the option-based function on the legitimate assignments to find the satisfiable option and calculate a function-based matching score.
These two types of scores are combined for making a conclusion. The question types and score calcu
lating methods are summarized in the Appendix C.
We make experiments on the AR-LSAT dataset and evaluate our system with label accuracy. The data split is (train/dev./test) = (1, 585/231/230) We first compare our system with powerful neural baselines and conduct analysis. Moreover, case study illustrates the reasoning process of our system by an explicit example. Lastly, we make error analysis to point out challenges in this task.
5.1 Model Comparison
Baseline Models We take various powerful neural models, including RNN-based models (i.e., LSTM) and powerful Transformer-based pretrained language models (i.e., BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), and the recent ALBERT (Lan et al., 2020)) as the baselines of our dataset and investigate their performance. The implementation details of these baselines are given in Appendix D.
Human Performance Since the dataset is based on a test designed for undergraduate students, we select nearly 100 instances in the AR-LSAT dataset and ask 10 undergraduate college students majoring in literature, commerce and law to answer these questions. We take their averaged performance as human performance and report it in Table 3.
Methods	Dev. Acc (%)	Test Acc (%)
Human Performance	-	59.7%
Random Guess	20.0%	20.0%
LSTM	22.5%	20.9%
BERT	23.4%	21.4%
XLNet	23.8%	22.5%
RoBERTa	24.2%	23.1%
ALBERT	24.4%	23.0%
ARM	34.2%	30.9%
Results and Analysis In Table 3, we compare our system (ARM) with baselines and human performance on the development and test set. As shown in Table 1, our model with context understanding and explicit reasoning process significantly outperforms RNN-based models and pretrained language models with  $34.2\%$  accuracy on the development set and  $30.9\%$  accuracy on the test
set. Results indicate that context understanding and reasoning are essential for this task.
Moreover, we observe that the RNN-based models and pre-trained models struggle to do well on this task, and achieve close performance with random guess. It is also noticed that the performance of both our system and baselines are still far from human performance, leaving significant opportunities for further exploration.
In this part, we further analyze the performance and variance of components of our system. To evaluate the performance of arguments extraction, we manually annotate the correct participants and positions in the development set as labels and calculate the accuracy and recall of our condition extraction method and report the results in Table 4. Moreover,
Table 3: Performance on the AR-LSAT dataset. Our model is abbreviated as ARM.
Acc. (%)	Recall (%)
Participants	96.17	92.88
Positions	84.42	85.79
we eliminate the neural semantic parsing method to evaluate its importance and extract functions by the symbolic parsing method. The results are shown in
Table 4: Performance of extraction of participants and positions on the development set.
Methods	Dev. Acc (%)	Test Acc (%)
ARM	34.2%	30.9%
ARM (w/o neural func.)	32.4%	30.2%
Table 5: Ablation of the the neural semantic parser.
Table 5. Eliminating neural semantic parsing yields no significant compromise in performance. This observation indicates that the neural semantic parsing model can improve performance by improving coverage of the functions and the symbolic parsing method can also provide reliable performance.
We present a case study in Figure 5 to illustrate the reasoning process of our system with interpretable results. Our system extracts correct arguments from the context, and interprets the rules into logical constraint functions. Afterwards, we perform deduction to find legitimate solutions. Lastly, our system matches the options with the legitimate solutions and calculates a score for each option. Option  $A$  achieves the highest score because it accords with legitimate assignments. This analysis
Passage: A professor must determine the order in which five of her students — Fernando, Ginny, Hakim, Juanita, and Kevin — will perform in an upcoming piano recital. Each student performs one piece, and no two performances overlap. The following constraints apply: Ginny must perform earlier than Fernando. Kevin must perform earlier than Hakim and Juanita. Hakim must perform either immediately before or immediately after Fernando.
Question: If Juanita performs earlier than Ginny, then which one of the following could be true?
Options: (A) Fernando performs fourth. √(B) Ginny performs second. (C) Hakim performs third. (D) Juanita performs third. (E) Kevin performs second
Participants & Positions	Fernando, Ginny, Hakim, Juanita, Kevin	first, second, third, fourth, fifth
Rules & Functions	(1) Ginny must perform earlier than Fernando.
(2) Kevin must perform earlier than Hakim and Juanita.
(3) Hakim must perform either immediately before or immediately after Fernando.
(4) Juanita performs earlier than Ginny	(1) Before (Ginny, Fernando)
(2) And {Before (Kevin, Hakim)}, {Before(Kevin,Juanita))}
(3) Or {Next (Hakim, Fernando)}, {Last (Hakim, Fernando))}
(4) Before (Juanita, Ginny)
Legal Assignments	1st	2nd	3rd	4th	5th
Fernando	F	F	F	T	F	F
Ginny	F	F	T	F	F	F
Hakim	F	F	F	F	T	F
Juanita	F	T	F	F	F	F
Kevin	T	F	F	F	F	F
Option Scores	(A) 1 (B) - 1 (C) - 1 (D) - 1 (E) - 1
Figure 5: A case study on the AR-LSAT dataset. Our system correctly extracts participants, positions, and rules from the context. Afterwards, it interprets rules into logical functions. After deduction, our system finds legitimate assignments and makes the correct prediction. Rules are highlighted in blue.
demonstrates that our system has better explicit interpretable reasoning ability.
We randomly select 50 wrongly predicted instances from the dev. set and summarize the error types.
The dominant error type is that some rules with complex semantics are not covered by current constraint logical function set. For example, given a rule "Each crew member does at least one task during the installation," we should map "At least" to function AtLeastNum. The second type of errors is caused by failing to extract correct participants or positions by the NER model and predefined matching pattern. The third error type is caused by the lack of basic commonsense knowledge, which is required for understanding the concept in the rules. For example, when a passage mentioned "Six entertainers should be scheduled at 9:00 A.M., 2:00 P.M., etc" and the rule is "Some participants should be scheduled in the morning," the system fails to match the morning with a specific time zone.
We would like to further highlight important directions to facilitate research on analytical reasoning.
One of the major challenges lies in deep understanding of the knowledge in the context, like parsing the rules into logically equivalent symbolic functions. Deriving machine-understandable functions from natural language is an essential step towards deeper understanding and reasoning. Although supervised semantic parsing has achieved
promising progress in recent years, obtaining complete human-annotated logical functions is impractical for this task. Therefore, further study can focus on function extraction with limited amount of annotated functions.
Furthermore, a better inference engine built upon logical functions is also essential because AR questions require deeper reasoning abilities far beyond just understanding the literal clues. Standard symbolic systems like expert systems can provide explicit reasoning, but they are difficult to deal with uncertainty in data. Although neural-based methods are more flexible at dealing with uncertainty, they still struggle to perform interpretable and explicit reasoning. It is promising to better integrate neural and symbolic systems to improve this task with deeper reasoning ability.
In this paper, we study the challenging task of analytical reasoning and introduce a dataset AR-LSAT to facilitate research on analytical reasoning. We analyze the knowledge understanding and reasoning ability required for this task and present a system, Analytical Reasoning Machine (ARM), which can comprehend the knowledge, including participants, facts and rules mentioned in the context and extract logically equivalent logical functions from the rules. Afterwards, it performs deep reasoning to find all the legitimate solutions to the problem posed and finally makes a prediction. Experiments show that our system outperforms strong Transformer-based baselines, which indicates that
knowledge understanding and deep reasoning is essential for this task. Results show that this task is very challenging for current neural-based models.
Jian Yin is the corresponding author. Wanjun Zhong, Daya Guo, Jiahai Wang and Jian Yin are supported by the Key-Area Research and Development Program of Guangdong Province (2020B0101100001)
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D. Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2580-2586. AAAI Press.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. pages 177-190.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Yifan Gao, Jingjing Li, Michael R Lyu, and Irwin King. 2021. Open-retrieval conversational machine reading. ArXiv preprint, abs/2102.08633.
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 1999. Learning to forget: Continual prediction with LSTM.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523-533, Doha, Qatar. Association for Computational Linguistics.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019a. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019b. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. 2020. R4C: A benchmark for evaluating RC systems to get the right answer for the right reason. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740-6750, Online. Association for Computational Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5189-5197. AAAI Press.
Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585-597.
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 271-281, Baltimore, Maryland. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jingjing Li, Zichao Li, Tao Ge, Irwin King, and Michael R Lyu. 2022. Text revision by on-the-fly representation optimization. ArXiv preprint, abs/2204.07359.
Jingjing Li, Zichao Li, Lili Mou, Xin Jiang, Michael R. Lyu, and Irwin King. 2020. Unsupervised text generation by learning from search. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23-33, Vancouver, Canada. Association for Computational Linguistics.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun-. som. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167, Vancouver, Canada. Association for Computational Linguistics.
Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. 2020a. Natural language inference in context-investigating contextual reasoning over long texts. ArXiv preprint, abs/2011.04864.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020b. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3622-3628. ijcai.org.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 task 11: Machine comprehension using commonsense knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages
747-757, New Orleans, Louisiana. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756-1765, Vancouver, Canada. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2Mind: Commonsense inference on events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 463-473, Melbourne, Australia. Association for Computational Linguistics.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Xin Sun, Tao Ge, Shuming Ma, Jingjing Li, Furu Wei, and Houfeng Wang. 2022. A unified strategy for multilingual grammatical error correction with pretrained cross-lingual language model. ArXiv preprint, abs/2201.10707.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641-651, New Orleans, Louisiana. Association for Computational Linguistics.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. 2021. Logic-driven context extension and data augmentation for logical reasoning of text. *ArXiv* preprint, abs/2105.03659.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018a. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018b. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, and Nan Duan. 2021. Syntax-enhanced pre-trained model. In Proceedings of the 59th Annual