Introduction
Critical reasoning, a type of logical reasoning not tied to formal logic, is a core ability of humans that is required for thoughtful reading of text.
It involves not only understanding what a passage explicitly says but also comprehending its underlying assumptions, argument structure, and supported conclusions.
Developing systems capable of critical reasoning as reliably as humans is one of the ultimate goals of natural language processing.
Recent studies have proposed datasets that evaluate logical reasoning including critical reasoning ability Yu et al. ( 2020 ); Liu et al. ( 2020 ) in reading comprehension.
Owing to the recent development of large language models (LLMs; Brown et al., 2020 ; He et al., 2023 ) , the performance of the state-of-the-art models is nearing that of humans Jiao et al. ( 2022 ); Wang et al. ( 2022 ) .

However, current multiple-choice questions in existing logical reading comprehension datasets may not sufficiently test the ability of critical reasoning.
The example illustrated in Figure 1 shows that even if a model can answer a question taken from the ReClor dataset Yu et al. ( 2020 ) that has questions for graduate admission examinations, it cannot answer an auxiliary question that queries the implicit rationale for eliminating a relevant alternative.
This behavior might be due to the model’s limited generalizability that is exposed by input perturbation Si et al. ( 2021 ); Lin et al. ( 2021 ); Shi et al. ( 2023 ) or characterized as shortcut reasoning Niven and Kao ( 2019 ); Geirhos et al. ( 2020 ) .
Because a single question cannot fully ask the rationale of why we select an option as the correct answer and eliminate the others as the incorrect ones, current datasets may not be sufficient to comprehensively evaluate the process of critical reasoning.

Recent studies propose methods for probing the reasoning process using auxiliary generation tasks such as in the form of simple commonsense facts Aggarwal et al. ( 2021 ) , logical graphs Huang et al. ( 2022 ) , and arithmetic equations Ribeiro et al. ( 2023 ) .
However, this line of approach may not be suitable to capture the implicit rationale of critical reasoning.
In particular, it cannot explicitly consider the selection and elimination process of relevant alternatives in logical reasoning.
In addition, the format of such auxiliary tasks is usually not the same as that of the main task, which may fail to evaluate the target abilities consistently.

As a first step to address these limitations, we construct a benchmark that comprehensively evaluates language models’ ability of critical reasoning in logical reading comprehension.
Our dataset, rationale understanding for logical reasoning evaluation (RULE), consists of main questions taken from ReClor and auxiliary subquestions that we newly create for this study.
The process of constructing our dataset is illustrated in Figure 2 .
Our core idea is that for each answer option in a main question, we crowdsource a free-form human-written rationale that explains why that option should be selected or eliminated, and use those rationales to create a set of subquestions that are associated with the main question. After manual filtering to ensure human answerability, in addition to 943 main questions, we obtain 3,003 subquestions for the test-only purpose.
The common multiple-choice format of the main questions and subquestions enables us to evaluate the models’ capability of critical reasoning concisely and consistently.

In our experiments using strong baseline models including LLMs, e.g., Flan-UL2, Tay et al. ( 2023 ) , LLaMA 2 Touvron et al. ( 2023b ) , and InstructGPT Ouyang et al. ( 2022 ) , we observe that the models cannot answer the main questions and subquestions consistently, showing a larger than 30% gap against humans in our strict consistency metric.
In particular, we find that the models struggle to answer eliminative subquestions, which are pertinent to the rationale of eliminating incorrect options, showing a large gap ( \approx 20% accuracy) between humans and the best-performing LLM.
Conversely, the models tend to correctly answer selective subquestions, which are pertinent to the rationale of selecting the correct option.
This clear contrast suggests that these models provide the correct answer without fully understanding why the other options are incorrect.
Our analysis using a follow-up task and manual annotations supports this observation.
We also compare our human-written rationales with model-generated ones using an LLM, finding that our rationales are likely to be more detailed and supportive than the model-generated ones.

Our contributions are as follows:
(i) Based on an existing logical reading comprehension dataset, we create a dataset including over 3,000 auxiliary questions designed to test a model’s consistent ability for critical reasoning. (ii) We evaluate cutting-edge models, including LLMs, across finetuned, few-shot, and zero-shot settings, showing that even the best model falls short of human performance, particularly lagging in understanding eliminative rationales for incorrect answer options. (iii) Our annotation analysis also highlights the model’s deficiency in understanding eliminative rationales and shows that our human-written rationales are of higher quality than model-generated ones. 1 1 1 Our dataset, evaluation scripts with model hypterparameters, and annotation results are publicly available at github.com/nii-cl/rule