Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
Authors: Akira Kawabata
Saku Sugawara
Venue: EMNLP 2023 Main (2023)
Keywords: natural language understanding
reading comprehension
evaluation
dataset
rationale
TL;DR: A new dataset for evaluating the understanding of rationale in logical reading comprehension

Abstract: To precisely evaluate a language model's capability for logical reading comprehension, we present a dataset for testing the understanding of the rationale behind critical reasoning.
For questions taken from an existing multiple-choice logical reading comprehension dataset, we crowdsource rationale texts that explain why we should select or eliminate answer options, resulting in 3,003 multiple-choice subquestions that are associated with 943 main questions.
Experiments on our dataset show that recent large language models (e.g., InstructGPT) struggle to answer the subquestions even if they are able to answer the main questions correctly.
We find that the models perform particularly poorly in answering subquestions written for the incorrect options of the main questions, implying that the models have a limited capability for explaining why incorrect alternatives should be eliminated.
These results suggest that our dataset encourages further investigation into the critical reasoning ability of language models while focusing on the elimination process of relevant alternatives.

Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
Akira Kawabata The Asahi Shimbun Company kawabata aaattaaaaaaaa....... com
Saku Sugawara
National Institute of Informatics sakuênii. ac.
Abstract
Passage
Statistical records much about the motives and methods of those who compile or cite them [--]. The poliice may underreport crime [..]
rates probably often reflect
To precisely evaluate a language model's capa- bility for logical reading comprehension, present a dataset for testing the understand- ing of the rationale behind critical reasoning. For questions taken from an existing multiple- choice logical reading comprehension dataset, we crowdsource rationale texts that explain why we should select or eliminate answer options, resulting in 3,003 multiple-choice subquestions that are associated with 943 main questions. Ex- periments on our dataset show that recent large language models (e.g., InstructGPT) struggle answer the subquestions even if they able to answer the main questions correctly. We find that the models perform particularly poorly in answering subquestions written for the incor- rect options of the main questions, implying that the models have a limited capability for explaining why incorrect alternatives should be eliminated. These results suggest that our dataset encourages further investigation into the critical reasoning ability of language mod- els while focusing on the elimination process of relevant alternatives.
overreport crime [..]] Politicians may magnify crime rates to
[..]] Newspapers often sensationalize [..].
Main Question
The argument proceeds by doing which one of the following?
deriving implications of genera inatanization that assumes
showing how evidence that apparently contradicts supports conclusion.
C. enumerating problems for which proposes general al solution.
D. citing examples support of conciusion.
Sub Question (on the rationale of eliminating Option A)
Why is method of drewing implications from assumnd true generallzation considared way which argument procsecis?
This passage giving examplee of cleim, generalizing arything
X
B. Nothing in this pessage contr adittadicts it's conciusion. d
C. The passage proceeds to give massns why erime rates are misleading. the passage approaches condiusion by citing examples.
Figure I: Example of ReClor (Yu et al., 2020) and its subquestion we create to test the understanding of implicit rationale. We find that even if the model can answer the original question correctly, it cannot answer subquestions that should be answerable.
However, current multiple-choie questions in existing logical reading comprehension datasets may not sufficiently test the ability of critical rea- soning. The example illustrated Figure/ 1 shows that even if model can answer question taken from the ReClor dataset (Yu et al., 2020) that has questions for graduate admission examinations, it cannot answer an auxiliary question that queries the implicit rationale for eliminating a relevant alter- native. This behavior might be due to the model's 's limited generalizability that is exposed by input per- turbation (Si et al., 2021; Lin et al., 2021; Shi al., 2023) or characterized as shortcut reasoning (Niven and Kao, 2019; Geirhos et al., 2020). Because a single question cannot fully ask the rationale of why we select an option as the correct answer and climinate the others the incorrect ones, current datasets may not be sufficient to comprehensively evaluate the process of critical reasoning.
1 Introduction
Critical reasoning, type of logical reasoning not tied to formal logic, is core ability of humans that is required for thoughtful reading of text. It involves not only understanding what passage ex- plicitly says but also comprehending its underlying assumptions, argument structure, and supported conclusions. Developing systems capable of crit- ical reasoning as reliably as humans is one of the ultimate goals of natural language processing. Re- cent studies have proposed datasets that evaluate logical reasoning including critical reasoning abil- ity (Yu et al., 2020; Liu et al., 2020) in reading comprehension. Owing the recent development of large language models (LLMs; Brown et al., 2020; He et al., 2023), the performance of the state- of of-the-art models is nearing that of humans (Jiao et al., 2022; Wang et al., 2022).
Recent studies propose methods for probing the

---

reasoning process using auxiliary generation tasks such as in the form of simple commonsense facts (Aggarwal et al., 2021), logical graphs (Huang et al., 2022), and arithmetic equations (Ribeiro et al., 2023). However, this line of approach may not be suitable to capture the implicit rationale of critical reasoning. In particular, it cannot explicitly consider the selection and elimination process of relevant alternatives in logical reasoning. In addi- tion, the format of such auxiliary tasks usually not the same as that of the main task, which may fail to evaluate the target abilities consistently.
and manual annotations supports this observation. We also compare our numan-written rationales with model-generated ones using an LLM, finding that our rationales are likely to be more detailed and supportive than the model-generated ones.
Our contributions are follows: (i) Based an existing logical reading comprehension dataset, we create a dataset including over 3,000 auxiliary questions designed to test a model's consistent abil- ity for critical reasoning. (ii) We evaluate cutting- edge models, including LLMs, across finetuned,
few-shot, and zero-shot settings, showing that even the best model falls short of human performance, particularly lagging in understanding eliminative rationales for incorrect answer options. (iii) Our annotation analysis also highlights the model's defi- ciency in understanding eliminative rationales and shows that our human- higher quality than model-generann ones.'
As a first step to address these limitations, we construct a benchmark that comprehensively evalu- ates language models' ability of critical reasoning in logical reading comprehension. Our dataset, ra- tionale understanding for logical reasoning evalua- tion (RULE), consists of main questions taken from ReClor and auxiliary subquestions that newly create for this study. The process of constructing our dataset is illustrated in Figure 2. Our core idea that for each answer option in a main ques- tion, we crowdsource a free-form Human-written rationale that explains why that option should be selected or climinated, and use those rationales to create set of subquestions that are associated with the main question. After manual filtering to ensure human answerability, in addition to 943 main ques- tions, we obtain 3,003 subquestions for the test- only purpose. The common multiple-choice format of the main questions and subquestions enables us to evaluate the models' capability of critical rea- soning concisely and consistently.
human-written written rationales are of
2 Related Works
Critical and Logical Reasoning Critical reason- ing is one of the core abilities of logical reasoning that humans perform, along with analytical reason- ing (Zhong et al., 2022) and abductive reasoning (Bhagavatula et al., 2020). This reasoning is re- lated to understanding the structure of practical arguments that is generally composed of ground (premise), warrant (rationale), and claim (conclu- sion). As formulated by Toulmin (2003), given facts or data the ground, provide the warrant that acts as bridge between the ground and the claim we are making. Recent research includes developing ways to model this behavior in tasks such as argument mining and question answering (QA) (c.g., ReClor). For example, Habernal et al. (2018) propose a task of identifying implicit ratio- nale (i.c., warrant) in arguments. However, Niven and Kao (2019) find that successful systems the argument reasoning task exploit superficial in- put features. Similarly, QA systems have been shown to exhibit shallow understanding by input perturbation (Si et al., 2021; Lin et al., 2021; Shi et al., 2023). For example, Lin et al. (2021) demon- strate that QA performance significantly decreases when incorrect options are replaced with irrelevant texts in adversarial manner. This means that successful models those datasets do not nec-
In our experiments using strong baseline models including LLMs, c.g., Flan-UL2, (Tay et al., 2023), LLaMA 2 (Touvron et al., 2023b), and Instruct- GPT (Ouyang et al., 2022), we observe that the models cannot answer the main questions and sub- questions consistently, showing a larger than 30% gap against humans in our strict consistency metric. In particular, we find that the models struggle to an- swer eliminative subquestions, which are pertinent to the rationale of eliminating incorrect options, showing large gap (= 20% accuracy) between humans and the best- L.M. Conversely, the models tend to correctly answer selective sub- questions, which are pertinent the rationale of selecting the correct option. This clear contrast sug- gests that these models provide the correct answer without fully understanding why the other options are incorrect. Our analysis using a follow-up task
'Our dataset, evaluation scripts with model hypterpa- rameters, and annotation results are publicly available at github. com/ni mi i -el-eelrlllll

---

Rationale Writing
Rationale Validation
Sub Question Generation
Human Validation
Option A
Rationale
Oplion
Main Quection
Passage
Serving
ino insiisaooa of Fout anquines
pappaaaeess thing nor
okis exarplee condudis
The arguren proceds by ahe ki cwing?
Sub Question
RationaleA
sryhing
PASS
Because
drow implioiizne generilaat not welccitee argamnnt procodds?
RationaleD
Option
Rationals D sppractes caing vartous
Option
Option implodoon of that mnberr
cinng acamplee
impicationa of that uaqumrcs taue
condicatoo
Figure 2: Our dataset construction process. We first ask crowdworkers to write the rationale for each answer option. After validating the collected rationale by aligning them to the source options, a large language model to generate subquestion texts. We finally ensure the human answerability of the generated subquestions.
essarily exhibit generalizable capabilities in other datasets. These findings necessitate the explain- ability of the (informal) logical reasoning process for better evaluation of intended reasoning abilities (e.g. the critical reasoning in this study).
test the understanding of the rationale necessary for answering the main questions correctly. In con- structing our dataset, we make three decisions in its design choices.
Source Dataset
Among existing datasets for test- ing logical reading comprehension, we use ReClor for the following reasons: (1) It covers various types of logical reasoning required in the multiple- choice format, (2) its context passages are of suf- ficient length to compose a meaningful rationale (e.g., the contexts in LogiQA (Liu al., 2020) are shorter), and (3) contains a sufficient number of examples to create an auxiliary benchmarking dataset. We cannot find other candidate datasets, but our approach is applicable to similar ones.
Reasoning Explanation Although some stud- ies explain the rationale behind commonsense and logical reasoning using graphs (Saha et al., 2021; Ribeiro et al., 2023), others explain it as decom- position (Khot et al., 2020; Dalvi et al., 2021; Geva et al., 2021), combination of supporting textual spans in the input (Yang et al., 2018; Inoue al., 2020), commonsense rules (Saha et al., 2022), or underlying facts (Aggarwal et al., 2021). The work most similar to ours is MetaLogic (Huang et al., 2022), which focuses on generating graphs explain- ing the logical relations between sentences in Re- Clor examples, aiming to model the valid reasoning process. In contrast, we employ free-text rationales that explain the process of critical reasoning, abling us to construct multiple-choice questions about the understanding of rationales. We also aim to faithfully test the models' performance on the main questions as well as auxiliary subquestions in the multiple-choice discrimination task, instead of the generation of the reasoning process in a differ- ent format from the original task.
Rationale Collection The task of writing im- plicit rationales from scratch for logical reason- ing questions is not straightforward because the reasoning process can involve multiple steps with differing granularity. Therefore, to facilitate ratio- nale writing, we use answer options in the multiple- choice questions. To answer question with four options, the reasoning process should involve the rationale of both identifying the correct option and eliminating the three incorrect options. By fo- cusing on the correctness of each option, we can decompose the complex task of rationale writing into smaller intuitive tasks. In addition, we collect human-written free-form rationales to expect ben- efits over model -edee-ggeerated rationales (Sun et al., 2022), in particular for covering the implicit pro- cess of critical reasoning.
3 RULE Data Collection
3.1 Design Choices
We construct new dataset, RULE (rationale un- derstanding for logical reasoning evaluation), to evaluate the consistent rationale understanding in logical reading comprehension. The dataset com- prises main questions and their auxiliary questions (subquestions). The subquestions are designed to
Task Format We also aim design auxiliary questions so that we can easily evaluate models on both main questions and subquestions in the same task format. To this end, we use four rationales

---

collected for a main question the four answer op- tions ifs subquestion. A single main question has at most four subquestions that share the same set of answer options, which can be seen as question- wise contrastive evaluation (Gardner et al., 2020; Ashida and Sugawara, 2022).
Rationale Validation To validate the collected rationales, we first focus on their specificity, which is critical for creating a set of reasonable subques- tions about a given main question. Because assess- ing the necessity of rationales may not be straight- forward, we analyze the reasoning types involved in understanding rationales in Section 5.
3.2 Collecting Rationales
For the validation, we conduct an aligoment test between a set of rationales and answer options. In this test, workers are presented with one main question, its four options, and one rationale. They are then asked to identify which one of the options is supported by the given rationale. If a rationale is insufficiently detailed and could potentially support other options, it would be difficult for workers to correctly match the rationale its corresponding option. We ensure that the worker who validates rationale is different from the one who wrote it.
We use crowdsourcing to collect rationales for cre- ating our subquestions. Appendix A shows our crowdsoureing instructions and examples.
Qualification We conduct a two-stage qualifica- tion test to recruit crowdworkers for our tasks. The first stage is a QA task to identify workers who carefully answer logical reading comprchension questions. The task consists of ten questions taken from ReClor, and workers achieving 2 80% accu- racy advance to the next test. In the second stage, workers are presented with a single ReClor ques- tion that is randomly sampled from a pool of ten questions. The task is to write four implicit ra- tionales (one sentence each) behind each option's 's (in)correctness. To guide them, we provide detailed instructions with eight writing examples.
This test enables us to refine our initial pool of 4,800 rationales down to 3,828, ensuring that each rationale is sufficiently specific to support its corresponding option.
3.3 Subquestion Construction
Question Generation We then generate ques- tion texts to construct subquestions using lan- guage model. Given one main question and one of its options, the model is instructed to generate subquestion that asks about the reason for the correctness of the option. For example, when we input the prompt "What mistake does the argument make in its reasoning?" and the incorrect answer option "It confuses probability and certainty," the model generates the question "What evidence is there that the argument does not make the mistake of confusing probability and certainty?" We use different prompts for the correct and incorrect op- tions avoid the problem of the model omitting negatives (c.g., "not") when generating climinative subquestions. For the generation, we use Instruct- GPT (text -davinci (olici-0030 003), which is one of the strong large language models. Appendix B shows an example of our prompt.
Through preliminary pilot studies, we define two essential criteria for writing rationales: specificity and necessity. Specificity requires rationales to be well informed and support the corresponding options exclusively. This requirement is crucial because non-specific rationales could support mul- tiple options, rendering them unsuitable for options in subquestions. Necessity emphasizes the impor- tance of ensuring that the rationale is essential for validating the option's correctness. Even if a de- tailed rationale provided, must be aligned with the main question's 's point to preserve its validity. Following these criteria, the authors manually assess the rationales provided by the workers. We identify 57 workers through this qualification pro- cess. These workers are invited to both the rationale writing and subsequent validation tasks.
Rationale Writing We take 1,200 questions from the training set of ReClor. As with the second phase of the qualification task, we present workers with context, question, and four options marked either correct or incorrect, and then ask workers write rationale sentences for each option. Of these qualified individuals, 50 were actively en- gaged in this task. We collect 4,800 rationales in total and send them to the rationale validation step.
Subquestion Construction Coupling the vali- dated rationales with generated question texts, we construct most four subquestions for single main question. Each subquestion corresponds each of the four answer options in the main ques- tion. The four answer options of the subquestions are identical to the four rationales written for the main question. The correct answer option of a sub-

---

question is the rationale written for the option that the subquestion is made from.
# Main / Sub Questions
943 3,003
# SubQ / MainQ
3.18
A subquestion must have four validated ratio- nales to compose the multiple-choice format. How- ever, when we look at a main question, all four rationales are not always valid, which could largely decrease the number of possible subquestions. To mitigate this issue, we create subquestion even if three out of the four rationales are valid, by re- placing the invalid rationale with the "None of the above choices" option. Through this process, we obtain 3,824 subquestions. We discard a main ques- tion if it has no valid subquestions.
Selective / Eliminative (S/E)
785 2,218
Avg. context length
73.8
Avg. question length
31,4 (15.5)
Avg. option length
23.5 (17.7)
Avg. correct option length
24.0 (18.6)
# Question vocabulary
8,843 (1,085)
# Option vocabulary
9,849 (9,652)
# SubQ w/ "None" (# answer)
1,102 (222)
Table I: Dataset statistics of our RULE dataset. S/E indi- cates the numbers of two types of subquestions written about the correct (sclective) incorrect (eliminative) options of their main questions, respectively. The ques- tion and option lengths of the main questions are sepa- rately reported in parentheses for comparison. "None" denotes "None the above choices."
3.4 Human Validation
As the final step of our data collection, we validate the answerability of the subquestions by humans. Despite the ensured specificity of rationales, the complexity of the subquestion texts could poten- tially make the subquestions unanswerable. To address this issue, we ask three workers to answer each subquestion to evaluate its human answer- ability. subquestion considered answerable at least two workers answer it correctly, or if all workers select "None of the above choices." In the latter scenario, we replace the correct answer in the question with "None of the above choices." This process results in 3,003 answerable subquestions with 943 main questions. We expect the number of questions our dataset can demonstrate statistical power for meaningful model benchmarking and comparison (Card et al., 2020).
sive evaluation of the models tested on ReClor, we use all of our main questions and subquestions as a test set. Our hypothesis is that if the models can effectively generalize to understand the rationale behind the correct answer, they should exhibit a similar degree of performance on both the main questions and subquestions.
Evaluation Metries In addition to the simple ac- curacy over the main questions (MainQ Accuracy) and subquestions (SubQ Accuracy), we calculate the accuracy across the subquestions written for the correct and incorrect original options (Selective and Eliminative SubQ Accuracy), respectively. We also calculate the Consistency score to see how of- ten model answers both the main question and all of its subquestions correctly and thereby shows the comprehensive capability critical reasoning. Because the SubQ accuracy is micro average, we also report macro average for reference (MainQ- wise SubQ Accuracy). To compute these scores for humans, we take majority vote of the three labels for each main question and subquestion.
We then ask different workers to answer the ques- tions, collecting three additional labels for each question to measure human accuracy.
3.5 Dataset Statistics
Table 1 shows the dataset statistics. Compared to the main questions (ReClor), our subquestions have longer questions and answer options. The subques- tions that have "None of the above choices" as the correct answer comprise 7.4% (222/3,003) of the dataset, which is comparable to a similar multiple- choice reading comprehension dataset (6.7% in CosmosQA: Huang et al., 2019). We also report the crowdsourcing details in Appendix C.
4.1 Models and Settings
The models we evaluate are either in the fully- finetuned setting on the training set of ReClor (ex- cluding our main questions), few-shot of ReClor, and zero-shot that uses only the task instruction.
4 Baseline Performance on RULE
We measure the baseline performance of recent state-ofthh-ar ahe models on our dataset. Because the main purpose of our dataset is to perform an exten-
Fully-Finetuned Models We use DeBERTa-v3 (large; He et al., 2023) and UnifiedQA-v2 (base, large, and 3B; Khashabi et al., 2020, 2022). Both

---

models are reported exhibit strong generalization performance on QA datasets.
only understand part of the necessary rationales for the comprchension process.
Although it is not our intended task setting, when we use a part of the subquestions for in-context learning, the highest SubQ accuracy is 70.1% % by InstructGPT in the five-shot setting. This result is still below the human accuracy by noticeable margin. Interestingly, the in-context learning on subquestions is not helpful for smaller models such as Vicuna 7B and 13B.
Few- and Zero-Shot Models We include recent LLMs such as FLAN-T5 (XXL; Chung et al., 2022), Flan-U1.2 (20B; Tay et al., 2023), Vi- cuna (7B and 13B; Chiang et al., 2023), LLaMA
2. (7B to 70B; Touvron et al., 2023b), Mis- tral (7B; Jiang et al.. 2023) and InstructGPT (text-davinci-000) Ouyang et al., 2022). In the few-shot setting, the input prompt has five Re- Clor exemplars. Because some models only accept a limited length of input, we only report one-shor results of those models. For reference, we report few-shot results using RULE examples. The zero- shot prompt only has the task instruction. We also include Chain-of-Thoughtt (CoT; Wei et al., 2022) and zero-shot CoT (Kojima et al., 2022) of Instruct- GPT. providing the models with explanatory exam- ples to potentially enhance their performance. In COT, the prompt includes ReClor exemplars each of which followed by the rationale of the cor- rect answer option that is collected in this study. Appendix D shows examples of our CoT prompt. In the few- and zero-shot settings, we follow the test split approach used by Ravichander et al. (2022) and split our dataset into five disjoint sets measure the variability of models' performance. Appendix E describes the details.
Looking at the best Selective and Eliminative SubQ Accuracies, we find that although the for- mer accuracy (five-shot LLaMA 2 70B, 90.0%) is close to the human performance, the latter accuracy (zero-shot Flan-UL2, 59.1%) is significantly below the human performance (78.9%). This contrast shows that answering the eliminative subquestions is difficult for the models, highlighting the limited capacity of LLMs: Even if the models can choose the correct answer option, they may not understand why incorrect answer options should be refuted. Consistency and MainQ-wise SubQ Accuracy also conform to this trend. Although the con- sistency by humans is not high (52.9%), proba- bly owing to the difficulty of the subquestions, a large margin still exists between the human con- sistency and the best consistency by InstructGPT (18.2%). MainQ-wise SubQ Accuracy provides bit more intuitive observation: The best model answers only 64.3% of the subquestions per main question, although humans get them wrong less often (81.5%). We report the detailed number of MainQ-wise SubQ Accuracy in Appendix F. Contrary to our expectations, CoT does not im- prove the performance of InstructGPT. Rather, it leads to a decline in the MainQ and SubQ accu- racies. This result is consistent with findings on the unreliable nature of CoT (Wang et al., 2023; Turpin et al., 2023), which may be exposed by the complexity of critical reasoning.
4.2 Results
Table 2 presents main results. In the fully- finetuned setting, we observe that the SubQ accu- racy does not significantly exceed the chance rate (25.0%), which is far below the zero-shot perfor- mance of UnifiedQA-v2 as well as the human per- formance. This degradation may be due to overfit- ting to ReClor examples, by which the models rely heavily superficial features of answer options that are not useful in answering the subquestions. In our dataset, a group of subquestions shares the same set of four rationales, which requires that the models closely examine the question texts.
Does the Model Answer "None of the above choices" Questions Correctly? Some of our sub- questions contain "None of the above choices," which might make the questions challenging. In particular, the model performance on this type of question might be strongly affected by the in- context learning of exemplars. To investigate this hypothesis, we calculate the accuracy of the sub- questions that include the "None" option. In the five-shot InstructGPT using RULE examples, we find that although the model achieves 62.7% ac-
In the few- and zero-shot settings, we observe that the highest accuracy is 80.3% on the main questions by ILaMA> 2 70B with five-shot exem- plars of ReClor and 65.7% on the subquestions by Flan-UL2 in the zero-shot setting. Both the MainQ and the SubQ accuracies are lower than the human accuracy by large margins (A 11.2%, 16.9%), highlighting severe limitation in the models' ratio- nale understanding; in most cases, the models may

---

MainQ Ace.
SubQ Acc.
Selective SubQ
Eliminative SubQ Acc.
MainQ-wise SubQ Acc.
Model
# Param
Consist.
Fully Finenieed on ReClor
DEBERTA V3- LARGE
304M
66.0
33.1
56.1
25.0
2.4
32.8
UNIIEDQA-V2- ---SEBAS BASE
220M
40.5
25.8
21.3
27.4
0.7
26.0
UNHHEDQA-V--AAE
770M
57.7
25.0
19,9
26.8
1.4
24.7
UNIREDQA-V2-3B
3B
66.8
25.3
21.8
26.6
1.4
25.2
Sive-Shot ReClor
VICUNA 13B
13B
46.2
50.0+4.4
78.2_3.0
40.1
5.6
49.4
FLAN-UL.2
20B
58.5
65.5
88.0_1.0
57.615.4
16.9
64.3
INSTRUCTGPT
N/A
7..8
65.3
88.4_2.5
57. 21.5
18.2
INSTRUCTGPT COT
N/A
67.8
63.212.1
88.5
54.2
17.2
61.8
LLAMA2 13B
48.5
44.6+a.2
75.3-a.4
33.8+4.0
5.3
LLAMA2 70B
70B
80.3
60.01..0
=0.0
49.4
MISTRAL
+9.9
55.3+a.4
83.6-3.4
45.4+a.6
Five-Shot on RULE (for reference)
VICUNA 13B
43.9+1.3
44.2+2.7
72.6-2.6
34.2+2.6
4.1
44.0
FLAN-UL2
20B
57.940.2
66.024.9
87.7-4.6
58.4=5.0
INSTRUCTGPT
70.210.4
70.1
90.0_3.5
63.0
LLAMA2 13B
13B
47.743.0
46.344.00
80.0=2.1
34.424.7
5.1
47.1
ILAMA2 70B
70B
78.9
64.014..
90.6_2.5
54.625.5
MISTRAL
7B
58.2+1.6
47.5
88. =3.0
46.7
9.4
57.2
Zero-Shot
UNIREDQA -V2-BBB
3B
45.5
47.9+1.1
71.6
39.4=2222
5.7
47.8
UNIREDQA-V2-111
11B
55.2
57.3+2.7
74.8
51.1
9.7
56.5
FLAN-T5-XXX
11B
60.0
64.3+4.0
86.2-6.4
56.5+3.3
14.7
63.4
VICUNA 13B
13B
44.2
49.5
77.1-1.7
39.7
6.2
49.4
FLAN-UL2
20B
56.2
65.715.2
84.5_4.4
59.1
14.7
64.2
INSTRUCTGPT
N/A
64.1
62.8+2.2
89.9
53.222.1
15.5
61.8
INSTRUCTGPT+ CoT
N/A
63.8
62.311.0
89.6
52.611.5
14.2
61.2
LLAMA2 13B
13B
43.8
44.4
75.3
33.5
4.7
44.5
LLAMA2 70B
70B
70.8
58.0+3.7
88.1-3.4
47.3+4.0
14.1
57.3
MISTRAL 7B
7B
$4.0
55.9
85.9=2.0
45.3+3.6
8.6
55.0
HUMAN
91.5
82.6
93.0
78.9
52.9
Table 2: Model performance
RULE dataset consisting of the main questions (MainQ) and subquestions (SubQ). We report the accuracy for the subquestions written about the correct option (Selective SubQ Acc.) and incorrect options (Eliminative SubQ Acc.) of the main questions. Consistency holds only when the model answers both the main question and its subquestions correctly. InstructGPT is text dax-t-dvvinn--00
Batch
Acc.
# None in shot
None Acc.
option the correct answer and 13.7% accuracy otherwise. These results demonstrate that using exemplars helps to answer those questions to some extent but not significantly. Table 3 reports the accuracy of five-shot InstructGPT across the five batches.
#1
70.1
0
10.3
#2
69.7
0
25.9
#3
72.9
0
0.0
#4
71.3
1
43.8
#5
66.0
1
40.6
We report the complementary results of the main experiment in Appendix G, in which the one-shot setting does not improve the model performance consistently. Appendix H shows the SubQ accuracy only for the main questions the models answer correctly. Appendix shows the performance plot across the question and option length.
Avg.
70.1
0.4
32.0
Table 3: Accuracy of the subquestions that have "None the above choices" as correct answer (None Acc), compared that of all subquestions (Acc). None in shot indicates how many "None" examples are included in the few-shot prompt for each test split.
Analysis
curacy for the subquestions that have the "None" option, it shows 32.0% when "None" is the cor- rect answer. This low accuracy is decomposed into 40.9% accuracy if the prompt includes the "None"
To qualitatively investigate the models' behavior observed in Section 4, we aim to answer the fol- lowing research questions.

---

Why Are the Eliminative Subquestions Diffi- cult? As discussed in the previous section, we find a performance discrepancy between the selec- tive and eliminative subquestions. We attribute this discrepancy to two potential reasons. First, the climinative subquestions are inherently complex because of the negation included in their question text, which the models may find difficult to handle (Ravichander al., 2022). Second, the model may lack the ability to comprehend why certain options are incorrect, which partially supported by stud- ies that highlight the susceptibility for distractors in the multiple-choice QA (Si et al., 2021).
Direct
Indirect
Total
Contextual
37 37/48 /4
28 / 22
65 / 69
External
22/20
13 / 11
35 / 31
Total
/9
41 333
100
Table 4: Annotation results of rationale types on 100 examples randomly sampled from all subquestions (left) and from the error examples by InstructGPT (right).
in the context, the model rationale looks convine- ing and close to the human rationale. Among the 15 examples where humans lose, we find five ex- amples unsatisfactory to validate the target option, implying that approximately 10% of unreasonable rationales are potentially included in our dataset.
To distinguish between the difficulty of compre- hending complex questions and that of refuting rel- evant alternatives in the climinative subquestions. we develop a follow-up task, rationale alignment. this task, given context, the main question, one of the main options, and four rationales, the model selects one out of the four rationales that validates the correctness of the given option. We use Instruct- GPT in the five-shot setting and report the average results from five different prompts. Appendix J provides the input prompt.
What Types of Reasoning are Required in the Rationale Understanding? To qualitatively an- alyze the collected rationales, we first sample 100 subquestions to annotate reasoning types. We de- fine two dichotomies: direct/indirect and contex- tual/external. Direct reasoning occurs if a rationale involves explicit description for the certification of a target option's (in)validity, whereas indirect reasoning only provides relevant facts for the va- lidity. Context reasoning includes facts (or their interpretation and summarization) described in the context, while external reasoning is pertinent to commonsense and norms that are not described in the context. For comparative error analysis, we also sample 100 subquestions among those that InstructGPT answers incorrectly.
Because the subquestion text is not used in this task, we expect that the results are not affected by the complexity of subquestion texts. The result is 89.7% and 31.5% accuracy for the correct and incorrect answer options, respectively, showing distinet difference between them. This discrepancy suggests the model's serious deficiency in compre- hending eliminative rationales.
Is the Model Better at Writing Rationales than Humans? Given that CoT does not improve the model performance, we are interested in the quality and potential usefulness of model-generated ratio- nales compared to our human-written rationales. We use similar prompt to that used in our CoT setting, instructing InstructGPT to generate ratio- nales for 50 options. We then randomly shuffle the order of human-written and model-generated rationales, and manually annotate which rationale is better in terms of necessity and specificity. The result is 35 wins by humans and 15 wins by the model among the 50 comparisons, showing that the human-writter rationales are likely to be more detailed and supportive than the model-generated ones. In particular, we find that the model ratio- nales struggle to capture the implicit rationale nec- essary for certifying the validity of the target option. When the rationale is explicit and described well
We report our annotation results in Table 4. The number of the direct and contextual rationales is the largest among the other types, which further increases when we look at the error cases of In- structGPT. We find that our dataset covers a suffi- cient number of indirect and external reasoning, i.e., various modes of rationale understanding. Error ex- amples for the four reasoning types are reported in Appendix K. Although we also examine the reason- ing types originally labeled in the ReClor dataset, we do not observe any remarkable trends in the subquestion accuracy (Appendix L).
Do the Rationales Help the Model to Answer the Main Questions? Because the collected ratio- nales are expected support the decision of select- ing and eliminating answer options, we investigate whether adding the rationales to the main questions improves the performance in the fiye-shot Instruct-

---

Limitations
Input
Accuracy
Context
72.2
We recognize the following limitations in this study.
Selective Rationale
91.4
Eliminative Rationale
66.0
Task Format In this study, we focus the multiple-choice QA task. This task format allows us flexibly ask about various linguistic phenom- ena and human reasoning by selecting and elimi- nating alternatives, and we consider solving such a discriminative task would be a minimal require- ment for human-like linguistic behaviors. However, it has an inherent limitation in assessing the ability of natural language understanding. For example, cannot evaluate the models' ability to produce an intended output.
Both
89.6
Table 5: MainQ accuracy of InstructGPT that uses the selective eliminative rationales the input.
GPT. We append the rationale the context, main question, and four options with the Ra i onale: label. The results are shown in Table 5. We ob- serve an improvement when the selective rationale is added; however, degradation occurs when we add the eliminative rationale, even if it is provided with the selective rationale. This result adds insight the observation by Sun al. (2022), showing that the model cannot eliminative rationales for answering main questions and becomes con- fused by those rationales. We also investigate the context-ablated setting in Appendix M.
A Annotation Analysis We conduct the annotation analysis in Section 5, in which we define the rea- soning types and manually review the sampled ex- amples. Although we make our annotation data and guideline publicly available for ensuring the reproducibility of annotation results, the results of our annotation analysis inevitably involve our subjective judgments.
6 Conclusion
Source Dataset We create our auxiliary ques- tions on top of an existing English logical read- ing comprehension dataset, ReClor. Although our methodology of the data collection (i.e., writing the rationale for selecting and eliminating alterna- tives) is widely applicable other datasets and languages, using the single dataset in the single language would limit the generalizability of our findings.
We construct dataset to evaluate the models' abil- ity of critical reasoning in logical reading compre- hension. We crowdsource free-form rationale for main questions taken from an existing dataset and use an L.M to generate subquestion texts. Result- ing questions ask about the underlying rationales for why certain answer option should be selected and the others should be eliminated. We find that LLMs are particularly bad answering climina- tive subquestions, highlighting that those models do not necessarily have the comprehensive ability critical reasoning. For future work, we will de- velop a more efficient pipeline for data collection and facilitate better rationale generation by LLMs.
Acknowledgments
We would like thank the anonymous review- ers for their helpful comments. This work was supported by JST PRESTO Grant Number JP- MJPR20C4 and JSPS KAKENHI Grant Number 17954.
Ethical Consideration
We use crowdsourcing in our data collection. We make sure to be responsible to the crowdworkers and to make fair compensation for their work. We do not collect any personal information other than worker IDs on the platform, which removed in our data release. Before the workers accept our tasks, we inform them of our purpose for the data collection. This study is approved by the internal review board of the authors' institutes.
References
Shourya Aggarwal, Divyanshu Mandowara, Vishwa- jeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. Explanations for Common- senseQA: New Dataset and Models. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 3050-3065, Online. Association for Computational Linguistics.

---

Mana Ashida and Saku Sugawara. 2022. Possible sto- ries: Evaluating situated commonsense reasoning under multiple possible scenarios. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3606-3630, Gyeongju, Republic of Korea, International Committee on Computational Linguistics.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Flazar, Ananth Gottumukkala, Nitish Gupta, Hammanch Hajishirzi, Gabriel Ihaarco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nel- son F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307-1323, Online. Association for Computational Linguistics.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In International Conference on Learning Representa- lions.
Robert Geirhos, Jörn-Henris Jacobsen, Claudio Michaelis. Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):66-6673.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Mor Geva, Daniel Khashabi. Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Mark Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. 2017. Developing, analyzing, and using distractors for multiple-choice tests in education: A compre- hensive review. Review of Educational Research, 87:003-4654117726533
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263-9274, Online, Association for Computa- tional Linguistics.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruc- tion of implicit warrants. In Proceedings of the 2018 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers). pages 1930- 1940, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing GPT-4 with 90%* Chat- GPT quality.
Pengcheng He. Jianfeng Gao, and Weizhu Chen. 2023, DeBERTav3: Improving deBERTa using ELECTRA- style pre-training with ding sharing. In The Eleventh International Confer- ence on Learning Representations.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- hert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros. Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Fdd H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-fineni language models. arXiv preprint 1141..
embed-
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense rea- soning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP--JCNLP)) pages 2391-2401. Hong Kong, China. Association for Com- putational Linguistics.
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hamnah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entail- ment trees. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 7358-7370, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Yinya Huang, Hongming Zhang, Ruixin Hong, Xiao- dan Liang. Changshui Zhang, and Dong Yu. 2022. Metal Metaloggie: Logical reasoning explanations with fine- grained structure. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 4698-477 4724, Abu Dhabi, United

---

Arab Emirates, Association for Computational Lin- guistics.
Intelligence, IJCAI-20, pages 3622-3628. Interna- tional Joint Conferences on Artificial Intelligence Organization. Main track.
Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. 2020. R4C: A benchmark for evaluating RC systems to get the right answer for the right reason. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740-6750, Online. Association for Computational Linguistics.
Nikita Nangia, Saku Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, and Samuel R. Bowman, 2021. What ingredients make for effective crowdsoure- ing protocol for difficult NLU data collection tasks? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1221-1235, Online. Association for Computational Linguistics.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulmier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv preprint 2310.06825.
Timothy Niven and Hung- Yu Kao. 2019. Probing neu- ral network comprehension natural language argu- ments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1658,4664, Florence, Italy. Association for Compu- tational Linguistics.
Fangkai Jiao, Yangyang Guo, Xuemeng Song, and Liqiang Nie. 2022. MERIt: Meta-Path Guided Con- trastive Learning for Logical Reasoning. In Find- ings of the Association for Computational Linguis- fics: ACI. 2022, pages 3496-3509, Dublin, Ireland. Association for Computational Linguistics.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 07730-27744, Curran Associates, Inc.
Daniel Khashabi, Yeganeh Kordi, and Hannaneh Ha- jishirzi. 2022. UnifiedQA-v2 Stronger generaliza- tion via broader cross-format training, arXiv preprint 12359.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. UNIFIEDQA: Crossing for- mat boundaries with single QA system. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Abhilasha Ravichander, Matt Gardner, and Ana Maraso- vic. 2022. CONDAQA: A contrastive reading com- prehension dataset for reasoning about negation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8729-8755, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence composi- tion. In The Foirty-Fourth AAAI Conference on Arti- ficial Intelligence, pages 8082-8090. AAAT Press.
Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Henghui Zhu, Rui Dong, Deguang Kong, Juli- ette Burger, Anjelica Ramos, zhiheng huang, William Yang Wang. George Karypis, Bing Xiang. and Dan Roth. 2023. STREET: A multi task struc- tured reasoning and explanation benchmark. In The Eleventh International Conference on Learning Rep- resentations.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199-22213. Curran Associates, Inc.
Jieyu Lin, Jiajie Zou, and Nai Ding, 2021. Using ad- versarial attacks reveal the statistical bias in Proceed-
Swarnadeep Saha, Peter Hase, Nazneen Rajani, and Mo- hit Bansal. 2022. Are hard examples also harder to explain? study with human and model-generated explanations. Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages
chine reading comprehension models. ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 333-342, Online. Association for Computational Linguistics.
Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang. Yile Wang, and Yue Zhang. 2020. LogiQA: A chal- lenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty- Ninth International Joint Conference on Artificial
Swarnadeep Saha, Prateek Yadav, Lisa Bauer, and Mohit Bansal. 2021. ExplaGraphs: An explanation graph generation task for structured commonsense reason- ing. In Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing,

---

pages 7716-7740. Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2307.0928 .02288.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don't always say what they think: Unfaithful explana- tions in chain ohain-of-thought prompting, arXiv preprint 2305.04388.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou, 2023. Large language models can be easily distracted by irrelevant context. In Proceed- ings the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31210-31227. PMLR.
Boshi Wang, Sewon Min, Xiang Deng, Jiamning Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Proceedings of the ō1st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 2717-2725 Toronto, Canada. Association for Computational Linguistics.
Chenglei Si, Ziqing Yang, Yiming Cui, Wentao Ma,
Ting Liu, and Shijin Wang. 2021. Benchmarking ro- bustness of machine reading comprehension models. In Findings of the Association for Computational Lin-
guistics: ACL-IJCNIPP 2021, pages 634-644, Online. Association for Computational Linguistics.
Siyuan Wang. Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. 2022. Logic-driven context extension and data augmentation for logical reasoning of text. In Find- ings of the Association for Computational Linguis- fics: ACL 2022, pages 1619-1629, Dublin, Ireland. Association for Computational Linguisties.
Jiao Sun, Swabha Swayamdipta, Jonathan May, and Xuezhe Ma. 2022. Investigating the benefits of free- form rationales. In Findings of the Association for Computaational Linguistics: EMNIP 2022, pages 5867-5882, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Yi Tay, Mostafa Dehghami, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng. Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. U1.2: Unifying language learning paradigms. In The Eleventh Inter- national Conference on Learning Representations.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837. Curran Associates, Inc.
Stephen E. Toulmin. 2003. The uses of argument. Cam- bridge University Press.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LLaMA: Open and efficient foundation language models. arXiv preprint 2302.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. ReClor: A reading comprehension dataset requiring logical reasoning. In International Confer- ence on Learning Representations.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Bahaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas
Wanjun Zhong, Siyuan Wang. Duyu Tang. Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang. Jian Yin, Ming Zhou, and Nan Duan. 2022. Analytical reasoning of text. In Findings of the Association for Computa- tional Linguisties: NAACL 2022, pages 2306-2319 Seattle, United States. Association for Computational Linguistics.
A Crowdsourcing Instructions and Examples
Figures 16 23 show the instructions and exam- ples we present to the crowdworkers. Figures 16 to 20 illustrate the rationale writing task, Figures 21 and 22 illustrate the rationale validation task, and Figure 23 illustrates the human validation task.

---

B Question Generation Prompt
Figure 14 shows an example of our prompt used for generating subquestions in Section 3.3.
C-2
C10
NH2
85
61
Crowdsourcing Details
N=3
63
106
To access pool of crowdworkers, we used Ama- zon Mechanical Turk. The crowdworkers who took the qualification test are based in the United States, United Kingdom, or Canada, have approval rate of at least 98%, and have least 1,000 approved tasks. We ensure that the average payments exceed $12.00 USD per hour for each task. The ratio- nale writing task costs $2.00 per main question (estimating that takes seven to ten minutes to write the rationales), the rationale validation task costs $0.30 per rationale (one minute), and the hu- man validation task $1.50 per five questions (five minutes). The rationale writing tasks, rationale validation tasks, QA validation tasks, and human performance tasks are taken by 48, 39, 52, and 24 workers, respectively. We use the crowdsourcing tool used in Nangia et al. (2021).
Nidd
171
100
300
400
N-11
43
C-3
Ca1
Nm=
Ned
133
*MainQ
Figure Distribution of correctly answered subques- tions (C) out of the total number of subquestions (N), for both InstructGPT (top) and humans (bottom).
D Chain-of-Thought Prompt
Figure 15 shows an example of the prompt used in our chain-of-thought experiment. We insert the rationale between the Answer: : label and the cor- rect option label, with an expectation that would help the model (InstructGPT) select the correct op- tion.
Proqunccy
Test Split Setting
QuestionLenghh (etakens)
The in-context learning performance of LLMs may vary depending on the exemplars of the prompt, but it incurs high computational cost (or finan- cial cost for proprietary models) if repeatedly evaluate the models on the entire dataset using var- ious sets of different exemplars to take the average performance. Because of this cost limitation, we follow the test split approach used by Ravichander et al. (2022), splitting our dataset into five disjoint sets and testing the models on each set with differ- ent exemplars to measure the performance variance across the disjoint sets. Note that we do not split the set of the main questions, because has only 943 examples; hence, in the few-shot setting, we take the average across five runs on main questions. the few-shot setting using ReClor, we sample questions disjointly from its training set, whereas
Figure 4: Distribution of the question length (#tokens) of the main questions.
in using RULE, the exemplars are sampled from the corresponding disjoint set.
F MainQ-wise SubQ Results of InstructGPT
Because single main question has multiple sub- questions in our dataset, we report the detailed numbers correctly-answere SubQ by Instruct- GPT in Figure 3.

---

MainQ Acc.
SubQ Acc.
Selective SubQ
Eliminative SubQ Acc.
MainQ-wise
Model
# Param
Consist.
Acc.
Shot ReClor
LLAMA 7B
7B
25.8
28.6+7.1
39.8-13.1
24.6
0.8
28.2
LLAMA 13B
13B
38.7
3..3
63.6=40
26.623.6
2.9
36.6
VICUNA 7B
7B
33,4+2.6
38.6+3.a
61.4-6.8
30.5+3.4
2.8
38.3
LLAMA2 7B
7B
36.4
35.2+5.9
63.6_6.5
25.1
1.7
35.6
One-Shot ReClor
UNITEDQA-VV-BASS
220M
2..4
34.7
42.8-7.7
31.926.2
0.7
34.6
SNIEEQQ-V2--LAS
770M
27.1
28.0
2..2
27.624.9
0.0
27.6
UNIREDQA-VV
31.32332
26.2+2.8
26.0_0.8
26.221.4
0.3
26.1
UNIREDQA-V2-IIB
44.1
37.2
53.8-16.8
31.4.4.4
2.0
36.5
FLAN-T5-XXL
61.3
63.7+a.
85.9-2.6
55.9+3.9
14.0
62.5
FLAN-UL2
58.0.0.3
66.0
87.4_3.9
58.5
17.7
LLAMA
26.6+0.9
32,4+4.2
50.7
26. +33.
32.5
LLAMA
32.7+2.4
33.8
$6.0
26.0
LLAMA 33B
56.1+1.0
49.3+4.3
80.1
38.5+4.33
7.4
49.6
LLAMA 65B
65.2
52.643.4
85.2-1.3
41.1
9.4
VICUNA 7B
7B
35.8
38.9122
65.3_3.1
29.5
38.1
VICUNA 13B
42.5
45.243.1
72.1=3.3
35.824.0
4.2
45.1
INSTRUCTGPT
67.810.
64.6119
87.8 2.0
56.3
17.5
INSTRUCTGPT + CoT
N/A
64.3
64.3+14
88.8-13
55.7
15.4
62.5
LLAMA2 7B
7B
35.011.1
34.812.4
61.7-4.2
25.222.6
2.2
35.2
LLAMA2 13B
13B
46.4
43.7+2.9
72.6-4.6
33.4=2.8
4.7
43.6
LLAMA2 70B
70B
77.2+0.2
61.34a.7
90.0-1.4
51.2
20.0
61.0
MISTRAL 7B
7B
52.6+1.4
53.4+2.a
81.9-25
43.4+2.7
7.4
52.7
Five-Shot on RULE (for reference)
LLAMA 7B
7B
29.1
34.912.5
64.3_4.
24.5
1.5
35.5
ILAMA 13B
13B
36.843.5
35.6+2.6
68.1_3.0
24.2
2.4
36.0
VICUNA 7B
7B
35.011.1
39.913.9
60.2_6.9
32.72444
3.2
39.8
LLAMA2 7B
7B
37.8
32.0+
62.3
21.1
1.5
32.4
One-Shot on RULE (for reference)
UNIFIEDAA- A- BASE
220M
27.7+2.7
36.5+1.
38,5
35.822.3
1.6
36.9
AAEEDDQA-V--AAEE
770M
28.3+1.4
27.4+1.
27.4-10.2
27.4=3.0
1.0
27.7
AV2-33
3B
35.0
30.1
35.8
28.224.0
2.0
30.6
UNI-IIDA-V2-IIB
11B
42.647.1
37.411.7
47.5
33.9
3.1
38.0
FLAN-T5-XXL
11B
60.610.5
64.1
85.8
56.523.9
14.0
63.2
FLAN-U1.2
20B
57.610.4
66.014.
87.4
58.5
17.2
64.9
LLAMA 7B
7B
28.212.4
32.9125
48.3
27.4
1.6
33.6
ILAMA 13B
13B
30.013.2
32.9113
50.9
26.6
1.7
33.4
LLAMA 33B
33B
53.3
48.4+4.6
79.7-14
37.3
6.0
48.4
LLAMA 65B
65B
62.8
52.8
84,9
41.5
7.5
52.0
VICUNA 7B
7B
34.7+1.8
41.2+2,4
65.3-64
32.6
3,4
41.2
VICUNA 13B
13B
40.7+2.5
41.741.
67.3_48
32.7-1.8
3.5
41.8
INSTRUCTGPT
N/A
65.4+1.9
66.5+1.
89.0
58.5+1.2
19.5
65.5
INSTRUCTGPT CoT
N/A
64.312.5
64.3
88,8_13
55.7
15.4
62.5
ILAMA2 7B
7B
32.813.3
33.4117
56.1
25.4
34.0
LLAMA2 13B
45.0.5.5
40.942.9
70.6-44
30.3
41.1
ILAMA2 70B
70B
76.020.2
61.5
90.4_a.1
51.4
19.2
61.1
50.8+1.1
53,8+a.
84.0
43.2+a.6
Zero-Shot
UNIFEDQA- -V2 BASE
220M
42,2+1.
48.5
39,9+1.3
UNIFIEDAA-V ---V2-LARGE
770M
42.9
55.0=4.9
18.5
3.3
LLAMA
7B
27.7
27.4+4.1
38.2-4.2
23.6+4..
27.1
LLAMA
31.7
36.043.3
59.3-2.1
27.8
VICUNA 7B
7B
36.7
40.5122
73.1
28.9
2.4
40.1
LLAMA2 7B
7B
32.3
35.949.9
66.4_4.7
25. 23.4
2.5
36.3
HUMAN
91.5
82.6
93.0
78.9
$2.9
81.5
Table 6: Complementary results of the model performance on our dataset including the models in the one-shot setting and omitting those the five-shot and zero-shot settings.
G Complementary Few-Shot and Zero-Shot Results
In Table 6, report the complementary results of few-shot settings, including the models on the

---

Frequency
Accuracy
Options Langth [etokems)
Figure 8: Distribution of the option length (#tokens) of the main questions.
InstructSPT Human
10
15
25
30
Question Length (#tokens)
Figure 5: Comparison between the model and human accuracy and question length for the main questions.
0.8
Accuracy
0.6
0.4
Pravency
0.2
3.0
20
Question Length (ttokkens
Options Length (#tokens)
Figure 6: Distribution of the question length (#tokens) of the subquestions.
Figure 9: Comparison between the model and human accuracy and option length for the main questions.
1.0
0.8
Accuracy
6.7
Accuracy
0.4
0.5
0.2
InstrucuPP Human
0.0
Human
20
40 50
10
Question Length (#tokens)
Options Length (#tokens)
Figure 7: Comparison between the model and human accuracy and question length for the subquestions.
Figure 10: Comparison between the model and human accuracy and option length for the subquestions.
Instruction
Corr. Opt.
Incorr. Opt.
shot setting. We also report the results of LLaMA (7B 65B; Touvron et al., 2023a) for reference.
Yes
89.7 (784)
31.5 (2,218)
No
88.7 (781)
28.1 (2,213)
H
Main Results of the Subquestions for the Correctly-Answere Main Questions
Table 7: Result of the rationale alignment task with and without the task instruction.
Table shows the main results the model perfor- mance on the subquestions for the main questions

---

K Reasoning Type Annotation
Table 10 shows examples of reasoning types we define in the annotation analysis. See Table 11 for full example that has the main question and two subquestions.
Rrouunnc
L ReClor Reasoning Types and Subquestion Aceuracy
Options Length (étokens)
Figure 11: Distribution of the option length (#tokens) of the subquestions.
Figure 13 shows the relation between the subques- tion accuracy and the reasoning types defined in the original ReClor dataset. Although we do not ob- serve significant performance differences, higher accuracy in Match Structures, Evaluation, Strengthen, and Weaken reasoning, and lower ac- curacy Sufficient Assumptions, Technique, and Role reasoning.
that are conrectly answered by the model. Overall, observe similar trends to the main results with the standard SubQ accuracy. Interestingly, the mod- els' SubQ accuracies do not significantly improve even when we focus only on the corsectly-answered main questions.
M Context-Ablation Analysis
We try to answer the question "Does the context help in answering subquestions?" in the context- ablation setting. By removing the context, we an- alyze the model performance on the subquestions (and the main questions for reference) to see the dependency between question texts and answer op- tions. The results in Table 9 show the performance reduction by approximately 4 points in the zero- shot setting and no reduction in the five-shot setting, This result implies question texts depend answer options to some extent, which potentially makes the subquestions difficult for the models, given the first analysis in this section.
I Relationship between Question and Option Length and Model Performance
In Figures 4 to 10, we plot the distribution of the questions and options length and the model perfor- mance according those lengths.
J Rationale Alignment Task
the rationale alignment task, we test Instruct- GPT in the five-shot setting. Similar to the main experiment, we report the average results from five prompts. Each prompt is composed of five exem- plars, with two exemplars presenting the correct option and three exemplars presenting the incorrect option. Figure 12 shows example of our prompt.
N
Similarity of Rationale with MainQ Option
In our process to validate specificity, even a ratio- nale has the same meaning as the MainQ's option, we can not exclude This implies that some ratio- nales might have the similar meaning as the option and not serve as valid rationale. To examine this potential issue, we sample 50 random questions from both the selective SubQ and the eliminative SubQ. We then count how many of these rationales are semantically similar the MainQ's option. We found three such instances in the selective SubQ and one in the eliminative SubQ, which are shown in Table 12.
The results with and without the task instruction are shown in Table 7. The performance gap be- tween the correct and incorrect options implies that such advanced models may simply infer the cor- rect answer without properly discriminating against incorrect options. Such a situation raises two is- sues: (1) the inability to reason logically like a human, and (2) the limitations of ability measure- ment using distractors. The first issue suggests that the model may not be able to make a clear distinction between what is correct and what is in- correct. The second issue that the alternatives in the multiple-choice QA task are generally expected to distinguish between test takers with and without sufficient knowledge (Gierl et al., 2017), but such an expectation may not be met in our dataset.

---

MainQ Ace.
SubQ Acc.
Selective SubQ
Eliminative SubQ Acc.
-ainQ-wise SubQ Acc.
Model
# Param
Consist.
Fully Finenieed on ReClor
DEBERTA -3- LARGE
304M
66.0
33.1
60.4
22.5
3.7
32.2
UNIIEDQA-V2 BASE
220M
40.5
25.8
19.7
26.7
1.8
25.0
UNIHEDQA-V--AA LARGE
770M
57.7
25.0
17.3
27.7
2.4
24.6
UNIREDQA-V2-3B
3B
66.8
25.3
19.9
25.7
2.1
24.1
Sive-Shot ReClor
FLAN-UL2
20B
58.5
66.3
89.6_4.6
57.2
28.8
65.1
LLAMA 7B
7B
2..8
28.5
34.3 34.3
24.2
3.4
27.2
LLAMA
38.7
37.024.0
65.0_6.7
25.9
37.2
LLAMA 33B
33B
58.511.2
47.7
7..5
38.325.8
10.6
47.3
LLAMA 65B
+9.1
54.4+1.7
85.5-2.8
46.9+6.6
16.0
54.1
VICUNA 7B
33.4
40.0
62.7=6.9
30.0
VICUNA 13B
46.2+0.7
48.5
76.0=8.8
42.2+6.4
48,1
INSTRUCTGPT
N/A
71.8
64.1
89.2=3.8
60.7=6.1
INSTRUCTGPT + CoT
67.8+0.5
62.9+3.1
89.5-4.2
55.8+3.0
24.9
LLAMA2 13B
48.5
45.8
79.1=4.0
33.1
11.1
LLAMA2 70B
80.310.4
59.912.5
90.7
50.5
MISTRAL
7B
59.9.0.9
54.7
84.5_2.8
46.822.8
15.0
53.7
Five-Shot on RULE (for reference)
FLAN-UL2
20B
47.940.2
67.2+6.2
89,4-14
57. #3.6
30.9
66.2
LLAMA 7B
7B
29.1
34,613.2
66.2
23.7
5.4
34.9
LLAMA 13B
13B
36.8
35.1
67.4-6.3
24.4
6.6
35.0
LLAMA 33B
33B
53.6
48.5+6.0
78.5
36.8
10.6
47.6
LLAMA 65B
65B
66.2+0.7
57.2+6.1
86.4-1.1
53.7+10.3
18.3
56.0
VICUNA 7B
7B
35.0
40.343.1
61.4_8.6
32.4-4.8
9.5
40.8
VICUNA 13B
13B
43.9
43.912.
73.0_3.1
34.223.7
9.4
43.4
INSTRUCTGPT
N/A
70.220.4
70.243.5
91.0-
64.322.4
32.7
69.3
INSTRUCTGPT + CoT
N/A
67.810.5
62.913.1
89.5_12
55.823.6
24.9
61.5
LLAMA2 13B
13B
47.7
46.9
79.4
33.644.6
11.0
47.0
LLAMA2 70B
70B
78.9
63.7+4..
90.8
55,2+9.2
26.5
63.0
MISTRAL 7B
7B
58.2 58.2+1.6
56.2
88.0
49.4
16.3
55.6
Zero-Shot
UNIFIEDA- A-V2-BASE
220M
30.4
43.0+2.9
51.0
39.8
8.7
43.4
UNIFIDQA-Y22 A-EED------LRGEEE
770M
41.4
43.9+4.1
59,9
39.0
7.9
42.5
UNFEEDQA-V2-3B
3B
45.5
49.3
75.4
38.923.1
12.6
49.5
UNIRIEQQA-V--IBB
11B
55.2
56.8129
77.6
53.0
17.5
55.9
FLAN-T5-XXL
11B
60.0
63.2134
87.3
59.22446
24.6
62.7
FLAN-U1.2
20B
56.2
65.315.9
86.0
60.5
26.2
63.7
LLAMA 7B
7B
27.7
27.0+4.6
37.5
23.6+3.9
3.1
26.9
LLAMA 13B
13B
31.7
35.5
55.9-40
27.5
4.3
35.9
LLAMA 33B
33B
$4.5
50.2+4.6
81.8-4.3
41.4-6.5
12.5
50.2
LLAMA 65B
65B
52.1
47.5+2.2
80.1
38.0
10.4
46.6
VICUNA 7B
7B
36.7
40.8+2.8
77.5-2.
29.4-2.8
6.6
40.3
VICUNA 13B
13B
44.2
48.9
76.9_3.4
40.4-1.8
13.9
48,2
INSTRUCTGPT
N/A
64.1
61.9129
89.8_2.9
55.8
60.8
INSTRUCTGPT COT
N/A
63.8
60.9
89.1
55.913.2
59.7
I.LMA22 13B
43.8
45.5
77.6
33.3
45.4
LLAMA2 70B
70B
57.2+4.0
88.2-a.2
49.9+6.4
56.3
MISTRAL
54.7
85.1
47.1
15.9
HUMAN
91.5
82.9
92.8
79.3
57.8
81.6
Table Main results the model performance on our dataset focusing on the subquestions for the main questions that correctly answered by the model.
Selective SubQ
Eliminat. SubQ
Setting
MainQ
SubQ
0-shot
41.0-23.0
58.8 -8.8_..2
86.4_s.7
53.4-..8 64.7+1.7
5-shot
42.5_29.7
70.5
86.9_3.1
Table 9: Context-ablated accuracy. The subscript values indicate the accuracy gap against the full- alll-inpt setting.

---

In a given context you'll be givin quaiotion, answer, and four relionales. Your bask to iduntily the cationale that eaplinns the coneetress of the provided option the best. if the option is wrong, choose the rationalo that explains why it Convarsely, the option is correct, choose the rationale that expltins why it coreec.
Conteat: Teadhers should wrything
And
lrying
ignerance: Thersforn, teacher doess not know enswer question sludent has asked should not pretand
answer Question: The conclusion
property drawn
following assumed?
Incorrect Option
Question: The condiusion property drown which one the following assumed?
Option: Students' respect for teacher is independent of the amount of knowledge they sit bule to that teacher
Exemplars with Task Instruction
Rationalc0: ranking students respect for honesty not rolovant the conclusion teacher shoulent protand know an answer to question they don't
know the aninner
Rationalet: 1: The aesumption that students' reepect for the teacher baced how much knowledge the teacher has
Rationale2: conduuioo that teachers shouldri prefend know the snswer to quession that they don't know, so the sesumptio that student's respect for
Ieacher is intefinked to the slutinn's peroaived ko knowlegga of the leacher
Rationale3: The condussoo does not have anything lo do with a toacher being affectivee.
Answer: The arowor Rationale2
Context: Miguel has four farily members who plan come to his gradustion Sunday aftemoon. but is thely that only three of them will be allowed attend. Nomally graduddin in held in the footbal stadkum, where theme no limil on the number of family membera who can allend. However the ceremorry is relocaled to the gyneasum if rains, and each graduate rensivis just three admission fiikels for use by family members.
Question: The condlusion the argument is most strongly supported which
the following is assumed?
Correct Option
Option: The weather
likethood of Rationale0: No mention is made whether un needed spaces can be transferre betweon soudents, and so this cannot be assumed to impact the number of spaces aorlable to Miguel's la family
has indicated that
Sunday atternoon.
Rationale:: Abnormally large class size may not precluce Migusi from having more than three family members antnnd as the feotball stadium is possible venue and has limiaaion on the number who may alland
Rationale:: family member who cannot allend the graduction has relevvence how
alloweed to sllend.
Rationales: Rain would preduce the use of the stadium wiich has finit of the number family members allanding and foros the use of the gymnsskuu, which imits number sttanding
Answer: The answar Rationale3
[..]]
relared living But shythm of serfously people the city. that experionca equal stress. surveys people living country bacome
Berousse permils
country supposed
Test Instance
Question: The statements above, if true, provide the most support for which one of the following?
Rationale0: This passage kind of di spules this line of thinkin. living in the country should have a slower rhythm yet they experience the same amount strees as diy dweller
Rationalet: 1: This passaga is not saying this spacifoally, just that a natural rhythm might not have as many benofits as people want beliove.
Rationale2: The passage kind of focuses on both the thoughts that living in the country should be hoalthier but surveys show that is not the case.
RationaleS: Nono of the above cholces
Answer:
Figure 12: Example of the prompt used for rationale alignment task.
400
0.6
0.5
300
#SAMPLE
200
0.3
0.2
100
0
Match Strucurres
Evaluation
Strengthen
Weaken
Most Strongly Supported
Explain Resolve
Identify a Flow
Principle
Necessary Assumptions
ConclusionMai Point
Implication
Match Flaws
Dispute
Others
Sufficient Assumptions
Technique
Role
Figure 13: Accuracy of InstructGPT and reasoning types originally annotated in the ReClor dataset.

---

Reasoning
Passage
Question
Option
Correct Rationale
Type
Direct
Today' family is de- clining in its ability to carry I..]. There must he return to the traditional values of commit- ment and responsibilityy. Jerod: We ought to leave what is good enough alone. Contemporary families may be less stable than traditionally. but most people do find that be bad. [...].
Trisha:
Trisha and Jerod dis- agree over whether the institution of the family is
longer
tradi-
FALSE
Both TTrisha and Jerod agree that families are no longer traditional, this is not what the argument about.
Contextual
tional.
Direct External
A just government never re- stricts the right of its citizens to act upon their desires except when their acting upon their de- sires is direct threat the health property of other of its citizens.
Which one of the following judgments most closely con- forms the princi- ple cited above?
A just government would not censor writings of Shake- speare, but it could censor magazines and movics that criticize the govern- ment.
FALSE
A just government would not censor magazines and movies that criticize the govermment because these things do not threaten the health property of its citizens.
Indirect Contextull
Doctor: The practice of using this therapy to treat the illness cannot be adequately supported by the claim that any therapy for treating the illness is more effec- tive than no therapy at all. What must also be taken into account is that this therapy is expensive and complicated.
Which one of the following most accu- rately expresses the main point of the doctor's argument?
The therapy's pos- sible elfectiveness in treating the ill- ness is not suffi- cient justification for using it.
TRUE
Therapy's other costs must be considered be- fore enlisting the treatment as it is not cheap and not simple.
Indirect External
On average, corporations that encourage frequent social events in the workplace show higher profits than those that rarely do. This suggests that the EZ Corpo- ration could boost its profits by having more staff parties during business hours.
Which one of the fol- lowing. if truc, most weakens the argu- ment above?
Frequent social ovents in a cor- porate workplace leave employees with less time to perform their assigned duties than they would otherwise have.
FALSE Frequent social events in corporate workplace can re- energize employees, like lunch break does.
Table 10: Examples of the reasoning types with passage, question, option, the correctness of the option, and its human- written rationale.
Combine Question Optin
follows.
Exemplars with Task Instruction Test Instance
Question: Which one the falkwing is an assumplen asumppion required by the department skoe manager's argument?
Option: Either fow customers would want frce git wrapping or most customers woule want
Answer: Whrch of the colliwing reasoning jusiflees that the department store menager's argument recuires the assumptio that etteer few customers would want
froe gif wrapping or most customers would
argument
battany focusee on clem that gisoline powered cane bailety-poweeed electric
buildig
Answer: Why welid conduule that Kints
building more generating foc itcl?
Question: The argument's concusioo follows logically if which one the following assumed?
Option work science fiction cannot achivve greatnees unlees contains compelling charasters.
Answer: the argument's condlusion logically assumed work science fiction cannot achiove greatness unless contains
compelling
Question: Which the following conclusions be validyy drawn from the pessage?
Option: Unless Cherles branch peychology. Charlee should not offer sclution Patrick's behavional problem.
Answer:
Figure 14: Example of the prompt used to generate subquestions.

---

Paragraph: Trisha: Today's family is declining in its ability to earry out its functions of child-rearing and providing stability for adult life. There must be return the traditional values of commitment and responsibility. Jerod: We ought to leave what is good enough alone. Contemporary families may be less stable than traditionally, but most people do not find that to be bad. Contemporary criticisms of the family are overblown and destructive.
Question: Trisha and Jerod disagree over whether the institution of the family is Options:
1) valued by most people.
MainQ
2) changing over time.
3) adequate as it is.
4) no longer traditional.
Question: What is the source of the disagreement between Trisha and Jerod regarding whether the institution of the family is adequate as it is?
Options:
Selective SubQ
1) The argument does not mention value to the people.
2) Trisha is arguing that things were better with traditional families and Jerod is arguing that they good now, the argument is about the quality of the relationship now.
3) Both Trisha and Jerod agree that families are longer traditional, this is not what the argument is about.
4) None of the above choices.
Question: What evidence there suggest that Trisha and Jerod's disagreement over whether the institution of the family is no longer traditional is not valid?
Options:
Eliminative
1) Both Trisha and Jerod agree that families are no longer traditional, this is not what the argument is about.
SubQ
2) Trisha is arguing that things were better with traditional families and Jerod is arguing that they are good now, the argument is about the quality of the relationship now.
3) The argument does not mention value to the people.
4) None of the above choices.
Table 11: Examples of the main questions and subguestions in our dataset. The options in bold indicate the correct answer.
Question Type
Option
Rationale
Delays in the communication of dis- coveries will have chilling effect on scientific research.
Delays communicating discoveries would limit the time other scientists have to investigate and contribute. All the information in the passage in- dicates that Kimmy is affluent and renowned.
Kimmy is a highly compensated and extremely popular television and movic actress.
Selective
Before new therapeutic agents reach the marketplace, they do not benefit patients.
The passage states that new therapies aid patients only after they are introduced to the marketplace.
The speed of eye orientation corre- lates with intelligence, not overall health.
The speed at which one can orient one's eye to a stimulus has been closely associ- ated with overall health.
Eliminative
Table 12: Rationales that are semantically similar to the M'inQ's option in Selective and Eliminative SubQs.

---

should do anythin cause students respect them, students sense samoono trying ride gnorance. Therefore, teacher who does know the answer queston student has asked should not pretend know answer.
Question: The concusioo properly drawn which
the following assumed?
Option0: Students respect honesty above all else.
Optiont: Students lose respect for teachers whenover they sense that the toachers are trying hide their ignorance.
OptionZ: Students' respect for teacher indpendent of
knowledge they bute teacher.
Exemplars with CoT
Option3: teacher
offective
rotains respect studonts.
Answer: The assumptio that studente respect for the teacher is based on how much knowledge the teacher has. Therefore the answer Option
Context: Miguel has four family members who
that only three will be allowed number famiy members who attend. Hawever, coremony by family mambers. stronply supported which the following assumed?
graduction Sunday afteroon. there
amonn. Normally graduation
the stadium relocatid the synmeskum rain,, and each gradusto rocsives just three admission
Question: The concission of argument
Optiond: Miguel has several friends who have fewer than three family members coming to graduation.
Optiont: Miguets graduating class much lorger than usual.
Option2: Miguel has fifh family member who unable
his gradustion.
Option3:
service indicated
very high Iklliood
Sunday anomoon..
Answer: Rain would preclude the use of the stedium which has no limit of the number of family members sttending and force the use of the gymnssium,
which limits number attending three. Therefore the
Option3
[-.]
more rhytim surveys show people living the country become as
country supposed seroossly as people living the ciy, that they experience an equal amount stress. Question: The statements above. true, provide the support which of following?
permii
healthy
living city.
Test Instance
Option0: Living the country neither hes thier not more relaxing than living in the city.
Option The amount of stress person expertances depends on that person's mythin of life.
Option2: People whose flythm of life slow and natunal reowerr quickey from liniiss.
Option3: Desplie what people believe,
drythm life unieeliity.
Answer:
Figure 15: Example of the prompt using the chain-of-thought approach.
We are collecting data assess how well machines understand questions that require complex logical reasoning such as in GMAT or LSAT. We appreciate your help!
In this task, you are given a passage, question about it, and four answer options where onc option is the correct answer and the others are incorrect (the check and cross marks indicate it). For each option, you asked to write rationale that is implicitly assumed by question writer and supports why option are correct or incorrect.
We recommend spending 5-10 minutes one HIT.
Guidelines for Writing Rationales
Write the rationale in one sentence.
Not only extract fact from the passage, but also write the reasoning process and background knowledge that not
written the passage explicitly, your own words.
Write concrete facts and knowledge (See an example below). For example, make sure to include proper nouns that appear in the passage and option you focus on.
You have to write rationales as self-contained: You shouldn' 't use pronouns that refer to expressions in the main question and options when writing rationales.
not include any text that explicitly states that the option correct incorrect. "This options correct because-", "-, so this option is wrong.")
UPDATED: When considering the rationale for question involving "if true X," focus the role that X plays in the passage under assumption that true, rather than whether actually true.
ensure the quality of rationale, please note the following two points.
Particularity
Avoid writing generic facts that applicable other options (e.g., "this option incorrect because doesn't match described events in the passage")
In other words, for each option, write rationale that only applies that option.
This is further rephrased to say that you should write rationales so that others should able to associate each rationale with
the choice.
UPDATED: When writing rationale, try incorporating
the content the corresponding option and explain
how it relates to the passage and the question.
Essentiality
Write rationale that is essentially necessary to justify whether the option correct or not. This means that the option changes its correctness if the given rationale is not valid.
Figure 16: Instructions for the rationale writing task (1/4).

---

Example
When Alicia Green borrowed neighbor' without permission, the police merely gave her warning. However, when Peter Foster did the same thing, he was charged with automablle thell. Peter came to the atlention of the police because the car he was driving was hit by speeding taxi. Alicia was stooped because the she was driving had defective tail lights. It not,
the taxi caused the damage this difference was not due to any difference in the blameworthiness of their behavior. Therefore Alicia should also have been charged with automoblle theft.
car Peter
got damaged
Question:
The statement that the Pater took got damaged Alicia took plays which of the following roles in the argument?
Answer Ontions
argument relies. Z A: demonstrates awareness of fact which possible objection might be based.
illustrates general principle on
C: presents reason that directly supports the conclusion.
X D: justifies the difference actual outcome in the
In this examole have to write rationales that explain
why is correct answer.
why is not correct answer.
why C not correct answer.
why is not correct answer.
For example you can write rationales with regard this question such as:
Good Rationales
A: The author assumes as a possible objection that the car damage caused the difference and denies it.
B: The statement that
Peter took got damaged and Alicia took did describes fact which occurred the passage, not principle.
C: The difference in the damage to the car does not support the conclusion but is supposed to be the rationalc for the
opposite conclusion instead.
D: The damage to the car should not justify differences in actual outcomes because whether the taken car got damaged was
just the reason why his theft was brought to light.
Bad Rationales
statement is
possible objection might based.
B: The statement is not a general principle.
C: The statement doesn't directly support the conclusion.
D: The statement doesn't justify difference the actual outcome two cases.
Reason
The bad rationales are (1) ambiguous and (2) they refer expressions used main question(e.g. "The statement")
Write concrete rationales that well explain the reasoning and background knowledge for answering question.
The following two examples focus essentiality and particularity.
Figure 17: Instructions for the rationale writing task (2/4).

---

Example(Essettality)
Black Americans are twice as likely to suffer from hypertension as white Americans.The The same is true when comparing Westernized black Africans white Africans.The researchers hypothesized that the reason why westernized black people suffer from hypertension the result the interaction of two reasons? one is the high salt content western foods, and the other is the adaptation mechanism of black genetic genes to the salt-deficlent environment.
Question
The following conclusions about contemporary westemized African blacks, if the item true, can it best support the researcher's hypothesis?
Answer Option
The blood pressure of Yoruba people West Africa is not high. Yoruba people have lived inland far away from sea salt and far away from the Sahara salt mine in Africa.
X
Bad Rationale
Yoruba people are not Westernized.
Reason
This rationale essential
That is because whether the Yoruba were westernized not is not mentioned in the article, and most importantly, is not relevant for determining the
of option.
Even if it were true that the Yoruba westernized, the fact that the Yoruba did not have higher blood pressure in salt- deficient environments contradicts the second reason researchers gave, making that option incorrect.
more appropriate rationale would be "the fact that the Yoruba do not have higher blood pressure in salt-deficient environments contradicts researchers' hypothesis that they are more likely to develop high blood pressure due to adaptation mechanisms to salt-deficient environments.
Examale(Patiicularitt)
The passage and question are the same as above.
Answer Option
The blood pressure of the descendants of Senegalese and Gambians usually not high, and the history Senegal and Gambia been short
Bad Rationale
Senegal and Gambia are African nations that are relevant to the study.
Reason
This rationale too generic.
Instead, please describe the rationale in detail as follows, for example.
"The fact that the descendants of Africans who are not salt-dericient do not have high blood pressure suggests that high blood pressure related chronic of salt."
Figure 18: Instructions for the rationale writing task (3/4).

---

Additional Examples.
For many people, leisure time an opportunity to get outdoors, have some fun and meet interesting people. Add two pieces of advanced 21stcentury technology global positioning system (GPS) devices and the Internet to get "" ""geocaching". The word geocaching comes from "geo"" (earth) and ""cache" (hidden storage). Geocachers log onto website to information about the location of cache -- usually waterproof plastic box containing small items such as toys and CDs along with notebook where "finders"" cache. Finders may take any of the items cache are expected to replace them with something of similar value. They then visit the website again and write a message to the owner. Geocaching became possible on May 2000, made public, Using an inexpensive GPS device, anyone on earth can send a signal to the satellites and receive information about their position. This is basically high-tech version of orienteering, the traditional pastime which uses maps and compasses instead GPS to determine one's location.
enter comments learn about the cache ""owner"", the person who created
satellite system developed by the Department Defense was
Geocachers are a very considerate group. Owners carefully choose a cache's location to give finders an enjoyable experience, such boautiful view good campsite. They also consider the environmental impsct their cache since it could result an increased number visitors to an area. for the content the caches, owners and finders must only use items that suitable for the whole family, as caches are found by geocachers of all ages
Question
Which of the following is true according the passage?
Answer Options
geocachers
B: Any item can be placed in the caches.
C: The caches should be in remote place.
D: Geocachers try to avoid damaging the environment.
Ratienales
A: All age groups participate in geocaching.
B: The finder expected out into the cache something valuable
in the cache.
C: Geocachers consider the placement of the cache based mainly on the fun for finder.
D: Geocachers also consider the consequences of increasing the number of visitors due to the popularity of geocaching.
we want to deal with the association between boys and girls properly, here are some "dos and don'ts" for you to follow. Keep normal and healthy state of mind. Our schools and classes are made of boys and girls. is very natural for boys girls to make friends with each other. should make many friends possible. should keep in with other sex in public instead of in secret.
Don't be too nervous or too shy. If you shy person, can also find way out. First of all, you make friends with students
have the same interest and hobby as you. As both of you have much common, you may have much talk about. If you keep doing like that, little by little, you will gladly find you are also as free to express yourself as others. Don't fall into the trap of early friendship as sign love and fall love with each other at an early age, In my opinion, early love is green apple that can't be eaten. An apple won't taste sweet until it is fully ripe. Boys and girls middle school are too young to carry the heavy duty love. keep of early love.
The
girls adolescence
rich feeling, They easy regard the
Question
The main idea of passage to
Answer Options
X A: tell students to keep away from early love.
associate between boys and girls. X C: tell students how to make friends.
B: give
advice
X D: teach boys how to talk with girls.
Rationales
The author recommend staying away from early part of encouraging healthy relationships between women.
B: The author offers several notes how maintain healthy relationships between men and women. Making friends juts a part of building proper relationships.
D: The author tells general ways how to talk with others.
Figure 19; Instructions for the rationale writing task (4/4).

---

studying reading comprenension technology: passage, question, four options, you caked write rationales explaining why each option incorrect.
The check and cross marks indieate the correciness of the options. Please read the full instruction before sterting. (We have updates red texts.)
Notes
Do not include any text that explicitly states that the option correct or incorrect, (a.g., "this option incorrect because
doesn't match described events the passag")
onsure quality rationale, please following two points.
Particularity
Avsid writing generic facts that applicable the other options (e.g., "this option incorrect because doesn't match described events the passage")
In other words, for each option, write rationale that only applies to that option.
This is further rephrased to say that you should write rationales so that others should be able to associate each rationale with the choice.
Essentiality
Write rationals essentially necessary to justify whother option la connect not: means that option changes correctress given
valid.
can accept up
Passage 1/1
Rationales Writing
Completed
public research facility should not patent its inventions. Such facilities, because right patent funds, should encourage invention has a motive to suppress the free flow ideas in order to profit from the patent. Retaining the right to patent Inventions is contrary to the public research facility's obligation to da work intended for the benefit of the public.
supported public
public research facility
Question
The claim that public research facility should not patent its inventions plays what role in the argument?
Options
1. x
is claim that later proven in the argument.
input goes here
is the conclusion of the argument.
input goes here
provides a roadmap for the passago.
Input goes here
It is an assumption upon which the argument is built.
Input goes here
Write rationales for all options!
your rationales
Figure 20: Rationale writing task interface. The workers given context, question, and four options along with their correctness, and are asked to provide rationale for each choice.

---

We assessing the quality of rationale writter multipe-choice read ng compreeennson questions.
In t-is task, others incorrect (the check and marks indicate it).
given passage, question about and four answer options where one option is the correct answer and the
In addition, you are given rationale ie, which corresponds to one of the options and correctness. You are asked to answer which option the given rationale corresponds to by checking the radio button.
We recommend spending about 1-2 minutes on one HIT.
Example
The town council of North Tarrytown favored changing the name of the town to Sleepy Hollow. Council members argued that making the town' association with Washington irving and his famous "legend" more obvious would increase tourism and result immediately in financial henefits for the town" inhabitants.
Quastion:
the wo..ld less than immediate per The council members' argument requires the assumption that
Anewer Options
immediate per capits inhabitants changing
capita revenue they would receive from the change.
X B: other towns the region have changec the names to reflect historicel asscciations and have, result, experienced a rise tourism.
C: the town can accomplish, at a very low cost per capita, the improvements in tourist facilities that an increase in touriam
would require.
many inhabitants would be ready supply tourists with information about Washington Irving and "legend".
Rationale
Creating tourist facilities not necessary to benefit from the financial gains of changing the towns
In this example, the answer is C. That's because the rationale explains why optionC is not correct, ie.. that builcing traveler's facility not assumed the passagee.
Figure 21: Instructions for the rationale validation task.

---

Evaluate explanation for reading comprehension
Instructions
this task, you given passage, question about and others are incorrect (the check and cross marks indicate it).
options where the correct and the
In addittinn, you are given rationale, which corresponds to one of the options and its correctness. asked
which option the given rationale corresponds by checking radio button. can accept up
batch.
Passage 1/1
Match the Rationale
Completed
university study reported that between 1975 and 1983 the length the average workweek certain country increased significantly. A governmenaal study, on the other hand, shows significant decline in the length of the average workweek for the same period. Examination the studies shows, however, that they used different methods investigation; there no need look further for explanation the difference in the studies' results.
Question
The argument's reasoning is flawed because the argument fails to
Options
X
recognize that varying economic conditions result in the average workweek changing in length
V
recognize that two different methods of investigation can yield identical results
X
distinguish between study produced for the purposes of operation government and study
produced as part of university research
distinguish between method investigation the purpose investigation
Rationale
It does not fail to distinguish the different studies, blatantly that one study was a university and one was by the government.
Answer which option corresponds to the ratinale!
Submit your answer
Figure 22: Rationale validation task interface. The workers select the option best supported by the provided rationale.

---

Solve reading comprehension questions!
This is a HIT of answering multiple choice reading comprehension questions. Spend up to five minutes for answering five questions rejection happens). You up HITs for this batch.
Passage 1/3
Our local community theater often produces plays by critically acclaimed playwrights. In fact, the production director says that critical acclaim is one of the main factors considered in the selection of plays to perform. So, since my neighbor Michaela' s new play performed by the theater this season, she must be critically acclaimed playwright.
Clark:
Question
The reasoning in Clark's argument is most vulnerable to criticism the grounds that the argument
Options
uses evidence source that there reason to believe unreliable
2.
fails to consider that several different effects may produced by a single cause
3. treats one main factor considered in the selection of plays to perform as though were condition that must be met in order for play to selected
4. takes condition necessary for playwright's being critically acclaimed to be condition sufficient for playwright's being critically acclaimed
5.
Invalid unanswerable question
Go the next passage
Figure 23: Human validation task interface. The workers are asked answer subquestions.


Reviews:
[{"review_id": "jjaoLTgwaa", "reviewer": "Reviewer_ivgz", "summary": "", "questions": "Question A. (Line 70-72) What are the examples of selection elimination process of relevant alternatives in logical reasoning?\n\nQuestion B. (Line 182-184) Why is faithfully testing the model’s performance on the main questions important?\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the invaluable feedback and constructive comments from the reviewer.\n\n\n>On the Annotation of Rationales: Annotators can write the rationales that are the semantically same with the option but with different expressions.\n\nWe did not have a filtering for such rationales. To confirm the degree of the similarity, we performed a manual annotation of our dataset, extracting 50 pairs of main options and rationales, both for selective and eliminative rationales.\n\nFor selective rationales, we identified three out of 50 pairs as semantically similar:  \n1. Main Option: “Delays in the communication of discoveries will have a chilling effect on scientific research.”  \nRationale: “Delays in communicating discoveries would limit the time other scientists have to investigate and contribute.”\n2. Main Option: “Kimmy is a highly compensated and extremely popular television and movie actress.”  \nRationale: “All the information in the passage indicates that Kimmy is affluent and renowned.”\n3.\nMain Option: “Before new therapeutic agents reach the marketplace, they do not benefit patients.”  \nRationale: “The passage states that new therapies aid patients only after they are introduced to the marketplace.”\n\nFor eliminative rationales, we noted only one similar pair:  \n1. Main Option: “The speed of eye orientation correlates with intelligence, not overall health.”  \nRationale: “The speed at which one can orient one's eye to a stimulus has been closely associated with overall health.”\n\n\n>The examples of selection and elimination process of relevant alternatives in logical reasoning (Question A): \n\nThe process of selecting or eliminating relevant alternatives in logical reasoning involves discerning incorrect possibilities and choosing the correct one.\nIn the context of multiple-choice questions, this process refers to ruling out incorrect options to identify the right option. Our dataset evaluates this capability by asking models to choose a rationale for each option.\nThe reasoning involving such a process might be:  \n\"Option A is incorrect because this passage provides examples to support a claim, not to generalize it. [...] Option D is correct as the passage builds its conclusion by citing various sources.\"\n\n\n>Why is faithfully testing the model’s performance on the main questions important? (Question B):\n\nWe apologize if our phrasing in L182-184 caused confusion. Our intent was to express that we aim to evaluate the model's understanding of the main questions through the auxiliary subquestions, ensuring a comprehensive evaluation. We will make this clearer in our revised manuscript.\n\n\nThank you again for the insightful comments and the opportunity to clarify our work. We hope that our responses adequately address the concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem is well-motivated by grounding on the prior work.\n2. The paper contributes valuable human-written free-from rationales datasets. The choice of the dataset is well-motivated and details about the annotation and qualification process are thoroughly performed. Especially, qualification and essential criteria for writing rationales are concrete. (It would be good to have examples of comparison between specific and unspecific samples and consistent and inconsistent samples)\n3. The experiment setting is concrete and detailed enough to reproduce and findings are well-organized in that there are also comparisons between findings from prior works.\n", "reasons_to_reject": "Annotation of rationales: Annotators can write the rationales that are the semantically same with the option but with different expressions. Can this kind of rationale be filtered out with current verification? Or is this kind of rationale added to the dataset? \n", "questions_to_authors": "Question A. (Line 70-72) What are the examples of selection elimination process of relevant alternatives in logical reasoning?\n\nQuestion B. (Line 182-184) Why is faithfully testing the model’s performance on the main questions important?\n\n", "paper_topic_and_main_contributions": "This paper proposes a dataset composed of auxiliary questions and rationales to test the model’s consistent ability for critical reasoning. The paper presents experiments that compare various models with different conditions on this task. Additional analysis on difficulties of eliminative subquestions and comparison between rationale writing ability also provides insights into the model’s behavior on reasoning tasks. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 40: reference info is missing\nLine 252-264: Explaining necessity with consistency is confusing\n"}, "extra_text": {}}, {"review_id": "smD28y6BcO", "reviewer": "Reviewer_i432", "summary": "", "questions": "Suggestion:\n\nA. I have a different hypothesis about why the score for eliminative subquestions is so much worse than selective subquestions: It is because the models ignore the word \"not\". It would be interesting to do an experiment to test this: Compare model predictions for eliminative subquestions with and without the word \"not\" and see if the scores are close and individual responses correlate. Although not necessary, it'd be a good addition to the paper (at least the appendix, if not the main paper).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we would like to express our gratitude for pointing out the fascinating hypothesis and  the related literature. Your comments are valuable to our work.\n\n\n>Comparison of Model Predictions for Eliminative Subquestions with and without the Word \"not\"\n\nWe investigated the hypothesis under the following settings.:\n- Extracted eliminative subquestions that include the word \"not\" (n = 2156).\n- Compared the performance of text-davinci-003 by providing 5-shot exemplars from ReClor as a prompt on the subset (vanilla) and on the subset where \"not\" is eliminated from the question (elim_not).\n\n\n**Results**\n\nAccuracy:\n- Vanilla: 57.33±1.61%\n- Elim_not: 36.26±4.77%\n\nCohen's Kappa: 0.207\n\n2*2 Matrix:\n\n|          | Vanilla Correct | Vanilla Incorrect |\n|----------|----------|----------|\n| **Elim_not Correct** | 566  | 670  |\n| **Elim_not Incorrect** | 216  | 704  |\n\nFrom the results above, we conclude that the presence of \"not\" significantly affects the model's reasoning. However, it is interesting that the model was still capable of answering 36% of the eliminative subquestions even when the polarity of the question was reversed (i.e., after the negation was removed).   \nWe will add these results in our revised manuscript.\n\nWe deeply appreciate your insightful comments, which have improved our work.\nThank you for taking the time to review our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is a very well-done study and evaluation design. I particularly liked the two features highlighted in the summary above.\n- This is a nice high-quality resource, and should be helpful for people exploring logical reasoning models.\n- I was impressed by the number and diversity of the models that they have benchmarked for this new task (both fine-tuned and few-shot).", "reasons_to_reject": "I don't see any reason to reject this work.\n", "questions_to_authors": "Suggestion:\n\nA. I have a different hypothesis about why the score for eliminative subquestions is so much worse than selective subquestions: It is because the models ignore the word \"not\". It would be interesting to do an experiment to test this: Compare model predictions for eliminative subquestions with and without the word \"not\" and see if the scores are close and individual responses correlate. Although not necessary, it'd be a good addition to the paper (at least the appendix, if not the main paper).", "paper_topic_and_main_contributions": "This work evaluates models' logical reasoning in the MCQ setting by probing it with additional questions about the reasoning behind selecting or eliminating individual choices. To this end, the authors build a new dataset, RULE. They start with questions from the ReClor dataset and annotate rationales for selecting and eliminating each of its answer choices. They then generate questions like for each choice which has one of the generated rationales as the answer.\n\nThese additional (sub-)questions have 2 key features:\n\n1. They are in the same MCQ format as the original question. So one can probe the models' reasoning on the original question by these additional questions.\n2. They are contrastive (minimally different), i.e., all subquestions share the same passage and answer choices, but they have different answers depending on the subquestion. This feature prevents models from taking simple shortcuts.\n\nThe authors have ensured the annotations are of high quality by human validation, and have also established a very high human score on the task.\n\nFinally, the authors benchmark a large number of recent (few-shot, fine-tuned, with/without CoT) models on ReClor and RULE and demonstrate that all the models struggle and are behind humans. In particular, they find that models are extremely bad at selecting why a given choice is incorrect.\n\nFinally, they explore (i) model-generated rationales and find humans are better at this task. (ii) using human-written rationales to improve models and find that selective subquestions help, eliminative ones hurt.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "(tangentially relevant): https://arxiv.org/abs/2204.05212 and https://arxiv.org/abs/2210.10860v1 also collect explanations for each answer choice, albeit for a different purpose. Unlike this work, their explanations for wrong answer argue for why answer should be accepted as opposed to eliminated.\n\nL040: missing reference\n"}}, {"review_id": "cDOjUQcPxx", "reviewer": "Reviewer_X3Ra", "summary": "", "questions": "1. I found the sub-question provided in Figure 1 a little difficult to parse. Did you try alternative ways of phrasing the sub-questions?\nFor example:\nWhy is “deriving implications of a generalization that it assumes to be true” not the correct answer for the question “The argument proceeds by doing which one of the following”?\n(I think this phrasing might be easier for both humans and machines to parse)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the time and effort you have dedicated to reviewing our manuscript. \n\n\n>License of ReClor:\n\nWe reached out to the authors of ReClor via email to inquire about the licensing for using ReClor as a part of our dataset. In response, we received confirmation that a portion of ReClor is permitted under the use item of ReClor, which can be confirmed on their project page. We will make sure to clarify this information in our revised manuscript.\n\n\n>Alternative Ways of Phrasing the Sub-questions:\n\nIn addition to generating from the main question (Q) and the options (Op) using InstructGPT, we also attempted to transform Q+Op into declarative sentences and subsequently into interrogative ones. Although we used T5 and QA2D (Demszky etal., 2018) for the transformation into declarative sentences, neither method generated high-quality declarative sentences.\n\nWe appreciate your suggested heuristic approach to constructing subquestions, emphasizing ease of parsing. While we understand and acknowledge the advantages of your proposed phrasing, our main intent was to ensure that the subquestions are self-contained (i.e., excluding phrases like \"which of the following\" from Q that necessitate providing the original options). In hindsight, a generation method that modifies the Q part of your heuristic to be self-contained may have been a better approach. For example: Why is “deriving implications of a generalization that it assumes to be true” not the correct answer for the question \"What method or technique does the argument employ?\"\n\n\nThank you once again for your valuable feedback.\n\nReference\nDemszky, D., Guu, K., & Liang, P. (2018). Transforming Question Answering Datasets Into Natural Language Inference Datasets. arXiv preprint. arXiv:1809.02922.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I think that the RULE dataset would be useful for future research, particularly for improving language models’ reasoning abilities\n2. The examination of language model limitations show areas where language models can potentially be improved", "reasons_to_reject": "1. The RULE dataset is built on the ReClor dataset. However, ReClor is not released under an open source/data license. It is unclear if the authors obtained permission to use and redistribute this data. Licensing information is also not provided for the RULE dataset, so it is unclear how and if this new dataset can be used by other researchers.\n\nEdit: The authors have clarified in there rebuttal that they have obtained a license for ReClor. However, it is still unclear if the RULE dataset will be made available and how it will be licensed\n\nEdit 2: The RULE dataset will be made publicly available under the CC BY-NC 4.0 license", "questions_to_authors": "1. I found the sub-question provided in Figure 1 a little difficult to parse. Did you try alternative ways of phrasing the sub-questions?\nFor example:\nWhy is “deriving implications of a generalization that it assumes to be true” not the correct answer for the question “The argument proceeds by doing which one of the following”?\n(I think this phrasing might be easier for both humans and machines to parse)", "paper_topic_and_main_contributions": "1. The authors introduce the RULE dataset which consist reading comprehension question from the ReClor dataset and new follow up questions to understand the rationale a selected reading comprehension question response\n2. Show limitations of existing language models in understanding why incorrect answers should be eliminated in reading comprehension question answering", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Broken citation on line 040 (“; ?”)\n2. You don’t mention “RULE” by name in the main text of Section 3. I would add a sentence introducing the dataset at the beginning and remaining readers what the name of the dataset is\n3. Annotator compensation is an important ethical consideration, I would move it from Appendix C to the Ethics Statement"}}]

Author rebuttal:

Meta-review: All reviewers agree that the new RULE dataset on understanding rationales behind critical reasoning is a valuable resource, and that the paper is well motivated and well-structured. 
The reviewers questions and concerns regarding licensing of the resource could be resolved during the discussion.
Decision: Accept-Main