RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Conclusion
We construct a dataset to evaluate the models’ ability of critical reasoning in logical reading comprehension.
We crowdsource free-form rationale for main questions taken from an existing dataset and use an LLM to generate subquestion texts.
Resulting questions ask about the underlying rationales for why a certain answer option should be selected and the others should be eliminated.
We find that LLMs are particularly bad at answering eliminative subquestions, highlighting that those models do not necessarily have the comprehensive ability of critical reasoning.
For future work, we will develop a more efficient pipeline for data collection and facilitate better rationale generation by LLMs.
Ethical Consideration
We use crowdsourcing in our data collection.
We make sure to be responsible to the crowdworkers and to make fair compensation for their work.
We do not collect any personal information other than worker IDs on the platform, which are removed in our data release.
Before the workers accept our tasks, we inform them of our purpose for the data collection.
This study is approved by the internal review board of the authors’ institutes.
Limitations
We recognize the following limitations in this study.
Task Format
In this study, we focus on the multiple-choice QA task.
This task format allows us to flexibly ask about various linguistic phenomena and human reasoning by selecting and eliminating alternatives, and we consider solving such a discriminative task would be a minimal requirement for human-like linguistic behaviors.
However, it has an inherent limitation in assessing the ability of natural language understanding.
For example, we cannot evaluate the models’ ability to produce an intended output.
Annotation Analysis
We conduct the annotation analysis in Section 5 , in which we define the reasoning types and manually review the sampled examples.
Although we make our annotation data and guideline publicly available for ensuring the reproducibility of annotation results, the results of our annotation analysis inevitably involve our subjective judgments.
Source Dataset
We create our auxiliary questions on top of an existing English logical reading comprehension dataset, ReClor.
Although our methodology of the data collection (i.e., writing the rationale for selecting and eliminating alternatives) is widely applicable to other datasets and languages, using the single dataset in the single language would limit the generalizability of our findings.
Acknowledgments
We would like to thank the anonymous reviewers for their helpful comments.
This work was supported by JST PRESTO Grant Number JPMJPR20C4 and JSPS KAKENHI Grant Number 22K17954.
Figure 1: Example of ReClor Yu et al. ( 2020 ) and its subquestion we create to test the understanding of implicit rationale.
We find that even if the model can answer the original question correctly, it cannot answer subquestions that should be answerable.; Example of ReClor Yu et al. ( 2020 ) and its subquestion we create to test the understanding of implicit rationale.
We find that even if the model can answer the original question correctly, it cannot answer subquestions that should be answerable.
Figure 2: Our dataset construction process.
We first ask crowdworkers to write the rationale for each answer option.
After validating the collected rationale by aligning them to the source options, we use a large language model to generate subquestion texts.
We finally ensure the human answerability of the generated subquestions.; Our dataset construction process.
We first ask crowdworkers to write the rationale for each answer option.
After validating the collected rationale by aligning them to the source options, we use a large language model to generate subquestion texts.
We finally ensure the human answerability of the generated subquestions.
Figure 3: Distribution of correctly answered subquestions ( C ) out of the total number of subquestions ( N ), for both InstructGPT (top) and humans (bottom).; Distribution of correctly answered subquestions ( C ) out of the total number of subquestions ( N ), for both InstructGPT (top) and humans (bottom).
Figure 4: Distribution of the question length (#tokens) of the main questions.; Distribution of the question length (#tokens) of the main questions.
Figure 5: Comparison between the model and human accuracy and question length for the main questions.; Comparison between the model and human accuracy and question length for the main questions.
Figure 6: Distribution of the question length (#tokens) of the subquestions.; Distribution of the question length (#tokens) of the subquestions.
Figure 7: Comparison between the model and human accuracy and question length for the subquestions.; Comparison between the model and human accuracy and question length for the subquestions.
Figure 8: Distribution of the option length (#tokens) of the main questions.; Distribution of the option length (#tokens) of the main questions.
Figure 9: Comparison between the model and human accuracy and option length for the main questions.; Comparison between the model and human accuracy and option length for the main questions.
Figure 10: Comparison between the model and human accuracy and option length for the subquestions.; Comparison between the model and human accuracy and option length for the subquestions.
Figure 11: Distribution of the option length (#tokens) of the subquestions.; Distribution of the option length (#tokens) of the subquestions.
Figure 12: Example of the prompt used for rationale alignment task.; Example of the prompt used for rationale alignment task.
Figure 13: Accuracy of InstructGPT and reasoning types originally annotated in the ReClor dataset.; Accuracy of InstructGPT and reasoning types originally annotated in the ReClor dataset.
Figure 14: Example of the prompt used to generate subquestions.; Example of the prompt used to generate subquestions.
Figure 15: Example of the prompt using the chain-of-thought approach.; Example of the prompt using the chain-of-thought approach.
Figure 16: Instructions for the rationale writing task (1/4).; Instructions for the rationale writing task (1/4).
Figure 17: Instructions for the rationale writing task (2/4).; Instructions for the rationale writing task (2/4).
Figure 18: Instructions for the rationale writing task (3/4).; Instructions for the rationale writing task (3/4).
Figure 19: Instructions for the rationale writing task (4/4).; Instructions for the rationale writing task (4/4).
Figure 20: Rationale writing task interface. The workers are given a context, question, and four options along with their correctness, and are asked to provide a rationale for each choice.; Rationale writing task interface. The workers are given a context, question, and four options along with their correctness, and are asked to provide a rationale for each choice.
Figure 21: Instructions for the rationale validation task.; Instructions for the rationale validation task.
Figure 22: Rationale validation task interface. The workers select the option best supported by the provided rationale.; Rationale validation task interface. The workers select the option best supported by the provided rationale.
Figure 23: Human validation task interface. The workers are asked to answer subquestions.; Human validation task interface. The workers are asked to answer subquestions.
Dataset statistics of our RULE dataset. S/E indicates the numbers of two types of subquestions written about the correct (selective) or incorrect (eliminative) options of their main questions, respectively.
The question and option lengths of the main questions are separately reported in parentheses for comparison. “None” denotes “None of the above choices.”
Model performance on our RULE dataset consisting of the main questions ( MainQ ) and subquestions ( SubQ ).
We report the accuracy for the subquestions written about the correct option ( Selective SubQ Acc. ) and incorrect options ( Eliminative SubQ Acc. ) of the main questions.
Consistency holds only when the model answers both the main question and its subquestions correctly.
InstructGPT is text-davinci-003 .
Accuracy of the subquestions that have “None of the above choices” as the correct answer ( None Acc ), compared to that of all subquestions ( Acc ). None in shot indicates how many “None” examples are included in the few-shot prompt for each test split.
Annotation results of rationale types on 100 examples randomly sampled from all subquestions (left) and from the error examples by InstructGPT (right).
MainQ accuracy of InstructGPT that uses the selective or eliminative rationales in the input.
Complementary results of the model performance on our dataset including the models in the one-shot setting and omitting those in the five-shot and zero-shot settings.
Result of the rationale alignment task with and without the task instruction.
Main results of the model performance on our dataset focusing on the subquestions for the main questions that are correctly answered by the model.
Context-ablated accuracy.
The subscript values indicate the accuracy gap against the full-input setting.
Examples of the reasoning types with a passage, a question, an option, the correctness of the option, and its human-written rationale.
Examples of the main questions and subquestions in our dataset.
The options in bold indicate the correct answer.
Rationales that are semantically similar to the MainQ’s option in Selective and Eliminative SubQs.