Title: QASA: Advanced Question Answering on Scientific Articles
Benchmark: QASA
Summary: This paper introduces QASA, a benchmark for advanced question answering in scientific articles, motivated by the dual process theory of human reasoning. It proposes a three-stage approach (associative selection, evidential rationale-generation, systematic composition) to tackle "full-stack reasoning". The dataset contains 1798 QA pairs on AI/ML papers, featuring diverse question types (surface, testing, deep) and requiring answers composed from multiple evidential rationales. Experiments show the proposed approach outperforms InstructGPT, emphasising the importance of the rationale generation step.
Contribution: Introduces the QASA benchmark for full-stack reasoning QA on scientific articles. Develop a question schema based on cognitive reasoning levels (surface, testing, deep) via a think-aloud study. Creates a dataset requiring a composition of long-form answers from multiple evidential rationales. Proposes and evaluates a three-stage computational approach (associative selection, rationale-generation, composition) mimicking dual process theory. Demonstrates the effectiveness of this approach and the importance of explicit rationale generation.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Full-stack reasoning (associative thinking + logical reasoning); Advanced question answering; Rationale generation; Answer composition from multiple evidences.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to answer complex questions about scientific articles by first selecting relevant paragraphs (associative selection), then extracting or generating key rationale points from each selected paragraph (evidential rationale-generation), and finally synthesizing these potentially disparate rationales into a single, comprehensive, non-redundant answer (systematic composition).
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: Inspired by cognitive science (dual process theory) to create a QA task that better reflects complex human reasoning compared to factoid or simple multi-hop QA. Aims to evaluate the ability to synthesise answers from multiple pieces of evidence spread across a document.
Task definition: Given a question about a scientific paper and the full paper text (as paragraphs), perform full-stack QA: select evidence paragraphs, generate an evidential rationale for each, and compose these into a final answer. The primary evaluation focuses on the quality of the final composed answer.
Task item definition: An instance includes a question (with type label: surface/testing/deep), the source scientific paper, a set of gold evidence paragraph identifiers, a set of gold evidential rationales (text snippets corresponding to each evidence paragraph), and a final composed gold answer (long-form text).
Task definition detail: Questions are diverse (surface, testing, deep) and collected from both readers and authors of AI/ML papers. Answers are often long-form and require composing information from an average of 1.67 (max 9) rationales.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: Papers sourced from S2ORC and arXiv (CS.AI domain). Questions collected from AI/ML graduate students, freelancers, and paper authors following specific guidelines and question schema. Answers, including evidence selection, rationale writing, and final composition, were annotated by qualified experts.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Simulates the process of deeply understanding a scientific paper to answer complex questions that go beyond simple fact retrieval.
Dataset size: 1,798 QA pairs total. Evaluation seems performed on the full dataset.
Dataset metadata available: Yes
Dataset metadata detail: Question Type (Surface/Testing/Deep + sub-types), Source Paper ID, Evidence Paragraph IDs, Evidential Rationale Text, Composition Required (True/False).
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragarph)
Response format detail: The key output is the final composed answer, which is a long-form text passage synthesized from intermediate rationale texts.
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric definition detail: Associative Selection: Precision, Recall, F1. Rationale Generation & Answer Composition: ROUGE-1, ROUGE-2, ROUGE-L (F1 scores). Human Evaluation: Pairwise win/tie/lose rates based on Groundedness, Completeness, Specificity, and Fluency.
Metric aggregation: Simple Mean
Metric statistics: Precision, Recall, F1, ROUGE-1, ROUGE-2, ROUGE-L, Human evaluation win/tie/lose rates (%).
Metric subscores: Yes
Metric subscores detail: Performance reported for each subtask (selection, rationale gen, composition) and the full-stack QA. Ablation study on training data sources. Analysis based on question types and compositionality requirements.
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Think-aloud study informed question taxonomy. Annotators were domain experts. Separate reader/author sessions enhanced question diversity. Detailed annotation process involved evidence selection, rationale generation, and composition steps. Manual checks confirmed high answer correctness (90%) and groundedness (87%) on a sample.
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning
Procedural notes: Details annotator recruitment (Upwork, professional networks), qualification, and background. Uses OpenAI embeddings for retrieval. Details models used for experiments. Describes construction of training data using public datasets and distillation from InstructGPT. Details human evaluation procedure and criteria.
Other notes: The explicit modelling of the three reasoning stages (selection, rationale generation, composition) and the empirical validation of the importance of the rationale generation step are key takeaways. The dataset's focus on scientific articles from AI/ML makes it specialised but highly relevant for evaluating models within this domain.