AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation
Authors: Tiancheng Huang
Ruisheng Cao
Yuxin Zhang
Zhangyi Kang
Zijian Wang
Chenrun Wang
Yijie LUO
Hang Zheng
Lirong Qian
Lu Chen
Kai Yu
Venue: ICLR 2026 Poster (2026)
Keywords: question answering
supervised fine-tuning
trajectory synthesis
Primary area: datasets and benchmarks
TL;DR: We annotate AirQA, a comprehensive QA dataset for AI Research with instance-level evaluation, and introduce ExTrActor, an automated framework for instruction data synthesis.

Abstract: The growing volume of academic papers has made it increasingly difficult for researchers to efficiently extract key information. While large language models (LLMs) based agents are capable of automating question answering (QA) workflows for scientific papers, there still lacks a comprehensive and realistic benchmark to evaluate their capabilities. Moreover, training an interactive agent for this task is hindered by the shortage of high-quality interaction trajectories. In this work, we propose AirQA, a human-annotated comprehensive paper QA dataset in the field of artificial intelligence, with 13,956 papers and 1,246 questions, that encompasses multi-task, multi-modal and instance-level evaluation. Furthermore, we propose ExTrActor, an automated framework for instruction data synthesis. With three LLM-based agents, ExTrActor can perform example generation and trajectory collection without human intervention. Evaluations of multiple open-source and proprietary models show that most models underperform on AirQA, demonstrating its quality. Extensive experiments confirm that ExTrActor consistently improves the multi-turn tool-use capability of small models, enabling them to achieve performance comparable to larger ones.

Published as a conference paper at ICLR 2026
AIRQA: A COMPREHENSIVE QA DATASET FOR AI RESEARCH WITH INSTANCE-LEVEL EVALUATION
Tiancheng Huang', Ruisheng Cao', Yuxin Zhang' Zhangyi Kang' Zijian Wang',
Chenrun Wang', Yijie Luo', Hang Zheng', Lirong Qian', , Lu Chen 1.2.3.4f Chen Kai Yu1.3,4
"X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
Shanggaii Innovation Institution, Shanghai. China
3 Jjangsu Key Lab of Language Computing, Suzhou, China
"Suzhou Laboratory, Suzhou, China (htc981, , chenl usz, kai yu) és ttt.sttuuuuuuuu edu.ecn
ABSTRACT
The growing volume of academic papers has made it increasingly difficult for researchers to efficiently extract key information. While large language mod- els (LLMs) based agents are capable of automating question answering (QA) workflows for scientific papers, there still lacks a comprehensive and realistic benchmark to evaluate their capabilities. Moreover, training an interactive agent for this task is hindered by the shortage of high-quality interaction trajectories. In this work, we propose AirQA, human-annotated comprehensive paper QA dataset in the field of artificial intelligence, with 13,956 papers and 1,246 ques- tions, that encompasses multi-task, multi-modal and instance-level evaluation. Furthermore, we propose EXTRACTOR, an automated framework for instruction data synthesis. With three LLM-based agents, EXTRACTOR ple generation and trajectory collection without human intervention. Evaluations of multiple open-source and proprietary models show that most models underper- form on AirQA, demonstrating its quality. Extensive experiments confirm that EXTRACTOR consistently improves the multi-turn tool-use capability of small models, enabling them to achieve performance comparable to larger ones.
perform exam-
INTRODUCTION
With the explosion of artificial intelligence (AI) publications, researchers must spend a significant amount time reading lengthy papers just to locate highly specific piece of information, which is both tedious and inefficient. The advent of large language models (LLMs). especially their re-
markable reasoning and planning capabilities (Ahn al.) 2024 Guo al. 2025 Huang al. 2024 Wang et al. 2023b,, has made it possible to automate the workflow of precise retrieval and question answering (QA) for academic papers (He et al. 2025 Othman 2025) Skarlinski et al.) 2024). De- spite recent advances, there remains notable absence of comprehensive and realistic benchmark, which covers diverse question types and multi-modal abilities. And training an interactive QA agent that focuses such task is difficult due to the scarcity of high-quality domain-specific trajectories.
Previous QA datasets on scientific papers usually focus on one narrow question type, such as query- ing technical details about single paper (Dasigi et al. 202 Lee et al. 2023) Pramanick et al. 2025) Singh et al. 2024), questions spanning across multiple documents following rule-constructed two- hop pattern (Li et al. 2024), or aiming the common paper retrieval requirements (Ajith et al., 2024 He et al. 2025). ). Accordingly, the evaluation function is usually tailored for one restricted type and lacks generalization
others. For example, M3SciQA (Li et al., 2024) designed one LLM-based prompt for long-form string evaluation with the reference answer, which is highly em- pirical and only serves its specific question type. On the other hand, most benchmark owners overly pre-process raw papers, and merely provide the cleaned text format for uniform input. This common
"Our code and data are publicly available at ht https:/// / / thub. com www.://mmmmm//ii///.p.... "The corresponding authors are Lu Chen and Kai Yu.

---

Published as a conference paper at ICLR 2026
Agent Trajectory
(Question
MALA023
Single-doc Detaii
Mult doz Analysis
Pager Retrieval
Comprehhensive QA
[Actoon] RAG
ICML
ICLR
AONGENETE FER LANGEEN
[Action] Web Search
Google
Text
Table
Image
Formula
Metadata link Berrhh lirrary
[Action] -0
What shortcoming doss REV overcome??
Whech TarT tuako CLICOTA mode
deman Gllbarcch dilleet doam Paak any hand quedsorc"
What is the forward process ol HydnaloRAA
hand annotate
EaRa
Appendis
EIEX
agent turing
Figure I: Left: An overview of the four question types and five element categories in our AirQA dataset. Right: An illustration of the bottleneck in multi-turn tool-use trajectory collection.
practice deviates from realistic scenarios, where real-world users may query other hyper-textual el- ements (illustrated in the bottom part of Figure D embedded in the raw PDF documents, such as figures, tables, formula, metadata, or even different combinations of them.
When tackling QA on academic papers, trivial methods (e.g. provide titles and abstracts alongside
the question (Dasigi et al.) 2021 Singh et al. 2024) will easily fail due to context limitation, as the scaling papers augment from a single paper to the entire conference volume. More advanced approaches adopt the popular RAG framework (Guu et al. 2020 Borgeaud et al. 2022 Izacard (ct al. 2022), but are not applicable when multi-turn reasoning over various chunked snippets are required. Meanwhile, agents which can predict executable retrievals or function-calling actions and interact with the outer environment for external knowledge exhibit significant potential in handling long-context multi-hop scenarios, making good choice for scientific QA under realistic and com- plicated settings (He et al. 2025 Nakano et al., 2022; Schick et al. (023). Unfortunately. manually annotating spec-ipec trajectories of interactions with the environment is both time-consuming and expensive, requiring domain expertise, while simple data generation with LLMs can't 't faithfully synthesize (action, observation) sequences with internal coherence and dependencies. As a result, the paucity of high-quality trajectory prevents the post-training of an effective QA agent.
To resolve the aforementioned bottlenecks, we propose a numan-annotate multi-modal multi-task Artificial Intelligence Research Question Answering dataset, AirQA, which encompasses 1,246 ex- amples and 13,956 papers in the domain of artificial intelligence, aiming at evaluating research capabilities in realistic scenarios. As illustrated in Figure [ our dataset contains 4 different question types and 5 different element categories, with 19 parameterized Python functions to support cus- tomized evaluation. Furthermore, to advocate agentic model post-training, we propose multi-agent framework, EXTRACTOR, for instruction data synthesis, which includes an explorer that generates natural language QA pairs based on contexts from papers, tracker that rewrites QA pairs into properly formatted examples, and an actor that interacts with the environment to collect trajectories.
We evaluate wide range of open-source and proprietary LLMs different baselines. Perfor- mances show that, though given several external information sources, LLMs struggle on our AirQA dataset, with the best model scoring only 44.14% overall, indicating that existing workflows still underdeveloped. With the proposed EXTRACTOR framework. we fine-tune models of differ- ent sizes. Results show that, with just 4,000 interaction trajectories, fine-tuned 7B model achieves performance comparable to untrained 14B model. Extensive experiments demonstrate that, the accuracy raises consistently as data scales up, highlighting the scalability of our framework.
To summarize, our contributions are threefold:
We propose AirQA, a human- annotaedd multi-modal multi-task multi-paper QA dataset with function- based instance- -eseanccespcccccccccc evaluations. To the best of our knowledge, AirQA is the first dataset that encompasses multiple question types, also the first to bring function- based evalua- tion into QA domain, enabling convenient and systematic assessment of research capabilities.
2

---

Published as a conference paper at ICLR 2026
We introduce EXTRACTOR document- boaeedent-bbsed framework aiming at the synthesis of QA exam- ples, interaction trajectories and instruction data, serving as an empirical method for improving the agent's multi-turn tool-using ability without the involvement of manual annotation.
We evaluate various LLMs and different QA baselines on our AirQA dataset, demonstrating the quality of our dataset, and indicating the insufficiency of current methods. Extensive exper- iments on instruction tuning reveal that, small models significantly benefit from our synthetic instruction data, validating the effectiveness of our proposed EXTRACTOR framework.
THE AIRQA DATASET
In this section, we introduce the task definition, the evaluation metrics, the construction and the statistics of our AirQA dataset.
2.1 TASK DEFINITION
To more effectively evaluate existing models and methods across a broader range of tasks, rather than limiting assessment to individual tasks, we carefully analyze real-world AI research scenarios, and systematically design the following four question types in AirQA to cover them up:
Table 1: Examples of different question types from our AirQA dataset.
Type
Question
Answer Format
Which downstream tasks does the CLiCoTEA out- perform other models in terms of zero-shot perfor- mance on the IGLUE benchmark?
Your answer should be Python list of strings, ev- cry string is the abbreviation of downstream task type mentioned in the paper.
single
According to this survey. what're the three most cent docoder-only LLMs for NL2Code? How many programming languages do their training datasets each contain?
Your answer should be Python dictionary of 3 key-value pairs, where each key is string, the LLM, each value the number program- ming languages.
multiple
retrieval
Which paper unifies reinforcement learning and imi- lation leurning methods under a dual frumework?
Your answer should be the exact title of the paper WITHOUT ANY OTHER EXPLANATION.
Among the text-to-SQU papers in ACL 2023, which best testsuite uccuracy on the SPI- DER dataset? Tell me the paper title and correspond- ing accuracy.
Your answer should be Python list of length two. with the first one being the title string and the sec- ond one being a float, the accuracy rounded 3 decimuls.
one achieves the
comprehensive
Querying detailed information from a specific paper. Besides text, we also explore different textual and non-textual aspects including table, image, formula and metadata to cover all elements that may appear in a scientific paper. We showcase one example for each category Figure ] Notably, a question may belong to multiple categories, requiring diverse capabilities.
Single-doc Detail
Posing questions across multiple papers. A simple idea for constructing multiple-doc questions is to merely combine several single-doc questions, but it overlooks the pos- sible relations between different papers, which actually what researchers pay more attention to. To imitate the real scenes where researchers scan across several documents to find the answer to a question, we propose two paradigms: 1) compare same aspects of different papers, and 2) find subtle points that are not fully illustrated in one paper, and explore the details the papers it cites.
Multiple-doc Analysis
Paper Retrieval Retrieving papers from specific conference in particular year, based on the description. Considering the search scale, while [Skarlinski et al.] (2024) argues that retrieval on fixed corpus not suitable as performance proxies for real scientific research tasks, we insist that dataset cannot contain an infinite number of papers. Without limitation, the answer would be ambitious, making the evaluation unfair. Only retrieval on a fixed corpus can ensure the objectivity of the dataset. Among these questions, 240 are directly transformed from author written questions in the LitSearch (Ajith et al. 2024) dataset with rule-based conversion.
Comprehensive QA A combination of the three aforementioned question types. Specifically, comprehensive QA question may combine a retrieval question with either a single question or a multiple question. As an integrated task, this combination designed for scenarios in which the
3

---

Published as a conference paper at ICLR 2026
user cannot directly provide the paper or has forgotten the specific paper to which the question refers, but recalls certain key points, thereby enabling retrieval. The solution can be divided into two main stages: retrieving the paper based on its description, and answering the detailed question.
We also exhibit one example for each question type in Table [ For brevity, in the following sections, refer to the four question types single, multiple, retrieval and comprehensive respectively. For retrieval and comprehensive questions, to ensure uniqueness of the retrieved paper, and to avoid ambiguity, we limit the scope of retrieval be one of ACL2023, ICLR2024 and NeurIPS2024.
2.2 EVALUATION METHOD
As mentioned in Section 5 most previous QA datasets depend on linguistics metrics and LLMs for assessment, favoring semantic coherence over factual correctness, which holds little value under current circumstances. While in our AirQA dataset, we mainly focus on judging the correctness of the answer as objective possible. We notice that, though the answers to different questions vary, they share common features. For example, when answering questions related to quantitative comparison, we only care about the number itself, rather than whether LLMs form complete sentence. In this case, the number is the "scoring point" of this question, which directly determines the quality of the answer. Inspired by the instruction following ability of LLMs, we adopt output reformatting by providing an answer format along with the question (as shown in Table D such as, "Your answer should be Python list of two floats, each rounded to 2 decimal places.". Our empirical check on 30 randomly sampled answers from Owen2.5-3B-Instree further shows that the model adheres to the answer format in over 90% of cases (28/30) indicating that output reformatting is quite applicable in practice even for smaller models. In this way, we guide LLMs to output the scoring points we primarily concern with, benefiting the following evaluation.
(Qwen et al. (025)
To evaluate scoring points of different kinds, we design 19 Python functions and complement them with optional keyword arguments (e-g. ignore_order r for list comparison) to support example- specific assessment. For each evaluation, the final result will be either 0 or 1, representing wrong and right. Based on whether they utilize LLMs for semantic judgment or not, and their functionalities, these functions can be classified into two types and six subtypes as shown in Table 8 An evaluation function is subjective if it involves LLMs, and is objective if not. Specifically, for logical functions, which combine multiple functions in one evaluation, the evaluation is classified as subjective long as there is one subjective function. For subjective functions, we select GPT-44-mini-2024-00 the backbone model for its relative stability. More details of the functions can be found in App. A.3
18 as
To further clarify the role and reliability of subjective evaluation, we highlight three points: 1) While LLM-based judgment is necessary certain cases, we design tailored prompts to support more fine- grained and targeted evaluation unlike previous datasets which mostly rely on a fixed prompt. e.g.,
We design a specialized prompt compare LaTeX formulas. 2) Existing studies (Wang al. 2023a]
Ho et al. 2025 Kamalloo et al. 2024) also suggest that, for QA tasks, bLsed evaluations
are more aligned with human judgments than metrics such as accuracy or F1. 3) Analysis on 66 examples shows that LL.M-based and human evaluations are largely consistent, with an agreement rate of approximately 83%.
2.3 DATASET CONSTRUCTION
Annotators To ensure professionalism, we employ 26 students with expertise in artificial intelli- gence. Their task is threefold: 1) read paper they are interested in, 2) pose an answerable question based on the textual and non-textual content of the paper (and additional papers if needed) they read, in accordance with the aforementioned question types, 3) wrap the question, the evaluation function, and other necessary information into an example file, as presented in App. A.I Example files are then sent into an automated inspection pipeline, and annotators are asked to rewrite unqualified ones.
Paper Collection Due to the professional background of the annotators, all papers are selected from the field of artificial intelligence to ensure accurate comprehension of the content. To facilitate reproduction, assign an uuid for each used paper based its title and its conference. We also generate a metadata file for each paper, containing the title, the abstract, the URL where the paper is downloaded, and other information. For further illustration, please refer to
A.2 A.2.
4

---

Published as a conference paper at ICLR 2026
2.4 DATASET STATISTICS
Example Classification We classify the examples in the AirQA dataset into four question types, five element categories and two evaluation types as discussed before. Table 2 shows that, the ex- ample numbers of the four question types are relatively balanced, while approximately half of the examples involve at least one element other than text (we classify am example as text if and only if it doesn't it include any other elements, and example can belong more than one categories). Regarding evaluation functions, Figure 2 shows that most of the evaluations are objective, meaning they do not require LLMs, which demonstrates the cost-effectiveness of our dataset.
Paper Usage To ensure the objectiveness of retrieval and comprehensive questions. we limit the retrieval scale to be one of ACL2023, ICLR2024 and NeurIPS2024. Besides including all papers from these three conferences in our collection, we also utilize another 707 papers the examples, summing up total of 13,956 papers. As shown in Table 2 in average, an example involve 1.63 papers, indicating the diversity of our dataset.
Table 2:
Statisties of examples. For the last two statistics, we only consider sin- gle and multiple questions.
bool
1.1%
0.2%
formalaa 5.1%
19.2%
logical 4.5
fuzzy 0.6%
legioal 3,3%
Statistics
Number
subjective 28.9%
Question Type
int 6.7%
single
351(28%)
multiple
323(26%)
float 5.7%
retrieval
288(23%)
comprehensive
284(23%)
objective
retrieval 23.3%
Element Category
text
621(50%)
string 15.2%
table
213(17%)
image
207(17%)
structured 14.8%
formula
127(10%)
metadata
122(10%)
Overall
1246(100%)
Avg. question length
34.52
Figure 2:
Distribution of different evaluation cate- gories. 'bool', 'int', 'string'. 'fuzzy', 'structured" stand for specific evaluation functions in 'match' subtype.
Max. question length
118
Avg. # papers per example
1.63
Max. papers per example
7
Comparison with Existing Datasets Table we compare AirQA with existing scientific QA datasets. It is evident that AirQA demonstrates several salient strengths: 1) More question types. AirQA designs four different question types to systematically cover realistic research scenarios, 2) More element types. AirQA contains wider variety of elements, including table, image, formula and metadata, 3) More precise evaluation. AirQA employs 19 parameterized functions, which can be classified into two types and six subtypes, facilitating customized evaluation.
Table 3: Comparison of our AirQA dataset and existing scientific QA datasets.
Task types
Question based on
Dataset
#QA
Evaluation Methods
Sgl.
Muln.
Ret::
Comp.
Full Text
Table
Image
Form.
Meta.
SchukalyReul Saikh el 2020
10K
DLEU, METEOR, ROUGE
x
QASPER Dadir
5,049
Fl
x
X
x
X
Precisioos Recall, ROUGE
x
X
X
CIDEs ROUGE.
2025
BERTS Soorv MRR, Recall, Rouge A AlignSenve, Promefieuu-2 ROUGE, BLEURT-20. BERTSeum, LLM judge
PAeeQA
Baumglitner al.
2025
579
X
X
V
X
X
4
X
SeIDQA Singheral al 5024
2,937
X
X
V
4
X
M3ScIQ4
X
2025
35K
XX
LitSencch Arth
307
Recall Precision, Accuncc
IQQA
SSamliskki at
7024
248
AlrQA (Ours)
1,246
Instance level Function
5

---

Published as a conference paper at ICLR 2026
3 EXTRACTOR FRAMEWORK FOR TRAJECTORY SYNTHESIS
In this section, we introduce our trajectory synthesis framework, EXTRACTOR, based on its three components: explorer, tracker and actor.
vector
action:: flttreee
actionz: Query
action:: Arower
relational DB
achio spRce
farer the on 6004 (dcooed) RR-LL
methed the accurncs parsentage." "arguments"
rollout
randomly sample b) use LLM to generater snippet (e.g.. table) QA pairs based it
yvaluator:
Explorer
2 Tracker
Actor
Figure 3: An overview of our EXTRACTOR framework, which consists of three stages to automati- cally extract QA pairs, formulate evaluation and answers, and filter valid agent trajectories.
3.1 OVERALL FRAMEWORK
To handle scientific QA, there are mainly three types of methods: 1) offering relevant information of the paper (c.g., title and abstract) alongside the question, 2) providing additional contexts via retrieval methods (c.g., RAG), and 3) equipping LLMs with supplementary tools, that they can obtain sufficient information during multi-turn interactions. As discussed in Section 4.2) methods of the former types significantly underperform the latter, even when supported by superior back- bone models. Therefore, for the following fine-tuning, we apply Agentic Hybrid baseline (further illustrated in Section [..] whose environment includes database and a vectorstore that produce execution or query results as observations when called by the agent with two predefined actions.
To mimic the real-world annotation and interaction scenarios, we split the synthesis process into three separate stages: 1) exploration stage, constructing natural language question-answer pair with given context, 2) tracking stage, choosing suitable evaluation function and fill in the formatted example file, and 3) action stage, interacting with the outer environment to collect trajectories.
3.2 EXPLORER
Above all, we randomly download 10,000 papers in the artificial intelligence domain from arXiv collect their metadata including titles and abstracts, and employ PyMuPDF (Artifex Software 2023) and MinerU WWag et al. 2024 to extract both textual and non-textual elements from the papers.
For the explorer, its goal is to generate rational question-anae answer pairs with sampled contexts. Based on different question types, we design three different modes: 1) For single type, we first randomly choose a paper and an element. Then, corresponding contexts are extracted according to the category of the element, and the explorer is expected to output long-form QA pair. 2) Regarding retrieval type, instead of contexts, the explorer only receives the title and abstract as inputs, and required to generate a QA pair in which the answer is the paper title and the question is designed to pinpoint the corresponding paper. 3) As for comprehensive type, we basically follow single type, while the only difference is that provide the title and abstract along with the context, and ask the explorer to elicit the paper where the element comes from in the question. To improve the quality of QA pairs, adopt chain-of-thougee (Wei al., 2023) and hand-written category- -based hint prompts.
3.3 TRACKER
Regarding the tracker, its purpose is to wrap the previously generated natural language QA pairs into example files in accordance with specific formats. As explorer settings for different question types vary, employ different tracker settings: 1) With regard single and comprehensive questions,

---

Published as a conference paper at ICLR 2026
provide the tracker with the QA pair along with the information of the evaluation functions, including descriptions, parameters and use cases. The tracker is then asked to choose the suitable evaluation function, fill in the parameters and the answer format, and refine the QA pair accordingly. 2) In terms of retrieval questions, as we restrict the answer to be the exact title of the chosen paper, we simply fill in the example file with fixed evaluation function, parameters and answer format. As for multiple questions, the manual annotation consistently involves new papers outside our sampled collection, requiring real-time download and processing, which incompatible with our explorer agent. For simplicity, we propose rule-based combination of single examples: a) merging questions and answer formats with natural language templates, and b) combining evaluation functions with the logical function eva pralus watee.atte... ..con junct ion (function details in App. A.3)
For more specific explorer and tracker prompts, please refer App.
3.4 ACTOR
As for the actor, it aims interacting with the environment to collect trajectories for instruc- tion construction. In the outer environment, we include a database and a vectorstore containing corresponding information of the papers. Following /Zeng et al. (2023), we employ ReAct (Yao itt al. 2023) framework with three actions to interact with the environment. For each synthetic example, we use LLM as an actor produce an interaction trajectory (uo, Au...., Mi aj..... Mn, an). where represents the user's instruction, or the observation from the environment, and a, denotes the response from the actor, including a thought and an action.
message list manner
To avoid exceeding context length, we adopt the idea of sliding window and chunk the message list based on window size of 5, generating multiple instruction data from one trajectory. During training, for each chunked list, we mask previous message history and train the last turn only.
We also observe that, some errors appear frequently in the collected trajectories (e.g., attempts to utilize undefined parameters). To ensure data quality, we remove instruction data that ends with wrong action. For other instruction data, we reserve previous wrong actions and corresponding error information in the message list to guarantee error correction capability and coherence of thoughts.
4 EXPERIMENT
4.1
EXPERIMENT SETTINGS
Baselines
To comprehensively assess the LLMs on AirQA, we implement 8 baselines, including:
Trivial Baselines: 1) Question Only baseline, only the question and the corresponding answer format are available, 2) Title-Abstract baseline, the titles and the abstracts of the correspond- ing papers are provided alongside, and 3) Full-Text with Cutoff baseline, raw textual contexts extracted from the papers are given in limited length.
Retrieval Baselines: 1) RAG baseline, the question is sent to the vectorstore to retrieve relevant chunks, and LLMS answer the question based on retrieved contexts, and 2) Text2SQL baseline, where LLMs first generate SQL, then answer the question based on the query results.
Agentic Baselines: We employ ReAct Yao et al. 2023) framework with three actions: RE- TRIEVE, QUERY and ANSWER, corresponding to retrieving from the vectorstore, querying the database and generating the final answer. With this framework, we implement I) Agentic RAG and 2) Agentic Text2SQL baseline that only interact with the vectorstore and the database, respec- tively. 3) Agentic Hybrid baseline with all actions. Details on actions can be found in App.
Note that for both base and fine-tuned models, all evaluations are performed on AirQA to enable a clear and fair comparison. Examples generated by EXTRACTOR are for training only.
LLMs and Slyper-Parameen
We evaluate various LLMs on AirQA. For closed-source ones,
use NPPT-4-20024006 ol-mini-2024 12, exp-03-25. Regarding open-source LLMs, employ Qwen2.5-72B -----7B--nnnnt (Qwen et al.) 2025),
and Gemini-2. 2..-- S-Pro-
Llama-3.3- -nn--nutt (Hugging Face Team 2023). and DeepSeek-Ri (Guo et al. 2025). 5). As for parametrr,,,,,rs, the temperature is set 0.7 and top p is fixed to 0.95. Specifically, for

---

Published as a conference paper at ICLR 2026
reasoning models, the temperature is set to 0.6. The maximum retrieved tokens in each turn and the cutoff for full-text input are both limited to 5K. The threshold of interaction turns is 20 and the window size for the message history is 5. For closed-source models, we directly call their API services, while for open-source ones, we deploy them on NVIDIA A800 Tensor Core clusters using
vLLM
Kwon al. 2023).
Instruction Tuning For instruction tuning. we choose the Qwen2.5 family (Qwen et al. 2025 as base models. For all three agents in our EXTRACTOR framework, we employ Qwen2.5 5-3BB Instruct. While for the target model, unless otherwise specified, we utilize Qwen2.5-777 B-een2.--B---- B-strructt as the backbone. As for the training framework, we employ LLaMA-Factory (Zheng et al. 2024).
Regarding our synthetic instruction data, transform them into standardized ShareGPT format following Vicuna (Chiang et al., 2023), and as mentioned before, we only compute the loss of the model's last output during learning rate of 1 x 10 apply AdamW optimizer (Loshchilov & Hutter 2019) with a cosine learning scheduler and train for one epoch. The entire training process is conducted on Ascend 910B4 NPUs with 64GB of memory each.
tuning by setting mask.history true. By default,
The detailed hapar-parameters used in LLLMA-Factory for instruction tuning are listed in Table
4.2
MAIN RESULTS
Evaluation of Base Models To figure out different baselines' performance on the AirQA dataset, we choose two widely used models, GPT-4o and
as representatives of pro- prietary and open-source LLMs. Table shows that: 1) Trivial baselines perform poorly. Under Question Only setting, LLMs can only answer 5% of the questions correctly, demonstrating the quality of our AirQA dataset. 2) Provided more information sources, LLMs produce better an- swers. With just a glimpse into the database or the vectorstore, retrieval baselines elevate the overall accuracy by at least 8% compared to Question Only baseline. 3) As the interaction turn increases, LLMs explore the backend environment better. While both agentic baselines outperform their retrieval counterparts, Agentic Tex12SQL baseline exhibits significantly greater improvement, in- dicating that for structured retrieval, more searches enable step-by-step problem-solving while for unstructured retrieval regarding the vectorstore, a single query is sufficient for most circumstances.
Table Performance of different baselines on AirQA
Question Type
Element Category
Evaluation
Bascline
AVG
sgl.
multi.
refr.
comp.
text
table
image
form.
meta.
obj.
subj.
OPT-44-2024-08.
Question Only
8.55
1.86
1.04
5.63
4.35
1.41
10.63
2.36
0.00
3.95
5.54
4.41
Tidle-Abstract
11.40
5,26
0.00
5.28
5.96
4.23
8.70
4.72
2.46
4.07
9.97
5.78
Full-Test w/ Cutoff
33.90
8.05
0.69
5.99
13.53
7.51
13.53
12.60
18.03
9.94
21.05
13.16
31.62
18.75
16.55
20.29
12.68
16.91
17.32
18.03
18.19
18.56
18.30
Tex22QQ
21.08
6.81
7.64
17.25
14.01
8.92
12.08
16.54
14.75
11.41
18.28
13.40
Agentic RAG
34.19
8.36
15.63
29.58
21.36
18.78
26.57
22.83
24.59
21.36
24.10
22.15
Agentic Tex12SQL.
42.17
11.15
18.40
38.38
23.19
21.60
28.99
33.07
47.54
26.44
31.02
27.77
Agentic Hybrid
45.58
10.53
52.13
35.56
39.61
23.00
33.86
47.54
38.76
29.09
35.96
Qwen2.5 2.nn2.55-78B. Instruct
Question Only
9.69
1.86
0.35
5,99
2.74
3.29
10.63
3.94
5.74
4.52
4.99
4.65
Tidle-Abstract
17.66
6.19
0.00
8.10
8.05
6.10
12.08
7.87
7.38
6.44
13.30
8.43
Texl-TText w/ Cutoff'
36.18
0.00
7.04
12.56
11.27
16.91
14.17
18.85
11.86
19.67
14.13
RAG Texi2SQL
31.91
7.43
18.75
21.83
22.06
11.27
19.32
20.47
21.31
19.55
21.88
20.22
22.22
4.02
11.11
13.38
13.85
8.45
15.46
10:24
11.48
12.43
14.13
12.92
Agentic RAG
32.76
9.60
15.63
30.28
22.06
15.96
25.12
25.98
18.85
21.02
25.21
22.23
Agentic Tex12SQL
43.02
11.46
43.40
40.14
36.07
21.13
29.95
35.43
49,18
35.37
31.59
34.27
Agentic Hybrid
39.03
10.84
55.21
37.32
41.71
13.15
28.02
30.71
45.90
37.74
28.53
35.07
While baselines such as Question Only bascline exhibit relatively low performance, Agentic Hy- brid baseline consistently outperforms the others, allowing for comparisons between different mod- els. In Table 5. observe that: 1) Proprietary models outperform open-source ones, show-
https:///ocss wl wtt.://mmm...mmmmemmaaeeee
htm1
8

---

Published as a conference paper at ICLR 2026
ing a stronger research capabillity, while open-source models achieve performance comparable to closed-source ones. 2) Reasoning models' performance on this method is not satisfactory, possibly due to the incompatibility between their fixed reasoning formats and our framework.
Table 5: Performance of Agentic Hybrid baseline with different backbone models on AirQA.
Question Type
Element Category
Evaluation
Model
AVG
sgl.
multi.
retr.
comp.
text
table
Image
form.
meta.
obj.
subj.
GPT-4o
45.58
10.53
52.13
35.56
39.61
23.00
25.60
33.86
47.54
38.76
29.00
35.96
ol-mini
37 04
12.07
45. 14
24.65
35.43
14.55
22.22
22.83
36.07
31.07
26.04
29.61
Claude-3.7-Somae
45.30
15.17
58.68
27.46
43.96
22.07
24.64
27.56
44.26
39.32
29.64
36.52
Semini-2.5-PP
51.85
18.58
67.01
40.49
51.53
29.58
29.95
33,86
53.28
46.55
38.23
44.14
Dwwen2.5-728. Instruct
39.03
10.84
55.21
37.32
41.71
13.15
28.02
30.71
45.90
37.74
28.53
35.07
Llama-3.3- Insttrrtttt
42.71
24.30
32.37
8.92
21.74
19.69
30.33
28.47
19.94
26.00
DeepSeck-RI
41.03
11.46
41.67
22:54
35.10
15.96
20.77
20.47
39.34
30:40
26.59
29.29
Evaluation of Fine-tuned Models We fine-tune three models, 3B, 7B and 14B, with instruction data extracted from the 4,000 trajectories. Table 6 shows that, all three exhibit improved perfor- mance after training. The observed reduction in the 14B model's performance gain is considered reasonable and acceptable, because: 1) The actor agent in our EXTRACTOR framework in effect --s-ruuttt scores only 31.94%
serves as a teacher agent. The selected teacher model on AirQA, which represents the upper performance bound achievable through distillation. 2) While this diminishing return has long been a meaningful and widely discussed research question beyond the scope of this paper, we can observe that with EXTRACTOR, small models produce significantly less errors in predicting actions. As shown in Table 7 7B's error action rate drops from 38. 18.69% to 6.85%, and similar improvement is observed on 14B, with error action rate dropping from 31.63% 6.64%, indicating the effectiveness of our framework.
Table 6: Performance of models trained using EXTRACTOR and evaluated on AirQA. "FT" denotes fine-tuning.
Question Type
Element Category
Evaluation
Size
FT?
AVG
sgl.
multi.
retr.
comp.
text
table
image
form.
meta.
obj.
subj.
X
7.98
2.48
12.85
6.69
9.5
3.29
6,28
4.72
7.38
7.91
6.09
7.38
3B
14.81
3.72
51.74
13.73
29.79
4.23
10.63
11.81
17.21
24.97
8.59
20.22
X 16.24
3.72
26.39
15.85
19.48
8.45
13.53
4.72
14.75
17.29
10.25
15.24
7B
/
21.08
4.95
51.04
22.18
33.66
7.51
14.01
13.39
25.41
27.01
16.90
24.07
X
25.07
7.74
46.18
25.35
31.88
10.33
22.22
24.59
28
17.45
25.52
14B
/
25.36
6.19
52.08
26.41
34.94
7.51
20.77
19.69
28.69
30.96
16.62
26.81
32B
X
36.47
11.76
52.78
28.17
38.81
13.62
24.15
26.77
37.7
34.8
24.93
31.94
4.3
ABLATION STUDY
Component Ablation To evaluate the effect of the two proposed components, sliding window and eтror removal, we fine-tune two additional models: (1) one without either component, which raw trajectories as training data and computes loss over all model outputs, and (2) another that applies sliding window but retains all error actions. As shown in Table 7 sliding window yields a more pronounced improvement the overall score, and drastic reduc-
Table 7: Component Ablation.
Setting
(%)
Rate! %
buse model
15.24
38.69
EXTRACTOR
sliding
wio erroe removal
tion is also observed in error action rate, defined as @ emor actions
EXTRACTOR
actions
who removal
24.08
20.32
demonstrating the value of error removal improving the model's ability to generate valid actions.
Synthetic Data Scale We fine-tune another four models with more instruction data extracted from IK, 2K, 4K and 10K trajectories. Figure 4 shows that, as the number of trajectory increase, the scores of all question types raise consistently, demonstrating the scalability of our method.
9

---

Published as a conference paper at ICLR 2026
2407
21/8
71.6
137
2013
17.41
11.55
Figure 4: Performance changes with the increasing number of synthesized trajectories.
Due to page limit, more experimental results and analysis regarding AirQA and EXTRACTOR are presented in App. and App.
5 RELATED WORK
PDF-based Scientific QA Datasets Previous research has consistently focused the develop- ment of high-quality scientific QA datasets. Some datasets emphasize the accurate retrieval of pa-
pers based on specific descriptions (Ajith al.] 2024 He al.) 2025). while others concentrate on extracting detailed information from the text Baumgärtnerr et al. 2025 Dasigi et al. 2021 Lee
al. 2023 Jin et al. 2019 Saikh al. 2020). Recent work advance this field by incorporating
non-textual elements (Pramanick et al., 2025] Singh et al., 2024). Li et al. (2024) further extend this approach by introducing a paradigm for generating cross-document questions, while [Skarlinski er al. (2024) pay more attention to the combination of paper retrieval and detailed QA. Our AirQA dataset innovates this field by encompassing various question types and element categories, while
designing function-basee based instance-level evaluation.
useful tool for aligning LLMs with human instructions, but it requires corresponding outputs for specific instructions, thus heavily relying on high-quality training data. While manual annotation an effective method, is hard to scale up due to time and cost constraints. Wan et al. (2024), [aag et al. (2023c) and Chen al. (2025) introduced different methods for crafting synthetic instruction data from seratch, while Zeng et al. (023) proposed extracting data from interaction trajectories to leverage existing datasets. In this work, we introduce multi-agent framework, EXTRACTOR, to automate both example generation and trajectory collection, facilitating instruction data synthesis.
Instruction Tuning and Synthetic Data Instruction tuning
CONCLUSION
In this work, we manually annotate a multi-modal multi-task multi-paper dataset (AirQA) with instance-level evaluation, and multi-agent framework (ExTRACTOR) for instruction data synthe- sis. Evaluations demonstrate the quality of our dataset, indicating the challenges current models face in scientific QA, while experiments on instruction tuning highlight the effectiveness of the frame- work. Future works include: 1) exploring other RI.-based methods for further improvements, and 2) extend the PDF-based tasks to other knowledge intensive domains, including law and medicine.
ACKNOWLEDGMMENT
We would like to thank Haoran Wang, Jingyi Zhang, Ye Wang. Yuxun Miao, Danyang Zhang, Hanqi Li, Zichen Zhu, Situo Zhang, Senyu Han, Dingye Liu, Wenjie Sun, Hanchong Zhang, Nan Jiang, Liangtai Sun, Da Ma, Hankun Wang, and Zhihan Li for their careful annotation the AirQA dataset. This work is funded by the China NSFC Projects (62120106006, U23B2057, 92370206, 62576212) and Shanghai Municipal Science and Technology Project (25X010202846).
REFERENCES
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges, 2024. URL https: /arxiv.viv.. wwwxxiioorr orq/ abs/ 2402. 00157
10

---

Published as a conference paper at ICLR 2026
Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. Lit- search: A retrieval benchmark for scientific literature search, 2024. URL
htg/abs/240/ 18940
Inc. Artifex Software.
Pymupdf a python binding for mupdf. https:/ /ttp:::///pp...pp readthedocs i en/eatsst/ 2023. Version 1.24.9, accessed on January 25, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL http:://arxvvvovr abs/abss/22200 13923
Tim Baumgärtner, Ted Briscoe, and Iryna Gurevych. PeerQA: A scientific question answering dataset from peer reviews. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Compu- tational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 508-544, Albuquerque, New Mexico, April 2025. Association for Computational Linguisties. ISBN 979-8- 89176- 189-6. doi: 10. 10.18653/v1 1/855/v1/202555 h/a00/.naaacllloo......a naacl-long. 22. URL
org/
2025. naacl - oog.ggg....................... 227
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irv- ing, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Lau- rent Sifre. Improving language models by retrieving from trillions of tokens, 2022. URL https://erxxt /ttp://////xxi..r///////////////
oww.oog///aoooo awwg///////// 2212. 04426
Mingyang Chen, Haoze Sun, Tianpeng Li. Fan Yang. Hao Liang, Keer Lu. Bin Cui, Wentao Zhang, Zenan Zhou, and Weipeng Chen. Facilitating multi- turl function calling for Ilms via composi- tional instruction tuning, 2025. URL /tL/ttts///axii.orrgg atwpaaxxvvvv......a abs/ 2410. 12952
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
03 -000-viina/////
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-ssecking questions and answers anchored in research papers, 2021. http::
//arxivvorg/////a abb./////////. abt. 0310
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, 7. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao. Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu Hui Li, Jianzhong Guo, Ji- ashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li. Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. Chen, L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuip- ing Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xi- aotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaox- iang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang.
11

---

Published as a conference paper at ICLR 2026
Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yut- ing Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhenda Xie. Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie. Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl incentivizes reasoning in Ilms through reinforcement learning. Nature, 645 September 2025. ISSN 1476-4687. doi: 10. 1001000/4115--025-000000 doi
URL
http://adt http::///....oo./g//........ org/10. 1038/333000 /015541155666 025 09422-2
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training, 2020. URL https://axxxv..orr/abb////
08909
Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang. Hang Li, and Weinan E, Pasa: An Ilm agent for comprehensive academic paper search, 2025. URL
abs/2501 10120
Xanh Ho, Jiahao Huang, Florian Boudin, and Akiko Aizawa. Lim-as-a-judge: Reassessing the performance of Iims in extractive qa, 2025.
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang. Hao Wang. Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Ilm agents: A survey, 2024.
URL httts::////axtt
.rgg http//bs//2222/// 02716
Hugging Face Team. Llama 3.3-70b-instruet model.
Betts::/huggingfaace co/ meta- aataa-lane 1laaa/aaaai///a LLLaaa 3. 3- 703 Instruct| 2023. Accessed: 2023-02-11.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yuu Armand Joulin, Sebastian Riedel, and Edouard Grave Atlas: Few-shot learning with retrieval augmented language models, 2022. URL htthhtts://aatt http://aaxvvv/// ort.
03299
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)), pp. 2567-2577, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 1..11853/v1/D11-11 1259. URL http:::///claaamoollogg//
219-12597
Ehsan Kamalloo, Shivani Upadhyay, and Jimmy Lin. Towards robust qa evaluation via open lims. In Proceedings of the 47th International ACM SIGIR Conference on Research and Developmens in Information Retrieval, SIGIR '24, pp. 2811 1-26. New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704314, 4. doi: 10. (0.11445/6627772.6577775
URL https:
7/doi org/10. 0010//0111//11111 365777777
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoic.. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jachyeon Kim, Hong-In Lee, and Moontae Lee. QASA: advanced question answering scientific articles. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). International Conference on Machine Learning, ICML 2023. 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19052- wwwpss//s00.... /etp/eeee3.n... html PMLR, 2023. URL httt:::::/ppoceeedngeo httpaaeessiaasrmrmmm....ssss///e
12

---

Published as a conference paper at ICLR 2026
Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, and Arman Cohan. M3sciqa: A multi-modal multi-documeee scientific qa benchmark for evaluating foundation models, 2024.
httttt/////aaaa org/ abs /ttp:/241..000 04075
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
/ /arxii..rrg/abs//7111
Hannes Mühleisen and Mark Raasveldt. duckdl:: DBI Package for the DuckDB Database Man- agement System, 2024. URL htttps://r.. /ttpp:://..uucddb.... www.kb.
org/
R package version 1.1.3.9017,
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022. URL https://arxiv.
axg/abs/211 12. 09332
Azizi Othman. Manus ai: Capabilities, limitations, and market position, 03 2025.
Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. Spiqa: A dataset for mul- timodal question answering on scientific papers, 2025. URL https://axxiv.org//
2407.055 09413]
Qwen,
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL htthhtts:// http//axxvvv..//. org. abs /e5./24112 2412. 15115
Tanik Saikh, Asif Ekbal, and Pushpak Bhattacharyya. ScholarlyRead: A new dataset for scien- tific article reading comprehension. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi. Hitoshi Isahara, Bente Mae- gaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. seille, France, May 2020. European Language Resources Association. ISBN
Mar- 34-4.
URL
ww2.:rrg/020..... 2020 Irec- 1. 1.6c//
Timo Schick, Jane Dwivedi-Y Yu,, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, / 02t.//222200777.1 04761
Shruti Singh, Nandan Sarkar, and Arman Cohan. Scidga: A deep reading comprehension dataset over scientific papers, 2024. URL https:: .rg/ abs 241111331111111818
Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, and Andrew D. White. Language agents achieve superhuman synthesis of scientific knowledge, 2024. URL httht::////aiiii/ 0ts//209.133743
Yuwei Wan, Yixuan Liu, Aswathy Ajith, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, and Ian Foster. Sciqag: A framework for auto-generated science question answering dataset with fine- me--graneed evaluation, 2024. URL holL:tttts://axxvvccoo
http://aa/2222.99990 09939
Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang. Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. Mineru: An open-source solution for precise document content extraction, 2024. /
Cunxiang Wang. Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkum Hu, Zheng Zhang, and Yue Zhang. Evaluating open- QA evaluation. In Thirty-eevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023a.
URL htt.://pss//pppreviieeee wttprr/wwwweeeeeeeooorreieeiiee
13

---

Published as a conference paper at ICLR 2026
Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xi- angzhou Guo, Chengming Li, Xiaohai Xu, et al. Milvus: A purpose-built vector data manage- ment system. In Proceedings of the 2021 International Conference on Management of Data, pp-
2614-2627, 2021.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solvv prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023b. URL
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: /ptoocee/ppoceeeeeeeee meuriiippnnnnnnnneoo...e...... #15ee243547deeee Fbd053c cetecccc44445aa c4a845aa--bst Abst htm
neuries httppsssccpppp..ppeeeee cc/paaper/222//hasii
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023c. URL http:htts: www.vvoorrgggbo..... wttgg/bb//21111 10560
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning large language models,
2023. URL /tttps://axiv..ooo org/ abs 22222120111111
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representationss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL htt.https:/// httttp:://ppenrevviit hegpaaereeeee..ttfofouuuu..ddd.
net htt//oformmmddeee Berpiddddd___uu_______ vluYUI-X
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang. Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun- ing: Enabling generalized agent abilities for llms, 2023. URL peetts://arxtie erg/ abs/ 2310. 12823
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- tics. httphttp .tg/ abs wttp://a///....1333333
ETHICS STATEMENT
To construct our AirQA dataset, use 13,956 papers in artificial intelligence domain. Most papers
utilized in our AirQA dataset can be downloaded from arXiv ? unavailable on arXiv, we use OpenReview Fl and ACL Anthelogy three websites are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0), and we use the papers in accordance to their usage terms, with no private, sensitive, or personally identifiable information used in this work.
For some conference papers
| as supplements. All
LLM USAGE STATEMENT
In accordance with the ICLR 2026 policy on the use of Large I Language Models (LLLMs), we disclose that LLMSs were used exclusively ceneraa-purpose writing aids, such for polishing grammar and improving readability. LLMs did not contribute to research ideation, experiment design, data analy- sis, or result interpretation, and thus played no role that would qualify authorship or substantive contribution.
httes://acllnnthaoogg..oo
14

---

Published as a conference paper at ICLR 2026
A AIRQA DATASET
A. EXAMPLE FORMAT
Each data instance in AirQA is represented as a JSON dictionary containing the following fields:
uuid: Globally unique uuid of the current task example.
question: The user's question about the given papers.
answer.formee The requirements for the output format of LLMs, c.g., "a Pythnn list of text strings" or "a single float number", so that LLMs can form their answer accordingly and we can evaluate the answer conveniently.
tags; A list of tags denoting different question types, element categories and evaluation types. Feasible tags include ["single", "multiple", "retrieval", "text", "image", "table", "formula", "metadata", "subjective", "objective"]. corresponding to question types, modal categories in Section 2.1 and evaluation types in Section 2.2
anchor.pde list of PDF uuids that are directly related to or explicitly mentioned in question, provided in single and multiple questions.
reference.pdf A list of PDF uuids that may may not help answer the question, only provided multiple questions.
conference: A list of conference name followed by year, provided in retrieval and compre- hensive questions.
evaluator: dictionary containing 2 fields, eval-fune and eval.kwargs which defines how to evaluate the model outputs. Concretely,
the "eval_ func" field defines the name of our customized Python function (or metric) which used to compare the predicted result and the expected ground truth; the "eval "orralkwargs" field defines the arguments for the corresponding evaluation function, which usually contain the gold reference answer and other optional parameters.
A.2 METADATA FORMAT
The metadata of each paper is organized in structured JSON format, capturing key bibliographic and content-related attributes, as shown below:
uuid: A universally unique identifier (UUID) assigned to this specific data sample.
title: The full title of the academic paper.
conference full: The complete official name of the conference where the paper was pub- lished or presented. For example, "Annual Meeting of the Association for Computational Lin- guistics".
The standardized or abbreviated name of the conference, commonly used in citations file naming. Examples include "ACL", "NeurIPS".
conference:
year: The year in which the paper published presented the conference.
volume: The volume information of the conference proceedings journal in which the paper appears, if applicable. For example, "NeurIPS 2023 poster".
bibteex A string containing the full BibTeX citation entry for the paper.
authors: An ordered list of the authors who contributed the paper.
pdf-url: A direct URL linking to the downloadable PDF file of the paper. The link should point to an actual PDF file and therefore must end with "'pdf"'.
p path: The local file system path where the PDF is saved. The file should be renamed using the UUID to ensure consistent and collision-free naming.
num.pages: An integer value indicating the total number of pages in the PDF document.
abstract: The abstract of the paper, which is concise summary of the research objectives, methodology, key findings, and implications.
15

---

Published as a conference paper at ICLR 2026
t tldr: "Too Long; Didn't Read" summary a brief, high-level summary of the paper's main contribution, typically one to two sentences.
tags: A list of keywords or topic tags associated with the paper.
Table The checklist of the 19 used evaluation functions, including their genres, categories, names, and descriptions.
Genre
Category
Function
Description
Evaluate the output against the answer using ex- act boolean match.
eval bool exact match
Evaluate the output against the answer using act float match with variable precision or toler- ance.
eval. float exact match
Evaluate the output against the answer using ex- act integer match.
eval. int_.exatt.mate exach_match
match
Evaluate the output against the answer using ex- act string match
eval, wtsisstttnneeeiee sxact_match
Evaluate the output against the answer using fuzzy match provided by FuzzyWuzzy.
string fuzzy match
objective
Evaluate the output against the answer recur- sively by parsing them both as Python-style lists or dictionaries.
wtt.stuuctuue.objee _exact_match
Evaluate whether the output is included in the an- swer list.
eval element included
Evaluate whether each element in the output list included in answer list.
set
Evaluate whether the output list overlaps with the answer list.
eval.clement _iitentt_iittuuu list_ oveelap
ppaper.rleevanee
Evaluate whether the retrieved paper is the same the reference answer.
retrieval
referenceeanneee .answer
nnl.referenceansweer _with_llm
Evaluate the output against the reference answer using LLMs.
eval. candidate reference referrnct.
Evaluate whether the output matches any candi- date reference answer.
.answer with Ilm
eval_scoring points_nww with
Evaluate whether the scoring points are all men- tioned the output using LLMs.
semantic
subjective
eval. paratia partiallscoring points .with_Ilm
Evaluate whether the scoring points are partially mentioned in the output using LLMs. Evaluate whether the reference answer and other scoring points all mentioned the output ing I.IMs.
eval reference answer and scoring-points withtoottttttthnlll..
Evaluate the mathematical equivalence between the output and the answer formatted in Latex us- ing LLMs.
camplex_mmat_f66e
formula
ula with Ilm
Evaluate the conjunction of multiple evaluation functions. The output passes the evaluation and only if all the elements the output pass the cor- responding sub-evaluations.
eval conjunction
Evaluate the disjunction of multiple evaluation functions. The output passes the evaluation if and only least one of the element in the output passes the corresponding sub-evaluation.
logical
cval.diinnnction
Evaluate the negation of an evaluation function. The output passes the evaluation if and only if doesn't pass the original evaluation function.
eval vval_negation
A.3 EVALUATION FUNCTION
In Table g we list the detailed names and descriptions of all 19 evaluation functions.
Here we present three representative cases corresponding to three distinct categories of evaluation functions: objective functions, subjective functions, and logical functions.
16

---

Published as a conference paper at ICLR 2026
"example";
"eaal Func": "eval_stringeexat__mo
eval kwargs":
"gold": "Italian",
"Lowercase": true
Listing 1: Objective Function Case
In case (Listing I the evaluation function compares the predicted answer with the gold answer "Italian", ignoring case sensitivity due to the "lowercase" parameter being set to true. This means "italian", "ITALIAN", or "ItAliAn" would all match "Italian".
"example";
reval_funch "eval @ewa__eeference_assser....
"eval_kver "eval_kwargs": {
ansfeerenc__anseer"
"Artificial intelligence is branch of computer science focused building systems that perform tasks that typically requize human
9 intelligence.",
"question": "What is artificial intelligence?"
Listing 2: Subjective Function Case
In case (Listing 2) the evaluation checks if the predicted answer conveys the same meaning as the reference answer about artificial intelligence, in response the question "What is artificial intelligence?".
The prompt used for conducting the subjective evaluation above is presented example among several possible formulations within this evaluation category, and is shown below.
Subjective Function Prompt Example
You are an intelligent judgement system who is expert determining whether a predicted answer matches the reference answer in terms of semantic meaning and intent, based the input question. You will he given the question, the reference answer, and the predicted answer. And you need to provide the final decision following format:
""xxt
Truc/Falsc
Notice that:
1. Remember to wrap the final judgement with triple backticks.
2. The final decision string must exactly be "True" "False" without any extra character or punctuation.
Any other text will considered as incorrect.
The structure and format of the predicted answer do not matter. We only care about the semantic content, compared to the reference answer. Minor differences in grammar. structure, or formatting should be ignored the core meaning preserved.
Now, let's start!
[Question]: {question]
[Reference Answer]: (reference answer)
[Predicted Answer]: [nnewer
Let's think step-by-step. and then provide the final judgement.
17

---

Published as a conference paper at ICLR 2026
"example":
"eaal =
"eval_disjunctions
eval wwval_kwargs:
"eval_func_list":
wwwesttiinneettttooeeeeeee @ with 11m"
"eval Lara___warrs_lsst"
"gold": "role-oriented routing", "lowercase": true
"reference sanreer""_ "It routes messages, requests, 4 or tasks based on the roles or
+ responsibilities of the recipients, rather
4 than simply by their identity static
4 attributes.",
"question": "What's the most important idea of S role-oriented routing?"
Listing Logical Function Case
In case (Listing 3) the disjunction evaluation function checks if at least one of the specified eval- uation functions returns a positive result. The first function, seval_stringeexact.mmm whether the predicted answer matches the gold standard "role-oriented routing" in case-insensitive manner. The second function, "eval_refcrence wwweeesswswerwhhhh..e.... answer sufficiently addresses the question about the most important idea of role-oriented routing, as described in the provided reference answer. If either condition is satisfied, the evaluation returns 1; otherwise, returns 0.
verifies
evaluates whether the predicted
A.4 ADDITIONAL DATASET STATISTICS
Here we include another two statistics on paper volume in Figure and question lengths in Figure
160
Number of Questions
Question
140
ICLR
ACL
120
23.9%
23.3%
100
3.2%
Others
NeurIPS
49.7%
Length
Figure 5: Conference distribution of the pa- pers used.
Figure 6: Distribution of question and answer
format lengths (in tokens) by count.
18

---

Published as a conference paper at ICLR 2026
B AGENTIC BASELINE
In this section, we present detailed information of the actions implemented in the agentic baselines. Each action can be called in Python-style manner, e.g., Retrieve (query="Is there any work about the topic structured RAG?", limit=4)
"act "action_type": "Retrieve",
"description": "Given query text, retrieve relevant context from the Milvus vectorstore.", "observation: "The osservation space is the retrieved top-ranked + entries from the Milvus vectorstore based on queries.", "parameters":
"query":
"type": "str",
"required"; true,
"description": "The query text will be enceded and used to 9 search for relevant context. You can rephrase the original
question obtain clear and structured query requirement."
b
"limit":
"type": "int", "required"; false,
"default": 5,
"description": "The number of top-ranked context to 9 retrieve. Please set it to positive integer to limit
the number of returned results. Extremely large 1imit values may be truncated."
}.
"use aise_cascs":
"example";
"query": "Is there any work about the topic structured 4 RAG?"
"explanation": "Retrieve top pieces about certain topic."
"example": {
"query": "What's the learning rate for training the S ResNet model?",
"limit": 4
b "explanation";
"Retrieve detailed information about the learning rate for training the ResNet model. The top most relevant entries will be returned based on the query."
"action_type" "Query",
"description"; "Generate SQL query to retrieve the desired information from the DuckDB database. Flease refer to the concrete database schema to produce a valid and executable + SQL.",
"bssevvation": "The observaticn space is the execution result of the SQL query. You do not need worry about the actual exccutien, we will perform for you. the SQL failed execute, we will return the error message. Extremely long SQL 4 output will be truncated.", "parameters";
"sq"":
19

---

Published as a conference paper at ICLR 2026
"type": "str", "required"; true, "description"; "The cencrete DuckDB SQL query to exccute and 7 retrieve results."
},
"use rise_cases".
"example": " "sql": "SELECT abstract FROM metadata WHERE paper_ic _id 123445678'
eexplanation" "Get the abstract of the paper with paper_id 9 '12345678' from the metadata table in the DuckDB database."
"example"; "sq1"
eages.page_ise.. pwwppaaappppeepp _age_number FROM images JOIN pages JOIN metadata ON i
"SELECT pages.
page_id AND re____e_____ddd_____
metadata. netadata.paper_id WHERE metadata.paper__. 112345678' AND images. aiagge_caption LIKE "Fiiuure 38';"
}
"explanation: "Find which page in the paper with paper_id 4 "12345678' contains Figure 3."
"action_type" "Answer",
"description": "When you take this action, the retrieved results
4 suffice answer the user question, PLEASE STRICTLY ADHERE TO THE ANSWER FORMAT FOR THE CURRENT QURSTION.",
ooserrvttinn: "There is no observation for this terminal action,
4 since it indicates the completion of the task and end of the interaction.",
"parameters";
"answer";
"type": "Any",
"required"; true,
"description": "The final answer user question."
}, "use_case""
"example"; "answer": 42
b.
"explanation"; "The final answer is 42."
"example":
"answer": ["Results", "Discussion")
"explanation";
"The final answer list of strings: 4 ('Results', "Discussion'),
Listing Detailed Action Format
20

---

Published as a conference paper at ICLR 2026
C
SUPPLEMENTARY EXPERIMENTS AND SETTINGS
C.I DETAILED HYPER-PARAMETERS FOR INSTRUCTION TUNING
Table 9: Hyper-parameters for Instruction Tuning.
Parametr
Default Value
Finetuning Type
LoRA
LoRA Target
all
LoRA Rank
16
LoRA Alpha
LoRA Dropout
0.05
Cutoff Length
4,096
Mask History
true
Gradient Accumulation Steps
16
Learning Rate
1 x 10-4
Train Epochs
1.0
Learning Rate Scheduler
Cosine 0.1
Warmup Ratio
C.2 PREPROCESSING
Besides collecting papers and metadata illustrated in Section 2.3 parse the papers with PyMuPDF (Artifex Software 2023) and MinerU (Wang et al. 2024 and populate rele- vant information into the relational database DuckDB (Mühleisen & Raasveldt 2024). Ad- ditionally, we segment raw documents into chunks of 512 tokens, encode the chunks with all Mi (Wang at.. 2020), and insert the vectors into the vectorstore Milvus (Wang)
et al. 2021)
C.3 ABLATION ON VLMs
To further investigate whether direct access to visual information improves performance, we imple- ment an additional action, "View", for the Agentic Hybrid baseline.
"action type": "Viex",
"description"; "You can retrieve the visual information of the paper + by taking this action. Please provide the paper id, the page number, and the optional bounding box.", "observation"; "The observation space is the image that you want to view. We will show you the image according to your parameters. 9 The errer message will be shown if there is any problem with 4 the image retrieval.",
"parameters";
"paper_idt {
"type": "str",
"required": true, "description": "The paper id to retrieve the image."
b. "page_number" "type": "int",
"required"; true,
"description": "The page number (starting from of the + paper to retrieve the image."
}
"bounding_box { "type": "L "List [f]cat]], "required": false,
21

---

Published as a conference paper at ICLR 2026
"default": 1],
"description": "The bounding box image to retrieve. + The format is [x_min, y_min, delta_x, delta_yl. The 7 complete page will be retrieved if not provided.
},
"use "use_case cases":
"example":
"paper_ist" "12345678",
"page_number":
"bounding_box": []
b "explanation"; "Retrieve the image of the third page of the paper with id 12345678."
"example";
"paper_id"; "12345678", "page_numberr: 5, "bounding_box":
51.1,
204.3,
333.0, 0,
13.8
1
"explanation": "Retrieve the image of the fifth page of the paper with id 12345678, with bounding box of [51.1, < 204.3, 384.1, 218.1]."
Listing 5: Details for "View" action.
Through this action, LLMs access the image content encoded in base64 format by specifying the paper ID page number, and the bounding box of the relevant region.
Table 10: Performance of Agentic Hybrid with "View" action AirQA dataset.
Size
"View"?
text
table
image
form.
meta.
AVG
X
31.88
10.33
17.87
18.90
21.31
24.24
32B
31.56
8.92
20.77
15.75
25.41
24.56
X
36.55
15.96
23.67
20.47
36.88
30.34
72B
36.55
15.96
24.64
22.83
36.07
30.78
We conduct experiments with Qwen2.5-VL. family (Bai et al. 2025) shows that, with direct access to images, the performance on visual questions improves, though further methods for enhancement remain to be explored.
backbone model. Table TO
C.4 FINE-TUNING BASED ON AIRQA
To examine the relative effectiveness synthetic data compared to manually annotated data, we fine-tune another model with examples directly from the AirQA dataset. Due to the unquantifi- able nature of manually annotated trajectories, we continue to use Qwen2.5-322 S-32B-Innstuue as teacher model for trajectory generation.
Table II indicates that, model fine-tuned on 400 examples from AirQA achieves performance com- parable that trained 1,000 automated generated examples. It is worth noticing that while
22

---

Published as a conference paper at ICLR 2026
Table 11: Comparison between models fine-tuned on examples directly from the AirQA dataset and examples generated by the EXTRACTOR.
Dataset
Count
sgl.
multi.
retr.
comp.
AVG
16.24
3.72
26.39
15.85
15.24
AirQA
400
15.67
4.95
48.26
13.73
19.98
EXTRACTOR
1000
15.95
2.48
48.96
14.44
19.74
manual examples to be more effective, they come at higher cost. It takes 20 minutes for human to generate an example and only 20 seconds for EXTRACTOR.
C.5 HUMAN STUDY
To provide reference for the difficulty of the AirQA dataset, we recruit 3 students with expertise in artificial intelligence to answer 98 questions sampled from our AirQA dataset. They are strietly prohibited from using any form LLM and are only allowed search the internet. Each question has a time limit of 20 minutes.
Table 12: Performance of human experts on AirQA dataset.
Question Type
Element Category
Evaluation
AVG
sgl.
multi.
refr.
comp.
table
image
form.
meta.
obj-
subj.
64.29
54.00
52.17
56.82
53.12
56.07
67.05
50.00
55.00
58.52
52.58
56.63
Results in [2soow that AirQA is a highly challenging dataset, even human experts are only able achieve relatively high score within the time limit, rather than a perfect one. Meanwhile, all three participants report difficulty in identifying the correct papers, particularly when multiple sources are involved, and two additionally note challenges understanding domain-specific terminology.
C.6 STATISTCS ON TIME COST
Evaluation Here provide empirical estimate: evaluating all 1,246 examples takes approx- imately 35 minutes and costs $0.056. This process can further accelerated using simple paral- lelization techniques if needed.
Agentic Baselines In Table 13 we list the time and cost per example for two models and three agentic baselines for reference.
Table 13: Statistics of the number of interactionss), accumulated prompt completion token(s), time consumption, and LLM cost per sample with different models and agentic methods on AirQA.
Prompt Token(s)
# Completion Token(s)
Model
RAG Method
Turn(s)
Time (s)
Cost (S)
Agentic RAG
7.53
39870
658
58
Qwwen2.5-72B-insrruet
Agentic Tex12SQL.
6.45
42991
790
70
Agentic Hyhrid
5.95
62533
767
62
Agentic RAG
4.59
13231
365
13
0.0367
GPT-4o
Agentic Text2SQL
7.26
32957
815
22
0.0905
Agentic Hybrid
5.08
35909
566
18
0.0954
Example Synthesis Though we have discussed in App. C.4 that LLM-based methods are far more efficient than manual annotation, we still believe that a rough cost analysis would be helpful to offer empirical reference for future work.
The results for each question type are computed by averaging the time and cost of generating 10 examples with 1-mii. A back-of-the-envelop calculation indicates that it costs roughly
23

---

Published as a conference paper at ICLR 2026
Table 14: Average time and cost for synthesizing example. The time and cost for "multiple" question type is estimated by directly doubling that of "single".
Time (s)
Cost ($)
sgl.
18.6
0.041
multi
37.2
0.082
retr.
5.2
0,002
comp.
18.5
0.039
$160 and 22 hours to produce 4,000 examples (1,000 per question type). Given the complexity of the tasks, this is reasonably efficient compared with human annotation, and can be further improved by using open-source models parallelization.
24

---

Published as a conference paper at ICLR 2026
D SUPPLEMENTARY ANALYSIS
D.1 ERROR ANALYSIS ON GPT-40
To further illustrate the bottleneck of our AirQA dataset, we randomly sample 60 examples (15 for each question type) where GPT--4o + Agentic Hybrid produces incorrect answers. Through manual analysis, we identify the following five root causes that ultimately lead to mistakes:
Lack Context: The agent fails to use the given tools to find relevant snippets.
Over Confidence: The agent chooses to generate the answer too early.
Missing Paper: For questions that involve multiple papers, the agent fails to realize/find other papers.
4. Textual Reasoning:
The agent successfully retrieves the key snippet but fails to understand it. 5. Visual Reasoning: The agent fails to retrieveeunderstand paratextual information.
Table 15: Error Analysis of Agentic Hybrid baseline with GPT-4o as backbone.
Category
sgl.
multi
retr.
comp.
Total
Lack of Context
5
3
9
6
23
Over Confidence
3
2
5
3
13
Textual Reasoning
2
1
1
5
Missing Paper
0
8
0
0
Visual Reasoning
5
0
1
The analysis shows that the dominant cause differs across question types, while generally, current models still exhibit limitations in long-term planning and multi-modal reasoning, suggesting the need for new methods that better balance planning and acting.
D.2 ERROR ANALYSIS ON FINE-TUNED 7B
Similarly, we conduct an error analysis fine-tuned (ween.5-7B-Innstree Besides the aforemen- tioned five causes, we identify another reason that ultimately leads to failures:
6. Repetition: The agent consistently predicts the same action.
Table 16: Error Analysis of Agentic Hybrid baseline with fine-tuned 7B as backbone.
Category
sgl.
multi.
retr.
comp.
Total
Lack of Context
4
4
9
3
20
Textual Reasoning
6
3
2
Over Confidence
2
3
2
3
10
Repetition
3
1
4
Visual Reasoning
2
1
0
3
Missing Paper
0
2
0
0
The analysis indicates that fine-tuned models still exhibit limitations in textual retrieval and com- prehension, highlighting areas for future improvement.
D.3 IMPROVEMENT ANALYSIS ON 7B
We further investigate which aspects of small models are improved by EXTRACTOR. We sample 10 examples for each question type where fine-tuned Qwen2.5-7B-Insst Instructt outperform untrained baseline, totaling 40 examples. For each example, we examine the source of improvement and categorize into of three main contributing factors:
25

---

Published as a conference paper at ICLR 2026
The model develops clearer understanding of the overall retrieval process and the specific actions required at each step, allowing to plan more effectively and identify useful intermediate steps.
Better Retrieval Strategy:
2. Better Retrieval Behavior: rectly interpreting schemas and providing appropriate tool parameters.
The model interacts more accurately with the environment, cor-
Better Understanding and Reasoning: The model demonstrates deeper comprehension of the question and context, producing more coherent and reliable reasoning.
Table 17: Improvement Analysis Fine-Tuned 7B.
Category
sgl.
multi.
retr.
comp.
Total
Strategy
13
Behavior
14
Understanding
13
The analysis shows that the dominant contributor varies across question types. While improved retrieval behavior is not always the primary factor, reductions in tool usage errors are consistently observed, enhancing both retrieval behavior and strategy as the model learns to use tools more effectively. This finding aligns with the component ablation in Section 4.3 and reinforces the claim that our training framework enhances the model's 's ability generate valid actions.
26

---

Published as a conference paper at ICLR 2026
E SYNTHESIS PROMPT
E. EXPLORER PROMPT
Here we showcase prompt templates of explorers discussed in Section 3.2
Explorer Prompt for Single and Comprehensive question types
You are an intelligent annotation system who is expert in posing questions. {description}
Your output should be in the following format:
[Thought]:
Your thought process.
"xxt
[Question]: Your question here.
[Answer]: Your answer here.
Notice that:
Remember to wrap your output (except [TThought)) with triple backticks.
Don't 't include the answer in the question or in the reasoning steps.
Your question should be objective possible.
Your answer should concise and clear.
[hint)
Let's 's think step-by-step, and then provide the final question and answer.
{context)
Here, (description) stands for the description of the task, {hint) } includes additional hints, and {context] represents the corresponding context for question generation.
For example, for single question type and table element category, the description "You will be given an Al research paper, and your task is to generate question based on the content of the table HTML format and the caption of the table.", ", the hint prompt is ". Try not to include the word "table" in your question.", and the context prompt is:
Context Prompt for Single type, Table category
The caption of the table is follows:
*"txt
{caption] <
The content of the table is follows:
"html
[content]
where the caption and the content are the raw text caption and the table content in HTML format respectively.
Explorer Prompt for Retrival question type
You are an intelligent annotation system who is expert in posing questions. You need to pose question based the title and abstract of paper, where the answer the question should be the titlc of the paper. That say, you need to describte the contribution respondents can identify the paper. Don't 't include the title itself in the question. Now let's start!
feature of the paper question, so that
[Title]: {title} [Abstract]: (abstract)
Your output should he in the following format:
Your thought process.
"xxt
Your question here.
27

---

Published as a conference paper at ICLR 2026
Note that, you should wrap your output with triple backticks.
For retrieval question type, we use the different explorer prompt shown above, and for multiple question type, there is no need of an explorer, as we simply combine two single questions.
E.2 TRACKER PROMPT
Here we present prompt templates of trackers discussed in Section
Tracker Prompt for Single and Comprchensive question types
You are intelligent annotation system who is expert in reviewing questions.
You will given question an answer. You should adjust the question and answer, adapting them to the evaluator's 's requirements. The descriptions, parameters and use cases of the evaluators are provided below:
{evaluator)
Note that:
If you want the predicted answer list to be exactly same with the gold answer list, use match", don't use eeva_ellmment list inncluded""" If your evaluation involves list matching, and the order doesn't 't matter, set 'ignore_ondee order' to 'true". If the order matters, set agnore_order" to ffasse. If you
eval_sstructured wtt.sttccturd.oojccttcooo
that the answer unique, there aren't other equivalent answers, and any rephrase will change the semantic meaning of answer, you can use 'eval string exact_matcl Otherwise, you should use eval reference answer with Ilm'. Generally, we recommend using "eval reference answer with Ilm for subjective questions, and staa_sstiieeeact.maaae for single-word answers.
Your output should he in the following format: [thought]: Your thought process. ""xtt
[question]: Modified question.
[evaluator]: The evaluator you choose.
[answer former:: The format that the respondent should follow in order to pass the evaluator. c.g. "Your answer should be single python list containing two strings, the first element of the list is the abbreviation of the baseline, the second element of the list the full name of this baseline, [answer]: Modified answer. [tag]: A single subjective" or "objective" without explanation. Whether the evaluator involves LLM, "sub- jective' if it involves LLM, otherwise "objective".
Note that:
Remember to wrap your output (except thought) with triple backticks. DON'T T INCLUDE ANSWERS, HINTS OR KEY POINTS IN [question] OR [answer IN ANY FORM, ESPECIALLY WHEN YOU TRY TO ILLUSTRATE [answer format] BY GIVING EXAMPLES. [answer_formn
[ormaa]
be provided the respondent along with the [question]. [question] and [an- swer swer_forma format] together form the who question that will be presented to the respondent. [question] focuses on the question itself, formal focuses on the format of the answer.
You should present [evaluator] in JSON format, given in the use cases. And your [answer] should be able to pass the evaluator.
question and answer based on the evaluator's requirements, don't change original meaning of the question and answer.
You can modify
When the question involves percentage, and the percentage is an exact value, not an approximate value. try "eval float exact match" "eval int exact match", while indicating the decimal places in formaj.
Here' re the original question and answer:
txt
28

---

Published as a conference paper at ICLR 2026
[question]:
{question] } [answer]: {answer]
Let's think step- and then provide the final arguments.
Here, (evaluator) contains the detailed information of the 19 evaluation functions, and fquestion and (answer) stand for the question-answer pair generated by the explorer. The evaluator prompt is in the following format:
Tracker Prompt for Single and Comprehensive question types
## [function]
### Description
deserripiion>
### Parameters [parameters]
### Use Case(s)
{use_case)
which contains the name, the description, the parameters and the use cases of the functions.
F LIMITATIONS AND BROADER IMPACTS
Although precise question answering datasets AirQA for academic papers can enhance research efficiency, this work still has certain limitations: 1) As large language models incorporate more aca- demic papers during pre-training, some questions can be answered solely based on their parametric knowledge; 2) The current dataset is limited to English-languag papers in the field of artificial intelligence, and its coverage remains to be improved: 3) While most questions can be evaluated using objective scoring functions, long-form answers inevitably rely large model-based evalua- tion, which may affect the consistency and stability of the evaluation results as the continual training and update of these LLMs. For broader social impact, LLM-based agents become increasingly robust and practical through more refined agent-level finetuning, their improved question-answering capabilities can help researchers significant time literature review and detail retrieval, avoid reinventing the wheel, and even assist in building personalized knowledge bases of academic pa- pers.
29


Reviews:
[{"review_id": "n6KhpuRqWN", "reviewer": "Reviewer_KUWD", "summary": "This paper introduce lots work to \n0) frame question answering as multi turn agentic calling \n1) create enviornments to generate agentic tractergories recognize the limits of human-input\n2) showcase scaling law to small 7b/14B model", "questions": "Why do you only compute loss on the model's last output? this means you are not doing any supervision to the actual tractegory \n\nFor the human baseline, does the error distrubtion is super similar to the gpt-4o error on this benchmark?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "There are novelty associated with this approach, constructing tools like vector search, SQL. The questions themselves \n\nThere are a human baseline that helps with establishing a good baselines. this is a really important baseline\n\nThe paper also mentioned error-analysis of gpt-4o, and classifier them into different classes, i think the lack of context and over confidence are really important to address in future work.", "weaknesses": "The multi-turn setup might be more ideal for RL instead of SFT, but I recognized most of the work are more on engineering the multi agent prompts, setting up the right environment, and reading through raw responses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a6DXK0bRkN", "reviewer": "Reviewer_msPL", "summary": "The paper introduces M4PQA, which is a framework for AI research question-answering. The benchmark is a human-annotated Multi-Modal Multi-Task Multi-Paper Question Answering dataset. The authors recruited 26 students and collected 1,246 examples and 13,948 papers in AI and created a set of questions with different types (single, double, etc.) and element categories (text, image, table, etc).", "questions": "No particular question at this time.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "M4PQA targets research QA and retrieval rather than generic QA. The function-based, instance-level evaluation is a sensible move away from free-form semantic metrics and reduces grading ambiguity. The agentic baselines and the released action space/environment help standardize comparisons for tool-using agents, and the dataset statistics (balance across question types; mix of elements) suggest measured coverage.", "weaknesses": "Evaluation relies substantially on synthetically generated content/trajectories. The paper does not describe where human work ends and LLM synthesis begins. The “26 students” section says annotators read papers, pose questions, and wrap evaluators, but elsewhere EXTRACTOR agents generate QAs and trajectories; it remains unclear, per item type, whether humans created or only vetted the questions and how often LLM-authored items passed review. Labeling is described as function-based with optional subjective paths; however, the paper should state which splits or item types invoke an LLM judge. \n\nReported performance also appears tightly coupled to tool access (search capabilities, for example). Table 4 shows large jumps when agents can query the environment; that’s informative, but it confounds “reasoning” with “tool use.” \n\nQwen fine-tuning with EXTRACTOR trajectories predictably improves performance of small models, but it doesn’t clarify whether gains come from better tool formats, better retrieval behavior, or overfitting to the dataset’s answer-format conventions; no cross-family or cross-benchmark transfer is shown. \n\nThe dataset appears calibrated to be hard, which is fine, but the paper implicitly values difficulty without probing whether item construction systematically reduces performance on weaker models and favors stronger ones (e.g., formatting-sensitive outputs).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffwIiisrpn", "reviewer": "Reviewer_b9vj", "summary": "This paper introduces M4PQA, a question-answering dataset containing 1,246 questions derived from documents in the field of artificial intelligence. The dataset is designed to evaluate the capabilities of large language model (LLM) agents in answering questions about research papers. It was manually curated by students with AI expertise and includes four categories of questions such as single-document, multiple-document, retrieval, and comprehensive, and targets different components of research papers (e.g., text, tables, figures). The authors demonstrate that state-of-the-art models such as GPT-4o and Qwen2.5 struggle to effectively answer questions from M4PQA under various prompting and agentic settings. Furthermore, the paper proposes Extractor, a framework that leverages LLMs to generate synthetic QA trajectories for agentic fine-tuning.", "questions": "- How reliable are the LLM-based (subjective) evaluation functions? Additionally, since LLM-based evaluations can be computationally expensive, doesn’t this approach introduce a significant cost whenever a method is evaluated on M4PQG? It would be helpful if you could provide an approximate cost estimate.\n- From line 430, it appears that Table 6 presents the results of the extractor framework. If this is correct, could you clarify why the caption currently states M4PQG?\n- In Section 3.2, could you elaborate on how the context extraction process is performed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a well-motivated and relevant problem.\n- The proposed dataset is comprehensive, encompassing a diverse range of question types and context variations.\n- The evaluation is thorough, testing the dataset across multiple models and agentic approaches.", "weaknesses": "- The effectiveness of the proposed extractor framework is not convincingly demonstrated. Since the framework is entirely LLM-based, its reliability and the quality of the generated questions are uncertain. The paper lacks dedicated experiments evaluating the quality of the synthetic trajectories produced. In addition, Table 11 indicates potential quality issues with the Extractor framework. A comparison with other automatic question-generation methods could have provided stronger empirical support.\n- If scalability is a primary motivation for introducing the extractor framework, a cost or efficiency analysis should be included to support this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8i5ngAHTm2", "reviewer": "Reviewer_ukJ4", "summary": "The authors introduce M4PQA, a human-annotated comprehensive paper QA dataset in the field of artificial intelligence, with 13,948 papers and 1,246 questions, that encompasses multi-task, multi-modal and instance-level evaluation. They also introduce EXTRACTOR, an automated framework for instruction data synthesis which can perform example generation and trajectory collection without human intervention.", "questions": "- Any reason not including ICML, AAAI or big vision conferences (e.g., CVPR)?\n\n- Small typo: figure 3 “based on it” \n\n- The main purpose of extractor is to automatically produce questions/trajectories/answers that are then used to finetune the LLM on these tasks? If you want to use it to extend your proposed dataset you still need a human expert in the loop to make sure the questions/trajectories/answers are correct.\n\n- Did you perform SFT only? Any RL methods that can be potentially be used (e.g., GRPO)?\n\n- Did you use the same papers as your dataset with extractor to generate questions/trajectories/answers? Which model was used for generating these trajectories?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- Expert annotated from students with AI expertise (n=26).\n\n- Expert validated study for difficulty (n=3).\n\n- Paper well written.\n\n- Experiments well designed and good ablations.\n\n- Authors address shortcomings of other scientific papers benchmarks.\n\n- Authors show that performance raises consistently as data scales up, highlighting the scalability of the framework.\n\n- Function-based evaluation into QA domain which is systematic and reproducible.", "weaknesses": "- Even though the work is good and addresses gaps in other datasets I am not confident how significant is the novelty compared to other similar works. Based on table 3, one can run on different benchmarks from the literature to assess on different question types (Single, Multi, Retrieval, ..., Image Form. Meta). This is the main reason for my score. \n\n- How is extractor validated? Any human intervention?\n\n- Maybe if the annotators had more specific instructions like identifying the key components of the papers and coming up with key questions would improve the quality of questions in terms of usefulness? Perhaps identifying questions that require some kind of thinking/logical deduction?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)