Deep Literature Survey Automation with an Iterative Workflow

## Abstract

Abstract Automatic literature survey generation has attracted increasing attention, yet most existing systems follow a one-shot paradigm, where a large set of papers is retrieved at once and a static outline is generated before drafting. This design often leads to noisy retrieval, fragmented structures, and context overload, ultimately limiting survey quality. Inspired by the iterative reading process of human researchers, we propose IterSurvey, a framework based on recurrent outline generation, in which a planning agent incrementally retrieves, reads, and updates the outline to ensure both exploration and coherence. To provide faithful paper-level grounding, we design paper cards that distill each paper into its contributions, methods, and findings, and introduce a review-and-refine loop with visualization enhancement to improve textual flow and integrate multimodal elements such as figures and tables. Experiments on both established and emerging topics show that IterSurvey substantially outperforms state-of-the-art baselines in content coverage, structural coherence, and citation quality, while producing more accessible and better-organized surveys. To provide a more reliable assessment of such improvements, we further introduce Survey-Arena, a pairwise benchmark that complements absolute scoring and more clearly positions machine-generated surveys relative to human-written ones. The code is available at https://github.com/HancCui/IterSurvey_Autosurveyv2 .

## 1 Introduction

Automatic literature survey generation has recently attracted growing attention due to its potential to help researchers quickly grasp new domains, identify key trends, and reduce the burden of manual reviews. Following  Wang et al. (2024b), current systems generally adopt a multistage pipeline (Liang et al., 2025; Yan et al., 2025; Wang et al., 2025): The process begins with a topic description, usually consisting of a few tokens, which is directly used to retrieve a large collection of candidate papers.
Due to the context window limitation of large language models (LLMs), the retrieved papers are divided into multiple groups, for each, an LLM agent generates a survey section outline based on the corresponding subset of papers.
These group-level outlines are subsequently merged into a global draft outline.
Once the draft outline is obtained, the system performs section-wise retrieval to collect references for section writing and then generates the corresponding text passages.
Finally, a global review and integration process is applied, in which the drafted survey is iteratively polished to improve readability and overall consistency.

The above approach takes a ”one-shot” planning paradigm, retrieves a comprehensive set of papers and construct a global outline from a single, static starting point. This approach, however, leads to several limitations.
First, retrieval can be imprecise and static due to reliance on a short topic description (often just a few tokens) as the retrieval query (Sun et al., 2019; Azad & Deepak, 2019; Wang et al., 2020). Such coarse queries fail to capture a field’s nuances and are never refined, leading to noisy and incomplete paper collections.
Second, the survey structure can be incoherent (Fabbri et al., 2019; Gidiotis & Tsoumakas, 2020; Yang et al., 2023a).
Since outlines are generated for each paper group independently and subsequently merged, the global structure lacks coherence and often misses important cross-group connections.
Third, injecting overly long contexts introduces distraction and context overload (Liu et al., 2023; Wu et al., 2024).
Feeding entire papers into LLMs not only exposes them to large amounts of peripheral information, such as dataset details or experimental setups, which distracts from the conceptual structure needed for survey writing, but also places unnecessary pressure on the limited context window of the model.

Figure: Figure 1: Overview of IterSurvey and Survey-Arena.
Refer to caption: x1.png

In contrast, human researchers rarely attempt to grasp an entire field in a single shot. Instead, they follow an iterative reading process: starting with a small set of core papers, summarizing key contributions, and gradually expanding to related directions as their understanding deepens (Bates, 1989; Asai et al., 2023).
Inspired by this workflow, we propose an iterative planning paradigm for automated survey generation. At its core lies a recurrent outline generation module that incrementally retrieves, organizes, and integrates evidence through a planning agent equipped with stability checks and stopping criteria, mitigating the brittleness of one-shot pipelines that rely on static queries and fragmented merges. Central to this process are paper cards, structured semantic abstractions that distill each paper into contributions, methods, and findings. Unlike conventional abstract-based inputs, these cards serve as fine-grained evidence units that guide both outline construction and section drafting, ensuring coherence and faithful citation across iterations. Finally, a global review and integration stage employs a reviewer–refiner loop to enforce consistency and clarity across sections, while an integrated figure–table generation pipeline compiles candidate visualizations, automatically checks them for layout and readability, and revises them to meet academic presentation standards.
This design inherits the advantages of iterative human reading: retrieval is progressively refined rather than static (Jiang et al., 2023), the outline develops as an organically coherent structure rather than a patchwork (Zhang et al., 2025a), and paper cards enforce fine-grained evidence grounding that avoids distraction from peripheral details (Cachola et al., 2020; Wu et al., 2024).

Comprehensive experiments validate the effectiveness of our incremental paradigm. IterSurvey consistently outperforms all baselines across multiple dimensions, with recurrent outline generation yielding more coherent structures and paper cards improving citation accuracy without sacrificing precision. These advantages are further confirmed by human evaluation, where experts also favor the outputs of IterSurvey over competing systems. While these results confirm the superiority of IterSurvey, we find that absolute scoring struggles to reliably quantify the performance gap against human-written surveys (Yang et al., 2023b; Oren et al., 2023; Ye et al., 2024). In the LLM evaluation community, similar concerns have led to the development of Chatbot Arena Chiang et al. (2024), which adopts pairwise human preference judgments to overcome the noisiness and inconsistency of absolute ratings. Inspired by this paradigm, we further contribute Survey-Arena, the first benchmark to our knowledge that evaluates synthesized surveys through direct, pairwise ranking against a corpus of human-written exemplars. This approach provides a more robust and interpretable assessment of system quality by directly positioning it relative to a human-level baseline.

Our contributions are threefold.

- •
We propose recurrent outline generation, which iteratively retrieves, reads, and updates outlines with paper cards and outline–paper grounding, while encouraging the model to explore new directions.
- •
We develop a new framework: IterSurvey, which produces finer-grained outlines and supports multi-modal inputs and outputs for more comprehensive surveys.
- •
We construct Survey-Arena, a pairwise evaluation benchmark that ranks machine-generated surveys alongside human-written ones, enabling more reliable and interpretable assessment of survey quality.

## 2 Related Work

#### Automated Survey Generation

Recent automated survey generation systems largely adopt a ”one-shot” paradigm, where a static outline is constructed upfront before content generation. This approach is evident in pipeline-based systems like AutoSurvey (Wang et al., 2024b), which employs a hierarchical paradigm, and SurveyForge (Yan et al., 2025), which utilizes a memory-driven scholar navigation agent. Other frameworks focus on enhancing this initial outlining step through reference pre-processing; for instance, SurveyX (Liang et al., 2025) introduces an AttributeTree to extract key information, while HiReview (Hu et al., 2024) generates a hierarchical taxonomy tree. Tackling the challenge from a technical scalability perspective, SurveyGo (Wang et al., 2025) leverages the LLM×MapReduce-V2 algorithm to handle long contexts within this paradigm. In contrast, our framework treats the outline not as a static blueprint but as an evolving knowledge structure. Through a dynamic, recurrent mechanism, the outline is continuously updated as the system iteratively engages with the literature, resulting in comprehensive and coherent synthesis.

#### Evaluation of Automated Surveys

Evaluating machine-generated surveys is inherently challenging. Building on insights from automated peer review (Yu et al., 2024; Jin et al., 2024; Weng et al., 2025), prior works (Wang et al., 2024b; Yan et al., 2025; Liang et al., 2025) commonly adopt an LLM-as-a-judge paradigm with manually designed criteria, assessing dimensions such as coherence, coverage, and factuality. Citation quality is typically measured with NLI-based protocols (Gao et al., 2023), and Yan et al. (2025) additionally evaluate coverage by comparing system outputs with human-written surveys. While absolute scoring by LLMs provides useful fine-grained signals, it has also been noted to suffer from inconsistency and calibration issues (Ye et al., 2024; Latona et al., 2024), making system-level comparisons less reliable. In contrast, pairwise judgment which is widely used in chatbot evaluation (Zhao, 2025; Chiang et al., 2024) and peer review (Zhang et al., 2025b), offers more stable and interpretable assessments, but has not yet been applied to survey evaluation. To fill this gap, we introduce Survey-Arena, the first benchmark that ranks machine-generated surveys against human-written exemplars, providing both robust comparison across systems and a clearer positioning relative to human-level quality.

## 3 IterSurvey

An overview of IterSurvey is shown in Fig. [1](https://arxiv.org/html/2510.21900v1#S1.F1), and its three core stages are detailed below.

### 3.1 Recurrent Outline Generation

Figure: Algorithm 1 Description of the recurrent outline generation process.

Outline generation is a central component of automatic survey construction, as it requires understanding the research domain, identifying its subfields, and synthesizing individual papers. Alg. [1](https://arxiv.org/html/2510.21900v1#alg1) shows the overview of the generation process. The outcome is a hierarchical framework that summarizes the domain, where each node in the hierarchy is represented by a title and an accompanying description.
Given a topic query, our goal is to enable the model to integrate retrieval with inductive reasoning, so that it can systematically explore the literature and produce a comprehensive outline for the target domain.
To this end, we design recurrent outline generation.

#### Paper Card Pool.

The paper card pool organizes retrieval keywords together with their associated papers in a structured mapping.
For each keyword $K_{i}$, we retrieve $n$ candidate papers and extract $m$ of the most relevant references, forming the set:

$$ $\mathcal{P}_{i}=\{p_{i}^{1},p_{i}^{2},\dots,p_{i}^{n+m}\}.$ $$

At iteration $i$, the system pops one keyword $K_{i}$ together with its associated paper set $\mathcal{P}_{i}$ from the pool.
Each paper $p_{i}^{j}\in\mathcal{P}_{i}$ is converted into a paper card

$$ $c_{i}^{j}=\texttt{PaperCard}(p_{i}^{j}),$ $$

which distills the paper into its contributions, methods, and findings.
The collection of paper cards is denoted as $\mathcal{C}_{i}=\{c_{i}^{1},c_{i}^{2},\dots,c_{i}^{|\mathcal{P}_{i}|}\}$.
Overall, the paper card pool can be represented as a mapping

$$ $\mathcal{Q}=\{\,K_{i}\mapsto\mathcal{C}_{i}\mid i=0,1,\dots\,\},$ $$

where each keyword $K_{i}$ is associated with the corresponding set of paper cards $\mathcal{C}_{i}$.

#### Outline updating.

The outline updating process begins with an empty initial outline, denoted as $O_{0}$. At each step, the outline is refined using the current outline $O_{i}$, the active keyword $K_{i}$, and a mini-batch of paper cards drawn from the pool.
Specifically, let $\mathcal{B}_{i}\subseteq\mathcal{C}_{i}$ be a batch of paper cards sampled from the set of cards associated with $K_{i}$.
The model produces a candidate update

$$ $\tilde{O}_{i+1}=g(O_{i},\mathcal{B}_{i},K_{i}),$ $$

where $g(\cdot)$ denotes the outline updating function.
This procedure is repeated iteratively, with batches $\mathcal{B}_{i}$ of paper cards popped from the paper pool $\mathcal{Q}$ under the current keyword $K_{i}$, until all cards associated with $K_{i}$ are consumed and integrated into the outline.
To ensure stability and promote refinement, the candidate update is accepted if its similarity to the previous outline exceeds $\tau$:

$$ $O_{i+1}=\begin{cases}\tilde{O}_{i+1},&\text{if }\texttt{Sim}(O_{i},\tilde{O}_{i+1})\geq\tau,\\ O_{i},&\text{otherwise}.\end{cases}$ $$

#### Keyword expansion.

When all keywords $K_{i}$ has been fully consumed, the system explores new directions by proposing additional keywords. The goal is to identify potentially relevant aspects of the domain that have not yet been covered.
Formally, new keywords are generated as

$$ $K_{i+1}=f(O_{i+1},K_{i},\dots,K_{0}),$ $$

where $f(\cdot)$ denotes a keyword generation function that takes the updated outline and the history of queries as input, and proposes candidate keywords for further exploration.
The corresponding paper set $\mathcal{P}_{i+1}$ is then retrieved and pushed into the pool $\mathcal{Q}$, thereby guiding the next iteration.

#### Stopping condition.

Let $N_{i}=|\mathcal{P}_{0}\cup\mathcal{P}_{1}\cup\dots\cup\mathcal{P}_{i}|$ denote the total number of consulted papers up to iteration $i$.
The process terminates when either (i) $N_{i}\geq N_{\min}$ and the stopping signal

$$ $s=h(O_{i+1},K_{i},\dots,K_{0}),\quad s\in\{0,1\},$ $$

indicates that the outline is sufficiently complete, or (ii) $N_{i}\geq N_{\max}$.
Here $h(\cdot)$ is a decision function which takes the evolving outline and the query history as input and outputs whether further exploration is necessary.
This design ensures that the outline is not terminated prematurely, while also preventing excessive exploration.

#### Post-processing.

After termination, the recurrent process produces a research-oriented outline $\tilde{O}$, which is further refined into a writing-oriented survey outline:

$$ $\hat{O}=\texttt{Refine}(\tilde{O}),$ $$

where $\texttt{Refine}(\cdot)$ reorganizes the structure, inserts standard survey components such as ‘Introduction’ and ‘Future Directions’, and ensures conformity with academic conventions. Finally, we perform paper–section relinking, where all consulted papers are reassociated with the corresponding sections of the final outline $\hat{O}$.
This guarantees that each section of $\hat{O}$ is grounded in concrete evidence, providing a reliable foundation for subsection drafting.

### 3.2 Section Drafting Guided by Paper Cards

A distinctive feature of our framework is that section drafting is entirely guided by paper cards, which serve as fine-grained, structured representations of the literature.
Given the refined outline $\hat{O}$, each section or subsection is written by conditioning on its description $d_{j}$ together with the relevant pool of cards.
Specifically, for a given subsection with description $d_{j}$, the system retrieves a set of additional reference papers $\mathcal{P}_{\text{sec}}^{j}$ and converts them into paper cards $\mathcal{C}_{\text{sec}}^{j}$.
In contrast to previous work, our framework benefits from the paper–section relinking established during outline construction: each subsection is already associated with a pool of consulted papers from earlier iterations.
This enriched evidence base, combining $\mathcal{C}_{\text{sec}}^{j}$ with the relinked cards, provides the model with a stronger foundation for subsection writing. Formally, the $j$-th subsection is generated as

$$ $S_{j}=\texttt{Draft}(d_{j},\mathcal{C}_{\text{sec}}^{j}\cup\mathcal{C}_{\text{link}}^{j}),$ $$

where $\mathcal{C}_{\text{link}}^{j}$ denotes the set of paper cards relinked to subsection $j$.
During drafting, the model is required to cite the provided references, and the citations are mapped to their corresponding papers.

### 3.3 Global Review and Integration

The final stage of survey generation goes beyond local drafting. It performs a global review-and-refine process that integrates sections into a coherent survey and enriches the survey with automatically generated figures and tables.

#### Textual Review-and-Refine.

We adopt a reviewer–refiner loop that involves two collaborative LLM roles. The reviewer takes the entire survey draft as input to capture the global context but then focuses its critique on a specific section or subsection. This design ensures that feedback on local content is always grounded in an understanding of the overall narrative. The reviewer provides detailed suggestions covering aspects such as clarity of exposition, consistency of terminology, logical alignment with preceding and following sections, and stylistic fluency. The refiner then incorporates these suggestions to revise the targeted section, producing a polished update that fits better into the survey as a whole. This loop is applied sequentially across all sections and iterated multiple times, progressively enhancing readability, improving cross-section coherence, and strengthening the global structural integrity of the survey.

#### Figure–Table Integration.

In addition to textual refinement, we extend the refinement process to include multimodal elements, to further enhance readability.
For each section, the model first generates visualization requirements, such as tables with structured comparisons or figures with explanatory diagrams, together with natural language descriptions.
Based on these descriptions, candidate figures and tables are synthesized.
The compiled outputs are then fed back to an LLM for quality assessment, enabling automatic detection of issues such as oversized layouts or unreadable text.
The LLM provides corrective suggestions, which are applied to improve the final visualizations.
Finally, the text is refined again to ensure that all generated figures and tables are properly referenced within the survey.

## 4 Experiments

### 4.1 Experimental Settings

#### Implementation Details.

Following  Wang et al. (2024b), we adopt GPT-4o-mini as our generation model for its balance of responsiveness and cost.
Our retrieval database contains 680K computer science papers from arXiv, with PDFs converted into structured Markdown using MinerU (Wang et al., 2024a) for consistent formatting. The details of the retrieval process are provided in App. [A.1](https://arxiv.org/html/2510.21900v1#A1.SS1).
In outline generation, the system consults 1000–1200 papers, with a maximum of 8 sections.
For section drafting, each subsection retrieves up to 60 additional relevant papers, combined with those linked during outline generation.
Finally, we apply two iterations of the review-and-refine loop to enhance coherence across sections and improve overall readability.
Illustrative outputs compared with AutoSurvey are provided in App. [A.8](https://arxiv.org/html/2510.21900v1#A1.SS8).

#### Baselines.

We compare IterSurvey with a set of baselines, ranging from simple retrieval-augmented generation (Naive RAG), which directly drafts from retrieved documents, to more advanced state-of-the-art systems. Specifically, we evaluate against AutoSurvey (Wang et al., 2024b), the first systematic framework for this task; SurveyForge (Yan et al., 2025), which combines heuristic outline generation based on the logical structures of human-written surveys with a memory-driven scholar navigation agent for high-quality retrieval; and SurveyGo (Wang et al., 2025), which employs the LLM$\times$MapReduce-V2 algorithm to address the long-context challenge. We also compare with SurveyX (Liang et al., 2025), which introduces an Attribute Tree-based outlining mechanism; however, due to access restrictions, we include SurveyX only in arena experiments. All methods are evaluated on the same retrieval database with generation hyperparameters aligned to their original settings for fairness.

### 4.2 Automatic Evaluation Results

#### Evaluation Setup.

We employ multiple complementary protocols to evaluate the quality of generated surveys. On the 20-topic suite from Wang et al. (2024b), we adopt multi-dimensional scoring with LLM-as-a-judge. Content quality is assessed along three dimensions: coverage, structure, and relevance followed from Wang et al. (2024b). Besides, citation quality is evaluated using the NLI-based protocol of Gao et al. (2023), reporting both recall and precision: Citation Recall measures whether all statements in the generated text are fully supported by the cited passages, while Citation Precision identifies irrelevant citations to ensure that references are pertinent and directly support the claims. To improve scoring stability and reliability, prompts are standardized and judges must provide a rationale before assigning scores. For additional robustness, we aggregate outputs from three judge models: GPT-4o, Claude-3.5-Haiku, and GLM-4.5V.(^1^11Specifically, we use chatgpt-4o-latest, claude-3-5-haiku-20241022, and glm-4.5v.) Full prompts are provided in App. [A.7](https://arxiv.org/html/2510.21900v1#A1.SS7).

#### Results.

The results on the 20 topics from Wang et al. (2024b) are reported in Tab. [1](https://arxiv.org/html/2510.21900v1#S4.T1). Statistical significance was confirmed via paired t-tests, indicating that IterSurvey consistently outperforms baseline models ($p<0.05$). We summarize the main observations below.

- •
Overall superiority. IterSurvey consistently outperforms all baselines across both content and citation quality, achieving the highest overall average score ($4.75$). This demonstrates that the proposed framework is effective and robust across multiple evaluation dimensions.
- •
Improved structural quality. On the structure dimension, IterSurvey achieves the best score ($4.72$). This improvement stems from the recurrent outline generation mechanism, which iteratively explores the literature and refines the outline, resulting in clearer organizational planning and stronger cross-sectional coherence.
- •
Enhanced citation quality. IterSurvey also achieves superior citation performance. While maintaining the same precision as AutoSurvey, it improves recall to $0.70$. This advantage is enabled by paper cards, which provide fine-grained summaries of individual papers and thus allow for retrieving and citing a broader yet still accurate set of supporting references.

Together, these results confirm that recurrent outline generation, paper cards, and outline–paper grounding synergize to produce surveys that are both structurally coherent and rigorously evidenced.

**Table 1: Comparison of different methods in terms of content quality and citation quality.**
| Methods | Content Quality | Citation Quality |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Coverage | Relevance | Structure | Avg. | Precision | Recall |  |
| NaiveRAG | $4.42_{\ \!\pm\!\ 0.50}$ | $4.85_{\ \!\pm\!\ 0.36}$ | $4.20_{\ \!\pm\!\ 0.73}$ | $4.49_{\ \!\pm\!\ 0.41}$ | $0.39_{\ \!\pm\!\ 0.16}$ | $0.40_{\ \!\pm\!\ 0.15}$ |
| AutoSurvey | $4.50_{\ \!\pm\!\ 0.29}$ | $4.80_{\ \!\pm\!\ 0.16}$ | $4.62_{\ \!\pm\!\ 0.24}$ | $4.64_{\ \!\pm\!\ 0.15}$ | $0.64_{\ \!\pm\!\ 0.08}$ | $0.64_{\ \!\pm\!\ 0.08}$ |
| SurveyForge | $4.57_{\ \!\pm\!\ 0.50}$ | $4.82_{\ \!\pm\!\ 0.39}$ | $4.60_{\ \!\pm\!\ 0.56}$ | $4.66_{\ \!\pm\!\ 0.40}$ | $0.59_{\ \!\pm\!\ 0.09}$ | $0.59_{\ \!\pm\!\ 0.09}$ |
| SurveyGo | $4.37_{\ \!\pm\!\ 0.49}$ | $4.83_{\ \!\pm\!\ 0.38}$ | $4.27_{\ \!\pm\!\ 0.63}$ | $4.49_{\ \!\pm\!\ 0.40}$ | $0.50_{\ \!\pm\!\ 0.11}$ | $0.63_{\ \!\pm\!\ 0.12}$ |
| \rowcolor[HTML]BCA88D IterSurvey | $\textbf{4.58}_{\ \!\pm\!\ 0.50}$ | $\textbf{4.95}_{\ \!\pm\!\ 0.22}$ | $\textbf{4.72}_{\ \!\pm\!\ 0.45}$ | $\textbf{4.75}_{\ \!\pm\!\ 0.30}$ | $\textbf{0.64}_{\ \!\pm\!\ 0.06}$ | $\textbf{0.70}_{\ \!\pm\!\ 0.07}$ |

### 4.3 Human Evaluation Results

To further assess the quality of the generated surveys, we conducted a blind, pairwise study (Novikova et al., 2018; Chiang et al., 2024) with seven PhD-level experts.
For each evaluation, experts were presented with an anonymized survey pair and asked to select the superior one based on multiple quality dimensions, including coverage, relevance, structural coherence, and overall quality, which is more objective and stable than ranking based on absolute scores (Herbrich et al., 2006; Sakaguchi et al., 2014). To control annotation cost, the human study was limited to direct comparisons between IterSurvey and two leading baselines: AutoSurvey and SurveyForge. Inter-rater agreement is reported in App. [A.2](https://arxiv.org/html/2510.21900v1#A1.SS2).
Results, as shown in Fig. [2](https://arxiv.org/html/2510.21900v1#S4.F2), indicate that IterSurvey is consistently preferred over AutoSurvey and SurveyForge by domain experts, especially in terms of structure and overall quality.
This trend aligns with our automatic evaluation, where recurrent outline generation also demonstrated stronger coherence and organization.
The consistency between expert judgments and automatic metrics further highlights the robustness of IterSurvey in generating high-quality surveys.

Figure: (a) IterSurvey vs AutoSurvey
Refer to caption: figs/iter_vs_auto.png

### 4.4 Survey-Arena: Pairwise Comparison and Ranking

#### Dataset construction.

Previous automatic evaluation methods typically assign an absolute score for each dimension, which struggles to fully capture the performance gap between machine-generated surveys and human-written ones. To move beyond absolute scores, we constructed the Survey-Arena benchmark. The benchmark spans ten research topics. For each topic, we manually selected five high-quality, human-written surveys to serve as a performance baseline. To ensure comparability, all surveys for a given topic were chosen from a narrow six-month submission window, a process that required careful verification to ensure each topic had a sufficient number of suitable papers. We further confirmed their quality and influence via non-trivial citation counts on Google Scholar. The retrieval database for all machine-generated surveys was correspondingly frozen to the same time period to guarantee fairness. The full list of topics and papers is available in the App. [A.4](https://arxiv.org/html/2510.21900v1#A1.SS4).

#### Evaluation protocol.

For each topic, all possible pairs of a machine-generated survey and a human-written survey are constructed. To ensure robust evaluation and mitigate positional bias, each pair is judged in both directions (A vs. B and B vs. A), following Li et al. (2024). A panel of three distinct LLMs, namely GPT-4o, Claude-3.5-Haiku, and GLM-4.5V, serves as the judges for each comparison. Elo scores are computed from these aggregated pairwise outcomes to generate rankings for all systems.

#### Results.

We report two key evaluation metrics: Avg. Rank, which indicates the mean position among all surveys, and $>$Human%, which reflects the proportion of topics where a system surpasses human surveys. The topic-wise outcomes from Survey-Arena are visualized in Fig.[3](https://arxiv.org/html/2510.21900v1#S4.F3), and the aggregated rankings are summarized in Tab.[2](https://arxiv.org/html/2510.21900v1#S4.T2).

Figure: Figure 3: Elo scores of Survey-Arena results across topics. The radar plot shows the Elo scores for each system across all topics, providing a topic-wise comparison.
Refer to caption: x2.png

Each system is evaluated by its average rank across all surveys (including 5 machine-written surveys and 5 human-written ones) and by the proportion of topics where it surpasses human surveys.
The results show that IterSurvey consistently achieves the best overall performance among automatic survey generation systems, with an average rank of $4.0$ and surpassing human-written surveys in $60\%$ of topics.
These findings highlight that IterSurvey not only outperforms competing methods but also approaches human-level quality across diverse domains.

#### Meta Evaluation.

To assess the reliability of Survey-Arena judgments, we compare the rankings produced by Survey-Arena for human-written surveys with citation counts on Google Scholar, which serve as an external signal of impact. Specifically, we compute Spearman’s $\rho_{s}$ by measuring the correlation between Arena-derived and citation-based rankings for each topic, and then report the average across topics. For relevance scoring, we treat citation counts as an indicator of relevance and compute nDCG directly over the ranking lists. As a comparison, we also use the rankings derived from absolute scoring and compute their consistency and nDCG. This allows us to evaluate how well the different ranking methods align with citation-based rankings.

Results are shown in Tab. [3](https://arxiv.org/html/2510.21900v1#S4.T3). Compared with the scoring-based approach, pairwise judgment achieves higher agreement with citation-based rankings, yielding a Spearman’s $\rho_{s}$ of $0.410$ and nDCG@$2/3=$ $0.834/0.873$. This indicates that when models are asked to directly compare two surveys, they more reliably identify the superior one, producing rankings that better align with human impact signals. These findings support pairwise evaluation as a more robust protocol for Survey-Arena.

### 4.5 Generalization on Survey-Lacking Topics

To examine whether automated survey generation can succeed in areas without existing surveys, we construct a subset of eight research topics (listed in App. [A.5](https://arxiv.org/html/2510.21900v1#A1.SS5)) where no human-written reviews are available. Such settings are common in emerging domains and pose greater challenges, since there are no canonical structures to imitate and the literature is often sparse and fragmented. This setup tests whether a system can autonomously organize the field into a coherent, well-grounded survey.

We compare IterSurvey against AutoSurvey and SurveyForge under this setup, and the results are presented in Tab. [4](https://arxiv.org/html/2510.21900v1#S4.T4).
Our method achieves the highest average score ($4.63$), consistently outperforming both baselines across content and citation quality.
Notably, IterSurvey shows clear advantages in structural quality ($4.63$) and citation recall ($0.67$).
These gains highlight the benefits of recurrent outline generation, which encourages iterative query expansion and literature exploration rather than relying on a fixed set of initial retrievals.
Combined with paper cards providing fine-grained evidence abstraction, this mechanism enables IterSurvey to construct coherent survey structures and incorporate broader supporting references even in areas where survey conventions are absent.

**Table 4: Comparison of different methods on survey-lacking topics.**
| Methods | Content Quality | Citation Quality |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Coverage | Relevance | Structure | Avg. | Precision | Recall |  |
| AutoSurvey | $4.00_{\ \!\pm\!\ 1.12}$ | $4.20_{\ \!\pm\!\ 1.20}$ | $4.00_{\ \!\pm\!\ 1.00}$ | $4.07_{\ \!\pm\!\ 1.11}$ | $0.55_{\ \!\pm\!\ 0.14}$ | $0.55_{\ \!\pm\!\ 0.09}$ |
| SurveyForge | $\textbf{4.50}_{\ \!\pm\!\ 0.50}$ | $4.75_{\ \!\pm\!\ 0.50}$ | $4.54_{\ \!\pm\!\ 0.54}$ | $4.60_{\ \!\pm\!\ 0.52}$ | $0.47_{\ \!\pm\!\ 0.12}$ | $0.47_{\ \!\pm\!\ 0.13}$ |
| \rowcolor[HTML]BCA88D<br>IterSurvey | $4.42_{\ \!\pm\!\ 0.58}$ | $\textbf{4.83}_{\ \!\pm\!\ 0.17}$ | $\textbf{4.63}_{\ \!\pm\!\ 0.63}$ | $\textbf{4.63}_{\ \!\pm\!\ 0.37}$ | $\textbf{0.60}_{\ \!\pm\!\ 0.06}$ | $\textbf{0.67}_{\ \!\pm\!\ 0.06}$ |

### 4.6 Ablation Study

**Table 5: Ablation study analyzing the contribution of each component in IterSurvey: $\Circle$ Recurrent outline generation; $\Square$ Paper Card; $\Diamond$ Review-and-Refine.**
| Methods | Content Quality | Citation Quality |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Coverage | Relevance | Structure | Avg. | Precision | Recall |  |
| Baseline | $4.00_{\ \!\pm\!\ 0.53}$ | $4.40_{\ \!\pm\!\ 0.48}$ | $4.20_{\ \!\pm\!\ 0.70}$ | $4.20_{\ \!\pm\!\ 0.44}$ | $0.58_{\ \!\pm\!\ 0.09}$ | $0.67_{\ \!\pm\!\ 0.09}$ |
| + $\Circle$ | $4.46_{\ \!\pm\!\ 0.52}$ | $4.80_{\ \!\pm\!\ 0.41}$ | $4.53_{\ \!\pm\!\ 0.52}$ | $4.60_{\ \!\pm\!\ 0.40}$ | $0.62_{\ \!\pm\!\ 0.08}$ | $0.59_{\ \!\pm\!\ 0.09}$ |
| + $\Circle$ + $\Square$ | $4.60_{\ \!\pm\!\ 0.51}$ | $4.80_{\ \!\pm\!\ 0.42}$ | $4.60_{\ \!\pm\!\ 0.52}$ | $4.69_{\ \!\pm\!\ 0.39}$ | $0.64_{\ \!\pm\!\ 0.09}$ | $0.71_{\ \!\pm\!\ 0.08}$ |
| + $\Circle$ + $\Square$ + $\Diamond$ | $4.73_{\ \!\pm\!\ 0.50}$ | $4.93_{\ \!\pm\!\ 0.41}$ | $4.80_{\ \!\pm\!\ 0.52}$ | $4.82_{\ \!\pm\!\ 0.39}$ | $0.65_{\ \!\pm\!\ 0.04}$ | $0.77_{\ \!\pm\!\ 0.04}$ |

We conducted an ablation study on five representative topics to analyze the impact of the three new modules of IterSurvey: Recurrent Outline Generation, Paper Card, and Review-and-Refine. Results are shown in Tab. [5](https://arxiv.org/html/2510.21900v1#S4.T5), revealing the following insights:

Recurrent Outline Generation yields stronger content quality.
We compare our recurrent outline generation with a one-shot paradigm, where retrieved papers are partitioned into groups, each group produces an outline independently, and the results are subsequently merged. The recurrent approach contributes significant improvements in content quality, with gains of $+0.46$ in coverage and $+0.33$ in structure over the baseline. This demonstrates that iterative exploration helps the model achieve broader coverage and stronger organizational coherence by progressively integrating evidence.

Paper Card improves citation quality.
We further examined the impact of paper card, we replace them with abstract-based inputs commonly used in retrieval pipelines. The results show that paper cards significantly improve citation grounding, raising recall from $0.59$ to $0.71$ while maintaining precision ($0.64$). This indicates that distilled paper-level evidence reduces distraction and enables the model to retrieve and cite a broader yet accurate set of references.

Review and Refine boosts overall performance.
Finally, we evaluate the review-and-refine stage by removing it from the pipeline. The full variant enhances all dimensions of content quality, raising the overall average from $4.69$ to $4.82$, and further improves citation recall from $0.71$ to $0.77$. These gains show that multi-round self-critique and revision help fill evidence gaps, eliminate unsupported claims, and polish the text into well-substantiated surveys. Together, recurrent planning, paper cards, and review-and-refine form the most effective configuration of IterSurvey.

## 5 Conclusion

In this work, we tackled the limitations of existing survey generation systems by introducing IterSurvey, a framework with recurrent outline generation, paper cards, and global review and integration. This design enables precise retrieval, coherent structure, and faithful citation grounding, while supporting multimodal outputs. Experiments on diverse topics show that IterSurvey outperforms state-of-the-art baselines in coherence, coverage, and citation quality. We also proposed Survey-Arena, a pairwise benchmark that complements absolute scoring for a more reliable assessment. Future work will extend our framework to broader domains, integrate richer multimodal evidence, and refine evaluation protocols toward human-level quality.

## Ethical Considerations

Our work focuses on automatic literature survey generation using large language models. While the system is designed to support researchers by synthesizing existing knowledge, it inevitably inherits limitations of current models, including potential citation errors, incomplete coverage, and occasional inaccuracies. Therefore, the generated surveys are intended as an assistive tool rather than a substitute for human scholarship, and should be used for reference only. For evaluation, all human experts involved in the study participated voluntarily and received fair compensation. All data used in our experiments were sourced from publicly available arXiv papers, which permit non-commercial use. We strictly avoided the use of private or sensitive data.

## Use of Large Language Models

We used large language models (GPT-4o, Claude-3.5-Haiku, and GLM-4.5V) in two ways:
(i) as evaluation judges for assessing survey quality, and
(ii) for limited language editing and refinement of the manuscript.
All substantive research ideas, experimental design, analyses, and final decisions were made solely by the authors,
who take full responsibility for the content of this paper.

## References

- Asai et al. (2023)
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi.
Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023.
URL [https://arxiv.org/abs/2310.11511](https://arxiv.org/abs/2310.11511).
- Azad & Deepak (2019)
Hiteshwar Kumar Azad and Akshay Deepak.
Query expansion techniques for information retrieval: A survey.
*Information Processing & Management*, 56(5):1698–1735, 2019.
ISSN 0306-4573.
doi: https://doi.org/10.1016/j.ipm.2019.05.009.
URL [https://www.sciencedirect.com/science/article/pii/S0306457318305466](https://www.sciencedirect.com/science/article/pii/S0306457318305466).
- Bates (1989)
Marcia J. Bates.
The design of browsing and berrypicking techniques for the online search interface.
*Online Review*, 1989.
URL [https://pages.gseis.ucla.edu/faculty/bates/berrypicking.html](https://pages.gseis.ucla.edu/faculty/bates/berrypicking.html).
- Cachola et al. (2020)
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S. Weld.
Tldr: Extreme summarization of scientific documents, 2020.
URL [https://arxiv.org/abs/2004.15011](https://arxiv.org/abs/2004.15011).
- Chiang et al. (2024)
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.
Chatbot arena: An open platform for evaluating llms by human preference, 2024.
URL [https://arxiv.org/abs/2403.04132](https://arxiv.org/abs/2403.04132).
- Fabbri et al. (2019)
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev.
Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.
In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pp. 1074–1084, Florence, Italy, July 2019. Association for Computational Linguistics.
doi: 10.18653/v1/P19-1102.
URL [https://aclanthology.org/P19-1102/](https://aclanthology.org/P19-1102/).
- Gao et al. (2023)
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen.
Enabling large language models to generate text with citations.
*arXiv preprint arXiv:2305.14627*, 2023.
- Gidiotis & Tsoumakas (2020)
Alexios Gidiotis and Grigorios Tsoumakas.
A divide-and-conquer approach to the summarization of long documents, 2020.
URL [https://arxiv.org/abs/2004.06190](https://arxiv.org/abs/2004.06190).
- Herbrich et al. (2006)
Ralf Herbrich, Tom Minka, and Thore Graepel.
Trueskill™: a bayesian skill rating system.
*Advances in neural information processing systems*, 19, 2006.
- Hu et al. (2024)
Yuntong Hu, Zhuofeng Li, Zheng Zhang, Chen Ling, Raasikh Kanjiani, Boxin Zhao, and Liang Zhao.
Hireview: Hierarchical taxonomy-driven automatic literature review generation.
2024.
- Jiang et al. (2023)
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig.
Active retrieval augmented generation, 2023.
URL [https://arxiv.org/abs/2305.06983](https://arxiv.org/abs/2305.06983).
- Jin et al. (2024)
Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang.
Agentreview: Exploring peer review dynamics with llm agents, 2024.
URL [https://arxiv.org/abs/2406.12708](https://arxiv.org/abs/2406.12708).
- Latona et al. (2024)
Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R. Davidson, Veniamin Veselovsky, and Robert West.
The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates, 2024.
URL [https://arxiv.org/abs/2405.02150](https://arxiv.org/abs/2405.02150).
- Li et al. (2024)
Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu.
Split and merge: Aligning position biases in llm-based evaluators, 2024.
URL [https://arxiv.org/abs/2310.01432](https://arxiv.org/abs/2310.01432).
- Liang et al. (2025)
Xun Liang, Jiawei Yang, Yezhaohui Wang, Chen Tang, Zifan Zheng, Shichao Song, Zehao Lin, Yebin Yang, Simin Niu, Hanyu Wang, et al.
Surveyx: Academic survey automation via large language models.
*arXiv preprint arXiv:2502.14776*, 2025.
- Liu et al. (2023)
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.
Lost in the middle: How language models use long contexts, 2023.
URL [https://arxiv.org/abs/2307.03172](https://arxiv.org/abs/2307.03172).
- Novikova et al. (2018)
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser.
RankME: Reliable human ratings for natural language generation.
In *NAACL-HLT*, 2018.
URL [https://aclanthology.org/N18-2012](https://aclanthology.org/N18-2012).
- Nussbaum et al. (2024)
Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar.
Nomic embed: Training a reproducible long context text embedder, 2024.
- Oren et al. (2023)
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto.
Proving test set contamination in black box language models, 2023.
URL [https://arxiv.org/abs/2310.17623](https://arxiv.org/abs/2310.17623).
- Sakaguchi et al. (2014)
Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme.
Efficient elicitation of annotations for human evaluation of machine translation.
In Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Specia (eds.), *Proceedings of the Ninth Workshop on Statistical Machine Translation*, pp. 1–11, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics.
doi: 10.3115/v1/W14-3301.
URL [https://aclanthology.org/W14-3301/](https://aclanthology.org/W14-3301/).
- Sun et al. (2019)
Haitian Sun, Tania Bedrax-Weiss, and William Cohen.
PullNet: Open domain question answering with iterative retrieval on knowledge bases and text.
In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pp. 2380–2390, Hong Kong, China, November 2019. Association for Computational Linguistics.
doi: 10.18653/v1/D19-1242.
URL [https://aclanthology.org/D19-1242/](https://aclanthology.org/D19-1242/).
- Wang et al. (2024a)
Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al.
Mineru: An open-source solution for precise document content extraction.
*arXiv preprint arXiv:2409.18839*, 2024a.
- Wang et al. (2025)
Haoyu Wang, Yujia Fu, Zhu Zhang, Shuo Wang, Zirui Ren, Xiaorong Wang, Zhili Li, Chaoqun He, Bo An, Zhiyuan Liu, et al.
Llmxmapreduce-v2: Entropy-driven convolutional test-time scaling for generating long-form articles from extremely long resources.
*arXiv preprint arXiv:2504.05732*, 2025.
- Wang et al. (2020)
Xiao Wang, Craig Macdonald, and Iadh Ounis.
Deep reinforced query reformulation for information retrieval, 2020.
URL [https://arxiv.org/abs/2007.07987](https://arxiv.org/abs/2007.07987).
- Wang et al. (2024b)
Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang.
Autosurvey: Large language models can automatically write surveys.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024b.
- Weng et al. (2025)
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang.
Cycleresearcher: Improving automated research via automated review, 2025.
URL [https://arxiv.org/abs/2411.00816](https://arxiv.org/abs/2411.00816).
- Wu et al. (2024)
Zijun Wu, Bingyuan Liu, Ran Yan, Lei Chen, and Thomas Delteil.
Reducing distraction in long-context language models by focused learning, 2024.
URL [https://arxiv.org/abs/2411.05928](https://arxiv.org/abs/2411.05928).
- Yan et al. (2025)
Xiangchao Yan, Shiyang Feng, Jiakang Yuan, Renqiu Xia, Bin Wang, Lei Bai, and Bo Zhang.
SURVEYFORGE : On the outline heuristics, memory-driven generation, and multi-dimensional evaluation for automated survey writing.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 12444–12465, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.609.
URL [https://aclanthology.org/2025.acl-long.609/](https://aclanthology.org/2025.acl-long.609/).
- Yang et al. (2023a)
Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian.
DOC: Improving long story coherence with detailed outline control.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 3378–3465, Toronto, Canada, July 2023a. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.190.
URL [https://aclanthology.org/2023.acl-long.190/](https://aclanthology.org/2023.acl-long.190/).
- Yang et al. (2023b)
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica.
Rethinking benchmark and contamination for language models with rephrased samples, 2023b.
URL [https://arxiv.org/abs/2311.04850](https://arxiv.org/abs/2311.04850).
- Ye et al. (2024)
Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen.
Are we there yet? revealing the risks of utilizing large language models in scholarly peer review, 2024.
URL [https://arxiv.org/abs/2412.01708](https://arxiv.org/abs/2412.01708).
- Yu et al. (2024)
Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Renjing Cui, Chengcheng Han, Qiushi Sun, Zhiyong Wu, Yunshi Lan, and Xiang Li.
Automated peer reviewing in paper sea: Standardization, evaluation, and analysis, 2024.
URL [https://arxiv.org/abs/2407.12857](https://arxiv.org/abs/2407.12857).
- Zhang et al. (2025a)
Dingchu Zhang, Yida Zhao, Jialong Wu, Baixuan Li, Wenbiao Yin, Liwen Zhang, Yong Jiang, Yufeng Li, Kewei Tu, Pengjun Xie, and Fei Huang.
Evolvesearch: An iterative self-evolving search agent, 2025a.
URL [https://arxiv.org/abs/2505.22501](https://arxiv.org/abs/2505.22501).
- Zhang et al. (2025b)
Yaohui Zhang, Haijing Zhang, Wenlong Ji, Tianyu Hua, Nick Haber, Hancheng Cao, and Weixin Liang.
From replication to redesign: Exploring pairwise comparisons for llm-based peer review, 2025b.
URL [https://arxiv.org/abs/2506.11343](https://arxiv.org/abs/2506.11343).
- Zhao (2025)
Zhimin Zhao.
Se arena: An interactive platform for evaluating foundation models in software engineering.
In *2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge)*, pp. 78–81, 2025.
doi: 10.1109/Forge66646.2025.00016.

## Appendix A Appendix

### A.1 Retrieval Setup

For the retrieval process, we implemented a lightweight database to provide the necessary functionality. The retrieval logic is based on vector similarity, using the nomic-ai/nomic-embed-text-v1.5 (Nussbaum et al., 2024) embedding model with all hyperparameters set to their default values. Given a query, the database computes the similarity between the query vector and all paper vectors, and returns the top-k most relevant entries. In addition, the database supports bidirectional lookup between a paper’s arXiv identifier and title, as well as filtering papers published prior to a specified cutoff date.

### A.2 Results of Inter-rater Agreement

To assess the reliability of human annotations, we computed Cohen’s kappa coefficients across four evaluation dimensions: Coverage, Relevance, Structure, and Overall, as shown in Tab. [6](https://arxiv.org/html/2510.21900v1#A1.T6). These results indicate substantial agreement among human annotators, supporting the consistency of the human evaluation process.

**Table 6: Inter-rater agreement among human annotators.**
| Dimensions | Coverage | Relevance | Structure | Overall |
| --- | --- | --- | --- | --- |
| kappa | 0.714 | 0.583 | 0.611 | 0.650 |

### A.3 Topics for Automatic Evaluation

We utilize 20 topics derived from AutoSurvey (Wang et al., 2024b). Each topic is paired with a human survey, as shown in Tab. [7](https://arxiv.org/html/2510.21900v1#A1.T7), which also reports the survey titles, arXiv IDs, and their latest citation counts from Google Scholar.

**Table 7: Topics for Automatic Evaluation**
| Topic | Human Survey | ArXiv ID | Citations |
| --- | --- | --- | --- |
| In-context Learning | A Survey on In-context Learning | 2301.00234 | 2396 |
| LLMs for Recommendation | A Survey on Large Language Models for Recommendation | 2305.19860 | 596 |
| LLM-Generated Texts Detection | The Science of Detecting LLM-Generated Texts | 2310.14724 | 308 |
| Explainability for LLMs | Explainability for Large Language Models: A Survey | 2309.01029 | 875 |
| Evaluation of LLMs | A Survey on Evaluation of Large Language Models | 2307.03109 | 4020 |
| LLMs-based Agents | A Survey on Large Language Model based Autonomous Agents | 2308.11432 | 1906 |
| LLMs in Medicine | A Survey of Large Language Models in Medicine | 2311.05112 | 217 |
| Domain Specialization of LLMs | Domain Specialization as the Key to Make Large Language Models Disruptive | 2305.18703 | 217 |
| Challenges of LLMs in Education | Practical and Ethical Challenges of Large Language Models in Education | 2303.13379 | 722 |
| Alignment of LLMs | Aligning Large Language Models with Human: A Survey | 2307.12966 | 435 |
| ChatGPT | Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond | 2304.13712 | 1254 |
| Instruction Tuning for LLMs | Instruction Tuning for Large Language Models: A Survey | 2308.10792 | 1174 |
| LLMs for Information Retrieval | Large Language Models for Information Retrieval: A Survey | 2308.07107 | 544 |
| Safety in LLMs | Towards Safer Generative Language Models | 2302.09270 | 13 |
| Chain of Thought | A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future | 2309.15402 | 290 |
| Hallucination in LLMs | A Survey on Hallucination in Large Language Models | 2311.05232 | 2599 |
| Bias and Fairness in LLMs | Bias and Fairness in Large Language Models: A Survey | 2309.00770 | 1009 |
| Large Multi-Modal Language Models | Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey | 2302.10035 | 285 |
| Acceleration for LLMs | A Survey on Model Compression and Acceleration for Pretrained Language Models | 2202.07105 | 101 |
| LLMs for Software Engineering | Large Language Models for Software Engineering: A Systematic Literature Review | 2308.10620 | 1058 |

### A.4 Topics for Survey-Arena

To construct the Survey-Arena benchmark, we select 10 topics, with several derived from AutoSurvey (Wang et al., 2024b) and SurveyForge (Yan et al., 2025). For each topic, we include 5 human-written surveys, requiring that their arXiv submission dates fall within a six-month window. We report their latest Google Scholar citation counts as a measure of impact, as summarized in Tab. [8](https://arxiv.org/html/2510.21900v1#A1.T8). For reproducibility, we also specify the exact arXiv version, since submission dates can vary considerably across different versions of the same paper.

**Table 8: Topics for Survey-Arena**
| Topic | Human Survey | ArXiv ID | Citations |
| --- | --- | --- | --- |
| Large Language<br>Models | Large Language Models: A Survey | 2402.06196v3 | 1133 |
| Large Language Models Meet NLP: A Survey | 2405.12819v1 | 86 |  |
| History, Development, and Principles of Large Language Models-An Introductory Survey | 2402.06853v2 | 73 |  |
| Recent Advances in Generative AI and Large Language Models | 2407.14962v1 | 68 |  |
| Exploring the landscape of large language models: Foundations, techniques, and challenges | 2404.11973v1 | 5 |  |
| Multimodal LLMs | MM-LLMs: Recent Advances in MultiModal Large Language Models | 2401.13601v3 | 381 |
| Multimodal Large Language Models: A Survey | 2311.13165v1 | 299 |  |
| The Revolution of Multimodal Large Language Models: A Survey | 2402.12451v1 | 98 |  |
| How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model | 2311.07594v1 | 43 |  |
| A Review of Multi-Modal Large Language and Vision Models | 2404.01322v1 | 39 |  |
| Multilingual LLMs | Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers | 2404.04925v1 | 83 |
| A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias | 2404.00929v2 | 55 |  |
| A Survey on Large Language Models with Multilingualism | 2405.10936v1 | 40 |  |
| Surveying the MLLM Landscape: A Meta-Review of Current Surveys | 2409.18991v1 | 12 |  |
| Multilingual Large Language Models: A Systematic Survey | 2411.11072v2 | 9 |  |
| LLMs Reasoning | A Survey of Long Chain-of-Thought for Reasoning Large Language Models | 2503.09567v3 | 130 |
| From System 1 to System 2: A Survey of Reasoning Large Language Models | 2502.17419v2 | 110 |  |
| Advancing Reasoning in Large Language Models: Promising Methods and Approaches | 2502.03671v1 | 19 |  |
| A Survey of Frontiers in LLM Reasoning | 2504.09037v1 | 17 |  |
| Thinking Machines: A Survey of LLM based Reasoning Strategies | 2503.10814v1 | 9 |  |
| Prompt Engineering<br>of LLMs | A Systematic Survey of Prompt Engineering in Large Language Models | 2402.07927v1 | 748 |
| The Prompt Report: A Systematic Survey of Prompt Engineering Techniques | 2406.06608v2 | 182 |  |
| Prompt Design and Engineering: Introduction and Advanced Methods | 2401.14423v4 | 117 |  |
| A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks | 2407.12994v1 | 60 |  |
| Efficient Prom pting Methods for Large Language Models: A Survey | 2404.01077v1 | 56 |  |
| Retrieval-Augmented<br>Generation for LLMs | Retrieval-Augmented Generation for Large Language Models: A Survey | 2312.10997v5 | 2583 |
| A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models | 2405.06211v3 | 559 |  |
| A Survey on Retrieval-Augmented Text Generation for Large Language Models | 2404.10981v2 | 119 |  |
| Retrieval-Augmented Generation for Natural Language Processing: A Survey | 2407.13193v2 | 77 |  |
| Retrieval Augmented Generation (RAG) and Beyond | 2409.14924v1 | 70 |  |
| LLM-based<br>Multi-Agent System | A survey on large language model based autonomous agents | 2308.11432v7 | 1623 |
| Multi-Agent Collaboration Mechanisms: A Survey of LLMs | 2501.06322v1 | 79 |  |
| Large language model agent: A survey on methodology, applications and challenges | 2503.21460v1 | 19 |  |
| Agentic large language models, a survey | 2503.23037v2 | 12 |  |
| A Survey on LLM-based Multi-Agent System: | 2412.17481v2 | 3 |  |
| LLM-Generated<br>Texts Detection | A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions | 2310.14724v2 | 210 |
| A Survey on Detection of LLMs-Generated Content | 2310.15654v1 | 69 |  |
| Towards Possibilities & Impossibilities of AI-generated Text Detection: A Survey | 2310.15264v1 | 46 |  |
| Detecting chatgpt: A survey of the state of detecting chatgpt-generated text | 2309.07689v1 | 22 |  |
| Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text | 2403.05750v1 | 13 |  |
| LLMs in Medicine | Large language models in healthcare and medical domain: A review | 2401.06775v2 | 246 |
| A Survey on Medical Large Language Models | 2406.03712v1 | 53 |  |
| A Comprehensive Survey of Large Language Models and Multimodal Large Language Models<br>in Medicine | 2405.08603v1 | 46 |  |
| Large Language Models for Medicine: A Survey | 2405.13055v1 | 37 |  |
| A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry | 2404.15777v4 | 32 |  |
| LLMs for<br>Recommendation | A Survey on Large Language Models for Recommendation | 2305.19860v4 | 508 |
| Recommender Systems in the Era of Large Language Models (LLMs) | 2307.02046v2 | 479 |  |
| A Comprehensive Survey of Language Modelling Paradigm Adaptations in Recommender Systems | 2302.03735v3 | 117 |  |
| Large Language Models for Generative Recommendation: A Survey and Visionary Discussions | 2309.01157v1 | 116 |  |
| How Can Recommender Systems Benefit from Large Language Models: A Survey | 2306.05817v4 | 104 |  |

### A.5 Topics for Survey-Lacking Test

We manually select 8 topics with no existing survey articles, as shown in Tab. [9](https://arxiv.org/html/2510.21900v1#A1.T9).

**Table 9: Topics for Survey-Lacking Test.**
| Topic |
| --- |
| Event Timeline Generation |
| Linear RNN in Natural Language Processing |
| Agent-flow Data Curation |
| Causal Mediation with Sparse Autoencoder Features in Transformers |
| Multi-Tenant Scheduling for MoE Inference |
| Benchmarking Tool-Using LLMs for Causal Tasks in the MCP Ecosystem |
| RAG for Mechanical Design: Cross-Modal Retrieval over CAD Trees and BOMs |
| Renderer-in-the-Loop Supervision for Multimodal Model |

### A.6 Detail of Naive RAG

Given a topic, the Naive RAG system first retrieves 1,500 papers from the same database as ours. It then employs an iterative prompting strategy, where the LLM generates content until the total length of the survey reaches 5,000 tokens (Wang et al., 2024b). The prompt used for generation is shown below.

### A.7 Prompts for Evaluation

### A.8 Comparison between AutoSurvey and IterSurvey.

Figure: Figure 4: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/p1.png

Figure: Figure 5: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/p3.png

Figure: Figure 6: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/ref1-2.png

Figure: (a) AutoSurvey
Refer to caption: autosurvey_compare/v1/v1-table.png

### A.7 Prompts for Evaluation

### A.8 Comparison between AutoSurvey and IterSurvey.

Figure: Figure 4: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/p1.png

Figure: Figure 5: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/p3.png

Figure: Figure 6: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/ref1-2.png

Figure: (a) AutoSurvey
Refer to caption: autosurvey_compare/v1/v1-table.png

### A.8 Comparison between AutoSurvey and IterSurvey.

Figure: Figure 4: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/p1.png

Figure: Figure 5: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/p3.png

Figure: Figure 6: LLM-generated survey comparison between AutoSurvey and IterSurvey.
Refer to caption: autosurvey_compare/v1/ref1-2.png

Figure: (a) AutoSurvey
Refer to caption: autosurvey_compare/v1/v1-table.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: please refer to the weaknesses
soundness: 3
strengths: 1. Innovative Iterative Paradigm:The core strength of the work is its departure from the conventional one-shot generation model. By thoughtfully mimicking the iterative reading and writing process of human experts, the proposed framework provides a more natural and effective solution to the complex task of survey generation.

2. Well-Designed Mechanisms for Fidelity and Flow:The introduction of "paper cards" ensures faithful, paper-level grounding of the survey content. Combined with the dedicated review-and-refine loop for visualization enhancement, the framework successfully addresses key challenges in maintaining textual flow and integrating multimodal information.

3. Comprehensive Evaluation and New Benchmark:The paper not only shows strong experimental results on established metrics but also introduces Survey-Arena, a valuable benchmark that moves beyond absolute scoring. This pairwise comparison tool offers the research community a more nuanced and reliable way to evaluate machine-generated surveys in relation to a human standard.
summary: This paper addresses the limitations of existing one-shot approaches to automatic literature survey generation, which often result in fragmented structures and information overload. Inspired by the iterative process of human researchers, the authors propose IterSurvey, a novel framework that employs a recurrent outline generation mechanism. In this framework, a planning agent incrementally retrieves and reads papers while dynamically updating the survey's outline to ensure coherence and exploration. A key feature is the use of "paper cards" to succinctly capture each paper's core elements, alongside a review-and-refine loop that integrates textual and multimodal content to enhance readability. Experimental evaluations demonstrate that IterSurvey produces surveys that significantly outperform state-of-the-art baselines in terms of content coverage, structural quality, and organization. Furthermore, the authors introduce Survey-Arena, a new pairwise comparison benchmark, to provide a more reliable method for assessing the quality of generated surveys against human-written ones.
weaknesses: 1. Ablation Studies: Ablation experiments are required to validate the effectiveness of the individual modules designed within the workflow.

2. Regarding Evaluation Metrics: In Table 1, several evaluation scores are very close to the baselines. The persuasiveness of the current results is insufficient; it should be tested whether the performance gap remains similarly close if the scale of the scoring metric is widened.

3. Human Alignment: For the LLM-as-a-judge evaluation method, its reliability must be demonstrated through comparison with human evaluation. This is particularly critical for highly creative tasks like survey writing, which cannot be assessed simply by checking items against a template. Evaluating more creative aspects is both more difficult and more valuable.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - In the recurrent outline generation module of IterSurvey, how is the similarity threshold τ—which determines whether to accept candidate outline updates—specifically calibrated, and what effects do different values of τ have on the stability of the outline and its exploratory capability?

- Regarding paper cards, which distill papers into their core contributions, methods, and findings, does the framework adopt a specific criterion for deciding which details to retain or discard during the distillation process, and how does it prevent the omission of critical information?

- The ablation analysis presented in Table 5 appears insufficient; it would be preferable to conduct a single-component ablation study to more clearly assess the contribution of each module.

- In the comparative evaluation with baselines (e.g., AutoSurvey, SurveyForge), why was the analysis limited to the “content quality” and “citation quality” dimensions, without including a comparison of computational efficiency (e.g., time consumption and resource utilization)?

- While IterSurvey performs effectively on emerging topics lacking existing surveys, does it incorporate adaptation strategies for domains with extremely sparse literature, such as newly developed interdisciplinary fields?

- Finally, to what extent does the underlying base model rigidly determine the overall performance of IterSurvey?
soundness: 3
strengths: - The integration of paper cards and a review-and-refine loop with visualization enhancements significantly improves textual flow, cross-sectional coherence, and multimodal integration, as evidenced by the experimental results.

- The proposed Survey-Arena, a pairwise evaluation benchmark that compares machine-generated surveys with human-written ones, offers a more reliable assessment and effectively addresses the limitations of absolute scoring methods.
summary: This paper proposes IterSurvey to fix flaws of one-shot automated survey systems (noisy retrieval, fragmented structure). It uses recurrent outline generation, paper cards (distilling papers’ key info), and a review loop. Experiments show it outperforms baselines (e.g., AutoSurvey) in coverage, coherence and citations. The authors also introduce Survey-Arena for reliable pairwise assessment against human surveys, noting IterSurvey is an assistive tool with data from public arXiv papers.
weaknesses: My main concerns are the limitations of the paper's coverage and the objectivity of the evaluation. If the author can address my questions, I would be happy to increase the rating:
- The retrieval database includes only 680K computer science papers from arXiv, resulting in limited coverage of other disciplines and constraining the framework’s generalization ability to non-CS domains.

- The iterative workflow—involving recurrent outline generation and multiple review-and-refine loops—requires repeated retrieval and model invocations, potentially leading to higher computational costs compared to one-shot systems.

- The human evaluation involves only seven PhD-level experts and compares performance against two baselines (AutoSurvey, SurveyForge), with a small sample size that may restrict the representativeness and robustness of the findings.

- The framework lacks evaluation on established benchmarks such as SurveyBench (proposed in SurveyForge), which would enhance the comparability and credibility of its performance assessment.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to the weakness.
soundness: 2
strengths: The paper abandons the static "one-time" planning mode and instead adopts a dynamic, iterative workflow which may iteratively improve the quality of the survey. Besides, injecting multimodal items (table and figure) into a survey is interesting.
summary: This paper proposes an auto-literature review generation framework *i.e.*, IterSurvey. The core idea of IterSurvey is to mimic the iterative research process of human researchers, through a cyclic outline generation module, incrementally conducting literature retrieval, reading, and updating to ensure the exploratory and coherent nature of the review. Additionally, to more reliably evaluate the quality of the generated reviews, the paper constructs a paired evaluation benchmark named Survey-Arena, which provides a more robust performance ranking by directly comparing machine-generated reviews with those written by humans.
weaknesses: 1. Lack of efficiency and cost analysis: The iterative framework proposed in the paper, especially the cyclic generation of outlines and multi-round "review and optimization," seems to require a large amount of computational resources and time. However, the paper does not provide a comparative analysis of the efficiency (such as the time required to generate a review) and cost (API usage) between IterSurvey and other baseline methods. 

2. The generation details of the "paper card" need to be supplemented: The process of constructing paper cards seems to need lots of time (using LLM to generate motivation, contribution, and so on). The authors need to add more details.

3. The authors need to verify the effectiveness of the iterative pipeline. For example, the performance improvement is seen with the increase in iterations.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. What is the difference of he iterative mechanism and review-and-refine between IterSurvey and Autosurvey [1].
2. What are the impact of iteration mechanism and paper card on the time consumption of the whole framework?
3. The cost of different methods should be compared.
4. Did you replicate the experiment independently, generating only one review per topic for comparison?

[1] Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, et al. Autosurvey: Large language models can automatically write surveys. Advances in neural information processing systems, 37:115119–115145, 2024.
soundness: 2
strengths: 1. An iterative mechanism is designed for survey generation and optimization.

2. Integration that includes multi-modal elements.

3. Introduce the Survey-Arena pairing evaluation benchmark.
summary: This paper develops a new framework: IterSurvey, which produces finer-grained outlines and supports multi-modal inputs and outputs for more comprehensive surveys, and constructs Survey-Arena, a pairwise evaluation benchmark that ranks generated surveys alongside human-written ones, enabling more reliable and interpretable assessment of survey quality.
weaknesses: 1. The methodology is not innovative enough, and the iterative mechanism or  review-and-refine are not novel contribution. What is the difference between it and autosurvey [1].

2. Does the introduction of iteration mechanism and paper card have a big impact on the time consumption of the whole framework?

3. The cost of different methods is not compared.

[1] Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, et al. Autosurvey: Large language models can automatically write surveys. Advances in neural information processing systems, 37:115119–115145, 2024.


Decision: Reject