AutoSurvey: Large Language Models Can Automatically Write Surveys
Authors: Yidong Wang
Qi Guo
Wenjin Yao
Hongbo Zhang
Xin Zhang
Zhen Wu
Meishan Zhang
Xinyu Dai
Min zhang
Qingsong Wen
Wei Ye
Shikun Zhang
Yue Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model; AI applications
Primary area: natural_language_processing
TL;DR: This paper introduces AutoSurvey, a speedy and well-organized methodology for automating the creation of comprehensive literature surveys in rapidly evolving fields like artificial intelligence.

Abstract: This paper introduces AutoSurvey, a speedy and well-organized methodology for automating the creation of comprehensive literature surveys in rapidly evolving fields like artificial intelligence. Traditional survey paper creation faces challenges due to the vast volume and complexity of information, prompting the need for efficient survey methods. While large language models (LLMs) offer promise in automating this process, challenges such as context window limitations, parametric knowledge constraints, and the lack of evaluation benchmarks remain. AutoSurvey addresses these challenges through a systematic approach that involves initial retrieval and outline generation, subsection drafting by specialized LLMs, integration and refinement, and rigorous evaluation and iteration. Our contributions include a comprehensive solution to the survey problem, a reliable evaluation method, and experimental validation demonstrating AutoSurvey's effectiveness.

AutoSurvey: Large Language Models Can Automatically Write Surveys
Yidong Wangl.2 Qi 23+
Wenjin Yao², Hongho Zhang', Xin Zhang'. Zhen Wu Meeshha Zhang' Xinyu Dai Min Zhang", Qingsong Wen Wei Ye Shikun Zhang Yue Zhanglt
Westlake University, "Peking University, 3 Nanjing University. "Harbin Institute of Technology, Shenzhen, "Squirrel Al
Abstract
This paper introduces AutoSurvey, a speedy and well-organized methodology for automating the creation of comprehensive literature surveys in rapidly evolving fields like artificial intelligence. Traditional survey paper creation faces challenges due to the vast volume and complexity of information, prompting the need for efficient survey methods. While large language models (LLMs) offer promise automating this process, challenges such context window limitations, parametric knowledge constraints, and the lack of evaluation benchmarks remain. AutoSurvey addresses these challenges through systematic approach that involves initial trieval and outline generation, subsection drafting by specialized L.Mss, integration and refinement, and rigorous evaluation and iteration. Our contributions include a comprehensive solution to the survey problem, reliable evaluation method, and experimental validation demonstrating AutoSurvey's effectiveness.
Introduction
Survey papers provide essential academic resources, offering comprehensive overviews of recent
research developments, highlighting ongoing trends, and identifying future directions [1] 2 3 4 However, crafting these surveys is increasingly challenging, especially in the fast-paced domain of Artificial Intelligence including large language models(LLMs) [5] 6 7 8 Figure Ta illustrates
significant trend: in just the first four months of 2024 alone, over 4,000 papers containing the phrase "Large Language Model" in their titles or abstracts were submitted to Xiv. This surge highlights a critical academie issue: the rapid accumulation of new information often outpaces the capacity for comprehensive scholarly review and synthesis, emphasizing the growing need for more efficient methods to synthesize the expanding literature. Moreover, as depicted in Figure [Th] while the number -athhred
of survey papers has rapidly increased, the growing difficulty of producing traditional survey papers-due to the sheer volume and complexity of data-remmt remains a significant challenge. This challenge is evidenced by the lack of comprehensive surveys in many fields (Figure Ic]. which hinders knowledge transfer and makes difficult for new researchers to efficiently navigate the vast amount of available information.
The advent of LLMs models, trained on extensive text corpora, demonstrate remarkable capabilities in understanding and generating human-like text, even in long-context scenarios [0 II 12]. Despite these advancements, the practical application of L.MMs to survey generation is fraught with challenges. Firstly, context
presents a promising avenue for addressing these challenges. These
"Equal contribution. @ @mail.comm qguo@smail.njun sma.l..j....du.cc Yidong Wang did this work
during his internship at Squirrel Al. *Correspondence to: wye@ pku.edu.cn. zhangsk pku.edu.en. zhangyue westlake.edu.oo edu.cc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

16000
13446
12000
8000
7306
4000
617
Year
(a) #Paper about LLM
800
714
Cluster D:
Emotion Recogrition
Cluster 7:
Ontelogy Enrichment
Cluster Embeedinng Techniques
Cluster Humer Detection
Empalhatic Besponse
Num
Cluster Duantum Theory
Cluster 10: Aftribute Value Extraction
Claster 4: Data Visudizwtion
Cluster 11: Anomaly Detection
Cluster S: Neural Scaling Luws
Regular Papers
200
Claster Mixture Expprrrss
19
20
21
22
23
24
(c) T-SNE visualization of surveys and papers about L.Mss. Clusters represent groups of papers identified through cluster- ing, which currently lack comprehensive survey coverage.
Year
(b) #Survey about LLM.
Figure 1: Depicting growth trends from 2019 to 2024 in the number of LLMs-related papers (a) and surveys (b) on arXiv, accompanied by a T-SNE visualization. The data for 2024 is up to April, with red bar representing the forecasted numbers for the entire year. While the number of surveys increasing rapidly, the visualization reveals areas where comprehensive surveys are still lacking, despite the overall growth in survey numbers. The research topics of the clusters in the T-SNE plot are generated using GPT-4 to describe their primary focus areas. These clusters of research voids can be addressed using AutoSurvey at cost of $1.2 (cost analysis in Appendix D) and 3 minutes per survey. An example survey focused on Emotion Recognition using LLMs is in Appendix F
window limitations: LLMs encounter inherent restrictions in output length due to limited processing windows [13 I5, 16 [7]. While several advanced large models, including GPT-4 and Claude 3, support inputs exceeding 100k tokens, their output is still limited to fewer than 8k tokens (the output length of GPT-4 is 8k, and the output length of Claude 3 is 4k). Writing a comprehensive survey typically requires reading hundreds of papers, resulting in input sizes far beyond the capacity of even the most advanced models. Moreover, a well-written survey itself spans tens of thousands of tokens, making it highly challenging to generate such extensive content directly with large models. Secondly, parametric knowledge constraints: Sole reliance on an LLM's internal knowledge is insufficient for producing surveys that require comprehensive and accurate references [T8] 19 20).
LLMs may generate content based on inaccuracies or even non-existent "hallucinated" references. Moreover, these models cannot incorporate the latest studies not included in their training data, which limits the breadth and depth of the surveys they generate. Thirdly, the lack of evaluation benchmark: after production, reliable metrics evaluate the quality of outputs from LLMs are lacking. Relying on human review for quality assessment is not only risource-intessie but also lacks scalability (2 22 23 This presents a significant obstacle to the widespread adoption of LLMs for academic synthesis, where rigorous standards of accuracy and reliability are paramount.
In response to these challenges, we introduce AutoSurvey: a speedy and well-organized methodology for conducting comprehensive literature surveys. Specifically, AutoSurvey's primary innovations include: logical parallel generation: AutoSurvey employs a two-stage generation approach paral- lelly generate survey content efficiently. Initially, multiple LLMS work concurrently to create detailed outlines. final, comprehensive outline then synthesized from these individual outlines, setting a clear framework for content development. Subsequently, each subsection of the survey is generated in parallel and guided by the outline, which significantly accelerates the process. To overcome potential transition and consistency issues due to segmented generation phases, AutoSurvey integrates systematic revision phase. After the initial parallel generation, each section undergoes thorough revision and polishing, ensuring smooth transitions and enhanced overall document consistency. The sections are then seamlessly merged to produce cohesive and well-organized final document. Real-time knowledge update: AutoSurvey incorporates a Real-time Knowledge Update mechanism using a Retrieval-Augeee Augmented Generation (RAG) approach [24 25 26). This feature ensures that every aspect of the survey reflects the most current studies. When survey topic is input by the user,

---

Survey Topic(()
Stage 1
Stags 2
Initial Retrieval Outline Generation
Subsection
retrivved publianioos
database publicatione
Ganeraae structured outline
relevant publications
Draft
ofe
Starp-4 4 Rigorous Evaluaaiio &
Saage: Integration Refinement
Select the bust surveyy
Fa Fx
Reftne
Boshuate the surveyss
Integrate the refined sections
Figure 2: The AutoSurvey Pipeline for Generating Comprehensive Surveys.
AutoSurvey leverages the RAG system to retrieve the latest relevant papers, forming the basis for generating structured and informed outline. During subsection writing, the system dynamically pulls in new research articles relevant to the specific content under development. This approach ensures that citations current and the survey content aligned with the latest developments in the field, significantly enhancing the accuracy and depth of the literature review. Multi-LLM Multi-LLM--ss-iie M-a-juudge evaluation: AutoSurvey employs the Multi-LLM-as-Judee strategy, leveraging the LLM-as-Judge method for text evaluation [22] 21 23). This approach generates initial evaluation metries using multiple large language models, which process a substantial corpus of high-quality surveys. These metrics are refined by human experts to ensure precision and adherence to academic standards. The Laati-LLL-as--JJucee method assesses generated content across two main dimensions: (1) Citation Quality. verifying the accuracy and reliability of the information presented. with sub-indicee indicators for Recall and Precision. (2) Content Quality, consisting of Coverage (assessing the extent of topic encapsulation), Structure (evaluating logical organization and coherence), and Relevance (ensuring alignment with the main topic). By utilizing multiple LLMMs, this strategy minimizes bias and ensures balanced and comprehensive assessment, upholding rigorous academic standards.
Extensive experimental results across different survey lengths (8k, 16k, 32k, and 64k tokens) demon- strate that AutoSurvey consistently achieves high citation and content quality scores. At 64k tokens, AutoSurvey achieves 82.25% recall and 77.41 % precision in citation quality, outperforming naive RAG-based LLMs (68.79% recall and 61.97% precision) and approaching human performance (86.33% recall and 77.78% precision). In content quality at 64k tokens, AutoSurvey scores 4.73 in coverage, 4.33 in structure, and 4.86 in relevance, closely aligning with human performance (5.00, 4.66, and 5.00 respectively). strong performance across all metrics. Furthermore, the Spearman's 's rho values indicate moderate positive correlation between the rankings provided by the LLMs and those given by human experts. The mixture of models achieves the highest correlation at 0.5429, indicating a strong alignment with human preferences. These results reinforce the effectiveness our multi-LLM scoring mechanism, providing reliable proxy for human judgment across varying survey lengths.
shorter lengths (8k, 16k, and 32k tokens), AutoSurvey also maintains
In conclusion, the best of our knowledge, AutoSurvey is the first system to explore the potential of large model agents in writing extensive academic surveys. It proposes evaluation criteria for surveys that align with human preferences, providing a valuable reference for future related research.
Methodology
In this section, we describe the methodology employed by AutoSurvey to automate the creation of comprehensive literature surveys. Our approach systematically progresses through four distinct Initial Retrieval and Outline Generation, Subsection Drafting, Integration and Refinement,

---

and Rigorous Evaluation and Iteration. Each phase is meticulously designed to address specific challenges associated with survey creation, thereby enhancing the efficiency and quality of the resulting survey document. The pseudo code of AutoSurvey can be found at Algorithm
Algorithm AUTOSURVEY: Automated Survey Creation Using LLMs.
I: Input: Survey topie T. publications databasc D
2: Output: Final refined and evaluated survey document Fbest
3: for cach survey generation trial t = to N do
4: Phase 1: Initial Retrieval and Outline Generation
5: Retrieve initial pool of publications Pimit + Retrieve(T, D)
6: Generate outline O + (T, Pinit)
7: Phase 2: Subsection Drafting
8: for each section O, in in parallel do
9: Retrieve relevant publications Psec <- (Retrieve(O;,
10: Draft subscction S; Draft(O;, Piec)
11: end for
12: Phase 3: Integration and Refinement
13: Refine the merged document improve coherence R. (efine(S.)
14: Merge subsection drafts into a single document Fe < Merge( R1, R Ra)
15: end for
16: Phase 4: Rigorous Evaluation and Iteration
17: Evaluate and select the best survey document Frea < Evaluate (1,, F2,,,,,.....
Fz,..., .a....,Fy. FN)
18: Return: Refined and evaluated survey Fbest
Initial Retrieval and Outline Generation The process begins with the Initial Retrieval and Outline Generation phase. Utilizing an embedding-
retrieval technique, AutoSurvey scans database of publications to identify papers most pertinent to the specified survey topic T. This phase is crucial for ensuring that the survey is grounded in the most relevant and recent research. The retrieved publications Pinit are then used generate structured outline O, which ensures comprehensive coverage the topic and logical structuring of the survey. To provide more detailed guidance for writing subsections, the outline generation includes not only titles for each subsection but also brief descriptions. These descriptions convey the main idea of each subsection, aiding in the overall clarity and direction of the survey. Given the extensive amount of relevant papers extracted during this stage, the total length of Pmn often exceeds the context window size of the LLM. To address this, papers are randomly divided according the LLM's context window size, resulting in the creation of multiple outlines. The model then consolidates these outlines to form the final comprehensive outline. Finally, the outline O of the entire survey is represented as O Outline P.nit).
Subsection Drafting With the structured outline in place, the Subsection Drafting phase com- mences. During this phase, specialized LLMs draft each section of the outline in parallel. This method not only accelerates the drafting process but also ensures detailed and focused content generation for each survey section, adhering to the thematic boundaries established by the outline. When writing the content of each subsection, the sub-outline O, of that subsection will be used to retrieve the necessary relevant reference papers Pace to provide information that aligns more closely with the main idea of the subsection. During the writing process, the model is required to cite the provided reference papers to support the generated content. The references in the generated content will be extracted and mapped to the corresponding arXiv papers (see Appendix B for details). The ith subsection St can be expressed S, Draft O,, P.ec).
Following the drafting phase, each section S, individually refined to enhance readability, eliminate redundancies, and ensure a seamless narrative. The refined sections R; are then merged into cohesive document F, which is essential for maintaining logical flow and coherence throughout the survey. During the refinement process, the model needs to polish each subsection based on the local context (considering the previous and following subsections) to improve readability, eliminate redundancies, and enhance coherency. Additionally, the model is required to check the correctness of the cited references in the content and correct any errors in the citations. This procedure can be represented by: F Merge(R,, Rg,.... R2,...,,,,, R,), where Rt Refine(S,).
Integration and Refinement

---

Rigorous Evaluation and Iteration The final phase involves rigorous evaluation and iteration pro- cess, where the survey document is assessed through Muuti-LLM-as-Juddee strategy. This evaluation critically examines the survey in several aspects. The insights gained from this evaluation are used to guide further refinements, ensuring the survey meets the highest academic standards. The best survey is chosen from N candidates. The final output of AutoSurvey is Fbest Evaluate F1, F Fx)).
The methodology outlined here-from initial data retrieval sophisticated multi-faceted evalua- tion-ensures that AutoSurvey effectively addresses the complexities of survey creation in evolving research fields using advanced LLM technologies.
3 Experiments
Setup We conduct comprehensive experiments to evaluate the performance of AutoSurvey, compar- ing it against traditional methods for generating survey papers. For the drafting phase of AutoSurvey, we utilize Claude-3- 3-iku, known for its speed and cost-effectiveness, capable of handling 200K tokens. For evaluations, we employ combination of GPT-4, Claude-3-Haile and Gemini-1.5-PP The evaluation covers the following key performance metrics:
To estimate the time it takes for humans to write a document, we use mathematical model with the following parameters: L (the length of the document), E (the number of experts), M (the writing speed of each expert), T, (the preparation time for research and data collection), Tw (the actual writing time, Te x ) and Te (the editing and revision time, Te 2 TTo) Assuming an ideal situation where
Survey Creation Speed:
10, tokens/hour, T, = 5 hours, and T. TTw. the total time Time calculated as:
2000
L
L
Time T, Tu + T,
(I)
Ex M
Fx M'
For Naive RAG and AutoSurvey, we count all the time of API calls. The speed is calculated as Speed (bous)-
Citation Quality: Adopted from (27), this metric assesses the accuracy and relevance of citations in the survey. Assuming set of claims = (c1,02,... c2....) extracted from the survey, the metric utilizes an NLI model h to decide whether claim e; is supported by its references Ref, = {re, .)., where each rs, represents one paper cited. h(c, Ref,) means that the references can support the claim, and h(c, Ref,) = 0 otherwise. Refer to Appendix C for more details. Citation quality encompasses two sub-metrics:
Citation Recall: Measures whether all statements in the generated text are fully
supported by the cited passages, which is calculated
E Ref,
Recall
[C|
Citation Precision:
Identifies irrelevant citations, ensuring that the provided citations
are pertinent and directly support the statements. Before listing the formula for preci-
sion, a function is defined as g(c,, ri (h(c,, (r.,
1)U (h(c, Ref, fri, }) 0), which measures whether the paper ri, is related to the claim c. The precision
Σ |Ref, h(C, Ref,) O (g(c,,
Precision
k=
Ref,
21=1
Content Quality: An overarching metric evaluating the excellence of the written survey, encompassing three sub-ind ibbiiniicators. Each sub-indiien indiiiiicttor is judged by LLMs according to 5-point rubric, calibrated by human experts to meet academic standards. Note that the detailed scoring criteria are provided in Table
Coverage: Assesses the extent to which the survey encapsulates all aspects of the topic.
Structure: Evaluates the logical organization and coherence of each section.
Relevance: Measures how well the content aligns with the research topic.
"Specifically, 1..4-112-pprreiewe claude- 202400070 and Gemini-1.. .---pr--peeiiee
5

---

Table 1: Content Quality Criteria.
Criteria
Stores
Coverage
Siore survey has very linited coverage. only touching on small portion the sopic and locking discussion key areas.
Score 2: The survey covers some parts of the topic but has noticcable omissions, with signtficant areas either undenepeeseneed or missing.. fully discussed.
Score The survey geneally cvmprehansive in covenge but still misses a few key points that
Score survey
togic compschensivaaly with only vay Score survey compreliensivelly covers all key and peripheral togics, providing detailed discussions and extensive information.
key
topics left out.
Structure
Score survey lacks logic, with no connections betwee sections, miking difficult understand the overall framewurk.
Soore 2:
The survey his weak logical flow with some content arranged in disordered or uns unrcasonable manner.
Soore The survey has generally ressonable logical structure, with most content arrznged ordecty, though some litkss and trasstions could be ingroved such repoated subsections,
Siore survey has good logical consistency, with comeatt arraoged natural transitions, only slightly rigid few parts.
Score 5: The survey is tightly structured and logically clear, with all sections and content arranged most reasonably, and transitions between adjucent sections smooch without redundancy.
Relevance
The content is ouadated or unrelated to the field it purports to review, offering no alignment with the toiic. digressions; Score survey generally lopoc, despiie a few unrelated details.
Score J:
Score survey somewhat topic but
subyect evident but consisteutly adhered
Score The survey mostly on lupic and fucused: the narnaivve has a consistent relevunce to the cure subject with infrequent digressions.
Soore The survey exceptionally locused and cotirely on topic; the article tightly centered on the subject, with every piece information commbuting comprehensive understanding the tpiic.
Baselines We compare AutoSurvey with surveys authored by human experts (collected from Arxiv) and naive RAG across 20 different computer science topics across 20 different topics in the field of LLMs (see Table 7) For the naive RAG, we begin with a title and survey length requirement, then iteratively prompt the model to write the content until completion. Note that we also provide the model with the same number of reference papers with AutoSurvey. To make a more comprehensive comparison, we additionally introduced two baselines: RAG+Reflection, which involves reflecting the generated survey content from RAG, and RAG+Query Rewriting, where the LLM reformulates the retrieval query based on the topic.
For AutoSurvey, we utilize a corpus of 530,000 computer science papers from arXiv as the retrieval database. During the initial drafting stage, we retrieve 1200 papers relevant to the given topic and split them into several chunks with window size of 30,000 tokens. The model generates an outline for each chunk and merges these outlines into a final comprehensive outline, using only the abstracts of the papers this stage. The outline predetermines the number of sections as 8. For subsection drafting, the models generate specific sections using the outline and 60 papers retrieved based on the subsection descriptions, focusing on the main body of each paper (up to the first 1,500 tokens). During the reflection and polishing stage, the same reference papers are provided to the model to ensure consistency and accuracy. The iteration number used for evaluation are excluded during the retrieval process. For more details of implementations, see Appendix B and the prompts are presented in Appendix F
set to 2. Note that human writing surveys
Table 2: Results of Naive RAG, Human writing and AutoSurvey. Both of AutoSurvey and Naive RAG use Claude-haiku as the writer. Note that human writing surveys used for evaluation are excluded during the retrieval process.
Survey ength (tokons)
Methods
Speed
Recall
Citation quality
Precision
Coverage
Structure
Content Quality
Relevance
Avg.
Human writing
0.16
$0.00
87.50
4.50
4.16
5,00
4.52
8k
Naive RAG
79.67
78.14_c as
71 71.92
4.40 is
3.861am
4.8610.33
4.33
Naive RAG: Reflection
22.25:220 as
76.84
4.46g0.57
4.0240.0
4.86.2..1
4.42
Naive RAG-Quesy Rewriting
80 80.99
71.83 so
4.84 2s.
4.05
4.88
4.58
AutoSurvey
107.00
82.48 72
77.42-00s
4.0040.4s
4.4640.40
4.840.3
4.61
Human writing
0.11
4.06
5,00
1.68
16k
Naive RAG
43.11
71.48
65 31 16.31-11..0
4.4620.40
3.66
4.78a0.4
4.25
Naive RAGi-Reflcation
79 07-67=224
73. 73.73
4.57
4 4.2840.30
4
4.834013
4.55
Naive RAG-Query Rewritigg
77. 77.78_a.e
66.29 (4.56
4.70,a.1
3.67
1.79
1.52
AutuSurvey
95.51
76.94
4.66ga.47
4.3320.50
4.86
1.60
Human writing
0.10
88.57
77.14
4.66
4.50
5.00
4.71
32k
Naive RAG
22.64
79. 10.88_4.0
68.00
4.41go
3.7510.7
4.06
4.23
Naive RAG: Reflection
80.50 os
72.181331
4.82
4.0810.11
1.40
4.44
Naive RAG=Query Query Rewriting
76. 76.56.44 as
65.30gase
4.61
3.962045
4.88
4.45
AutoSurvey
91.46
83.14_9 41
78.04.334
4.73300.1
4.2610.00
4.81051
4.58
Human writing
0.07
86.33
77.78
5:00
4.66
5.00
4.88
61k
Naive RAG
12.56
68.794..00
11.97518.4
4.4an
3.6640.4
4.06
1.19
Naive RAGi-Rellcttoo
73 15.12:2244 as
68 68.361345
4.66g0.31
4.06
4.76a0.1
4.47
Naive RAG-Query Rewriting
69. 60.77
62.21
4. 4.4520.35
3.5840.57
1.099029
4.51
AutoSurvey
73.59
82.25
4.73g0.44
4,33g0.7
4,86
1.62

---

Main Results The results of our experiments comparing human writing, Naive RAG, and AutoSur- vey for generating academic surveys are summarized Table 2
The key findings are:
AutoSurvey significantly outperforms both human writing and Naive RAG in terms of speed. For instance, AutoSurvey achieves speed of 73.59 surveys per hour for 64k-token survey, compared to 0.07 for human writing and 12.56 for Naive RAG, highlighting the larger gap in speed for longer context generation.
AutoSurvey demonstrates superior citation quality compared other baselines, with per- formance close to human writing. For an 8k-token survey, AutoSurvey achieves citation recall and precision scores of 82.48 and 77.42, respectively, surpassing Naive RAG (78. recall, 71.92 precision). While human writing achieves the best performance, ours is close to human's across different lengths. We also observe significant decline in citation quality for other baselines as the survey length increased, whereas AutoSurvey maintains stable performance. We investigate this phenomenon in our ablation study.
AutoSurvey excels in content quality, scoring 4.60 on average for a 16k-token survey. achieves 4.66 for coverage, 4.33 for structure, and 4.86 for relevance, matching human writing and surpassing Naive RAG.
The experiments indicate that AutoSurvey provides a balanced trade-off between quality and efficiency. achieves near-human levels of coverage, relevance, and citation quality while maintaining significantly lower time cost. While human writing still leads in structure and overall quality, the efficiency and performance AutoSurvey make it compelling alternative for generating academic surveys. Naive human writing and AutoSurvey, making it the least preferred method among the three for generating high-quality academic surveys, particularly in terms of structure, due to the lack of outline.
LLM, though effective, falls short in several key areas compared to both
To verify the consistency between our evaluation method and human evaluation, we conduct meta-evaluation involving human experts and our automated evaluation system. Human experts judge pairs of generated surveys to determine which one is superior. This process, referred to as a "which one is better" game, serves as the golden standard for evaluation. We compare the judgments made by our evaluation method against those made by human experts. Specifically, we provide the experts with the same scoring criteria used in our evaluation for reference. The experts rank the generated 20 surveys, and we compare these rankings with those generated by LLM using Spearman's rank correlation coefficient to measure consistency between human and LLM evaluations.
Meta Evaluation
The results of this meta-evaluuttin are presented in Figure 33 The table shows the Spearman's 's rho val- ues, indicating the degree of correlation between the rankings given by each LLM and the human experts. The Spearman's rho values indicate moderate posi- tive correlation between the rankings provided by the
0.6
0.55
Spoararn 's the
0.45
0.4
LLMs and those given by the human experts, with the mixture of models achieving the highest correlation at 0.5429. These results suggest that our evaluation method aligns well with human preferences, provid- ing reliable proxy for human judgment.
0.35
GPT-4
Claude
Gemini
Mixture
Judge
Figure 3: Spearman's 's rho values indicating the degree of correlation between rankings given by LLMs and human experts. Note that value over 0.3 indicates a positive correla- tion and over 0.5 indicates strong positive correlation.
Ablation study To assess the impact of various components on the performance of AutoSurvey, we conduct an ablation study by systematically remov- ing key components: the retrieval mechanism, the reflection phase, and iterations. Additionally, we evaluate the influence of using different base LLMs to demonstrate that even with less optimal LLM like Claude-haiku, AutoSurvey's 's performance remains comparable to human- -eman-generraed surveys.
Table 3 demonstrates that removing the retrieval mechanism significantly degrades citation quality, highlighting its critical role in ensuring accurate and relevant references. The absence of the reflection phase slightly impacts the overall content quality, particularly in structure and coherence.
7

---

Table Ablation study results for Aut with different components removed.
Citation
Quality
Content Quality
Methods
Recall
Precision
Coverage
Structure
Relevance
Avg.
AutoSurvey
83.484s.05
77. 77.15=6.05
4.740.45
4.16g0.73
4.93+0.30
4.37
AutoSurvey w/o retrieve
60.11
51.66 51.6516.30 16.33
1..49
1..14
1.8810.32
4.44
AutoSurvey w/o reflection
83.23
76.36
4.7620.42
4.13
4.88
4.56
Table 4: Performance of AutoSurvey with different base LLM writers.
Citation Quality Recall
Content Quality Structure
Base L.M writer
Precision
Coverage
Relevance
Avg.
GPT-4
20.25
78.83 28.03
4.820.54
4.4620.49
4.86
4.70
Claude-haiku
82.45=2.77
76.31
4.6620.47
4.26
4.8620.3
4.58
Gemini-1 Commini-1.5-ppe 1.5-po
78.13
71.24
4.86
4.33
1.93
4.69
Human
85.86
80.51
4.71
4.43
5
4.70
Table shows the performance of AutoSurvey when using different LLMs as the base writer. The results indicate that all three LLMs (GPT-4, Claude-haiku, and Gemini-1.5-Pro perform well, with GPT-4 slightly outperforming the others in terms of overall content quality. Importantly, with the less optimal hauudehhiiku AutoSurvey's performance remains competitive with human standards.
Figure presents the effect of different iteration counts on the performance of AutoSurvey. The re- sults show that increasing the number of iterations from 5 leads slight improvement overall content quality, with diminishing returns after the second iteration.
4,93
4.9
1.87
4.87
4.8
473
4.3
Average
4.67
Coverage
4.6
4.63
463
4.65
Sturtue
4.58
Relevance
To assess whether the generated survey can provide useful information enrich the knowledge, we cre- ated 50 multiple-choice questions about 5 topics. These questions primarily involve knowledge related to literature, such as identifying which paper pro- posed a particular method. We compared the accu- racy of the Claude model under the following condi- tions: (1) directly chooses the answer without provid- ing any reference materials, (2) has access to a 32k
4,27
427
427
427
4.2
Ite:=1
Ito=2
Itc:=3
Ite:=5
Iteration
Figure 4: Impact of Iteration on AutoSurvey Performance.
survey generated by naive RAG-based LLMs, (3) has access to 32k survey generated by AutoSurvey, and (4) refer to 20 papers (30k tokens in total) retrieved using the options provided (Upper-bound, directly retrieving the answers).
The results are shown in Table and ing topic-related materials can effectively improve the accuracy of answers. Providing option-relace papers can be considered an upper bound for perfor- mance (73.60%). AutoSurvey improves accuracy by 9.2% compared to directly answering and is 2.4% higher than using naive RAG-based I.L.--gnerated surveys. This demonstrates that our method can ef- fectively provide topic knowledge.
and we find provid-
Table 5: Performances given different refer- ences.
Methods
Accuracy
Direct
58.40
Naive RAG-based LLMss
65.20 20008.00
Upper-bound
73.60
AutoSurvey
67.60
As mentioned in the main results, the naive RAG- based generation method demonstrates a notable de- cline in citation quality as the survey length increases. contrast, AutoSurvey maintains stable citation quality across varying survey lengths. Such phenomena may be attributed to the streaming generation process, where each step must reference previous content, leading to the accumulation of errors 28 To validate this, we segment the extracted claims into 20% intervals and calculate the citation recall for each segment. The results in Table/jindicatt that the recall of Naive RAG gradually decreases as the generated text length increases, while AutoSurvey maintains stable performance.

---

Table 6: Performance of AutoSurvey with different base LLM writers.
Method
20%
40%
60%
80%
100%
Naive RAG
76.79
73.17
71.52
64.08
49.85
AutoSurvey 82.86
84.89
79.04
82.27
82.29
In summary, the ablation study underscores the critical role of the retrieval mechanism and reflection phase in AutoSurvey. Furthermore, the performance is influenced by using different LLMs as the base writer and varying the iteration count. Nevertheless, AutoSurvey consistently performs well across various configurations, showcasing its robustness and efficiency.
User study To evaluate the real-world performance of AutoSurvey, we launch a free trial and distribute anonymous survey to 141 users. The survey primarily assesses user perceptions of generation quality and whether the generated surveys contribute to their practical work. Users are asked to rate the relevance, structure, and usefulness to their actual work of the generated survey on a scale from 1 (poor) to (excellent). A total of 93 valid responses are collected. Results reveal that users generally perceive AutoSurvey relevant to topic, well structured, and useful, with most ratings skewed towards 4 and 5. Compared to other aspects, the proportion of scores below 5 in the "Structure" higher, indicating that there remains room for improvement in further refining the structure. We are pleased to observe that the majority of users think the generated surveys beneficial their practical work, indicating the utility and practical value of AutoSurvey.
511
49
(a) Relevance
(b) Structure
(c) Usefulness
Figure Distribution of user ratings terms of relevance, structure and usefulness.
4
Related Work
Long-form Text Generation The ability to effectively process and generate long-form text is critical challenge for large language models (LLMs) due to the need to maintain coherence and logical flow over extended passages of text [29] 30 31 32]. Several works try address the challenge by directly extending the context window with different Positional Encoding Techniques [33] 34). However, modifying position encoding strategies requires retraining the model, which is costly. Another solution using memory-augmennted techniques. RecurrentGPT 35 enables the generation of arbitrarily long texts by simulating the recurrence mechanism of RNNs using natural language prompts to store previous contextual information. Temp-Lora (36] enables long text generation by embedding context information into a temporary Lora module updated progressively during generation rather than relying on an extensive context window. These methods effectively establish relationships among tokens and maintain contextual understanding. but still face the issue of long generation times. To further accelerate the generation process, Hierarchical Modeling Techniques have been explored extensively to capture inherent hierarchical nature of long-form text [37 38 Despite such efficiency. it ignores the long dependency of text and may degrade the content quality 399. To tackle the drawbacks, AutoSurvey, similarly using a Hierarchical generation paradigm, creates well-organized outline for guidance and refines the generated content to improve the quality.

---

Automatic Writing Due to the high costs associated with manual writing, automated writing has attracted substantial research interest in recent years. Compared to traditional methods, which primarily focus on training models to generate linguistically coherent text 40 41 the emergency of
large language models (L..Ms) has opened up new possibilities for automated writing, drawing more
attention to broader aspects like faithfulness, logical structure, style, and ethics [42 43 44 45 For
example, Retrieval-Augmented Generation techniques are useful for generating claims with citations IRP framework [47 generates expository text by iteratively performing content planning,
127 46].
fact retrieval, and paraphrasing to ensure factuality and stylistic consistency. Several works focus on the outline creation to improve the structure of generated content. PaperRobot [48] incrementally writes key elements to generate a paper abstract. STORM (20) designs a refined outline based on multiple rounds of wiki-page- pal---aaggeeeeee Q&A to facilitate wiki-like article generations. These methods have only been explored in shorter texts (<4k). In contrast, Autosurvey shows its effectiveness in generating long content (64k), with focus academic reviews.
Limitation
In addition to directly using recall and precision to evaluate citations, also perform a manual analysis, providing a more comprehensive view of the citation quality. We examine 100 unsupported claims and their corresponding references and find that the emors mainly fall into three categories: (I) Misalignment, (2) Misinterpretation, and (3) Overgeneralization. Misalignment occurs when the connection between them is incorrectly made, such an irrelevant citation. Misinterpretation happens when the claim and source are related, but the claim incorrectly represents the information from the source. Overgeneralization occurs when claim extends the conclusions of the source material to a broader context than is supported. Among the three types of errors, overgeneralization accounts for the largest proportion (51%), indicating that LLMs still rely heavily on their parametric knowledge for writing. Misinterpretation has a small proportion (10%), suggesting that LLMSs are capable understanding the content of the references most cases, avoiding the creation of claims that significantly deviate from the references.
Misalignment (39%) An example is citing the "General Data Protection Regulation (GDPR)" in context where the referenced paper does not propose GDPR but merely mentions it in the content.
Misinterpretation (10%); ): An example is claiming that "In-context learning allows LLMs to adapt to new tasks by simply conditioning on a few demonstration examples, without the need for any parameter updates or fine-tuning." based on paper that focuses on meta-out-offconeee learning and mentions the limitations of in in-context learning.
Overgeneralization (51 %): An example is that "in-context learning can also benefit from advance- ments other learning paradigms, such as multi-task learning," " based on a paper that discusses multi-task sew-shot learning but does not explicitly address its influence on in-context learning.
Among the three types of errors, overgeneralization accounts for the largest proportion (51%), indicating that LLMs still rely heavily on their parametric knowledge for writing. Misinterpretation has small proportion (10%), suggesting that LLMs are capable of understanding the content of the references in most cases, avoiding the creation of claims that significantly deviate from the references. Additional potential societal impact and ethical considerations are discussed in Appendix B
6 Conclusion
In this paper, we introduce AutoSurvey, a novel methodology leveraging large language models to automate the creation of comprehensive literature surveys. AutoSurvey addresses key challenges such as context window limitations and parametric knowledge constraints through a systematic approach involving initial retrieval, outline generation, parallel subsection drafting, integration, and rigorous evaluation. Our experiments show that AutoSurvey significantly outperforms Naive RAG and matches human performance in content and citation quality, while also being highly efficient. This advancement offers a scalable and effective solution for synthesizing research literature, providing a valuable tool for researchers in rapidly evolving fields like artificial intelligence.
10

---

References
[1] Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei- Ling Shyu, Shu-Ching Chen, and Sundaraja S Iyengar. A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR), 51(5):1-36 2018.
[2] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models, ACM Transactions Intelligent Systems and Technology. 2023.
[3] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.1822, 2023.
[4] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1-41, 2022.
[5] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444 2015.
[6] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[7] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 1.Xi:22333.08774 2023.
[8] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yo Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-
4026, 2023.
[9] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[10] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. 2023.
[11] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Lon- glora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference Learning Representations, 2023.
[12] Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. Advances in Neural Information Processing Systems, 36, 2024.
[13] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173. 2024.
[14] Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.00169, 2023.
[15] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210-31227. PMLR, 2023.
[16] Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source llms truly promise? In NeurlPS 2023 Workshop Instruction Tuning and Instruction Following, 2023.
11

---

[17] Tianle Li, Ge Zhang, Quy Duc Do. Xiang Yue, and Wenhu Chen. Long-context Ilms struggle with long in-context learning, arXiv preprint arXiv:2404.0224 020600 2024.
[18] Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint trXiv:2310.07522 2023.
[19]
Ziwei Ji, Naycon Lcc, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.
ACM Computing Surveys, (5(12):1-33 2023.
[20] Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), 2024.
[21] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. 2024.
[22] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[23] Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. Kieval: A knowledge-grounded interactive evaluation framework for large language models. 2024.
[24] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augee -agmeented generation for large language models: A survey, arXiv preprint arXiv:2312. 10997, 2023.
[25] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmenter generation for knowledge-intensive nlp tasks. Advances Neural Information Processing Systems, 33:9459-9474 2020.
[26] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yuu Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969-7992, 2023.
[27] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[28] Jian Guan, Xiaoxi Mao. Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. Long text generation by modeling sentence-level and discourse-leved coherence. In Chengging Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), August 2021.
[29]
Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric P. Xing, and Zhiting Hu. Progres- sive generation of long text with pretrained language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tüt Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2021, Online, June 11, 2021, pages 4313 4324. Association for Computational Linguistics, 2021.
[30] Yushi Bai, Xin Lv, Jiajie Zhang. Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao
Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. CoRR, abs/2308. 14508, 2023.
12

---

[31] Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji- Ji-Rong Wen. BAMBOO: A comprehensive benchmark for evaluating long text modeling capacities of large language models. CoRR, abs/2309. 13345, 2023.
[32] Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-content language models understand long contexts? CoRR, abs/2311 1121111.4999 .00939, 2023.
[33] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. In Marilyn A. Walker, Heng Ji. and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 464 468. Association for Computational Linguisties, 2018.
[34] Xing Wang. Zhaopeng Tu, Longyue Wang, and Shuming Shi. Self-attention with structural position representations. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-UCNNP 2019, Hong Kong, China, November 3-7, 2019, pages 1440--409 Association for Computa- tional Linguistics, 2019.
[35] Wangchunshu Zhou, Yuchen Elcanor Jiang. Peng Cui, Tiannan Wang. Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text. arXiv preprint MxXv:2235511330 2023.
[36] Y Wang, D Ma, and D Cai. With greater text comes greater necessity: Inference-time training helps long text generation. arXiv preprint arXiv:2401.11504, 2024.
[37] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint 18X.:v1805.04833 2018.
[38] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiemnon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
[39] Yapei Chang, Kyle Lo. Tanya Goyal, and Mohit lyyer. Booookscore: A systematic exploration of book-length summarization in the era of llms. arXiv preprint arXiv:2310.00785 2023.
[40] Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang, Xiujun Li, Michel Galley, Chris Brockett, Mengdi Wang, and Jianfeng Gao. Towards coherent and cohesive long-form text generation. arXiv preprint ar arx; 181 100511, 2018.
[41] Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. Discourse-aware neural rewards for coherent text generation. arXiv preprint arxiv: 1et11805..00 0076666 2018.
[42] Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. arXiv preprint 230.:22333311313 2023.
[43] Jinxin Liu, Shulin Cao, Jiaxin Shi, Tingjian Zhang, Lei Hou, and Juanzi Li. Probing structured semantics understanding and generation of language models via question answering. arXiv preprint arXiv:2401. 005777, 2024.
[44] Chiyu Zhang, Honglong Cai, Yuexin Wu, Le Hou, Muhammad Abgul-Mageed, et Distilling text style transfer with self-explanation from llms. arXiv preprint arXiv:2403.01 106, 2024.
[45] Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. Large pre-trained language models contain human-like biases of what is right and wrong do. Nature Machine Intelligence, ((3):258-263) 2022.
[46] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chad- wick, Mia Glaese, Susannah Young, Lucy Comppell-Gillinghaal Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022.
13

---

[47] Nishant Balepur, Jie Huang, and Kevin Chen- Chuan Chang. Expository text generation: Imitate, retrieve, paraphrase. arXiv preprint arXiv:2305.03276, 2023.
[48] Qingyun Wang, Lifu Huang. Zhiying Jiang, Kevin Knight, Heng Ji, Mohit Bansal, and Luan. Paperrobot: Incremental draft generation of scientific ideas. arXiv preprint 19Xi::1905007700 2019.
[49] Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training reproducible long context text embedder, 2024.
A Detail of Topics and Human-writing Surveys
We select 20 surveys from different topies within the LLM field. During the selection process, we prioritize both the breadth of the topics and the citation count (from google scholar) of the surveys. The basic information of surveys are listed in Table 7
Table 7: Survey Table
Tople
Survey Title
Citations
In-contest Learning
surve in-contest learning
LLMs for Recommendation
Survey on Large Language Models Recommendation
55
LLM- Generated Texts Detection
Survey of Deuectigg LLM GMMeraeree Texts
42
Explaimaility for LLMs
Explainahility for Large Language Models
25
Esaluaaion LLMs
Survey Evoluation Large Language
LLMs-based Agents
Survey on Large Language Model based Autenomous Agmis
LLMr in Medicine
Survey of Large Language Models in Medicine
234
Domsin Specialization of LLMs
Domain Specialization the Key Make Lange Language Models Disnuptive
Challenges L.Mss Education
Practical Edhical Challenges
Languge Models Education
Alignment of LLMs
Aligning Large Language Models with Human
53
ChaGGT
Survey CharGPT and Beyond
144
Large Language Models Language Models
Instraction
for
LLMs Information Retrieval
Retrival
22
Safety in LLMss
Towards Sufer Generalive Language Models: Safety Risks, Evaluntions, and Improvements
[7
Chain of Thought
Survey of Chain of Thought Keasoning
13
Hallucination LLMs
Survey Hallscination Large Language Models
116
and
LLMs
and Fairness Lange Language
Large Multi Modal Language Models
Large scale Mula Modal Pre trained Models
61
Acceleration for LLMs
Survey on Model Compression and Acceleration for Pretrained Language Models
22
LLMs for Sollware Engineering
Lange Languae Models for Software Engineering
19
B Details of Implementations
We adopt nomic-embed-text--1.. (4),, a widely used embedding model in RAG applications. To build our database, we store the embeddings of the title and abstract for each paper. Since the context window length is 8k, which is longer than any individual abstract, we embed the raw text directly without chunkings. During generation, related papers are retrieved by the abstract and ranked by their similarity to the query. When generating subsection content, the model needs to write the corresponding paper titles where citations are required. After generation, each title will be embedded as query and be mapped to the closest paper title in our database. This approach allows the LLMs to use their own parameter knowledge to generate citations without references while ensuring the existence of the generated citations. When calling API, we set temperature = and other parameters as default. Even with the same parameters, the final length of the generated surveys can vary. Therefore, papers with lengths from 8k to 16k are classified into the 8k category, those from 16k to 32k into the 16k category, and so on.
C Details of Evaluation
For citation quality, we define sentence with at least one citation as a claim and extract all the claims from the generated survey. For human evaluations, we invite three PhD students and of them have experience in writing M.-Mlleeettd surveys. We provide them with the same scoring criteria, along with explanations of the specific metries. They are asked to score based on these criteria, and the final rankings of the generated surveys are determined by the total scores.
14

---

D Cost Analysis
We present the average number of tokens generate 32k-tokens survey, along with the cost of using different LLMs in Table 8
Table 8: Cost of AutoSurvey
Input tokens
Output tokens
Claude-haiku Gemini-L.5-pro
GPT-4
3009.7K
112.9K
0.89$
11.72$
33.48$
E
Societal Impact and Ethical Considerations
By integrating various specialized databases, our approach can generate academic surveys across different fields, potentially filling the gaps in existing reviews. However, as method relies on the performance of large models, it inevitably contains citation errors. Therefore, the generated survey content is intended for reference only. All personnel involved in the evaluation process participated voluntarily and received ample compensation. All data used in our experiment is sourced from arXiv and is allowed for non-commercial use.
Prompt used in AutoSurvey
RDUGH_ DUNTOOUNNINNEPPPPP
You want to write overall and comprehensive academic survey about [TOPIC].
You are provided with a list of papers related to the topic below:
[PAPER LIST]
You need to draft outline based on the given papers. The outline should contains a title and several sections. Each section follows with brief sentence to describe what to write in this section.
supposed to be comprehensive and contains [SECTION NUM] sections.
The outline
Return in the format: <format>
Title: [TITLE OF THE SURVEY] Section 1: [NAME OF SECTION 1]
Description 1: [DESCRIPTION OF SENTCTION 1]
Section K: [NAME OF SECTION K] Description K: [DESCRIPTION OF SENTCTION K] </format>
outline:
SUBSECTION_ DOSEECTIOO.OUTTIIE_PPRRO
You want to write overall survey about [TOPIC].
You have created a overall outline below:
[OVERALL OUTLINE]
The outline contains a title and several sections.
Each section follous with a brief sentence to describe what to write in this section.
15

---

You need to enrich the section [SECTION NAME].
The description of [SECTION NAME]: [SECTION DESCRIPTION] You need to generate the framwork containing several subsections based on the overall outlines.
Each subsection follous with brief sentence to describe what to write in this subsection.
These papers provided for references:
[PAPER LIST]
Return the outline in the format: <format>
Subsection [NAME OF SUBSECTION
Description 1: [DESCRIPTION OF SUBSENTCTION 1]
Subsection K: [NAME OF SUBSECTION K] Description K: (DESCRIPTION OF SUBSENTCTION K]
</format>
Only return the outline without any other informations: 11
BERCING_OUTLINEEPPROO
You want to write overall survey about [TOPIC],
You are provided with a list of outlines as candidates below:
[OUTLINE LIST]
Each outline contains a title and several sections. Each section follows with a brief sentence to describe what to write in this section.
You need to generate final outline based these provided outlines to make the final outline show comprehensive insights the topic and more logical.
Return the in the format:
<format>
Title: [TITLE OF THE SURVEY]
Section 1: [NAME SECTION
Description 1: [DESCRIPTION OF SENTCTION 1]
Section K: [NAME OF SECTION K] Description K: (DESCRIPTION OF SENTCTION K] </format>
Only return the final outline without any other informations: 12>
SUBSECTION QUSSECTIO__WRTING.RRAO WWW.WWN_IIIIINIIIIII
You wants write overall and comprehensive survey about TOPIC].
You have created a overall outline below:
[OVERALL OUTLINE]
Below are a list of papers for reference:
[PAPER LIST]
Now you need to write the content for the subsection: " "[SUBSECTION ".ME]*.
16

---

The details of what to write in this subsection called [SUBSECTION
NAME] is in this descripition:
[DESCRIPTION]
Here is the requirement you must follow:
The subsection is recommended contain than [WORD NUM] words.
2. When writing sentences that are based on specific papers above, you cite the "paper_title" in '[]' format to support your content.
Here's concise guideline for when to cite papers in a survey:
1. Summarizing Research: Cite sources when summarizing the existing literature.
2. Using Specific Concepts or Data: Provide citations when discussing specific theories, models, or data.
3. Using Established Methods: Cite the creators of methodologies
you employ in your survey.
4. Supporting Arguments: Cite sources that back up your arguments.
conclusions
Only return the content more than [WORD NUM] words you write for the subsection [SUBSECTION NAME] without any other information:
122
want to write overall comprehensive survey about [TOPIC
].
Below are a list of papers for references:
[PAPER LIST]
You have written a subsection below:
[SURSECTION]
The sentences that are based on specific papers above are followed with the citation of "paper_title" in "[]".
For example 'the emergence of large language models (LLMs) [PaLM: Scaling language modeling with pathways]'
Here's concise guideline for when to cite papers in a survey:
1. Summarizing Research: Cite sources when aummarizing the existing literature.
2. Using Specific Concepts or Data: Provide citations when discussing specific theories, models, or data.
3. Using Established Methods: Cite the creators of methodologies you employ in your survey.
4. Supporting Arguments: Cite sources that back up your conclusions and arguments.
Nor you need check whether the citations of "paper_title" in this subsection is correct.
Once the citation can not support the sentence you write, correct the paper_title in '[}> or just remove it.
change any other things except the citations. Only return the subsection with correct citations:

---

3.2
You want to write overall comprehensive survey about [TOPIC ].
Now you need to help to refine one of the subsection to improve th ecoherence of your survey.
You are provied with the content of the subsection along with the previous subsections and following subsections.
Previous Subsection:
[PREVIOUS]
Following Subsection:
[FOLLOWING]
Subsection to Refine:
[SUBSECTION]
Now refine the subsection to enhance coherence, and ensure that it connects more fluidly with the previous and following subsections
Remember that keep all essence core information the subsection intact. Do not modify any citations [ following the sentences!!!!
Only return the whole refined content of the subsection without any other informations: 113
MLI_PROMPT
Claim:
[CLAIM]
Source:
[SOURCE]
Claim:
[CLAIM]
the Claim faithful to the Source? A Claim is faithful to the Source if the core part in the Claim can be supported by the Source. An Only reply with 'Yes? 'oo 133
SAIVE_RACPPROMT
)
You are an expert artificial intelligence who wants to write overall and comprehensive survey about [TOPIC).
You are provided with list of papers related to [TOPIC] below:
[PAPER LIST]
Here is the survey content you have written:
18

---

[SURVEY CONTENT]
Hers is the requirement of the survey:
1. The survey must be more than (SURVEY LEN] tokens! containing serval sections. Each section contains several subsections.
5. Cite several paper provided above to support the content you srite.
Here the format of your writing:
1. '##" indicates the section title
2. *### indicates the subsection title
3. Only cite the "paper_title" in '[]. An example of citation:
the emergence of large language models (LLMs) [Language models few-shot learners; Language models are unsupervised multitask
learners; PaLM: Scaling language modeling with pathways]'
You need to continue writing the survey adding section subsection.
Do not stop until the length of survey is more than [SURVEY LEN] tokens!!!
Return the content you write:
19

---

Comprehensive Survey Emotion Recognition using Large Language Models
## 1. Introduction to Emotion Recognition and Large Language Models Emotion recognition has been crucial and active research area in field of affective computing, which to understand, interpret, and respond to human emotions [1].
enable machines
Emotions play a fundamental role in human cognition, decision- making, and social interaction [2], and the ability to automatically recognize and interpret emotions has wide range of applications, including healthcare, education, entertainment, human computer interaction [3). The importance of emotion recognition is evident in various real- applications. In healthcare, emotion recognition can be used to monitor patient mental health, provide personalized therapy, and improve doctor- patient communication [4]. In education, emotion recognition can help identify student engagement and frustration levels, enabling adaptive learning environments that cater to individual needs [6]. be used
In the entertainment industry, emotion recognition analyze viewer responses and tailor content to evoke desired emotionat emotion recognition, the field faces several challenges that have hindered its widespread adoption and implementation [7]. One of the primary challenges is the inherent complexity and subjectivity emotions, which can vary across individuals, cultures, and contexts [8]. Emotions are often expressed through multiple modalities, including facial expressions, vocal cues, body language, and physiological signals, and integrating these diverse sources of information Additionally, the availability of high-qualle annotated emotion datasets is persistent challenge in the field [10]. Many existing datasets are limited in size, lack diversity, have inconsistent
emotional responses [6]. Despite the significant benefits of
a significant challenge [9]. diverse,
subjective emotion labeling, which lead to biases and poor generalization of emotion recognition models [11].
### 1.1 Background Emotion Recognition
### 1.2 Large Language Models and their Capabilities
### 1.3 Emotion Representation in LLMs
### 1.4 Multimodal Emotion Recognition using LLMs
## 2. Techniques and Approaches for Emotion Recognition using LLMs
Fine-tuning LLMs on Emotion Datasets
### 2.2 LLM-based Prompting Methods for Emotion Recognition
### 2.3 Integrating LLMs with Other Modalities for Multimodal Emotion Recognition
# 3. Enhancing LLM based Emotion Recognition
### Data Augmentation Improving Emotion Recognition
### 3.2 Prompt Engineering for Emotion Recognition
### 3.3 Integrating External Knowledge for Emotion Recognition
## 4. Challenges, Limitations, and Ethical Considerations
### 4.1 Model Biases and Hallucinations
20

---

#*# 4.2 Interpretability and Explainability
### 4.3 Ethical Considerations
## 5. Applications and Future Directions
### 5.1 Assistive Robotics
### 5.2 Mental Health Assessment
#*# Customer Service and User Experience
#*# 5.4 Symbolic Reasoning and Long-tailed Emotions
### 5.5 Robust Evaluation Frameworks
## References
[1] Affective Computing for Large Scale Heterogeneous Multimedia Data Survey
[2] Emotion Recognition in Conversation Research Challenges, Datasets and Recent Advances
[3] A Comprehensive Survey Affective Computing; Challenges, Trends, Applications, and Future Directions
[4] Affective Computing for Healthcare Recent Trends, Applications, Challenges, and Beyond
[5] Automatic Sensor-fre Affect Detection A Systematic Literature Review
[6] Affective Video Content Analysis Decade Review and New
Perspectives
[7] Emotion Recognition from Multiple Modalities Fundamentala and
Methodologies
[8] The Ambiguous World of Emotion Representation
[9] Multimodal Affective Analysis Using Hierarchical Attention
Strategy with Word-Level Alignment
[10] Expression, Affect, Action Unit Recognition Aff-Wild2, Multi-
Task Learning and ArcFace
[11] Feature Dimensionality Reduction Video Affect Classification Comparative Study
21

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes] See limitation section
Justification: [NA]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA] No theory in this paper
22

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes] All details provided
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuii wwwtii/gguie///eeeeceoocc CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes] See experiment section
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:::///nuriipee www.e/iw....p..uu.ppeeee httpp///scccpp.tttttttttttteeeea
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We just use apis of LLMs, and their owners have safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "hqqhfejZH1", "reviewer": "Reviewer_LdXj", "summary": "This paper introduces a fast automating way to write literature surveys based on LLM.It aims to solve the challenges of large volume, complexity, context window limitations, parametric knowledge constraints, and lack of evaluation benchmark.The AutoSurvey pipeline contains initial retrieval & outline generation, subsection drafting, integration and refinement, rigorous evaluation and iteration. The experiments are conducted compared to human experts and naive RAG-based LLM and evaluated on survey creation speed, citation quality, content quality. The experiments show that AutoSurvey is much faster than human writing and RAG, matches  human writing and outperforms RAG in citation and content quality.", "questions": "In Experiments, how do you obtain the citation and content quality score?\nWhat is the novelty of this method and the bottlenecks this paper breaks through? \nDoes this method address the limitations of automatic survey generation? I would suggest a more thorough summary on it.\nMinor questions:\nWhere do the forecasted numbers of publication in 2024 in Figure 1 come?", "limitations": "Yes. The limitations are discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is relatively novel and meaningful to adopt LLM effectively in automatic creation of survey papers. The paper is well structured, and clearly written. The general pipeline of AuthoSurvy is logical, from outline generation to integration and refinement, which ensembles the human writing process. The experiments and analysis are clear.", "weaknesses": "- About evaluation metric, it is unclear how the citation quality and content quality are obtained, although the metrics and scores are defined by formulas and words. If you use an LLM-based procedure to get the scores for citation quality and content quality, what are the prompts you give to the LLM? Please give a clearer introduction on this.\n- The algorithm and methodology this paper introduces are too straightforward. Although it is a good application of LLM, I doubt the value of sharing this knowledge with the scientific society, especially for Neurips.\n- Regarding experiment comparison, I highly recommend comparing Autosurvey to not only naive RAG, but more advanced methods, which could make the results more convincing. \n- In table 2, apart from the speed, the other benefits and improvements that AuthoSurvey brings are not significant enough.\n- Minor issues: typo in line 66, page 2: Muti -> Multii", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n**W1: \"About evaluation metric, xxx.\"**\n\nWe appreciate the reviewer’s feedback on the need for a clearer explanation of the evaluation metrics. The citation quality and content quality scores are both obtained using LLMs. The prompts involved in the evaluation can be found in the code we provide, we will include these prompts and a more detailed explanation in the revised version of the paper.\n\nApart from the formulas introduced in the paper, some specific details of the evaluation are as follows:\n\n- **Citation Quality:** Specifically, we first use regular expressions to extract all sentences from the survey. Sentences that contain citations are considered claims and will be evaluated. The papers cited in the claims will be used as sources for the NLI (Natural Language Inference) model's reasoning. The NLI model determines whether the sources support the validity of the claims and returns 0 or 1.\n  \n- **Content Quality:** We directly provide the LLMs with the scoring criteria and the generated results, allowing the model to score between 1 and 5 based on these criteria. Each time, only one aspect will be scored.\n\nNote that, in our experiment, we use three different models for evaluation. Each model evaluates the survey five times, and the average result is taken. Overall, a single survey requires 3 x 5 = 15 evaluations in total.\n\n**W2: \"The algorithm and methodology this paper introduces xxx.\"**\n\nThank you for your feedback. We believe that AutoSurvey is a novel and significant application of AI, particularly relevant to NeurIPS 2024's emphasis on impactful AI applications (https://nips.cc/Conferences/2024/CallForPapers). While the methodology may appear straightforward, it effectively addresses the complexities of automatic survey generation, including context window limitations, real-time knowledge updates, and rigorous evaluation. These contributions are crucial for advancing AI's practical utility in academic research, aligning well with the conference's focus on innovative and practical AI applications.\n\n**W3: Regarding experiment comparison, I highly recommend comparing AutoSurvey to not only naive RAG but more advanced methods, which could make the results more convincing.**\n\n**A3:**\n\nTo address your concerns regarding experiment comparison, we have supplemented the original naive RAG-based experiments by including refinement and query rewriting stages. In Naive RAG + Refinement, the LLM is required to enhance the continuity of the written content with previous sections and check for factual errors based on the references retrieved. In Naive RAG + Query Rewriting, references are first retrieved using the topic, after which the LLM rewrites the query based on the references to assist in writing subsequent content\\ \\\n**Due to the length constraints of the rebuttal, we present the experiments details in Author rebuttal above.**\n\n**W4: In table 2, apart from the speed, the other benefits and improvements that AuthoSurvey brings are not significant enough.**\n\n**A4:**\nWhile speed is a significant advantage of AutoSurvey, we believe the improvements in citation quality and content quality are also noteworthy. AutoSurvey consistently achieved higher citation recall and precision compared to naive RAG-based LLMs and approached human performance levels. Moreover, the content quality metrics, including coverage, structure, and relevance, show that AutoSurvey produces well-structured and relevant surveys. These improvements, combined with the efficiency gains, make AutoSurvey a valuable tool for academic survey generation.\n\n\n**W5: Minor issues: typo in line 66, page 2: Muti -> Multi**\n\n**A5:**\nThank you for pointing out this typo. We will correct \"Muti\" to \"Multi\" in the final version of the paper.\n\n\n### Questions\n\n**Q1: \"In Experiments, how do you obtain the citation and content quality score?\"** \n\n**A1:**\nRefer to the response to Weakness 1.\n\n\n**Q2: \"What is the novelty of this method and the bottlenecks this paper breaks through? Does this method address the limitations of automatic survey generation? I would suggest a more thorough summary on it.\"**\n\n\n**A2:**\nThe novelty of AutoSurvey lies in its systematic approach to addressing key challenges in automatic survey generation as we wrote in Introduction of our paper, we provide some additional explanations h.\n\n- **Context Window Limitations:** Due to the considerable length of the survey (typically around 32k tokens), it exceeds the output window limitations of most closed-source models. For example, a single API call for GPT-3.5 cannot return more than 4k tokens. The parallel framework of AutoSurvey addresses this issue effectively.\n- **Parametric Knowledge Constraints:** The model requires Retrieval-Augmented Generation (RAG) to mitigate the issue of hallucinations during generation. However, a key problem lies in how to effectively retrieve the necessary literature（e.g., there is no significant performance improvement for naive rag + query rewriting）. AutoSurvey offers a 530K-size paper database and guides the retrieval of references based on the writing of an outline.\n- **Lack of Evaluation Benchmarks:** \nAs far as we know, there is no universally accepted standard in the academic community for evaluating the quality of a survey. AutoSurvey proposes an evaluation metric and utilizes multiple LLMs to mitigate bias, aligning the evaluation with human standards.\n\n**Q3: \"Minor questions: Where do the forecasted numbers of publication in 2024 in Figure 1 come?\"**\n\n\n**A3:**\nThe forecasted numbers of publications in Figure 1 are based on extrapolating the data from the first four months of 2024. We used a linear regression model to project the annual totals based on this data. We will include this explanation in the caption of Figure 1 and the corresponding section of the paper for clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CcPsfvqsQB", "reviewer": "Reviewer_jk6Z", "summary": "In this paper, the authors propose an automated system based on LLMs to draft literature surveys on a given topic. The core idea behind the approach involves decomposing the task of writing a survey into multiple smaller subtasks in 4 stages. The first stage focuses on retrieving relevant papers from a database using embedding-based retrieval, leveraging the topic and abstract of papers for selection. An LLM is then prompted given the papers to generate a plan / outline for drafting the survey paper. To address the limited context window, the papers are randomly divided and processed across multiple LLM calls to generate several outlines. These outlines are then merged in a using another LLM call. In the next stage the subsections identified in the outline are populated by retrieving the relevant papers specified in the outline and providing it to an LLM to extract information relevant to the subsection. This is followed by an iteration and refinement stage where the goal is improve the overall coherence and readability of the drafted survey. Finally this is followed by an Evaluation phase where multiple LLMs are used to rate the overall generated content and the generated survey with the highest rating is picked. The authors conduct comparison experiments of their methods against Human and Naive RAG based LLM approaches and show that their proposed approach is faster and leads to better citation and content quality in some settings.", "questions": "Are there any experiments that show how useful these generated academic surveys are to people working in the field? Which aspect of the generated survey is the most helpful for humans?\nHow well does this technique handle multimodal information (i.e figures/graphs) or information from structured sources such as tables?How often do these appear in the generated surveys?", "limitations": "The technique appears to be more effective at summarizing works in the literature for a survey, rather than performing the detailed analysis and comparison of works that characterize a high-quality survey.\nThe technique seems heavily biased by the retrieval stage. This could limit its ability to reference and use relevant works whose topics/abstracts do not semantically match well with the topic of the survey.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper attempts to tackle a new problem of automatically writing academic surveys and provides metrics and evaluation criteria that can be used to compare different techniques on this task.\nThe authors show that their technique can generate academic surveys much faster than humans / naive RAG based LLM generation.\nThe authors perform experiments that depict that the proposed technique outperforms naive RAG based LLM generation in content & citation quality.", "weaknesses": "The paper lacks a significant novelty component. The concepts of decomposing tasks into smaller subtasks for LLMs, using Retrieval-Augmented Generation (RAG), iteratively refining generated content with LLMs, and employing multiple LLMs as evaluators are well-established in the literature. The extension of these ideas to the application of automatically writing surveys seems to be the only novel contribution of this work. \n\nThe evaluation criteria seem to overly rely on the ability of ML models (NLI and LLMs) to judge the quality of the survey. A well-written human survey, for instance, cannot be assessed merely based on the number of relevant papers cited or the presentation of content. Subtle nuances, such as comparing and weighing the pros and cons of different methods proposed in the literature through the use of statistical tools and analysis, are often distinguishing elements of a good survey. The human evaluations proposed in the paper do not appear to account for these qualities. The evaluation metric requires a closer examination.\n\nThe experimental analysis is weak. Conclusions are drawn too early without much deliberation and analysis. For instance the authors do not provide a deeper analysis on why the Auto-survey performs better/at-par than human-writing for shorter context lengths while being significantly worse at longer context lengths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n**W1: \"The paper lacks a significant novelty component. The concepts of decomposing tasks into smaller subtasks for LLMs, using Retrieval-Augmented Generation (RAG), iteratively refining generated content with LLMs, and employing multiple LLMs as evaluators are well-established in the literature. xxxx.\"**\\\n\n\n**A1:**\nThank you for your feedback. We believe that AutoSurvey is a novel and significant application of AI, particularly relevant to NeurIPS 2024's emphasis on impactful AI applications (https://nips.cc/Conferences/2024/CallForPapers). While the methodology may appear straightforward, it effectively addresses the complexities of automatic survey generation, including context window limitations, real-time knowledge updates, and rigorous evaluation. These contributions are crucial for advancing AI's practical utility in academic research, aligning well with the conference's focus on innovative and practical AI solutions.\n\n\n**W2: \"The evaluation criteria seem to overly rely on the ability of ML models (NLI and LLMs) to judge the quality of the survey. A well-written human survey, for instance, cannot be assessed merely based on the number of relevant papers cited or the presentation of content xxxxx.\"**\\\n\n\n**A2:**\nWe appreciate this insightful critique. After extensive literature review, we found no existing tools that adequately address this challenge, prompting us to develop the evaluation methods presented in our paper. This is a foundational approach, and we recognize the need for further refinement, which we outline as future work. Additionally, manual evaluation of lengthy surveys is labor-intensive and prone to bias, while ML models provide consistent and scalable assessments. We will include a discussion of these points in the revised manuscript.\n\n\n**W3: \"The experimental analysis is weak. Conclusions are drawn too early without much xxxxx\"**\\\n\n**A3:**\nFirst, I would like to point out a mistake. Autosurvey does not show a significant performance drop with increased length. instead, it is the naive RAG generation that shows a decline at longer context lengths. Such phenomena may be attributed to the streaming generation process, where each step must reference previous content, leading to the accumulation of errors. To validate this, we segmented the extracted claims into 20% intervals and calculated the citation recall for each segment. The results indicate that the recall of Naive RAG gradually decreases as the generated text length increases, while AutoSurvey maintains stable performance.\n\n| Claims | 20% | 20%~40% | 40%~60% | 60%~80%| 80%~100%|\n| -------- | -------- | -------- |-------- |-------- |-------- |\n|   Naive RAG-based LLM generation (64k)  | 76.79     | 73.17     |71.52     |64.08     |49.85     |\n|  AutoSurvey (64k)  |82.86     | 84.89    |79.04     |82.27     |82.29  |\n\nReferences: \n    [1] Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence (ACL 2021)\n\n\n\n### Questions\n\n**Q1:** Are there any experiments that show how useful these generated academic surveys are to people working in the field?\n**A1:**\nWe appreciate this important question. In fact, Autosurvey has recently been available for use and has already been called upon over 1,000 times. According to proactive feedback from users, the generated surveys have indeed been helpful in their work, particularly in saving time and providing a comprehensive overview of the literature. We will conduct a user study, and include the detailed results of the feedback in the revised version of the paper.\n\n**Q2:** Which aspect of the generated survey is the most helpful for humans?\\\n**A2:**\nAs mentioned in response to Q1, the most helpful aspects of the generated surveys include the comprehensive coverage of relevant papers and the structured organization of content. These features facilitate quick understanding and navigation of the topic. The combination of regularly updated paper databases and RAG (Retrieval-Augmented Generation) technology effectively ensures the real-time knowledge of the generated survey.\n\n**Q3:** How well does this technique handle multimodal information (i.e., figures/graphs) or information from structured sources such as tables?\\\n**A3:**\n\nOur current implementation primarily uses text-based large language models to write surveys. However, it is straightforward to replace the base model with a multimodal GPT, enabling the system to understand and incorporate figures, graphs, and tables.\n\n### Limitation\n\n**L1:** The technique appears to be more effective at summarizing works in the literature for a survey, rather than performing the detailed analysis and comparison of works that characterize a high-quality survey.\\\n**A1:**\n\nWe acknowledge this limitation and are actively working on enhancing the analytical capabilities of AutoSurvey. Incorporating more sophisticated analysis techniques is a priority for our future work. \nIn practice, for novices who are just getting acquainted with a particular field, a comprehensive and \"summary-like\" survey of the work within that domain can also be considered a high-quality and helpful survey.\n\n**L2:** The technique seems heavily biased by the retrieval stage.\\\n**A2:**\nThe references retrieved indeed matters. To enhance the quality of the references, the autosurvey generates descriptions for each subsection when drafting the outline, which helps in improving the quality of the references. We additionally compared the naive approach with query rewriting methods (see response to W3 from reviewer LdXj or directly see the Author Rebuttal) and found that, without content planning, rewriting the query did not lead to a significant performance improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ECOVuXE41", "reviewer": "Reviewer_EgXr", "summary": "This paper presents a methodology for generating automatically surveys called AutoSurvey.  AutoSurvey leverages the power of Large Language Models (LLM) and a Retrieval Augmented Generation (RAG) approach using as external resource a database of publications. Based on those publications, an outline is generated which is used as a guide for generating the sections of the paper. After the section generation the generated survey is refined. This process is repeated for several iterations.  AutoSurvey is compared with two other baselines. The first baseline involves human-written surveys while the second one is a RAG-based LLM. An evaluation technique is also defined called Multi-LLM-as-Judge which is a combination of various LLMs accessing the generated responses. The surveys are evaluated based on speed, citations and content quality. Experiments have been conducted for comparing AutoSurvey with the baselines and the results are presented in the paper. Moreover, Multi-LLM-as-Judge evaluation results are also compared with evaluation from human experts.", "questions": "•\tIn Figure 1 you mention that it requires 3 minutes for generating a survey, which according to your speed calculation is 20 while in Table 2 AutoSurvey has speed more than 73. How do you explain that?\n\n•\tIn Figure 1 you mention a cost of $1.2 for AutoSurvey, does this mean that one has to pay this to use AutoSurvey? \n\n•\tThe publications are basically in the computer science domain and more specifically related to LLMs. Have you tried to use publications for other domains?\n\n•\tIt is not clear what do you mean in lines 257-258 “ …(4) can refer to 20 papers (30k tokens in total) retrieved using the options provided (Upper-bound, directly retrieving the answers).” \n\n•\tIn the ablation study, which parts of AutoSurvey does the reflection study include?", "limitations": "Publications are limited to computer science domain and more specifically about Large Language Models.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "•\tThe paper is well-written and easy to follow. \n\n•\tThe presented results support the claims of the paper.\n\n•\tCode is provided.", "weaknesses": "-\tDatabase used for the retrieval is not provided. \n-\tThe collection of publications seems specific to the field of computer science and related to Large Language Models. \n-\tBased on the ablation study, the retrieval technique has more impact on AutoSurvey. It seems that the reflection part does not really influence the results.\n-\tDetails about the naïve RAG-based LLM used as baseline is limited. \n-\tMore baseline techniques could have been used for comparison to better support the paper.\n-\tLine 66: Muti-LLM-as-judge -> Multi-LLM-as-judge", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n**W1:** Database used for the retrieval is not provided.\\\n**A1:** \n\nWe appreciate the reviewer's observation. Due to the large size of the database, which amounts to 17GB even after text extraction from pdf, we are unable to provide it as supplementary material on OpenReview. However, we will make it available on GitHub along with the code.\n\n**W2:** The collection of publications seems specific to the field of computer science and related to Large Language Models.\\\n**A2:** \n\nWhile our initial implementation focused on the computer science domain and LLMs due to their rapid evolution and extensive research, the methodology of AutoSurvey is domain-agnostic. Our method only integrates domain knowledge through RAG without involving any modifications to the model itself. Therefore, it is not limited to a specific domain.\nWe are currently extending our approach to other fields by incorporating diverse databases relevant to different domains. This will be highlighted in the revised paper.\n\n**W3:** Based on the ablation study, the retrieval technique has more impact on AutoSurvey. It seems that the reflection part does not really influence the results.\\\n**A3:** \n\nThe performance of citation quality in surveys is influenced both by retrieval and reflection. Since autosurvey first uses a large number of references to write outlines, which assist in retrieval and writing, it already demonstrates good citation quality in the subsection drafting stage. The reflection module primarily helps the model to correct factual errors, so its improvement may not be very significant. In fact, as referenced in the response to the weakness 3 from Reviewer LdXj (or directly see the Author Rebuttal above), the citation quality of the naive RAG improves after adding the reflection module.\n\n\n**W4:** Details about the naïve RAG-based LLM used as baseline is limited.\\\n**A4:** \n\nThe model first utilizes the survey topic as query to retrieve relevant references. Each time, based on the references and the content already written, the model generates the subsequent section or subsection until the total word count meets the requirement. The specific prompts can be referred to in the appendix.\n\n**W5:** More baseline techniques could have been used for comparison to better support the paper.\\\n**A5:** \n\nOn the basis of the naive RAG, we have also added two additional baselines for comparison (navie rag + reflection and navie rag + query rewriting). Due to context limitations of openreview, the experimental results can be found in our response to the Weakness 3 from Reviewer LdXj.\n\n**W6:** Line 66: Muti-LLM-as-judge -> Multi-LLM-as-judge\\\n**A6:** \n\nWe apologize for the typographical error and will correct \"Muti-LLM-as-judge\" to \"Multi-LLM-as-judge\" in the revised manuscript.\n\n\n### Questions\n**Q1:** In Figure 1 you mention that it requires 3 minutes for generating a survey, which according to your speed calculation is 20 while in Table 2 AutoSurvey has speed more than 73. How do you explain that?\\\n**A1:** \n\nIn the experimental section, we mentioned, **\"For naive RAG-based LLM generation and Autosurvey, we count all the time of API calls.\"** However, Figure 1 shows the total time taken for the entire generation process (including the retrieval stage). Actually there many factors (such as vector database framework, whethere GPU is used, and the embedding model) can significantly impact the retrieval time. Note that the drafting of each subsection involves the retrieval stage. The time spent on retrieval for both naive RAG and AutoSurvey is consistent. Therefore, we only record the time taken for API calls.\n\n\n**Q2:** In Figure 1 you mention a cost of $1.2 for AutoSurvey, does this mean that one has to pay this to use AutoSurvey?\\\n**A2:** \n\nThe $1.2 cost mentioned in Figure 1 reflects the computational expense incurred when using cloud-based LLM services to generate a survey. This cost can vary depending on the specific LLM service and its pricing model. Given that the price of the model varies, we will report the total number of tokens consumed in the revised paper.\n\n**Q3:** The publications are basically in the computer science domain and more specifically related to LLMs. Have you tried to use publications for other domains?\\\n**A3:** \n\nWhile our initial implementation focused on the computer science domain and LLMs due to their rapid evolution and extensive research, the methodology of AutoSurvey is domain-agnostic. Our method only integrates domain knowledge through RAG without involving any modifications to the model itself. Therefore, it is not limited to a specific domain.\nWe are currently extending our approach to other fields by incorporating diverse databases relevant to different domains. This will be highlighted in the revised paper.\n\n**Q4:** It is not clear what you mean in lines 257-258 “…(4) can refer to 20 papers (30k tokens in total) retrieved using the options provided (Upper-bound, directly retrieving the answers).”\\\n**A4:** \n\n\nThe experiment in Table 5 aims to assess whether Autosurvey can provide topic-relevant knowledge. The assessment uses multiple-choice questions where the model needs to select the correct option based on provided references and a question. The option could be the title of a paper or the method used in the paper. The upper bound is established by directly using the options as a query to retrieve references, ensuring the most relevant content is retrieved and the performance. can be viewed as the upper-bound.\n\n**Q5:** In the ablation study, which parts of AutoSurvey does the reflection study include?\\\n**A5:** \n\nIncluded in stage 3: Integration & Refinement.\n\n### Limitations\n\n**L1:** Publications are limited to computer science domain and more specifically about Large Language Models.\\\n**A1:** \n\nSee response to Q3", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fq55w4HQeh", "reviewer": "Reviewer_VjJG", "summary": "This paper introduces a framework AutoSurvey that uses LLMs to automatically write scientific literature surveys. The process contains mainly four steps with retrieval, content generation and evaluation. The authors compare the framework with human writing and naive rag based LLM generation, in terms of speed, citation quality (recall and precision of citations) and content quality (5 scale scoring by different LLMs as judges). Empirical results show the performance gain of the proposed method. The authors also did additional ablation studies on the robustness of the framework and the variations between different LLMs als base models for the framework. Those ablation studies show that the framework could be sensitive to the components of the framework.", "questions": "1. In Table 2, you also include the speed of human writing. Where does the result come from?\n2. As far as I understand, the framework is conducted by a single LLM, or is it possible to have multiple LLMs involved in the process?", "limitations": "Limitations are already presented in sec5.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Good paper writing. The scopes are made clear and very easy to follow and reader-friendly.\n2. Solid experiments. Additional ablation studies also provide valuable insights to the framework.\n3. Very interesting usecase to employ LLMs to write literature surveys. The novelty should be acknowledged.", "weaknesses": "1. As far as I understand, the authors only experiment with the Claude-3-Haiku model in the framework for paper writing. Although the authors employ different models in the evaluation process, the main findings could be biased based on only one model.  \n\n2. In evaluation of the citation quality, the metric used an NLI model but didn't report the details of the performance of the NLI model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n**W1:** As far as I understand, the authors only experiment with the Claude-3-Haiku model in the framework for paper writing. Although the authors employ different models in the evaluation process, the main findings could be biased based on only one model. \\\n**A1:** \n\nWe appreciate the reviewer's concern regarding the potential bias from using a single model for paper writing. While Claude-3-Haiku is the primary model employed in our experiments, we also conduct supplementary tests with other models, including GPT-4 and Gemini-1.5-Pro. Additionally, we used different LLMs as judges to mitigate bias, and the results of human evaluations also confirm the consistency with human preferences. Overall, we think the entire framework demonstrates generalization performance across different models.\n\n\n**W2:** In evaluation of the citation quality, the metric used an NLI model but didn't report the details of the performance of the NLI model.\\\n**A2:** \n\nThe metric is first introduced by the paper \"Enabling Large Language Models to Generate Text with Citations\". The author uses a 7B model for NLI and demonstrates that its evaluationis consistent with human preferences. In contrast, we directly employed a closed-source large model for NLI inference. The reasoning and inference capabilities of the closed-source model are significantly superior to those of the 7B-scale model.\n\n### Questions\n\n**Q1:** In Table 2, you also include the speed of human writing. Where does the result come from?\\\n**A1:** \n\nThe speed of human writing included in Table 2 is derived from a mathematical model that estimates the time required for humans to write a document based on various parameters, such as document length, number of experts, and writing speed. These estimates are supplemented by empirical data from surveys and interviews with experienced researchers who provided insights into their writing processes and time requirements. Overall this is an ideal model, while the actual writing speed is usually much slower.\n\n**Q2:** As far as I understand, the framework is conducted by a single LLM, or is it possible to have multiple LLMs involved in the process?\\\n**A2:** \n\nThe framework is designed to be flexible and can indeed involve multiple LLMs. While our initial implementation primarily utilized a single LLM for the writing process, the architecture of AutoSurvey allows for the integration of multiple LLMs to handle different tasks, such as retrieval, outline generation, and section drafting. Actually, in our practice, we found that using Claude-4 Haiku to write the outline and GPT-4 to draft the subsections can effectively reduce costs. We will elaborate on this capability in the revised paper, highlighting the potential benefits of using multiple LLMs in the survey generation process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear All Reviewers,

We sincerely appreciate all of your thoughtful comments and valuable suggestions on our manuscript. Your expertise and detailed review have been crucial in improving the quality and clarity of our work. We are grateful for your recognition of the strengths in our research, including its novelty, clarity, and solid analysis. In response to your feedback, we have thoroughly examined and addressed each concern you raised.

Due to the length constraints of the rebuttal, we present the additional experiments below and in the attached PDF.

we have supplemented the original naive RAG-based experiments by including refinement and query rewriting stages. In Naive RAG + Refinement, the LLM is required to enhance the continuity of the written content with previous sections and check for factual errors based on the references retrieved. In Naive RAG + Query Rewriting, references are first retrieved using the topic, after which the LLM rewrites the query based on the references to assist in writing subsequent content.

| Survey Length (#tokens) | Methods | Recall| Precision | Coverage | Structure | Relevance |
|-------------------------|---------|-------|-----------|----------|-----------|-----------|
|    8k   | Naive RAG-based LLM generation + Refinement | 82.25 | 76.84 | 4.46 | 4.02 | 4.86 | 
| |    Naive RAG-based LLM generation + Query Rewriting | 80.99 | 71.83 | 4.84 | 4.05 | 4.88 | 
|      16k |Naive RAG-based LLM generation + Refinement | 79.67 | 73.73 | 4.57 | 4.28 | 4.83 |
| |    Naive RAG-based LLM generation + Query Rewriting | 77.73 | 66.29 | 4.70 | 3.67 | 4.79 |
|     32k | Naive RAG-based LLM generation + Refinement | 80.50 | 72.18 | 4.82 | 4.08 | 4.49 |
|  |   Naive RAG-based LLM generation + Query Rewriting | 76.56 | 65.36 | 4.61 | 3.96 | 4.88 | 
|  64k|     Naive RAG-based LLM generation + Refinement | 73.12 | 68.36 | 4.66 | 4.06 | 4.76 | 
|  |   Naive RAG-based LLM generation + Query Rewriting | 69.77 | 62.21 | 4.45 | 3.88 | 4.69 |

After adding the refinement stage, both citation quality and structure improved. The effect of query rewriting is not obviously enhanced, possibly due to the model's lack of a clear planning of the content to be written, leading to lower quality of rewritten queries. Overall, these baselines still lag behind AutoSurvey, especially when surveys get longer. This gap may be attributed to the streaming generation process, where each step must reference previous content, leading to the accumulation of errors. To validate this, we segmented the extracted claims into 20% intervals and calculated the citation recall for each segment. The results indicate that the recall of Naive RAG gradually decreases as the generated text length increases, while AutoSurvey maintains stable performance.

| Claims | 20% | 20%~40% | 40%~60% | 60%~80%| 80%~100%|
| -------- | -------- | -------- |-------- |-------- |-------- |
|   Naive RAG-based LLM generation (64k)  | 76.79     | 73.17     |71.52     |64.08     |49.85     |
|  AutoSurvey (64k)  |82.86     | 84.89    |79.04     |82.27     |82.29  |

References:
[1] SELF-RAG: Learning to Retrieve, Generate and Critique through self-reflection (ICLR 2024)
    [2] Query Rewriting in Retrieval-Augmented Large Language Models (EMNLP 2023)
    [3] Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence (ACL 2021)

Meta-review: The task is exciting -- AI writing technical surveys automatically. All reviewers lean positive. Seems like a clear accept.

Authors should incorporate all the discussion into the revised version. Also, they promised a user study to evaluate the meaningfulness of generated surveys to real users -- that will make the paper further strong.
Decision: Accept (poster)