Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
Authors: Chenglei Si
Diyi Yang
Tatsunori Hashimoto
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
Idea Generation
Human Study
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advancements in large language models (LLMs) have sparked optimism about their potential to accelerate scientific discovery, with a growing number of works proposing research agents that autonomously generate and validate new ideas. Despite this, no evaluations have shown that LLM systems can take the very first step of producing novel, expert-level ideas, let alone perform the entire research process. We address this by establishing an experimental design that evaluates research idea generation while controlling for confounders and performs the first comparison between expert NLP researchers and an LLM ideation agent. By recruiting over 100 NLP researchers to write novel ideas and blind reviews of both LLM and human ideas, we obtain the first statistically significant conclusion on current LLM capabilities for research ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility. Studying our agent baselines closely, we identify open problems in building and evaluating research agents, including failures of LLM self-evaluation and their lack of diversity in generation.

Published as a conference paper at ICLR 2025
CAN LLMs GENERATE NOVEL RESEARCH IDEAS? A LARGE-SCALE HUMAN STUDY WITH 100+ NLP RESEARCHERS
Chenglei Si, Diyi Yang, Tatsunori Hashimoto Stanford University {clsi, diyly, thashim/ @sstafford.edl
ABSTRACT
Recent advancements in large language models (LLMs) have sparked optimism about their potential to accelerate scientific discovery, with a growing number of works proposing research agents that autonomously generate and validate new ideas. Despite this. no evaluations have shown that LLM systems can take the very first step of producing novel, expert-level ideas, let alone perform the entire research process. We address this by establishing an experimental design that evaluates research idea generation while controlling for confounders and performs the first comparison between expert NLP researchers and an LLM ideation agent. By recruiting over 100 NLP researchers to write novel ideas and blind reviews of both LLM and human ideas, obtain the first statistically significant conclusion on current LLM capabilities for research ideation: we find LLM- --geeerrttedd ideas are judged more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility. Studying our agent baselines closely, we identify open problems in building and evaluating research agents, including failures of LLM self-evaluation and their lack of diversity in generation.
INTRODUCTION
The rapid improvement of LLMs, especially in capabilities like knowledge and reasoning, has enabled many new applications in scientific tasks, such as solving challenging mathematical problems (Trinh assisting scientists in writing proofs (Collins et al. 2024), retrieving related works (Ajith Press et al. 2024), and generating code to solve analytical or computational tasks (Huang
al.
2024),
et al.
2024
al.
2024 Tian al. 2024). While these are useful applications that can potentially increase the
productivity of researchers, it remains an open question whether LLMS can take on the more creative and challenging parts of the research process.
We focus on this problem of measuring the research ideation capabilities of LLMs and ask: are current LLMs capable of generating novel ideas that are comparable to expert humans? Although ideation is only one part of the research process, this is key question to answer, as it is the very first step to the scientific research process and serves as a litmus test for the possibility of autonomous research agents that create their own idcas. Evaluating expert-level capabilities of LLM systems is challenging (Bakhtin et al., 2022; Collins et al., 2024),, and research idcation takes this to an extreme. Qualified expert researchers difficult to recruit scale, evaluation criteria can be highly subjective, and it is difficult even for experts to judge the quality of research ideas (Beygelzimer et al.) 2021).
We address these challenges directly, recognizing that for important, high-stakes tasks like research ideation, there is no substitute for a large-scale expert evaluation. We design a carefully controlled comparison of human and L.M ideas that overcomes sample size and baseline problems present in earlier small-scale evaluation studies. Our study recruited a large pool of over 100 highly qualified NLP researchers to produce human baseline ideas and perform blind reviews of human and LLM ideas. To reduce the possibility that confounding variables affect our outcome measures, we enforce strict controls that standardize the styles of human and LLM ideas and match their topic distribution.
We compare our human expert baseline with simple and effective LLM agent that incorporates retrieval augmentation and adopts recent ideas in inference- time scaling, such as overgenerating and
We release our agent implementation and all human review scores at: https: h/tp:s////i thub.
Novi Scl Resea rcher The last two authors advised this project equally.

---

Published as a conference paper at ICLR 2025
Idea Generation
Blind Experts (N-79)
NLP Topics
Human Experts
Condition Human Ideas (N-49)
Novelty Score:
Bias Coding Safety Muttiiggall
Condition Al Ideas (N=49)
Novelty Score: 5.64
Factuality
Al Agent
Math
Uncortainty
Condition Ideas Humma (N=49))
Novelty
Figure 1: Overview: recruit 79 expert researchers perform blind review of 49 ideas from each of the three conditions: expert-written ideas, Al-generated idcas, and Al-generated ideas reranked by human expert. We standardize the format and style of ideas from all conditions before the blind review. We find Al ideas are judged as significantly more novel than human ideas (p < 0.05).
Novelty
Excitement
Feasibility
Effectiveness
Overall
Human
Figure 2: Comparison of the three experiment conditions across all review metrics. Red asterisks indicate that the condition is statistically better than the Human baseline with two-tailed Welch's t-tests and Bonferroni correction. All scores are on I to 10 scale. More detailed results are in Section
reranking LM outputs. These measures allow make statistically rigorous comparisons between human experts and state-of-the-arr LLMs (Figure I).
Our evaluation-centriic approach complements many recent methods-centric works that attempt to instantiate research agents. These works rely on fast and lower-cost evaluation surrogates either by decreasing the number of expert reviewers (Back al. 2024 Li al. 2024 Wang al. 2024 Yang et al. 2024). constraining the length and detailedness of the ideas (Wang et al. 2024 Yang et al. 2024 or relying on L-M--as-a-judge (Lu et al.) 2024). They do not perform the large-scale human comparison studies that are needed to answer the motivating question of our work. Our work takes
the opposite approach, performing year-long and high-cost evaluation that provides human expert baselines and a standardized evaluation protocol to serve foundation for future follow-up studies and methods work.
Through nearly 300 reviews across all our conditions, we find that Al-generated ideas are judged as more novel than human expert ideas (p 0.05), which holds robustly under multiple hypothesis correction and across different statistical tests (Figure 22. Apart from evaluating the ideas, also analyze the L.M agent, showing limitations and open problems despite excitement about inference- time scaling of LLMs, find that they lack idea diversity when scale up idea generation, and they cannot currently serve as reliable evaluators.
2 PROBLEM SETUP
The central experiment of our work is comparison of human- and LLM -enerrred ideas. While this goal is simple, there no existing consensus on how to formulate the task of research ideation and evaluation, and begin by defining the key aspects of experiment design.
2

---

Published as a conference paper at ICLR 2025
We think of research idea evaluation as consisting of three separate components: 1). the idea itself, generated in response to our instructions, 2). the writeup which communicates the idea, and 3). the evaluation of the writeup by experts. We outline our experiment design in each of these three parts with particular focus on potential confounders, such as the area of research, the format of a research idea, and the evaluation process.
Ideation Scope and Instructions Any experiment on ideation must carefully balance the realistic- ness and interestingness of a research idea with the practical realities of eliciting ideas from a large population. In case, these tradeoff's are more pronounced, we have designed our ideation experiments so that the resulting ideas can be executed by experts in a follow-up set of experiments.
These constraints have led us to study prompting- based NLP research as a testbed for our study, Prompting research has been popular in recent years of NLP and Al research [Schulhoff et al. (2024). This class of projects strikes reasonable trade-off among our constraints. The most impactful prompting projects like chain-of-thought have had a major influence on LLM performance (Wei et al. 2022.. and prompting projects are executable with minimal computing hardware.
We further structure our ideation process to avoid selection-biatt bass- confounders in ideation. If we simply ask LLMs and humans to produce ideas on 'prompting topics', we may find that LLMs and humans differ in the types of research ideas they produce (for example, LLMs may naturally suggest more projects on safer topies, which might be judged as less exciting by humans). This would lead us to simply measure misalignment in research topic preference between LLMs and humans, which is not the goal of our study. To address this possibility, we define a set of seven specific research topics extracted from the Call For Papers page of recent NLP conferences such as COLM. Specifically, our topics include: Bias, Coding. Safety, Multilinguullty, Factuality, Math, and Uncertainty (see Appendix A.3 for complete description of these topies).
Each human and LLM participant of the ideation experiment receives the same set of natural language instructions including the same topic description, idea template, and demonstration example ensure a fair comparison. For human participants, we additionally allow them to select preferred topic from the list, and for each selected topie, we generate corresponding LLM idea. This exactly matches the idea topic distribution between the LLM and human participants, while ensuring that human experts are able select topies according to their expertise.
Idea Writeup An idea can only be evaluated if it is written up be communicated, but this writing process introduces many additional potential confounders. Human researchers may write in ways that subtly signal quality research, such including more examples and implementation details. The format of the writeup functions as a way to scaffold what contents should be included and the level of detailedness. Ideally, we want both human and LLM participants to provide all the necessary implementation details for their generated ideas.
We take inspiration from guidelines used in grant submissions and introduce a template to specify the structure and detailedness of idea proposals. Specifically, we construct template that includes fields for the title, problem statement, motivation, proposed method, step-by-step experiment plan, test case examples, and the fallback plan. Both the LLM agent and the human idea writers are instructed to follow this template and our provided demonstration examples to produce a project proposal as the output (see Appendix A.4 for the full template and Appendix A.5 for the demo example).
Even with these templates, there may be subtle writing style cues that affect the outcome measure. For example, humans may tend to write in a more engaging and informal tone. To reduce this possibility further, we developed a style normalization module that uses an LLM to convert all ideas into the same writing and formatting style without changing the original content. Our small-scale human study shows that such a normalization approach leads to a 50% accuracy for expert human judges who are asked to distinguish AI ideas from human ideas. Finally, the use of an L.M style anonymizer has the possibility of substantively changing the content of the ideas. To rule this out, the first author of this paper manually verified each human idea proposal to ensure all contents of the original ideas were preserved. We present the full prompt used in Appendix A.6
Review and Evaluation Reviewing research ideas is notoriously subjective, so we want to design a review form that defines all review criteria clearly to standardize and anchor the evaluations as much
3

---

Published as a conference paper at ICLR 2025
possible. At the same time, want review criteria and measured variables to capture all the desiderata of high-quality research ideas.
We follow best practices from AI conference reviewing (e.g., ICLR and ACL) when designing the review form, where we define four breakdown metrics including novelty, excitement, feasibility, and expected effectiveness, apart from the overall score. For each metric, we ask for numerical score on a 1-10 scale along with a free-text rationale. We provide clear definitions and grounding for each numerical scale calibrate all reviewers' standards (see Appendix A.7 for the full review form). In the next two sections, we instantiate how our LLM agent generates ideas and how our expert participants generate and review the ideas.
IDEA GENERATION AGENT
We build a simple but effective LLM ideation agent to compare with the human expert baseline. Rather than focusing on innovating the agent itself, we adhere to minimalist design principle, aiming to understand the current capabilities of LL in idea generation. Our research ideation agent has three essential components: paper retrieval, idea generation, and idea ranking, which we will describe in detail below.
3.1 PAPER RETRIEVAL FOR RAG
To ground idea generation, the agent needs to retrieve papers related to the given research topic, so that it will be aware of related works when generating new ideas. To do so, leverage retrieval-augmenned generation (RAG), which has demonstrated effectiveness on many knowledge-intensivee tasks (eewis et al., 2020 Shi et al. 2024). Concretely, given a re- search topic (e.g., "novel prompting methods that can improve factuality and reduce halluci- nation of large language models"), we prompt an LLM to generate sequence of function calls to the Semantic Scholar API. We use claude- -oieee----soont--220 bone model for our agent but the pipeline should generalize to other LLMs as well. The paper retrieval action space includes: (Keywordquery (keywords) P PaperQuery (paperId) GetRe ferences (paper Id) ). Each action generation is grounded on the previous actions and executed results. We keep the top k 20 papers from each executed function call and stop the action generation when a max of N = 120 papers have been retrieved. We then use the LLM to score and rerank all retrieved papers based on three criteria: 1) the paper should be directly relevant to the specified topic; 2) the paper should be an empirical paper involving computational experiments; 3) the paper is interesting and can inspire new projects. The LLM is prompted score each retrieved paper on a scale of I to 10 based on these criteria and we use the top-ranked papers for the next step of idea generation.
as the back-
3.2 IDEA GENERATION
Our key insight for idea generation is generate many candidate ideas possible. Our intuition that only a small fraction of all generated ideas might be high-quality, and we should be willing to expend inference-time compute to generate more candidates so that we can later use reranker to discover the "diamond in the rough". This aligns with existing results showing that scaling inference compute with repeated sampling can boost LLM performance on various coding and reasoning tasks (Li et al. 2022; Brown et al. 2024). Specifically, we prompt the LLM to generate 4000 seed ideas on each research topic. The idea generation prompt includes the demonstration examples and the retrieved papers. We craft & = 6 demonstration examples by manually summarizing exemplar papers (Yasunaga et al. 2024 Madaan et al. 2023] Weller et al., 2023) Weston Sukhbaatar 2023), Zheng al. 2024:] Dhuliawala et al. 2023) into our desired idea format. For retrieval augmentation. we randomly select k = 10 papers from the top-ranked retrieved papers and concatenate their titles and abstracts prepend to the idea generation prompt. We also append the titles of all previously generated ideas to the prompt to explicitly ask the LLM to avoid repetitions.
To remove duplicated ideas from this large pool of candidate ideas, first perform round of dedupli- cation by encoding all seed ideas with all - LM-L6-V from Sentence-Transfomen Transformers (Reimers) & Gurevych 2020) and then computing pairwise cosine similarities. We set similarity threshold of
4

---

Published as a conference paper at ICLR 2025
0.8 for the idea deduplication based manual inspection. This leaves about 5% non non-duplicated ideas out of all the generated seed ideas. We expand more on this duplication issue later in Section 7.1
3.3 IDEA RANKING
The next step is for our ideation agent rank all the remaining ideas so that we can find the best ones among them. To build such an automatic idea ranker, we use public review data as a proxy. Specifically, we scraped 1200 ICLR 2024 submissions related to LLMs (with keyword filtering) along with their review scores and acceptance decisions. We explored multiple ways of predicting the scores and decisions of these submissions and found that LLMs are poorly calibrated when asked directly to predict the final scores or decisions, but can achieve non-trivial accuracy when asked to judge which paper is better in pairwise comparisons.
We converted the ICLR submissions into our stan- dard project proposal format and randomly paired up accepted and rejected papers and asked LLMs to predict which one is accepted. On this task, 1.aaude3.55Soont
N
Top-10
Bottom- 10
Gap
6.28
5.72
0.56
2
6.14
5.24
0.90
accuracy of 71.4% with zero-shot prompting. For compari- son, GPT-4o achieves 61.1% and Claude-3-Opus achieves 63.5%, and we do not observe significant gains from additional prompting techniques like few-shot or chain-of-thought prompting. We therefore choose the Claude-3, 5-Sonnet zero-shot ranker.
achieves
3
5.83
4.86
0.97
4
5.94
4.99
0.95
6.42
4.69
1.73
6
6,11
4.81
1.30
Table 1: Average ICLR review scores of top- and bottom- 10 papers ranked by our L.MM ranker, with different rounds (N) of pairwise comparisons.
In order to obtain reliable scores for all project proposals based on pairwise comparisons, we adopt Swiss system tournament where all project proposals are paired with those whose accumulated scores are similar, and if the proposals are judged be better, they gain an additional point. We repeat this for N rounds so the total score of each project proposal will be within the [O, N] range. As a sanity check, we use the Claude 3. 5 Sonnet ranker to rank the 1.2K ICLR LLM-related submissions and compare the average review scores of the top 10 ranked papers and the bottom 10 ranked papers in Table [ We see a clear separation between the top and bottom ranked papers, indicating the effectiivneess of the LLM ranker. We choose N 5 for all our experiments since it gives the best ranking result on this validation set. The top-ranked project proposals from the agent will be directly used for the AT Ideas condition of the human study.
Since our AI ranker is still far from perfect, we also introduce another experiment condition where the first author of this paper manually reranked the generated project proposals instead of relying on the LLM ranker, and we call this the AI Ideas + Human Rerank condition. 17 out of the 49 ideas in the AI Ideas LLM agent (Table 1 in Appendix Appendix[A.II,, while the other 32 are different, indicating the discrepancy between the L.LM ranker and the human expert reranking.
Human Rerank condition overlap with the AI Ideas ranked by the
EXPERT IDEA WRITING AND REVIEWING
In this section, shift focus the human branch of idea generation comparison. We present the details of our human study, including information about the recruited experts, the human idea generation task, and the subsequent review process.
4.1 EXPERT RECRUITMENT
We recruit our expert participants (including for idea writing and reviewing) by sending sign-up forms to several channels, including: 1) the OpenNLP Slack channel with 1426 NLP researchers from 71 institutions; 2) Twitter (X); 3) Slack channels of various NLP groups by direct communication with the group members; and 4) official chat app of the NAACL 2024 conference. Our study including all recruitment materials has been approved by IRB.
We provide randomly sampled idea pairs and their similarities in Appendix A.10 We also provide additional implementation details about the ideation agent in Appendix A.8
5

---

Published as a conference paper at ICLR 2025
Idea Writing Participants (N=49)
Idea Reviewing Participants (N=79)
Metric
Mean
Median
Min
Max
SD
Mean
Median
Min
Max
SD
papers
12
10
2
52
9
1.5
13
2
52
10
citations
477
125
4553
861
635
327
7276
989
h-index
4
7
i10-index
5
4
0
32
6
7
5
32
Table Research profile metrics of the idea writing and reviewing participants. Data are extracted from Google Scholar at the time of idea or review submission.
Metric
Mcan
Median
Min
Max
SD
Human Ideas
Familiarrity (1-5)
3.7
4.0
1.0
5.0
1.0
Difficulty (1-5)
3.0
3.0
1.0
5.0
0.7
Time (Hours)
5.5
5.0
2.0
15.0
2.7
Length (Words)
901.7
876.0
444.0
1704.0
253.5
AI Ideas
Length (Words)
1186.3
1158.0
706.0
1745.0
233.7
AI Human Rerank Ideas
Length (Words)
1174.0
1166.0
706.0
1708.0
211.0
Table Statistics of the 49 ideas from each condition.
We performed screening on the participants based on their provided Google Scholar profiles and recruited N = 49 experts for writing ideas, and = 79 experts for reviewing ideas. Each idea writer asked write one idea within 10 days and we compensate $300 for each, with a $1000 bonus for the top 5 ideas as scored by the expert reviewers. Each idea reviewer is assigned 2 to 7 ideas to review and we collected N 298 unique reviews total. They are given one week to finish the reviews and we compensated $25 for each review written by the idea reviewers.
4.2 EXPERT QUALIFICATIONS
Our pool of participants is highly qualified and diverse. The 49 idea writers come from 26 different institutions and 73% of them current PhD students. The reviewers come from 32 institutions and 87% of them are PhD students and Postdocs. We provide the detailed statistics in Appendix A.13 We use their Google Scholar profiles to extract several proxy metrics, including the number of papers, citations, h-index, and i10-index at the time of their submission. Table shows that our idea writers have an average 12 papers and 477 citations, while every reviewer has published at least two papers and has average citation of 635 and h-index of 7. Moreover, based their survey responses, 72 out of the 79 reviewers have previously reviewed for conferences. These statistics indicate that our participants are highly qualified and have substantial research experience.
4.3 IDEA WRITING
We report statistics of our idea writers' ideas to measure their quality. As shown in Table 3 idea writers indicate moderately high familiarity with their selected topic (3.7 on indicate the task moderately difficult (3 on to 5 scale). They spent average of 5.5 hours the task and their ideas are 902 words long on average. These indicate that participants are putting substantial effort into this task. We show the distribution of their selected topics in Appendix A.3
scale), and
4.4 IDEA REVIEWING
Review Assignment We let all reviewer participants select their top two preferred topics as well as their preferred reviewing load (from 2 to 7). We then randomly assign them to ideas within their selected topics and all ideas are anonymized. In the assignment, we balance the number of ideas from each condition for each reviewer and ensure that each reviewer gets at least one human idea and one AI idea. Every idea is reviewed by to 4 different reviewers. We also avoid assigning ideas written by authors from the same institution to avoid any potential contamination. Each reviewer wrote an average of 3.8 reviews from 2 or 3 conditions, across 1 to topies (full statistics in Appendix 1.14

---

Published as a conference paper at ICLR 2025
Metric
Mean
Median
Min
Max
ST
Ours
Familiarity (1-5)
3.7
3.0
1.0
5.0
0.9
Confidence (1-5)
3.7
4.0
1.0
5.0
0.7
Time (Minutes)
31.7
30.0
5.0
120.0
16.8
Length (Word)
231.9
208.0
41.0
771.0
112.1
ICLR 2024
Confidence (1-5)
3.7
4.0
1.0
5.0
0.8
Length (Word)
421.5
360.0
14.0
2426.0
236.4
Length (Word; Strengths & Weaknesses)
247.4
207.0
2.0
2010.0
176.4
Table 4: Statistics of collected reviews, with ICLR 2024 reviews baseline (for the 1.2K submissions that mentioned the keyword "language models").
Review Quality Check Apart from ensuring reviewer qualifications, we also compute statistics to measure the quality of the reviews in Table On average, the reviewers indicated familiarity of 3.7 (out of 5) in their selected topic and confidence of 3.7 (out of 5) in their reviews. This is comparable with the 1.2K ICLR 2024 submissions related to language models, where the reviewers also have an average confidence of 3.7 out of 5. Moreover, reviewers spent an average of 32 minutes on each review, with each review being about 232 words long.
Since our review forms are different from the ICLR review forms, we compare them with the ICLR reviews where we remove the summary and question sections and only count the lengths of the strengths and weaknesses sections. This way, the ICLR reviews have average length of 247, similar to our collected reviews. As an additional measure of review quality, out of the 298 unique reviews that we have collected, 80 of them provided links existing papers in their rationales to justify why the proposed method is not novel. These results further validate the high quality our review data.
5 MAIN RESULT: AI IDEAS ARE RATED MORE NOVEL THAN EXPERT IDEAS
In this section, present our main finding. Consistently across three different statistical tests accounting for the possible confounders, we find that Al ideas have higher novelty scores than human ideas while being comparable on all other metries.
Independent Data Point. In Test 1, treat each review as an independent data point and aggregate all reviews from the same condition. We treat the Human Ideas the baseline condition and compare it with AI Ideas and AI Ideas + Human Rerank using two-tailed Welch's ' t-tests with Bonferroni correction. We show the barplot in Figure 2 and the detailed numerical results in Table 5 Both AI Ideas (u 5.64 + a 1.76) and AT Ideas + Human Rerank (μ 5.81 + = 1.66) are significantly better than Human Ideas 4.84 1.79) on the novelty score (p 0.01). In this particular test, the Al ideas in both conditions are also significantly better than human ideas on the excitement score (p < 0.05), and the AI Ideas Human Rerank condition also significantly better than Human Ideas in terms of the overall score (p < 0.05). We do not observe significant differences between Al-generated ideas and numan-written ideas on the other metrics.
Test 1: Treating Each Review
Test 2: Treating Each Idea as an Independent Data Point. Since we collect multiple reviews for each idea, one could argue that we should not treat each review as independent data point. To account for this potential confounder, we perform Test where we average the scores of each idea and treat each idea as one data point. This way, the sample size for every condition will be N = 49, namely the number ideas. We treat the Human Ideas as the baseline condition and compare it with AI Ideas and AI Ideas with Bonferroni correction. Under this test (Table [4in Appendix AA.15, we still see significant results (p 0.05) where both AT Tdeas (μ 5.62 + 1.39) and AT Tdeas Human Rerank (μ 5.78 + a = 1.07) have higher novelty scores than Human Ideas (μ = 4.86 a = 1.26).
Human Rerank using two-tailed Welch's 's t-tests
Test 3: Treating Each Reviewer as an Independent Data Point. Another possible confounder is that different reviewers might have different biases, for example, some reviewers may be more lenient
7

---

Published as a conference paper at ICLR 2025
Condition
Size
Mcan
Median
SD
SE
Min
Max
p-value
Novelty Score
Human Ideas
119
4.84
1.79
0.16
1
8
AI Iceas
109
5,64
1.76
0.17
1
10
0.00
Ideas
Rerank
109
5.81
1.66
0.16
2
10
0.00
Excitement Score
Fuman Ideas
119
4.55
1.89
0.17
8
AI Ideas
109
5.19
1.73
0.17
1
9
0.04
Ideas Human Rerank
109
5,46
1.82
0.17
9
0.00
Feasibility Score
Human Ideas
119
6.61
1.99
0.18
1
10
AI Iceas
109
6.34
1.88
0.18
2
10
1.00
Ideas Human Rerank
109
6.44
1.63
0.16
10
1.00
Expected Effectiveness Score
Human Idcas
119
5.13
1.76
0.16
1
8
AI Ideas
109
5,47
1.58
0.15
1
10
0.67
Ideas Human Rerank
5.55
1.52
0.15
9
0.29
Overall Score
Human Ideas
119
4.68
1.90
0.17
1
9
AI Ideas Ideas Human Rerank
109
4.85
1.70
0.16
1
9
1.00
109
5.34
1.79
0.17
1
9
0.04
Table 5: Scores across all conditions by treating each review
independent datapoint (Test 1). Size is the number of reviews for each condition and the p-values are computed with two- tailed Welch's t-tests with Bonferroni correction. We bold results that are statistically significant ("p < 0.05:** p < 0.01:"** < 0.001). Al ideas are judged as significantly better than human ideas in terms of novelty and excitement while being comparable on all other metrics.
than others. To account for such reviewer biases, perform Test 3 where treat each reviewer as one data point and compute their average score on each condition. Then for each reviewer, we get their mean score difference between the Ideas condition and the Human Ideas condition, as well as the difference between the AI Ideas + Human Rerank condition and the Human Ideas condition. This way, we only analyze the differences among the different conditions. That is, if the differences are significantly higher than zero under the one-sample (-test, that indicates reviewers are giving higher scores one condition compared to the other. Using this test (Table [5 in Appendix 15).),, we also see significant results (p < 0.05) that Al ideas in both the AT Ideas and AI Ideas Human Rerank conditions are rated more novel than Human Ideas. Therefore, conclude that AI ideas generated by our ideation agent are judged as more novel than human expert generated ideas, consistently across all three different statistical tests.
6 IN-DEPTH ANALYSIS OF THE HUMAN STUDY
In this section, move beyond the statistical comparisons and dive into other aspects of our collected data. Specifically, we focus on the quality of human ideas and the extent of reviewer agreement.
6.1 HUMAN EXPERTS MAY NOT BE GIVING THEIR BEST IDEAS
We first investigate whether human experts are submitting their best ideas to us. We did post- study survey to understand how idea-writing participants came up with their ideas. Out of the 49 participants, 37 of them came up with the idea the spot, while the other 12 already had the idea before the study. Furthermore, we asked the survey question: "How does this idea compare to your past research ideas (ideas that you actually worked on)? Please answer with percentile. E.g., this idea is one of my top 10% ideas." Our participants indicated that on average their submitted ideas about the top 43% of all their past ideas. This implies that our collected ideas are likely the median-level ideas from these expert researchers, which is reasonable given that most of them came up with the idea within the 10-day time constraint of the task.
We also include results of fitting linear mixed-effects models in Appendix 16 which reinforces conclusions. Additionally, plot the breakdown of all metrics by topic in Appendix A.17
8

---

Published as a conference paper at ICLR 2025
6.2 REVIEWING IDEAS IS INHERENTLY SUBJECTIVE
Finally, we acknowledge that reviewing is inherently subjective, and reviewing based on ideas rather than executed papers might be even more subjective. We investigate this using nnteewreveewer agreement. Specifically, we randomly split reviewers of each paper into half, use one half to rank the top and bottom 25% of all ideas, and then measure agreement with the held-out set of reviewers. As shown in the first block of Table reviewers have a relatively low agreement (56.1% despite the fact that we have provided detailed explanations for each metric in our review form. As a baseline comparison, the NeurIPS 2021 reviewer consistency experiment found 66. 66.0% accuracy using this reviewer agreement metric in the balanced setting (Beygelzimer et al., 2021 Lu et al. 2024). We also computed the reviewer agreement using the same metric the I.2KICLR 2024 submissions related to language models, which has balanced accuracy of 71.9%. While our reviewer agreement higher than random (50%), it is generally lower than conference reviewing, most likely due the higher subjectivity involved when evaluating ideas without seeing the actual experiment results.
Apart from the above quantitative analysis, we also provide some qualitative analysis of our collected data. We provide summary of free-text reviews in Appendix A.18, and provide four pairs of Al and human ideas along with full reviews in Appendix A.19
7 LIMITATIONS OF LLMS
Our ideation agent is motivated by two potential strengths of LLMs: their ability to scale by generating vast number of ideas far more than any human could- and the possibility of filtering these ideas to extract the best ones from the large pool. In theory, this approach could lead to high-quality ideas by leveraging inference scaling. However, we present empirical evidence that this naive assumption about scaling idea generation has significant limitations.
7.1 LLMS LACK DIVERSITY IDEA GENERATION
We adopted over-generate and rank paradigm in idea generation. This raises the question: is there an upper limit to how many new ideas LLMs generate? To answer this question, we take a closer look at 4000 generated seed ideas for each topic.
Accumulation of Nun-Duplicaaa Actosa Gerer
200
275
:25
We encode all raw ideas with all ail-MiniLM-L6-1 from TraneenceTaansoormees For each idea, we compute its cosine similarity with all previously generated ideas on the same topic. We consider an idea as a duplicate it has a similarity of above 0.8 with any of the previously gen- erated ideas. In Figure we show that as the agent keeps generating new batches of ideas, the accumulated non- duplicate ideas eventually plateau. In fact, out of the 4000 generated seed ideas, there are only 200 non-duplicate unique ideas. This sets a bottleneck on our inference-time scaling since increasing the number of generated ideas simply leads to repeating duplicate ideas.
200
Accumultted Dupicates
75
1000
1990
2003
2000 3500
10tal Number of Moeas Generated
Figure 3: The accumulated non- duplicate ideas saturate the agent keeps generating new ideas. All data points are averaged across all topics.
7.2 LLMS CANNOT EVALUATE IDEAS RELIABLY
Most prior works have adopted LLM-as-a-judge for evaluating research ideas Lu et al. (2024) motivated by the observation that LLMs can have a higher agreement with human evaluators than the inter-human agreement. However, we offer some empirical evidence that LLMs cannot evaluate ideas reliably yet.
Concretely, we use the average review score of each idea to rank the top and bottom 25% of all our collected human and AI ideas, and use this to benchmark various LLM evaluators. Specifically, we obtain the LLM predicted scores of all ideas and set the median score as the threshold to measure their accuracy on our balanced idea ranking data.

---

Published as a conference paper at ICLR 2025
compare several differ- ent LLM evaluators: 1) directly giving the review criteria and
In the second block of Table
Consistency
Random
50.0
prompting for final score (Yang et al. 2024 Li al. 2024
NcurIPS'21
66.0
Baek et al. 2024); 2) our pairwise ranker as described in Sec-
ICLR'24
71.9
tion and 3) the "Al Scientist" reviewer agent et al. 2024
Ours
56.1
All of these LLM evaluators have a lower agreement than our
GPT-40 Direct
50.0
expert reviewers' scores. Even the best LLM evaluator - our own Claude-3.5 pairwise ranker only achieves an accuracy of 53.3%, lower than our inter-reviewer consistency of 56. 1%.
GPT-4o Pairwise
45.0
Claude-3.5 Direct
51.7
Claude-3.5 Pairwise
53.3
"AI Scientist" Reviewer
43.3
Even if Al-human agreement eventually matches or exceeds human- human agreement, simply meeting this baseline does not imply that Al-as-a-reviewer is meaningful, since we may be trading variance for bias, where reviewers are more consistent but rely spurious correlations (Durmus et al. 2022). Our findings Table are consistent with these brittleness concerns, find significant drop in Al-human agreement scores under our study compared to the original studies. Finally, even though Claude-3.5 pairwise agreements may close to human agreement, many other pieces of evidence throughout the paper leads us to be cautious about the use of LLM-as-a-judge include our findings on the significant discrepancy between the agent's top-ranked ideas and the human expert's 's top-ranked ideas (Appendix (A.II] and how the AI Ideas Human. Reran condition tends score higher than the AI Ideas condition on all metrics in Section
Table 6:
Review score consis- tency among human reviewers (first block) and between humans and AI (second block).
such a complex and subjective task. These
RELATED WORK
Research idea generation and execution. Several prior works explored methods to improve idea generation, such as iterative novelty boosting (Wang et al. 2024), multi-agent collaboration (Baek) retrieval and revision (Yang et al., 2024). While some of them
al., 2024), and share similar components as our ideation agent, these works focus improving the idea generation methods over vanilla prompting baselines, without comparisons
any human expert baselines. Beyond idcation, another line of work uses LLMs for executing experiments by generating code given the research problems (Huang et al. 2024 Tian et al. 2024). or combining idea generation with code generation to directly implement Al -generrated ideas (Lu et al., 2024: Li et al. 2024). These works either use automatic evaluation on pre-defined set of problems and benchmarks, setting a constrained problem space; rely proxy metrics like LLM evaluators, which often unreliable.
LLM for other research-related tasks. LLMs have also been used for several other research-relate tasks, such generating code to perform data-driven discovery (Majumder al. 2024 Hu et al.
2024
Guo et al.
2024 Gu et al. 2024 [fargan et al. 2024), automatic review generation (D) Arcy
al.
2024
Liang al. (0024), related work curation (Kang & Xiong| 2024 Ajiih et al., 2024 Press
et al.
2024 Lehr et al.
2024), experiment outcome prediction (Lchr et al. 22224] Zhang et al.
2024
Manning et al. 2024] Hewitt al.) 2024). and future work recommendation (Zhang al. 2024).
Unlike these works, we tackle the more creative and open-ended task of research ideation.
Computational creativity. Our work also connects the line of work on examining Al's novelty and diversity in creative tasks. Previous findings include AI writings being less creative than professional writers (Chakrabarty et al., 2024): L.M generations lacking collective diversity (Zhou al. 2024] Anderson et 20244: and human-Al collaboration reducing diversity (Padmakumar & He 2024).
In contrast, we focus on the human-A AI comparison on the challenging task of research ideation with expert participants.
CONCLUSION
We compared research ideas generated by AI agent with ideas written by expert researchers and observed the robust finding that expert reviewers rate Al ideas as statistically more novel than expert ideas. We recognize several limitations of the current study, including the quality of the human baseline, the subjectivity of idea evaluation, and the limited scope. We discuss future steps to address these limitations in Appendix and discuss various ethical considerations in Appendix A.2
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We thank all participants who wrote and reviewed ideas for us. Many of them also provided insightful feedback on various aspects of this study. This project would not have been possible without their support. We thank Rose Wang, Dora Zhao, Irena Gao, Isabel Gallegos, Ken Liu, Aryaman Arora, Harshit Joshi, Shi Feng, Tianyu Gao, Xinran Zhao, Yangjun Ruan, Xi Ye, Mert Yuksekgonul, and members of Tatsu Lab and SALT Lab for their helpful feedback on the early version of this draft. We thank our undergraduate intern Isha Goswami and faculty administrator Eric Alejandro Pineda for assisting with review data collection and financial logistics.
This work was supported by gifts from Open Philanthropy, Tianqiao and Chrissy Chen Institute, Meta, IBM, and Amazon, and grants from ONR, NSF IIS-2247357, and CNS-2308994.
REFERENCES
Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. LitSearch: A Retrieval Benchmark for Scientific Literature Search. ArXiv, abs/2407. 18940, 2024.
Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. Homogenization Effects of Large Language Models on Human Creative Ideation. In Proceedings of the 16th Conference on Creativity & Cognition, 2024.
Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. ResearchAgent Iter- ative Research Idea Generation over Scientific Literature with Large Language Models. ArXiv, abs/2404.07738, 2024.
Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sandra Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David J. Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378:1063 1074, 2022.
Alina Beygelzimer, Yann Dauphin, Percy Liang. and Jennifer Wortman Vaughan. The neurips 2021 consistency experiment. https::/blog neurips newrips.cc//22//111 2t////222/1//00 2021. Neural Information Process-
Chhe-neurigg
ing Systems blog post.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher e' and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. ArXiv, abs/2407.21787 2024.
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art or Artifice? Large Language Models and the False Promise of Creativity. In CHI, 2024.
Katherine M. Collins, Albert Qiaochu Jiang, Simon Frieder, Li Siang Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B. Tenenbaum, William Hart, Timothy Gowers, Wenda Li, Adrian Weller, and Mateja Jamnik. Evaluating language models for mathematics through interactions. Proceedings of the National Academy of Sciences of the United States of America, 121, 2024.
Mike D' D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. MARG: Multi-Agent Review
Generation for Scientifie Papers. ArXiv, 0tss/2001042255 2024.
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of- Verification Reduces Hallucination in Large Language Models. ArXiv, abs/2309. 11495, 1495, 2023.
Esin Durmus, Faisal Ladhak, and Tatsunori B. Hashimoto. Spurious Correlations in Reference-Free Evaluation of Text Generation. In Annual Meeting of the Association for Computational Linguistics, 2022. URL (tRt.httts:///a semant oowpecoooo..o.oooo/ooorooroooa:a

---

Published as a conference paper at ICLR 2025
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang. Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrry Heer, and Tim Althoff. BLADE: Benchmarking Language Model Agents for Data-Driven Science. ArXiv, abs/2408 096677, 2024.
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. DS-Agent: Auto- mated Data Science by Empowering Large Language Models with Case-Based Reasoning, In ICMI., 2024.
Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer. Predicting Results of Social Science Experiments Using Large Language Models. Preprint, 2024. URL https::/docseed
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Kun Kuang, Yang Yang. Hongxia Yang, and Fei Wu. InfiAgent- Beennhhh Evaluating Agents on Data Analysis Tasks. In ICML, 2024.
Qian Huang, Jian Vora, Perey Liang, and Jure Leskovec. AleggenBBncch: Evaluating Language
Agents on Machine Learning Experimentation. In ICMI,, 2024.
Tal Ifargan, Lukas Hafner, Maor Ker, Ori Alcalay, and Roy Kishony. Autonomous LLM-driven research from data to human-verifiable research papers. ArXiv, abs/2404.17605, 2024.
Hao Kang and Chenyan Xiong. ResearchArena: Benchmarking LLMs Ability Collect and Organize Information as Research Agents. ArXiv, abs/2406. 10291, 2024.
Steven A. Lehr, Aylin Caliskan, Suneragiri Liyanage, and Mahzarin R. Banaji. ChatGPT Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor. Proceedings of the National Academy of Sciences of the United States of America, 121 35, 2024.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensie NLP Tasks. In NeurIPS, 2020.
Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. MIR-Copilot: Autonomous Machine Learning Research based Large Language Models Agents. ArXiv, abs/2408. 14033, 2024.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom, Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey, Cherepanov, James Molloy, Daniel Jaymin Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de, Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with AlphaCode. Science, 378:1092 1097, 2022.
Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis. NEJM AI, 1(8), 2024.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The Al Scientist: Towards Fully Automated Open-Ended Scientifie Discovery ArXiv, abs/2408.06292. 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative Refinement with Self-Feedback. In NewrIPS, 2023.
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discovery- Bench: Towards 2024.
Discovery with Large Language Models. ArXiv, abs/2407.01125,
12

---

Published as a conference paper at ICLR 2025
Benjamin S. Manning, Kehang Zhu, and John J. Horton. Automated Social Science: Language Models as Scientist and Subjects. SSRN Electronic Journal, 2024.
Vishakh Padmakumar and He He. Does Writing with Language Models Reduce Content Diversity? In ICLR, 2024.
Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofrr Press, and Matthias Bethge. CiteME: Can Language Models Accurately Cite Scientific Claims? ArXiv, Abs/2407.12861. 2024.
Nils Reimers and Iryna Gurevych. Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. In EMNLP, 2020.
Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki. Sweta Agrawal, Chau Pham, Gerson C. Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander Da Costa, Saloni Gupta, Megan L. Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Miserlis Hoyle, and Philip Resnik. The Prompt Report: A Systematic Survey of Prompting Techniques. ArXiv, 1atp22406066605. 2024.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen tau Yih. REPLUG: Retrieval-Augmennte Black-Box Language Models. In NAACI, 2024.
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Min Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du Tianhua Tao, Ofir Press, Jamie Callan, A. Huerta, and Hao Peng. SciCode: A Research Coding Benchmark Curated by Scientists. ArXiv, has/22077.11668 2024.
Trieu H. Trinh, Yuhuai Wu, Quoe Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476 482, 2024.
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. SciMON: Scientific Inspiration Machines Optimized for Novelty. In ACL, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoe Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In NeurlPS, 2022.
Orion Weller, Mare Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. "According to ....." ": Prompting Language Models Improves Quoting from Pre-Training Data. In EACL, 2023.
Jason Weston and Sainbayar Sukhbaatar. System Attention (is something you might need too). ArXiv, abs/2311.11829, 2023.
Zonglin Yang. Xinya Du, Junxian Li, Jie Zheng. Soujanya Poria, and E. Cambria. Large Language Models for Automated Open-domain Scientific Hypotheses Discovery. ACL Findings, 2024.
Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovee, Percy Liang, Ed Huai hsin Chi, and Denny Zhou. Large Language Models as Analogical Reasoners. In ICLR, 2024.
Xingjian Zhang, Yutong Xie, Jin Huang, Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Ergen, Dongsub Shim, Honglak Lee, and Qiaozhu Mei. MASSW: A New Dataset and Benchmark Tasks for Al-Assisted Scientific Workflows. ArXiv, abs/2406. 06357, 2024.
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze -ze Cheng, Ed Huai hsin Chi, Quoe Le, and Denny Zhou. Take Step Back: Evoking Reasoning via Abstraction in Large Language Models. In ICLR, 2024.
Ruiqi Zhong, Charles Burton Snell, Dan Klein, and Jacob Steinhardt. Describing Differences between Text Distributions with Natural Language. In ICMI., 2022.
13

---

Published as a conference paper at ICLR 2025
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal Driven Discovery of Distributional Differences via Language Descriptions. In NeurlPS, 2023.
Yilun Zhou, Caiming Xiong, Silvio Savarese, and Chien-Sheng Wu. Shared Imagination: LLMs
Hallucinate Alike, ArXiv, abs/2407.16604 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DISCUSSION
In this section, we discuss some high-level questions readers might have and suggest ways to address them.
Question 1: Do these collected expert ideas represent their best ideas? One might argue that these ideas submitted by our idea- idea-writing participants might not represent their best ideas as we discussed in subsection 1, since most of them came up with the idea on the spot within a short period. In order to address this concern, we have designed an experiment where we will compare Al ideas with papers accepted at top-tier AI conferences. To avoid any possible contamination, we target the upcoming EMNLP 2024 conference, which will release the accepted papers in October 2024. We have generated AI ideas with our agent on 23 topics from the EMNLP Call For Papers page in July 2024 and cached them. We pre-registered our analysis plan which also includes the link to the cached ideas. Apart from comparing the quality of these ideas, we will also compute the overlap between Al-generated ideas and accepted papers on the same topics.
Are evaluations based solely on ideas subjective? In this current study, we focused solely evaluating the ideas themselves. Ideas that sound novel and exciting might not necessarily turn into successful projects, and our results indeed indicated some feasibility trade-offs of AI ideas. We view the current study as preliminary evaluation of Al-generated ideas. In the next phase, we will recruit researchers to execute some Al and human- -eneratee ideas into full projects. This will enable reviewers to assess the complete experimental outcomes, providing a more reliable basis for evaluation. Furthermore, it will allow us to analyze whether our initial idea evaluations align with the assessments of the actual project outcomes.
Question 2:
Why do you focus only on prompting- research in NLP? The scope of our study limited to prompting research ideas within NL.. We chose this design to facilitate the next phase of our execution experiment, where prefer research ideas that are less resource-demandinn can be executed relatively quickly. We believe that the evaluation protocols we established should be applicable to other research domains well, although the conclusions could be different depending on the research fields. Future work should consider extending such human study to other research domains and it would be interesting compare how the conclusions differ.
Question 3:
and
Can you automate idea execution as well? It is tempting to envision an end-to-end automated research pipeline where Al agents can implement Al-generated ideas to directly evaluate their effectiveness. Apart from speeding up scientific discovery, one could also imagine using such exccution agents automatically verify experiment results in existing papers or new submissions. We have also explored building an LLM agent to generate code to implement the generated ideas. Specifically, we provide template codebase that consists of: (1) loading datasets from Huggingface or generating synthetic test examples; (2) implementing baseline methods; (3) implementing the proposed method; (3) loading or implementing the evaluation metrics; (4) running experiments on the testset with the baselines and the proposed method, so that the output of the agent will be report of the bascline performance as well as the proposed method's 's performance. While this agent can generate code that compiles and executes, we find that the automated experiments can be misleading because the agent often skips or modifies steps in the baselines or proposed methods. In some cases, the metric functions also not correctly defined. This highlights the core challenge: just comparing the final experiment results is not enough; we have to verify the faithfulness of the implementations as well. Performing such implementation verification is not trivial task, and we leave future work. We provide detailed description of our idea execution agent in Appendix A.30
Question 4:
15

---

Published as a conference paper at ICLR 2025
A.2 ETHICAL CONSIDERATIONS
Publication Policy. The growing use of Al to generate research ideas raises serious concerns about the potential abuse of these technologies by students or researchers who may flood academic conferences with low-quality or poorly thought-out submissions. The availability of LLM-generated content could lead to a decline in the overall quality of academic discourse, as some individuals might take lazy approach, relying AI to both generate ideas and review submissions. This would undermine the credibility and integrity of the review process. The risks are real. Without proper oversight, we could see deluge of submissions that lack depth intellectual merit. To prevent this, it is essential to hold researchers accountable for the outputs generated through AI tools. Rigorous standards must be applied equally both Al-assisted and human-gen -geerrrte research to ensure that the use of LLMs does not result in misleading, superficial, or unethical academic contributions.
Intellectual Credit. The use of LLMs to generate research ideas introduces significant ambiguity around the concept of intellectual credit. Traditional frameworks for attributing credit in research, based human authorship and contribution, become less clear when AI plays a significant role in idea generation. Questions arise around how to distribute credit between the developers of the LLM, the researchers who designed the frameworks for its use, and the researchers who integrate Al- Al-generated ideas into their work. Furthermore, it becomes increasingly difficult to trace the origins of Al-generated contributions, especially when they draw from vast datasets composed of numerous sources. This complexity calls for a broader rethinking of how intellectual credit is assigned in Al-driven research. While a complete overhaul of legal and academic norms is beyond the scope of this project, we advocate for the adoption of transparent documentation practices. Researchers should clearly disclose the role Al played in the idea generation process, specifying which models, data sources, and frameworks were used, and outlining the level of human involvement. This could ensure that the credit distribution in Al-supported research is as transparent and fair as possible.
Potential for Misuse. Al-generated research ideas, especially those that introduce novel concepts, have the potential to be misused in ways that could lead to harmful or destabilizing outcomes. For instance, ideation agents could be leveraged generate adversarial attack strategies other unethical applications. This concern aligns with broader arguments from those focused on existential risk (X-risk), who argue that Al-driven innovation could be primary route to destabilizing the status quo, posing risks at a societal or even global level. Our stance is that such discussions on safety should be evidence-ber based to the extent that it is possible, and careful evaluation work is an important component of keeping these discussions grounded in actual, measured capabilities of these systems. We advocate for continued safety research specifically targeting these types of concerns -uch as the development of Reinforcement Learning from Human Feedback (RLHF) systems or anti-jat anti-jailbreak mechanisms for research ideation agents. Additionally, we believe it would be meaningful to create safety benchmarks that assess the ethical and safe application of Al-generated ideas.
Idea Homogenization. Our analysis showed that current LLMs lack diversity in idea generation. This raises important concerns that wide adoption of LLMs can result in idea homogenization, where the generated ideas only reflect a narrow set of perspectives or have systematic biases. Over time, this could lead to a reduction in the richness and diversity of research outputs globally. Future work should develop ways to either improve LLMs themselves or refine our idea generation methods to promote idea diversity. It's also important to note that our evaluation primarily assesses the quality of the typical ideas being generated, and may not fully capture the long tail of unique or novel ideas that would be truly transformative.
Impact on Human Researchers. The integration of Al into research idea generation introduces a complex sociotechnical challenge, as research is fundamentally a community- -riven, collaborative effort. By introducing AI, particularly LLMs, into this social system, we risk unforeseen consequences. Overreliance on Al could lead to a decline in original human thought, while the increasing use of LLMs for ideation might reduce opportunities for human collaboration, which is essential for refining and expanding ideas. To mitigate these risks, future works should explore new forms of human-Al collaboration, and our results on human reranking of AI ideas show that even naive Au collaboration approaches can be effective. Beyond reranking. humans can play a critical role in the ideation process by providing intermediate feedback, taking Al Al-generated ideas as inspiration for further development, and bringing their unique expertise into the process. Understanding how to integrate I.LMs into this collaborative process without disrupting the social fabric of research
16

---

Published as a conference paper at ICLR 2025
will be important ongoing problem, requiring careful consideration of the broader sociotechnical implications.
Impact on Human Researchers. The of Al to generate research ideas raises concerns about the potential displacement of human researchers and the devaluation of human creativity. There is risk that researchers may become overly reliant AI, leading to decline in original human thought and innovation. Furthermore, the dynamics of research collaboration could be fundamentally altered. For example, increasing use of LLMs for ideation might discourage collaboration among human researchers. To address this, we highlight the value of human-Al collaboration. We presented preliminary results where human reranking on top of Al-generated ideas can bring additional values. Apart from reranking, there are many other possible ways for humans to contribute to the collaborative ideation process, for example, by providing intermediate feedhack to generated ideas, or taking Al ideas as inspirations for further improvement. Moreover, human researchers often brainstorm together and collaborative discussion helps refine ideas. How adapt LLMs in collaborative idea generation an interesting open question that we leave to future work.
17

---

Published as a conference paper at ICLR 2025
A.3 LIST OF RESEARCH TOPICS
We selected the following list of research topics for our research ideation task:
1. Bias: novel prompting methods to reduce social biases and stereotypes of large language models
2. Coding:
novel prompting methods for large language models to improve code generation
3. Safety:
novel prompting methods to improve large language models' robustness against adversarial attacks or improve their security or privacy
4. Multilingual:
novel prompting methods improve large language models' performance on multilingual tasks or low-resource languages and vernacular languages
5. Factuality: novel prompting methods that can improve factuality and reduce hallucination of large language models
6. Math: novel prompting methods for large language models to improve mathematical problem solving
7. Uncertainty: novel prompting methods that can better quantify uncertainty or calibrate the confidence of large language models
We these topic descriptions to elicit ideas from both human participants and our LLM agent.
show the distribution our idea writing participants' selected topics in Table
Topic
Count
Bias
Coding
9
Safety
Multilingual
10
Factuality
Math
4
Uncertainty
Total
49
Table 7: Idea topic distribution.
18

---

Published as a conference paper at ICLR 2025
A.4 PROJECT PROPOSAL TEMPLATE
We give the following project proposal template to both the Al agent and human idea writers,
1. Title: A concise statement of the main research question to be used as the paper title.
Problem Statement: Clearly define the problem your research intends to address. Explain clearly why this problem is interesting and important.
Explain why existing methods the inspiration behind the new proposed method. You should also motivate why the proposed method would work better than existing baselines on the problem.
Motivation:
good enough solve the problem, and explain
4. Proposed Method: Explain how the proposed method works, describe all the essential steps.
5. Step-by-Step Experiment Plan: Break down every single step of the experiments, make sure every step is executable. Cover all essential details such as the datasets, models, and metrics to be used. If the project involves prompting. give some example prompts for each step.
6. Test Case Examples: Give at least two concrete examples. The first example should show how the baseline method fails on the test case. there are multiple baselines, give examples for all of them. The second example should show how the proposed method succeeds on the test case. For each test case, include the input (test example and the full prompt) and the expected output. You should also provide an explanation for why the outputs from the proposed prompt are better. If the proposed method has multiple steps, break them down into intermediate steps.
7. Fallback Plan:
Propose some alternative plans for what should the students do if the proposed method doesn't 't manage to satisfy the success criteria. For example, you can suggest additional analysis to help debug why the proposed method didn't 't work, which could inform alternative new methods, or just turn the project into an analysis paper instead by offering some interesting ablation and insights.
19

---

Published as a conference paper at ICLR 2025
A.5 PROJECT PROPOSAL DEMO EXAMPLE
We present manually written demonstration example used for project proposal generation. The example is summarized from an existing paper (Dhuliawala et al. 2023). This same example given to both the Al agent well as the idea-writing experts.
1. Title:
Chain-of- Verification Reduces Hallucination in Large Language Models
Problem Statement:
Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models.
3. Motivation:
majority of the methods for reducing hallucination can be divided into roughly three categories: training-tima correction, generation-tine correction, and via augmentation (tool-use), We want to take a simpler approach that fully leverages the power of L.LM itself. Our key motivation is that large language models, when suitably prompted, can both generate and execute plan of how to verify themselves in order to check their own work, and finally incorporate this analysis into an improved response.
Proposed Method: Our overall process, which call Chain-of- oorrefiiiiation (CoVe), thus performs four steps:
(1) Generate Baseline Response: Given a query, generate the response using the LLM.
Plan Verifications: Given both query and baseline response, generate a list of verification questions that could help to self-analyze there are any mistakes in the original response.
(3) Execute Verifications: Answer each verification question in turn, and hence check the answer against the original response to check for inconsistencies or mistakes.
(4) Generate Final Verified Response: Given discovered inconsistencies (if any), generate revised response incorporating the verification results.
Each of these steps performed by prompting the same L.LM different ways to obtain the desired response.
Step-by-Step Experiment Plan:
1: Gather Datasets: We choose datasets that evaluate factual correctness, including the Multi- SpanQA dataset closed-book QA and the FactScore dataset generating biographies.
2: Construct Prompts: For the baseline, we use direct prompting where, given a query, we generate left-to-right usual using the LLM,, with no special tricks. Given that such baseline generations are typically prone to hallucination, CoVe attempts to identify these hallucinations and correct them the following steps:
Plan Verifications:
Conditioned on the original query and the baseline response, the model prompted to generate series of verification questions that test the factual claims in the original baseline response.
(2) Execute Verifications: Given the planned verification questions, the next step is to answer them in order to assess any hallucinations exist. The planning prompt conditions the baseline response in the first step. The verification questions generated from planning are answered in the second step, where crucially the context given to the LLM prompt only contains the questions and not the original baseline response, hence preventing the LLM from repeating those answers directly.
(3) Generate Final Verified Response: Finally, the improved response that takes verification into account is generated. This executed by final few-shot prompt where the context takes into account all of the previous reasoning steps, the baseline response, and verification question-answer pairs, so that the corrections can take place.
3: Select Models: We test GPT-3.5 well the open-source L.AMA-2-70B-ccatt.
and GPT-4 from the OpenAl API, as
20

---

Published as a conference paper at ICLR 2025
4: Get Results: Get answer predictions from the models these datasets with both the baselines and the proposed method.
5: Analyze Results: Compare whether the new method improves the performance of LLMs in these tasks as compared to the baselines.
6. Test Case Examples:
Baseline Prompt Input (Direct Prompting): Name some politicians who were born in NY, New York.
Baseline Prompt Expected Output (Direct Prompting): Here are some politicians who were born in NY, New York:
1. Hillary Clinton former secretary of state and former Democratic presidential nominee.
2. Donald Trump- former president of the United States.
3. Michael Bloomberg former Mayor of New York City and former Democratic presidential candidate.
Proposed Prompt Input (Chain-of- Verii-faatiiiiiaiiionn Prompting; Step Plan Verifications):
Generate a set of verification questions to verify each answer.
Proposed Prompt Expected Output CChain-of-Veriffcca Prompting; Step 1: Plan Verifications):
Where was Hillary Clinton born?
Where was Donald Trump born?
Where was Michael Bloomberg born?
Proposed Prompt Input
Prompting; Step 2: Execute Verifica-
tions): Answer each of the verification questions.
Proposed Prompt Expected Output Veriifit-eii Prompting: Step Execute Verifications):
1. Hillary Clinton born Chicago, Illinois, United States on October 26, 1947.
2. Donald Trump born June 14, 1946, in Queens, New York City, New York, United States.
Michael Bloomberg was born on February 14, 1942, in Boston, Massachusetts, United States.
Proposed Prompt Input (Chain-of- Verification Prompting; Step 3: Final Response): (Prepend the original question and the baseline response and the verification QA pairs) Given the verification answers, generate a revised response incorporating the verification results.
Proposed Prompt Expected Output CChain-ff-eerificalie
Prompting; Step 3: Final Response): Here are some politicians who were born in NY, New York:
1. Donald Trump former president of the United States.
2. Alexandria Ocasio-Cortez. Democratic member of the U.S. House of Representatives.
Explanation: Given user query, large language model with direct prompting generates a baseline response that may contain inaccuracies, e.g., factual hallucinations. To improve Chain-of-Verification this, Chain-of- Verification first generates a plan of set of verification questions to ask,
and then executes that plan by answering them and hence checking for agreement. We find that individual verification questions are typically answered with higher accuracy than the original accuracy of the facts in the original longform generation. Finally, the revised response takes into account the verifications.
Fallback Plan:
If the proposed method does not help compared to the baseline, analyze each step of the CoVe process to see the verification questions are relevant, if the answers to the verification questions correct, and whether the generated final verified response is indeed improved over the baseline response by considering the verification QA pairs. This can help us debug the proposed method or turn this into interesting analysis the model's ability verify and correct its responses.
21

---

Published as a conference paper at ICLR 2025
A.6 STYLE STANDARDIZATION PROMPT
Style Standardization Prompt
You writing assistant specialized in editing academic writing. I will give you student's 's research idea and idea template. Your task is to edit the student's idea to follow the template's format. Student idea: (Insert the student's idca here) Template: (Insert the template idea here)
Make sure that you only edit the wording and formatting, including things like punctuation, capitaliza- tion, linebreaks, and bullet points. Also make sure to edit any informal wording and phrasing use vocabulary that sounds like the template's 's writing style. No other changes allowed beyond these. The main subsections should indexed clearly without indentation at beginning. The title subsection does not need indexing: other subsections, including problem statement, motivation, proposed method, step-by-step experiment plan, test case examples, and fallback plan, should be indexed to 6. Each subsection then have sub sub-bullets for sub-subsections if applicable. Leave empty line after each subsection.
You should use as indentation and make sure to use appropriate nested indentation for sub-bullets. All bullets should have clear hierarchy so people can casily differentiate the sub-bullets. Only leave empty lines between subsections and remove any extra line breaks. If many bullet points are clustered together in paragraph, separate them clearly with indentation and appropriate bullet point markers. Change to new line for each new bullet point.
For the fallback plan, do not list a bunch of bullet points. Instead, condense them into one coherent paragraph.
For line breaks, avoid Raw String Literals Double Backslashes when using "In", and change them spaces or tabs.
For in-line citations, the citation mentioned author's 's last name (like "(Si et al., 2023)" or "(An et al., 2024)"), you should keep them there; but if the citation is just number (like "[1]" or "[3,4,5]"), you should just remove it and do some necessary rephrasing to make the sentence still sound coherent without the references.
Apart from minor rephrasing and changing formatting, do not change any content of the idea. You must preserve the exact meaning of the original idea, do not change. remove, or add any other details. Do not drop any subsections (including test case examples). Do not rename any models, datascts, or methods. Do not drop clarification examples in brackets and do not drop any data source mentions (e.g., Chatbot Arena Wildchat)! Note that when indexing test case examples, each case example could have multiple steps of inputs and outputs and you shouldn't i give separate indices to them. Each test case example should be a whole set of input-output pairs for the baseline(s) and proposed method. For the proposed method subsection, avoid any big changes. If the subsection comes in as a coherent paragraph, you don't t have to break it down into bullet points. the subsection is already in bullet points, you should keep it that way. the subsection is mix of both, you should keep the bullet points and the coherent paragraph as they are. Keep all the clarification and examples mentioned in all the subsections and do not remave any of them (including those in brackets).
For model selection, if any version of Claude is mentioned, change the latest version of Claude (Claude-3.5) any version LLaMA mentioned, change to latest version LLaMA-3, Do not make any other model changes.
Now directly generate the edited student idea to match the format of the template.
22

---

Published as a conference paper at ICLR 2025
A.7 IDEA REVIEW FORM
We use the following review form to elicit reviews from all expert reviewers. Reviewers have one week of time finish each review.
1. Name
Institution
3. Email
4. Consent
Honor Code: confirm that will not use ChatGPT, Claude, Gemini, or any other Al tools when writing my reviews.
Familiarity: Before reviewing the idea, please indicate how familiar you are with the given topic on scale of 5 (this is just for us understand potential confounders).
1. You have never read about this topic before
2. You have read at least one paper on this topic
3. You have read multiple papers on this topic but have not published any paper on
4. You have co-authored at least one paper on this topic
5. You have co-authored multiple papers on this topic or have published least one first-author paper on this topic
7. Experience: Have you reviewed for major NLP or Al conferences before (e.g.. *ACL, COLING, NeurlPS, ICLR, ICML, AAAI)?
8. Full Research Idea Proposal
Novelty Score: Whether the idea is creative and different from existing works on the topic, and brings fresh insights. You are encouraged to search for related works online. You should consider all papers that appeared online prior to July 2024 as existing work when judging the novelty.
1. Not novel at all - there are many existing ideas that are the same
2.
3. Mostly not novel - you can find very similar ideas
4.
5. Somewhat novel there are differences from existing ideas but not enough to turn into a new paper
6. Reasonably novel there are notable differences from existing ideas and probably enough to turn into a new paper
7.
8. Clearly novel major differences from all existing ideas
9.
10. Very novel very different from all existing ideas a very interesting and clever way
10. Novelty Rationale: Short justification for your score. If you give low score, you should specify similar related works. (Your rationale should be at least 2-3 sentences.)
11. Feasibility Score: How feasible it is to implement and execute this idea as research project? Specifically, how feasible the idea is for typical CS PhD student to execute within 1-2 months of time. You can assume that we have abundant OpenAI/ Anthropic API access, but limited GPU compute.
Impossible: the idea doesn't 't make sense or the proposed experiments are flawed and cannot be implemented
23

---

Published as a conference paper at ICLR 2025
2.
3. Very challenging: there are flaws in the proposed method or experiments, or the experiments require compute/human resources beyond any academic lab
4.
5. Moderately feasible: It can probably be executed within the given time frame but would require careful planning, efficient use APIs or some advanced computational strategies to overcome the limited GPU resources, and would require some modifications to the original proposal to make it work
6. Feasible: Can be executed within the given constraints with some reasonable planning
7.
8. Highly Feasible: Straightforward to implement the idea and run all the experiments
9.
10. Easy: The whole proposed project can be quickly executed within a few days without
requiring advanced technical skills
12. Feasibility Rationale: Short justification for your score. If you give low score, you should specify what parts are difficult to execute and why. (Your rationale should be at least 2-3 sentences.)
13. Expected Effectiveness Score: How likely the proposed idea is going to work well (e.g., better than existing baselines).
1. Extremely Unlikely: The idea has major flaws and definitely won't work well
2.
3. Low Effectiveness: The idea might work in some special scenarios but you don't 't expect to work in general
4.
5. Somewhat ineffective:
There might be some chance that the proposed idea can work better than existing baselines but the improvement will be marginal inconsistent
6. Somewhat effective: There is decent chance that the proposed idea can beat existing baselines by moderate margins on few benchmarks
7.
8. Probably Effective: The idea should offer some significant improvement over current methods on the relevant benchmarks
9.
10. Definitely Effective: You very confident that the proposed idea will outperform existing methods by significant margins on many benchmarks
14. Expected Effectiveness Rationale: Short justification for your score. (Your rationale should be least 2-3 sentences.)
15. Excitement Score: How exciting and impactful this idea would be if executed as a full project. Would the idea change the field and be very influential.
1. Poor: You cannot identify the contributions of this idea, it's not interesting all and you would fight to have it rejected at any major Al conference
2.
3. Mediocre: this idea makes marginal contributions and very incremental
4.
5. Leaning negative: it has interesting bits but overall not exciting enough
6. Learning positive: exciting enough to be accepted at major Al conference, but still has some weaknesses or somewhat incremental
7.
24

---

Published as a conference paper at ICLR 2025
8. Exciting: would deepen the community's understanding or make major progress in this research direction
9.
10. Transformative: would change the research field profoundly and worth best paper award major Al conferences
16. Excitement Rationale: Short justification for your score. (Your rationale should be at least 2-3 sentences.)
17. Overall Score: Overall score: Apart from the above, you should also give an overall score for the idea on scale of 10 defined below (Major AI conferences in the descriptions below refer to top-tier NLP/AI conferences such as *ACL, COLM, NeurIPS, ICLR, and ICML.):
1. Critically flawed, trivial, wrong, would be waste of students' time to work it
2. Strong rejection for major Al conferences
3. Clear rejection for major Al conferences
4. Ok but not good enough, rejection for major conferences
5. Decent idea but has some weaknesses or not exciting enough, marginally below the accep- tance threshold of major Al conferences
6. Marginally above the acceptance threshold of major Al conferences
7. Good idea, would be accepted by major AI conferences
8. Top 50% of all published ideas on this topic major conferences, clear accept
9. Top 15% of all published ideas this topic major Al conferences, strong accept
10. Top 5% of all published ideas on this topic at major Al conferences, will be seminal paper
18. Overall Rationale: You should also provide rationale for your overall score. (Your rationale
should be at least 2-3 sentences.)
25

---

Published as a conference paper at ICLR 2025
19. Confidence: Additionally, ask for your confidence in your review scale of 1 to 5 defined as following:
1. Your evaluation is an educated guess
2. You are willing to defend the evaluation, but it is quite likely that you did not understand central parts of the paper
3. You are fairly confident that the evaluation is correct
4. You are confident but not absolutely certain that the evaluation is correct
5. You are absolutely certain that the evaluation is correct and very familiar with the relevant literature
20. Time: How many minutes did you spend this task?
26

---

Published as a conference paper at ICLR 2025
A.8 IDEA GENERATION AGENT: ADDITIONAL IMPLEMENTATION DETAILS
Seed Idea Generation Due to the max output length limit of the LLM API, we first generate a large number of shorter seed ideas. We keep the seed ideas short that ideas given the same output token budget. We provide a demonstration example of the seed idea in Appendix A.9 Then, we perform duplication and expand each remaining seed idea into full project proposal following our standard template in Appendix A.4
explore more different
Retrieval Augmentation We apply retrieval augmentation to the idea generation prompt order to increase diversity in the idea generation. To maximize diversity, we apply retrieval augmentation half the time when generating seed ideas, and we randomly select is = 10 papers from the top 20 retrieved papers when applying retrieval augmentation.
Idea Filtering After expanding seed ideas into full project proposals, we did some basic filtering to remove any project proposals that failed the novelty and feasibility checks:
1. Novelty: We use the literature review module retrieve the top 10 most relevant papers to the generated idea and ask the LLM to compare each of them to the generated idea. The idea will filtered as long as any one of the retrieved papers is judged as equivalent.
2. Feasibility: The idea will be filtered
requires extensive manual labor or hardware resources beyond the capacity of a typical academic lab. The idea will also be filtered if it involves any inconsistency in the experimental setups or assumptions. For example, if the idea assumes only Blxck-box API access of the LLMs, then it shouldn' involve experiments that need internal weight access.
This filtered out about 1% of the generated project proposals.
27

---

Published as a conference paper at ICLR 2025
A.9 DEMONSTRATION EXAMPLE: SEED IDEA GENERATION
We present a demonstration example used for seed idea generation. The example is summarized from existing paper (Dhuliawala et al. 2023).
Title:
Chain-of- Verification Prompting
Problem:
Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models.
Existing Methods:
majority of the methods for reducing hallucination can be divided into roughly three categories: training-time correction; generation-time correction; and via augmentation (tool-use).
Motivation:
A key observation is that large language models, when suitably prompted, can both generate and execute plan of how verify themselves in order to check their own work, and finally incorporate this analysis into improved response.
Proposed Method:
Our overall process, which we call
(CoVe), thus performs four core steps:
Generate Baseline Response: Given a query, generate the response using the L.I.M. Given both query and baseline response, generate a list verification questions that could help self-analyze if there any mistakes in the original response.
(2) Plan Verifications:
(3) Execute Verifications: Answer each verification question turn, and hence check the answer against the original response to check for inconsistencies or mistakes.
(4) Generate Final Verified Response: Given the discovered inconsistencies (if any), generate a revised response incorporating the verification results.
Each of these steps is performed by prompting the same LLM in different ways obtain the desired response.
Experiment Plan:
Compare with zero-shot prompting, Chain-of- Chain-of-Thought, and few-shot prompting on the MultiSpan dataset closed-book QA and FactScore dataset generating biographies.
28

---

Published as a conference paper at ICLR 2025
A.10 GENERATED SEED IDEAS AND THEIR NEAREST NEIGHBORS
We present several randomly sampled generated seed ideas (see Appendix|A.8) for the definition of seed ideas) on the topic of "novel prompting methods that can better quantify uncertainty or calibrate the confidence of large language models". For each idea, we show the most similar idea (nearest neighbor) based on the embedding similarity, along with the similarity score. In practice, we set a threshold threshold of 0.8 for determining whether two ideas duplicates.
Idea 1:
Title: Adaptive Precision Boundary Probing
Problem: LLMs often provide uncertainty estimates that are either too coarse-graing or inappropri- ately precise, failing to adapt to the inherent ambiguity or precision requirements of different queries. Existing Methods: Existing uncertainty quantification methods typically use fixed precision scales or calibration techniques that don't adapt to the specific context and precision requirements of each query.
Motivation: Human experts adjust the precision of their uncertainty estimates based on the nature of the question and the available evidence. We can incorporate this adaptive approach to improve LLM uncertainty quantification.
Proposed Method: We introduce Adaptive Precision Boundary Probing (APBP), a dynamic prompt- ing technique that iteratively refines the precision of uncertainty estimates. Given query, APBP starts with a coarse-grained confidence interval. It then prompts the model to assess whether this interval is appropriately precise given the query's context and the model's 's knowledge. If the model determines that greater precision is warranted, APBP iteratively narrows the interval, prompting the model each step to justify the increased precision. Conversely, if the model recognizes high ambiguity or limited knowledge, APBP widens the interval. Throughout this process, the model is asked to explicitly reason about the factors influencing the appropriate level of precision, such as the specificity of the query, the reliability of relevant knowledge, and potential sources of ambiguity. The final output an uncertainty estimate with precision level tailored to the specific query and the model's 's knowledge state. Experiment Plan:
We will evaluate APBP on a diverse set of tasks with varying inherent precision requirements, including numerical estimation, date prediction, and open-ended text generation. We Ill compare APBP against fixed- -ree-ireciioon uncertainty estimation methods, measuring both calibration accuracy and the appropriateness of precision levels judged by human experts.
Nearest Neighbor of Idea 1:
Title: Contextual Confidence Oscillation
Problem: Current methods for quantifying uncertainty in large language models often fail to capture the dynamic nature of confidence across different contexts within a single query. Existing Methods: Most existing approaches use static confidence scores or calibration techniques that don't It account for intra-query contextual shifts.
Motivation: Human confidence often fluctuates as we process different parts of a complex question or task. By mimicking this oscillation, we potentially capture more nuanced and accurate representation of model uncertainty.
Proposed Method: We propose Contextual Confidence Oscillation (CCO), novel prompting technique that encourages the model to continuously re-evaluate and express its confidence as it processes a query. The prompt is structured as series of checkpoints, where the model must pause its reasoning, reflect on its current confidence level, and explain any changes since the last checkpoint. This creates a confidence trajectory that can be analyzed for patterns, sudden drops, or gradual increases. Additionally, we introduce 'confidence disruptors' intentionally ambiguous or challenging sub-queries inserted various points to test the model's ability to recognize and express increased uncertainty when appropriate.
Experiment Plan: We will evaluate CCO against standard uncertainty quantification methods on range of tasks, including multi-step reasoning problems, ambiguous queries, and long-form text analysis. We'll measure not just overall accuracy of uncertainty estimates, but also the correlation between confidence oscillations and query. We'll also analyze how well the model's 's expressed confidence trajectory aligns with its actual performance across different segments of complex tasks.
difficulty levels of different parts of each
29

---

Published as a conference paper at ICLR 2025
Similarity: 0.70
Idea 2:
Title: Quantum Superposition Confidence Prompting
Problem: Current LLMs struggle to accurately quantify uncertainty across multiple possible answers,
often defaulting to overconfidence in single response.
Existing Methods: Existing approaches typically involve single-path reasoning limited branching. failing to capture the full spectrum of uncertainty.
Motivation: Inspired by quantum mechanics, where particles can exist in multiple states simultane- ously, we propose a method that allows L.MMs to consider multiple answer possibilities concurrently. Proposed Method: We introduce Quantum Superposition Confidence Prompting (QSCP), where the LLM is instructed to generate multiple potential answers simultaneously, assigning confidence scores to each. The prompt encourages the model to 'exist multiple states,' exploring contradictory an- swers and their implications concurrently. For example: 'Imagine you are in a quantum superposition of multiple expert personas. Each persona will provide an answer to the following question, along with confidence score (0-100%). Ensure the personas explore contradictory viewpoints. Question: [INSERT QUESTION]' The L.M then generates responses from multiple personas, each with its own confidence score. The final uncertainty is derived from the distribution of these scores, providing a more nuanced understanding of the model's confidence across possible answers.
Experiment Plan: Compare QSCP against standard prompting, thain-of-thought,, and other uncer- lainty quantification methods on diverse cresstoon-answrinng datasets, Evaluate using metrics such as calibration error, Brier score, and a novel "quantum uncertainty score' that measures the spread and coherence of the generated answer superposition.
Nearest Neighbor of Idea 2:
Title: Quantum Superposition Prompting
Problem: Traditional methods for uncertainty quantification in large language models often fail to capture the full range of possible interpretations and outcomes, especially for queries with inherent ambiguity multiple valid perspectives.
Existing Methods: Current approaches typically focus on generating a single response with an associated confidence score, or best, small set of discrete alternatives.
Motivation: Drawing inspiration from the principle of superposition in quantum mechanics, we propose a method represent and reason about multiple possible outcomes simultaneously, providing richer and more nuanced uncertainty quantification.
Proposed Method: We present Quantum Superposition Prompting (QSP), a novel framework for exploring and quantifying uncertainty in language model outputs. QSP begins by prompting the model to generate 'superposition' of possible interpretations or approaches to the given query. Each element in this superposition is assigned complex amplitude, representing both its probability and its relationship to other elements. The model then guided through a series of "measureeent prompts, designed to collapse this superposition along different bases of interpretation. These measurements yield probability distributions over outcomes, capturing different facets of uncertainty. QSP employs techniques inspired by quantum computing, such as interference and entanglement, to model how different interpretations interact and influence each other. The final uncertainty quantification derived from the full set of measurements, providing a multi-dimensional representation of the model's uncertainty that captures ambiguity, conflicting evidence, and the interdependence of different interpretations.
Experiment Plan: We will evaluate QSP on tasks that inherently involve multiple valid perspectives or ambiguous interpretations, such as ethical dilemmas, creative writing prompts, and open-ended analytical questions. Metrics will include the diversity and coherence of generated superpositions, the ability to capture human-judged ambiguities, and improvements in uncertainty calibration compared to classical methods.
Similarity: 0.77
Idea 3: Title: Fractal Uncertainty Decomposition
30

---

Published as a conference paper at ICLR 2025
Problem:
LLMs often provide overly simplistic uncertainty estimates that fail capture the hierar- chical and nested nature of uncertainty in complex knowledge domains.
Existing Methods: Current uncertainty quantification methods typically produce flat, single- dimensional confidence scores that don't 't reflect the multi-layered structure knowledge and uncer- tainty.
Motivation: By recursively decomposing a query into sub-components and assessing uncertainty multiple levels of granularity, we can construct a more comprehensive and structurally informed uncertainty estimate.
Proposed Method: We introduce Fractal Uncertainty Decomposition (FUD), prompting technique that recursively breaks down a query into hierarchical structure of sub-queries, assessing uncertainty at each level. Given an initial query, FUD prompts the model to identify key sub-components or aspects of the question. For each sub-component, the model provides an answer and confidence estimate. If the confidence for a sub-component below a certain threshold, FUD recursively applics the same decomposition process to that uub-component. This continues until either maximum depth is reached or all sub-components have high confidence. The resulting structure is a tree of nested confidence estimates. FUD then aggregates these estimates bottom-up, using combination of statistical methods and prompted meta-analysis by the model. The final output is both an overall uncertainty estimate and detailed map of the uncertainty structure, showing how confidence varies across different aspects and levels of the query.
tasks such as scientific expla- nation, historical analysis, and technical troubleshooting. We will compare its performance to flat confidence estimation methods and other hierarchical approaches. Evaluation metrics will include traditional calibration measures, as well as new metrics designed to assess the quality and informa- tiveness of the uncertainty decomposition. We will also conduct case studies to demonstrate how FUD can provide more actionable and interpretable uncertainty information in real-world scenarios.
Experiment Plan: We will evaluate FUD on complex,
Nearest Neighbor of Idea 3:
Title: Semantic Fractal Decomposition
Problem: Current uncertainty quantification methods for large language models often fail to capture the hierarchical and self-similar nature of conceptual understanding, leading to inconsistent confi- dence estimates across different levels of abstraction.
Existing Methods: Existing approaches typically focus on flat, single-level uncertainty estimates or simple hierarchical decompositions that don't 't fully capture the complex, nested nature of semantic understanding.
Motivation: Drawing inspiration from fractal geometry, where patterns repeat at different scales, we propose method that recursively decomposes concepts and queries into self-similar sub-components, allowing for a more nuanced and scale-invariant approach to uncertainty quantification.
Proposed Method: We present Semantic Fractal Decomposition (SFD), prompting technique that guides the model to recursively break down a given query or concept into smaller, self-similar components. At each level of decomposition, the model is asked to provide confidence estimate. The process continues until a predefined depth is reached or the model indicates it can no longer mean- ingfully decompose the concept. The final uncertainty estimate is then constructed by aggregating these multi-level confidence scores using novel fractal dimension-inspired algorithm. This approach allows for capturing uncertainty that may be present different semantic scales and provides more robust and consistent measure of the model's 's confidence across varying levels of abstraction.
Experiment Plan: We will evaluate SFD on a diverse set of tasks ranging from simple factual queries to complex, multi-faceted questions in domains like philosophy, science, and law. We will compare its performance against traditional flat confidence estimation techniques and simpler hierarchical methods. Key metrics will include the consistency of uncertainty estimates across related queries at different levels of abstraction, the correlation between aggrel-aggre confidence scores and actual model performance, and the interpretability of the decomposition process.
Similarity: 0.81
31

---

Published as a conference paper at ICLR 2025
A.11 OVERLAP BETWEEN AI RANKING AND EXPERT RERANKING
We show the overlap between the AI Ideas condition and the AI Ideas + Human Rerank conditions in Table We note that 17 out of the 49 ideas in the AI Ideas Human Rerank condition are also ranked as top ideas in the AI Ideas condition by the Al ranker, while the other 32 are not.
Topic
Overlap
New
Bias
Coding
Safety
Multilingual
5
5
Factuality
Math
Uncertainty
Total
Table 8: Overlap of ideas between AI Human Rerank and conditions, broken down by topic.
A.12 QUALITY CONTROL OF HUMAN EXPERT IDEAS
Each expert is instructed to choose one the seven specified topies and write one idea on it within 10 days, following the given template in the annotation document. We included an honor code statement to ask the participants to not use any AI tools in their idea writing. We collected N = 50 ideas originally and manually checked all of them for quality control. We filtered out one of them as being essentially a paraphrase of an existing paper's ss abstract. We compensated the participant nevertheless but excluded them from the review task.
32

---

Published as a conference paper at ICLR 2025
A.13 PARTICIPANT DETAILS
We show the detailed position breakdown of our 49 idea-writing participants in Table and the positions of our 79 reviewer participants in Table TO
Other
Postdoc
8%
8%
Other
Master
PhD
71%
79%
Figure 4: Positions of our idea writer (left) and reviewer (right) participants.
Position
Count
Postdoc
PhD
36
Master
Undergraduate
Research Scientist
Machine Learning Engineer
Table 9; Positions of the 49 idea writing participants.
Position
Count
Postdoc
PhD
Research Scientist 5
Master
Machine Learning Engineer
Table 10: Positions of the 79 idea reviewing participants.
33

---

Published as a conference paper at ICLR 2025
We show the institutions of the idea writing participants in Table 11
Institution
Count
Stanford University
11
University of Southern California
6
University of Maryland
3
University of Illinois Urbana-Champaign
Johns Hopkins University
3
Columbia University
2
Camegie Mellon University
2
University of Pennsylvania
Princeton University
Penn State University
Portland State University
Stony Brook University
University of Chicago
University of Washington
UC Berkeley
UCSD
Massachusetts Institute of Technology George Washington University
Yale University
University of Toronto
Georgia Institute of Technology
National University of Singapore
Peking University Tsinghua University LinkedIn
Norm Al
Table 11: Institutions of the 49 idea writing participants.
34

---

Published as a conference paper at ICLR 2025
We show the institutions of the idea reviewing participants in Table
12
Institution
Count
Stanford University
25
UC Berkeley
UT Austin
4
University of Maryland
Princeton University
3
University of Washington
University of Southern California
3
Carnegie Mellon University
3
University of Chicago
2
Johns Hopkins University
2
UCLA
2
Georgia Institute of Technology
2
University of Illinois Sarbana-Champaige
2
Tsinghua University
2
Stony Brook University
Ohio State University
National University of Singapore
University of Michigan
Dartmouth College
Massachusetts Institute of Technology
University of Pennsylvania
University of Toronto
Portland State University
Penn State University
New York University
Columbia University
UC Santa Barbara
Brown University
Amazon
LinkedIn
Norm AI
AMD
Table 12: Institutions of the 79 reviewer participants.
35

---

Published as a conference paper at ICLR 2025
A.14 14 REVIEW ASSIGNMENT STATISTICS
We list the details of the review assignment in Table 13
Metric
Mean
Min
Max
SD
# Reviews
3.8
2.0
7.0
1.3
# Conditions
2.5
2.0
3.0
0.5
# Topics
1.5
1.0
3.0
0.6
Table 13: Statistics of the review assignment.
36

---

Published as a conference paper at ICLR 2025
A.15 ADDITIONAL STATISTICAL TESTS
We present two additional statistical tests that account for potential confounders by treating each idea one data point and each reviewer data point, respectively.
Condition
Size
Mcan
Median
SD
SE
Min
Max
p-value
Novelty Score
Human Ideas
49
4.86
5.00
1.26
0.18
1.50
7.00
AI Ideas
49
5.62
5.50
1.39
0.20
1.50
8.33
0.03*
Ideas Human Rerank
5.78
6.00
1.07
0.15
3.00
8.33
0.00**
Excitement Score
Human Ideas
49
4,56
4.33
1.16
0.17
2.00
7.00
AI Ideas
49
5.18
5.50
1.33
0.19
2.50
7.33
0.08
AI Ideas Human Rerank
5.45
5.50
1.36
0.19
1.00
7.33
0.00*s
Feasibility Score
Human Ideas
49
6.53
7.00
1.50
0.21
3,00
9.00
AI Ideas
49
6.30
6,00
1.27
0.18
2.50
8.50
1.00
AI Ideas Human Rerank
49
6.41
6.50
1.06
0.15
4.00
9.00
1.00
Expected Effectiveness Score
Euman Ideas
49
5,10
5.33
1.14
0.16
3,00
7.00
AT Ideas
49
5.48
5.50
1.23
0.18
2.00
7.50
0.58
Ideas
Rerank
49
5.57
5.50
0.99
0.14
3.00
7.50
0.17
Overall Score
Human Ideas
49
4.69
4.67
1.16
0.17
2.00
6.67
A1 Ideas
49
4.83
5.00
1.34
0.19
1.50
7.50
1.00
Ideas
Rerank
5.32
5.50
1.24
0.18
2.00
7.50
0.06
Table 14: Scores across all conditions by averaging the scores for each idea and treating each idea as one data point (Test 2). Size is the number of ideas for each condition, and the p-values are computed with two-tailed Welch's t-tests with Bonferroni correction. We bold results that statistically significant (*p < 0.05;"* p < 0.01). AI ideas are judged as significantly better than human ideas in terms of novelty while being comparable all other metrics.
Mean Diff
p-value
Novelty Score
Al Ideas Human Ideas
70
0.94
0.00**
AI Ideas Human Rerank Human Ideas
65
0.86
0.00**
Excitement Score
AI Ideas vs Human Ideas
70
0.73
0.01
AI Ideas + Human Rerank Human Ideas
65
0.87
0.00**
Feasibility Score
AI Ideas vs Human Ideas
70
-0.29
0.36
AI Ideas + Human Rerank vs Human Ideas
65
-0.08
0.74
Effectiveness Score
AI Ideas vs Human Ideas
70
0.42
0.16
AI Ideas + Human Rerank vs Human Ideas
65
0.39
0.16
Overall Score AI Ideas Human Ideas
70
0.24
0.36
AI Ideas Human Rerank vs Human Ideas
0.66
0.01
Table 15: Mean score differences between AI ideas and human ideas by treating each reviewer data point (Test 3). All p-values are computed with one-sample t-tests with Bonferroni correction. We bold results that statistically significant ("p < 0.05;** p 0.01).
37

---

Published as a conference paper at ICLR 2025
A.16 MIXED-EFFECTS MODELS
One way to combine all the statistical tests above is to fit linear mixed-effects model where we treat the condition as the fixed effect and other factors including reviewer and idea as random effects, while also accounting for the differences among different topics. This way, we can rely on the regression to account for all the possible confounders the random effects. Specifically, for each metric, we fit the following linear mixed -exeeteffects model:
model = smf. mi w..mixedee xedim ("Score Condition", df,
[roup=-df["oppio]] re_forrmla==~~Cndiiiii "0 + (ReviewerID) , vc vc_formmla=(RRevieeeee "IdeaID": "o C (IdeaID) "))
This mixed-effects model analyzes the relationship between Score and Condition, while accounting for the hierarchical structure of the data. Fixed effects estimate the average effect of Condition on Score. Random intercepts for Topic allow for varying baseline scores across topics, and random slopes for Condition within each topic allow the effect of Condition to vary by topic. Additionally, variance components for ReviewerlD and IdealD account for variability in scores specific to individual reviewers and ideas, respectively.
The results are shown in Table [6] The intercepts the mixed-effects models represent the estimated mean score of the baseline condition, which in this context is the Human Ideas. The coefficients for Condition(AT Ideas] and Condition[AT Ideas models represent the difference in the mean score for each metric between the Al ideas and the baseline (human ideas). For example, a positive coefficient of 0.761 for the novelty score means that AI Ideas, on average, score 0.761 points higher than Human Ideas the novelty score metric; conversely, a negative coefficient -0.330 for the feasibility score means that AI Ideas, score 0.330 points lower than Human Ideas feasibility in the mixed-effects model represents the variability in the outcome metric that can be attributed to differences between the topics, which is relatively small in general. Similarly, the idea variance and reviewer variance in the mixed-effects model represent the variability in the outcome metric that can be attributed to differences between individual ideas and between reviewers. respectively. The reviewer variances are high in general, suggesting that there is substantial variability in how different reviewers rate the same ideas. This implies that reviewer differences play a significant role in the observed scores, with some reviewers consistently giving higher or lower ratings.
Human Rerank] in the mixed-effects
average. The topic (group) variance
Overall, the results from the mixed-effects models confirm our main conclusion that AI ideas are rated as significantly more novel than human ideas.
38

---

Published as a conference paper at ICLR 2025
Coet.
SF
p
Novelty Score
Intereept
4,826
0.217
0,000
Condition[AI Ideas]
0.756
0.331
0.023
Condition[Al Ideas
Rerank]
0,902
0,305
0.003
Idea Var
0,412
0.178
Reviewer Var
0.803
0.202
Excitement Score
Intereopt
4,493
0.212
0.000***
Condition[AI Ideas]
0.626
0.303
0.039*
Condition[AI Ideas Human Rerank]
0.879
0.298
0.003
Idea Var
0,495
0.227
Reviewer Var
0.782
0.167
Feasibility Score
Intercept
6.595
0.224
0.000
Condition[AT Ideas]
-0.300
0.294
0.307
Condition[AI Ideas Human Rerank]
-0.183
0.314
0.561
Idea Var
0,476
0.188
Reviewer Var
1.035
0.261
Expected Effectiveness Score
Intercept
5,156
0.211
0.000***
Condition[AA Ideas]
0.310
0.140
0.027
Condition[AI Ideas Human Rerank]
0,383
0,242
0.114
Idea Var
0.200
0,151
Reviewer Var
0.469
0.141
Overall Score
Intereept
4,660
0.242
0.000
Condition[AL Ideas]
0.137
0.294
0.640
Condition(AA Ideas Human Rerank]
0.610
0.320
0.056
Idea Var
0.262
0.154
Reviewer Var
1.071
0.225
Table 16: Results of linear mixed-effects models. We bold results that are statistically significant ("p < 0.05;** p < 0.01;*** < 0.001). Our main conclusion on Al ideas being more novel than human ideas still holds here.
39

---

Published as a conference paper at ICLR 2025
A.17 SCORE BREAKDOWN BY TOPIC
We show the breakdown of all scores across all conditions by topic. Note that due to the smaller sample sizes for the per-topic breakdown, most results are not statistically significant and only offer an intuitive understanding of the trends.
50140
Deheeer
Figure 5: Breakdown of all scores by topic.
40

---

Published as a conference paper at ICLR 2025
A.18 ANALYSIS OF FEXE-TEXT REVIEWS
Following recent practices of using LLMs to extract patterns from text corpora (Zhong et al. 2022] 2023), we use Claude-3.5 extract and cluster the main points from all reviews. We then manually verified and labeled each cluster.
Many reviews reinforce our quantitative finding that AI ideas tend to be more novel. For example. reviewers noted: "The idea of [..]] is quite novel in an in-context learning setting.", "The idea of exploring [..]] using an LLM-based iterative approach novel.", "The idea of [..]] when constructing prompts to improve cross-lingual transfer is one that have not heard of before.". "I like the idea to [..]], and think it will be helpful for other researchers in the community.", "Combining [...] is unique way of attempting to preserve the gist of the information while likely losing specific identifiers.", and "Safeguarding using [...] is clearly novel. Similar ideas have not been seen in the related work.".
Next, we summarize some common failure modes of AT ideas:
1. Being too vague on implementation details. For example, one reviewer noted: "I'm m not super clear on the details of this lattice and how the model will be prompted, I'm not super sure how well the model will complete these subtasks and how well- this particular structure to completing the overall task." " and another reviewer noted: ""For analyzing the effectiveness of the method, the proposal only provides a very ad-hoe + hand-wavey suggestion to compare responses across predefined questions." another case, the Al idea is criticized for not considering practical implementation details: "I think in each of the steps, there is something hard to execute. For example, in step Constellation Formation, how do we do the weighted sum?" Similarly, other reviews noted: "It's unclear how CLIP is connected to the language model and how training a CLIP model would enable the LM to understand images.", and "There's ' no mentioning defensive strategies and refine the model's responses using these strategies." Such vagueness often makes it difficult for reviewers to make confident judgments: "Because this idea too general and vague, I can't 't really answer the previous question. An idea needs certain level of details to be determined if it fits for conference/jouraal but this one misses them."
how prompt the model to generate
2. Misuse of datasets. For example: "I'm not sure about the datasets picked. StereoSet is not a QA dataset; it simply contains statements. Also, don't 't understand why Dialogue NLI responses require empathy.". "I'm concerned the datasets proposed are the right test cases for security of the code (since they are really just Ml/prrgrrmmmig problems, not system-level programming).", and "the choice of datasets might not be the best show the effect of incorporating multiple perspectives, especially TruthfulQA and ScienceQA, which seems to have a single correct interpretation and answer." In another example, the benchmark datasets chosen are considered too easy by the reviewer: "none of the chosen datasets (MATH, GSM8K, and MMLU) uses complicated math concepts".
3. Missing inappropriate baselines. For example: "The proposed method needs to be compared to simply asking the model to think of one (or several) facts about the question before answering using more turns. This could be additional baseline verify the scoring process is meaningful." and "Although the proposal includes some baselines that should be compared to, it does not mention some methods which seem do quite well with LLMss." Sometimes, "the chosen baselines may not be suitable", for example. because they are not directly comparable with the proposed method.
4. Making unrealistic assumptions. For example: "The assumption that model can (mostly) accurately flag its own hallucinations is quite tricky.", "there is a presupposed assumption that hallucinations in LLMs are ungrounded and independent of the data they trained on, which is generally not considered true", "TTh big issue for the effectiveness of the proposed method that, it asserts very strong assumptions on downstream tasks, such as there must exist only two extremes.", "Some assumptions (e.g.. [...]) are unlikely to be true in practice, especially when low-resource languages and less represented cultures are included in the study.", and "A major assumption in this approach is that the model is able to [...]. However, [...]'.
5. Being too resource-demanding Despite the fact that we explicitly prompted the agent to consider feasibility when generating ideas, some of the generated ideas are still too demandi-- For example, one reviewer noted: "The biggest issue to feasibility
41

---

Published as a conference paper at ICLR 2025
see is that the project calls for fine-tuning BLOOM (See step 5). BLOOM has 176B parameters so it's going to take quite a lot of GPUs to fine-tune. From systems perspective, see this causing delays." In some other cases, manual data annotation is being criticized for feasibility: "The bottleneck seems to be the dataset collection process if there are no existing datasets that fit the requirements of the paper.", and "the manual evaluation by native speakers or cultural experts could be time-consuming and resource-intensive"
6. Not well-motivated. For example: "Not well-motivated and there is not clear intuition that this work can work increase the factuality.", "And in general the method is not well-motivated and needs reasons why retrieving from model itself is meaningful by use cases or specific tasks.", and "The idea simply doesn't I make sense to me. Given current LLMs' ability, I'm pretty sure they can simply recite code like inserting data to binary search tree."
7. Not adequately following existing best practices. For example: "The proposal does not seem to include awareness of what has been previously tried, or more strategic ways to evaluate success/failures...
We contrast these with some of the unique strengths and weaknesses of human ideas:
Human ideas are generally more grounded in existing research and practical consider- ations, but may be less innovative. For example, these ideas might be applying existing techniques to new problems: "Multilinguality as a debiasing method has already been considered in the literature, although not necessarily in the prompt engineering framework." Sometimes people apply incremental changes to existing techniques: "The overall idea shares quite similar idea with program-of tfthhugt (PoT). The only difference is that there is an additional step where an LLM is prompted to decide whether to use code or not." Some ideas try to combine existing techniques: "Query decomposition and RAG separately are well studied, if there is no existing work that combines both (which I' I'm not aware of), then it's reasonably novel." As some reviewers noted, human ideas tend build known intuitions and results: "There are already existing works on using available lexicons to improve the translation capabilities LLMs general."
2. Human ideas tend to be more focused common problems datasets in the field. For example: "The problem of models not handling negation properly is a very common problem, especially among propriety LMs such claude-3-5-sonnet.. "The data exist. This project mainly entails plugging in these datasets to a prompt template and finetuning for a bit. There little left unspecified, and it should be quite simple to execute on.". "I haven't found any work using this idea to solve this specific problem, but [...] is definitely not new.". and "While existing works have explored the problem of calibration in long-form answers (e.g. [...]), the specific method for calibration is different."
3. Human ideas sometimes prioritize feasibility and effectiveness rather than novelty and excitement. For example, reviewers noted: "I don't 't think this will be a groundbreaking finding, but it will probably work." and "while the idea is promising and could lead to signif- icant improvements, it may not be groundbreaking enough to be considered transformative or worthy of best paper award".
42

---

Published as a conference paper at ICLR 2025
A.19 RANDOMLY SAMPLED HUMAN AND AI IDEAS WITH REVIEWS
We randomly sample four pairs of ideas from different topics to ground our numerical results with actual examples. In each pair, there the full project proposal of each idea along with the full set of reviews in the Appendix, but we list their titles. topics, and average scores here for quick reference (we reveal whether each idea is Al- Al-generated or numan-written in Appendix A.28):
Al idea and human idea. To space, we include
1. Modular Calibration for Long-form Answers: Appendix A.20 Topic: Uncertainty; Average Overall Score: 5.5
2. Semantic Resonance Uncertainty Quantification: Calibrating LLM Confidence through Multi-Path Reasoning: Appendix A.21
Topic:
Uncertainty; Average Overall Score:
3. Translation with LLMs through Prompting with Long-Form Context: Appendix A.22
Topic: Multilingual; Average Overall Score: 4
4. Linguistic Pivot Constellation: Enhancing Cross-Lingual Transfer for Low-Resource Lan-
guages and Dialects: Appendix (A.23 Topic:
Multilingual; Average Overall Score: 6.7
5. LLM Directed Retrieval Querying for Improving Factuality: Appendix A.24 Topie: Factuality; Average Overall Score: 4.7
6. Semantic Divergence Minimization: Reducing Hallucinations in Large Language Models
through Iterative Concept Grounding: Appendix [A.25]
Topic:
Factuality; Average Overall Score: 3.3
7. Autoprompting: Generate Diverse Few-shot Examples for Any Application: Appendix A.26
Topic:
Coding; Average Overall Score: 5
8. Temporal Dependency Unfolding: Improving Code Generation for Complex Stateful Sys-
tems:
Appendix A.27
Topie: Coding; Average Overall Score: 6.7
43

---

Published as a conference paper at ICLR 2025
A.20
EXAMPLE IDEA: MODULAR CALIBRATION FOR LONG-FORM ANSWERS
Modular Calibration for Long-form Answers (Part
Calibrating the confidence of Large Language Models (LLMs) when generating long-form answers, such essays and code, remains open challenge in the field of natural language processing.
1. Problem Statement:
2. Motivation: While numerous methods have been developed to calibrate the performance of LLMs multiple-chiice questions open-domain questions with short answers, extending these approaches tasks requiring lengthy responses presents significant difficulties. For instance, code generation tasks (e.g.. the HumanEval dataset), traditional confidence extraction methods perplexity may prove inadequate due to the substantial variation in answer length across questions. Verbalized confidence can be affected by instruction tuning artifacts or unclear scope, while the reliability of metrics such Expected Calibration Error (ECE) and Macro averreadeeegee Calibration Error (MacroCE) may be compromised by differences in task settings. Our
to propose novel pipeline for confidence long-form answers, drawing inspiration from methods used for
extraction and calibration of short or fixed-set answers. This approach will enable us to monitor the model's long-form answer generation process and apply targeted external augmentation when necessary, thercby enhancing both performance and efficiency.
Proposed Method: We introduce Modular Calibration, a process comprising four core steps:
Prompt the model elaborate the original question in relation the answer, identifying which components the question are addressed the long-form form response.
Extend:
2. Decompose: Instruct the LLM to break down the extended question and long-form answer into multiple modules.
3. Extract Confidence: Utilize verbalized confidence or perplexity to determine the confidence level for each modulc.
Merge: Based relationships between the modular questions/answers and the overall questions/answers, prompt the model to combine the modular confidence scores into an overall score representing the confidence in the long-form answer.
Each of these steps executed by prompting the same LLM in different ways elicit the desired response.
4. Step-by-Step Experiment Plan:
Gather Datasets: Select datasets featuring long answers with correctness annotations, Poten- tial candidates include GSM8K, Code Gen, and Essay Writing.
2. Construct Prompts:
(a) Establish baseline using direct prompting, where a query presented without special techniques.
(b) Analyze outputs to refine prompts for the Extend and Decompose steps.
(c) For the Confidence step, employ vanilla perplexity or verbalized confidence extraction. performance is unsatisfactory, explore advanced methods built upon these techniques, such as those presented in recent research (e.g., FaR paper).
Select Models: Evaluate GPT-3.5 (Text- Davinci-003) and GPT-4 from the OpenAl API, as well as the open-source LLaMA-3-70B-- --.ah.a
4. Get Results: Obtain confidence predictions from the models the selected datasets using
both baseline methods and proposed Modular Calibration approach.
5. Analyze Results: Compare the calibration performance of L.LMs using the new method against the baselines (e.g., g., the perplexity of the entire long-form answer). Conduct qualitative and quantitative analyses on each component the Modular Calibration process.
44

---

Published as a conference paper at ICLR 2025
Modular Calibration for Long-form Answers (Part
Test Case Examples:
Test Case I: Verbalized Confidence Prompting
Input: <Q> <A> Confidence (0-1)
Oulput: [Model generates confidence between and I]
Test Case 2: Modular Calibration Step (Extend)
Input: Given the answer, you extend the question and elaborate what points covered the answer?
Output: The answer these points of the question: (1) how fast A runs; (2) how runs; (3) is faster than B.
Test Case 3: Modular Calibration Step 2 (Decompose)
Input:
Please decompose the above extended question and answers into modules.
Output:
How fast runs: [relevant excerpt from the original answer]
*. How fast runs: [relevant excerpt from the original answer]
[Additional modules as needed)
Test Case 4: Modular Calibration Step (Extract)
Input: How fast runs: [relevant excerpt from the original answer] Confidence (0-1)
Oulput: 0.9;
[Additional confidence scores for other modules]
Test Case 5: Modular Calibration Step 4 (Merge)
Input: For each of these points related to question confidence is: 0.9, 0.6, ... What
the overall confidence for the whole problem?
Output: [Model generates an overall confidence score]
6. Fallback Plan: If the proposed Modular Calibration method does demonstrate improvement over the baseline, will exccute each sub-question and module individually to assess whether calibration enhanced
component. This approach will facilitate debugging of the proposed method and potentially yield interesting insights into the relationships between performancce/allibattio of decomposed modules and overall problems. Alternatively, we may analyze the model's 's ability to effectively decompose questions and answers into appropriate modules. These analyses will inform potential refinements the method or provide valuable insights into limitations and capabilities of LLMs handling complex, long form responses.
45

---

Published as a conference paper at ICLR 2025
Reviewer
Novelty: 6 (reasonably novel there are some notable differences from existing ideas and probably enough to turn into a new paper)
Rationale: Focus the long-form setting novel at the moment. The idea of obtaining modular confidence estimates different claims long-for form output, and synthesizing them into single uncertainty estimate not that complicated, but does seem to he underexplored.
Feasibility: 8 (Highly Feasible: Straightforward to implement the idea and run all the experiments.) Rationale: The only part of the project that seems challenging is obtaining correctness annotations for one of the datasets (e.g.. Essay Writing). GSM8K and code datasets like HumanEval seem like very natural long-form output settings to try out the idea. Other than this, iterating on the prompts for decomposition verbalized UQ for cach of the modules will be important, but the author mentions this.
Expected Effectiveness: (Somewhat effective: There is decent chance that the proposed idea can beat existing baselines by moderate margins on few benchmarks.)
Rationale: It's possible that first obtaining verbalized uncertainty estimates for each module, and then synthesizing into single score, will outperform the standard baselines of self-consistency over the entire long-forn form output (using majority vote the confidence score). However, I don't expect this be dramatically better. If the paper instead set out with goal actually producing UQ estimates for each claim, then almost no prior work does this, and the baselines would be less strong.
Excitement: 5 (Leaning negative: it has interesting bits but overall not exciting enough)
Rationale:
like the straightforward possible way ohtain uncertainty estimates for long-form generation with an I.M.M This means the project could produce some useful engineering artifacts, but it doesn't 't really push the idea to its logical conclusion. Therefore don't consider "exciting enough". There is some mention of ussing the uncertainty estimates to possibly condition on information" but this not fleshed the fine graigg uncortainty estimates could used selectively retrieve factual information from Wikipedia etc. on a nowlledge-intessii task.
This
it could be more interesting. For example, studying how
Overall Score:
5 (Decent idea but has some weaknesses or not exciting enough, marginally below the acceptance threshold of major Al conferences)
Rationale: like focus long-form generations. However, this proposal is very straightforward baseline and extension of existing work to the long-form form generation setting (just produce the long generation, decompose it, apply verbalized uncertainty on each claim, and finally aggregate them). I could the paper being well-cited, but don't interesting/novel angle here.
Confidence: (You are absolutely certain evaluation is correct and very familiar with relevant literature)
46

---

Published as a conference paper at ICLR 2025
Reviewer 2
Novelty: 6 (reasonably novel there are some notable differences from existing ideas and probably enough to turn into a new paper)
Rationale: While existing works have explored the problem of calibration in long-form form answers (c.g- https//aaxii.or//bb//////4/ 00.00444444 specific method calibration different. Also seems related 14251) where task was different (getting factuality score) but the idea of breaking long form generations into smaller units, evaluating each separately and then combing does seem related.
FactScore (https://arxiv. https///arii..rr/bb///2222
Feasibility: (Highly Feasible: Straightforward implement idea run experiments.) Rationale: The idea seems simple enough to implement with API access, considering all the steps involved in the method can he done via prompting with API. The proposal does mention using LLaMA3- 70B as additional model, which would require GPUs guess.
Expected Effectiveness:
(Somewhat effective: There is decent chance that the proposed idea can beat existing baselines by moderate margins on few benchmarks.)
Rationale: Since it has been shown that LLMs are quite well calibrated when asked to verbalize the confidence for short answers, I'm guessing the calibration scores would be pretty good for individual modules. Also LLMs might be decent combining confidence scores (especially with detailed instructions and some examples the prompt), so overall the method might work well. But it's unclear it would do better than the methods proposed in http://aaiiv.ooggab/////20..000
Excitement:
(laarnnng positive: exciting enough he accepted major Al conference, hut still has some weaknesses or somewhat incremental)
Rationale: If the method does work well in getting calibration for long-form answers, think that would be pretty exciting. One thing which is missing from the proposal (and why the score was not higher) was that it does not touch upon the issue that for long- long-form answers won't have binary correct/incorrect decision but answers can be partially correct.
Overall Score:
6 (Marginally above the acceptance threshold of major Al conferences)
Rationale: The overall idea makes sense to me, but the score is not higher right now because: (a) it's unclear what exactly meant by 'oduless' especially for essay writing which the proposal mentions one tasks: (b) the issue partial correctness which was mentioned above.
Confidence: 3 (You are fairly confident that the evaluation is correct)
47

---

Published as a conference paper at ICLR 2025
A.21
EXAMPLE IDEA: SEMANTIC RESONANCE UNCERTAINTY QUANTIHICATION
Semantic Resonance Uncertainty Quantification (SRUQ) (Part 1)
Problem Statement: Current uncertainty quantification methods for Large Language Models (LLMs) often rely on simple statistical measures or model-specific attributes, which may not capture the nuanced semantic uncertainties in complex reasoning tasks. This limitation can lead overconfident poorly calibrated model outputs, potentially resulting in unreliable decision-make making in critical applications.
Existing approaches typically use softmax probabilities, entropy measures, or ensemble disagreement quantify uncertainty. However, these methods often fail to capture the semantic nuances and reasoning complexities in tasks that require deep understanding and multi-step reasoning. Human experts, on other hand, gauge their uncertainty by considering 'resonates' with their broader knowledge and experience. By mimicking this process in LLMs, we can potentially develop a more robust and semantically grounded approach to uncertainty quantification.
Motivation:
well their reasoning
We propose Semantic Resonance Uncertainty Quantification (SRUQ), which prompts the LLM generate multiple independent reasoning paths for given problem, then quantifies uncertainty based on the semantic coherence and mutual reinforcement among these paths. The process involves five key steps:
Proposed Method:
Generating diverse solution attempts using different prompting strategies.
2. Cross-evaluating each solution attempt against the others, assessing logical consistency and mutual support.
3. Constructing a 'resonance graph" where nodes are solution attempts and edges represent semantic reinforvement.
Computing resonance score based on graph properties like connectivity centrality.
5. Mapping the resonance score to calibrated uncertainty estimate.
48

---

Published as a conference paper at ICLR 2025
Semantic Resonance Uncertainty Quantification (SRUQ) (Part
Step-by-Step Experiment Plan:
1. Dataset Preparation
Utilize three datasets covering different reasoning tasks:
(a) GSM8K for mathematical problem-solving
(b) EntailmentBank for logical deduction
(c) HotpotQA for multi-hop question answering
Split each dataset into train, validation, and test sets if already done.
2. Baseline Implementation
Implement three baseline uncertainty quantification methods:
(a) Softmax probabilities
(b) Monte Carlo Dropout
(c) Ensemble disagreement (using different few-shot prompts)
Generate predictions and uncertainty estimates on the validation and test sets for each bascline.
3. SRUQ Implementation
(a) Generate diverse solution attempts using different few-shot prompts and temperature settings.
(h) For each pair of solutions, prompt the L.M evaluate their consistency and mutual support.
(c) Construct the resonance graph using the pairwise evaluations.
(d) Compute the resonance using graph centrality measures (e.g.. PageRank).
(c) Map the resonance score to calibrated uncertainty estimate using isotonic regression on the validation
4. Evaluation
Compare SRUQ against the baselines using the following metries:
(a) Expected Calibration Error (ECE)
(b) Brier score
(c) Area Under the Precision-Recall Curve (AUPRC) for uncertainty ranking
Evaluate the correlation between uncertainty estimates and actual errors.
5. Analysis
Visualize the resonance graphs for high and low uncertainty examples.
Analyze the relationship between graph properties and prediction accuracy. Investigate cases where SRUQ significantly outperforms or underperforms compared to haselines.
6. Ablation Studies
Vary the number of solution attempts. Compare different graph centrality measures.
Evaluate the impact of the cross-evaluation step.
Generalization Test
Test the generalization of SRUQ on out-of-distribution samples by applying the method trained on one dataset to examples from the other datasets.
49

---

Published as a conference paper at ICLR 2025
Semantic Resonance Uncertainty Quantification (SRUQ) (Part 3)
Test Case Examples:
Baseline Example:
Input: Q: If train travels at 60 miles per hour, how far will it travel in 2.5 hours?
Softmax Output: The train will travel 150 miles in 2.5 hours. (Confidence: 0.92)
Explanation: The softmax prohability is high, but it does not capture the reasoning
process or potential uncertainties the calculation.
SRUQ Example:
Input: Q: If train travels at 60 miles per hour, how far will it travel in 2.5 hours?
Solution Attempts:
1. Distance = Speed x Time
Distance 60 miles/hour hours
Distance 150 miles
hour, the train travels 60 miles In hours, it's 120 miles
In 0.5 hours, it's 30 miles
Total: 120 + 30 150 miles
3. Let's break it down:
hour: 60 miles
hour: 60 miles
0.5 hour: miles
Sum: 60 + 60 + 30 150 miles
Cross-Evaluation All three solutions are consistent and mutually supportive. They use different approaches but arrive at the same result.
Resonance Graph: Fully connected graph with high edge weights
Resonance Score: 0.95
Calibrated Uncertainty: 0.05
Final Output: The train travel 150 miles hours. (Uncertainty: 0.05)
Explanation: SRUQ generates multiple solution paths, evaluates their consistency, and quantifies uncertainty based on their semantic resonance. The high resonance score indicates low uncertainty, which then calibrated to provide final uncertainty estimate.
analysis paper exploring why semantic resonance might not capture uncertainty effectively. We could investigate the quality and diversity of generated solution attempts, potentially improving the prompting strategies. Additionally, we could examine the effectiveness of the cross evaluation step, possibly incorporating ex- structured reasoning. Furthermore, graph properties and actual uncertainty, which might reveal insights about how LLMs represent confi- dence internally. could also consider combining SRUQ traditional uncertainty quantification methods, creating hybrid approach that leverages both statistical and semantic information.
Fallback Plan: If SRUQ does not significantly outperform baselines,
pivot
ternal knowledge
could explore the relationship between
50

---

that the proposed method should do Published as a conference paper at ICLR 2025
Reviewer
Novelty: 6 (reasonably novel there are some notable differences from existing ideas and probably enough to turn into a new paper)
Rationale: haven't seen (and couldn't 't find) any prior work which exactly has the same idea in this proposal. The proposed idea is definitely related using consistency among multiple solutions estimate uncertainty (e.g. https://arxiv.orr http://ariiv.r//aa////0003 18711 does this across solutions decoded from different layers) but have not seen the idea of constructing resonance graph and using graph properties to estimate uncertainty.
Feasibility: (Highly Feasible: Straightforward implement idea run the experiments.) Rationale:
The proposed method, SRUQ, should be pretty easy to implement given that LLM API access is abundant. SRUQ involves multiple steps all of which can he done through prompting via API -getting multiple solutions, prompting LLMs get consistency score between each pair of solutions The parts which cannot be implemented through API are the baselines c.g. Monte Carlo dropout, and would require GPUs. do a comparison to baselines, iimaiine SRUQ will also have to be done on open models which could also require GPUs.
Expected Effectiveness: (Somewhat effective: There is decent chance that the proposed idea beat existing baselines by moderate margins on few benchmarks.)
Rationale: Although the proposal includes some baselines that should be compared to, it does not mention some methods which seem to do quite well with LLMs (especially getting better with scale) e.g. methods like P(True) (htps://arsiv.or//ab///2 207.0000022222221 verbalized confidence (httss//aaxivxorr/ab///..0 14975). It's not clear/obvious better than these baselines.
6 (Learning positive: exciting enough to be accepted at major Al conference, but still has weaknesses somewhat incremental)
Excitement:
Rationale: While the method novel and feasible, I'm not too excited by it since some of the other existing methods out there mentioned above (like https//arxiv.or//bb//222 0.222,,, 14975) are much simpler and work quite well. Compared to that SRUQ is more complex, and hence maybe has less chance of being very impactful (unless it works really better).
Overall Score: 6 (Marginally above acceptance threshold of major Al conferences)
Rationale: The above accept score is assuming the idea does work better than the baselines on some category of tasks. Overall, given that the idea is novel, the proposal includes comparison to other baselines well analysis & ablations, think that could be enough get accepted into an Al conference.
Confidence: You are confident but not absolutely certain that evaluation correct)
51

---

Published as a conference paper at ICLR 2025
Reviewer 2
Novelty: 6 (reasonably novel there are some notable differences from existing ideas and probably enough to turn into a new paper)
The proposed approach shares some similar ideas with self-consistency (which suggests the consistency of sampled LLMs outputs is relatively well calibrated). But the approach more generalized and fine-grained than existing work if approach uses more advanced 'mutual support evaluation' beyond simply comparing the final answers.
Rationale:
(Moderately feasible: It can probably be executed within the given time frame but would require careful planning, efficient use APIs or some advanced computational strategies to overcome the limited GPU resources, and would require some modifications to the original proposal to make work.)
Feasibility:
Rationale: There lacks some important details in terms of the coss-evaluation part. How is the mutual support evaluated (by prompting or some other methods?). This part is crucial for implementing the whole pipeline this approach.
Expected Effectiveness:
(Somewhat effective: There is decent chance that the proposed idea can beat existing baselines moderate margins few benchmarks.)
Rationale: think has some chances beat proposed baselines. If cross-evaluation part properly executed. Again, the success of this proposal is highly dependent on that part.
Excitement:
6 (1 (Learning positive: exciting enough be accepted at major Al conference, but still has weaknesses somewhat incremental)
Rationale: If this idea actually works, at least tells something new about how to use multiple samples to provide better confidence estimation than simple consistency. But the idea itself still somewhat incremental given the existence of current consistency- -baseed calibrators.
Overall Score: 6 (Marginally above the acceptance threshold of major Al conferences)
Rationale: Overall there are some incremental contributions, but not too exciting. The algorithm itself can be neat. think it can be worth a borderline acceptance.
Confidence: 4 (You confident but absolutely certain that the evaluation correct)
52

---

Published as a conference paper at ICLR 2025
Reviewer 3
Novelty: 6 (reasonably novel there are some notable differences from existing ideas and probably enough to turn into a new paper)
Rationale: I think the idea is reasonable and indeed identifies some limitations of current works uncertainty estimation. However, the consistency between reasoning paths somehow similar to self-consistency reasoning from Google SelfCheckGPT.
Feasibility: 7
Rationale: 1 think it could be easy implement and quickly be tried PhD students undergrads. Also, in test case example, setting straightforward and well-defined.
(Somewhat effective: There is decent chance that the proposed idea can beat existing baselines by moderate margins few benchmarks.)
Expected Effectiveness:
Rationale: Based my experience, the consistency-bas grounded, can work pretly in current uncertainty estimation questions. believe working this on the reasoning path level could also work to some extent.
methods, although not fully thcoretically
Excitement: 6 (Learning positive: exciting enough be accepted at major Al conference, but still
some weaknesses or somewhat incremental) Rationale: Overall, this idea identified a good research question, although the method might not be very exciting to me.
Overall Score: 6 (Marginally above the acceptance threshold of major Al conferences) Rationale: The novelty the actual application of this method in area limited, could an inspiring idea.
Confidence: 4 (You confident but not absolutely certain that the evaluation correct)
53

---

Published as a conference paper at ICLR 2025
A.22
EXAMPLE IDEA: TRANSLATION WITH LLMs THROUGH PROMPTING WITH LONG-FORM CONTEXT
Translation with LLMs through Prompting with Long-Form Context (Part 1)
Problem Statement: Stable generation text in low-resource languages an unsolved issue in large language models.
While LLMs can often produce surprisingly good translations despite not being explicitly trained for this task, this does hold for lower-resource languages. LLMs are both more likely to generate tar-tt text (text another language than intended) when prompted to translate to a lower-resource language, and show increased instability in translation quality across prompt templates in lower-resource languages.
Motivation:
Our proposed method investigates use of long-form templates to improve generated translation quality and reduce off target translations in lower-resource languages. We propose to provide additional prompt context by translating multi-sentent input, with additional views of the target language with the langid template provided as context. We do multiple stages:
Proposed Method:
Querying the language model to first generate a paragraph containing the source sentence to be translated.
Prepending monolingual text in the target language, with langid: tags, above the translation prompt.
3. Presenting both these additional sources of content, prompting the L.M for translation.
Step-hy-Step Experiment Plan:
1. Choose datasets: Evaluate the FLORES-20 datasets, which allow for wide language coverage on the Wikipedia domain, as well the WMT-21 test for news and law/medical domain.
2. Choose languages: Opt for English-centric translation with:
high-resource languages with diffcrent scripts (French, German, Russian, Chinese, Japanese)
mid-resource languages (Farsi, Vietnamese, Arabic, Korean, Hebrew)
low-resource languages with considerably lower likelihood of incidental bilingualism (Gujarati, Thai, Tajik, Sindhi, Pashto)
3. Choose models:
Include the API-based GPT-3.5 (Text-Davinci-O00 and GPT-4 model from OpenAl and Gemini from Google, as well as the open- open-weight LLaMA-3, Gemma, and Aya models which enable additional analysis.
4. Gather translation results:
Systematically compare standard MT prompt templates proposed method across different models and language pairs. Additionally ablate the steps of the new method (removing langid templates; replacing langid templates with endonymic langid tags: provide only the generated paragraph; only the monolingual content).
Perform analysis: Evaluate whether the new method improves the performance of L.LMs in these tasks as compared to the baselines using multiple standard automatic metrics MT (chrF, COMET, BLEU) and token-level LID to measure off-target translations. Assess which component(s) are necessary for this improvement and whether not there are changes across language pair direction and language resource levels scripts.
54

---

Published as a conference paper at ICLR 2025
Translation with LLMs through Prompting with Long-Form Context (Part 2)
Test Case Example:
Baseline Prompt Template:
[English] This is an example. [Spanish]
Baseline Prompt Expected Output:
[Spanish] Este es un ejemplo.
Proposed Prompt Input (step 1):
Write paragraph containing the following sentence: [Eaglish] This is an example.
Expected model output (step 1):
This is example link. Italics and Bold: use asterisks and underscores to indicate spans of emphasis. Use asterisk (or underscore) for italics two asterisks (or two underscores) for bold. For example...
Proposed Prompt Input (step 2):
[Spanish] Computadoras simples 1o suficientemente pequeñas para residir en dispositivos móviles.
[Spanish] Las computadoras portátiles, tales come tabletas, netbooks, notebooks, ultrabooks, pueden ser alimentadas por pequeñas baterias.
[8panish] Las computadoras personales sus diversas formas son iconos de la llamada era de información y son lo que la mayoria de la gente considera como kcomputadorav.
Translate this paragraph from Knglish to Spanish: [8nglish] This is an example link. Italics asterisks and underscores to indicate spans of emphasis. Use one asterisk (or underscore) for italics and two asterisks (or two underscores) for bold. For example...
Hold: use
[Spanish]
Proposed Prompt Expected Output:
[Spanish] Este es un enlace de ejemplo. Cursiva negrita: utilice asteriscos y guiones bajos para indicar intervalos de énfasis. Utilice asterisco guión bajo) para cursiva y dos asteriscos dos guiones bajos) para negrita. Por ojemplo...
55

---

Published as a conference paper at ICLR 2025
Translation with LLMs through Prompting with Long-Form Context (Part 3)
Test Case Example (Continued):
Proposed Prompt Input (step 3):
Which of the following sentences are translations of the following English sentence? Multiple sentences chosen. [Baglish] This is an example.
1. Este es un enlace de ejemplo.
2. Cursiva y negrita: utilice asteriscos y guiones bajos para indicar intervalos de énfasis.
3. Utilice un asterisco (o guión bajo) para cursiva y dos asteriscos (o dos guiones bajos) para negrita.
4. Por ejemplo...
Proposed Prompt Expected Output:
The sentence "This is an example." can translated to Spanish as:
1. Este un ejemplo.
2. Por ejemplo...
These two options correctly translate the meaning of "This is an example." into Spanish.
If the proposed method does not help compared to the baseline, analyzing the results of step would likely provide further insights into how the template should be modified. In addition potentially identifying off-targ off-target errors, may be that the model is unable to identify correct translations even if they have been generated, and results are likely to vary across languages based on their training data. Using the generated paragraph as provided context and still querying the model to translate at only the sentence level could be compared. Restricting monolingual text to be retrieved text sentence could be explored. Adding few-shot examples in the prompt
6. Fallback Plan:
within the domain of the comparing other MT prompt templates may also help debug the proposed method. Including an additional query where the model is first asked to label each generated token by langid and then asked to re- translate the source including those tokens which are correctly labelled in target may reinforce langid and guide generation in the target language. Performing layer-wise analyses of likelihood of generating the next token in- in-language and in-script for open-weight models may also help debug where and why off-target issues persist.
56

---

Published as a conference paper at ICLR 2025
Reviewer
Novelty:
(somewhat novel there are differences from existing ideas but not enough to turn into new paper)
Rationale: While I'm not aware of papers that have used this exact prompting strategy. I don't 't think that this proposal will enough to justify a publication. think that there should variety strategies suggested + an analysis of multiple prompting strategies rather than suggesting one strategy. think that a thorough analysis of the effects of additional context langids could potentially turn this into paper.
Feasibility: 9
Rationale: Such a project that only uses LLM APIs could executed very quickly without much expertise in coding/architectuse The only time-consuming part might be iterating and adjusting the prompts in the ablation studics.
Expected Effectiveness: 7
Rationale: think that this proposal could work well guide LLMs to translate the desired target language, since this is known problem with current prompt-based MT strategies (as the writers have suggested).
Excitement: 5 (Leaning negative: has interesting bits overall exciting enough)
Rationale: I'm not sure how well this method will transfer to future models, and this could be a limiting factor in the longevity of this research. (But this is a limitation of all prompting research...)
Overall Score: 5 (Decent idea but has weaknesses
exciting enough, marginally below the
acceptance threshold major conferences)
Rationale: think that the work should focus on the ablation studies and comparison of multiple prompting strategies analysis, rather than focusing one new strategy.
Confidence: 3 You fairly confident that the evaluation correct)
57

---

Published as a conference paper at ICLR 2025
Reviewer 2
Novelty: (not novel at all - there are many existing ideas that are the same)
Rationale: There are multiple existing works prompting LLMs on low-resource transla- tion, usually using few-shot demo.
wttprr/cednngs.....rrss///////// https//aaxxiv.og//dd/22... 14857 Also work explaining why few-shot prompt would work: https//arkiv..or//////233....
Feasibility: 5 (Moderately feasible: It can probably be executed within the given time frame but would require careful planning, efficient of APIs some advanced computational strategies overcome the limited GPU resources, and would require some modifications work.)
original proposal to make
Rationale:
The prompting experiment is mostly feasible given one can afford the API calls. The model, prompts, and evaluation metries conerete, although unclear if the proposed experiment is useful for proving the rescarch idca, c.g., few high-resource languages listed for research idea that focuses on low-resource languages.
Expected Effectiveness: 3 (Low Effectiveness: The idea might work in some special scenarios but you don't I expect to work in general.)
Rationale: The proposed experiment can help find a set relatively high-pertonn performing prompts, but it unclear among the prompts proposed if any of them will bring any improvement.
Excitement: 3 (Mediocre: this idea makes marginal contributions and is very incremental)
Rationale: The ability do prompting/few-shot translation fundamentally tied the training data, see https://arxi.orrgpdfff ortprii..rg//pd//230... 10266, so trying to solve this problem from the prompting space inherently limited.
Overall Score: 3 (Clear rejection for major Al conferences)
Rationale:
There similar work on prompting LLMs generate translation low- re-oouroucc languages, hence the idea is not very novel. Morcover, terms of the goal to generate figh-quality low- ww-resourcc translation, the gains likely are not going to come from prompting.
Confidence: 4 (You confident but not absolutely certain that the evaluation correct)
58

---

Published as a conference paper at ICLR 2025
A.23 EXAMPLE IDEA: LINGUISTIC PIVOT CONSTELLATION; ENHANCING CROSS-LINGUAL TRANSFER FOR LOW-RESOURGE LANGUAGES AND DIALECTS
Linguistic Pivot Constellation (LPC): Enhancing Cross-Lingual Transfer for Low- Resource Languages and Dialects (Part
1. Problem Statement: Large language models struggle with cross-lingual transfer, especially for low-resource languages and dialects. This limitation hinders the models' ability perform well multilingual tasks involving languages, potentially exacerbating digital language divides.
2. Motivation: Current approaches often rely parallel data multilingual pretraining, which are limited for many language pairs. Inspired by how polyglots leverage similarities between known languages learn new ones, we propose creating network conceptual bridges across languages. This method could potentially overeome the limitations of existing approaches by loveraging the model's 's broad knowledge to create connections between known and unknown linguistic territories.
We introduce Linguistic Pivot Constellation (LPC), novel prompting technique that constructs dynamic network of linguistic pivot points.
Proposed Method:
given task, LPC first identifies conceptually similar languages or dialects to the target language. It then generates a constellation of prompts in these pivot languages, each capturing different aspect of the task. The model is guided 'triangulate" the correct response by considering these multiple perspectives. For example, translate dialect, I.P. might use prompts in related languages, regional lingua francas, and etymologically connected languages.
4. Step-by-Step Experiment Plan:
Data Collection
Gather datasets for translation and question-answering tasks across diverse set of low-resource languages and dialects.
Utilize the FLORES- 101 dataset for machine translation and the TyDi QA datasct for question answering.
2. Baseline Implementation
Implement standard few-shot prompting and existing cross-lingual transfer methods (e.g.. zero-shot cross-lingual transfer) as baselines.
3. LPC Implementation
(a) Create language similarity matrix based on language families and geographical prox- imity.
(b) Implement a function to select the most relevant pivot languages for given target language.
(c) Design prompts for each pivot language that capture different aspects of the task.
4. Prompt Construction
(a) Select 3-5 pivot languages based on the similarity matrix.
(b) Generate task task-specific prompts in each pivot language.
(c) Combine these prompts into 'constellation' prompt that includes original task the target language.
5. Model Selection
Use GPT-4 as the primary model for experiments.
Test GPT-3.5-turbo for comparison.
6. Experiment Execution
(a) Run the baseline methods.
(b) Run the LPC method with varying numbers of pivot languages (1, 3, and 5).
(c) Record the model outputs and performance metrics.
59

---

Published as a conference paper at ICLR 2025
Linguistic Pivot Constellation (LPC): Enhancing Cross-Lingual Transfer for Low- Resource Languages and Dialects (Part 3)
4. Step-by-Step Experiment Plan (Continued):
7. Evaluation
Evaluate the results using task- specific metrics:
BLEU score for translation tasks
Fl score for question answering tasks
8. Analysis
Analyze the effectiveness of different pivot language combinations and the method's scalability to extremely low-resource scenarios.
Compare LPC performance against baselines across different language families and resource levels.
5. Test Case Examples:
Test Case
Translate the following Sicilian sentence to English: "Unni c'è fumu c'è focu.'
Baseline Prompt Input:
Baseline Prompt Expected Output: Where there's smoke, there's fire.
Proposed Prompt Input:
We will translate Sicilian sentence to English. To help with this task, consider the following related phrases:
In Italian:
"Dove d'è fumo c'è fucco.'
In Neapelitan: Addd ce sta ' furmo ce sta 'o ffucco.'
In Latin: Ubi fumus, 1bi ignis.'
Now, translate the Sicilian sentence English: 'Unni e'è fumu e'è focu.'
Proposed Prompt Expected Output: Where there's smoke, there's fire.
Explanation: The LPC method provides context from related languages (Italian, Neapolitan, and Latin), which help the model better understand and translate the Sicilian phrase. This is especially useful for low-resource languages like Sicilian, where direct translation data might be limited.
If the LPC method does not significantly outperform baselines. will pivot the project towards in-depth analysis cross-lingual transfer mechanisms. We will investigate the relationnhip between language similarity and transfer offectiveness, the impact of pivot language selection on performance, and how different aspects of language (lexical, syntactic, semantic) transfer across the constellation. This analysis could provide valuable insights into the strengths and limitations of large language models in cross-lingual tasks, potentially informing future research directions in multilingual Natural Language Processing.
Fallback Plan:
60

---

Published as a conference paper at ICLR 2025
Reviewer
Novelty:
Rationale: The idea of using linguistic similarity matrix to form conceptual bridges when constructing prompts improve cross- lingual transfer one that have not heard of before. think this could be an interesting way of leveraging existing information about related languages for NLP tasks in general.
Feasibility: 8 (Highly Feasible: Straightforward to implement the idea and run all the experiments.) Rationale: think the idea makes sense, but more details should be shared ahout how exactly this language similarity matrix is constructed and what algorithms will be used for determining language similarity. More details should provided how the prompts for different languages will obtained and how the data will be collected, which might be a time bottleneck.
Expected Effectiveness: (Somewhat effective: There is decent chance that the proposed idea beat existing baselines by moderate margins on few benchmarks.)
Rationale: think this idea could work well just by providing more context different languages. The effectiveness sounds like it might be highly variable on the selection of pivol languages, though.
Excitement: 7
Rationale: I think that this could be interesting beyond context of prompting, such as the use of pivot languages in traditional machine translation.
Overall Score: 7 (Good idea, would he accepted by major Al conferences)
Rationale: think that the idea sufficiently novel, and if it is executed well with good results, could produce a quality paper a top NI.P conference.
Confidence: 3 (You are fairly confident that the evaluation is correct)
61

---

Published as a conference paper at ICLR 2025
Reviewer 2
Novelty: (clearly novel major differences from all existing ideas)
Rationale: The LPC method introduces novel way of leveraging related languages and dialects to improve longull transfer. While rross-lingual transfer and language similarity have been explored, the idea dynamically creating constellation of prompts using pivot languages for specific tasks fresh and innovative approach.
Feasibility: 5 (Moderately feasible: It can probably be executed within the given time frame but would require careful planning, efficient of APIs some advanced computational strategies overcome the limited GPU resources, and would require some modifications work.)
original proposal to make
Rationale: implementing LPC could be challenging due to the complexities involved in selecting optimal pivot languages and designing effective prompts for each. While the concept is sound, the practical exccution such as building the language similarity matrix and dynamically generating prompis-may require substantial effort and experimentation.
Expected Effectiveness:
(Somewhat effective: There is decent chance that the proposed idea can beat existing baselines moderate margins few benchmarks.)
Rationale: The LPC method has the potential to improve cross- longull performance, especially in low-resource languages. By leveraging linguistic similarities, the model might better understand and translate languages with limited training data.
Excitement: 7
Rationale: The L.P. method exciting hecause it tackles a critical challenge multilingual NLP- ina--iproiing performance for low-resource languages. If successful, it could significantly en- hance the accessibility and usability of Al models across diverse linguistic contexts, particularly in underrepresented languages.
Overall Score: 6 (Marginally above the acceptance threshold of major Al conferences)
Rationale: The idea is promising candidate for exploration in the field of multilingual NLP. It introduces novel approach that could potentially improve cross-lingual transfer, particularly for low-resource languages and dialects. However, the challenges in implementation and the uncertain effectiveness the method warrant a cautious overall rating.
Confidence: 4 (You are confident but not absolutely certain that the evaluation is correct)
62

---

Published as a conference paper at ICLR 2025
Reviewer 3
Novelty: (clearly novel major differences from all existing ideas)
Rationale: Leveraging language similarity is often quite well studied in machine translation, but there hasn't 't been studying using similar language demonstration in multilingual in in-context learning. would interesting to how the model behavior change with different pivots.
Feasibility: 8 (Highly Feasible: Straightforward to implement the idea and run all the experiments. Rationale: The implementation will mostly involve building the similarity matrix and formatting the prompts. The similarity matrix should be able get from some existing works. The prompt formatting and experiments part should be pretty straightforward with enough API quota.
Expected Effectiveness: (Somewhat effective: There is decent chance that the proposed idea can beat existing baselines by moderate margins few benchmarks.)
Rationale: The idea is pretty interesting, but it's exactly sure whether similar languages model, since it requires model understand similarity between languages and reason over the relationship between target language and the given languages.
informative enough
Excitement: 8 (Exciting: would deepen the community's understanding make major progress in this research direction)
Rationale: would be informative to the community to see whether such demonstration can lead to good performance for in-conexxt learning. Even if this idea doesn't 't work, the analysis will be quite informative.
Overall Score: (Good idea, would he accepted by major conferences)
Rationale: This work studies an important problem for the multilingual community. The experiment results and analysis will be quite informative for multilingual in in-context learning.
Confidence: 4 You confident but absolutely certain that the evaluation correct)
63

---

Published as a conference paper at ICLR 2025
A.24 EXAMPLE IDEA: LLM DIRECTED RETRIEVAL QUERYING FOR IMPROVING FACTUALITY
LLM Directed Retrieval Querying for Improving Factuality (Part 1)
Problem Statement: Large language models can generate flexible, long-form language generations, but LLM--gnnrated responses often contain hallucinated or factually inconsistent content. Particularly in high-risk settings, there is need for methods to improve the factuality of LLMs.
Motivation: A common framework for improving the factuality of LLM generations is retrieval augmented generation (RAG). a RAG framework, retriever takes a query as input and retrieves external knowledge from a high-quality knowledge base from reliable sources. The retrieved content is incorporated into the prompt for generating the response. One issue with this approach is that the quality of the generation can be bottlenecked by the quality of the retrieved content. Retrieval can be challenging for tasks where query objective underspecified additional reasoning (or multi-step reasoning) on the query is required to retrieve content that supports the query.
3. Proposed Method:
Our method refines the query by using LLM decompose the problem into qub--uuetionns and generate candidate answers expand each que-quesiion. The key steps include:
1. Decomposing the original question into sub-questions using an L.M.M
Generating candidate answers for cach sub-question using the LLM.
3. Expanding each sub-question with generated candidate answers to create retrieval queries.
Retrieving passages each expanded query.
5. Filtering retrieved passages based on retrieval model score.
Aggregating filtered passages across sub questions.
7. Prompting the generative LLM with the aggregated passages as context to answer the original question.
Step-hy-Step Experiment Plan:
1. Choose RAG datasets where the retrieval task has underspecifficc/uiqqee objectives requires multi-hop reasoning. such as BIRCO and HotpotQA.
2. Select a retriever, such as an E5 or BGE model, and generative I.M,, such as GPT or I.L.M-3.
Establish Baseline:
(a) Use the example question as the query to the retriever to retrieve relevant content from the retrieval passage pool.
(b) Construct prompt that provides the retrieved context passages and the question.
(c) Prompt the generative L.M to answer the question using the context.
Implement Proposed Method:
(a) Prompt the generative LLM to decompose the question into sub-questions.
(b) For each sub-question, prompt the generative LLM to generate candidate answers.
(c) Use semantic similarity to cluster the generated candidate answers and sample for semantic diversity.
(d) Construct retrieval queries by expanding each sub-question with sampled candidate answers.
(c) Retrieve passages using each query and aggregate results for each sub nub-question.
()) Deduplicate retrieved passages and filter based on retrieval model score.
(g) Prompt the generative LLM with filtered passages as context to answer the original question.
64

---

Published as a conference paper at ICLR 2025
L.LM Directed Retrieval Querying for Improving Factuality (Part
Test Case Examples:
Test Case 1:
Original Question: In which region is the village after which lager "Fucking Hell" is named?
Baseline:
Retrieval Query: In which region is the village after which lager "Fucking Hell" named?
Fucking Hell is German palc lager, Pilsner, with an alcohol content of 4.9%. is named after Fucking, the previous name of the village of Fugging in Austria; hell the German word for 'pale' and typical description of this kind of beer. The beer's name was initially controversial. Both the local author- ities in Fucking and the Europcan Union's 's Trade Marks and Designs Registration Office initially objected the name. It was eventually accepted and the lager is sold internationally.
Retrieved Passage:
Prompt: Given the retrieved passage(s) as context and the question, answer the question using the context.
Answer:
The village after which the lager "Fucking Hell" is named is located in Austria.
Proposed Method:
8 sabb-Quettions:
1. What village is the lager "Fucking Hell" named after?
2. In which country is this village located?
3. In which specific region state within that country is the village located?
* Example Retrieval Query: What village is the lager "Fucking Hell" named after?
The lager "Fucking Hell" is named after the village previously known Fucking, which is now called Fugging, in Austria.
* Retrieved Passages:
1. Fucking Hell is German pale lager, Pilsner, with alcohol content of 4.9%. named after Fucking, the previous name of village Fugging Austria; hell is the German word for 'pale' and a typical description of this kind of beer. The beer's 's name was initially controversial. Both the local authorities in Fucking and the Europcan Union's Trade Marks and Designs Registration Office initially objected the name. internationally.
eventually accepted and lager sold
2. Fugging, spelled Fucking until 2021, is an Austrian village in the municipality of Tarsdorf, located the Innviertel region of western Upper Austria. is
(21 mi) north of Salzhurg and 4 km (2.5 mi) east Inn river, which forms part of the German border.
Prompt: Given the retrieved passage(s) as context and the question, answer the
question using the context.
Answer: The village after which the lager "Fucking Hell" is named is located in the Innviertel region of western Upper Austria.
Fallback Plan: If the proposed method docs not satisfy the success criteria, alternative approaches could be explored. These may include quantifying the difficulty of various examples and analyzing whether this correlates with method improvement. The method is likely be effective for questions about esoteric facts, where model or its generated answers are more likely to disagree. Additionally, the method may be more beneficial for questions requiring information from multiple passages. Further analysis could help debug why the proposed method did not work, informing alternative new methods transforming the project into analysis paper by offering interesting ablations and insights.
likely to have internal knowledge
answer.
65

---

Published as a conference paper at ICLR 2025
Reviewer
Novelty: (not novel at all there are many existing ideas that are the same) find this idea is extremely similar to "GenDec: A robust generative method for Multi-hop reasoning" by Wu et al. (2024). Link: https:/aankvvorrghh////2222..2. 1666v
Rationale:
Feasibility: 8 (Highly Feasible: Straightforward implement idea run experiments.) Rationale: Technically, this idea can be quickly re-produced based on the aforementioned paper. Though the motivations and evaluations are different from the existing work, it shouldn' 't take too long ligure them out.
Expected Effectiveness: (Low Effectiveness: The idea might work in some special scenarios but you don't I expect it to work in general.)
Rationale: Given that the idea is too similar to existing one, the author may need to create new but related idea follow-up study of the aforementioned paper. This idea does have different motivation from aforementioned one, so uses different evaluation methods, though.
Excitement: 2
Rationale: Reviewers may argue the originality and novelty of this idea if it's submitted to venue. They may not find exciting, either.
Overall Score: (Critically flawed, trivial, or wrong. would be waste of students' time to work on it)
Rationale: The students should probably think one-step-further of the existing study and they may eventually find way to improve the existing system.
Confidence: (You are absolutely certain that the evaluation is correct and very familiar with the relevant literature)
Reviewer 2
Novelty: 6 (reasonably novel there are some notable differences from existing ideas and probably enough to turn into a new paper)
Rationale: Query decomposition and RAG separately are well studied, if there is no existing work that combines both (which I'm not aware of), then it's reasonahly novel.
Feasibility: 10 (Fasy: The whole proposed project can he quickly executed within a few days without requiring advanced technical skills.) Rationale: It's just series of prompting which should be easy for CS PhD student.
Expected Effectiveness: (Probably Effective: The idea should offer some significant improvement over current methods on the relevant benchmarks.)
Rationale: This method involves multiple finc-grained retrieval operations and should naturally outperform existing retrieval methods without decomposition.
(Learning positive: exciting enough be accepted at major AI conference, still has some weaknesses or somewhat incremental) Rationale: Although believe in the effectiveness of the proposed method, the high latency compared baselines is concern training end-to-end model to reduce latency might be good add-on.
Excitement:
(Good idea, would he accepted by major Al conferences) Rationale: This is good idea. If there is no identical existing work and the authors conduct compre- hensive experiments, it would be good paper.
Overall Score:
Confidence: (You are confident but not absolutely certain that evaluation correct)
66

---

Published as a conference paper at ICLR 2025
Reviewer 3
Novelty: 5 (somewhat novel there are differences from existing ideas but not enough to turn into new paper)
Rationale: The idea aims to tackle question by breaking down and solving it one with RAG. seems be more specialized way of CoT with RAG.
Feasibility: (Moderately feasible: It can probably be executed within the given time frame but would require careful planning, efficient use of APIs or some advanced computational strategies to overcome the limited GPU resources, and would require work.)
modifications the original proposal make
The idea assumes question can be broken down into subquestions where each subquestion is independent of the others. In cases where they are not independent, the method might suffer from issues or inefficiency. But maybe the distribution of these questions is more like long tail and predominantly questions that can be easily broken down. And there case where the question high-level mathematics and difficult to the point where it breaks down into non-linear scale of question text token?
Rationale:
Expected Effectiveness: 5 (Somewhat ineffective: There might be some chance that the proposed idea can work better than existing baselines but the improvement will marginal or inconsistent.)
Rationale: The main question is how the sub sub-questions are created. We can break the question into conditioned parts from p(go (o.......... ....g).....tt.. p(gg (go, ...91 where assume them to be dependent, or we I.IM reason about their dependency. We can also ask the question by asking leveled sub-questions like "where is this person from" into "which country is this person from", "which city this person from "which district is this person from". concern is that different methods might affect the performance differently.
Excitement:
(Learning positive: exciting enough be accepted at major AI conference, but still some weaknesses somewhat incremental)
Rationale: The idea seems exciting as prevents LLM from shortcutting the question and hallucinating. But it needs more method formulation on how the question should be broken down. The very baseline implementation will just degrade to CoT reasoning with RAG for each step. Because this could just be subset of CoT methods
Overall Score:
6 (Marginally above the acceptance threshold of major Al conferences)
Rationale: believe there could be more comparison with CoT motivation. Why should this be better with prompting the model step by step using RAG, and why are they different? And for problem formulation, would be great we can list more edgy examples of how questions help pilot the prompting methods.
divided
Confidence: 4 (You are confident but not absolutely certain that the evaluation is correct)
67

---

Published as a conference paper at ICLR 2025
A.25 EXAMPLE IDEA: SEMANTIC DIVERGENCE MINIMIZATION: REDUCING HALLUCINATIONS IN LARGE LANGUAGE MODELS THROUGH ITERATIVE CONCEPT GROUNDING
Semantic Divergence Minimization: Reducing Hallucinations in Large Language Mod- els through Iterative Concept Grounding (Part 1)
1. Problem Statement: Large language models often generate hallucinations by diverging from the core semantic content of the input, especially in complex reasoning tasks. This problem undermines the reliability and trustworthiness of L.LMs in critical applications that require accurate and factual responses.
Motivation: Current approaches like chain-of- chain-of-thought thought prompting focus on generating intermediate steps but do not explicitly constrain semantic drift. By continuously grounding generated content to the original semantic space of the input, capabilities. This method leverages the LLM's own ability extract and comparc semantic concepts, creating a self-corretting mechanism that does not require external knowledge bases or complex architectures.
reduce hallucinations while preserving reasoning
Proposed Method: We introduce Semantic Divergence Minimization (SDM) prompting. For each reasoning step, we prompt model
1. Generate a candidate next step.
Extract key semantic concepts from the original input.
3. Measure semantic similarity between the candidate step and extracted concepts.
If similarity below threshold, regenerate step with explicit instructions incorporate more relevant concepts.
5. Repeat until convergence maximum iterations.
This creates a semantic 'gravity well' that keeps reasoning tethered to the input's conceptual core.
68

---

Published as a conference paper at ICLR 2025
Semantic Divergence Minimization: Reducing Hallucinations in Large Language Mod- els through Iterative Concept Grounding (Part 2)
4. Step-by-Step Experiment Plan:
Dataset Preparation:
Use two datascts: HotpotQA for multi-hop hop reasoning and GSM8K for complex math word problems.
For HolpoiQA, utilize the dev set (7,405 questions).
For GSM8K, employ the test set (1,319 problems).
2. Baseline Implementation:
Implement two baselines:
Standard prompting: directly asking the model to answer the question.
Chain-of-thought (CoT) prompting: asking the model to show its work step-by-step before giving the final answer.
3. SDM Implementation:
Implement SDM method with the following sub-steps each reasoning iteration:
Generate next step.
Extract key concepts from input.
Mcasure semantic similarity.
Regenerate if below threshold.
Repeat until convergence maximum iterations.
4. Prompt Engineering:
Design prompts each step SDM. For example:
"Generate the next step in solving this problem:"
"Extract key concepts from the original question:"
"Rate semantic similarity between these concepts generated step on scale of 0-10:"
"Regenerate the step, focusing
these key concepts:"
5. Hyperparameter Tuning:
Experiment with different similarity thresholds (e.g.. 5, 6, out of 10) and maximum iteration limits (e.g.. 3. 7) to find optimal balance between performance and computational cost.
6. Model Selection:
Use GPT-4 as the primary model due to advanced reasoning capabilities.
Test GPT-3.5-turbo for comparison.
7. Evaluation:
HotpotQA, use official evaluation script to compute Exact Match (EM) and Fl scores.
For GSM8K, use accuracy as the metric.
Manually review subset of responses to assess qualitative improvements reasoning and reduction of hallucinations.
8. Analysis:
Compare SDM results with baselines.
Analyze correlation between semantic similarity scores answer correctness.
Examine cases where SDM significantly improves or fails improve over baselines.
69

---

Published as a conference paper at ICLR 2025
Semantic Divergence Minimization: Reducing Hallucinations in Large Language Mod- els through Iterative Concept Grounding (Part 3)
Test Case Examples:
Example (HotpotQA): Input: Who was born first, Alec Guinness or Billic Holiday? Baseline Output: Alcc Guinness was born first. He was born on April 2, 1914, while Billic Holiday was born on April 7, 1915.
SDM Output: The SDM method provides more structured and transparent reasoning process, reducing the risk of hallucination by grounding each step in the key concepts (birth dates) from the original question.
Example (GSM8K): Input: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and sells the rest the farmers' market daily for $2 per egg. How much money does she make every week?
Baseline Output: Let's solve this step by step: Janet makes $182 every week.
SDM Output: The SDM method not only provides a clcar step-by-step solution but also explicitly verifies that all key concepts from the original question incorporated, reducing the risk of introducing irrelevant information hallucinating facts.
Fallback Plan: If the proposed SDM method does not significantly outperform baselines, we can pivot the project in several ways. First, could conduct in-depth analysis of where and why SDM fails, potentially uncovering insights about LLM reasoning processes. We might find that SDM works better certain types questions or reasoning tasks, which could lead to more nuanced application of the method. Second, we could explore variations of SDM, such as using different prompts for concept extraction or similarity measurement, or incorporating dynamic threshold that adjusts hased the complexity of the question. Third, could combine SDM with other prompting techniques like self-consistency create hybrid approach. Finally, if the semantic grounding aspect proves challenging, we could focus to analyzing how LLMs interpret and maintain semantic consistency throughout multi-step reasoning, which could provide valuable insights for future work on reducing hallucinations.
chain-of-thoure
70

---

Published as a conference paper at ICLR 2025
Reviewer
Novelty: (clearly novel major differences from all existing ideas) Rationale: The use of semantic similarity to constrain CoT--ttyld generation is very new. have not seen similar work
(Moderately feasible: can probably executed within given time frame would require careful planning, efficient use of APIs or some advanced computational strategies to overcome the limited GPU resources, and would require some modifications to the original proposal to make it work.)
Feasibility:
Rationale: The pipeline feasible to me. The major challenge would finding the similarity threshold for each dataset.
Expected Effectiveness: 3 (Low Effectiveness: The idea might work in some special scenarios but you don't expect to work general.)
Rationale: some drawbacks this pipeline. First, manually tuning the similarity threshold seems not the best practice for scalable applications. The GSM8K math dataset contains pretty elementary math problems. In that case, the semantic similarity threshold should be set very high, since these basic math concepts involved in the prompt and the CoT breakdown would be determined as highly similar by most existing embedding methods. This brings question of whether this similarity threshold non-trivial at all for some tasks.
6 (I (Learning positive: exciting enough be accepted at major Al conference, but still has weaknesses somewhat incremental)
Excitement:
Rationale: Constraining CoT breakdowns is novel and deserves more work and exploration. While the use of semantic similarity has many drawbacks (such as tuning the threshold, task-sensitive, non-scalable), it can still show us some valuable results about constraining CoT breakdowns.
Overall Score:
5 (Decent idea but has some weaknesses acceptance threshold major Al conferences)
exciting enough, marginally below the
Rationale: There are some clear drawbacks inherent to the method, as discussed carlier. If the authors can overcome these limitations, this idea could yield some interesting findings useful for our understanding of CoT behavior and could pass above major conference threshold.
Confidence: (You are fairly confident that the evaluation is correct)
71

---

Published as a conference paper at ICLR 2025
Reviewer 2
Novelty:
Rationale: Generally this method is way of rejection sampling to improve factuality. is somewhat not too different from previous literature for "constrained decoding" for improving factuality: Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation Don't Say What You Don't Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search
Feasibility: 9
Rationale: Simple prompting approach easy to implement. Evaluation is simple.
Expected Effectiveness: 3 (Low Effectiveness: The idea might work in some special scenarios but you don't 't expect it to work in general.) Rationale: 1. Right now most LLMs hallucinate a subtle way: they say things in semantically correct or reasonable ways, the precise fact is incorrect. Using semantic similarity as a measurement to gauge/control hallucination might not be able to solve the problem. 2. The rejection sampling is based on another LLM-whal if the LLM also hallucinates?
Excitement: 3 (Mediocre: this idea makes marginal contributions and very incremental) Rationale: The method is not that novel and think the method not that effective and might not solve the problem at all.
Overall Score:
3 (Clear rejection for major AT conferences)
Rationale: The experiment design is kind simple and the evaluation not comprehensive. think the idea is in the range of 4 but the experiment plan further reduces my score.
Confidence: (You absolutely certain that the evaluation is correct and very familiar with the relevant literature)
Reviewer 3
Novelty: (mostly not novel you can find very similar idcas)
Rationale: The idea of extracting key semantic concepts, measuring the relevance of the candidate next step, and possibly rejecting/revising the step very similar incorporating self-critique into multi-step reasoning problems. Different versions are already commonly used, especially for solving math problems.
Feasibility: 8 (Highly Feasible: Straightforward implement the idea and run all the experiments.) Rationale: The proposed approach should be straightforward to implement: it only requires prompt engineering to extract semantic concepts and evaluate the relevance of a candidate next step.
Expected Effectiveness: 3 (Low Effectiveness: The idea might work in some special scenarios but you
don't expect to work general.)
Rationale: Compared chain-of-thought prompting, there's a reasonable chance method could work better: could help identify when a reasoning step becomes irrelevant to the original question. However, since such self-critique methods have already been explored, it's unlikely that this instantiation will work significantly better than previous concepts and measuring semantic similarity seems vague, and it's not reflected the provided examples.
Also, the proposed idea of extracting relevant semantic
Excitement: 2
Rationale: The proposed method is similar existing works; doesn't 't contain novel insights that would meaningfully boost current LM performance or introduce new ideas worth building on. would not be an exciting paper.
Overall Score:
2 (Strong rejection for major Al conferences)
Rationale: Similar the reasoning above: the proposal too similar existing works, it docsn't introduce new ideas or insights, and is unlikely to meaningfully improve current LM performance.
Confidence: 4 (You confident but not absolutely certain that the evaluation is correct)
72

---

Published as a conference paper at ICLR 2025
A.26
EXAMPLE IDEA: AUTOPROMPTING: GENERATE DIVERSE FEW-SHOT EXAMPLES FOR ANY APPLICATION
Autoprompting: Generate Diverse Few-Shot Examples for Any Application (Part 1)
Problem Statement: Adding natural language capabilities to existing software requires manually crafting few-shot prompts, which is tedious and does not guarantee high coverage.
Integrating natural language capabilities into software applications often necessi- tates manually creating few-shot prompts, process that time-consuming and may not ensure comprehensive coverage. An "Autoprompting" system capable of automatically generating diverse and relevant few-shot examples tailored to specific applications would significantly reduce manual effort, improve coverage and versatility, and enable rapid prototyping and iteration of natural language capabilities. Large Language Models iteratively test different functionalities of an application and make adjustments to few-shot prompts akin human developer. This approach would ulti- mately democratize the integration of such capabilities across wide range of applications and industries.
Motivation:
Proposed Method: This method leverages Large Language Model (LLM) with coding capabilities. involves the following core steps:
1. Extract all user-facing functions and gather their documentation and unit tests, available.
Generate diverse natural language prompts to utilize each function, defining the expected
output.
3. Generate code from the natural language prompts and execute the corresponding functions.
If code fails:
Update the code and retry
If the code runs but produces an incorrect result, update it using insights from unit tests general reasoning.
5. Once you have a few exemplar prompts for all (or desired) functions, generate prompts that compose multiple functions together and repeat step 4.
By iteratively refining code generation from natural language and leveraging available documentation and tests, this process aims to create an LLM capable of correctly implementing functions based on natural language instructions.
Step-hy-Step Experiment Plan:
Applications: When collecting applications from GitHub, prioritize those with clear, well- written documentation and comprehensive test suites. Include applications from different domains and with varying levels of complexity ensure diverse dataset.
Few shots and feasibility: Create manual few-shot examples to understand the complexity of the functions and the quality of the documentation. Begin by creating 4-5 examples for any function, which could also
starting point for the LLM.
Extract functions and metadata: Utilize static code analysis tools to ensure accurate and comprehensive extraction of functions, documentation, and test cases. Consider extracting additional metadata, such as function signatures, dependencies, and comments, they can provide valuable context.
NL Module:
Generate diverse user utterances and incorporate techniques to handle variations in natural language. For each user utterance, generate the expected outcome. Consider generating negative inputs.
improve the model's l's ability handle invalid ambiguous
Execution Module: Incorporate sandboxing containerization techniques ensure secure and isolated execution environment when executing the generated code. Implement logging and reporting mechanisms capture and analyze errors unexpected behavior.
73

---

Published as a conference paper at ICLR 2025
Autoprompting: Generate Diverse Few-Shot Examples for Any Application (Part 2)
Step-hy-Step Experiment Plan (Continued):
Incorporate techniques such as code summarization, call graph analysis, and lype inference to provide more contextual information to the agent. Specifically, in any code snippet, if there other user-defined functions, retrieve their metadata and use it in the next iteration prompt generation.
Exploration:
Utilize vector database other structured storage mechanism that supports efficient retrieval and querying for storing few-shot examples and their outputs. Incorporate mecha- nisms for versioning and updating stored data the codebase and underlying models evolve.
Store:
Experiments: Once few-shot examples for different functionalitics and their compositions are obtained, simulate different users with various intents and calculate goal complction and error rates using different models. Initially, start with strong model, and once few-shot examples are available, test with weaker and open-source models.
Test Case Examples: Select toy application from GitHub implemented Python JavaSeript.
Direct prompting: Provide the few-shot examples created and check the goal complction and error rates for the following scenarios.
Toy example: Calculator app and different utterances to try.
Provide complete user utterance with ambiguity. For example:
* Can you add 4 to 8.
Divide by and multiply it by 6.
Provide utterance with some ambiguity. For example:
Take 6 and add them, and then subtract 8. Also, add to the first here the
"first" one is ambiguous as it could be the intermediate answer (6+9=15).
Provide utterance that is related to the function. For example:
Please add A and The correct result would be refusing to answer instead of generating add("A' "J").
6. Fallback Plan: If the proposed methodology does not yield satisfactory results, there are several areas to investigate. First, examine the documentation to ensure it adequately explains the basic functionality of each function. Then, practices. Suhsequently, evaluate each module separately. For the NI. module, verify the examples are diverse and that the generated test cases are aligned. For the execution module, ensure that the correct emror messages are being passed and explore ways to enhance them. The exploration module is the most challenging aspect; if any function has high dependency other functions, traversing will be difficult. Therefore, initially focus on examples with limited function dependency and gradually increase the complexity.
the coding style confirm it aligns with recommended
74

---

Published as a conference paper at ICLR 2025
Reviewer
Novelty:
Rationale:
The proposed method is similar to htpp://arriioooggbb2211003333 https//ackanhholoyy.og/// findings-acl.2. ac1.216/
Feasibility: (Feasible: Can be executed within given constraints some reasonable planning.) Rationale: The experiments can he done with sufficient API access, The dataset collection needs some planning but is in general feasible do. Setting up the vector database may take extra time.
(Somewhat ineffective: There might some chance that the proposed idea can work better than existing baselines but the improvement will be marginal or inconsistent.) The proposal is vague as it docsn't It mention what's 's the final evaluation metric, and does not provide sufficient description of the compared baseline. The prompt in the direct prompt bascline is confusing as well. Overall it's hard discuss the effcctiveness.
Expected Effectiveness:
Rationale:
Excitement: 4
Rationale: Given that the proposed method is vague, I am unsure about its contributions and effective- and therefore feel less excited about it.
Overall Score: 4 (Ok but not good enough, rejection for major Al conferences)
Rationale: The descriptions are confusing and I'm not really sure what's the focus or contribution. The title problem statement mentioned ensuring "diversity"""hip descrihe how this ensured in later subsections. The "Test Case Examples" doesn't 't explain how the components the "Step-hy-Step Experiment Plan" are used.
coverage" as the goal but doesn't t
Confidence: 3 (You are fairly confident that the evaluation is correct)
75

---

Published as a conference paper at ICLR 2025
Reviewer 2
Novelty:
Rationale: Mapping natural language to custom applications is hugely impactful capability, and doing automatically is really interesting. like the focus autoprompting for these types of translations, as the task is feasible since builds off some of "few-shot prompting" that developers might normally do
functionality, with more automatic process that has real system checks/verifications (e.g., running the applications through containers). A related work from HCT tries to enable individual developers to add such NI. functionality to their own applications via DSL + NI program signatures (https//facckieyagg me/reactgennie/) This work is distinguished, it would empower adding such NL functionality to any application, without changing the code.
Feasibility: 4
Rationale: The project infrastructure seems more difficult than simply choosing some prompting methods. It would be an iterative process choosing real example applications from Github, and developing the few-shot prompts manually to get feel this task. Then, some modules seem like 1-2 week tasks (Execution Module, Exploration, Storage) which estimate would make the project more like 3 4 months to complete all modules AND to do the evaluations.
Expected Effectiveness:
Rationale: The baseline here is a zero-shot prompt, asking to do the NL intent and feeding in the documentation of the APL. Assuming the author is correct to say that such NL function mapping requires good few & diverse few-shot examples, expect the method to work well. It uses number of external systems enrich the code dataset give the L.M context and some ways, Autoprompting is allowing an agent to make use all these SWE tools understanding the software, which then will allow it to maximize its understanding and better retrieve good few-shot examples for the task at hand.
system
inform. So in
Excitement: 7
Rationale: Seems like an impactful and ambitious outcome completed. am curious how such an approach fits into the conversation about general agents, which can leverage APl/toll/unncions calls. It's little unclear from the toy example why existing function-calling models can't can't translate NL intents into.
Overall Score: 6 (Marginally above the acceptance threshold of major Al conferences)
Rationale: The results would be really exciting and the technical infrastructure to enable the Auto- prompting agent would be impressive. However, I' I'm missing bit of which cases will be really difficult for other generalist weh/system agents, but where finding the few-shot examples for this task is really needed. Thus, core idea of the method doesn't seem clarified enough to result really clear takeaway on the method.
Confidence: 3 (You fairly confident that the evaluation correct)
76

---

Published as a conference paper at ICLR 2025
A.27 EXAMPLE IDEA: TEMPORAL DEPENDENCY UNFOLDING: IMPROVING CODE GENERATION FOR COMPLEX STATEFUL SYSTEMS
Temporal Dependency Unfolding: Improving Code Generation for Complex Stateful
Systems (Part
Generating code for complex, stateful systems or applications with intricate temporal dependencies remains challenging for current code generation models. Most existing approaches focus on generating individual functions or small code snippets without fully considering the temporal aspects and state changes in larger systems. This limitation hinders the applicabilityy of Al-assisted programming in areas such as distributed systems, game development, and real-time applications.
1. Problem Statement:
Motivation: Many real-world applications require careful management of state over time. Existing code generation models struggle with capturing the full complexity of temporal dependencies and state changes in larger systems. A method that can effectively reason about and generate code for systems with complex temporal dependencies could significantly improve the applicability of Al-assisted programming critical areas. Our proposed Temporal Dependency Unfolding method is inspired by how human developers approach complex system design, first identifying key states and their relationships before implementing the detailed logic.
We propose Temporal Dependency Unfolding, novel prompting technique that guides the model generate code by explicitly reasoning about state changes and temporal relationships. The method consists of five steps:
Proposed Method:
1. State identification: Prompt the model to identify key states and variables that change time in the target system.
2. Temporal Graph Construction: Guide the model to create conceptual graph of how these states evolve and interact time.
3. Staged Code Generation: Generate code in stages, focusing on different temporal slices or state transitions in each stage.
Consistency Verification: After stage, prompt the model verify temporal consistency and make necessary adjustments.
5. Integration:
Finally, guide the model to integrate the stage-wise wise generated code into cohesive system, ensuring proper handling all temporal dependencies.
4. Step-by-Step Experiment Plan:
Dataset Preparation:
Create dataset of programming tasks that involve complex temporal dependencies. Include tasks from three domains: 1) Multi-threaded applications, 2) Game logic, and 3) Distributed systems.
For each domain, prepare 50 task descriptions, each with a clear specification of the desired functionality and temporal requirements.
Baseline Implementation:
Implement two baseline methods:
Direct prompting: Simply provide the task description to the model and ask it to generate the code.
Chain-of Thouugh (CoT) prompting: Append "Let's approach this step-by-step to the task description.
Use GPT-4 for both baselines.
77

---

Published as a conference paper at ICLR 2025
Temporal Dependency Unfolding: Improving Code Generation for Complex Stateful Systems (Part 2)
4. Step-by-Step Experiment Plan (Continued))
3. Temporal Dependency Unfolding Implementation:
Implement our proposed method with the following sub-steps for each task:
(a) State Identification: Prompt GPT-4 with 'Identify the key states and variables that change over time in this system:'.
(b) Temporal Graph Construction: Prompt with 'Create a conceptual graph showing how the identified states evolve and interact over time:'.
(c) Staged Code Generation: For each major state transition identified, prompt with 'Generate code for the following state/transition: [state/transition]]
(d) Consistency Verification: After each stage, prompt with "Verify the temporal con- sistency of the generated code and suggest any necessary adjustments:".
(c) Integration: Finally, prompt with 'Integrate the generated code segments into cohesive system, ensuring proper handling of all temporal dependencies.".
Evaluation Metries:
Correctness: Percentage of generated code that passes predefined test cases.
Temporal Consistency: Manual evaluation of how well the code handles temporal dependencies (scale 1-5).
Code Quality: Automated metrics like cyclomatic complexity and maintainability index.
Exccution Efficiency: Runtime performance on benchmark inputs.
5. Human Evaluation:
Recruit experienced developers to review subset of 30 generated solutions (10 from each domain).
They will rate the code on scale of 1-5 for readability, maintainability, and correct handling of temporal dependencies.
6. Experiment Execution:
For each task in the dataset:
(a) Generate solutions using both baseline methods and our Temporal Dependency Unfolding method.
(b) Apply all evaluation metrics to the generated solutions.
(c) Collect human evaluations for the subset of solutions.
7. Analysis:
(a) Compare the performance of Temporal Dependency Unfolding against the baselines across all metrics.
(b) Analyze the effectiveness of each step our method (State Identification, Temporal Graph Construction, etc.) by examining intermediate outputs.
(c) Identify patterns in tasks where our method shows significant improvement or underper- forms
(d) Correlate automated metrics with human evaluations validate their reliability.
78

---

Published as a conference paper at ICLR 2025
Temporal Dependency Unfolding: Improving Code Generation for Complex Stateful Systems (Part 3)
5. Test Case Examples:
Test Case 1:
Bascline Prompt Input (Direct Prompting): Generate Python code for a simple multi- threaded producer-consumer system with shared buffer. The producer should generate random numbers and add them the buffer, while the consumer should remove and process these numbers. Implement proper synchronization to avoid race conditions.
Baseline Prompt Expected Output (Direct Prompting): [Python code for simple producer-consumer system]
Proposed Prompt Input (Temporal Dependency Unfolding; Step State Identification): For a multi-threaded producer-consumce system with a shared buffer, identify the key states and variables that change over time in this system:
Proposed Prompt Expected Output (Temporal Dependency Unfolding; Step I: State Identification): [List of key states and variables]
Proposed Prompt Input (Temporal Dependency Unfolding; Step 2: Temporal Graph Construction): Create a conceptual graph showing how the identified states evolve and interact time for the producer--consmmrr system:
Proposed Prompt Output (Temporal Dependency Unfolding: Step 2: Temporal Graph Construction): [Conceptual graph state evolution and interactions]
Proposed Prompt Input (Temporal Dependency Unfolding; Step 3: Staged Code Gener- ation): Generate code for the producer functionality the producer-consumer system, focusing on its interaction with the buffer and synchronization mechanisms:
Proposed Prompt Output (Temporal Dependency Unfolding; Step 3: Staged Code
Generation): [Python code for producer functionality]
Proposed Prompt Input (Temporal Dependency Unfolding; Step Consisteney Verifi- cation): Verify the temporal consistency of the generated producer code and suggest any necessary adjustments:
Proposed Prompt Output (Temporal Dependency Unfolding; Step 4: Consistency Verifi-
cation): [Verification and adjustment suggestions]
Proposed Prompt Input (Temporal Dependency Unfolding: Step 5: Integration): Inte- grate the generated producer code with consumer and main control logic create complete producer-consunc system, ensuring proper handling of temporal depen- dencies:
Proposed Prompt Output (Temporal Dependency Unfolding: Step S: Integration):
[Complete Python code for producer-connsmmrr system]
The Temporal Dependency Unfolding method produces a more compre- hensive and robust solution compared to the baseline. It explicitly handles temporal dependencies, includes proper synchronization, and provides mechanisms for graceful lermination. The staged approach allows for better handling of edge cases and improved overall system design.
Explanation:
6. Fallback Plan: If the Temporal Dependency Unfolding method does not show significant im- provement the baselines, can pivot the project several ways. First, could conduct an in-depth analysis where and why the method fails. which could provide valuable insights into the limitations current language models in handling temporal reasoning tasks. This analysis could involve examining the intermediate outputs (state identification,, temporal graphs) to understand where the reasoning breaks down. Second, could explore combining method with other techniques, such retrieval-auge aaureevll-ugmeettee generation, to see if providing relevant examples improves performance. Third, we could focus on developing new evaluation framework specifically designed to assess temporal reasoning in code generation, which could be a valuable contribution to the field even if our primary method docsn't It outperform baselines. Lastly, we could investigate whether the method performs better on certain types of temporal dependencies specific programming domains, which could lead to more targeted approach for improving code generation those areas.
79

---

Published as a conference paper at ICLR 2025
Reviewer
Novelty: 6 (reasonably novel there are some notable differences from existing ideas and probably enough to turn into a new paper)
Rationale: The construction of Temporal Graph sounds novel. The research question is also relatively underexplored, necessary for coding domains like distributed systems.
Feasibility: 6 (Feasible: Can be executed within the given constraints with some reasonable planning.) Rationale: The data collection part should he the most challenging part. Collecting high-quality coding problems that involve complex temporal dependencies could be hard. Also, the human evaluation might also take time execute.
Expected Effectiveness: (Somewhat effective: There is decent chance that the proposed idea can beat existing baselines by moderate margins few benchmarks.) Rationale: With specific prompting techniques, the proposed method should outperform baselines in terms of temporal dependencies.
Excitement: 7
Rationale: think this should be exciting than most of the borderline papers since working on new problem. The collected data should also be super useful.
Overall Score: 7 (Good idea, would be accepted by major Al conferences) Rationale: Again, working novel problem makes it better than most of the prompting papers.
Confidence: You are confident not absolutely certain that evaluation correct)
80

---

Published as a conference paper at ICLR 2025
Reviewer 2
Novelty: (somewhat novel there are differences from existing ideas but not enough to turn into new paper)
Rationale: Although I
entirely familiar with the field of generating temporally adaptive programs, suspect some similar ideas can be found software engineering works (e.g. ICSE). More concretely on the method, rather similar to code generation with intermediate state reasoning, which has been explored in several multi-step, conversational code generation works, e.g:
1. 7heng. Tianyu, al. "Opencodeinterpreter:: Integrating code generation with execution and refinement."
Cao. Liuwen, et "Beyond Code: Evaluate Thought Steps Complex Code Generation." Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024.
3. Nijkamp, Erik, et al. "Codegen: An open large language model for code with multi-turn program synthesis."
Feasibility: 3 (Very challenging: there are flaws in the proposed method or experiments, or the experiments require compute/human resources beyond academic lab)
Rationale: It would be pretty hard to collect such datasets (e.g.. would mostly require whole repository). further, would difficult to generate executable test cases verify the multiple problems created. Especially because the task targets temporally dependent modules in the program, may necessitate domain experts to carefully construct examples and tests, which would demand a lot of time and costs.
semporall-ddpeendent
Expected Effectiveness: (Somewhat ineffective: There might some chance that the proposed idea can work better than existing baselines but the improvement will be marginal or inconsistent.) Rationale: am not very confident that the model can solve this complex temporally -eneedel programming problems with reasonable correctness. Furthermore, because the current method is basically prompting, which may have very low performance upper bound. Therefore, don't expect the proposed method to improve significantly on code generation.
Excitement: 4
Rationale: Overall, don't expect this method to bring substantial improvements, hence less excited about the potential of this method. would still an interesting problem solve, particularly in bringing more challenging coding problems and proposed corresponding methods. With this being said, given the current performance of models, building a solid benchmark regarding this temporal code generation problem may be more exciting than proposing method that is expectedly not working.
Overall Score: 4 (Ok not good enough, rejection major Al conferences)
Rationale: The task of temporal code generation is not the most urgent issue of current code generation models, and the proposed method is expected to not bring much improvement. The method needs to be further refined and go beyond simple prompting convince the audience of the potential of this thread methods.
Confidence: 3 (You are fairly confident that the evaluation is correct)
81

---

Published as a conference paper at ICLR 2025
Reviewer 3
Novelty: 10 (very novel very different from all existing ideas in very interesting and clever way) Rationale: This idea studies very novel problem in LLM-based based code generation. Temporal dependen- cics in code generation should be specifically studied in the of LLMs.
Feasibility: (Moderately feasible: can probably executed within given time frame would require careful planning, efficient use of APIs or some advanced computational strategies to overcome the limited GPU resources, and would require some modifications to the original proposal to make it work.)
Rationale:
Constructing a reasonable dataset challenging within short time. Also, human evaluation might take more time. Whether LLM can construct high-quality graphs in this case also to be examined.
Expected Effectiveness: (Somewhat effective: There is decent chance that the proposed idea can beat existing baselines by moderate margins on few benchmarks.) Rationale: One needs to build reasonable metrics to show effectiveness. Also, one might need to tune prompts carefully to construct high-quality graphs in this case.
Excitement: (Exciting: would deepen the community's understanding or make major progress this research direction)
Rationale: This is novel and could have huge impact those code generation cases requiring temporal dependencies. But one needs to justify why such use are important, and why temporal dependency is the prohlem in such cases.
9 (Top 15% of all published ideas on this topic at major Al conferences. strong accept) Rationale: Considering its novelty, valuable dataset, and comprehensiveness of experiment and evaluation design, this could be impactful work. But one needs to make experiment results concrete by re-examining whether each step works well in practice.
Overall Score:
Confidence: 4 You are confident but not absolutely certain that the evaluation is correct)
82

---

Published as a conference paper at ICLR 2025
A.28 IDENTITIES OF EXAMPLE IDEAS
We reveal whether each example idea is Al-generated or human-written:
Human ideas: Example A.20
Example A.22
Example
A.24
Example A.26
A.26
AI ideas: Example
A.21 A.21
Example
A.23
Example
A.25
Example
A.27
83

---

Published as a conference paper at ICLR 2025
A.29 ABLATIONS IDEA GENERATION
In this section, we investigate several factors that influence the performance of our retrieval-augmented generation (RAG) approach for idea generation. We ablate:
The maximum number of papers to retrieve (N).
2. The number of papers (k) to include in the prompt.
3. Different base models and prompt setups for idea generation.
For ablating the impact of the maximum number of papers to retrieve (N), we vary N among (30, 60. 120, 180) and measure the average relevance score of the top 20 retrieved papers (on scale of 1-10), as judged by an LLM. The results in Table T7 show that N 120 consistently achieves strong performance and that the relevance of the top retrieved papers tends to plateau beyond this point.
Table 17: Ablation of the maximum number of papers retrieve (N). We report the average relevance score of the top 20 retrieved papers (on a scale of 1-10) for two topics (Multilingual and Uncertainty) as judged by an LLM.
N
Multilingual
Uncertainty
30
6.80
6.75
60
7.05
7.50
120
8.10
8.45
180
8.10
8.40
We additionally ablate the impact of the number of papers (k) add to the prompt for RAG by varying & in 50,5, 10, 20), and measure how it affects the diversity of generated ideas for the topic of uncertainty prompting. In Table [8 the "Non-Duplicates (%)" metric reflects the proportion of unique ideas generated. We observe that varying has minimal impact on the diversity.
Table 18: Ablation the number of retrieved papers & added the prompt, showing the diversity of generated ideas on the uncertainty prompting topic. "Non-Duplicates (%)" indicates the proportion of unique ideas.
k:
Non-Duplicates (%)
18.8
5
18.4
10
19.1
20
19.4
To examine whether the diversity issue exists across different base models and prompt configurations, we measure the idea diversity of multiple other models and prompt setups. We use a temperature of 1.0 for all the generations and compute the percentage of non-duplicate ideas of 2K generated ideas on the topic of uncertainty prompting. Table 19 compares four different base models.
Table 19: Comparison of different base models regarding idea diversity on the topic of uncertainty prompting. We measure the percentage of non-duplicate ideas out of 2K generated ideas using temperature of 1.0.
Base Model
Non -on-Duplisates (%)
Souude-3.5-Sonell
19.1
GPT-4o
59.5
ol-mini
22.6
Llama-3. 1.teaa...14000
-40SB-Instrru Instruct
51.1
84

---

Published as a conference paper at ICLR 2025
We find that different models have very different non-duplicate rates, with GPT-4o and Llama-3.1- 405B-Inntrruc showing substantially higher diversity than Claude-3.5-Sonne and ol-mini. However, chose Claude-3.5-Somee
the base model of our agent because the quality of its generated ideas outperforms the others in our pilot study. Specifically, we randomly sampled 10 ideas from Claude-3.5-Somee and GPT-4o for round of pilot expert scoring and found that Claude-3.5-Sonnet achieved an average score of 5.4 (on a 1-10 scale), whereas GPT-4o scored 4.8.
Next, ablate several different prompt setups for idea generation using the Seaude-3.5-Sonne backbone. Specifically, we analyze the impact of applying RAG and the impact of appending previously generated ideas for deduplication.
Table 20 shows that appending previously generated ideas in the prompt and instructing the model to avoid repetition significantly reduces idea duplication. However, including retrieved papers in the prompt has minimal impact on diversity.
Table 20: Comparison of different prompt setups for idea generation with the Claude-3.5. -laude-3.5-Somee 3.5-Sonnet backbone. report the percentage non-duplicate ideas out 2K generated ideas on the topic of uncertainty prompting.
Prompt Setup
Non-Duplicates (%)
No RAG; No prev
7.6
No RAG; Prev
18.8
RAG (k=5); Prev
18.4
RAG 10); Prev
19.1
RAG (k=20); Prev
19.4
85

---

Published as a conference paper at ICLR 2025
A.30 ATTEMPT ON IDEA EXECUTION AGENT
For our execution agent, the input is the generate idea (the full project proposal), and the output is Python file that be executed with our specified command. Since there is often common pipeline of implementing prompting-bar bbaeed research ideas, we provide a manually crafted code file example as template. We attach the full template below:
import random
from tqdm import tqdm
3 from utils import call_api, Load_model
model
import random
sendom.seed (2024)
7 P Step 1: Generate synthetic test examples
def (mhrrate_testset>> ():
=
"input": "Natalia sold clips to her friends in April, and then she sold half as many clips in May.
How many clips did Natalia sell altogether in April and May?",
" "output": "Natalia sold 48/2 <<48/2-24>524 el clips May. Natalía sold 48+24 <<48424-72>>72 clips altogether in April and May. FEF 72"
15
"input":
"Weng earns $12 an hour for babysitting. Yestorday, she just did 50 minutes of babysitting. How much did she carn?",
" "output"; "Weng carns 12/50 $<<12/60=0 2>>0.2 per minute. Working 50 minutes, she earned 0.2 50 4<c0.2250=10>>10. #### 10"
" "input": "Tim has 30 less apples than Martha, and Harry has half as many apples as Tim. If Martha has apples, how many apples does Harry have?",
20
"output": "Tim has 68-30 <<68-30-38>>38 apples. Harry has 38/2 <<38/2-19>>> apples. 444 19"
23
"input": "Four people lost a total of 103 kilograms of weight. The first porson lost 27 kilograms. The second person lost kilograms less than the first person. The two remaining people lost the same amount. How many kilograms did each of the last two people lose?",
ootpptt:: "Second person 27 7 <<27=7-20>>20 kg 103 27 20 <<103-27-20=56>555 kg 56/2 <<56/2=28>>28 kg The last two people each lost 28 kilograms of weight. #### 28"
28
return test_data
29
## Step Implement the baseline method model_name, seed, question):
def geeeline_method(cllle
33
#F zero-shot chain-of-though.
prompt = "Answer the following question: ()An".forma (question) prompt +- "Think step by step." prompt_messages [["role": "user", "content": prompt:]
86

---

Published as a conference paper at ICLR 2025
response, call api (client, model_name name, prompt messages, temperature=0., max_token==2000, seed=seed, json_cutppu-=False) return response.strip
38
## Step 3: Implement the proposed method
(cinet,,,,,,,, model_name, seed, question, def (rint_all=False))
intermediate_outputt..
if print_all: print ("question: An", question)
## collaborative reasoning step 1: task decomposition
prompt "Please break down the following task into smaller
sub-tasks or steps:: ()" format (question)
50
prompt_nessages = [["rolo"; "user", "conteet"" prompt
dccomposition, call_api (elleent (client, model_name, prompt_messagos,
temperaturo=0., max_token==2000, seed=seed, [son_cuttut-Faalse
intermediate__oupple "\n"
"task decomposition: Vn" decomposition
if print_all: print ("decompositions Xn", decomposition)
## collaborative reasoning step 2: sub-task information generation
57
prompt "For each of the following sub-tasks, please generate relevant information or intermediate results: An(]" format (decomposition)
prompt_messagen [["role": "user", "content": prompt]]
intermediate, call_apt (client, model_name, caompt_message,, temperature=O., max_toxens=2000, seed=seed, jscn_cutput-False, outusss += "sub-task results:\n" Intermediate + "\n"
if print_all: print "Wintermediate:\n"
intermediate)
#F coll collaborative reasoning step 3: result combinati
prompt "Given the follow! ng intermediate resulta: \n(), please conbine them to generate the sinal answer for the task: An()".forman format (intermediate, question)
prompt messages = r1"re""" "usez", "content": prompt answer, - call_api (client, model_name, prompt_messages, temperature=0., max_tokens=2000, seed=seed, jsen_output=Faase) intermediatee_ourputs "result combination: \n" answer + "\n" if print_all:
68
print ("initial answer:\n", answer)
#F collaborative reasoning step reflection and refinement prompt = "Given the task: (1 AnPlease reflect on the generated answer:\pll An AnAnAre there any gaps or inconsistencies in the answer? If s0, please identify and address them and give me improved answer. not, you don't have to edit anything and can just return the original answer. " format (question, answer)
73
prompt_ned messages = [["role": "user", "content": prompt]] call_api aallapencilinnt (client, model name, temperaturo=0., max_tokens=2000, seed=seed, jsen_cutput=False) intermediate_ourgee
76
+= "refloction and refinement: \n"
fiswor
if print_all:
print ("final answer: \n", final_answer)
79
80
return final answer strip (), intermediate.outtptte
87

---

Published as a conference paper at ICLR 2025
## Step 4: Define the style evaluator
def style_ovaluator (client, nodel_name, seed, question, baselinc_prediittonn
proposed_predicttonn FF define all the components that the proposed method outputs should have
# and the advantages of the proposed method over the baseline method
87
## just need to check the style is correct
prompt = "Given the task: 11 "a""fformttt (question)
prompt += "The baseline methed produced the following output:\n() An'n". , format (basel ine_lredeiee ction)
90
prompt +- "The proposed new method produced the following output:An() \n'n". format (proposed_predicttil iinn))
prompt += "Now determine if the proposed method is better by
checking is it has satisfied the following criteria: \n"
92
prompt += "1. The proposed nethod's a output should produce all the intermediate components including: task docomposition, sub-task information generation, result combination, and reflection and refinement. An"
prompt += "2. The proposed method should provide more detailed and comprehensive answer than the baseline methed. An" prompt +- "Just tell me 'yes' or 'no' for whether the criteria are met, nothing else is needed." prompt_nessage
94
I1"role": "user", "content": prompt response, call spi (client, model name, meggpge_msagee, temperature-0., max_tokens=l, seed=seed, json_output-False)
judgment False if response strnipp) (lwer "yes": return True
100
101
102
return judgment
104
105
Define the output evaluator def out_e__e uator (cl ient, model_name seed, question, gold_label,
Step 5:
106
predi cn))
#F check if the predict ctiin is correct giver the gold 1abel prompt "Given the following question and reference answer,
108
determine if the prediction is correct. Just tell me 'yes' or
'no', nothing else is needed, AninQuestion: () (/Annneeeeeence Answer; () (DNnkaPrediction: .frmmmt
format (question,
gold_label, prediction)
109
prompt_messages = [["role"; "user", "content": prompt]]
110
response, = call_api (client, model_name, prompt_messages, _____mssages,,
temperature=0., tax_tokens=1, seed=seed, json_cutput-False)
112
judgment False
113
if cesponse.strip lower = "yes":
114
return True
return judgment
117
118
119
## Step 6: Define the function that runs the experiments to obtain model predictions and performance
120
# you shouldn't need to modify this function in most cases
121
def run_oxperiment (client, model_name, seed, testset); sample_sizc = len
122
proposed_predictions
125
126
propose_.correctees:
88

---

Published as a conference paper at ICLR 2025
style_check check
131
for i in tqdm ( (sample_size)
132
question testset [i] [""nput") sstrip
cold_label = testset [2] ["outpur"] strrip(
135
gaaseline_preiictie baselineemechoo seed, question)
model_nama
136
proposee_predictionn.oo
intermediate
proposed nethod(ciieni model name, seed, questien)
baseli iaselinnepredcccionn append (basel inaeeline_predicee cticiiction
138
proposd__rediccioon..eee (proppss__radiiiioooio..
139
140
baseline bat_he__crrrcetsssseee (wtp:/ttteeaaaattttoorraeeeeeeee www.eaaalrrrrooooo (client, model_name, seed, question, gold_laben
baseline (heeline_predcction))
141
appond (ourput_ovaluatoo) (client, model_name, seed, question, gold_label, fropes:_prrcddiiiooo__.aaaa)))
143
wtw.eeecccccee...ppeeded////eeea (client, model_name, seed, question, baseline_prediction prg.oeddppeedccctoooooieeeeeeee
145
return suseline_correctnnn proposed p correctness, style_ check
146
147
Step 7: Execute the experiments and compare performance == main
if
150
testset generate_ testset
151
print ("simulated () test examples for
ovaluation. " format (len (testset)))
153
model_nand name -claadde--oppss222225-------3---
154
seed 2024
155
client 1cad_mode, (modei_nane)
print ("using model: ", modei_name
158
#F output correctness
139
baseline_comme cor.eeccrreeessss,oeeoee proposed correctness, style_check
experrmennt (client, model_ seed, testset)
print ("baseline correctness: sum cumbaaeelneeeeorrecceessss)
len
161
print ("proposed correctness: ", sum sun[propped_ccorrottnsss
len (ooppssee_oorreeneeesss
162
print ("style check pass rate: sum (sttyleccheck) len (sttle__chcc))
As seen above, we have defined two different evaluator functions. The checks whether all components mentioned in the proposed method are implemented, and the output_evaluatoo. () checks whether the model predictions are correct.
()
We provided this template along with instructions to tell the execution agent to modify all necessary functions for the provided idea. tested the agent on 30 safety prompting ideas and 30 factuality prompting ideas.
As shown Table 21 over half of the generated code files can be successfully executed, and the results indicate that only handful of these proposed methods actually outperform the baselines. While this shows some promise that the execution agents can automatically implement and identify effective methods out of all the generated ideas, careful manual check reveals many pitfalls.
89

---

Published as a conference paper at ICLR 2025
Safety
Factuality
Total
30
30
Executed
17
18
Passed Style Evaluator
13
14
Proposed Method Works Better
5
Table 21: Success rate our exccution agent on 30 randomly sampled Al ideas on safety and factuality.
For example, the agent implemented the following baseline for the task of identifying malicious and adversarial prompts, which is a keyword based approach that is trivial to beat with any LLM-based methods.
## Step 2: Implement the baseline method
def baseline melien( nodel_name, seed, prompt) Keyword-based heuristic
keywords = ("ignore", "pretend", "roleplay", "disregard", "override"]
for keyword in keywords:
if keyword in prompt. (oxer)) return "Adversariai"
return "Benign"
In another example, for the same task of detecting adversarial prompts, the agent implemented the following evaluator function:
## Step 5: Define the output evaluator
2 def gobd_aabel, prediction):
(client, model_ name, seed, raput_textt
prompt "Given the following text and reference sent iment
classification, determine if the predi sted classification is correct. Just tell me yes'' or 'no', nothing else is
() ()An/neeference: () (DAnnnrediction
format (input (input_texts text, gold_label, predictien) messages [["role": "user", "content": prompt]]
response, call_api (cllent, model_name, prompt_messages, temperature=0., max_tokens=1, seed=seed, json_cutput-Falsd
judgment False if response.atrip () llver "yea": return True
10
return judgment
The agent is supposed to inject adversarial triggers into sentiment classification data to test whether the proposed method can detect those adversarial prompts while maintaining sentiment classification accuracy. However, the agent only evaluates the accuracy the original sentiment classification task but not the task of adversarial prompt detection.
Given these errors, we believe more work needed to carefully verify the code implementations produced by the execution agent rather than blindly trusting their executed results, and we leave such attempts future work.
90


Reviews:
[{"review_id": "05SiJAQyBq", "reviewer": "Reviewer_3XGM", "summary": "This paper evaluates if LLM based RAG agents are capable of generating novel research ideas for prompting techniques by comparing them with expert researchers. The key contributions of their work are \n\n1) Evaluating the capabilities of LLMs to generating research ideas and analyzing their strengths (novelty) and weaknesses (feasibility and diversity)\n2) They focus on removing confounders by conducting a large scale human evaluations and controlling experimental setting", "questions": "1) RAG pipeline ablation tests: \"We keep the top k = 20 papers from each executed function call and stop the action generation when a max of N = 120 papers have been retrieved.\" , \"For retrieval augmentation, we randomly select k = 10 papers from the top-ranked retrieved papers\" . Can you please provide some small scale (self-evaluated) tests ablating these parameters to ensure their optimality?\n\n2) Researcher Background: I understand that the researcher familiarity numbers are provided for idea generation and reviewing processes, however it is slightly different from expertise. So, a better way to ensure the robustness of results is to look at the distribution of the expertise of the reviewers. \n\n3) Interestingness: Can you please clarify the meaning of the term? Is it that the insight behind the proposed idea a non-trivial one or is it that the human evaluating it finds it personally interesting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Extensive human evaluation with over 100 NLP researchers\n2) Great care taken to remove confounders through style normalization, review form, grant submission form for research ideas etc. \n3) Interesting insights like lack of ability of LLMs to generate diverse research ideas.", "weaknesses": "1) Not strong enough motivation for restricting to prompting based research. This could have been a confounder since we did we are not sure about the ability of LLMs to generate research where the solutions are more complex. \n\n2) Usefulness: The paper does not talk about the usefulness of the generated ideas research ideas, which is more important than novelty. Research ideas are born out of deep insights into the limitations of previous research. This paper neither gives the discussion/limitations section of research papers to the LM as context nor does it do any analysis of the usefulness of the generated ideas. Using abstracts and paper titles sounds like an anti-causal way of generating ideas which I think could affect the usefulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2tTPQh85E", "reviewer": "Reviewer_HdAN", "summary": "- Study: This paper presents a novel study comparing research ideas generated by Large Language Models (LLMs) with those crafted by expert NLP researchers.\n- Methodology: The study involves hired expert researchers who generate ideas and conduct blind reviews of both human- and AI-generated ideas.\n- Conclusion: This work is the first to conclude that AI-generated ideas are judged as significantly more novel than those from human experts (p < 0.05).", "questions": "- The authors used reviews from Openreview, are there any licensing issues?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "* Thorough Study Design: This study is detailed and carefully planned, taking a year to complete and requiring significant resources. It includes human expert baselines and a clear evaluation process, with strong statistical methods that make the results reliable.\n\n* Key Conclusion: The study makes an important finding about the differences between AI-generated and human-generated research ideas, adding to our understanding of AI's potential in coming up with research ideas. — more discussions on the absolute quality, in addition to the comparison between AI and human, would be more apprieciated (e.g. are the quality of both ideas bad).\n\n* Interesting Findings: The study also reveals some surprising insights, such as AI-generated ideas becoming repetitive as more ideas are created, and that current AI systems aren’t yet good at reliably evaluating research ideas", "weaknesses": "* While this work is interesting and presents important conclusions, it does not center on machine learning methodology or technical analysis. Although it is relevant and intriguing for the ML community, the paper leans more towards evaluating scientific idea generation, aligning more with applications rather than with core ML research.\n* The study is limited by its exclusive focus on prompting-based NLP research, which restricts the generalizability of its findings to other fields or other ML directions.\n* Although the authors worked to standardize evaluation criteria, research ideation remains inherently subjective, as reflected in the study's inter-reviewer agreement of 56.1%", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zpCfDDvDiC", "reviewer": "Reviewer_YqJH", "summary": "They constructed a meticulously controlled experiment comparing human-generated and LLM-generated ideas, thus overcoming the sample size and baseline issues present in previous small-scale evaluation studies. The study recruited 138 outstanding natural language processing researchers to generate human baseline ideas and conduct blind reviews (49 were responsible for writing ideas, and 79 for blind review). To reduce the impact of confounding factors, the study strictly controlled the style of the ideas and standardized the topic distribution between human and LLM-generated ideas. After a year-long high-cost evaluation, the authors provided a human expert baseline and a standardized evaluation protocol, laying a foundation for future related research.\n\nIn nearly 300 reviews, the authors found that under multiple hypothesis tests and different statistical tests, AI-generated ideas were considered more novel than those of human experts, but less feasible. In addition to evaluating the ideas, the study also analyzed the LLM agent, revealing its limitations and unresolved issues. Despite high hopes for expanding LLMs' reasoning abilities, the authors found that they still show deficiencies in the diversity of idea generation, and the results of human consistency analysis indicate that LLMs cannot yet serve as reliable evaluators of scientific idea quality.\n\nIn summary, the contributions of the paper are as follows:\n\n1. Standardized some evaluation criteria for scientific ideas generated using large language models while demonstrating the limitations of LLMs as judges in evaluating scientific ideas.\n2. Conducted a blind review evaluation of the quality of scientific ideas generated by LLMs through the recruitment of a large number of scientists, verifying both the potential and shortcomings of LLMs in the field of scientific research.", "questions": "1. Could you provide the specific prompt used for the Agent to generate ideas?\n2. Observing that some researchers in Table 3 and Table 4 have a familiarity level of 1 with the domain (which could be considered as low-quality annotation), what are the experimental results in Table 5 after removing this data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The results are quality assured due to significant investment. The author recruited a total of 138 excellent natural language processing researchers and verified their quality through Google Scholar. On average, each idea took the researchers 5.5 hours to generate, and there was a reward of $300. As for the evaluators, each evaluation opinion had a reward of $25. This incentive mechanism, to a certain extent, ensures the quality of idea generation and evaluation.\n2. It is relatively fair. By standardizing the topics and providing human experts and large language models with the same writing templates, and finally passing both outputs through the same style transformation module for rewriting, the consistency of style between the two is ensured, minimizing human bias towards writing style as much as possible.\n3. It is very meaningful. This large-scale manual evaluation verifies the ability of large language models to generate scientific ideas comparable to those of human experts, providing motivation for the development of subsequent work.", "weaknesses": "1. The author concluded that ideas generated by large language models lack diversity. However, using the same prompts, especially longer prompts, it is easy for large models to produce similar responses. Therefore, the lack of diversity may not be solely due to the models themselves; it could also be due to a lack of diversity in the prompts. The author needs to test more prompts to sufficiently reach such a conclusion for this experiment.\n2. The paper only verifies the generation quality of the author's simple implementation of an agent, lacking evaluation against other baselines.\n3. The research is limited to the NLP field and lacks studies in other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aU97yCNEqn", "reviewer": "Reviewer_r2Ms", "summary": "This paper investigates the potential of LLMs to autonomously generate innovative research ideas comparable to those created by human experts. \nIn a controlled study, over 100 NLP researchers evaluated ideas generated by both humans and an LLM agent across metrics like novelty, excitement, feasibility, and effectiveness. The findings reveal that LLM-generated ideas were rated significantly more novel than human ideas, though slightly lower on feasibility.\n\nThe authors addressed multiple experimental challenges, including ensuring a balanced comparison between human and LLM ideas by standardizing topics and using blind reviews. \nThey also tested the robustness of results through various statistical methods. The LLM utilized a retrieval-augmented generation approach and scaled inference techniques to produce numerous ideas, from which a ranking mechanism selected the best ones. Despite the promising novelty scores, the study identifies several limitations of LLMs in research ideation, particularly a lack of diversity in ideas and challenges with reliable self-evaluation.\n\nThe study highlights open questions in designing research agents and emphasizes the potential and current limitations of LLMs in academic innovation.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The topic is interesting, as employing LLMs to generate research ideas holds practical value for researchers. Additionally, the findings offer valuable insights that can help researchers determine whether to adopt LLMs as part of their research toolkit.\n\n2. The study’s experimental design is well-structured, effectively controlling for potential confounding factors. Through the use of blind reviews and standardized styles and topics, the authors achieve a fair and balanced comparison between ideas generated by humans and those by LLMs.\n\n3. This paper incorporates a large-scale human evaluation, involving over 100 expert NLP researchers to create and assess research ideas. This approach provides a robust evaluation of the ideas across multiple metrics.", "weaknesses": "1. As noted in L434-L435, I appreciate that the authors acknowledge the subjectivity involved in reviewing ideas rather than executed papers. However, it would still be valuable to compare LLM-generated ideas with those from published papers (as ground truth), as the latter have been deemed at least ``feasible''.\n\n2. According to L175-L179, the LLM agent was created with a minimalist approach, incorporating only paper retrieval, idea generation, and ranking. However, in real-world research, high-quality ideas often benefit from iterative refinement and prototyping, which are essential for improving clarity and feasibility. As such, I view this paper as an initial investigation into whether LLMs, in their current, straightforward form, can generate high-quality research ideas. The limitations observed, such as the lower feasibility of LLM-generated ideas, could potentially be addressed by a more sophisticated agent framework that includes stages for refining and prototyping ideas.\n\n3. I would not consider this as a major weakness, but given the insights provided, it would be beneficial if the authors offered best practices or recommendations for using LLMs effectively to generate research ideas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)