GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation
Authors: Tao Feng
Yihang Sun
Jiaxuan You
Venue: ICLR 2025 Poster (2025)
Keywords: Idea Evaluation
View-graph
Lightweight model
Label propagation
Graph prediction
Primary area: learning on graphs and other geometries & topologies
TL;DR: GraphEval is a lightweight, graph-based LLM framework for idea evaluation, offering two methods: GraphEval-LP (training-free) and GraphEval-GNN (minimally trained).

Abstract: The powerful capabilities of Large Language Models (LLMs) have led to their growing use in evaluating human-generated content, particularly in evaluating research ideas within academic settings. Existing solutions primarily rely on prompt-based LLM methods or fine-tuned lightweight language models for idea evaluation. However, these methods are often unstable and struggle to comprehend the complex semantic information embedded in the ideas, impeding their ability to perform high-quality evaluations. To address the above challenges, we propose $\texttt{GraphEval}$, a lightweight graph-based LLM framework for idea evaluation. Our insight is that a complex idea can be broken down into comprehensible viewpoint nodes using prompts from small LLMs. These viewpoint nodes can then be linked together through edges created from LLM-based relation extraction and/or BERT similarity scores. The created viewpoint-graph can be used to conveniently propagate scores across view-nodes to improve the robustness of the idea evaluations. In particular, we propose two lightweight graph-based methods for idea evaluation: (1) GraphEval-LP: a training-free label propagation algorithm that propagates evaluation scores from known view-nodes to unknown nodes; (2) GraphEval-GNN: a Graph Neural Networks (GNN) that is trained to predict the evaluation scores given the observed graph with minimal computation resources. Moreover, to overcome LLM's limitation in objectively assessing the novelty of ideas, we further propose a novelty detection model to GraphEval-GNN to enhance its capability in judging idea novelty. Experiments on two datasets show $\texttt{GraphEval}$ improves F1 scores by at least 14% with low computation and API costs. Additionally, $\texttt{GraphEval}$ can effectively detect plagiarized ideas.

Published as a conference paper at ICLR 2025
GRAPHEVAL: A LIGHTWEIGHT GRAPH-BASED LLM FRAMEWORK FOR IDEA EVALUATION
Tao Feng",, Yihang Sun Jiaxuan You!
University of Illinois at Urbana-Champann Champaign "Peking University "Equal Contribution
ABSTRACT
The powerful capabilities of Large Language Models (LLMs) have led to their grow- ing use in evaluating human-generated content, particularly in evaluating research ideas within academic settings. Existing solutions primarily rely on prompt-based LLM methods or fine-tuned lightweight language models for idea evaluation. How- ever, these methods are often unstable and struggle comprehend the complex semantic information embedded in the ideas, impeding their ability to perform high- quality evaluations. To address the above challenges, we propose GraphEval a lightweight graph-based LLM framework for idea evaluation. Our insight is that a complex idea can be broken down into comprehensible viewpoint- using small prompted LLMs. These viewpoint-nodes can then be linked together through edges created from LLM-based relation extraction and/or BERT similarity scores. The created viewpoint- graphh can be used to conveniently propagate scores across viewpoint-node to improve the robustness of the idea evaluations. In par- ticular, we propose two lightweight graph-based methods for idea evaluation: (1) GraphEval-LP: a training-fres label propagation algorithm that propagates quality labels from known viewpoint-nodes Graph Neural Network (GNN) that is trained to predict the quality labels given the observed graph with minimal computation resources. Moreover, to overcome LLM's limitation in objectively assessing the novelty of ideas, we further propose a novelty detection model to GraphEval-GNN to enhance its capability in judging idea novelty. Experiments by at least 14% with low computation and API costs. Additionally, GraphEval 1 can effectively detect plagiarized ideas. Our codes for GraphEval is released at https: /ttthubb.bbt/tg com/ /uluululabuuue http:/llllauuueeeeeeeeeeeeeeeeee GraphEaap
unknown nodes; (2) GraphEval-GNN: a
two datasets show GraphEval improves FI scores
INTRODUCTION
With the advancement of LLMs. many tasks traditionally performed by humans, such idea evaluations (Liang et al., 2024; Lin et al., 2023a), label annotation (Wang et al., 2024; Goel et al., 2023), providing feedback to intelligent systems (Stamper al., 2024; Mai al., 2023), now handled by I.Mss. Among these applications, the use of L.MMs to substitute humans in idea evaluations (Lu et al., 2024; Back al., 2024) carries substantial potential, where researchers can obtain much faster feedback, as well as considerable risks, where the preference and bias of LLMs could affect the development of scientific domain. Concretely, is well known that many reviews for paper submissions are now written with the help of LLMs, which is explicitly allowed by ICLR 2025 as well. Unfortunately, existing LLMs are often biased to be "nice and helpful" while being highly sensitive to the prompt, illustrated by Figure 1. Therefore, this paper aims to highlight pressing research question: how do we improve the fidelity of -hsed idea evaluation?
Most existing research attempts to address the problem of LLM-- idea evaluation by designing better prompt strategy (Brown, 2020; Wei et al., 2022: Wang et al., 2022: Yao et al., 2024), so that more background knowledge, feedback, example, Research Agent (Back et al., 2024) evaluates the ideas based on its five criteria added in the prompt. AI Scientist (Lu al., 2024) introduces some prompt tricks like self-reflection (Shinn et al., 2024), providing few-shot examples (Wei et al., 2022), and response ensembling (Wang et al., 2022) enhance the idea evaluations. However, these prompt-based evaluation methods still
inductive bias can be incorporated to an LLM. For

---

Published as a conference paper at ICLR 2025
OOriginal Prompt, Positive Prompl, Negative Prompil:
Query: Title- "In-Comiest Policy teeraton", Absimct - "This work
-System Prompe- aru are an Al researcher who is reciewing a puper's tidle and abstract thst submined a prsstigions ML
presents In-Context Policy Iteration, algorithum for performing
critical and good and accept
yoar decision.
Reinforeemest Learning (RL.), in-context. using foundation models..." "
paper is good you
Original Prompt Result: Overall Score (0-100y- 78, Accept (Poster)
Ita paper &s had you are unsure, give had scores and reject it.
Positive Prompt Resalt: Overall Score (0-100)- Accece (Oral)
-lastruction= Please evaluate the paper draft based on the following six dimeusions....
Negative Prompt Resel:: Overall Score (0-100)- 75, Accept (Poster)
Figure I: Current LLMs are highly sensitive to prompts and show biases in evaluations. This figure illustrates that even minor variations in the LLM's prompts (Original Prompt, Positive Prompt, Negative Prompt) for the same idea can lead to drastic changes in the final LLM evaluation results. Moreover, the LLM tends to always give friendly evaluations like AAccept' and rarely gives negative evaluations such as 'Reject'. This observation demonstrates that the LLM evaluation is biased.
Idea
Evalunte Ruse on GraphEvvll (Qurs)
(I) Graph Neural Networks (GNNs) have emerged is powerful rool for msddeling graphb-trruetredd dats. (3) Recent advancements
Mean
Averuge
Entire
80
Pooling
Information
60
demonstrated that GNNs outperform traditional neural networks on tasks such node classification and prediction.
2
75
LLM
(3) The Transformer model, originally developed for compater vision, has also been widely adopted in natural language processing tasks.
Split
Min
Pooling
Overall Score 85,
(4) In many appllcations, GNNs eshibit superior scalability compared
Overall 37.5,
Accept
convolutional neural networks.
Roject
Figure 2: GraphEval performs better idea evaluation than the existing LLMd method by focusing on both the global and local information of the idea. In this figure, the part highlighted
in red in the idea contain factual errors. The existing LLM-based method shown on the far left focuses solely on the global information of the idea, which often leads to overlooking factual errors interspersed within the idea. In contrast, GraphEval decomposes the idea into viewpoints to obtain scores for each viewpoint, then employs Mean Pooling and Min Pooling to extract global and local information of the idea, respectively. Finally, GraphEval 1 derives a fair and unbiased evaluation based on these two aspects of information.
limited because: 1) they are highly sensitive different prompts (Errica et al., 2024; Zhang et al., 2024a) and are prone to hallucinations (Sansford et al., 2024; Yao et al., 2023a); 2) they also require LLMs to possess advanced capabilities (Santu et al., 2024; Liang et al., 2024) to fully understand and judge a complex research idea, which often requires PhD-level humans in the real-world; 3) they could overlook factual inaccuracies interspersed among the ideas. As illustrated in Figure existing I.L.-based methods directly analyze the entire set of information, therefore easily missing the factual errors within the idea, leading to a biased evaluation.
Many studies in human psychology (Knauff & Wolf, 2010; Dijkstra et al., 2014) indicate that people often find it difficult understand abstract ideas, which can lead to random cognition and decision- making. However, two methods can significantly enhance human understanding of abstract ideas: 1) by breaking down complex abstract ideas into simpler viewpoints, it becomes easier for humans to understand (Cowell et al., 2019; Rips et al., 2012); 2) showing the connections between the complex ideas and other ideas can also improve human's 's understanding on these complex ideas (Huang, 2020; Hayes & Kraemer, 2017; Khatin-Zadeh & Farsani, 2022).
Inspired by the psychological findings above, we propose GraphEval, a lightweight graph-bases LLM framework for idea evaluation, which breaks down complex ideas into simple and compre- hensible viewpoints, and bridges different viewpoints into a viewpoint-graph. Specifically, we first deconstruct complex and difficull-t-uunderstaadd ideas into simple viewpoints using a prompt-based approach with a (small) LLM. Then, we treat each viewpoint as a node and construct edges through LLM--bsed relation extraction and/or BERT similarity scores. Finally, we create a viewpoint- graph by joining the viewpoints and edges across different ideas. Based on this, we propose two lightweight graph-based methods for idea evaluations: 1) GraphEvvl-LLP: It is training-free framework based on graph label propagation algorithm. It operates by transferring quality labels from labeled nodes to unlabeled nodes through weighted edges, ultimately predicting the final evaluation of an idea based on the labels of the viewpoint-subgree framework based on Graph Neural Networks (GNN) that requires minimal training. It models idea evaluations graph prediction problem using GNNS, obtaining evaluation results by predicting the attributes or classifications of viewpoint -uppoittssuurrpphh.... Moreover, in order to objectively assess the novelty of ideas, we have also added a plagiarism detection mechanism to the GraphEval-GNN,
associated with it. 2) GraphEval-GNN: It is deep learning
2

---

Published as a conference paper at ICLR 2025
Specifically, have incorporated temporal information into the features of the niewpoint-nodee and deliberately constructed plagiarized ideas along with their negative evaluation labels as negative samples in the GNN training process. By doing so, enable the GraphEval to learn to give lower evaluation to those ideas that are plagiarisms of previous research.
In summary, main contributions
follows:
To the best of our knowledge, we are the first to investigate -aseed idea evaluation from a graph perspective, offering new insights into graph-enhanced LLM research.
We propose a lightweight graph-based LLM framework called GraphEval for idea evaluations, which includes GraphEval-LP and GraphEval-GNN methods. It breaks down the complex ideas into simple viewpoints, connects the viewpoints into a viewpoint-gras -gaaph, and models the idea evaluation as a node-level prediction task on the viewpoint-graphe
Extensive experiments on two datasets have demonstrated that GraphEval 1 can achieve at least a 14% improvement in Fl score with low computation cost and API cost compared with other baselines. Additionally, GraphEval can effectively detect plagiarized ideas and provide fair evaluation.
2 RELATED WORKS
Automatic Idea Evaluation. The rapid growth of research idea production and intricate knowledge specialization challenge conventional scientific feedback mechanisms (Liang et al., 2024), prompting researchers to explore Al for automated idea evaluation to accelerate the academic innovation cycle. For example, Sun & Li (2021): Li et al. (2019) investigated the use of CNNs for evaluating academic innovation and design, while Siemon (2023); Bell al. (2024) analyzed the automated idea evaluation process from a Human-Compute Interaction perspective. In addition, numerous studies employed fine-tuned lightweight language models (e.g., BERT (Devlin, 2018)) to evaluate complex texts, such as dialogues (Thida, 2021), tweets (Pota et al., 2021), and the novelty of ideas (Just et al., 2024; Yuan et al., 2022). However, most of these methods require extensive training on large-scale data and face limitations in generalizability (Sun & Li, 2021; Just et al., 2024). Conversely, recent studies have sought to leverage the domain knowledge and logical capabilities of LLMS to create idea evaluators (Ubonsiri, 2024; Back al., 2024: Du al., 2024: Lin al., 2023a). Du al. (2024) proposed using a prompt-based approach to allow LLMs to act as reviewers and meta-reviewers in order to assess the level of papers/ideas based on different evaluation criteria. Xu et al. (2024) utilized representations from specific layers of LLMs for evaluation, Shankar et al. (2024) aligned LLM evaluators with human preferences through feedback, and Lu et al. (2024) enhanced the srciision-mkinng ability of LLM-based evaluators via self-reflection, few-shot learning, and response integration. Furthermore, Si et al. (2024) measured the consistency gap between LLMS and human expert reviews. However, when faced with the inherently complex semantic information of research ideas (Back et al., 2024) and the subjectivity of the evaluation task (Si et al., 2024), the decision- maaiin consistency between LLMs and human reviewers remains limited, often leading to LLMs struggling to provide high- quality feedback (Si et al., 2024; Liang et al., 2024; Lu et al., 2024). Recently, some research works have been evaluating long-form texts, such as biographies of people (Min et al., 2023) and complex mathematical reasoning texts (Lightman et al., 2023). These studies divide the long text into multiple subsets and evaluate each of them. Inspired by these works, we decompose the obscure ideas into simple, understandable viewpoint nodes using LLMs, and further evaluate the idea based on graph algorithms.
Graph for LLMs. The use of graphs in conjunction with LLMS is an emerging research area, with several established directions. These include integrating LLMs with path selection mechanisms to learn unified graph representations (Shang et al., 2024); constructing graph-based text indexes using L.LMs to answer questions over private text corpora (Edge et al., 2024); and utilizing L.MSs for knowledge graph creation (Yang et al., 2024; Zhu et al., 2024; Carta et al., 2023; Trajanoska et al., 2023) and completion (Yao et al., 2023b). In addition, Zhang et al. (2024b) proposed the NLGift benchmark, which focuses on the evaluation of LLM graph reasoning generalization; Perozzi et al. (2024) introduced GraphTokens which can explicitly represent structured data for LLMs; Shi et al. (2024) introduced novel recommender that synergizes LLMs and KGs to enhance recommendations and provide interpretable results. In terms of apen-source software, various graph databases are supported by both the LangChain (LangChain, 2024) and LlamaIndex (llmmandexx, 2024) libraries.
3

---

Published as a conference paper at ICLR 2025
Idea List
Viewpoint-Grapp Extraction
GraphEvab-LP
Label propagation
La6el prediction
(a) Develop GNN model that dynamically
L4 i
topology of dynamic graphs. (b) Investigate
Viewpuint
Sum Arginax
methods interpret bow node representations
13
Viewpoint subgraph
node
Label
learned. (e) Design framework for training
die
Edge feature
GraphEval-GNN
GNNs
while preserving data
privacy, (d)
Explore the use of GNNs
GrapbCony
L layer
Softmax
AGG
g
predicting future events in spatiotemporal data.
Figure 3: Overview of GraphEval methodology. GraphEval first transforms the ideas into viewpoint-graph via Viewpoint-Graph Extraction, which contains multiple viexpoint-subgraaph,, viewpoint-nodes. and edges between newpoint-nodes, Then two lightweight GraphEval imple- mentations named GraphEval-LP and GraphEval-GNn GNN are employed to evaluate the ideas. Note that AGG denotes the acronym for aggregation function.
However, leveraging LLMs to extract diverse viewpoints embedded in research ideas, structuring them as graphs, and using these for idea evaluation remains direction that warrants further exploration.
3. VIEWPOINT-GRAPH EXTRACTION: A GRAPH STRUCTURE FOR DIVERSE RESEARCH VIEWPOINTS AND THEIR RELATIONSHIPS
Problem Setup We consider a predefined set of quality labels Stabel for evaluating research ideas (e.g., categorical values [Reject, Accept (Poster), Accept (Oral)]). Given set of ideas [Do, Di,..., DnI. only a subset of these ideas has known quality labels during training, and our objective is to predict the quality labels for the remaining ideas test time.
Framework Overview Figure 3 provides an overview of the proposed GraphEval framework. The key insight of our approach is that by leveraging the summarization capabilities of LLMs (Jin et al., 2024; Ghosh et al., 2024), we can extract niewpoint-subgrapp from each idea's 's text, which serves as
representation that captures diverse viewpoints and the semantic relationships between them. Additionally, we connect multiple hiewpoint-subgraphh to construct a larger graph structure, the viewpoint-graph. graph, which acts as an extensible datahase, encompassing existing research viewpoints and their intricate interrelations. This allows to apply label propagation or GNN algorithms to evaluate ideas in the test set, using only the quality information from the training set ideas.
Viewpoint Extraction through Prompted LLMs The key challenges in LLM-based research idea evaluations are twofold: (1) Research ideas inherently encapsulate complex semantic information (Baek et al., 2024; Si et al., 2024), as a single idea often contains multiple distinct viewpoints rooted in different concepts, interconnected through intricate logical relationships that collectively define the idea. (2) Idea evaluation is fundamentally subjective (Si et al., 2024), which presents a significant challenge for ILMSs comprehension and reasoning abilities (Santu al., 2024; Liang al., 2024), often resulting in severe biases and a lack of alignment with human evaluations (Lu et al., 2024).
To address these challenges, we utilize LLMs to extract fine-grained components from research ideas, which we refer to as viewpoints. A viewpoint can be an idea, argument, or fact embedded within the research content. These viewpoints are semantically independent, evaluable units that are made as granular as possible to ensure they cannot be further decomposed. For a given research idea D,, we employ a prompted LLM Lp extract a list of viewpoints: [r]: vf...., example of viewpoint extraction is illustrated in Appendix A.
. o = (D.)). A simple
By extracting viewpoints, we decompose semantically intricate research ideas into fine- aie-ggainedd and semantically independent units. In this process, we utilize prompted LLMs to extract elements as an objective and straightforward NLP task (Manning, 1999; Rush, 2015), relying solely on the models' summarization and abstraction capabilities (Jin et al., 2024; Kurisinkel & Chen, 2023). This approach typically induces fewer biases compared to subjective judgment tasks that necessitate deeper comprehension and reasoning of complex text (Zhang et al., 2023; Zheng et al., 2023). Additionally, allows us to leverage smaller LLMs (c.g., those with 7 billion parameters) to implement the GvaphEval framework, resulting in significant resource savings.
4

---

Published as a conference paper at ICLR 2025
Table I: LLM-based relation extraction yields relatively few relevant relationships from the viewpoints, resulting in siewpoint-subgraphs with overly sparse edges. We conduct LLM-based viewpoint and relation extraction on 300 research ideas, quantifying the average length of each idea and extracted viewpoints, the average number of viewpoints extracted per idea, the average edge number and edge density of each viewpoint-subgraph,, with text length measured in word count. The LLM used is Mistral (7B) Instruct v0.3 (Jiang et al., 2023).
Avg. Idea Len.
Avg. Viewpoint Num. Avg. Viewpoint Len.
Avg. Edge Num.
Avg. Edge Density
174.60
8.83
20.19
3.71
10.73%
Viewpoint-subgraph Construction through Prompted LLMs To identify the semantic relation- ships between viewpoints extracted from an idea, we utilize prompted LLMs for relation extraction (Wei et al., 2024; Jinensibieke et al., 2024). Specifically, we treat each viewpoint a node in graph, referred to as a viewpoint-node. We then input the viewpoint list into the prompted LLM, instructing it extract semantically related viewpoint pairs. These pairs are subsequently considered as edges connecting the corresponding viewpoint-nodess We refer to the graph constructed from a research idea as a hetpwoont-subbraaae
To validate the feasibility of using prompted LLMs for relation extraction, we collect 300 submissions the ICLR conferences between 2021 and 2023, treating the abstracts of these academic papers as representations of research ideas (Si et al., 2024; Baek et al., 2024). We perform L.LM-base viewpoint and relation extraction on these research ideas. As shown in Table 1, the relation extraction yields relatively few relevant relationships from the viewpoints, resulting in viewpoint-subgraahh with overly sparse edges. This leads to an excess of isolated viewpoint-nodes in each niewpoint-subgrann and deficiency in the inherent relational information. Additionally, the LLM-based relation extraction incurs extra resource costs.
To address these issues, propose a method for relation extraction based embedding similarity.
Viewpoint-subgraph Construction through BERT-based Encoder To automatically identify logical relationships between viewpoints, we a BERT-based encoder E' to obtain embed- dings of equal dimensions e for each viewpoint-node v (Qasim et al., 2022: Lin et al., 2023b): [e1, 62, ........ cn] E([11, 122 ..., U,])). Then, we compute the cosine similarity between their embeddings: s(c,, ez) = Te, Ille,, (Izacard et al., 2021; Li et al., 2023). Each viewpoint-node is connected to the top-k nodes with the highest embedding cosine similarity using weighted undirected edges, with the edge weights set to the cosine similarity (Harnoune et al., 2021). This way, we construct the neewpoint-subgraphe which serves as a high-granularity representation of the research idea. Additionally, by controlling the value of k. we can regulate the edge density, allowing for the construction of -ubrrrppsss that are more suited to specific downstream tasks.
Viewpoint-graph Construction through Connecting Viewpoint-subgraphs After transforming the ideas in both the training and test set into suwppoit--uubrraphs. we connect them to construct a larger graph. Specifically, similar to the construction of viewpoint-subgraphe for each viewpoint- node, we connect similarity using undirected weighted edges. We refer to the graph constructed as the viewpoint-geat which integrates the diverse viewpoints of different research ideas and the interrelations between them. The viewpoint-gre G can be represented by node list and an edge list:
the top-m nodes from different subgraphs with the highest embedding cosine
}}} G = [[(vo, eo), .... (Un, e,)] [( Uky Wkok)) ), ..., (Ukmn y URim+ Wkmm Prmn
Notably, the viewpoint-graph is scalable, allowing new viewpoint-subgraphs to be integrated in linear time, providing a theoretical foundation for its expansion as new ideas are generated.
4 GRAGHEVAL-LP: A SIMPLIFIED AND LIGHTWEIGHT IMPLEMENTATION
After obtaining the viewpoint-graph G, we would like to validate its efficacy by first applying a simple and lightweight algorithm, label propagation (Raghavan et al., 2007: Zhang et al., 2017), to evaluate the ideas in the test set. Our results in Section 7 show that this simple algorithm is already very effective with idea evaluations. We refer to this evaluation framework GraphEval-I.P.
Initialization and Regularization For each viewpoint-node v; in G, we maintain a vector d, where each dimension corresponds to quality label in Stabet. Thus, the dimensionality of d, is given by
5

---

Published as a conference paper at ICLR 2025
|dy] -Stabell- For viewpoint-noke nodes extracted from ideas in the training set. assign value of to the dimension corresponding to the idea't 's label, while all other dimensions are set to 0. In contrast, viewpoint-nodee extracted from the test set initialized as vectors. Additionally, the edge weights wi in G to ensure that the sum of the weights of all edges connected to any given viewpoint-node ti equals 1, i.e., DjEN() Wij where (i) represents the set of neighbors of u,-
regularize
Label Propagation We perform multiple iterations of label propagation graph G until the labels no longer change. Specifically, in each iteration, each node updates its vector by adding the weighted vectors of its neighboring nodes:
((++))
(1
wiyd JEN(1)
Σ
(2)
Zi
Where df is the vector of node v at iteration 6, and Z, is normalization factor that ensures the updated vector is properly scaled.
Label Prediction After completing label propagation, we sum the vectors of the viewpoint-nodes corresponding to each idea in the test set. The predicted label ý is then determined by selecting the
dimension with the highest value in the summed vector, i.c., ÿ = arg max, (Ei=1 d.), where indexes the dimensions of the vector and k means the number of viewpoints for given research idea.
GRAGHEVAL-GNN: A GRAPH NEURAL NETWORK-BASHE SOLUTION
Although label propagation is effective, it does learn how to properly propagate evaluation scores from known nodes to unknown nodes. Therefore, we further propose a learning-bann approach, GraphEval-GNN, which trained to predict the evaluation scores for a viewpoint-node
Method Overview. As shown in Figure 3, GraphEval-GNN models viewpoints as viewpoint-nodes, while the relationships between viewpoints are represented by edge features. We apply GNN to embed the node and edge features and use them for training and testing.
Initialize node/edge features. As illustrated Sec. 3, we initialize the viewpoint-node features he by converting viewpoints into embeddings using BERT. Since the relationships between viewpoint- nodes encompassing the similarity relations obtained from BERT, we initialize the edge features wn using this relational attribute.
Predict via a weighted GNN. We implement the predictive model fa over viewpoint-nodes using weighted GNN, shown in Figure 3. The objective of the GNN is to learn expressive node embeddings h, through an iterative weighted aggregation of the local network neighborhoods. The 1-th iteration of the GraphCony((), the node embeddings update of the 1-th layer, is represented as:
N(o))) h U0 MEEAN (RELLU w (H)g")))))) E & (())))
(3)
where h is the node embedding after iterations, ())) have been initialized as explained above, (C) denotes the direct neighbors of v, and U (()) are learnable parameters.
Since the evaluation of idea determined by all the viewpoints extracted from we further model the LLM evaluation problem as a sub- nub-graph prediction problem and aggregate all the node embeddings into a subgraph embedding. Moreover, as introduced in Figure 2, we consider MEAN Pooling and Max Pooling simultaneously to extract global and local information of the idea. Specifically, the sub-gr -obpgraph probability distribution dD. of idea D; can be made through GraphPred(.)) the form of:
Sn, SOFTMAX (MLP (CONCAT (MBAN { h9 : L.(D)) MAX (n.) v Lp(D.))))
We have summarized the detailed training process of GraphEval in Algorithm 1. In addition. in the testing of GraphEval, we choose the category with the highest output probability as the result of the L.M evaluation.

---

Published as a conference paper at ICLR 2025
Algorithm 1 Training of GraphEval
Require: Dataset Derain = {(x, v)). A weighted GNN fg. Edge weights wa. Number of GNN
layers L.
Initialize the embeddings of viewpoint node, h using BERT.
2: for each iteration do
3: N + SamppeeMiiiidgeBBachhhh Purii))
4: Mask the Viewpoint--ubggapphs in Drain that are in N, and obtain the labels of the viewpoint-
subgraphs in T
5: for 1 1 to L do
h + GraphConv(h (1) w.) with fo
6:
Backward (Criterion (D., fer) E &N))
7:
GraphEval for idea novelty assessment. Assessing the novelty of ideas is crucial, as plagiarized or derivative ideas can sometimes mislead LLMS into giving them higher evaluation scores. As concrete example, if the same idea is being evaluated by LLM twice, LLM will always assign the same evaluation score, since does not take the novelty aspect into account when evaluating ideas.
To address this issue, enforce our GraphEval-GNN to learn that ideas and viewpoints appearing later time and exhibiting high similarity to earlier ones should be evaluated with lower scores. Specifically, our approach focuses two key aspects. First, incorporate temporal features into the viewpoint representations, enabling the model to capture the chronological sequence of viewpoints. Second, we artificially generate duplicated ideas and viewpoints that direct combinations of existing viewpoints in the viewpoint-graph, label them with lower evaluation scores as negative samples, and include them in the GNN training process.
EXPERIMENTAL SETUP
Task. Following the works of Si et al. (2024); Baek al. (2024), we treat the abstract of an academic paper as a representation for the research idea, since it typically offers a concise summary of the research problem, the scientific methods employed, the experimental design, and the key contributions. Specifically, we provide each method with the abstracts and titles the academic papers, tasking them with evaluating the review decision: Reject, Accept (Poster), Accept (Oral), or Accept (Spotlight).
Datasets. We employ two datasets to thoroughly evaluate the proposed GraphEval framework:
ICLR Papers:
We collect abstracts and review decisions from paper submissions the ICLR conferences between 2021 and 2023. From this, we randomly select 300 papers as the training set for learning- based methods and 50 papers the test set.
AI Researcher Dataset: We the dataset collected by Si al. (2024) in Al Researcher as additional test set, which contains academic papers focusing on the domain of "novel prompting methods." " Note that due the scarcity of Accept (Oral) and Accept (Spotlight) labels in this dataset, we combine them into single label, thereby transforming the task into three-class classification problem.
The details of the datasets can be found in Appendix C.
Baselines. To gain a comprehensive understanding of the performance of our proposed framework in evaluating research ideas, we have adopted several baselines:
We provide several criteria for assessing research ideas in the prompt. Addi- tionally, we present specific standards for the four review decisions and include one example for each as few-shot examples for in- mo-coteet learning (Brown, 2020). Moreover, we include the label distribution of the dataset to help the LLMs understand the frequency of each review decision.
Prompted LLM:
CoT prompt: Drawing inspiration from Wei et al. (2022), we modify the prompt used for prompted LLM to adopt a CoT format, guiding it to complete the idea evaluation step by step.
CoT-SC: consisensst with CoT (CoT-SC) is an ensemble approach that samples i.i.d. CoT, then returss the most frequent output (Wang al., 2022).
7

---

Published as a conference paper at ICLR 2025
ToT prompt: Tree of Thoughts (ToT) is extension of CoT (Yao al., 2024). Similar to CoT, we divide the evaluation process into multiple steps. At each step, we sample branch = i.idd. CoTs, and pass the most frequent output the intermediate result to the next step.
Research Agent: We adopt the idea evaluation method from Research Agent (Back et al., 2024) as one of our baselines, where the research problem, scientific method, and experiment design of an idea are each scored based on five criteria. Building on this, we further introduce final decision step that synthesizes the above evaluation results to provide a comprehensive review decision.
Fine-tuned BERT: In addition to LLM-based methods, we fine-tune DistilBERT model (Sanh et al., 2019) using collected paper abstracts and review decisions as a baseline to validate the competitiveness of our approach compared to learning- naaning-basedd methods.
For all LLM-based baselines (except Fine-tuned BERT), we use two LLMs of different sizes: Mistral (7B) Instruct v0.3 (Jiang et al., 2023) and Qwen 2 Instruct (72B) (qwe, 2024). All prompts used in the methods can be found in the Appendix B.
Evaluation Metries. To comprehensively evaluate the consistency between the idea evaluation methods and human reviewers, we calculate the accuracy, macro precision, macro recall, and macro F1 score for each method. Additionally, we record the average token cost per evaluation as a measure of resource consumption. Note that for Mistral (7B) Instruct vO.3, the API cost is $0.20 per 1M tokens, and for Qwen Instruct (72B), the API cost is $0.90 per 1M tokens. 1 We calculate the average cost per evaluation for each method according to these pricing standards. To intuitively illustrate the resource consumption of each method, we normalize the average costs by setting the highest-cost method to 1, which we refer to normed cost. A smaller normed cost indicates lower resource expenditure.
Implementation Details. During the training phase, we configured the graph neural network as a two-layer weighted GNN with a hidden dimension of 64. The batch size is set to 64, and the maximum number of training epochs is limited to 1000. We employ the Adam optimizer (Diederik, 2014) for training and gradually reduce the learning rate from 1e-3 to using a LambdaLR scheduler. Our proposed method is implemented using PyTorch² and PyTorch Geometric (PyG)', with all experiments conducted on a single NVIDIA A 100 Tensor Core GPU. For the LLMs, we utilize API calls from Together AI obtain responses. Additionally, the average GPU memory usage of GraphEval-GNN for the two tasks is 372MB, whereas Fine-tuned BERT utilizes 4.84 GB on average.
7 EXPERIMENT RESULTS
7.1 COMPARISON WITH EXISTING BASELINES.
We report the performance of our methods and baselines in Tables 2 and
(1) Across all datasets, GraphEval-GNN significantly outperforms all baselines: for the ICLR Papers dataset, it achieves a 10%-72% accuracy advantage and an 18%-42% macro Fl score advantage: for the AI Researcher dataset, it achieves a 13%-53% accuracy advantage and a 14%-48% macro Fl score advantage. Moreover, its normed cost on both datasets demonstrates its utilization of resources comparable to the minimum expenditure level. This indicates that by leveraging smaller LLM (7B parameters) to convert semantically complex research ideas into more granular viewpoint-graphh and utilizing GNN algorithms to extract global and local information, we achieve precise evaluation of research ideas.
(2) Regarding the prompt-based baselines, they generally achieve lower accuracy and macro Fl scores. Our observations indicate that all these methods tend to overestimate the quality of ideas, with very few ideas being rejected. This aligns with findings in previous works (Lu et al., 2024; Si et al., 2024). Furthermore, we find that using larger-scale LLMs does not consistently improve evaluation performance; rather, it often leads to a decline: In experiments, the 72B model tends to provide consistent or similar decisions and overall scores for different ideas. This suggests that L.LMs exhibit
'The API pricing referenced here is based on the rates provided by https//wwww.ogglteerr///rpgngg "nttps://pytorchh org/
wtwp:rrrnceeeommeiiiireeeeee rewdteedocs.e... readthedos..i//e//atteet
8

---

Published as a conference paper at ICLR 2025
Table 2: GraphEval-GNN consistently outperforms all baselines in the ICLR Papers Dataset while utilizing resources comparable the minimum expenditure level. Bold and underlined text denotes the best and second-best results. Specifically, for accuracy. macro precision, macro recall, and macro score, higher values indicate more precise predictions of the labels for research ideas the test set. Conversely, for the normed cost, lower values represent reduced resource expenditure. Since Fine-tuned BERT is not an LLM-based method, its token cost and normed cost are not calculated.
Dataset
ICLR Papers
Method Metric
Accuracy
Precision
Recall
F1 Score
Token Cost
Normed Cost
Prompted L.MM (7B)
16.00%
4.55%
16.67%
7.14%
1968. 22
0.06
Prompted LLM (72B)
6.00%
4.09%
31.25%
6.35%
1735.30
0.24
CoT prompt (7B)
16.00%
5.00%
16.67%
7.69%
2443.28
0.07
CoT prompt (72B)
6.00%
5.36%
27.08%
5.05%
2415.62
0.33
CoT-SC (7B)
20.00%
5.21%
20.83%
8.33%
3121.72
0.10
CoT-SC (72B)
4.00%
1.19%
25. 25.00%
2.27%
3428.
0.47
ToT prompt (7B)
8.00%
4.95%
18.75%
6.47%
8963,92
0.27
ToT prompt (72B)
4.00%
1.06%
25.00%
2.04%
6211.46
0.85
Research Agent (7B)
12.00%
8.11%
22.92%
10.05%
7909.18
0.24
Rescarch Agent (72B)
6.00%
5.30%
31.25%
7.17%
7278.72
1.00
Fine-tumed BERT
66.00%
27.22%
28.39%
26.01%
GraghEval-LP (Ours)
70.00%
37.61%
32.55%
32.16%
2672.95
0.08
GraghEval-GNN (Ours)
76.00%
56.59
42.63% %
43.59%
2672.95
0.08
Table 3: GraphEyal-GNN consistently outperforms all baselines in the AT Researcher Dataset while utilizing resources comparable to the minimum expenditure level. Bold and underlined text denotes the best and second-best results. Since Fine-tuned BERT is not an LLM-based method,
its token cost and normed cost not calculated.
Dataset
Al Researcher
Method Metric
Accuracy
Precision
Recall
Score
Token Cost
Normed Cost
Prompted LLM (7B)
26.98%
50.40%
36.44%
24.75%
1961.41
0.06
Prompted LLM (72B)
30.16%
52.88%
41.99%
28.33%
1717.57
0.23
CoT prompt (7B)
30.16%
51.44%
37.09%
21.18%
2410.06
0.07
CoT prompt (72B)
23.81%
50.51%
34.97%
22.86%
2263.92
0.31
CoT-SC (7B)
31.75%
26.61%
41.67%
26.43%
2854.44
0.09
CoTSCC (72B)
25.40%
52.36%
37.75%
24.37%
3157.40
0.43
ToT prompt (7B)
30.16%
42.66%
29.04%
19.89%
9829.14
0.30
ToT prompt (72B)
25.40%
51.78%
39.38%
22.93%
6071.98
0.83
Research Agent (7B)
27.42%
19.78%
38.19%
24.24%
7887.44
0.24
Research Agent (72B)
20.63%
14.53%
32.03%
18.71%
7345.06
1.00
Fine-tuned BERT
60.00%
54.44%
54.44%
53.33%
GraghEyal-L.P (Ours)
70.47%
61.11%
55.56%
56.97%
2541.17
0.08
GraghEval-GNN (Ours)
73.33%
81.67%
73.33%
67.13%
2541.17
0.08
significant bias when faced with subjective judgment tasks that necessitate a deeper understanding and reasoning of complex text (Zhang al., 2023; Zheng et al., 2023), irrespective of their model size and capability. On the other hand, the GraphEval framework mitigates bias by transforming the LLM's task into the objective and straightforward task of extracting elements.
(3) Compared to CoT-SC, the ToT prompt and Research Agent, which utilize more complex prompting techniques, do not demonstrate significant advantages. This suggests that prompting techniques have limited efficacy in enhancing the capabilities of LLMs for tasks requiring complex comprehension and reasoning.
(4) Although fine-tuned BERT achieves better results compared to other prompt-based baselines, still falls short of the performance level of GraphEval. This is due to the construction of the viewpoint-graph, which allows GraphEval-LDP -L and GraphEval-GNN to obtain quality information about viewpoints locally and capture the intricate interrelations among diverse viewpoints globally, thereby leading improved performance.
(5) GraphEval-LP consistently achieves the second-best results across both datasets, and it does not require training, making it efficient and lightweight. GraphEval-L.P effectively demonstrates the

---

Published as a conference paper at ICLR 2025
Without Novelty Assessment
With Novelty Assessment
Values
0.4
Accuracy
Precision
Recall
F1 Score
Figure 4: Novelty assessment can significantly improve the performance of GraphEval when detecting plagiarized or derivative ideas. We compare two variants of GraphEval on the ICLR Papers dataset and evaluate their performance four metrics.
strong utility of the constructed viewpoint-gras grapph for research idea evaluation, owing to the inherent correlations between research ideas, such as shared common viewpoints. These implicit correlations cannot be effectively leveraged by prompt-based methods or fine-tuned language models.
(6) The comparison between GraghEval-LP and GraghEval-GGig GGNN demonstrates that: 1) Deep learning can enhance the performance graphs when applied for LLM evaluation tasks; 2) Although the introduction of GNN has improved performance, it also results in increased computational cost. Therefore, in our paper, we propose these two implementations to provide options for users with different needs.
7.2 GRAPHEVAL FOR NOVELTY ASSESSMENT.
To evaluate the effectiveness of novelty assessment on the ICLR Papers dataset, we artificially construct 80 plagiarized ideas for testing. Specifically, we employ three strategies to evenly create these 80 ideas: 1) We directly copy highly-rated ideas from the dataset; 2) We randomly replace some viewpoints in highly-rated ideas with viewpoints from other ideas; 3) We substitute some viewpoints in highly-rated ideas with those of their neighboring nodes based on the similarity of embeddings. Subsequently, we select 10 of the above idcas and construct negative samples using the method mentioned in Sec 5, which are then provided to GraphEval for training. We compare the impact of considering Novelty Assessment on the performance of GNN across four metrics, as shown in Figure 4. We can observe that Novelty Assessment significantly improve the performance of GraphEval when detecting plagiarized or derivative ideas.
8 CONCLUSION
In this paper, we propose novel lightweight graph- based LLM framework, CraphEval, for idea evaluation, addressing the complexities and subjectivity inherent in this task. Drawing inspiration from human psychology, we break down complex ideas into simpler viewpoints and model the relationships between them using viewpoint-graphs. Our framework includes methods: GraphEval-LP, training-free -fren approach utilizing label propagation, and GraphEval-GNN, a deep learning-bai based method using Graph Neural Networks. Both methods effectively leverage viewpoint -raiint-ggahsss to predict idea evaluations, while GraphEval-Gii GaNNEN also incorporates plagiarism detection mechanism that ensures fair and objective assessment novelty.
Through extensive experiments two datasets, we demonstrate that GraphEval not only achieves significant improvement in accuracy and macro score compared to multiple baselines but also op- erates with low resource expenditure. Our work pioneers the integration of graph-based approaches with LLMs for idea evaluation, providing new insights for enhancing LLM-based evaluations with graph representations.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Qwen2 technical report. 2024.
Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprini arXiv:2404.07738, 2024.
J Jason Bell, Christian Pescher, Gerard J Tellis, and Johann Füller. Can ai help in ideation? theory-based model for idea screening in crowdsourcing contests. Marketing Science, 43(1):54-72. 2024.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Salvatore Carta, Alessandro Giuliani, Leonardo Piano, Alessandro Sebastian Podda, Livio Pompianu, and Sandro Gabriele Tiddia. Iterative zero-shot Ilm prompting for knowledge graph construction. arXiv preprint arXiv:2307.01128 2023.
Rosemary A Cowell, Morgan D Barense, and Patrick S Sadil. A roadmap for understanding memory: Decomposing cognitive processes into operations and representations. Eneuro, 6(4), 2019.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0423 2018.
Kingma Diederik, Adam: A method for stochastic optimization. (No Tite),, 2014.
Katinka Dijkstra, Anita Eerland, Josjan Zijlmans, and Lysanne S Post. Embodied cognition, ab- stract concepts, and the benefits of new technology for implicit body manipulation. Frontiers in psychology, 5:757, 2014.
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, al. Llms assist nip researchers: Critique paper (meta-) reviewing. arXiv preprint arXiv:2406. /6253, 2024.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local global: A graph rag approach query-focused summarization. 2024. URL https: / arxiv. org/ abs . 16130.
Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. What did i do wrong? quantifying Iims' sensitivity and consistency to prompt engineering. arXiv preprint arXiv:2406. /2334, 2024.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304. 20.00544 2023.
Akash Ghosh, Arkadeep Acharya, Raghav Jain, Sriparna Saha, Aman Chadha, and Setu Sinha. Clip- syntel: clip and llm synergy for multimodal question summarization in healthcare. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 220319 2024.
Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Soña Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, al. Llms accelerate annotation for medical information extraction. In Machine Learning for Health (ML4H)), pp. 82-100. PMLR, 2023.
Ayoub Harnoune, Maryem Rhanoui, Mounia Mikram, Siham Yousfi, Zineb Elkaimbillah, and Bouchra El Asri. Bert based clinical knowledge extraction for biomedical knowledge graph construction and analysis. Computer Methods and Programs in Biomedicine Update, I: 1:100042, 2021.
Justin C Hayes and David JM Kraemer. Grounded understanding of abstract concepts: The case of stem learning. Cognitive research: principles and implications, 2:1-15, 2017.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval,
pp. 639-648, 2020.
11

---

Published as a conference paper at ICLR 2025
Yanli Huang. How to represent abstract concepts? from the perspective of conceptual metaphor theory. Hum Psychol, 1(2)27-37, 2020.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXhh:2112.0911111 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv. org/abs/ 2010 06825.
Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. A comprehensive survey on process-oriented automatic text summarization with exploration of Ilm-based methods. arXiv preprint arXiv:2403. 02900, 2024.
Dawulie Jinensibicke, Mieradilijiang Maimaiti, Wentao Xiao, Yuanhang Zheng, and Xiangbo Wang. How good are 1lms at relation extraction under low-resuurce scenario? comprehensive evaluation. arXiv preprint arXiv:2406. /62, 2024.
Julian Just, Thomas Ströhle, Johann Füller, and Katja Hutter. Ai-based novelty detection in crowd- sourced idea spaces. Innovation, 26(3):359-388 2024.
Omid Khatin-Zadel and Danyal Farsani. The understanding of abstract concepts: perspective from distributed models of conceptual representation. Discover Psychology, 2(1):34, 2022.
Markus Knauff and Ann G Wolf. Complex cognition: the science of human reasoning, problem- solving, and decision-making, 2010.
Litton Kurisinkel and Nancy F Chen. Lim based multi-document summarization exploiting main- event biased monotone submodular content extraction. arXiv preprint arXiv:23100034114 2023.
LangChain. Langchain graphs. httpp::/ 1angcha in. com/ www.//sseeceee graph/, 2024.
Baorui Li, Yi Wang, Kesheng Wang, and Jinghui Yang. Application of enn deep learning product design evaluation. In Advanced Manufacturing and Automation VIII 8, pp. 517-526. Springer, 2019.
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320, 2023.
Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? large-scale empirical analysis. NEJM AI, 1()):1l02240019, 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023.
Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. Automated scholarly paper review: concepts, technologies, and challenges. Information fusionn, 98:101830, 2023a.
Sheng- Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452. 2023b.
Sheng-Chieh
Llamalndex.
Llamaindex knowledge graph index. httpl///xxapleseeeeeee sttrddee__ttutte_eeeeee html, 2024.
12

---

Published as a conference paper at ICLR 2025
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.00292. 062422 2024.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. Chatgpt as factual inconsistency evaluator for text summarization. arXiv preprint arXiv:2303./502/ 2023.
Jinjie Mai, Jun Chen, Guocheng Qian, Mohamed Elhoseiny, Bernard Ghanem, al. Lim as a robotic brain: Unifying egocentric memory and control. 2023.
Christopher D Manning. Foundations of statistical natural language processing. The MIT Press, 1999.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint ar0iv: 14251, 2023.
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. Entity-level factual consistency of abstractive text summarization. arXiv preprint arXiv:2102.09111 2021.
Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for llms. arXiv preprint arXiv:2402.05862, 2024.
Marco Pota, Mirko Ventura, Hamido Fujita, and Massimo Esposito. Multilingual evaluation of pre-processing for bert-based sentiment analysis of tweets. Expert Systems with Applications, 181: 115119, 2021.
Rukhma Qasim, Waqas Haider Bangyal, Mohammed A Alqarni, and Abdulwahab Ali Almazroi. A fine-tuned bert-based transfer learning approach for text classification. Journal of healthcare engineering, 2022(1):349812) 2022.
Usha Nandini Raghavan, Réka Albert, and Soundar Kumara. Near linear time algorithm detect community structures in large-scale networks. Physical Review E Statttisticall, Nonlinear, and Soft Matter Physics, 76(3):036106, 2007.
Lance Rips, Edward Smith, and Douglas L Medin. Concepts and categories: Memory, meaning, and metaphysics. The Oxford handbook of thinking and reasoning, 1:177-209, 2012.
A Rush. A neural attention model for abstractive sentence summarization. arXiv Preprint, CoRR, abs/l509.000l 2015.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108, 2019.
Hannah Sansford, Nicholas Richardson, Hermina Petric Maretic, and Juba Nait Saada. Grapheval: knowledge-graph based Ilm hallucination evaluation framework. arXiv preprint arXiv:2407.10793, 2024.
Shubhra Kanti Karmaker Santu, Sanjeev Kumar Sinha, Naman Bansal, Alex Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Guttikonda, Mousumi Akter, Matthew Freestone, et al. Prompting iims to compose meta-review drafts from peer-review narratives of scholarly manuscripts. arXiv preprint arXiv:2402. 15589, 2024.
Wenbo Shang, Xuliang Zhu, and Xin Huang. Path-Ilm: A chortest-path-bert 11m learning for unified graph representation. arXiv preprint arXiv:2408.0544 2024.
Shreya Shankar, JD Ceanffrrescu-Pereiin Bjöm Hartmano, Aditya G Parameswaran, and Ian Arawjo. Who validates the validators? aligning -lm-assitted evaluation of Im outputs with human prefer- ences. arXiv preprint arXiv:2404. 240.:22404..1222. 2024.
Guangsi Shi, Xiaofeng Deng, Linhao Luo, Lijuan Xia, Lei Bao, Bei Ye, Fei Du, Shirui Pan, and Yuxiao Li. Lim -ln-pedered explanations: Unraveling recommendations through subgraph reasoning. arXiv preprint 4406 15859, 2024.
13

---

Published as a conference paper at ICLR 2025
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Chenglei Si, Diyi Yang. and Tatsunori Hashimoto. Can ilms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04 2024.
Dominik Siemon. Let the computer evaluate your idea: evaluation apprehension human human-computer collaboration. Behaviour & Information Technology, 42(5):459-477, 2023.
John Stamper, Ruiwei Xiao, and Xinying Hou. Enhancing llm-based feedback: Insights from intelligent tutoring systems and the learning sciences. In International Conference on Artificial Intelligence in Education, pp. 32-43. Springer, 2024.
Lina Sun and Sitan Li. Cnn-based evaluation method of academic innovation effect of american research universities. In 202/ IEFE International Conference on Industrial Application of Artificial Intelligence (IAAT), pp. 355-360. IEFE, 2021.
Aye Thida. Bert-based dialogue evaluation methods with ruber framework. In Advances in Artifi- cial Intelligence: Selected Papers from the Annual Conference of Japanese Society of Artificial Intelligence (JSAI 2020), volume 1357, pp. 133. Springer Nature, 2021.
Milena Trajanoska, Riste Stojanov, and Dimitar Trajanov. Enhancing knowledge graph construction using large language models. arXiv preprint 2..iv:2305.04676, 2023.
Thanyalak Ubonsiri. AI-GENERATED EVALUATION: THE INFLUENCE OF CHATGPT EVAL- UATION ON INDIVIDUALS' DECIIIIOSS IN THE IDEA EVALUATION PHASE. PhD thesis, Innsbruck, 2024.
Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. Human-lin collaborative annotation through effective verification of Ilm labels. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-21, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint wxxiv:2033.1177., 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of-thouuhh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Kangda Wei, Aayush Gautam, and Ruihong Huang. Are Ilms good annotators for discourse-level event relation extraction? arXiv preprint wxXiv:2407.19568 2024.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference machine learning, pp. 6861-6871. PMLR, 2019.
Yi Xu, Bo Xue, Shuqian Sheng, Cheng Deng, Jiaxin Ding, Zanwei Shen, Luoyi Fu, Xinbing Wang, and Chenghu Zhou. Good idea or not, representation of Ulm could tell. arXiv preprint arXiv:2409.13711 2024.
Rui Yang, Boming Yang, Sixun Ouyang, Tianwei She, Aosong Feng, Yuang Jiang, Freddy Lecue, Jinghui Lu, and Irene Li. Graphusion: Leveraging large language models for scientifie knowledge graph fusion and construction in nlp education, 2024. URL https: /ttp//ixivvv..co/. org abs/ 2407 10794.
Jia- Yu Yao, Kun-Peng Ning, Zhen- Hui Liu, Mu-Nan Ning, and Li Yuan. Lim lies: Hallucinations are not bugs, but features adversarial examples. arXiv preprint arXiv:2310.01469, 2023a.
Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion. arXiv preprint arXiv:2308. 13916, 2023b.
14

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171-212, 2022.
Collin Zhang, John X Morris, and Vitaly Shmatikov. Extracting prompts by inverting Ilm outputs. arXiv preprint arXiv:2405. /501, 2024a.
Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing Hc, and Yulia Tsvetkov. Can Ilm graph reasoning generalize beyond pattern memorization' arXiv preprint arXiv:2406. 15992, 2024b.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's song in the ai ocean: a survey on hallucination large language models. arXiv preprint arXiv:2309.01219 2023.
Zh Zhang, Xiao- Yuan Jing, and Tie-Jian Wang. Label propagation based semi-supervised learning for software defect prediction. Automated Software Engineering, 24:47-69, 2017.
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. Why does chatgpt fall short providing truthful answers? arXiv preprint arXIv:2304.10513, 2023.
Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Hua- jun Chen, and Ningyu Zhang. Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities. World Wide Web, 27(5):58, 2024.
15

---

Published as a conference paper at ICLR 2025
A A SIMPLE EXAMPLE OF VIEWPOINT EXTRACTION
Here, present simple example of viewpoint extraction in Figure 5. For given research idea i, we employ a prompted LLM Lp to extract list of viewpoints: [Vo, U1,..., Un = (p)).
Viewpoint List Vo Vi, V2
Stare-of-the-ar computer systems
prodict a Eixed set of predotermined object callegoties.
Research Idea i
Prompted-I.LM Lp
sfate-of-th--rt compuseer vision systems are limited by being trained prodict a fixed set predetemained object categorics.
genemiitt and
State-of-the- art computer vision systems are trained to predict a fixed ser of predetermined object categories, supervision limits generality and usubility since additional labelod data needed specify other visual concept...
restricted
Additional laheled data is needed to speeify visual concepts that included fixed predetemnined object caregories.
Figure 5: Example of viewpoint extraction from a research idea. This figure illustrates how prompted LLM extracts fine-graine fine-grained viewpoints from research idea. Each viewpoint represents an independent, evaluable unit such as idea, argument, or fact. The viewpoints capture distinct components of the research idea that contribute to its overall understanding.
B PROMPT USAGE
Here, we present the prompts used in our method and the baselines.
B. PROMPTS USED IN GRAPHEVAL
We present the prompts used in LLM- viewpoint extraction and relation extraction in Table and Table 6, respectively.
B.2 PROMPTS USED IN BASELINES
We present several criteria for evaluating research ideas, along with specific standards for the four review decisions outlined in the prompts used for the baselines. The idea evaluation criteria and decision prompt templates can be found in Table and Table 9.
The prompt used in the prompted LLM is presented in Table 10, while the prompt used in the CoT prompt and CoT-SC shown Table 11.
For the ToT prompt, we decompose the problem into eight steps: novelty evaluation step, validity evaluation step, significance evaluation step, rigorousness evaluation step, clarity evaluation step, ethical evaluation step, overall score step, and final discussion step. The prompts used for the novelty evaluation step, validity evaluation step, overall score step, and final discussion step are presented in Tables 12, 13, 14, 15; the prompts for the remaining steps similar to those displayed in Tables 12 and 13.
Building the work of Back al. (2024), our Research Agent baseline divides the task into four steps: problem validation step, method validation step, experiment validation step, and final decision step, with the corresponding prompts presented in Tables 16, 17, 18, 19.
16

---

Published as a conference paper at ICLR 2025
Table 4: Viewpoint extraction prompt template. Here, for brevity and clarity, we have omitted portions of the system's s input and LLM'ss answer from the one-shot demonstration.
You are required to act an Al annotator and extract the Viewpoints embedded in the sentences of the provided academic paper abstract. Below, you will be given an abstract from academic paper. You need break it down sentence by sentence and extract the Viewpoints embedded in each sentence. The extracted Viewpoints can be an idea, argument, or fact. Each sentence may contain one or more Viewpoints be extracted.
The extracted Viewpoints should be as granular as possible to ensure they cannot be further broken down.
When extracting Viewpoints from a sentence, pay attention to the context within the abstract. Replace pronouns with the nouns they represent and complete any omitted sentence compo- nents to ensure the independence of the Viewpoints is not compromised. This means that each extracted Viewpoint should not contain pronouns whose referents cannot be found within that Viewpoint.
Below is an example interaction that can serve as a reference for the format and method of extracting Viewpoints:
System's s Input:
[The Start of Abstract]
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories.
[The End of Abstract]
Your Answer:
[Sentence 1]
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories.
[Extracted Viewpoints in Sentence 1]
[State-of-the of-the-art computer vision systems trained to predict fixed set of predetermined object categories.]
[Sentence 2]
Table Details of the Datasets. We present the data sizes and label distributions for the datasets used in our experiments. For the Al Researcher Dataset, due to the scarcity of Accept (Oral) and
Accept (Spotlight) labels, we have combined them into a single label.
Dataset
Data Size
Reject
Poster
Oral
Spotlight
ICLR Papers (Training)
300
55%
25%
10%
10%
ICLR Papers (Test)
50
64%
24%
8%
4%
AI Researcher Dataset
66
53.03%
27.27%
19.70%
C DETAILS OF THE DATASETS
In this section, we present the details of the ICLR Papers and AI Researcher Dataset used in our experiments, as shown Table 5.
Specifically, we control the label distribution of the training set in the ICLR Papers by increasing the representation of Accept (Oral) and Accept (Spotlight) papers. This adjustment enables learning- based methods to effectively capture features of less-represented samples under long-tail distribution conditions.
For the AI Researcher Dataset (Si et al., 2024), due to the scarcity of Accept (Oral) and Accept (Spotlight) labels, we have combined them into a single label, thus transforming the task into a three-class classification problem. Additionally, given the limited data volume in this dataset, we record the performance metrics of the methods across the entire dataset when testing prompt- based methods obtain more comprehensive evaluation. For testing other methods, split the dataset into training and testing sets in an 85%:1554 ratio and conduct multiple experiments to average the results, thereby reducing bias.
17

---

Published as a conference paper at ICLR 2025
Table 6: Relation extraction prompt template. Here, for brevity and clarity, have omitted portions of the system's s input and LLM'ss answer from the one-shot demonstration.
You are required to act academic paper along with a set of extracted Viewpoints. These Viewpoints can represent an idea, argument, or fact.
Al annotator. You will be provided with an abstract from
Your task is to identify pairs of related Viewpoints and provide a suitable logical connector to describe the relationship between each selected pair. Then, you need to indicate whether the logical relationship you selected belongs to the "supporting relationship" or "opposing relationship" category. The "supporting relationship" can describe logical connections such continuation, cause-effect, exemplification, etc., while the "opposing relationship" is used to describe contrast, contradiction, etc.
The format of Viewpoint Pair is as follows: ([ ([Viewpoint]], [logical connector), ("supporting"
or "opposing"] "J. [Viewpoint2]]
You need to refer to the given academic paper abstract to determine the relevance between Viewpoints and the appropriate logical relationship for related Viewpoints based on the context. You need to list all the Viewpoint Pairs you find. Below is an example interaction that can serve as a reference for the format and method of constructing Viewpoint Pairs: System's ss Input:
[The Start of Abstract]
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories.
(The End of Abstract]
[The Start of Extracted Viewpoints]
[State-of-the-att computer vision systems are trained to predict fixed set of predetermined object categories.]
[The End of Extracted Viewpoints]
Your Answer:
(The Start of Viewpoint Pairs]
[[State-of-the-ane computer vision systems are trained predict fixed of predetermined object categories.]. [however], [opposing]. [The generality and usability of state-of-the-an computer vision systems are limited by being trained to predict fixed set of predetermined object categories.]}
[The End of Viewpoint Pairs]
Table 7: Hyperparameter Settings for Experiments. This table lists the hyperparameters, their descriptions, and the values used during our experiments. Parameter Name
Description
Value
temperature t
The temperature coefficient when calling the LLM
0.1
intra graph degree
Degree of each view-node in Sub-viewpoint-grapp construction through embedding similarity
inter graph degree
Number of view-nodes each node 10 connects to in the Viewpoint-graph construction, from a different sub- graph
max.iters
Number of iterations in Label Prop- (ICLR Papers)
agation for GraphEval-LP
2 (AI Researcher)
D HYPERPARAMETER CONFIGURATION
The hyperparameter settings used in our experiments presented in Table
18

---

Published as a conference paper at ICLR 2025
Table 8: Idea evaluation criteria prompt template. We outline several criteria for assessing research ideas in the prompts used for the baselines. Criteria
Texts
Does it introduce a new problem or perspective that has not been explored before? Does introduce new techniques or represent significant advancement compared to existing methods? How does it align with or diverge from current research trends?
Novelty
Does it include solid theoretical foundations, robust algorithms, and detailed methodologics to address the research problem? Are the under- lying principles well-defined and logically consistent?
Validity
Consider its potential contribution and impact on the research community in its specific domain and beyond. How does it compare existing works in terms of impact?
Significance
Are the research design and methods clearly described and justified? Is the methodology robust and appropriate for addressing the research questions? Are the results well-analyzed and interpreted? Do the findings support the claims made in the paper?
Rigorousness
How well do the title and abstract summarize the paper? Are they clear, concise, and informative? Does the paper effectively convey its significance and main contributions? Are the title and abstract well- aligned with each other and accurately represent the core idea and content of the paper? Is the content well-structured and easy to follow?
Clarity
Does it adhere to ethical guidelines and responsible research practices? Are potential negative consequences biases addressed?
Ethical Considerations
Table 9; Idea evaluation decision prompt template. We present specific standards for the four review decisions in the prompts used for the baselines. Decision
Texts
Papers this category lack sufficient novelty, contain fundamental flaws in methodology, or fail to present significant contribution to the field. For example, a paper that proposes a minor tweak to existing methods without offering substantial improvement may fall under this category.
Reject
These papers offer incremental contributions, demonstrate solid theoret- ical or experimental work, and may be of interest niche audience. They have clear and understandable results but may not present break- throughs.
Accept (Poster)
Papers in this category present more significant contributions to the field, with clear and convincing evidence of effectiveness. The methodology is robust, and the findings impactful. These papers well-executed and can be of interest to a broader audience.
Accept (Oral)
Papers that represent groundbreaking work major advancement in the field, offering novel insights or techniques with broad applicability and significant potential impact. These papers stand out in terms of both innovation and technical quality.
Accept (Spotlight)
E GENERALIZATION EXPERIMENT
E.1 GENERALIZATION ON LONG FORM TEXT EVALUATION TASK
To validate GraphEval's capability in text evaluation forms beyond research ideas, we conducted experiments on a long form text evaluation task (Min et al., 2023). Specifically, we used human- annotated data from the FActScore dataset, where each entry contains "atomic facts" about celebrities generated by LLMs, along with assessments from human annotators on whether these "atomic facts" " supported by the materials provided to the annotators. Based on the "atomic facts" and human annotations from the training set, our method needed to predict the labels of "atomic facts" in the test set that partitioned off. We selected topics such as Ramesses IV, Lanny Flaherty, and Florencia
19

---

Published as a conference paper at ICLR 2025
Table 10: Prompted L.M prompt template. We provide several criteria for assessing research ideas in the prompt. Additionally, we present specific standards for the four review decisions and include one example for each as few-shot examples for in-context learning. Moreover, we include the label distribution of the dataset to help the LLMs understand the frequency of each review decision.
[System Prompt]
You an Al researcher who is reviewing paper's 's title and abstract that submitted prestigious ML venue. Be critical and cautious in your decision. If paper's title and abstract are bad you unsure, give it bad scores and reject it! [Instruction]
Please evaluate the paper draft based on the following six dimensions:
{i {idea evaluat ion criteria prompt temp semplate)
You will classify the paper into one of the following four categories based on the evaluation:
{i {idea evalua aalluattnn uutiion decision prompt temp 1 atte
**ooe: The approximate distribution of decisions for papers at this ML venue is as follows:
(label dist bunion of the dataset). Please take this decision distribution into account and make your judgment carefully. [Examples for Evaluation Standards] {one example per decision]
[Input]
Here is the paper draft to evaluate: Title - (title); Abstract (abstract); (abtacaaaaattt [Output]
You only need to give an overall score (0-100) and select review decision. No detailed analysis is required.
The output format should follow these rules:
Overall Score (0- (0-100)= (score)
(one decision from "Reject", "Accept (Poster)", "Accept (Oral)", "Accept (Spotlight)") An example of the output:
Overall Score (0-100)= 82
Reject
Bertotti, and divided the training, validation, and test sets in 7:1:2 ratio. We compared GraphEval and some applicable baselines on this dataset in Table 20. The experimental results the table verify that our approach performs well on the long form text evaluation task, demonstrating good adaptability to various tasks.
E.2 GENERALIZATE ABILITY ACROSS DIFFERENT TIMES
To explore the temporal generalization performance of GraphEval on the dataset, selected papers from before 2022 in the ICLR Papers dataset as the training and validation sets, and papers from 2023 as the test set. We compared the performance of GraphEval with other classic baselines in Table 21. The results in the table validate GraphEval's temporal generalization ability in the task of idea evaluation.
F ADDITIONAL ABLATION STUDY
EFFECTS OF VARIOUS LIGHTWEIGHT GRAPH NEURAL NETWORK ARCHITECTRRES
To compare the impact of different lightweight GNN architectures on the performance of GraphEval, we selected two classic lightweight GNN frameworks, SGC (Wu et al., 2019) and LightGCN (He al., 2020), to replace the current heterogeneous graph structure in GraphEval. We named these two baselines GraphEval-SGo and Craphhval-LightGee with GraphEval-GNT of the lightweight frameworks was inferior to that of GraphEval -NNN which is due to their sacrifice of individualized node information in order optimize memory usage and speed.
respectively. We compared these baselines the ICLR Papers dataset, shown in 22. We observed that the performance
20

---

Published as a conference paper at ICLR 2025
F.2 COMPARATIVE IMPACT OF ALTERNATIVE RELATION EXTRACTION METHODS
We proposed hybrid relation extraction method named Hybrid to compare with our fully similarity- based approach, GraphEval. Specifically, the hybrid method uses Prompted LLMs mentioned in Section 3 to connect nodes within viewpoint-suhgraphs. while the edges between viewpoint-subgrappe are still based on similarity. The results of the two relation extraction methods on the ICLR Papers dataset are presented in Table 23, showing that GraphEval-GNN performs better than Hybrid. This might be due to the difficulty of ensuring adequate edge density when connecting nodes within viewpoint-subgraphe using Prompted LLMs. Additionally, this connection method may increase the likelihood of hallucinations produced by LLMs and increase the token cost of LLMs, thus affecting the final impact on idea evaluation and the actual expenses.
G SCALABILITY GENERALIZATION
To validate the generalization capability of GraphEval-GNN on large-scale datasets, we conducted experiments on the ASAP- dataset (Yuan et al., 2022). The ASAe-Review dataset an open peer review dataset that includes 5,192 ICLR papers from 2017-2020 obtained through OpenReview and 3,685 NeurIPS papers from 2016-2019 accessed through NeurIPS Proceedings. A detailed introduction this dataset, along with its composition, can be found in Section 3.1 and Table 2 of (Yuan et al., 2022). Similar to the settings described in Section of our paper, we used the abstracts of all papers the dataset as inputs and the review decisions of the papers the predicted labels, which included Accept (Oral), Accept (Spotlight), Accept (Poster), and Reject. We divided the dataset into training, validation, and test sets in the proportions of 70%, 10%, and 20%, respectively. It is important to note that for NeurIPS papers, since only accepted papers are included and no specific labels such as Oral, Spotlight, or Poster and ratings are provided, we have to assign all paper labels as Accept (Poster). This approach ensures the accuracy of the sample because over 85% of the papers accepted at the NeurlPS conference are designated as posters. As shown in Table 24, we compared the performance of GraphEval-GNN with that of Fine-tuned BERT and Prompted LLM on this dataset. We observed that GraphEval-GNN still maintains the best performance on this large-scale dataset, with accuracy 9.8% better than the strongest baseline, Fine-tuned BERT. Furthermore, although the rare labels of Accept (Oral) and Accept (Spotlight) (less than 4%) make it difficult for all methods perform well in terms of macro Fl score, GraphEval-GNN still achieved an 8% improvement in macro Fl score compared to Fine-tuned BERT. These observations demonstrate the robust generalization capability of GraphEEvl-GNN on large-scale datasets.
H ACCURACY EVALUATION OF VIEWPOINTS
In order to evaluate the accuracy of viewpoints generated from ideas, we explore from two perspec- tives. First, we use a noseet approach (Luo et al., 2023; Gao et al., 2023), allowing a large LLM assess whether each viewpoint is consistent with the original idea. Specifically, we employ the 1LaMa-3.1 (405b) LLM", which has shown excellent performance in evaluation tasks, as the evaluator. Using the prompt from Table 25, we evaluate the consistency between the viewpoint and the idea, with an output of 1 indicating consistency and 0 indicating inconsistency. We calculate the proportion of samples judged consistent and average this across all samples to determine the consistency rate. We finally achieve consistency rates of 99.47% and 99.82% for the ICLR Papers and AI Researcher datasets, respectively. These rates, very close to 100% demonstrate the high degree of consistency between the generated viewpoints and the original ideas achieved by our method.
Additionally, we measure the accuracy of viewpoints from an entity-level perspective. Specifically, we first aggregate the constructed viewpoints and then assess their entity-level accuracy with respect to the idea using entity-level factual consistency metrics (Nan et al., 2021). We report the results on the datasets ICLR Papers and AI Researcher in Table 26. From the table, entity-level Precision, Recall, and F1 Score between the viewpoints and the idea exceed 0.9 on both datasets, which also validates the accuracy and rationality of our viewpoints.
observe that the
21

---

Published as a conference paper at ICLR 2025
Table 11:
CoT prompt template. We modify the prompt used for prompted LLM to adopt CoT format, guiding it to complete the idea evaluation step by step.
[Instruction]
Please evaluate the paper draft step by step based on the following dimensions. For each step. carefully think through and evaluate the corresponding dimension, and then provide ratings for each dimension (1-10). You must give an overall score (0- 100) along with the 6 dimension scores. No detailed analysis is needed, but ensure that your evaluation for each step is based on logical reasoning. [Inpur]
Here is the paper draft to evaluate: Title - -tittettte Abstract (abstract); (abtataaaaaaaatssa [Step 1: Evaluate Novelty] First, evaluate the novelty of the paper. ftext for novelty in the idea evaluation criteria prompt template.) Novelty Rating (I-10);
[Step 2: Evaluate Validity]
Next, evaluate the validity of the paper. ftext for validity in the idea evaluation criteria prompt template.} Validity Rating (1-10);
[Step 3: Evaluate Significance]
Then, evaluate the significance of the paper. ftext for significance in the idea evaluation criteria prompt template.] Significance Rating (1-10):
[Step 4: Evaluate Rigorousness] Now, evaluate the rigorousness of the paper. ftext for rigorousness in the idea evaluation Rigorousness Rating (1-10):
criteria prompt template.]
[Step 5: Evaluate Clarity]
Next, evaluate the clarity of the paper. ftext for clarity in the idea evaluation criteria prompt template.} Clarity Rating (1-10):
[Step 6: Evaluate Ethical Considerations]
Lastly, evaluate the ethical considerations of the paper. ftext for ethnic in the idea evaluation criteria prompt template.} Ethical Considerations Rating (1-10):
[Step 7: Final Overall Score]
After completing all the dimension evaluations, summarize your assessment and give an overall score that reflects the paper's general quality and performance across all dimensions.
Overall Score (0-100):
[Step 8: Final Decision]
Based on the overall score and individual ratings, choose the most appropriate review decision.
Carefully consider how the paper performs in each dimension, and select from the following
categories:
(idea evaluation decision prompt template}
Decision:
**Note:** The approximate distribution of decisions for papers at this MI venue is as follows:
[label distribution of the dataset). Please take this decision distribution into account and make your judgment carefully. [Examples for Evaluation Standards] (one example per decision: [Output]
The output format should follow these rules:
Novelty Rating (1-10):
Validity Rating (1-10):
Significance Rating (1-10):
Rigorousness Rating (1-10):
Clarity Rating (1-10):
Ethical Considerations Rating (1-10):
Overall Score (0-100):
Decision: {one decision from "Reject", "Accept (Poster)", "Accept (Oral)", "Accept (Spot- light)"}
22

---

Published as a conference paper at ICLR 2025
Table 12: ToT prompt template: Novelty Evaluation Step
[Instruction]
Please evaluate the novelty of the paper draft provided. (text for novelty in the idea evaluation criteria promet template.
You only need give novelty rating (0-10). No detailed analysis is required. [Input]
Title: (title)
Abstract: {abstract}
[Output]
Please generate rating for the novelty of this paper (1-10)
An example the output:
Novelty Rating (1-10): 5
Table 13: ToT prompt template: Validity Evaluation Step
[Instruction]
Please evaluate the validity of the paper draft based on the provided title, abstract, and novelty rating.
ftext for validity in the idea evaluation criteria prompt template.]
You only need to give novelty rating (0-10). No detailed analysis is required. [Input]
Title: {title)
Abstract: (abstract}
Novelty Rating (1-10): (novelty rating)
[Output]
Please generate rating for the validity of this paper (1-10)
An example of the output:
Validity Rating (1-10): 5
Table 14: ToT prompt template: Overall Score Step
[Instruction]
Please evaluate the overall quality of the paper draft based on the provided title, abstract, and ratings (novelty, validity, significance, rigorousness, clarity, and ethical considerations). The overall score should reflect the general quality of the paper and how well it performs across all the evaluation dimensions.
You only need to give an overall score (0-100). No detailed analysis is required. [Input]
Title: {title)
Abstract: (abstract]
Novelty Rating (1-10): (novelty result}
Validity Rating (1-10): [validity result)
Significance Rating (1-10): (significance result)
Rigorousness Rating (1-10): (rigorousness result)
Clarity Rating (1-10): (clarity result)
Ethical Considerations Rating (1-10): (ethical considerations result] [Output]
Please generate an overall score for this paper (0-100).
An example of the output:
Overall Score (0-100): 80
23

---

Published as a conference paper at ICLR 2025
Table 15: ToT prompt template: Finale Decision Step
[Instruction]
Please determine the final decision for the provided paper draft based the provided title, abstract, overall score, and individual ratings (novelty, validity, significance, rigorousness, clarity, and ethical considerations). The decision should reflect the overall quality of the paper and how well it performs across all evaluation dimensions. Select the most appropriate option from the following four categories: (idea evaluation ** The approximate distribution of decisions for papers at this MI venue is as follows: (label di sistriii bution of the dataset). Please take this decision distribution into account and make your judgment carefully. [Examples for Evaluation Standards] {one example per decision]
decision prompt temp 1 aate
[Input]
Title: {title)
Abstract: (abstract)
Novelty Rating (1-10): nnovelty result)
Validity Rating (1-10): {validity result}
Significance Rating (I-10): (significance result]
Rigorousness Rating (1-10): (rigorousness result]
Clarity Rating (1-10): fclarity result]
Ethical Considerations Rating (I-10); (ethical considerations result]
Overall Score (0-100): (overall score) [Output]
Decision: {one decision from "Reject", "Accept (Poster)", "Accept (Oral)", "Accept (Spot- light)"}
An example of the output:
Decision: Accept (Poster)
24

---

Published as a conference paper at ICLR 2025
Table 16: Research Agent prompt template: Problem Validation Step. Criteria is presented in Table 10 of the paper by Back al. (2024).
[System Message]
You are an AI assistant whose primary goal is to summarize the research problem in an academic paper based on its title and abstract, and to assess the quality and validity of the research problem across various dimensions. Your evaluations and feedback will help researchers refine their research problems, thereby enhancing the impact and scope of their work.
[User Message] extract its research problem and the rationale for the research problem. You are required to evaluate the research problem based on the following dimensions: Clarity, Relevance, Originality, Feasibility, and Significance, with focus whether it is clearly, accurately, and understandably defined.
You will be provided with the title and abstract of an academic paper, and you need to
The academic paper title and abstract to be evaluated are as follows:
Paper Title: {title}
Paper Abstract: fabstract} Now, please proceed with systematic evaluation focusing on Clarity, Relevance, Originality, Feasibility, and Significance:
First, carefully read the provided title and abstract, and extract the research problem and its rationale.
Next, generate review and feedback that is constructive, helpful, and concise, focusing the research problem's 's Clarity, Relevance, Originality, Feasibility, and Significance. Finally, rate the problem that your ratings are discerning and critical to avoid uniformly high scores (4-5) unless fully justified. The definitions for each evaluation criterion (criteria} Output:
5-point Likert scale, with being the lowest Ensure
follows:
First, summarize the research problem and its rationale from the provided paper. After evaluating the content, provide your review, feedback, and ratings the following format: Research Problem: {research problem}
Rationale:
(research problem rationale)
Review: {review)
Feedback: [feedback]
Rating (1-5):
Gelevance-(rating! Originality- (ratiallity((ating)) Feasibility- (ratiing)
Digificance-(ratiin (rating)
25

---

Published as a conference paper at ICLR 2025
Table 17: Research Agent prompt template: Method Validation Step. Criteria is presented in Table 10 of the paper Back al. (2024).
[System Message] You are an Al assistant whose primary goal is summarize the scientific method used in research paper based on its title and abstract, and to evaluate the quality and soundness of the method across various dimensions. Your feedback will help researchers refine their methods, thereby enhancing the impact and reach of their work.
[User Message]
You will be provided with the title and abstract of an academic paper. From this, you are required to summarize its Scientific Method and Scientific Method Rationale. You need to evaluate the method for its Clarity, Validity, Rigorousness, Innovativeness, and Generalizability, focusing on whether the method is described clearly, precisely, and understandably, ensuring that it can be replicated and easily comprehended.
As part of your evaluation, you may refer to the research problem of the paper, which will help you better understand the context of the method and conduct more comprehensive assessment.
The academic paper title and abstract be evaluated and the research problem are as follows: Paper Title: {title)
Paper Abstract: fabstract]
Research problem: (research problem}
Rationale: (research problem rationale}
Now, please proceed with the systematic evaluation of the method based on Clarity, Validity, Rigorousness, Innovativeness, and Generalizability:
First, carefully read the provided paper title and abstract, keeping in mind the context provided by the research problem, and summarize the scientific method and its rationale.
Next, generate review and feedback that should be constructive, helpful, and concise, focusing on the method's Clarity, Validity, Rigorousness, Innovativeness, and Generalizability. Finally, provide ratings on a 5-point Likert scale, with 1 being the lowest, Ensure that your ratings are discerning and critical, avoiding tendency toward uniformly high scores (4-5) unless fully justified. The definitions of each evaluation criterion are as follows: (criteria) Output:
First, summarize the scientific method and its rationale. After evaluating the content, please provide your review, feedback, and ratings in the following format: Scientific Method: {scientific method)
Rationale: (scientific method rationale]
Review: (review)
Feedback: [feedback]
Rating (1-5): (rating-) Validity-(rating) Rigorousness-rratnng) Innovativeness-
(rating) Generalizability
26

---

Published as a conference paper at ICLR 2025
Table 18: Research Agent prompt template: Experiment Validation Step. Criteria a is presented in Table the paper by Back et al. (2024).
[System Message]
You are an Al assistant whose primary goal is to summarize the experimental design an academic paper based its title and abstract and meticulously evaluate the experimental design across various dimensions. Your evaluations and feedback will help researchers refine their experimental approaches, thereby amplifying the quality and impact of their scientific contributions.
[User Message]
You will be provided with the title and abstract of an academic paper. From this, you are required summarize its experiment design and experiment design rationale. You are going to evaluate the experiment design for its Clarity, Validity, Robustness, Feasibility, and Reproducibility in validating a scientific method to address a research problem, focusing on how well it is described in clear, precise, and understandable manner, enabling others to grasp the setup, procedure, and expected outcomes.
As part of your evaluation, you refer to the research problem and scientific method, which will help in understanding the context of the designed experiment for more comprehensive assessment.
The academic paper title and abstract to be evaluated, along with the research problem and scientific method, are follows:
Paper Title: {tittee
Paper Abstract: fabstract]
Research problem: {research problem]
Rationale: (research problem rationale]
Scientific Method: /scientific method} Rationale: (sci escientio ent fic method rationale)
Now. proceed with your systematic evaluation of Clarity, Validity, Robustness, Feasibility, and Reproducibility: Start by thoroughly reading the provided paper title and abstract, keeping in mind the context provided by the research problem and scientific method mentioned above. Summarize the experiment design and its rationale.
Next, generate a review and feedback that should be constructive, helpful, and concise, focus- ing the Clarity, Validity, Robustness, Feasibility, and Reproducibility of the experiment. Finally. provide ratings on a 5-point Likert scale, with I being the lowest, Ensure that your evaluation is discerning and critical, avoiding a tendency toward uniformly high scores (4-5) unless fully justified:
(criteria)
Output:
First, summarize the experiment design and its rationale. After evaluating the content, please provide your review, feedback, and ratings in the following format: Experiment Design: (experiment design]
Rationale: (experiment design rationale) Review: (review)
Feedback: (feedback)
Rating (1-5): Cratiny- Validity- Robustness- (rating Feasibility-Iratin
ppproducibilityyrraiin
27

---

Published as a conference paper at ICLR 2025
Table 19: Research Agent prompt template: Finale Decision Step. Building on the work of Back et al. (2024), we further introducc a final decision step that synthesizes the evaluation results from the aforementioned steps to provide comprehensive review decision.
You are an Al assistant. You will be provided with the title and abstract of an academic paper, along with a summary of its research problem, scientific method, and experiment design. Additionally, you will receive reviews, feedback, and ratings (on a scale of 1-5) for the research problem, scientific method, and experiment design across various dimensions.
Based on the provided paper title and abstract, as well as the evaluations of its research problem, scientific method, and experiment design, your task is assign an overall score (0-100) to the paper.
You will also classify the paper into one of the following four categories based on the evalua- tion:
{idea evaluation decision prompt template}
**Note:** The approximate distribution of decisions for papers at this ML venue is as follows: (label distribution of the dataset). Please take this decision distribution into account and make your judgment carefully. [Examples for Evaluation Standards] (one example per decis sini [[nput]
Paper Title: {title}
Paper Abstract: fabstract]
Research Problem: fresearch problem]
Research Problem Rationale: (research problem rationale]
Research Problem Review: (research problem review]
Research Problem Feedback: (research problem feedback]
Research Problem Rating: (research problem rating)
Scientific Method: (scientific method}
Scientific Method Rationale: (scientific method rationale)
Scientific Method Review: fscientific method review)
Scientific Method Feedback: (scientific method feedback)
Scientific Method Rating: (scientific method rating) Experiment Design: (experiment design]
Experiment Design Rationale: (experiment design rationale)
Experiment Design Review: (experiment design review]
Experiment Design Feedhack: (experiment. design feedback) Experiment Design Rating: [Output]
(experiment design rating]
You only need to give overall score (0-100) and select review decision. No detailed analysis is required. The output format should follow these rules: Overall Score (0-100)= (score}
{one decision from "Reject", "Accept (Poster)", "Accept (Oral)", "Accept (Spotlight)") An example of the output:
Overall Score (0-100)= 82
Reject
Table 20: Comparative performance results on the Fact Verification dataset. Bold text denotes the best results. For all CeetrissAAcurri Macro Precision, Macro Recall, and Macro F1 Score higher values indicate more precise predictions.
Model
Accuracy
Precision
Recall
F1 Score
Prompted LLM (7B)
49.79%
57.19%
52.27%
47.59%
Prompted LLM (72B)
59.52%
63.13%
60.35%
56.33%
Finetuned-Bert
70.27%
69.74%
68.54%
68.64%
GraphEval-LPP GraphEyal-GNN
82.83%
83.41 %
83.04%
82.40%
85.00 %
90.00 %
83.00%
84.00%
28

---

Published as a conference paper at ICLR 2025
Table 21: Comparative performance results under the setting of idea evaluation of different years, Bold text denotes the best results. For all metrics--Acuracyy, Macro Precision, Macro Recall, and Macro F1 Score higher values indicate more precise predictions.
Model
Accuracy
Precision
Recall
Fl Score
Prompted LLM (7B)
16.67%
20.63%
26.12%
18.25%
Prompted LLM (72B)
14.29%
11.25%
32.47%
11.76%
Benttued-Beert
48.41%
42.46%
36.14%
31.57%
GraphEval-LP
63.20%
52.38%
48.60%
44.72%
GraphEval-GNN
76.19%
48.25%
57.38%
51.32%
Table 22: Performance comparison of different lightweight graph models.
Model
Accuracy
Precision
Recall
F1 Score
GraphEval-SGCC
61.0%
27.7%
23.3%
27.3%
GraphEval-LighhGCN
54.0%
23.43%
25.05%
26.70%
GraphEval-GNN 76.0%
38.40 %
37.30%
44.80%
Table 23: Performance comparison of GraphEyal-GNN via two different alternative relation
extraction methods. Model
Accuracy
Precision
Recall
F1 Score
Hybrid
62.0%
25.08%
27.60%
25. 35.46%
GraphEval-GNN
76.0% %
38.40%
37.30%
44.80 %
Table 24: Comparative performance results for different models on the ASAP-Review dataset. Bold text denotes the best results. For all metrics Accuracy Macro Precision, Macro Recall, and Macro Fl Score-higher values indicate more precise predictions.
Model
Accuracy
Precision
Recall
Fl Score
Prompted LLM (7B)
22.00%
11.04%
28.57%
12.83%
Prompted LLM (72B)
4.00%
4.00%
17.86%
3.04%
Finetuned-Bert
61.17%
29.81%
30.37%
29.86%
GraphEval-GNN 67.02%
33.11%
32.86%
32.20%
Table 25: Prompt template of viewpoint accuracy evaluation.
[Instruction]
Decide if the following Viewpoint, derived from the idea, is consistent with the Idea. Note that consistency means all information the viewpoint is fully supported by the idea.
[Input]
Idea: {idea}
Viewpoint: (viewpoint)
[Output]
Explain your reasoning step by step, identifying if each part the viewpoint aligns with the idea, then answer: Is the viewpoint consistent with the idea? Answer with only for yes or 0 for no.
Table 26: Performance of entity-level factual consistency metrics for ICLR Papers and AI Researcher datasets.
Dataset
Precision
Recall
F1 Score
ICLR Papers
0.9339
0.9288
0,9314
AI Researcher
0.9472
0.9004
0.9232
29


Reviews:
[{"review_id": "htzCcSPPBI", "reviewer": "Reviewer_zpR7", "summary": "This paper aims to develop a graph-based LLM framework for evaluating research ideas. Specifically, the authors first break down a whole idea into multiple viewpoints (nodes) and then connect them (making edges) based on their embedding-level similarity as well as the relation extraction approach, from which the authors construct the viewpoint graph. Then, by utilizing this viewpoint graph through simple label propagation or GNNs, the proposed approach predicts the quality of the research ideas. In addition to this, the authors propose the simple trick to evaluate the novelty of the ideas, by generating the labels for them and then training the GNN with them. The authors show that the proposed approach can not only predict the quality of the research ideas better than existing methods but also it can predict the novelty of the ideas.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The proposed approach of evaluating the research ideas over the graph-structure (by breaking down them into multiple sub-ideas connected with other sub-ideas) is novel and sound.\n* The proposed approach is superior to other research idea evaluation methods.\n* This paper is very well-written and easy to follow.", "weaknesses": "I don't see any major weaknesses. But, there are some points that can make this paper more solid:\n* The authors mainly evaluate the proposed method on only the research idea evaluation task, and, while this task is very important and less explored, this point may limit its generalizability to other tasks and domains (i.e., I believe it can be well applicable to other tasks related to evaluating long text and it is worth trying it).\n* The authors can discuss some works on evaluating the long-form text [1, 2], as some of them tend to divide the long text into multiple subsets (similar to the proposed approach) and evaluate each of them. \n* The authors can incorporate more analyses, to showcase the efficacy of the proposed approach more. One experiment that is worthwhile to try is, to showcase the generalizability of the proposed approach (or the constructed viewpoint graph) to new papers in 2023 by training it with papers before 2022.\n\n---\n\n[1] FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP 2023.\n\n[2] Let's Verify Step by Step, arXiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jGYVQ7ptZF", "reviewer": "Reviewer_oRRW", "summary": "This paper presents a framework addressing the limitations of LLMs in evaluating research ideas, focusing on stability, semantic comprehension, and objectivity. The authors introduce two core methods—GraphEval-LP (label propagation) and GraphEval-GNN (a GNN-based approach). Both are lightweight, with GraphEval-GNN incorporating a novelty detection component to assess plagiarism risks.", "questions": "1. In the domain of automated paper review generation, several existing works relevant to this field are missing from the discussion [1-3].\n\n2. The paper contains informal language in some variable names, such as \"max_iters\" and \"cos sim\".\n\n3. The framework represents edges between viewpoints within a single document and across multiple documents as undirected. Could the authors clarify how (or if) temporal edges are integrated into the graph?\n\n4. In line 288, what is $n$ when predicting the label $\\hat{y}$?\n\n5. Lines 270-274 describe the initialization process is ambiguous. The initialization node features are formed with node labels? Could the authors clarify this initialization procedure?\n\n6. The task tackled in this study does not appear complex but the dataset used for evaluation is quite small. Given the availability of larger, relevant datasets (e.g., provided by ReviewAdvisor [1], which covers ICLR and NeurIPS), could the authors explain why a larger dataset was not employed?\n\n7. What is the ratio of positive to negative samples in the dataset? The authors note that LLMs are inclined to accept most papers, yet the baseline methods report very low accuracy (below 20% on the ICLR dataset), suggesting a high prevalence of negative samples. Could the authors provide details on dataset composition and how they ensured a fair experimental comparison?\n\n8. The paper lacks implementation details, and no code or datasets are available.\n\n9. Could the authors specify the exact prompts used in the CoT and ToT baselines?\n\n10. GraphEval framework relies on a 7B parameter LLM while Fine-tuning BERT (with much fewer parameters) achieves comparable performance. How about the performance of fine-tuning 7B models?\n---\n\n[1] Yuan, Weizhe, Pengfei Liu, and Graham Neubig. \"Can we automate scientific reviewing?.\" Journal of Artificial Intelligence Research 75 (2022): 171-212.\n\n[2] Du, Jiangshu, et al. \"Llms assist nlp researchers: Critique paper (meta-) reviewing.\" arXiv preprint arXiv:2406.16253 (2024).\n\n[3] Lin, Jialiang, et al. \"Automated scholarly paper review: concepts, technologies, and challenges.\" Information fusion 98 (2023): 101830.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The viewpoint-graph breaks down complex ideas into interconnected, evaluable components.", "weaknesses": "1. Some components of model lack clear explanations.\n2. The experimental evaluation is limited; for instance, there is no ablation study, and the dataset size is small.\n3. The construction of the graph relies solely on textual information. Evaluating a paper’s acceptance potential based exclusively on textual relevance is not entirely reasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xMbCnn7ZL", "reviewer": "Reviewer_d4yk", "summary": "This paper proposes GraphEval, a lightweight graph-based LLM framework for idea evaluation, which utilizes LLM's summarization and abstraction capabilities for improving idea evaluation. Through the viewpoint graph extraction, label propagation or GNN algorithms, and plagiarism detection mechanism, GraphEval breaks down complex ideas into multiple simple viewpoints, connects the viewpoints into an entire viewpoint-graph, and evaluates ideas via label propagation or a weighted GNN. Extensive experiments on two datasets demonstrate that GraphEval can significantly enhance the accuracy of idea evaluation while effectively detecting plagiarized ideas to provide a fair evaluation.", "questions": "1. Could the authors discuss the similarities and differences between GraphEval and GraphRAG [1]? GraphRAG breaks a document into chunks, extracts a knowledge graph from raw text, builds a community hierarchy, generates summaries for these communities, and leverages these structures in RAG tasks. This seems similar to the viewpoint-graph extraction proposed here.\n2. In the left of Figure 1, are the colors of the positive and negative prompts correctly marked? Specifically, is \"If a paper is good or you are unsure, give it good scores and accept it.\" intended as a negative prompt?\n\n**Reference**  \n[1] From Local to Global: A Graph RAG Approach to Query-Focused Summarization, 2024.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper introduces an innovative method for idea evaluation with a lightweight graph-enhanced LLM framework, addressing the inherent complexities and subjectivity of this task.\n2. This paper breaks down complex ideas into simpler viewpoints to construct viewpoint-graphs and evaluates ideas with label propagation or GNN algorithms under the low computation and API costs. Meanwhile, the proposed plagiarism detection mechanism integrates the temporal information for more accurate and fair idea evaluation.\n3. The paper is well-written and organized, with a clear problem setup, methodology, and experimental evaluation.", "weaknesses": "1. The paper lacks a thorough theoretical analysis or justification for the proposed mechanisms, such as the viewpoint-graphs extraction and the plagiarism detection mechanism. Whether viewpoints extracted by LLMs are reasonable? If so, how to verify the accuracy of these viewpoints? Is there an overlap between viewpoints from different complex ideas? How are the inherent LLM issues—like hallucinations and token limitations—addressed or mitigated in this framework?\n2. The paper does not provide comprehensive ablation and hyperparameter studies, such as the impact of different LLMs/GNNs and varying values of $k$ and $m$ (c.f., line 247 and line 255). Moreover, it is beneficial to provide an analysis of the computational complexity, efficiency, and resource requirements of the proposed framework, including training/inference time, which would be helpful in assessing its scalability and practical applicability. \n3. The paper seems to underutilize the reasoning and generative capabilities of LLMs, which could improve GraphEval’s interpretability and effectiveness in idea evaluation tasks.\n4. Several typos are present in the paper. For example, In line 302, \"... As illustrated in Sec 3, ...\" lacks a period after \"Sec\". In lines 414-422, the use of **macro** F1 score to evaluate the accuracy is inconsistent with the content \"..., and **micro** F1 score ....\" (in lines 399-401).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z1h9IGqDva", "reviewer": "Reviewer_5M8a", "summary": "The paper presents a novel framework that leverages graph structures to enhance large language models in the evaluation of complex research ideas. The authors propose two core methods: GraphEval-LP, a label propagation-based approach, and GraphEval-GNN, a GNN model for predicting evaluation scores. The framework addresses known limitations in LLM-based evaluation, such as sensitivity to prompts and biases toward positive feedback, by breaking down ideas into viewpoint nodes and constructing graphs to propagate scores across these nodes. Experiments on two datasets demonstrate that GraphEval improves F1 scores compared to baselines, while also detecting plagiarized ideas.", "questions": "Is it possible to try alternative methods for relation extraction, such as unsupervised learning or hybrid approaches, which might address the sparsity issue in graph construction, especially when dealing with less structured idea content?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of graph-based methods to enhance LLMs in research idea evaluation is innovative.\n2. The methodology is explained in detail, with clear steps for viewpoint extraction, relation identification, and the integration of label propagation and GNN techniques.\n3. The results demonstrate significant improvements in F1 scores over multiple baselines.", "weaknesses": "1. While the approach works well on specific datasets (ICLR Papers and AI Researcher), the generalizability to other domains or different types of research content is unclear. Given the topic of the paper is research idea validation, the experiment turned out to detect whether this paper will be accepted. The claim and experiment are a bit stretched.\n2. The paper mentions that the LLM-based relation extraction yields sparse edges between viewpoint nodes (with a 10.73% edge density on average). This sparsity may limit the effectiveness of GraphEval in certain scenarios, particularly where relationships between ideas are less explicit. The authors address this by using BERT embeddings, but further experiments with alternative relation extraction methods could improve robustness.\n3. While GraphEval-GNN offers significant improvements in performance, its scalability is questionable for larger datasets. Training GNNs on extensive viewpoint-subgraphs may become computationally prohibitive, especially if the graph structure grows significantly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)