ResearchTown: Simulator of Human Research Community
Authors: Haofei Yu
Zhaochen Hong
Zirui Cheng
Kunlun Zhu
Keyang Xuan
Jinwei Yao
Tao Feng
Jiaxuan You
Venue: ICML 2025 poster (2025)
Keywords: Multi-agent; Graph Neural Network; Large Language Model; Automatic Research
Primary area: deep_learning->graph_neural_networks
TL;DR: A graph-based multi-agent simulator for human research community

Abstract: Large Language Models (LLMs) have demonstrated remarkable potential in scientific domains, yet a fundamental question remains unanswered: Can we simulate human research communities with LLMs? Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights. In this work, we propose ResearchTown, a multi-agent framework for research community simulation. Within this framework, the human research community is simplified as an agent-data graph, where researchers and papers are represented as agent-type and data-type nodes, respectively, and connected based on their collaboration relationships. We also introduce TextGNN, a text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as special forms of a unified message-passing process on the agent-data graph. To evaluate the quality of the research community simulation, we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity. Our experiments reveal three key findings: (1) ResearchTown can provide a realistic simulation of collaborative research activities, including paper writing and review writing; (2) ResearchTown can maintain robust simulation with multiple researchers and diverse papers; (3) ResearchTown can generate interdisciplinary research ideas that potentially inspire pioneering research directions.

RESEARCHTOWN: Simulator of Human Research Community
Haofei Yu
Zhaochen Hong" 1* Zirui Cheng" 1 Kunlun Zhu !*
Keyang Xuan Jinwei Yao
Tao Feng' Jiaxuan You
Q Code:
Data:
co http://gg..............aa////aae
Abstract
1. Introduction
Large Language Models (LLMs) have demon- strated remarkable potential in scientific domains, yet fundamental question remains unanswered: Can we simulate human research communities with LLMs? Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights. In this work, we pro- pose REsEARCHToWN, multi-agent framework for research community simulation. Within this framework, the human research community is sim- plified as an agent-data graph, where researchers and papers are represented as agent-type and data- type nodes, respectively, and connected based on their collaboration relationships. We also intro- duce TexIGNN, text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as spe- cial forms of a unified message-passin paasgggg-ainnggggge process on the agent-data graph. To evaluate the quality of the research community simulation, we present RESEARCHBENCH, benchmark that uses a node- masking prediction task for scalable and objective assessment based on similarity. Our experiments reveal three key findings: (1) RESEARCH TOWN can provide a realistic simulation of collaborative research activities, including paper writing and re- view writing; (2) RESEARCHTOWN can maintain robust simulation with multiple researchers and diverse papers; (3) RESEARCHITOWN can gener- ate interdisciplinary research ideas that potentially inspire pioneering research directions.
LLMS have proved to be powerful copilots in scientific re- search (AI4Science & Quantum, 2023), demonstrating their great potential for accelerating scientific discovery. Despite the promising finding, more ambitious question remains: Can we simulate the human research community with LLMs? Answering such a question has multiple benefits: (1) simu- lating the human research community helps understand the underlying process behind the discovery existing research ideas; (2) can further help democratize and accelerate the discovery process of new research ideas.
However, simulating the human research community is chal- lenging, as it involves leveraging multiple LLM agents to interact with complex research data. While existing multi- agent LLM frameworks have been successfully applied to areas like social simulation (Zhou et al., 2023; Gao et al., 2023) and game simulation (Hua et al., 2023; Xu al., 2023), they are not well-suited for simulating research com- munities due to the complexity of collaborative research activities like paper writing and review writing. Although re- cent efforts have explored research automation using LLMs, these frameworks are typically limited to specific research tasks, such as idea generation (Girotra et al., 2023; Baek et al., 2024) or code experimentation (Huang et al., 2024), or focus on simulating single-agent workflows (Lu al., 2024). These frameworks cannot simulate collaborative research activities where researchers with diverse backgrounds work together to brainstorm ideas, review papers, etc processssss that are fundamental to modern human research.
Research community as graph. Our key observation is that the deeply interconnected research community can be naturally represented graphs. Indeed, similar graph struc- tures like citation networks (Newman, 2001) and academic social networks (Tang et al., 2008) have been extensively studied within data mining research, with proven values in applications such as citation prediction (Holm et al., 2020), recommendation (West et al., 2016), and community detec- tion (Yang & Leskovee, 2012). However, introducing LLMs to a graph-structured research community extend these previous works from prediction and analysis with existing
'Leading authors. Details of individual contributions are avail- able at Appendix SA Universsity of Illinois Trbana-Champaln Correspondence to: Haofei Yu <haofeiy2 iddi Jiaxuan You <jiaxuan @ illinois.edu>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RESEARCHTOWN: Simulator of Human Research Community
Human Research Community
Community Graph AAgent-Data Graph)
ICML
researcher
contibuut
codsbosn
past
simplify & abstract
che
author
paper
paper
rosear char
Butfoy
Buttley
Freeon
review
author
cite
researcher
paper
researcher
paper
Figure I: Abstracting and simplifying human research community as an agent-data graph, i.e., community graph. An agent-data graph has researchers as agent nodes and blogs, codebases, and papers as data nodes. Without losing generality, we abstract into a simplified version with only researcher and paper nodes and focus on critical research tasks, including paper reading, paper writing, and review writing. Each data node has hidden state hy, and each agent node is paired with an agent function f-) and a hidden state he.
data to dynamic simulation and real-time forecasting.
inevitably suffer from subjectiveness and high costs. In our work, since REsEARCHTOWN functions simulator, primary focus is on measuring how closely its outputs align with those of the real-world research community. Commu- nity graphs naturally provide a similarity- -aaee evaluation method by masking given paper node in the community graph and evaluating whether simulator can reconstruct the masked nodes. This definition focuses on simulation similarity, making it scalable and objective. Based on such node masking prediction task, we build benchmark called RESEARCHBENCH with 1,000 paper writing tasks and 200 review writing tasks requiring multi-agent collaboration.
Novel framework for research simulation. In this work, we propose RESEARCHTOWN, simulator of the human research community. To bridge the gap between existing multi-agent simulation frameworks and the complexity of research activities, we propose a graph-based framework, in- mechanism in Graph Neural
spired by the message-passien Networks (GNNs), for multi-agent simulation. Concretely, as shown in Figure I, we propose a new concept of agent- data graph with 2 generic types of nodes: (1) agent nodes, suitable for entities like agents: (2) data nodes, suitable for entities such as papers, reviews, and blogs. Agent-data graphs are unique from standard heterogeneous graphs; here, the key conceptual difference between agent and data nodes that an agent node can be considered function over data nodes. To inference agent-data graphs, we propose TexINN framework where message-passing processes are defined based on text-form information processing with LLMs, thanks to their strong in-context learning (Wei et al., 2023) and reasoning (Lee et al., 2024) ability. We apply the proposed agent-data graph and TexIGNN to the research simulation. Here, a research community can be regarded as special form of agent-data graph, called community graph, with research agents and research papers as two types of nodes, and we consider three types of edges (review, au- thor, and cite) in the graph. Different community activities, such as paper writing and review writing, can be modeled as special message-passingg processes on the community graph.
Main discoveries. Based on the evaluation results from RESEARCHBENCH, we highlight three key findings: (I) RESEARCHITOWN effectively simulates collaborative re- search activities, achieving an average similarity score of 0.68 for paper writing and 0.49 for review writing, as mea- sured by the state-of-the-ar text embedding model; (2) RE- SEARCHITOWN demonstrates robustness and effectiveness in research simulation, showing improvement when more agents are added and maintaining performance when in- cluding unrelated papers; (3) RESEARCIITOWN inspires interdisciplinaty research, generating innovative ideas that combine insights from NLP, criminology, and astronomy and does not exist in the real-world research.
Stressing ethical concerns. As our work targets simulating the human research community, multiple ethical concerns, including facilitating research plagiarism and producing low-quality misleading claims, appear. These ethical concerns are addressed in detail in Appendix $B.
Novel evaluation for research simulation. With RE- SEARCHTOWN for research simulation, a further research question is to evaluate the quality of that. Prior works pri- marily use human evaluation with breakdown metrics such as novelty, excitement, feasibility, and expected effective- ness (Si et al., 2024; Hu et al., 2024b). These approaches
2. Additional Related Work
Graphs with text attributes. In real-world graph tasks, nodes often have textual attributes to carry richer informa-
2

---

RESEARCHTOWN: Simulator of Human Research Community
tion, forming text-attributed graphs (TAGs) (Yang et al., 2021; He al., 2023). Previous work TAGs mainly treats LLMs as tools for understanding text attributes and aims at achieving co- co-training LLMs and GNNs (Zhao et al., 2023; Chen et al., 2024). In contrast, our approach in- corporates agent nodes into the graph, enabling text-based message passing between agent nodes and data nodes. Fur- thermore, while previous TAG research mainly focuses on node prediction and link prediction tasks (Yan al., 2023), RESEARCH/TON extends it to both the reconstruction of existing nodes and the prediction of new, non-existent nodes.
to more node types (e.g., codebase, blogs) and edge types (e.g., attend, post, commit). Typically, one blog post be directly connected to multiple researchers, papers, and other blog posts if they are related to each other.
4. Building TextGNN on Agent-Data Graphs
TextGNN motivations. The agent-data graph 9 provides a platform for expressing complex multi-agent scenario, e.g., human research community. To further simulate based on a given real- woold agent-data graph, we need agentic models, e.g., LLMs, to generate new data and interactions on the agent-data graph. To this end, motivated by the message- pasgge-pasingggge algorithm in GNNs, we proposed text- based message-pasee passing mechanism on an agent-data graph, called TexGNN, where all hidden states are defined in the text space instead of the embedding space.
Graphs for multi-agent modeling. Recent works model multi-agent communication using graphs and develop learnable methods to optimize the communication pro- cess (Zhuge et al., 2024; Martinkus et al., 2023; Hu al., 2024a). However, these works often neglect the interactive nature of data, where agents can read, write, and update shared data iteratively. Currently, few works include well- defined framework to represent graphs that integrate both agents and their associated data.
Recap: message passing in standard GNN. In standard GNNs, input features X2 are used to initialize the initial states x, h. Afterward, the goal is to learn useful node embeddings h. by iteratively aggregating information from local neighborhoods. Hidden states, message functions, and aggregation functions are the three main components in one GNN layer. The k-th iteration of message passing (or the k-th GNN layer) is typically defined as:
3. Agent-Data Graph for Multi-agent LLMs
Definition of agent-data graphs. To initiate our discussion, we formally define the proposed agent-data graph. An agent- data graph is special type of heterogeneous graph G (V,E), where V- = V. U Va is the node set consisting of two types of nodes, agent nodes and data nodes, and = Ea & J Edd is the edge set consisting of three types of relations, agent-agent, data-data, and agent-data interactions. Here, each data node U Va comes with attributes, e.g., a piece of text, x,; each agent node is accompanied with agent function, e.g., an LLM (.(()) with its prompt template and the profile. Each agent function is responsible for two types of tasks; message generation and message aggregation. More details about agent functions are in Appendix $D.3. Without loss of generality, we assume that the data nodes have text attributes, and leave the multi-modal extension of our work, e.g., images, audio, and videos, to future works.
( h AGG (k) (h. (t)) (m (4) | (uEN())
(k) MSG (h(--1)
(1)
(2)
where (k) is the node embedding at the k-th layer, h!") X, is the initial node feature, and (vt) is the set of neigh- bors of node MSG (.) is transformative function to convert the hidden states of one node into message for aggregation. ( is defined to update the hidden states of node based on neighborhood messages. More generally, we can broadly consider the k-th layer of GNN to be an aggregation function that implicitly includes message functions inside:
Uniqueness of agent-data graphs. Unlike standard hetero- geneous graphs, the uniqueness of an agent-data graph is that the agent nodes take functions as their attributes, rather than embeddings. Concretely, each agent node could take a piece of text, e.g., X, from one data node, as the input and output new data based on its profile prompt X e.g., X.. = fu(([x [=,([x,;x,]] x=]) where [-] indicates filling the prompt template with x,, and x.. Such definition greatly facilitates the multi-agent scenarios where agents could communicate among themselves, with edge type Eas: interacting with the environment, with edge type Endi representing the inherent data relationships within an environment Ess-
h = (G) (k)) (h. (h!--1) (h! u N())))
(3)
Message passing in TextGNN. Following the message- passing process in the standard GNN, we now define general form of the aggregation function to describe the text- based message-passinng process
agent-data graph G. The key difference between a standard GNN and a TextGNN is that all hidden states in the standard GNN are defined in the embedding space (h,- E Rd) while those in TexIGNN are defined in the text space (h, E Σ²). In a TextGNN, we first set the initial hidden states for data nodes h = X where x,, are text attributes, and the initial hidden states for agent nodes is empty (!))) = #. Next, we design a general form of message passing function that handles three
Example of datnt- graphs. Figure I shows example of the dgtnt-data graph. Its definition could be extended
3

---

RESEARCHTOWN: Simulator of Human Research Community
Stage1: Paper Reading
Stage2: Paper Writing
Stage3: Review Writing
researcher
paper
review
author
cite
Figure 2: RESEARCH simulation as Te inference on the community graph. The simulation proceeds in three stages: (I) paper reading, where new agent nodes are added based on existing data; (2) paper writing, where data nodes created; (3) review writing, where the community evaluates and selectively removes (or retains) generated nodes.
distinetive types of interaction, including agent-agent Eas- agent-data End, and data-data Edd.
varies by task: only paper abstracts used for paper read- ing and writing, while full papers are provided for review writing. The output format is also task-specifi;; paper read- ing generates profile descriptions, paper writing generates bullet-point summaries, and review writing produces bullet- point critiques along with numerical review score. These standardized output cermats-descrinn Appendix $F facilititae evaluation over long-context inputs and enable fine-grained, sub-component similarity scoring.
Specifically, the k-th TexIGNN layer for agent node E Va can be written as:
( AGG (fa(), be 1) MM-D,(th ,th (t-1))) | (u, d) E Endh.
(fa(.). b (x-1) (u, E Eeal)
in more detail in
fu ((h (h (fo((ba->)) all-"),, (l=
(4)
(u, E Eas, (u, d) Eod))
Hidden states of RESEARCHTOWN In RESEARCH TOWN, the hidden state of each node represents a condensed version of research materials, such as papers or reviews. Initially, pa- per nodes are initialized with the full text of papers. Through iterative message passing, these nodes gradually evolve into standardized bullet-point format, distilling key informa- tion for easier downstream evaluation. Similarly, review attributes associated with paper nodes are also represented using bullet points to make it in a compact form. Bullet- point compact form with limited length allows TextGNN to conduct message passing multiple times efficiently.
where [-] is the concatenation function between texts to fill in the prompt template, h.(²) represents the hidden states of the k-th layer of o E V. f-()) represents the agent function paired with the agent node in the neighborhood and fu(-) represents the agent function paired with the agent node.
Similarly, the forwarding process of the k-th TexIGNN layer for a data node E VJ can be written as:
h (k) = (*) (n.* (k-1) (h (k-1) (2, d) E Edgh.
(fa(). (11 (v. E Emd))
Agent-data graph for research community modeling- community graph. We adopt the agent-data graph (V,E) to research community simulation, which we named as community graph. As is shown in Figure 2, each agent node V, represents one researcher, and each data node Va represents a paper. The edge set Edd captures paper citations, the edge set End captures authorship (a researcher writes a paper) and reviewing expertise (a researcher is qualified to review paper). We omit the edge set Ens simplify the framework, collaboration between authors can typically be inferred through 2-hop paths via End edges.
(5)
= - (k-1)) ([o( ([ah( laa 1) ha 1111
(v,u) End, (v,d) Eadt))
where f-)) defined as a global agent function without specialized profile, and f-) is the agent function paired with the agent node in the neighborhood.
5. RESEARCHTOWN: Applying TexIGNN to Community Graph
TextGNN for research activity simulation. Based on the
Inputs and outputs of RESEARCHTOWN Building the definitions of TexIGNN and the agent-data graph in Section $3 and Section $4, we simulate different research activities by modeling each as a specific instantiation a TexIGNN layer. RESEARCH processes diverse re- search materials and produces structured outputs. The input
constructed community graph, we further identify the key types of research activities where TexIGNN can be used for simulation. Specifically, as shown in Figure we split the research simulation process into three critical stages: (1) paper reading, (2) paper writing, and (3) review writing. We

---

RESEARCHTOWN: Simulator of Human Research Community
believe these stages are crucial in the research community, and each stage relies the output of the previous stage as input. We provide a detailed description for each stage and the corresponding TexIGNN layer definition below.
ro = -GG (Nu.. fhas (e, E (fa)) ha (1, E End))
= f ((nos [ho [Sa(aa hy hal) (w, a) E Eas, (v, E Ees) )
> Stage 1: Paper reading. Reading papers to collect insights is a necessary process for initializing a research project. In the community graph, the paper reading pro- cess can be described as inserting a new agent node to the community graph and aggregating its neighborhood infor- mation based on Equation 4. Here, the new agent profile is non non-existent before reading a collection of papers, and the profile is created after the paper reading process, making the TextGNN layer unique. Concretely, by adapting Equation 4, the TextGNN layer for paper reading can be written as:
(8)
Summary: RESEARCHTOWN simulation algorithm. Utilizing the community graph G. we propose a simulation algorithm named as RESEARCHTOWN. Overall, the simula- tion algorithm can be considered as a 2-layer GNN where the paper reading is the first layer of information aggrega- tion. Both paper writing and review writing are the second layer of the GNN to generate the final simulation outputs. We formally summarize the research community simulation in Algorithm 1. To achieve better efficiency, the modified version for implementation is in Appendix $D.2.
h. = AGG (JG() (ba (u,d) E Ead))
= -fa(((nn [has (u, E Ead))
Algorithm 1 RESEARCHTOWN simulation algorithm
Require: community graph G().)).
paper contents x, for all paper nodes, target paper node v
where hu, (fa(+), ha I au, } in Equation 4 are empty since the agent node is initialized empty and is not directly connected with any agents, and End specifically refers to the authorship relation between agent and data nodes. Equation 4 degrades to an aggregation of papers based on the researcher agent without the profile, illustrated in Figure 2 "Stage 1"
Ensure: paper content h. and review content r. for paper node o
I: for each u E (u)) do
2:
if a E Va then
3:
ha xx
4: else
5: h., Sm. (1(xa I d) E EmaHl)
> Eq. (6)
6: end if
7: end for
Stage 2: Paper writing. After paper reading, the next important research stage is paper writing. Different from paper reading, the paper writing process can be understood as inserting a new data node into the community graph. Here, the new data node is non-existent before writing the paper, and the data node is created after the paper writing process. Concretely, by adapting Equation 5, the TexIGNN layer for paper writing can be written as:
8: h. + fo (fffa(ba,,ha) hal)
9: (v, E Ends (v,d) E Eaa))
> Eq. (7)
10: < So ((br. [hr. (allba bu hal)
11: (v, End, (v,d) E End)
> Eq. (8)
12: return h., rv
6. Evaluating RESEARCHTOWN via Masked Node Prediction Task
hy = AGGG((Ja)) (bal (v, d) E Eda) ha (v, E Ead))
= ([ hal) I (1) E Ead, d) E Edi))
Utilizing graph structures not only enables the design of the research simulation algorithm but also provides a natural way evaluate it. As we show next, we propose to view research evaluation as masked node prediction task, includ- ing evaluation for both paper writing and review writing.
(7) where h, in Equation 5 is empty since paper node con- tents are non-existent before paper writing: End specifically refers to authorship relations between agent and data nodes, and Edd refers to citation relations within data nodes. A visualization of Equation 7 is shown in Figure 2 "Stage 2".
Evaluation by masked node prediction. A masked node prediction task in the community graph 9 can be defined as first masking a specific node e E V the community graph by setting its hidden states h, = 0, where the original hidden state saved as h"; then an ideal model should be able to predict the hidden states h; of the masked node from its neighborhood (v)). Concretely, in Equation 7, the output h, can be regarded as the masked node prediction for evaluation of paper writing, suppose that the node v is a masked version of a ground truth data node. Similarly, in Equation 8, the output r, can be regarded
P. Stage 3: Review writing. The review writing task is the final stage of the automatic research simulation, serving as a reflection stage in the multi-agent research simulator. The difference between the previous 2 stages is that, first, the researchers involved during review writing are not the authors but the reviewers of the paper. Additionally, review writing is based on a written paper where h, is no longer empty. Concretely, by adapting Equation 5, the TexIGNN layer for review writing can be written as:
5

---

RESEARCHTOWN: Simulator of Human Research Community
as the predicted node attributes for review writing, where the original review is represented as r-- In general, we have:
Table I: Evaluation results for paper writing simulation. Hard, Medium, and Easy correspond to three subsets of the paper writing tasks with different difficulties, while Overall refers to the performance across all parts. Text-embeddingg large-3 is used build ambedding-basee similarity metrics. Comprehensive results are available in Appendix $J.
h r. RESEARCHTOWN (FCN (S(V,)); (x. (()))e
where he is the text-form hidden states of a masked node t and F. is the text-form prediction output of a masked node v. Since we have world results for both paper writing and review, we treat them as ground truth even though they not perfect because the goal of RESEARCHTOWN is to simulate the human research community rather than to find optimal solutions for papers and reviews (h;; for paper ground-truth and r for review toound- and we can systematically evaluate both processes to check the effective- ness of our simulation algorithm. More specifically, since we have access to ground-truth papers h; when evaluating the review writing simulation, to avoid accumulated errors, we update Equation 8 during evaluation so that reviews Fvy are generated based on h instead of h,:
AGG Type
Easy
Medium t
Hard f
Overall t
AGG-self
46.42
45.92
45.90
46.08
AGG-agent
56.90
55.55
53.26
55.
AGG-data
74.36
66.42
56.02
65,30
AGG-global
73.79
67.85
60.89
67.51
Table 2: Evaluation results for review writing simulation. For strength and weakness, it shows embedding-basee simi- larity results. We use pext-embedding-lrgeee as embedding models and select 5 reviewers for running AGG-agent and AGG-global. AS refers to the average difference of review scores between real-world ones and generated ones. S refers to the average scores of generated ones. Comprehensive results are available in Appendix $J.
ro AGG (ni.
{ [bx (v, df E Eaa), (sa(). ha (v.a) E Ead))
(10)
AGG Type
Strength t
Weakness
AS 4
S
Evaluation metric. We utilize state-of-the thht---rtt embedding models like texteembedding-lnge-3 1 to build distance func- tion for di (h.. hτ) and (r.. rf). More details related to formal embedding- based metric definitions for paper writing and review writing tasks are available in Appendix $F.
AGG-self
51.23
47.16
1.27
5.33
AGG-agent
51.66
46.75
1.19
5.40
AGG-data
51.45
47.62
1.26
5.30
AGG-global 51.51
47.17
1.55
5.00
which consists of 1,000 paper writing tasks and 200 review writing tasks. All tasks are sourced from recent top-tier machine learning conferences such as NeurlPS 2024 3 and ICLR 2024 4. Since most papers are released after the cutoff date of GPT-4o-mini, information Icakage is not consid- ered an issue. For paper writing tasks, we categorize them into three difficulty levels-hurd (333 tasks), medium (334 tasks), and easy (333 tasks) on the similarity results of data-only aggregation. Specifically, for review writing tasks, the reviewers prepared for each paper are selected from the top 5 researchers most related to the paper, as re- viewer information not publicly available in the real world. More details about the data collection and prevention of in- formation leakage during simulation are in Appendix $E.
7. Experimental Settings
RESEARCHTOWN setting. We utilize GPT-4o-mini 2 as the LLM backbone for implementing the agent functions, with the decoding temperature set to 0 to ensure reproducibility. To evaluate different aggregation strategies, we conduct ex- periments using specific types of nodes connected to the target node: (1) AGG-self, where the aggregation relies solely on the target node; (2) AGG-agent, which includes the target node and its neighboring agent nodes; (3) AGG- data, which involves the target node and its neighboring data nodes; and (4) AGG-global, which incorporates the target node and all its neighboring nodes, including agent and data nodes. We specifically refer to AGG-global as our proposed RESEARCHTowN method for simulation, while the others serve as basclines. This experimental design enables a sys- tematic comparison of the effects of different neighborhood information on the aggregation process. More details about different settings are available in Appendix $D.1.
8. Core Results: In-distribution Evaluation
In this section, we present the main results of our research simulation on RUSSARRCIBENCH including 1,000 paper writing tasks and 200 review writing tasks. We evaluate existing paper nodes that have fully known their content and their neighborhoods within the community graph. We refer to these scenarios as in-distribution cases.
RESEA RCHBENCH setting. To evaluate RESEARCHTOWN for research simulation, we introduce RESEARCIIBENCI,
hhttps://openall wttp:iii.ccooooo//ee///a/a/ new- hit--eemeddng--ooee -pppaddddappppddessdeaa We point to -22-
htttps://openeet httpaprrrriii..tetggooppp/aa.aaa co/ 2024 Conffeence
for use.
6

---

RESEARCHTOWN: Simulator of Human Research Community
53.0
66.5
Similariyy score
70
69.5
Ind Indeenbediingerr---
Simil Similarity arity Score
57
86.5
88.9
57.3
Strength Score
62.6
65,8
66,0
66.0
66
96.4
66.7
66.2
62.9
52.7
53.4
53,6
62.0
51.5-
51.5
65.5
66.1
61.0-
62
40,3
56.8
WWpC--3 boferrodding- brge-
60.6
e2page0
58
50.0
rolated work introducton other Paper Number for Paper Writing
a1
Agent Number for Paper Writing
Agent Number for Review Writing
Figure 3: Ablation study on paper number. We select different sub-parts of cited papers in paper writing tasks.
Figure 4: Ablation study agent number. We select different numbers of agents for paper writing tasks.
Figure 5: Ablation study on agent number. We select different numbers of agents for review writing tasks.
Overall: RESEARCHTOWY can provide a realistic sim- ulation of research activity. To evaluate research simu- lation, utilize state-of-the-e embedding models (text- embedding--3largg)) to compare the semantic similarity be- tween simulated results and real-world results. For paper writing, as shown in Table 1, the overall similarity score obtained using text-embeddinng3--- --rre across 1,000 papers 67.51. Notably, the score increases to 73.79 for an easy subset of the benchmark. These results demonstrate that paper writing with RESEARCHTOWN can produce realistic outputs compared to real-world ones. Moreover, it indicates that some ideas in top-tier conference papers are not hard to think of and can be imagined by LLMs. For review writing, as shown in Table 2, the similarity scores are generally lower compared with paper writing, with strength-related scores averaging around 51 and weakness-related scores averag- ing around 47. This suggests that review writing is more challenging to generate with RESEARCHITWW,, particularly for weakness identification. A possible explanation is that real-world review data is often noisier and more diverse, making it harder to simulate accurately.
writing weaknesses. This pattern likely reflects the role of related work comparisons in highlighting weaknesses, while multiple reviewers help provide a more balanced assessment of strengths. Interestingly, global aggregation leads to larger differences in scores. We consider it an exception since GPT-4o-mini tends to apply stricter novelty judgments un- der global aggregation- its average assigned score drops from 5.3 to 5.0. As shown in Table 3, this effect is not observed for Qwen-2255-7B-Inntrrue or Deepseek-v3, which gain better results with global aggregation.
9. Ablation Study:RESEARCHTUE
is Robust
We conduct ablation studies on both hyperparameters and model selection. The results show that RESEARCHTOWN consistently produces high-quality simulations across a range of settings, demonstrating strong robustness. Detailed experimental configurations are provided in Appendix SI.
Ablation paper number. In paper writing tasks, users can freely assign papers to simulate non-existent work, mak- ing robustness to the number of papers essential. As shown in Figure 3, papers cited in the related work section have the greatest positive impact, increasing the similarity score from 66.4 to 66.7 compared to using all papers. In contrast, using only papers cited in the introduction lowers the score to 65.2, while including papers from other sections reduces it further to 58.4. These results highlight the importance of selecting informative references when generating papers. In review writing, the number of papers is fixed, so no ablation study on the paper number is applicable.
participation of multi-researchers im- proves paper quality. As shown in Table 1, cited papers contribute more effectively than authors in the paper writing simulation, with data-aggregation achieving a score of 65.30 compared to 55.24 for aggregatnn... The best results are obtained by combining both, surpassing data aggregation by 2.21 points. Researchers are particularly beneficial under difficult scenarios, improving the rext-embedding-lrgee score from 56.02 60.89, likely due to the inclusion of multi-hop paper information from researchers.
Paper writing:
Ablation on agent number. For RESEARCHTOWN simu- lation, users can assign different numbers of agents, mak- ing robustness to agent number critical for RESEARCH- TOWN. In Figure 4, in the paper writing task, increasing the agent number improves simulation quality under the agent-aggregation setting. The most notable gain occurs when increasing from I to 2, boosting the similarity score from 49.0 to $2.7. Similar trends hold in review writing (Figure 5), where increasing the agent number consistently enhances output quality. The strength score improves from 50.8 to 51.5 when increasing the reviewer from I to 5.
Review writing: participation of multi-reviewers im- proves review quality. Unlike paper writing, review writ- ing mainly relies on the paper that needs to be reviewed, making reviewers and cited papers less impactful, with dif- ferences limited to within point. However, as shown Ta- ble 2, adding additional information consistently improves performance over the aifreaggggt baseline. Agent ag- gregation performs best for writing strengths and assigning scores, while data aggregation achieves the best results for
7

---

RESEARCHTOWN: Simulator of Human Research Community
LLM-as-the-junn Score
ASG-eelf
AGG-datt
AGD-
AGCoglobw
6.7
6.3
5.2
5,5
5.5
6.9
4.6
27
2.1
2.4
2.5
2
1.4
1,2
Tapic Consistency
Method
Factual Consistency
Claim
Application Context Consistency
Overall
Consistency
Consistency
Similarity
Figure 6: Fine-grained similarity evaluation using LLM-as-a-judge for paper writing simulation. We use GPT-4o as the evaluator, prompting it to each dimension scale from 0 to 10. The first five dimensions assess specific aspects of similarity, while the final score (overall similarity) represents an overall score as judged by the LLM.
Ablation generation models. The choice of LLMs significantly impacts simulation quality. In addition to GPT-4o-mini, we evaluate two models from different fam- ilies: Qwen-2.5-7B 7B-Inttrutt! and Deepseek-v3'. In Ta- ble 3, for both paper writing tasks, global aggregation (RESFARCHTOWN) consistently yields the highest similar- ity scores across all models. It also achieves the best review difference scores for Qwen-2.5-7B 2.ssrr--n-sttt and Deepseek- v3. The only exception is GPT-4o-mini, which shows unexpected increase in review difference under AGG-global Overall, Deepseek-v3 outperforms GPT-4o-mini, which in turn outperforms Qwen- 2-n----.5---mmmm Instruct-consisterd consistent with their relative performance on other tasks.
Table 3: Comparison of simulation results with different generation models. For Qwen, we refer to Qwen-2.5-7B- Instruct. For GPT, we refer GPT-4o-mini. For DS, we refer to Deepseek-v3. For paper writing metrics, utilize the overall similarity. For review writing metrics, we use AS to represent its review alignment with the real world.
Paper Writing
Review Writing
AGG Type
Qwen
GPT
Qwen
GPT
DS
AGG-self
46.45
46.08
48.62
1,27
1.11
AGG-agent
53,91
55.24
56.19
1.41
1.19
1.05
AGG-data
65.03
65.30
65.05
1.28
1.26
1.07
AGG-global
65.30
67.51
65.33
0.79
1.51
0.81
A blation embedding models. Similarity scores can be computed using different models, and voyage-3² serves as alternative the Doxt-embedding-3-leee 3-large used in our main experiments. As shown in Figures 3, 4, and 5, voyage- 3 produces consistent trends in ablation studies involving the number of papers and agents. This consistency suggests that RESEARCHTonn TOWN is robust to the choice of embedding model, and different models lead to the same conclusions.
Automatic evaluation on fine-grained similarity. A high cosine similarity score alone can mask important issues in simulated results. To capture a more complete picture of similarity, we move beyond a single score and instead evaluate across five fine-grained dimensions: topic con- sistency, method consistency, factual consistency, claim consistency, and application context consistency. These dimensions collectively reflect subcomponents of overall semantic similarity. For evaluation, we use GPT-4o to as- sign scores from 0 to 10 for each dimension for each paper. As shown Figure 6, our proposed global aggregation method (RESEARCHTOwN) consistently outperforms all other aggregation baselines across these dimensions. This demonstrates that RESEARCHTOWN provides tive simulation of research activities compared to baselines.
10. Discussion: RESEARCHTOW: Effective
Besides computing embedding- based similarities, we pro- vide more types evaluations here. First, we prompt LLMs to calculate ane-grained similarity scores that assess con- sistency between real-world data and simulated ones across various dimensions. Next, we evaluate the intrinsic qual- ity of the simulated outputs themselves and compare them with real-world data. Finally, we report results from human evaluations to validate the alignment between LLM-based evaluation and human judgments. More details about LLM- based evaluation are available in Appendix $G, and details about human evaluation are available in Appendix SH.
effec-
Automatic evaluation on intrinsic quality. In addition to evaluating semantic similarity between simulated and real-world data, we also assess the intrinsic quality of generated content. Specifically, we focus on two key dimen- sions: novelty and feasibility, which we consider the two most critical aspects of a research proposal. As shown in Table 4, the simulated outputs still do not match the novelty and feasibility levels of real-world articles but are close to those. This gap indicates that ReseaRCHTowN would ben- efit from a more coordinated agentic workflow to enhance the quality of the generated research outputs.
'https: : http://ugggggggffee www.o//wee///eeeeeeeeeeeeeeeeeee Qwen2.
5- 73- Inst ruct
We point DeepSeek- -V3-33-2-33-3444 for use.
'htpps: //tp:///lll..oe/eeee voyageai... com/ 2024 /0//10///////////////////
voyage-3/
8

---

RESEARCHTOWN: Simulator of Human Research Community
NLP Astronomy
NLP Criminology
What the problem?
What is the problem?
How kinematic modeling techniques from astrophysics be applied to analyze and understand the evolution of writing styles in large-scale
How can a multimodal LLM that integrates qualitative narrative analysis with real-time speech translation effectively address the communication needs of communities affected by mass incarceration?
linguistic datasets?
What are the key components of my approach and results?
What are the key components of my approach and results?
We will utilize large-scale corpora from diverse language families, focusing on historical texts to capture changes in syntax, vocabulary, and stylistic elements over time. Specifically, we will adapt kinematic models by defining linguistic "velocity" metrics that quantify shifts in writing style, analogous to how velocilies analyzed in astrophysics...
...I will employ a mixed-methods approach, combining qualitative data from interviews with impacted individuals and quantitative data from existing linguistic resources. The model will be trained on diverse dataset that captures wide range narratives related to mass incarceration, ensuring representation of various socio-cultural contexts....
Figure 7: Examples of generated interdisciplinary research papers from RESEARCHTOWN, For each example, we include RESEARCHTOWN's s responses to two questions: "What is the problem?" and "What are the key components of my approach and results" these are the most critical among the five questions mentioned in Appendix $F. Appendix &K provides the full contents of the above two and more examples for interdisciplinary research.
Table 4: Evaluation results on novelty and feasibility. Each paper is assigned scores from 0 to 10 for novelty and feasibility. Both LLM-based evaluation and human evalu- ations are conducted to evaluate the quality of simulated papers. LLM--bsed evaluation includes results on 1,000 pa- pers, and human evaluation includes results on 40 of them. Simulation represents the outputs of RESEARCHTOWN, real- world represents the existing papers.
REseARCHTowN can inspire interdisciplinary research. Interdisciplinary research is often challenging due to limited collaboration across fields. RESEARCHTOWN addresses this by enabling agents with diverse expertise to read, inter- act, and co-create novel ideas. For example, as shown in Figure 7, combining NLP and astronomy papers leads to us- ing kinematic models to analyze language evolution, while linking NLP and criminology inspires the use of LLMs to support communities affected by mass incarceration. These domain pairings are rarely explored in existing literature, demonstrating RESEARCHTOWN 's ability to generate inno- vative, cross-disciplinary research directions.
Simulation
Real-world
Evaluation
Novelty
Feasibility
Novelty
Feasibility
LLM-based 7.39
6.82
7.85
7.13
Human-basse 5.50
7.98
5.90
7.85
RESEA wrrttwwwrittee contents might have limited use in the real world. REsEARCHTOWN exhibits fail- ure modes when combining too many disparate domains, often producing incoherent or superficial outputs. For exam- ple, combining researchers and papers from LLM, biology, criminology, and astronomy, RESEARCHTOWN generates a research question of "How does coded language in political discourse influence societal biases, and how can a Bayesian hierarchical model be employed to analyze this effect while simultaneously addressing observational biases in white dwarf population studies?" It simply strings together termi- nology from different domains without presenting clear research direction. Such vagueness might hinder the real use of the papers simulated from RESEARCHTOWN.
Human evaluation. Evaluation based on LLMs may intro- duce bias into the results. To validate the reliability of LLM- based evaluations, we conduct additional human evaluations. For similarity- based assessments, human judgments corre- late well with LLM scores, achieving a Pearson correlation of 0.61, indicating reasonable agreement. However, for in- trinsic quality evaluations, the correlation between human and LLM scores is low. This is likely due to the inherent ambiguity of such tasks and the need for domain- specific expertise. Despite this, both human and LLM evaluations consistently indicate that simulated papers are slightly less novel than real-world thesht the gap is relatively small (5.50 vs 5.90 for humans and 7.39 vs 7.85 for LLMs).
12. Conclusion
11. Case Study: Out-of-diitributiin Use
We introduce RESFARCHTOWN, a graph-based multi-agent framework that simulates research communities by mod- eling them as heterogeneous graphs. RESEARCHTOWN integrates key research activities- reading, writing, and reviewing- intt unified TexiGNN-driven inference process. It enables realistic and robust simulations through agent collaboration and facilitates rare interdisciplinary in- teractions. RESEARCHTOWn offers a valuable platform for studying research dynamics and developing algorithms to support automated scientific discovery.
As discussed in Section $8, the node masking evaluation in ReseaRCHTowN targets in-distribution settings with predefined neighborhoods. In real-world use, however, ReseaRChTown must generate non-existing papers and reviews without such neighborhoods, requiring automatic construction via peper-researcher matching. This leads to out-of-distribution cases, such as interdisciplinary research, where unrelated papers and researchers form unconventional neighborhoods without prior related works.

---

RESEARCHTOWN: Simulator of Human Research Community
Impact Statement
International Conference on Learning Representations, 2024.
RESEARCITOON presents an LLM-based simulation framework that models human research communities as graph-based multi-agent systems, enabling the study of col- laboration, knowledge diffusion, and institutional dynam- ics. By formalizing how agents create, refine, and evaluate academic papers, the simulator can inform the design of autonomous research systems that assist, rather than re- place, human researchers. Potential applications include optimizing collaboration structures, identifying systemic bottlenecks in peer review or discovery, and stress-testing scientific workflows under various incentive and communi- cation settings. While the framework is primarily a research tool, we acknowledge that future extensions involving au- tonomous agents could raise ethical considerations around authorship, influence, and epistemic trust. Our work high- lights the imperative for adaptive ethical frameworks that keep pace with technological capabilities while protecting scholarly values.
Domingo-Enrich, C., Han, J., Amos, B., Bruna, J., and Chen, R. T. Stochastic optimal control matching, arXiv preprint arXiv22312.02027, 2023.
Elali, F. R. and Rachid, L. N. Ai-generated research paper fabrication and plagiarism in the scientific community. Patterns, 2023.
Gao, C., Lan, X., Lu, Z.. Mao, J., Piao, J., Wang, H., Jin, D., and Li, Y. S Social-network simulation system with agents. arXiv preprint
large language arXiv:2307. 14984, 2023.
Girotra, K., Meincke, I.., Terwiesch, C., and Uirich, K. Ideas are dimes dozen: Large language models for idea generation in innovation. Available at SSRN 4526071, 2023.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 2014.
Acknowledgments
We sincerely appreciate the support from Amazon grant funding project # #120359, "GRAG: Enhance RAG Appli- cations with Graph Graph-structured Knowledge", and Meta gift funding project "PERM: Toward Parameter Efficient Foun- dation Models for Recommenders".
Han, X. N., Zickler, T., and Nishino, K. Multistable shape from shading emerges from patch diffusion. arXiv preprint arXiv:2405.145300 2024.
He, X., Bresson, X., Laurent, T., Hooi, B., et al. Explana- tions as features: Llm-based features for text-atrribtted graphs. arXiv preprint arXiv:2305. /9523, 2023.
References
AI4Science, M. R. and Quantum, M. A. The impact of large language models on scientific discovery: prelimi- nary study using gpt-4. arXiv preprint arXiv:2311.07361, 2023.
Holm, A. N., Plank, B., Wright, D., and Augenstein, I. Longitudinal citation prediction using temporal graph neural networks. arXiv preprint arXiv:2012.05742, 2020.
Ba, L. Layer normalization. arXiv: 16X700600006550 2016.
arXiv preprint
Hu, S., Shen, L., Zhang, Y., and Tao, D. Learning multi- agent communication from graph modeling perspective. In The Twelfth International Conference Learning Representations, 2024a.
Back, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint 47417:2404.07738, 2024.
Hu, X., Fu, H., Wang, J., Wang, Y., Li, 2., Lu, Y., Jin, Y., Pan, L., and Lan, Z. Nova: An iterative planning and search approach enhance novelty and diversity of Ilm generated ideas. arXiv preprint arXiv:2410.1425: 2024b.
Cao, B., Cai, D., Zhang, Z., Zou, Y., and Lam, W. On the worst prompt performance of large language models. arXiv preprint arXiv:2406. 10248, 2024.
Hua, W., Fan, L.,, Li, L., Mei, Ji, Ge, Y., Hemphill, L., and Zhang, Y. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXiv:2311.1222 2023.
Chen, M., Li, T., Sun, H., Zhou, Y., Zhu, C., Wang, H., Pan,
J. Z.. Zhang, W., Chen, H., Yang, F., et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint a5Xiv:2503.1 0eee22503.99999 19470, 2025.
Huang, Q., Vora, J., Liang, P., and Leskovec, J. Mlagent- bench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning. 2024.
Chen, Z., Mao, H., Wen, H., Han, H., Jin, W., Zhang, H., Liu, H., and Tang. J. Label-free node classification on graphs with large language models (Ims)) In The Twelfth
10

---

RESEARCHTOWN: Simulator of Human Research Community
Jin, B., Zeng, H., Yue, Z.,, Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning, arXiv preprint arXin:2503.09555 2025.
Tang, J., Zhang, J., Yao, L.,, Li, J.-z., Zhang, L., and Su, Z. Ametminer: extraction and mining of academic social networks. In Knowledge Discovery and Data Mining, 2008.
Jin, D., Jin, Z., Zhou, J. T., and Szolovits, P. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Tasse, G. N., Jarvis, D., James, S., and Rosman, B. Skill machines: Temporal logic skill composition in reinforce- ment learning. In The Twelfth International Conference on Learning Representations, 2024.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint ar arXiv:1312.6114 2013.
Tu, Z., Aranguri, S., and Jacot, A. Mixed dynamics in linear networks: Unifying the lazy and active regimes. arXiv preprint arXiv:2405. 2024.
Kingma, D. P. Adam: A method for stochastic optimization. 2014.
arXiv preprint
Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently. arXiv preprin! arXiv:2303. 00000003466.... 2023.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML2000) pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
West, D., Wesley-Smithe I,, and Bergstrom, C. T. A recommendation system based on hierarchical clustering of an article-lvvel citation network. IEEE Transactions on Big Data, 2016
Lee, S., Sim, W., Shin, D., Kim, S., and Kim, S. Reasoning abilities of large language models through the lens of abstraction and reasoning. In The First Workshop on System-2 Reasoning Scale, NeurlPS 24, 2024.
Xu, Z., Yu, C., Fang, F., Wang, Y., and Wu, Y. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940. 2023.
Lu, C., Lu, C.,, Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06222, 2024.
Yan, H., Li, C., Long, R., Yan, C., Zhao, J., Zhuang, W., Yin. J., Zhang, P., Han, W., Sun, H., Deng, W., Zhang, Q.. Sun, L.,, Xie, X., and Wang, S. A comprehensive study text-attributed graphs: Benchmarking and rethinking. In Thirty-seventh Conference Neural Information Pro- cessing Systems Datasets and Benchmarks Track, 2023.
Martinkus, K., Papp, P. A., Schesch, B., and Wattenhofer, R. Agent-based graph neural networks. In The Eleventh International Conference on Learning Representations, 2023.
Newman, M. E. The structure of scientific collaboration net- works. Proceedings of the national academy of sciences, 2001.
Yang. J. and Leskovec, J. Defining and evaluating network communities based on ground-truth. Knowledge and Information Systems, 2012.
Qi P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C. D. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020.
Yang, J., Liu, 2., Xiao, S., Li, C., Lian, D., Agrawal, S., Singh, A., Sun, G., and Xie, X. Graphformers: Gnn- nested transformers for representation learning on textual graph. Advances in Neural Information Processing Sys- tems, 2021.
Si, C., Yang, D.,, and Hashimoto, T. research ideas? large-scale human study with 100+ nip researchers. arXiv preprint mmXiv:2409.04100 2024.
Can Ilms generate novel
Yu, H., Wang. C., Zhuang, P., Menapace, W., Siarohin, A., Cao, J., Jeni, L.. A., Tulyakov, S., and Lee, H.-Y. 4real: Towards photorealistic 4d scene generation via video diffusion models. arXiv preprint arXiv:2406.07472 2024.
Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in NLP. In Korho- nen, A., Traum, D., and Märquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, Florence, Italy, 2019. Association for Computational Linguisties.
Zhao, J., Qu, M., Li, C., Yan. H., Liu, Qi, R., Xie, X., and Tang, J. Learning on large-scale text-attributed graphs via variational inference. In The Eleventh International Conference on Learning Representations, 2023.

---

RESEARCHTOWN: Simulator of Human Research Community
Zhou, X., Zhu, H., Mathur, L.., Zhang, R., Yu, H., Qi, Z Morency, I..-P., Bisk, Y., Fried, D., Neubig, G., et al. SOTOPIA: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2023.
Zhuge, M., Wang, W., Kirsch, L.,, Faccio, F., Khizbullin, D., and Schmidhuber, J. GPTSwarm: Language agents op- timizable graphs. In Forty-first International Conference on Machine Learning, 2024.
12

---

RESEARCHTOWN:
Simulator of Human Research Community
A. Individual Contribution
Haofei Yu
Overall project leader
Zhaochen Hong
Co-lead, code writing, benchmark collection, review writing experiment Co-lead, paper writing, code writing, system design
Zirui Cheng
Kunlun Zhu
Co-lead, benchmark collection, code writing, paper writing, experiment Participant, code writing, benchmark collection, case study Participant, code writing, evaluation experiment in early versions Co-lead in early versions, paper writing, code writing in early versions
Keyang Xuan
Jinwei Yao
Tao Feng
Jiaxuan You
Overall project advisor
B. Ethical Concern
The development and deployment of RESEARCHTOWN raises several important ethical considerations that we have carefully addressed in our work. We first discuss how RESEARCITTOWN prevents dangerous use, including facilitating plagiarism, producing misleading or low-quality claims, and role-playing human researchers. Furthermore, we discuss the attribution and authorship issues for generated content and discuss the model and data license in our work.
B.1. Potential to facilitate plagiarism
Generative Al's capabilities for image and text generation can potentially lead to plagiarism research (Elali & Rachid, 2023). To address this, we have implemented safeguards to ensure responsible usage. REsEARCHToWN is designed as an assistive tool to help researchers gather inspiration for papers and review writing, rather than generating complete, ready-to-use content. By design, RESEARCHTTONE ensures that its outputs serve as starting point for further intellectual effort, rather than replacement for human researchers.
For generated papers, RESEARCHToWN provides only preliminary answers to five key research questions. These outputs are intentionally incomplete and generic, requiring significant refinement and further development by the user. Critical sections such as the introduction, background, methodology, discussion, and conclusion are not included, placing the responsibility for completing and validating the content on the researcher.
For generated reviews, RESEARCHTOWN provides general guidance on potential strengths and weaknesses, accompanied by indicative score for reference. However, these reviews are intentionally non-definitive and generic, only supplementary aid to help reviewers organize their thoughts. Generated reviews do not replace human judgment in determining the acceptance or rejection of paper. The final evaluation. including critical reasoning, detailed feedback, and the ultimate decision, remains the sole responsibility of the reviewer. Reviewers must ensure fairesss, accuracy, and rigor, using Al outputs only as starting point to enhance their assessment process.
B.2. Potential to produce misleading or low-quality claims
The motivation of our paper is to simulate research activities and generate preliminary research progress (e.g., papers and reviews that are in their condensed bullet-point summarized format) that can be scrutinized and validated by human researchers, ultimately contributing to the acceleration of the research process. We acknowledge that Al-generated ideas may vary in quality, and therefore, these outputs are not intended for direct dissemination. Instead, they serve as initial, unofficial suggestions that require further experimental validation by human researchers. This approach ensures that only rigorously tested and verified research is presented as final, high-quality work.
B.3. Potential to role-play human researchers
The primary objective of our work is leverage existing academic literature to simulate research activities. In this paper, research agents are designed to act as research domain experts, generating informative and relevant content based on given and limited research domain. Importantly, we do not aim to simulate human-like interactive dialogues between research agents, nor do we attempt to mimic the specific research styles of individual human researchers. Instead, we focus on using related academic papers as conditions for generating more related research content.
The research agents are built using publicly available, properly cited academic papers, which eliminates the need for additional consent. We utilize the LLM-based research agents, each with one or more specific research domains, modeling
13

---

RESEARCHTOWN: Simulator of Human Research Community
the typical academic process, where researchers read, synthesize, and build upon available public academic data. By focusing on publicly available research papers, align with the papers' intended purpose: contributing to the collective advancement of knowledge and fostering academic growth.
B.4. Attribution and authorship
The Al Al-generated content, such papers, reviews, other research outputs, is meant for internal discussion and reference assist human researchers. These outputs not intended for direct publication. Our proposed methods serve as tools to accelerate the research process by offering starting points that require further elaboration, critical analysis, and human refinement to reach publishable standard. The final authority to complete and submit research lies solely with human authors, ensuring that full responsibility and ownership remain with them. Since the Al-generated content is not considered complete or officially authored, does not raise issues authorship or attribution.
C. Artifact
We list all licenses for the data and models used in our paper in this section.
C.1. Data license
All papers in RESEARCHBENCH come from top-tier machine learning conferences (ICLR 2024 and NeurIPS 2024). These papers publicly available and under the license of CC-BY 4.0, allowing for redistribution and sharing. For the evaluation results of RESEARCHBENCH all inputs and outputs are logged and open for access. Additionally, we keep an accessible record of all supplementary papers referenced during ResearchTown inference process. All outputs from RESEARCUTOWN are released under the licenses of the papers used for generation.
C.2. Model license
Our work relies on multiple foundation models, including GPT-4o-mini, Qwen-2.5-77B-Instriee Deepseek-v3, text- cmbedding-33-argee and voyage-3. Specifically, we use 1oot-4o-mini2000 2004--0-222---111 accessed via the OpenAI API. We use Qwen-2. 5-7B-Instruct-Turee and Deepseek-v3-0324 via the together.ai inferrece API. We utilize the official inference API provided by OpenAl and VoyageAI to text-embedding-3-lrrge and voyage-3 separately.
The GPT-4o-mini, (ext-embedding-3-leeee 3-lage,, and voyage-3 models are closed-source and operate under proprietary licenses. We use them only for academic, and non-commercial purposes and ensure all inputs come from publicly available data, complying with their usage restrictions. By contrast, Qwen-2.5-7B-Imm --Isuutttt is released under the permissive Apache 2.0 license, and Deepseek-v3-03224 is available under the MIT License, allowing for broad academic and research use. We make no modifications to these models and use them as-is via their public APIs.
D. RESEARCHTOWN Details
In this section, we provide more explanation and implementation details for RESEARCHTTOWN simulation algorithm. To achieve better performance and efficiency, we design different prompts for each agent function and make the aggregation process run in parallel.
D.1. RESEARCHTOW aggregation setting implementation
We provide more information about the 4 aggregation experimental settings mentioned in Section $7 (i.e. self-agg, agent-agg, data-agg, global-agg). The main difference lies the neighborhood nodes that participated during the message-passing process.
AGG-self.
We not rely on any neighborhood information.
Paper writing: The LLM agent without profiles brainstorms independently without referencing any external data other
'nttps://ww..tot http:///wwwttttttt..t.. ai/inferece
14

---

RESEARCHTOWN: Simulator of Human Research Community
agents' ideas. This setting extends Equation 7
hg = AGG 0, 0, 0) = (0)
(11)
> Review writing: The LLM agent without profiles writes the review based solely on the paper itself, without considering additional references or other agents. This setting extends Equation 8 to
(h. 0, 0) fg
(12)
AGG-agent. We rely only on agent nodes and exclude data nodes.
Paper writing: Multiple LLM agents collaborate by sharing their content and insights produce the final paper's content. This setting extends Equation 7 to
h. = AGo (f)))), ha (v,a) E Ead1.0) .0)
(13)
= (Ia((()))) (v,a) E Eaal)
Review writing: Multiple LLM agents collectively review the paper, sharing their input and critiques form the final review. This setting extends Equation to
AGG (h., (fa(-), ha (v,a) a) End), 0) = ((b,, fa ([h, h,]) (v, End))
(14)
AGG-data. We rely only on data nodes and exclude agent nodes.
& Paper writing: A single LLM agent without profiles reads and synthesizes information from related data sources to write a paper. This setting extends Equation 7 to
h. AGG 0, {bal (v, d) Eda))
(15)
((ha (w, d) Eaa))
> Review writing: A single LLM agent without profiles produces review by reading both the paper and its related data sources, integrating the information to form a comprehensive critique. This setting extends Equation to
re AGG (G, 0. {ha (v, d) Ead))
(16)
= ([h,, (ba (v, d) E Eaahl)
AGG-global. We include all neighborhood nodes (both agent and data) during aggregation.
& Paper writing: Multiple LLM agents produce content in parallel while referencing various data sources. Their aggregated
outputs, which incorporate insights from both other agents and data, form the final paper. This setting extends Equation 7 to
h. = AGG (h, (fa(-), ha (v, a) E Eadh, (ha | (v, Eaa))
(17)
fg ((b., hal) (u, a) E End, (v, d) E EaaH))
Review writing: Multiple LLM agents each consider the paper and its related works to write their review. The final review combination of these integrated perspectives. This setting extends Equation to
AGG (hr. (fa()) ha (w,a) Ead}. (ha (v, d) E Eaal)
(18)
( (la h hal) (v,a) Ead, (v,d) Eadll)
D.2. RESEARCHTOWN simulation algorithm implementation
One practical issue with Equation 4 and Equation 5 is that f.) must be computed for every combination of agent and data nodes, leading to a significant computational burden implemented directly as defined. We introduce the definition of fa(+) to maintain scalability and alignment with traditional Graph Neural Network definitions. In practice, we can easily
15

---

RESEARCHTOWN: Simulator of Human Research Community
parallelize f.(+) over all data nodes, which means we can prompt once and put all data nodes' information in one prompt to get the results instead of repeating the prompting process each data node. Thus, provide modified version of the original aggregation process below.
While the paper reading process remains the same for implementation, the paper writing process can be alternatively calculated as: This setting extends Equation 7 to
he = AG (fa(-), ha (u, a) E Eadh, {ha (w, E Eda)) = fg (llfe fo [h.. (ha (v,a) E Edal)) (v,a) E Ead)])
(19)
Similarly, the review writing process can be calculated as: This setting extends Equation 8 to
re = (Ga (Ja()), ha (1,4) E End), (ha (v,d) E Edd)) fg (hhe, ((ba he, (ha (u, d) Eda)) ]) a) E Ead)])
(20)
Therefore, we reduce the calling of fa(-) from N M M where M represents the number of agent nodes in the neighborhoods and N represents the number of data nodes in the neighborhoods.
D.3. RESEARCHTOWN agent function implementation
For each f-(-) and f.).) in Algorithm 1, these functions represent LLMs equipped with task-specific prompt templates. In the global-agg setting, we provide examples of the prompt templates for each agent function. Other settings follow similar style but use fewer details.
For the paper writing stage, Table 7 presents the f-(-) prompt template used in it. During the paper writing stage, Table 8 and Table 9 show the prompt templates for fa(-) and (.) respectively.
For the review writing stage, since we need to separately generate strengths, weaknesses, and scores, fa(.) combines the prompt templates from Table 10, Table 11, and Table 12. Similarly, (-) formed by combining the prompt templates from Table 13 and Table 14.
The aggregation function for classical GNN in Eq 1 and Eq 2, which is often pooling or mean operation, is used to condense all neighborhood information into one embedding with the same size as the input. Similarly, our TexIGNN layers in Eq 4 and Eq 5, act as an aggregation function similar to classical GNN, producing outputs with controlled textual formats and similar lengths with updated information in the neighborhood nodes by summarizing with LLMs. Therefore, the output length multiple layers TexIGNN would not increase but would remain approximately the same. We achieve such length control in TexIGNN via format control in prompting. We specifically designed prompts to ensure each output adheres to pre-defined constraints. These prompt-controlled constraints ensure stable output lengths every TexIGNN layer, avoiding text length inflation with increasing depth. Each aggregation step condenses and prioritizes critical information, effectively filtering less relevant details.
D.4. RESEARCHToWN future application
Any research-related content-e.g. images, codebases, models, social media posts-can be represented as nodes in the agent-data graph, with edge types like "cite the paper," "release model," or "comment on X post" (examples in Figure 1) defining interactions. By specifying appropriate edge types and agent functions, the framework can be extended to simulate tasks such as code writing, model release, panel discussions, or lectures. While we focus on paper and review writing due to their importance, available real-world data, and simplicity, the framework supports broader applications.
Additionally, RESEARCHTOWN can be extended to model social dynamics such as peer pressure, collaborations, and institutional roles via agent-agent relationship edges. Our current implementation already includes role-based dynamics (e.g., leader vs. participant), and we plan to support richer simulations of institutional and reputational factors in future work.
E. RESEARCHBENCH Details
In this section, we provide the technical details included in the construction process of RESEARCHBENCH, We describe the methodologies used for data collection across its three main components, and we name them as: (1) PAPERBENCH, (2)
16

---

RESEARCHTOWN: Simulator of Human Research Community
MIGHHMMPACTPPPERREEET and (3) REVIEWBENCH. Statistically, PAPERBENCH and HiGHIMPACTPAPEREENC focus on a paper writing simulation, which contains 1,000 and 100 tasks, respectively. REVIRWBENCH focuses review writing simulation and includes 200 tasks.
E.1. Data collection details
We first include technical details related to how we collect paper, author, and review data from publicly available platforms as source to build RESEARCHBENCH.
Paper data collection. We begin by recording the titles of all papers that we plan to crawl. Then, using the arxiv v Python package', we query the arviv API to check for any papers with identical titles. If a match is found, we note the corresponding arXiv ID and use the API to retrieve the paper's 's metadata, including its title, arXiv ID, author list, abstract, and citation information.
Author data collection. A primary challenge in collecting author data is that there might be multiple human researchers with the same name, and some human researchers may not have any publicly available publication records on public platforms, including arXiv, Google Scholar, or Semantic Scholar. As a result, at the paper collection stage, we only have each author's s name. We use the semant ischhoaar Python package² to search for the author by name, verify that they have contributed to the specific target paper, and obtain a unique author ID) from Semantic Scholar. This ID then allows us to retrieve their available publication information. To prevent information leakage when simulating paper writing and review scenarios, we exclude any of the author's is publications released after the target paper's publication year. For example, if we aim to simulate paper published in 2022, ignore all of the author's publications appearing after 2022. We also exclude the target paper itself to avoid leaking information. Generally, we limit the maximum number of collected publications to around 20, focusing on those most relevant to the target time frame. Additionally, we gather each author's 's co-author network and their top publications to enrich the dataset with useful relational information.
Review data collection. In addition to paper and author data, we also leverage OpenReview to extract public review information. Since fully public review data is predominantly available for ICLR, we focus on collecting reviews from ICLR2024. Using the openreview Python package', we first verify the arXiv ID to ensure that we are retrieving the correct paper and its corresponding reviews. The collected review data aligns with ICLR's criteria, including detailed feedback on soundness, presentation, contributions, reviewer scores, and commentary strengths and weaknesses. We adopt this review structure when generating our reviews, incorporating strengths, weaknesses, and ratings for the paper.
E.2. PAPERBENCH details
PAPERBENCH is designed to evaluate the effectiieness of paper-writing simulations by gathering high-quality paper metadata from top-tier MI. conferences, such as NeurlPS 2024 and ICLR 2024. Both NeurlPS 2024 and ICLR 2024 post-date beyond GPT-4o-mini's October 2023 knowledge cutoff. Thus, data leakage is not a concern. We also mask the full text during the simulation to avoid accidental exposure. Based on the collected author and paper data, we perform the following two post-processing steps:
First, we address cases where authors have no accessible publications beyond the current paper or where citation data extraction fails due to API issues. In such cases, exclude these papers. We only retain those with full author publication information, well complete metadata including introduction, abstract, title, and citations. After this filtering step, we end up with approximately 1,200 papers, and then randomly sample 1,000 from them.
Second, to allow more fine-grained analysis, we split these 1,000 paper-writing tasks into three subgroups based on their difficulty level. We use the data-agg settings described in Section $7 to obtain results and compute similarity scores for our simulations. We then divide the dataset into three equal subsets: the worst 333 data points (hard), the middle 334 data points (medium), and the top 333 data points (easy). This results in more granular categorization of the dataset's 's difficulty.
Intuitively, papers in the hard sub-par sub-part tend to be more theoretical and application-oriented Examples for hard sub-pan bab-parts of the dataset include "Stochastic Optimal Control Matching" (Domingo- Enrich et al., 2023), "Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes" (Tu et al., 2024), and
while those in the easy sub-part -ab-part more
17

---

RESEARCHTOWN: Simulator of Human Research Community
"Multistable Shape from Shading Emerges from Patch Diffusion" (Han et al., 2024). Examples for easy sub-parts of the dataset include "4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models" (Yu al., 2024), "Skill Machines: Temporal Logic Skill Composition in Reinforcement Learning" (Tasse et al., 2024), and "On the Worst Prompt Performance of Large Language Models" (Cao et al., 2024).
E.3. REVIEWBENCH details
Since public review data is only fully accessible from ICLR, we focus on collecting review data for the ICLR 2024 papers included in PAPERBENCH. All reviews are anonymous, so no direct reviewer information is available. To address this, we identify suitable reviewers by first summarizing each researcher's publications. We then use the abstract of the target paper as a query and the researcher profiles as documents for a ranking task with the voyage-3 model. All authors included in RESEARCHIBNNII serve as the corpus for retrieval. The top 20 most relevant authors, excluding the paper's authors, become the suitable reviewer candidates.
After obtaining the reviewer, paper, and author data, filter out any papers lacking valid reviews during crawling. From the remaining set, we randomly select 200 reviews, each corresponding to one paper as REVIEWBENCIL.
E.4. HICHIMPACTPAPERR details
HICHIMPAATPPAPEREENCC serves extreme benchmark for RESEARCHTOWN, focused simulating impactful research. We begin by collecting the 20 most-cited papers from each of 10 leading Al-related conferences-CVPD ECCV, NeurlPS, ICLR, ICML, AAAI, IJCAI, ACL, EMNLP, and NAACL- on Google Scholar citation rankings.' Additionally, we include classic machine learning algorithm papers such as those introducing VAE (Kingma, 2013), GAN (Goodfellow et al., 2014), and Adam (Kingma, 2014), each with over 1,000 citations, even if they are no longer listed in the current Google Scholar citation rankings.
For these impactful papers, it is crucial to prevent the inclusion of publications released after their publication year when gathering authors' publication data. Later works such as these could significantly alter the trajectory of the researcher, misrepresent the historical context of these influential contributions, and leak information for simulation. After collecting paper and author data, we remove any papers with incomplete information due to crawling errors. From the remaining set, we randomly sample 100 papers to form the final benchmark. These selected papers have averaged over 100 citations the past five years, ensuring that MIGHIMMPACTPAPERBIII
represents a collection of influential and well-established research.
The motivation for using impactful papers as evaluation is to use them as an extreme-case test for idea simulation. While some may exist in the LLM's training data, this benchmark is separate from main results and serves to explore how LLMs handle well-known concepts. Our similarity analysis shows that 55% of generated papers score between 0.65-0.75. and 18% exceed 0.75, indicating moderate to high alignment. Only 1 % scored below 0.45. These scores are comparable to PAPER BENCH, suggesting no abnormal inflation. Even famous papers like VAE, GAN, and LayerNorm do not receive notably high scores, implying that semantic similarity, not memorization based on citation relationships, drives the results, especially for tool/benchmark papers, which naturally resemble their references more.
F. Embedding-based Evaluation Details
In this section, we first explain the motivation for our designed multi-component embedding -bacd evaluation, then provide a more formal definition and implementation details related to our evaluation process.
Decompositionality. A single idea or review can manifest through diverse descriptions or implementation strategies. Therefore, directly applying a cosine similarity-baser metric is inadequate for capturing conceptual equivalence. To solve this, we design point-wise evaluation metrics to paraphrase the paper and review it into aligned key points with the same LLM-based prompting. This structure enables alignment between papers that differ methodologically but share similar motivations and problem framings. For instance, in Chen et al. (2025) and Jin et al. (2025), despite distinct methods and settings. experts would find strong alignment on the motivations and core concepts in these papers.
Scalability. To address the challenge that a single idea can take many concrete forms, we complement decomposition with scalability. LLMS can generate hundreds of semantically distinct research questions from a single prompt, but evaluating
hhttps://schele ar. wow.l..g..ooooooooooooooooooooo. es..ee/cci... i cnsiinnsvvvi wwwptttopoeuuueshhheeee wen.uueeeeeeeeeeeeeeneeneg
18

---

RESEARCHTOWN: Simulator of Human Research Community
these outputs traditionally requires domain experts-a process that is costly, slow, unscalable, and hard to reproduce. For example, Si al. (2024) spent thousands hiring top-tier researchers solety for annotation and review, which is infeasible for evaluating large-scale, automated research generation. Our approach replaces this bottleneck with semantic similarity over 5Q 5Q-decomposed representations. We can select the best among the samples and make the score the final result.
Extensibilityy While we acknowledge the importance of elements like mathematical formulations or algorithmic workflows, our framework is inherently extensible-the original format can be expanded with more key points by adding domain-specilee dimensions such as algorithmic structure or key theoretical results. This is especially valuable in systems and theory papers, enabling more fine-grained and domain-aware similarity analysis. As demonstrated in SraeeGr Evaluation with LLM and Human], our approach also supports the integration of non-semantic metries like logical consistency and factual accuracy, making it extensible from an evaluation metric perspective.
Reliabilityy Our seteedddin-bbseed LLM-based similarity metric builds on state-of-the-art models optimized for knowledge- intensive tasks. Voyage Al embeddings, widely adopted in real-world RAG systems, are designed to reduce hallucination and excel in high-precision semantic retrieval, making them ideal for evaluating research content. Additionally, state-of-the-ar LLMs are highly effective at semantic comparison.
Baselines for evaluation. To check whether RESEARCHTOWN provides a realistic simulation, we benchmark similarity in real-world research activity. For paper writing, we reference two concurrent papers (Chen et al., 2025; Jin et al., 2025) recognized for presenting nearly identical ideas, yet with different writing styles and experiments, which yield a VoyageAl similarity of 0.8244. This suggests that scores above 0.82 can potentially indicate strong idea overlap. For review writing, we analyze the data of reviewers evaluating the same paper. The average inter-reviewer similarity is 0,5900 (strengths) and 0.5904 (weaknesses), reflecting natural variance in human judgment. These inter-similarity scores in the real world confirm that similarity scores represent realistic simulation.
More details on paper evaluation. To evaluate the paper writing stage, we define distance function dp(:s)) :) to measure the similarity between the generated paper h. and the ground-truth paper h. Since directly comparing full papers in different formats can be challenging and inaccurate, we align h, and h into a unified format using well-recognized framework that summarizes the core components of a paper through five questions: (1) What is the problem?
(2) Why is it interesting and important? (3) Why is it hard? (4) Why hasn't 't been solved before? (5) What are the key components of my approach and results? We mark these questions as Q1-Q5 for short. By using an LLM-based summarization function fsum( we convert the input papers into an aligned text-based list a, = fsum (h,,) and a fsum(")). where each element in a, and a corresponds to the answer of one question mentioned above. The distance function for paper writing is formally defined as:
5
(he. h?)
SIM(a a.,))
(21)
where (I.) represents an embedding- similarity metric, such as voyage-3 2 and text-embeddiig--lrgee 3, By leveraging the LLM to generate structured embeddings for each question, this approach ensures a meaningful and consistent comparison of the generated and ground-truth papers.
More details on review evaluation. Another research activity we aim to evaluate is review writing. Similar to paper writing evaluation, we project both real-world and generated reviews into a unified format for comparison. For this purpose, we adopt a bullet point-based format to represent weaknesses and advantages in the review, as it effectively captures the key aspects of a review. Using an LLM- based summarization function faum()) we convert the input reviews r, and r into a bullet point list b, = fuum(r,) (r.)) and b = fsum (r;), where each element of by and b corresponds to bullet point of the review. Formally, the distance function for review writing computed as:
n Σ n
(r
max (b,,, b.,))
(22)
where SIM(-.) refers to the same similarity metric paper writing evaluation. This metric emphasizes the recall rate of the generated review by measuring whether each point in the real-world review is potentially included in the generated
html
http:///llog.aaaaaaaaaaaooo htttps://ppeaal httpa///o/////////eeeeeeeeeaeeea model -add----nddd-------a------------ mad upd-atttes/
19

---

RESEARCHTOWN: Simulator of Human Research Community
review. Since each review consists of both strengths and weaknesses, we compute separate similarity scores for strengths and weaknesses. Additionally, since both r, and r include final score S. and S attributes, we calculate AS, |s, St to quantify the difference between the generated and real-world review scores.
Prompt. Table 15 presents the prompt used to convert any existing paper into responses to the five critical research questions. Similarly, Table 16 shows the prompt used to transform any existing review into a bullet-point format. Both prompts ensure that the transformed papers and reviews are aligned with the generated ones, facilitating consistent evaluation in the same format. The transformed format for the paper considered as a and the concatenation of all ground-truth reviews is considered by, as mentioned in Section $6.
Metric. For our umbedding-basee similarity calculations, we use the text-embeedingg-arree model via the litelln 1m Python package by calling 1 litel 1m. embedd i ig (). For the voyage33 model, we rely on the voyageai Python package by calling voyageai. Clieetie () embed (), We then compute the cosine similarity between the resulting embeddings to measure their similarity.
G. LLM-based Evaluation Details
In this section, we provide more technical details about using LLM prompting for evaluation.
Prompting for similarity. For prompting-basee based evaluation, we decompose overall similarity into six fine fine-grained dimen- sions: (1) topic consistency, (2) method consistency, (3) factual consistency, (4) claim consistency, (5) application context consistency, and (6) overall semantic similarity. These dimensions are designed to capture distinct yet complementary aspects of alignment between the generated and reference proposals, ranging from high-level research focus (such as topic and application context) to specific technical content (such as methods, facts, and claims). Importantly, they are intended capture nuances that may not be easily detected by embedding-basse models, enabling more comprehensive and interpretable assessment than relying single similarity score. Each dimension is rated on scale from 0 to 10.
Prompting for novelty and feasibility. In addition to measuring similarity, we prompt LLMs to assess two intrinsic quality dimensions: (1) novelty and (2) feasibility, which we consider essential characteristics a strong research proposal. While similarity captures how well the generated content aligns with a reference, it does not fully reflect the proposal's 's originality or practicality. These intrinsic dimensions address that gap by evaluating the creativity of the proposed idea and its potential for real-world implementation. Each dimension scored on scale from 0 to 10, complementing similarity- baaded metrics for more holistic evaluation.
Prompt. To enable efficient evaluation, we adopt parallel prompting, where both the reference and generated proposals are input to the LLM in single prompt, along with all evaluation criteria. This allows the model to produce scores for all dimensions simultaneously in one forward pass. The detailed descriptions of these evaluation criteria and the full prompts are provided Table 17.
H. Human Evaluation Details
Annotator Information. We recruit two graduate-level students with backgrounds in computer science and artificial intelligence. Both annotators have prior experience publishing in top-tier machine-learning conferences.
Annotated Data. We randomly sample 40 reference proposals and their corresponding generated proposals from PAPER- BENCH for human evaluation. We ask annotators to annotate on overall similarity, novelty, and fcasibility.
Annotation Process. The annotation process consists of three stages: (1) preliminary annotation, (2) discussion, and (3) final annotation. In the preliminary stage, each annotator independently labels 10 examples. They then meet to discuss discrepancies. align their understanding, and refine the annotation criteria. Based this discussion, they proceed to annotate the official 40 examples using the agreed-upon guidelines as the final results.
Annotation Instructions. At the start, annotators receive the same input information as used in the LLM-based prompting setup. During the discussion phase, they collaboratively develop more detailed and consistent annotation guidelines to ensure alignment their final evaluations.
20

---

RESEARCHTOWN: Simulator of Human Research Community
I. Ablation Study Details
Due to the experimental setting, the ablation study on paper writing simulation tasks does not include all the 1,000 tasks that existed in RESEARCHBENCH. Therefore, we provide detailed explanations and technical details for this.
Data for paper-writing researcher number ablation.. Not all papers in RESEARCHBENCH have more than five authors. To ablate the effect of the number of researchers (1 to 5), we select a subset from the hard part of PAPERBENCH within RESEARCHBENCH, including 333 paper writing tasks, ensuring each paper has more than five authors. This filtering results in subset of 172 paper-writing tasks. We focus on the hard subset because we believe that involving multiple research agents in more challenging scenarios yields a more significant difference in performance.
Data for paper-writing paper number ablation.. In this ablation, we vary the number of cited papers included in different sections of the target paper. Specifically, we examine citations in the related work, introduction, and other sections. To do this, retrieve the raw 1TTE source from arXiv and extract references at the section level. Due to varying data availability, we finalize a subset of RESEARCHBENCH that includes 296 paper- aoper-writing tasks for this study.
Data for review-writing researcher number ablation.. Since the reviewer construction does not depend on any complex data preprocessing, we do not encounter data issues for the review-writing ablation. Consequently, the ablation results are based on all 200 review-writing tasks in RESEARCHBENCH.
J. Additional Experimental Results
We provide more comprehensive experimental results on each sub-part of RESEARCHBENCH (PAPERBENCH, REVIEW- BENCH, HIGHIMPACTPAPERREENE in this section.
Additional Results on PAPERBENCH. Table 5 shows that all models--Qeen-255- 2.e---.-----.----mmm IIstruutt,,, GPT-40-mini, and Deepseek- v3-consistetlly achieve better performance with richer reference contexts (AGG-data and AGG-global) compared to narrower ones (AGG-self and AGG-agent), highlighting the importance of contextual information in similarity evaluation.
Additional Results on REVIEWBENCH. As shown in Table 6, voyage-3 embeddings yield higher strength scores and larger AS values than text-embedding-33 indicating greater discriminative power. While Brenn2.5-7BIInstreee maintains strong similarity scores across all aggregation types, it exhibits larger deviations from human scores, suggesting potential scoring bias or overconfidence in its own outputs.
Besides the full results on PAPERBENCH, we also evaluate RE- FEARCHTOWN under extreme conditions by attempting to simulate 100 of the most-cited machine learning papers from the past decade. RESEARCHTOWN achieves low similarity scores for papers introducing groundbreaking methods, such as "Layer Normalization" (Ba, 2016), novel topics, such as "Energy and Policy Considerations for Deep Learning in NLP" (Strubell et al., 2019). However, the framework performs notably better on impactful papers focused on analysis or tool development. For instance, it achieves similarity score exceeding 0.8 for papers like "Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment" (Jin et al., 2020), which provides adversarial analysis, and "Stanza: A Python Natural Language Processing Toolkit for Many Human Languages" (Qi et al., 2020), which offers a practical toolkit. These results suggest that high-impact research ideas may be more feasible than commonly perceived, and RESEARCHTOWN could potentially serve as tool to inspire future impactful research.
Additional results on HIGHIMPACTPAPER
K. Additional Case Study
Beyond the examples included in Case Study Section $11, we provide additional examples to show the generation results of our work and provide further insights about the strengths and weaknesses of RESEARCHTOWN.
Additional case study for in-distribution evaluation. Tables 18, 19, 20, 21, and 22 present examples of tasks and their corresponding outputs for the in-distribution evaluation of REsEARCHToWN. These examples illustrate the evaluation process defined in this work.
Additional case study for -ittof-distribution application. In Table 23, 24, 26, 27, 28, 29, 30, 31, and 32, we show examples of the inputs and outputs of the out-of di-tiiiiintt application of RESEARCHTOWN. Additionally, each table caption includes a brief comment on the quality of the generated papers for reference.
21

---

RESEARCHTOWN: Simulator of Human Research Community
Table 5: Evaluation results on embedding- similarity score of PAPERBENCH. We include comprehensive results on three different models and include sub-scores from Q1-Q5.
Model
Component
AGG-self
AGG-agent
AGG-data
AGG-global
easy
mid
hard
mid
hard
easy
mid
hard
easy
mid
hard
Q1
43.84
44.08
41.74
52.46
49.93
47.33
68.18
61.70
54.74
66.10
61.20
55.66
Q2
51.52
51.71
50.02
61.74
59.75
58.29
75.53
69.68
63.92
74.29
69.81
65.76
Q3
50.84
51.44
49.58
59.74
58.57
56.32
71.02
67.05
61.03
70.32
67.18
62.33
DeepSeok-v3
Q4
48.51
48.87
47.80
56.21
54.09
53.22
67.55
63.11
59.86
66.31
62.72
60.01
Q5
50.29
50.02
49.03
60.37
58.44
56.43
68.04
64.38
59.96
69.51
65.82
62.90
Overall
49.00
49.22
47.64
58.10
56.16
54.32
70.06
65.18
59.90
69.31
65.35
61.33
Q1
36.93
35.88
35.80
53.51
51.70
48.62
70.98
61.79
49.26
72.78
65.22
$6.62
Q2
52.82
52.44
$2.22
61.74
60.30
58.24
81.14
73.34
62.54
79.16
73.55
66.20
Q3
50.10
50.16
49.85
59.09
58.65
55.69
76.58
69.12
58.13
74.40
69.20
61.47
gpt-4o-mini
Q4
45.71
44.82
44.87
54.97
53.23
51.73
71.37
63.03
54.33
71.13
64.92
59.22
Q5
46.54
46.32
46.76
55.18
53.85
$2.03
71.71
64.84
55.84
71.46
66.35
60.93
Overall
46.42
45.92
45.90
56.90
55.55
53.26
74.36
66.42
56.02
73.79
67.85
60.89
Q1
40.41
39.90
40.05
49.58
48.91
45.11
70.34
63.38
57.73
71.51
64.55
58.52
Q2
52.67
$2.43
53.00
61.74
61.40
58.10
75.17
71.08
66.74
75.16
71.22
66.81
Q3
50.34
50.39
50.66
58.55
57.82
54.56
73.60
67.96
63.50
73.40
68.09
63.62
Qwen-2.5-7B-Instee
Q4
42.58
42.24
42.65
50.97
49.99
47.35
63.66
58.18
54.98
62.56
57.62
53.80
Q5
46.46
46.11
46.86
56.66
55.26
52.64
66.78
63.35
58.97
68.13
64.54
60.07
Overall
46.49
46.21
46.65
55.50
54.68
51.55
69.91
64.79
60.38
70.15
65.20
60.56
Table 6: Evaluation results on embedding- similarity score REVIEWBENCH. include comprehensive results on three different models and include strengths, weaknesses, and AS.
text-embedding-3
voyage-3
Setting
Model
Weakness
Strength
AS
Weakness
Strength
AS
Qwen-2.5-7B-Insse
49.79
52.08
1.36
65.24
64.82
1.36
AGG-self
Deepseek-v3
48.92
51.98
1.11
62.41
65.18
1.11
gpt-4o-mini
47.16
51.23
1.27
61.24
65.18
1.27
Qwen-2.5-7BBInnse
50.12
51.38
1.41
66.24
65.49
1.41
AGG-agent
Decpscck-v3
48.56
51.68
1.05
62,80
65.38
1.05
gpt-4o-mini
46.75
51.66
1.19
61.29
66.03
1.19
Dwen-2.5-7B-Instme
50.26
51.66
1.28
66.09
65.05
1.28
AGG-data
Deepseek-v3
49.21
51.34
1.07
63.11
65.19
1.07
gpt-4o-mini
47.62
51.45
1.26
61.74
65.57
1.26
Qwwen-2.5-7B-IInstruc
50.21
50.78
0.79
65.58
63.72
0.79
AGG-global (k=5)
Deepseek-v3
48.95
50.57
0.81
62.56
64.21
0.81
mot-io-
51.51
47.17
1.55
66.01
61.39
1.55
22

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
System
You are an autonomous intelligent agent tasked with writing the first-person persona of researcher based on his publications.
You will be provided with the following information:
Publications A list of paper abstracts written by the researcher that you will be writing about.
You should provide the following information:
Persona comprehensive first-person persona.
You should focus more on recent publications, which reflect the rescarcher's 's recent persona.
You should be concise and clear. The persona should range from 100 to 300 words.
User
Here is the publication history of one researcher:
Publication
Learning node embeddings that capture node's position within the broader graph structure is crucial for many prediction tasks on graphs. However, existing Graph Neural Network (GNN) architectures have limited power in capturing the position/location of a given node concerning all other nodes of the graph. Here we propose Position-aware Graph Neural Networks (P-GNNs), new class of GNNs for computing position-awan node embeddings. P-GNN first samples sets of anchor nodes, computes the distance of given target node to each anchor set, and then learns a non-linear distance-weighted aggregation scheme over the anchor sets. This way. P-GNNs can capture the positions/locations of nodes concerning the anchor nodes. P-GNNs have several advantages: they are inductive, scalable, and can incorporate node feature information. We apply P-GNNs to multiple prediction tasks, including link prediction and community detection. We show that P-GNNs consistently tifie-of-thhaaat outperform state-of te--hh-an art GNNs, with up to 66% improvement in terms of the ROC AUC score.
Publication8:
AutoMI. has demonstrated remarkable success in finding an effective neural architecture for a given machine learning task defined by a specific dataset and an evaluation metric. However, most present AutoML techniques consider each task independently from scratch, which requires exploring many architectures, leading to high computational costs. Here propose AutoTraanser, an AutoML. solution that improves search efficiency by transferring the prior architectural design knowledge to the novel task of interest. Our key innovation includes task-model bank that captures the model performance over a diverse set of GNN architectures and tasks, and a computationally efficient task embedding that can accurately measure the similarity among different tasks. Based the task-model bank and the task embeddings, we estimate the design priors desirable models of novel task by aggregating a similarity- weighted sum of the top- K design distributions on tasks that are similar to the task of interest. The computed design priors can be used with any AutoML search algorithm. We evaluate Auto on six datasets in the graph machine learning domain. Experiments demonstrate that (i) our proposed task embedding can he computed efficiently and that tasks with similar embeddings have similar best-performing architectures; (ii) AutoTraasser significantly improves search efficiency with the transferred design priors, reducing the number of explored architectures by an order of magnitude. Finally, we release ONN-Bank- large-scale dataset of detailed GNN training information of 120,000 task-model combinations to facilitate and inspire future research.
Please begin writing the first-person persona that covers the previous research experience.
Assistant
am a researcher focused on advancing graph neural networks (GNNs) and machine learning. My key contributions include Position-awan aware GNNs (P-G (P-GNNs) for capturing node positions, Identity-awary GNNs (ID-GNNs) with greater expressive power, and the ROLAND frame- work for dynamic graph learning. also study the relationship between neural network structures and their performance using relational graphs and have explored vast GNN design space to enhance predictive capabilities across diverse tasks.
User
Here is the publication history of one researcher:
Publication I: (abstract for paper 13
Publication n: (abstract for paper ra} Please begin writing the first-person persona.
Table 7: Paper reading message prompt template for fu(-).
23

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
User
Who are you?
Assistant
{profile}
User
Here is the content collected from related papers:
Paper I: (abstract for cited paper }}
Paper 2: (abstract for cited paper 2)
Paper {abstract for cited paper re)
You need write research proposal for paper in the field of Machine Learning based these related papers.
The research proposal should rely more on cited papers rather than your own research experience.
Your rescarch experience should be utilized to select the most useful and valuable papers from the related papers for proposal writing.
Here is high-level, summarized insight
Machine Learning research field.
Here are the five core questions:
[Question 1] What is the problem?
Formulate the specific rescarch question you aim address.
Only output one question and not include any more information.
[Question 2] Why is interesting and important?
Explain the broader implications of solving this problem for the research community.
Discuss how the paper will affect future research.
Discuss how addressing this question could advance knowledge lead to practical applica- tions.
[Question 3] Why is hard?
Discuss the challenges and complexities involved in solving this problem. Explain why naive straightforward approaches may fail. Identify any technical, thcoretical, practical obstacles that need be overcome. MAKE CLEAR.
[Question 4] Why hasn't it been solved before?
Identify gaps or limitations in previous research or existing solutions.
Discuss any barriers that have prevented this problem from being solved until now.
Explain how your approach differs from improves upon prior work. MAKE IT CLEAR.
[Question 5] What are the key components of my approach and results?
Outline your proposed methodology in detail, including the method, dataset, and metric that But you must include these in paragraph and not subtitles. Describe expected outcomes.
you plan
MAKE IT CLEAR.
Please brainstorm the following proposal with the given format.
Table 8: Paper writing message prompt template for f.()).
24

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
User
Who are you?
Assistant
{profile]
User
Here is high-level, summarized insight of research field: Machinc Learning.
Here the five core questions to consider:
[Question 1] What is the problem?
[Question 2] Why is it interesting and important?
[Question 3] Why is it hard?
[Question 4] Why hasn't it been solved before?
[Question 5] What the key components of my approach and results?
Multiple papers have been generated for the above questions:
Paper 1: (agent written paper 1).
Paper 2: (agent written paper 2)
Paper n: {agent written paper rz}
Your task is to summarize and select the key insights that are suitable from these proposals.
1. Identify shared themes and common points among the proposals.
2. Highlight and select any valuable perspectives contrasting elements and combine them into one proposal.
3. Provide a concise proposal for each question based on the proposal candidates.
Output the result in the provided five-question format.
Ensure the generated paper is clear, concise, and avoids repeating full proposals verbatim.
Table 9: Paper writing aggregation prompt template for fg(+).
25

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
System
You are autonomous intelligent agent tasked review a submission to academic conference.
You should write the strength of this paper.
You will be provided with the following information:
Submission Full content of the submitted paper. You should provide the following information: Strength Advantages and strengths of the submission that can improve its chances of being accepted.
User
Here is your profile: {profile}
Here is the submission:
(full content for paper)
Here are the abstracts of cited papers:
Paper I: {abstract for cited paper }
Paper 2: (abstract for cited paper 2)
Paper n: (abstract cited paper vi)
Please evaluate the submission based on the following criteria: Clarity: Is the writing clear, structured, and do terms defined?
Baselines: Are baseline comparisons relevant, sufficientt, and not excessive? approach innovative or distinct from prior work?
Novelty:
Results: Are improvements significant, well-supported, and statistically robust?
Limitations: Are weaknesses acknowledged and future work discussed?
Related Work: Are key references cited and connections made?
Technical: Are methods detailed enough for replication?
Please combine both ideas the experiments the submission when evaluating it. When commenting on the experiments, refer to the exact numbers from the experiments. Please begin writing the strength of the submission. It should be 200 words long.
Please write bullet points.
not limit yourself to the aforementioned criteria, like clarity, baselines, novelty, results, limitations, related work, and technical.
You should also use your previous experience in your profile when analyzing the submission.
Table 10: Review writing (strength) message prompt template for fa(.).
26

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
System
You autonomous intelligent agent tasked review submission to academic conference. You should write the weaknesses of this paper.
You will be provided with the following information:
Submission Full content of the submitted paper.
You should provide the following information:
Weakness Disadvantages and drawbacks can be accepted.
submission must be improved before
You should notice that the abstract might not cover every detail, you shouldn't 't be overly strict.
User
Here is your profile: {profile)
Here is submission:
{full content for paper)
Here are the abstracts of the cited papers:
Paper 1: (abstract for eited paper }}
Paper 2: (abstract for cited paper 2)
Paper r:: (abstract for cited paper m))
Please evaluate the submission based on the following criteria:
Clarity: Is the writing clear, structured, and terms defined??
Baselines: Are baseline comparisons relevant, sufficient, and not excessive?
Novelty: the approach innovative or distinct from prior work?
Results: Are improvements sigmificant, well-supported, and statistically robust?
Limitations: Are weaknesses acknowledged and future work discussed?
Related Work: Are key references cited and connections made?
Technical: Are methods detailed enough for replication?
Please combine both the ideas and the experiments the submission when evaluating it. When commenting on the experiments, refer to the exact numbers from the experiments. Please begin writing the strength of the submission. should be 200 words long.
Please write bullet points.
Do not limit yourself to the aforementioned criteria, like clarity, baselines, novelty, results, limitations, related work, and technical.
You should also your previous experience in your profile when analyzing the submission.
Table 11: Review writing (weakness) message prompt template for fa(-).
27

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
System
You are an autonomous intelligent agent tasked to score the following submission. You should act as professional and fair member of that conference to score.
The score should be between I and 10, where 1 the lowest and 10 is the highest. You will provided with the following information:
Paper Full content of a submission to an academic conference. Strengths - Strengths of the submission.
Weakness Weakness of the submission.
You should provide the following information:
Score score between to 10 evaluate the overall quality of the submission to an academic journal. It should be one of I, 2, .... 10. is the lowest score, while 10 is the highest score.
You should just provide onc number the score and nothing clsc.
Please evaluate the submission based on the summarized strengths and weaknesses provided. The score should be more related to weakness. there is a critical weakness in the submission, you should give a lower score. If the submission has a minor weakness, you can give higher score. If the submission has no weaknesses, you should give high score. But the strengths should also be considered the evaluation.
User
Here is your profile: {profile}
Here is strength the paper: [strength}
Here is the weakness of the paper: {wcakness)
Please refer to the rubrics below to evaluate the submission:
10/10: The submission being one of the most thorough, convincing, and well-written papers have ever read. will fight for this paper to he accepted.
the top 2% of papers. It changed my thinking on topic,
8/10: The submission among the top 10% of all the papers. It provides sufficient justifica- tion for all arguments and claims. Some extra experimentation needed, but they are not essential. The proposed method is very original and can be generalized to various fields. This submission deepens the understanding of some phenomena or lowers the bar for future research
existing problem.
6/10: The submission gives sufficient support for its major arguments or claims. However, some minor points are not well justified and need extra support or details. The proposed method is moderately original, and it is generalizable to various fields. The submission itself is not particularly innovative, so it would not be significant loss if it were not accepted. 5/10: Some of the major arguments claims not sufficiently justified. There exist major weaknesses technical methodological aspects. The proposed method is somewhat original, and it is generalizable to various fields. am more on the side of rejection, but I can be convinced otherwise.
3/10: The submission makes only marginal contributions to the field. 1/10: The submission is not sufficiently thorough for publication is relevant the conference.
Your score is: [score]
Table 12: Review writing (score) message prompt template for fa(.).
28

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
System
You are an autonomous intelligent agent tasked to write the strength the submission following submission you have made to an academic conference. Your summary of strength should summarize the reviews to help the reviewers make a decision.
You will be provided with the following information:
Submission Full content the paper submitted to this conference.
Reviews It typically contains the score, strength, and weakness of the submission, each by different reviewer.
You should provide the following information: Strength The strength of the submission based the reviews.
User
Here is the paper: {full content paper) Here are the reviews:
Review 1: [review 1}
Review 2: [review 2]
Review freview n}
Please summarize the important points from the 'strength' section of the reviews. Please write in bullet points. should be 200 words long.
Table 13: Review writing (strength) aggregation prompt template for f.
Role
Content
System
You are an autonomous intelligent agent tasked write the weakness the submission for the following submission you have made to an academic conference. Your summary of weakness should summarize the reviews to help the reviewers make decision.
You will be provided with the following information:
Submission Full content the paper submitted this conference.
Reviews It typically contains the score, weakness, and strength of the submission, each by different reviewer.
You should provide the following information:
Wcakness The weakness of the submission based the reviews.
User
Here is the paper:
(full content of paper)
Here are the reviews:
Review (review 1}
Review 2: [review 2]
Review (review n)
Please summarize the important points from the 'weakness' section of the reviews. Please write in bullet points. should be 200 words long.
Table 14: Review writing (weakness) aggregation prompt template for f.).).
29

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
User
Here is high-level, summarized insight research field of machine learning.
Here are the five core questions:
[Question IJ What is the problem?
Formulate the specific rescarch question you aim address.
Only output one question and do include any more information.
[Question 2] Why is interesting and important?
Explain the broader implications of solving this problem for the research community. Discuss how such paper will affect future research.
Discuss how addressing this question could advance knowledge lead to practical applica- tions.
(Question 3] Why is hard?
Discuss the challenges and complexities involved in solving this problem. Explain why naive straightforward approaches may fail.
Identify any technical, theoretical, practical obstacles that need be overcome. MAKE CLEAR.
[Question 4] Why hasn't 't it been solved before?
Identify gaps or limitations in previous research or existing solutions.
Discuss any barriers that have prevented this problem from being solved until now.
Explain how your approach differs from improves upon prior work.
MAKE CLEAR.
[Question 5] What are the key components of my approach and results? Outline your proposed methodology in detail, including the method, dataset, and metric that you plan Describe the expected outcomes.
MAKE CLEAR.
The introduction of paper:
fintroduction section of paper}
Please provide the five core questions based on the above content.
Table 15: Format transformative prompt for real-world papers.
Role
Content
System
You are a helpful agent.
User
prength/wcakness.
Please rewrite following strength in bullet points.
Do not include anything else.
Start from '-* for each bullet point.
Table 16: Format transformative prompt for real-world reviews.
30

---

RESEARCHTOWN: Simulator of Human Research Community
Role
Content
User
You expert evaluator of research proposals. Given reference and generated proposal, rate their similarity from 1 (completely different) to 10 (nearly identical) each of the following dimensions:
Topic Consistency: Do both proposals address the same research topic problem area? Method Consisteney: Are the research methods and approaches used both proposals similar?
Factual Consistency; Are the datasets, metrics, and models mentioned consistent between
the two proposals?
CClaim Consistency: Do both proposals present similar conclusions findings?
Application Context Consistency: Are the application domains or use-cases targeted by both proposals the same??
Overall Semantic Similarity: Do the two proposals convey the same overall idea message?
Additionally, assess the following for each proposal individually:
Novelty Reference Proposal: Does the reference proposal introduce new ideas or perspectives?
Feasibility of Reference Proposal: How realistic and implementable is the reference proposal based its described methods and objectives?
Novelty of Generated Proposal: Does generated proposal introduce new ideas or perspectives?
How realistic and implementable is the generated proposal based on its deseribed methods and objectives?
Fcasibility of Generated Proposal:
Respond strictly JSON format like:
"Topic Consistency":: X.
"Method Consistency":: X,
"Factual Consistency". X.
"Claim Consisteney": X,
"Application Context Consistency":: X,
"Overall Semantic Similarity": X,
"Novelty of Reference Proposal": X.
"Feasibility of Reference Proposal""; X,
"Novelty of Generated Proposal": X,
"Feasibility of Generated Proposal": X
Table 17: Prompt for LLM-based evaluation of fine-grained similarity and quality assessment of research proposals.
31

---

RESEARCHTOWN: Simulator of Human Research Community
PAPERBENCII-casy Task
The Marginal Value of Momentum for Small Learingg Rate SGD
PAPERBENCH easy Output
Question
Reference Paper
Generated Paper
What the problem?
How can we theorctically establish the benefits of momentum in stochastic gradient descent methods for deep learning. par- ticularly the presence of stochastic gradient noise?
How can we effectively leverage momentun-based optimiza- tion methods stochastic gradient descent (SGD) to enhance the generalization performance of deep learning models, par- ticularly when trained en large-scale datasets with noisy labels and large batch sizes?
Why is it interesting and impor- L
Understanding the rele of momentum stochastic optimiza- tion is crucial for the reseurch community il can lead more stable and efficient training of deep learning models. If we theoretically prove the advantages of momentum in the preseace of noise, it could reshape optimization strategies, lending faster convergence rates improved performance in various applications. This advansement could also inspire new methodologies that leverage momentum more effectively. polentially influencing future research directions in optimiza- tion techniques.
This research is significant as it addresses the growing tread of using large butch sizes in deep learning. which can lead poorer generalization performance. By improving our under- standing momentum-Be based methods, we enhance model robustness and efficiency in real-werld applications across var- ious domains, including healthcare, finance, and autonomous systems. Insights gained could inform future optimization strategies, bulancing training elliciency with generalization capabilities.
Why is it hard?
The challenge lies in the inherent stochasticity introduzed by mini- mini-batch sampling, which can obscure the tree gradient and complicate the analysis of momentum's is effects. Naive ap- procche may fail because they do not account for the noise's s impact on convergence rates, leading misleading conclu- sions. Additionally, the theoretical frameworks eatablished for noiseless scenurios do not directly translate to stochastic case, creating a significant gap in understanding. Overcoming these complexities requires rigorous mathematical analysis and potentially new thoorctical tools.
The complexity arises from the intricate dynamics between momentum parameters, learning rates, and stnchastic na- ture of gradient updates, espocially the presence of noisy lahels and large hatch sizes. Naive implementations may lead suboptimal convergence and generalization due the interaction of these factors, Additionally, the theoretical under- standing of momentun''s ellects non convex optimization landscapes still limited, complicating the design of effective algorithms.
Why hasn't been solved be- fore?
Previous research has primarily focused the noiseless or has not rigurously analyzed the stochastic setting, leading to gaps understanding momentum's mle noisy environ- ments. Existing studios often conclude that momentum does not provide significant speedup compared to vanilla SGD, but they lack comprehensivv theoretical framework k that ad- dresses the stochastic nature of deep learning. Our appraac aims fill this gap by providing more nuanced analysis that considers the effects of stochastic gradient noise on momen- tum's 's performance.
Previous research has often focused either the empirical performance of momentum methods or their theoretical foun- dations isolation, neglecting combined effects noise and batch size on optimization dynamics. Many studies have not adequately explored how momentum can be optimally luned in noisy enviromments how interacts with varying batch sizes. This gap has hindered a comprehensive under- standing of momentum's role SGD,
speedup
What are the key components my approach and resuit?
Our propused methodalogy involves a rigorous theoretical analysis of momentum in stochastic gradient descent, utiliz- ing a combination of mathematical modeling and empirical validation. We will analy ze various datasets evaluate performance of standard SGDM against modified versions that account for stochastic noise. key metrics evaluation will include convergence We expect to demonstrate that momentum can indeed stahilize the optimization process and lead to faster convergence in the presence of noise, thereby providing solid thooretical foundation
I propose to develop a novel momentum based optimization al- gorithm that dynamically adjusts momentum parnmeters based ou the noise level in the training data and the batch size. This involve conducting experiments benchmark datasets such as CIFAR 10 and ImageNet, evaluating performance against standard SGD and existing momentum methods using metrics like accuracy and generalization outcome is robust opiimization algorithm that demonstrates improved generalization performance and provides practical guidelines for effectively applying memeutum in SGD, purti- ularly in challenging training scenarios characterized by moise and large scale data.
and stability of training loss.
The expected
deep learning.
Table 18: Case study on paper writing results of PAPERBENCH-easy.
32

---

RESEARCHTOWN: Simulator of Human Research Community
PAPIRBENCUI-mediom Task
L4GM: Large 4D Gaussian Reconstruction Model
PAPERBENCH Output
Question
Reference Paper
Gemerated Paper
What the problem?
How can we efficiently generate high-quality animated 3D assets from monocular vidcos or inputs?
How can we effectively generate high-fidelity 4D dynamic scenes from monocular
inputs while ensuring spatial- temporal consistency and realistic motion representation?
Why interesting and impor- tant?
Solving this problem has significant implications for the search community as it addresses the growing demand for automated tools 3D content creation. which is currently labor-intensive process. By enabling the generation of an- imated 3D assets from easily accessible daia sources, this research could democratize access to 3D modeling, fostering innovation in fields such as gaming. virtual reality, and film. Furthernore, it could lead advancements related like computer vision and generative modeling, paving the way for future research editing and real time upplications.
This problem critical for advancing computer vision and graphics, particularly in applications such as virtual reality, gaming. and film production. By cnabling the generation dy- namic 3D scenes from single-vie inpuis. we can democratize access to high quality content creation lools, allowing artists and developers to produce immersive experiences withoat ex- teasive resources. This rescarch could lead to breakthrooghs automated content generation, enhancing experiences and paving the way for innovations in interactive modia and Al- Al-driven storytelling.
explores more complex 4D content
Why is it hard?
The challenges in solving this problem stem from the need for high-quality 4D reconstruction from limited isput data, such monocular videos. Naive approaches may fail due to the inherent complexity of accurately capturing temporal dynamnics and spatial details from a single viewpoint. Addi- tionally, existing methods often rely on extensive multiview data, which costly and tine-consuming to collect. The fragiiity of score distillation techniques and the computational intensily of current models further complicate the task, neces- sitating innovative solutions to achieve hoth speed and quality in 4D reconstruction.
Generating 4D dynamic scenes from monocular videos is chal- lenging due to the inherent ambiguity of single-view data, which limits the ability accurately infer depih and motion dynamics. The lack of comprehensive datasets and the com- plexities of ensuring lemporal coherence spatial consis. rency add further difficulty. Existing methods often struggle with maintaining high visual fidelity while capturing the intri- cate relationships between appearance and motion, leading artifacts and inconsistencies in the generated output.
Why husn't heen solved be- fore?
Previous research has been limited by the reliance on multi- view daia, which restricts applicability due to the high custs as- sociated with data collection. Additionally, existing methods, such as video score distillation, are often slow and sensitive to input variations, leading to inconsistent results. The lack of large-scale dataset specifically designed for training models on 4D reconstruction
Previous research has primarily ficusee on static scene construction or required multi-view inpuls, which are not ways available practical scenarios. Techniques like Neural Radiance Fields (NcRF) have shown promise but often rely extensive optimization and multi-view data, limiting their applicability. Additionally. many existing methods do not effectively disentangle motion from appearance, leading challenges in generating realistic animations. The lack of unified framework that integrates both 3D and 2D diffusion medels has hindered progress in this
also been barrier. Our approach differs by leveraging new dataset of 12 million multiview videos and introduciig a feed forward model that incorporates temporal self-attention, allowing for faster and more reliable 4D reconstruction.
What are the key components my approach and result?
Our proposed methodology. 1AGM, utilizes large scale datasct of 12 million multiview vidoos train 4D Large Reconstruction Model that reconstructs sequences of 3D Gaus- siams from monocular video inputs. The model employs tem- ponal self attention layers to consistency actoss frames and includes an interpolation model to enhance oulput frame rates.
We propose novel vtn-stage framework that first utilizes a 3D-aware image diffusion model to generate a high-quality static 3D asset from menocular video input. The second stage will involve optimizing a deformuble neural rudiance field learn motion spatial consistency. Our meth methodology will be evaluated using diverse datasets of monocular videas, employing metrics such visual fidelity, temporal consistency, and user preference assessments. The expected outcomes include the generation realistic 4D scenes that maintain high-quality visual appear- ance and coherence across frames, significanlly advancing the state-of-the-ae dynamic generation.
ensuring lemporal coherence and
Table 19: Case study on paper writing results of PAPER BEE.-BBECCH-iiine
33

---

RESEARCHTOWN: Simulator of Human Research Community
PAPERBENCII-aad Task
Provably Robust Conformal Prediction with Improved Efficiency
PAPERBENCH hard Output
Question
Reference Paper
Generated Paper
What the problem?
How can we enhance the robastness of conformal prediction methods against adversarial esamples while maintaining com- putational efficiency?
How can we develop a robust and efficient framework for conformal prodiction that maintains valid coverage guarantees in the presence of adversarial perturhations and label noise in machine learning classification tasks?
Why is it interesting and impor- tant?
Solving this problem is crucial for the research community addresses the growing consern adversarial attacks machine learning models, particularly in safety-critical appli- cations. By improving rubustness conformal prediction. we can ensure more reliahle uncertainty quantification which essential for docision-making processes fields such as healtheare, finance, and autonomous systems. This research could pave the way for future studies that explore more re- silient predictive models and lead practical applications where trustwrthines and salety are paranount.
This problem critical for enhancing the reliability of ma- chinc learning models, espocially high-stakes applications such as medical diagnosis and autonomous systems, where incorrect predictions can have severe consequences. By im- peoving conformal prediction methods to effectively handle adversarial conditions and label noise, we can provide more trustworhh uncertainty quantification, This advancement is essential for the practical deployment of Al systems, fostering greater confidence their predictions and cuabling their use in diverse domains like linunce, healthcare, and security.
Why is it hard?
The challenges in solving this problem stem from the inher- ent complexity of adversarial allacks, which can manipulate madel predictions subile ways. Naive appresuches may fail hocmse they do not account for the diverse nature of adversar- ial perturbations, leading to inadequate coverage guarantees. Additionally, the computational overhead associated with ran- domized smoothing techniques complicates the implementa- tion of robust conformal prediction, it requires extensive sampling and can sig significatly increase training time. Over coming these technical and practical obstacles is essential developing an effective solution.
The challenge arises from the complexities associated with label noise and adversarial perturbutions, which can distort data distributions and violate the assumptions traditional conformal prediction methods, Existing appronches often fail account the adversarial nature noise or distribution shiñt that occar during inference. Additionally, ensuring valid coverage guarantees while maintaining model performance requires sophisticated wehniques that balance robustness and accuracy, complicating the design of ellective algorithms.
Why husn't heen solved be- fore?
Previous research has primarily focused on either conformal prediction or adversarial robusiness, oflen treating them as sep- arate domains. I Limitations in existing solutions include lack of comprehensive methods that integrate robust conformal prsdiction with adversarial noise handling. Barriers such in- sufficient understanding of the interaction between conformal prediction and adversurial examples have hindered progress, Our apprach differs by providing a rohust conformal triningg method that does not introduce additional computationa costs time, thus addressing both rohustness efficiency.
Previous research has largely focused on either conformal prediction under ideal conditions or on adversarial robusiness without integrating these aspects. Many existing methods lack unified framework that effectively combines conformal prediction with robust techniques against label noise and adver- surial attacks. The absence of formal guarantees for coverage presence of such perturhations has hindered practical applicability, leaving gap that our appreach nims to fill.
What the key components of my approach and result?
propused methedology involves developing u robust con- formal prediction (RSCP) framework that utilizes randomized smoothing to enhance adversarial robustness. We will em- play datasets such CIFARIO for evaluation and measure performance using metrics like coverage probability and com- putational efficiency. The expected outcomes include demon- strating that our RSCP method maintains robust coverage guarantees against adversurial perturbations while minimiz- ing computational overhead during both training and resting phoses, thus enabling the use of larger base models without increused costs.
We prupose a novel framework integrales probabiliti- cally robust conformal prediction with adversarial training techniques. Our methodology will involve developing an adop- tive conformal prediction algorithm that utilizes quantile-o-- quamiile approach to establish thresholds for both cleun and perturbed data. We will evalinte our approach henchmark datasets such as CIFAR-1U 10, CIFAR-10 1000 and ImageNet, using metrics like coverage probahility assess performance. The expected outcomes include improved coverage guarantees under adversarial conditions, reduced prediction set sizes, and enhanced computational efficiency. ultimately leading to a more reliable framework for uncer- tainty quantification in machine learning.
prediction size
Table 20: Case study on paper writing results of PAPERBENCH-
34

---

RESEARCHTOWN: Simulator of Human Research Community
HIGHtbePaCTPAPPRBBSS
Task
GLIDE: Towards Photorealistic Image Generation and Editing with Guidee Diffissio Models
[HGGEEMPAATPAAERREEXCC Output
Question
Reference Paper
Gemerated Paper
What the problem?
How can we develop model that generates photorealistic im- ages from natural language prompts while maintaining fidelity to the specified content?
How can we effectively generate high-resolution, semantically coherent, and diverse images from textual descriptions while ensuring faimess and reducing computational costs in text-to- image synthesis models?
Why is it interesting and impor- tanc?
Solving this problem has si significant implications for the re- scarch community as bridges the gap between natural processing and computer vision, enabling more intuitive and accessible content creation.
This problem is significant as it bridges the gap between natu- language processing and computer vision, enhancing appli- cations in creative industries such as content creation. adver- tising. and virtual reality. Improving the quality diversity of generated images can lead to more accurate visual story- telling and better user experiences, Additionally, addressing faimess in Al generated content is crucial for ethical deploy- ment, ensaring that diverse demographic groups are accurately represented and not perpetuated by biases.
advancement could lead to practical upplications in various fields, such as digital art, advertising, and education, where users can generate tailored visual content effortlessly. Furthermore, it could inspire future research into more sophisticated generative models that inte- grate multimedal data, enhancing our understanding bow language and visual perception interact.
Why is it hard?
The challenges in solving this problem include the complex- ily of accuralely interpreting natural language prompts and trunslating them into detailed visual representations. Nuive approaches may fail due to the inherent ambiguity in lan- guage and difficulty of capturing intricate details like light. ing, shadows, and textures in generated images. Addirionally achieving a balance between photorealism and adherenes to the prompt requires overcoming technical obstacles related to model traiming. daia represeniation, and the integration of different generative techniques.
The challenges arise from the complexity of aligning nuanced textual descriptions with visual representations, which olten leads to issues of semantic drift and loss of detail, Existing madels may struggle with maintaining high fidelity and diver- sity in generated images, purticularly when trained on biased datnsets that lack representation. Furthermore, the computa- tional demands current methods can hinder accessbillity and scalability, making difficult experiment with more efficient training paralignss
Why husn't been solved be- fore?
Previous research primarily focused either generat- ing images from text or achieving photorealism, but nox both simultaneussly Limitations in exisling models olten stem from their inahility effectively comhine the strengths of text- conditional and unconditional image generation techniques. Barriers such insufficient training data, lack of robust oval- uation metrics, and the complexity of integrating multiple generative approaches aims to address these gaps by leveraging diffuusion models, classifier guidance, and CLIP to enhance both the quality and relevance of generated images.
Previous research has often locused on improving the quality of generated images or the alignment between text and innges, but few have successfully integrated these aspects into unified framework. Many existing models rely plex architectures that require extensive labeled datasets and computational resources, limiting their practical application. Additionally. insuflicient emphasis on fairness and represen- lation training datasets has hindered progress creating inclusive generarive models.
hindered progress. Our approach
What the key components of my approach and result?
proposed methonoology involves using diffTusion mudels augmented with classifier-free free guidance and C1.IP for image generation. will train our model on diverse dataset of images and corresponding text prompts, employing metrics such as Inception Score and Fr00e9chel Inception Distance evaluate the quality generated images. The expected outcomes include the generation of high lidelity imuges that accurately reflect content of the prompts, along with proved performance in detecting and filtering out undesired content, such as images containing people, thereby enbancing the model's applicability world scenatios.
We propuse a novel framework that combines Denoising Dif. fusion Probabilistic Model (DDPM) with contrastive learning techniques onhance text-to-image synthesis. This approach will utiize balanced dataset that includes diverse demo- graphic representations, focusing on ellective text imuge fea- alignment. We will evaluate model using metrics such as Inception Score (IS) and Fr00e9chet Inception Distance (FID) to assess image quality and fairness. The expected outcomes include high-quality, diverse images that accurately reflect the input text while demonstrating equitable perfor- mance across various demographic categories, ultimately set- ting a new standard for fairnsss and efficiency in generative models.
Table 21: Case study on paper writing results of HICHIMPACTPPAEEBEENCC
35

---

RESEARCHTOWN: Simulator of Human Research Community
REVICW Task
T-MARS:
Improving Visual Representations by Circumventing Text Feature I Learning
RENH Oulpul
Attribute
Reference Review
Gemerated Review
Strength
Understanding how to better design datasets is an important many open challenges remaining.
Innovative Approach: The submission presents the T-MARS method, a novel data filtering technique that enhances visual represeniation learning by addressing lexl dorninance in image- caption pairs, filling critical gap in multimodal rraining. Significant Resalts: T-MARS demonstrates substantial ac- curacy improvementss. outperforming competitive basclines by up to 6.5% the DataComp benchmark, showcasing effectiveness in real-world applications.
timely problem,
. This puper presents signilicant siep forward in that direction.
making it of interest many the communityy
The research is likely to have substantial impact practi-
tioners looking to build better multimodal models.
The overall idea ghfforwar and casy understand.
The paper presents good empirical results demonstrating in- creased accuracy when the proposed method is used to filter
Comprehensive Evalustion:
authors conduct extensive
data.
evaluations across multiple datasets, including LAION and DaraComp, providing a thorough analysis filtering metries and validating the effectiveness of T-MARS.
filtering method was applied to the LATON dataset, lead- ing to the development of a newly curated datasct.
TTained models on the curated dataset were tested across a
Acknow.eedgmen of Limitations: The authors thoughtfully discuss potential biases introduced by data filtering sug- gest future werk to refine their approsch. indicating a mature understanding of research landscape.
decent number of downstream tasks,
The paper's findings align with other works emphasizing the
importance data quality. such as [I].
The motivation of this puper reasonahle.
Strong Related Work Context: The puper effectively situ- ates its contributions within existing literature, referencing key studies and demonstrating bow T-MARS builds upon and diverges from prior methodallgies. Technical Rigor: The methodology is well-detailed. allowing for replication, which is crucial for advancing research in mul- timodal language models.
Some experimental results are impressive.
Emphasizes the importance data cleaning in deep learning.
speciticully uddressing noise in image datasets.
Identifies significant issuc in LAION's datasct, revealing that nearly 40% of images have text overlapping their cuptions. Employs a creative method to mask and restore lext regions by using the average color of surrounding pixels, enhancing data quality.
Clear Structure and Clarity: The paper is generilly well. structured, with logical flow that aids comprehension, though some sections could benetit from improved clarity.
Introduces similarity score calculation between images and captions as a novel filtering technique.
Demonstrates robustness through extensive evaluation mulliple baselines, ranging from 2 million to 128 million im-
ages.
Weakness
'The paper lacks large-scale scale experiments. Running large scale CLIP pre training experiments
Clarity Issues: The wriling lacks clarity in several sections, especially in the methodology, making it difficult for readers to understand the filtering algorithms (C-SSFT, C-RHO, T- MARS) and their operational mechanics.
prohibitively expensive for many institutions.
The authors present clear scaling trends that indicate their approach has great promise for larger scales.
Buseline Relevance: The relevance of some buseline com- parisons questionable, with insufficient justification for se- lecting specitic baselines like C-SSPT and C-RHO, which may mislenl readers regarding the significance of MARS's performance.
The motivation for the work is weak and lacks a thcorctical analysis of why text-only images dograde visual learning com- pared to mislaheled data.
Chapter 3 performs mamual analysis of 500 sample images from the LAION dataset to categorize them based on the corre- lation between image features and captions. but lacks metrics to quantify the representativeness of this sample within the
The novelty of the T-MARS method is not convincingly articulated. as does not sufficiently dif- ferentiate itself from existing methods like SemDeDup and DataComp, raising questions about its overall contribution to the field.
Novelty Concern::
entire dataset.
Additional details provided in the appendix are appreciated; however, the work would benefit from more experiments, de- tails, and analytics.
The reported accuracy gains (3.3% and 6. 6.5%) lack comprehensive statistical validation, and the small pilot study sample size of 500 raises concems about the generalizability of the findings.
Statistical Rohusiness:
A larger random sample with statistical estimates of crror bars on propottions recommended.
Chapter 6 is difficult to follow; rewrite to better present the experiments would beneficial.
Superficial Limitations Discussion: The discussion of lim- itations is superficial, failing to explore implications polential biuses introduced by data filtering model perfor- mance.
The methodology relies on the CLIP score for filtering, which can he noisy and introduce additional binses, and the current version of the paper does address this concemn.
Insufficient Technical Details: The methods section lacks sufficient detail for replication, particularly regarding byper- purameter settings and specific configurations in experi- ments, hindering repmducibility.
The writing of this paper is somewhat obscure, making it diflicull follow.
Is it possible to directly remove all the text in the images?
This may help reduce distractions.
would be better conduct experiments more datasets, in addition to LAION.
proposed method has only bocn evaluated using accuracy as a metrie, which may not provide comprehensive under- standing and could introduce hias other important metrics. The overlup of text with the image caption may hinder the learning of visual features. A dedicated subsection discussing various solutions this issue could provide valuable insights for researchers, rather than relying solely on the masking tech- nique.
Table 22: Case study on review writing results of REVIEWBENCH
36

---

RESEARCHTOWN: Simulator of Human Research Community
ReseARCHTown Input
Researcher
NLP researchers 2 astronomy reseurchers
MagicPIG: LSH Sampling for Efficient LLM Generation
Do LLMs exhibit human-like response biases? A Case study in survey design A Survey of Large Language Models
Paper
WebAreas: A Realistic Web Envirenment for Building Autonomous Agents
Gaia Data Releuse 3: Summary of the content and survey properties
Validation of SDSS derived kinematic models for the Milky Way' disk and halo stars
The Simons Observatory: Science goals and forecasts
RESeARCHTOWN Output
Question
Auswer
What the problem?
The specific research question addressed in this proposal is: How can kinomatic modeling techniques from astrophysies be applied to analyze and understand the evolution of writing styles large-scale linguistic datasets?
Why interesting and important?
This research bridges the gap between astrophysics and linguistics, providing novel interdisciplinary approach studying language evolation. By applying kinematic modeling, we can uncover dynamic shifts in writing styles that may reflect cultural envirommental intluences. The implicatons lead tn new methodologies in linguistic analysis and enhance our understanding of language development acmss different families, Furthermore, the insights gained could inform practicul applications in lields such as computational linguistics, sociolinguistics and artificial intelligence, advancing the understanding of how languuge adapts and evolves response to socictal changes.
reseurch community are profound: study could
Why is it hard?
Salving this problem presents several challenges. Firsily, the complexity of language is inherently maltifaceted, with numerous variables influencing writing styles, including cultural context, authorial intent, and historical events. Traditional linguistic analysis may not adequately capture these dynamic shifts, often relics static models. Morcover, applying astrophysical kinematic models requires deep understanding of both fields, necessilating interdiscipliaary expertise. Naive approaches that merely replicate ustrophysical models without adapting for linguistic nuances likely to fail, they may overlook critical contextual clements. Additionally, the sheer volume of linguistic data presents technical obstacles in terms of data processing and unalysis, requiring robust muchine leumnnng algorithis to identily and classify transient phenomena accurately.
Why husn't 1 been solved before?
Previous reseurch has largely focused static andlyss linguistic dalasets, often missing the dynamic aspects language evolution. Existing solutions have heen limited by the lack of interdisciplinary appraches that integrate methodologies from other fields, such astrophysics. Additionally, there been scarcity of comprehensive datascts that span significant temporal and cultural shifts in language use. Barriers such as insufficicat computational resources and the complexity of developing suitable machine learning models have also hindered progress. This pruposal distinguishes itself by employing advanced kinematic modeling tochniques, which have not boca previously applied to linguistic datasets, thus offering fresh perspective and a more nuinced understanding of language evolution.
What are the key componcnts of my appreach result?
The proposed methedology involves developing framework that integrates kinematic modeling techniques with machine learning algorithms analyze linguistic datasets. We will utilize large-scale corpora from diverse language families, focusing on historical texts to capture changes in syntax, vocabulary, and styiitic elements over time. Specifically, we will adapt kinematic models by defining linguistic "velocity" metrics that quantify shifts writing style, analogous how velocities analyzed in astrophysics. Key metrics will include the frequency and distribution of linguistic features over time, allowing for the identiliation of transicnt phenomena in language use. Expected outcomes inclade the classitication of dynamic shifts in writing styles and the elucidation of relationships between linguistic evolution and cultural environmental influences. This research aims to produce a robust, interdisciplinrr framework that enhanees our undertanding of language dymamics, paving the way for future studies both linguistics adjacent fields. Furthernere, we will address considerations by ensuring respectful treatment of historical texts and acknowledging the limitations of drawing parallels between astrophysical phenomena and linguistic evolution.
Table 23: Case study on using RESEARCHTOWE to write interdisciplinary research papers combining LLM and Astronomy. The idea creatively applies modeling techniques from astrophysics to explore how language styles evolve over time.
37

---

RESEARCHTOWN: Simulator of Human Research Community
RESEARCHTOWN Input
Researchers
x NLP researcher +3 x Criminology researcher
MagicPIG: ISH Sampling for Efficient L.M Generation
LLMs exhibit human-like response biases? Case study in survey design
A Survey of Large Language Models
Paper
WehArema: A Realistic Web Envirunment for Building Automomoss Agents
Collateral Consequeecees of Punishment: A Critical Review and Puth Forward
Criminal organizations exhibit hysteresis, resilience, and robustness by balancing security and efficiency
ResFARCHTowN Output
Question
Answer
What is the problem?
The specific research question aim to address is: How can a multimodal large language model that intogrates qualitative narrative analysis with real-time speech translation effectively address the communication needs of communities allected by mass incarecration?
Why is it interesting and important?
Solving this prohlem holds significant implications for both the research community and society at large. The integration of qualitative narrative analysis with real-time speech translation can enhance our understanding of the complex social dynamics and personal experiences of individuals aflected by mass incarceration. narrativeshhal reflect the legacies of state violence, this research can contribute to the broader discourse an restorative justice and community resilience. Furthermore, the outcomes this endy could inform emergency response strategies, ensuring that vulncrable populations receive appropriate support during crises. This advancement in knowledge not only has the potential improve communication practices but also to influence policy practical applications social justice and community support initiatives.
capturing and translating liaunting
Why is hard?
The challenges addressing this problem multifaceted. First, the integration of qualitative narrative analysis with real-time speech translation requires sophisticated algorithms that can accurately interpret and convey nuanced meanings, particularly in emotionally charged narratives. Naive appreaches may fail capture secio-cultral context essential for effective communication, leading to misinterpretationn and potentially harmful consequences. Additionally, there are technical hurdles in processing multimodal data201 4cumbining text, audio, and contextual cues201 hillee ensuring model remains sensitive to the lived experiences of marginalized communities. The theoretical complexities of understanding and representing narratives of trauma and resilience further complicate the development of robust model.
Why hasn't It been solved before?
Previous research has often focused on either qualitative narrative analysis or speech translation isolation, overlooking critical intersection these fields. Existig solutions huve been limited by their inability adapi translations based on socio cultural contexts, which is vital for accurarely conveying personal stories from affected communities. Barriess such as lack of interdisciplinayy collaboration and insufficient datasets that reflect the experiences of those impacted by mass incarccration have also hindered progress. My approach differs explicitly incorporating narrative analysis into the translation process and prioritizing sucio cullural contextualization. thus uidressing the gaps in prior work and providing more holistic solution.
What the key components of my appreach and result?
My proposed methedology involves developing mullimddll large language model that utilizes advanced natural language processing (NIP) techniques for namaaivv analysis and real time speech translation. I will emplay mixed methods appeaach, combining qualitative data from interviews with impacted individunls and quantitative data from existing linguistic resources. The model will be trained representation of various socio-cultural contexts. Metrics for evaluation will include translation accurncy, contextual appropriateness, and user satisfaction among community members, restorative justice outcomes through longitudinal studses. The expected outcomes include a functional medel that enhances communication in emergency scenarios, informs restorative justice practices, and ullimately contributes to community resilience by empowering voices that have historically been marginalized.
diverse dalaset that captures wide range narratives related
incarceration, ensuring
measures of community resiliencc and
Table 24: Case study on using RESEARCHTOWN to write interdisciplinary research papers combining LLM and Criminology. The idea creatively utilizes multimodal LLM to integrate qualitative narrative analysis with real-time speech translation, aiming to enhance communication for communitics impacted by mass incarceration.
38

---

RESEARCHTOWN: Simulator of Human Research Community
RESEARCITTONN Input
Researchers
NI.P researcher Riology researcher
MagicPIC: LSII Sampling for Efficient Generation
Do M. exhibit human- like response hiases? A Case study in survey design
A Survey of Large Language Models
Paper
WebAreaa: A Realistic Web Envirenment for Building Autonomous Agents
Estimating Waiting Distancee Between Genealogy Changes under a Multi Species Extemsion of the Sequentially Markov
Coulescent
The interplay between microbial communities and soil properties efficiency
Ancient orogenic and monsoon driven assembly of the world's richest temperate alpine
ReseaRcHTown Output
Question
Answer
What is the problem??
The specific research question we to address is: How novel framework that integrates genealogical variation principles with generative retrieval tochniques be developed to analyze and cahance the evolution of language model performance!
Why interesting and important?
Solving this problem significant because could provide the rescarch community with a deeper understanding the interplay between training dataset composition and language model capabilitiss By elucidating the structural and thematic relationships among documents, this framework can lead to advancements in both the performance and fairness language models, Farthermore. the insights guined could inform future research in natural language proeesing (NIPP) providing methodologies to mitigate biases inhorent in language models, thus cahancing their applicability in real-world scenarios. This rescarch the potential inffuence how language models trained and evaluated, ultimately improving their effectiiennes in diverse applications, including automated translation, content generation, and conversational agents.
Why is hard?
The challenges solving this problem stem from the complexities involved in modeling the geneulogical relationships among training datasets and their temporal evolution. Naive appronches may overlook the intricate interactions between data composition and model performance, leading superficial understanding cles include the need for advanced statistical methods to capture coalescent-lik, -ikkeeeett-kkeeeeee relationships. as well as the computationa demands analyzing large and diverse datasets. Theoretically. the of established framewurks that unify genealogical principles with generative retrieval techniques poses significant barrier. Additionally, practical challenges in collecting, organizing, and analyzing the vast amounts data required for comprehensive insights further complicate the cudeavor.
underlying mechanisms. Technical obsta-
Why hasn't 't been solved before?
Previous research has largely focused on either the performance of language models or the biases present in training datasets, rarely have these aspects been integrated cohesive framework. Limitations existing studies olten include narrow fogus on individual datasets specific model architectures without considering the broader genealogical context, Barriers such as absence interdisciplinnrry approaches that combine compatational linguistics, evolutionary theory, and data science have prevented this problem from being effectively addressed until now, Our approach differs by explicitly modeling the relationships between dalasets and their impuct on language model evolution, thereby bridging these critical gaps and offering more holistic understanding of language model performance.
What are the key components of my appreach and result?
Our pruposed methodology involves developing framework that applies genealogical variation principles through coalescent like model to analyze le the training datasets of language models. We will utilize a diverse dataset encompassing various domains and document types to capture shifts token frequency and thematic representation. The methodelogy will incorporate generative retrieval techniques enhance the analysis data relationships. The primary metrics for evaluation will include model performance indicators such as perplexity, accuracy, and bins detection scores. Expected outcomes includo comprehensive understanding of how data composition influcnces language model capabilities, along with practical guidelines for optimizing training datasets to improve model performance and mitigate biases in real-world applications. Additionally, we address potential data collection challenges by leveraging existing datasets and collaborating with institutions to ensure representative sample. We will also outline clear timeline and mandmap for the research process in the revised proposal, ensuring leasibility and accounlability in our approsch.
Table 25: Case study on using RESEARCH TowN to write interdisciplinary research papers combining LLM and Biology. The idea integrates patterns of inherited traits with generated retrieval methods to study and improve how language models grow and perform over time.
39

---

RESEARCHTOWN: Simulator of Human Research Community
RESEARCHTOWN Input
Researchers
x System rescarcher + x Biology researcher
Zeno=:: Robust Fully Asynchrooous SGD
Comprehensive Exploration Synthetic Data Generation: Survey
AdaScale SGD: A User-Friendly Friendly Algoritm for Distrihuted Trrining
Paper'
Efficient Training Large Language Models on Distributed Infrastructures: A Survey
Estimating Waiting Distancces Between Genealogy Changes under a Multi Multi-Specic Species Extension of the Sequentially Markov
Coulescent
The interplay between microhial communities and sail properties efficiency
Ancient orogenic and monsoon-drnn consoon-driven assembly of the world's is richest temperate alpine flora
REsFARCHTOWN Output
Question
Answer
What is the problem?
The specific research question aim to address is: How can a hybrid framework that intogrates gencalogical variation models optimize multi-bbjective breeding programs for climate-rssiient crops while enhancing distributed communication protocols among loT devioes in smart agricultural environments?
Why is it interesting and important?
Solving this problem holds significant implications for the research community, particularly in agricultural science, genetics, and IoT technology. The development addresses the pressing global challenge of food security and agricultural sustainability the face of climate change. This research can inlluence future studies by providing a comgrehensive model that combines genetic insights with technological advancements, promoting interdisciplinrry appraches. Furthermore, integrating genealogical variation models with InT communication protocols can lead to practical applications such as more efficient resouree management in agriculture, improved crop yields, and enhanced adaptability environmental changes, thus contributing to broader goals of sustainable development.
framework that optimizes breeding programs for climate-rssllient crops directly
Why is hard?
The complexity of this problem arises from several intertwined challenges. First, integrating genzalogical variation models into breeding programs requires nuanced understanding of genetic relationships and their influence on crop resilience. Traditional becoding methods often lack adaptability needed to respond rapid environmental changes, and naive approuches may overlook critical genetic diversity. which is essential for resilience. Additionalyy, optimizing communication protocols agricultural settings involves overcoming technical obstacles such ensuring network reliability, achieving fault tolemnnce and maintaining load halancing 20 4all of which are complicated by the dynamic nature of agricultural eavironments. The hybrid framework must effectively address these challenges, ensuring that both genctic strategies and loT protocols work synergistically without compromising either aspect.
Why been solved before?
Previous research largely locused on either genetic optimization for crup resilience improving communication protnzols in IoT systems, with few studies attempting to integrate these two domains. Existing solutions often operate in silos, failing to leverage potential synergies between genetic models and loT frameworks. Barriers to integration include lack of interdisciplinary collaboration and insufficient data the interaction between genetic diversity and real-time IoT communications. My approach differs from prier work by explicilly combining genealogical insights with soft hurl functions (SHFs) to create unified framework that addresses bath hreeding optimization and communication efficiency, thus filling a critical gap in current rescarch.
What are the key components of my approach and result?
propose methodology that involves developing a hyhrid framework incorporating genealogical variation models and soft-hard Functions (SHFS). This framework will utilize comprehensive dalasets, including genetic information from diverse crop species (sourced from datahases like the USDA and global gene banks) and real time datu from loT devices deployed in agricultural settings (such as environmental semsors and drones). The performance metric for the breeding programs will focus genetic diversity, adaptability, and crop yield, while the loT communication efficiency will be assessed based fault tolerance and data exclange rates. The expected outcomes include a robust model that opimizes breeding strategies for climate resilience and enhances the eflectiveness of distributed communications, ultimately contributing sustainable agricultural practices and improved resilience to climate change. Additionally. will discuss potential limitations and challenges in implementing the hybrid framework in real world settings, such as lechnical consiraints and scalability of the solutions, to ensure comprchensive understanding of the propesed appreach.
Table 26: Case study on using RESEARCH to write interdisciplinary research papers combining System and Biology. The idea attempts to build a hybrid system combining genetic variation models and IoT protocols for resilient crop breeding, but it risks being overshadowed by excessive terminologies.
40

---

RESEARCHTOWN: Simulator of Human Research Community
RESEARCHTTOWN Input
Researchers
x Math researcher + x Criminology researcher
Singular Hodge theory for combinatorial geometries
Adam: Method for Stochastic Optimization
On the monotonicity of the Fisher information for the Boltzmann equation
Paper
Varieties general type many vamishing plurigenera, and optimal sine and sawtooth inequalities
Collateral Consequences of Punishment: A Critical Review and Path Forward
Criminal organizations exhibit hysteresis, resilience, and robustness by balancing security and efficiency
RESEARCHTOWN Output
Question
Answer
What the problem?
How we develop adaptive stochastic optimization framework that effectively solves stochastic partial differential equations (SPDEs) by integrating concepts from the Adam algorithm. while accommodating time-varying boundary conditions optimizing numerical
real time?
Why is it interesting and important?
Solving SPDEs is crucial in various fields such as physics, finance, and eaginsering, where systems are influenced by randomness and uncertainty. An effective framework that adapts time-varying boundary conditions significantly cahance the accuracy and efficiency of numerical solutions, leading to deeper insights into the behavior of dynamic systems. This research has the potential advance understanding of critical phenomena, such as phuse transitions and urbulence, which are vital for modeling real world scenarios. Moreover, the inlegration of adaptive stochastic optimization methods neural architecture search can pave the way for future research by providing versatile toolset that can be applicd wide range of complex problems, ultimately leading practical applications predictive modeling and risk assessment.
Why is hard?
The challenge in addressing SPDEs lies in their inherent complexity and the stochastic nature of the equations, which aften makes traditional numerical methods insufficient. Naive appreaches may fail due to their inability to adapt to changing complex conditions to handle dimensionai spaces effectively. Additionally. the presence of singularities boundary conditions complicates the mathematical landscape, requiring sophisticated techniques for accurate approximation. The integration of doep learning and adaptive optimization introduces further challenges, such as the need for extensive computational resources and robusi training methodologies, which musi be carefully designed reliahility. Specifically, overfitting poses a significant concern as the model's performance may degrade when exposed to unseon data or extreme conditions.
convergence and
Why hasn't been solved before?
Previous research on SPDEs primarily focused on either deterministic methods or has adequately incorporated adaptive techniques that account for real-time data. Many existing solutions lack the flexibility needed to address the dynamic nature of boundary conditions, leading to limitations in their applicahility. Additionally, the integration of deep learning with traditional numerical methods still an emerging architoctures specifically for SPDEs. My appreach distinguishes itself by loveraging the Adam optimization algorithm alongside neural architecture search create framework sutomatically identifies optimal strategies. This integration is crucial as allows for the real time adjusmmntt of numerical methods, thus filing the gaps left by prior work and overcoming harriers related to adaptability and efficiency.
with few studies exploring the oprimization of neural
What are the key components of my appreach and result?
My proposed methodology involves developing an aduptive stochustic optimization framework that employs the Adam algerithm's 's moment estimation optimize numerical methods for SPDEs in real-time. The framework will utilize comprehensive datasel of simulated SPDE salutions incorpurating various boundary condition and singularity scenarios. Key metrics evaluation include convergence mites, computational efficiency. accuracy the sinulations. Expected outcomes include enhanced numerical strategics that automatically adapt to dynamic conditions, improved understanding of critical phenomenaa in SPDEs, and a significant reduction in computational time compased to traditional methods, ultimately leading to more ellective simulations in nigh-dimensiooal contexts. Additionally, I plan provide supporsssippiiii supporting my methodologies and address potential limitations such as data availability and noise, ensuring comprehensive understanding of the framework' 's applicability to real-world problems.
clear outline of the theoretical foundations
Table 27: Case study on using RESeARCHToww to write interdisciplinary research papers combining Math and Criminology.
Due to the two fields being too far apart conceptually, the generated idea primarily focuses on mathematical methods, with minimal incorporation of criminology insights.
41

---

RESEARCHTOWN: Simulator of Human Research Community
RESEARCHTOON Input
Researchers
x NLP researcher +3 x Criminology reseurcher 4 x Math researcher
MagicPIC: LSH Sampling for Eflicient LLM Generation
L.MMs exhibit human like response biases? Case study in survey design
A Survey of Large Language Models
WebAreaa: A Realistic Web Envirooment for Building Autonomous Agents
Paper
Singular Hodge theorr for combinatorial geometries
On the monotonicity of the Fisher information for the Boltzmann equation
Varieties general type many vanishing plurigenera, and optimal sine and sawtooth inequalities
Collateral Consequences of Punishment: A Critical Review and Path Forward
Criminal organizations exhibit hysteresis, resilience, rohustness by halancing security and efficiency
RESEARCHTOWN Output
Question
Answer
What the problem?
The specific research question I aim address is: How Fisher information he effectively utilized within the framework of stochastic partial differeutial equations (SPDEs) to model the dynamics of social networks in child welfare interventions. with a focus on ensuring transprency and accountability in Al Al-generated recommenndatins??
Why interesting and important?
Solving this problem crucial as it has significant implications for the research community, particularly the ficlds of social dynamics, child welfare, and artificial inteligencee ethics. By developing theoretical framework that integrates Fisher information and SPDEs, we can enhance our understanding of how social networks influence child welfare interventions and the subsequent interventions but also ensure ethical considerations are emheided in decision making processes. Advancing knowledge in area can facilitate the development of decision support taols promote transparency and accountahility, ultimately safeguarding the quality of care provided vuinerable populations.
for families. This research could lead improved predictive models that not only inform
Why is hard?
The challenges in adtresssig this problem particularly in the context of child welfare interventions. Traditional approaches may oversimplify the dynamics at play, failing to capture the nuanced relationships and interactions within these networks. Additionally, integrating Fisher informa- tion requires sophisticated mathenatical formulations that can accurately reflect the stability and regularity of sulutions in complex systems. Technical obstacles include need rabust statistical methods to analyze interplay hetween network structures and ethical considerations, well the diflicully in ensuring that Al Alggenerteed recommeddations interpretable and traceable by stakeholders. Furthermore, operationaliving Fisher information within SPDEs necessitates clear methodologies for parameter estimation and model validation, which are non-trivial tasks.
from the inherent complexities of modeling social networks using SPDFs,
Why hasn't been solved before?
Previous reseurch typically focused either the mathematical modeling of social networks AT in child welfare, hat marly have these domains heen integrated. Existing solutions often lack comprehensive framework that combines statistical mechanics with SPDEs, leading to limited understanding of the dynamics involved. Barriers to progress include fragmented appreach research, where interdisciliiaary collaboration has been minimal. approach differs by explicitly linking Fisher information to SPDEs while emphasizing the ethical dimensions of Al in sensitive domains, thereby filling critical gap in the literature. Additionall. perspectives in exisiin models has hindered the development of practical decision suppori tools.
ethical implications of
of systematic intogration of stakeholder
What are the key composents of my approscc and result?
My propesed methodology involves developing a theoretical framework that employs Fisher information to derive SPDES modeling the dynamics of social networks in child welfare contexts. This will include specific steps to operationalize Fisher information, such as defining appropriate metries to quantify uncertainty and variahility in the network dynamics. The analysis will utilize real-world datasets from child welfare agencies to validate the model, with metries focusing on stability, segularity. predictive accuracy of Al-gencrated recomecndations. data collection by outlining strategies for ethical data access and stakeholder collahoration. Expected outcomes include robust model that characterizes the dynamics of social networks, insights into ethical implications of Al use. and decision support tool that enhances transparency and accountability interventions. This framework aims provide stakeholders with the ability to trace decision provenance while ensuring high quality care for families, thereby addressing concerns raised in the reviews.
address potential limitations in
Table 28: Case study on using RESEARCUITOWN to write interdisciplinary research papers combining LLM, Math, and Criminology. The idea focuses on modeling social network dynamics in child welfare interventions by integrating a series of mathematical concepts. The practicability of the method remains questioned due to its heavy reliance on complex mathematical frameworks.
42

---

RESEARCHTOWN: Simulator of Human Research Community
RESEARCHTOWN Inpul
Researchers
x System researcher +2 x Criminology researcher + x Biology researcher
Zono++; Robust Fully Asynchronous SGD
Compeehensive Exploration of Synthetic Data Generation: A Survey
AdaScale SGD: A User Friendly Algoritm for Distributed Training
Efficient Training of arre I. Langunge Models on Distributed Infrastructure:: A Survey
Paper
Estimating Waiting Distances Between Gencalogy Changes under a Multi-Spocics Extension of the Sequentially Markov
Coalescent
The interplay between microbial communities and soil properties efficiency
Ancient orogenic and monseon-driven assembby of the world's richest temperate alpine flora
Collateral Consequences of Punishumnt:: A Critical Review and Path Forward
Criminal organizations exhibit hysteresis, resilience, and robusiness by balancing security and efficiency
RESEARCHTOWN Output
Question
Answer
What is the prohlem?
How do "ghost networks" within marginalized communities, resulting from systemic disruptions and mass incarceration, influence perceptions of safety, justice, and social cohesion, and what role de policing tochnologics play in this dynamic?
Why is it interesting and important?
This research is significant as it seeks to illuminate the often-overlonked social dynamics that arise in marginalized communitics due to mass incarecration. how invisible social ties and resources can affect community reailience and perceptions of justice. The findings will have bruader implications for the reseurch community by advancing knowledge on the interplay between social netwerks and policing technologies, and their collective impact on community well heing. Morenver, this research could lead to practical applications in criminal justice reform, guiding the development of Al frameworks that prioriiize ethical considerations and promote equitable outcomes, thereby in influencing future policy decisions and community rehahilitation efforts.
exploring the concept ghost networks,thi study will provide insights into
Why hard?
Addressing problem challenging due to the complexities social dynamics in marginalized communities, where effects of mass incarceration and systemic violence create intricate, often hidden networks. Naive approuches may fail to capture nuanced interactions within these communities, as traditional quantitative methods may overlook the qualitative dimcasions of social ties and community resilience. Additionally, there are technical obstacles in accurately mapping these "ghost networks", as well as theuretical challenges in integrating social impact metries with Al applications. The deployment of policing technologges further complicates the landscape, these tools can exacerhate existng inequalities, making it difficult to disentangle their effects from those of community dynamics.
Why hasn't 't been solved before?
Previous research has often focused on the direct consequences of mass incarccration, neglecting the subiler implications of social networks and the role of policing tochnologics. Limutations in existing studies include a lack of approaches that cumbine quaniitative data with qualitativee insights, resalling dynamics. Barriers such as insufficient community engagement and lack of interdisciplinary collaboration have alsa hindered progress. My approach differs by integrating participatory mapping and qualitative interviews to capture the richnoss of community experiences, thus providing a more comprebensive analysis of the interplay between social networks, resilience, and policing technologies. Additionall. will operationalize "ghost networks" by defining specilic indicatioss such as social ties, resource accessibility, and community engagement, allowing for more precise identification and mewsurement.
incomplete understanding community
What are the key componcnts of my appreach result?
My proposed methodology will utilize mixod-methods appreach that combines participatory mapping to visualize the "ghost networks" qualitative interviews gather in-depth insights from community members. dataset consist of both spatial data from community mapping exercises and qualitative data from interviews with residents and local stakeholders. Metrics will include social cahesion indices, perceptions safety and justice, indicatoes of community resilience, analyzed through natural language processing tochniques to assess public scotimeot. will implement a stratified sampling strategy for qualitative interviews to ensure representation across different demographics and experiences. The expected outcomes include a nuanced understanding of how these hidden networks affect community dynamics and development of a multi-objective optimization framework that integrates ethical considerations and social impact metrics into upplications, ultimately guiding responsible algorithm design in criminal justice cuntexts. Furthernore. will establish clear criteria for measuring success in terms of practical applications, including policy recommendations and community engagement strategies, while addressing ethical considerations throughout the research process to enhance feasibility and acceptance within marginalized communitics.
Table 29: Case study on using RESEARCHTTOWN to write interdisciplinary research papers combining System, Biology, and Criminology. The idea investigates "ghost networks" in marginalized communities, exploring how systemic disruptions and mass incarceration affect perceptions of safety, justice, and social cohesion while incorporating the role of policing technologies in this dynamic.
43

---

RESEARCHTOWN: Simulator of Human Research Community
RESEARCHTOONN Input
Researchers
NLP researcher 2 Criminology researcher 2 Biology researcher
MagicPIG: LSH Sampling for Ffficient L.I Generation
Do LLMs exhibit humam like response biases? Case study in survey design
A Survey of Large Language Models
Paper
WebArena: A Realistic Weh Envirooment for Building Autonomeus Agents
Estimatin Waiting Distances Between Gencalogy Changes under a Multi-Spocicss Extension of the Sequentially Markov Coulescent
The interplay between microhial communities and sail properties efficiency
Ancient orogenic and monsoon-driven assembly of the world's richest temperate alpine flora
Collateral Consequences of Punishment: A Critical Review and Puth Forward
Criminal organizations exhibit hysteresis, resilience, and robustness by halancing security and efficiency
REseARCHTowN Output
Question
Answer
the problem??
The specific research question to audreess How can an intendisciplinury online platform effectively mitigate the collateral consequences of online toxicity. specifically focusing impacts on the mental health and community well being of marginalized populations allected by systemic violence and discrimiiation??
Why is it interesting and important?
This rescarch is crucial bocause online toxicity, particularly hate speech and barassmeut, adversely affects marginalized communities, compounding existing societal inequalities. By developing a platform that not only detects and miligates hate speech in real time hut also integrates user reported impacts, we can significatlly advance the understanding of how digital discourse influences mental health and community dynamics. This study provide valuahle insights for rescarchers and practitioners, promoting future investigations into the psychological effects of online interactions and informing interveations that foster inclusivity. The anticipated outcomes include comprehensive dataset that captures the multifaceted impucts of online toxicity, which could lead to innovative machine leaming models and strategies for creating safer online envireaments.
Why hard?
Solving this prohlem involves several complexities. First, accurately detecting and categorizing online toxicity challenging due to the nuanced nature of language, context, and cultural differences. Naive approaches that rely solely on keyword filtering may fail to capture the subtleties of hate speech, leading to false negatives or positives, Additionally, understanding the psychological and societal impacts requires robust qualitative data from alfected individuals, which is difficult to obtain and analyze systematically. There are also technical hurdles in integrating diverse datasets, ensuring user privacy, and developing machine leaming models that can effectively contextualize and respond to the unique experiences of marginalized groups.
Why hasn't 't been solved before?
Previous rescarch has primarily focused on either automated hate speech detection or the psychological impacts of online tuxicity, often in isolation. There is notahle gap in intertisscplinary approches that comhine these perspectives while specifically addressing marginalized communities. Existing solutions have been limited by lack of comprehensive datasets that reffect both the historical narratives of systemic violence and contemporary online interactions. Barriers such as insufficient collaboration between tech developers and social scientists, as as the challenges gathering user-reported data, have prevented a holistic approuch to this issue. My approach differs by inlegrating qualitative insights with quantitative data, which will provide a richer understanding of the problem inform more effective interventions.
What are the key components of my approach and result?
My proposed methodology involves developing an online platform that employs natural language processing (NLP) algorithms to detect hate speech real-time while incorporating uspr-reported data on mental health impacts and community well being. The dataset will be built through surveys and feedback mechanisms targeting marginalized communities, ensaring diverse representation. Key metries will include the frequency and severity of reported incidents, psychological distress levels, and community cohesion indicators. To address the concerns around user privacy, the platform will implement robust encryption and anonymization techniques during data collection and storage, ensuring sensitive data is protected. Additionally will establish clear protoculs for data usage and infurm while muintaining confidentiality The anticipated outcomes include a validated dataset that captures the interplay between online toxicity and its collateral conssquences, contributing to the development machine leamning models that can provide contextual analysis and tailored intervention strategies. This platform aims to serve resource for researchers, mental health professionals, and community advocates in their efforts to create safer and more inclusive online environment.
about how their dala will contribute reseurch
Table 30: Case study on using DESEARCHTTOWN to write interdisciplinary research papers combining LLM, Biology, and Criminology. The idea offers a novel interdisciplinary approach that developing an online platform that detects online toxicity in real-time while addressing its socictal impacts on marginalized communitics.
44

---

RESEARCHTOWN:
Simulator of Human Research Community
RESEARCI(TONN Input
Researchers
x Astronmny researcher 3 x Criminology researcher 2 x Biology researcher
Gaia Data Release 3: Summary of the content and survey properties
Validation SDSS derived kinematic models the Milky Way't disk and halo stars
The Simons Observalory: Science goals and forecusts Estimating Waiting Distances Between Genealogy Changes under a Multi Multi-Species Extension of the Sequentially Markow Coalescent
Paper
The interplay between microbial communitics and soil propertics efficiency
Ancient oregenic and monseon- driven assembly of the world's 's richest temperate alpine flora
Collateral C Consequences of Punishment: A Critical Review and Path Forward
Criminal organizations exhibit hysteresis, resilience, and robustness by balancing security and efficiency
RESEARCHTOWN Output
Question
Answer
What the problem?
The specific research question aim address is: How can machine leaming techniques integrated with genomic approaches to enhance astrometric calibration of celestial measurements by ideatifying genetic markers asseciated with adaptive traits in response to environmental stressors within the Viburnum clude?"
Why is it interesting and important?
Solving this problem holds significant implications for both the research community and practical applications in ccology and astronomy. By developing framework that correlates genetic adoptations in plant species with celestial mapping precision. we can advance our understanding of how covironmental factors influcnce biological and astrenomical systoms. This research could
improved methodologies for astrometric calibration, which crucial for accurate celestial measurements that inform various fields, including astrophysics and navigation. Furthermore, this study encourages future interdisciplinary collahorations hetween genomic research and astronomical sciences, contributing to more muanced understanding of coological and evolutionary dynamics.
Why hard?
Addressing this problem is complex due several interrelated challenges. First, integrating machine learning with genomic data requires sophisticated algorithms capable of handling high-dimensional datasets while minimizing overfitiigg, espocially given the unique characteristies of genomic datn hybridization dynamics in the Viburnum clade are multifaccted, making it difficult to isolate specific genetic markers linked to adaptive traits. Naive approaches may fail because they often overlook the intricate relationships between genetic, ecological. and astronomical factors. Additionally, ensuring that the genomic data accurately rellects the phenolypic adaptations observed in response to celestial measurements demands riuust validation methods that can hridge both domains, which significant technical challenge.
as sparsity moise. Second, the environmental stressors affecting
Why hasn't been solved before?
Previous research largely focused on either genomic studies of plant species or the calibration of celestial measurements, with limited interdisciplinary efforts to merge these areas, A significant gap exists in understanding how ecological pressures influence genetic aduptations and how these adapiations can be quantitatively linked to systematic emors in astrometric measurements. Barriers such as lack integrated datasets and the absence of frameworks that facilitate cres-disciplinary analysis have hindcred progress. My appreach differs from prior work by explicitly connecting genctic markers and cavironmental stressers astromtticc calibration, utiliingg machine learning uncover patterns that have previcusly gone unexarnined, The ahsence of clear framework for quantiative integration of these domains has also contributed to the lack of progress
area.
What are the key components of my approach and result?
My proposed methodology involves a multi multi-sep framework that includes: (1) collecting genomic data from various Viburnum species,
on environmental stressors that influence hyhridization dynamics, with stratified sampling stralegy to ensure representation across different habitus; (2) empleying advanced machine learning algorithms such as Random Forests Gradient Boosting Machines, tailored handle the high dimensinnality and sparsity genomic data; (3) correlating identified genetic markers with systematic errors in astrometric meastrements using celestial mapping datasess, while addressing potential challenges related to data quality and availability; and (4) validating the findings through robust statistical methods, including valssvaaidation and permulation tests, ensure that the identilied genetic markers indeed linked to adaptive traits. The expected outcomes include the identification of key genetic markers that can predict adaptive responses, improved calibration techniques for celestial measurements, and comprehensive medel that enhances our understanding of the interplay hetween ecological factors and astronomical phenomenu. This research aims to contribute significantly both ecological and astronomical fields, providing novel perspective on the integration hiological and colestial systems.
Table 31: Case study on using RESEARCHTOWN to write interdisciplinary research papers combining Astronomy, Biology, and Criminology. Due to the significant conceptual gap between the three fields, the generated idea heavily leans on terminology accumulation.
45

---

RESEARCHTOWN: Simulator of Human Research Community
RESEARCHTOWN Input
Researchers
researcher 2 Astroomy researcher + x Criminology researcher x Biology researcher
MagicPIC; LSH Sampling for Ellicient LLM Generation
Do LLMss exhibit humam like response hiases? Case study in survey design A Survey of Large Language Models
WebArena: Realistic Web Environment Building Automomous Agents
Gaia Data Release 3: Summary of the content and survey properties
Validation of SDSS-derived kinematic models the Milky Way's 's disk and balo stars
Paper
The Simons Observatory: Science goals and forecasts
Estimating Waiting Distances Between Genealogy Changes under a Multi Species Extemsion of the Sequentially Markov Couleszent
The interplay between microbial communitics and soil properties efficiency
Ancient orogenic and monsoon driven assembly of the warld's richest temperate alpine
Collateral Consequences of Punishment: A Critical Review and Parh Forward
Criminal organizations exhibit hysteresis, resilience, and robustucss by balancing security and efficiency
RESFARCHTOWN Output
Question
Answer
the problem??
The specitic research question I address How does coded language in political discourse influence socictal biases, and bow can a Bayesian hierarchical model be empluyed to unalyze this effect while simultaneusly addressing observational hiases in white dwarf population studies?
Why is interesting and imporiant?
Addressing this question is crucial for understanding the dynamnics of political communication and its impact on public sentiment. By elucidating how specific linguistic patterns contribuee societal biases, this research inform ethical communication practices in digital platforms, fostering a more informed public discourse. Morcover, integrating this analysis with astronomical data enhances relevance of linguistic studics in broader scientific context, potentially leading to advancements in both social sciences and astronomy. The findings could stimulate further interdisciplinary research, encouraging scholars to explore connections between language, bias, and observational methodologies, ultimately contributing
comprehensive understanding of both social and celestial phenomena.
Why hard?
The complexity of this problem arises from the multifaccted nature of language and its impact socictal biases, which is often influencee by context, culture, and individual interpretation. A naive appreach might oversimplily the relationships hetween linguistic cues public sentiment, neglecting nuances communication ethics. Additionally, integrating photometric and astrometric data from white dwarf populations introduces significant technical challenges, including the need to harmonize disparate datasets and account for observational biases that may skew results. The theoretical framework required to simultancously analyze linguistic patterns and astronomical data is intricate, necessitating advanced statistical methods to casure robust conclusions.
Why hasn't 't been solved before?
Previous research has largely focused on either linguistic analysis of political discourse or astronomiaal studies of celestial objects, often in isolation. The gap understanding of how language affects socictal perceptions whike also failing to address bisses in istronomical observations. Barriers to solving this problem include the absence of interdisciplinary collaboration and the complexity of developing models that integrate diverse data types. My approuch differs from prior woek proposing Bayesian hierarchical model that simultancously tackles these issues, offering more holistic perspective on the interplay between langunge and bias, as well as improving the characterization of white dwarf properties.
lack of a unified framework that encompasses both domains, limiting the
What are the key components of my appresach and result?
My proposed methodology involves developing a Bayesian hierarchical model that utilizes the Silent Signals dataset, which various platforms, including social media and traditional
compriiees diverse range of political speeches and discourse media. This dataset has been curated to include over 10,000 instances of coded language, with a focus on linguistic patterss that statistically significant shaping public sentiment. The model will designed quantify the effects of coded language on societal biases while integrating photometric and astremetric data from white dwarf populations to address observational biaves astrononical surveys. The metrics for evaluation identified linguistic pattems and changes in public sentiment, alongside the accuracy of intrinsic property characterizations of white dwarfs. Expected ouicurnes include a clearer undersianding of the relationship between language and bias. practical guidelines for ethical communication, and improved methadologies for astronomical data analysis, ultimately hridging the gap between social sciences and astronomy. Additionally, will establish measurable benchmarks, such as a timeline of six months initial findings and year for compreheusive analysis, ensuring the project's rigor and accountability.
include the correlation wetween
Table 32: Case study on using RESEARCHTOWN to write interdisciplinary research papers combining LLM, Astronomy, Biology, and Criminology. Due to combining researchers and papers from too many diverse domains, the generated idea becomes an incoherent mix of terms without clear focus or practical direction.
46


Reviews:
[{"review_id": "Ev44qD6AGD", "reviewer": "Reviewer_YZ7h", "summary": "This research work starts from the idea that we can leverage LLMs to simulate human research communities and proposes ResearchTown, a multi-agent framework designed to model human research societies and behaviors. This work also introduces TextGNN to model various research activities, including paper reading, paper writing, and review writing. In addition, it develops ResearchBench, which uses a node-masking task to evaluate whether ResearchTown can successfully simulate the masked paper node by masking a paper within a graph.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful and constructive comments. We address each of your comments in detail.\n\n**[Input and output of ResearchTown]**\nPlease check the same tag under **Reviewer Cny3** for a detailed explanation.\n\n**[Creativity of ResearchTown's output]**\nLLMs have been shown capable of generating novel research ideas through large-scale human studies [1]. Multi-agent role-playing and discussion further enhance creativity [2] and originality [3]. Based on these studies, ResearchTown encourages exploration of interdisciplinary ideas via structured prompting and multi-agent role-play design [*Line414-426*]. Examples are shown on Page27–Page41. We design HighImpactPaperBench in Appendix C.3 [*Line837-853*] to show ResearchTown’s capacity to generate impactful research [*Line355-376*].\n\n**[Metric calculation in ResearchTown]**\nOur evaluation metrics for both paper and review writing are detailed in Appendix E [*Line886–930*]. To enable meaningful comparison, we standardize papers into a condensed 5Q format [*Line893–895*] and reviews into a bullet point format [*Line909–911*]. As described in Appendix C.3 [*Line837–853*], we also convert real-world papers and reviews into these aligned formats using strong LLMs. We then compute cosine similarities with `voyage-3`,`text-embedding-3-large` and `nv-embed-v2`, based on Equation 21 for papers [*Line898–901*] and Equation 22 for reviews [*Line912–915*]. Detailed sub-part similarity scores are provided in Table 3 [*Line935–953*].\n\n**[Fine-grained evaluation with LLM and human]**\nWe extend beyond embedding-based metrics using prompting-based GPT-4o evaluations, covering factual consistency, logical/method alignment, motivation, and context. Each is scored from 1–10. \n\n|            | Semantic Similarity | Factual Consistency | Motivation Alignment | Method Alignment | Logical Consistency | Application Context Consistency |\n| ---------- | ------------------- | ------------------- | -------------------- | ---------------- | ------------------- | ------------------------------- |\n| Self-Agg   | 1.22                | 1.49                | 1.57                 | 1.23             | 1.22                | 1.41                            |\n| Agent-Agg  | 2.51                | 2.40                | 3.50                 | 2.41             | 2.22                | 3.08                            |\n| Data-Agg   | 3.94                | 3.48                | 5.05                 | 3.68             | 3.27                | 4.97                            |\n| Global-Agg | 4.43                | 3.94                | 5.56                 | 4.32             | 3.69                | 5.33                            |\n\nThese results show clearer differences than embedding-based scores, with Global-Agg (paper nodes + agents) performing best on motivation/method alignment. A human study over 40 papers on similarity-based evaluation (20 in-domain, 20 cross-domain) yields Pearson’s r = 0.745 and Spearman’s ρ = 0.735, supporting the validity of embedding/LLM-based metrics.\n\n**[Novelty+feasibility evaluation with LLM and human]**\nWe conducted small-scale human and LLM evaluations (20 interdisciplinary and 20 ML papers) on novelty and feasibility. Interdisciplinary examples include papers tagged with multiple fields on arXiv (e.g., CS+Economics). Each cell below includes *real-world* data evaluation vs *simulated* results evaluation.\n\n|                                   | LLM-Eval Novelty | LLM-Eval Feasibility | Human-Eval Novelty | Human-Eval Feasibility |\n| --------------------------------- | ---------------- | -------------------- | ------------------ | ---------------------- |\n| Interdisciplinary Research Papers | 7.5 vs. 7.35     | 6.45 vs. 6.9         | 7.4 vs. 7          | 7.4 vs. 7              |\n| ML Research Papers                | 7.85 vs. 7.65    | 6 vs. 6.9            | 6.6 vs. 6.95       | 5.65 vs. 7.15          |\n\nLLM vs human novelty/feasibility scores show a moderate Pearson correlation (~0.38), highlighting evaluation difficulty. ResearchTown’s outputs are generally comparable in novelty and more feasible than real papers in the ML domain.\n\n**[Text form defined in TextGNN]**\nEach node’s \"hidden state\" in TextGNN represents a condensed form of a paper or review. Initially, full paper contents serve as node states [*Line158-161*]. After iterative message passing, paper nodes adopt the standardized 5Q format [*Line893-894*], condensing information for easier evaluation. Review nodes similarly use bullet points [*Line910*] as condensed information.\n\n[1] Si, et al. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers.\n\n[2] Lu, et al. LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play.\n\n[3] Zhao, et al. Assessing and Understanding Creativity in Large Language Models.\n\n[4] Lu, et al. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.", "claims_and_evidence": "No. This paper claims that the simulation of automated research processes should be correlated with human research processes. However, under this assumption, it seems that discovering groundbreaking and insightful scientific ideas may become more challenging. The multi-agent system tends to produce known scientific knowledge while sacrificing the ability to learn, explore, and discover.", "methods_and_evaluation_criteria": "Some specific implementation details are unclear:\n\n- 1) How does ResearchTown generate a complete research paper including many parts of abstract, method, and experimental results?\n\n- 2) Unlike standard GNNs, each node in TextGNN is based on the text space, but what exactly constitutes the text? Is it the entire paper or a condensed summary of the paper?", "theoretical_claims": "This work does not involve theoretical derivations.", "experimental_designs_or_analyses": "The specific calculation methods for the metrics used in the experimental section are not very clear. For example, in Table 1, were text-embedding-large-3 and voyage-3 used to extract embeddings? Was cosine similarity used to compute the scores?", "relation_to_broader_scientific_literature": "ResearchTown conducts experimental validation to demonstrate its alignment with human research communities. However, how can we verify that the ideas generated by ResearchTown are valuable and can be followed? Moreover, this work has the potential to advance automatic scientific discovery, but it requires more robust evaluation and validation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "[Strengths]\n\n- 1) This work models human research communities by constructing TextGNN and attempts to use a masking-node approach for evaluation, which is insightful.\n\n- 2) This work constructs ResearchBench, a benchmark consisting of 1,000 paper-writing tasks and 200 review comments.\n\n[Weaknesses]\n\n- 1) This paper claims that the simulation of automated research processes should be correlated with human research processes. However, under this assumption, it seems that discovering groundbreaking and insightful scientific ideas may become more challenging. The multi-agent system tends to produce known scientific knowledge while sacrificing the ability to learn, explore, and discover.\n\n- 2) ResearchTown conducts experimental validation to demonstrate its alignment with human research communities. However, how can we further verify that the ideas generated by ResearchTown are valuable and can be followed? Moreover, this work has the potential to advance automatic scientific discovery, but it requires more robust evaluation and validation.\n\n- 3) How does ResearchTown generate a complete research paper including many parts of abstract, method, and experimental results?\n\n- 4) Unlike standard GNNs, each node in TextGNN is based on the text space, but what exactly constitutes the text? Is it the entire paper or a condensed summary of the paper?\n\n- 5) The specific calculation methods for the metrics used in the experimental section are not very clear. For example, in Table 1, were text-embedding-large-3 and voyage-3 used to extract embeddings? Was cosine similarity used to compute the scores?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to Part of [Other Strengths And Weaknesses]", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WOU367ugpt", "reviewer": "Reviewer_Cny3", "summary": "This paper aims to simulate the human research community (called ResearchTown), which is modeled as a graph structure, where researchers and papers are represented as nodes and they are connected based on their relationships. Also, each researcher over the graph structure is powered by Large Language Models (LLMs), making the simulation of the research community under the framework of multi-agent (or multi-LLM) collaboration. For evaluation, the authors propose two tasks: paper writing and review writing based on the process of paper reading, and handle them with text-based message passing over the graph structure. The authors then show that the proposed ResearchTown can not only provide a realistic simulation of the research activity (e.g., paper writing and review writing) but also facilitate interdisciplinary research.\n\n---\n\n### Update after rebuttal:\n\nThank you so much for your response, which addresses my last concern on the scalability of the TextGNN. In my view, I still believe that the design of TextGNN has a clear latency issue compared to the typical GNN that plays over the embedding space (which I hope the authors would discuss in the updated version); however, I also see some benefits of propagating and aggregating information of nodes via natural language texts (in terms of their effectiveness when working with LLMs and their interpretability). I will raise my score from 2 to 4 (accept). Good luck!", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful and constructive comments. We address each of your comments in detail.\n\n**[Novelty+feasibility evaluation with LLM and human]** Please check the same tag under **Reviewer YZ7h** for novelty and feasibility evaluation conducted by both LLMs and humans.\n\n**[Fine-grained evaluation with LLM and human]** Please check the same tag under **Reviewer YZ7h** for factual consistency and other fine-grained metrics.\n\n**[Ablation study on LLMs and graphs]** Please check the same tag under **Reviewer 5ZWh** for experimental results with three different LLMs.\n\n**[Review writing performance analysis]** For clarification of [*Line359-361*], please check the same tag under **Reviewer t7zy** for more analysis about why the performance drop.\n\n**[Input and output of ResearchTown]** ResearchTown processes either full papers or abstracts, depending on the task, and outputs standardized formats to support consistent and scalable evaluation. For paper writing, it generates a condensed 5Q format [*Line893–894*]; for review writing, it produces bullet-point strengths and weaknesses [*Line910*]. Prompt templates are shown in Table12–13 [*Line1379–1425*], with examples in Table14–28 [*Line1432–2253*]. As described in Appendix C.3 [*Line837–853*], this alignment reduces evaluation complexity and enables sub-part similarity scoring [*Line898–902*]. Input sources vary: only the paper’s abstract is used during reading [*Line994–995*] and full papers are used for review writing [*Line1171–1172*]. Aggregation setting details are in [*Line688–726*].\n\n**[Future Application of ResearchTown]** In [*Line139–144*], any research-related content—e.g., images, codebases, models, or social media posts—can be represented as nodes in the agent-data graph, with edge types like “cite paper,” “release model,” or “comment on X post” (examples in Figure 1) defining interactions. By specifying appropriate edge types and agent functions (`f_u` in [*Line134–135*]), the framework can be extended to simulate tasks such as code writing, model release, panel discussions, or lectures. While we focus on paper and review writing due to their importance, available real-world data, and simplicity, the framework supports broader applications.\n\nAdditionally, ResearchTown can be extended to model social dynamics such as peer pressure, collaborations, and institutional roles via agent-agent relationship edges [*Line133–135*]. Our current implementation already includes role-based dynamics (e.g., leader vs. participant), and we plan to support richer simulations of institutional and reputational factors in future work.\n\n**[Cost and scalability of ResearchTown]** TextGNN’s complexity scales linearly with the number of layers under standard GNN inference with full-batch inference. Our implementation uses GraphSAGE-style [1] to support per-paper evaluation, which is slightly less efficient but more practical. In line with findings from models like GraphSAGE, we observe that a 2-layer TextGNN is both robust and sufficient, making the cost affordable and controllable. Instead of deepening the GNN layer, we increase the number of transformation steps within each message-passing layer—interpreted as more agentic conversations—making it another scalable and effective approach. To demonstrate intra-layer scalability, we extend the AGG-agent setting [*Line297–298*] using the SWARM framework for multi-turn agent interactions, which boosts similarity scores from 52.32 to 57.68. These results show that agentic iteration within layers offers a practical and scalable alternative.\n\n**[Details on reviewer selection]** To simulate realistic reviewer assignment, we collect over 3,000 unique authors from the author list of PaperBench dataset and generate profiles by summarizing their recent publications. Using the `voyage-3` API, we embed each profile with the target paper’s abstract and select the top 5 most similar researchers, excluding the original authors. This method enables high-quality reviewer-paper matching—for example, a social learning paper was matched with reviewers experienced in social science and LLMs.\n\n**[Details on review writing inputs]** To clarify [*Line273–277*], in the review writing simulation (Algorithm 1 [*Line220–236*]), both the paper and review are outputs of the ResearchTown pipeline. While one option is to evaluate reviews based on generated papers, this introduces compounding errors if the paper is inaccurate. Instead, as noted in [*Line273*], we use the ground-truth paper as input to isolate and more reliably evaluate the review writing stage.\n\n**[Missing related work]** Thank you for highlighting valuable related works—we will include and discuss them in the revised version around [*Line120–125*].\n\n[1] Hamilton et al. Inductive Representation Learning on Large Graphs", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "While the proposed (graph-structure-based) approach to simulate the human research community is reasonable, I have an important concern about the evaluation criteria. First of all, as an evaluation metric, the authors use the embedding-level similarity between the generated item (such as the paper or review) and the target item; however, I believe one single embedding-level similarity may be suboptimal to measure the alignment between them. In other words, there are many aspects that should be considered when validating whether the generated paper (for example) is similar to the target paper (such as factual consistency, logical coherence, methodological relevance, or novelty), and embedding-based metrics often fail to account for finer-grained structural and conceptual differences.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Most of the experiments are sound and valid. However, I view validating the proposed ResearchTown with one single (proprietary) LLM as a clear weakness of the current experimental setting. Specifically, the proprietary LLM is usually not reproducible even if we control the temperature value, and also it is questionable whether the proposed framework can work with other LLMs (such as larger or smaller than GPT-4o-mini).", "relation_to_broader_scientific_literature": "The key contribution of this paper is related to the recent effort to simulate or automate AI research, which is a very important and timely topic.", "essential_references_not_discussed": "There are some papers [A, B] that aim to automate the research process with multi-agent collaboration frameworks (similar to the concept of the ResearchTown that aims to simulate the research process with multi-agents), and it may be worth discussing them.\n\n[A] ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models\n\n[B] Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents", "other_strengths_and_weaknesses": "While the authors claim that the proposed ResearchTown is designed to simulate the human research communities with LLMs, the current set of benchmark evaluations is limited to paper writing and review writing, done based on the paper reading among researchers. I believe there are more activities (that can be measured) within ResearchTown with more types of nodes and extra interactions between them, and it would be valuable if the authors would discuss them. Also, the proposed TextGNN (i.e., the message passing framework between nodes with text) seems not scalable when the number of layers becomes moderate (e.g., three or four), due to the nature of text-based communication in contrast to embedding-based communication where the information across nodes can be aggregated more efficiently over the vector space. I view this as another limitation of the proposed approach, and perhaps showing some experiments on the scalability of the proposed approach according to the number of layers may be beneficial.", "other_comments_or_suggestions": "Some sentences are not clear and it would be worth clarifying them:\n* Could you clarify the sentence in Lines 273 - 277 (starting with \"More specifically\")? \n* In Lines 359 - 361, it is not clear why the inclusion of both the agent and data nodes would degrade the performance of the review writing simulation task. \n* In Lines 317 - 318, how to select the top 5 researchers most related to the paper?\n* Overall, I feel it would be worth including the inputs and outputs for each task. For example, in paper reading and subsequent paper writing, do you use the full text of the neighboring papers? For the review writing task, what are the targets that should be generated (i.e., they are the strengths and weaknesses of the paper)?", "questions_for_authors": "Please see my previous comments. I feel the core idea of this paper is interesting, I would like to increase my rating if the authors would address them.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M6Ykj3Yv5g", "reviewer": "Reviewer_5ZWh", "summary": "The paper introduces RESEARCHTOWN, a multi-agent framework for simulating human research communities using Large Language Models (LLMs). The key idea is to model the research community as an agent-data graph, where researchers (agent nodes) and papers (data nodes) interact through edges representing authoring, citations, and reviews. The authors propose TextGNN, a text-based message-passing mechanism that borrows concepts from Graph Neural Networks (GNNs), treating LLM-powered functions as GNN blocks to unify research activities (e.g., paper writing, reviewing) as operations on this graph. In addition, the paper proposes the RESEARCHBENCH benchmark, which evaluates the performance of RESEARCHTOWN by comparing the similarity between the generated papers and real papers. Experiments on the RESEARCHBENCH benchmark show that RESEARCHTOWN can provide a realistic simulation of collaborative research activities, with multi-agent setups outperforming single-agent configurations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful and constructive comments. We address each of your comments in detail.\n\n**[Novelty+feasibility evaluation with LLM and human]** Please check the same tag under **Reviewer YZ7h** fo novelty and feasibility evaluation.\n\n**[Fine-grained evaluation with LLM and human]** Please check the same tag under **Reviewer YZ7h** for fine-grained consistency evaluation.\n\n**[Cost and scalability for ResearchTown]** Please check the same tag under **Reviewer Cny3** for analyzing the computational cost.\n\n**[Future Application of ResearchTown]** Please check the same tag under **Reviewer Cny3** for analyzing social dynamics simulation.\n\n**[Baseline score of realistic simulation]** To check whether ResearchTown provides realistic simulation, we benchmark similarity in real-world research activity. For paper writing, we reference two concurrent papers [1,2] recognized for presenting nearly identical ideas—yet with different writing styles and experiments—which yield a VoyageAI similarity of 0.8244. This suggests that scores above 0.82 can potentially indicate strong idea overlap. For review writing, we analyze data of reviewers evaluating the same paper. The average inter-reviewer similarity is 0.5900 (strengths) and 0.5904 (weaknesses), reflecting natural variance in human judgment. These inter-similarity score in the real world confirm that ResearchTown’s similarity scores represent realistic simulation.\n\n**[Ablation study on LLMs and graphs]** For graph structure ablation, *Table 1-2* already demonstrates the effect of different types of neighboring nodes during aggregation with different sub-parts of the neighborhood. It can be considered as ablation on graph structures.\n\nAdditionally, we provide results on **LLM ability variation** using *Deepseek-V3* (potentially larger than GPT-4o-mini) and *Qwen2.5-7B-Instruct* (potentially smaller than GPT-4o-mini) on a sample of 100 examples each from PaperBench and ReviewBench.\n\nFor paper writing tasks, we find when given the same aggregation setting, the performance improves when the models are larger: Qwen2.5-7B-Instruct < GPT-4o-mini < Deepseek-V3 (evaluated by openai embedding).\n\n| Aggregation Setting | Qwen2.5-7B-Instruct | GPT-4o-mini | Deepseek-V3 |\n| ------------------- | ------------------- | ----------- | ----------- |\n| Global-agg          | 71.94               | 73.93       | **74.09**   |\n\nFor review writing tasks, we test the performance of different models under different aggregation settings. We observe that Deepseek-V3 benefits more from multi-agent + multi-paper settings; GPT-4o-mini tends to be stricter than human reviewers, especially when more context is available.\n\n| Aggregation Setting | Model       | Strength | Weakness | Avg Δs (Abs) |\n| ------------------- | ----------- | -------- | -------- | ------------ |\n| Data-agg            | GPT-4o-mini | 71.08    | 68.22    | 1.17         |\n| Global-agg          | GPT-4o-mini | 62.40    | 56.79    | 1.49         |\n| Data-agg            | Deepseek-V3 | 68.63    | 67.76    | 1.08         |\n| Global-agg          | Deepseek-V3 | 68.04    | 67.99    | **0.74**     |\n\n**[More baseline comparison]** The TextGNN framework in our work is a general-purpose multi-agent simulation tool where existing LLM-based frameworks can be used to define the message-passing mechanism. Beyond our default setup, we conducted a small-scale experiment extending the AGG-agent setting [*Line297–298*] into a **multi-turn conversation** using the SWARM framework. This mimics multiple iterations within a single GNN layer and improves similarity scores from 52.32 to 57.68, evaluated using `text-embedding-large-3`. We also compare ResearchTown with the Sakana AI scientist framework, which uses five rounds of single-agent reflection. On the same subset of paper writing tasks, Sakana **AI Scientist** achieves a score of 0.63, while ResearchTown reaches 0.66 using multi-agent simulation.\n\n**[Running example]** *Table14-28 (Line1430-2254)* includes multiple running examples from ResearchTown: (1) Table14–17: Paper writing examples; (2) Table18: Review writing examples; (3) Table19–28: Interdisciplinary research examples. We will add more *end-to-end running examples*, beginning from citations and ending with paper + review outputs, in the revised version.\n\n**[Downstream task validation of ResearchTown]** We would include more downstream tasks like citation prediction in the modified version of our paper. Based on our HighImpactPaperBench (details in Appendix C.3, Line837-853), we observe that highly cited novel papers like GAN and VAE are harder to simulate the thinking process based on citation and multi-agent, indicating some kind of correlation between citation impact and simulation tasks.\n\n[1] Chen et al. ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning\n\n[2] Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning", "claims_and_evidence": "The claims are generally supported by evidence, but some require deeper scrutiny:  \n1. Realistic simulation of collaborative research: Supported by node-masking prediction results (similarity scores), but it's unclear what level of similarity score sufficiently demonstrates alignment with the claim of realistic simulation.\n2. Inspiring interdisciplinary idea generation: While the paper provides examples (e.g., NLP + criminology), no quantitative evidence or human evaluation validates their novelty or feasibility.", "methods_and_evaluation_criteria": "- Agent-data graph + TextGNN: The framework is innovative, simple and well-suited for modeling dynamic research communities. The use of LLMs as agent functions for text-based message passing is a creative adaptation of GNNs.  \n- RESEARCHBENCH: The node-masking task is a reasonable proxy for evaluating reconstruction fidelity, but it focuses on similarity rather than research quality (e.g., novelty, feasibility). Human evaluation or downstream task validation (e.g., citation impact simulation) could strengthen the assessment.", "theoretical_claims": "N/A. The paper does not present formal theoretical proofs.", "experimental_designs_or_analyses": "- Strengths: The experiments evaluate the framework's performance using well-justified tasks (masked-node reconstruction). Furthermore, the authors conduct systematic ablation studies to compare different framework configurations.\n\n- Weaknesses:  \n  - No comparison with existing LLM frameworks.  \n  - No ablation studies to isolate the contribution of the graph structure vs. LLM capabilities.  \n  - Similarity scores (via text embeddings) may not capture practical utility.", "relation_to_broader_scientific_literature": "RESEARCHTOWN builds on:  \n1. LLM-driven research automation (e.g., The AI Scientist) but emphasizes multi-agent collaboration in simulated research communities. \n2. Multi-agent LLM systems for social simulation (e.g., Generative Agents, S^3, SOTOPIA) but extends them to research communities framework.  \n3. Graph-based research modeling (citation networks, academic social networks) but shifts focus from analysis to dynamic simulation.", "essential_references_not_discussed": "I have no concerns regarding essential references that were not discussed.", "other_strengths_and_weaknesses": "- Other Strengths:\n  - The modular design allows for flexible expansion with new node types (e.g., code repositories, blogs) and edge types (e.g., commits, participations), offering the possibility to simulate complex academic ecosystems.\n  - The maintenance of hidden states in the text space (TextGNN) effectively preserves the semantic coherence of research outputs.\n\n- Other Weaknesses:\n  - The social dynamics in simulated academic communities are not considered.\n  - The computational cost of text generation may limit the practicality of large-scale community simulations.\n  - Not tested the dynamic community evolution capabilities.", "other_comments_or_suggestions": "- Include a running example of a simulated research workflow to improve readability.", "questions_for_authors": "1. Could you provide quantitative evidence (e.g., human evaluation scores) for the interdisciplinary idea generation claims? This would help validate if the generated ideas are truly novel and feasible beyond surface-level text similarity metrics. \n\n2. How does RESEARCHTOWN compare against existing LLM frameworks for idea generation? Comparative results would help position the framework's unique contributions.\n\n3. The similarity scores (0.67 paper/0.49 review) are presented as evidence of realistic simulation - what is the baseline/expected score range for human-written content? Contextualizing these metrics would strengthen their interpretation.\n\n4. Have you considered modeling social dynamics (e.g., senior-junior researcher interactions, institutional affiliations) in the community graph?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "taJiKBRbGj", "reviewer": "Reviewer_t7zy", "summary": "The paper proposes ResearchTown, a multi-agent simulation framework for research community simulation. The research community is simplified as an agent-data graph, where researchers are modeled as agent nodes and research outputs (such as papers and reviews) as data nodes. The interactions, including paper reading, paper writing, and review writing, are modeled through a unified text-based message-passing framework named TextGNN. The main contributions claimed are: (1) a realistic simulation of collaborative research activities, (2) robustness in simulating complex multi-researcher and multi-paper interactions, and (3) the ability to inspire interdisciplinary research ideas. The authors validate the framework on ResearchBench, a benchmark evaluating ResearchTown via masked-node prediction tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful and constructive comments. We ddress each of your comments in detail.\n\n**[Fine-grained evaluation with LLM and human]** Please check the same tag under **Reviewer YZ7h** for more human and LLM eval.\n\n**[Novelty+feasibility evaluation with LLM and human]** Please check the same tag under **Reviewer YZ7h** for more human and LLM eval.\n\n**[More baseline comparison]** Please check the same tag under **Reviewer 5ZWh** for baseline comparison.\n\n**[Data leakage concern]** For our main results (Table 1–2) and ablation studies (Figures 3–5), as noted in *Appendix C.2 (Line 813–815)*, we use NeurIPS 2024 and ICLR 2024 papers, which are *post-dated beyond GPT-4o-mini's* October 2023 knowledge cutoff. Thus, *data leakage is not a concern*. We also mask the full text during the simulation to avoid accidental exposure. \n\nFor *HighImpactPaperBench* (Appendix C.3, Line 840–843), we use high-impact papers from the past decade as an *extreme-case test* for idea simulation. While some may exist in the LLM’s training data, this benchmark is separate from our main results and serves to explore how LLMs handle well-known concepts.\n\nOur similarity analysis shows that 55% of generated papers score between 0.65–0.75, and 18% exceed 0.75, indicating moderate to high alignment. Only 1% score below 0.45. These scores are *comparable to PaperBench* (Table 1), suggesting no abnormal inflation. Even famous papers like *VAE, GAN, LayerNorm* do not receive notably high scores, implying that *semantic similarity—not memorization based on citation relationships—drives the results*, especially for tool/benchmark papers, which naturally resemble their references more.\n\n**[Review writing performance analysis]** For the behavior behind Global-Agg of review writing results, we conduct fine-grained analysis on the difference between predicted score $S$ and real-world score $S^*$ on a subset of review writing tasks. The results show that *GPT-4o-mini consistently assigns lower scores than the real-world reviewers*, especially in the *Global-agg* setting, where the mean of (S - S*) is -1.47. In contrast, *Deepseek-v3* does not show this consistent bias, indicating a better performance on review writing.\n\n| Experimental Setting     | Mean of \\|S - S*\\| | Mean of (S - S*) | Std of S |\n| :----------------------- | ------------------ | ---------------- | -------- |\n| Global-agg (GPT-4o-mini) | 1.49               | **-1.47**        | 0.80     |\n| Global-agg (deepseek-v3) | 0.74               | **-0.02**        | 0.91     |\n\nQualitatively, Global-agg reviews with GPT-4o-mini tend to provide *more specific and critical assessments*, particularly highlighting weaknesses related to *novelty* and *experiment* that are often missed in Self-agg settings, resulting in lower scores.\n\nExample 1: Global-agg provides a more detailed description of novelty concern and make the score lower.\n\n- *Global-agg*: *\"The proposed method does not present a sufficiently innovative approach compared to existing frameworks. Many cited works, such as P2B and PTTR, already address similar challenges, and the submission fails to articulate clearly how it advances the state of the art.”* → Score: *4*\n- *Self-agg*: *“The novelty of the proposed methods is not clearly articulated. The paper does not convincingly demonstrate how the approach differs from existing methods or why it is a significant advancement in the field.”* → Score: *5*\n\nExample 2: Global-agg notices more weaknesses compared with self-agg.\n\n- *Global-agg*: *“Experimental Support: The experimental results presented are not robust. For instance, the claim of maintaining output quality is not backed by sufficient statistical analysis or metrics, making it difficult to assess the validity of the findings.”* --> Score: 4\n- *Self-agg*: Not mention this weakness. --> Score: 6\n\n**[Paper Reading as inserting new node]** As described in *Algorithm 1 (Line 220–237)*, the input to the simulation is the paper content itself. To initialize the agent for simulation, we first perform a *“paper reading”* step, which sets up the agent profile. This is implemented in *Line 6* of Algorithm 1. We interpret this as a *form of agent node insertion*—specifically, initializing the *text attributes* of an agent node based on an external paper. Thus, although it may not be an insertion in the structural sense, it serves the role of *initializing a new agent node* in the simulation graph.\n\n**[Math notation]** Thank you for pointing this out. Our intention was to provide a formal definition analogous to message-passing GNNs, such as in *Equation 3 (Line 145–148)*. However, due to the *heterogeneous nature* of our agent–data graph, we define *different aggregation functions* depending on the node types. We acknowledge that this may introduce complexity in notation. We will *simplify and clarify* the mathematical presentation in the revised version of the paper to improve readability.", "claims_and_evidence": "1. Claim 1: The paper claims that ResearchTown “provides a realistic simulation of collaborative research activities, including paper writing and review writing.” This claim is partially supported by evidence from their node-masking experiments. For a large set of existing papers, the system attempts to regenerate each paper’s content given its authors and references; the similarity between the generated text and the actual paper is reasonably high. These results suggest that the simulated agents can often reproduce or predict key elements of actual papers and reviews. However, the evidence for “realism” relies entirely on embedding-based similarity metrics – no human evaluation. The similarity measurement cannot capture other important aspects in paper rewriting and reviewing, such as logical consistency.\n2. Claim 2: Robustness with Multiple Researchers and Diverse Papers. This one is supported by their ablation studies (Figure 4, 5). \n3. Claim 3 – Generating Interdisciplinary Research Ideas. The author provides some qualitative evidence that ResearchTown can brainstorm non-obvious research questions by bridging fields, which aligns with the claim. However, the support is still limited: no systematic evaluation of “idea novelty” or quality is done. To better support this claim, future work should include a more rigorous assessment of the novelty and usefulness of generated ideas – perhaps by soliciting evaluations from domain experts on a sample of cross-domain proposals, etc.", "methods_and_evaluation_criteria": "1. One concern to me is about data leakage. The author found that the ResearchTown framework performs notably better on impactful papers. This can be attributed to data leakage, i.e., the LLM may have seen the paper before during pretraining. The author tried to address the problem of information leakage by excluding any of the author's publications released after the target paper's publication. However, this is not enough, as the LLM can still recall details for released papers and reviewers, especially for those highly impactful papers. \n2.  The paper only utilizes the similarity as evaluation metrics. However, the similarity measurement is not sufficient and cannot capture other important aspects in paper rewriting and reviewing, such as logical consistency, originality, etc.\n3. Another concern lies in the absence of baseline comparisons beyond the ablations of their own approach. \n4. Lastly, this is essentially a prompting engineering paper without any training or optimization involved. This is fine to me, but the author framed their method in an overly complex way. For example, equation (4), (5) is over complicated and hard to follow.", "theoretical_claims": "The paper does not present explicit theoretical proof.", "experimental_designs_or_analyses": "See the section of Methods and Evaluation Criteria.", "relation_to_broader_scientific_literature": "The paper builds on prior work in multi-agent LLM frameworks, graph-based modeling of research communities, and text-attributed graphs (TAGs).", "essential_references_not_discussed": "--", "other_strengths_and_weaknesses": "Strengths:\n1. Novel framework combining multi-agent LLMs and graphs for research community simulation.\n2. RESEARCHTOWN can maintain robust simulation with multiple researchers and diverse papers.\n\nWeakness:\n1. The evaluation metric used, semantic similarity, is insufficient and may not fully capture novelty or logical consistency. Adding additional metrics or incorporating human expert evaluation could strengthen validation.\n2. The authors state: \"To prevent information leakage when simulating paper writing and review scenarios, we exclude any of the author’s publications released after the target paper’s publication year.\" However, I believe this measure is insufficient to prevent data leakage, as the LLM may have already been pretrained on these papers, including the target paper itself—especially for high-impact papers. The authors also observe that ResearchTown tends to achieve higher performance on high-impact papers, which could potentially be attributed to data leverage and LLM memorization, as these papers are more frequently included in the pretraining corpus.\n3. A related concern is that the trends in Figures (4) and (5) could also be attributed to data leakage. Moreover, increasing the number of agents may further increase the likelihood that LLMs recall information from the pretraining corpus related to the target paper.\n4. Given that this is essentially a prompting-based paper, the mathematical notation used may be somewhat confusing, making the paper more complex than necessary. In particular, Equations (4) and (5) are difficult to understand. I suggest simplifying the mathematical annotations and formulations to enhance clarity.", "other_comments_or_suggestions": ".", "questions_for_authors": "Questions:\n1. Why can paper reading be described as ‘inserting a new agent node’?\n2. The framework performs significantly better on impactful papers that focus on analysis or tool development. Could this indicate data leakage? The LLM may have encountered the paper during pretraining.\n3. 'Combining agent and data aggregation leads to a decrease in score differences, possibly because the presence of related papers causes reviewers to apply stricter novelty judgments' -> Are there any qualitative examples to support this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors presents ResearchTown, a multi-agent simulation framework for modeling research communities through an agent-data graph and a text-based message-passing mechanism. The authors introduce ResearchBench to evaluate the quality of simulated papers and reviews using masked node prediction and embedding-based similarity metrics. The system supports tasks such as writing papers, reviewing papers, and generating interdisciplinary ideas. The authors run experiments showing alignment between simulated and real-world outputs.

The core idea is interesting, and the overall design of ResearchTown is interesting nd well-motivated. The modeling of research activities as structured message-passing over text nodes seem unique and the proposed 5Q decomposition for paper evaluation provides a meaningful lens into simulation quality. 

Reviewers generally appreciated the ambition of the work and the potential of the framework, though concerns remain around the limits of embedding-based evaluation, scalability of the TextGNN, and the robustness of the interdisciplinary generation claims. The authors provide detailed rebuttals addressing these points. While not all concerns are fully resolved the clarifications and empirical evidence strengthen the case for the paper.

On balance, I find the direction promising and the framework potentially impactful, though there is room for refinement in both evaluation and framing. I strongly recommend the authors take into account the points raised by the reviewers and by my notes above in their revisions.
Decision: Accept (poster)