CycleResearcher: Improving Automated Research via Automated Review

Authors: Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, Linyi Yang
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Large Language Models, Automation of Scientific Discovery, AI Scientist

Abstract:
The automation of scientific discovery has been a long-standing goal within the research community, driven by the potential to accelerate knowledge creation. While significant progress has been made using commercial large language models (LLMs) as research assistants or idea generators, the possibility of automating the entire research process with open-source LLMs remains largely unexplored. This paper explores the feasibility of using open-source post-trained LLMs as autonomous agents capable of performing the full cycle of automated research and review, from literature review and manuscript preparation to peer review and paper refinement. Our iterative preference training framework consists of CycleResearcher, which conducts research tasks, and CycleReviewer, which simulates the peer review process, providing iterative feedback via reinforcement learning. To train these models, we develop two new datasets, Review-5k and Research-14k, reflecting real-world machine learning research and peer review dynamics. Our results demonstrate that CycleReviewer achieves promising performance with a 26.89\% reduction in mean absolute error (MAE) compared to individual human reviewers in predicting paper scores, indicating the potential of LLMs to effectively assist expert-level research evaluation. In research, the papers generated by the CycleResearcher model achieved a score of 5.36 in simulated peer reviews, showing some competitiveness in terms of simulated review scores compared to the preprint level of 5.24 from human experts, while still having room for improvement compared to the accepted paper level of 5.69. This work represents a significant step toward fully automated scientific inquiry, providing ethical safeguards and exploring AI-driven research capabilities. The code, dataset and model weight are released at https://wengsyx.github.io/Researcher.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730432164469

thread_type:
official_review

<thread_object>
creator: reviewer_5wha

<content>
summary:
The authors introduce two core components: CycleResearcher, a policy model that autonomously performs research tasks, and CycleReviewer, a reward model that simulates the peer review process. Experimental results suggest that CycleReviewer can outperform individual human reviewers in scoring consistency, and CycleResearcher shows promise in generating research papers that approach the quality of human-written preprints.

strengths:
**Valuable Datasets**: The introduction of the Review-5k and Research-8k datasets could be highly beneficial to the research community. These datasets provide resources for training and evaluating models in academic paper generation and review, potentially fostering further advancements in automated research tools.

**Innovative Use of Preference Data**: Utilizing preference data to iteratively train the CycleResearcher model is an interesting approach. This method allows the model to improve over multiple iterations, aligning more closely with human standards through reinforcement learning.

**Ethical Safeguards**: The inclusion of a detection model to identify AI-generated papers addresses ethical concerns related to the misuse of automated research tools. By implementing such safeguards, the authors demonstrate a commitment to responsible AI deployment.

**Automation of the Research Lifecycle:** The paper attempts to automate the full research cycle, from idea generation to peer review and revision. This holistic approach is ambitious and, if successful, could significantly impact the efficiency of scientific research.

weaknesses:
**Quality of Generated Papers**: Upon examining the samples provided in the Appendix (Sections E.1 and E.2), it is evident that the generated papers contain hallucinations and inaccuracies. For instance, in the generated abstracts, there are claims of outperforming state-of-the-art methods without substantial evidence or appropriate citations. This raises concerns about the reliability of the CycleResearcher model in producing high-quality, factual research papers.

**Counterintuitive Results with Model Scaling**: In Table 3 (Section 4.2), the CycleResearcher-12B model achieves a higher acceptance rate than the larger 72B and 123B models. This is counterintuitive, as larger models typically perform better due to increased capacity. The paper does not provide sufficient analysis or explanations for this phenomenon, leaving readers questioning the scalability and efficacy of the approach.

**Insufficient Ethical Considerations**: While the authors mention the implementation of a detection tool for AI-generated papers, the paper lacks a deep exploration of the ethical implications of automating research. Issues such as accountability, potential misuse, and the impact on the scientific community are not thoroughly addressed. A dedicated discussion in the Ethics Considerations section would strengthen the paper.

questions:
**Explanation for Performance of Smaller Models**: In Table 3, why does the CycleResearcher-12B model receive the highest acceptance rate compared to the 72B and 123B models? This result is unexpected given that larger models generally have better performance. Could the authors provide an analysis of this outcome, possibly including case studies or error analysis to understand the limitations of larger models in this context?

**Evaluation Stability of CycleReviewer**: What is the temperature setting used for the CycleReviewer during evaluation? Additionally, have the authors experimented with running the CycleReviewer multiple times to assess the variability or deviation in the review scores and feedback? Understanding the stability and consistency of the CycleReviewer is important for gauging its reliability in the automated review process.

**Addressing Hallucinations in Generated Papers**: Given the observed hallucinations and inaccuracies in the sample generated papers (Appendix E), what strategies do the authors propose to mitigate these issues? Are there mechanisms in place to fact-check or verify the content produced by the CycleResearcher before it is submitted for automated review?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730574966807

thread_type:
official_review

<thread_object>
creator: reviewer_gavj

<content>
summary:
The paper introduces an iterative training framework for automatic generation and review of research papers using open-source LLMs. The core of their approach consists of two main components:
1. CycleResearcher: A policy model that generates the paper, prompted by abstracts of related work.
2. CycleReviewer: A reward model that writes several peer reviews and returns scores according to ICLR criteria.


The authors initialize these models by supervised fine-tuning on scraped conference papers and ICLR reviews. They then improve the CycleResearcher using reinforcement learning (specifically iterative Simple Preference Optimization, SimPO), using CycleReviewer as a reward model.


The paper claims three main contributions:
1. Development of an iterative reinforcement learning framework that mirrors the real-world research-review-revision cycle.
2. Creation of two new datasets: Review-5k. Research-8k
3. Empirical results showing:
   - CycleReviewer produces scores that are closer to averages of multiple human reviewers than scores by individual human reviewers
   - CycleResearcher-12B achieved paper quality scores surpassing preprint level and approaching accepted paper level

The paper implements some ethical safeguards: they train a model to detect papers generated by LLMs they publish; they promise to implement a licensing agreement such that downloading model weights requires sharing institutional affiliations and agreeing not to use models for official peer reviews or submissions without disclosure.

strengths:
* Training LLMs with reinforcement learning on parts of the AI research process is a novel and significant contribution.
* The paper includes numerous experiments and ablations. The overall methodology is sound (with exceptions, see weaknesses).
* The authors achieve strong results on the metrics they choose. It is somewhat impressive that their system achieved an acceptance rate of 31.07%, similar to ICLR 2024's acceptance rate.
* Authors use open-source models with a large range of scale (from 12B to 123B).

weaknesses:
* The writing is overclaiming the extent to which the paper covers the full research process. Authors write that the paper “explores performing the *full* cycle of automated research and review”, however the paper omits crucial part of the process: actually running experiments. Instead, the authors train models to write complete papers purely from abstracts of past work, with completely hallucinated experiment design and results.  
* I do not think that the task authors train models for — hallucinating experiment results and writing papers for them — is well motivated. Using models for this purpose will not contribute real knowledge to the scientific field. I think this is dual use technology, if not a completely malicious one. I could imagine the paper could be reframed to center on demonstrating this imminent failure of the reviewing system and raising an alarm, allowing the scientific community to adapt. In current form, the paper is probably net-negative.  
* Automated evaluation of papers produced by CycleResearcher is hard to trust, since CycleResearcher was trained with RL against the same reward model as used at test-time. Reward model overoptimization (Gao et al, 2022 \- [https://arxiv.org/abs/2210.10760](https://arxiv.org/abs/2210.10760)) should be the expected result of RL, however the authors do not run any experiments to investigate to which extent their evaluation is influenced by this. For example, the authors could train a held-out reward model on a held-out dataset of reviews and then evaluate CycleResearcher on both the reward model used for RL training and this new held-out reward model.  
* The claim that CycleResearcher surpasses the quality of preprint papers and approaches quality of accepted papers is not well supported, due to the concerns about reward model overoptimization mentioned above. Human reviewers rate CycleResearcher’s papers significantly lower (4.8) than the automated reviewer made by the authors (5.36). The authors could have reported the actual historical average score of ICLR2024 accepted papers.   
* I have a number of concerns about the human evaluation procedure.  
  * When the authors evaluate their CycleResearcher with the AI Scientist, they seem to only use rejection sampling (best of N) for CycleResearcher. This is not a fair comparison.  
  * Overall, human evaluation is conducted on a small scale (10 papers total, three human reviewers, 2 methods: this paper and baseline)  
  * I do not think 30min per review (including reading, writing comments & providing scores) is enough\!  
* I think it’s misleading to use the term “revision” for parameters updates of the policy model (Figure 2). The paper refers to this revision as part of the full research process (“Research-Rebuttal-Revision”) but this does not actually involve revision of papers based on reviews.

questions:
1. What exactly are the prompts, based on which CycleResearcher generates papers during evaluations?  
2. Why do smaller CycleResearcher models get better scores in the evaluation?  
3. How many samples in automated evaluation?  
4. Please include the average real score of accepted papers given by human ICLR2024 reviewers.  
5. How do you compute the acceptance rates, e.g. one mentioned in line 128?  
6. For human evaluation:  
   1. Please report the N used in best-of-N / rejection sampling.  
   2. Please clarify whether each paper is evaluated by one or several humans.  
   3. How are the human experts chosen?  
   4. What do you mean by saying “excluding formatting considerations” in the assessment, and why is it omitted?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730888904625

thread_type:
official_review

<thread_object>
creator: reviewer_czsx

<content>
summary:
The paper presents CycleResearcher and CycleReviewer, which is a cohesive system intended to make steps towards automatic scientific discovery. In particular the novelty of the approach lies in the encapsulation of the entire research pipeline from research to review, in order to better model the entire system of research generation and get better outcomes. The authors contribute two datasets for research and review, and use these to train the system of researcher and reviewer, and evaluate them using various methods and metrics.

strengths:
Originality: The idea to design both a researcher and a reviewer is novel and interesting. 

Quality: The usage of recent preference optimization methods is a nice technical plus. The work contributes datasets to the direction of scientific peer reviewing, which is a resource that is rather helpful for the field. RL details and how they fit in is nice. 

Clarity: Figures are well-designed and artistically pleasing. Appreciate the various different ways that are used to evaluate the methods (qualitative, ablations, etc.)

Significance: The automation of scientific research and reviewing is a very interesting and timely topic. In particular, due to the massive increase in submissions year-to-year, progress towards the paper's direction is well appreciated.

weaknesses:
Originality: N/A

Quality: 
One big issue of the paper is the method in which the authors obtain the "ground truth" review score: "for each submission, we use the average of the other n − 1 reviewers’ scores as an estimator of the true score."
In my opinion (and what feels like a general consensus in the community), it's pretty clear that this isn't the correct approach in determining a ground truth quality of a paper. Different reviewers have different expertises and opinions, and may disagree substantially based on their backgrounds, but this is a positive quality of peer review rather than a negative one. Thus, the metric used to judge the "loss" of a review score can be used to train proxies of reviewers, sure, but it does not make sense to then take the trained system and use the same metric to compare it against actual human reviewers. For instance, if human reviewers vary differently based on their perspectives and CycleReviewer is just doing some "hedge" where most scores are around the median score for all papers, it might achieve better than human performance on the MAE metric that is used in the evaluation. Furthermore, focusing on the score ignores perhaps the more important points of paper reviewing, such as being able to highlight errors in the paper or provide advice for making changes that are adopted in future versions. I think the true objective of reviewing in the paper's cycle paradigm matches these objectives more, although I recognize that they are even harder to quantify. Even so, I think the paper is overclaiming by saying that the lower MAE suggests that "LLMs can surpass expert-level performance in research evaluation".

And since I don't necessarily agree with the evaluation metric for the reviewer, this casts doubt on the results for the CycleResearcher because the CycleReviewer is reviewing the CycleResearcher. Also, since CycleResearcher is optimized on CycleReviewer, then saying that CycleResearcher does better on CycleReviewer than humans or AI scientist doesn't mean much. The qualitative study in section 4.3 is helpful to remove some of these doubts though. 

Clarity: There are a reasonable amount of typos and grammatical errors in the document. For instance, CycleReviewer is replaced with WhizeReviewer in Section 4.1. The title of section 4.1 should be "Experiments on Paper Review Generation", etc. The paper would benefit from a pass over to correct grammatical mistakes in general to make it easier to read. 

Significance: The claims are very catchy that the system can generate better reviews and papers than humans. However, given the questionable-ness of the metric, I think these are discounted to a degree.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731057790741

thread_type:
official_review

<thread_object>
creator: reviewer_7lzg

<content>
summary:
The paper explores the use of open-source large language models to automate the entire research process, from literature review and manuscript preparation to peer review and revision. The proposed framework includes CycleResearcher, which performs research tasks, and CycleReviewer, which simulates the peer review process. The study demonstrates that CycleReviewer can outperform human reviewers in predicting paper scores, and CycleResearcher can generate papers of quality close to human-written preprints. The models are trained using two new datasets, Review-5k and Research-8k, which capture the complexities of peer review and research paper generation. The results indicate that this approach can significantly enhance the efficiency and quality of scientific research, while also providing ethical safeguards to prevent misuse.

strengths:
The introduction of CycleResearcher and CycleReviewer models to automate the entire research process, including literature review, manuscript preparation, peer review, and revision, is highly innovative. This framework mimics the real-world research cycle, enhancing the efficiency and consistency of scientific inquiry.
Performance Improvement: The CycleReviewer model demonstrates a significant improvement in predicting paper scores, outperforming human reviewers by 26.89% in mean absolute error (MAE). This indicates that the model can provide more accurate and consistent evaluations than individual human reviewers.
Quality of Generated Papers: The CycleResearcher model generates papers with an average quality close to human-written preprints, achieving an acceptance rate of 31.07%. This shows that the model can produce high-quality research outputs that are competitive with human-generated content.
Large-Scale Datasets: The development of the Review-5k and Research-8k datasets, which capture the complexities of peer review and research paper generation, provides valuable resources for training and evaluating models in academic paper generation and review.

weaknesses:
Generalizability Across Domains: The models are primarily designed for machine learning-related research, and their generalizability to other scientific fields remains unexplored. This limitation suggests that the framework might not perform as well in domains outside of machine learning.
Reward Design: The paper highlights the issue of reward definition, where the policy model might exploit loopholes in the reward model to maximize rewards without genuinely improving the quality of the generated research. This behavior could undermine the long-term goal of producing high-quality research outputs.
Complexity of Implementation: Implementing the framework requires significant computational resources and expertise in reinforcement learning and LLMs. This complexity might be a barrier for widespread adoption, especially for smaller research teams or institutions with limited resources.

questions:
The framework is primarily designed for machine learning-related research. How do you envision adapting CycleResearcher and CycleReviewer to other scientific fields, such as biology or social sciences, where the nature of research and evaluation criteria might differ significantly?
The paper mentions the potential issue of reward hacking, where the policy model might exploit loopholes in the reward model. Could you elaborate on the specific strategies you are considering to mitigate this issue and ensure that the generated research outputs maintain high academic rigor and novelty?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732458874091

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses and Summary of Revisions

comment:
We sincerely thank all reviewers for their careful and constructive feedback, which has helped us significantly improve our work. We are encouraged by the reviewers' recognition of several key strengths:

- "The introduction of CycleResearcher and CycleReviewer,..., is highly innovative." (*Review 7LzG*)
- "The paper includes numerous experiments and ablations. The overall methodology is sound" (*Review GAvj*)
- "The paper includes numerous experiments and ablations" (*Review CzSX*)
- "The introduction of the Review-5k and Research-8k datasets could be highly beneficial to the research community" (*Review 5wHA*)


Based on your constructive comments, we have made substantial improvements to strengthen our paper's contributions and address the limitations. The major enhancements include:

1. **Automated Research Pipeline**: We implemented a complete automated scientific discovery pipeline with real experimental capabilities, including code generation and execution through GPT-o1-preview (The API call implementation code for specific experiments comes from AI Scientist and aidir. We have integrated these into one unified system to produce a paper with complete and real experimental results.). Full codebase and experiment logs are included in supplementary materials (Appendix G).

2. **Extended Dataset Coverage**: We created a new Researcher-14K dataset incorporating recent NeurIPS 2024, ACL and CVPR papers, enhancing model performance across scales. With this expansion, our 72B model achieved improved results (average score from 5.20 to 5.38, acceptance rate from 25.81% to 33.64%), demonstrating the value of comprehensive training data (Appendix F.1).

3. **Reward Model Validation**: We implemented an independent evaluation framework using a separate reward model trained on the Review-5k test set. Results show minimal performance differences (~0.14 in average score), validating our system's robustness beyond reward exploitation (Appendix F.2).

4. **Comprehensive Benchmarking**: We added complete ICLR 2024 benchmark comparisons:
   - ICLR 2024 Submissions: 5.54
   - ICLR 2024 Accepted: 6.44
   - CycleResearcher-12B: 5.36 (auto) / 4.8 (human)
   - AI Scientist: 4.31 (auto) / 3.6 (human)
This provides clear context for our model's current capabilities (Section 4.3).


5. **Enhanced Ethics Framework**: We significantly expanded our ethics considerations with:
   - Mandatory disclosure requirements
   - Institutional affiliation verification
   - Publisher verification license
   - 95%-accurate detection system
   - Comprehensive usage guidelines
   - Human-AI collaboration protocols
(Section 7: Ethics Statement, Appendix A)

6. **Evaluation Stability Analysis**: We conducted comprehensive stability testing for CycleReviewer:
   - Temperature setting: 0.4
   - Three independent evaluation runs
   - Consistent performance metrics (std < 1%)
This validates the reliability of our evaluation framework.

7. **Domain Adaptation Discussion**: We clarified our framework's current focus on machine learning while outlining concrete steps for cross-domain expansion through publisher collaborations and domain-specific training data collection (Appendix Limition).

8. **Terminology Refinement**: We updated our terminology from "revision" to "refinement" to more accurately reflect the model's learning process and avoid confusion with paper revisions (Throughout paper).

With the generous guidance from our esteemed reviewers, we have been able to substantially enhance our paper's technical depth, experimental rigor, and practical impact. We believe these comprehensive improvements address the key concerns raised while maintaining the innovative contributions of our work. We would be honored if the reviewers find these enhancements worthy of a more favorable evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733242434751

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Ethical Considerations and Mitigation Measures for CycleResearcher

comment:
On behalf of all authors, I would like to summarize and clarify the mitigation measures implemented in our work to address potential ethical concerns for all program committee members, ethics committee members, and potential users. All the content discussed has been included in the Ethics Section and Appendix A of our paper.

Our goal is to advance scientific research automation through LLMs. We believe that AI capabilities will continue to advance through the efforts of all researchers. Since 2022, LLMs have been widely adopted, comprehensively improving society's productivity levels. LLMs' capabilities have evolved from grammar correction, literature summarization, and code debugging to today's research assistance. As one of the pioneers in automated scientific research, CycleResearcher's primary purpose is to contribute to social and human welfare!

However, everything has its pros and cons. When malicious individuals misuse CycleResearcher, it may cause negative impacts to some extent. We have carefully considered a series of negative impacts and provided protective measures to mitigate them:

> **Safety and Potential Harmful Consequences**

Improper use of CycleResearcher may lead to negative impacts such as generating dangerous drug formulations or researching attack methods. **Before open-sourcing the weights, we implemented additional SafetyLock [1] security strategies to make the model reject such content and conducted additional red team exercises for testing**. Specifically, we tested all 11 safety categories from [2] and additionally tested experiments related to producing dangerous drugs and cyber attack methods. **We will only open-source it after ensuring the attack success rate is reduced to an acceptable range**. Meanwhile, we have also established strict usage guidelines that clearly limit and regulate the model's use scenarios.

> **Accountability and Authorship**

We provide three mitigation measures: watermarking, high-performance detection tools, and licensing.

**Watermarking**:
All versions of CycleResearcher we trained disclose the use of LLMs at the end of papers:

```
This paper was written with the assistance of CycleResearcher, including but not limited to the introduction, related work, experimental design, and experimental results sections. A portion of the content may have been generated using large language models (LLMs).
```

**High-Performance Detection Tools**:
In Sections 3.4 and 4.4, we implemented a highly available paper detection tool based on Fast-Detect-GPT [3], **maintaining close to 95% Accuracy and F1 Score for content generated by CycleReviewer and CycleResearcher**. It can successfully identify content generated by CycleReviewer and CycleResearcher in about 2 seconds with just 1 GPU, making widespread deployment in paper submission systems feasible. **This rapid detection mechanism, combined with our disclosure requirements, provides dual protection for maintaining academic integrity**.

**Licensing**:
For code, datasets, and model weights, we implement a comprehensive licensing and supervision system. Before accessing the model, **users must publicly disclose their institutional affiliations and explicitly state their intended use**. Additionally, our license agreement requires all users to agree that **publishers may submit queries to verify whether specific authors accessed our model within a given timeframe when there are reasonable concerns about potential misuse**. This verification process aims to maintain academic integrity while protecting user privacy by only confirming model access without revealing specific usage details.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors study the problem of automatically evaluating and writing scientific papers. They introduce 1) CycleReviewer, a model that takes in a paper and returns a score from 1-10 (trained to match the score a human peer reviewer would give), and 2) CycleResearcher, an automated paper writing model that trains against the reward provided by CycleReviewer.

This was an interesting but contentious paper. Reviewers all found the work intriguing, but raised questions about the ethical implications of the work, the overclaiming in the writing, as well as the rigor of the evaluations. The authors significantly revised their manuscript and experiments in response. These responses partially addressed the reviewers' concerns, but not fully. I thank the authors and reviewers for their efforts during the rebuttal phase.

After discussion, I recommend acceptance but with reservations. I **strongly** encourage the reviewers to incorporate the following feedback into their camera-ready version and to consider the ethics of releasing their model publicly.


1. The main claims of the paper -- that CycleResearcher produces papers that are comparable in quality to human-authored papers -- rely mainly on CycleReviewer scores. CycleReviewer is trained from historical data to mimic human reviewer scores. However, this historical data wholly consists of (one would hope) human-written papers, which are significantly out-of-distribution to the generated papers. For example, the initial version of this manuscript used LLMs to autoregressively generate all of the experimental results (which, as the reviewers pointed out, involves a lot of hallucination). Nevertheless, CycleReviewer gave those papers high scores. In human peer review, there is a certain benefit of the doubt given that the authors actually correctly did what they said they did; reviewers do not often, for instance, read through every line of the submitted code to ensure that it is correct. For generated papers, a much higher standard of scrutiny needs to be applied. Furthermore, since CycleResearcher is trained directly to optimize CycleReviewer scores, it is also not surprising that it does well by the metric of CycleReviewer scores. 

    In response to similar reviewer comments, the authors (a) prompted OpenAI o1 to generate and then execute code, instead of just making up experimental results; and (b) ran a n=3 human reviewer evaluation study. While (a) seems like a good step towards more reliable results, it still needs to be validated. There is no analysis of the code, and its correctness, provided in the paper. Likewise, there are few details provided on (b) -- for example, was the "author" of the papers blind to the reviewers? Were the reviewers able to pick up errors in the experiments/results? What validation was done to show that the human study was reliable? Both (a) and (b) need to be elaborated upon and made much stronger to convince readers that CycleResearcher can actually do end-to-end research.


2. I recognize that running a rigorous evaluation of the generated papers is difficult and could be the topic of a different project. Without introducing more experiments, I strongly recommend that the authors shift the framing of their paper to more accurately reflect their experiments. For example, these results present an interesting example of how one can generate papers that optimize for automated review metrics without necessarily having the results in those papers be anywhere correct. The current, already-toned-down claim that CycleResearcher generates papers "with an average quality level close to human-written preprints" is not substantiated by the current experiments.