Paper Copilot: Tracking the Evolution of Peer Review in AI Conferences
Authors: Jing Yang
Qiyao Wei
Jiaxin Pei
Venue: ICLR 2026 Poster (2026)
Keywords: peer review
review dynamic
aiml
community
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Submissions are rising fast, and venues use different rules, data formats, and update times. As a result, signals of progress get split across places, and key moments (rebuttal, discussion, final decision) are easy to miss, making analysis hard. We present Paper Copilot, a system and scalable peer-review archive that pulls data from official sites, OpenReview, and opt-in forms into a single, standardized, versioned record with timestamps. This lets us track trends over time and compare venues, institutions, and countries in a consistent way. Using the archive for ICLR 2024/2025, we see larger score changes after rebuttal for higher-tier papers, reviewer agreement that dips during active discussion and tightens by the end, and in 2025 a sharper, mean-score–driven assignment of tiers with lower decision uncertainty than expected at that scale. We also state simple rules for ethics—clear sourcing and consent, privacy protection, and limits on use for closed venues. Together, we provide a clear, reusable base for tracking AI/ML progress, and, with this data, enable validation, benchmarking, and otherwise hard-to-run studies.

Published as a conference paper at ICLR 2026
PAPER COPILOT: TRACKING THE EVOLUTION OF PEER REVIEW IN AI CONFERENCES
Jing Yang*
Qiyao Wei University of Cambridge
Jiaxin Pei
University of Southern Californi
Stanford University
ABSTRACT
The rapid growth of Al conferences is straining already fragile peer-review system, leading to heavy reviewer workloads, expertise mismatches, inconsistent evaluation standards, superficial or templated reviews, and limited accountability under compressed timelines. response, conference organizers have introduced new policies and interventions to preserve review standards. Yet these ad-hoc changes often create further concerns and confusion about the review process, leaving how papers are ultimately accepted-and how practices evolve across years-largely opaque. We present PAPER COPILOT, a system that creates durable digital archives of peer reviews across wide range of computer-science venues, an open dataset that enables researchers to study peer review at scale, and a large-scale empirical analysis of ICLR reviews spanning multiple years. By releasing both the infrastructure and the dataset, PAPER COPILOT supports reproducible research on the evolution of peer review. We hope these resources help the community track changes, diagnose failure modes, and inform evidence-base improvements toward a more robust, transparent, and reliable peer-review system.
INTRODUCTION
The rapid growth of submissions to top-tier Artificial Intelligence (AI) and Machine Learning (ML.) conferences- now exceeding 10,000 per venue annually (Yang) 2025 has placed unprecedented pressure peer review. To address issues of scale and transparency, many conferences have adopted open or semi-open platforms like OpenReview. Yet practices remain inconsistent: some venues publish reviews and scores, while others keep them private. This inconsistency has sparked ongoing debate around fairness, accountability, and the effectiveness of different review models (Cortes & Lawrence 2021).
Meanwhile, the nature of review data itself has evolved. Whereas reviews were initially limited to single numerical rating, nowadays reviews include multiple dimensions such as soundness, technical correctness, and presentation quality (Beygelzimer et al. [021]. Since the rise of Open- Review (Soergel et al. 2013) and especially following the widespread adoption of formal rebuttal phases, the need to analyze score dynamics over time has grown Yangg 2025). Yet community discourse remains scattered seittered-sprad across Twitter, Reddit, Zhihu, Xiaohongshu, and other social platforms in fragmented, venue-specific threads. As submission volumes and review complexity continue to increase, the absence of structured tools that unify and visualize the review data has become a bottleneck for both transparency and timely author decision-making.
From the standpoint of conference organizers, internal dashboards provide visibility into score trends and acceptance distributions (Cortes & Lawrence 2021). But these views are rarely made accessible to authors. During the critical post-review phase- often only one or two weeks long-authon are expected to analyze reviewer feedback, prepare rebuttals, and potentially decide whether withdraw or revise their submissions. With scattered resources, authors must resort to scraping statistics (Sun) 2020) or aggregating posts from scattered social media. This inefficiency hinders timely decision- making and may even lead to missed opportunities for meaningful rebuttal or clarification. A system that actively collects review statistics, visualizes score dynamics, and contextualizes submissions relative to their peers would therefore offer immediate utility to thousands of authors.
"Corresponding Author, email ji inngang.....mm carl wwrkk@gaii com

---

Published as a conference paper at ICLR 2026
In addition, while the OpenReview ecosystem has grown, many conferences still use closed-fome review systems and opt not to release reviews or scores publicly. Motivations for withholding reviews may include protection of intellectual content, reviewer anonymity, historical precedent. However, survey studies of peer review in MI have documented that such opacity exacerbates known problems: inconsistent reviewing standards, lack of calibration across reviewers, and limited accountability for low-quality or biased reviews (Shah) 2022). These shortcomings directly affect fairness and community trust, and they compound when authors have no visibility into aggregate statistics, A practical workaround is to allow voluntary, anonymized community submissions. If designed carefully, such a system can extract value from partially open data while respecting privacy and consent.
Beyond the review process itself, the AT/ / MI. community lacks robust infrastructure for tracking who is shaping the field over time. Traditional platforms like Google Scholar and DBLP focus primarily on paper-level or author-level
citttat,,ctations,, references, and publication profiles (Scholar) DBLP). While Google Scholar does display author affiliations (and, by extension, country information), these systems were not designed to support temporal analysis or dynamic tracking across institutions or regions. Their scope remains limited, offering little visibility into broader affiliation-, institution-, country-level dynamics. Tracking such information has historically been left to organizations due to the technical and logistical challenges involved, as seen in efforts like CSRankings (Berger) or university rankings.
As AI becomes increasingly high-profile and competitive, attention is increasingly concentrated around specific venue cycles. This shift highlights a growing need to understand who is actively driving progress over shorter timeframes: which institutions are rising, who remains active in the field, and what geographic regions are gaining influence. Yet no existing academic tool provides this kind of dynamic, multi-scal scale view. Ranking systems often update annually and draw from non-trans con-transparent data sources, making them ill-suited to capture the rapid and evolving dynamics of the field.
To address these needs and challenges, we introduce Paper Copilot, a system with open dataset for tracking peer review dynamics and talent trajectories across AI/ML. Our contributions include:
System for tracking AI/M progress. A venue-configurable pipeline (Fig. D that unifies multi-source inputs into standardized, versioned paperlists and powers interactive, multi- granularity analytics (vemue/insiiution/countyy for longitudinal progress tracking.
Scalable peer-review archive. A unified archive built from open, semi-open, and opt-in community data with temporal snapshots, capturing multi-dimennional review metadata and daily dynamics (ICLR 2024/2025), to preserve and analyze cross-venue review evolution at scale.
Findings ethical considerations. Empirical results show a 2025 shift toward sharper, score-driven tiering under volume pressure and characteristic rebuttal-phass dynamics (score shifts, consensus evolution); we articulate safeguards on sourcing/consent, privacy, misuse, and bias.
2 RELATED WORKS
2.1 PEER REVIEW DATASETS AND ANALYSIS
There a growing body of work on creating datasets to study academic peer review. PeerRead (Kang et al. 2018). collected 14.7K paper drafts with reviews and decisions from NLP/ / ML venues including ACL. NeurlPS and ICLR, enabling research on review-decision decision alignment and NLP applications like review score prediction. More recently, MOPRD (Lin et al. 2023) introduced multidisciplinary open peer review dataset spanning several journals and computer science conferences, capturing the full reviewing process (including review comments, rebuttals, meta-reviews and final decisions) across domains. Other efforts have targeted specific aspects: for example, ORB (Szumega et al. 2023) curated a list of more than 36,000 scientific papers with their more than 89,000 reviews and final decisions in high energy physics.
Bharti et al. (2022) leveraged NLP approach to estimate reviewer confidence from review text, illustrating analysis that can be done when reviews are available. Peer Review Analysis (Ghosal et al. 2022) provided 1,199 review reports labeled the sentence level for aspects like critique, suggestion,
2

---

Published as a conference paper at ICLR 2026
and sentiment, serving benchmark for automated review assessment. Ribeiro et al. (2021) 1) used a private dataset of two non-disclosed conferences to perform RSP. PDP. and sentiment analysis to extract review polarities. In general, these datasets have been invaluable in enabling studies review quality, bias, and consistency. However, each is often limited to a particular venue or a snapshot in time. We advance beyond prior datasets by unifying multiple venues over long temporal window, thereby supporting shift over years, or how an author's successive submissions fare over time). It also introduces the notion of tracking review process "dynamics,"" since our data include timestamps and sequences (allowing one reconstruct the timeline of reviews and discussions for open-review conferences). This complements earlier one-off studies that examined review consistency and randomness in single years (Beygelzimer et al. 2021] Cortes & Lawrence 202T), by providing a broader data foundation to study such effects in the aggregate.
cooss-confeeence and longitudinal analyses (e.g., how review score distributions
2.2 PEER REVIEW TRACKING SYSTEMS
Our work is related to platforms that manage or expose the peer review process. OpenReview.net the most prominent system enabling open peer review for many MI. conferences (ICLR, NeurlPS, UAI, etc.). providing public APIs to fetch submissions, reviews, and comments (OpenReview 2024). We extensively leverages ReenReview as data source, but whereas OpenReview is focused on facilitating the review process for active conferences, our system designed to track and analyze review outcomes across conferences and years. In some sense, Paper Copilot serves meta-layer on top of systems like OpenReview and other open-access portals and proceedings (Foundation [0244] Anthology), aggregating their outputs for comparative analysis. A few individual conferences have released summary statistics (e.g., acceptance rates, score distributions) in blog posts or slides, but these are ad-hoc and not standardized (Beygelzimer al. 202I). To our knowledge, there was no public web resource before Paper Copilot that allowed users to conveniently browse and compare peer review results (scores, decisions, etc.) for a wide range of AI ML venues in one place. Our system fills this gap by acting as centralized peer review statistics dashboard for the community. It somewhat analogous to "CS Conference Stats" websites that track acceptance rates (Berger), but we go further by including detailed review metrics and providing the underlying dataset.
2.3 AUTHOR PROFILING AND METASCIENCE
Studying author career trajectories and publication patterns has been a long-standing interest in bibliometrics (Fan et al. 2024, De Frutos Belizon et al.) 2025). Large-scale academic databases and knowledge graphs such as Google Scholar, Microsoft Academic Graph, Semantic Scholar's 's Open Research Corpus, and AMiner-haad enabled analyses of collaboration networks, citation trajectories, and researcher mobility (Scholar). For example, AMiner (Tang et al. 2008), an academic social network mining system that compiled tens of thousands of author profiles and their affiliations over time, allowing for queries on career moves and productivity. recent dataset from AMiner contains over 57,000 scholars' career paths (with 42,230 affiliations) for studying academic job transitions. These tools, however, focus on publication and citation data; they generally do not incorporate the peer review stage. We brings new dimension by linking authors the review process outcomes of their submissions. This creates opportunities for "talent trajectory" studies in AI / ML for instance, one could track how an early-career researcher improves their paper quality (as reflected in review scores) over successive submissions, or examine whether consistent high reviewers' ratings correlate with future citation impact (Vinkenburg et al.) 2020). Our dataset, combined with external bibliometric data, could also help identify patterns such as mentorship lineages (e.g.. how students of certain groups perform in reviews) or geographic trends in research feedback (Guevara et al., 2016). In the broader context of metascience, our work contributes an open resource to empirically study questions about peer review and researcher development. This aligns with recent calls for transparency and data-driven policy in peer review (Bianchi & Squazzoni 2022). By making our data easily accessible, we hope to facilitate further meta-research on how review processes influence and reflect the growth of the AT/ MI. research community.
3

---

Published as a conference paper at ICLR 2026
Purtine Dead
Paper Caplitt System
Venue Configaation Pool
Gox Cours Coniguenice
Site
Shucare Owaviow
Oanne wokepett CP: Form (oo Coli Coogle
Netwo Caratio
Scurce Aisiigiirss
Global Liser
Dosan
Corerrity
S
0 1001
Worker Fock
1 1010
Colcciss
100
Clour Stroage
pippodia
Daltere
Chock
Figure 1: Paper Copilot System Architecture
3 SYSTEM
Paper Copilot is a modular system for large-scale collection and presentation of peer review dynamics and talent trajectories. Figure[]uttinng its architecture: venue configurations define scalable sources, processed through assigners, worker pools, and parallel bots generate standardized paperlists. Versioned datasets are validated, uploaded to cloud storage, and served globally through optimized backend, database, and frontend components. The following sections detail data collection, system backend/frontend, and maintenance.
Data Collection We use hybrid data ingestion pipeline to accommodate diverse conference policies, including: 1) OpenReview API: For open-review venues (e.g.. ICLR, NeurlPS), scheduled scripts pull submission metadata, review scores, confidences, and discussion threads. We store timestamped snapshots to track score changes (e.g., pre-/post-rebuttal). 2) Webvite Scraping: For venues without APIs (e.g., CVPR, AAAI), lightweight scrapers extract accepted papers, authors, and other publicly available metadata from official websites or proceedings. 3) Community Contributions: For closed-review venues, authors voluntarily submit reviews via structured form, yielding 6,584 valid reviews per submission to date.
All data flows through cleaning and normalization module that standardizes fields, deduplicates entries, and prepares them for integration. Adding new venues requires minimal configuration.
Backend and Frontend The backend is built on a LAMP stack (Linux, Apache, MySQL, PHP) and deployed via a -Panel-managed hosting environment, which simplifies cron scheduling, backups. and database management. Conference data suta--uumissioons reviews, authorship stored in MySQL relational database with normalized schema linking papers, reviews, and authors. Pre-computed aggregates (c.g., average score, acceptance rate per venue/year) are cached to support fast retrieval for analyties and display.
The user-facing website is built with WordPress, extended by custom PHP and JavaScript for dynamic rendering. All pages support filtering by score range, decision status, or specific metadata fields, and provide links to external resources such as OpenReview, official proceedings, or ArXiv when available. Each conference currently has two primary views: 1) Statisties: Interactive charts display trends in paper counts, acceptance rates, score distributions, and reviewer score rankings. 2) Puper list: Tabulated entries show paper titles, primary areas, authors, affiliations, departments, countries, citations, and acceptance types (c.g., oral/poster).
Beyond individual venues, the system supports cross-venue, longitudinal analysis at the institutional level. Users can explore hierarchical structures trackknn contributions over time from institutions to departments, laboratories, and regions. This enables global and fineggraiedd insights into talent flows, geographic shifts, and institutional participation across the field. For more advanced exploration, provide a companion search tool that can run locally. Users can query by score thresholds, keywords, or metadata fields, operating either the live database ISON exports.
4

---

Published as a conference paper at ICLR 2026
Maintenance and Updates The entire Paper Copilot platform from backend to frontend is fully automated and designed for scalability, reliability, and long-term meta-research with minimal manual intervention. Cron jobs manage regular updates to review dynamics; scraper failures trigger alerts; and the database is routinely backed up to ensure data integrity. Static pages are cached to reduce server load, and the modular pipeline allows easy adaptation conference policies evolve. Developed and maintained at low cost, the system's 's modular architecture enables rapid scaling to new venues with minimal overhead. All processed datasets are publicly available, detailed in Section 4.2 supporting transparency, reproducibility, and community-drinn -riven enhancements to features such as search and visualization.
DATASET
The Paper Copilot open dataset spans decades of peer review data and accepted paper metadata across dozens of AI / ML conferences and continues to grow. In this section, we detail the dataset's 's contents, structure, and coverage. We describe how temporal review profiles are represented, the coverage of authors and venues, and provide examples of insights which we release the data and the visualizations included.
derive. We also explain the format in
4.1 PEER REVIEW DYNAMICS
Review Dimensions As AI ML research expands, review criteria have grown in both number and complexity, varying across venues and years. In addition to core metrics like rating and confidence, many conferences now assess dimensions such as soundness, correctness, novelty, contribution, presentation, etc. These evolving schemas reflect a shift toward more granular peer evaluation.
Reviewer-Author Discussion Score Dynamics For venues with fully open review processes (e.g.. ICLR), we leverage the public API track the temporal evolution of review scores, confidence, and comments throughout the discussion and rebuttal phases. This allows us to construct reviewer profiles over time and analyze how feedback shifts during the review cycle. By timestamping each review snapshot, our system captures pre- and post-rebuttal score changes at the individual reviewer level, enabling fine-gre graiina analysis of reviewer behavior and consistency. Figure T illustrates these dynamics using ICLR 2025 public data. Although ICLR reviews are public, historical snapshots are not preserved on the official platform older versions are overwritten during the discussion phase. Paper Copilot continuously archives these updates in real time, providing what is, to the best of our knowledge, the only publicly accessible archive of review score dynamics available anywhere on the internet.
For closed-review venues that do not disclose review revisions (e.g.. CVPR, ICCV, ICML), we invite authors to voluntarily submit their initial and final review scores. This opt-in process enables partial reconstruction of rebuttal impact, even when the official platform lacks transparency. By combining automated tracking and community community-contributee data, system enables venue-level comparisons of rebuttal effectiveness and decision volatility across time.
Closed Venue Review Disclosure Due the closed nature of most conference reviews, public access to detailed review data remains limited. As part of our community engagement, we launched an opt-in questionnaire alongside our score collection process assess author willingness to anony- mously share their reviews. This study was conducted across major venues, including CVPR, ICML., ICCV, and ACI., and will continue with NeurIPS 2025.
To characterize community attitudes toward transparency, we conducted user study across four major conferences in 2025. In total, we collected 1,860 responses, of which 1,115 authors (59.9%) consented to publicly release their anonymized review scores. Consent rates were broadly consistent across venues: 53.5% at CVPR (191 of 357), 60.7% at ICML (628 of 1,034), 59.4% at ICCV (151 of 254), and 67.4% at ACL (145 of 215). These results indicate strong support-roughl. 60% overall- for releasing anonymized review data. We plan to continue this initiative across future years and venues to build a longitudinal dataset that inform empirical studies of peer review models.
5

---

Published as a conference paper at ICLR 2026
4.2 DATASET FORMAT AND RELEASE
To ensure usability and reproducibility, we release the dataset in structured JSON format, with one record per paper containing its metadata, review scores, timestamps, and author/affiliation information. Each paper record includes over 30 fields covering metadata, authorship, review scores, rebuttal dynamics, and final decisions. More details of data acquisition can be found in the supplementary material. The main dataset is available on GitHub A separate repository, GitHub hosts the temporal data with parsing code for tracking review dynamics over time, due size constraints. We are also collecting ICLR 2026 review data, which will be integrated into the archive in future updates if the conference continues to adopt fully open review model.
THE EVOLUTION OF PEER REVIEW IN AI CONFERENCES.
In this section, analyze the evolution of peer review in ICLR, frontier AI conference. Our analysis aims at two key questions: (1) how has the conference decision-making mechanism evolved across years and (2) what are the discussion dynamics of ICLR during the rebuttal period?
5.1
THE EVOLUTION CONFERENCE DECISION-MAKING MECHANISMS
Volume Change
Submissions Entropy
Rating Average Changes
Rating Std Changes
12000
Submiesions
80c
Area Challs
Soolight
12
Number Area Chairs
5000
Decision Entroyy
7
42000
Rozing Average
Rating Std
Raject
of
2000
402
*2025
Reject
Witdrww
0.5
Withdroww
Onal
400B
2000
0.4
40017
soodight
02
2002
2350
8000
1000
2016
20107
Figure 2: ICLR review dynamics from 2017 to 2025. Despite stable score distributions, decision entropy rises with
2025, where it drops sharply. This deviation suggests Area Chairs play more decisive role, increasingly relying on mean scores under high submission pressure.
Over the past decade, ICLR has experienced explosive growth- from just 490 submissions in 2017 to 11,672 in 2025 (Figure 2 leftmost panel). To manage this scale, the number of Area Chairs (ACs) expanded proportionally, rising from 31 to 823 over the same period. This rapid expansion underscores the systemic pressure imposed by volume growth.
When analyzing the final decisions made by ACs, we quantify uncertainty using decision entropy. which measures unpredictability in tier assignment given paper's mean score. Formally, for year and score bin b:
H
Σ
Pt.b. log Pt.b.s:
H = Σ We We..He..
SE (Reject, Poster, Spotlight Oral}
where P is the empirical probability of status for papers with binned mean score, and WA. are bin-level weights. Higher H, indicates greater uncertainty; lower values indicate sharper, score-driven decisions.
As shown in Figure (centerreet H. typically increases with submission volume X,, and is well approximated by logarithmic scaling:
H. = a log X, +
This trend aligns with an ordered-login model for score-to-status mapping:
P(status = s x) = (T, - k4x) - a - KyD),
where a is the logistic CDF and K, reflects decision sensitivity to mean score in year t. As X, grows, He typically softens, yielding higher entropy.

---

Published as a conference paper at ICLR 2026
Yet ICLR 2025 is clear outlier. Despite the largest submission volume (X2025 11,672), H2025 fell below the fitted log trend. This implies a stronger N2025, i.., the venue relied more deterministically average scores, reducing uncertainty rather than increasing it. The residual,
resid2025 22225 (a log X2025 b),
strongly negative, showing that scale alone cannot explain the drop.
Turning allocation (Figure 2 center-right and rightmost panels), average ratings have grown more tier-separated, with Spotlights converging toward Orals. Meanwhile, rating variance rises for rejected/withdrawn papers but drops for accepted ones, especially Spotlights. This shows ACs increasingly favor high and stable scores, sharpening thresholds and limiting borderline flexibility.
Together, these results indicate structural change in 2025: ACS assumed ing sharper score-dependent rules. The system shifted from probabilistic tiering to near-determinstti mappings, improving efficiency but potentially narrowing the role of reviewer justifications, rebuttals, or confidence.
decisive role, enforc-
5.2 DISCUSSION DYNAMICS
ICLR 2025 Discussion Dynamics
Acceptance Rate by Score Combo
fre-rebuttal + -a Post-rebuttal
colored by combo min-max range
10
S,S.8.8 Gavge6.50, max-mins3)
max-min--))
6.K.6& (avg=6.40, mas-mine))
laugee 33, max-mine51
Roiing Avsrage
2.C.B.B
b.hi (avgue 25, max-inn
f,S,8 (avg-6.05, max minss)
18
5.5.666. levged co. max-mn=3)
5,5.8 lavg-o do, mux-min-33
Perlig Bire 18oerttinee Focusi
8.8.8 lovgen 00, min-min)0)
6.6.6 (aug mas-min-))
1,6.6,8 1mxp-5.75,
mac-min-55
weindross
Raject
eester
Speciight gral
5,8,5.8 lavg-5,75. mas-min-1
G.6il Cougu maa-nin-1)
Papers
Score Updates
Reviewer Evolution
R,5 3,5.6.6. (avg=5.60,
ma-min::5
Percentage Changes (86
ki
Rating Mm Max Difference
5.5.6,6 5.30-5.30, max- mar-min-))
-0-
5.5.6.4 (ivgens, max-min1)
1avr--3.33,
8,5.8 (auqu 1.13, mas mas
2.4
3.5.3.8 large3.25, mas-nie-d)
eac-min3)
30
1.5
155,3.6 lavged 2.25 man-min-11
Wihdiow
5.5.5.5. (avg=5 20 mas min-1)
20-
2.2.6.8 (auge mas-mine5)
13
Spotlight
3.5.6. targe5.00 5.05 mac-min-3)
Drai
Discussion End
S.S.5 laegus. max-miis-)
5.5.5.5 lavg-1.00 0b, max-min-d)
(avg-1.05,
Rating
Confituice
Soundress
Contritution Presentation
10
20
30
20%
40m
si7s uns 1074
Days
Acceptance
Figure 3: Rebuttal dynamics in ICLR 2025. (top-left) Ridge plots of pre- vs. post-rebuttal ratings across final decision tiers. (top-mid) Sankey plot tracking score trajectories from initial rating bins to final bins and final statuses. (lower-left) Distribution of score updates across review dimensions. (lower-mid) Reviewer consensus evolution measured by min-max rating difference. (right) Accep- tance rates conditioned on reviewer score combinations with frequency cutoff=30.
To complement the volume and entropy analysis, we examine how the rebuttal phase shaped reviewer behavior and final decisions in ICLR 2025 (Figure 3)
Score shift before and after rebuttal The ridge plot (top-left) shows that score shifts after rebuttal were most pronounced in higher tiers. Spotlight and Oral papers experienced clear upward movement, whereas Withdrawn and Rejected papers showed little change. The Sankey diagram (top-right) further traces the evolution of scores from initial assignments to post-rebuttal updates and final status. Here, green flows denote upward shifts in mean score and red flows denote downward shifts. The dominant pattern is that many borderline and mid-tier papers received upward adjustments, while downward movements remain less common but non non-negligible.
Reviewer score change patterns and consensus At finer granularity, score updates were not uniform across review dimensions (bottom-left). Over half of papers (54.8%) saw changes in the overall rating, whereas dimensions such confidence, soundness, contribution, and presentation
7

---

Published as a conference paper at ICLR 2026
shifted in only --10-3%% of cases. This suggests that rebuttals primarily influenced holistic ratings rather than specific dimension scores.
Reviewer consensus dynamics (bottom-center) show that rating min min-max differences consistently narrowed as discussions unfolded. Notably, consensus levels dropped sharply at the onset of the discussion phase (marked by the vertical dashed line), reflecting the divergence that arises when reviewers actively debate rebuttal content. Toward the end of the process, consensus recovered, with Oral papers converging most strongly by the decision deadline, while Rejected papers retained higher levels of disagreement. This indicates that high-stakes papers underwent deeper discussion and alignment, while weaker papers received less effort consensus-building
Acceptance rate and score patterns Finally, we inspect acceptance rates at the borderline using reviewer score combinations (bottom-right). After filtering for frequency (cutoff 30), find notable asymmetry: when the average score is slightly above the borderline, lower min-max ranges correlate with higher acceptance rates; conversely, when the average is slightly below, higher min-max ranges correlate with acceptance. A plausible explanation is that below the borderline, a single strongly positive reviewer can sway the outcome, while above the borderline, persistent disagreement often signals unresolved concerns leading to rejection. While compelling, this empirical pattern warrants further validation in future analyses.
Taken together, these findings highlight the dual role of rebuttals: amplifying changes for borderline cases and driving consensus formation for stronger papers. The corresponding plots for ICLR 2024 are provided in the appendix, enabling year-over-year comparison.
6 ETHICAL CONSIDERATIONS
The development of a dataset tracking peer review and talent trajectories necessitates a careful and transparent approach to ethical considerations. We are committed to responsible data stewardship and have outlined our mitigation strategies for key ethical challenges below.
6.1 DATA SOURCING, LICENSING, AND CONSENT
One potential concern revolves around the methods of data collection and adherence to licensing and Terms of Service (ToS). Our data is aggregated from three distinct sources, each with a different consent and licensing model. 1) Public APIs and Data Sources: A significant portion of our data is sourced from platforms with explicit data-sharing policies, such OpenReview, which provides public API for accessing review data. Data from such sources collected in accordance with their established terms. Web Scraping: Automated scraping can violate the ToS of some platforms, such as Google Scholar. To ensure full compliance, we have audited our data collection pipelines and ceased all automated data collection that conflicts with site 'robots.txt" files or explicit ToS. 3) Coamunit--Contributt
Data: For closed-revieee venues, we rely on voluntary, opt-in submissions from authors. Each author who contributes data does so via a form that includes explicit consent questions. Authors can choose whether their submission contributes only to anonymized, aggregate statisties or if their anonymized scores can be displayed in detailed tables showing individualized statistics.
6.2 PRIVACY, ANONYMITY, AND RE-IDENTIFICATION RISK
Another concern, potentially related to data licensing, is about protecting the identities of both authors and reviewers.
We do not collect, store, attempt to de-anonymize reviewer identities. The vast majority of review data comes from platforms like OpenReview, where reviewer identities are already anonymized by default. There are potentially concerns that author trajectories, especially when combined with public timelines or multiple affiliations, could create re-identification risks. To mitigate this, our primary analysis focuses large-scale, aggregated trends at the institutional and geographic levels rather than individual-level data. In addition, we are committed to implementing privacy- preserving measures, such applying differential privacy aggregate statistics.
8

---

Published as a conference paper at ICLR 2026
6.3 POTENTIAL FOR MISUSE AND DUAL-USE RISKS
The dataset's potential for misuse, particularly in hiring and evaluation, is significant "dual-use" risk. A worst-case scenario involves the data being used unfairly judge job candidates, especially early-career researchers. Our stated goal is to increase transparency in the peer review process and empower researchers with broader view of the academic landscape, not create a performance ranking tool. To guard against misuse, we will take the following steps: 1) Explicit Use Guidelines: The Paper Copilot website will feature prominent guidelines strongly discouraging the use of its data for hiring, promotion, or other high-stakes evaluations of individuals. 2) Focus on Aggregates: The platform will prioritize the visualization of aggregated, large-scale trends over individual-level data. 3) Community Dialogue: We are committed to engaging in an ongoing dialogue with the AI/ML community to monitor how the data used and to develop stronger safeguards as needed.
6.4
DATA INTEGRITY, INACCURACY, AND BIAS
The dataset is subject to several sources of potential inaccuracy and bias.
Sampling and Representation Bias. For closed-review venues, the reliance on community sub- missions introduces a self-selection bias. While this is currently the only feasible method for tracking review dynamics at scale without official data access, we recognize its limitations. To rigorously quan- tify this sampling bias, we plan to conduct comparative analysis between our community-subinn submitted data and the official ground-trute data from NeurIPS 2025 once it is released.
Data Inaccuracy. Inaccurate data, whether from parsing errors or malicious submissions, could harm researchers' reputations. The L.L.-based affiliation extraction has non-zero error rate. We will mitigate this by: 1) Clearly labeling the source of all data (e.g., "Official API", "Community- Submitted"). 2) Providing transparent metrics on the known error rates of our extraction models. 3)
Implementing validation checks for community submissions to flag anomalous entries.
Demographic and Geographic Bias. The dataset could contain demographic and geographic biases, posing a special risk to researchers from certain groups (e.g.. by geography, gender, or race). This bias can manifest in several ways. For closed-review conferences, the dataset's reliance on voluntary community submissions may skew it toward certain regions or institutions that are more culturally inclined to share data. Furthermore, some research groups have internal policies that discourage sharing review data even when it is possible to opt-out, which could lead to the systematic underrepresentatioo of those groups in our dataset.
Such imbalances could lead to skewed analyses, where trends observed in overrepresented groups are incorrectly generalized to the entire research community, potentially disadvantaging those in underrepresented regions. To actively address this, we are committed to the following actions:
Transparency via Datasheets: We will publish and maintain a datasheet that transparently documents the known geographic and institutional distributions in our dataset, highlighting potential skews so that researchers can interpret the data with proper context.
We will perform regular audits to quantify these biases. Where possible, we will compare the distribution in our data against available ground-truth statistics from conference organizers measure the extent of the disparity.
Regular Bias Audits:
Platform-Ievel Context:
On our platform, any analyses derived from potentially biased data sources, such as community submissions, will be clearly marked with disclaimers that explain the data's limitations and advise against overgeneralization.
7 CONCLUSION
Paper Copilot provides unified, large-scale platform for tracking peer review dynamics and talent trajectories across major AI / ML conferences. It enables comparative analysis score trends, rebuttal effectiveness, and institutional influence, with datasets already adopted by both academia and industry. Key limitations include partial reliance on voluntary data contributions for closed-review venues and the need to further improve the accuracy of talent trajectory reconstruction. Future work

---

Published as a conference paper at ICLR 2026
will focus expanding venue coverage and enhancing metadata quality for comprehensive and precise analysis.
Scope and Clarifications. The core contribution of this work lies in the combination of standard- ized multi-source infrastructure for peer-review data collection and normalization, the ICLR Daily Snapshot dataset that preserves temporal review states otherwise overwritten by existing platforms, and the empirical analysis enabled by this record. Although the system is publicly deployed through a web interface, the scientific contribution not the interface itself, but the underlying archive, data pipeline, and metascientific analysis. All main quantitative results in this paper are derived exclusively from official ICLR data; community-contrilute data remains explicitly separated, clearly labeled, and is not used to support the paper's ' scientific claims. We further emphasize that statisties such as entropy are used here as descriptive measurements review dynamics rather than as definitive causal evidence, and multiple interpretations remain possible, including both review-pree proccess pressure and improved consistency in evaluation. The system was also designed to extend naturally to ICLR 2026, and we continue archive its score evolution; however, because the ICLR 2026 review cycle encountered an OpenReview reviewer-information leakage issue, we treat the 2026 record in this work only as reference signal rather than part of the main analysis. Finally, broader capabilities of the system, such as tracking talent trajectories across institutions and years, are part of the extensible scope of the platform but not presented here as current empirical claims.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
ACL Anthology. Acl anthology, https://Aclaanthoooyy.oorr Accessed: 2025-05-15.
Emery D. Berger. Csrankings: Computer science rankings.
htttps:/ccsrankknggs....
Accessed: 2025-05-14.
Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The neurips 2021 consistency experiment. Neural Information Processing Systems blog post, https://blog. neurips. 2021.
cc/2021/12208//heeeeeenn 2e0r conssloooossenceeeeeereieee
Santosh Kumar Bharti, S Varadhaganapathy, Rajeev Kumar Gupta, Prashant Kumar Shukla, Mohamed Bouye, Simon Karanja Hingaa, and Amena Mahmoud. Text-based emotion recognition using deep learning approach. Computational Intelligence and Neuroscience, (022(1)) 2022.
Federico Bianchi and Flaminio Squazzoni. Can transparency undermine peer review? a simulation model of scientist behavior under open peer review. Science and Public Policy, 49(5):791-88) 2022.
Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment. arXiv preprini arXiv:2109.09774 2021.
DBLP. dblp computer science bibliography. monthly snapshot release of may 2025. https:
/bblp 2025-05 01. xml. gz Accessed: 2025-05-14.
Jesus De Frutos Belizon, Felix Guerrero Alba, and Gonzalo Sanchez Gardey. Exploring the academic research career: A bibliometric and content analysis of the academic life cycle. Innovative Higher Education, pp. 1-30, 2025.
Yangliu Fan, Anders Blok, and Sune Lehmann. Understanding scholar-trajectoriies across scientific periodicals. Scientific Reports, 14(1):5309, 2024.
Computer Vision Foundation. Cvf open open access. httpps://openaccess.tt thecvf. com 2024. Accessed: 2025-05-14.
Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. Peer review analyze: A novel benchmark resource for computational analysis peer reviews. Plos one, 17(1):e0259238, 2022.
Miguel Guevara, Dominik Hartmann, Manuel Aristarán, Marcelo Mendoza, and César A Hidalgo. The research space: using career paths to predict the evolution of the research output of individuals, institutions, and nations. Scientometrics. 109:1695-11000 1709, 2016.
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine Van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A dataset of peer reviews (peerread): Collection, insights and nlp applications. arXiv preprint arXiv: 880409635, 2018.
Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. Moprd: A multidis- ciplinary open peer review dataset. Neural Computing and Applications, 35(34):2411--4422 2023.
OpenReview. Openreview.net.
2024. Accessed:
2025-05-14.
Ana Carolina Ribeiro, Amanda Sizo, Henrique Lopes Cardoso, and Luís Paulo Reis. Acceptance decision prediction in peer-review through sentiment analysis. In EPIA Conference on Artificial Intelligence, pp. 766-777. Springer, 2021.
Google Scholar. Google scholar. http::::///colar.. www.te.oooog........... com/ Accessed: 2025-05-14.
Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 6566)776-87, 2022.
David Soergel, Adam Saunders, and Andrew McCallum. Open scholarship and peer review; a time for experimentation. 2013.
11

---

Published as a conference paper at ICLR 2026
Cha Sun.
lc120000-peenrevieddaatt
https://github: htt.h//.....//accccuool......
ICCRR000--peeeReer1eeaaaaa/// 2020. Accessed: 2025-05-14 14.
Jaroslaw Szumega, Lamine Bougueroua, Blerina Gkotse, Pierre Jouvelot, and Federico Ravotti. The open review-based (orb) dataset: Towards automatic assessment of scientific papers and experiment proposals in high-energy physics. arXiv preprint arXiv:2312.04577 2023.
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnemmnne: extraction and mining of academic social networks. Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 990-998, 2008.
Claartje Vinkenburg. Sara Connolly, Stefan Fuchs, Channah Herschberg, and Brigitte Schels. Mapping career patterns in research: A sequence analysis of career histories erc applicants. PloS one, 15())xe233252, 2020.
Jing Yang. Position: The artificial intelligence and machine learning community should adopt more transparent and regulated peer review process. In Forty-second International Conference on Machine Learning (ICML) Position Paper Track, 2025. URL
Eorum? Serum?dd=gnygaarPP2
12


Reviews:
[{"review_id": "DimkKmxeE3", "reviewer": "Reviewer_hF5U", "summary": "The paper introduces Paper Copilot , a system designed to track and analyze the peer review process at AI conferences. It collects data from public APIs, website scraping, and voluntary community submissions to create an open dataset. The authors use this dataset to conduct an empirical analysis of ICLR review dynamics, aiming to bring more transparency to the field.", "questions": "n/a", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a system architecture for aggregating peer review data from diverse sources, including public APIs like OpenReview and opt-in community contributions for closed-review venues.\n- It provides a dataset that captures temporal review dynamics, such as tracking score and confidence changes throughout the discussion and rebuttal phases, which is not always preserved by official platforms.", "weaknesses": "- The reliance on voluntary, \"community-contributed\" data for all closed-review venues is the paper's biggest weakness. The authors acknowledge this self-selection bias, but it's a fundamental limitation. It's hard to trust that this data (e.g., 1,860 responses for four major conferences ) is representative, which limits the reliability of any insights drawn about those venues.\n- I knew this (similar) project before and I have used the corresponding website. At that time, my biggest concern is the potential bias, which (I think) is inevitable but sadly could falsely affect the judgement of some researchers (especially those junior ones). So, I personally think that such project have clear advantages and disadvantages at the same time that should be critically balanced.\n- Data integrity and accuracy are potential issues, stemming from known error rates in LLM-based affiliation extraction and a reliance on scraping methods that may conflict with platform Terms of Service.\n- I am not sure whether ICLR would accept such form of paper as this paper looks more like a position paper rather than a typical ICLR technical paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0eKvY9BKUs", "reviewer": "Reviewer_c5VV", "summary": "This paper introduces a system and dataset, Paper Copilot, aimed at tracking and analyzing the evolution of peer review in AI/ML conferences. The authors note that with the surge in submissions, the current peer review system is under immense pressure, facing issues of low transparency, inconsistent standards, and heavy reviewer load. To address this, the authors have built a hybrid data collection pipeline (combining the OpenReview API, web scraping, and community contributions) to create a longitudinal dataset. A unique contribution of this dataset is its capture of temporal dynamics, such as pre/post-rebuttal score changes.\n\nThe paper's core empirical contribution is a large-scale analysis of ICLR (2017-2025). The analysis reveals several key findings, including a decision entropy anomaly in 2025 (suggesting AC decisions are becoming more reliant on mean scores), the clear role of the rebuttal phase in boosting scores for high-rated papers, and an asymmetry observed for borderline papers. Finally, the authors discuss the ethical considerations related to the system and dataset, particularly regarding data sourcing, privacy, and potential self-selection bias.", "questions": "1. Your analysis identified a shift toward \"score determinism\" in ICLR 2025. During your research, did you find any concrete reasons for this shift (e.g., specific instructions from conference organizers to ACs, or the introduction of new reviewing policies)? Furthermore, beyond diagnosing problems (like opacity or score dependency), can your analysis offer any specific, actionable solutions or intervention recommendations for conference organizers?\n\n2. Could you please clarify in detail the relationship with the Yang (2025) position paper? Was that paper merely a \"proposal\" or \"call to action,\" or did it already include the design, data, or preliminary analysis of the Paper Copilot system? For the reviewers, clarifying the precise incremental contribution of this ICLR submission is quite important.\n\n3. The \"talent trajectory\" analysis was presented as a core motivation but seems to be missing from the empirical results. Could you comment on the status of this analysis? Do you have preliminary findings you can share? Or is this purely future work? If the latter, perhaps you might consider revising the paper's framing and claims to align them more closely with the presented contributions.\n\n4. The authors mention making the dataset available on GitHub. To better evaluate this contribution, a clear data and code availability statement would greatly help reviewers assess the reproducibility and contribution to the community.\n\n5. Regarding the self-selection bias of community-contributed data, you mention a plan to compare against (future) official NeurIPS 2025 data. This is a good plan, but have you conducted any preliminary analysis to check the representativeness on the currently collected data? For example, comparing the institutional, national, or track distribution in your collected data (for CVPR/ICCV) against known public statistics (e.g., from accepted paper lists) from those conferences?\n\n**I highly look forward to the authors' discussion. If my concerns can be satisfactorily addressed, I am open to increasing my score.**", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper addresses a problem of critical importance and significant interest to the machine learning community. The challenges of maintaining a fair, transparent, and effective peer review system at scale are widely recognized, and this work provides a valuable quantitative lens through which to examine these issues. \n\n2. A major strength lies in the creation of a unique dataset that captures the temporal dynamics of the review process. By archiving review snapshots that are often overwritten, the authors provide a durable and highly valuable resource for future metascience research. This contribution has the potential to enable a new class of studies on reviewer and author behavior.\n\n3. The analysis of ICLR data is thorough and leads to several non-trivial conclusions. The finding on \"decision entropy\" (Section 5.1) is particularly compelling, offering strong quantitative evidence for a hypothesis that many in the community have anecdotally suspected: the review system may be simplifying its decision-making process under load. Similarly, the analysis of rebuttal dynamics and the \"asymmetry of borderline papers\" provides practical insights for authors, reviewers, and ACs.\n\n4. The authors' transparent and thoughtful discussion of the ethical dimensions of their work (Section 6) is highly commendable. Acknowledging issues related to data sourcing, privacy risks, potential for misuse, and self-selection bias demonstrates a mature and responsible approach to research.", "weaknesses": "1. This paper's main contribution seems to be positioned more as a resource/tool (the Paper Copilot website and dataset) rather than a novel methodology or benchmark. While the analysis is insightful, it primarily applies (relatively standard) statistical methods to a novel dataset. This makes the paper's positioning somewhat ambiguous; it feels more akin to a \"Demo Paper\" than a typical ICLR main conference paper.\n\n2. The paper references a Position Paper (Yang, 2025) whose title (\"...community should adopt a more transparent and regulated peer review process\") suggests that the core motivation and possibly even the high-level framework for the Paper Copilot system itself may have been outlined previously. The fundamental issue is that if the Paper Copilot system itself (the core concept and initial framework) was the primary contribution of that prior Position Paper, the novelty of the system component in the current submission is substantially diminished. I would hope the authors could clarify in the paper or in the rebuttal the specific, concrete and incremental technical contribution (e.g., the complex implementation details, the novel longitudinal data collection pipeline, and the empirical analysis) presented here, clearly distinguishing it from the system's initial concept outlined in the Position Paper. \n\n3. The paper claims in its abstract and introduction that the Paper Copilot system supports \"tracking talent trajectories\" as one of its core motivations (e.g., analyzing institutional or national influence). However, the core empirical analysis (Section 5) seems to lack analysis on this topic; all analysis focuses on the review process itself (entropy, score dynamics). I hope the authors might consider adjusting the paper's framing and claims to be more consistent with the presented contributions, or perhaps supplement this analysis in the rebuttal, as there currently appears to be a disconnect between the two.\n\n4. For closed-review conferences, the system relies on voluntary community contributions. This is a potentially significant limitation, as the authors acknowledge in the ethics section. While the authors conducted a user study on \"consent rate\" (59.9%), this does not guarantee that the distribution of actually submitted data is representative (e.g., submitters might be skewed toward high-scoring papers or specific institutions). This impacts the generalizability of any conclusions that might be drawn from this data in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BNGOJ8cDgQ", "reviewer": "Reviewer_reMW", "summary": "The paper introduces Paper Copilot, a pipeline-based system for collecting and analyzing peer-review data across AI/ML conferences. It aggregates multi-source information to enable large-scale statistical analysis of review dynamics, yielding insights such as the 2025 shift toward more score-driven decisions. It is also worth noting that Paper Copilot has already gained significant visibility in the community. Many researchers, myself included, have used or browsed the platform and found its analytics informative and impactful. In essence, the authors have built a system that performs large-scale statistical and visualization tasks similar to what platforms like OpenReview could provide, but with a stronger focus on analysis and accessibility.", "questions": "I do not have significant concerns about this paper, but I would like to offer a few suggestions. \n\n• Why does Figure 5 in the Supplementary Material appear to be identical to Figure 3 in the main manuscript?\n\n• Paper Copilot collects some review scores voluntarily uploaded by authors (I have contributed such data myself). However, the reliability of this information largely depends on self-reporting, and anyone could, in principle, submit arbitrary data. The authors are encouraged to estimate the extent of this potential bias. For example, they could compare the statistics collected on the platform with the actual post-acceptance distributions in later stages to assess consistency.\n\n• The name Paper Copilot does not seem closely aligned with the system’s functionality. Without prior knowledge of the project, one might assume it is a writing-assistance tool similar to Cursor or other AI copilots.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "• As mentioned in the summary, this paper corresponds to a complete solution and a mature product. This is something that many AI papers lack. It is not just a paper but a real, influential system that has already helped many researchers.\n\n• Since my research area is different, I am not very familiar with the LAMP stack, but considering that a mature web service is already running, these technologies should be reliable.\n\n• The authors present several interesting findings, such as the observation that in ICLR 2025, under the large submission volume, acceptance decisions were more influenced by mean scores, suggesting limited reviewing capacity. These statistics and data may become valuable references for future work.", "weaknesses": "• Although I fully acknowledge the influence and practical value of Paper Copilot, I must point out several weaknesses when considering it as an ICLR submission. The most significant issue is that the work is highly engineering-oriented. Beyond the statistical analyses, there is little conceptual or methodological novelty. The paper essentially combines software engineering (LAMP stack, web scraping), basic statistics, and visualization. While the results are interesting, the overall framing feels somewhat utilitarian. It focuses mainly on acceptance outcomes rather than on the fundamental purpose of peer review itself, which is to help manuscripts improve and to strengthen trust in published research. This is the main reason I would rate the paper a 6 rather than an 8.\n\n• The authors introduce an entropy-based metric and visualize its trend over time. When observing the sharp entropy drop in 2025, they attribute it to limited reviewing capacity. However, this conclusion seems overly speculative. The reasoning appears circular since the authors assume that reviewing pressure led to algorithmic decision-making, design an analysis around that assumption, and then use the results to confirm it. In fact, the entropy decline could also result from improved reviewer quality or enhanced matching processes, possibly aided by AI tools that reduce variance among reviewers and area chairs, leading to more consistent decisions. Building on that, the authors’ interpretation that low entropy represents an undesirable state may not be universally valid. As Schrödinger once said, “life feeds on negative entropy,” and in physics, low entropy generally denotes higher order and structure. In peer review, high entropy indeed reflects confusion and stress, but low entropy can have two possible explanations: (1) decision-making rigidity due to over-reliance on reviewer scores under pressure, or (2) improved efficiency from better reviewer–paper matching and enhanced consistency across evaluations. In the latter case, lower entropy could actually indicate a healthier and more coordinated review process.\n\n• There are also some writing issues. First, in the abstract, the sentence beginning with “We present PAPER COPILOT, …” contains too many clauses, which makes it hard to understand and potentially ambiguous. In addition, the introduction part is not very clear and is quite difficult to follow. If I did not already know what Paper Copilot is, I might not be able to figure out the background, purpose, and motivation of the system within limited reading time. There are also a few typos, for example, in line 199 “ArXiv” should be “arXiv.”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nyfB6cOj4m", "reviewer": "Reviewer_Pkuc", "summary": "This paper introduces Paper Copilot, an open-source framework for collecting, visualizing, and open-sourcing peer review data from major AI conferences. The paper addresses the lack of transparency in current peer review practices, arguing for a collective and standardized framework to track peer review processes and make them easily accessible to the general public and researchers. Currently, peer review information remains largely closed (except at certain conferences like ICLR) or difficult to access due to variations across fields and the structure of review systems at different conferences. Paper Copilot seeks to consolidate these heterogeneous sources into a single centralized platform. The paper makes this infrastructure public, releases a dataset for research, and presents interesting quantitative results that explain shifts in peer review practices.", "questions": "Minors.\n- Check L269. A comma is missing.\n\nQuestions.\n- Is it possible for malicious users to manipulate the survey data? Any user can submit erroneous results to the survey.\n\n- Is Figure 2 reproducible for other venues such as NeurIPS?\n\n- Can you provide a snapshot of how the review dataset looks like? \n\n- Visualization of Fig1 need to be further improved. Better caption illustrating the flow of the system is needed. Also, the text inside the figures are too small.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The reviewer finds this paper timely and necessary for the academic community, introducing a valuable open-source asset to researchers working on peer review practices. \n- The reviewer has experienced collecting review data across different venues, and understands the difficulty of collecting a unified review dataset. The difficulty stems from different fields (categories, review criteria) used across venues, and the proposed framework seems to have an independent bot to process these fields for each venue. Utilizing this framework would be of real help to the community if it is well maintained.\n\n- The results and visualizations in Figures 2 and 3 are excellent and easy to understand. Some findings, especially the results for ICLR25, are surprising.\n- Most of the questions that I had at the beginning of reading this paper were well addressed throughout the paper. The paper is easy to follow and limitations were well discussed.", "weaknesses": "- My primary concern with this work is the limited depth of discussion on re-identification risks. While the authors acknowledge the possibility of re-identification, several questions remain. For instance, to what extent will the authors employ differential privacy? Additionally, given recent advancements in LLMs, there is a significant possibility that tracking reviews across multiple venues could enable re-identification of reviewers based on their writing styles and fields of interest. I believe these issues and potential mitigation strategies warrant further discussion.\n\n- Section 6.3, \"Potential for Misuse and Dual Use Risks,\" outlines possible measures the authors can take to prevent platform misuse. While I understand these represent the best efforts the authors can make, this remains a weakness that cannot be easily addressed.\n\n- Most of the results are derived from ICLR due to its open-ness, which I do think is a weakness despite being the only option.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)