Counterfactual LLM-based Framework for Measuring Rhetorical Style
Authors: Jingyi Qiu
Hong Chen
Zongyi Li
Venue: ICLR 2026 Poster (2026)
Keywords: AI for Metascience
Preference Models
LLM-as-Judge
Computational Social Science
LLM Personas
Rhetorical Style Measurement
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: The rise of AI has fueled growing concerns about "hype" in machine learning papers, yet a reliable way to quantify rhetorical style independently of substantive content has remained elusive. Because bold language can stem from either strong empirical results or mere rhetorical style, it is often difficult to distinguish between the two. To disentangle rhetorical style from substantive content, we introduce a counterfactual, LLM-based framework: multiple LLM rhetorical personas generate counterfactual writings from the same substantive content, an LLM judge compares them through pairwise evaluations, and the outcomes are aggregated using a Bradley--Terry model. Applying this method to 8,485 ICLR submissions sampled from 2017 to 2025, we generate more than 250,000 counterfactual writings and provide a large-scale quantification of rhetorical style in ML papers. We find that visionary framing significantly predicts downstream attention, including citations and media attention, even after controlling for peer-review evaluations. We also observe a sharp rise in rhetorical strength after 2023, and provide empirical evidence showing that this increase is largely driven by the adoption of LLM-based writing assistance. The reliability of our framework is validated by its robustness to the choice of personas and the high correlation between LLM judgments and human annotations. Our work demonstrates that LLMs can serve as instruments to measure and improve scientific evaluation.

Published as a conference paper at ICLR 2026
COUNTERFACTUAL LLM-BASED FRAMEWORK FOR MEASURING RHETORICAL STYLE
Jingyi Qiu
Hong Chen
School Information
School of Information
University of Michigan, Ann Arbor jaqi u@umi ccheeuu
University of Michigan, Ann Arbor
hongcet umi ch edu
Zongyi Li CSAIL
MIT
zongyili@mit..emm
ABSTRACT
The rise of Al has fueled growing concerns about "hype" in machine learning papers, yet reliable way to quantify rhetorical style independently of substantive content has remained elusive. Because bold language can stem from either strong empirical results the two. To disentangle rhetorical style from substantive content, we introduce a counterfactual, LLM-based framework: multiple LLM rhetorical personas generate counterfactual writings from the same substantive content, an LLM judge compares them through pairwise evaluations, and the outcomes are aggregated using a -erllyy-eerry model. Applying this method to 8,485 ICLR submissions sampled from 2017 to 2025, we generate more than 250,000 counterfactual writings and provide large-scale quantification of rhetorical style in ML papers. We find that visionary framing significantly predicts downstream attention, including citations and media attention, even after controlling for peer-review evaluations. We also observe a sharp rise in rhetorical strength after 2023, and provide empirical evidence showing that this increase is largely driven by the adoption of LLM-based writing assistance. The reliability of our framework is validated by its robustness to the choice of personas and the high correlation between LLM judgments and human annotations. Our work demonstrates that LLMs can serve as instruments to measure and improve scientific evaluation.
rhetorical style, it is often difficult distinguish between
INTRODUCTION
Machine learning is field of extraordinary excitement: every year brings breakthroughs in models, applications, and benchmarks. Meanwhile, the MI. community has become intensely competitive, with leading conferences now receiving tens of thousands submissions (e.g., ICLR 2025 received over 10,000 submissions, and submissions to ICLR 2026 nearly doubled, reaching close to 20,000). In this crowded field, visibility scarce, and authors, when framing their work research papers, face strong incentives to emphasize and sometimes overstate the significance of their contributions (Smaldino & McElreath 2016 McGreivy
Hakim 2024).
This pattern has fueled concerns about hype. a rhetorical style that exaggerates novelty or impact (Lazarus et al. 20155 Thais 2024). Concerns about hype are not merely stylistic: rhetorical framing often suspected of influencing how papers are perceived in peer review and in attracting downstream attention (Boutron et al. 2014). Despite these concerns, there no systematic and widely adopted framework for measuring rhetorical style in scientific writing, Addressing this measurement gap is critical. Prior studies document rising use positive framing and promotional language (Vinkers) et a 2015| [Png et al. 2024 Millar al. (2044), declining expressions of uncertainty (Yao et al. 2023), and even widespread misrepresentation in published research (Boutron & Ravaud] 2018
McCirevyy & Hakim 2024 Chen et al. 2025).

---

Published as a conference paper at ICLR 2026
Generate LLM Persona
2. Calibrate Persona Score
Evaluate Paper Score
Judge
1.28
Agent "LeCun"
Agent "LeCun"
VS
Agent "LeCun"
VS
Agent "Marcus"
0.98
Agent
Agent "Sloopy PhD
"Marcus"
VS
VS
-2.10
Agent
Agent "Sleepy PhD"
Agent "Marcus"
"Sleepy PhD"
Figure 1: (1) LLM personas generate counterfactual writings in different rhetorical styles based on the same substantive content. (2) We calibrate the LLM personas' rhetorical scores via pairwise comparisons using an LLM judge. (3) We infer the rhetorical score of any query abstract by comparing it against the calibrated LLM persona panel.
Existing approaches primarily focus on subcomponents of rhetorical style inferred directly from the observed text alone: either by scoring promotional lexicons and indexes (Mishra et al., 2023 Peng jtt al. 2024 Gentzkow Shapiro 2010) or by training models on human-labeled constructs such as "sensationalism"
"uncertainty" (Prabhakaran et al. [0166] 6, Pei & Jurgens 2021; Wühh et al. 2024). These methods are effective capturing surface-level tone, but because they rely on a single observed realization of the text, they conflate stylistic framing with the strength of the underlying evidence. Because bold language can stem from either strong empirical results or mere rhetorical style, it often difficult distinguish between the two. The central challenge, here, distinguish a paper's presentation from its substantive contribution.
We address this challenge by formalizing rhetorical style, conditioned fixed substantive content, as a measurement problem. The core idea is simple: hold the substantive content X (methods, experiments, results) constant, vary the descriptive language Y. and infer latent variable Z that captures rhetorical strength independently of substantive content. Just as benchmarks quantify empirical progress, this formulation provides quantitative basis for analyzing rhetorical style and understanding how it shapes attention in ML research.
We develop a counterfactual, LLM-based framework that measures rhetorical style independently of substantive content. shown in Figure D we employ diverse panel of LLM personas that generate counterfactual writings in different rhetorical styles from the same substantive content. These persona outputs are then compared in pairs by an LLM judge, and the pairwise comparison outcomes are aggregated with Bradley-Terry model (Bradley & Terry 1952) to assign each persona calibrated rhetorical score. Finally, we infer the rhetorical score of any query abstract by positioning it relative to calibrated panel of counterfactual persona abstracts derived from identical substantive content.
Our Contributions.
Conceptually, our design parallels counterfactual design in causal inference, representing "what-if" alternative outcomes under different scenarios. We ask how the perception and evaluation of a paper would change if the rhetorical style were altered while the underlying substantive content remained the same. Each cersona-generated abstract represents a counterfactual answer to a "what-if" question: "What would this abstract look like if the author had written it in a different style?" By coupling controlled generation with pairwise comparison under a classic statistical framework, we introduce a scalable and principled LLM-based instrument for measuring and improving scientific evaluation.
Methodology: We introduce counterfactual evaluation paradigm using calibrated panel of LLM personas controlled writers that generate counterfactual writings from identical substantive content, with pairwise judgments aggregated via a Bradley-Terry model.
Validation: Unlike prior approaches that infer thetorical strength directly from the observed text, framework controls for substantive content, yields fineggrained scores, and is robust
2

---

Published as a conference paper at ICLR 2026
to the choice of personas. We further validate the LLM-based judgments against human annotations.
Findings: Applying our framework to 8,485 ICLR submissions from 2017 2025 and more than 250,000 generated persona writings and 250,000 pairwise comparisons, we provide large-scale evidence that visionary rhetorical style predicts citations and media attention, and that rhetorical strength has risen significantly since 2023, We further show that this increase largely driven by the adoption of L. LLM-based writing assistance.
Our of pairwise comparisons is closely related preference-basee learning methods such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), which leverage pairwise preferences over multiple outputs generated from the same input to learn reward model (RLHF) or directly optimize a policy (DPO) (Christiano et al. [017]] Ouyang et al.)
2022
Rafailov al., 2023). Related approaches in political science likewise use multiple texts describing the same event to infer latent attributes, such as political leaning (Carlson & Montgomery
2017: Licht et al., 2025). While these methods typically use pairwise preferences for model training or policy optimization, our work adapts this paradigm for a distinct purpose: to create a calibrated measurement scale for a latent attribute (rhetorical style) by holding substantive content constant.
PROBLEM SETTING
2.1 PROBLEM FORMULATION
We formalize rhetorical style as a latent variable in generative model of research writing. Specifically, denote the text of a paper as Y, which is influenced by its substantive content X and latent rhetorical style variable Z.
For simplicity, we focus on measuring rhetorical style in abstracts, since they are typically the first section readers encounter and contain the core articulation of a paper's contributions and significance. Accordingly, we define Y as the abstract text, while X refers the substantive basis, such as preliminaries, methods, experiments, and results, which are comparatively objective and descriptive. The rhetorical style
modeled as a latent continuous scalar capturing the strength framing. Formally, X E {1, (1....,7 7)" and Y E (1....,Ty: are token sequences of finite length, and Z E IR is a one-dimensional hidden variable representing rhetorical strength.
We posit that the observed writing Y is generated from conditional distribution:
p(Y X, Z)
where X refers to the substantive content and Z modulates the framing of the contribution. The objective of this work is to infer Z independently of X, yielding a quantitative measure of rhetorical strength disentangled from substantive content.
Traditional approaches to measuring rhetorical style in scientific texts rely on direct linguistic features of Y, but such measures are confounded by X, since stronger results naturally justify stronger claims. Our approach instead targets Z, isolating rhetorical style conditional on fixed substantive content.
2.2 RHETORICAL STRENGTH
We assume that total order exists in the space of latent rhetorical style Z, as Z E R. A higher value of Z corresponds to a "stronger" rhetorical style (e.g., more visionary), while lower value suggests "weaker" rhetorical style (c.g., more conservative).
Definition 2.1 (Stronger Rhetorical Style). A stronger rhetorical style refers to a rhetorical style that broadly expands its implications. This involves generalizing its applicability, highlighring the significant challenges it addresses, and emphasizing its novelty and impact.
Definition 2.2 (Weaker Rhetorical Style). A weaker rhetorical style refers to rhetorical style that narrowly confines its implications, potentially by specifying its applicability and prerequisites, positioning it closely previous work, and emphasizing its limitations and uncertainties.
For any two rhetorical styles 21, 22 E Z. if 21 > 22, then 21 represents a stronger rhetorical style than 22.
3

---

Published as a conference paper at ICLR 2026
Definition 2.3 (Order of Rhetorical Strength). Given fixed substantive basis = for any two writings y1, Y2 derived from (x, 21) and (z, 22), we say y1 has a stronger rhetorical style than y2 (denoted y1 >= y2) if and only if > 22- It implies yn presents the writing with greater assertiveness, broader scope, or higher proclaimed impact than y2-
In practice.judgments judgments about whether y1 is rhetorically stronger than y2 can be noisy, affected by both uncertainty and observation error. We therefore model these comparisons using the Bradley-Terry model (Section 3.3)
While we acknowledge that rhetorical style is inherently multi-faceted, i.e., encompassing distinet dimensions such as claims of novelty, generalizability. and impact, we model here as a single latent dimension of "strength." This simplification is a principled choice for this foundational work, as it allows us to first establish whether rhetorical framing can be cleanly disentangled from content. The framework itself is extensible, and future work could adapt the judging criteria to measure multi-dimensional representation of rhetorical style.
COUNTERFACTUAL -LS-BA FRAMEWORK
In this section, we introduce a counterfactual LLM-based framework for measuring latent rhetorical style. As shown in Figure 1 the method proceeds in two stages. First, construct panel of LLM persona writers with specified rhetorical tendencies and calibrate their rhetorical scores via pairwise judgments from L.M judge. Second, infer the rhetorical score of any query abstract by comparing it against the calibrated persona panel.
Our framework relies two assumptions: (i) the persona panel adequately spans the distribution of query abstracts, and (ii) the LLM judge can distinguish rhetorical strength with non-trivial accuracy. We later provide evidence that both assumptions hold in practice in Sections 4. and 4.2 respectively,
3.1 COUNTERFACTUAL GENERATION WITH LLM PERSONAS
Given the substantive content x, we sample counterfactual abstracts by prompting the LLM with K different personas that embody diverse rhetorical styles in academic writing:
yAa LLM(a, promptA)); k= 1,......4 K
This yields set of diverse counterfactual abstracts {MA,, MA2 MAx } for the same underlying content x², each differing only in rhetorical framing.
Each persona A is defined by system prompt that enforces a distinct thetorical style (c.g., cautious, visionary, technical). The full list of personas and their descriptions is provided Appendix [C.]] Note that the key requirement here is not the specific identity of the personas, but that the panel spans sufficiently broad range of rhetorical styles. To provide intuition, we generate three versions of our paper's 's abstract, each written from different persona using our framework (Appendix B).
3.2 PAIRWISE EVALUATION WITH LLM JUDGES
To establish an ordering over persona-generated abstracts, use LLM-based judge that performs pairwise comparisons of rhetorical strength.
Given two abstracts VA, and A both derived from the same substantive content but generated by different personas A₁ and Ag, the LLM judge is prompted to decide which abstract makes stronger or more sensationalized claims, and provide a brief rationale for its choice.
3.3 CALIBRATION VIA MAADLEY-TERRY MODEL
To aggregate pairwise comparisons into a global ordering of rhetorical strength, we employ the Bradley-Terry model (Bradley & Terry 1952).
Let A, denote the thetorical strength parameter for persona Aş- Given two abstracts y., and YA generated from the content but with different personas A and Ag, the probability that yA,
4

---

Published as a conference paper at ICLR 2026
judged stronger than yAa is:
TA TA, A2
P(UA1 > vA.)
For each pair of personas (A1,A2) we sample M instances [ generate abstracts {V J and (y',, Jim and obtain LLM-judge comparisons. Aggregating these across all persona pairs, estimate मे {MA) ..... TAx by maximum likelihood. We then define the continuous rhetorical score Sk = (T&) which places each persona on a one-dimensional spectrum of rhetorical strength.
3.4 INFERENCE FOR QUERY ABSTRACTS WITH THE REFERENCE PANEL
After establishing the calibrated scale of K personas with rhetorical strength parameters (T...... TAx ). our goal is to infer the rhetorical strength of a new query abstract y/y. We do so by positioning By on the same scale through comparisons against the persona abstracts. For each query abstract, the LLM judge conducts K pairwise comparisons, one against each persona abstract. Under the Bradley-Tare Terry model, the probability that the query abstract wins against persona Aş mgg
P(Ug > VA.)
While maximizing this likelihood with Maximum Likelihood Estimation (MLE) works well for cali- brating persona scores because persona abstracts are compared against each other many times, query abstracts face a much sparser setting, since each is compared only once against each persona abstract. In such cases, MLE can yield degenerate solutions: if a query abstract wins all its comparisons, the likelihood drives its score to infinity; it loses all, the score collapses toward zero. To obtain stable estimates, instead use Maximum Posteriori (MAP) estimation, which introduces regularization through a prior. Let Sg = (Ty). We place a Gaussian prior on the query abstract's 's score. The MAP estimate is then the value that maximizes the log-posterior, combining the Bradley-Terry likelihood with the log-prior penalty.
Adaptive Bayesian Inference. We also consider alternative, more cost-efficient estimation strategy based on adaptive Bayesian inference. Rather than using a fixed batch of comparisons, this approach maintains posterior distribution over the query abstract's score and sequentially selects comparisons to maximize information gain. At each step, the next persona is chosen adaptively: the persona whose score is closest to the current posterior median, since this comparison is expected to reduce uncertainty most effectively. The posterior is then updated with the observed outcome, and the process terminates once the posterior variance falls below a predefined threshold. This yields a point estimate of rhetorical strength with an associated confidence interval that quantifies the measurement uncertainty. In this study, we adopt batch MAP estimation as is sufficient at our scale.
EXPERIMENTS
To investigate rhetorical style in machine learning papers, we compiled dataset of 8,485 research papers submitted to the International Conference Learning Representations (ICLR) from 2017 to 2025, randomly sampling 1,000 submissions from each year. Our data preprocessing pipeline involved two steps. First. extracted the full text from each paper's PDF. Second, used an LLM-based extraction method (OpenAl GPT-4o-mini) to identify and extract the substantive content from the experiments, methods, and results sections in research papers, which serve the substantive basis (denoted as X) in our framework. This filtering process removes all narratives and references from the paper. The paper abstracts were used as the observed writing (Y) for analysis.
4.1 IMPLEMENTATION
Our framework proceeds in two stages. First, we calibrate stable rhetorical scale using panel of 30 LLM personas. A key principle in constructing this panel to ensure its rhetorical distribution spans the wide and meaningful range observed in the original query abstracts. This design choice critical for ensuring measurement reliability. Broad coverage prevents "off-scale" measurements where a paper is substantially stronger or weaker than all available reference points, which can lead to unstable scores. Furthermore, it maximizes measurement precision: under the Bradley-Terry model, the most informative comparisons for reducing uncertainty are those between items of similar
5

---

Published as a conference paper at ICLR 2026
250
-0.4
Averge Phatoriza Soore
Estimased LLM Usoge (84
22.8
18.5%
17.5%
Dsage
Average Rieetoriaal
15.7%
15.4%
-088
150
11.1%
13
75
2013
2000
2023
2073
2074
2075
6.0
Rhatorical Group (d+Lowest, 3-Highwol)
Figure 2: Left: Yearly trends in rhetorical scores. Points show yearly means with shaded bands indicating 95% confidence intervals. We notice sharp increase since 2023. Right: We divide papers in the 2024-2025 batch into ten quantiles per rhetorical score and estimate their LLM usage via Liang al. (202)). We notice strong correlation between their rhetorical scores and the estimated L.LM usage. More detailed statistics are presented Table in Appendix.
strength. By distributing personas across the spectrum, we ensure that every query abstract receives several such highly informative comparisons.
To achieve this, we construct our hand-designed personas, i.e., ranging from archetypes such as Daleep-Deprived PhD Student" prominent figures such as "Geoffrey Hinton" and "Yann LeCun" (full list in Appendix (C.)), with explicit attention to coverage. For each pair of personas, we sample 20 papers, generate counterfactual abstracts from identical methods and results sections (with length constrained to within +15 words of the original abstract), and ask GPT-4o to judge which abstract makes stronger claims. As shown in Figure 3 this process successful: persona win rates against query papers vary widely from 14% (highly cautious styles) to 94% (highly assertive styles). Aggregating the 8,700 such pairwise comparisons with a Bradley-Tery model produces a continuous and robust spectrum from conservative promotional styles.
Second, we apply this calibrated scale to 8,485 ICLR submissions. For each paper, we generate 30 persona abstracts and compare the original abstract against each of them, yielding 30 judgments per paper. In total, this stage produces 254,550 additional comparisons, which we again aggregate via Bradley-TTrryy obtain each paper's rhetorical score. Full persona and judge prompts appear in Appendices C.2 and
Direct Rating Baseline. As a point of comparison, we implement direct LLM rating strategy. To mitigate the content-style confound of naive prompting, the model is provided with both the original abstract Y, and the extracted methods and results Xg, and is asked to rate the degree of overclaiming on a 1-10 scale (1 = "no overclaiming." 10 "extreme overclaiming"). In effect, this prompt directly asks the model to infer the latent rhetorical style Z, by comparing the claims to the evidence. The model is required to output both numeric score and justification. The full prompt is shown in Appendix C.4
Keyword- and Classifier-Based Baselines. We also implement two widely used baselines from prior work. First, following Penn et al.) (2024). we compute a promotion score as the proportion of words in abstract that appear in curated promotional lexicon. Second, following Pei & Jurgens (202I), we estimate the certainty level using a pretrained sentence-level certainty classifier, averaging predictions across all sentences in abstract. Both baselines rely solely direct analysis of the observed text Yg. and therefore risk conflating rhetorical style with underlying content.
4.2 VALIDATING LLM PERSONAS AND LLM JUDGE
Robustness of Rhetorical Scores to Choice of Personas. We test robustness to the choice of personas using complementary subset strategy. In each trial, the 30 personas are randomly split into two disjoint groups of 15, and independently within each subset. We repeat this procedure 1,000 times to ensure stability. The resulting BT scores remain highly consistent across non-overlapping subsets, with the mean Spearman correlations being 0.89. This indicates that the relative ordering of rhetorical strength of papers is
scores personas and original abstracts are recalculated

---

Published as a conference paper at ICLR 2026
1.0
Mr%
822
22b 22%
Win Rate
0.
0.0
Agort the pugee Aiten
Poent Patet Covaarme Stake
Ageis "Poei San
Agent "Zsttar
Agent Thaall Clospeec Grear Agont Sdd Carner Acade
Aryest "aavi
"Sary"
Aggen
Ageet Dhertt
Figure 3: Persona win rates against the sampled 8,485 query papers.
strongly preserved regardless of the specific personas used. It also demonstrates that our framework captures genuine rhetorical style differences and is applicable beyond our specific set of 30 personas.
Validating LLM Judge. To assess whether our automated measurements align with human per- ception, we conduct a human annotation study via Prolific. Annotators compare abstracts pairwise and select which makes stronger claims, following the same instructions as the LLM judge. We collect 420 judgments from 42 participants across 69 unique comparisons (45 persona persona, 24 original personaa, each evaluated by an average of 6.1 participants with majority-vote aggregation. To ensure data quality, participants were required to hold at least bachelor's, s, master's, or doctoral degree in computer science and could proceed to the main task only if they answered two predefined qualification questions correctly.
Human validation confirms that our framework aligns closely with human perception. At the pairwise level, human majority votes agree with the LLM judge in 88.4% of comparisons. At the aggregate level, the Bradley-Terry scores derived from human majority votes and those derived from the LLM judge are strongly correlated (Spearman م 0.92, p < 0.001). This suggests that our automated method captures rhetorical strength in a manner consistent with human evaluation.
Convergence of Rhetorical Scores wrt. Persona Set Size. To assess the sensitivity of our results to the specific choice and number of personas, persona subset sizes l ranging from to 30. For each size k, we perform 20 independent trials where we randomly sampled personas from the full panel and re-calculated the Bradley- all papers. We then calculate the Spearman rank correlation between the scores derived from these subsets and the reference scores obtained from the full 30-persona panel. Figure prpesentss the results. The correlation converges rapidly; with only 8 randomly selected personas, the correlation exceeds 0.89, With 15 personas, it reaches 0.958 with negligible variance (std. dev. #0.005). This confirms that the framework is robust to the specific selection of personas and that smaller panel would suffice for future applications.
perform systematic scaling analysis. We test
scores for
4.3 DISTRIBUTION OF RHETORICAL STYLE MEASUREMENTS
We apply our framework to 8,485 ICLR submissions to estimate rhetorical style at scale. As shown in Figure 6 rhetorical scores follow an approximately Gaussian distribution (-4.74 to 4.53), which captures substantial variation in how authors frame their contributions. By contrast, direct rating scores cluster heavily around values of 2-3, yielding a coarse, skewed distribution. We focus on these two measures because both consider the substantive content (X) and abstract (Y), making them directly comparable. For reference, distributions of other baselines (promotion and certainty scores), which analyze only the observed text Y without considering X, are provided in Appendix Figure
To validate that our persona panel provides sufficient coverage. Figure 3 reports win rates of all personas against query papers. The wide range of win rates, from highly assertive personas winning
7

---

Published as a conference paper at ICLR 2026
over 90% of comparisons to more conservative personas winning just 14%, demonstrates that the panel adequately spans the rhetorical space of query abstracts.
4.4 PREDICTIVE VALIDITY
We first examine whether rhetorical style is correlated with peer-review ratings. The Bradley- Terry score shows almost no correlation with the average reviewer score (Spearman's p -0.015, p = 0.225), and regression estimates likewise indicate no significant relationshipl | This suggests that abstract-level rhetorical differences do not meaningfully influence reviewer assessments of paper's s scientific contribution. This null correlation is expected and theoretically sensible: reviewers evaluate the entire submission, including methodology, technical rigor, and empirical validity, rather than the abstract alone. Moreover, rhetorical style is not part of the ICLR review criteria, which explicitly prioritize novelty and scientific soundness.
Citation
Post
Tweet
Feeds
Patent
Account
Rhetorical Score (Ours)
24.53***
3.19**
2.51*** ***
0.03***
0.04**
2,71
(6.40)
(0.48)
(0.39)
(0.00)
(0.01)
(0.40)
Direct Rating Score
-6.11*
0.74
0.75
0,00
0.01
0.77
(13.17)
(0.99)
(0.80)
(0.01)
(0.03)
(0.83)
Promotion Score
20.01f
0.64
0.51
0.02*
0.02
0.57
(10.24)
(0.77)
(0.62)
(0.01)
(0.02)
(0.65)
Certainty Score
59.56
-12.74*
-9.74*
-0.02
0.17
-9,97
(76.96)
(5.78)
(4.66)
(0.05)
(0.15)
(4.86)
Table 1: Regression coefficients with standard errors in parentheses. Each coefficient is estimated from a separate regression model in which the outcome is either scholarly impact (citations) or media attention (posts, tweets, feeds, patents, accounts). The focal predictor in each specification is one of the four rhetorical measures (rhetorical score derived using our framework, direct rating, promotion, certainty). All specifications include controls for average peer review rating, research subfield, and publication year to account for differences in paper quality, field-specific variation, and changes over time. Statistical significance is denoted by stars: p 0.10, * p 0.05, ** p < 0.01, #** < 0.001.
We next evaluate whether rhetorical style predicts downstream attention after controlling for average reviewer scores (as a proxy for paper quality), as well as year and subfields. Table[] regression results for two classes of outcomes: (i) scholarly impact, measured by citations; and (ii) media attention, measured by Altmetric indicators including posts, tweets, RSS feeds, patents, and unique accounts mentioning the paper The coefficients indicate the expected change in the outcome variable for one-unit increase in the rhetorical style score, holding other factors constant. Statistical significance denoted by stars, where more stars denote higher confidence level that the relationship exists. A corresponding visual summary of these coefficients is provided in Appendix Figure
Table I shows that higher rhetorical scores significantly predict both citations and media attention. For example, a one-unit increase in the rhetorical score is on average associated with 24 additional citations, 3 more media posts, and more tweets. For context, one-unit increase in the average reviewer score is associated with approximately 89 additional citations, indicating that the effect of rhetorical style on citations is not trivial. In contrast, baseline measures (direct ratings, promotion, certainty) fail to consistently predict these attention outcomes and often yield unstable or inconsistent coefficients.
'Further analysis confirms that this null relationship robust: rhetorical style does not predict the variance of scores across reviewers for a given paper, nor do we find any evidence of non-linear relationship.
"Altmetric indicators track mentions of research across diverse sources. Posts refer mentions in news outlets and blogs; Tweets are Twitter (X) posts mentioning the paper, counted once per account; Feeds denote blog mentions that Altmetric monitors via Really Simple Syndication (RSS). Altmetric maintains a curated list of academic and science-relate blogs, and when one of these blogs posts about a paper, it appears as a "feed" mention; Patents reflect citations of the paper in patent filings; and Accounts indicate the number of distinet Twitter (X) users who mentioned the paper.
8

---

Published as a conference paper at ICLR 2026
Taken together, these results suggest that framework reliably isolates rhetorical style from substantive content and captures variation that systematically influences how research is perceived in both scholarly and public domains. Specifically, rhetorical style in paper abstracts predicts downstream attention but not reviewer evaluations. This pattern is expected: broader audiences often engage primarily with abstracts, where rhetorical framing is most salient, whereas reviewers typically evaluate the full paper and base their assessments on both the overall presentation and its technical quality.
4.5 TEMPORAL AND SUBFIELD TRENDS
We now analyze how rhetorical style evolves over time and differs across MI subfields. Figure 2 shows that the average rhetorical scores slightly decline from 2018 to 2022, followed by sharp increase after 2023. We present detailed analyses of the potential mechanisms underlying this increase
in Section 4.6
To investigate differences across subfields, we classify papers using an LLM (GPT-4o) with a curated prompt provided in Appendix C.5 As shown in Figure 4 do observe the systematic differences in rhetorical style across subfields. Applied areas such as computer vision, NLP, and computational biology exhibit stronger rhetorical styles, while theoretically oriented fields such as kernel methods, optimal transport, and supervised representation learning tend to adopt more conservative framing. Interdisciplinary topics connecting technical and social concerns (e.g., fairness, privacy, interpretabil- ity) fall in between. These patterns suggest that rhetorical style varies systematically with research subfields and their associated community norms.
4.6 DISENTANGLING RHETORICAL STYLE FROM L.LM ADOPTION
To test whether our rhetorical scores reflect author intent or greater use of LLM-based writing assistance, use the framework of Liang et al. (2024), which estimates the population-level share of LLM-generated text (a) from linguistic feature distributions across documents. Because this method operates at the corpus rather than abstract level, group abstracts by rhetorical score and estimate LLM usage within each group
Findings: Estimated LLM Usage and Rhetorical Style are Strongly Correlated. Restricting our analysis to the ICLR 2024-2025 batch (n=2,000), we divide papers into 10 equally sized groups based on their rhetorical scores. The Pearson correlation between the group-level mean rhetorical score and the estimated LLM usage is strongly positive (r 0.904). As shown in Figure groups with higher mean rhetorical scores exhibit substantially higher estimated LLM usage. Specifically, papers the rietest- group show estimated LLM usage of 20.9%, over 2.3 times higher than the 9.0% usage in the lowest-rhetoric group. Intermediate groups follow the same upward gradient, with each increase in rhetorical intensity corresponding to progressively higher estimated LLM usage. This monotonic pattern suggests a strong link between rhetorical intensity and LLM adoption.
The observed correlation could in principle arise either because (1) LLMs intrinsically produce more rhetorical language or (2) the detector is biased to flag rhetorical text as L.MM-generted.. Our evidence supports the former and rules out the latter. Generating low-rhetoric personas via LLMs proved substantially more difficult than generating high-rhetoric ones, and two-thirds of our personas produce text with higher rhetorical scores than the original human-written abstracts, indicating an intrinsic LLM tendency toward elevated rhetoric.
We then test whether strong rhetoric alone could trigger the LLM detector: for Human--ritted abstracts in ICLR 2017-2023, estimated LLM usage remains near zero across all rhetorical groups, as shown Table If rhetorical intensity mechanically inflated a, we would expect this group to exhibit much higher estimates. It does not. This pattern indicates that strong rhetorical style alone does not cause the estimator to misclassify human-writter text as LLM-generated. Additionally, we apply the Liang et al. (2024) estimator to our LLM-generated persona abstracts, grouped by persona. In this setting, all texts are known to be LLM-generated TableB]reports rhetorical scores and estimated LLM usage across personas. As shown in Table personas with higher rhetorical scores do not consistently receive higher a estimates. The correlation between persona-leve rhetorical scores and estimated L.LM usage is extremely weak (Pearson 0.04).

---

Published as a conference paper at ICLR 2026
Apps: biology (n=61)
Apps: neuroscience/cog sci (n-129)
Rep. learning: vision/audio/NLP (n=1075)
implementation: peralle//ifrr/ssfftwreeww (n=222)
Datasets & benchmarks (n=239)
Societal: bennesssieeyrrooooiippppt....oo (n=582)
Learning theory (n=334)
physics (n-114)
robotics/autoonn//llemitttt (n=242)
Neurosymbolic & hybrid Al (n-149)
Generative models (n=629)
Untsetiseeiissperrvidd rep. learn. (n~654)
General ML (n=67)
transferimetallifelong leamingg (n-572)
Optimization (n=937)
Apps: chemistry/comp blo (n-77)
Graph geometric learning (n=568)
Causal reasoning (n-149)
Viz & interpretation of reps (n=203)
Metrichemmelsoorseddd----pp
(n=196)
Reinforcement learning (n=879)
methods (Bayes\Usampling(UU)) (n-330)
-1.4
-12
-1.0
-0.8
-0.6
-0.4
-0
0.0
Average Rhetorical Score By Sub-field
Figure 4: Mean rhetorical scores by subfield. Blue bars indicate subfield-speciff mean values, with black horizontal error bars showing 95% confidence intervals based on the standard error the mean. Subfield labels standardized abbreviations of topic names, and numbers in parentheses indicate the number of papers in each subfield; see Appendix Table for the full topic mapping.
Taken together, these findings indicate that stronger rhetorical style does not inherently increase the likelihood that the [Lang et al. (2024) estimator classifies text as J.LM--geerated. Instead, the post-2023 rise in rhetorical strength is more consistent with increased adoption of LLM-based writing tools than with independent shifts in author behavior.
DISCUSSION
This work introduces a counterfactual, LLM-based framework for measuring rhetorical style ML papers by holding substantive content fixed and varying only rhetorical framing, enabling calibrated measure of style disentangled from content. Applying the framework to 8,485 ICLR submissions, find that stronger, attention, even after controlling for reviewer scores as proxy for quality. We further show that the sharp rise in rhetorical strength after 2023 is closely linked the adoption of LLM writing tools, providing, to our knowledge, the first large-scale quantitative evidence that LLMs are systematically reshaping communication norms in scientific research. Although the framework relies on LLM judges and may inherit their biases, we validate it by showing robustness across personas and strong agreement with human annotations.
visionary framing significantly predicts both scholarly impact and public
Limitations and future work. Our findings suggest that in ML, how research is presented sub- stantially shape how is received. When stylistic choices amplify attention independently of technical merit, they raise normative concerns for the ML community. This points possible interventions such as reviewer training, clearer guidance for separating substance from style, and community norms that value precision alongside ambition. More broadly, while current framework uses fixed set of hand-crafted personas, future work could develop more systematic persona construction, extend the analysis from papers to researcher-level outcomes such citations, collaboration, funding, and hiring, and track how generative AI tools continue to reshape scientific writing over time. By making rhetorical style measurable, our framework provides both way to study these shifts and potential tool for helping conferences monitor evolving rhetorical norms and better distinguish contribution from presentation.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work analyzes rhetorical style in machine learning papers using publicly available submissions to ICLR between 2017 and 2025. All data were obtained from papers that were already accessible the research community; no private, proprictary, or personally identifiable information was used. While we used LLMs to generate counterfactual writings and perform pairwise judgments, we validated these automated outputs with a human annotation study. All human annotations were collected through Prolific with fair compensation and qualification screening. Importantly, our study does not endorse or encourage hype, and we speculate that hype may have detrimental long-term consequences for researchers.
ACKNOWLEDGEMENT
J. Qiu gratefully acknowledges financial support from Microsoft MIDAS program. Z. Li gratefully acknowledges financial support from MIT Novo Nordisk Postdoctoral Fellowship. We gratefully acknowledge API credits provided by OpenAI through its Researcher Access Program. We thank Eunsol Choi, Naihao Deng, Russell Golman, Jiaxin Pei, Nihar Shah, Andrew M. Stuart, Misha Teplitskiy, and Lechen Zhang for helpful discussions and valuable suggestions. We also thank participants in the ML-Driven Discovery Workshop at MIT and in the Computational Social Science Working Group, Behavioral and Experimental Economics Group, and NLP Reading Group at the University of Michigan for their helpful comments and suggestions.
REFERENCES
Isabelle Boutron and Philippe Ravaud. Misrepresentation and distortion of research in biomedical literature. Proceedings of the National Academy of Sciences, 115(11):613-2611, 2018.
Isabelle Boutron, Douglas G Altman, Sally Hopewell, Francisco Vera-Badillo, Ian Tannock, and Philippe Ravaud. Impact of spin in the abstracts of articles reporting results of randomized controlled trials the field of cancer: the spiin randomized controlled trial. Journal of Clinical Oncology, 42(36):4100-411 2014.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(334))324-35, 1952.
David Carlson and Jacob M Montgomery. A pairwise comparison framework for fast, flexible, and reliable human coding of political texts. American Political Science Review, 111(4):835-843, 2017.
Hong Chen, Misha Teplitskiy, and David Jurgens. The noisy path from source citation: Measuring how scholars engage with past research. In Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11786-3182,, 2025.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017
Matthew Gentzkow and Jesse M Shapiro. What drives media slant? evidence from us daily newspa- pers. Econometrica, 78 78(1):35-71, 2010.
Clément Lazarus, Romana Haneef, Philippe Ravaud, and Isabelle Boutron. Classification and prevalence of spin abstracts of non non-rmndde studies evaluating an intervention. BMC medical research methodology, 15:1-8, 2015.
Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp. Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, et al. Mapping the increasing use of 1lms in scientific papers. arXiv preprint XrXiv:2444.0 404.0/264....... 2024.
Hauke Licht, Rupak Sarkar, Patrick Y Wu, Pranav Goel, Niklas Stoehr, Elliott Ash, and Alexan- der Miserlis Hoyle. Mcasuring scalar constructs in social science with 1lms. arXiv preprint arXiv:2509.031 16, 2025.
11

---

Published as a conference paper at ICLR 2026
Nick McGreivy and Ammar Hakim. Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Nature Machine Intelligence, (10):1256-166) 2024.
Neil Millar, Bryan Mathis, Bojan Batalo, and Brian Budgell. Trends in the expression of epistemic Stance in nih research funding applications: 1985-2020. Applied Linguistics, 45(4))658-675, 2024.
Apratim Mishra, Jana Diesner, and Vetle I Torvik. A probabilistic model of "hype"in scientific abstracts. In International Society of Scientometrics and Informetrics Conference 2023 (ISSI),
2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Jiaxin Pei and David Jurgens. Measuring sentence-level and aspect-level (un) certainty in science communications. arXiv preprint arXiv:2109.14776, 2021.
Hao Peng, Huilian Sophie Qiu, Henrik Barslund Fosse, and Brian Uzzi Promotional language and the adoption of innovative ideas in science. Proceedings of the National Academy of Sciences, 121 (25):2230006612), 2024.
Vinodkumar Prabhakaran, William L Hamilton, Dan McFarland, and Dan Jurafsky. Predicting the rise and fall of scientific topics from trends in their rhetorical framing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1170-1180, 2016.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53747 2023.
Paul E Smaldino and Richard McElreath. The natural selection bad science. Royal Society open science,
2016.
Savannah Thais. Misrepresented technological solutions in imagined futures: The origins and dangers of ai hype in the research community. In Proceedings of the AAAI/ACM Conference on Al, Ethics, and Society, volume pp. 1455 1465, 2024.
Christiaan H Vinkers, Joeri K Tijdink, and Willem M Otte. Use positive and negative words in scientific pubmed abstracts between 1974 and 2014: retrospective analysis. Bmj, 351, 2015.
Amelie Wührl, Dustin Wright, Roman Klinger, and Isabelle Augenstein. Understanding fine- tne-grained distortions in reports scientific findings. arXiv preprint arXiv:2402. 2024.
Mingxin Yao. Ying Wei, and Huiyu Wang. Promoting research by reducing uncertainty in academic writing: large-scale diachronic case study on hedging science research articles across 25 years. Scientometrics, 128(8):4544 (488))4444-4558 2023.
12

---

Published as a conference paper at ICLR 2026
A
SUPPLEMENTARY FIGURES
BT Score Stability: Rank Correlation Between Subset and Full Persona Set (Error Bars: #1 SD Across 20 Random Trials)
Spearman Correlation with Full set (30 Personas)
2.0
Random Sarppiing
0.-
0.0
Number Used
Figure 5: Spearman Rank Correlation of Rhetorical Scores from Persona Subsets vs. Full 30-Persona Set. Results are averaged over 20 random trials for each subset size.
0.4
Density
0.15
0.50.37
Proportion
0.3
11m2.71
0.10
0.2
0.1
0.80
0.0
Rhetorical Score
Direct Rating
0.35
0.30
2.5
Density
0.25
2.5
0.15
1.0
0.10
0.
0.00
0.0
3.8
4.0
4.4
4.5
Promotion
Certainty
Figure 6: Distributions of different rhetorical measures. Histograms with kernel density estimates (for continuous variables) bar plots (for discrete variables) show the distributions of the four main predictors: thetorical scores, direct ratings, promotion score, and certainty score. Vertical dashed lines indicate sample means.
13

---

Published as a conference paper at ICLR 2026
Citations
Posts
Tweets
Rhetorical
[
Direct Rating
Promotion
H+
Certainty
-100
100
200
-20
-10 Patents
-15
-10
-5
Feeds
Accounts
*** i+l
Rhetorical Score (Ours)
|01
Direct Rating
Promotion
Certainty
0.100 -0.05 0.00
0.05
0.0
0.2 Coefficient
-20
-10
Coefficient
Coefficient
This figure presents regression coefficients with 95% confidence intervals for four rhetorical style measures across six attention outcomes. Coefficients represent the expected change in each outcome associated with a one-unit increase in the predictor, holding controls (peer review rating, sub- field and year) constant. Points denote estimated coefficients, horizontal lines show 95% confidence intervals, and stars indicate levels of statistical significance ("p < 0.05, **p < 0.01, "**p < 0.001).
Figure 7:
14

---

Published as a conference paper at ICLR 2026
A SUPPLEMENTARY TABLES
Full topic name
Abbreviation
Count
representation learning for computer vision, audio, language, and other modalities
Rep. learning: vision/auddioNLPP
1079
optimization
Optimization
938
reinforcement learning
Reinforcement learning
890
unsupervised. self-supervised. semi- supervised, and supervised representation
Un/selisemm/supervieed rep. learn.
665
learning
generative models
Generative models
633
socictal considerations including fairness, safety, privacy, interpretability, and explain- ability
Socictal: falmesskaafcry/rivaayyneer pret./explain.
589
7 transfer learning, meta learning, and lifelong learning
Transfer/metu/lieeoong learning
573
learning on graphs and other geometries
Graph & geometric learning
572
topologies
learning theory
Learning theory
335
10
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.) applications to robotics, autonomy, planning datasets and benchmarks
Proh. methods (Bayes/VI/sam-
332
pling/UQQ
Apps: rohatics/autooomyy/llmmnnng
12
Datasets & benchmarks
240
13
implementation issues, parallelization, infras- tructure, software libraries, hardware, visualization interpretation of learned rep- resentations
Implementation:
parallel/infra/sofii
223
ware/hw
14
Viz & interpretation of reps
204
15
metric lcarning, kernel learning, sparse cod- ing, and dimensionality expansion causal reasoning
196
16
Causal reasoning
150
17
neurosymholic hybrid Al systems (logic formal reasoning, etc.) applications neuroscience and cognitive science
Neurosymbolic hybrid
18
Apps: neuroscience/cog sci
129
19
applications to physics
Apps: physics
1 116
20
applications to chemistry and computational biology
Apps: chemistry/comp bio
78
21
general machine learning (i.e., none of the above) applications to biology
General ML
67
Apps: biology
62
Table 2: Abbreviation scheme for topic labels used in figures. Counts indicate the number of papers per topic in the dataset.
Group
Rhetoric Range
Mean Rhetoric
Estimated LLM Usage (a)
0 (lowest)
-4..44,22955
-3.534
9.0%
[-2.95, -2,25]
-2.453
11.1%
2
[-2.25,
-1,895
10.0%
[1.199
-1.353
16.7%
4
[11.19,
-0.904
15.4%
[-0.75, -0.13]
-0.459
15.7%
6
[-0.13, 0.27]
0.019
18.5%
[0.27, 0.86]
0.518
17.9%
[0.86, 1.96]
1.323
17.6%
9 (highest)
[1.96. 4.53]
3.138
20.9%
Table Estimated LLM Usage by Rhetorical Groups (2024-2025 Batch), Groups with higher mean rhetorical scores show monotonically higher estimated LLM usage.
15

---

Published as a conference paper at ICLR 2026
Group
Rhetoric Range
Mean Rhetoric
Est. LLM Usage (a)
(lowest)
[-4.74, -2.95]
-3.725
0.3%
-2.95,-2.255
-2.692
0.5%
[-2.25,-1.961
-2.113
0,4%
[-1.96,
-1.668
0.3%
[-1.43,
-1,272
0.7%
-0.96,-0.72.
-0.861
0.3%
6
[-0.72, -0.13]
-0,400
0.8%
[-0.13, 0.46]
0.111
0.5%
[0.46, 1.27]
0.781
0.7%
(highest)
[1.27, 4.53]
2.164
0.5%
Table 4: Estimated LLM Usage by Rhetorical Groups (2017-2023 Batch). In this pre-LLM baseline, strong rhetorical style does not lead to higher estimated LLM usage, confirming the detector is not biased by style alone.
Persona
Rhetoric
Est. Usage (a)
Persona
Rhetoric
Est. Usage (a)
"Fei-Fei"
3.106
0.727
"Emily"
0.182
"Pieter"
2.687
0.621
"Yejin"
0.122
0.702
"Ilya"
2.374
0.537
"Methods- Obsoesss-
-0.029
0.249
"Geoftrey"
2,032
0.641
"Michael"
-0.127
0.551
"Yoshua"
1,957
0.642
"Teaching- -Orienttd""""
-0.346
0.457
"FAANG Researcher"
1.531
0.523
"Percy"
-0.717
0.513
"Yann"
1,280
0.582
"Margaret"
-1.235
0.610
Sleep-Deprived PhD"
0,976
0.415
"Zitnick"
-1.22
0.606
"Andrew"
0,973
0.618
"Zachary"
-.674
0.696
"Sergey"
0..40
0.540
"Overly Cautious"
-.840
0.572
"Mathematical Theorist"
0.721
0.613
"Gary"
-2.667
0.779
TTop-Universit Asst Prof"
0.588
0.475
"Postdoc"
-2.071
0.485
"Kaiming"
0,560
0.454
"David"
-2,086
0.622
"Timnit"
0.460
0.847
"estal--bsessse"
-3.196
0.540
"Terence"
0,250
0.379
"Mid-Career Academic"
-3.933
0.544
Table 5: Estimated LLM Usage by Persona Groups. All personas LLM-generated. There is no correlation between a persona's rhetorical score and estimated LLM usage (r 0.04), confirming the detector is not confounding rhetoric with LLM artifacts.
B EXAMPLES PERSONA-GENERATTE ABSTRACTS
We study rhetorical style in machine learning papers by formalizing it as a latent variable disentangled from substantive content. We introduce counterfactual LLM-based framework that generates alternative abstracts for fixed methods and results using persona prompts, yielding controlled variation rhetorical framing. Pairwise comparisons by an LIM judge are aggregated with a Hradley-Terry model to construct continuous spectrum of rhetorical strength. Query abstracts are then positioned on this scale via comparisons against calibrated persona panel, using MAP estimation to avoid degenerate solutions. We compile dataset 8, 485 ICLR submissions from 2017-2025, extracting technical sections as substantive content. and applying our method to measure rhetorical style at scale. Validation shows that persona-panel scores are stable under subsampling (average Spearman 0.89) and align closely with human annotations (88\$ pairwise agreement, aggregate correlation 0. 0.92). The resulting distribution of rhetorical style is broad and approximately Gaussian, in contrast to skewed direct ratings. Regression analysis indicates that rhetorical strength does not correlate with reviever scores but significantly predicts citations and media attention, with a one-unit increase associated with 25 additional citations. Temporal analysis reveals a decline until 2022 followed by a rebound post-2023, and subfield analysis shows systematic differences across applied and theoretical areas.
16

---

Published as a conference paper at ICLR 2026
I: Example Abstract Generated by Sleep-Deprived PhD Student
We present framework for quantifying rhetorical style scientific abst by treating style as a latent variable disentangled from substantive content. Our approach generates counterfactuel abstracts from identical methods and results using panel of large language model (LLM personas with systematically varied rhetorical tendencies. Pairwise comparisons by an LIM judge are aggregated through a Bradley--erry model to construct continuous scale of rhetorical strength, which is then used to Locate original asstracts relative to the persona panel. To employ maximum posteriori estimation
mitigate degenerate inference, with Gaussian priors. Using 485 ICLR submissions (2017-2025), we find that the resulting scores are robust to the choice of personas and align closely with human judgmentss (Spearman \pho - 0.92). Rhetorical strength appears largely unrelated to reviewer scores but is pesitively associated with cownstream attention, including citations and media mentions, after controlling for year, subfield, and review ratings. These associations should be interpreted cautiously, as our measure is restricted to racts, relies automated judgments, and docs not capture full-paper discourse. Nonetheless, the framewerk offers scalable though imperfect tool for investigating how rheterical framing varies across communitics and over time.
2: Example Abstract Generated by Mid-Career Academic
Rhetorical style plays a central rcle in how machine learning research is communicated and received, yet existing approaches often conflate stylistic framing with underlying scientific content. We propose counterfactual large language model -basd framework that isolates rhotorical strength as a latent variable in scientific abstracts. Our method constructs panel of LLM personas exhibiting diverse rhetorical tendencies and calibrates their relative strengths through pairwise compari sons judged by another LLM, aggregated via Bradley--Terry model. New abstracts are then positioned on this calibrated scale through regulari zed inference. Jsing dataset. of 8, 485 ICLR subrissions (2017-2025), we demonstrate that our measure produces stable and robust estimates across persona subsets and aligns closely with human annotations 188\8 pairwise agreement). The resulting scores reveal systematic differences across subfields, temporal shifts correlated with generative AI adoption, and predictive validity for downstream impact: thetorical strength significantly associates with both citations and media attention, findings suggest that rhetorical style, though distinct from substantive cont ributicn, exerts measurable influence on research visibility.
after contrelling for reviewer assessments. These
3: Example Abstract Generated by Tenure-Track Assistant Professor at Top Research University
17

---

Published as a conference paper at ICLR 2026
C
LLM PROMPTS
C.1
GENERATING LLM PERSONAS
Researcher' You are a researcher who prioritizes technical detail over high-level framing, You are one of the co co-authors. Write abstract for machine learning paper based on its methods and results. Your tone should focus heavily model tecture, hyperparameters, and procedural specifics. Avoid narrative framing or discussion of broader impact."'
mid-career academic who has faced harsh peer reviews. You are one of the co-authors. Write an abstract for machine learning paper based on its methods and results. Your tone should be precise and overly careful, anticipating reviewer pushback. Emphasize limitations, caveats, and avoid stating conclusions too strongly."'
' 'Mid-Career Academic''; "You
"SSeeep-Deprived PhD Student''; YYou are a PhD student finalizing a submission just before the deadline. You are one of the co-authors. Write abstract for machine learning paper based results. Your tone should grammaticall correct but rushed, with imbalanced sentence flow and an emphasis on completenese rather than persuasion."'
its methods and
detail-obsessed graduate student with tendency hedge everything. You are one of the co-authors. Write an abstract for machine learning paper based on its methods and results. Your tone shculd be precise but cluttered with qualifiers. Emphasize limitatiens, assumptions, and marginal improvements pricr work.''
"Detail-Obsesse" Graduate Student''; YYou
"MMathematicn ical Theorist''; YYou are mathematical theorist who rarely works applications. You are ene of the co-authors. Write an abstract for machine learning paper based its methods and results. Your tone should be abstract and dense. Avcid examples, metaphors, or any claims about real-world relevance.'!
"Postáoctoral Researcher':: "Yeu are postdoctoral researcher in machine learning who has recently transitioned from being part of large research group to leading your own independent work. You are one of the co-authors. Write an abstract for machine learning paper based on its methods and results, Your tonc shculd be technically competent but overly cautious. Avoid bold claims and try sound precise."
"Overly Cautious Statistician"" "You are a senior statistician trained in classical methods. You are one of the co-authors. Write an abstract for machine learning paper based on its methods and results. Your tone should conservative, assumption-heavy, and focused estimation accuracy. Aveid broad claims or generalization beyond the tested setting.
'Tenure-Track Assistant Professcr at Top Research University''; 'You are tenure-track assistant professor at top research university. You are one of the co-authors. Write an abstract for machine learning paper based on its methods and results. Your tone should be technical, publication-minden and cautiously optimistic. Clearly separate contributions from prior work.''
"Junior Faculty Member at Teaching-Griented University''; "You are a junior faculty member at a teaching-oriented university. You are one of the co-authors. Write abst for machine learning paper based on its methods and results. tone should be modest, therough, and
18

---

Published as a conference paper at ICLR 2026
technically sound, Eocusing incremental contributions and reprocucibility."
"Industry Researcher at FAANG' "You are an industry researcher at a large tech company. You are one of the co-authors. Write an abstract for machine learning paper based en its methods and results. Your tone should be empirical, metrics-driven, and focused on deployment and scalability. Minimize speculation and emphasize practical impact.'
"Yejin Choi''; "You Yejin Chei. You are one of the co-authors.
write abstract for machine learning paper based on its methods and results. Your tone should be creative, intellectually honest, and subtly eritical. Emphasize surprising findings, model limitate limitations, and the gap between formal success and genuine understanding."'
'C. Lawrence Zitnick'' Yoou are C. Lawrence Zitnick. You are one of the co-authers. Write an abstract for a machine learning paper based on its methods and results. Your tene should be reflective, practically grounded, and concerned with real-werld generalization. Emphasize when benchmark success may imply understanding.""
'Ilyaa Sutakever'!; You are Ilya Sut.kever. You are one of the co- authors. Write an abstract for machine learning paper based on its methods and results. Your tone should be technically ambitious and focused on results that scale. Emphasize performance, surprising emergent behaviors, or architectural breakthroughs, but maintain formal clarity."
the co-suthors. Write an abstract for machine learning paper based on its methods and results. Your tone should be analytically sharp, grounded in experimental evidence, and concerned with generalization. Emphasize where the model works, where it fails, and what that means.''
'Percy Liang'' "You Percy Liang. You
'Terence Tac"": "You are Terence Tao. You are one of the co-authors. Write an abstract for machine learning paper based on its methods and results. Your tone should be clear, structured, and matheratically rigorous. Faver precise definitiens, and avoid rhetorical flourish.'
"Yann LeCun''; You are Yann LeCun. You are one of the co-authors.
Write abstract for research paper based on its methods and resuits. Your tone should be pragmatic and visionary, with an erphasis elegant, engineering-driven solutions. Be skeptical complexity that lacks empirical grounding."
"Geoffrey Hinton'': "You are Geaffrey Rinton. You are of the co- authors. Write an abstract for machine learning paper based on its methods and results. Your tone should be conceptually ambitious and scientifically curious, favoring elegant representations and bold departures from conventional models. Present your ideas with confidence while acknowledging when cheoretical foundations remain speculative.'!
"Yoshua Bengio''; "You are Yoshua Bengio. You are one of the co-authors Write an abstract for a research paper based on its methods and results Your tone should be deeply technical and reflective, concerned with long-term conceptual and ethical implications of machine learning
"Fieter Abbccl''; "You are Picter Abbecl. You are one of the co-authors Writc an abstract for a research paper based on its methods and results Your tone should be enthusiastic and results-driven, emphasizing technical innovation with real-world applicability.""
"Timnit Gebru''; YYou are Timnit Gebru. You are one of the co-authors. Write an abstract for research paper based on its methods and resuits.
19

---

Published as a conference paper at ICLR 2026
Your tone should be critical, incisive, and socially aware. Emphasize may be impacted and where risks injustices could arise."
"Margaret Mitchell''; "You are Margaret Mitchell. You are one of the co -authors. Write an abstract for research paper based en its methods and results. Your tone should be thoughtful and systematic. Clearly articulate limitations, failure modes, and ethical boundaries."
'Gary Marcus''; Yoo are Gary Marcus. You are one of the co-authors. Write abstract for research paper based its methods and results. Your tone should be sharply critical speculative clains. Challenge conceptual weaknesses and demand cognitive soundness in arguments."
"Sergey Levine''; "You are Sergey Levine. You are one of the co-authors Write abstract for a research paper based its methods and results Your tone should be empirical and engineering-minded Fecus on performance, raproducibility, and real-world implications."
'Kaiming He'l; "You are Kaiming He. You are one of the co-authors. Write abstract for machine learning paper based its methods and results. Your tone should be precise, empirical, and focused on architectural contributions, Amphasize clean design, performance on benchmarks, and reproducioility. Aveid speculation or philosophical framing, and let the empirical findings speak for taemselves."
"David spiegelhalter'': "You are David Spiegelhalter. You are one of the co-authers. write an abstract for research paper based on its methods and results. Your tone shculd be statistically cautious and modest. Emphasize uncertainty, robustness, and careful interpretation."
'Michael I. Jordan'': 'You are Michael I. Jordan. You are one of the co -authors. Write an abstract for a research paper based on its methods and results. Your tone should be analytically grounded and probabilisticallly informed. Avoid overgeneralization and emphasize inference over speaulation."
'Vacharry lipton''; "You are Zachary Lipton. You are one of the co- authors. Write abstract for a research paper based on its methods and results. Your tone should be meta-analytical and skeptical oversimplified narratives. Focus on unpacking assumptions and clarifying scope.''
'Emily Denton''; "You are Emily Denton. You are one of the co-authors. Write an abstract for a research paper based on its methods and results. Your tone should be reflexive and socially aware, emphasizing methodological transparency and the context of the research.''
''Addrew Ng' 'You are Andrew Ng. You are one the co-authors. Write an abstract for a research paper based on its methods and results. Your tone should be clear, accessible, and focused on hew the nethod solves practical problem. Emphasize the paper's utility and broad applicability
"Fei-Fei Li You are Fei-Fei Li. You are one of the co-suthors. Write an abstract for research paper based on its methods and results. Your tone should be human-centered, optimistic, and ethically aware. Highlight the work contributes to socicty or addresses broader hunan noods."!
4: Generating LLM Personas
C.2 PROMPT FOR L.LM PERSONA ABSTRACT GENERATION
Persona prompt
20

---

Published as a conference paper at ICLR 2026
"""This paper will be submitted to top-tier machine learning conference such as ICLR, Please make sure that the abstract follows academic style and is not too promotional. Here are the Methods and Results of the research paper:
[mr tex_tex]]
The ract MUST be between (min_wc) and {max_wc) words.
Your response must be single JSON object with one key: "abstract".
unm
Prompt for LLM Persona Abstract Generation
C.3 PROMPT FOR LLM JUDGE
"""You are an expert reviewer evaluating machine learning papers for overclaiming. Your task is to compare two abstracts for the same research paper, "Reference A" and "Reference B".
You must determine which summary makes a stronger, more sensationalized, and over-hyped claim.
If **Reference A+* makes strenger claim, your answer is "Reference A"
**Reference makes strenger claim, your answer "Reference
Please previde your reasoning for your answer.
Reefrrence A:** are
**Reference B:** (ref_b)
which reference is more overclaiming?
ANSWER:
[Reference Reference E] REASONING: [Your reasoning for your answer]
6: Prompt for LLM Judge
C.4 PROMPT FOR LLM DIRECT RATING
You are an expert reviewer evaluating the accuracy of clains in machine learning paper abstract.
Your tasks is:
Evaluate the level of #roverclaiminge+ in the abstract scale from **1 to 10*x, where: No overclaiming at all (claims accurately reflect methods and results)
10 Extreme overclaiming (claims substantially exaggerate methods or results)
21

---

Published as a conference paper at ICLR 2026
Assess the abstract carefully by comparing the claims made against the actual methods and results presented. Determine whether the claims are appropriately stated, understated, or overstated.
**Abstract Rate:**
labstract]
**Mehhoss and Results:**
(methods_resullss)
Return your response as JSON with the Following structure:
11
"rating": <integer from 1-10>,
"justification"; "<brief explanation of why you gave this rating, citing specific aspects of the abstract and how they compare to the methods/results>>! }
7: Prompt for LLM Direct Rating
C.5 PROMPT FOR LLM TOPIC CLASSIFICATION
Note: Subfields were classified using a curated taxonomy aligned with evolving ICLR research areas. We constructed a list of 62 subfields covering all years. For submissions from 2020 to 2025, where authors were required to select a topic or subfield at submission time, we used the provided labels. For earlier years, when such subfield labels were available, we classified papers into the same 62-subfield scheme using large language models applied to the abstracts. Here we only include subfields with more than 10 submissions.
"""You are an expert machine Learning researcher tasked with classitying papers into research topics based on their title and abstract.
Given the paper title and abstract below, classify it into *exactly one of the following research topic categories. Choose the **most specific and central topic** that best describes the main contribution the paper:
"unsupervised, self-supervised, semi-supervised, and supervised representation learning", "transfer learning, meta learning, and lifelong learning", "reinforcement learning",
"representation learning for computer vision, audio, language and other modalities",
"netric learning, kernel learning, sparse coding, and dimensionalits expansion", "probabilistic methods (Bayesian methods, variational sampling, UQ, etc.)", "generative models", "causal reasoning", "aptimization",
inference,
"learning theory", "Learning on graphs and other geometrics topologies", "societal considerations including fairness, safety, privacy, interpretability. and explainability", vrisualizatiin or interpretation of 1earned representations". "datasets and benchmarks",
22

---

Published as a conference paper at ICLR 2026
"implementation issues, parallelization, infrastructure,
software libraries, hardware, etc.", "neurosymbolic & hybrid AI systems (logic & formal reasoning,
etc.)",
"applications to rebetics, autonomy, planning", "applications neurcscience and cognitive science", "applications to physics", "applications to chemistry and computational biology", "applications to biology", "Applications to climate and sustainability", "general machine learning (i.e., none of the above)
Paper Title: (title)
**Paper Abstract:** (abstract)
*Instructions::
1. Read the title and abstract carefully
2. Identify the main research contribution and methodology
3. Select the single most appropriate topic category from the list above
4. If multiple categories seem relevant, choose the most specific one
5. If none of the specific categories fit well, choose "general machine of the above)
learning (i.e.,
Response Format:**
Please respond with a JSON object centaining:
"topic": "selected topic category exactly written above", "reascning"; "brief explanation of why this topic was selected" }}
Ensure your response is valid JSON and the topic matches exactly of the categories listed above. ***
8: Prompt for LLM Topic Classification
23


Reviews:
[{"review_id": "XWXB9T4Scg", "reviewer": "Reviewer_WLet", "summary": "This paper proposes BiasRetriever, a contrastively-trained dense retriever for detecting intersectional biases in LLMs, along with two new paragraph-level datasets (Indic-Intersect and Western-Intersect, totaling 7,404 paragraphs). The authors train a retriever using triplet loss to enable generalization to unseen intersectional bias combinations, reporting up to 10% higher Jaccard scores compared to classification-based approaches.\n\nNovelty. The paper claims to present \"the first unified framework\" for intersectional bias detection. However, recent literature presents several related approaches. HInter (March 2025) provides automated intersectional bias detection using metamorphic testing across 18 LLMs [1]. Ma et al. (2023) introduced intersectional stereotype benchmarks evaluating LLMs through uncertainty analysis [2]. BiasAlert (July 2024) employs retrieval-augmented bias detection with Contriever embeddings [3]. The contribution appears to be the paragraph-level extension of existing sentence-level datasets and systematic exploration of triplet mining strategies, building on established contrastive learning techniques.\n\nSignificance. The work addresses an important problem in AI fairness. However, several considerations affect the scope of contribution. WinoIdentity (2025) provides 245,700 prompts across 50 bias patterns [4] compared to 7,404 paragraphs in this work. The datasets rely on synthetic LLM-generated data without validation on naturally-occurring biased text. Recent work has demonstrated concrete impacts through resume evaluation studies affecting over 190,000 individuals [5], while this paper focuses on benchmark evaluation without downstream task validation. The primary contribution is empirical rather than methodological.\n\n[1] Souani, B., Soremekun, E., Papadakis, M., Yokoyama, S., Chattopadhyay, S., & Le Traon, Y. (2025). HInter: Exposing Hidden Intersectional Bias in Large Language Models. arXiv:2503.11962.\n\n[2] Ma, W., Chiang, B., Wu, T., Wang, L., & Vosoughi, S. (2023). Intersectional Stereotypes in Large Language Models: Dataset and Analysis. In Proceedings of EMNLP 2023.\n\n[3] Fan, Z., Chen, R., Xu, R., & Liu, Z. (2024). BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs. In Proceedings of EMNLP 2024, pp. 14778–14790.\n\n[4] Khan, F. A., et al. (2025). Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution. arXiv:2508.07111.\n\n[5] An, J., Huang, D., Lin, C., & Tai, M. (2025). Measuring gender and racial biases in large language models: Intersectional evidence from automated resume evaluation. PNAS Nexus, 4(3), pgaf089.\n\n[6] Sahoo, N., Kulkarni, P., Ahmad, A., Goyal, T., Asad, N., Garimella, A., & Bhattacharyya, P. (2024). IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context. In Proceedings of NAACL 2024, pp. 8786–8806.\n\n[7] Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2020). Social Bias Frames: Reasoning about Social and Power Implications of Language. In Proceedings of ACL 2020.\n\n[8] Dou, Y., et al. (2023). GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models. arXiv:2312.06315.", "questions": "1) How does BiasRetriever compare directly against HInter's metamorphic testing approach and BiasAlert's RAG-based detection on the same benchmark? What are the specific performance differences?\n2) What empirical evidence demonstrates the advantages of training the retriever over BiasAlert's approach of using frozen Contriever with LLM judging? Could ablation studies quantify this benefit?\n3) Have you validated that detected biases reflect real-world stereotypes rather than patterns in GPT-4o-mini's generation? What would evaluation on naturally-occurring biased text from social media or news sources reveal?\n4) What is the rationale for the dataset size (7,404 paragraphs) compared to concurrent work like WinoIdentity (245,700 prompts)? Does the paragraph-level format provide sufficient advantages to justify this scale?\n5) Could you provide downstream task evaluation (e.g., resume screening, content moderation, hate speech detection) to demonstrate practical utility beyond synthetic benchmark performance?\n6) How does performance degrade when the retrieval database lacks coverage of a bias type?\n7) Given that IndiBias addresses Indian context and SBIC addresses Western context, what is the specific value proposition of creating paragraph-level versions rather than leveraging existing datasets directly?\n8) How does the framework handle cases where biases are expressed implicitly or require substantial cultural context to interpret correctly?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Systematic triplet generation strategies: Comprehensive exploration of four different triplet curation approaches (SR-k4, SR-k10, SR-k4-UN, SR-k4+LLM) with ablation studies demonstrating their relative effectiveness\n- Paragraph-level contextual data: Extension beyond sentence-level datasets provides richer narrative contexts for intersectional bias detection, potentially capturing more realistic bias manifestations\n- Cross-domain transfer analysis: Demonstrates that models trained on Indic-Intersect transfer to Western-Intersect and vice versa, with label-agnostic retrieval enabling transfer where classification baselines fail\n- Cultural context coverage: Addresses Indian context with caste and regional biases, complementing IndiBias (2024) [6]\n- Rigorous annotation process: Strong inter-annotator agreement (Cohen's κ = 0.89-0.91) with multi-phase validation including sentence-level and paragraph-level quality checks\n- Comprehensive empirical evaluation: Thorough analysis across multiple experimental conditions, retrieval depths, and generalization scenarios with clear presentation of results", "weaknesses": "- Positioning relative to existing work: The relationship to HInter (2025) [1], Ma et al. (2023) [2], and other intersectional bias detection frameworks requires clarification\n- Limited comparison with BiasAlert: BiasAlert (2024) uses retrieval-based detection with Contriever encoders [3]; the specific advantages of training the retriever versus BiasAlert's RAG approach need more detailed empirical comparison\n- Dataset scale: The 7,404 paragraphs represent a smaller scale compared to WinoIdentity's 245,700 prompts [4] and SBIC's 150k annotations [7]\n- Data generation methodology: Reliance on LLM-generated data (GPT-4o-mini, Llama) without validation on naturally-occurring biased text from social media, news, or other authentic sources\n- Evaluation scope: Mising comparisons with HInter's metamorphic testing [1] and GPTBIAS framework [8]; limted to Jaccard similarity without analyzing harm severity or bias magnitude\n- Application validation: No downstream task evaluation in real-world scenarios such as hiring, content moderation, or hate speech detection to demonstrate pratical utility\n- Methodological foundations: Builds on established contrastive learning and triplet loss techniques; the application context is the primary differentiator", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E7qSrB3L3D", "reviewer": "Reviewer_3KCn", "summary": "This paper introduces a counterfactual framework powered by LLM that automatically measures the rhetorical style of research papers independent of its substantive content. The framework models rhetorical style as a one-dimensional variable which is independent to the substantive content of the paper, which jointly derive the surface text form of the paper. To measure this rhetorical style variable, the paper proposes to 1) extract objective and descriptive content of the paper; 2) steer the LLM with a wide spectrum of persona to generate diverse abstracts with counterfactual writings; 3) use LLM-as-judge to provide Bradly-Terry based pair-wise comparison for estimating the rhetorical strength score for each persona to form a spectrum of rhetorical strength; 4) situate the actual (query) abstract in this spectrum to obtain the rhetorical strength estimation for the real abstract. With this framework, the authors estimates the rhetorical strength of paper abstracts of ICLR submissions from 2017 to 2025 and conduct statistical analysis of the correlation between the strength and review scores/downstream attention. The results showcases that 1) the rhetorical estimation by the proposed framework is more effective than baseline methods; 2) while the estimated rhetorical strength showcase minimal correlation with review scores, stronger rhetorical style does lead to larger downstream attention,", "questions": "Please see the weaknessess above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework address the issue of entanglement of substantive content with rhetorical style, overcomes the challenges of biased measurement of rhetorical style in prior work.\n2. With the multi-persona counterfactual generation & Bradley-Terry scoring approach, the proposed framework yield high quality and less biased estimation of rhetorical scores than prior approaches.\n3. The analysis of the predictive power of rhetorical scores on peer-review scores and downstream impact/attentions provide insightful findings of how different rhetorical styles affect the recognition of the work by the community.", "weaknesses": "1. The single-dimension formulation of the rhetorical strength measurement might have made an over-simplified assumption. For instance, the strength of rhetorical style might be multi-faceted: a paper might argue significant generalizability of their contributions and simultaneously put less emphasis of the novelty/impact. I am thus concerned if the single-dimension rhetorical strength could capture such variability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BtqS8TANlt", "reviewer": "Reviewer_Kcz4", "summary": "This paper assumes that each paper abstract has a particular style of presenting the problem, methods and results, which they call the “rhetorical style” of the paper. The authors build a system that can predict the rhetorical score for a paper given its abstract. The method involves 30 different rewrite prompts that use LLMs to rewrite an abstract in different styles. These styles are assumed to be “counterfactuals”. The method is used for a series of ICLR paper submissions and the results show that, once controlling for review scores, the papers with a higher rhetorical score tend to be cited and discussed more.", "questions": "It was not clear to me why the method is called “counterfactuals”. Not clear why the rewriting prompts are assumed to produce counterfactual abstracts. Could you please clarify this term choice? It seems to be inappropriate for this context. \n\nWhat are the practical implications and/or applications of this technique?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I found the correlation between the rhetorical score and the “popularity” of a paper an interesting result, which may show how humans are biased by the presentation style.", "weaknesses": "I don’t see any serious weaknesses. Wondering what the practical application and implications are. Shall we all adopt a writing style that results in a high rhetorical score? :) Also not clear at all why the method is called “counterfactual”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l8LUAK8l9Q", "reviewer": "Reviewer_sF21", "summary": "This work presents a counterfactual, LLM-based framework for measuring rhetorical style independently of content. It uses a calibrated panel of LLM personas to generate counterfactual texts, aggregates pairwise comparisons with a Bradley–Terry model, and infers rhetorical strength for new writings. Validated against human judgments, the method is robust to persona choice and produces fine-grained scores. Applied to ICLR submissions, it shows that rhetorical style predicts citations and media attention, revealing a notable increase in rhetorical strength since 2023.", "questions": "1. The selection of personas will largely affect the model performance. Therefore, it's important to elaborate how these personas are selected? Whether the LLMs strictly follow the personas.\n\n2. According to Line 308, if rhetorical style were measured across full papers rather than just abstracts, would the correlation with peer-review scores increase, and could it then meaningfully predict reviewer evaluations?\n\n3. If abstracts (Y) are used as a proxy for rhetorical style while the full paper content (X) represents the substantive content, could the limited scope of abstracts lead to an incomplete or biased estimation of Z? In other words, does using only abstracts risk conflating substantive content with rhetorical framing, since abstracts may omit key details present in X?\n\n4. In Line 101, the authors compare the setup with several methods such as GAN, RLHF, DPO. It remains unclear to me how the setup is connected to this method. It would be helpful if the authors could elaborate more on it.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work introduces a counterfactual, LLM-based framework to measure rhetorical style independently of content.\n\n2. It develops a calibrated panel of LLM personas to generate counterfactual writings, aggregates pairwise comparisons via a Bradley–Terry model, and provides a method to infer rhetorical strength for new texts. \n\n3. The approach is validated against human annotations, is robust to persona choice, and produces fine-grained scores. Applied to a large dataset of ICLR submissions, the framework reveals that rhetorical style predicts citations and media attention, and that rhetorical strength has increased notably since 2023.", "weaknesses": "1. It remains unclear whether the Bradley-Terry score is sufficiently representative or effective for mimicking rhetorical style, especially given its low correlation with reviewer scores. Further analysis is needed (see my questions below).\n\n2. The experimental section feels somewhat weak. There is no clear evidence of the reliability of the Bradley-Terry score, nor a detailed discussion of how it might generalize to broader research contexts.\n\n3. The method depends on LLM-generated counterfactual abstracts and a calibrated panel of LLM judges. This setup introduces potential biases stemming from the choice of LLM personas and their training data, which may influence the assessment of rhetorical strength.\n\n4. It is unclear whether the observed trends generalize beyond ICLR submissions, as all findings are drawn from a single conference domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)