Latent Principle Discovery for Language Model Self-Improvement
Authors: Keshav Ramji
Tahira Naseem
Ramón Fernandez Astudillo
Venue: NeurIPS 2025 poster ()
Keywords: self-improvement
language models
self-correction
principles
latent chain-of-thought
Primary area: deep_learning
TL;DR: We discover model-generated constitutions and train language models to intrinsically self-correct their responses by using these principles; repeating this process iteratively enables self-improvement.

Abstract: When language model (LM) users aim to improve the quality of its generations, it is crucial to specify concrete behavioral attributes that the model should strive to reflect. However, curating such principles across many domains, even non-exhaustively, requires a labor-intensive annotation process. To automate this process, we propose eliciting these latent attributes that guide model reasoning toward human-preferred responses by explicitly modeling them in a self-correction setting. Our approach mines new principles from the LM itself and compresses the discovered elements to an interpretable set via clustering. Specifically, we employ a form of posterior-regularized Monte Carlo Expectation-Maximization to both identify a condensed set of the most effective latent principles and teach the LM to strategically invoke them in order to intrinsically refine its responses. We demonstrate that bootstrapping our algorithm over multiple iterations enables smaller language models (7-8B parameters) to self-improve, achieving +8-10\% in AlpacaEval win-rate, an average of +0.3 on MT-Bench, and +19-23\% in principle-following win-rate on IFEval. We also show that clustering the principles yields interpretable and diverse model-generated constitutions while retaining model performance. The gains that our method achieves highlight the potential of automated, principle-driven post-training recipes toward continual self-improvement.

Latent Principle Discovery for Language Model Self-Improvement
Keshav Ramji", Tahira Naseem, Ramón Fernandez Astudillo IBM Research Al
Abstract
When language model (LM) users aim to improve the quality of its generations, it is crucial to specify conerete behavioral attributes that the model should strive to reflect. However, curating such principles across many domains, even non- exhaustively, requires labor-intensive annotation process. To automate this process, we propose eliciting these latent attributes that guide model reasoning toward human-preferred responses by explicitly modeling them in self-correction setting. Our approach mines new principles from the LM itself and compresses the discovered elements an interpretable set via clustering. Specifically, we employ form of posterior-regullrized Monte Carlo Expeetation-Maxiiizatii both identify a condensed set of the most effective latent principles and teach the LM to strategically invoke them in order to intrinsically refine its responses. We demonstrate that bootstrapping our algorithm over multiple iterations enables smaller language models (7-8B parameters) to self-improve, achieving +8-10% in AlpacaEval win-rate, an average of +0.3 on MT-Bench, and +19-23% in principle- following win-rate on IFEval. We also show that clustering the principles yields interpretable and diverse model-generated constitutions while retaining model per- formance. The gains that our method achieves highlight the potential of automated, principle-driven post-training recipes toward continual self-improvement.
Introduction
Modern language models (Grattafiori al., 2024; OpenAI, 2024; DeepSeek-Al et al., 2025) have achieved striking fluency and coherence in open-ended generation, yet guiding them to satisfy multiple, possibly overlapping numan-defined criteria remains core challenge. Conventional approaches to align language models (LMs) rely on human annotations distinguishing between chosen and rejected generation, even when their gap in quality may be nuanced and multi-faceted Constitutional AI and other related paradigms (Bai et al., 2022b; Guan et al., 2025) consider a human- curated "constitution" of high-level attributes which models' responses should follow. While these frameworks enable models to be steered toward safer behavior, the static nature of their constitutions requires experts to anticipate nuances advance and update rules manually as edge cases surface. As use cases proliferate, new failure modes arise reliably synthesizing task-specific "amendments" and collecting annotations a costly and time-consuming process leading to brittleness and limited adaptability. We aim to automate the process of discovering these attributes for model improvement, obviating the need for human intervention or explicit domain adaptation.
Automatically discovering behaviors for ielrovmmpoeeeettt can be seen as meta-level reasoning process. Recent efforts to induce reasoning capabilities in LMs have often focused on domains such math and code where gold reference answer exists and candidate answers are more easily verifiable (DeepSeek-AA, 2025). The availability of verifiable responses has also been capitalized for teaching self-correction (Kumar et al., 2025). However, in this work, we focus on open-ended text
*Correspondence to keshav ramji (ammj01: dim.. com.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Model M
AlpacaEval Win-rates with SaaPLe
Principle Discovery (E (t-Stsp)
Input Corpus Xp 1f
10
Cluster
Principle based Trajectories 2,
StaPLe Model
30
Principles
34
AlpacaFuni
17
No. 2
Principla (M-Steg)
28
Liwne-. 2----immme
Grante 48 I129 Owwn2 invtrut
STaPLe Constitution
STaPLe Model
Figure 1: We introduce Self-Taught Principle Learning (STaPLe). (Left) Our Monte Carlo EM algorithm alternates between on-policy discovery and learning of latent principles guiding self- correction behavior. The principles may also be clustered to compressed set, yielding human- interpretable constitutions C2 and models trained to follow them M,. (Right) The STaPLe algorithm induces self- improvement in AlpacaEval win-rate three iterations for all three language models.
generation tasks that are challenging to verify; identifying situations for a human to intervenc and induce refined response can be especially tricky in such cases.
We introduce a novel approach to discover expressive principles, treating them as latent attributes in the self-correction setting to bridge an initial attempt and a target response. We find that the language model itself serves as an effective principle generator to improve its responses, contrasting prior works which rely on human-defined principles strong model supervision. We design Expectation- Maximization algorithm, Self-Taught Principle Learning (STaPLe), which first leverages rejection sampling to identify principle candidates for self-correction and choose the induced refinement that closest to the gold, then trains over these trajectories to learn this principle-gus guided refinement behavior. Iteratively applying this method results in model trained elements produced from itself, implicitly learning the refinement goal to enable its self-correction abilities at inference-time. We also show that the discovered principles can be compressed smaller set for human readability by applying hierarchical clustering through posterior regularization, without compromising downstream performance.
dynamic constitution of
validate the efficacy our method over several iterations on cnonuction-follwiing benchmarks including MT-Bench (Zheng et al., 2023) and AlpacaEval (Li et al., 2023), and leverage Prometheus- v2.0 (Kim et al., 2024) to analyze win-rates with gre-grained, principle-followee show that STaPLe outpaces baseline methods such as Self-Taught Reasoner (STaR; Zelikman et al. (2022)) (adapted for non- verifiable responses) and prompted refinement approaches like Self-Refine (Madaan et al., 2023). It continues to self- self-improve in performance over multiple iterations, before saturating. We also find that clustering largely matches or outperforms training on all principles.
rubrics. Our results
Our key contributions can be summarized as follows:
We propose a Monte Carlo Expectation-Mil Maximization (EM) algorithm for iterative latent principle discovery and learning. enable language model self ceff-improvement.
We find that on-policy generated principles effective stimuli for self-correction in smaller LMs, and training to learn them improves the performance on MT-Bench, AlpacaEval--2.0
Clustering the set discovered principles prior to training retains the performance gains of the full distribution while yielding interpretable constitution.
2 Related Work
Principle-Driven Language Modeling Early work in principle-driven alignment demonstrated that embedding high-level rule sets or "constitutions" into the training loop can steer model behavior without direct human labels for each generation. Constitutional AI (Bai et al., 2022b) introduced

---

two-stage process in which a pre-trained model first generates critiques of its own outputs against a static constitution curated and synthesized a priori and learns from these critiques. then trains a preference model from on-policy data and performs RI. Dromedary (Sun et al., 2023) extended this idea by introducing Self-Align, an algorithm which generates prompts, applies a small set of human written principles with in-context learning. and then fine-tunes the model to learn the principle-ge -uiided responses; this was later extended by SALMON (Sun et al., 2024) to design an instructable, principle- following reward model (RM). ConstitutionalExperss (Petridis et al., 2024) and SPRI (Zhan et al., 2025) address the problem of mapping prompts to principles. Deliberative Alignment (Guan et al., 2025) introduces CoT reasoning over safety specifications, and trains models to learn these reasoning traces via SFT and online RL with a safety RM.
More recent efforts have sought to leverage models to draft and refine constitutions with limited human supervision. LEAPP (Zhang et al., 2024) showed that models can propose new principles via self-critique given the gold response, synthesizing a list of task-specific principles that can be used for in-context learning. SAMI (Fränken et al., 2024) introduced a self-alignment mechanism where a strong model is used to generate small set of principle candidates, and the target model is trained to maximize the mutual information between the constitution and the model's responses through an InfoNCE loss. IterAlign (Chen et al., 2024b) and ICAI (Findeis et al., 2025) also leverage strong frontier models such as GPT-4/GPT-40 (OpenAI, 2024) for constitution proposal. The former uses red-teaming framework to identify model weaknesses, and uses the strong model to propose a principle towards helpfulness, harmlessness, and honesty; the latter considers principles as specifications over preference annotations, injecting them to reconstruct the annotation. Most recently, DeepSeek introduced a pointwise generative reward model (Liu et al., 2025) over self- self-generated principles, demonstrating that such RM can be successfully used to improve inference-time scaling.
Self-Correctio and Seff-Improvement The recent emergence of large reasoning models such as 03 and DeepSeek-RI (OpenAI, 2025; DeepSeek-Al, 2025) has lead to a growing exploration into the ability of models to perform intrinsic refinement of their own outputs with internal feedback. However, much of the prior literature below either trains models over the improved responses alone or performs prompted self-refinement at inference-time model-generatte critiques followed by revision, showing that alternating these two stages yields gains in instruction following (Zelikman et al., 2022). Self-Refine (Madaan et al., 2023) explored the setting of prompted multi-turn refinement: after producing an initial answer, the model is induced to critique itself, reflecting on the weaknesses of the current response on specific dimensions, proposing actionable changes, and reflecting them in the corrected response. ProMiSe (Ramji et al., 2024) extended this ability to smaller language models with weaker self-critique and refinement capabilities, showing that greedily refining responses relative to one attribute in a sequential manner improves performance, while demonstrating that training on synthetic dialogues modeling this self-refinement process improves dialogue question answering. APO (D' Oosterlinck et al., 2024) addressed the notion of minimally contrastive preference pairs, reinforeing the notion that revision along fewer attributes yields a better signal for preference optimization. Recently. reinforcement learning strategies have been explored train models to directly perform self-correction. RISE (Qu et al., 2024) frames self- correction through a multi-turn Markov decision process (MDP), performs best-of-N sampling over sequential attempt rollouts, and uses offline RL to train the model to correct over these trajectories. SCoRe (Kumar et al., 2025) improves over this by multi-turn RL formulation that boosts the quality of the initial attempt and leverages reward shaping to incentivize self-correction to improve the refined response. Beyond individual output corrections, recent work has shown that models can bootstrap their underlying capabilities in an iterative fashion over time. Several works suggest that sampling diverse responses or environmental interactions, filtering based on feedback or majority voting, and training on these on- on-policy generations can boost performance (Huang et al., 2023; Patel et al., 2024). SPIN (Chen et al., 2024c) introduced self-play fine-tuning approach, wherein the model compares its generations against the ground-trutt annotated responses in the SFT dataset yielding a preference pair, fine-tunes with a contrastive objective, and repeats this process iteratively. Huang et al. (2025) theoretically formalizes the self self-improvement phenomenon through a "sharpening" process, wherein the model's 's policy moves towards maximizing its generations' self-reward.
rather than learning this ability. STaR first leveraged
Latent Chain Thought Learning Chain-of-thought (CoT) prompting (Wei et al., 2023; Kojima et al., 2022) elicits an explicit, verbalized step-by-step walkthrough of the reasoning trace guiding the model from the input to the final response. Simultancously, STaR (Zelikman et al., 2022) leveraged
3

---

gold condiiioond rationalization of the CoT and fine-tuned the model to learn this reasoning behavior. The notion of rationalization as a latent variable modeling problem was previously explored in ELV (Zhou et al., 2020), under the framework of labeling examples with explanations through variational EM algorithm. More recent approaches also treat chain-of-thought as latent variable that may be trained over rather than purely to be induced at inference-time TRICE (Phan et al., 2023) casts intermediate reasoning steps as latent codes, training the model to marginalize over them so that it internally develops coherent reasoning trajectories, through Markov Chain Monte Carlo (MCMC) EM algorithm. LaTRO (Chen et al., 2024a) demonstrated that models can self-reward latent reasoning paths generating candidate thought sequences, scoring them by task success, and reinforeing the most effective ones through a variational framework. Concurrent work introduced BoLT (Ruan et al., 2025), showing that leveraging these implicit traces supervision leads to gains in data efficiency and performance for continued pre-training on complex reasoning benchmarks by converting latent chain-of-thought into self-supervised learning signal.
3 Self-Taught Principle Learning
We propose STaPLe, a self- -eflovmppoveetttt mechanism for (1) discovering human-interprette prin- ciples by the model itself, aimed towards response revision, and (2) training language models to intrinsically invoke such principles and subsequently perform self-refinement of their generations (if needed) inference time.
We view these principles as latent reasoning traces that bridge the gap between an initial model response and reference target. In the vein of the Self-Taught Reasoner (STaR) (Zelikman et al., 2022), we leverage the gold response as a "hint" to propose principles and guide response refinement decisions. However, our formulation is generic and allows for the of non-verifiable gold responses as hints. In particular, we use the proximity of the generated response to the reference response as a signal of correctness. Any similarity metric can be used to measure this proximity the exact match metric, used for verifiable rewards, can be seen as one such instantiation.
Given a dataset D {( yi y/ ())) where y is the gold response and y is model's initial response for the 7th sample, we aim to learn a latent response-improvement reasoning trace such that the probability of producing a response close to the gold reference maximized. The latent reasoning trace, or principle, 2 is also verbalized as natural language, i.e. discrete text tokens from vocabulary V We implement STaPLe to optimize the following marginal log- likeliiohhoo::
(()) = log f x, y = log Σ E-EV ZEV*
Σ
p(y |x,y' 2, 12) p(x2 x ;0
(1)
where y² is model refinement of the initial response yt generated with aid of latent principle 2. The distribution p(y& x. v² y²y²) acts as prespecified parameter-free validator model indicating the probability of the current revision y² matching the gold response yo This can be implemented in many different ways, e.g. LLLM-as-a-jddge but for the remainder of this work, it will be based on string similarity function f, with the rule
(1)2,10)) - f(vt, if f(r", otherwise.
f
ply
xx |,,y 2, v²
10,
We parametrize p(v?, 2 x, y.)) by the language model itself, with parameters f. The gradient for the STaPLe objective is:
Vo L(0) Vo log
Σ plu ,,,0,, p(u x. 00
VeE SEV-
P(US | 2,12)
Σ Σ
Vop(y? ,2 x1,00:0)
Ply x.y')
"TE"" =EV*
p I, 2, 2 p(y2,
x. v.v':0) Vo logp(u?, 11,,1 x,)
Σ
p x,
FEV* =EV*
Ep(s /4/x00 ,9) (Vo Voggp((? | 000
(3)

---

Gold Respouse
Iapus
Similarity Puzetion /
Crisiquo
Crisiques
2.
Criliquey
>/
Supervised
Save Tinjsttryy
Pinc-tuning
Inpur, y², 2.
The figure above depicts the principle discovery (E-step) phase. We sample an initial response y' on-policy, then "hint" with the gold response to elicit candidate principles =(1:N)- Then, sample critiques the initial response (only used in rejection sampling, and not included in the fine-tuning trajectories), which we use to obtain principle-guided refined responses V(i:Ny The best refined response g² selected based on similarity to the gold response. We save the resulting trajectory, which is used for supervised fine-tuning in the principle learning (M-step) stage.
Figure 2:
Optimizing this objective can related to the repeated application of two alternating stages: discovery of the posterior distribution over principles and refinements (E-step), and 2) a principle learning stage (M-step). This is depicted in Figure/ 1. For details about the relationship EM, see Appendix D.
(EM) learning and comprises
In principle discovery stage (E-step), we need determine the posterior p(y², x. y', y However, obtaining the true posterior would require an intractable marginalization over y Instead, we use a Monte Carlo approximation of the expectation the STaPLe objective for which we only need samples of this posterior. To sample from the posterior, one usual method is rejection sampling, which allows us to sample from any unormalized score
p(y²
.y g(v² 2.
It rests to determine what is good score g approximate the posterior samples. A straightforward option using the numerator the Bayes theorem. Here, we instead use cycle consistency score
ply², v. ;00) p(y Ix,y' ",.y',,,"") p(y².2 2 |2, vv))
(5)
This allows distribution of principles sharper and easier to learn, at the cost of bias. As with the Bayes theorem option, this uses the validator p xr yy 2, y²) upweight samples that yield back the original gold answer yo given the refinement y² and the principle z. In this case, we still need to define posterior proposal p(y²: z vr y², G;; One possibility is to just use the language model's policy Ty at the current phase of training and prompt it to produce the principle and refinement. However, this bears the danger of collapsing into the trivial solution y² " To avoid this, we purposefully factor the principle-proposing proposing distribution as
proposal conditioned the hint yc, which can be expected to make the
p(y2.2 x. y² c p(y² 12.y", 22,) p(= 1..
(6)
so that sampling happens in two stages and the gold response is only seen by the principle generation stage 2, Setting this distribution as the proposal, we have that sample pair
2, p(2 x. ..:.,yc:00
Ui p(y² x. 2n.
gets accepted with probability:
We acknowledge that in absence of any constraint on this can still lead copying via However, practice, the prompt that elicit the principle creates a contextual bias against exact copy, with evidence included in Appendix K.2. Moreover, we also experiment with explicit clustering constraints over z, which makes exact copy impossible, and show that both versions of our approach perform similarly.
5

---

x. 3. 2n v2) ply
Pn
max VEV
ply 1..yy 2.,",,2'" 2²
(8)
where we use a sample approximation for the maximum as in conventional rejection sampling strategies. We include derivation of this rejection sampling rule in Appendix E.
In the practical implementation of the algorithm, there are two further aspects to consider. We include additional filtering where we compare the initial response y² to the gold reference y using the similarity metric; if they are sufficiently close, we accept the response without further refinement and without sampling Moreover, only keep the sample with the highest p(y" 2, y 2n yn (i.c. Best-of-N) as opposed to keeping all accepted samples or using lower variance estimates such as Bao-Blackwellizznn
This can be seen form of hard-EM following the insight that Best-of-N can be seen as rejection sampling for temperature + O. We provide further details in Appendix F.
The principle díscovery stage yields principle-augmenter dataset (x U ,, v2) E D', Note that no refinements improve upon the initial generation relative to the gold response, we discard the sample; thus, the dataset 'D' only consists of those samples on which principle improved the quality of the response towards the gold.
In the principle learning stage (M-step), we use the data D' collected in the principle discovery stage for supervised fine-tuning of the language model. In particular, we train the model to maximize the log-likelihood of the refinement trajectories in DD. The corresponding EM update can be written as:
(t++1) arg max Ea (..))) [log p(u", 2|x. ;0]
(9)
Each optimization stage includes multiple gradient updates, with the data collected in the E-step being seen multiple times based on the number of epochs, after which new batch of on-policy data collected. The two stages are repeated, achieving incremental improvements till no further gains are seen with respect to the gold references or the maximum number of iterations reached. Qualitatively, this should result in the fine-tuned LM being able to invoke principles conditioned on prompt, and learning to produce high-quality responses conditioned on both the prompt and the invoked principle. We also draw a connection between STaPLe (this EM procedure) and variance-reduced self-play; this is discussed further in Appendix G.
3.1 Posterior Regularization via Clustering
To maximize the human interpretability of principles and their application relative to specific domains, it beneficial to have compressed set, or constitution, distill to the model. However, the E-step described above, results in thousands of unique principles. We seek to project this set into a constrained subspace where the resulting principles serve as representatives for desirable attributes to be reflected. This can be achieved via posterior regularization (PR) in latent variable modeling, For a set of posteriors that satisfy the constraints Q, the canonical posterior regularization framework solves the problem:
( p(u? q- (v², 2 ...V".9 = arg min (L x. v PEQ
(10)
which aims to project our unconstrained posterior into the space of posteriors that satisfy the con- straints. From Ganchev al. (2010), obtain that the primal solution is given by:
2 n, V. . x p(y?.2 = x. v. (. exp (-)g(z))
(11)
where 9(2) are the constrained features, which only concerns the principles, and À Lagrange multiplier that must be set such that the expected value of the features under p satisfies the constraints.
Consider the following cluster-based basse definition of the constraints: assume access to clustering algorithm which yields a set of clusters {C1, C2. ... Cx}. For each cluster, a representative element I is chosen, forming the set 7 = { Ex} }. Now define g(2) = 1(z & Ž) as a binary feature function. Thus, to ensure that the regularized posterior only places mass on the representative elements, we can enforce the constraint set Q {q Eg]g(2)] = 0}
For our case, the posterior regularized objective has trivial solution of À + do for elements not in the cluster representative set 2, which only leaves the problem of computing new partition function.

---

Since we only need to sample from the posterior and we already do this through rejection sampling, retaining only the representative elements as a further stage of rejection sampling can be seen as form of posterior regularization.
In particular, we consider hierarchical (agglomerative) clustering for several of its benefits: (1.) it requires no assumptions about number of clusters or cluster shape a priori, (2.) the algorithm is deterministic, ensuring that the same clusters would be obtained for a given configuration, and (3.) the algorithm is relatively fast, only taking a few seconds in practice over thousands of principles. To ensure that the clustering performed in a semantically-awe aware manner, we first obtain sentence embedding with an encoder-only model and perform clustering over these embeddings, consolidating principles that are lexically close.
Given the principle-augment elements over = {Ci) we aim to replace & with the element 3 Z that is closest meaning to the original principle. Qualitatively, we want the set Z to comprise the human-readable constitution, minimizing semantic overlap in its labels. We take the medoid as the cluster representative:
dataset (2 v/ Pi. vf) 'D' and set 2 of cluster representative
Emedood = {mp. mi = arg min Σ mECN
lle, e,lle, & [], (Medoid Representatives) jEC,
It suffices to retrieve the corresponding cluster C, for sample i and replace 2, with 2, E Z Meedoid- The resulting dataset from this augmentation, (z4, v\ 2,, vt) E D is then used to train the model.
4
LMs Can Self-Improve with Iterative Principle Discovery
4.1 Experimental Setup
Mixed-Domas Input Corpus. We form a corpus of 100k samples for the principle discovery
phase, consisting of four datasets: Anthropic HH-RLHF (Bai al., 2022a), UltraFeedback (Cui et al., 2024), TL;DR (Stiennon et al., 2020), and HotpotQA (Yang et al., 2018), taken in equal proportion (i.e. 25k samples of each dataset, drawn randomly) and deduplicated by prompt. For all datasets, we use the existing, uubicly--vvallable human- annootttd gold responses; for preference datasets, we take the chosen response to be the gold answer y². To run STaPLe, we use the first 50k samples for iteration 1, to heavily bootstrap off the first iteration, and then use 10k samples for each iteration thereafter, such that the input prompts are unseen for each iteration.
Models and Hyperparameters. We evaluate three performant small language models: Llama- 3. -nnttuctttttuct (Grattafiori et al., 2024; Meta, 2024), Granite-311-8B-Innttne -8B--Inttttt (Granite Team, 2024; Granite Team and IBM, 2024), and Qwen2.5-7BBInsttee
(Qwen, 2025). We also evaluate our method on Qwen3-32B to show that our approach scales to larger sizes (Appendix L6-v2 model (Sentence Transformers, 2021) from SentenceTransformer N). We use the all-MiniL the embedding model to compute medoids in our clustering approach. We use the Rouge-L Fl score (Lin, 2004) to compare the similarity of candidate responses relative to the reference answer. We also include an ablation in Appendix P.2 using a prompted Phi-4 (Abdin et al., 2024) judge to score responses, leveraging additional compute to improve the quality of rejection sampling. We discuss all other major STaPLe algorithm and model training hyperparameters in Appendix C.
Baselines. We compared our method against several baselines in both the single-iteratee and multi-iteration settings, in addition to the scores of each model's initial policy.
1. Prompted self-refinement to directly produce a self-critique and revision, akin Self-Refine, without any principle or specific feedback criterion provided priori.
2. Supervised fine-tuning on the gold responses of the first 50k samples in the mining corpus.
3. Following SCoRe, we adopt STaR-like baseline for intrinsic self-correction we apply the STaPLe algorithm and perform supervised fine-tuning on the best refined response (without principle-based refinement trajectory). This will henceforth be referred to as "STaR".
7

---

Table 1: Comparison of the STaPLe algorithm (unconstrained and constrained) against the baselines.
The scores reported below are an average over 5 runs for all benchmarks. T1 and T2 refer the first and second turns and WR is Win Rate.
Model
MT-Bench (avg)
MT-Bench (T1)
MT-Bench (T2)
AlpacaEval
IFEval (WR)
Llama-3,1 1.B--...1-----tuuer
Initial Policy
8.09
6.83
26.9
Self-Refine
7.40
6.75
26.1
51.2%
Gold-only SFT
7.47
6.83
26.4
56.2%
STaR Iter 4
7.56
7.00
31.8
62.3%
STaPLe Iter 4
7.71
8.13
7.30
33.4
68.9%
Construined STaPLe Iter 4
7.70
8.13
7.28
34.9
69.1 %
Granite-3.1 --tiit--r-------ttrucc Instruct
Initial Policy
7.83
8.59
7.08
30.2
Self-Refine
7.86
8.63
7.10
31.7
57.1%
Gold-only SFT
7.86
8.68
7.05
30.1
55.8%
STaR Iter 4
7.96
8.68
7.25
35.6
62.1%
STaPLe Iter
8.04
8.69
7.41
38.4
67.6%
Constrained STaPLe Iter
8.03
8.65
7.41
38.8
68.4%
Qwen2.5-7B-Imss Instruct
Initial Policy
6.83
7.34
6.31
30.4
Self-Refine
6.91
7.41
6.40
30.7
58.4%
Gold-only SFT
6.89
7.43
6.35
30.0
56.9%
STaR Iter 4
7.14
7.63
6.66
37.8
68.4%
STaPLe Iter
7.24
7.64
6.85
40.2
73.4%
Constrained STaPLe Iter
7.22
7.60
6.84
39.9
72.1%
We compare the STaPLe and STaR algorithms over four iterations this performed over the same number of samples per iteration, i.e. 50k samples in the first iteration and 10k samples for each subsequent one. Naturally, the other baselines are performed for a single iteration. evaluate both the unconstrained (without clustering) and constrained (with clustering) versions of the STaPLe algorithm, and compare their performance.
Evaluation. We evaluate on the MT MT-Bench (Zheng al., 2023) and AlpacaEvall2.0-1 (Li al., 2023; Dubois et al., 2024) datasets, instruction instruction-following evaluations designed to reflect alignment abilities of LLMs chat settings; these are scored using the GPT-4o model (OpenAI, 2024). We also use the Prometheus-88 Baieeuus-xxxB--a 8x7--xx2.BB-- --0v..0 model (Kim et al., 2024) on the IFEval (Zhou et al., 2023) dataset, for fine-grained evaluation on principle-following rubrics, with additional experiments in Appendix At inference time, if principle was invoked intrinsically given prompt, the response is parsed to only score the refined generation, following the principle proposal this is done similarly for the STaR baseline. Otherwise, the IFEval results, the win-rate with respect to the principle invoked for example, the principle is "Directness". the Prometheus judge assesses which response is more direct between the candidate generation and the generation from the initial policy. Given that the STaR baseline does not explicitly invoke principle, we use the same principle invoked by the STaPLe model for that sample'.
score the full generated response, and special parsing is performed. For
4.2 Results
Latent Principle Learning Improves Response Quality. The STaPLe algorithm outperforms the baselines on all benchmarks, across all models, as seen Table 1. The MT-Bench average exceeds the best baseline by an average of +0.11 over the three models, with the Turn 2 increasing by an average of +0.22. The AlpacaEval win-rates improve over the initial policy by +5.3-7%, and improves over the STaR baseline by +1.6-2.8%% Furthermore, the IFFval win-rates on principle-followin of the refined generations against the base policy using Prometheus improve by +5-6.6%. This suggests that training models to explicitly invoke the principle
expressive form of latent attribute is
'While the STaR- like baseline may not have seen this exact principle, it is valuable to analyze the effect of explicitly training principle labels to how well the final responses reflect those principles invoked.

---

Figure 3: Principle discovery rates of the STaPLe algorithm in the unconstrained (left) and constrained (right) settings. This represents the fraction of the trajectories saved from the principle discovery process (E-step) that contain unique principle label that was unseen in previous iterations. Constrained STaPLe Principle Discovery Rate
STaPLe Principle Discovery Rate
0.9
0.9
Lleme-3.1 3.1 3.1-00-Instruct
0.8
0.8
Granita-3.1 Seritta-3.1-8- 38-insrrcttt
Frection wr Unique Princioles
Fraction wi Unique Princioles
Qwen2.55 5-B-issuttt
0.7
0.7
0.6
0.5
9.5
0.4
0.4
0.3
0.3
0.2
Jbmo-3.1-98--nsmi
0.1
Granite-3. 88-Instruutt
9.1
Qwen2.5-78.lnssrrut
0.0
0.0
Iteration
Iteration
effective, as opposed to implicitly learning over this by simply training only on the refined responses (i.e. the STaR baseline). The Self-Refine baseline improves performance for the Granite and Qwen models, but not Llama-8B, suggesting that it is not as effective in zero-shot self-refinement without pre-identified principles. This corresponds with a higher IFEval win-rate for those models with strong self-refinement abilities.
Iterative Principle Discovery Enables Self- Self-Improvement. The results in Table I demonstrate the performance of our method in the fourth iteration of the Monte Carlo EM algorithm. Our algorithm outpaces the STaR baseline by sizable margin throughout the execution of both algorithms; we include the full set of results in Appendix H. iteration 3, the STaPLe scores outperform STaR and the initial policy on average across the three models by +0.16 and +0.29 on MTT-Bench (avg.); +3.6% and +9.2% on AlpacaEval win-rate; and +7.9% and +21.0% on IFEval principle-following win-rate, respectively. We observe slight diminishing returns for Llama-8B and Granite-8B, as in iteration 4, the scores either remain at a similar level or drop slightly; however, Qwen-7B continues to improve on all three benchmarks. We further analyze pinciple-following quality in Appendix I and stepwise win-rates of iteration t against iteration t 3-1 in Appendix J to reinforce the self- ooff-improvemett induced by STaPLe. In Appendix K, we demonstrate that the model's intrinsic self-correction ability improves over the iterations. Moreover, we include an ablation in Appendix M showing that self-refinement improves when providing the constitution for the model to select principles from at inference- time.
Clustering Balances Interpretability and Performance. In Table 1, we also include the per- formance of "constrained" STaPLe - the version of the algorithm with agglomerative clustering following the E-step during each iteration, and use the medoids of each cluster as a representative principle to yield dataset 'D. We find that this largely matches the performance of the "unconstrained" version, in fact outperforming it in AlpacaEval and IFEval win-rates for Llama-8B and Granite-8B. The full results can be found in Appendix P.I. where we ablate across different label replacement schemes (medoids, modes, and a perplexity-based method). For both versions, we observe a strong correlation in the trend (avg. p Rs 0.95-0.96) between the MT-Bench (avg.) and AlpacaEval results.
4.3 Analysis of Principle Discovery
It is also valuable to study the nature of the principle discovery process and the constitutions that we have aligned the language model toward. We include the full constitutions and perform more qualitative analysis on their distribution of elements when performing label replacement ("density" of the constitution) in Appendix L. In Figure 10, we show that the number of principles in the constitution under Constrained STaPLe decreases over the iterations, suggesting that the model converges to learning a relatively stable distribution of principles. In particular, the size of the constitution by iteration 4 is roughly 50% of the iteration size, or even smaller.
This finding is reinforced by an analysis of the principle discovery rate the fraction of refinement samples with new, unseen principles- in Figure 3. We show that this rate decreases over the iterations

---

under both the unconstrained and constrained versions of the STaPLe algorithm, suggesting that all models learn to re-use principles accordingly. The observation that constrained STaPLe helps to accelerate this convergence to a condensed set of principles reinforces the motivation behind the introduction of clustering as a posterior regularization mechanism. This also highlights one of the advantages of using the LM to approximate the posterior distribution, as the changing nature of the learned posterior can be observed over the iterations and elicited via on-policy sampling.
5 Discussion and Limitations
The STaPLe algorithm guides largely autonomous self self-improvement process, with the exception of a few hyperparameters that are to be set, discussed in Appendix C. As a result, the algorithm does not require human supervision beyond the labels in the curated (publicly-available) mining corpus. This work focuses primarily on smaller, non-reasoning language models, which struggle with self-refinement. Nonetheless, we find that STaPLe is also effective for the Qwen3-32B reasoning model to self-improve (see Appendix N), an encouraging sign for further scaling its applications.
While the distribution of principles is mined relative to the mining corpus' task distribution, the STaPLe algorithm itself is task-agnostic, and can be used for any distribution of datasets where a reliable gold reference exists, with paired preference data. However, designing task-awan version of the STaPLe algorithm may reveal further insights into the model's 's task dese-depnndent self- corfecorretton mechanisms while inducing a curriculum. In this work, we focus the two-turn self-correction setting at the same time, interesting insights could be extracted regarding the compositional nature of principles when extending to further refinement attempts, which also yields more diverse trajectories (combinatorially many possible), even over a condensed set of principles.
A core aim of alignment research is to balance human-readability with machine- readabiiiy. The STaPLe algorithm succeeds in achieving this by discovering principles that are useful to the model for self-correction, while compressing them to smaller set via clustering for human reader to analyze. We believe that this work and the notion of LM self-improvement keeps with the theme of the Bitter Lesson (Sutton, 2019), when facilitated in relatively autonomous fashion. Specifically, limit the influence of human- driven priors or constraints on the algorithm; this is reflected further by our clustering technique, and our ablation in Appendix P.3 to fully automate this as well. At the same time, we acknowlede ledge the value of human oversight on the alignment process; as such, we believe that co-man----the-loo analysis of the principles a post-processing mechanism following the E-step of each iteration would be valuable to further mitigate misalignment risks or potentially harmful principles (see Appendix R).
aim to
Our work introduces principles as an expressive form latent chain-of-thoughtt relying on the power and scale of pre-training to bootstrap our mechanism in post-training. One can view our constitutions as general-purpost abstractions that inform generation quality - such a notion can be easily extended to new domains, such difficult reasoning. for multi-aspect verification. It provides a concrete verbalization of the dimensions along which Pareto improvement is desirable to maximize cnsrucction-oolowinn capabilities. Moreover, they can serve to guide further training, including online reinforcement learning with STaPLe as a warm-up, and integration with memory frameworks for continually discovering and revising target attributes. The evidence that the discovered constitutions are useful both in iteratively training LMs for self-correction as well as through their inference-time application (see Appendix M) presents promising outlook for this research direction.
6 Conclusion
We introduced a new language model self-improvement method which uses model-generated latent principles to learn intrinsic self-correction. These serve a purpose akin to reasoning chain-of- thought, boosting the quality of LM generations on seanmennt-fousser benchmarks. Furthermore, our posterior-regularizee Monte Carlo EM algorithm shows that the model can continue to improve over multiple iterations, while simultaneously compressing the principles to human- buman-interpretable constitution. We also show that our clustering approach balances performance with the diversity of the generated constitution, thus adding valuable utility to the STaPLe algorithm. The efficacy of STaPLe highlights the potential for constitutional alignment with self- self-geeerated principles to improve model responses in an interpretable manner with minimal human supervision.
10

---

The authors would like to thank Sara Rosenthal, Yikang Shen, Radu Florian, and Salim Roukos for valuable input and feedback during this work. KR would also like to thank Brian Williams and Arduin Findeis for helpful discussions related to this work and the impacts of principle discovery, and Sriram Tolety for feedback on a draft of this paper.
Acknowledgments.
References
M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, J. R. Lee, T. Lee, Y. Li, W. Liu, C. C. T. Mendes, A. Nguyen, E. Price. G. de Rosa, O. Saarikivi, A. Salim, S. Shah, X. Wang, R. Ward, Y. Wu, Yu, Zhang. and Y. Zhang. Phi-4 technical report, 2024, URL https: /ttp:////axiv.../ org/ harg/abs/2en ab1. 2412208905........
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield- Dodds, D. Hermandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, Mann, and J. Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022a. URL https: //arxiv.orge abs /tab////22442 05862.
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, Perez, J. Kerr, Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. HlaffieldDDddds B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and Kaplan, Constitutional ai: Harmlessness from feedback, 2022b. URL https: / //tpxivvvvv///////////e/////aa./ org/ abs 22000088378
H. Chen, Y. Feng, Z. Liu, W. Yao, A. Prabhakar, S. Heinecke, R. Ho, P. Mui, S. Savarese, C. Xiong, and H. Wang. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding, 2024a. URL https: /ttp:v////xxiv/ org. htt.aa//////.eeeee/e..e..e
X. Chen, H. Wen, S. Nag. C. Luo. Q. Yin, R. Li, Z. Li, and W. Wang. IterAlign: Iterative constitutional alignment of large language models. In K. Duh, H. Gomez, and S. Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1423-1433, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10. 11.18553/1/2024. naacl-long.77 78. URL https:ttpso //t///aaaaaatooogooyoooooo org/2024. 2024.
Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning convertsweak language models to strong language models. In Proceedings of the 41st International Conference on Machine Learning, ICML 2.M.'24./0 JMLR.org, 2024c.
G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, Z. Liu, and M. Sun. Ulrafeedback: Boosting language models with scaled ai feedback, 2024. URL https: //arxiv. org/ /337000000177 01377.
P. Dayan. Reinforcement comparison. In D. S. Touretzky, J. L. Elman, T. J. Sejnowski, and G. E. Hinton, editors, Proceedings of the 1990 Connectionist Models Summer School, pages 45-51, San Matco, CA, 1990. Morgan Kaufmann.
ReepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning, 2025. URL https: / arxiv. org/
12948.
DeepSeek-AL, Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai,
J. Liang, J. Guo, J. Ni, J. Li, J. Wang, J. Chen, J. Chen, J. Yuan, J. Qiu, J. Li, J. Song, K. Dong,
K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Xu, L. Xia, L. Zhao, L. Wang,
L. Zhang, M. Li, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, N. Tian, P. Huang, P. Wang, P. Zhang, Q. Wang, Q. Zhu, Q. Chen, Q. Du, R. Chen, R. I.. Jin, R. Ge, R. Zhang, R. Pan, R. Wang, R. Xu, R. Zhang. R. Chen, S. S. Li, S. Lu, S. Zhou, S. Chen, S. Wu, S. Ye, S. Ye, S. Ma.
11

---

S. Wang, S. Zhou, S. Yu, S. Zhou, S. Pan, T. Wang, T. Yun, T. Pei, T. Sun, W. L. Xiao, W. Zeng.
W. Zhao, W. An, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, X. Li, X. Jin, X. Wang, X. Bi,
X. Liu, X. Wang, X. Shen, X. Chen, X. Zhang, X. Chen, X. Nie, X. Sun, X. Wang. X. Cheng,
X. Liu, X. Xie, X. Liu, X. Yu, X. Song, X. Shan, X. Zhou, X. Yang, X. Li, Su, X. Lin, Y. K. Li,
Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Y. Zhang, Y. Xu, Y. Xu, Y. Huang, Y. Li, Y. Zhao, Y. Sun, Y. Li,
Y. Wang, Y. Yu, Y. Zheng. Y. Zhang, Y. Shi, Y. Xiong, Y. He, Tang. Y. Piao, Y. Wang, Y. Tan,
Y. Ma. Y. Liu, Y. Guo, Y. Wu, Y. Ou, Y. Zhu, Y. Wang. Y. Gong, Y. Zou, Y. He, Y. Zha, Y. Xiong,
Y. Ma, Y. Yan, Y. Luo, Y. You, Y. Liu, Y. Zhou, Z. F. Wu, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu,
Z. Huang, Z. Zhang, Z. Xie, Z. Zhang, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Xu, Z. Wu,
Z. Zhang, Z. Li, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Gao, and Z. Pan. Deepseek-v3 technical report, 2025. URL attps://arke
org httggab///41..../ 19437.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 12 2018. ISSN 0035-9246. doi: 10.11 11111//2177-6111177780006600 //doi.org/10 1111/j 2167-61611111111 1977 197...1066.......1
URL https:
K. D'Oosterlinck, W. Xu, C. Develder, T. Demeester, A. Singh, C. Potts, D. Kiela, and S. Mehri. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment, 2024. URL htttps://aaatt wwwpxiiiorrgaaaeo httg//bb/24088 06266.
Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: simple way to debias automatic evaluators. arXiv preprint wrXiv:2404.04475, 2024.
A. Findeis. T. Kaufmann, E. Hüllermeier. S. Albanic, and R. D. Mullins. Inverse constitutional AI: Compressing preferences into principles. In The Thirteenth International Conference on Learning Representations, 2025. URL /tttps::/opeereviiie www.:nnnnuumme... 1ee-iddd999 9FRukPP3Cn.
J. Fränken, E. Zelikman, R. Rafailov, K. Gandhi, T. Gerstenberg, and N. D. Goodman. Self- supervised alignment with mutual information: Learning follow principles without preference labels. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet. J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 61328 61371. Cur- ran Associates, Inc., 2024. URL https:t neurips cc cwcpapprppppeeeeeeeeeeeeeeeeeeee filfffeesss walleeepaaae paper/ 2022//5lee/006885177222222226662 ortp::/e--00000000--------o---r- Cowfeeeeneeereeeeeeeeee pdf.
K. Ganchev, Graça, J. Gillenwater, and B. Taskar. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 1(67):2001-2049, 2010. URL http:
//jmlr. wwt.:rrrrrppaper/////// wttpar/ss/v/////eev1oe1e/ wgggggnnceviio html.
I. Granite Team. Granite 3.0 language models.
http:://ww.rriiistaaaaappppoooo uploads//022/10 1ttp//100pppppee/ pdf, Oct. 2024.
Granite Team and IBM.
https://huggingface co/ibm-granite/ granite-3.1 Sb-instruct, Dec. 2024. Release Date: December 18, 2024.
A. Grattafiori et al. The llama 3 herd of models, 2024. URL https:::/arxiii.r///aa 21783.
M. Y. Guan, M. Joglekar, E. Wallace, S. Jain, B. Barak, A. Helyar, R. Dias, A. Vallone, H. Ren, J. Wei, H. W. Chung, S. Toyer, J. Heidecke, A. Beutel, and A. Glaese. Deliberative alignment: Reasoning enables safer language models, 2025. URL
T. Head, M. Kumar, H. Nahrstaedt, G. Louppe, and I. Shcherbatyi, scikit-optimize: Sequential model-based optimization in python, Sept. 4 2020. URL https::/eod..org/11.528//// 4014775.
A. Huang, A. Block, D. J. Foster, D. Rohatgi, C. Zhang, M. Simchowitz, J. T. Ash, and A. Krishnamurthy. Self-improvemmnn in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
// /ppeneeeeew
12

---

J. Huang, S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han. Large language models can self- improve. In H. Bouamor, Pino. and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051-1068, Singapore, Dec. 2023. Association for Computational Linguisties. doi: 10.18333/1122033 emnlp-main. URL https: htaps:///ccaarna acl.aaoooottoyyyyyoo
2223..2232222 emnlp-main. 67/.
Y. Katsis, S. Rosenthal, K. Fadnis, C. Gunasekara, Y-S. Lee, L. Popa, V. Shah, H. Zhu, D. Contractor, and M. Danilevsky. Mtrag: multi-turn conversational benchmark for evaluating retrieval- augmented generation systems, 2025. URL htt.::tts///// http://rxxvv.oorggbaaa/aa/ htgg/bbb/250111 03468.
S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo. Prometheus 2: An open source language model specialized in evaluating other language models, 2024. URL https: / arxiv. org/ abs/2405. 01535.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 22199-222111 22213. Curran As- sociates, Inc., 2022. URL httpps:/pproceedinggg neurips cc/ wtc/paper_ffee files//l//eee / p22pee//2// /t////bbb2999cdaaa
11w11111 Paper- Conference. pdf.
A. Kumar, V. Zhuang. R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs, L. M. Zhang, K. McKinney, D. Shrivastava, C. Paduraru, G. Tucker, D. Precup, F. Behbahani, and A. Faust. Training language models to self-correct via reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.o wwt.teettuuureee htte//ormmmeie wow.ooridmCceiiiii CEEERCAA
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Sto- ica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloiing models. https://git thub, com/ tatsu-lab/ alpacapppaaaaaaaaaaaaaaaaaaaaaaa 5 2023.
C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: http:///caaanttooocceo org/w04-1/000
T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. pling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024. URL https::///// /ttp:////ppeeeeeeeeeee nnt./ forum? www.ooidoounosiaoo ddrrmmiiddxxSSwwr[eee
Saleh, P. J. Liu, and J. Liu, Statistical rejection sam-
Z. Liu, P. Wang, R. Xu, S. Ma, C. Ruan, P. Li, Y. Liu, and Y. Wu. Inference-time scaling for generalist reward modeling, 2025. URL https:////aan //arxiv. org/ abs a204 02495.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. URL https: http///onrrevieeeee / net wtt.:eeee///eeee wow.ooriduduaae -ig.iiidi-BBgy.......
A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yaz- danbakhsh, and Clark. Self-refine: Iterative refinement with self-feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine. editors, Advances in Curran Associates,
Neural Information Processing Systems, volume 36, pages wtteae/fllessseaeeeeeeeeee pptppp/pee/////leee/ Inc., 2023. URL https: htttps://proceedinggs neurips. wwpprp/ppppppppeeeeeeeeeeeeeeeee files/llllleeell 9laefff772211555000000eeccreeeee
Meta.
https: httpss///hggingaace... commeta Laaaamm///maaaaaa. 1-8B- Instruct,, July 2024. Release Date: July 23, 2024.
-lama-3.1-8bbinsttee
OpenAl. Gpt-4o system card, 2024. URL httth: org wttp://////ab/1..22222222 httt::://aaai..ooggaa/////22222
OpenAI. OpenAl o3 and 04-mini System Card.
System card OpenAl, Apr.
2025. URL htt.h:t/t///ltt https/:///ddo.oo.i....aaaa
o3- and- o4-minimn ceini-systeen system-------e---- card, pdf.
13

---

A. Patel, M. Hofmarcher, C. Leoveanu Condree, M.-C. Dinu, C. Callison-Burch, and S. Hochreiter. Large language models can self imff-ie web agent tasks, 2024. URL https:t//aaaaae org/
Petridis, B. Wedin, A. Yuan, J. Wexler, and N. Thain. ConstiutiionaExperts: Training mixture of principle-based prompts. In L. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 574-582, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653//1/2224.aall www.:0244..l--ho...53. URL https::t////.... http:p::///aaaahhllogo..o
arg/2024.20 short...t....... 52/.
D. Phan, M. D. Hoffman, D. Dohan, S. Douglas, T. A. Le, A. Parisi, P. Sountsov, C. Sutton, S. Vikram, and A. Saurous. Training chain-of-thought via latent-variable inference. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Y. Qu, T. Zhang, N. Garg, and A. Kumar. Recursive introspection: Teaching lan- guage model agents how to self-improve. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neu- ral Information Processing Systems, volume 37, pages 55249-55285. Curran Associates, Inc., 2024. URL https: htapps://pree //pror/ed///cceeneeee/ wwwpiiiinnnnnnnnnnneeeeee neurips. wwwppppr.pp.ppp... seaper_fflese solffiles/pppp htilee/paperr/224/E11. 09949228810024400844444411888880 Paperrrrreeeee---eereeeeeeeeee
Conference pdf.
Qwen. Qwen2.5 technical report, 2025. URL
wwwpa/wwwv.orr///////aaa/ 15115.
K. Ramji, Y.S. Lee, R. F. Astudillo, M. A. Sultan, T. Nascem, A. Munawar, R. Florian, and S, Roukos. Self-refinement language models from external proxy metrics feedback, 2024. URL https://arxiv org/ abs//403,.../////////////// 00827.
Y. Ruan, N. Band, C. J. Maddison, and T. Hashimoto. Reasoning to learn from latent thoughts, 2025.
URL https: //ta/////xxiv wwwexxiv..o//aaae org/abs/2003 18866.
Sentence
all-MiniLM-LL-vv. sentence- www.eeetrrraasfferrsseeeemmeeeee aasaaaallMMMMM aatprr//aaaMMMiiMaaaa--a L6-v2, 2021.
Transformers.
https://huggingfaes co/
N. Stiennon, Ouyang. J. Wu, D. Ziegler, Lowe, C. Voss, A. Radford, Amodei, and Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3008 30211. Curran Associates, Inc., 2020. URL https: http::/t///////tt //proceddge//o neurips corrips.coo cc/ paper_ wtt.::/eeeeapppeee////////
Paper. pdf.
Z. Sun, Y. Shen, Q. Zhou, H. Zhang Z. Chen, D. Cox, Y. Yang, and C. Gan. Principle- driven self-alignment of language models from scratch with minimal human supervision. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 2511-2565. Curran Associates, Inc., 2023. URL 0764db1151 018.8005510000a2222222222211---0
neurips. cc/ cccpaper__p.. lalllaapppeeeeeeellllllllpaaaaee filffiies/ppeel hllee/paperr/223/t111 Conference. pdf.
Z. Sun, Y. Shen, H. Zhang, Q. Zhou, Z. Chen, D. D. Cox, Y. Yang, and C. Gan. SALMON: Self- alignment with instructable reward models. In The Twelfth International Conference Learning Representations, 2024. URL https: //tp://///e///ee/e///// nwt. forum wiw.ddddsseB......
R. S. Sutton. Temporal Credit Assignment in Reinforcement Learning. Ph.d. dissertation, University of Massachusetts, Amherst, Amherst, MA, 1984.
R. S. Sutton. The bitter lesson. https: / http://wwwcccc cs. wtt.:cc...eteeeee utexas. eateaa..ddu///// htth://eunsooccurssss/attttt bitter_lessed deeesssoo.f.......eeee 2019.
J. von Neumann. dom digits. InBook_ InBook_Computing Computing.gni.oocm 1961 1
Various
techniques
used in connection with ran- URL https: https://mmnp... htmp///pppann.....mmmmmmmmmmmmmm www..aan.ppd...........o http//dffflee////////
1951.
PariousTechni www.useecciiuuesseeeeeeeoaaa inestssediCoooneeti onwi wwweeeiiiiiittttnnnnneoo shRandomDigita.... pdf. J. Res. Natl. Bur.
Stand. Appl. Math. Series, vol. 3, pp. 36-38 (1955).
14

---

G. C. G. Wei and M. A. Tanner. A monte carlo implementation of the em algorithm and the poor man's data augmentation algorithms. Journal of the American Statistical Association, 85(411): 699-704, 1990. ISSN 01621459, 1537274X. URI http: jstor org/ wtrg/stable/222000
J. Wei, X. Wang. D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https: //arxiv. org/ abs/2201. 11903.
A. Yang, A. Li, B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng,
D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, Yang, Yang. J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang. Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu. Qwen3 technical report, 2025. URL abw./5/...... 09388.
https:/ /ttp:///axxvvvvi
Z. Yang, Qi, Zhang, Y. Bengio, W. Cohen, Salakhutdinov, and C. D. Manning. Hot- potQA: A dataset for diverse, explainable multi-hop question answering. In E. Riloff, D. Chi- ang, Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium, Oct. Nov. 2018. Association for Computational Linguistics. doi: 10.18653/ 12599 URL fttps://att http:/aalllaaan htt///lconntoooyyyyy org/ (trg/D18-10 1259/.
E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 15476-15488. Curran Associates, Inc., 2022. URL attps::/pproceet naw.r 239.a17204tbb111b6ffte2ee------2 htt.ap/ee...ooaaeaaaaaaarra---oe pdf.
neurips ct/ wwwpppppppeaaeeeeeeeeeeeeeeee filest wttpreeerrreeeseprrrreeeeeeeeeee ppppp///02//fillle/eee
H. Zhan, M. Azmat, R. Horesh, J. Li. and M. Yurochkin. Spri: Aligning large language models with context-situted principles, 2025. URL https://arxiii org/ abaaaba/22000 2502. 03397.
T. Zhang, A. Madaan, L. Gao, S. Zheng, S. Mishra, Y. Yang, N. Tandon, and U. Alon. In-context principle learning from mistakes, 2024. URI. /ttp:///aaxvvvvio org/ abs 2402.05402.
L. Zheng, W.-L. Chiang, Y. Sheng, Zhuang, Z. Wu, Zhuang, Z.
Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 46595 Curran As- sociates, Inc., 2023. URL /ttptpss:/ppooeeeeiiiee nwwrips. cc/ cccpppe.cc/pppprrr files/ /211e///apr//223/ 2ttp//11111122777922777666665363
Datasets and Benchmarks. pdf.
J. Zhou, T. Lu, Mishra, Brahma, Basu, Y. Luan, D. Zhou, and Hou. Instruction-folloiing evaluation for large language models, 2023. URL https: /tatps///axxv..... org/ http//abb///////////////
W. Zhou, J. Hu, H. Zhang, X. Liang, M. Sun, C. Xiong, and J. Tang. Towards interpretable natural language understanding with explanations as latent variables. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's contributions are reflected Sections (Algorithm) and (Empir- ical Results), as well as the Appendices including ablations and supporting experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work are included in Section 5, and discussed further in
the ethics statement Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification:
The theoretical contribution is discussed, with proof included, in Appendix G.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide a formal algorithm description in Appendix A, reproducibility statement in Appendix B, hyperparameter details in Appendix C, and the prompts used in our work in Appendix O. Along with the discussions of the method and experimental setup in Sections 3-4, we believe these to be sufficient to reproduce our work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
17

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes. we provide details to reproduce our results with the algorithm (Section and Appendix A), experimental setup details (Section 4.1 and Appendix S), hyperparameters and model training details (Appendix C), and prompts (Appendix O). We will also open- source the code used to generate the results for the STaPLe algorithm and for the baselines we have reported, with README file explaining how is to be run. This also reflected in reproducibility statement in Appendix B. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
gew.dee/ooeeuummaae caaiimiisionoPoccyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
onnoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We discuss the experimental setup in Section 4.1, and the hyperparameters used to induce the results presented are included in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, our results Tables and are an average score over runs of each benchmarks. The other results and ablations included in the Appendices are over single run, due to the costs of running each benchmark several times, given MTT-eench and AlpacaEval use closed-source OpenAl models judges.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources are described along with the hyperparameters and model training details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't ' make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps://neuripee cc Catublic/Ettisstiiii deliddeine
Answer: [Yes]
Justification: We believe that our work conforms with the Code Ethics; we further provide ethics statement for our work in Appendix R.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
19

---

Answer: [Yes]
Justification: The impacts of our work are discussed in Sections 5 and 6, following from the motivation described Section 1. We further discuss the socictal impacts and any possible harms in our ethics statement, in Appendix R.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not have any artifacts (data, models) that pose high risk for misuse. The model-generated constitutions are included in Appendix L for informative purposes, and we do not believe that they pose a strong misuse risk. We detail potential misuse of the proposed algorithm and risk mitigation strategies in Appendix R.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use usel-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
20

---

Justification: In addition to citing the models and datasets used in Section 4.1, we include dedicated appendix section to discussing the details and licenses of each of these assets used, in Appendix S.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
snperswithcode
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced are the model-generated constitutions and the language models trained to follow them. The former is included Appendix I,, while the details of obtaining the latter included in Section 4.1 (experimental setup), Appendix A (algorithm), and Appendix C (training details and hyperparameters)
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
21

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
The primary focus of this work self-improvement of LLMs, so LLMs are used for iterative on-policy synthetic data generation and training in the core methodology this work, described Section 3 and Algorithm Aside from this, LLMs are used a judge to assess the quality of custruction-followiiin evaluations) and principle-following in all experiments. Furthermore, in Appendix P.2, include ablation where similarity to the gold response.
Justification:
(as part of the MT-Bench and AlpacaEval
Phi-4 LLM-as-a-juee to score candidate response
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / //w.ripppp..../ for what should or should not be described.
22

---

Appendix
Table of Contents
Formal Description of STaPLe Algorithm
24
B Reproducibility Statement
24
C STaPLe Hyperparameters and Training Details
24
Derivation of the Monte Carlo EM Gradient
25
E Derivation of Rejection Sampling Rule
26
F Hard EM and Best-of-N in Monte Carlo EM
26
Self-Play Equivalence
27
II Complete Table: Self- Improvemrv Multiple Iterations
29
1 Prometheus Win-rates on MT-Bench and AlpacaEval
30
J Stepwise Win-rate Analysis
31
K Intrinsie Self-Correction
32
K.1 Refinement Rate Analysis
33
K.2 Precision Analysis in Delf-Refinemen
33
L. Model-Generated Constitutions
Granite-3. B.aeee...--B---tttttte -n-estrret-Geeeerteee Constitution
34
L.2
Llama-3.1 -lama-3.1-8B-Bll ----Instruct-Geneeeeee
Constitution
36
L.3 Qwen2.5-7B B-w7-----ttuut-eeeeeeeeeee Iestruct-Generated Constitution
37
L.4 Number of Clusters over the Iterations of STaPP. Algorithm
39
Interpreting Model-GGnerated Constitutions
39
M Inference-time Application of Discovered Constitutions
39
N Self-mmprovnmnnt on Reasoning Models
40
STaPLe Algorithm Prompts
41
Principle Mining Prompt
41
0.2 Critique Generation Prompt
41
0.3
Principe-Connditioned Refinement Prompt
41
P
Ablations
42
P.1 Label Replacement Method
42
P.2 LLM-as-a-Judge Rejection Sampling
44
P.3
Bayesian Hyperparameter Optimization for Clustering Distance Threshold
46
Qualitative Examples of Principle-Guided Self-Correctin
46
Q.1 Llama-3,1 1---B-Insstrctt IFEval Examples
46
Q.2 Granite-3.1-88 --struutt IFEval Examples
50
R Ethics Statement
52
S Details of Models and Datasets Used
54
23

---

A Formal Description of STaPLe Algorithm
We provide full, formal description of the STaPLe algorithm below. We use y² and y² notationally to avoid confusion with the sample indices. We use general variables for components which may be ablated on: the similarity function f. clustering algorithm and label replacement scheme R. We leave the . step in terms of the dataset D' for generality, although if clustering were to be performed, would use D instead.
Algorithm Self-Taugh Principle Learning (STaPLe)
Require:
Dataset D = { Mi
( >>>- pretrained LM parameters 0 number of EM iterations
T, number of principle samples N, similarity threshold similarity function ((s-); (optional) embedding model EMB, clustering algorithm C, and cluster representative scheme
1: for (= 0....,
T do
2: E-step: initialize D' +- 2. for each (I,, ) D do Sample initial response yt ~ Tocc ~TT xz). f(y vf < T then
3:
4:
5:
D needs refinement
6:
Draw principles {2+ for j = ,...., do
(1) fj=1 ppcu (z 2i, 24, vi v2
7:
Generate critique cf t Critique(yl 2(()).
8:
9:
Sample refinement V
2,(1) Tacs 24, 02.88.22 2(1) )
10:
end for j +- argmaxj [(y; 2.(5) ,14)
11:
(as, vi) + (z,1", V1 2.6))
13:
fly v > MV then
14:
Add trajectory (Iz; y 24, vi) to D'.
15:
end if
end if
17:
end for
(Optional): Cluster the principles smaller set in augmented dataset Đ Clusters C' + C (EMB((2.))
Assign cluster representatives (e.g. Medoid) Ž R(C) Augment dataset +- Rep(D', Z)
19:
g(++1) + arg max M-step:
Σ
log pe (y², x, y
0
(2.a) JED!
end for
Ensure: Final LM parameters ((T)
B Reproducibility Statement
In addition the algorithm description above (Algorithm and experimental details Section 4. we include the hyperparameters used and model training details in Appendix C and the prompts used in the STaPLe algorithm in Appendix O. We make all evaluation results available in tabular format throughout the main paper and the appendices for comparability. We also will publicly release the code for the STaPLe algorithm, to further facilitate reproducibility of our self eef--imprveement method.
C STaPLe Hyperparameters and Training Details
STaPLe Algorithm Hyperparameters. We use a Rouge-L threshold of 0.4 for the similarity threshold (f(v. C) - if the initial response exceeds this threshold, we do not pursue refinement). For the ablation using a Phi-4 judge in Appendix P.2, the threshold was set to be (on scale of 1-10). The other major hyperparameters involved in the execution of the STaPLe algorithm N, the number

---

of principles to sample, and the distance threshold for the clustering algorithm. STaPLe requires an inference time budget of 3N + for the Rouge-L version, and 4N + + for the LLM-as-a-judge version we set N = 16 to balance runtime per iteration of the algorithm with sufficient exploration of diverse principles. During principle discovery, we sample principles, critiques, and responses at a temperature of 0.7; the maximum number of tokens for principle proposal and critique is set at 500, and is set at 1024 for the refined response. We use x Nvidia GPUs for the principle discovery phase, with separate vLLM (Kwon et al., 2023) instance per GPU.
We set distance threshold to avoid setting specific target number of clusters when performing agglomerative clustering. The currett results involve manually setting distance threshold, where the authors analyzed the resulting set of clusters and for each of the first three iterations, ensured that there are at least 30 clusters. Fortunately, given the speed of agglomerative clustering, this is fairly easy to do. For the first iteration, the Fuclidean distance thresholds were set at 8 (Ilama and Qwen) and 6 (Granite); for iterations 2-4, the thresholds were decreased to 7 and 5, respectively. Alternatively, one could automate this process by designing an objective over the diversity (semantic or surface-level) of the cluster medoid labels and performing hyperparameter search. This is explored further in Appendix P.3 using Bayesian hyperparameter optimization tools to identify an appropriate, model-specific distance threshold. The threshold Tppt. for the perplexity difference label-replacement scheme, described in Appendix P.I,, was set at 0.2.
Model Training. We perform full supervised fine-tuning for epochs learning rate of x 10-6 with the AdamW optimizer (Loshchilov and Hutter, 2019), with sequence length of 4096. All experiments were performed on 8xH100 Nvidia GPUs.
D Derivation of the Monte Carlo EM Gradient
Recall that the conditional log- ligeliill is defined as:
((0) log p(y x. (x,y')
log Σ E
1 p(y2, xr v²
(12)
BPEV- SEV-
The usual EM formulation is given from the Variational Lower Bound (ELBO):
(()) = log ply
x. (x,y')
logg Σ
Σ ply
1 p(y2,2 2 |x, v.y')
V²EV" =EV"
p( y2, v.v')
= log Σ
E2, 12,8")
(Z. 1²2 xx,
S-E SEV*
2 Σ Σ u(2, |,,1²) log ply v² x,8²))
g(2 82 xz
8'EV- SEV*
= logg(yy x (2
- - KL(q 11 p(x x. vt ()
= Eg ( {log ply |x,y² 2, )²) + log 2 2 v.y') + H(g).
(13)
From the fourth line it is clear that the bound is tight only if the KL is zero i.e. the variational distribution is equal to the posterior over the latent
q(2, y² xx, (,,y)) p(y2, 1 1,,
(14)
Once this is determined (E-step) rests to optimize the last line (M-step). For a single gradient update, this yields the STaPLe update rule
V VoL(0) Ep(p layy { V log 12. :2,'
(15)
25

---

Since we do not derive a closed form solution for ply², u..''''' y but rather we only obtain samples from it, this algorithm is better understood related Monte Carlo EM (Wei and Tanner, 1990).
E Derivation of Rejection Sampling Rule
We here refresh rejection sampling and show the corresponding STaPLe formulas. Rejection sampling or accept/reject sampling (von Neumann, 1951) allows to sample from distributions with an intractable partition function
ply, 2 1.....80 ))=== gly. 2, I. y² .1
(16)
Z
by sampling from a proposal distribution q(y², , 2) and accepting the sample with probability
1 ply N. .0
Pr. M
(192,2)
(17)
where
plu2 2 v. y 9
M
max =VV==
g(y²,20
(18)
scales the ratio to ensure that it a valid probability. Note Z cancels out when replacing the last equation into the previous, making the probability not dependent from the partition function.
For STaPLe have defined
gV ",,,,,,,'""" ply ".,',,,,' p152 = xr ,
(19)
and
<(v,2)= $(02
(20)
leading to
Pn
ply .2, (llp 21
-EV
max
2-p(y 22.
p x, 3 2, 12 5(y 2 xz
9)
Z p(y22
C x, 3 3m V
(21)
max =EV
p(yG
Ir, x,,y 22
F Hard EM and Best-of-N in Monte Carlo EM
Hard Expectation Maximization (Dempster et al., 2018) refers to EM algorithms where the posterior the latents is replaced by the greedy most probable output of the posterior e.g. obtained via Viterbi decoding for classical Hidden Markov Model latent variable models. In other words, we collapse the posterior prove that this coincides with Best-of-N choice according to the score p(y x;, y Zn. v2 under conditions that algorithm meets.
its most likely output. We can the result in (Liu et al., 2024, S 3.2)
For this, we consider our score-based distributions with an additional temperature term T in our score
... 9082 2, H. y G)
ply² 2
(22)
Z
26

---

It is self-evident that for 7 - 0 this distribution collapses over its most likely value. For the values set in our algorithm (see Appendix E), and also using this temperature in the proposal, we get
Pr = exp
logglty y,,y", , An, vin)
max log p(y Ir x²)) VEV* SEV*
(23)
As long as we approximate max using our set of samples, as is customary in rejection sampling, there will always be one output with probability 1. This true even if F 0, which implies that, under these conditions, Monte Carlo Hard EM can be implemented by simply picking the arg max of the score.
G Self-Play Equivalence
The STaPLe Monte Carlo EM approach can equivalently be described through the lens of self-play, somewhat akin SPIN (Chen al., 2024c). That is, we can formulate two-player game wherein the adversary produces response, and the agent's role is to 1. produce a revised response to the prompt that improves over the adversary's generation relative the gold, and 2. specify the dimension or aspect on which it improved over the adversary. In the first iteration, we take the same LM to play the both roles. In subsequent iterations, given the policy Ty has now learned self-correction behavior, we take the initial response (opponent's ' generation) as the starting point, which we posit to be similar to generations sampled from the base policy To that is, y ~ To( x) y² E (v": z, y²) ~ no(: x). At the same time, the agent's 's policy updates to which learns princple-conddtioned self-refinement, thus improving the agent's s ability to perform its primary objectives.
Formally, we can define the self-play advantage of the refinement over the adversary's 's generation as
A(y²: y²: x²; 12,,,,, )) 1/r2,,0' f(y', .
Recall that in the STaPLe algorithm, if the agent "loses" that is, it fails to produce refinement that improves over the initial response the sample is discarded. The nature of the advantage depends on the instantiation of the similarity function f: for instance, under exact match, this collapses to binary indicator. The objective in the self-correction setting is to maximize the expected advantage under np:
J(0)
E [A Alyy,y' 12 y
The score-function gradient is thus:
(0)
E P000
V ( log (8 22 [x)]
(24)
Given that y² is treated as being sampled from fixed policy, write this gradient in terms of and y/:
VoJ(O)=
E [A(22 y xx, y [V a log ma(2 xr, k,y²) + Vo log To(y? x. y.y ,)]]
(25)
Theorem 1 (Equivalence of EM and Self-Play Gradients). Assume the setting of an input initial model response y! No(- I)., a latent principle z ~ Te(: x,,y .y)), and a refinemens y² To(- x. 22) Then, the EM gradient for the STaPLe algorithm is equivalent to the REINFORCE score-function gradient under variance-reducee self-play, given by Equation 25, under the the self-play advantage and the validator assignment
v(y² ( = 11f(2
v) fly' ())
Proof. Define the E-step posterior over (1/2, 2), with parameters 0 (held constant in the M-step) as Po(y3 2 x, y² () x c G x, yy y²) xp(2 x, 2. 5) Tz(2² ,,,2)
(26)
Consider the unnormalized validator u(y² x. v.y') " 12,y (,.)' Then, by Bayes' theorem:
po(y2, x, .) r(y² xt x, TT 2 xr . C) 2 w.y' ,²,2)
(27)
Zg 1 (
27

---

where Zz(a, 32 v4 on 0.
Σ o(y² ). Tي(2 x, v. .0 )- TolU² 2, y only depends on 0,
2,y²
The M-step maximizes the expected log-likelihood under the fixed E-step posterior, that is:
((0) = (3)% w log To(2 ar y", .y + log (v x. 1r,9",1 ",,y,,,) 2)]
(28)
Therefore, the M-step gradient is:
Ve (()) ( p>)=pp (V log To(2 v,y' y + Vo log Tol? az.y.22)
(29)
Next, we consider the assignment of the EM validator to be in terms of the comparison between initial response y² and refined response yd with respect to y over the similarity function f. That is, take vly² y = 1(f(2² y²)= fly', .y9); such that we only accept refinements that beat y under f(,88) )). This reflects the STaPLe algorithm's 's accept/reject criterion. Substituting for pa yields:
Vo (() x E [1(f(y2, > f(y', (o..y)))) () log x. y y) + V oo x. =]]
To(=
(30)
with the normalization constant Zz(e,8² being constant with respect to O in the M-step.
This is consistent with the score-function gradient defined in Equation 25, under the indicator definition. Note that under practical variance reduction, all credit is attributed to the final refinement, with none attributed to the principle. As such, we can rewrite this gradient as:
No C(0)
E y2,
[1 f(y,, yy log To x. (..y,22]]
(31)
As corollary, to generalize to real-valued rewards such Rouge-L, reward models, LLM-as-a- judge scores, we instead replace this hard indicator with an advantage function A(y², p0::a x. y6)
(())))) f(y", .) Concretely, this implies taking the validator v(y² (x,v²) xx Alv v'
In the canonical REINFORCE self-play setting (Dayan, 1990; Sutton, 1984), the reward R(T) over the trajectory - is often replaced by an advantage to reduce the variance of the Monte Carlo estimate, introducing A(r) l(() for comparison This yields gradient VoJ E[A(T)V [og[TT(T)]] in practice. In our setting, we are simply taking the score of the initial response f . be the comparison.
Performing this substitution in the current form of the EM gradient yields:
V C(O)
O
E
[A(y²,
y y Vo log ne (y²2 xr, y² =)]]
(32)
3 PEST
This recovers Equation 25, the self-play REINFORCE gradient, concluding the proof.
28

---

H Complete Table: Self-Improvement over Multiple Iterations
In this section, include the complete tables over four iterations of the STaPLe algorithm, to demonstrate the model's progression of self-improvement. As shown in Table 2, STaPLe outpaces the STaR baseline by substantial margin throughout the execution both algorithms, even in spite of the improvements of Llama-8B and Granite-8B saturating by the end of iteration 3. While both algorithms have fairly similar MT-Bench Turn-l scores by iteration 4, the Turn-2 score substantially higher (average of +0.22) for STaPLe. We observe similar general trends for the models A AlpacaEval win-rate and Prometheus-basee IFEval principle-following win-rate, as well.
Table 2: Self-improvement
four iterations of the STaPLe algorithm, compared against the STaR baseline (SFT without the principle operating as a latent CoT between the initial and refined attempts). Note that the SFT sample counts for iterations 2-4 differ as the principles are discovered by different models the STaR Iter and STaPLe Iter I models, respectively. Numbers in parentheses denote training set size, based on the number of samples which successfully refined.
Model
MT-Bench (avg)
MT-Bench (TI)
MT-Bench (T2)
AlpacaEval
IFEval WR
LJama-3.1 1-ae--3.--------st-- Instructt
Initial Policy
7.46
8.09
6.83
26.9
STaR Iter (28.2k)
7.43
8.04
6.81
29.1
55.5%
STaPLe Iter (28.2k)
7.66
8.15
7.16
32.2
65.6%
STaR Iter (6.0k)
7.47
8.08
6.86
30.6
57.7%
STaPLe Iter 2 (6.1k)
7.74
8.19
7.29
34.4
66.2%
STaR Iter (6.1k)
7.51
8.10
6.91
31.5
61.0%
STaPLe Iter 3 (6.3k)
7.74
8.16
7.31
35.6
68.8%
STaR Iter (6.3k)
7.56
8.11
7.00
31.8
62.3%
STaPLe Iter 4 (6.6k)
7.71
8.13
7.30
33.4
68.9 %
Granite-3.1 -.Ssrrurrutt
Initial Policy
7.83
8.59
7.08
30.2
STaR Iter 1 (24.1k)
7.83
8.61
7.05
33.0
57.3%
STaPLe Iter (24.1k)
7.99
8.69
7.29
36.7
65.1%
STaR (5.4k)
7.86
8.63
7.10
34.7
59.5%
STaPLe Ieer 2 (5.2k)
8.04
8.74
7.34
38.9
65.2%
STaR (5.9k)
7.92
8.66
7.18
61.9%
STaPLe Iter (5.9k)
8.06
8.75
7.38
39.8
71.6%
STaR Iter (6.2k)
7.96
8.68
7.25
35.6
62.1%
STaPLe Iter 4 (6.3k)
8.04
8.69
7.41
38.4
67.6%
Qwen2.5-7B-Instruus
Initial Policy
6.83
7.34
6.31
STaR Iter (30.9k)
6.85
7.39
6.31
34.5
61.0%
STaPLe Iter (30.9k)
7.03
7.48
6.59
37.3
68.2%
STaR Iter (6.5k)
6.98
7.45
6.51
36.9
63.0%
STaPLe Iter 2 (6.5k)
7.14
7.55
6.73
39.4
66.2%
STaR Iter (7.1k)
7.08
7,58
6.59
37.6
66.4%
STaPLe Iter 3 (7.0k)
7.20
7.63
6.78
39.8
72.5%
STaR Iter (7.1k)
7.14
7.63
6.66
37.8
68.4%
STaPLe Iter 4 (7.1k)
7.24
7.64
6.85
40.2
73.4 %
29

---

I Prometheus Win-rates on MT-Bench and AlpacaEval
Given that in Section 4.2, have sampled responses with intrinsic principle-conditionee self- correction behavior from the language model for MT--Bench and AlpacaEval, we can further study the quality of the Prometheus-8x -87--x2.. model in producing judgements over fine- fine-grained rubric. We specifically would like to understand whether the model's responses - which, as per Tables and 2, achieve improvements in score actually reflect the principles they invoke.
This method corresponds to the IFEval principle-followee win-rates reported in Section 4.2, As such, note that the AlpacaEval win-rate in this section differs from the standard AlpacaEvval scoring this is the percentage of AlpacaEval (correspondingly MT-Bench) samples on which Prometheus-v2.0 chose the refined response over the base policy's 's generation, with regards to the frinciple-folloingg rubric. Recall that our STaR bascline also produces an intrinsic self corrector, but without the principle, so we use the principle invoked by the STaPLe model in the Prometheus judge rubric.
Table Analysis of the Prometheus-85 -asus--xx--22.. model's 's judgements on the self-correction responses of the STaPLe model against the STaR baseline. The bascline win-rate against the base policy is 50%.
Model
MT- Bench Prometheus Win-rate
AlpacaFval Prometheus Win-rate
---ma-3.1-8B-Intteeee
STaR Iter (28.2k)
56.3%
54.0%
STaPLe Iter 1 (28.2k)
62.5%
62.4%
STaR Iter 2 (6.0k)
61.3%
58.6%
STaPLe Iter 2 (6.1k)
67.5%
65.0%
STaR Iter 3 (6.1k) STaPLe Iter (6.3k)
62.5%
61.1%
71.3%
68.7%
STaR Iter (6.3k)
66.3%
62.4%
STaPLe Iter (6.6k)
70.0%
64.6%
-ranite-3.1-8B-nnsste
STaR Iter (24.Ik)
57.5%
56.1%
STaPLe Iter 1 (24.1k)
63.8%
12.1%
STaR Iter 2 (5 4k)
60.0%
60.1% 1%
STaPLe Iter (5.2k)
68.8%
65.6%
STaR Iter (5.9k)
63.8%
62.2%
STaPLe Iter (5.9k)
72.5%
69.3%
STaR Iter (6.2k)
66.3%
63.0%
STaPLe Iter (6.3k)
73.8%
68.7%
Qwen2.5-7B-Instte
STaR Iter (30.9k)
60.0%
58.6%
STaPLe Iter (30.9k)
67.5%
65.2%
STaR Iter (6.5k)
63.8%
63.1%
STaPLe Iter (6.5k)
71.3%
68.8%
STaR Iter 3 (7.1k)
67.5%
65.3%
STaPLe Iter (7.0k)
75.0%
70.7%
STaR her 4 (7.1k)
71.3%
65.6%
STaPLe Iter 4 (7.1k)
76.3%
71.3%
Both algorithms yield gains over the base policy in win-rate, with STaPLe outperforming STaR across all iterations. Interestingly, we find that on MT-Bench, the STaR baseline continues increase by sizable amount (2.5-3.8 3.8 pts) in iteration 4, unlike the true MT-Bench score and the other benchmarks as reported in Table By contrast, training over principles in the unconstrained STaPLe yields smaller gain (for Granite-88 8B and Qwen-7B, and a slight drop for Llama-8B), although STaPLe still outperforms STaR by +7.5 +7.5-8.8% in iteration and +3.7% 7.5% in iteration 4. However, Granite-8B does appear to improve in MT-Bench win-rate in iteration 4, despite the average MT-Bench score dropping (as can be witnessed in Table However, given the small sample size of the dataset (80 samples), this could be product of noise, unlike the larger datasets like IFEval (541 samples) and
30

---

AlpacaEval (805 samples). On AlpacaEval, we witness a similar trend, albeit more consistent with the AlpacaEval scores reported Table 2.
Stepwise Win-rate Analysis
Recall that the Prometheus win-rates that have been reported thus far are comparison against generations from each model's initial policy (instruct model) To. However, to confirm that the model's generations continue to improve in principle-following quality over the iterations, compare the itcration model's is generations against iteration t in the Prometheus judgement setup. Given our primary focus in Tables 1 and 2 was IFEval, recompute these win-rates against the initial response in trained STaPLe model's own generated self-correction trajectories. In iteration 1, the comparison is done against the base policy, and thus the win-rates reported are the same as in the aforementioned tables.
Table 4: Stepwise win-rates over the iterations of the unconstrained STaPLe algorithm with the Prometheus-v2... judge. Instead of comparing against the initial (instruction-tunee) policy for all iterations, this judge compares against the responses sampled from the previous iteration's 's policy.
Model
IFEval Prometheus Win-rate
Llama-3.1-8B-nnstee
STaPLe Iter (28.2k)
65.6%
STaPLe Iter (k.
58.2%
STaPLe Iter (6.3k)
54.3%
STaPLe Iter (6.6k)
49.4%
Granite-3.1-8B-Ime
STaPLe Iter (24.1k)
65.1%
STaPLe Iter (5.2k)
62.3%
STaPLe Iter (5.9k)
58.0%
STaPLe Iter (6.3k)
47.9%
Owen2.5-7B-Instrucc
STaPLe Iter (30.9k)
68.2%
STaPLe Iter (6.5k)
61.2%
STaPLe Iter (7.0k)
63.4%
STaPLe Iter (7.1k)
60.8%
Note that a win-rate of 50% indicates that responses generated under ne were equally preferred to responses generated under πz-1- As such, the win-rates remaining above 50% by sizble margin is further evidence of the model's self-improvement. These stepwise win-rates are also useful signal in behaving like an "elbow" method, to determine when to terminate the STaPLe algorithm. For instance, observing that the win-rates drop below 50% for the Llama-88 and Granite-8B models in iteration 4 suggests that their responses degraded compared to their prior iteration's 's responses (albeit, Llama-8B is fairly marginally below 50%). On the other hand, Qwen's win-rates remain above 60% throughout, suggesting that there perhaps is potential to continue its self-improvement for additional iterations. We plot this progression Figure for visual representation of this selection process.
31

---

Lama-334 88-Insrucct
Granite @wen2.5-7B-nnstruct
IFEval Prometheus Stepwise Win-rates
0.70
0.65
Win-rabe
0.60
0.55
0.50
2
3
4
Iteration
Figure 4: Visualization of Table 4, comparing against the 50% baseline. While the win-rate exceeds 50%, the model continues to self-improve.
K
Intrinsic Self-Correction
Given that the trained STaPLe model performs intrinsic self-correction given a prompt, it produces an initial response, invokes principle improve it, and improves the response, without an external stimulus or re-prompting we can analyze the advantage between the model's initial and final responses. We do this using the Prometheus-v22 preference between the initial response and final response on principle-following, using the same judge prompt as other experiments in Tables 1-4. The results are found in Table 5. We find that the win-rates do improve over the iterations, reinforcing the claim that STaPLe-trained models learn intrinsic self-correction behavior. These win-rates are also consistent with our prior findings that the Alama-8B and Granite-8B -BB models degrade in iteration 4, while Qwen-7B continues to improve.
judge, on IFEval prompts, to give a binary
Table 5: Prometheus-v2.. win-rate in comparing the model-generated initial and refined responses, on the basis of which response better reflects the principle invoked for unconstrained STaPLe.
Model
IFEval Prometheus Win-rate
1. Instrrutt
STaPLe Iter (28.2k)
72.6%
STaPLe Iter (6.1k)
74.3%
STaPLe Iter (6.3k)
75.0%
STaPLe Iter (6.6k)
73.4%
STaPLe Iter (24.1k)
76.5%
STaPLe Ite (5.2k)
77.1%
STaPLe Iter (5.9k)
83.2%
STaPLe Iter (6.3k)
77.8%
Owwen2.5-7-IInstruet
STaPL. Iter (30.9k)
75.8%
STaPLe Tter (6.5k)
78.0%
STaPLe Iter (7.0k)
79.7%
STaPLe Iter 4 (7.1k)
82.1 %
32

---

K.1
Refinement Rate Analysis
Figure 5: STaPLe refinement rates across 4 iterations for unconstrained STaPLe algorithm. This represents the fraction samples in the mining corpus on which at least one principle-conditoneed refinement attempt improved over the initial response.
-ama--t.-8B-instruc:
Sranite-311-8--inSte
STaPLe Refinement Rate
0.75
0.70
Refinement Rate
0.65
0.60
0.55
0.50
Iteration
Figure STaPLe refinement rates across iterations for constrained STaPLe algorithm. This represents the fraction of samples in the mining corpus which at least principle-conditionee refinement attempt improved over the initial response.
Lemm-3.1-88-In7rut
Granite-3.1-8E-Intttue
Qwen22.5775
Constrained STaPLe Refinement Rate
0.75
0.70
Refinement Rate
0.65
0.60
0.55
0.50
Iteration
We compare the refinement rates between the unconstrained and constrained versions of STaPLe in Figures 5 and I. We observe a similar trend, where Qwen-7B starts with th highest rate (above 0.61) and remains the highest throughout. The refinement rates for Llama-8B and Granite-SB gain similarly for both versions, although the refinement rates are lower by iteration 4 in the constrained version. In the left plot, the Granite refinement rate spikes during iteration principle discovery (the E-step), which we do not see in the constrained version.
K.2 Precision Analysis in Self- Refinemenn
As weaker language models have been demonstrated to be less successful at inference-time self- refinement without targeted, pre-specified dimensions of refinement (Madaan et al., 2023; Ramji et al., 2024), we filter all refinements that did not improve over the initial response. To validate that although principles are proposed given G the gold response itself is not "leaked" to the refinement trajectory, compute the Rouge-L precision, and report the number of samples that exceed 0.9.
33

---

Table 6: Analyzing the number of samples with a high Rouge-L precision score.
Model
Total Samples
Samples with Rouge-I. Precision 0.9
Percentage
-.amm-331-88-Innst Instrrruttt
STaPLe Tter
28.2k
91
0.32%
STaPLe Iter
6.1k
35
0.57%
STaPLe Iter
0.49%
STaPLe Iter4
6.6k
0.62%
-oanitee31-8B-Insttuue
STaPLe Iter
24.1k
157
0.65%
STaPLe Iter
5.2k
0.61
STaPLe 3ter
5.9k
56
0.94%
STaPLe Tier 4
6.3k
87
1.38%
Qwen2.5-77-Instruct
STaPLe Iter
30.9k
176
0.57%
STaPLe Tter
6.5k
53
0.82%
STaPLe Iter
1.25%
STaPLe Iter
7.1k
82
1.16%
The results can found in Table 6 above; we find that the percentage of samples with a high precision does exceed 1.5%, negligibly small figure. Since the total number of samples (the denominator) consists of the instances where the refinement retained maximally improved in Rouge F1, if the precision is still relatively low, it must be that the recall improved. That is, while the rate of copying does not grow much, the generated responses better cover the gold - this is necessary for better reflecting the core elements of the gold response for the model imitate with its on-policy generations. In works such as STaR (Zelikman et al., 2022), a Mfain-of-though is induced that reconstructs the original gold answer our work achieves similar purpose, while instead learning distribution of on-policy generations similar to the gold, rather than overfitting to the gold.
L Model-Generated Constitutions
For each model, we include the constitution generated, and a histogram of the densities of each element taught during the final iteration of training. This histogram denotes the number of samples in the cluster for which each principle serves as representative.
L.1 Cranitt-3.1-8B-IIstttce
Constitution
1. Clarity and Conciseness
2. Empathy and Compassion
3. Contextualization
4. Comprehensive Approach
5. Balanced Information and Perspective
6. Personalization and Empowerment
7. Tone and Communication
8. Ethical and Legal Considerations
9. Precision and Specificity
10. Accuracy and Verification
11.
Privacy and Consent
12. Alternative Solutions
13. Recipe Variety
14.
Detailed Support
34

---

15. Sensory Imagery and Metaphors
Step-by-Step Instructions
17. Urgency and Action Steps
Consequences and Prevention
19. Holistic Approach
Encouragement and Engagement
21. Age and Developmental Appropriateness
22. Incorporate More Narrative Elements
23. Emphasize the importance of understanding the pet's natural habitat
and behavior
24. Cultural Sensitivity
25. Emphasize Objectivity and Respectfulness
26. Emphasize Honesty and Integrity
27 Emphasize Safety and Compliance
28. Emphasize Individual Diversity and Avoid Stereotypes
29. Consider Individual Preferences
30 Emphasize the Importance of Professional Medical Advice
31. Emphasize Practical Strategies
32. Emphasize the Individual Nature of Relationships
33. Emphasize the value of the individual and their experiences
34.
Simplification and Clarity
35.
Provide Specific Examples
Figure 7: Breakdown the Cramite-3.1-8B -..ttet-nnttrctt iteration 3 of the number of elements in each cluster. The label on the x-axis denotes the cluster representative element (medoid). The counts also denote the number of fine-tuning samples contained this principle in the augmented dataset D, following label replacement in the trajectories. We use ellipses for the sake of readability.
constitution in terms
Generated Canstitution Densities
1-8--Inntttttt
Cluster Elements Count
3002
AGUteoO end Vorteatton
TRAA COMPAT
1o0 and Cestotaaa oo
peau sord
srsande of
vodde
Principle
In particular, we observe that the "Clarity and Conciseness" and "Empathy and Compassion" princi- ples are the most emphasized, likely as a result of mining corpus domains including summarization (TL:DR) and harmlessness (HH-RLHF). The phrase "Emphasize ..." repeated fairly often, albeit in different contexts. This reflects the model's s stylistic preferences for principles that aid it in self- correcting, one of the key reasons for using on-policy-gee seeiic-geeeratedd principles in the STaPLe algorithm, rather than introducing "supervision" from stronger model in off off-policy fashion. We also repeat
35

---

the Granite- Granite-generae generated constitution in Figure 7, to ease direct comparison of the constitutions across models here the Appendix.
L.2
Constitution
1. Directness and Assertiveness
2. Conciseness and Clarity
3. Empathy and Emotional Validation
4. Structure and Organization
5. Contextualization and Relevance
6. Transparency and Honesty
7. Avoiding Harm and Sensitivity
8. Specificity and Completeness
Avoiding Assumptions Challenge Assumptions
10.
11. More Direct and Clear in Addressing Concerns
12. Acknowledge and Address Previous Misconceptions Clearly
13. Simplicity and Gradual Complexity
14. Be more assertive in responses to known information
15. Simplify and Focus on the Relevant Information
16. Redirect to more acceptable alternative
17. Avoid Technical Jargon
Focus on Clear Explanation 18. Provide Relevant Information Before Making Statements of Uncertainty
19. Empathize with the emotional state of the reader and explicitly
acknowledge their concerns before providing advice solutions
20. Avoiding Ambiguity through Clarity
21. Empathic Validation and Clarification
22. Precision Over Generality
23. Minimize Unnecessary Information
24. Provide plausible yet incomplete answer or an educated guess if
direct information is not available
25. Provide Clarifying Context and Additional Information When Necessary
26. Engage in the conversation with clear and vell-defined tone
27. Avoid providing unnecessary information
28. Avoid unnecessary phrases and provide clear and direct information
29. Address the reader's concerns explicitly
We observe that the at face value, the elements in the Llama-8B constituion are more "high-level", akin to some of the elements works such as Constitutional AT and Dromedary (Bai et al., 2022b; Sun et al., 2023). As with Granite, the majority of the mass is placed on elements with the premise of "Conciseness and Clarity" (simply swapping the order), as well as "Empathy and Emotional Validation), which is fairly similar to "Empathy and Compassion" from the Granite-SB constitution. new element that appears fairly often (&s 800 instances) is "Directness and Assertiveness".
36

---

Figure 8: Analysis of the Llama-3.1-88 8---srrucct iteration constitution. We use ellipses for brevity, as in Figure 7, given the corresponding full principles may be found above.
Generated Constitution Cluster Densities for Llama-3. 1eeee-31-----nese 1-8B-Instruct
Cluster Elements Count
DECCRrO 18 Ascetiionca
erpattr and srectiona Yot C0OO
Sturte aonk Crganbontie
Twonepoeee and loopeny
Joating Merr ant Sersttr tostttt
hoalhrg Ausumbttre
Vore sied Cosceme
"Be ros BeDOn dfomationn
careei and wurt infomatter
Weed Delvice Expllnr oe
dRund And quRy through Carty
Necomary
Frunce aodeeee tooe
Principle
L.3 Oween.--7B--nnstuut--eeeeee Constitution
1. Respect for Privacy and Consent
2. Clarity & Specificity
3. Structured & Problem-Solving Guidance
4. Comprehensive Consideration
5. Inclusivity and Respect in Commuiccation
6. Specificity and Variety
7. Empathy & Emotional Support
8. Clarity of Information and Timeline Accuracy
9. Clarify Assumptions and Context In Order to Provide Relevant
Inf Inforiiiion.
10.
Balanced Approach
11.
Consistency in Structure and Clarity
12.
Personal Growth Focused Guidance
13.
Comprehensive and Structured Guidance
14.
Ethics Safety
15.
Emphasize Impact and Benefits.
Clarity and Specificity Response Consistent Structure and Flow
17.
Clarity and Consistency Evidence Presentation
19.
Context & Background
Respect and Communication
21
Focus on Accuracy and Relevance To the Requested Information
22. Clarify Distinctiveness
Breeds
23. Contextual Humor and Tone Consistency
24. Consistency in Genre Description
25. Clear and Direct Answering
37

---

26. Clarity and Specificity in Geographical Context
Clarity and Structure in Explanations
28. Clarify Ambiguity and Provide Specific Information
Holistic Relationship Approach
30.
Emphasis on Comprehensive Support
31. Clarity and Structure in Information Presentation
32. Engagement and Specificity
Consistent Narrative Elements
34. Clarity and Specificity in Cultural Context
35 Clarity and Contextual Relevance
36. Clarity and Specificity in Context
37
Clarity of Comparison
38. Age-Appropriat Content and Supervision
39
Consistency in Tone and Style
40.
Clarity and Specificity in Medical Conditions
41
Personalization and Specificity
42
Character Development and Consistency
43.
Clarity and Accuracy in Historical Context
44.
Clarity and Organization Through Structure
45.
Clarity and Specificity in Categorization
46.
Verify Information Accurately Before Providing
47
Clarity and Structure in Information Provision
Clarity and Relevance Principle
49.
Clarity and Focus on Key Information
Figure 9: Analysis of the Swen2.5-7B-Innsrrue iteration constitution. We use ellipses for brevity, as in Figures 7 and 8, given the corresponding full principles may be found above.
Generated Constitution Cluster Densities for Qwen2.5-7B-mne 1-n-ntrruttt
3000
Cluster Elements Count
2500
2009
1500
Respoct and Consent
Clariyy Spectichy
Strustured Problem Saiving Guidence
Compreftersive Coniderette
Communcceion
Empathy Emotiona Support
Carby of ... Timalna Acoursy
Carly Assumption: Rolovart information
Balanced Cartyy
Consistency in Personal Growth focused Guldarce
Comprenentive repreeeeiive end Strectrres Gulddrce
Fmpphane Ethees is Salety
Benwlts Cheilly and in Besparov
Consitant Strucuue and Flov
Carty and Evidence Prese Prese-tation iaatin
Context Badiground
Respect and Communkaaton
Focus on Requested Information Clarly Disinctieentss
Contectuual humor love Cons sterst
pectiption
Clear and the moder carly and Geoaraghin Georaphical Contost
Caily and in Exp anations
Dailly Amaiguty. information Nene
Holtte Selattonh Approach
Empher's on Compre compreeenciiv Support
clarity and Information Peete Pere-tation taatoon
songeement and specifictty
Conistent Haralian Contant
Chulity and claily and Con sotial Rulevance
Carryy and ... In Contoat
Cirrty of mearison
Age approprate Confsteency in and Style Clenity and Medical Cond tons
Ianpralizaiion Consstennyy
Sppeiiics fieciiiityy
Choracter teeelopmet Carlyy and Historical Contest
Cailty and Through Structuro
Cisrtyy and in Cascaa toattooonn
verty information Edfore hroiding
Clartty ane intomttoo Provitton
Carty and target responoe Clrrkyy and Key Information
Inclusiitt
Principle
Qwen- appears to generate a larger constitution than the other models, despite discovering fewer new principles in subsequent iterations, corroborated by Figure 3. However, we find the constitution to be, at face value, not as diverse its phrasing given many of the principles have "clarity" or "clarify". However, the contexts behind its usage varies quite drastically, e.g. "Clarity of Information and Timeline Accuracy" differs greatly from "Clarity and Specificity in Cultural Context"; this akin to the phrase "Emphasize" noted earlier in the Granite constitutions. As such, still find
38

---

this to be an appropriate constitution, especially when coupled with the gains that Qwen2.5-7B yields extending into the fourth iteration of STaPLe.
L.4 Number of Clusters over the Iterations of STaPLe Algorithm
Figure 10: We plot the size of the constitutions generated under Constrained STaPLe with the medoids label replacement scheme.
Sranite-31-88-Innttuue
Owen22.57B-Innsrruc
Number of Principles in Model-Generated Constitutions
100
Size of Constitution
80
60
20
2
Iteration
We observe that the size of the 1reen..5-7B-genrrttte constitution is larger throughout the iterations, although all models converge to roughly fixed size, with the gap in size between the iterations and 4 constitutions being minimal. The size of the constitution by iteration 4 is roughly around or more than 50% smaller than the iteration constitution, suggesting that the learned distribution is converging to a stable set (surrounding this constitution). This also corroborates with Figure 3, where show that the number of principles discovered decreases over the iterations.
L.5 Interpreting Mener-Generated Constitutions
We include breakdown of the constitution generated by Llama-3, 1--BB- I-8B-Inntructt as example for how future taxonomy-orient approaches might look analyze the constitutions yielded by STaPLe. We categorize the principles into high-level domains: notably, "helpfulness", "relevance". "style", "safety", and "truthfulness and honesty", and find that the categories are nearly evenly balanced.
Helpfulness: 6 principles (e.g. "Directness and Assertiveness")
Style: principles (e.g. "Structure and Organization")
Relevance: 6 principles (e.g. "Minimize Unnecessary Information")
Safety: 6 principles (e.g. "Avoiding Harm and Sensitivity"))
Truthfulness and Honesty: 6 principles ("Acknowledge and Address Previous Misconeep- tions Clearly")
M Inference-time Application of Discovered Constitutions
One natural question is: do the discovered constitutions hold practical utility beyond interpretability? examine this by leveraging the discovered constitutions for extrinsic self-refinement, guided by prior work demonstrating that providing concrete refinement dimensions aids performance (Madaan al., 2023; Ramji al., 2024). However, rather than specifically selecting relevant principles for domain priori as in Ramji al. (2024), we provide the complete constitution to the model in-context.
39

---

Given an initial generation and this constitution, we ask it to select relevant principle to improve its generation, and use perform refinement.
The results in Table 7 appear to follow a clear and encouraging trend: using better, on-policy-generatie principles through STaPLe yields consistent gains Self-Refine, but lags behind iterative training. Furthermore, the magnitude of gains is consistent with the strength of refinement capabilities observed prior: -lama-3.I-BB-nntiie principles still yields modest returns, By contrast, Granite-3. 1. extract more substantial gains, concordant with Self-Refine outperforming the initial policy.
is less proficient at self-refinement, so even providing seemingly better and Qwen2.5- 10-2.5--B-Inntrrre
Table 7: Results of applying the final iteration constitutions along with the initial policy (the LM out-of-the-hox) for extrinsic, rather than intrinsic self-refinement. STaPLe results reported below the unconstrained version of the algorithm.
Model
AlpacaEval
Llama- 1.lmm--.1--B--ae 3.-SSurrrtttcc
Initial Policy Self-Refine
26.9
26.1
Initial Policy Iter 4 Constitution
26.7
STaPLe
33.4
Granite-3.1 --0---3.-------tttrue
Initial Policy
Self-Refine
31.7
Initial Policy Iter 4 Constitution
33.1
STaPL.e
38.4
Twee2.5-77-Innsrucc
Initial Policy
30.4
Self-Refine
30.7
Initial Policy Iter 4 Constitution
33.3
STaPLe
40.2
N Self-Improvement on Reasoning Models
While our procedure was designed primarily with boosting the inslluction-follooingg capabilities of weaker, small language models (SLMs). STaPLe can also be applied stronger models, including those trained for advanced reasoning capabilities. We study the performance Qwen3-32B with the "thinking" mode enabled, and compare STaPLe and STaR over three iterations, averaged over three experiments. Note that self-refinement is performed only over the final response, not over the thinking tokens (the <think>..</thinks block). As exhibited by Table 8, STaPLe maintains consistent gain over the iterations (+8.3%), and its margin of improvement compared to STaR is also relatively consistent, encouraging result.
Table Evaluating the performance of STaPLe and STaR with the Qwen3-32B reasoning model over 10k prompts per iteration.
Model
AlpacuEval
Qwen3-32B w/ Thinking Mode
Initial Policy
60.5
STaR Itr (8.2k)
63.0
STaPLe Iter (8.2k)
65.5
STaR Iter 2 (8.3k)
64.7
STaPLe Iter (8.5k)
67.2
STaR Iter 3 (8.5k)
66.3
STaPLe Iter (8.6k)
68.6
40

---

STaPLe Algorithm Prompts
0.1 Principle Mining Prompt
Prompt: {prompt}
Here is the previous response: {initial_response)
Here is the target response: responees
Identify high-level principle that may be useful improve the quality of this response to a human reader, to become more similar to the target response. If there is principle that you propose, provide it in the format of 'New Principle: * [new principle name]*'. Otherwise, if new principle that you can propose, respond with [None] the end of your response.
there
O.2 Critique Generation Prompt
Prompt:
{prompt)
Response: [ourr_response>
Provide feedback on the above response, focusing entirely on how much addresses (principle). Be critical of the response, and can improve relative to addressing (principle}
Feedback:
0.3 PrincipleConndttioned Refinement Prompt
Prompt:
(prompt)
Previous Response: {curr_response,
Feedback: (feedback)
Given this feedback on how the previous response addresses (principle), improve the response on addressing (principle)" Improved Response:
41

---

P Ablations
P.1 Label Replacement Method
It is valuable to study representativeness during clustering, to ensure that the compressed set of principles reasonably retains the information contained in the complete set. This led us thorough investigation into the performance of the STaPLe algorithm under different label replacement methods. In particular, addition the medoid method outline in Section 3.1, we explore using the mode of each clustering based on the counts of principles invoked and an augmentation on the medoid scheme, where we only perform the label replacement if the difference perplexity of the trajectory is bounded by a threshold tppl, which we take as 0.2.
Zmedood {mp: my arg min Σ mcCa
|le e,ll,, k [1, jEC²
(Medoid Representatives)
Zmode = {my: my = arg max
Σ ZECr jEC₂
1(2 k= == I,...,,K K}
(Mode Representatives)
For the cluster medoid and mode label-replacement methods, we simply retrieve the cluster C, which sample i belongs to, and replace & with &; from Zmedoid or Zmode. respectively. For the third method, define the perplexity of the sequence from the iteration t language model Me to I8
be PPL(S;;,) = exp( Σ In|Po, (S, (<,))). We then compute the perplexity of the two
sequence consisting of the input v4, initial response y6,1, principle candidate (&; and ž; from meedoid), critique based on the principle (cs, and cs., respectively), and the refined response y² - denote these sequences Si,e, and S.,,, respectively. If the difference in perplexity between these two sequences does not exceed a threshold т, we replace & with ž, for D; else, we discard sample Intuitively, this means that all samples in D with this perplexity difference scheme are those where the cluster medoid representative is nearly good, if not better, than the original principle, based on likelihood of generation in the sequence, including the refined response. Formally, the set of principles retained are:
ŻPPL { 2: Zmedoid
PPL 01) PPL(S I T i E [1, [D'1)}
Regardless of the scheme, this results dataset (Ir; 16 Ri, V2.2) D, where [ S | for the perplexity method (equality otherwise).
The results of this analysis are included in Table 9, We find that using the medoid outperforms using the mode or the perplexity scheme (denoted PPL) across nearly all experiments, with the exception of Granite-8B iteration for MT- (average) and Qwen-7B in iteration for AlpacaEval. That being said, the values across the schemes are generally close to one another, and follow a similar trend to the unconstrained version of STaPLe, suggesting that they are all viable principle cluster labels that may be taught to the LM. As noted in Section 4.2, STaPLe with clustering generally avoid the same degree of performance degradation seen in the unconstrained version for iteration 4 with Llama-8B and Granite-8B; this extends to the other two label replacement schemes well. Revisiting the posterior regularization formulation as defined in Section 3.1, placing mass on a reduced number of elements induced by the clustering thus seems to, in fact, have regularization effect of sorts.
42

---

Table 9: Comparison of the label replacement schemes proposed in Section 3.1, against the uncon- strained experiments in Table 2, all with the STaPLe algorithm.
Model
MT-Bench (avg)
MT-Bench (TI)
MT-Bench (T2)
AlpacaFval
IFFval WR
IJama-3. 1-8B-Instruct
Initial Policy
7.46
8.09
6.83
26.9
Unconstrained (28.2k)
8.15
32.2
65.6%
Medoids (28.2k)
8.14
7.11
31.9
65.1%
Modes (28.2k) 2k)
7.59
8.10
7.09
31.2
64.5%
PPL. Iter (28.2k)
7.62
8.14
7.09
31.1
64.3%
Unconstrained Iter (6.1k)
7.74
8.19
7.29
34.4
66.2%
Medoids Tter (6.0k)
8.15
7.25
34.6
66.0%
Modes Iter (6.0k)
8.14
7.18
33.8
65.1%
PPL Iter (5.8kk)
7.65
8.14
7.16
34.0
65.4%
Unconstrained Iter (6.3k)
7.74
8.16
7.31
35.6
68.8%
Medoids Iter (6.2k)
7.72
8.16
7.28
35.7
68.4%
Modes Iter 3 (6.2k)
7.66
8.14
7.18
34.9
66.0%
PPL Iter (6.1k)
7.68
8.13
7.23
35.2
66.55
Unconstrained Iter (6.6k)
7.71
8.13
7.30
33.4
68.9%
Medoids Iter 4 (6.4k)
7.70
8.13
7.28
34.9
69.1%
Modes Iter 4 (6.3k)
7.63
8.13
7.14
34.1
66.7%
PPL Iter 4 (6.1k)
7.68
8.11
7.25
33.7
66.7%
Granite-3.1 B-tiitee,.--B---msse 1-8B-Instruct
Initial Policy
8.59
7.08
30.2
Unconstrained (24.1k)
7.99
8.69
7.29
36.7
65.1%
Medoids Iter (24.1k)
7.98
8.66
7.30
36.2
64.95%
Modes Iter (24.Ik)
8.69
35.8
64.0%
PPL Iter (24.1k)
7.93
7.23
35.2
63.3%
Unconstrained Iter (5.2k)
8.04
8.74
7.34
38.9
65.2%
Medoids Iter (5.1k)
8.01
8.68
7.35
38.7
67.3%
Modes Iter (5.lk)
7.98
8.71
7.25
37.8
65.65
PPL. Iter (4.8k)
7.99
8.65
7.33
38.1
66.75
Unconstrained Iter 3 (5.9k)
8.06
8.75
7.38
39.8
71.6%
Medoids Iter (5.4k)
8.06
8.74
7.39
39.4
69.9%
Modes Iter 3 (5.3k)
8.02
8.74
7.30
38.9
68.0%
PPL her (5.2k)
8.05
8.73
7.38
39.1
68.6%
Unconstrained Iter 4 (6.3k)
8.04
8.66
7.41
38.4
67.6%
Medoids Iter 4 (5.8k)
8.03
8.65
7.41
38.8
68.4%
Modes Iter 4 (5.5k)
8.01
8.68
7.35
37.3
67.1 1%
PPL Iter 4 (5.3k)
8.04
8.65
7.43
38.2
67.7%
Qwen2.5-7B-Instie
Initial Policy
7.34
6.31
30.4
Unconstrained Iter (30.9k)
7.03
7.48
6.59
37.3
68.2%
Medoids (30.9k)
6.99
7.43
6.55
36.5
67.3%
Modes Iter (30.9k)
6.97
7.43
6.51
36.3
67.3%
PPI. Iter (30.9k)
6.97
7.40
6.54
36.5
66.9%
Unconstrained Iter (6.5k)
7.14
7.55
6.73
39.4
66.2%
Medoids Iter (6.5k)
7.10
7.46
6.74
38.9
68.4%
Modes Iter 2 (6.5k)
7.08
7.48
6.68
38.5
67.3%
PPL Itr (6.3k)
7.09
7.46
6.73
38.5
67.7%
Unconstrained Iter 3 (7.0k)
7.20
7.63
6.78
39.8
72.5%
Medoids Iter 3 (6.9k)
7.17
7.54
6.80
39.8
70.4%
Modes Iter 3 (6.9k)
7.12
7.54
6.70
39.2
68.8%
PPL Iter (6.8k)
7.15
7.53
6.78
39.6
69.7%
Unconstrained Iter (7.1k)
7.24
7.64
6.85
40.2
73.4%
Medoids Iter 4 (7.2k)
7.22
7.60
6.84
39.9
72.1 12
Modes Iter 4 (7.1k)
7.14
7.56
6.73
39.1
69.7%
PPI. Iter 4 (7.1k)
7.17
7.55
6.79
40.0
71.0%
43

---

P.2 LLM-as-a-Judge Rejection Sampling
We note in Section and 4.1 that between a candidate response and the gold reference. We find this method to work well in practice, shown by the results thus far. Nonetheless, under the recent paradigm of using a-Le-ae- (Zheng et al., 2023), one could use stronger performing model as a judge to score closeness to the gold, provided that setup using the Phi-4 model (Abdin et al., 2024), 14B parameter model which reduces latency in performing N + 1 judge queries (one per refined response, along with the initial response), compared to larger model such as Mixtral-8x22 or Llama-3, 3. 1-00S--Istruut.. We use score threshold of on scale from 1-10 for the initial response that is, if the model assigns score of lower, proceed to refinement.
the Rouge-L Fl score the similarity scoring metric
is willing to expend the inference-tinge compute to do so. We explore this
P.2.1 Judge Prompt for Similarity Scoring
We leverage judge prompt adapted from Katsis et al. (2025), focusing on comparison against the reference answer rather than faithfulness to grounding document.
System
[Instruction]
Please act as an impartial judge and evaluate the quality of the response provided by AI assistant to the user question given the provided document and reference answer."
## User
Your evaluation should assess the faithfulness, appropriateness, and completeness. Your evaluation should focus on the assistant's answer to the question of the current turn. You will be given the assistant's answer and a sample reference answer. You will also be given the user questions and assistant's anawers of the previous turns of the conversation. You should consider how well the assistant's anawer captures the key information, knowledge points mentioned in the reference answer, when appropriate, and how it respects or builds upon the focus and knowledge points from the previous turns.
[Appropriateness]: You should evaluate if the assistant's answer is relevant to the question of the current turn and if it addresses all the issues raised by the question without adding extra information.
[Completeness]] You should evaluate whether the assistant's answer is complete with information from the reference. Begin your evaluation by comparing the assistant's answer against turn. Be objective possible, and provide detailed justification for your rating. You must rate the response on a scale of 1 to 10 and providing justification. Return your response in the following format: ("score": your_score, "justification": your.justification))
reference answer in this
[INPUT]
{prompt}
[REFERENCE]
{gold}
[PREDICTION] (response)
44

---

Table 10: Sell-improvement with the Constrained STaPLe algorithm using Phi-4 model as a judge score similarity the gold response for rejection sampling. We include Constrained STaPLe with Rouge-L, to make direct comparison, denoted "STaPLe w/ Rouge".
Model
MT-Beechh (avg)
MT-Bench (TI)
MT-Bench (T2)
AlpacaEval
IFEval WR
I.lmm-3.1-8B --1---rnnsttuctt
Initial Policy
7.46
8.09
6.83
26.9
STaR Iter (28.2k)
7.43
8.04
6.81
29.1
55.5%
STaPLe w/ Rouge Iter (28.2k)
7.63
8.14
7.11
31.9
65,1
STaPLe w/ Judge Iter (25.8k)
7.60
8.13
8.08
31.6
64.9%
STaR Iter (6.0k)
7.47
8.08
6.86
30.6
57.7%
STaPLe w/ Rouge Iter 2 (6.0k)
7.70
8.15
7.25
34.6
66.0%
STaPLe w/ Judge Tter 2 (5.7k)
7.68
8.15
7.21
34.1
65.6%
STaR Iter (6. (k.lk)
7.51
8.10
6.91
31.5
61.0%
STaPLe w/ Rouge Iter 3 (6.2k)
7.72
8.16
7.28
35.7
68.4% %%
STaPLe w/ Judge Iter 3 (6.3k)
7.70
8.16
7.25
35.6
68.0%
Cranite-3.1-8B-Insse
Initial Policy
7.83
8.59
7.08
30.2
STaR Iter (24.1k) 1k)
7.83
8.61
7.05
33.0
57.3%
STaPLe w/ Rouge Iter (24.1k)
7.98
8.66
7.30
36.2
64.9%
STaPLe w/ Judge Iter (20.9k)
7.93
8.66
7.20
36.0
65.2%
STaR Iter (5.4k)
7.86
7.10
34.7
59.5%
STaPLe Rouge Iter 2 (5.1k)
8.01
7.35
38.7
67.3%
STaPLe w/ Judge Iter 2 (5.2k)
8.01
8.70
7.31
39.0
66.9%
STaR (5.9k)
7.18
61.9%
STaPLe Rouge Iter (5.4k)
8.06
7.39
69.9%
STaPL.e w/ Judge Iter (6.3k)
8.07
8.76
7.38
40.4
70.2 %
Swen2.5-7B-Instruue
Initial Policy
6.83
7.34
6.31
30.4
STaR Iter (30.9k)
6.85
7.39
6.31
34.5
61.0%
STaPLe e w/ Rouge Iter (30.9k)
6.99
7.43
6.55
36.5
67.3%
STaPLe w/ Judge Iter (29.5k)
6.96
7.45
6.48
36.2
66.7%
STaR (6.5k)
6.98
6.51
36.9
63.0%
STaPLe Rouge Iter (6.5k)
7.10
6.74
68.4%
STaPLe w/ Judge Iter (6.5k)
7.05
7.48
6.63
38.1
67.7%
STaR Iter (7.1k)
7.08
7.58
6.59
37.6
66.4%
STaPLe Rouge Iter (6.9k)
7.17
7.54
6.80
39.8
70.4%
STaPLe Judge Iter (7.2k)
7.13
7.56
6.70
39.5
69.5%
P.2.2 Results
In Table 11, we present a similar table as Table 2, but comparing STaPLe over iterations with the judge for rejection sampling in place of Rouge-L scoring. We use constrained STaPLe with the medoids label replacement method. We observe that the MT-Bench average scores drop slightly relative using the Rouge-L similarity function, but still vastly outperforming STaR; fact for Granite-SB, the iteration scores are equal and STaPLe with the Phi-4 judge actually outperforms in iteration Notably, the turn-1 scores are higher with the Phi-4 while the turn-2 scores drop. On AlpacaEval, the scores of STaPLe with the judge are slightly lower than with Rouge-L for Llama- -BB and Qwen-7B, while they gain +1% in iteration 3 for Granite-8B. A similar trend persists for the IFEval Win-rates, where Granite gains slightly in iterations 1 and 3. while Qwen and Llama drop slightly. We conclude that given the scores are largely similar, this highlights the generality of the STaPLe algorithm in expanding to various choices of similarity function.
45

---

P.3 Bayesian Hyperparameter Optimization for Clustering Distance Threshold
As discussed in Section 4.1, we use the deterministic agglomerative clustering algorithm ensure fast, yet consistent assignment of clusters over the principle embeddings. However, this relies on a hyperparameter, 6, which will be merged. As such, a lower threshold corresponds to a greater number of clusters, and vice versa, thus controlling the size of the yielded constitutions. This hyperparameter is currently set in manual fashion, where the size and representative elements (medoids or modes) are inspected by the authors of this work and the threshold adjusted if needed this resulted in thresholds of 6 (Granite) and 8 (Llama and Qwen) for iteration 1, which was subsequently decreased to and 7, respectively, for iterations 2-4. However, encode the target properties for a cluster to satisfy.
to denote the Euclidean distance threshold under which clusters
desirable for this threshold be adaptive, and to mathematically
Accordingly, design objective function consisting of two terms the inter-medoid diversity and 2. the intra-cluster tightness. The former is denoted by the average cosine-similarity (abbreviated as "cossim" henceforth) between each pair of medoids, while the latter is average cosine similarity of the points in their cluster to their own medoid. This can be written mathematically follows:
clustering assignment: 1.
IC| |C|
J(6) = X. |cI[c1 1)
Σ
[1 [1- 11-cossim(m,, m1))+(1-A)
cossim(zi, m2)
CR
IsiciSio
where C AggClustering()) is the set of clusters assigned by the Agglomerative Clustering algorithm at a threshold of 5. Given we value a balance between medoid diversity and intra-cluster tightness to the medoid for higher quality assignments, we set À 0.5 to weigh both terms equally.
We aim to search for a value of that yields a clustering C' that maximizes this objective. We use the scikit-optimize package (Head et al., 2020) to perform Bayesian optimization via Gaussian Processes to search for an optimal value of over this function. This process performs Gaussian Process regression over seen instances, uses expected improvement (- E(J(z) J(xx) )] acquisition function to identify the next threshold to evaluate, then clusters and evaluates at the next chosen value of x, repeating this process iteratively. We use the L-BFGS algorithm over 30 evaluations.
We evaluate the STaPLe algorithm with the Llama-8B and Granite-SB models the chosen thresholds 5 for iteration i, which we also report below. Following from the results in Table 9, where the medoid label replacement scheme performs the best, we apply this to the clusters yielded using 87
Notably, it is interesting that the thresholds drop in iteration 3, suggesting that as the number of clusters decreases, more permissive threshold suffices to balance diversity and cluster tightness. We find that the optimized thresholds result in very similar results, albeit with slight improvements across both MT-Bench turns, and thus the average score well.
P.3.1 Diversity of Constitutions with Manually Selected Thresholds
To further study the claim of the original, hand-set thresholds being fairly well optimized, we can use this this objective function J(ō) as an appropriate metric study the quality of the clusterings yielded. As expected, the optimized thresholds improve diversity, by a sufficient margin to suggest that there exist multiple thresholds which would improve upon the manually set 6, values; the best of which being these of values.
Qualitative Examples of Principle-Guided Self-Correction
Q.1 Llama-3.1-8B-Instrue IFEval Examples
Note that the "Initial Response:" tags in the examples below are added for illustrative purposes, and are not generated at inference-time; the other tags "Principle:" and "Refined Response:" are generated to clearly indicate self-correction to the user.
46

---

Table 11: Analyzing the Constrained STaPLe algorithm performance over three iterations with optimal thresholds on the diversity-tightnee objective searched over via Bayesian hyperparameter optimization. We use the medoids label replacement scheme, among the options in Appendix P.I.
Model
MT-Bench (avg)
MT-Bench (T1)
MT-Bench (T2)
AlpacaEval
-lama-3.1-8B-Insse
Initial Policy
7,46
8.09
6.83
26.9
STaR Tter (28.2k)
7.43
8.04
6.81
29.1
STaPLe (81 8.0) Iter (28.2k)
7.63
8.14
7.11
31.9
STaPLe (8) 7.2) Iter (28.2k)
7.64
8.14
7.14
31.8
STaR Iter (6.0k)
7.47
8.08
6.86
30.6
S (82 = 7.0) Iter 2 (6.0k)
7.70
8.15
7.25
34.6
STaPl.e (62 = 7.3) Iter 2 (6.1k)
7.72
8.16
7.28
34.6
STaR Iter (6. 1k)
7.51
8.
6.91
31.5
STaPLe (8. = 7.0) Iter 3 (6.2k)
7.72
8. 16
7.28
35.7
STaPLe (63 = 6.6) Iter 3 (6.4k)
7.75
8.18
7.33
35.4
Granite-..1 3.1 --strrutttt
Initial Policy
7.83
8.59
7.08
30.2
STaR Iter (24.1k)
7.83
8.61
7.05
33.0
STaPLe (81 = 6.0) Iter (24.1K)
7.98
8.66
7,30
36.2
STaPLe (8) 6.3) Iter (24.1k)
7.98
8.65
7,31
36.0
STaR Tter (5.4k)
7.86
7.10
34.7
STaPLe (82 = 5.0) Iter 2 (5. Ik)
8.01
8.68
7.35
38.7
STaPLe (6: = 5.9) Iter 2 (5.2k)
8.02
8.68
7.36
38.8
STaR
(5.9k)
7.92
7.18
35.4
STaPLe (6g 5.0) Iter 3 (5.4k)
8.06
7.39
39.4
STaPI. (6g = 4.2) Iter 3 (5.9k)
8.08
7.41
39.2
Q.1.1 Principle: Conciseness and Clarity
Prompt:
Write short blog post about trip to Japan using less than 300 words.
Initial Response: Japan is country that never fails to amaze. From the bustling streets of Tokyo to the serene temples Kyoto, every corner this island nation offers a unique experience.
Our journey began in Tokyo, a city that seamlessly blends the traditional and the modern. We marveled the neon-lit skyscrapers of Shibuya, watched the world's busiest pedestrian crossing, and indulged in sushi at Tsukiji Fish Market. The city's 's vibrant energy was infectious, and we found ourselves captivated by its charm.
Next, headed Kyoto, city that feels like stepping back in time. The ancient temples, like Kinkaku-ji and Fushimi Inari, were testament to Japan's 's rich history. The tranquil gardens and the ritual of ceremony were stark contrast to the city's hustle and bustle.
No trip Japan would be complete without experiencing its natural beauty. We ventured to Hakone, mountainous region known for its hot springs and stunning views of Mt. Fuji. The scenic train ride through the Japanese Alps was highlight our trip.
Japan's culinary scene is as diverse as its landscapes. From the humble bento box to the exquisite kaiseki, every meal was gastronomic delight. The attention to detail and the emphasis scasonal ingredients were evident in every dish.
Our trip to Japan was more than just vacation; it was a cultural immersion. The country's blend of tradition and modernity, its respect for nature, and its culinary artistry left indelible mark
47

---

Table 12: Analyzing the constitutions yielded by the Constrained STaPLe algorithm, both with and without the Bayesian hyperparameter optimization process detailed above.
Model
J(8)
STaPLe (01 8.0) Iter (28.2k)
0.6437
STaPLe (8) = 7.6) Iter (28.2k)
0.6502
STaPLe (62 7.0) Iter (6.0k)
0.6625
STaPLe (62 7.3) Iter (6.1k)
0.6732
STaPLe (63 7.0) Iter (6.2k)
0.6889
STaPLe (6g 6.6) Iter (6.4k)
0.7054
Graanite331188---nntt
STaPLe (81 = 6.0) Iter (24.1k)
0.6036
STaPLe (8) = 6.3) Tter (24.1k)
0.6151
STaPLe (62
Iter (5.lk)
0.6241
STaPLe (62
5.9) (5.2k)
0.6482
STaPLe (6a 5.0) Iter (5.4k)
0.6765
STaPLe (6g 4.2) Iter (5.9k)
0.6894
Japan, with its myriad charms, is place that will forever hold special place in our hearts.
In less than 300 words, we've tried to encapsulate the essence of Japan country that is complex as it is captivating. From its bustling cities to its serene temples, from its technological marvels its natural wonders, Japan is land of contrasts that offers something for every traveler.
Principle: Conciseness and Clarity
Japan, land of contrasts, captivated with its blend of tradition and modernity. Our journey began in Tokyo, where neon-lit skyscrapers and bustling streets of Shibuya contrasted with the serenity of Kinkaku-ji, or the Golden Pavilion. We savored sushi at Tsukiji Fish Market, testament to Japan's 's culinary artistry.
Refined Response:
Next, Kyoto's 's ancient temples and tranquil gardens offered stark contrast to Tokyo's vibrant en- ergy. The Fushimi Inari shrine, with its thousands of vermilion tori gates, was spiritual highlight.
In Hakone, we experienced Japan's natural beauty. The scenic train ride through the Japanese Alps, coupled with the soothing hot springs, provided serene escape.
Japan's culinary scene, from humble bento boxes exquisite kaiseki, showcased the country's respect for seasonal ingredients and meticulous attention to detail.
Our trip to Japan was a cultural immersion, leaving an indelible mark. The country's 's contrasts from bustling cities serene temples, technological marvels natural wonders make it must-visit destination for every traveler.
While the initial response is a reasonably well-written blog about Japan, it exceeds the 300 word limit, thus failing to follow the instruction. The trained STaPLe model recognizes this intrinsically. invokes the "Conciseness and Clarity" principle by observing that concision is the primary lacking element of the initial response, and then produces refined response. The resulting response falls well under the 300 word limit, while paraphrasing much of the prior blog post's 's core elements, thus accomplishing the task.
48

---

Q.1.2 Principle: Specificity and Completeness
Prompt: I'm interested in a college with open enrollment and regional accreditation. Which college would you recommend? Don't include the keywords "DuPage" and "Dade" in your response. Let's make a constrained writing problem: be sure the letter p appears at least 15 times in your response.
Initial Response: I'd recommend considering the University of the People (UOP).
Principle: Specificity and Completeness
I'd recommend considering the University of the People (UOP). non- non-profit institution with open enrollment and regional accreditation by the Higher Learning Commission (HI.C). UOP offers affordable, accredited bachelor's and master's degrees in business administra- tion, computer science, and health science. Their tuition-free model and flexible schedule make an excellent choice for non-traditioall students.
Refined Response:
UOP's open enrollment policy means that anyone with high school diploma equivalent can apply, regardless of previous academic experience. This inclusivity aligns with your request for college with open enrollment.
The university's 's regional accreditation by the HLC ensures that its degrees are recognized and respected by employers, academic institutions, and professional organizations. This accreditation crucial for maintaining the value and transferability of your degree.
UOP's online, asynchronous learning format allows students to balance their education with work and personal commitments. This flexibility is particularly beneficial for adult learners and those in remote areas.
Moreover, UOP's commitment to affordability is commendable. As a non-profit, the university offers tuition-free degrees, making higher education accessible broader range of students.
In summary, the University of the People is college with open enrollment and regional accredita- tion that meets your criteria. Its flexible, affordable, and accredited programs make an excellent choice for students seeking high-quality education without the burden of high tuition fees.
The initial response produces the name of college, but does not provide particularly informative response. It also does not meet the criterion including the letter "p" at least 15 times in its response. The model intrinsically recognizes the need for greater specificity in its answer, and produces more substantial and useful revised response.
Q.1.3 Principle: Directness and Assertiveness
Prompt:
Write a cover letter for a job and end with exactly "Call me 631-481-4867" No other words should follow that phrase.
Initial Response: Dear Hiring Manager,
am writing express my interest in the [Job Title] position [Company Name], as advertised. With Bachelor's degree in [Relevant Field] and over [Number of Years] years of experience in [Relevant Skills), am confident that would be a valuable addition to your team.
In my previous role at [Previous Company Name], I was responsible for [Key Responsibility 1]. which allowed me to hone my skills in [Skill 1]. I also led team of [Number of Team Members] in [Project/nnititivee] resulting in [Achievement]. am particularly drawn to [Company Name] because of its reputation for [Company Value/Mission), which aligns with my own professional
49

---

goals.
I am excited about the opportunity to bring my unique blend of skills and experience to [Company Name] and am eager to contribute to your team's 's success. am available at your earliest convenience for interview and can be reached at 631-481-4867.
Thank you for considering my application. look forward to the opportunity discuss how can contribute to your team.
Sincerely,
[Your Name]
Call me at 631-481-4867.
Principle: Directness and Assertiveness
Refined Response: Dear Hiring Manager,
am excited to apply for the [Job Title] position at [Company Name]. With a Bachelor's s in [Relevant Field] and [Number of Years] years of experience in [Relevant Skills), am confident in my ability to excel this role.
In my previous role at [Previous Company Name], honed my skills in [Skill 1] through [Key Responsibility 1]. and led team of [Number of Team Members] to achieve [Achievement]. am drawn to [Company Name] because of its commitment to [Company Value/Mission), which resonates with my professional aspirations.
am eager to bring my unique blend of skills and experience to [Company Name] and contribute to your team's success. am available for an interview at your earliest convenience and can be reached 631-481-4867.
Thank you for considering my application. look forward to discussing how can contribute to your team.
Sincerely, [Your Name]
Call me at 631-481--480 -48477
This minor rewrite reflects a more assertive tone, as indicated in the principle invoked by the model, "Directness and Assertiveness". Specifically, when comparing phrasing such as "discussing how I can contribute ... opposed "the opportunity to discuss the former is evidently more direct and confident in pursuing this job opportunity.
Q.2
Granite-3.1-8B-Insttuee IFEval Examples
Q.2.1 Principle: "Urgency and Action Steps"
Please rewrite the following sentence in serious tone, similar to the president of the united states, and without an upward bias or inflation alarmism. Please also include the keywords "ink" and "memoirs" in the sentence: "The main points to consider are the following: First, the ECB's own forecasts, which currently have a strong upward bias, must be revised downwards. Second, the ECB must not allow itself to be influenced by the inflation alarmism that is currently being disseminated some quarters.
Prompt:
Initial Response: In the context our ongoing analysis, it is imperative to acknowledge the significance of the European Central Bank's 's projections, which have been marked by a pronounced upward trend. It crucial that these forecasts are recalibrated to reflect more balanced perspective. Furthermore, it is essential that the ECB remains steadfast in its
50

---

decision-making process, unswayed by the inflation alarmism that has been circulating in certain circles.
Principle: Urgency and Action Steps
Refined Response: In the course of our meticulous examination, it is of paramount importance to recognize the profound implications of the European Central Bank's projections, which have been characterized by marked upward trajectory. is incumbent upon us to adjust these forecasts to better reflect a more nuanced and balanced outlook. Moreover, is of critical significance that the ECB maintains its resolute stance in the face of the inflation alarmism that has been pervasive in certain sectors. This is matter of utmost urgency, we endeavor navigate the intricate complexities of our economic terrain. In this regard, have recently put ink to paper in my memoirs, which offer an in-depth exploration of our economic policies and the challenges we have encountered. These memoirs stand as a testament to our unwavering dedication to transparency and accountability, and I urge to engage with them as we forge ahead in our economic journey.
While the initial response was good, using verbiage such as "imperative", "crucial", "essential" and "steadfast", it doesn't it really make a call to action while avoiding an "alarmist" tone. It also does reflect the keywords "ink" and "memoirs", principle "urgency and action steps", which informs the call to action, along with a changed tone. Despite seeming reflect slight "upward bias", using keywords like "meticulous examination" and "maintains its resolute stance, it balances this with phrases such as "critical significance", "incumbent upon us", which retain seriousness and a balanced outlook, as desired.
desired in the prompt. The model invokes the
Q.2.2 Principle: Tone and Communication
Prompt: Can you give me a zany, bullet point TLDR of this article: //en. wikipedia. wrg/wiki/Dota._'
https:
Make it zany, but do not include the keywords 'icefrog', 'blizzard', 'lawsuit' in the response.
Initial Response:
Dota 2 a game where you play as hero and battle out with other players.
It's like superhero showdown, but with magic and spells!
You can choose from over 100 heroes, each with their own unique abilities.
The game is played
map called the "Acon of Strife" which is like giant battlefield. You and your team have to destroy the enemy's base, called the "Ancient", to win.
There also creeps, which like minions that help you fight.
You earn gold by killing creeps and enemies, which you can use to buy items to make your hero
powerful.
The game is super competitive, with professional teams and tournaments.
It's like the Olympics of video games, but with more magic and less spandex.
Dota 2 free to play, but you can buy cosmetic items to customize your hero's look.
It's like dressing up your action figure, but with more strategy and less plastic.
Dota 2 has huge community of players, and they love to stream their games on Twitch.
It's like watching live-action superhero show, but with more keyboards and less popcorn.
Dota 2 is a game that will make you feel like powerful wizard, without the risk of turning into toad.
Principle: Tone and Communication
Refined Response:
Dota 2 is a wild game where you pick hero and duke it out with other players!
It's like a superhero smackdown, but with magic and spells!
You can choose from over 100 heroes, each with their own special powers.
The game played on a map called the "Acon of Strife" which is like giant playground.
You and your team gotta destroy the enemy's base, called the "Ancient", to win.
51

---

There are also creeps, which are like tiny sidekicks that help you fight.
You earn gold by defeating creeps and enemies, which you can use to buy cool stuff to make
your hero even more awesome.
The game is super competitive, with professional teans and tournaments.
It's like the Super Bowl of video games, but with more magic and less helmets.
Dota 2 is free to play, but you can buy fancy stuff to make your hero look super cool.
It's like dressing up your action figure, but with more strategy and less plastic.
Dota 2 has a huge community of players, and they love to show off their skills on Twitch.
It's like watching a live-action superhero show, but with more keyboards and less popcorn. Dota 2 is a game that will make you feel like a powerful wizard, without the risk of turning into toad.
So grab your magic staff and get ready to battle it out in the Aeon of Strife!
The model associates a "zany" response with a more child-like tone, and to reflect the "Tone and Communication" principle, changes certain phrasing. For example "battle it out" - "... duke it out ..."; "have to destroy" >> "gotta destroy"; "which are like minions" - "which are like tiny sidekicks", etc. It also adds this final sentence "So grab your magic staff and get ready battle it out in the Acon of Strife!", which can encourage a reader enthused by the "zany" tone to adopt the video game.
R
Ethics Statement
Our findings suggest that most users can use STaPLe to improve the quality of the model's responses by eliciting and training the model to follow desirable latent attributes. As such, we hope that this induces positive societal impact by way of producing a set of model-preferred labels which are used effectively to perform self-correction in an expressive, and thus interpretable manner. However, we caveat this by noting that a principle label alone does not fully model the latent reasoning process that a human may use self-correction, but rather, only serves as a stimulus to indicate the most relevant direction that a refined response should "step" towards for improvement.
An adversarial user could potentially use this process as a means deliberately misalign the model by using the principle discovery phase as means to steer the model further away from desirable responses. That is, one could select another objective aside from the gold response to use as a self-correction target; this would likely yield drastically different principles and results. Training on such trajectories would induce self-degradation behavior at inference-time collapsing the quality of the model's responses, rather than the desired self-improve inff-mpproeeent of its self-
abilities. We observe that this a potential risk for all such principle-driven alignment strategies, even with human-curated or strong model-generatee principles, but is especially the case with self-generated principles, given the generator a relatively weaker language model.
As mitigation strategy for this potential negative impact, continuing from our discussion in Section we suggest human oversight by way of human-in-thee-lop feedback. Specifically, an external set of reviewers can assess the quality and safety of the principles generated at the end of the E-step of each iteration after clustering before training the model to follow One could feasibly provide multiple candidate constitutions e.g. one constitution per label replacement strategy described in Appendix P.I. or under different clustering thresholds (the impact of which is explored in Appendix P.3) and the annotators can select the best one and make edits to it as appropriate. For instance, if an annotator were to discard an element, one could simply discard all samples with labels that fall under that cluster. Thus, we acknowledge the role that clustering plays in making informed assessments over the constitution; as such, constrained STaPLe is more controllable in comparison to the unconstrained version. While this reintroduces human oversight to balance performance with safety, it would add minimal human labor overhead, as judging a constitution for safety would require substantially fewer annotation hours than curating one, presenting an advantage over methods such as Constitutional AI. We believe that this strategy would be effective in enforcing responsible usage of STaPLe.
The above human-in in-the-loop proposal is also an effective strategy to mitigate bias amplification over the iterations. Allowing annotators to discard elements that they assess would propagate biases or stereotypes would ensure that these behaviors are not learned by the model and then invoked in subsequent iterations, avoiding the cascading effect. Again, clustering and the label replacement scheme plays important role here by ensuring that we do not train principles that are hyper-
52

---

specific to particular sample. This is especially relevant when there may be noisy or adversarial prompts designed to induce undesirable behavior. We suggest that users inspect the model-generated constitutions to assess their principles and the alignment of these labels with their values before training over these elements the M-step.
Even when using STaPLe to improve responses towards the gold, it is possible that this reference answer is noisy i.c. incorrect (verifiable settings) still undesirable in some aspect (preference settings). Given the algorithm's 's generality, dataset selection is left to the user we encourage users to analyze the gold responses to filter samples with lower quality gold responses accordingly during pre- vre-prcesssing This could be done by way of human annotation (using Likert scale annotations on multiple attributes, akin UltraFeedback), or using trained model-based filters for undesirable qualities such as profane language.
We believe that the promise of STaPLe in facilitating self-improvement in language models by alignment to model-generated constitutions outweighs the possible negative impacts. We further suggest that the strategies detailed above specifically, the introduction of some human oversight into the STaPLe algorithm - would largely mitigate these risks and promote responsible usage.
53

---

S
Details of Models and Datasets Used
As noted in Section 4.1, we use the following large language models in our experiments:
Llama-3.1-88 1-8t-Instruct (Grattafiori et al., 2024); this model is available under the custom 1lama-3.1 Community License* which includes provisions for commercial usage.
Granite-3.J B.aitee-..-BB-nnn 188-Insttutct (Granite Team, 2024); this model available under the permissive,
Apache 2.0 open-source license.
Qwen2.5-777
- Instruct (Qwen, 2025); this model is also available under Apache 2.0.
similarity scoring function between candidate response generated on-policy by one of the above models to the gold response. We instantiate this judge with the Phi-4 language model (Abdin al., 2024), which is made available under the permissive MIT license. In Appendix N, we examine the efficacy of our method on the Qwen3-32B reasoning model (Yang et al., 2025), which is available under the Apache 2.0 license.
Furthermore, in Appendix P.2, explore the use of LLM-as-a-judde
We also provide further details of the datasets used in the mining corpus, expanding description in Section 4.1:
Anthropic HH-RLHF: this dataset consists of total of 161k preference pairs (chosen- rejected) over helpfulness and harmlessness as described in Bai et al. (2022a). HH-RLHF is available under the MIT license.
UltraFeedback (Cui et al., 2024): this dataset consists of 64k prompts; for each prompt, responses are sampled from four different language models. For each response, Likert-scale annotations obtained over four attributes - helpfulness, honesty, mnstruction-followiin and truthfulness with corresponding rationales. For the STaPLe algorithm, we only least 3, forming list of gold responses. We
consider samples where all Likert scores then score against the gold in the by taking the average over the multiple reference answers. UltraFeedback has been made available under the MIT license.
TL:DR (Stiennon et al., 2020): this dataset consists of Reddit posts detailing a situation, along with two candidate summaries, in the "comparisons" part, which we use. They include "choice" label, which
select gold response (summary). We use the train set, consisting of 92.9k samples. TL;DR is available under the CC-BY-4.0 license.
HotpotQA: this dataset focuses on Wikipedia-based question answering. We use the train set of the "fullwiki" split, consisting of 90.4k samples; these contain a question, context, supporting facts, and gold response. HotpoiQA is available under CC-BY-SA-4.0.
Lastly, discuss the details behind the evaluation datasets and evaluation framework.
MT-Bench consists of 80 prompts, testing multi-turn, open-ended response generation capabilities for chat assistants. is available under the Apache 2.0 license, in the FastChat GitHub repository. We use GPT-4o (OpenAI, 2024) as the judge model.
A.p---C (Li et al., 2023) consists of 805 samples testing instruction - abilities, using length-controlled win-rates through generalized linear modeling approach (Dubois et al., 2024). It is released under the Apache 2.0 license.
IFEval (Zhou et al., 2023) consists of 541 prompts, similarly testing instruction-follloiigg abilities. It is released under the Apache 2.0 license.
We used the Prometheus-8x7 Sateeeuussx7--22.... language model (Kim al., 2024) fine-grained judge to compare the quality of the STaPLe models' generations their principle-following ability. This model is available under the Apache 2.0 license.
"mtpss/huuggingfacccc/meet co///////..a.ccccataaaaaaaaa Hamaa/Lama-3311220 3.1-70B L
54


Reviews:
[{"review_id": "uVe3oJjAej", "reviewer": "Reviewer_u4GT", "summary": "The paper proposes a framework for self-refinement of large language models.\n\nThey use the model itself prompted with a gold response to generate principles (in natural language) and critique its own initial response to a input against that generated principle, and then generate a refined response based on the critique. \n\nThese refined responses are used with supervised fine-tuning to self-improve the model to align to the self-generated principles of a good response derived from the model by looking at the gold response. \n\nThe paper poses this solution as an EM setup given the iterative self-improvement where the prompting approximates the estimation.\n\nThey also empirically arrive at clustering the generated principles (natural language) embeddings to get a reduced set for interpretability.", "questions": "None", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Weakness\n1. The paper is posing things as EM/posterior estimation but approximating or choosing empirical setup, the connection is not very relevant. The EM is sparsely related because its repeatedly fine-tune (maximizing step to generate critiques and responses for fine-tuning, and then the learning step, to SFT optimize with it). Figure 2.\n\n2. Evaluation: The baseline is considered with fine-tuning on 50k gold responses but the proposed method uses the 50k with 4 additional iterations each in which 10k gold responses are used to generate critiques and responses to fine-tune with. A fair baseline would be to use all the 50k + 40k input-gold response pairs in SFT which is not considered in the paper.\n\n3. Setup does not seem well motivated: The setup is not very useful - Why would we use Gold supervision to explicitly generate critiques vs just using SFT with them and implicitly learning it? Other than interpretability of the clustered generated set of principles, there is no reason it is not already implicit in the learning. Also, using gold annotations is more expensive - RLHF with preference annotation would be cheaper to collect. It would align the model without explicit principles. The explicit principles do not seem very useful. The paper also does not compare with RLHF.", "was_revised": true, "final_justification": "The rebuttal fails to address flaws in Setup and Evaluation:\n\n\n1. \"which relies on learning explicit verbalization of principles and refinement, greatly improves\" - There is no clear justification with technical analysis to support the results. \n\n``` The results of SFT training on gold-only 50k samples reported in Table 1 in the paper, and on 90k samples in the table above, are clear indicators that our STaPLe method, which relies on learning explicit verbalization of principles and refinement, greatly improves performance over simple SFT on gold responses.```\n\n2. The setup is mentions that it can be done with RL. The paper is purely hypothetical, and does not compare against RLHF, and the setup of regular RLHF is more cost-effective - and this work offers poorly validated.\n\n```STaPLe is equivalent to the REINFORCE gradient for self-play under this definition. However, we view our work as a cold-start phase for such a training and we leave purely online RL training as future work. We also hypothesize that without a cold-start phase, where principles are elicited using hints and are then inserted into an un-hinted trajectory for training, the model would have taken too long (or forever) to arrive at the point of generating good/relevant principles.```\n\n```We do not compare against existing RLHF works on constitution learning due to clear differences in setup and goals. A key difference is that we do not prescribe any principles a priori. The primary efficiency from a labor standpoint is not just in annotation, but also in curating a constitution```\n\nI strongly do not support its acceptance, and maintain my score and initial arguments.", "rebuttal": "We would like to thank Reviewer u4GT for taking the time to review our work and for their valuable feedback. We address the concerns raised in the review through our responses below:\n\n\\\n**1. EM Approach in STaPLe Algorithm.**\n> The paper is posing things as EM/posterior estimation but approximating or choosing empirical setup, the connection is not very relevant. The EM is sparsely related because its repeatedly fine-tune (maximizing step to generate critiques and responses for fine-tuning, and then the learning step, to SFT optimize with it). Figure 2.\n\nWe acknowledge that our implementation approximates the EM algorithm, since exact computation of posterior marginals over latent variables is intractable in our case; However, our setup is close to what is referred to as Monte Carlo EM [1][2] – where these marginals are replaced with sample based expectations i.e. a set of latent variables, sampled from posterior, is used to optimize the gradients in the M-Step; In particular, we sample 16 latent [principles, response] pairs via rejection sampling, given current model parameters and gold responses and then maximize the likelihood over these samples. \n\n[1] William Ruth, A review of Monte Carlo-based versions of the EM algorithm, 2024 https://arxiv.org/abs/2401.00945\n\n[2] Greg C. G. Wei and Martin A. Tanner. A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. Journal of the American Statistical Association, 85(411), 1990.\n\n\\\n**2, SFT Baseline.**\n> The baseline is considered with fine-tuning on 50k gold responses but the proposed method uses the 50k with 4 additional iterations each in which 10k gold responses are used to generate critiques and responses to fine-tune with. A fair baseline would be to use all the 50k + 40k input-gold response pairs in SFT which is not considered in the paper.\n\nThank you for this suggestion! In addition to the baseline on 50k samples we report in Table 1 (“Gold-only SFT”), here we train all 3 models on 90k samples as proposed, and compare the result against the Iteration 4 scores for STaR and STaPLe {unconstrained, constrained} in the table below:\n\n| Model                              | MT-Bench (Avg.) | AlpacaEval |\n| ---------------------------------- | --------------- | ---------- |\n| Llama 8B SFT (90k)                 | 7.50            | 28.2       |\n| Llama 8B STaR Iter 4               | 7.56            | 31.8       |\n| Llama 8B STaPLe Iter 4             | 7.71            | 33.4       |\n| Llama 8B Constrained StaPLe Iter 4 | 7.70            | 34.9       |\n\n| Model                                | MT-Bench (Avg.) | AlpacaEval |\n| ------------------------------------ | --------------- | ---------- |\n| Granite 8B SFT (90k)                 | 7.94            | 30.4       |\n| Granite 8B STaR Iter 4               | 7.96            | 35.6       |\n| Granite 8B STaPLe Iter 4             | 8.04            | 38.4       |\n| Granite 8B Constrained StaPLe Iter 4 | 8.03            | 38.8       |\n\n| Model                             | MT-Bench (Avg.) | AlpacaEval |\n| --------------------------------- | --------------- | ---------- |\n| Qwen 7B SFT (90k)                 | 7.00            | 32.3       |\n| Qwen 7B STaR Iter 4               | 7.14            | 37.8       |\n| Qwen 7B STaPLe Iter 4             | 7.24            | 40.2       |\n| Qwen 7B Constrained StaPLe Iter 4 | 7.22            | 39.9       |\n\nWhile SFT on 90k samples does improve slightly over the SFT on 50k samples (the results included in Table 1), it is still behind the STaR iteration 4 results, and as such, lags further behind STaPLe in both the unconstrained and constrained settings. \n\n\\\n**3. Learning Principles from Human Data.**\n> Setup does not seem well motivated: The setup is not very useful - Why would we use Gold supervision to explicitly generate critiques vs just using SFT with them and implicitly learning it? Other than interpretability of the clustered generated set of principles, there is no reason it is not already implicit in the learning. Also, using gold annotations is more expensive - RLHF with preference annotation would be cheaper to collect. It would align the model without explicit principles. The explicit principles do not seem very useful. The paper also does not compare with RLHF.\n\nThe results of SFT training on gold-only 50k samples reported in Table 1 in the paper, and on 90k samples in the table above, are clear indicators that our STaPLe method, which relies on learning explicit verbalization of principles and refinement, greatly improves performance over simple SFT on gold responses.\n\nAs for the use of gold annotations, we do not need any new annotations for training, we use [query, response] pairs from existing open source datasets, of which there is an abundance; in STaPLe we provide a way to make the most out of those existing datasets (beyond simple SFT) without any additional annotations, with the added benefit of automatically curating a data-dependent constitution.\n\nWhile we train our model via an EM-based SFT framework, we agree that RL methods like PPO or GRPO could be used with the similarity score/judgment as a reward signal. In fact, in Appendix F, we show that our method is equivalent to self-play reinforcement learning by defining the advantage in terms of the difference in score against the gold response; the MC-EM gradient obtained by STaPLe is equivalent to the REINFORCE gradient for self-play under this definition. However, we view our work as a cold-start phase for such a training and we leave purely online RL training as future work. We also hypothesize that without a cold-start phase, where principles are elicited using hints and are then inserted into an un-hinted trajectory for training, the model would have taken too long (or forever) to arrive at the point of generating good/relevant principles. \n\nWe do not compare against existing RLHF works on constitution learning due to clear differences in setup and goals. A key difference is that we do not prescribe any principles a priori. The primary efficiency from a labor standpoint is not just in annotation, but also in curating a constitution. Our method serves as a means to identify qualities a model should reflect in its generations; the constitutions we report in the appendix should be viewed as such a list, based on the domains and tasks included in the data mixture. \n\nTraining models to follow known principles — desirable attributes for a model’s responses to reflect — is a core goal of alignment research [3, 4] at present. Our goal is to both discover such principles from the model, and train the model to follow them — implicitly learning them from gold annotations or preference data does not accomplish this task as we do not have annotations indicating *why* the gold/preferred response is desirable [5]. This motivates us to elicit principles from the data to enable human oversight of this alignment process. \n\n[3] Bai et al., Constitutional AI: Harmlessness from AI Feedback, 2022, https://arxiv.org/abs/2212.08073\n\n[4] Guan et al., Deliberative Alignment: Reasoning Enables Safer Language Models, 2025, https://arxiv.org/abs/2412.16339\n\n[5] Findeis et al., Inverse Constitutional AI: Compressing Preferences into Principles, 2025, The Thirteenth International Conference on Learning Representations. \n\n---\nWe hope these responses have addressed your concerns — if so, we would like to respectfully ask you to reconsider your assessment. We would also be happy to address any further concerns you may have. Thank you again for your time!"}, {"review_id": "m1I2MjEoG8", "reviewer": "Reviewer_pKck", "summary": "These principles are treated as latent variables in an Expectation-Maximization-like process. At each iteration, the model generates better answers by conditioning on the gold answer (hinting), extracts candidate principles, filters them using similarity metrics (like Rouge-L or LLM judges), clusters them into a distilled constitution, and fine-tunes on the best ones. Over four iterations, the method shows consistent performance improvements on multiple benchmarks (MT-Bench, AlpacaEval, IFEval) for several open-source 7–8B models, outperforming simple self-refinement and STaR-style baselines.", "questions": "Please refer to the weakness for questions", "limitations": "Please address the questions in the weakness section", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe work shows that small open models can bootstrap themselves without external teachers. Modelling principles as latent variables distinguishes the method from prior self-refinement approaches, and the principled EM derivation plus a clustering-based regulariser add technical depth . The experimental section is relatively thorough with ablations on clustering and scoring functions.\n\nWeakness:\n1.\tThe model gets a hint by seeing the correct answer while generating principles in the propised method.  What if this risks making the model dependent on that answer that it might just learn to copy parts of it rather than actually reason towards it.  Also, the authors say that they filter out poor candidates, but they don’t hsow how often the golden answer leaks into the principles of how much this affect the performance. \n2.\tThe paper frames the method as EM-learning but the scoring presented in the paper doesn’t seem trainable.  Can you clarify on this?\n3. the clustering step is happening after principles are generated and filtered which means, its not part of the training process if I understand it correct.  Also the clustering threshold is tuned using another optimizer?  Can you explain?\n4.\tThe paper compares against basic self-refinement and instruction tuned models, but no results on newer methods like RISE.\n5.\tDo you have experiments on how the model generated principles could introduce biases and errors?", "was_revised": true, "final_justification": "I went through the author's response, and I am satisfied with their response to my questions.  I strongly advise the authors to include them in their updated paper. I maintain my score of \"accept\" for this submission.", "rebuttal": "We would like to thank Reviewer pKck for taking the time to review our work and for their valuable feedback. We appreciate that the reviewer finds the mathematical framework underlying our approach to “add technical depth”, and that the experimental section is “relatively thorough”. We address the concerns stated in the review with our responses below:\n\n\\\n**1. Copying in Self-Refinement.**\n>  The model gets a hint by seeing the correct answer while generating principles in the propised method. What if this risks making the model dependent on that answer that it might just learn to copy parts of it rather than actually reason towards it. Also, the authors say that they filter out poor candidates, but they don’t hsow how often the golden answer leaks into the principles of how much this affect the performance.\n\nWe would like to clarify that the model only sees the gold response when proposing a principle, not during self-refinement. As noted in the footnote on page 4, we find that the principle generated never looks like the gold response itself, in part because the prompt to propose a principle asks the model to suggest an attribute of the gold response that would make the initial response more similar to this gold target. \n\nIn practice, smaller / weaker language models are not very successful at self-refinement without targeted dimensions of refinement [1,2], necessitating our filtering of all refinements that did not improve over the initial response. Note that filtering is performed on the refined responses induced by the principle and its critique, not directly on the principles themselves. \nBelow, to confirm that $y^2$ does not copy $y^G$, we compute the Rouge-L precision, and report the number of samples that have a score above 0.9 — we find this number to be very small (less than 1.5% of the data), which we find to be negligible. We would like to point out that high overlap with the gold data is not an undesirable property -- after all, the gold responses are from license-friendly train sets that one would ordinarily perform SFT with -- however, we find on-policy generations to be effective.    \n\n| Model  | Total Samples | \\# Samples with Rouge-L Prec > 0.9 | Fraction |\n| :--------------------------: | :-------------: | :----------------------------------: | :--------: |\n| Llama-3.1 8B STaPLe Iter 1 | 28.2k         | 91  | 0.32%    |\n| Llama-3.1 8B STaPLe Iter 2 | 6.1k          | 35  | 0.57%    |\n| Llama-3.1 8B STaPLe Iter 3 | 6.3k          | 31   |  0.49%    |\n| Llama-3.1 8B StaPLe Iter 4 | 6.6k          | 41  | 0.62%    |\n\n| Model  | Total Samples | \\# Samples with Rouge-L Prec > 0.9 | Fraction |\n| :--------------: | :-------------: | :-----------: | :--------: |\n| Granite-3.1 8B STaPLe Iter 1 | 24.1k         | 157  | 0.65%    |\n| Granite-3.1 8B STaPLe Iter 2 | 5.2k          | 32    | 0.61%    |\n| Granite-3.1 8B STaPLe Iter 3 | 5.9k          | 56  | 0.94%    |\n| Granite-3.1 8B StaPLe Iter 4 | 6.3k          | 87 | 1.38%    |\n\n| Model  | Total Samples | \\# Samples with Rouge-L Prec > 0.9 | Fraction |\n| :---------------- | :-------------: | :------: | :--------: |\n| Qwen2.5 7B STaPLe Iter 1 | 30.9k  | 176  | 0.57%    |\n| Qwen2.5 7B STaPLe Iter 2 | 6.5k   | 53    | 0.82%    |\n| Qwen2.5 7B STaPLe Iter 3 | 7.0k | 88   | 1.25%    |\n| Qwen2.5 7B StaPLe Iter 4 | 7.1k  | 82   | 1.16%    |\n\nSince all samples in “total samples” are those where the refined generation improved over the initial response in Rouge-L F1, and the fraction of samples where the precision increased to above 0.9 did not increase substantially, it must be that the recall improved. That is, while the rate of copying does not grow much, the generated responses better cover the gold -- this is necessary for better reflecting the core elements of the gold response for the model to imitate with its on-policy generations. In works such as STaR [3], a chain-of-thought is induced that reconstructs the original gold answer — our work achieves a similar purpose in learning a distribution of on-policy generations similar to the gold, rather than overfitting to the gold. \n\n[1] Madaan et al., Self-Refine: Iterative Refinement with Self-Feedback, 2023, Thirty-seventh Conference on Neural Information Processing Systems\n\n[2] Ramji et al., Self-Refinement of Language Models from External Proxy Metrics Feedback, 2024, https://arxiv.org/abs/2403.00827\n\n[3] Zelikman et al., STaR: Bootstrapping Reasoning with Reasoning, 2022, Advances in Neural Information Processing Systems\n\n\\\n**2. EM Scoring Clarification.**\n> The paper frames the method as EM-learning but the scoring presented in the paper doesn’t seem trainable. Can you clarify on this?\n\nThat is partially true – the distribution over latent variables (the principle and the model response given the principle) is trainable (and we train over this), but the distribution over observed gold given the generated response is fixed and is defined via a similarity scoring function. However, this does not interfere with the EM framing of the method. The E-Step uses both distributions to generate samples from the posterior via rejection sampling, thus giving an approximate expectation over latent trajectories, while the M-Step updates just the trainable distribution to maximize the likelihood of these trajectories.\n\n\n\\\n**3. Clustering in the EM Framework.**\n> the clustering step is happening after principles are generated and filtered which means, its not part of the training process if I understand it correct. Also the clustering threshold is tuned using another optimizer? Can you explain?\n\nClustering is performed after the E-step (data generation phase) but before the M-step (model training phase), as a data augmentation procedure. A hierarchical clustering model is trained on the embeddings of the principles generated in the E-step, and the cluster representatives are chosen according to the schemes outlined in Appendix M.1 (primarily, using the cluster medoid). The self-correction trajectories generated in the E-step are augmented by replacing the principle with the cluster representative of the cluster it belongs to. The resulting trajectories (using the clustered labels) are then used for SFT training in the M-step.\n\nThe results in the main paper use thresholds that we (the authors) set. However, to automate this process, we also describe and implement a Bayesian optimization procedure in Appendix M.3, which optimizes for balancing diversity across the clusters with tightness within the cluster. This method searches over threshold values, fits an Agglomerative Clustering model based on the chosen threshold for each point, and calculates this diversity-tightness objective function based on the resulting clustering, repeating iteratively by choosing next threshold value using Gaussian Process regression. In fact, we find this to perform slightly better on ablations performed on the Llama-8B and Granite-8B models. \n\n\n\\\n**4. Baselines.** \n> The paper compares against basic self-refinement and instruction tuned models, but no results on newer methods like RISE.\n\nThe baselines chosen are prominent methods in self-refinement and multi-iteration self-improvement, specifically those with code that is made available open-source (e.g. Self-Refine, STaR). Notably, while some recent methods like RISE and SCoRe share a broad focus on self-improvement and self-correction, neither has made their code available for reproduction, and both methods are non-trivial to implement. We consider a similar set of baselines as RISE; given our work’s premise of on-policy chain-of-thought generation given the gold response, akin to STaR, we include a STaR-like baseline. \n\n\n\\\n**5. Errors and Refinement Quality.**\n> Do you have experiments on how the model generated principles could introduce biases and errors?\n\nWhile it was rare for intrinsic principle-based refinement to hurt response quality over the initial response (“errors”), we did find that the initial response generated by the model during self-correction became slightly worse in comparison to responses sampled from the base policy, on the basis of our Prometheus evaluation on IFEval (no degradation would be around 50%, subject to slight variance). We report this behavior in the table below:\n\n| Model  | IFEval WR (initial vs base) | \n| :---------: | :-----------: | \n| Llama-3.1 8B Constrained STaPLe Iter 1 | 44.6% |\n| Llama-3.1 8B Constrained STaPLe Iter 2 | 47.7% |\n| Llama-3.1 8B Constrained STaPLe Iter 3 | 45.3%  | \n| Llama-3.1 8B Constrained StaPLe Iter 4 | 42.6%   | \n\n| Model  | IFEval WR (initial vs base) |\n| :-------------: | :---------: | \n| Granite-3.1 8B Constrained STaPLe Iter 1 | 48.2%    | \n| Granite-3.1 8B Constrained STaPLe Iter 2 | 45.9%   | \n| Granite-3.1 8B Constrained STaPLe Iter 3 | 46.8%    |  |  |\n| Granite-3.1 8B Constrained StaPLe Iter 4 | 45.5%     |  |  |\n\nAs such, comparing the refined response against generations from the base policy — rather than the initial response generated from the model during inference — was important to confirm that the quality of the refined responses generated over the iterations does indeed improve, as reflected in Tables 2 and 3. Given the refined response is what a user would follow and trust (as in recent reasoning works following a token like “Wait” [4], where our principles serve as a similar delimiter), we focus on evaluating the refinement in this work. We also compare the refined responses across iterations (“stepwise”) in Table 4, which further confirms that the refined responses do become better, before saturating, as evidenced in Figure 1 and Table 1. \n\n[4] Muennighoff et al., s1: Simple test-time scaling, 2025, https://arxiv.org/abs/2501.19393\n\n-----\n\nWe hope these responses have addressed your concerns — we would be happy to address any further concerns you may have. Thank you again for your time!"}, {"review_id": "GAoq01q41j", "reviewer": "Reviewer_Hwww", "summary": "This paper proposes a fully automated \"principle-driven\" self-improvement framework - Self-Taught Principle Learning (STaPLe): let the small language model (7-8 B) discover the \"latent principles\" that can improve the quality of answers, and then learn how to call these principles during reasoning to criticize and revise its initial answers, thereby achieving continuous self-improvement.\n\nDuring each iteration, the model (i) drafts an answer $𝑦_1$, (ii) samples candidate principles 𝑧 given the gold answer $𝑦^G$, (iii) produces revised answers $𝑦^2$  that pass a rejection filter, and (iv) fine-tunes itself to invoke the selected 𝑧 before answering. A hierarchical clustering step keeps only one medoid principle per cluster, yielding a compact, human-readable “constitution”.\n\nFour iterations on mixed alignment data improve MT-Bench and AlpacaEval scores by 15-23 pp over the base model and surpass STaR / Self-Refine baselines.", "questions": "Fundamental Questions:\n\n1. The paper is entirely based on fine-tuning (SFT), and no experiments were conducted on the RAG of \"retrieval of principles and splicing them during reasoning\". This reminds me that this paper is similar to **MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems**, both of which let the model \"propose hypothesis → self-revise\". STaPLe builds a single model + Monte-Carlo EM, and learns to \"call principles\" and internalize them through SFT. MetaMind builds a multi-agent loop with explicit metacognitive control. Therefore, STaPLe can be seen as a \"single-model version of principle bootstrapping\", providing another perspective of non-multi-agent cooperation. I am curious about the comparison of the effects of explicit enhancement and fine-tuning internalization under the same setting. It will be beneficial to compare with this work to discover RAG vs. SFT. You can also try retrieving $top-𝑘$ principles at inference instead of SFT. Even a small comparison would clarify when full fine-tuning is necessary.\n\n2. When comparing the method proposed in this paper with the baseline method, does it consider the difference in iterative convergence rounds of different methods?\n\n3. The author emphasizes that the algorithm is also applicable to \"unverifiable\" tasks, because the similarity function can be replaced by Rouge-L or LLM-as-judge scores, and principle learning still uses \"making the answer closer to the reference\" as a signal. Is this strongly related to the reward rule setting of reinforcement learning? Further exploration is recommended.\n\nResearch Questions:\n\n1. Could you demonstrate STaPLe on tasks without explicit references, using LLM-judged pairwise wins as $𝑓(⋅)$? A convincing result would strengthen Significance.\n\n2. A quantitative breakdown (factuality, style, safety, etc.) of the final medoids would improve interpretability claims.\n\n3. If time allows, test the clustered constitution on a larger 34 B model; showing gains would boost my Significance rating.", "limitations": "Principle sampling uses $𝑦^𝐺$; scalability to settings without references (e.g. open-ended creative tasks) is only sketched.\n\nThe author admits that the principle is only effective for improving a subset of samples, and if the gold standard noise or malicious targets will lead to self-degradation, human review is required. Author also did not analyse potential bias amplification when principles encode cultural preferences.\n\nNo theoretical analysis is given for \"data sufficiency\", and it mainly relies on experience curves, so generalization still needs to be verified by more tasks.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\nConceptual novelty – views principles as latent variables and optimises them via MC-EM, unifying discovery ↔ execution.\n\nFull automation – removes manual constitution engineering, lowering domain-transfer cost.\n\nInterpretability – hierarchical clustering keeps a 50 % subset of principles while matching performance, yielding a concise rule set.\n\nConsistent gains – +19-23 pp on AlpacaEval-2.0 and +0.11-0.18 on MT-Bench across three distinct 7-8 B backbones.\n\nWeaknesses\n\nI can understand the quantitative definition of the potential principle $z$, but I doubt whether this principle can be obtained only through prompts. In addition, I doubt that the principle of screening by the similarity of $y^i$ and $y^G$ is not necessarily accurate, after all, this paper starts from the reasoning of \"principles\". Through similarity screening, I doubt that the principles learned are some \"plagiarism\" or \"specific and non-generalizable\" \"detailed guidance\" rather than principles. There should be a more accurate evaluation model, etc., for a deeper analysis.\n\nThe paper is only verified on Llama-3, Granite, and Qwen at the 7–8B level, and has not tested the migration of these principles to larger models. Therefore, there is uncertainty in scale extrapolation, and rediscovery/clustering may be required to match stronger representations.\n\nThe author gives a visualization of the \"constitution\" generated after the iteration of the three models in the appendix, and uses the convergence of cluster size (~50% compression) to show that the principle set tends to be stable, but only qualitative examples are given for \"which principles are most effective\", lacking systematic statistics and further analysis.", "was_revised": true, "final_justification": "I think the paper is thorough in its content, and the argument for the method itself is solid. However, the specific method motivation and contribution to its extension are insufficient—that is, the reasons for fine-tuning and the generalizability of the principles. But the authors' clarifications are acceptable. In final, I believe this work has to compare with principle-based inference-time optimization method, otherwise it's incomplete. But as long as it does, this work indeed has contribution to the LLM self-improvement field. Based on this, although this work is slightly incomplete by now, I will approve the direction of this work and leave the final justification to AC and other reviewers based on the present condition.", "rebuttal": "We would like to thank Reviewer Hwww for taking the time to review our work and for their valuable feedback. We appreciate that the reviewer agrees with the conceptual novelty of our work, and values our method’s interpretability, full automation, and that it achieves consistent gains across models. We address the concerns and questions posed in the review through our responses below:\n\n\\\n**1. Filtering in self-refinement.**\n\nAs the model sees the gold response only when it proposes a principle, and not during self-correction, the suspicion that the refined response $y^2$ copies the gold could only hold if the principle can transfer the full gold sequence. However, we instead find that the principle is fairly short and general in nature, and instead operates as a summary of the aspects contained in the gold response that are lacking in the model’s original response. This is in part due to the prompt for principle proposal asking the model to suggest an attribute of the gold response that would help guide the initial response to be more similar to the gold; this avoids the suggested copying effect. Furthermore, the fact that model performance improves on multiple unseen test datasets – where no hints are available at test time and hence copying via principles is not even possible – indicates that the model is actually learning to invoke principles for better refinement.\n\nAs for generalization, our clustering method is precisely towards this goal — rather than training the model to follow specific principles which may not be relevant in future iterations, clustering ensures that only the high-level attributes that the model should follow are trained upon, while simultaneously aiding interpretability for human overseers of the alignment process. \n\nTo affirm that our method produces refined responses that do not copy the gold, we compute the number of samples where the Rouge-L precision score exceeds 0.9, for each of the main experiments. This is an effective measure of the fraction of $y^2$ that appears verbatim in $y^G$:\n\n| Model | Total Samples | \\# Samples with Rouge-L Prec > 0.9 | Fraction |\n| :----: | :-----: | :----: | :---: |\n| Llama-3.1 8B STaPLe Iter 1 | 28.2k  | 91 | 0.32%  |\n| Llama-3.1 8B STaPLe Iter 2 | 6.1k  | 35  | 0.57% |\n| Llama-3.1 8B STaPLe Iter 3 | 6.3k   | 31  | 0.49% |\n| Llama-3.1 8B StaPLe Iter 4 | 6.6k   | 41  | 0.62% |\n\n| Model  | Total Samples | \\# Samples with Rouge-L Prec > 0.9 | Fraction |\n| :----: | :----: | :----: | :----: |\n| Granite-3.1 8B STaPLe Iter 1 | 24.1k | 157 | 0.65% |\n| Granite-3.1 8B STaPLe Iter 2 | 5.2k  | 32 | 0.61% |\n| Granite-3.1 8B STaPLe Iter 3 | 5.9k | 56 | 0.94% |\n| Granite-3.1 8B StaPLe Iter 4 | 6.3k | 87   | 1.38% |\n\n| Model | Total Samples | \\# Samples with Rouge-L Prec > 0.9 | Fraction |\n| :----: | :-----: | :----: | :----: |\n| Qwen2.5 7B STaPLe Iter 1 | 30.9k | 176 | 0.57% |\n| Qwen2.5 7B STaPLe Iter 2 | 6.5k | 53  | 0.82%  |\n| Qwen2.5 7B STaPLe Iter 3 | 7.0k | 88  | 1.25%  |\n| Qwen2.5 7B StaPLe Iter 4 | 7.1k  | 82  | 1.16% |\n\nWe find the fraction of samples with a high precision to be less than 1.5% across all experiments, which is negligibly small. \n\nTo extend beyond Rouge as a measure of similarity, we also include an ablation in Appendix M.2 where we use a Phi-4 model as a judge to compare against the gold; we filter out all samples that do not improve in the judge’s score, and find this to perform comparably to our Rouge method. There are two reasons why the availability of a reference response is important in our setup:\n\n1. Recently, the use of gold responses to compute verifiable rewards has become very successful in reasoning domains – our use of the existing gold response, along with a judge/score, should be seen as an extension of the notion of verifiable reward to non-reasoning domains. Furthermore, an LLM judge that has access to a gold response should intuitively be more accurate than a reference-free judge.\n2. We also use the gold response as a hint to elicit a relevant principle -- given we have access to such a response, we do not want to unnecessarily limit the strength of the scoring method by not using it during judgment. \n\n\\\n**2. Scalability.**\n\nWhile we do not suggest that the principles are necessarily transferable across models — our proposed method’s success at automatically discovering principles extends across various models, and different model families altogether. \n\nNonetheless, based on your feedback, we run our STaPLe method for 3 iterations with the Qwen3 32B model, and report the results below, alongside a STaR baseline for this model. Note that we keep the thinking mode on for response generation, which results in an even stronger initial policy; nonetheless, self-refinement is performed on just the response, not the thinking tokens (the \\<think\\>...\\</think\\> block):\n\n| Model | AlpacaEval Win-rate |\n| :-----: | :-----: |\n| Qwen-3 32B Initial Policy | 60.46 |\n| Qwen-3 32B STaR Iter 1 | 62.98  |\n| Qwen-3 32B STaPLe Iter 1| 65.49  |\n| Qwen-3 32B STaR Iter 2| 64.73  |\n| Qwen-3 32B STaPLe Iter 2| 67.18  |\n| Qwen-3 32B STaR Iter 3 | 66.33   |\n| Qwen-3 32B STaPLe Iter 3 | 68.82 |\n\nThese results are encouraging, suggesting that even a 32B parameter reasoning model can self-improve its alignment by a substantial amount (+8.36% on AlpacaEval). STaPLe also still outperforms the STaR baseline at each iteration, If desired, we would be glad to run further experiments with models larger than the 7-8B parameter tier.\n\n\\\n**3.Retrieving principles.** \n\nThank you for the reference to the MetaMind paper — we looked into this further and found that this paper was released on arXiv after the NeurIPS deadline, so we were not aware of this work. The framework of retrieving principles raises an interesting point — however, it assumes that one has access to a constitution or an a priori list of principles. It is also unclear what the criterion (or which model) would be responsible for selecting the top-k principles. By contrast, our work instead makes principle selection intrinsic to the model — it can use the principles generated in previous iterations, but can also create new principles for self-refinement as needed, if the principles it has learned do not apply. Nonetheless, we will certainly keep this point in mind for future work on principle-guided refinement and applications of the generated constitutions. We will also cite the MetaMind work along with a discussion of its similarities and differences from STaPLe in the final version of the paper.\n\n\\\n**4. Convergence over multiple iterations.** \n\nGiven that the method of data curation for the STaR baseline relies on applying self-correction and training on the refined response, we end up with fairly similar volumes of data used for SFT for both the STaR and STaPLe methods, as shown in Table 2. While we do find that the scores of our STaPLe method does plateau sooner than STaR (in the fourth iteration), its scores are already substantially better than the STaR baseline for a similar number of samples. \n\n\\\n**5. Reinforcement learning comparison.** \n\nIndeed, one can view the function $f(\\cdot)$ comparing a response against the reference as a reward function; in Appendix F, where we relate our framework to self-play in reinforcement learning, we define the self-play advantage as the difference in closeness to the reference, with the initial response’s score as a baseline. We ultimately show that our method is theoretically equivalent to self-play, as the MC-EM gradient induced by our framework and the self-play gradient using REINFORCE are equivalent. We also experiment other instantiations of $f(\\cdot)$, specifically using a Phi-4 LLM-as-a-judge (in Appendix M.2). We leave further exploration with reinforcement learning algorithms for principle-conditioned self-correction and alignment to future work. \n\n**6. Interpretation of generated constitutions and generalization across tasks.**\n\nWe note that STaPLe is a data-driven discovery process, so the set of principles we obtain is subject to the task / prompt distribution chosen. We have a multi-task mixture consisting of samples spanning safety, summarization, knowledge-based QA, general purpose instruction-following, truthfulness, and helpfulness. As such, our results demonstrate that our method can obtain principles across various use cases, while clustering again helps to distill the core aspects the model has been trained to reflect. Naturally, expanding the size of the prompt mixture used for principle discovery can aid in further generalization; nonetheless, the constitutions induced by smaller LMs under our current mixture already illustrate generalizable insights.\n\n In Appendix K, we include an analysis of the number of samples which have the respective medoid elements in the constitution during iteration 4 SFT — that is, the data-driven “weightage” of each principle during its last iteration of training. For further analysis, we include below a breakdown of the Llama 8B constitution in terms of the high-level domains each element of the constitution belongs to, as desired. We categorize them as “helpfulness”, “relevance”, “style”, “safety”, and “truthfulness and honesty”, and find that the categories are nearly evenly balanced. \n\n* Helpfulness: 6 principles (e.g. \"Directness and Assertiveness\")\n* Style: 5 principles (e.g. \"Structure and Organization\")\n* Relevance: 6 principles (e.g. \"Minimize Unnecessary Information\")\n* Safety: 6 principles (e.g. \"Avoiding Harm and Sensitivity\")\n* Truthfulness and Honesty: 6 principles (\"Acknowledge and Address Previous Misconceptions Clearly\")\n \n---\nWe hope these responses have addressed your concerns — if so, we would like to kindly ask you to further your support of our work. We would also be happy to address any further concerns you may have. Thank you again for your time!"}, {"review_id": "uq8bWBuavw", "reviewer": "Reviewer_9mwi", "summary": "The paper introduces STaPLe (Self-Taught Principle Learning), a method designed to enable self-improvement of language models (LMs). The core idea behind STaPLe is that a model can automatically discover and learn latent principles (rules or attributes) that help it improve its responses. STaPLe employs an Expectation-Maximization (EM) algorithm consisting of two main steps: E-step (principle discovery) and M-step (principle learning). In the E-step, the model uses rejection sampling to identify potential principles and improve its responses, while the M-step fine-tunes the model based on the discovered principles. The paper demonstrates that STaPLe outperforms existing approaches like STaR in various benchmarks (MT-Bench, AlpacaEval, IFEval), showing that self-improvement based on latent principles is an effective strategy for enhancing LM performance.", "questions": "Please see weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\nSTaPLe makes contributions in principle-augmented fine-tuning, which is both timely and important, by leveraging EM algorithm to enable iterative learning of principles and response refinement. This approach allows the model to improve its performance by learning principles automatically, differentiating it from methods that rely on predefined principles. Another key strength of STaPLe is its use of posterior regularization via clustering, which compresses the learned principles while maintaining their interpretability. This process helps to make the principles more understandable to humans, enhancing transparency. Additionally, STaPLe’s agnostic approach to similarity metrics offers flexibility, allowing the model to use various metrics for performance improvement, making it adaptable to different tasks and ensuring robust performance across various use cases.\n\n**Weaknesses**\n\n1. **Reliance on another type of human involvement.** The authors claim that their approach aims to avoid conventional forms of human involvement, such as relying on “human annotations distinguishing between a chosen and rejected generation” or “human-curated constitutions.” However, their proposed method fundamentally depends on gold responses provided by users, another form of human involvement. Given this, the manuscript lacks a clear justification of why requiring gold responses from users is practically more efficient, scalable, or less burdensome compared to the forms of human involvement it aims to replace. Without this, the practical usage and broader applicability of the proposed method could be limited, as it merely substitutes one form of human dependence with another.\n2. **Insufficient positioning and relevant baseline.** The positioning (Section 2) could be improved by explicitly discussing the similarities and differences between STaPLe and closely related approaches, especially Findeis et al. (2025). In my view, both works aim at deriving latent principles for aligning language models, but differ in their learning source (and of course, the methodology): STaPLe derives them from gold responses, whereas Findeis et al. derive them from pairwise comparisons. In addition, given the first weakness (reliance on gold responses), it would be helpful to empirically evaluate Findeis et al.’s comparison-based approach as an additional baseline. Such an evaluation would clarify the efficiency trade-offs and practical advantages of deriving latent principles from gold responses versus pairwise comparisons.\n3. **Analysis of representatives.** While the clustering mechanism in STaPLe reduces the number of unique principles, it is unclear whether the compressed principles effectively *represent* the original set. The analysis demonstrates compression but lacks a direct evaluation of *representativeness*, which is essential to confirm that the reduced set still captures all critical principles for improved interpretability.\n\n\n**Minor comments**\n\n1. The method description relies heavily on numerous equations, which may be difficult for a broader audience to fully grasp. It would be beneficial to include an illustrative figure that shows the effect of each process, as the currently included figure only demonstrates the overall process flow and does not highlight the impact or outcome of each step in the method.\n2. (Very minor) It would improve readability if there were a direct comparison table showing performance based on different similarity scores. Currently, the tables for each similarity score are presented separately (one in the main text and another in the appendix), which makes it harder to quickly compare them. Combining this information in a single table could enhance clarity.\n\n**References**\n\n- Findeis et al. (2025). “Inverse Constitutional AI: Compressing Preferences into Principles.”", "was_revised": true, "final_justification": "I am maintaining my score, as the authors did not respond to my follow-up questions, leaving the discussion incomplete. This is problematic because key concerns regarding the paper's claims (**Human annotations**) and evaluation (**focused analysis comparing the quality of the extracted principles**) remain unaddressed.", "rebuttal": "We would like to thank Reviewer 9mwi for taking the time to review our work and for their valuable feedback. We are encouraged that the reviewer finds our premise to be timely and important, appreciates the interpretability introduced through our clustering method, and the adaptability of our algorithm to various tasks and scoring mechanisms. We address the concerns raised in the review in our responses below:\n\n\\\n**1. Human annotations.**\n> The authors claim that their approach aims to avoid conventional forms of human involvement, such as relying on “human annotations distinguishing between a chosen and rejected generation” or “human-curated constitutions.” However, their proposed method fundamentally depends on gold responses provided by users, another form of human involvement. Given this, the manuscript lacks a clear justification of why requiring gold responses from users is practically more efficient, scalable, or less burdensome compared to the forms of human involvement it aims to replace. Without this, the practical usage and broader applicability of the proposed method could be limited, as it merely substitutes one form of human dependence with another.\n\nWe would like to clarify that our method only requires human annotations that are already public, and widely available through the open-source datasets available on Hugging Face. Our method does not aim to replace preference annotations — in fact, we use the chosen response as the target ($y^G$) for the preference dataset HH-RLHF in our multi-task prompt mixture. In alignment tasks, we seek to reflect human generations and preferences, so we need a form of reference that is reliable — hence, a gold response for QA settings, or a chosen response in preference pairs. Thus, from our perspective, it is no more costly to use human-written gold responses than it is to use human preference data— it is as simple as downloading the data — our method could feasibly be run using purely preference data or only using gold responses, rather than a mixture of both. Our choice of mixture was to capture a diverse set of domains (safety, summarization, truthfulness, general purpose instruction-following, grounded QA, etc.). What we instead seek to replace is the process of distilling desirable attributes for responses (equivalently, the “lens” of preference) into a readable set that the model is simultaneously trained to reflect — which our method yields at the end of each iteration. \n\n\\\n**2. Comparison with Inverse Constitutional AI.** \n> The positioning (Section 2) could be improved by explicitly discussing the similarities and differences between STaPLe and closely related approaches, especially Findeis et al. (2025). In my view, both works aim at deriving latent principles for aligning language models, but differ in their learning source (and of course, the methodology): STaPLe derives them from gold responses, whereas Findeis et al. derive them from pairwise comparisons. In addition, given the first weakness (reliance on gold responses), it would be helpful to empirically evaluate Findeis et al.’s comparison-based approach as an additional baseline. Such an evaluation would clarify the efficiency trade-offs and practical advantages of deriving latent principles from gold responses versus pairwise comparisons.\n\nAs the reviewer notes, STaPLe and ICAI (Findeis et al. 2025) broadly aim at eliciting principles from available human data (preference annotations) — however, as aforementioned, our method does not solely rely on access to gold responses, and can use preference annotations as well. ICAI uses the rejected (dispreferred) response, whereas STaPLe uses an initial response generated on-policy from the target LM as the baseline. Both methods also make use of clustering as a means to distill relevant principles down to an interpretable set. However, the primary aim of ICAI is to reconstruct preference annotations (in a single iteration) to verify the constitution’s correctness, so that they may be used to interpret human preferences at inference time. By contrast, STaPLe uses the principles as a learning signal over multiple iterations, to imbue a distribution of responses that reflect similar attributes as their gold references, and learns a self-correction behavior that intrinsically calls those principles. Another important difference is the principle generator — ICAI uses strong, frontier-level models (GPT-4o and GPT-3.5-turbo), whereas STaPLe uses on-policy-generated principles from the small LM (7-8B parameters) itself, without external guidance or supervision from any other model. STaPLe is also fully automated — with the Bayesian optimization framework, even the size of the constitution can be purely discovered from the data without human threshold-setting during hierarchical clustering — in contrast to ICAI, which performs k-means clustering specifically aiming for 5 clusters. Ultimately, while ICAI is designed with the purpose of better understanding human preferences (rather than improving the LM’s intrinsic alignment as reflected on MT-Bench/AlpacaEval), STaPLe teaches non-frontier LMs attributes that should be reflected, resulting in a better aligned language model. It is unclear how ICAI as a baseline would aid in improving response quality; one would need to perform a method like Self-Refine using the ICAI-generated principles, which would not be reflective of the ICAI method itself. \n\nWe appreciate the reviewer’s suggestion for more explicit positioning of our work with respect to other approaches — although PDF updates are not permitted during this rebuttal cycle, we would be happy to add such a discussion in our camera-ready version. \n\n\\\n**3. Representativeness of constitutions.** \n> While the clustering mechanism in STaPLe reduces the number of unique principles, it is unclear whether the compressed principles effectively represent the original set. The analysis demonstrates compression but lacks a direct evaluation of representativeness, which is essential to confirm that the reduced set still captures all critical principles for improved interpretability.\n\nThis is a valuable point, and we find that measuring or evaluating information retention is a technical challenge in its own right. We designed the perplexity-based label replacement scheme in Appendix M.1 with this aim in mind — to replace individual principle instances with a representative (medoid induced by clustering) such that the difference in perplexity (“surprisal” effect) when replaced is minimized. To this effect, we find this perplexity-based scheme to yield largely comparable performance to just using the assigned medoids; we primarily report the latter for the sake of simplicity and efficiency. Additionally, in the aim of creating better clusters and therefore, better representative elements, our Bayesian optimization framework in Appendix M.3 addresses the tradeoff between diversity of the clusters with tightness -- ensuring the elements are similar in embedding space, in a purely automated manner. We would be happy to try other methods of measuring and optimizing for representativeness, if you have any in mind! \n\n\\\n**4. Clarity.** \n> The method description relies heavily on numerous equations, which may be difficult for a broader audience to fully grasp. It would be beneficial to include an illustrative figure that shows the effect of each process, as the currently included figure only demonstrates the overall process flow and does not highlight the impact or outcome of each step in the method.\n> (Very minor) It would improve readability if there were a direct comparison table showing performance based on different similarity scores. Currently, the tables for each similarity score are presented separately (one in the main text and another in the appendix), which makes it harder to quickly compare them. Combining this information in a single table could enhance clarity. \n\nThank you very much for the suggestions! As aforementioned, although the guidelines for this rebuttal period do not permit updating our paper with new figures/tables, we would be more than happy to make these changes for our camera-ready version. Specifically, we would be glad to illustrate the result of each step of the process, such that one can interpret how principles are discovered from the LM, refinements induced given the principles and critiques, the refinements filtered based on the scoring function $f(\\cdot)$, the clustering mechanism being used to replace the principles in the trajectories with representative elements, and then performing fine-tuning on the augmented trajectories. \n\n---\nWe hope these responses have addressed your concerns — if so, we would like to kindly ask you to further your support of our work. We would also be happy to address any further concerns you may have. Thank you again for your time!"}]

Author rebuttal:

Meta-review: The paper studies the self-refinement of large language models by proposing a fully automated principle-driven self-improvement framework. Reviewers provided useful feedback, and the authors should compare to inference-time principle-based optimization methodology during the revision.
Decision: Accept (poster)