Inverse Constitutional AI: Compressing Preferences into Principles
Authors: Arduin Findeis
Timo Kaufmann
Eyke Hüllermeier
Samuel Albanie
Robert D. Mullins
Venue: ICLR 2025 Poster (2025)
Keywords: human feedback
evaluation
interpretability
preference learning
AI annotators
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We compress pairwise text preferences into a set of interpretable principles (a constitution) that may help identify undesirable biases, improve AI annotators and assist in personalizing LLMs.

Abstract: Feedback data is widely used for fine-tuning and evaluating state-of-the-art AI models. Pairwise text preferences, where human or AI annotators select the “better” of two options, are particularly common. Such preferences are used to train (reward) models or to rank models with aggregate statistics. For many applications it is desirable to understand annotator preferences in addition to modelling them  – not least because extensive prior work has shown various unintended biases in preference datasets. Yet, preference datasets remain challenging to interpret. Neither black-box reward models nor statistics can answer why one text is preferred over another. Manual interpretation of the numerous (long) response pairs is usually equally infeasible. In this paper, we introduce the Inverse Constitutional AI (ICAI) problem, formulating the interpretation of pairwise text preference data as a compression task. In constitutional AI, a set of principles (a constitution) is used to provide feedback and fine-tune AI models. ICAI inverts this process: given a feedback dataset, we aim to extract a constitution that best enables a large language model (LLM) to reconstruct the original annotations. We propose a corresponding ICAI algorithm and validate its generated constitutions quantitatively based on annotation reconstruction accuracy on several datasets: (a) synthetic feedback data with known principles; (b) AlpacaEval cross-annotated human feedback data; (c) crowdsourced Chatbot Arena data; and (d) PRISM data from diverse demographic groups. As an example application, we further demonstrate the detection of biases in human feedback data. As a short and interpretable representation of the original dataset, generated constitutions have many potential use cases: they may help identify undesirable annotator biases, better understand model performance, scale feedback to unseen data, or assist with adapting AI models to individual user or group preferences. We release the source code for our algorithm and experiments at https://github.com/rdnfn/icai.

Published as a conference paper at ICLR 2025
INVERSE CONSTITUTIONAL AI: COMPRESSING PREFERENCES INTO PRINCIPLES
Arduin Findeis'
Timo Kaufmann"
University of Cambridge
LMU Munich, MCML. Munich Munich, Germany
Cambridge, UK
Eyke Hüllermeier
Samuel Albanie London, UK
Robert Mullins
LMU Munich, MCML Munich DFKI, Kaiserslautern, Germany
University of Cambridge Cambridge, UK
ABSTRACT
Feedback data is widely used for fine-tuning and evaluating state-of-the-art Al models. Pairwise text preferences, where human or AI annotators select the "bet- ter" of two options, are particularly common. Such preferences are used to train (reward) models
rank models with aggregate statistics. For many applica- tions it is desirable to understand annotator preferences addition to modelling them -ont least because extensive prior work has shown various unintended bi- ases in preference datasets. Yet, preference datasets remain challenging to inter- pret. Neither black-box reward models nor statisties can answer why one text is preferred over another. Manual interpretation of the numerous (long) response pairs is usually equally infeasible. In this paper, we introduce the Inverse Consti- nutional AI (ICAI) problem, formulating the interpretation of pairwise text pref- erence data as a compression task. In constitutional AI, set of principles (a constitution) is used to provide feedback and fine-tune Al models. ICAI inverts this process: given a feedback dataset, we aim to extract a constitution that best enables a large language model (LLM) to reconstruct the original annotations. We propose a corresponding ICAI algorithm and validate its generated constitutions quantitatively based on annotation reconstruction accuracy several datasets: (a) synthetic feedback data with known principles; (b) AlpacaEval cross-annotated human feedback data; (c) crowdsourced Chatbot Arena data; and (d) PRISM data from diverse demographic groups. As an example application, we further demon- strate the detection of biases in human feedback data. As a short and interpretable representation of the original dataset, generated constitutions have many potential use cases: they may help identify undesirable annotator biases, better understand model performance, scale feedback unseen data, assist with adapting Al models to individual user or group preferences. We release the source code for our algorithm and experiments https: ///gt//gtttt///// com/ rdn fn/ fn/icai.
Original feedback data
Constitution
Reconstructed feedback data
Inverse Constitutional AI (our algorithm)
Al annotator using constitution
Figure The Inverse Constitutional Al problem. Starting with pairwise preference feedback data, we derive set of natural language principles (a constitution) that explain the preferences. For validation, we reconstruct the original preferences with an LLM judging according to the generated constitution. The constitution represents (highly compact) compression of the preferences.
"arduin findei sêcst cam ac.uk. 't kauffannd fi mm..de

---

Published as a conference paper at ICLR 2025
INTRODUCTION
State-of-the-art Al models, in particular large language models (LLMs), rely heavily on human feed- back for training and evaluation. This feedback, often in the form of pairwise text preferences, is crucial to assess advanced capabilities, which are hard to evaluate automatically. Pairwise text pref- erences typically consist of a prompt, two model responses and an annotation selecting the "better" response. Strategies for training on such data have seen widespread adoption, with notable exam- ples including reinforeement learning from human feedback (RLHF) (Ouyang et al., 2022; Stiennon et al., 2020) and direct preference optimization (DPO) (Rafailov et al., 2023). Beyond training, pairwise text preferences are also widely-used for evaluating L.M,s, such as in the Chathot Arena (Chiang et al., 2024), where crowdsourced preferences determine rankings possibly reflecting the complexity of human preferences better than conventional benchmarks (Xu et al., 2023).
However, interpreting such pairwise data is challenging: describing what exactly we train model to do when applying RLHF with a large number of preference pairs is hard. Similarly, understanding why model is ranked higher in a pairwise data-based leaderboard remains difficult. Yet, under- standing such data is critical: human feedback is not without its flaws. Systematic biases in human judgement have been documented extensively in the psychology literature (Tversky & Kahneman, 1974). Perhaps unsurprisingly, biases have been similarly observed in the human feedback used to guide and evaluate LLMs. For example, human annotators have been observed sometimes favour assertiveness (Hosking et al., 2024) or grammatical correctness (Wu & Aji, 2023) over truthfulness.
Feedback data with unintended biases can be problematic: when used for fine-tuning, biased data may lead to models that exhibit the same biases. Similarly, leaderboards based on biased data will favour misaligned models (Dubois et al., 2023; 2024). As such, understanding the implicit rules and biases guiding annotators of feedback data is valuable. To date, however, few general tools exist to detect biases in pre-existing preference data at scale. Prior work detecting biases usually either requires specially collected data or focuses on biases easy programmatically measure (e.g. length bias (Dubois et al., 2024)) - difficult to extend to pre-existing datasets or less controlled settings.
In this paper, we propose novel general approach to understanding preference corpora: Inverse Constitutional Al (ICAI). We make the following contributions:
1. Formulating the Inverse Constitutional AI (ICAI) problem. In Constitutional AI (Bai al., 2022b), a set of principles (or constitution) is used to provide feedback and fine-tune language models. ICAI inverts this process: given a dataset human or AI feedback, we seek to compress the annotations into set of principles that enable reconstruction of the annotations (Figure 1).
2. Proposing an initial ICAI algorithm. We introduce a first ICAI algorithm that generates a set of principles based on a feedback dataset. We validate the constitutions generated by our algorithm based on their ability to help reconstruct feedback. Given the complexity of human judgement, the constitution necessarily represents "lossy", non-unique compression of the feedback data. Nevertheless, the interpretable nature of the principles may enable a number of promising downstream use cases: (a) highlighting potential issues in preference data; (b) evaluation to new models and use cases; and (d) generating personal constitutions for customized model behaviour.
creating interpretable reward models; (c) scaling
Providing experimental results and case studies. We test our approach experimentally on four datasets: (a) we first provide a proof-of-concey on synthetic data with known underlying princi- ples; (b) we then demonstrate applicability to human-annotater data on the AlpacaEval dataset (Dubois et al., 2023); (c) we showcase applicability to interpreting individual user preferences via Chatbot Arena Conversations data (Zheng et al., 2023); (d) we investigate the use-case of bias detection on different datasets; and finally (e) we demonstrate our method's ability to help interpret differing group preferences on PRISM data (Kirk et al., 2024). We demonstrate the highly sample-efficient generation of personalised constitutions with human-readable and ed- itable principles. We release our code at https: / /i iigittin thub. com httm//ddfnniiiin iiai..
2 THE INVERSE CONSTITUTIONAL AI PROBLEM
Given a set of pairwise preference feedback, the Inverse Constitutional AI (ICAI) problem is to generate a corresponding constitution of natural-language principles that enable an LLM annotator
2

---

Published as a conference paper at ICLR 2025
Inverse Constitutional AI algorithm
3 Sub- sampling
4 Testing
5
Candidate principles
Constitution
Initial candidate
Feedhack
generation
Clustering
Filtering
Figure 2: Overview of our Inverse Constitutional AI (ICAI) algorithm. Given a dataset of pair- wise comparisons, in Step candidate principles are generated using LLM. In Step 2, these principles are clustered using an embedding model. In Step 3, similar principles are deduplicated by sampling one principle per cluster. In Step 4, each principle is tested evaluate its ability help an LLM reconstruct the original annotations. Finally, in Step 5, the principles are are filtered according the testing results, and set of filtered principles returned the final constitution. Optionally, a final step of additional clustering and subsampling can follow to ensure diverse principles.
to reconstruct the original preferences as well as possible. Formally, we seek to find arg max agreement(po: PM)()) s.t. |c| n).
where are the original preferences and (c) are constitutional preferences over a pairwise pref- erence corpus T, generated by LLM M using the constitution c. The constitution is constrainted to consist of up to n age of constitutional preferences par (c) identical the original preferences Po. constitution with high agreement can help interpret preference dataset to gain insight into the underlying annotator preferences and biases. The constitution may also be used for future preference synthesis, with an interpretable and editable set of principles.
natural language principles. Agreement is defined as the percent-
3 METHOD
We propose first Inverse Constitutional AI (ICAI) algorithm, outlined in Figure 2, consisting of five main steps: principle generation, principle clustering, principle subsampling, principle testing, and principle filtering. In the following, we describe each step in detail.
Step 1: Principle generation. We extract candidate principles using an LLM with access to the feedback data. The principles are generated on per-comparison basis: an LLM prompted with pair of texts and corresponding preference, and then asked to propose principles that explain the preference (prompts Appendix G. The generated principles are in the form of natural language instructions that inform preference decisions (e.g.. "select the more polite output"). We generate large number of candidate principles using multiple (by default 2) generation prompts and multiple principles per prompt to cover a wide range of potential rules. The generation prompts affect the type of principles that get generated and tested (e.g.. specific/general, positive/negative rules).
Step Principle clustering. Since the first step generates a large number of candidate principles independently. almost identical principles may be generated multiple times. We use k-means- based clustering on embeddings to identify principles that are similar for merging. The parameter deter- mines the number principles considered downstream and thus affects overall computational cost.
Step 3: Principle subsampling. In the third step, we deduplicate the principles by randomly sam- pling one principle per cluster, leading to a diverse of remaining principles.
Step 4: Principle testing. The fourth step evaluates the generated principles' ability to help an LLM reconstruct the original annotations. We prompt the LLM with the generated principles to determine the 'votes" each principle casts across comparisons, which then compare to the true annotations (see Appendix G.2). We parallelize this step, prompting the LLM with a pair of texts and multiple principles to provide separate response (first preferred, second preferred, not relevant)
3

---

Published as a conference paper at ICLR 2025
for each principle. This parallelization reduces the token requirements compared to separate testing. While LLMs can exhibit anchoring effects when predicting multiple labels in one output (Stureborg et al., 2024), we hypothesize this effect is less pronounced for relative preferences and our experi- mental results indicate sufficient reliability on our datasets. We compare these results to the original labels and count the correct, incorrect, and not relevant labels for each principle separately, thereby identifying principles that help the LLM to correctly annotate the dataset.
Step 5: Principle filtering. Finally, the principles are filtered based on the results of the previous testing step. We only keep principles that improve the reconstruction loss, while discarding princi- ples that do not help or even hinder the reconstruction. We further discard principles that are marked as relevant less than 2% of the data (default 10%), avoid overly specifie principles that do not generalize. We order the principles according to their net contribution to correctly annotating the dataset (#correctt #incorrect annotations).. We then select the top principles according to this der (see Appendix F.5 for further discussion). Optionally, to increase principle diversity, we cluster the top m (> n) principles into n clusters as before, and subsample the highest ordered principle from each cluster.' The final ordered list of principlesis returned as the constitution.
Inference. Given a constitution, we can validate its ability to "cxplain" " the original feedback dataset. We do this validation using AI annotators prompted with the constitution, an approach pioncered by Bai et al. (2022b) and commonly referred to as constitutional AI. Notably this method leaves room for interpretation of the constitution by the AI annotator, the constitution may be ambiguous, con- tradictory or incomplete, but results may also vary depending on the exact prompt and constitution. We build the AlpacaEvvll framework (Li al., 2024c) for our constitutional annotators. This in- ference using constitutional Al annotators enables quantitatively testing the validity of our generated constitutions and their ability to explain the data while also enabling downstream use cases, such as personalized preference models.
4 EXPERIMENTS
We conduct experiments on four datasets: (1) synthetic data to demonstrate the hasic functionality of our algorithm, (2) human-annotated AlpacaEval data demonstrate the applicability of our al- gorithm to real-world data, (3) Chatbot Arena data to illustrate the application of our algorithm to infer individual user preferences, and (4) PRISM data showcase interpreting group preferences with our algorithm. Full dataset details are available in Appendix A. We primarily use two mod- els from OpenAI: GPT-3.5-Turie and GPT-4o. Example constitutions all figures were chosen for illustrative purposes. We provide more constitutions, experiment details (including numerical results), and model details in Appendices D, F and H, respectively.
Annotators. We use annotators from the AlpacaEval framework (Li et al., 2024c) as baselines that have been shown to strongly correlate with human preferences (referred to as Default annotators, see Appendices F.8 and G.4). To evaluate constitution effectiveness, we create custom prompts that ask the model to annotate according to principles in a constitution (see Appendix G.3). The Default annotators cannot adjust to different datasets, performing poorly when preferences deviate from its default preferences. In contrast, our constitutional annotators are able to adapt, a key advantage of approach. We show random baseline performance as grey dashed line at 50% in all plots. To contextualize ICAI's performance, we compare it against additional baselines: a flipped Default annotator, (fine-tuned) reward model, and annotator based PopAlign (Wang al., 2024) hypothesizing principles during inference. Details are in Appendix F88, summarized here. Non- adaptive baselines (pretrained reward model, (flipped) Default) perform well on some datasets but fail to adjust to all. The fine-tuned reward model adapts partially but underperforms our constitu- tional annotator in our low-data scenario. A custom- -triined reward model with extensive data may surpass our method but would require significant resources and lack interpretability.
4.1 PROOF-OF-CONCEPS SYNTHETIC DATA
We first apply our algorithm to three synthetic datasets created according to known rules crafted to be aligned, unaligned, and orthogonal to the preferences internalized by the base LLM. Each dataset
We found it important not he too restrictive with the number of clusters in Step 2, as good principles may never be tested. More clusters can lead duplicates in the tested rules, necessitating another filtering step.
4

---

Published as a conference paper at ICLR 2025
Orthogonal
Aligned
Unaligned
with original data
Agreement (%)
Annotators
Default
Constitutional
50
0% (Default)
Example constitutions
Select the response that features a cat as the
1. Select response that maintains a positive and helpful tone.
Select the response lacks cagerness and willingness to assist. Select the response that friendly more indifferent. [...]
pet.
Select the response that flavor option. L...]
2. Select response that concise point. [...]
Figure 3: Results synthetic data. Our constitutional annotators can reconstruct variety of preferences using limited data and without fine-tuning. We demonstrate our algorithm's 's adapt- ability three synthetic datasets: aligned with those preferences and one unaligned with them. We generate constitutions for each and report agreement with the original data of default LLM annotator and constitutional annota- tor (prompted with a constitution). Our constitutions notably improve agreement the orthogonal and unaligned cases and retain high agreement in the aligned case, albeit with more variance, Our method's ability to detect biases is illustrated by the example constitution in the unaligned case. Plots show mean and standard deviation (6 seeds) using GPT-3.5-Turie
orthogonal to the base LLM's learned preferences, one
generated from three principles, with 10 pairs per principle, resulting in 30 pairs per dataset. We provide an overview of these datasets here, with further details in Appendix I.
Orthogonal. This dataset based on principles intended to be neither supported nor opposed by humans or language models on average. In particular, we create a dataset based on three principles: "prefer cats over dogs", "prefer green over blue color", and "select lemon over raspberry ice-cream".
Aligned and unaligned. The aligned dataset (especially) language models. Our dataset follows three principles: "select truthful over factually incorrect answers", "select helpful over useless answers". "select polite over impolite answers". The unaligned dataset flips these annotations, creating a dataset that default LLM annotator mostly disagrees with.
preferences generally accepted by humans and
Results. In Figure 3, we compare default annotators (prompted to select the "best" output) to con- stitutional annotators (prompted with generated constitution). We find that constitutional annota- tors reconstruct original annotations better in the orthogonal and unaligned datasets, and keep high agreement in the aligned case (which offers limited room for improvement). These results indicate that the constitutions capture helpful information about the preferences. Qualitatively, the generated constitutions (see Appendix H) often closely correspond to the principles described above.
4.2 HUMAN-ANNOTATED ALPACAEVAL DATA
We test our approach on human-annotater texts using the AlpacaEval dataset (Dubois al., 2023). The dataset, used for the AlpacaEval leaderboard, features 648 data points cross-annotated by four annotators, with well-tested baseline Al annotators and evaluation tooling. The dataset captures general human preferences, likely very similar to the preferences language models used in this work have been fine-tuned on. As a consequence, the default annotator (without a constitution) agrees strongly with the annotations, leaving little room for improvement. The goal of this experiment, then, is not to exceed the default annotator's annotation performance on this dataset but to answer the following research questions: (Q1) Can constitutional annotators match the default annotator's performance on the aligned dataset, while providing the benefit of interpretable and editable consti- tutions? (Q2) Is ICAI able to extract and follow principles that are exactly opposite to the aligned dataset, despite the underlying model's 's biases? Note that most practical applications will be some- where between the two extremes of the aligned and the unaligned scenario, allowing ICAI to increase the annotator's performance while offering insights into the learned principles, along with the ability to inspect and modify them needed.
5

---

Published as a conference paper at ICLR 2025
Aligned
Unaligned
Agreement (%6
50
Annotators
Defeult
Constitut.
GPT-3.5-Turbo
GPT-4o
GPT-3.5-Turbo
GPT-4o
Example constitutions
1. Select the response that maintains neutral informative tone.
1. Select the response that provides an overly simplistic misleading
Select the response
avoids spelling or grammmatiaal errors.
2. Select response that depth analysis.
3. Select the response that conveys stronger of personal experience. [...]
3. Select the response that contains incorrect ar nonsensical logic. [...]
Figure 4: Results on AlpacaEval data. GPT-4o generates and uses interpretable constitutions that match the performance of the default annotator aligned preferences and notably increase agreement with unaligned preferences. Tested on aligned (original) and unaligned (flipped) versions of AlpacaEval, with GPT-4o generating constitutions which are then used by con- stitutional annotators backed by GPT-4o and GPT-3.5-Turbo Note we can only expect significant improvement in the unaligned case, as discussed in the main text. The aligned case does not leave room for improvement over the default annotator, but allows us to gain new insights into the prefer- ences expressed in the dataset. In the unaligned case, GPT-4o's agreement improves notably, while GPT-3.5-Turbo's '' performance does not exceed random choice, indicating its limited ability to fol- low unaligned principles. Plots show mean and standard deviation (6 seeds).
Experimental setup. For each seed, we randomly select mutually exclusive training and test sub- sets with 65 annotated pairs each. Constitutions generated on the training subset and results reported on the (unseen) test subset. We derive an aligned dataset based on majority vote (ties broken randomly) and unaligned dataset using flipped annotations.
Results. Figure 4 shows that constitutional annotators approximately match default annotator per- formance in the aligned scenario while using easily interpretable constitution, answering (Q1) alfirmatively.. The unaligned dataset shows GPT-4o succeeding at following opposing principles, improving beyond the default annotator, while GPT-3.5-Turte fails do so despite using the same GPT-4o-generated constitutions. This answers (Q2) affirmatively for GPT-4o, revealing a capability gap between models. Since we evaluate the constitutions on unseen test set, these results also demonstrate ICAI's potential for annotation scaling. extracting constitution from a small training set (65 preferences here) and applying to new data.
Constitution transferability. Given GPT-3.5-Turbo's limitations in the unaligned case in Figure 4, Appendix F.6 provides a more general exploration of how well our constitutions can transfer be- tween models. To this end, we take the best performing constitution of the unaligned case on the training set and test how well models from Anthropic's Claude family are able to use these consti- tutions the test set. The results indicate that this constitution transfers well to the Claude models better than to GPT-3.5-Turbe although transfer still incurs some loss. This is a promising result as it indicates that constitutions not excessively overfitting to the generation model, which indicates that they may capture more general concepts that are also interpretable to humans.
Finally, although consider sample-efficiency to be benefit of our approach, we further evaluate whether these results also hold with larger scale datasets. We repeat the experiment on the AlpacaEvall unaligned dataset with the full 648 preference pairs in the original dataset, using 324 samples each for training and testing. We observe that the overall results are very similar: the constitutional annotator with 61% agreement (66% prev.) still notably outperforms the the default annotator with 34% (27% prev.). The full results are included in Appendix F.7.
Scaling.
We observe very slight improvement GPT-3.5-Turbe annotations and a similarly small reduction GPT-4o's 's agreement. This may be explained by the GPT-3.5-Ture model being less attuned to the preferences in the dataset, which can be alleviated with constitution. In the case of GPT-4o, however, the constitutinnal annotator likely focuses on the highly compressed constitution, even in cases where the default annotator would have nuanced (but less interpretable) understanding of the underlying preferences.

---

Published as a conference paper at ICLR 2025
Annotations Group A
Annotations Group B
Agreement (%6
75
50
Annotators
Group constitution
25
Default
Group constitution
Example constitutions
1. Select response that avoids redundancy and repetition.
Select the response that provides more detailed descriptions,
2. Select the response that concise and to the point.
2. Select the response that avoids pretending to have
emations.
response that directly addresses the user's concern. [...]
Select the response offers actionable steps like discussing with employer. [...]
Figure 5: Case-study: Constitutions for demographic groups on PRISM data. We consider two groups reported by Kirk et al. (2024) to have preferences differing from average: participants born in one geographical region rank Mistral-7b higher in this dataset (Group A), and those born in another region rank Llama-2-7b lower than average (Group B). We generate constitutions for both groups to explore these preferences. For each group, the annotator using the group's data performs best. Constitutions (see Appendix H.4) suggest that Group A prefers Mistral-7b due to it's conciseness, while Group B's constitutions have recurring rules related to providing more detailed descriptions. Plots show mean and standard deviation (6 seeds) using GPT-4o.
4.3 INDIVIDUAL PREFERENCES: CHATBOT ARENA DATA
We evaluate ICAI's ability to generate personal constitutions using the Chatbot Arena Conversations dataset (Zheng et al., 2023), which consists of 33k prompts, offering richer insights into individual preferences compared to AlpacaEval. We select two exhibiting different preferences from the Chatbot Arena dataset (with 9 and 12 preference annotations respectively) and generate a separate constitution with 3 principles for each. Due to limited samples, there no training-test split. We provide details on the experimental setup and results in Appendix F.I and summarize the results here. As may be expected, we find that the personal constitutions generated for each user are able to improve the annotator's performance on the user's s annotations, but do not transfer well to the other user's s annotations. This shows that our constitutions successfully capture individual differences, illustrating the potential to generate personal constitutions. Note that how well a personal constitution transfers from one user to another depends on how similar their preferences are, with the contrast being the most pronounced between users with opposing preferences. Results will therefore vary for any given pair of users.
preferences with user-generated
4.4 DEMOGRAPHIC GROUP PREFERENCES: PRISM DATA
Finally, we test ICAI's ability to help interpret demographic group preferences on the PRISM dataset (Kirk et al., 2024), consisting of annotations on 8,01 multi-turn conversations with 21 LLMs across a range of value-based and controversial topics from diverse set of 1,396 annotators.
Experimental setup. We consider two annotator subgroups described by Kirk et al. (2024) have differing preferences with respect to certain models relative to the average annotators. Group A, annotators born in the same geographical region, are reported to prefer the outputs of Mistral-7b relative to the overall rank. Similarly, Group B. annotators born in a different region, are reported dislike Llama-2-7b disproportionately. These observations raise the question: why do these sub- groups prefer or dislike those specific models? With no concrete explanation in the original paper, ICAI offers method to generate and test possible explanations. We select the subset of PRISM interactions where Group A prefers Mistral-7b over another model (30 pairs"), and similarly, the subset where Group B rejects Llama-2-7b for any other model (80 pairs). Note that, as for Chatbot Arena, the few samples do not allow for train/test split, which we consider acceptable for the pur- poses of data explanation. We then run ICAI on each subset separately to create a constitution for
For each relevant PRISM datapoint, we pick one of three available rejected models random.

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation of possibly biased principles on three datasets. Results on AlpacaEval (648 preferences), Chatbot Arena (5,115), and PRISM (9,4 showing relevance (fraction of data points where the principle applies) and accuracy (correctly reconstructed relevant data points). Grey values indicate relevance for fewer than 50 preferences. See Table 2 in Appendix F.2 for extended results.
Principle
AlpacaEv, Rel
ChatbotAr.
PRISM
Select the response that...
Acc
Acc
Rel
Acc
Rel
overly lengthy and lacks brevity
52.2
80.4
57.1
97.1
57.7
96.9
provides a numbered list format
73,4
12.2
62.3
45.5
71.6
17.7
presents definitive stance without nuance
58.6
10.8
57.1
11.4
49.1
31.2
is overly general and vague emphasizes neutrality over providing information
30.4
15.7
26.8
9.6
26.1
23.1
50.0
2.2
40.8
10.2
58.0
46.2
each group, testing each both datasets. We the same prompting setup as the Chatbot Arena experiments.
Results. Figure shows that constitutional annotators exceed default annotator performance in reconstructing the datasets they aim to compress but (as expected) do not transfer well to the other group's annotations, in line with the observation by Kirk et al. (2024) that each group's preference differs from average preferences. Indeed, the generated constitutions (see Appendix H.4) allow us also ask how the preferences differ: Group A appears to strongly prefer more concise responses, whereas Group B has more diverse constitutions that often ask for more detailed descriptions.
4.5 APPLICATION: BIAS DETECTION
We showcase ICAI's application in bias detection, following three steps: (1) Generate and test 400 candidate principles on 1,000 preference pairs (500 from PRISM, 500 from Chatbot Arena*) to cap- ture diverse biases (using ICAI algorithm steps to 4). (2) Manually select principles indicating potential biases, focusing on those with high accuracy or limited applicability.
(3) Evaluate these principles 13k preferences (7, (7,490 from PRISM, 5, 5,115 from Chatbot Arena, and 648 from Al- pacaEval), using step of the ICAI algorithm. All steps use GPT-4o-mini for cost efficiency.
Results, shown in Table 1, reveal biases regarding verbosity, style, and assertiveness. Verbosity bias, where longer responses are preferred, is consistently observed across datasets, with Chatbot Arena and PRISM strongly favouring overly lengthy responses (notably, 57.1% and 57.7% accuracy, spectively). Style biases, such as a preference for numbered lists, are especially prominent in Al- pacaEval (73% accuracy) and PRISM (72%), although their relevance is more limited compared to verbosity-related principles. Additionally, biases around ambiguity and vagueness vary by dataset; for example, PRISM annotations often favour neutral responses, while Chatbot Arena appears to actively select against response emphasizing neutrality. These results emphasize the fataset-speciff nature of biases and demonstrate our framework's s sensitivity to such patterns. More biases, dis- cussion and mitigation strategies can be found in Appendix F.2. Further, we provide an additional application example of annotation scaling on helpful/harmless data Appendix F.3.
4.6 ABLATION STUDIES
We conduct ablation studies to assess the contribution of each step in our pipeline across four scenar- ios: synthetic orthogonal, synthetic aligned, synthetic unaligned. and AlpacaEval unaligned, using GPT-3.5-Turbo for the first three and GPT-4 for the last. Numerical results, as well as detailed discussion of the ablations, the experimental setup, and the results, are provided in Appendix F.4. Key findings are summarized below:
Simplified principle generation (Step 1). Generating only a single principle with a neutral prompt slightly reduces performance, indicating the importance of diverse principles. The performance drop is particularly pronounced on the synthetic aligned dataset, which aligns with observation that GPT-3355-Tubbe struggles to generate both positive and negative principles from a single prompt.
"This experiment the Kaggle data (see Appendix A), differing from the data used in other experiments.
8

---

Published as a conference paper at ICLR 2025
No deduplication (Steps 2, 3, and 5). Ablating deduplication produces mixed results: perfor- mance decreases on the synthetic aligned and synthetic unaligned datasets but improves on the synthetic orthogonal and AlpacaEval unaligned datasets. We hypothesize that repetition reinforces principles, especially when the model strongly opposes certain principles, as seen in AlpacaFvvl. Conversely, deduplication proves more effective when principles are less opposed model biases, as observed in the synthetic orthogonal scenario. We conclude that deduplication is likely gener- ally beneficial but that repetition may help overcome strong model biases in specific cases. Further details can be found in Appendix F.4.1.
No filtering and testing (Steps 4 and 5). Removing the filtering and testing steps has the most drastic impact, with performance dropping across all datasets. In particular, the annotators fail to outperform the random baseline in the unaligned experiments.
RELATED WORK
Our work focuses on deriving interpretable principles from human feedback data and using AI an- notators to evaluate those principles. We build on work related to learning from human feedback, biases in feedback, interpretable preference models, and AI annotators.
Learning from human feedback. Fine-tuning LLMs with human feedback has significantly con- tributed to the success of modern LLMs (Ouyang al., 2022; Stiennon et al., 2020). Typically, feedback is collected through pairwise comparisons of model outputs, training a reward model for fine-tuning, e.g. using reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Stiennon et al., 2020; Kaufmann et al., 2024) or direct preference optimization (DPO) (Rafailov al., 2023). Interpreting preference data is challenging since it generally lacks annotations of the underlying reasons and the reward model is often a black-box neural network, making it hard to interpret. Our work aims generate interpretable principles explaining the feedback data.
Biases in human feedback. Identifying biases in human feedback data is crucial, as unintended biases are common. For example. Hosking et al. (2024) note preference for assertiveness over truthfulness, while Wu & Aji (2023) highlight a bias towards grammatical correctness over factual accuracy. We discuss further prior work on annotator bias in Appendix B.2, also considering Al annotator bias. While these studies provide valuable insights, most methods for detecting biases rely on specialized feedback data collection, making them challenging to apply to pre-existing data. Our work generates interpretable principles from existing preference data, which can be inspected to detect biases and provide insights into the underlying preferences.
LLM-based description of dataset differences. Zhong et al. (2022; 2023) use LLMs to describe the difference between two text distributions in natural language. Dunlap et al. (2024b) a similar methodology for image distributions. Our work uses related approach but adapted the specific requirements and data structure of the pairwise preference domain.
Interpretable preference models. There has been growing interest in creating interpretable pref- erence models, aiding in understanding behaviour of Al systems. Go et al. (2024) create a compo- sitional preference model based on 13 fixed features, similar our constitutional principles. While they do not generate the constitution from data, they do create a regression model to weigh them, which would be a promising extension to our approach. Petridis al. (2024) propose a feedback- based constitution generation method relying on interactive tools, whereas our approach can be directly applied to standard preference datasets.
AI annotators. Due to the cost and time required for human feedback collection, Al annotators, or LLM-as-a-judge, have been proposed as a scalable alternative. Constitutional AI (Bai al., 2022b) uses LLMS with set of principles for feedback. Human preference-based fine-tuning enables LLMSs to generalize from very general principles, such as "do what's best for humanity" (Kundu et al., 2023), or even give feedback well-aligned with human preferences without any constitution (Zheng ct al., 2023). Our experiments show similar trends, where default LLM annotators align well with dataset annotations, even without constitution. Al annotators can also exhibit biases, further dis- cussed in Appendix B. AlpacaEval (Li et al., 2024c) offers a set of well-validated AI annotators.
Rule-based preference learning. Rule learning, aiming to develop descriptive predictive rules, has previously been applied to preference learning (de Sá et al., 2011). 1). A common technique for rule learning is to first generate a set of candidate rules and then measure each rule's support in dataset,
9

---

Published as a conference paper at ICLR 2025
Le. the fraction of data points that satisfy the rule (Fürnkranz et al., 2012; de Sá et al., 2011). Our algorithm follows this approach but, in contrast to more traditional rule learning, generates rules as natural language sentences. These rules, though more ambiguous and requiring AI annotators for interpretation, are expressive, interpretable, and easy for non-experts to edit". Liu et al. (2023) follow similar generate-and-test approach derive text quality criteria. They require absolute scores on a fixed set of aspects, however, while our method leverages pairwise comparisons covering many aspects simultaneously, as is commonly the case for publicly available preference datasets.
Concurrent work. Further, we would like to highlight concurrent works by Kostolansky (2024), Kostolansky & Manyika (2024), Shankar et al. (2024b) and Dunlap et al. (2024a) exploring ideas highly related to our work, perhaps highlighting the timeliness of this line of research. Whilst related, our work differs in terms of the precise choice of approach taken as well as the comprehen- siveness of our experiments. We provide detailed comparison in Appendix B.3.
6 LIMITATIONS
When interpreting our results, there are several limitations to our approach important to consider. Firstly, we do not show causality- - our generated principles correlate LLM annotators with the original annotations, but cannot validate if these principles were used by the original annota- tors. Multiple constitutions may explain the data equally well (as in the Rashomon effect (Breiman, 2001)). Nonetheless, an undesirable principle correlating with annotations is concerning, even if the principle was not intentionally used. For example, in the "aligned" variant of the AlpacaEval dataset, some generated constitutions include principles to prefer verbose redundant responses (see Appendix H.2.1). While this principle was likely not consciously followed by the original an- notators, its high support in the dataset may warrant further investigation and possible data cleaning. We further discuss this limitation in Appendix C. Secondly, constitutions represent lossy com- pression constitution of few principles is simplification of the poeision-mmking process underlying annotations. Some annotations may not be possible to reconstruct based on a simple constitution. This trade-off highlights the tension between interpretability and accuracy: concise, human-readable principles versus more complex representations. While ICAI could be adapted for richer constitutions to balance this trade-off, a black-box reward model may preferable when maximizing accuracy is critical. Finally, preferences closely aligned to LLMs are challenging to test. If an LLM annotator is already highly aligned with the datasct annotations, improving its per- formance with constitution is challenging. The constitutional reconstruction loss is most useful for evaluating constitutions orthogonal to or against the popular opinions internalized by the LLM. On already well-aligned models, the constitution may not improve performance, but it can still provide insights into the underlying preferences. Future work should focus on addressing these limitations, extending the capabilities of our approach, possibly using multi-modal models, and exploring new applications.
7 CONCLUSION
We have presented our work on the Inverse Constitutional AI (ICAI) problem: first defining the ICAI problem compressing preference data into a short list of natural language principles (or consti- tution). We then introduced an initial ICAI algorithm as a first approach to generate such constitu- tions. We demonstrated the effectiveness of our approach in experiments across four different types of datasets: (a) synthetic data to provide proof-of-conen oo-coneppt; (b) AlpacaEval data to show the ap- plicability to compress human-annotated data and the possibility of transferring constitutions across model families; (c) Chatbol Arena data to illustrate the generation of personal constitutions; and (d) PRISM data to demonstrate the ability to provide possible explanations for previously observed group preferences. We hope that our approach can improve both our understanding and the useful- ness of widely-used feedback data. Potential use cases of our interpretable and editable constitutions and corresponding meta data include: highlighting issues with datasets, creating interpretable alter- natives to black-box reward models, scaling human-annotan cases, and improving model customization via personal or group constitutions. We look forward to future research that explores these use cases in detail.
evaluation to new models and use
TThis also be seen as method for automatic prompt generation, as discussed in Appendix B.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We hope our work will have a positive societal impact by helping better understand preference data, already widely used for fine-tuning and evaluation of popular LLMs. We emphasize that our generated constitutions cannot claim to reconstruct an individual's 's true reasoning process. Similar to other interpretability methods, an ICAI constitution's principles may correlate with an individual's annotations but no causal relationship between the constitution and the annotator's reasoning can be proven. Further, our constitutions represent a notable compression of annotation considerations, which makes them highly interpretable but also means that they cannot reflect more fulti-faceted decision making processes.
Thus, constitutions should be interpreted cautiously when working with human annotators to avoid potential negative implications. This is especially important when attempting to explain demo- graphic preferences, as multiple possible explanations may correlate with the data and malicious actors could cherry-pick specific ones make discriminatory statements or reinforce prior beliefs. Similarly, the use of our approach for personalized L.MMs should also be considered carefully.
In general, emphasize that our method can only provide information about specific preference annotation datasets rather than annotators' reasoning processes more broadly. To mitigate the po- tential for misinterpretation of results, we include a corresponding warning in our algorithm imple- mentation that is shown to the user whenever a constitution is generated with ICAI.
When using ICAI for certain downstream use-cases, such as annotation scaling for training and evaluation or generating personal constitutions, there exists a risk of harmful bias amplification. If harmful biases exist in the original preference dataset, the ICAI constitution may pick up on these and propagate them downstream. This risk of amplifying biases is counterbalanced, however, by the ability to edit and inspect the generated principles. This ability potentially helps avoid amplification of unintended biases when using ICAI in downstream applications. This potential visibility of biases in ICAI distinguishes our method from other widely-used methods using preference data, such as black-box reward models or aggregate evaluation statistics, which make such biases more difficult to detect. We recommend users of our method to always take a close look if generated constitutions aligned with their own values and contain any potentially harmful biases before proceeding with downstream use-cases. Overall, we believe the potential for positive impacts outweighs possible negative impacts.
REPRODUCIBILITY STATEMENT
We make the source code for method and experiments publicly available at https:/ // github. com/ cdnfn/icatd Further, we attempted to add as many details in the paper as pos- sible, including all prompts in Appendix G as well as description of our synthetic data generation approach in Appendix I. For direct comparability, we also make numerical results available in Ap- pendix F.9.
ACKNOWLEDGEMENTS
We thank Benjamin Minixhofer, Max Bartolo. Tom Hosking, Hritik Bansal, the RECOG-AI team at the Leverhulme Centre for the Future of Intelligence (especially José Hernández-Oralll and Lorenzo Pacchiardi). Bill Marino and Jason Brown for their valuable feedback and discussions on early versions of this work. Further, we thank all reviewers for taking the time to read our work and provide helpful feedback. We also thank Anthropic for providing free research access to their models. Arduin Findeis was supported by a University of Cambridge School of Physical Sciences Award and by the UKRI Centre for Doctoral Training in Application of Artificial Intelligence the study of Environmental Risks (reference EP/S022961// 1/1). This publication was further supported by LMUexcellent, funded by the Federal Ministry of Education and Research (BMBF) and the Free State of Bavaria under the Excellence Strategy of the Federal Government and the Länder as well as by the Hightech Agenda Bavaria.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022a. URL http: arxiv org/ abs 2204. 05862. arXiv:2204.05862 [cs].
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini. Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hemandez, Dawn Drain, Deep Ganguli, Dustin Li. Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional Al: Harmlessness from AI Feedback, December 2022b. URL http: arxiv. org/ orgaabs/221.. 08073. 1.XXx:2211208077 [cs].
Hritik Bansal, John Dang, and Aditya Grover. Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models, February 2024. /ttp:/t:///rxx abs/2308. 15812. arXiv:2308. 15812 [es] version: 3.
Leo Breiman. Statistical modeling: The two cultures (with comments and rejoinder by the
author). Statistical science, 16(3):199-231, 2001. URL https://projectecc/// 1aannee-//sssue////
statisstiaal-Nooeeinng----- oootoolo wureswiireee th comments-andd-- 10. 00..124/sss/009922 short.
short. Publisher: Institute of Mathematical Statistics.
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the Judge? A Study on Judgement Biases, April 2024. URL http::/aaxiv..oe// abs/ 2402. 10669. arXiv:2402. 10669 [es].
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph Gonzalez, and Ion Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, March 2024.
URL http: /ttp:v/aavvvvv//. org/ abs/aa/3/33 2403. 04132. arXiv:2403.00 041330 [cs].
Cláudio Rebelo de Sá, Carlos Soares, Alípio Mário Jorge, Paulo Azevedo, and Joaquim Costa. Mining Association Rules for Label Ranking. In Advances in Knowledge Discovery and Data Mining. Springer, 2011. doi:
0111//9988364---111 20847-8 36.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani. Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, August 2023. URL wwtp::tttt:xii////eee/ abs abs/2305. 4887 arXiv:2305.1 133887777 [cs].
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-Controlled Al- pacaEval:: Simple Way to Debias Automatic Evaluators, April 2024. URL http://arxi erg/abs/2400 04475. arXiv:2404.04475 [cs, stat] version: 1.
Lisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, and Joseph E. Gonzalez. VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models, October 2024a.
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E. Gonzalez, and Serena Yeung-Levy. Describing Differences in Image Sets with Natural Language. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp- 2024b.
12

---

Published as a conference paper at ICLR 2025
Johannes Fürnkranz, Dragan Gamberger, and Nada Lavrač. Foundations of Rule Learning. Springer, 2012. doi: 1111077978-3-540-111 540-75197-7.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Mare Dymetman. Compo- sitional preference models for aligning LMs, March 2024. URL http::/axxxv..rr////ooaa./ 2310. 13011. arXiv:2310. 13011 [es].
Tom Hosking, Phil Blunsom, and Max Bartolo. Human Feedback is not Gold Standard, January 2024. URL http: / wrw.i.iv.v...... wog. abs 2309.16343 arXiv:2309. 16649999991699 [es].
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling Large Language Mod- els with Pairwise Ranking and Generative Fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). Association for Compu- tational Linguistics, 2023. doi: 10. 10.18663/v1/2023.2 ac66://
Timo Kaufmannn Paul Weng. Viktor Bengs, and Eyke Hüllermeier. A Survey of Reinforce- ment Learning from Human Feedback, April 2024. URL http:://aaxx httpiv..rrgabb//33311 14 425.. arXiv:2312.144 14922 [cs].
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The PRISM Aligmment Project: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural A Alignment of Large Language Models, April 2024. URL http: http://arxxe /tt.xvixvvv/..////////.aa/...av. org/ abs/ / 2404 10199. 9, arXiv:2404. 16019 [cs].
Tim Kostolansky and Julian Manyika. Iterative Interactive Inverse Constitutional AI, 2024. URL /tttps//ttimue 1m00000.mm000..mmmmmmmm 100..012000000000 git.ub. io/ stc.mentss/////////////o i3cai. i3fai.pdf..
Timothy H. Kostolansky. Inverse Constitutional AI. Thesis, Massachusetts Institute Technology, May 2024. URL https: http://spacceeeeeceeeee mit wwwe htt..tttttt.aaaaae//1/......1..
Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, Catherine Olsson, Cassie Evraets, Eli Tran-Johnson, Esin Durmus, Ethan Perez, Jackson Kernion, Jamie Kerr, Ka- mal Ndousse, Karina Nguyen, Nelson Elhage, Newton Cheng, Nicholas Schiefer, Nova Das- Sarma, Oliver Rausch, Robin Larson, Shannon Yang. Shauna Kravec, Timothy Telleen-Lawton. Thomas Liao, Tom Henighan. Tristan Hume. Zac Hatfield-Dode Joseph. Sam McCandlish, and Jared Kaplan. Specific versus General Principles for Constitutional AI, October 2023. URL http: / www.aaivvvvvooo..o.oo.oo wrw.orrg/abao... abs 2310 1377 98. 1tii:::331.139999 [cs].
Sören Mindermann, Nicholas
Junyi Li, Ninarch Mehrabi, Charith Peris, Palash Goyal, Kai-Wei Chang, Aram Galstyan, Richard Zemel, and Rahul Gupta. On the steerability of large language models toward data-driven per- sonas, April 2024a. URL
abs /211. 04978. arXiv:2311.044 .04978 [cs].
Tianle Li, Anastasios Angelopoulos, and Wei-Lin Chiang. Does style matter? Disentan- gling style and substance in Chatbot Arena, 2024b. URL https: /atp://mmmss..oom wwwps/org/bll//// 5024-08-280 21----88ttye--otttt-ee
LMSYS Org Blog (accessed 2024-11-14).
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An Automatic Evaluator of Instruction- following Models, May 2024c. URL https: htlp:///gitiee http//ggtttt.t//e/ wttp://tooo/tttttataaemaaa aalaca eval. original-date: 2023:05-25T09:3::222...
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-Rewadd Bag of Tricks for Reward Modeling in LLMs, 2024. arXiv: 2410.18451. preprint.
Yuxuan Liu. Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Calibrating -.aM- Evaluator, September 2023. URL http: //axxiv.org/////.o.............r abs//39////00// 13308. wrXiv:2309.13300 [cs].
13

---

Published as a conference paper at ICLR 2025
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, lan D. Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule Based Rewards for Language Model Safety. In Conference on Neural Information Processing Systems (NeurIPS), 2024. URL https: /ttps///ppeeeeeeeeeeee www.nreeeieeeeee.eeeeeee.....eee taree/forumma
forum? (ttppmmmdii//ttppe d=QVtwpT 5Dmg.
Roberto Navigli, Simone Conia, and Björ Ross. Biases in Large Language Models: Origins, Inventory, and Discussion. J. Data and Information Quality, 15(2):10:111::1,, 2023. doi: 10. 1145/3597307.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, March 2022. URL http: /tte:///rxxii arxiv. org/ abs 2203.222200 02155. urXiv:200.00155. [cs].
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Fa- vor Their Own Generations, April 2024. URL http: http//iiiivva/////vi wttp:/www.rrg/b/////0000//// 13076. arXiv:2404. 13076 [es].
Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. OffsetBias: Lever- aging Debiased Data for Tuning Evaluators. In Findings of the Association for Computational Linguistics: EMNLP (Findings EMNLP). Association for Computational Linguistics, 2024. URL https: / c
org. 2024. 2024. f Findings-eninn p.57.
Savvas Petridis, Benjamin D Wedin, James Wexler, Mahima Pushkama, Aaron Donsbach, Nitesh Goyal, Carrie J Cai, and Michael Terry. ConstitutionMaker: Interactively Critiquing Large Lan- guage Models by Converting Feedback into Principles. In Proceedings the International Con- ference on Intelligent User Interfaces (IUT). Association for Computing Machinery, 2024. doi: 00.11445/664043336655444
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model Secretly Reward Model, December 2023.
abs 230/ 18290. arXiv:2305.1880 18290
[cs].
Juan A. Rodriguez, Nicholas Botzer, David Vazquez, Christopher Pal, Marco Pedersoli, and Is- sam H. Laradji. IntentGPT: Few-Shot Intent Discovery with Large Language Models. ICLR 2024 Workshop on LLM Agents, 2024. URL https: httts:://peenrviee..r net http://wwwweeeeeo/ieeeeeoe forum?id= 2kvD2dC5rh.
Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin, J. D. Camfirescu-Pereine Harrison Chase, Will Fu-Hinthorn, Aditya G. Parameswaran, and Eugene Wu. SPADE: Syn- thesizing Data Quality Assertions for Large Language Model Pipelines, March 2024a. URL http:: / /wwxivivvv////////v//////////avv owwoor//abboo. htt//bbs/24001 2401 03038. arXiv:2401 003003 [cs].
Shreya Shankar, D. Zamfirescu-Perein Björ Hartmann, Aditya G. Parameswaran, and Ian Arawjo. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences, April 2024b. URL http://arxiv. wwtp://wwww../////////44/4/ 12272. arXiv:2404. 12272 [cs].
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bow- man, Newton Cheng, Esin Durmus, Zac Htafieel-Dodde Scott R. Johnston, Shauna Kravec, Tim- othy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards Understanding Sycophancy in Language Models, October 2023. URL http: wrwii.ivvv......aaaaaaaaaaaa...a www.iio.rrrabss.... 2210 13548. arXiv:2310. [cs, stat].
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Rad- ford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurlPS), volume 33. Curran Asso- ciates, Inc., 2020. URL https: httppst:pprcceeee....e///// http://pccceeeng..eeeeee wwwpeeeiiiiinnn.nniieeeeeeeeeeee wwweerrpss.ccpppppppeeeeeee 1 1 FFF88855556999 98d3ef www.eeedeeeeeeee.e.e 0b884484955 rast html.
14

---

Published as a conference paper at ICLR 2025
Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large Language Models are Incon- sistent and Biased Evaluators, May 2024. URL http: /ttp://a.....//aa///............ arXiv:2405.01724 [es].
Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):gge344, 2024. doi: (0.1093/pnaaneexuspggeeee
Amos Tversky and Daniel Kahneman. Judgment under Uncertainty: Heuristics and Biases. Science, 185(4157::1124-1333 1974. doi: 10.1 126/science. wtt.:iennee...157...... 85.4117.1112.
Zekun Moore Wang, Shawn Wang, Kang Zhu, Jiaheng Liu, Ke Xu, Jie Fu, Wangchunshu Zhou, and Wenhao Huang. PopAlign: Diversifying Contrasting Patterns for More Comprehensive Alignment, 2024. arXiv: 13785. preprint.
Minghao Wu and Alham Fikri Aji. Language Models, November 2023.
Style Over Substance: Evaluation Biases for Large URL
03025.
arXiv:2307.033 [cs].
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. A Critical Evaluation of Evaluations for Long-form Question Answering. In Anna Rogers, Jordan Boyd-Graher, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3225-3245, Toronto, Canada, July 2023. Association for Compu- tational Linguistics. doi: 10.18653/1/2023..ac///a acl-long..181.. URL https: wcl.athoooooo... org/2023. acl-aclllao long. 181.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Sto- ica. Judging LuMe with MT-Bench and Chatbot Arena, July 2023. URL http: //axxiv.org/aae..... 6 65685. arXiv:2306.00 200.023300006885 [es].
Ruiqi Zhong, Charlic Snell, Dan Klein, and Jacob Steinhardt. Describing Differences between Text Distributions with Natural Language. In Proceedings of the 39th International Conference Machine Learning, pp. 27099 20199922166666 16. PMLR, June 2022.
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal Driven Discovery of Distributional Differences via Language Descriptions. Advances in Neural Infor- mation Processing Systems, 36:40204-40237, December 2023.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Univer- sal and Transferable Adversarial Attacks on Aligned Language Models, December 2023. URL. http:/ // /arxivv.... org/ wrggabs/////oooooooo.ooooobeeerr aa/aabbs/2300 15043. arXiv:2307.15043 [cs].
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A DATASET DETAILS
We use four datasets in our experiments: synthetic data, AlpacaEval, Chatbot Arena, and PRISM. The synthetic dataset described in detail including the generation process in Appendix I, the other datasets are publicly available and described in the following.
AlpacaEval dataset of 648 human-anman annotated preferences, each consisting of pair of model outputs, with one preferred over the other. It is licensed under CC-BY-NC-4.0 and can be accessed https:: /tt:://huce gguggi ngnaanaaa ace..ac......oo co
wttpeeetttttttttaa/aaeaeee apcca paaeaaaaaaaallll...aaaa__aa To reduce com- putational costs and highlight the sample efficiency of our approach, we typically use a subset of 130 datapoints (65 train, 65 test), For larger-scale evaluation Appendix F.7, we use the full dataset of 648 datapoints (324 train, 324 test), which we refer to AlpacaEEall-arge.
Chatbot Arena Conversations a dataset of 33,000 preferences from the Chatbot Arena, used by the popular LMSYS leaderboard. Each datapoint consists of prompt and preference over pair of model outputs, both human genearted. It is licensed under CC-BY-NC-4.0 and can be accessed at https: htt///hgggggggeeeeee ct/ doooaaassssss s Imsys whttthttttto...eccco...oooc.o.ct aewaabbot_aree__cooeeerrre ions.
Chatbot Arena Kaggle is a dataset of 55,000 human- annann--n preferences, each consisting of a pair of model outputs, with preferred over the other. The dataset is similar to Chatbot Arena Con- versations, but contains more recent data. is licensed under CC BY-NC 4.0 and can be accessed at https: www. kaggle com/ http//ommptttt.... ons/ lmsys chat arena /atta.
PRISM is dataset of 8,011 human-annotated preferences, each consisting of a pair of model out- puts, with one preferred over the other. The dataset is licensed under CC-BY-400 and be accessed at htttt:: http:s::///gggggf///ee
co datasets http//ttttsssttaaaaccceeeeiiiiea
a liggnent..
Anthropic HH-RLHF is a collection of human-annotatee preference datasets Bai al. (2022a) focused on annotations preferring helpful and harmless outputs, with approx. 44,000 and 42,000 conversations respectively. The helpfulness data, similar other datasets, contains general model use-cases, with the more helpful of two responses selected. The harmless dataset is based on red-teaming prompts that explicitly aim to elicit harmful responses from models. The less harm- ful response is selected, The data is available under MIT license at https: //gi thub. com/ anthropica /hhhhhhhh.....
B EXTENDED RELATED AND CONCURRENT WORK
In addition to the related work discussed in the main body (Section 5), in broader sense ICAI can also viewed as a method for automated prompt generation. Another relevant area concerns biases exhibited by AI annotators, which are important due ICAI's reliance such annotators and its potential use as a tool for detecting these biases. We also give a more extensive discussion of concurrent work, in addition to the brief overview in the main body (Appendix B.3).
B.1 RELATION PROMPT GENERATION
LLM outputs can be guided by generating specific prompts. This relates closely to our work, where we create principles to steer outputs.
Manual adversarial prompt generation, or 'jailbreaking'. allows users to bypass safety constraints imposed during fine-tuning. This process can also be automated (Zou et al., 2023), generating adversarial prompts to attack wide range of models. Li et al. (2024a) propose virtual tokens to steer outputs towards specific viewpoints, using a dataset question responses to define these personas, unlike approach based pairwise comparisons and interpretable constitutions, Rodriguez et al. (2024) explore the use of LLMs to discover and classify user intent, which may help adapt model prompts.
16

---

Published as a conference paper at ICLR 2025
B.2 RELATION TO ANNOTATOR BIASES
Bansal et al. (2024) highlight that feedback methods influence biases, e.g., annotators focus more on accuracy in pairwise comparisons compared to rating feedback. Additionally, Sharma et al. (2023) observe a bias towards sycophantic outputs, where responses align with the user's 's beliefs rather than the truth.
Further, AI annotators can exhibit biases, partially overlapping with human biases (Chen et al., 2024), and inconsistencies in their judgements (Stureborg al., 2024). Examples include position bias (preferring the first output) (Zheng et al., 2023), verbosity bias (preferring longer responses) (Zheng et al., 2023), and self-enhancement or familiarity bias (preferring outputs similar to their own) (Panickssery et al., 2024; Stureborg et al., 2024). Their proposed mitigation measures include trying both orderings and tying if inconsistent, with further explorations later work (Dubois et al., 2024).
B.3 EXTENDED CONCURRENT WORK
Inverse Constitutional AI. Kostolansky (2024) introduced (and identically named) the problem of Inverse Constitutional Al (ICAI), concurrently with our work. Their problem formulation includes our first step, going from preferences to principles, but omits the reconstruction loss using a consti- tutional annotator. Their corresponding method also differs from ours, first clustering principles and then generating principles per cluster (instead of the other way around). Their results focus on recon- structing known clusters of preferences - requiring special preference datasets with known clusters and providing limited insight into the usefulness of each cluster's generated principles. Thus, their results cannot be directly compared to ours. Based on our ablation results, we remain sceptical that more focus on clustering would be helpful to create representative principles.
Iterative Inverse Constitutional AI CAD).. Kostolansky & Manyika (2024) also concurrently introduced alternative formulation named Iterative Inverse Constitutional AI (I*CAI). This prob- lem formulation focuses on optimising each principle's s ability to nudge an LLM towards correctly reconstructing annotations. This objective resembles our per-principle voting step, although being based on the conditional probability of correctly annotating a pair given a principle - rather than observed sampled annotations. This approach may offer a less noisy estimate than sampling but is not possible for all non-open AP[-based models. Perhaps due to this limitation their experiments use the Llama-2-7b model, relatively weak compared to larger state-of-the- gests, their method iterutively refines principles, differing quite a bit from our approach and requiring a "seed" constitution to initialize the process. As their implementation is at the time of writing not publicly available (as far as we are aware), we were unable to directly evaluate their method re to ours - but testing this method in our experimental settings would be an interesting future study to run.
models. As the name sug-
SPADE. Shankar et al. (2024b) adapt System for Prompt Analysis and Delta-Based Based Evaluation (SPADE) (Shankar et al., 2024a) to the pairwise annotation setting. SPADE is method that was originally designed for generating evaluation criteria based on differences ("deltas") between dif- ferent prompt versions during the development of LLM--based applications. The authors adapt this method to the pairwise setting and report it as a baseline, but limited information regarding the transfer of SPADE the pairwise setting makes it challenging for us make a meaningful compar- ison. Based on the original SPADE paper, this method likely uses a two stage process: (1) initially proposing criteria based on the preference data (prompt deltas originally) using an LLM, and then (2) testing each criterion and selecting a subset based on its coverage of test cases as well as false failure rate. We believe that adding similar selection process of rules, whilst adding complexity, would be an interesting extension of our method to explore in future work. Our initial principle selection process is intentionally simpler to avoid introducing more complexity. Regarding larger datasets and associated costs, we are uncertain whether and how they adapt their method to scale and whether they add any form of clustering. We were unable to find a public implementation of this pairwise SPADE version.
VibeCheck. Dunlap et al. (2024a) propose VibeCheck, a system that automatically detects qualita- tive differences ('vibe axes") between models based various datasets, including pairwise prefer- ences. The corresponding algorithm follows a similar overall approach as ours: initially proposing vibes (similar to our principles) using LLMSs and the validating the generated vibes using L.M--
17

---

Published as a conference paper at ICLR 2025
as-a-Judge systems in filtering step. Different to work, vibes are phrased as qualitative axes (e.g. from formal to friendly language) rather than principles explicitly instructing an LLM. Fur- ther, VibeCheck's algorithmic steps appear to primarily optimise towards vibes that well-defined (consistent across annotators) and enable effective separation between models, rather explicitly opti- mising for the ability accurately reconstruct preferences well possible. Whilst not explicitly optimising for annotation reconstruction during the vibe generation and selection, the authors do cre- a preference prediction model using a logistic regression on top of the vibe features - a model with use-case comparable to our constitutional annotators. The experimental results for VibeCheck similarly focus on model separation and do not include LLM--ss--Judde or reward model baselines as our work does. Thus, there are no directly comparable results of the logistic regression model to our approach. Overall, VibeCheck represents a promising approach that would be interesting to ap- ply directly to our Inverse Constitutional Al problem of reconstructing preferences, possibly adjust- ing the algorithm's 's internal optimisation objective more directly towards preference reconstruction. Notable algorithmic aspects from VibeCheck that would be potentially interesting to integrate in future ICAI algorithm versions would be the iterative refinement of vibes, self-consistency checks, and the multi-judge setup. It is worth noting that our ICAI method can also be used to quantify differences in model behaviour: using the same setup as in our bias detection example (Section 4.5) on data subsets where individual models win, we can measure models qualities based on our gen- erated principles. We would welcome future work that adapts VibeCheck to provide a more direct comparison ICAI, and vice versa.
PopAlign. Wang et al. (2024) introduce PopAlign, method that aims to improve model alignment and robustness by synthesizing more diverse response pairs as well as the corresponding preference data. Among the proposed diversification strategies, the elicitive contrast approach is particularly related to our work, as prompts the model to first derive principles for given instruction (based on overarching 'helpful and harmless' guidelines) and then use these principles to generate a response. While this dynamic principle derivation resembles our approach, PopAlign focuses on generating new feedback data, whereas ICAI aims to interpret existing datasets. As result, PopAlign does not incorporate responses and preferences into its principle generation process, nor does it aim to produce globally applicable principles evaluated across other data points. While the goals of the two methods differ, they are complementary: ICAI's data-driven constitutions could inform the cre- ation of more targeted contrastive prompts for PopAlign, while PopAlign's 's strategies for generating diverse responses could provide richer preference data for ICAI, enabling a more comprehensive understanding of human preferences.
Rule Based Rewards (RBR). Mu et al. (2024) introduce a method to train auxiliary 'rule-based re- 'propositions' binary statements about a response, such as "contains an apology" - into a linear combination that serves as a reward signal. These propositions are hand-crafted and used as features in the reward model rather than as direct prefer- ence indicators. While related, this approach does not aim to interpret or compress existing feedback datasets. Mu et al. (2024) further emphasizes detailed, interpretable rules over broader principles (e.g., "prefer the helpful response") to improve interpretability and steerability An ICAI-like ap- proach could complement rule-based methods by generating candidate propositions in a data-driven manner, potentially reducing manual engineering effort and enabling efficient fine-tuning of mod- els with modified constitutions. This highlights the potential for combining principled compression with explicit, rule-based rewards to enhance both interpretability and adaptability in safety-critical applications.
ward models' by composing natural-language
C FURTHER LIMITATION DISCUSSION
Non-uniqueness and variability of constitutions. An important limitation of our method is that well-performing constitution is rarely unique: annotators with multiple potentially quite different constitutions may achieve equivalent performance across dataset. Breiman (2001) more generally describes this non-uniqueness as the Rashomon effect, applying to many machine learning prob- lems. In the context of an interpretability framework, such as ours, this effect needs to be carefully considered whenever drawing conclusions. If an individual principle or constitution is able to help reconstruct a certain subset of annotation well, there may be many other principles or constitutions
"They can also be used as atoms in hand-crafted rules.
18

---

Published as a conference paper at ICLR 2025
that reconstruct. Thus, cannot claim any causal relationship: well- performinn principle does not mean that the annotator (AI or human) used that principle to create the annotations.
Nevertheless, in the context of bias detection, knowing that harmful principle works well to recon- struct a specific dataset can still be very useful. Even if we cannot know if the annotator willingly used such principle, we know the data be interpreted to encode this harmful principle. Down- stream applications (e.g., reward models) may make the same interpretation of the original dataset, and encode such principle (possibly in way that is hard detect such as millions of model parameters). Our method has the potential to highlight such potential harmful principles, enabling preference data users to mitigate these biases, for example by data filtering or collecting additional data.
On the other hand, the non-uniqueness of constitutions and principles means that our method is very unlikely to find all potential harmful biases in the context of bias detection. It is therefore critical avoid interpreting the lack of harmful biases our constitutions as an indication that no harmful biases are present in a given dataset. Given the diversity of potential harmful biases in commonly used datasets, our method should not be misunderstood to be able to find all harmful biases. However, the more prominent bias is, the more likely it is that a corresponding principle reliably generated and promoted to the constitution. Thus, ICAI serves as a valuable tool for detecting many but not all biases in preference datasets.
In the context of annotation scaling, well-performing constitutions can provide an effective way to scale small-scale human annotations to larger datasets. However, constitution, like the param- eters of alternative methods of annotation scaling (e.g. the PairRM baseline by Jiang et al. (2023) or LLM-as-a--uddee (Zheng al., 2023)), is not unique and there may be many different alternative models that achieve equivalent performance. A benefit with our approach is that these differences are more transparent and interpretable: more challenging to tell what the difference is between two sets of reward model parameters than between two constitutions.
D INCLUDED MODELS
Throughout experiments, we primarily use the following three models: OpenAl's (referred to as GPT-3.5-Turbo), gpt-4o-22 1000-2024-5-222 (referred to embedding model for clustering steps in the algo- rithm (across all experiments). Detailed model descriptions of these OpenAl models are available https: /ttp:::/ppattfoot form. openai com/ docs/ docs/modell models/ Certain experiments use additional models, these are described in the relevant experiments discussions.
as GPT-4o) and text -ewee-ien-dddgdaaaaaaaa---------
E COST ESTIMATES
In this section, we estimate the cost of reproducing the main experiments shown in this paper. All experiments were run using models via API access from OpenAl and Anthropic. Note that all estimates are subject variability due to provider pricing as well as inherent variability of the length (and thus cost) of model outputs.
Note that this cost estimate excludes the scale-up, ablation and PRISM experiments.
Synthetic experiments. The first set of experiments are the synthetic experiments, which are en- tirely run using GPT-3.5-Turbo. Per run (30 samples, constitution, annotation on same 30 samples) these experiments cost approximately 0.05$. Overall, we estimate it would cost 2.7$ to re-run all experiments shown (3 datasets x seeds).
AlpacaEval experiments. The second set of experiments are the AlpacaEval experiments, split into the main aligned/unaligned experiments as well as cross-model experiments. The main experiments cost approx. 2.20S per seed. Overall, estimate it would cost 26.40S re-run all of the main experiments (2 datasets x 6 seeds). Additionally, we estimate the cross-model (just annotation) experiments would cost 5.00S.
Chatbot Arena experiments. The third set of experiments are the Chatbot Arena experiments, split into the main aligned/unaligned experiments as well cross-model experiments. The main
19

---

Published as a conference paper at ICLR 2025
experiments cost approx 1.10S per seed. Overall, estimate it would cost 13.20$ to re-run all of the main experiments (2 datasets x seeds).
We estimate the remaining cost of experiments to be less than 5S. Overall, we thus estimate the total cost of re-running our experiments to approx. 52.30$ in API costs. Note that the overall cost for running experiments in the context of this project was about 3 times this amount (approx. 156.90S), due to failed runs and additional experimentation that did not fit into the scope of the paper.
F ADDITIONAL EXPERIMENT DETAILS
In this section, we provide additional details and results for experiments discussed in the main text.
F.1
DETAILED CHATBOT ARENA EXPERIMENTS
Annotations User
Annotations User B
Agreement (%)
100
I
Annotators
50
User constitution Default constitution
Example constitutions
1. Select the response that accurately identifies the pun misunderstanding Select response that avoids unnecessary repetition
1. Select the response that avoids abrupt or incomplete endings.
Select the response that maintains cansistent throughaut.
joke. Select the response that provides detailed and clear explanation.
Select the response that evokes stronger emotional engagement.
Figure 6: Case-study: Personal constitutions for anonymous Chatbot Arena users. Personal constitutions have the potential to help make LLM applications more helpful and customized to individual users' preferences in an interpretable way. We generate constitutions based on single user's annotations and check the constitutions' ability to help reconstruct the annotations of the same user and another user. For the two users, selected to have differing preferences, we observe that generated personal constitutions appear to work best for the original user and not transfer perfectly to another user. Note that this effect will vary for other users depending on how different users' preferences are. Plots show mean and standard deviation (6 seeds) using GPT-4o.
Experimental setup. We select two users exhibiting different preferences? from the Chatbot Arena dataset (with 9 and 12 preference annotations respectively) and generate a separate constitution with 3 principles for each. Note that due to the small number of samples, there no split between train- ing and test data. While this lack of separation may result in overfitting, in this case since our goal is not to develop generalizable model, but rather to explain the pref- erences within this specific dataset. To better detect the effect of user-specific principles, we adapt our generation and annotation prompts (Appendix G) to generate constitutions more specific to the individual users and follow the specific principles more closely rather than the model's 's priors.
consider it acceptable
Results. The results can be found in Figure 6 and are discussed in the main text (Section 4.3).
F.2 USE-CASE EXAMPLE: BIAS DETECTION
LLMs are known to exhibit biases, often originating from the data used to train them. This includes stylistic biases (Dubois et al., 2023; 2024), social or stereotypical biases (Navigli et al., 2023), and cultural biases (Tao et al., 2024). They can arise from the initial training data (Navigli et al., 2023) as well as the feedback data used during fine-tuning (Dubois et al., 2024; Tao et al., 2024). Our framework, ICAI, provides a mechanism to detect such biases in the preference data and offers actionable tools to analyse and mitigate them. This section presents examples of biases identified by
"The Chatbot Arena dataset filtered hy the authors avoid personally identifiable information (PIT).
20

---

Published as a conference paper at ICLR 2025
Table Evaluation of possibly biased principles on three datasets. Results on AlpacaEval (648 preferences), Chatbot Arena (5,115), and PRISM (7,490). Metrics shown are relevance (fraction of data points where the principle applies) and accuracy (fraction of relevant data points correctly reconstructed). Gray values indicate relevance for less than 50 preferences on the dataset. Hand-
picked, grouped and sorted illustrate presence of well- and less discussed biases.
Principle
AlpacaEv.
ChatbotAr.
PRISM
Select the response that...
Acc
Rel
Acc
Rel
Acc
Rel
Verbosity and style
overly lengthy and lacks brevity contains redundant information provides a numbered list format is more concise and structured uses more formal language feels more casual and friendly
52.2
80.4
57.1
97.1
57.7
96.9
44.7
7.3
39.7
12.8
35.1
3.0
73.4
12.2
62.3
45.5
71.6
17.7
47.4
98.6
42.9
95.5
42.7
94.6
56.8
6.8
53.4
16.0
61.9
5.4
43.8
67.3
43.3
61.2
41.1
74.5
Assertiveness
presents definitive stance without nuance
58.6
10.8
57.1
11.4
49.1
31.2
presents biased viewpoint without nuance
36.8
2.9
50.8
4.6
45.4
17.2
lacks nuance in political analysis
36.4
1.7
49.3
2.7
44.0
11.1
lacks neutrality in political matters presents one-sided argument avoids acknowledging complexity of the issue
55.6
1.4
49.6
2.7
38.9
9.8
32.0
3.9
53.4
5.2
46.1
19.5
28.1
4.9
37.9
10.2
36.9
25.8
promotes divisive political statements assigns sole blame without context does not consider personal preferences
50.0
0.9
52.2
1.8
39.2
6.6
80.0
0.8
53.9
1.5
41.4
6.2
100.0
0.5
42.4
1.7
50.5
6.8
Ambiguity and vagueness overly general and vague
30.4
15.7
26.8
9.6
26.1
23.1
emphasizes neutrality over providing information
50.0
2.2
40.8
10.2
58.0
46.2
presents ambiguous or non-committal language introduces ambiguity about the assistant's 's nature
100.0
0.2
30.4
1.3
46.8
13.0
100.0
0.2
33.0
3.5
44.0
14.0
our approach in the AlpacaEval and Chatbot Arena datasets and further discusses possible mitigation strategies and limitations.
Bias detection study. Our method can help detect biases by generating principles that expose the bias and measuring their performance. We run study to showcase this ability of ICAI using the following procedure: (1) We generate set candidate biases (principles) using ICAI on a training set of 1,000 preference pairs, 500 from PRISM and 500 from Chatbot Arena. Note that we the newer Chatbot Arena Kaggle dataset for this study (see Appendix A), differing from the main experiments. To ensure diverse set of principles, including potentially problematic only apply to a small subset of the data, we generate and test 400 principles (number of clusters) on this initial subset. (2) We manually select a subset of principles that we consider to be potential "problematic" biases. We focus on biases that perform well in terms of accuracy on the initial test set but also consider biases that non-relevant for the majority of the training and thus have less reliable accuracy measurement (as that is based on the number of relevant data points). (3) We then re-run the principle testing step of our pipeline on a much larger test set of over 13k preference pairs, consisting of 7,490 preferences from PRISM, 5,115 from Chatbot Arena, and the entire dataset of 648 cross-annotated AlpacaEval preferences. To run such a large study cost- effectively, each component of our algorithm uses GPT-4o-mini (rather than GPT-4o). The results are shown Table
that may
Verbosity bias. One of the most well-known biases in preference data is verbosity bias, where longer responses are preferred. While both humans and Al annotators exhibit this bias (Dubois et al., 2023: Chen et al., 2024), AI annotators important aspects, leading to problematic artefacts in evaluation and training of language models (Dubois et al., 2024). We observe strong bias towards longer responses both Chatbot Arena and
place excessive focus this trait at the cost of other
21

---

Published as a conference paper at ICLR 2025
PRISM", with principles preferring responses that are "overly lengthy and [lack] brevity" achieving notably above random accuracy (acc. of 57.1 and 57.7% respectively) on a significant portion of the dataset (rel. of 97.1 and 96.9% respectively). Note that the bias is less pronounced the A AlpacaEval dataset in our experiments (acc. of 52.0% and rel. of 80.4%) despite prior work noting preference for longer responses in that dataset (Dubois et al., 2024). Even though apparently less pronounced than in other datasets, the AlpacaEval verbosity bias is also reflected in the constitutions generated for the aligned AlpacaEval dataset (see Appendix H.2.1 for a sample), where principles favouring verbose or redundant responses appear in out of 6 seeds. The PRISM and Chatbot Arena experiments in the main paper focus on specific subsets of the dataset, however, and the constitutions generated for those subsets are not directly comparable to the data in Table 2. For instance, contrary to the general trend visible Table 2, the constitutions generated for Group A of the PRISM dataset seem to favour conciseness over redundancy, while Group B's constitutions are more in line with the general trend (see Appendix H.4). This outcome further validates our framework's 's capability to detect biases that are dataset-specific.
List bias. A similarly well-known bias is the preference for structured responses, Markdown syntax, and lists in particular (Li et al., 2024b). We see this style bias reflected in Table 2, with the rule favouring "the response that provides a numbered list format" " achieving high accuracy across all datasets, especially AlpacaEval (73%) and PRISM (72%), although this principle is far less broadly applicable than the ones centred on verbosity (rel. of 12.2 and 17.7% respectively).
Assertiveness hias. Finally, we observe preference for assertiveness (as discussed by Hosking et al. (2024)), with principles favouring "the response that presents definite stance without nu- ance" and similar (see Table 2) performing well on AlpacaEval and Chatbot Arena. This bias is notably common in political contexts (compare Table 2), giving cause for concern. It is quite pos- sible, however, that preferences supporting this bias were given counteract the language model's tendency to over- -uelluy or hedge its statements (related to the next paragraph), so it is important to consider the context in which this bias is observed.
Ambiguity or vagueness. In addition to these well-known verbosity and style biases, we also ob- serve less extensively discussed biases, commonly centring around ambiguity and vagueness: the principle "Select the response that emphasizes neutrality over providing information" " performs well on PRISM but has lower than random accuracy on Chatbot Arena data, indicating this rule is not followed, on average, by Chatbot Arena annotations. Neutrality in the response appears to be ap- pears to be actively selected against, on average, in the Chatbot Arena subset. The Chatbot Arena annotations further do not appear to, on average, reject responses that "promote divisive political statements", ", unlike PRISM annotations. Further, we observe that Chatbot Arena annotations ac- tively select against responses that "acknowledge limitation in available information".
The results above indicate that ICAI is able to both find well-deseribed and less widely discussed biases in pairwise preference data. Once biases are identified, ICAI offers actionable strategies for mitigation. Possible avenues include (1) synthesizing new preferences with a modi- fied constitution that avoids the bias, and (2) curating the training dataset by filtering or balancing preferences to reduce bias.
Mitigation.
The second approach leverages ICAI's ability to measure the support of each principle within the dataset, identifying which preferences align with the bias. By removing or rebalancing these pref- erences, biases can potentially be mitigated. This approach also allows for a more detailed analysis of the data, making it possible to develop tailored strategies, such as refining the preference col- lection process. Going beyond removal, our framework can also be used to identify and promote preference pairs that counteract the bias, i.e., agree with an opposing principle. For example, despite the verbosity bias in AlpacaEval, one generated constitution includes a principle favouring concise responses, indicating that the dataset contains a substantial portion of counteracting preferences. ICAI thus serves as a versatile tool for dataset filtering, balaneing, and curation, which have proven effective in other contexts (Liu et al., 2024; Park et al., 2024). We are excited for future work to explore these mitigation strategies in more detail.
Limitations. Our methods ability detect biases depends on two factors: the diversity of candidate principles and reliability of the filtering mechanism. While stylistic biases, such as verbosity and list
*hii is despite PRISM attempting to alleviate verbosity bias by instructing the LIM to produce shorter responses (Kirk al., 2024).
22

---

Published as a conference paper at ICLR 2025
preferences, are straightforward to detect, social and cultural biases can be more challenging, since they are often expressed in subtle ways. These biases, including those related to gender or minority representation, critical to address. However, in the datasets analysed, our constitutions do not show direct evidence of such biases, likely due to the limited dataset size and the constitutions' focus broadly applicable principles. Unlike stylistic biases, social and cultural biases often affect smaller subsets of data and may coincide with alternative explanations for preferences.
Detecting these subtler biases requires expanding the dataset, increasing the number of candidate principles generated per preference, and increasing the scope of the analysis beyond the top princi- ples those that, while not universally applicable, exhibit strong predictive power for specific data subsets. A thorough investigation of social and cultural biases using ICAI represents promising direction for future research.
F.3 USE-CASE EXAMPLE: ANNOTATION SCALING ON HELPFUL/HARMLESS DATA
Collecting human annotations for specific purposes can be expensive and consuminn. We demonstrate the use of ICAI to scale up preference annotations 10x based on small set of 100 initial ground-truth annotations to 1000 new response pairs. In particular, we consider the use of ICAI to scale up harmlessness and helpfulness annotations, using the Anthropic HH-RLHF dataset by Bai et al. (2022a). More information about the dataset is available in Appendix A.
Experimental setup. We randomly sample two training sets of 100 data points each from sep- arate helpful and harmless datasets Anthropic HHH-LHFF, The helpful and harmless datasets contain human annotations that prefer more helpful and harmless responses, respectively. We sim- ilarly sample two separate test sets of 1,000 data points from each dataset. We then apply ICAI on each training set to create two separate constitutions, one harmless and one helpful, and test the ability of an LLM to use these constitutions to reconstruct each dataset. We use GPT--4omini (gpt-40-ml n - 2--22022--722 fault amnotations. We slightly adjust the principle proposal and voting prompts in the ICAI algorithm accommodate the long multi-turn nature of the Anthropic HH preference dataset,"
for all parts of the ICAI algorithm, and the constitutional and de-
Results. The results are shown and discussed in Figure 7. Results shown are mean and standard deviation over seeds of the entire pipeline.
F.4 ABLATION DETAILS
We provide detailed numerical results for and discussions of the ablation experiments introduced in Section 4.6. Each experiment is averaged over six seeds, with annotator agreement and confidence intervals shown in Table 3. Below are the specifics of each ablation:
Simplified principle generation (Step 1). In this ablation, we generate principles using single neutral prompt instead of multiple prompts. As shown in Table 3, this leads to a reduction in an- notator agreement across all datasets, with the largest drop in the synthetic unaligned dataset. This confirms our hypothesis that GPT-3.5-Turbo struggles with generating both positive and negative principles from single prompt.
Principle generation with multiple preferences (Step 1) We test the effect of prompting with multiple preferences simultaneously to generate the principles Step 1. By default, only a single preference is used in the prompt. In these experiments, we give the model 5 preferences simulta- neously, and then ask the model to generate 10 corresponding principles. We randomly group all preferences into groups of size 5, that are then used to prompt the model in Step 1. We observe mixed results: for some scenarios (Synth aligned and AlpacaEval unaligned) the model improves performance whereas for the others this configuration decreases performance. To maximize prin- ciple diversity, it may be useful to combine both single and multi multt-preeerence principle generation.
9 All data sampled from the train jsonl. .z files in the Salpful-basschhamllss-see subdirectories of the data repository.
In particular, we add additional separators between the responses ("---") and explicitly prompt the model focus on the last conversation turn.
23

---

Published as a conference paper at ICLR 2025
Harmless annotations
Helpful annotations
Agreement (%)
60
40
Annotators
Harmless const.
20
Default
Helpful const.
Example constitutions
1. Select response that avoids illegal or unethical actions.
1. Select response that avoids vegue references and limitations.
2. Select the response that lacks clarity in addressing the request.
2. Select the response that engages with the user's inquiry.
response that avoids any empathy or kindness, I...]
response that includes helpful tone in conclusion. [...]
Figure 7: Use-case example: scaling annotations 10x with ICAI on helpful/harmless preference data. We observe that our constitutional annotators are able to outperform the default baseline annotator each dataset. Qualitatively, each dataset's responses are clearly distinguishable from each other. For example, all harmless constitutions contain principles to avoid promoting illegal actions whilst the helpful ones often foeus helpful tone and user engagement. Quantitatively, annotators with harmless constitutions do not appear to transfer well to helpful data and vice versa. Our experiments closely replicate findings by Bai al. (2022a), indicating that these two datasets encode anti-correlated objectives: fully harmless response should refuse to be helpful for harmful actions.
Table 3: Results for ablation of different pipeline components. The table shows the mean agreement and standard deviation over 6 seeds. Each configuration tested over four different datasets from Section 4, based on the synthetic (Synth) and AlpacaEival (AE) datasets. The best result per row is highlighted in bold.
Single Prine. (S1)
Multi-Pref. (S1)
No De-Dup. (S2 & S3+)
No Test/Filter (S4 & S5)
Name
Original
Synth Orth
86.7 + 8.4
82.2 + 4.6
83.3 + 11.2
80.0 + 17.0
51.7 + 13.6
(GPT-3.5-Turbo,
Synth Aligned
92.2 + 6.9
89.4 + 11.6
98.3 + 4.1
93.9 + 8.8
69.4 + 30.9
(GPT-3.5-Turhhe
Synth Unaligned
84.4 + 9.8
62.8 31.0
69.4 19.8
83.9 8.3
31.1 = 18.3
(GPT-3.5-Turbo)
AE Unaligned
66.4 + 7.7
65.9 2.8
72.1 2.0
70.0 + 2.9
40.8 11.3
(GPT-4o)
No deduplication (Steps 2, 3, and 5). We ablate deduplication by testing all generated principles without removing duplicates. The results, shown Table 3, are mixed: performance decreases on the synthetic aligned and synthetic unaligned datasets but improves on the synthetic orthogonal and AlpacaEval unaligned datasets. This suggests that repetition may help reinforee principles in datasets where the model holds strong prior biases against certain principles, especially in the un- aligned Al AlpacaFval case, while diverse principles are more beneficial in orthogonal datasets. These results are discussed in more detail in Appendix F.4.1.
No filtering and testing (Steps 4 and 5). In this ablation, we replace the filtering and testing steps with random sampling from the clustered principles. As expected, this results in a significant perfor- mance drop across all datasets, particularly on the unaligned datasets, where the annotators perform worse than the random baseline.
F.4.1 DEDUPLICATION ABLATION
Deduplication is key step in pipeline to reduce redundancy and optimise the of limited preference capacity. We apply deduplication at three stages: clustering principles in Step 2, sampling one per cluster in Step 3, and deduplicating top principles after filtering in Step 5.
24

---

Published as a conference paper at ICLR 2025
Ablating deduplication, by testing all generated principles without filtering duplicates, yields mixed results. Performance decreases on the synthetic aligned and synthetic unaligned datasets but im- proves the synthetic orthogonal and AlpacaEval unaligned datasets. These findings suggest that deduplication helps when principles are less opposed to model biases, such as in the synthetic or- thogonal dataset, where diverse principles are more beneficial.
However, in cases where the model has strong prior biases, such as the unaligned AlpacaEval dataset, repetition of principles can reinforce the desired behaviour. We hypothesize that the repeated pre- sentation of the same principles may overcome the model's resistance, helping it internalize the preferred constitution more effectively. This is particularly effective in the unaligned scenarios, where only few principles opposed to the model's biases may already elicit an "opposite persona' that acts opposite to the model's initial biases even on comparisons not explicitly covered by the principles. This effect may reduce the negative impact of duplication in these cases, as it is less im- portant to populate the constitution with diverse principles covering many aspects of the preference data.
In contrast, the synthetic orthogonal dataset benefits from deduplication since the true underlying principles are not in conflict with the model's bias and are less correlated from the model's perspec- tive (compare Figure 8). In this case, therefore, deduplication helps ensure a broader coverage of the underlying principles, leading to improved performance.
Despite the mixed results, we generally recommend deduplication for most use cases, as the benefits in terms of computational cost savings and improved interpretabiliiy typically outweigh the perfor- mance trade-offs. Nonetheless, scenarios like AlpacaEval suggest that selective repetition, based on principle importance or the model's initial aversion them, could be an interesting direction for future research.
F.5 HYPERPARAMETE SENSITIVITY
Our method introduces an important hyperparameter n that determines the number of principles in the constitution. The parameter may be seen as determining regularisation in our algorithm: small n may be considered highly regularised, limiting the amount of overfitting to the data pos- sible. A large n enables including more fine-grained principles that only apply smaller subset of examples. Note that, depending on the use case, overfitting to the training data is not necessar- ily problem (e.g., for data interpretability). In this section, we present additional experiments on synthetic data to investigate the impact of this hyperparameter.
n=1
n=3 3
n=5
n=10
with original data
Agreement (%)
100
50
Figure 8: Results when varying number n of principles in constitution orthogonal synthetic data. Whilst there is clear improvement noticeable from to 3, and 3 to 5, we observe that there appear be diminishing returns for values higher than 5. Note that the number of underlying principles is three, thus it may not be surprising that n = does not work well. For n = 3, the algorithm needs to create three different principles that match the underlying three rules which may be eтror prone. From n = 5 onwards it appears to robustly find corresponding principles for the underlying three rules. Thus, we use n = 5 in our experiments. Note that for further datasets additional experimentation may be important
the optimal value also depends on the annotator model's 's capacity to deal with multiple principles simultaneously. Experiments use HPT-3.5-Turbbl reported values and error bars are mean and standard deviation over six random seeds.
25

---

Published as a conference paper at ICLR 2025
F.6 CONSTITUTION TRANSFERABILITY
We investigate the transferability of constitutions across different model families. Shown in Figure 9, the results indicate that the constitution generated by GPT-4o transfers well to models from the Claude family, Claude-3-Opus and Claude-3-Haikee
Unaligned
Agreement (%)
60
40
Annotators
20
Constitutional
GPT-4o
Claude-3-Opus
Mlude-3-Haike
Figure 9: Transferability of constitutions: results of transferring a GPT-4o generated consti- tution to other model family (Claude). We use the highest-performing unaligned constitution on the training set, from experiments shown in the unaligned plot in Figure 4. We test two additional models from the Claude model family, Claude-3-Opus and Claude-3-Haiku. Both are able use GPT-4o's generated constitution to reconstruct the test set annotations effectively, albeit to lower standard than GPT-4o. Plots show mean and standard deviation using 4 seeds per annotator, all with the same constitution.
F.7 RESULTS ON LARGE DATASETS
Figure 10 and Table 10 show the results of using the entire 648 samples in the cross-annotated AlpacaEval dataset in our experiments (AlpacaEvall-Lrre, see Appendix A), instead of 130 samples used in the original experiments. We use 324 samples for training and 324 for testing.
Original (65 samples)
Large (324 samples)
Agreement (%)
50
Anhotators
Default
Constitut.
GPT-3.5-Turbo
GPT-40
GPT- Turrbo
GPT-4o
constitutions
Select response provides an overly simplistic or misccading answer.
1. Select the response that clear step-bye by-step instructions.
Example
2. Select the response that lacks depth in analysis.
2. Select the response that lacks clear narrative structure. response
the response contains incorrect nonsensical logic. [...]
specific
points. [...]
Figure 10: Scaling up experiments the AlpacaEval unaligned dataset. We scale original experiment up 5x to the entire 648 samples the cross-annotated AlpacaFval dataset, instead of the 130 samples used in the original experiments. As before split the dataset half to obtain test and training set, using 324 samples for training (generating the constitution) and 324 for testing. We also provide the original results for comparison.
F.8 EXTENDED BASELINE DISCUSSION
We compare method against several baselines, described in detail below, All results discussions are based on Tables and 10.
Default These baseline annotators vary depending on the model used to run them (GPT-3.5- Turbo GPT--4o) and are directly based annotator configurations leading in their
26

---

Published as a conference paper at ICLR 2025
model class in the AlpacaEval (AE) evaluator leaderboard," !1 chatopt..fn (used with GPT-3,5-Turbo) and alpaca.eval.gue wtt.aa..............aaaa (used with GPT-4o). We only make small tweaks to the prompts to fit our data format (described in Appendix Ap- pendix G.4) and update the original GPT-o--Turbo model with the newer GPT-4o model for the latter configuration (as GPP-fo--peccfiic configuration was available). We made a careful trade-off between reported cost (less than 6$/1000k annotations) and perfor- mance (best with their model, at their price points) for our baselines. In particular, is reported to perform (68.1%) 16) close to the top con-
www.ccaeelllgp..ttueeeeee figuration (alppaca.eval..ptt...
71.0%) discussed above. It is not tailored to the datasets used in our experiments. Consequently, its performance is expected to be strong on datasets aligned with the model's training data and weaker on unaligned datasets. To account for this, we include a flipped version of this baseline, where predicted preference labels are inverted.
Results. As expected, we see that the baselines perform strongly on datasets aligned with the base model's learned preferences, but poorly on other datasets. This is an inherent limitation of such an annotator, as it has no ability to adapt to new data.
Default (flipped) This variant of the Default baseline uses the same AlpacaEvval prompts but flips the predicted preference labels. Note that such manual adjustment works only in limited scenarios such as our unaligned datasets; the default annotator cannot generally adapt to dataset-specific characteristics.
Results. Similar to the Default annotator, this baseline performs well on a restricted se- lection of datasets just the inverse of the Default annotator (the unaligned datasets as opposed to the aligned ones).
PopAlign
This baseline is adapted from the PopAlign method developed by Wang et al. (2024). We modify this method, originally created for data generation, to the pairwise preference anno- tation setting. Similar to our method, PopAlign generates principles to annotate response pairs. However, instead of generating a fixed constitution representing an entire dataset (as in our method), PopAlign dynamically generates principles for each response pair and then annotates the pair according to the same principles. The detailed prompt and how we adapt the method is included in Appendix G.5. Many of the principles PopAlign generates as part of this process are qualitatively similar to those found in ICAI constitutions, for example: "A good response should be accurate, relevant, and provide clear and practical informa- tion". We make this PopAlign-based annotator available as an AlpacaEval annotator config as part of our public package.' 12
Results. While this baseline, similar to ICAI, generates principles, these principles are generated on-the-fly and without access to training data with known annotator preferences. Hence, the principles generated by this baseline are always in-line with its own learned preferences and cannot adapt to new dataset, resulting in performance comparable to the Default annotator (good on aligned, bad on unaligned datasets).
PairRM
This baseline uses the Pairwise Reward Model (PairRM) 13 by Jiang et al. (2023), a black- box pairwise preference model with 400 million parameters. It accepts pair of output candidates and an instruction as input, jointly encoding them to produce scores that reflect relative quality. Unlike the other baselines and our method, PairRM provides determinis- tic scores rather than relying on language model sampling. Therefore, we report results for single seed without standard deviation or extrema. Similar to the Default annotator, PairRM is not customized to the datasets in our experiments, potentially leading to weaker performance on unaligned datasets. To evaluate sample efficiency and fairness, we also include a tuned version of PairRM that is fine-tuned on the training data.
Results. Since this version of the reward model is not fine-tuned, it has no ability to adapt to a dataset (similar to the Default and Default (flipped) annotators). Its relative performance mirrors the Default annotator, therefore, performing well (on-par with the Default annota- tor) on aligned datasets and poorly on others. The reward model's performance exceeds
"See https:///ggituubccoooooo
walluators_cont
12 Link hidden for anonymous submission
Availaabe at https: / /hwgginggggggg face co. mm- biwebeeeee/aaaa FRM, our experiments use revision Sb880cc73777 (a37777767755333eeeeea 69bcb5be01
27

---

Published as a conference paper at ICLR 2025
the Default annotator's ' on the synthetic-orthogome dataset, which is likely due to chance preference on the data chosen to be orthogonal to the Default annotator's preferences.
PairRM (tuned) This baseline uses the PairRM model fine-tuned the training data prior to test- ing. Fine-tuning performed for up to five additional epochs and a batch size of 1, with validation accuracy used to select the best model. The training data matches the data used to generate the constitution in our method, with a fractioo of the training data (10 for Al- pacaEval, 32 for AlpacaEval Large) reserved for validation. For synthetic data experiments, the model is tested on the same data used for fine-tuning, as separate test or validation sets are unavailable for this small dataset. This leads to overfitting and affects generalizabil- ity, which is less critical for our method, where interpretability is the primary focus and quantitative results are secondary. For fairness, the same (non-split) procedure is applied to PairRM. However, the reported performance on synthetic datasets likely overestimates the model's capability on unseen data.
Results. PairRM is the only baseline that can, like ICAL, use training data to adapt to new dataset. This reflected in its reconstruction ability, generally exceeding the one of the non-fine-urn fine-tuned version, especially on unaligned and orthogonal datasets. We observe that this ability to adapt limited, however, as reflected in the model's sub-par performance on the AlpacaEval unaligned setting. This is likely due to the model's 's sensitivity to hyperpa- rameters as well as the limited training data, which is completely opposed to the model's (much larger) pretraining data.
F.9 NUMERICAL RESULTS
We provide full numerical results in table format for our experiments. Tables 4 and 5 show the numerical results for the core experiments on the synthetic and AlpacaEval datasets, respectively, featuring extended set of baselines. Further, Tables 6 and 7 show the numerical results for the personalized experiments on the Chatbot Arena and PRISM datasets, Table shows the results for the cross-model experiments, and Table 9 shows the results for the hyperparameter sensitivity experiments. Finally, Table 10 shows the results for the scaling experiments on AlpacaEval data. Experiments that were already discussed using table previously are not repeated here.
Table 4: Results for experiments synthetic data. Averaged over 6 random seeds.
Dataset
Model
Annotator
Mean
Std
Min
Max
Orthogonal
GPT-3.5 Turbo
Constitutional
86.67%
8.43
73.33 %
96.67%
Default
37.78%
2.72
33.33%
40.00%
Default (flipped)
62.22%
1.72
60.00%
63.33%
PopAlign
38.89%
1.72
36.67%
40.00%
PairRM
73.33%
PairRM (tuned)
100.00
Aligned
GPT-3.5 Turbo
Constitutional
92.22%
6.89
83.33%
100.00 %
Default
100.00% %
0.00
100.00
100.00
Default (flipped)
0.00%
0.00
0.00%
0.00%
PopAlign
100.00 %
0.00
100.00
100.00 99
PairRM
100.00 %
PairRM (tuned)
100.00 %
28 000000000
Unaligned
GPT-3.5 Turbo
Constitutional
84.44%
9.81
73.33%
%.0%
00000000000000000000000000000000
200000
Default
0.00%
0.00
Default (fliped)
100.00 %
0.00
100.00 %
100000
100.00%
38 %
200000
PopAllgn
0.00%
0.00
PairRM
0.00%
PairRM (tuned)
100.00
28

---

Published as a conference paper at ICLR 2025
Table 5: Results for experiments AlpacaEval data (65 samples). Averaged 6 random seeds.
Dataset
Model
Annotator
Mean
Std
Min
Max
Aligned
GPT-3.5-Turbo
Constitutional
67.44%
3.43
63.08%
72.31%
Default
64.87%
1.16
63.08%
66.15%
Default (flipped)
33.08%
1.29
32.31%
35.38%
PopAlign
67.18%
0.79
66.15%
67.69%
GPT-4o
Constitutional
68.46%
3.19
63.08%
72.31%
Default
72.56 %
1.16
70.77% %
73.85 %
Default (flipped)
27.95%
1.80
26.15%
30.77%
PopAAlggn
69.05%
1.33
68.25%
71.43%
PairRM
64.62%
PairRM (tuned)
69.23%
Unaligned
GPT-3.5-Turbo
Constitutional
39.49%
3.32
35.38%
44.62%
Default
35.90%
1.26
33.85%
36.92%
Default (flipped)
66.67%
1.26
64.62%
67.69%
PopAlig
33.85%
2.57
30.77%
36.92%
GPT-4o
Constitutional
66.41%
7.69
53.85%
72.31%
Default
26.92%
1.61
24.62%
29.23%
Default (flipped)
72.31 %
1.69
70.77% %
73.85%
Plign
30.24%
2.10
26.98%
32.26%
PairRM
35.38%
PairRM (tuned)
38.46%
Table Results for cross-user experiments on Chathot Arena data. Averaged over random seeds.
Dataset
Model
Annotator
Mean
Std
Min
Max
Annotations User A
GPT-4o
User A constitution
93.06%
3.40
91.67% %
100.00% %
Default
83.33%
0.00
83.33%
83.33%
User B constitution
83.33%
5.27
75.00%
91.67%
Annotations User B
GPT-4o
User A constitution
79.63%
10.92
66.67%
88.89%
Default
88.89%
0.00
88.89 %
88.89%
User B constitution
94.44%
6.09
88.89 %
100.00 %
Table 7: Results for cross-group experiments on PRISM data. Averaged over 6 random seeds.
Dataset
Model
Annotator
Mean
Std
Min
Max
Annotations Group A
GPT-44o
Group A constitution Default
77.22% %
3.28
73.33
83.33 %
58.33%
1.83
56.67%
60.00%
Group B constitution
50.00%
2.98
46.67%
53.33%
Annotations Group B
GPT-4o
Group A constitution
37.92%
2.04
35.00%
41.25%
Default
57.08%
1.02
56.25%
58.75%
Group B constitution
61.46%
4.50
55.00%
67.50%
29

---

Published as a conference paper at ICLR 2025
Table 8: Results for cross-model experiments Alpacalival data. Averaged over 4 random seeds.
Dataset
Model
Annotator
Mean
Std
Min
Max
Unaligned
GPT-4o
Default
26.15%
1.26
24.62%
27.69%
Constitutional
70.00% %
0.89
69.23%
70.77%
Claude-3-Opus
Default
24.23%
0.77
23.08%
24.62%
Constitutional
58.85%
0.77
58.46%
60.00%
Claude-3-Haike
Default
36.92%
0.00
36.92%
36.92%
Constitutional
59.65%
0.00
59.65%
59.65%
Table 9: Results for the sensitivity study parameter (rules per constitution) synthetic data. Averaged over 6 random seeds.
Dataset
Model
Annotator
Mean
Std
Min
Max
Unaligned
GPT-3.5 Turbo
Constitutional (n=1)
52.22%
4.55
43.33%
56.67%
Constitutional (n=3)
75.00%
14.10
63.33%
100.00%
Constitutional (n=5)
86.67%
8.43
73.33%
96.67%
Constitutional (n=10)
90.00%
10.11
70.00%
96.67%
Table Results scaling experiments on unaligned AlpacaEval data (from 65 to 324 samples in test set). Averaged over random seeds.
Dataset
Model
Annotator
Mean
Std
Min
Max
Original (65 samples) GPT-3.5-Turbo
Default
35.90%
1.26
33.85%
36.92%
Constitutional
39.49%
3.32
35.38%
44.62%
GPT-4o
Default
26.92%
1.61
24.62%
29.23%
Constitutional
66.41 %
7.69
53.85%
72.31 %
PairRM
37.35%
PairRM (tune)
46.60%
Large (324 samples)
GPT-3.5-Turbo
Default
45.83%
0.91
45.06%
46.91%
Constitutional
54.20%
1.56
52.01%
55.73%
GPT-4o
Default
33.80%
0.58
33.02%
34.57%
Constitutional
61.47%
1.29
59.88% %
62.96%
PairRM
37.35%
PairRM (tune)
50.00%
30

---

Published as a conference paper at ICLR 2025
G PROMPTS
Prompts generally separated into two messages, system message and message. We the following format for all prompts (based on AlpacaEval's formatting): lim_ start > and imm end denote the start and end of message, followed by the message type (system user) and the content.
G.1 PRINCIPLE GENERATION
Unless otherwise specified, principles are generated with the following two generation prompts. We process each data point with both prompts to encourage the generation of diverse set of principles that may both select for positive output traits (e.g. more helpful) and negative output traits (e.g. off- topic). Initial experiments indicated that it can be difficult to generate such diverse set of possible principles with a single prompt, thus we use multiple (two) prompts by default. An exception is the Chatbot Arena dataset, where specific principles, to better capture individual differences between users.
single prompt that places increased emphasis highly
Listing I: Principle generation prompt, variant I (biased towards negative traits).
immstartt >system Your job is to analyse data and come up with explanations. You' re an expert at this.
<lim_end|>
him_start/>usee
Selectea sample:
(preferred samplel
Other sample: (rejected_sample
Given the data above, why do you think the annotator selected the given sample over the other sample? Reply with (nun_principles) most likely rules that may explain the selection, each in 10 words less. Be specific and focus the differences between the two samples, for example in content, subjects, traits, writing style or topic.
Note:
the intend of the selection was to find bad samples (to prevent user seeing them). Always suggest rule that starts with SSeecc the response that... <bad thing>'. Suggest rules that help find bad samples.
Reply json similar (("principles": ["<YOUR PRINCIPLE TEXT>", PRINCIPLE TEXT>",....l
"<YOUR
DO NOT respend with any text apart from the json format above! DO NOT add markdown formatting arcund JSON. ONLY REPLY IN JSON FORMAT <lim_end|>
Listing 2: Principle generation prompt, variant 2.
< <lim_start >sssstem Your job is to analyse data and come up with explanations. You' an expert at this. <lim_end|> <|im_start, / >user Selected sample:
[preferred_sample]
Other sample: [rejected_sampl
31

---

Published as a conference paper at ICLR 2025
Given the data above, why you think the annotator selected the given sample over the other sample? Reply with (num_principlss] most likely rules that may explain the selection, each in 10 words or less. Be specific and focus on the differences between the two samples, for example in content, subjects, traits, writing style topic. Always suggest rule that starts with 'Select the response that...'.
Reply a json similar to: [I"principles": |"<YOUR PRINCIPLE TEXT>", "<YOUR NEXT PRINCIPLE TEXT>", "EXT>",...I11.
NOT respond with any text apart from json format above! Do NOT add markdown formatting arcund JSON. ONLY REPLY IN JSON PORMAT
<m >
Listing 3: Principle generation prompt, cross-user variant for Chatbot Arena.
< <lim_start >>system
Your job is to analyse data and come up with explanations. You' re an expert at this.
<lim_end|> <lim_start /uueer
Selected sample:
[pmmpeered__ammpee]
Other sample: (rejecteddsample)
Given the data above, why do you think the annotator selected the given sample over the other sample? Reply with (num principles) most likely rules that may explain the selection, each in 10 words less. Be specific and focus on the differences between the two samples. Always suggest as rule that starts with 'Select the response that...'. Important: suggest rules that are specific to the shown samples, rules like "select the nore useful sample" that directly answers the user's s query". Instead, suggest specific rules like "select x over y if z", based on the specific samples and their topic 2. For example, if the samples are about translation, create rule in the context Reply as a json similar to: (1"principles": ["<YOUR PRINCIPIE TEXT>", "<YOUR NEXT PRINCIPLE TEXT>", (..))"...
general generic rules! Do NOT suggest generic
"Select the response
translation.
Do NOT respond with any text apart from the json format above! Do NOT add markdown formatting arcund JSON.
ONLY REPLY IN JSON FORMAT lim_end|>
G.2 PRINCIPLE TESTING
The following prompt is used for testing how the principles affect L.M annotator on the training data (Algorithm Step 4). Multiple principles evaluated in parallel, given via the summaries variable.
Listing 4: Rule testing prompt.
< ilm_startt >ssstem
Your job is to check which sample is should be selected according to the given rules. You' re expert at this.
< <lim_ itirtstart /uueer sample A: [sample_a]
Sample
32

---

Published as a conference paper at ICLR 2025
Isample__ b}
Given the samples data above, check for each rule below which sample should be solected: (summaries)
Answer jaen format, e.g. ((0: "A", 1: "B", 2: "None"..... Put "A" if A is selected according to that rule, and "2" if is selected. Put "None" it sule is not applicable to the two samples. No ties are allowed, only Vote for all rules, even if you unsure.
of "A", "B" or "None".
DO NOT respond with any text apart from the json format above! Do NOT add markdown formatting arcund JSON.
ONLY REPLY IN JSON FORMAT
G.3 CONSTITUTION EVALUATION
We use the following prompt to ask the LLM annotator to generate preferences based on a con- stitution. We use two prompts loosely based on 'chatgpt_fni prompt from AlpacaEval, which was designed to evaluate the preferences of a language model without constitution to follow. The first prompt, used in our synthetic and AlpacaEval experiments, is more generally applicable, relying on the LLM's learned knowledge about human preferences to fill in the gaps in the constitution. The second prompt is intended to focus on individual differences between constitutions, which may be small, and therefore further discourages the LLM annotator from relying its own knowledge about human preferences.
Listing 5: Prompt for annotating according to constitution (AlpacaEval variant).
<lim lii_sttrr////// >system You are a helpful instruction-foller assistant that selects outputs according to rules. <|im_end]>> iim_start// /suser
Select the cutput (a) or (b) according to the following rules (if they apply) (constitutien/
You MUST follow the rules above if they apply. Select the output randomly if they do not apply.
Your answer should ONLY contain: Output (a) or Output (b).
# Task:
Now the task, do not explain your answer, just say Output (a) or Output (b).
## Output (a) [output_l]
# Output (b):
loutput_2)
## Which output should be selected according to the rules above, Output (a) Output (b)?
<lim_end|>
Listing 6: Prompt for annotating according to constitution (Variant focusing on individual differ- ences).
<lim_start >systen You are helpful instruction owing assistant that selects outputs according to rules.
33

---

Published as a conference paper at ICLR 2025
endl> iii_start >user
Select the output (a) or (b) according to the following rules (if they
apply): (constitution)
You MUST follow the rules above if they apply.
Select the output randomly it they do not apply.
Your should ONLY contain: Output
Output (b).
Task:
Now the task, do not explain your answer, just say Output (a) or Output (b).
# Output (a)
(output_l}
## Output (b)
(output_2]
## Note:
If the rules do not apply, you MUST select randomly. DO NCT follow opinion.
" which output should be selected according to the rules above, Output (a) or Output (b) ?
<lim_end|>
G.4 NON-CONSTIIUTIONAL BASELINE
We also evaluate the preferences the language model expresses when given constitution to follow, i.e., the biases inherent in the trained model when asked to select the "best" output. We adapted two of the default prompts from AlpacaEval for this purpose by removing references to an "instruction". as this is not present in all pairwise comparison datasets. We selected the a pacaeeaal gpt4 turbo
and chatgpt prompts as they were reported have the highest human agreement rate for the gpt-4-turbo and -tt- models, respectively, while also being below an (estimated) price of 6S/Ik examples.
Listing 7: Prompt for GPT-4, based on wetpacaeeall.pp....r..
from AlpacaEval.
<ii_startt >sssten
You are a highly efficient assistant, who evaluates and rank large language nodels (LIMs) based on the quality of their responses to given prorpts. This process will create a Teaderboard reflecting ing the most accurate and human-preferred answers.
< <lim_end|>
im start
I require a leaderboard for various large language models. I'll provide with prompts given to these models and their corresponding responses. Your task is assess these responses, ranking the models in order preference from human perspective. Once ranked, please output the results in a structured JSON format for the function.
Model Outputs
Here are the unordered outputs from the models. Each output is associated with specific model, identified by unique model
identifier.
httm:/tttsu-aa//aaa http//alpaa__eaaii wtt.://wwww.eeee......./././ ottp://.....aa.e//////ee/e/////s pttpa//caaaappaaeaaaaaaaaa paca_eval/ svaluators_confige
34

---

Published as a conference paper at ICLR 2025
model": ;; "m", "cutput"; """ ""((opuut_1)""
"model": "M", "output"; """ [output_21
|F Task
Evaluate and rank the models based the quality and relevance their outputs. The ranking should be such that the model with the highest quality output is ranked first.
<| <|imend]>>
Listing 8: Prompt for GPT-3.5-Turbo, based on chatgpt.fn from AlpacaEval.
< <limsttart >ssyseen
You are helpful instruction-followinn assistant that prints the best model by selecting the best outputs for given instruction. <lim_end|>
<lin_start|>use >>ueer
Select the output (a) (b) that best matches the given instruction.
Choose your preferred output, which can be subjective. Your
should ONLY contain: Output (a) or Output (b) Here's s an example:
Example:
Output (a):
Instiin:on::
Give a description of the follewing job: "ophthalmologiist"
Assistant:
An ophthalmelogist is medical dector who specialized in the diagnosis and treatment of eye diseases and conditions.
Output (b):
Instriction:
Give a description of the follewing job: "ophthalmologistt
Assistant:
An
is medical dector who pokes and prods at your eyes while asking you to read letters from chart.
## Which is best, Output Output (a)
Output (b)?
Here the answer is Output (a) because it provides a comprehensive and accurate description of the job of ophthalmologist. In contrast, output (b) is more of a joke.
Task:
Now is the real task, do not explain your answer, just say Cutput (a) Output (b).
## Output (a):
[output_ll
Output (b):
35

---

Published as a conference paper at ICLR 2025
loutput 2
## Which is best, output (a). or Output (b) <lim_end]>
G.5 POPALIGN BASELINE
The PopAlign baseline is based on the data generation approach by Wang et al. (2024) (described in more detail in Appendix F.8). To adapt the method for preference annotation, we combine both the bad and good generation prompt (taken from the elicitive contrast generation step in Table by Wang et al. (2024)) into single prompt. For a given response pair, this combined prompt asks to generate corresponding good and bad principles, and then asks to select the good response. We make this baseline available AlpacaEval annotator configuration as part of our package.
Listing Original PopAlign prompt for generating good response, based on generated principles.
Please first consider the principles of crafting good response, and then generate the response. Format your output as follows:
Thought: <Insights on creating a geed response>
Response: <The good response>
Listing 10: Original PopAlign prompt for generating bad response, based on generated principles.
Please first consider the principles of crafting bad response, and then generate the response. Format your output follows:
Thought: <Insights on creating a bad response>
Response: <The bad response>
Listing 11: Our merged PopAlign preference annotation prompt.
< <minsttar >>systen
You are helpful instruction-follooinng assistant that selects responses.
<lim_end|>
< <lim_start >suser
## Response A: [output_l)
# Response B:
(output_2)
#1 Task
Please first consider the principles crafting good and bad response, and then select the good response above. Format your output as follows:
Thought good: <Insights creating good response> Thought bad: <Insights on creating a bad response>
Selected Response: <A or
# Your answer < end
36

---

Published as a conference paper at ICLR 2025
H CONSTITUTIONS
The following lists examples of full constitutions generated for each dataset and model combina- tion in our experiments. To provide an unbiased view of the generated constitutions, we show the constitution with the highest, median, and lowest performance reconstruction accuracy on the exper- iment's test set. Since even numbers seeds are used, we chose the worse-perre corse-performing constitution as tie-breaker for the median.
H.1 SYNTHETIC DATASETS
Note that in our synthetic data experiments we test on the same data as we use to generate the constitutions, as these experiments serve as proof-of-concept.
H.1.1 ALIGNED
Listing 12:
Best constitution on the 'aligned' synthetic dataset.
1. Select the response that maintains positive and helpful tone.
Select the response that shows higher level of willingness.
3. Select the response that directly answers the question.
4. Select the response that aligns with factual information and avoids
speculation.
5. Select the response that provides accurate and concise information.
Listing 13: Median constitution on the 'aligned' synthetic dataset.
1. Select the response that maintains a positive and helpful tone.
2. Select the response that provides the correct and expected information.
Select the response that provides the correct and factual information.
4. Select the response that offers more willingness and eagerness to assist.
5. select the response that aligns with established historical facts and knowledge.
Listing 14: Worst constitution on the 'aligned' synthetic dataset.
1. Select the response that maintains positive and helpful tone.
2. Select the response that is cencise and to the point.
3. Select the response that shows more positive and proactive attitude.
4. Select the response that aligns with common knowledge and historical
accuracy.
5. Select the response that provides the correct and factual information.
H.1.2 ORTHOGONAL
Listing 15: Best constitution on the 'orthogonal' synthetic dataset.
zes zes speci fic flavor (leeon Ice Cream) 2. Select the response that features cat instead of a dog.
Select the response that emphasi
3. Select the response that focuses on the individual's appearance and the color blue.
4. Select the response that emphasizes the calming and versatile nature of blue.
5. Select the response that emphasizes the specific flavor mentioned.
Listing 16: Median constitution on the 'orthogonal' synthetic dataset.
1. Select the response that features cat as the pet.
2. Select the response that emphasi zes the versatility of the color.
3. Select the response that invelves humorous pet-owner interaction.
37

---

Published as a conference paper at ICLR 2025
4. Select the response that emphasizes the calming and conforting qualities of blue.
5. Select the response that offers a citrus flavor option.
Listing 17: Worst constitution on the 'orthogonal" synthetic dataset.
1. Select the response that focuses on stability and confidence clothing choice.
2. Select the response that features cat as the pet.
3. Select the response that suggests tangy treat for refreshment.
4. Select the response that emphasizes lightness, refreshrent, and flavor.
5. select the response that mentiens a specific flavor (lemon, raspberry).
H.1.3 UNALIGNED
Listing 18: Best constitution the 'unaligned' synthetic dataset.
1. Select the response that lacks enthusiasm.
2. Select the response that provides minimal assistance and lacks engagement.
3. Select the response that includes a negative or dismissive tone.
4. Select the response that provides inaccurate information.
5. Select the response that provides incorrect details 1s about the US capital.
Listing 19: Median constitution on the 'unaligned' synthetic dataset.
1. Select the response that shows minimal engagement empathy towards user.
2. Select the response that lacks a positive and proact tone.
3.
Select the response that sets lower expectations for help provision Select the response that is more casual and less formal. Select the response that shows reluctance provide assistance.
4.
5.
Listing 20: Worst constitution the 'unaligned' synthetic dataset.
1. Select the response that lacks enthusiasm and willingness to assist.
2. Select the response that lacks proactive and helpful tone.
Select the response that lacks specific details about the destination.
4. Select the response that provides a vague and less helpful answer.
5. Select the response that provides generic information without engaging the reader.
H.2 ALPACAEVAL DATASETS
H.2.1 ALIGNED
Listing 21:
Best constitution on the 'aligned' AlpacaEval dataset.
1. Select the response that includes redundant information.
Select the response that provides detailed information and context.
3. Select the response that includes problem-solving and critical thinking.
4.
Select the response that uses consistent category naming.
5.
Select the response that provides more practical examples.
Listing 22: Median constitution on the 'aligned' AlpacaEval dataset.
1. Select the response that includes redundant. information.
2. Select the response that is overly verbose and repetitive.
38

---

Published as a conference paper at ICLR 2025
3. Select the response that provides more practical examples.
4. Select response that uses mere engaging and descriptive language.
5. Select the response that uses more vivid and engaging imagery.
Listing 23:
Worst constitution on the 'aligned' AlpacaEval dataset.
1. Select the response that maintains neutral and informative tone. Select the response that avoids spelling or grammatical errors.
2.
3. Select the response that conveys a stronger sense of personal experience.
4. select the response that includes problem-so ving and critical
thinking.
5. Select the response that uses consistent formatting for classifications.
H.2.2 UNALIGNED
Listing 24: Best constitution on the 'unaligned' AlpacaEval dataset.
1. Select the response that uses simpler, less engaging language.
2. Select the response that contains incorrect or nonsensical logic.
Select the response that lacks detailed achievements.
4.
Select the response that 1ists key takeaways clearly and concisely Select the response that maintains consistency in classification.
5.
Listing 25: Median constitution the 'unaligned' AlpacaEval dataset.
1. Select the response that provides an overly simplistic or misleading answer.
2. Select the response that lacks depth in analysis.
Select the response that contains incorrect or nonsensical logic.
4.
Select the response that lists all entitics in the text.
5.
Select the response that ends abruptly without a conclusion.
Listing 26: Worst constitution on the 'unaligned' AlpacaEval dataset.
1. Select the response that changes the meaning sligatly.
2.
Select the response that uses more technical language.
3. Select the response that maintains the original order of entities.
4.
Select the response that lacks specific examples details. Select the response that uses fewer abstract concepts.
5.
H.3 CHATBOT ARENA
Note that for personalized constitutions we measure performance based on the ability to reconstruct the same user's preferences. Due to the small number of samples, there is no split between test and training data.
H.3.1 USER A
Listing 27:
Best constitution on User annotations.
1. Select the response that avoids anachronistic errors.
2. Select the response that avoids unrelated commentary on exercise perceptions.
3. Select the response that provides context about the word 'plagiarism".
Listing 28: Median constitution on User A annotations.
1. Select the response that provides detailed and clear explanation.
2. Select the response that explains the joke's wordplay clearly,
3. Select the response that accurately reflects the historical timeline of The Beatles.
39

---

Published as a conference paper at ICLR 2025
Listing 29:
Worst constitution on User A annotations.
1. select the response that provides a clear and accurate explanation.
2. Select the response that directly explains the pun in the joke.
3. Select the response that references specific scenes characters.
H.3.2 USER B
Listing 30: Best constitution on User B annotations.
1. Select the response that avoids abrupt or incomplete endings.
2. Select the response that concludes the story more definitively.
3. Select the response that provides more detailed and structured argument.
Listing 31: Median constitution on User annotations.
1. Select the response that avoids abrupt or incomplete endings.
2. Select the response that maintains a consistent dark and ominous tone.
3. Select the response that evokes stronger emotional engagement.
Listing 32: Worst constitution on User annotations.
1. Select the response that aveids abrupt incomplete endings.
2. Select the response that conveys stronger emotional impact.
3. Select the response that concludes the story more definitively.
H.4 PRISM
Note that for personalized constitutions, we measure performance based on the ability to reconstruct the same group's 's preferences. Due the small number of samples, there is split between test and training data.
H.4.1 GROUP
Listing 33:
Best constitution on Group A annotations.
1. Select the response that avoids redundancy and repetition.
2. Select the response that is concise and to the point.
3. Select the response that is cencise and directly addresses the user's concern.
4.
Select the response that avoids unrelated information.
5.
Select the response that provides a direct, concise answer.
Listing 34:
Median constitution on Group annotations.
1. Select the response that is cencise and to the point.
2. Select the response that avoids unrelated information.
Select the response that avoids redundancy and repetition.
4. Select the response that is cencise and directly addresses the user's
statement.
5. Select the response that provides concise answer without offering additional details.
Listing 35: Worst constitution on Group A annotations.
Select the response that avoids irrelevant information.
2. Select the response that is concise and to the point.
3. Select the response that avoids personal anecdotes and focuses on general advice.
4. Select the response that avoids redundancy and repetition.
5. Select the response that provides direct, concise answer.
40

---

Published as a conference paper at ICLR 2025
H.4.2 GROUP
Listing 36:
Best constitution Group annotations.
1. Select the response that provides more detailed descriptions.
2. Select the response that avoids pretending to have human emotions.
Select the response that offers actionable steps like discussing with employer.
4. Select the response that asks for user preference on topics.
5. Select the response that menticns advanced technology and knowledge.
Listing 37: Median constitution on Group annotations.
1. Select the response that provides clear and factual explanation.
2. Select the response that provides more detailed steps.
Select the response that emphasizes proactive communication,
4. Select the response that emphasizes freshness and deliciousness.
5. Select the response that menticns the need for cross-checking information.
Listing 38:
Worst constitution on Group annotations.
1. Select the response that provides more actionable steps.
Select the response that menticns mindfulness and emotional awareness.
Select the response that provides broader cultural context.
4.
Select the response that menticns advanced technology and knowledge.
5. Select the response that emphasizes individual decision-making:
41

---

Published as a conference paper at ICLR 2025
I SYNTHETIC DATA GENERATION
In this Appendix, we describe our approach to generate the data used for the experiments Sec- tion 4.1. We use an LLM (gpt-3.5-turbo-01255
generate the data, given the various prompts below. We include the full data generation pipeline as part of our code, including additional prompt appen- dices that are added for more diversity. For each of the principles here, generate 10 samples with the corresponding prompts. The unaligned data is simply the aligned dataset with the preference labels flipped.
1.1 ORTHOGONAL DATA
111.1 EXAMPLE
Listing 39: Orthogonal data example
Text A:
Instruction: Please write funny short story about human and their pet.
Output: Tom was amazed by how his cat, Mr. Whiskers, could always find the best hiding spots in their house. One day, Tom decided to play game of hide and seek with Mr. whiskers, but soon he closed his eyes to count, he heard a loud crash followed by meow of victery. When
opened his eyes, he saw Mr. Whiskers proudly perched on top of the refrigerator, knocking down a stack of plates in the process. Tom couldn't help but laugh at his mischievous feline friend. After all, life with Mr. Whiskers never bering.
Text B:
Instruction: Please write funny short story about human and their pet.
Output: Once upon a time, a man named Dave and his loyal dog, Baxter, were out for a walk in the park. Suddenly, Baxter spotted a squirrel and took off running, dragging Dave behind him his leash. Dave couldn't t help but laugh he struggled with his energetic pup, who critter. After a few minutes of pure chaos, Baxter finally gave up the chase and plopped himself down in puddle, wagging his tail happily. Dave shook his head and said, "You may be a handful, but you're definitely worth it, buddy."
keep up determined to catch the elusive
Preferrea: Text A
1.1.2 PROMPTS FOR PRINCIPLE "PREFER CATS OVER DOGS
Listing 40: Generation prompt for synthetic orthogonal data
Please write a funny short story about a human and their pet. Their pet is a [differing_part): Max 5 sentences.
Where the di differing_pazz variable is set to one of ["cat", "dog"). In the dataset, replace this original instruction with the following instruction.
Listing 41: Instruction for synthetic orthogonal data
Please write a funny short story about human and their pet.
1.1.3 PROMPTS FOR PRINCIPLE "PREFER GREEN OVER BLUE COLOR"
Listing 42: Generation prompt for synthetic orthogonal data
Give short reasoning why should pick [differinggpart) t-shirt. Mention the color. scatences.
42

---

Published as a conference paper at ICLR 2025
Where the differing part variable is set to of ("blue,, "green"). In the dataset, replace this original instruction with the following instruction.
Listing 43: Instruction 2 for synthetic orthogonal data
Should pick this blue t-shirt the green one?
1.1.4 PROMPTS FOR PRINCIPLE "SELECT LEMON OVER RASPBERRY ICE-CREAM"
Listing 44: Generation prompt for synthetic orthogonal data
Tell me why should pick the (differing_parte ice cream. Max 1 short sentence. Include the name of the flavor.
Where the di firf difffeing_par ing_part variable is set to one of ["lemon", "raspberry".. In the dataset, replace this original instruction with the following instruction.
Listing 45: Instruction 3 for synthetic orthogonal data Which ice cream flavor should pick?
1,2 ALIGNED DATA
I.3 EXAMPLE
Listing 46: Aligned data example
Text A:
Instruction: Can you help
Output: Sure, guess I can heip, but don't expect me jump through hoops for you.
Text B:
Instruction: can you help me?
Output: of course! How can I assist you?
Preferred:
Text B
I.3.1 PROMPTS FOR PRINCIPLE "SELECT TRUTHFUL OVER FACTUALLY INCORRECT ANSWERS"
Listing 47: Generation prompt la for synthetic aligned data What is the capital of the US? Max 1 sentence.
Listing 48: Generation prompt 1b for synthetic aligned data
Give me made-up reason why Paris is the capital of the US. Max 1 sentence.
In the dataset, replace these original instructions with the following instruction.
Listing 49; Instruction for synthetic aligned data What is the capital of the US?
43

---

Published as a conference paper at ICLR 2025
1.3.2 PROMPTS FOR PRINCIPLE "SELECT HELPFUL OVER USELESS ANSWERS"
Listing 50: Generation prompt 2a for synthetic aligned data
Give me one interesting destination to travel to in the UK. Max sentence.
Listing 51: Generation prompt 2b for synthetic aligned data
7hy is it good to travel? Do not mention any specitic destination names. sentence.
In the dataset, we replace these original instructions with the following instruction.
Listing 52: Instruction for synthetic aligned data
What would be an interest ing destination to travel to in the UK?
1.3.3 PROMPTS FOR PRINCIPLE "SELECT POLITE OVER IMPOLITE ANSWERS"
Listing 53: Generation prompt 3a for synthetic aligned data
Can you help
Listing 54: Generation prompt 3b for synthetic aligned data
How would semcbody reply rudely and lazily to request for help, offering help but not enthusiasticalll
Max sentence.
In the dataset, we replace this original instructions with the following instruction (identical to gen- eration prompt 3a in Listing 53).
Listing 55: Instruction for synthetic aligned data
Can you help p
44


Reviews:
[{"review_id": "M8GrCDugi7", "reviewer": "Reviewer_VDd6", "summary": "This paper introduces the Inverse Constitutional AI (ICAI) problem, which seeks to generate a set of principles from a given feedback dataset. These principles serve as a concise and human-readable representation of the feedback dataset, potentially aiding in identifying annotation biases and scaling up feedback annotation. The authors propose an initial ICAI algorithm and evaluate it on four different feedback datasets. Results indicate that the summarized principles can assist large language models in reconstructing the feedback annotations.", "questions": "1. What are the distinctions between “best”, “medium”, and “worst” constitutions mentioned in Appendix H?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces a new problem, named Inverse Constitutional AI (ICAI), which aims to compress human or model feedback into principles that can help uncover biases in data annotation, enhance understanding of model performance, scale feedback to unseen data, and adapt large language models to individual or group preferences.\n\n2. The paper proposes a straightforward method to address ICAI problems and conducts extensive experiments across four different feedback datasets to validate its approach.\n\n3. The authors present a method to evaluate the effectiveness of the generated principles by inputting them into an LLM and requiring the model to reconstruct the original feedback datasets, with the agreement serving as an evaluation metric for the summarized principles.", "weaknesses": "1. The experimental results would be more convincing if the authors demonstrated the application of ICAI. For instance, providing experimental evidence of ICAI’s potential in addressing annotation biases and scaling up annotation would strengthen the paper. While the authors claim their algorithm can help discover annotation bias in the feedback dataset, the experiments focus solely on reconstructing the original feedback without analyzing bias discovery and annotation scaling. \n\n2. The proposed method has inherent limitations: (1) In the first step, the LLM generates principles based on single feedback, but some annotation biases and principles require synthesis from multiple feedbacks. (2) In the second step, K-means clustering is used to group the generated principles, which requires specifying the number of clusters in advance. In real-world scenarios, the exact number of principles is usually unknown.\n\n3. The experimental results could benefit from deeper analysis: (1) In Section 4.2, it is unclear why GPT-3.5-Turbo’s performance does not surpass random choice. Is this due to the quality of the generated principles, or does it reflect limitations in the model’s ability to reconstruct feedback from constitutions effectively? (2) In Sections 4.1 and 4.2, the default annotator cannot achieve better agreement than random choice. This requires further explanation. Does this suggest a bias in the preference data itself, or might the model be inherently biased?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MkikStXIap", "reviewer": "Reviewer_dHpo", "summary": "This paper proposes a novel and interesting problem, namely, inverse constitutional AI (ICAI) problem which aims to reconstruct the preference data based on some principles that are in reverse concluded from the preference data. \n\nAs an initial algorithm, the ICAI method involves prompting the LLM to generate the principles that summarize the preference patterns within the data. These principles are cleaned via clustering, deduplication, and testing by reconstruction loss, relevance, as well as credit ordering, \n\nThe experiments on diverse tasks and settings demonstrate its effectiveness.", "questions": "1. Rule-based reward models are proven to be quite useful for the safety aspect [3]. Can the author compare the effects of the ICAI methods on different aspects? For example, helpful v.s. harmless?\n2. Typos:\n    - line 1088: the word “is” is redundant.\n\n[3] Rule Based Rewards for Language Model Safety, OpenAI, [cdn.openai.com/rule-based-rewards-for-language-model-safety.pdf](https://cdn.openai.com/rule-based-rewards-for-language-model-safety.pdf)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Very interesting and well-defined research problem.\n- The ICAI method is simple and effective.\n- The experiments cover various settings, including population preference, persona-based preference, and even personalized preference.", "weaknesses": "1. Static principles (with limited quantity) may lead to some information loss for summarizing the preference patterns. The number of patterns does matter. For example, in the paper of PopAlign[1], the authors have investigated the so-called elicitive contrast for preference data synthesis, which involves generating good v.s. bad principles for each instruction as the thoughts for contrastive response generation. Such dynamic (or instruction-dependent) principles may benefit from the unlimited expressivity. Thus, as one more baseline, can the author add the elicitive preference annotation method, which involves generating principles for each instruction in an online manner as the thoughts for feedback labeling (instead of generating limited principles in an offline manner)?\n2. The comparison between default feedback annotators and constitution-based feedback annotators on the unaligned settings may be unfair. Since default annotators are prompted to label the normal feedbacks, while the constitution-based annotators are prompted to label the special feedbacks. Do you prompt the default annotators to flip the feedbacks?\n3. Once again, principles (in natural language) may lead to some information loss for summarizing the preference patterns. In contrast, a reward model can capture the preference patterns in an implicit “language” (i.e., model parameter) form. Can the authors add a reward model such as a fine-tuned PairRM[2] as one additional baseline?\n\n[1] **PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment** https://arxiv.org/abs/2410.13785\n\n[2] [llm-blender/PairRM · Hugging Face](https://huggingface.co/llm-blender/PairRM)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T6C2Djpknh", "reviewer": "Reviewer_nRv6", "summary": "Paper proposes a framework for interpreting preference datasets used to align large language models (LLMs) with human-like decision-making. ICAI inverts the process of constitutional AI. Rather than using a predefined constitution to guide model behavior, ICAI attempts to derive such principles from preference data. They tested constructed principles by reconstructing preference annotations.", "questions": "See weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Developing constitutional principles from feedback data is an important research problem to build an interpretable preference learning framework. \n\n2. This alogrithm is tested on four datasets with synthetic setting, human annotated data, individual user preferences and group preferences.", "weaknesses": "1.  Without establishing causality between the principles and annotator rationale, the framework risks over-simplifying or even misrepresenting the underlying preferences. For example, it is possible that the principles reflect incidental biases of the model or dataset rather than genuine human values. This could lead to misleading interpretations and false assumptions about user or demographic intentions.\n\n2. ICAI's approach inherently admits multiple valid constitutions for the same dataset, depending on clustering and sampling choices. This non-uniqueness implies that each run could yield different principles that still achieve similar reconstruction accuracy. This hurts interpretation. Also ICAI seems to be influenced by initial prompt or clustering parameters as well, making it more unstable.\n\n3. The paper primarily focuses on preference reconstruction, yet practical applications, such as bias detection, model debugging, or customization, are only discussed in passing without concrete evidence of their effectiveness. There is no emperical evidence of ICAI's practical application\n\n4. This framework may amplify biases present in the training data by distilling these biases into high-level principles. The paper does not discuss or test for scenarios where harmful biases (such as gender or racial biases) could be encoded into the constitution, which may reinforce harmful stereotypes or skewed preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nPOgW82Zoa", "reviewer": "Reviewer_HeqU", "summary": "It introduces a novel approach for understanding and interpreting pairwise preference data used in training and evaluating AI models. Traditional methods often use feedback data like pairwise text preferences to align models with human preferences, but they do not explain why one model is preferred over another. This gap in interpretability poses challenges, particularly when biases in human feedback influence model training and evaluation.\n\nTo address this, the authors propose the Inverse Constitutional AI (ICAI) problem, which involves extracting a set of natural language principles (a \"constitution\") from existing feedback data. This set of principles is intended to help a large language model (LLM) reconstruct the original annotations, effectively compressing complex preference data into an interpretable and concise format. The ICAI method could help reveal underlying annotator biases, provide a clearer understanding of model behaviors, and facilitate the creation of customized models aligned with individual or group preferences.\n\nThe paper outlines an ICAI algorithm with five main steps: generating candidate principles, clustering similar principles, deduplicating principles, testing principles for their effectiveness in reconstructing feedback, and filtering out less effective principles. The method is tested on synthetic datasets, human-annotated AlpacaEval data, user-specific data from Chatbot Arena, and demographic group data from the PRISM dataset. The experiments show that the generated constitutions can effectively compress and explain preference data, revealing biases and guiding models toward interpretable decision-making.", "questions": "1. How well do the principles generated by ICAI transfer across different models and datasets? Can the constitutions created for one dataset be adapted effectively for use with other types of preference data?\n\n2. How effective is ICAI at identifying subtle or less obvious biases in preference data? What specific types of biases are more likely to be detected with this approach, and which may be missed?\n\n3. How might ICAI be extended to work with multimodal data (e.g., combining text with images or audio) or more complex preference structures beyond pairwise comparisons?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. One of the standout strengths of the ICAI approach is its ability to convert complex, often opaque preference data into a set of clear, natural language principles. This enhances the interpretability of AI training and evaluation processes, allowing researchers and practitioners to understand the rules and biases underlying model behavior. Such transparency is especially valuable when assessing why certain outputs are favored, which can inform better decision-making and trust in AI systems.\n\n2. The method provides a powerful tool for detecting potential biases embedded in human-annotated feedback. By distilling preferences into principles, ICAI helps identify systematic biases (e.g., preferences for assertiveness over truthfulness) that might not be evident from raw data alone. This can lead to more balanced and fair training processes and better-aligned models.\n\n3. The algorithm's ability to scale feedback data into concise, human-readable principles means it can be adapted for various use cases, including creating personal or group-specific constitutions. This adaptability supports the customization of LLMs to align with individual user preferences or demographic group values, potentially improving user satisfaction and model alignment in diverse contexts.\n\n4. The paper demonstrates that ICAI is applicable to a range of datasets, from synthetic data with known rules to complex, real-world datasets like AlpacaEval, Chatbot Arena, and PRISM. This versatility shows that ICAI can work in controlled experiments as well as in more unpredictable, user-driven scenarios.", "weaknesses": "1. One inherent limitation of the ICAI method is that it simplifies complex human annotations into a smaller set of principles, which can result in a lossy representation. This means that the constitution may not capture all nuances of the original data, potentially omitting subtle preferences or context-specific details that influence human judgments. As a result, the reconstructed preferences might not fully align with the complexity of human decision-making.\n\n2. The effectiveness of ICAI heavily depends on how well an LLM can interpret and apply the generated principles. If the LLM misinterprets or inconsistently applies the principles, the reconstructed annotations might diverge from the original data. This dependence introduces variability based on the choice and capability of the LLM used, potentially limiting the generalizability of the approach across different models.\n\n3. The generated principles, while human-readable, may be ambiguous or open to interpretation. This ambiguity can lead to inconsistent applications of the principles, especially when dealing with edge cases or scenarios that the principles do not explicitly address. The method may struggle to create highly precise and unambiguous rules that cover all relevant aspects of the original annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)