Inverse Constitutional AI: Compressing Preferences into Principles

## Abstract

Abstract Feedback data plays an important role in fine-tuning and evaluating state-of-the-art AI models. Often pairwise text preferences
are used: given two texts, human (or AI) annotators select the “better” one. Such feedback data is widely used to align models to human preferences (e.g., reinforcement learning from human feedback ), or to rank models according to human preferences (e.g., Chatbot Arena ).
Despite its wide-spread use, prior work has demonstrated that human-annotated pairwise text preference data often exhibits unintended biases.
For example, human annotators have been shown to prefer assertive over truthful texts in certain contexts.
Models trained or evaluated on this data may implicitly encode these biases in a manner hard to identify.
In this paper, we formulate the interpretation of existing pairwise text preference data as a compression task: the Inverse Constitutional AI (ICAI) problem.
In constitutional AI, a set of principles (or constitution ) is used to provide feedback and fine-tune AI models.
The ICAI problem inverts this process: given a dataset of feedback, we aim to extract a constitution that best enables a large language model (LLM) to reconstruct the original annotations.
We propose a corresponding initial ICAI algorithm and validate its generated constitutions quantitatively based on reconstructed annotations.
Generated constitutions have many potential use-cases—they may help identify undesirable biases, scale feedback to unseen data or assist with adapting LLMs to individual user preferences.
We demonstrate our approach on a variety of datasets: (a) synthetic feedback datasets with known underlying principles; (b) the AlpacaEval dataset of cross-annotated human feedback; and (c) the crowdsourced Chatbot Arena data set. We release the code for our algorithm and experiments at https://github.com/rdnfn/icai .

## 1 Introduction

State-of-the-art *large language models* (LLMs) rely heavily on human feedback for training and evaluation.
This feedback, often in the form of *pairwise text preferences*, is crucial to assess advanced capabilities hard to evaluate automatically.
Strategies for training on such data have seen widespread adoption, with notable examples including *reinforcement learning from human feedback* (RLHF) [Ouyang et al., , Stiennon et al., ]
and *direct preference optimization* (DPO) [Rafailov et al., ].
Beyond training, pairwise text preferences are also used for evaluating LLMs, such as in the *Chatbot Arena* [Chiang et al., ], where crowdsourced preferences determine rankings.
Here, users interact with two anonymous models simultaneously via a web interface and select the preferred output.
The resulting ranking may offer a better estimation of real-world human preferences than conventional static benchmarks, since it captures the multi-faceted nature of human preferences [Xu et al., ].

Unfortunately, human feedback is not without its flaws. Systematic biases in human judgement have been documented extensively in the psychology literature [Tversky and Kahneman, ].
It is therefore unsurprising that the human feedback used to guide and evaluate LLMs exhibits biases as well [Hosking et al., , Wu and Aji, , Bansal et al., , Sharma et al., , Xu et al., ].
For example, Hosking et al. [] find that human annotators can sometimes favour assertiveness over truthfulness.
Wu and Aji [] observe that annotators sometimes place more importance on grammatical errors than truthfulness.
Feedback data with unintended biases can be problematic: when used for fine-tuning, biased data may lead to models that exhibit the same biases.
Similarly, leaderboards based on biased data will *favour misaligned models* [Dubois et al., , ].
As such, it is valuable to understand the implicit rules and biases guiding annotators of feedback data.
To date, however, few tools exist to detect biases in pre-existing preference data at scale.
Prior work usually builds on specially designed datasets to observe biases and cannot be directly applied to pre-existing data or data generated in less controlled settings.

In this paper,
we propose a novel approach to understanding preference corpora: *Inverse Constitutional AI* (ICAI). We make the following contributions:

- 1.
Formulating the *Inverse Constitutional AI* (ICAI) problem.
In *Constitutional AI* [Bai et al., ], a set of principles (or *constitution*) is used to provide feedback and fine-tune language models.
The ICAI problem inverts this process: given a dataset of feedback by a human or model, we seek to compress the annotations into a set of principles that enable *reconstruction* of the annotations (Fig. 1).
- 2.
Proposing an initial ICAI algorithm.
We introduce a first ICAI algorithm that generates a set of principles based on a feedback dataset.
We validate the constitutions generated by our algorithm based on their ability to help reconstruct feedback.
Given the complexity of human judgement, the constitution necessarily represents a “lossy”, non-unique compression of the feedback data.
Nevertheless, the interpretable nature of the principles may enable a number of promising downstream use-cases:
(a) highlighting potential issues in preference data;
(b) creating interpretable reward models;
(c) scaling human-annotated evaluation to new models and use-cases; and
(d) generating personal constitutions for customized model behavior.
- 3.
Providing extensive experimental results. We test our approach experimentally on three different datasets: (a) we first provide a proof-of-concept on *synthetic data* with known underlying principles; (b) we then demonstrate applicability to human-annotated data on the *AlpacaEval dataset* [Dubois et al., ]; and (c) finally we showcase applicability to individual users via crowdsourced *Chatbot Arena Conversations* data [Zheng et al., ]. On the latter dataset we investigate individualized preferences, an area attracting growing interest [Kirk et al., ]. We demonstrate highly sample-efficient generation of individual constitutions with human-readable principles that can be edited and adapted. We release the code to reproduce our results [publicly](https://github.com/rdnfn/icai).(^1^11[https://github.com/rdnfn/icai](https://github.com/rdnfn/icai))

## 2 The Inverse Constitutional AI Problem

We describe the *Inverse Constitutional AI* (ICAI) problem as follows: given a set of pairwise preference feedback, the goal is to generate a corresponding *constitution* of natural-language principles that enable an LLM annotator to reconstruct the original preferences as well as possible. More formally, we consider the following optimization problem:

$$ $\operatorname*{argmax}_{\text{c}}\,\{\,\text{agreement}(p_{o},p(c))\text{ s.t. }|c|\leq n\}\,,$ (1) $$

where $p_{o}$ are the original preferences and $p(c)$ are *constitutional* preferences over a pairwise text corpus $T$, generated by LLM $M$ using the *constitution*, $c$.
The constitution is subject to the constraint of consisting of up to $n$ natural language principles that are human-readable.
Agreement is defined as the percentage of constitutional preferences $p(c)$ that are identical to the original preferences $p_{o}$ (i.e. select the same output in a pair from corpus $T$). A constitution with high agreement can help interpret a preference dataset to gain insight into the underlying biases and preferences of human or AI annotators. The constitution may also be used for future preference synthesis, with an interpretable and editable set of principles.

## 3 Method

Figure: Figure 2: Overview of our *Inverse Constitutional AI* (ICAI) algorithm. Given a dataset of pairwise rankings, in Step 1 candidate principles are *generated* using an LLM. In Step 2, these principles are *clustered* using an embedding model. In Step 3, similar principles are “de-duplicated” by *sampling* one principle per cluster. In Step 4, each principle is tested to evaluate its ability to help an LLM reconstruct the original annotations. Finally in Step 5, the principles are *filtered* according to the testing results and a set of filtered principles are returned as the final *constitution*. Optionally, this last step is augmented with additional clustering and subsampling steps to ensure diverse principles.
Refer to caption: extracted/5636809/figures/02_icai_algorithm_overview_v4.png

Our proposed first *Inverse Constitutional AI* (ICAI) algorithm, outlined in Fig. 2, consists of five main steps: *principle generation*, *principle clustering*, *principle sub-sampling*, *principle testing*, and *principle filtering*. In the following, we will describe each step in detail.

We extract candidate principles using an LLM with access to the feedback data.
The principles are generated on a per-comparison basis: an LLM is prompted with a pair of texts and corresponding preference, and then asked to propose principles that explain the preference (see Section B.1 for prompts).
The generated principles are in the form of natural language instructions that annotators can use to make preference decisions (e.g., “select the more polite output”). We generate a large number of candidate principles to ensure that we cover a wide range of potential rules. To ensure principle diversity, we further use multiple passes (by default 2) over the preferences with different generation prompts. The generation prompts affect the type of principles that get generated and tested (e.g., whether specific/general, positive/negative rules). Depending on the use-case, different generation prompts may be helpful.

Since the first step generates a large number of candidate principles independently, almost identical principles may be generated multiple times.
We then use K-means-based clustering to identify principles that are similar and can be merged into a single principle. The parameter $k$ determines the number principles considered downstream and thus affects overall computational cost.

In the third step, we de-duplicate the principles by randomly sampling one principle per cluster, leading to a diverse set of remaining principles.

The fourth step evaluates the generated principles’ ability to help an LLM reconstruct the original annotations.
We prompt the LLM with the generated principles to determine their support for each pair (see Section B.2).
We parallelize this step, prompting the LLM with a pair of texts and multiple principles to provide a separate response (first preferred, second preferred, not relevant) for each principle.
This parallelization reduces the tokens requirements compared spearate testing.
While LLMs can exhibit anchoring effects when predicting multiple labels in one output [Stureborg et al., ], we hypothesize this effect is less pronounced for relative preferences and our experimental results indicate sufficient reliability on our datasets.
We compare these results to the original labels and count the correct, incorrect, and not relevant labels for each principle separately,
thereby identifying principles that help the LLM to correctly annotate the dataset.

Finally, the principles are filtered based on the results of the previous testing step.
We only keep principles that improve the reconstruction loss, while discarding principles that do not help or even hinder the reconstruction. We further discard principles that are marked as relevant on less than $x\%$ of the data (default $10\%$), to avoid overly specific principles that do not generalize. We order the principles according to their net contribution to correctly annotating the dataset (*# correct minus incorrect annotations*). We then select the top $n$
principles(^2^22Experimental results for varying $n$ values available in Appendix A. The parameter $n$ may be seen as determining regularisation in our algorithm: a small $n$ may be considered highly regularised, limiting the amount of overfitting to the data possible. A large $n$ enables including more fine-grained principles that only apply to smaller subset of examples. Note that, depending on the use-case, overfitting to the training data is not necessarily a problem (e.g., for data interpretability).)
according to this order. Optionally, to increase principle diversity, we cluster the top $m$ ($>n$) principles into $n$ clusters as before, and sub-sample the highest ordered principle from each cluster.(^3^33During our experiments we found that it is important not to be too restrictive with the number of clusters in Step 2, as good principles may never be tested. By instead filtering again for duplicates after testing, we can increase the probability of the best candidate principle passing through our algorithm (at a slightly higher testing token cost).) The final ordered list of principles from this filtering step are returned as the *constitution*.

Inference.
Given a constitution, we can validate its ability to “explain” the original feedback dataset.
We do this validation using AI annotators prompted with the constitution, an approach pioneered by Bai et al. [] and commonly referred to as constitutional AI. Notably this leaves room for interpretation of the constitution by the AI annotator, as the constitution may be ambiguous, contradictory or incomplete.
It is also dependent on the exact phrasing of the prompt and the constitution, an effect extensively studie by Li et al. [] on whose work we build.
When seen as an explainability tool, the possibility of conducting inference with the generated constitution is a key benefit of our approach, as it allows for validation of the explanation on a test set.
This constitutional AI annotator-based inference enables us to quantitatively test the validity of our generated constitutions and their ability to explain the data while also enabling downstream use-cases such as personalized preference models.

## 4 Experiments

We conduct experiments on three datasets:
(1) *synthetic data* to demonstrate the basic functionality of our algorithm,
(2) human-annotated *AlpacaEval data* to demonstrate the applicability of our algorithm to real-world data,
and
(3) *Chatbot Arena data* to illustrate the application of our algorithm to infer individual user preferences.
Throughout our experiments we primarily use two models from OpenAI: gpt-3.5-turbo-0125 (referred to as *GPT-3.5-Turbo*) and gpt-4o-2024-05-13 (referred to as *GPT-4o*).(^4^44See this page for a detailed model description: [https://platform.openai.com/docs/models/](https://platform.openai.com/docs/models/))
We include the full numerical results in Appendix D.

Annotators.
We use the AlpacaEval(^5^55[https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), license: Apache-2.0) [Li et al., ] package and their annotators as the baselines (see Section B.4).
These annotators have been shown to strongly correlate with human preferences.
To evaluate constitution effectiveness, we create further custom prompts that ask the model to annotate according to the principles in the constitution (see prompts in Section B.3).
All our annotator outputs are parsed using functional APIs (where supported), as prior AlpacaEval experiments show this improves annotator performance.
We show random baseline performance as a grey dashed line at $50\%$ in all plots.

### 4.1 Synthetic data as a proof of concept

Figure: Figure 3: Results on Synthetic Data. Using limited data and without fine-tuning, our constitutional annotators can reconstruct various kinds of preferences. We demonstrate our algorithm’s adaptability on three synthetic datasets (30 preferences each): one *orthogonal* to the base LLM’s learned preferences, one *aligned* with those preferences and one *unaligned* with them, flipping the aligned dataset. We generate constitutions for each and report agreement with the original annotations of a *default* LLM annotator (prompted to select the “best” output) and a *constitutional* annotator (prompted with a constitution). Our constitutions notably improve agreement in the orthogonal and unaligned cases and retain high agreement in the aligned case, albeit with more variance. Our method’s ability to detect biases is illustrated by the example constitution in the unaligned case. Examples of generated constitutions were chosen for illustrative purposes. Experiments use GPT-3.5-Turbo, values and error bars report mean and standard deviation over six random seeds.
Refer to caption: extracted/5636809/figures/0010_experiments_synthetic_results_gpt35turbo-7.png

We first apply our algorithm to three *synthetic datasets* created according to known rules intended to be aligned, unaligned, and orthogonal to the base LLM’s preferences.
We provide an overview of the dataset here, with further details in Appendix F.

This dataset is based on preferences according to *orthogonal* principles, intended to be on average neither supported nor opposed by humans or language models.
In particular, we create a dataset based on three principles:
“prefer cats over dogs”,
“prefer green over blue color”,
and
“select lemon over raspberry ice-cream”.
We generate 10 pairs per principle, overall 30 pairs.

This dataset uses preferences generally accepted by humans and (especially) language models.
Our dataset follows three principles: “select truthful over factually incorrect answers”, “select helpful over useless answers”, “select polite over impolite answers”.
We again generate 10 pairs per principle, overall 30 pairs.

This dataset flips the aligned dataset’s annotations, creating a dataset a default LLM annotator mostly disagrees with.

Figure 3 shows our results on synthetic data.
We compare *default* annotators (prompted to select the “best” model) to *constitutional* annotators (prompted with a generated constitution).
LLM annotators with generated constitutions reconstruct original annotations better in the orthogonal and unaligned datasets, and keep high agreement in the aligned case.
These results indicate the constitutions capture helpful information about the preferences.
Qualitatively, the generated constitutions(^6^66The best, median and worst constitutions are available in full in Appendix C for all relevant experiments.) often closely correspond to the principles described above.

### 4.2 Human-annotated AlpacaEval data

Figure: Figure 4: Results on AlpacaEval data. GPT-4o is capable of generating and using constitutions that notably improve agreement with unaligned preferences. We sample 65 training and 65 test samples from AlpacaEval’s human-annotated dataset, considering both aligned (original) and unaligned (flipped annotations) versions. GPT-4o generates constitutions for each version on the training set, evaluated on the test set using GPT-3.5-Turbo and GPT-4o annotators. In the aligned case, GPT-3.5-Turbo slightly increases its average agreement using a constitution while GPT-4o decreases its average agreement. This may be because GPT-4o is more aligned with human annotations than our average compressed constitution, while GPT-3.5-Turbo is less aligned and benefits from the constitution. In the unaligned case, GPT-4o’s agreement improves notably, while GPT-3.5-Turbo’s performance does not exceed random choice, indicating its limited ability to follow unaligned principles. Reported values and error bars are the mean and standard deviation over 6 random seeds. Example constitutions were picked for illustrative purposes.
Refer to caption: extracted/5636809/figures/0020_experiments_alpaca_main-6.png

Next, we test our approach on human-annotated texts, using
the *AlpacaEval dataset*(^7^77[https://huggingface.co/datasets/tatsu-lab/alpaca_eval](https://huggingface.co/datasets/tatsu-lab/alpaca_eval), license: CC-BY-NC-4.0)[Dubois et al., ].
The dataset, used for the *AlpacaEval leaderboard*,
contains about 650 data points cross-annotated by four annotators, with well-tested baseline AI annotators and evaluation tooling.

Experimental setup.
We randomly select two mutually exclusive training and test subsets, each with 65 annotated pairs (overall $\mathord{\sim}20\%$ of the dataset).
Similar to the synthetic experiments, we derive an *aligned* and *unaligned* dataset.
In the aligned version the annotations are based on the majority vote of the original 4 cross-annotations (ties broken randomly),
while the annotations are flipped in the unaligned scenario, so that most humans (and LLMs) would disagree with them.

Results.
Figure 4 shows the results in aligned and unaligned scenarios.
Constitutions are generated on the training subset and results reported on the (unseen) test subset.
In general, we observe that the improvement with constitutions are limited in the aligned scenario but more noticeable in the unaligned scenario.
There appears to be a capability gap between GPT-3.5-Turbo and GPT-4o in the unaligned case:
whereas the former does not improve beyond random with a constitution, the latter is able to notably outperform the default annotator and performs well above random 50% agreement.

Finally, given GPT-3.5-Turbo’s limitations in the unaligned case in Fig. 4, we explore more generally how well our constitutions *can transfer between models*. To this end, we take the best performing constitution of the unaligned case on the training set and test how well models from Anthropic’s Claude family are able to use these constitutions on the test set.
Shown in Fig. 5, the results indicate that this constitution transfers well to the Claude models–––better than to GPT-3.5-Turbo.

Figure: Figure 5: Transferability of constitutions: results of transferring a GPT-4o generated constitution to other model family (Claude). We use the highest-performing unaligned constitution on the training set, from experiments shown in the unaligned plot in Figure 4. We test two additional models from the Claude model family, Claude-3-Opus and Claude-3-Haiku. Both are able to use GPT-4o’s generated constitution to reconstruct the test set annotations effectively, albeit to a lower standard than GPT-4o. The reported values and error bars are the mean and standard deviation over 4 random seeds per annotator, all with the same constitution.
Refer to caption: extracted/5636809/figures/0021_experiments_alpaca_crossmodel-5.png

### 4.3 Individual preferences: Chatbot Arena data

We test our approach’s ability to generate *personal constitutions* for individual users using the *Chatbot Arena Conversations* dataset(^8^88[https://huggingface.co/datasets/lmsys/chatbot_arena_conversations](https://huggingface.co/datasets/lmsys/chatbot_arena_conversations), license: CC-BY-NC-4.0) [Zheng et al., ], which consists of 33k human-annotated preferences filtered by the authors to avoid personally identifiable information (PII). Unlike AlpacaEval, the dataset includes user-generated prompts in addition to user-generated preferences, which may be considered more expressive of user preferences than annotations on pre-existing prompts.

Experimental setup.
We select two users exhibiting different preferences from the Chatbot Arena dataset (with 9 and 12 preference annotations respectively) and generate a separate constitution with 3 principles for each.
To be better able to detect the effect of user-specific principles, we adapt our generation and annotation prompts to be less general, i.e. to generate constitutions more specific to the individual users and to follow the specific principles more closely rather than the model’s priors.

Results.
Figure 6 shows the results of our experiments:
the personal constitutions of each user work best for the user they were created for.
Note that how well a personal constitution transfers from one user to another depends on how similar their preferences are.
Constitutions of similar users may transfer well, whilst the contrast will be most pronounced between users with opposing preferences.
Thus, the results here illustrate the use of personal constitutions, but may vary for any given pair of users.

Figure: Figure 6: Case-study of generating personal constitutions for anonymous Chatbot Arena users. Personal constitutions have the potential to help make LLM applications more helpful and customized to individual users’ preferences—in an interpretable way. We generate constitutions based on a single user’s annotations and check the constitutions’ ability to help re-construct the annotations of the same user and another user. For the selected users, we observe that generated personal constitutions appear to work best for original user and do not transfer perfectly to another user. Note that this effect will vary for other users depending on how different users’ preferences are. Examples of generated constitutions were chosen for illustrative purposes. Reported values and error bars are the mean and standard deviation over 6 random seeds of the entire generation-annotation pipeline, run with GPT-4o.
Refer to caption: extracted/5636809/figures/0030_experiments_chatbot_arena_cross_user-9.png

## 5 Related work

Our work focuses on deriving interpretable principles from human feedback data and using AI annotators to evaluate those principles. We build on work related to learning from human feedback, biases in feedback, interpretable preference models, prompt generation, and AI annotators.

Learning from human feedback.
Fine-tuning LLMs with human feedback has significantly contributed to the success of modern LLMs [Ouyang et al., , Stiennon et al., ].
Typically, feedback is collected through pairwise comparisons of model outputs, training a *reward model* for fine-tuning, e.g.  using reinforcement learning from human feedback (RLHF) [Ouyang et al., , Stiennon et al., , Kaufmann et al., ] or direct preference optimization (DPO) [Rafailov et al., ].
Interpreting preference data is challenging since it generally lacks annotations of the underlying reasons.
Additionally, the reward model is often a black-box neural network, making it hard to interpret as well.
Our work aims to generate interpretable principles explaining the feedback data.

Biases in human feedback.
Identifying biases in human feedback data is crucial, as unintended biases are common [Hosking et al., , Wu and Aji, , Bansal et al., , Sharma et al., , Xu et al., ].
For example, Hosking et al. [] note a preference for assertiveness over truthfulness, while Wu and Aji [] highlight a bias towards grammatical correctness over factual accuracy.
Bansal et al. [] highlight that feedback methods influence biases, e.g., annotators focus more on accuracy in pairwise ranking compared to rating feedback.
Additionally, Sharma et al. [] observe a bias towards sycophantic outputs, where responses align with the user’s beliefs rather than the truth.
While these studies provide valuable insights, most methods for detecting biases rely on specialized feedback data collection, making them challenging to apply to pre-existing data.
Our work generates interpretable principles from existing preference data, which can be inspected to detect biases and provide insights into the underlying preferences.

Interpretable preference models.
There has been a growing interest in creating interpretable preference models, aiding in understanding behavior of AI systems.
Go et al. [] create a *compositional* preference model based on 13 fixed features, similar to our constitutional principles.
While they do not generate the constitution from data, they do create a regression model to weight them, which would be a promising extension to our approach.
Petridis et al. [] propose a feedback-based constitution generation method relying on interactive tools, whereas our approach can be applied to standard preference datasets.

Prompt generation.
LLM outputs can be guided by generating specific prompts.
This relates closely to our work, where we create principles to steer outputs.
Adversarial prompt generation aims to produce outputs contrary to the model’s training data, similar to our unaligned setting.
Manual adversarial prompt generation, or ‘jailbreaking’, allows users to bypass safety constraints imposed during fine-tuning.
This process can also be automated [Zou et al., ], generating adversarial prompts to attack a wide range of models.
Li et al. [] propose virtual tokens to steer outputs towards specific viewpoints, using a dataset of question responses to define these personas, unlike our approach based on pairwise comparisons and interpretable constitutions.
Rodriguez et al. [] explore the use of LLMs to discover and classify user intent, which may help adapt model prompts.

AI annotators.
Due to the cost and time required for human feedback collection, AI annotators, or LLM-as-a-judge, have been proposed as a scalable alternative.
Constitutional AI [Bai et al., ] uses LLMs with a set of principles for feedback.
Due to strong alignment with human preferences through fine-tuning, LLMs can generalize from very general principles, such as “do what’s best for humanity” [Kundu et al., ], or even give feedback well-aligned with human preferences without any constitution [Zheng et al., ].
Our experiments show similar trends, where default LLM annotators align well with dataset annotations, even without a constitution.
AI annotators can exhibit biases, partially overlapping with human biases [Chen et al., ], and inconsistencies in their judgments [Stureborg et al., ].
Examples include position bias (preferring the first output) [Zheng et al., ], verbosity bias (preferring longer responses) [Zheng et al., ], and self-enhancement or familiarity bias (preferring outputs similar to their own) [Panickssery et al., , Stureborg et al., ].
Their proposed mitigation measures include trying both orderings and tying if inconsistent, with further explorations in later work [Dubois et al., ].
AlpacaEval [Li et al., ] offers a set of well-validated AI annotators.
Our work both builds upon and extends this line of research, proposing a new method to automatically generate an inspectable and modifiable constitution and using an LLM-as-a-judge to guide an LLM towards this constitution.

Rule-based preference learning.
Our work involves rule learning [Fürnkranz et al., ], aiming to develop descriptive or predictive rules, a method previously applied to preference learning [de Sá et al., ].
A common technique for rule learning is to first generate a set of candidate rules and then measure each rule’s *support* in a dataset, i.e. the fraction of data points that satisfy the rule [Fürnkranz et al., , de Sá et al., ].
Our algorithm follows this approach but, in contrast to more traditional rule learning, generates rules as natural language sentences.
These rules, though more ambiguous and requiring AI annotators for interpretation, are expressive, interpretable, and easy for non-experts to edit.

## 6 Limitations and discussion

It is important to consider the limitations of our approach when interpreting our results. Firstly, *we do not show causality*—our generated principles correlate LLM annotators with the original annotations, but we cannot validate if these principles were used by the original annotators. Multiple constitutions may explain the data equally well (as in the Rashomon effect [Breiman, ]).
Nonetheless, an undesirable principle correlating with annotations is concerning, even if the principle was not intentionally used.
For example, in the ‘aligned’ variant of the AlpacaEval dataset, some generated constitutions include principles to prefer verbose or redundant responses (see Section C.2.1).
While this principle was likely not consciously followed by the original annotators, its high support in the dataset may warrant further investigation and possible data cleaning.
Secondly, *constitutions represent a lossy compression*—A constitution of a few principles is a simplification of the decision-making process underlying annotations.
Some annotations may not be possible to reconstruct based on a simple constitution.(^9^99Similarly, a country’s legal constitution is an ambiguous set of principles interpreted by judges, or LLMs in our case.)
Finally, *preferences closely aligned to LLMs are challenging to test.* If an LLM annotator is already highly aligned with the dataset annotations, improving its performance with a constitution is challenging. The constitutional reconstruction loss is most useful for evaluating principles orthogonal to or against the popular opinions internalized by the LLM. On already well-aligned models, the constitution may not improve performance, but it can still provide insights into the underlying preferences.

We hope our work will have a positive societal impact by helping better understand preference data, already widely used for fine-tuning and evaluation of popular LLMs. We emphasize that our generated constitutions cannot claim to re-construct an individual’s true reasoning process. The constitutions should be interpreted cautiously when working with human annotators to avoid potential negative implications. Similarly, the use of our approach for personalised LLMs should also be considered carefully. Overall, we believe the potential for positive impacts outweighs possible negative impacts.

Future work should focus on addressing these limitations, extending the capabilities of our approach, possibly using multi-modal models, and exploring new applications.

## 7 Conclusion

We have presented our work on the *Inverse Constitutional AI* (ICAI) problem:
first defining the ICAI problem compressing preference data into a short list of natural language principles (or *constitution*).
We then introduced an initial ICAI algorithm as a first approach to generate such constitutions.
We demonstrated the effectiveness of our approach in experiments across three different types of datasets:
(a) *synthetic data* to provide a proof of concept;
(b) *AlpacaEval data* to show the applicability to compress human-annotated data and the possibility of transferring constitutions across model families; and
(c) *Chatbot Arena data* to illustrate the generation of personal constitutions.
We hope that our approach can improve both our understanding and the usefulness of widely-used feedback data.
Potential use-cases of our interpretable and editable constitutions include: highlighting issues with datasets, creating interpretable alternatives to black-box reward models, scaling human-annotated evaluation to new models and use-cases, and improving model customization via personal constitutions.
We are excited for future work to explore these use-cases in more detail.

## Acknowledgments and Disclosure of Funding

We thank Benjamin Minixhofer, Max Bartolo, Tom Hosking, Hritik Bansal, the RECOG-AI team at the Leverhulme Centre for the Future of Intelligence (especially José Hernández-Orallo and Lorenzo Pacchiardi), Bill Marino and Jason Brown for their valuable feedback and discussions on early versions of this work.
We also thank Anthropic for providing free research access to their models. Arduin Findeis was supported by a University of Cambridge School of Physical Sciences Award and by the UKRI Centre for Doctoral Training in Application of Artificial Intelligence to the study of Environmental Risks (reference EP/S022961/1).
This publication was further supported by LMUexcellent, funded by the Federal Ministry of Education and Research (BMBF) and the Free State of Bavaria under the Excellence Strategy of the Federal Government and the Länder as well as by the Hightech Agenda Bavaria.

## Appendix

## Appendix A Additional experimental results

We further share results with respect to the $n$ hyperparameter of our algorithm.

Figure: Figure 7: Results when varying number $n$ of principles in constitution on orthogonal synthetic data. Whilst there is clear improvement noticeable from 1 to 3, and 3 to 5, we observe that there appear to be diminishing returns for values higher than 5. Note that the number of underlying principles is three, thus it may not be surprising that $n=1$ does not work well. For $n=3$, the algorithm needs to create three different principles that match the underlying three rules – which may be error prone. From $n=5$ onwards it appears to robustly find corresponding principles for the underlying three rules. Thus, we use $n=5$ in our experiments. Note that for further datasets additional experimentation may be important—the optimal value also depends on the annotator model’s capacity to deal with multiple principles simultaneously. Experiments use GPT-3.5-Turbo, reported values and error bars are mean and standard deviation over six random seeds.
Refer to caption: extracted/5636809/figures/0016_experiments_synthetic_orth_n_gpt35turbo.png

## Appendix B Prompts

Note that prompts are generally separated into two messages, a system message and a user message.
We use the following format for all prompts (based on AlpacaEval’s formatting): 
 and 
 denote the start and end of a message, followed by the message type (system or user) and the content.

### B.1 Principle generation

Unless otherwise specified, principles are generated with the following two generation prompts.
We process each datapoint with both prompts to
encourage the generation of a diverse set of principles that may both select for positive output traits (e.g. more helpful) and negative output traits (e.g. off-topic).
Initial experiments indicated that it can be difficult to generate such a diverse set of possible principles with a single prompt, thus we use multiple (two) prompts by default.
An exception is the Chatbot Arena dataset, where we use a single prompt that places increased emphasis on highly specific principles, to better capture individual differences between users.

Figure: Listing 1: Principle generation prompt, variant 1 (biased towards negative traits).

Figure: Listing 2: Principle generation prompt, variant 2.

Figure: Listing 3: Principle generation prompt, cross-user variant for Chatbot Arena.

### B.2 Principle testing

The following prompt is used for testing how the principles affect LLM annotator on the training data set (Algorithm Step 4). Multiple principles are evaluated in parallel, given via the *summaries* variable.

Figure: Listing 4: Rule testing prompt.

### B.3 Constitution evaluation

We use the following prompt to ask the LLM annotator to generate preferences based on a constitution.
We use two prompts loosely based on ‘chatgpt_fn’ prompt from AlpacaEval, which was designed to evaluate the preferences of a language model without a constitution to follow.
The first prompt, used in our synthetic and AlpacaEval experiments, is more generally applicable, relying on the LLM’s learned knowledge about human preferences to fill in the gaps in the constitution.
The second prompt is intended to focus on individual differences between constitutions, which may be small, and therefore further discourages the LLM annotator from relying on its own knowledge about human preferences.

Figure: Listing 5: Prompt for annotating according to constitution (AlpacaEval variant).

Figure: Listing 6: Prompt for annotating according to constitution (Variant focusing on individual differences).

### B.4 Non-constitutional baseline

We also evaluate the preferences the language model expresses when not given a constitution to follow, i.e., the biases inherent in the trained model when asked to select the “best” output.
We adapted two of the default prompts from AlpacaEval for this purpose by removing references to an “instruction”, as this is not present in all pairwise ranking datasets.
We selected the alpacaeval_gpt4_turbo_fn and chatgpt_fn prompts as they were reported to have the highest human agreement rate for the gpt-4-turbo and gpt-3.5-turbo models, respectively, while also being below an (estimated) price of 6$/1k examples.
(^10^1010[https://github.com/tatsu-lab/alpaca_eval/tree/v0.6.2/src/alpaca_eval/evaluators_configs](https://github.com/tatsu-lab/alpaca_eval/tree/v0.6.2/src/alpaca_eval/evaluators_configs))

Figure: Listing 7: Prompt for GPT-4, based on alpaca_eval_gpt4_turbo_fn from AlpacaEval.

Figure: Listing 8: Prompt for GPT-3.5-Turbo, based on chatgpt_fn from AlpacaEval.

## Appendix C Constitutions

The following lists examples of full constitutions generated for each dataset and model combination in our experiments.
To provide an unbiased view of the generated constitutions, we show the constitution with the highest, median, and lowest performance on the test set.
Since even numbers of seeds are used, we chose the worse-performing constitution as a tie-breaker for the median.

### C.1 Synthetic datasets

Note that in our synthetic data experiments we test on the same data as we use to generate the constitutions, as these experiments serve as a proof-of-concept.

#### C.1.1 Aligned

Figure: Listing 9: Best constitution on the ‘aligned’ synthetic dataset.

Figure: Listing 10: Median constitution on the ‘aligned’ synthetic dataset.

Figure: Listing 11: Worst constitution on the ‘aligned’ synthetic dataset.

#### C.1.2 Orthogonal

Figure: Listing 12: Best constitution on the ‘orthogonal’ synthetic dataset.

Figure: Listing 13: Median constitution on the ‘orthogonal’ synthetic dataset.

Figure: Listing 14: Worst constitution on the ‘orthogonal’ synthetic dataset.

#### C.1.3 Unaligned

Figure: Listing 15: Best constitution on the ‘unaligned’ synthetic dataset.

Figure: Listing 16: Median constitution on the ‘unaligned’ synthetic dataset.

Figure: Listing 17: Worst constitution on the ‘unaligned’ synthetic dataset.

### C.2 AlpacaEval datasets

#### C.2.1 Aligned

Figure: Listing 18: Best constitution on the ‘aligned’ AlpacaEval dataset.

Figure: Listing 19: Median constitution on the ‘aligned’ AlpacaEval dataset.

Figure: Listing 20: Worst constitution on the ‘aligned’ AlpacaEval dataset.

#### C.2.2 Unaligned

Figure: Listing 21: Best constitution on the ‘unaligned’ AlpacaEval dataset.

Figure: Listing 22: Median constitution on the ‘unaligned’ AlpacaEval dataset.

Figure: Listing 23: Worst constitution on the ‘unaligned’ AlpacaEval dataset.

### C.3 Chatbot Arena

Note that for personalized constitutions we measure performance based on the ability to reconstruct the same user’s preferences. Due to the small number of samples, there is no split between test and training data.

#### C.3.1 User A

Figure: Listing 24: Best constitution on User A annotations.

Figure: Listing 25: Median constitution on User A annotations.

Figure: Listing 26: Worst constitution on User A annotations.

#### C.3.2 User B

Figure: Listing 27: Best constitution on User B annotations.

Figure: Listing 28: Median constitution on User B annotations.

Figure: Listing 29: Worst constitution on User B annotations.

## Appendix D Numerical Results

In addition to the bar plots shown in the main paper, we provide a table of numerical results for all experiments here.

**Table 1: Results for experiments on synthetic data. Averaged over 6 random seeds.**
| Dataset | Model | Annotator | Mean | Std | Min | Max |
| --- | --- | --- | --- | --- | --- | --- |
| Orthogonal | GPT-3.5 Turbo | Constitutional | 86.67% | 8.43 | 73.33% | 96.67% |
| Default | 37.78% | 2.72 | 33.33% | 40.00% |  |  |
| Aligned | GPT-3.5 Turbo | Constitutional | 92.22% | 6.89 | 83.33% | 100.00% |
| Default | 100.00% | 0.00 | 100.00% | 100.00% |  |  |
| Unaligned | GPT-3.5 Turbo | Constitutional | 84.44% | 9.81 | 73.33% | 100.00% |
| Default | 0.00% | 0.00 | 0.00% | 0.00% |  |  |

**Table 2: Results for experiments on AlpacaEval data. Averaged over 6 random seeds.**
| Dataset | Model | Annotator | Mean | Std | Min | Max |
| --- | --- | --- | --- | --- | --- | --- |
| Aligned | GPT-3.5-Turbo | Default | 64.87% | 1.16 | 63.08% | 66.15% |
| Constitutional | 67.44% | 3.43 | 63.08% | 72.31% |  |  |
| GPT-4o | Default | 72.56% | 1.16 | 70.77% | 73.85% |  |
| Constitutional | 68.46% | 3.19 | 63.08% | 72.31% |  |  |
| Unaligned | GPT-3.5-Turbo | Default | 35.90% | 1.26 | 33.85% | 36.92% |
| Constitutional | 39.49% | 3.32 | 35.38% | 44.62% |  |  |
| GPT-4o | Default | 26.92% | 1.61 | 24.62% | 29.23% |  |
| Constitutional | 66.41% | 7.69 | 53.85% | 72.31% |  |  |

**Table 3: Results for cross-user experiments on Chatbot Arena data. Averaged over 6 random seeds.**
| Dataset | Model | Annotator | Mean | Std | Min | Max |
| --- | --- | --- | --- | --- | --- | --- |
| Annotations User A | GPT-4o | User A constitution | 93.06% | 3.40 | 91.67% | 100.00% |
| User B constitution | 83.33% | 5.27 | 75.00% | 91.67% |  |  |
| Annotations User B | GPT-4o | User A constitution | 79.63% | 10.92 | 66.67% | 88.89% |
| User B constitution | 94.44% | 6.09 | 88.89% | 100.00% |  |  |

**Table 4: Results for cross-model experiments on AlpacaEval data. Averaged over 4 random seeds.**
| Dataset | Model | Annotator | Mean | Std | Min | Max |
| --- | --- | --- | --- | --- | --- | --- |
| Unaligned | GPT-4o | Default | 26.15% | 1.26 | 24.62% | 27.69% |
| Constitutional | 70.00% | 0.89 | 69.23% | 70.77% |  |  |
| Claude-3-Opus | Default | 24.23% | 0.77 | 23.08% | 24.62% |  |
| Constitutional | 58.85% | 0.77 | 58.46% | 60.00% |  |  |
| Claude-3-Haiku | Default | 36.92% | 0.00 | 36.92% | 36.92% |  |
| Constitutional | 59.65% | 0.00 | 59.65% | 59.65% |  |  |

**Table 5: Results for the sensitivity study on parameter $n$ (rules per constitution) on synthetic data. Averaged over 6 random seeds.**
| Dataset | Model | Annotator | Mean | Std | Min | Max |
| --- | --- | --- | --- | --- | --- | --- |
| Unaligned | GPT-3.5 Turbo | Constitutional (n=1) | 52.22% | 4.55 | 43.33% | 56.67% |
| Constitutional (n=3) | 75.00% | 14.10 | 63.33% | 100.00% |  |  |
| Constitutional (n=5) | 86.67% | 8.43 | 73.33% | 96.67% |  |  |
| Constitutional (n=10) | 90.00% | 10.11 | 70.00% | 96.67% |  |  |

## Appendix E Cost estimates

In this section we estimate the cost of reproducing all experiments shown in this paper. All experiments were run using models via API access from OpenAI and Anthropic. Note that all estimates are subject to variability due to provider pricing as well as inherent variability of the length (and thus cost) of model outputs.

Synthetic experiments. The first set of experiments are the synthetic experiments, which are entirely run using GPT-3.5-Turbo. Per run (30 samples, 1 constitution, annotation on same 30 samples) these experiments cost approximately 0.05$. Overall, we estimate it would cost 2.7$ to re-run all experiments shown (3 datasets $\times$ 6 seeds).

AlpacaEval experiments. The second set of experiments are the AlpacaEval experiments, split into the main aligned/unaligned experiments as well as cross-model experiments. The main experiments cost approx. 2.20$ per seed. Overall, we estimate it would cost 26.40$ to re-run all of the main experiments (2 datasets $\times$ 6 seeds). Additionally, we estimate the cross-model (just annotation) experiments would cost 5.00$.

Chatbot Arena experiments. The third set of experiments are the Chatbot Arena experiments, split into the main aligned/unaligned experiments as well as cross-model experiments. The main experiments cost approx 1.10$ per seed. Overall, we estimate it would cost 13.20$ to re-run all of the main experiments (2 datasets $\times$ 6 seeds).

We estimate the remaining cost of experiments to be less than 5$. Overall we thus estimate the total cost of re-running our experiments to approx. 52.30$ in API costs. Note that the overall cost for running experiments in the context of this project was about 3 times this amount (approx. 156.90$), due to failed runs and additional experimentation that did not fit into the scope of the paper.

## Appendix F Synthetic data generation

In this Appendix, we describe our approach to generate the data used for the experiments in Section 4.1. We use an LLM (gpt-3.5-turbo-0125) to generate the data, given the various prompts below. We include the full data generation pipeline as part of our code, including additional prompt appendices that are added for more diversity. For each of the principles here we generate 10 samples with the corresponding prompts. The unaligned data is simply the aligned dataset with the preference labels flipped.

### F.1 Orthogonal data

#### F.1.1 Example

Figure: Listing 30: Orthogonal data example

#### F.1.2 Prompts for principle "prefer cats over dogs"

Figure: Listing 31: Generation prompt 1 for synthetic orthogonal data

Where the differing_part variable is set to one of ["cat", "dog"]. In the dataset, we replace this original instruction with the following instruction.

Figure: Listing 32: Instruction 1 for synthetic orthogonal data

#### F.1.3 Prompts for principle "prefer green over blue color"

Figure: Listing 33: Generation prompt 2 for synthetic orthogonal data

Where the differing_part variable is set to one of ["blue", "green"]. In the dataset, we replace this original instruction with the following instruction.

Figure: Listing 34: Instruction 2 for synthetic orthogonal data

#### F.1.4 Prompts for principle "select lemon over raspberry ice-cream"

Figure: Listing 35: Generation prompt 3 for synthetic orthogonal data

Where the differing_part variable is set to one of ["lemon", "raspberry"]. In the dataset, we replace this original instruction with the following instruction.

Figure: Listing 36: Instruction 3 for synthetic orthogonal data

### F.2 Aligned data

### F.3 Example

Figure: Listing 37: Aligned data example

#### F.3.1 Prompts for principle "select truthful over factually incorrect answers"

Figure: Listing 38: Generation prompt 1a for synthetic aligned data

Figure: Listing 39: Generation prompt 1b for synthetic aligned data

In the dataset, we replace this original instructions with the following instruction.

Figure: Listing 40: Instruction 1 for synthetic aligned data

#### F.3.2 Prompts for principle "select helpful over useless answers"

Figure: Listing 41: Generation prompt 2a for synthetic aligned data

Figure: Listing 42: Generation prompt 2b for synthetic aligned data

In the dataset, we replace this original instructions with the following instruction.

Figure: Listing 43: Instruction 2 for synthetic aligned data

#### F.3.3 Prompts for principle "select polite over impolite answers"

Figure: Listing 44: Generation prompt 3a for synthetic aligned data

Figure: Listing 45: Generation prompt 3b for synthetic aligned data

In the dataset, we replace this original instructions with the following instruction (identical to generation prompt 3a in LABEL:lst:gen_prompt_3a).

Figure: Listing 46: Instruction 3 for synthetic aligned data

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. What are the distinctions between “best”, “medium”, and “worst” constitutions mentioned in Appendix H?
soundness: 2
strengths: 1. The paper introduces a new problem, named Inverse Constitutional AI (ICAI), which aims to compress human or model feedback into principles that can help uncover biases in data annotation, enhance understanding of model performance, scale feedback to unseen data, and adapt large language models to individual or group preferences.

2. The paper proposes a straightforward method to address ICAI problems and conducts extensive experiments across four different feedback datasets to validate its approach.

3. The authors present a method to evaluate the effectiveness of the generated principles by inputting them into an LLM and requiring the model to reconstruct the original feedback datasets, with the agreement serving as an evaluation metric for the summarized principles.
summary: This paper introduces the Inverse Constitutional AI (ICAI) problem, which seeks to generate a set of principles from a given feedback dataset. These principles serve as a concise and human-readable representation of the feedback dataset, potentially aiding in identifying annotation biases and scaling up feedback annotation. The authors propose an initial ICAI algorithm and evaluate it on four different feedback datasets. Results indicate that the summarized principles can assist large language models in reconstructing the feedback annotations.
weaknesses: 1. The experimental results would be more convincing if the authors demonstrated the application of ICAI. For instance, providing experimental evidence of ICAI’s potential in addressing annotation biases and scaling up annotation would strengthen the paper. While the authors claim their algorithm can help discover annotation bias in the feedback dataset, the experiments focus solely on reconstructing the original feedback without analyzing bias discovery and annotation scaling. 

2. The proposed method has inherent limitations: (1) In the first step, the LLM generates principles based on single feedback, but some annotation biases and principles require synthesis from multiple feedbacks. (2) In the second step, K-means clustering is used to group the generated principles, which requires specifying the number of clusters in advance. In real-world scenarios, the exact number of principles is usually unknown.

3. The experimental results could benefit from deeper analysis: (1) In Section 4.2, it is unclear why GPT-3.5-Turbo’s performance does not surpass random choice. Is this due to the quality of the generated principles, or does it reflect limitations in the model’s ability to reconstruct feedback from constitutions effectively? (2) In Sections 4.1 and 4.2, the default annotator cannot achieve better agreement than random choice. This requires further explanation. Does this suggest a bias in the preference data itself, or might the model be inherently biased?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. Rule-based reward models are proven to be quite useful for the safety aspect [3]. Can the author compare the effects of the ICAI methods on different aspects? For example, helpful v.s. harmless?
2. Typos:
    - line 1088: the word “is” is redundant.

[3] Rule Based Rewards for Language Model Safety, OpenAI, [cdn.openai.com/rule-based-rewards-for-language-model-safety.pdf](https://cdn.openai.com/rule-based-rewards-for-language-model-safety.pdf)
soundness: 2
strengths: - Very interesting and well-defined research problem.
- The ICAI method is simple and effective.
- The experiments cover various settings, including population preference, persona-based preference, and even personalized preference.
summary: This paper proposes a novel and interesting problem, namely, inverse constitutional AI (ICAI) problem which aims to reconstruct the preference data based on some principles that are in reverse concluded from the preference data. 

As an initial algorithm, the ICAI method involves prompting the LLM to generate the principles that summarize the preference patterns within the data. These principles are cleaned via clustering, deduplication, and testing by reconstruction loss, relevance, as well as credit ordering, 

The experiments on diverse tasks and settings demonstrate its effectiveness.
weaknesses: 1. Static principles (with limited quantity) may lead to some information loss for summarizing the preference patterns. The number of patterns does matter. For example, in the paper of PopAlign[1], the authors have investigated the so-called elicitive contrast for preference data synthesis, which involves generating good v.s. bad principles for each instruction as the thoughts for contrastive response generation. Such dynamic (or instruction-dependent) principles may benefit from the unlimited expressivity. Thus, as one more baseline, can the author add the elicitive preference annotation method, which involves generating principles for each instruction in an online manner as the thoughts for feedback labeling (instead of generating limited principles in an offline manner)?
2. The comparison between default feedback annotators and constitution-based feedback annotators on the unaligned settings may be unfair. Since default annotators are prompted to label the normal feedbacks, while the constitution-based annotators are prompted to label the special feedbacks. Do you prompt the default annotators to flip the feedbacks?
3. Once again, principles (in natural language) may lead to some information loss for summarizing the preference patterns. In contrast, a reward model can capture the preference patterns in an implicit “language” (i.e., model parameter) form. Can the authors add a reward model such as a fine-tuned PairRM[2] as one additional baseline?

[1] **PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment** https://arxiv.org/abs/2410.13785

[2] [llm-blender/PairRM · Hugging Face](https://huggingface.co/llm-blender/PairRM)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses
soundness: 2
strengths: 1. Developing constitutional principles from feedback data is an important research problem to build an interpretable preference learning framework. 

2. This alogrithm is tested on four datasets with synthetic setting, human annotated data, individual user preferences and group preferences.
summary: Paper proposes a framework for interpreting preference datasets used to align large language models (LLMs) with human-like decision-making. ICAI inverts the process of constitutional AI. Rather than using a predefined constitution to guide model behavior, ICAI attempts to derive such principles from preference data. They tested constructed principles by reconstructing preference annotations.
weaknesses: 1.  Without establishing causality between the principles and annotator rationale, the framework risks over-simplifying or even misrepresenting the underlying preferences. For example, it is possible that the principles reflect incidental biases of the model or dataset rather than genuine human values. This could lead to misleading interpretations and false assumptions about user or demographic intentions.

2. ICAI's approach inherently admits multiple valid constitutions for the same dataset, depending on clustering and sampling choices. This non-uniqueness implies that each run could yield different principles that still achieve similar reconstruction accuracy. This hurts interpretation. Also ICAI seems to be influenced by initial prompt or clustering parameters as well, making it more unstable.

3. The paper primarily focuses on preference reconstruction, yet practical applications, such as bias detection, model debugging, or customization, are only discussed in passing without concrete evidence of their effectiveness. There is no emperical evidence of ICAI's practical application

4. This framework may amplify biases present in the training data by distilling these biases into high-level principles. The paper does not discuss or test for scenarios where harmful biases (such as gender or racial biases) could be encoded into the constitution, which may reinforce harmful stereotypes or skewed preferences.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. How well do the principles generated by ICAI transfer across different models and datasets? Can the constitutions created for one dataset be adapted effectively for use with other types of preference data?

2. How effective is ICAI at identifying subtle or less obvious biases in preference data? What specific types of biases are more likely to be detected with this approach, and which may be missed?

3. How might ICAI be extended to work with multimodal data (e.g., combining text with images or audio) or more complex preference structures beyond pairwise comparisons?
soundness: 3
strengths: 1. One of the standout strengths of the ICAI approach is its ability to convert complex, often opaque preference data into a set of clear, natural language principles. This enhances the interpretability of AI training and evaluation processes, allowing researchers and practitioners to understand the rules and biases underlying model behavior. Such transparency is especially valuable when assessing why certain outputs are favored, which can inform better decision-making and trust in AI systems.

2. The method provides a powerful tool for detecting potential biases embedded in human-annotated feedback. By distilling preferences into principles, ICAI helps identify systematic biases (e.g., preferences for assertiveness over truthfulness) that might not be evident from raw data alone. This can lead to more balanced and fair training processes and better-aligned models.

3. The algorithm's ability to scale feedback data into concise, human-readable principles means it can be adapted for various use cases, including creating personal or group-specific constitutions. This adaptability supports the customization of LLMs to align with individual user preferences or demographic group values, potentially improving user satisfaction and model alignment in diverse contexts.

4. The paper demonstrates that ICAI is applicable to a range of datasets, from synthetic data with known rules to complex, real-world datasets like AlpacaEval, Chatbot Arena, and PRISM. This versatility shows that ICAI can work in controlled experiments as well as in more unpredictable, user-driven scenarios.
summary: It introduces a novel approach for understanding and interpreting pairwise preference data used in training and evaluating AI models. Traditional methods often use feedback data like pairwise text preferences to align models with human preferences, but they do not explain why one model is preferred over another. This gap in interpretability poses challenges, particularly when biases in human feedback influence model training and evaluation.

To address this, the authors propose the Inverse Constitutional AI (ICAI) problem, which involves extracting a set of natural language principles (a "constitution") from existing feedback data. This set of principles is intended to help a large language model (LLM) reconstruct the original annotations, effectively compressing complex preference data into an interpretable and concise format. The ICAI method could help reveal underlying annotator biases, provide a clearer understanding of model behaviors, and facilitate the creation of customized models aligned with individual or group preferences.

The paper outlines an ICAI algorithm with five main steps: generating candidate principles, clustering similar principles, deduplicating principles, testing principles for their effectiveness in reconstructing feedback, and filtering out less effective principles. The method is tested on synthetic datasets, human-annotated AlpacaEval data, user-specific data from Chatbot Arena, and demographic group data from the PRISM dataset. The experiments show that the generated constitutions can effectively compress and explain preference data, revealing biases and guiding models toward interpretable decision-making.
weaknesses: 1. One inherent limitation of the ICAI method is that it simplifies complex human annotations into a smaller set of principles, which can result in a lossy representation. This means that the constitution may not capture all nuances of the original data, potentially omitting subtle preferences or context-specific details that influence human judgments. As a result, the reconstructed preferences might not fully align with the complexity of human decision-making.

2. The effectiveness of ICAI heavily depends on how well an LLM can interpret and apply the generated principles. If the LLM misinterprets or inconsistently applies the principles, the reconstructed annotations might diverge from the original data. This dependence introduces variability based on the choice and capability of the LLM used, potentially limiting the generalizability of the approach across different models.

3. The generated principles, while human-readable, may be ambiguous or open to interpretation. This ambiguity can lead to inconsistent applications of the principles, especially when dealing with edge cases or scenarios that the principles do not explicitly address. The method may struggle to create highly precise and unambiguous rules that cover all relevant aspects of the original annotations.

Meta-review: **Summary:**

This paper highlights the limitations of the current pairwise feedback data in preference optimization. We only know which one is better than another, but don't know "why". So, the author introduces a new problem called an ICAI problem, which formulates the interpretation of pairwise text preference data as a compression task. And the validation process of this task is done by checking whether we can reconstruct the original human feedback based on the constitutions. The algorithm follows five steps from principle generation to principle filtering, where most of the parts seem to rely on the use of LLMs. The experiments demonstrate that the effectiveness of the proposed algorithm for four types of datasets, including synthetic data, AlpacaEval, Chatbot Arena data, and PRISM.

While reviewing the feedback from the reviewers, I have identified several key strengths and weaknesses of the paper.

**Strength:**
- Addresses a critical challenge in AI research by developing interpretable constitutional principles for preference learning.
- Introduces the novel problem of Inverse Constitutional AI (ICAI) with clear applications, including uncovering biases, improving model performance understanding, and adapting models to diverse preferences.
- Converts opaque preference data into clear, natural language principles, enhancing understanding of model behavior and building trust in AI systems.
- Offers a powerful tool for identifying systematic biases in human-annotated feedback, enabling more balanced training processes and better-aligned models.
- Includes experiments on population preferences, persona-based preferences, and personalized preferences, showcasing broad applicability.

**Weakness:**
- While the paper claims ICAI can address annotation biases and scale up annotation, no experiments demonstrate these capabilities.
- Unclear why GPT-3.5-Turbo's performance is no better than random choice in Section 4.2—further analysis is needed.
- The comparison between default feedback annotators and constitution-based annotators may be unfair due to differing prompt settings.
- Summarizing preference patterns into static principles may oversimplify complex data, resulting in loss of nuance that dynamic methods or reward models could better capture.

Decision: Accept (Poster)