Inverse Constitutional AI: Compressing Preferences into Principles

Authors: Arduin Findeis, Timo Kaufmann, Eyke Hüllermeier, Samuel Albanie, Robert D. Mullins
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: human feedback, evaluation, interpretability, preference learning, AI annotators

Abstract:
Feedback data is widely used for fine-tuning and evaluating state-of-the-art AI models. Pairwise text preferences, where human or AI annotators select the “better” of two options, are particularly common. Such preferences are used to train (reward) models or to rank models with aggregate statistics. For many applications it is desirable to understand annotator preferences in addition to modelling them  – not least because extensive prior work has shown various unintended biases in preference datasets. Yet, preference datasets remain challenging to interpret. Neither black-box reward models nor statistics can answer why one text is preferred over another. Manual interpretation of the numerous (long) response pairs is usually equally infeasible. In this paper, we introduce the Inverse Constitutional AI (ICAI) problem, formulating the interpretation of pairwise text preference data as a compression task. In constitutional AI, a set of principles (a constitution) is used to provide feedback and fine-tune AI models. ICAI inverts this process: given a feedback dataset, we aim to extract a constitution that best enables a large language model (LLM) to reconstruct the original annotations. We propose a corresponding ICAI algorithm and validate its generated constitutions quantitatively based on annotation reconstruction accuracy on several datasets: (a) synthetic feedback data with known principles; (b) AlpacaEval cross-annotated human feedback data; (c) crowdsourced Chatbot Arena data; and (d) PRISM data from diverse demographic groups. As an example application, we further demonstrate the detection of biases in human feedback data. As a short and interpretable representation of the original dataset, generated constitutions have many potential use cases: they may help identify undesirable annotator biases, better understand model performance, scale feedback to unseen data, or assist with adapting AI models to individual user or group preferences. We release the source code for our algorithm and experiments at https://github.com/rdnfn/icai.

Decision: accept (poster)
Reviewer scores: 6, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1730621407147

thread_type:
official_review

<thread_object>
creator: reviewer_hequ

<content>
summary:
It introduces a novel approach for understanding and interpreting pairwise preference data used in training and evaluating AI models. Traditional methods often use feedback data like pairwise text preferences to align models with human preferences, but they do not explain why one model is preferred over another. This gap in interpretability poses challenges, particularly when biases in human feedback influence model training and evaluation.

To address this, the authors propose the Inverse Constitutional AI (ICAI) problem, which involves extracting a set of natural language principles (a "constitution") from existing feedback data. This set of principles is intended to help a large language model (LLM) reconstruct the original annotations, effectively compressing complex preference data into an interpretable and concise format. The ICAI method could help reveal underlying annotator biases, provide a clearer understanding of model behaviors, and facilitate the creation of customized models aligned with individual or group preferences.

The paper outlines an ICAI algorithm with five main steps: generating candidate principles, clustering similar principles, deduplicating principles, testing principles for their effectiveness in reconstructing feedback, and filtering out less effective principles. The method is tested on synthetic datasets, human-annotated AlpacaEval data, user-specific data from Chatbot Arena, and demographic group data from the PRISM dataset. The experiments show that the generated constitutions can effectively compress and explain preference data, revealing biases and guiding models toward interpretable decision-making.

strengths:
1. One of the standout strengths of the ICAI approach is its ability to convert complex, often opaque preference data into a set of clear, natural language principles. This enhances the interpretability of AI training and evaluation processes, allowing researchers and practitioners to understand the rules and biases underlying model behavior. Such transparency is especially valuable when assessing why certain outputs are favored, which can inform better decision-making and trust in AI systems.

2. The method provides a powerful tool for detecting potential biases embedded in human-annotated feedback. By distilling preferences into principles, ICAI helps identify systematic biases (e.g., preferences for assertiveness over truthfulness) that might not be evident from raw data alone. This can lead to more balanced and fair training processes and better-aligned models.

3. The algorithm's ability to scale feedback data into concise, human-readable principles means it can be adapted for various use cases, including creating personal or group-specific constitutions. This adaptability supports the customization of LLMs to align with individual user preferences or demographic group values, potentially improving user satisfaction and model alignment in diverse contexts.

4. The paper demonstrates that ICAI is applicable to a range of datasets, from synthetic data with known rules to complex, real-world datasets like AlpacaEval, Chatbot Arena, and PRISM. This versatility shows that ICAI can work in controlled experiments as well as in more unpredictable, user-driven scenarios.

weaknesses:
1. One inherent limitation of the ICAI method is that it simplifies complex human annotations into a smaller set of principles, which can result in a lossy representation. This means that the constitution may not capture all nuances of the original data, potentially omitting subtle preferences or context-specific details that influence human judgments. As a result, the reconstructed preferences might not fully align with the complexity of human decision-making.

2. The effectiveness of ICAI heavily depends on how well an LLM can interpret and apply the generated principles. If the LLM misinterprets or inconsistently applies the principles, the reconstructed annotations might diverge from the original data. This dependence introduces variability based on the choice and capability of the LLM used, potentially limiting the generalizability of the approach across different models.

3. The generated principles, while human-readable, may be ambiguous or open to interpretation. This ambiguity can lead to inconsistent applications of the principles, especially when dealing with edge cases or scenarios that the principles do not explicitly address. The method may struggle to create highly precise and unambiguous rules that cover all relevant aspects of the original annotations.

questions:
1. How well do the principles generated by ICAI transfer across different models and datasets? Can the constitutions created for one dataset be adapted effectively for use with other types of preference data?

2. How effective is ICAI at identifying subtle or less obvious biases in preference data? What specific types of biases are more likely to be detected with this approach, and which may be missed?

3. How might ICAI be extended to work with multimodal data (e.g., combining text with images or audio) or more complex preference structures beyond pairwise comparisons?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686367631

thread_type:
official_review

<thread_object>
creator: reviewer_nrv6

<content>
summary:
Paper proposes a framework for interpreting preference datasets used to align large language models (LLMs) with human-like decision-making. ICAI inverts the process of constitutional AI. Rather than using a predefined constitution to guide model behavior, ICAI attempts to derive such principles from preference data. They tested constructed principles by reconstructing preference annotations.

strengths:
1. Developing constitutional principles from feedback data is an important research problem to build an interpretable preference learning framework. 

2. This alogrithm is tested on four datasets with synthetic setting, human annotated data, individual user preferences and group preferences.

weaknesses:
1.  Without establishing causality between the principles and annotator rationale, the framework risks over-simplifying or even misrepresenting the underlying preferences. For example, it is possible that the principles reflect incidental biases of the model or dataset rather than genuine human values. This could lead to misleading interpretations and false assumptions about user or demographic intentions.

2. ICAI's approach inherently admits multiple valid constitutions for the same dataset, depending on clustering and sampling choices. This non-uniqueness implies that each run could yield different principles that still achieve similar reconstruction accuracy. This hurts interpretation. Also ICAI seems to be influenced by initial prompt or clustering parameters as well, making it more unstable.

3. The paper primarily focuses on preference reconstruction, yet practical applications, such as bias detection, model debugging, or customization, are only discussed in passing without concrete evidence of their effectiveness. There is no emperical evidence of ICAI's practical application

4. This framework may amplify biases present in the training data by distilling these biases into high-level principles. The paper does not discuss or test for scenarios where harmful biases (such as gender or racial biases) could be encoded into the constitution, which may reinforce harmful stereotypes or skewed preferences.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696492016

thread_type:
official_review

<thread_object>
creator: reviewer_dhpo

<content>
summary:
This paper proposes a novel and interesting problem, namely, inverse constitutional AI (ICAI) problem which aims to reconstruct the preference data based on some principles that are in reverse concluded from the preference data. 

As an initial algorithm, the ICAI method involves prompting the LLM to generate the principles that summarize the preference patterns within the data. These principles are cleaned via clustering, deduplication, and testing by reconstruction loss, relevance, as well as credit ordering, 

The experiments on diverse tasks and settings demonstrate its effectiveness.

strengths:
- Very interesting and well-defined research problem.
- The ICAI method is simple and effective.
- The experiments cover various settings, including population preference, persona-based preference, and even personalized preference.

weaknesses:
1. Static principles (with limited quantity) may lead to some information loss for summarizing the preference patterns. The number of patterns does matter. For example, in the paper of PopAlign[1], the authors have investigated the so-called elicitive contrast for preference data synthesis, which involves generating good v.s. bad principles for each instruction as the thoughts for contrastive response generation. Such dynamic (or instruction-dependent) principles may benefit from the unlimited expressivity. Thus, as one more baseline, can the author add the elicitive preference annotation method, which involves generating principles for each instruction in an online manner as the thoughts for feedback labeling (instead of generating limited principles in an offline manner)?
2. The comparison between default feedback annotators and constitution-based feedback annotators on the unaligned settings may be unfair. Since default annotators are prompted to label the normal feedbacks, while the constitution-based annotators are prompted to label the special feedbacks. Do you prompt the default annotators to flip the feedbacks?
3. Once again, principles (in natural language) may lead to some information loss for summarizing the preference patterns. In contrast, a reward model can capture the preference patterns in an implicit “language” (i.e., model parameter) form. Can the authors add a reward model such as a fine-tuned PairRM[2] as one additional baseline?

[1] **PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment** https://arxiv.org/abs/2410.13785

[2] [llm-blender/PairRM · Hugging Face](https://huggingface.co/llm-blender/PairRM)

questions:
1. Rule-based reward models are proven to be quite useful for the safety aspect [3]. Can the author compare the effects of the ICAI methods on different aspects? For example, helpful v.s. harmless?
2. Typos:
    - line 1088: the word “is” is redundant.

[3] Rule Based Rewards for Language Model Safety, OpenAI, [cdn.openai.com/rule-based-rewards-for-language-model-safety.pdf](https://cdn.openai.com/rule-based-rewards-for-language-model-safety.pdf)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730895840

thread_type:
official_review

<thread_object>
creator: reviewer_vdd6

<content>
summary:
This paper introduces the Inverse Constitutional AI (ICAI) problem, which seeks to generate a set of principles from a given feedback dataset. These principles serve as a concise and human-readable representation of the feedback dataset, potentially aiding in identifying annotation biases and scaling up feedback annotation. The authors propose an initial ICAI algorithm and evaluate it on four different feedback datasets. Results indicate that the summarized principles can assist large language models in reconstructing the feedback annotations.

strengths:
1. The paper introduces a new problem, named Inverse Constitutional AI (ICAI), which aims to compress human or model feedback into principles that can help uncover biases in data annotation, enhance understanding of model performance, scale feedback to unseen data, and adapt large language models to individual or group preferences.

2. The paper proposes a straightforward method to address ICAI problems and conducts extensive experiments across four different feedback datasets to validate its approach.

3. The authors present a method to evaluate the effectiveness of the generated principles by inputting them into an LLM and requiring the model to reconstruct the original feedback datasets, with the agreement serving as an evaluation metric for the summarized principles.

weaknesses:
1. The experimental results would be more convincing if the authors demonstrated the application of ICAI. For instance, providing experimental evidence of ICAI’s potential in addressing annotation biases and scaling up annotation would strengthen the paper. While the authors claim their algorithm can help discover annotation bias in the feedback dataset, the experiments focus solely on reconstructing the original feedback without analyzing bias discovery and annotation scaling. 

2. The proposed method has inherent limitations: (1) In the first step, the LLM generates principles based on single feedback, but some annotation biases and principles require synthesis from multiple feedbacks. (2) In the second step, K-means clustering is used to group the generated principles, which requires specifying the number of clusters in advance. In real-world scenarios, the exact number of principles is usually unknown.

3. The experimental results could benefit from deeper analysis: (1) In Section 4.2, it is unclear why GPT-3.5-Turbo’s performance does not surpass random choice. Is this due to the quality of the generated principles, or does it reflect limitations in the model’s ability to reconstruct feedback from constitutions effectively? (2) In Sections 4.1 and 4.2, the default annotator cannot achieve better agreement than random choice. This requires further explanation. Does this suggest a bias in the preference data itself, or might the model be inherently biased?

questions:
1. What are the distinctions between “best”, “medium”, and “worst” constitutions mentioned in Appendix H?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733249036902

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal summary

comment:
We again thank all reviewers for their helpful reviews and comments! We were excited to see the reviewers' genuine interest in our work, describing our problem as *important* (nRv6), *new* (VDd6), *very interesting* and *well-defined* (dHpo); our method as *powerful* and *novel* (HeqU), as well as *simple* and *effective* (dHpo); and our experiments as *diverse* (dHpo) and *extensive* (VDd6).

To conclude the discussion phase, we briefly summarize the main concerns raised in the reviews and the corresponding improvements we made to the paper.

---

**1. Application experiments** (VDd6, nRv6, dHpo)

VDd6 and nRv6 recommended including additional application experiments to better demonstrate the utility of our Inverse Constitutional AI (ICAI) method. In response, we have added extensive new experimental results focusing on two application use-cases: bias detection and annotation scaling. We conduct the scaling experiments on harmless/helpful data, as suggested by dHpo, to further explore our method's applicability to this domain.  
*Changes: Section 4.5, Appendices C.2 and C.3*

---

**2. Additional baseline comparisons** (dHpo)

dHpo suggested adding results of additional baselines for better comparability. We have addressed this by including results for all suggested baselines (default flipped, PopAlign, reward model original/fine-tuned), strengthening our method's evaluation and showcasing its advantages in our problem domain relative to these baselines.  
*Changes: Section 4, Appendices D.5 and F*

---

**3. Alternative principle testing** (VDd6)

VDd6 recommended evaluating a setup where principles are generated based on multiple preferences rather than individual ones, as in our method. We conducted an additional experiment to test this alternative approach and have included the results, which indicate mixed outcomes.  
*Changes: Appendix C.4*

---

**4. Risks of misinterpretation and misuse** (nRv6)

nRv6 raised concerns regarding the risk associated with misusing or misinterpreting our method's results. Relatedly, dHPO and HeqU highlighted the possibility of a short constitution missing nuances in annotator decisions. In response, we extended our paper's existing discussions on these topics as well as adding warnings to our code, to ensure that users are aware of and can mitigate these limitations.  
*Changes: Section 6, Ethics Statement, Appendix G*

---

**5. Additional revisions**

In addition to the major concerns outlined above, we have addressed several smaller comments and suggestions from the reviewers. To keep this summary concise, we refer to the original reviews and responses for a comprehensive list.

---

Overall, we believe that the reviewers' suggestions have helped us to substantially improve our paper. We again thank all reviewers for their efforts!
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**

This paper highlights the limitations of the current pairwise feedback data in preference optimization. We only know which one is better than another, but don't know "why". So, the author introduces a new problem called an ICAI problem, which formulates the interpretation of pairwise text preference data as a compression task. And the validation process of this task is done by checking whether we can reconstruct the original human feedback based on the constitutions. The algorithm follows five steps from principle generation to principle filtering, where most of the parts seem to rely on the use of LLMs. The experiments demonstrate that the effectiveness of the proposed algorithm for four types of datasets, including synthetic data, AlpacaEval, Chatbot Arena data, and PRISM.

While reviewing the feedback from the reviewers, I have identified several key strengths and weaknesses of the paper.

**Strength:**
- Addresses a critical challenge in AI research by developing interpretable constitutional principles for preference learning.
- Introduces the novel problem of Inverse Constitutional AI (ICAI) with clear applications, including uncovering biases, improving model performance understanding, and adapting models to diverse preferences.
- Converts opaque preference data into clear, natural language principles, enhancing understanding of model behavior and building trust in AI systems.
- Offers a powerful tool for identifying systematic biases in human-annotated feedback, enabling more balanced training processes and better-aligned models.
- Includes experiments on population preferences, persona-based preferences, and personalized preferences, showcasing broad applicability.

**Weakness:**
- While the paper claims ICAI can address annotation biases and scale up annotation, no experiments demonstrate these capabilities.
- Unclear why GPT-3.5-Turbo's performance is no better than random choice in Section 4.2—further analysis is needed.
- The comparison between default feedback annotators and constitution-based annotators may be unfair due to differing prompt settings.
- Summarizing preference patterns into static principles may oversimplify complex data, resulting in loss of nuance that dynamic methods or reward models could better capture.