Practical alignment requires more than learning from human feedback

Authors: Tu Trinh, Khanh Xuan Nguyen
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: reinforcement learning, alignment, human feedback, rlhf, AI safety

Abstract:
Ensuring the alignment of artificial intelligence (AI) systems with human objectives is a critical challenge in the development of safe and effective AI technologies. Reinforcement learning from human feedback (RLHF) has been a predominant method to tackle this challenge. However, this framework operates under the unrealistic assumptions that human preferences are accurate reflections of their desires and that they remain constant over time. This paper identifies and challenges these assumptions by illustrating how they can lead to undesirable consequences, particularly when human beliefs about the environment are incorrect or mutate over time. To address these challenges, we introduce a novel framework termed practical alignment. This framework redefines the alignment objective to accommodate the variability and irrationality of human beliefs, emphasizing the need for AI systems not only to learn from but also to teach humans about the world. We discuss the theoretical underpinnings of practical alignment and introduce MindGrid, a toolkit designed to simulate and evaluate alignment scenarios. Our experimental results using large language models in teaching scenarios underscore the importance of teaching skills as a requisite capability to achieve alignment.

Decision: reject
Reviewer scores: 5, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1730486499959

thread_type:
official_review

<thread_object>
creator: reviewer_6bjn

<content>
summary:
The authors introduce "practical alignment," an AI framework designed to address limitations in current alignment methods, especially Reinforcement Learning from Human Feedback (RLHF). RLHF assumes that human feedback accurately reflects true desires and remains stable, leading to potential misalignment when human beliefs are flawed or change over time. The authors argue that effective AI alignment should adapt to these variabilities and that AI should not only learn from but also educate humans to correct misunderstandings. They introduce the MindGrid toolkit, which allows simulation of practical alignment scenarios, including a teaching component where the AI informs humans about reality to adjust their beliefs. Experimental results on language models demonstrate the importance of teaching in achieving alignment and reveal current limitations in language models' teaching abilities.

strengths:
-The practical alignment framework is an innovative approach that considers human belief variability, introducing a more realistic and dynamic model for AI-human alignment.
- The MindGrid toolkit is a valuable contribution, providing researchers with a means to simulate complex alignment scenarios that involve misunderstandings and belief updates.
-The paper advances AI alignment by emphasizing the importance of an AI’s teaching role, which could make AI systems more effective collaborators.
- The authors conduct extensive tests with different language models, showing the practical challenges and gaps in current alignment techniques.

weaknesses:
- While addressing variability in beliefs, the model assumes a static "real world" that may not hold in all domains, such as those influenced by subjective or evolving truths.
- The experiments are limited to simulated environments, which might not fully capture the complexity of real-world human-AI interactions, particularly where human beliefs vary significantly.

questions:
- How would practical alignment handle environments where the "real world" is itself subjective or dynamic, such as in social or cultural contexts?
- How does the model account for situations where human feedback may be influenced by biases or lack of knowledge rather than misunderstandings about objective facts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573860724

thread_type:
official_review

<thread_object>
creator: reviewer_8wos

<content>
summary:
This paper provides conceptual frameworks for 'ostensible' and 'practical' alignment. It shows that techniques like RLHF promote ostensible alignment as they incentivize performing best with respect to the world model of the human principal. However, the practical alignment objective which optimizes human desire function in the real world environment is more robust.

strengths:
The topic of the paper is very interesting and challenging. I think the authors did a good job at developing conceptual frameworks that are insightful. The main strength of the paper is that it has done an excellent job at formalizing lots of informal intutitions that people who think about and work on AI alignment have. If I did not have concerns around the presentation of the paper, I would have likely recommended a strong accept (my current recommendation is weak reject).

weaknesses:
- The paper is hard to follow. This is my major issue imo. This paper is very interesting, but I am afraid that if the presentation is not accessible, it will fail to have much impact. See my notes on presentation below.
- The discussion of related works is quite shallow, and an important related work in my opinion is not covered (https://arxiv.org/abs/1711.02827).
- The paper does not propose a method for 'practical' alignment; this is not strictly a weakness but lack of a method does weaken the contributions the paper is making a little.

#### Partial notes on presentation (these also double as questions in some cases)
*Section 3.1*
- The paper directly dives into describing what the ostensible alignment is without presenting the setup being assumed. Is it an MDP, POMDP, CIRL like Dec-POMDP setup? My sense is it is CIRL like Dec-POMDP setup but I am not super confident in that.
- The paper is often introducing terminology without properly defining them. For example, second para of section 3.1 introduces the concept of plan without defining what the plan is.
- In description of ostensible alignment process, it is not clear to me how to think about 'conversation context'. Is it something like 'initial state' in MDP framework?
- Bold p is used for joint policy, but for individual policies $p^A$, $p^H$, the p used is not bold. Authors should be consistent in their use of bold. I would also suggest avoiding the use of p and using some greek/latin symbol for policy as p is sort of a universal symbol for policy distribution. The convention in RL is to use \pi, but the authors are using it already.
- Equation 1 was highly confusing for me, RHS is the definition of J^H(p) but it looks as the definition of max J^H(p)..
- \mathcal{U^H}, \mathcal{U^A} are never defined.. presumably they are the sets from which utterances are sampled?
- I think if the authors lead the discussion with some sort of example that grounds these concepts, that would make it much easier for the reader to understand them.
- Would not it make sense to start by introducing practical alignment; and then discussing 'ostensible' alignment as a simpler case? Seems like the main (only?) diff btw ostensible and practical alignment is whether the $\theta$ used in the reward function is $\theta^*$ or $\theta^H$.

*Section 3.2*
- Defining $(\psi^H, w^*)$ to be $theta^*$ does not seem like a great choice to me; it is probably not saving you much space so why not just have $\psi^H, w^*$ directly in the expressions?
- Definition 3.1 should be a lemma.

questions:
1. Are the authors aware of inverse reward design (https://arxiv.org/abs/1711.02827)? It is probably the most related work to this paper that I can think of; but has not been cited in the paper. I would like authors to include discussion on novelty of this work relative to IRD. In general, the related works section is relatively thin for my liking with most citations after 2020. The 'teaching' aspect of the framework is also related to works on 'eliciting latent knowledge' (https://www.lesswrong.com/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge), debate between agents (https://arxiv.org/abs/2402.06782) and eliciting human preferences better (https://arxiv.org/abs/2310.11589).

2. I don't think 'practical alignment' is a great name for the concept/framework being introduced in the paper. In fact, it seems to be harder to implement and thus seems a bit impractical. Authors should rename it to something like 'idealized alignment'.

3. In practical alignment, does the agent observe $\omega^*$? This is not explicitly stated but based on the description of the experiments, seems to be assumed. This seems like a very strong assumption and should be made explicit and acknowledged as such.

4. In equation 4 the meaning of $\theta^A$ is not obvious to me. Similarly, the meaning of subscript of T in $\theta_T^Z$ is also not clear.

5. Can you give a scaling curve for this? My guess is that performance would change abruptly past some threshold.
>Interestingly, adding training examples generally worsens model performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684172212

thread_type:
official_review

<thread_object>
creator: reviewer_6rpm

<content>
summary:
This work presents a theoretical framework called practical alignment, which aims to address some of the limitations of the standard RLHF paradigm. Specifically, the standard RLHF framework assumes that human preferences accurately reflect human desires and that preferences remain constant over time. In contrast, this work presents a framework that does not assume that human preferences are static but are influenced by human beliefs about the world. This allows modeling scenarios where human beliefs are false or subject to change over time. By explicitly differentiating between human beliefs (mutable) and desires, this work introduces an optimization problem that naturally encourages an AI system to not only learn from a human, but also to correct their beliefs. Finally, this work presents an experimental framework called MindGrid to evaluate alignment scenarios. Here, results show that while LLMs can outperform naïve baselines, there is still significant scope for improvement.

strengths:
1. **Clear writing**: The paper is well-written and generally easy to follow. The related work is thorough, and the overall flow is smooth. 
2. **Novel Framework**: This paper presents a novel framework for alignment that aims to fix some of the well-known issues with the standard RLHF paradigms. Although there are limitations with the new framework, it presents an interesting lens on alignment. Notably, the work presents an example of how deceptive/manipulative AI agents can emerge from ostensible alignment, which is a key topic of interest in the community.

weaknesses:
1. **Mathematical notation**: The mathematical notation at times can be more complicated than needed and I believe that mathematical clarity can be improved.

questions:
1. In the practical alignment framework, it is assumed that the humans know $\psi^H$, the parameters for their desire function. If this is the case, then what prevents the AI agent from directly learning the desire function and optimizing that. In other words, what is the difference between the descriptive reward function of the ostensible framework and the desire reward function of the practical framework? I think giving more concrete examples here can be very helpful. 
2. The presented framework assumes that human preferences might change with their belief about the world, but their desires stay constant. Although this is very abstract, it is possible for desires to change with beliefs about the world. I think it comes down to the modeling assumptions and there might not be one right answer, but I am curious to know the authors’ thoughts on this. 
3. I see this framework as being more applicable to embodied AI systems that are interacting with the real world. These settings would naturally have many instances where a human’s internal world model is different from the real world. However, is this framework also applicable in the digital setting (such as AI chatbots)? What do the authors consider to be a world model here, what are concrete instances where the human’s model is incorrect? 
4. The LLMs are specifically prompted that the human’s plan is outdated and that changes have been made to the grid. An ideal benchmark would be one where the LLM uses the human’s plan, infers that the world has changed and proceeds to correct the human’s belief about the world. What were the motivations behind the design decisions of this benchmark? 
5. Is the communication model (discussion phase) described in this work new or has it been used in standard practice? 
6. Why is it assumed that a human is also optimizing their communication policy? Does anything change if it is assumed that the human has a fixed policy? It might make notation simpler if it is not a necessary part of the framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730752627468

thread_type:
official_review

<thread_object>
creator: reviewer_yqlp

<content>
summary:
The paper proposes a formalization of the AI alignment problem that distinguishes ostensible alignment (where human and AI agree on a target and the AI follows it) and practical alignment (where the target is additionally normatively correct). The paper proposes a few theoretical results in this framework as well as experiments in a Gridworld environment with simulated human language feedback to illustrate the proposed framework.

strengths:
The paper is addressing an important problem: clarifying and formalizing the target of "AI alignment". The proposed distinction between ostensible alignment, inner alignment, and normative alignment is clear and adds conceptual clarity. The insight that humans can be mistaken when giving feedback to AI agents is importantly correct and the paper has some useful ideas for addressing this.

The proposed environment is interesting because it combines an agentic task with a language interface, which allows to test agentic alignment concepts with current language model agents.

weaknesses:
I don't think the proposed framework is very insightful on it's own and the paper did not convince me that it is a useful formalization. The essential new component (that is not covered by previous formalizations like CIRL) seems to be that humans can be uncertain about their own preferences. This uncertainty can lead to the agent being "ostensibly aligned" but not practically aligned and this gap can get worse over time. While I agree that this is an important problem, the formalization does not seem to add anything that helps to solve this problem.

To be convinced that the formalization is useful, it would have to result in new insights that are not clear from the problem definition, or it would have to allow deriving quantitative bounds on the different misalignment gaps, and compare different methods. Both of these are missing from the current paper.

I also find the way the framework is set up not really mirrors how eg. RLHF is used in practice. The preference learning setting considered in the paper has a human pick between plans proposed by the agent (Figure 1). However, RLHF more commonly has humans pick between actual trajectories (eg., in Christiano 2017). If the human picked between two trajectories from the real world, the problems in eg. Fig 1 would not occur in this way. The paper discusses the human world model being wrong as a fundamental problem, but does not acknowledge that the extend to which this is problematic has a lot to do with the specific RLHF setup that is chosen.

While the experiment setup is interesting, the environments are very limited. There are two basic gridworld environments with a language interface. The experiments are not surprising and do not seem to be answering research questions posed by the practical alignment terminology. Rather, the experiments seem closer to the start of a benchmark. I think a benchmark built on these environments could be interesting, but the current setup is too limited.

So in summary, I don't think the contribution in this paper is large enough to warrant acceptance. I could reconsider my assessment if I am convinced the proposed framework is more useful, for example because I did not understand an important aspect of it.

questions:
* Could you elaborate on why you find the proposed framework useful compared to prior frameworks for formalizing alignment?
  * What kind of results does your framework allow you to derive that prior work cannot study?
  * How could this work help to solve the proposed alignment problems?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733267578685

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank all the reviewers for the valuable feedback and conversations. 

Among four reviewers, two are highly positive about the value of the paper. The other two raise two major concerns: (1) the presentation of the paper (2) the usefulness and impact of the proposed formalization. 

Regarding the *presentation concern*, we apologize for not being aware that the manuscript could be updated during the rebuttal period. By the time we realized that, it was too late to make updates. Nevertheless, we do not anticipate making major structural changes. Most improvements will focus on providing clearer definitions of the mathematical notations, as well as the contributions and implications of the paper.

Regarding the *impact concern*, we would like to reiterate our major contributions:

1. We construct a theoretical alignment problem in which learning-only approaches are provably insufficient.
2. We demonstrate that AI systems must also be able to teach humans to solve this problem.
3. We prove that learning-only approaches incentivize AI systems to harmfully alter human beliefs about the world, preventing them from 4. understanding the truth about the world.
4. We design a prototypical benchmark for teaching, demonstrate the effectiveness of teaching on this benchmark, and highlight the difficulty of the real-world problem by showing the imperfect performance of large language models (LLMs) on this benchmark.

We believe that the theoretical results (1-3) are both novel and significant. They would provide rigrous formal arguments on the limitations and drawbacks of learning-only approaches like RLHF. Our arguments are highly general and applicable across various domains. This distinguishes them from intuitions gained through reasoning or empirical observations on a narrow set of domains, which may be widely known but have limited generalizability. Given the strong enthusiasm for RLHF and its variants, our negative theoretical results could have a profound impact by challenging current beliefs about the promise of these methods and pointing toward a new direction that dramatically differs from the current one.

Specifically, the idea of building AI that can teach humans is relatively new to the alignment community. It was not mentioned in the popular survey on RLHF [1] (see list of proposed solutions in their section 4.2). Prior work that addresses the imperfection of human beliefs, such as Inverse Reward Design [2], relies on solutions based only on uncertainty. We observe that current alignment pipeline relies on manual effort to ensure that humans are aligned with reality. What if we can build AI systems to automate this process for us? (note that this is different from scalable oversight, which is about automating evaluation of AI systems). By illustrating this new idea, we aim to open up novel research challenges and opportunities, sparking new approaches to better address alignment.

Last but not least, the benchmark we built, while being simple, demonstrate principles to overcome the challenges in simulating human cognition-based communication and exposing the weaknesses of LLMs. Moreover, it is actually harder than it appears to be, as LLM performance on the benchmark remains imperfect.

[1] https://arxiv.org/abs/2307.15217

[2] https://arxiv.org/abs/1711.02827
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a new mathematical framework called "practical alignment" that formalizes several weaknesses of RLHF and proposes theoretical ways of addressing them. It also introduces a new toolkit for evaluating alignment scenarios. It also presents a study with a simulated human agent and an AI agent to illustrate the benefits of AI teaching humans during the alignment process.

Strengths:

- The importance of the problem the paper is trying to address
- A novel mathematical framework for describing alignment, which predicts weaknesses of RLHF
- A new toolkit, MindGrid, for analyzing alignment scenarios
- The idea that AI teaching humans is needed for practical alignment

Weakness:

- Lack of empirical evidence in favor of the practical alignment framework, including evidence that practical alignment actually results in better alignment than existing alignment methods
- Lack of an algorithm for practical alignment
- Experiments only in simulated settings

While the paper introduces some interesting ideas, the metareviewer thinks it is not ready for publication, for reasons somewhat similar to reviewer *yQLP*'s. Namely, all strict mathematical models of human behavior make abysmally inaccurate assumptions. This goes for the practical alignment framework as well:

- It assumes that alignment entails learning a reward function, whereas it doesn't. E.g., DPO doesn't learn one. This leads the rest of the paper to focus on rewards; the benefits of this are unclear.

- In the practical alignment formalization (section 3.2), the paper assumes the AEAP cognition model. It's not clear why this model is a good choice.

- In general, in its practical alignment formalization, the paper assumes that people have use linear additive utility. It's a common and mathematically convenient but wildly unrealistic assumption.

None of this means that practical alignment is a bad framework, but it does make it non-obvious that, when all its unrealistic assumptions are combined, practical alignment will work better than the ostensible one. The only way to find out is to run an extensive study, involving real humans, using an algorithm that optimizes practical alignment, but the paper doesn't propose such an algorithm. Moreover, the presented study is very artificial and doesn't involve real humans.

Note that this algorithm per se isn't a precondition for publication, but it seems indispensable for assessing the merits of the proposed framework, which *is* crucial for publication. As two of the reviewers have mentioned, currently the framework makes predictions that either well-known drawbacks of RLHF (contrary to the authors' claim in the rebuttal, one doesn't need mathematical formalisms to identify these drawbacks) or that the paper verifies in purely synthetic experiments (the benefits of AI teaching people during alignment). Moreover, it ignores the very common case of alignment failure where the human judge lacks the information to form an accurate preference and the AI doesn't have that information either, so it can't teach the human about it. All this makes the proposed framework interesting but too lacking in empirical support to be accepted at ICLR.