Online-to-Offline RL for Agent Alignment

Authors: Xu Liu, Haobo Fu, Stefano V Albrecht, QIANG FU, Shuai Li
Primary area: reinforcement learning
Keywords: reinforcement learning, agent alignment

Abstract:
Reinforcement learning (RL) has shown remarkable success in training agents to achieve high-performing policies, particularly in domains like Game AI where simulation environments enable efficient interactions. However, despite their success in maximizing these returns, such online-trained policies often fail to align with human preferences concerning actions, styles, and values. The challenge lies in efficiently adapting these online-trained policies to align with human preferences, given the scarcity and high cost of collecting human behavior data. In this work, we formalize the problem as *online-to-offline* RL and propose ALIGNment of Game AI to Preferences (ALIGN-GAP), an innovative approach for the alignment of well-trained game agents to human preferences. Our method features a carefully designed reward model that encodes human preferences from limited offline data and incorporates curriculum-based preference learning to align RL agents with targeted human preferences. Experiments across diverse environments and preference types demonstrate the performance of ALIGN-GAP, achieving effective alignment with human preferences.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729973577392

thread_type:
official_review

<thread_object>
creator: reviewer_qpd3

<content>
summary:
This paper conceptualizes a new problem setting for reinforcement learning, namely, the online-to-offline setting, which aims to align RL agents with targeted human values in offline data after extensive online training. The proposed method aims at solving challenges in this setting, e.g., agent unlearning caused by rapid reward shifting. Experiements on Game AI verify the effectiveness of the proposed method.

strengths:
- The paper proposes a novel problem setting, namely, online-to-offline RL. The authors clearly explain the distinction between online-to-offline and offline-to-online RL. The practical importance of the newly proposed setting is also well explained.
- The proposed method is simple and intuitive, and it well handles the issue of sharp reward shifting in the online-to-offline setting, which is demonstrated by extensive experiments.

weaknesses:
- The newly conceptualized setting is termed "online-to-offline", however the proposed method for this setting, namely, reward curriculum, needs access to the online environment in the offline phase, which seems inconsistent with the offline concept. Although, considering the object of this new setting, i.e., leveraging potentially insuficient offline data to align the online learned agent, it is somewhat reasonable to assume access to the online environment, this may still limit the application of the proposed method to some real-world scenarios that are truly offline in the second phase.
- The work only focuses on the Game AI context. It would be nice to see if the method works on more practical applications.

questions:
- What causes the agent's failure to learn if we don't explicitly specify "what is preferred by humans but not yet learned by the agent" by Eq. (5)? Since when the preferences of humans and the agent are aligned, $R_{\phi}(\tau)$ would be similar to $R_{\phi}(\tau')$ and so there'll be no significant reward shifting. The motivation to use RM score calibration therefore seems a bit unclear, in contrast to the use of curriculum approach.
- It would be interesting to look into why exactly the agent unlearns when the reward signal shifts rapidly. This reminds me of the recent studies of plasticity loss in the field. I think these work may help understanding this phenomenon.
- Figure 9 is not very persuasive for the purpose of showcasing the effectiveness of the aligning method, since there are only 4 pairs of static images. I think it would be nicer to accompany the paper with videos that compares the performances of different alignment methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730028398684

thread_type:
official_review

<thread_object>
creator: reviewer_ahtp

<content>
summary:
The paper  introduces ALIGN-GAP, an innovative framework that enables alignment of high-performing RL agents trained online with specific human preferences. This framework consists of two main components: a reward model trained on limited offline human data to capture preferences and a calibrated curriculum-based learning approach to gradually adjust the agent's goals from maximizing environmental rewards to human-aligned behaviors.
Through experiments in diverse game environments, the ALIGN-GAP approach demonstrates effective alignment across various human styles, achieving behavior that closely resembles human preferences.

strengths:
1) **Significance of the Problem**: The paper addresses a crucial and emerging issue in reinforcement learning — aligning agent behavior with human preferences, especially in cases where human data is limited and offline. By tackling the alignment problem in online-to-offline RL contexts, this work directly addresses gaps in current RL applications and sets a foundation for broader, practical deployment of RL agents in human-interactive settings.

2) **Well-structured and Methodologically Clear**: The paper is logically structured, with each section building effectively on the last. The clear delineation between problem formulation, related work, methodology, and experimental results enhances readability and comprehension, making it easy to follow the progression from theoretical motivation to practical implementation.

3) **Robust Experimental Results**: The experimental results demonstrate the effectiveness of ALIGN-GAP across a variety of environments and human preference types, showcasing the method’s versatility. The use of both qualitative and quantitative evaluations provides a compelling case for the approach, with detailed analysis that underscores the alignment improvements made by ALIGN-GAP over baselines.

4) **Rationale Behind the Proposed Method**: The approach is intuitively sensible, leveraging a transformer-based reward model and calibrated curriculum learning in a way that aligns well with recent advancements in human-preference modeling. The stepwise, calibrated shift from performance-based to human-aligned behaviors ensures that the agent maintains learned skills while adapting effectively, reinforcing the practical value and feasibility of the proposed solution.

weaknesses:
1) **Preference Learning Paradigm Clarity**: In Section 4.1, the method constructs a preference learning framework where human trajectories are set as the “winner” samples and agent trajectories as the “loser” samples. This choice is somewhat unclear because agent trajectories may still exhibit successful behavior, albeit not fully aligned with human preferences. This setup raises concerns about whether such binary labeling could unintentionally degrade or "unlearn" the agent's effective skills. It would be beneficial if the authors could further clarify how the method differentiates between genuinely preferred human behavior and successful agent behavior to avoid compromising the agent's performance.

2) **Impact of Agent Pretraining**: Building on the previous point, it remains unclear how essential the agent pretraining phase is to the final performance. Would the method still be effective if the agent was trained exclusively with the human preference-aligned reward model, without prior environmental training? Additionally, an analysis of how different levels of pretraining influence the results would provide insight into the dependence on the agent's initial training stage and could help clarify the method’s requirements.

3) **Potential Time and Computational Costs**: The ALIGN-GAP approach may be computationally intensive due to the need for separate alignment processes for each human player’s unique preferences. This potential cost in terms of training time, especially for applications requiring frequent preference realignments, raises questions about scalability. It would be valuable for the authors to discuss potential optimizations, such as reusing alignment data across similar preferences or developing faster adaptation techniques to make this approach more time-efficient for practical use.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662806992

thread_type:
official_review

<thread_object>
creator: reviewer_rthm

<content>
summary:
The paper is well-written and clearly presents ALIGN-GAP, a simple method for aligning reinforcement learning (RL) agents with human preferences through an online2offline framework. ALIGN-GAP leverages a Transformer-based reward model to extract human preferences from limited offline data and utilizes a curriculum-based preference learning strategy to gradually shift the agent’s objectives from maximizing environmental rewards to aligning with human values. Extensive experiments across D4RL locomotion tasks and Atari games demonstrate that ALIGN-GAP effectively aligns agent behaviors with human preferences, outperforming several baseline methods. The thorough evaluations, including ablation studies, highlight the robustness and generalizability of the approach. However, the method lacks of theoretical foundations and also suffers from a limited range of experiments. While it performs well in the selected D4RL and Atari game environments, the scope of tasks is relatively narrow.

strengths:
The strengths of the paper include its clear and well-organized writing, which makes the proposed approach easy to follow. Additionally, the method itself is intuitive and relatively simple, combining straightforward components like a reward model and curriculum learning for preference alignment. This simplicity makes the approach accessible and potentially easier to implement, enhancing its appeal for practical applications.

weaknesses:
The method lacks of theoretical foundations and also suffers from a limited range of experiments. While it performs well in the selected D4RL and Atari game environments, the scope of tasks is relatively narrow.

questions:
1. **Data Requirements**: The method relies on high-quality offline human data to build a preference model. How sensitive is ALIGN-GAP to the quality and quantity of this data? Would noisy or sparse data significantly impact the model’s performance?
2. **Theoretical Guarantees**: The paper lacks a rigorous theoretical analysis. Are there any formal guarantees on the stability or convergence of the agent's alignment process, especially when switching between environmental and human-aligned rewards?
3. **Computational Cost and Efficiency**: Given the multi-component nature of ALIGN-GAP (e.g., reward calibration, curriculum learning), what is the computational cost of this method compared to simpler baselines? Could this limit its feasibility for resource-constrained applications?
4. **Evaluation Metrics**: The paper evaluates alignment through performance similarity to human proxies. However, this measure is somewhat subjective. Are there more objective or standardized metrics that could improve the evaluation of alignment quality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700556570

thread_type:
official_review

<thread_object>
creator: reviewer_9gas

<content>
summary:
The paper proposes an online-to-offline reinforcement learning (RL) framework, ALIGN-GAP, aimed at aligning RL agents’ behavior with human preferences, specifically within Game AI environments. This framework incorporates a transformer-based reward model to encode human preferences from limited offline data and a calibrated preference curriculum that incrementally transitions the agent's learning objectives from maximizing environmental rewards to prioritizing human-aligned behaviors. The experimental results across various game environments highlight ALIGN-GAP’s effectiveness in reducing the alignment gap between agents and humans.

strengths:
- The approach of fine-tuning a well-trained agent to match human preferences is novel and previously unexplored. The paper presents various challenges for this process and achieves strong performance through well-designed reward calibration and curriculum learning.

- Provides a theoretical justification for the reward calibration method, lending a solid foundation to the alignment strategy.

- Through comprehensive testing, ALIGN-GAP shows consistent improvement in agent-human alignment, proving its practicality and effectiveness in real-world scenarios like gaming.

weaknesses:
- I'm curious if fine-tuning a well-trained agent is necessary for aligning it with human preferences. Could training an agent offline, like AlignDiff [1], using the rollout and replay buffer data from the well-trained agent, be more advantageous?
Here’s my perspective in more detail:

  - Fine-tuning a well-trained agent to align with human preferences is challenging, particularly due to unlearning risks. While techniques the authors proposed like reward calibration and curriculum learning help, they may not fully solve the problem. Research (e.g., RLPD [2]) suggests that training a new agent using pre-collected data is often more stable and effective than fine-tuning.
  - Thus, instead of direct fine-tuning, using the well-trained agent’s replay buffer and rollout data to train a new, preference-aligned policy—either offline or online with offline data settings like RLPD—could be more effective.
  - The paper criticizes AlignDiff for needing manually designed attributes, but this seems inconsistent. For example, the MuJoCo experiments also rely on manually crafted attributes (such as health, movement, and cost), similar to AlignDiff. In Atari tasks, the paper relies on UMAP for unsupervised clustering of important attributes, but I think manually designing attributes might be simpler and more intuitive compared to relying on additional UMAP training.

  Therefore, it would be helpful if the authors could explain the specific benefits of the proposed method compared to using AlignDiff’s offline training (or an RLPD-like setting), supported by experimental results.

- Although the introduction of this paper suggests a focus on game AI settings, the experiments were primarily conducted on MuJoCo locomotion tasks and two Atari games. Expanding the range of experiments to include more varied game environments could help build a more cohesive argument.

[1] Dong, Zibin, et al. "AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model." The Twelfth International Conference on Learning Representations.

[2] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." International Conference on Machine Learning. PMLR, 2023.

questions:
- In Section 5.4, $\lambda$ in the expression for $\alpha(t)$ appears to be an important hyperparameter, but I couldn't find any information in the paper on how to tune $\lambda$. Could you explain how you tuned $\lambda$ for the experiments with the Linear / Exponential \$\alpha(t)$? 
- In equation (5), it seems that $R_{\text{calibrated}}$ might need to be scaled differently depending on the environment. Does it empirically perform well with just simple subtraction, without considering such adjustments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732265383725

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Updates in the Revised Version

comment:
We thank all reviewers for their valuable feedback and constructive suggestions, which have greatly helped us improve the quality of our work. Below, we summarize the main changes and additions in the updated version of our paper, addressing the points raised during the review process:

1. **Comparison with RLPD-like alignment without pretraining:**
We have added a detailed comparison with RLPD-like alignment approaches that do not utilize a pre-trained agent. In Appendix A.5, we demonstrate that ALIGN-GAP, by leveraging a well-trained agent for post-training alignment, achieves superior alignment efficiency and effectiveness compared to re-training from scratch. This supports ALIGN-GAP's focus on Game AI applications where pre-trained agents are commonly available and cost-efficient alignment is essential.

2. **Expanded experimental validation:**
While our main experiments focus on commonly used Game AI benchmarks (D4RL locomotion tasks and Atari games), we have extended the Atari experiments to include the Alien game, showcasing consistent advantages of ALIGN-GAP in Appendix A.3. The findings confirm ALIGN-GAP's consistent performance advantages across different environments. We also outline future plans to test ALIGN-GAP in more complex, application-oriented game scenarios.

3. **Computational cost analysis:**
In Appendix A.6, we provide a comparison of the computational overhead introduced by ALIGN-GAP components, such as reward calibration and reward curriculum. The result shows that these components add minimal overhead relative to the overall training process, ensuring ALIGN-GAP remains computationally efficient and feasible for practical applications.

4. **Enhanced visualization for alignment:**
To provide a more intuitive understanding of ALIGN-GAP's alignment performance, we have created videos corresponding to Figure 9 (shown in https://anonymous.4open.science/r/test2-08C1/video.mp4), showcasing the alignment of agents to diverse human preferences. In future revisions, we plan to include a comprehensive collection of video comparisons across additional environments and a broader range of human preferences.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers agree this work presents a novel and intuitive approach for fine-tuning pretrained game agents to align with offline human preferences. In particular, the reward curriculum is insightful and the experiments demonstrate that this approach makes a significant impact.