Online-to-Offline RL for Agent Alignment
Authors: Xu Liu
Haobo Fu
Stefano V Albrecht
QIANG FU
Shuai Li
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
agent alignment
Primary area: reinforcement learning
TL;DR: We propose an approach for aligning online-trained RL agents with human preferences.

Abstract: Reinforcement learning (RL) has shown remarkable success in training agents to achieve high-performing policies, particularly in domains like Game AI where simulation environments enable efficient interactions. However, despite their success in maximizing these returns, such online-trained policies often fail to align with human preferences concerning actions, styles, and values. The challenge lies in efficiently adapting these online-trained policies to align with human preferences, given the scarcity and high cost of collecting human behavior data. In this work, we formalize the problem as *online-to-offline* RL and propose ALIGNment of Game AI to Preferences (ALIGN-GAP), an innovative approach for the alignment of well-trained game agents to human preferences. Our method features a carefully designed reward model that encodes human preferences from limited offline data and incorporates curriculum-based preference learning to align RL agents with targeted human preferences. Experiments across diverse environments and preference types demonstrate the performance of ALIGN-GAP, achieving effective alignment with human preferences.

Published as a conference paper at ICLR 2025
RL FOR AGENT ALIGNMENT
Xu Liu Haobo , Stefano V. Albrecht', Qiang Fu², Shuai Li
Shanghai Jiao Tong University', Tencent", University of Edinburgh" shuailia e jtu..eu.dd edu. cn
(hacbofu, Leonfu etencent. com, ste. a www.aaeebraaalaeeee com com
ABSTRACT
Reinforcement learning (RL) has shown remarkable success in training agents achieve uihhpperfrmmig policies, particularly in domains like Game Al where simulation environments enable efficient interactions. However, despite their suc- cess in maximizing these returns, such online-trained policies often fail to align with human preferences concerning actions, styles, and values. The challenge lies in efficiently adapting these online-trained policies to align with human prefer- ences, given the scarcity and high cost of collecting human behavior data. In this work, we formalize the problem online-to- RL and propose ALIGNment of Game Al to Preferences (ALIGN-GAP) an innovative approach for the align- ment of well-trained game agents to human preferences. Our method features carefully designed reward model that encodes human preferences from limited offline data and incorporates curriculum-based preference learning align RL agents with targeted human preferences. Experiments across diverse environments and preference types demonstrate the performance of ALIGN-GAL achieving ef- fective alignment with human preferences.
INTRODUCTION
Modern reinforcement learning (RL) has achieved significant successes in optimizing strategies across diverse environments, driven by innovative algorithmic designs and the integration of deep learning techniques LLi 2017: Wang et al. 2022). Despite the high sample complexity typically associated with RL training, practical applications can mitigate online training costs and develop effective policies through approaches such as reward shaping (Schäfer et al.) (022) and pre-training on substantial offline datasets (Levine et al., 2020; Prudencio et 20233 Melnroe et al. 2024 An- dres al. 2024). In the realm of Game AI, the availability of cost-effective game engines facilitates efficient interactions, enabling RL agents to achieve high returns under carefully designed reward functions (Shao et al. 2018 Fan et al.) 2020). However, integrating these agents into real-world applications often uncovers a misalignment between the agents' behaviors, styles, and values and those humans, particularly given the varied value orientations of different individuals. As an ex- caline-trained agent learns behavioral policy based on given
ample illustrated in Figure [ reward function. However, diverse values can exist for human game players, e.g., a cautious be- ginner, speed-runner focused completion time, accomplishments. This diversity in values could lead to a variety of play styles and behaviors among different player groups. This discrepancy can make the agents appear unnatural from a human per- spective, necessitating alignment of agents with specific human preferences.
achievement hunter aiming for in-game
To align agents with human preferences in aspects such as human-like behaviors and action styles, it is essential to encode human values and preferences from behavior data, which may be signifi- cantly off-policy for the aaline-trained agents (Ziegler al., 2019). The high cost and scarcity of collecting human behavior data further hinder the effectiveness of direct fine-tuning processes. Fur- thermore, the variety of behavioral preferences, styles, and values among humans necessitates align- ment with specific groups or even individual players, intensifying the challenges associated with data collection (Gabriel, 2020). The inherent sample complexity of RL algorithms presents significant challenges in fine-tur tuning. Besides, the potential suboptimality of human- preferred behaviors, when assessed through the agents' value estimations, further complicates the process (Nguyen-Tame Tang et al.
human-preferred
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Online-Traiine Agent
Game Beginner
Speedrunner
Achievement Hunter
reward dhaping: -8-10 15, @0,0,
Slippery run: Navigate slippery surface from starting flag to reach the finish line X without falling into pits Inexperienced players may sometimes slide unpredictably, Collect stars i and open boxes @ (which have a 50/50 chance of containing star) for bonus achievements. Reaching the finish line or falling into pit ends the game.
An example for agent behavior (left) and diverse human behaviors (right) in grid-based game. A beginner prioritizes avoiding pitfalls; a speed-runner optimizes for quick completion; an achievement hunter seeks collect stars and open boxes for possible collectable items.
Figure 1:
2021 Additionally, the shift between the state-action distributions of human- numan--referred styles and those of RL agents can lead to "unlcarning" phenomena during fine-tuning, whereby agents lose their previously acquired understanding of environmental dynamics (Kumar al., 2020 Kostrikov et al. 2021] Nakamoto et al.) 2024). To address these challenges, we conceptualize the issue as an online-re coiineett-offfing offolii RL problem. Here, "online" refers to the process of achieving optimal near- optimal policies through extensive online RL training imitation of high-return trajectories, as eval- uated by reinforcement learning rewards from the environment. The "offline" component involves aligning these well-trained policies with human preferences using offline behavior data, which can often scarce. This framework contrasts with dfffine-to-onlle RL, where key distinctions can be summarized under main categories:
Assumptions regarding offfine data: Offline-to-online RL typically utilizes abundant but po- tentially suboptimal offline data, which provides extensive coverage of environmental dynam- ics (Kumar et al.] 2020 Kostrikov et al., (021). Online-to-offlin RI., however, focuses on offline preference data reflecting specifie human behaviors or values. This data is often limited and costly to collect, which may hinder its utility for fine-tuning RL policies.
Optimization objectives: The goal of offline-to-onlin RL to enhance online learning by utiliz- ing offline pre-training to address challenges in sample complexity (Zhang et al. [023] Hansen- Estruch et al. 2023; McInroe et al., 2024). In contrast, online-to-offlin RL aims to align online- trained agents with human preferences encoded in offline behavior data, prioritizing the integration human values into the learning process.
In this work, we introduce ALIGNment of Game AI to Preferences (ALIGN-GAP) aimed at align- ing RL agents with human styles and values within the Game Al context. Our method comprises two stages: first, inspired by techniques from large language model alignment, extract human preferences from offline human data by training a weweffomme--stuucuure reward model. This model trained with pair-wise sampling of state-action (sub- )trajectories from both online-trained agents and human data. Subsequently, to bridge the gap between the objectives of online learning and aligning with human preferences, we introduce calibrated preference curriculum learning strategy. This approach begins by calibrating the reward model scores to identify behaviors that align with human preferences but are not yet learned by the agent. It then gradually shifts the agent's objec- tives, moving from maximizing environmental returns to prioritizing human preferences encoded by the reward model. ALIGN-GAP achieves strong performance on D4RL. locomotion and Atari games, outperforming baseline fine-tuning methods in alignment on 17 of 21 tasks.
2 RELATED WORK
Online RL with offline data. Prior works have explored various strategies to utilize previously collected offiine datasets accelerate the training of online RI. algorithms. These approaches can
2

---

Published as a conference paper at ICLR 2025
generally be divided into two main categories: offline-to-onine RL and online RL with offline data. Offlinetto-onling RL involves pre- training a policy and a value function using offline data (Ku- mar et al.) 2020: Kostrikov et al. 2021; Agarwal et al.] 2020 Nakamoto et al.] 2024 Zhang et al. 2023), which allows for effective utilization of abundant yet sub-optimal offline datasets to establish well well-initialized policies. This reduces the need for extensive online exploration, thereby improving
sample complexity and overall efficiency (Prudencio et al. 2023 Levine et al. 22200 McInroe et al., 2024 Zhong et al. 2022). To address the distributional shift between the offline dataset and online interactions, several techniques such as balanced sampling (Lee et al. 2022] Guo et al. (023), adap- tive conservatism (Kostrikov et al.. 2021 Nakamoto et al., (024), and actor-critic alignment (Yu & Zhang, 2023 Wang et al. 2024b) have been proposed to enhance stability and performance. In contrast, online RL with offline data involves training an agent from scratch on top of a replay buffer filled with online data and offline data (Song et al., 2022), mixed with approaches such as imitation learning (Andres et al., 2024) and symmetric sampling (Ball et al. 2023). In our work, we seek to extend beyond improving agent performance by addressing how to align well-trained agents with human preferences using a limited amount of human offline data. Here, the challenge shifts from maximizing environmental returns aligning agents with specific human preferences, where offline data may be of high quality but insufficient for extensive RL fine-tuning.
Learning from human preferences. With the evolution of large language models (LLMs), align- ing AI capabilities with human values has become focal point of research. In the domain of RL. modern algorithms enable agents to learn effective strategies through extensive interaction with complex environments (Arulkumaran et al., 2017 Schulman et al.. 2017: Haarnoja et al.] 2018). A primary step for aligning these capable agents with human values involves representing and encod- ing human preferences, especially from pre-collected human data. Inverse reinforcement learning has been used to infer reward functions that could explain the behavioral patterns hidden within the data (Eysenbach et al.] 2020; Xue al. (021), although this often requires significant amount of demonstration data (Arora & Doshi 2021; Adams et al., 2022). A promising alternative is the development of reward models that encode human preferences with transformer-basee architec- tures (Guan et al. 2022; Dong al., 2023 Faal et al.) 2023 Kim et al. (023), which can help capture long contextual relationships and dependencies effectively through pair-wise sampling and 2022, Wang et al. 2023). These preference signals have found successful ap- plications in Al model alignment, particularly in LLM and vision language model alignment (Guan
training (Ouyang et al.
ct al.
2022
Liu et al. 2023 Wang et al. 2024b).
Aligning agents with humans. Popular methods for aligning fundamentally capable base models with human preferences include using reward models and proximal policy optimization (PPO) in reinforcement learning from human feedback (RLHF) framework (Achiam al.) 2023 Wang al. 2024c]a). By formalizing the alignment challenge as a bandit problem, initiatives like InstructGPT improved its alignment capability on issues such hallucination and toxicity (Ouyang al. 2022
RLHF has been adopted in various subsequent LLMS (Zheng et al.. 2023 Wang et al. 2024a Yang et al., 2024). Moreover, directly using offline human data through approaches like direct policy optimization (DPO) allows for supervised fine-tuning that aligns base models more closely with
human expectations of being helpful and harmless (Ratailov et al. 2024 Xu et al. 2024). Further explorations have relaxed the form of offline data (Ethayarajh et al. 2024) and integrated reward signals improve alignment (Zhong al. (024).. In the field of reinforcement learning, there has been an exploration of aligning agents with humans in domains such as autonomous driving and gaming (Wurman et al.) 2022 Dong et al., (023). However, existing approaches often require hand- engineering attributes for specific environments and struggle to provide unified formulation across different domains. In this work, we aim to identify the challenges and bottlenecks in aligning RL agents with humans through an onlinett--fffinee RL framework and propose alignment approach under this setting to establish general strategy for agent-human alignment.
3 PRELIMINARIES
3.1 REINFORCEMENT LEARNING
Reinforcement learning (RL) involves training an agent to make decisions by interacting with an environment in way that maximizes cumulative rewards over time. This is formalized within
3

---

Published as a conference paper at ICLR 2025
the framework of a Markoy Decision Process (MDP) PPteeran 1990), represented as M (S, A. p.r,p. where S.A denote the state and action spaces, p(s s. )) is the state transition probability, r(s, a.) represents the reward function, p(s) is the initial state distribution, and y E (0, is the discount factor which prioritizes immediate rewards over future ones.
The objective in RL is to learn policy S - (AA) that maps states probability distribution over actions in order to maximize the cumulative discounted reward starting from any initial state 8. This objective can be expressed through the value function:
V*(s) E (st))
Σ rrr (st,
|
(I)
The action-value function, or Q-function, further refines this by evaluating the expected return fol- lowing a specific action in state under policy
Q" (s, ((())
Σ r'r (5t, 0z) |so = s, =
(2)
and deep RL algorithms often use function approximators, such as neural networks, to estimate the Q-function and conduct policy learning. We denote the parameterized policy as To, where A represents the parameters of the network.
Despite the effectiveness of RL algorithms such as proximal policy optimization (Schulman et al., 2017) and soft actor-critic (Haarnoja et al., (018), these methods often suffer from high sample com- plexity (Wang et al., 2022), necessitating numerous interactions with the environment for conver- gence. In real-world applications, this requirement can be mitigated through engineering optimiza- tions that reduce the cost of environmental interactions, allowing the training of competent agents lower overall costs (Liang et al.] 2018). This is particularly viable in Game AI contexts, where inter- actions with the environment sampling (Vinyals et al. 2017 Guss et al.] (219)). Such setup facilitates the development of well- trained online policies at reduced costs, achieving high returns under specified reward functions and enhancing practical applications in real-world gaming environments (Jayaramireddy et al. 2022).
mediated by a game engine that enables rapid, large-scale parallel
3.2 REPRESENTING AND LEARNING FROM HUMAN PREFERENCES
crucial for effective integration to align agents with human users by accurately and efficiently identifying and representing human preferences. A traditional approach to understanding these pref- erences involves using inverse reinforcement learning (IRL) to infer reward functions that explain observed human behaviors (Ab Azar et al. 2020). However, IRL faces challenges such as high sample complexity, often requiring extensive demonstration data to accurately infer reward func- tions (Adams et al.) 2022. Liu et al. 2024 Additionally, IRI. can suffer from ambiguity, as mul- tiple reward functions can explain the same observed behavior, limiting its ability to reflect human preferences accurately (Arora & Doshi 2021).
Recent advancements LLM alignment offer a promising alternative for recognizing human pref- -truufrrrer-tttuuuueeddd reward model, preferences can be learned directly from data through pairwise comparisons of human choices (Ouyang et al.] [022]] Wang et al. 2023). The objective function for training such a reward model can be formulated as:
erences. By employing
(3) where r& represents the reward model parameterized by and (a. Vwo 31) denotes a sample triplet consisting of an input prompt x, preferred (winning) completion You, and a less preferred (los- ing) completion y/- The dataset D comprises such comparisons. This training objective leverages contrastive learning format, demonstrating effectiveness in encoding human preferences. The transformer- based architecture of the reward model further enhances its ability to scale, efficiently handling long sequences and complex structures (Zhong et al. 2024 Wang et al. 2024a).
L = - (().)~-)))> [log(a (2, Bw) - (2 y)))
4 WULINE-TO-OFFLAN RL FOR AGENT ALIGNMENT
In this section, we describe our approach to aligning trained reinforcement learning agents with human preferences using offline human data. The proposed approach utilizes two primary
4

---

Published as a conference paper at ICLR 2025
Model Weights
Initialization
Agent
Online Trained Agent
Human-Aligned Agent
Online Pretraining
Yeurriculum
Preference Curriculum Learning
RM Score Online Trained Agent
callibrated
Alignment Score
Online Environment
Calibration
Tenvy
TRM
Human Interaction
Agent Trajectories
State-Action Subsequence
000 0000 Human Players
Pairwise Sampling From Agent and Human
Transformer
Transformer-Sruccureed Structured
Online Game Environment
Human Trajectories
Figure 2: The framework of ALIGN-GAP for aligning RL game agent with human preferences. The gray section represents the training of the reward model, while the orange section represents the calibrated curriculum preference learning component.
strategies: First, we extract human preferences embedded in offline data by employing a transformer- based reward model (Section 4. Following the development of the reward model, we identify several challenges associated with aligning agents these extracted preferences. To address these challenges, we introduce a calibrated preference curriculum learning approach. This strategy focuses on identifying behaviors that align with human preferences but are not yet adopted by the agent and transitions the agent's focus from maximizing environmental rewards to aligning with human preferences (Section 4.2). The framework and process of our method are illustrated in Figure 2
4.1 PREFERENCE EXTRACTION FROM ONLINE AGENT AND OFFLINE HUMAN DATA
Extracting preferences with reward model. To align the behaviors and styles of an online- trained agent with human preferences, the initial step involves extracting these preferences from offline human data, particularly in comparison to the behaviors and styles of the online agent. In the realm of L.MM alignment, preference extraction can achieved by training a reward model, typically structured
transformer with linear head for predicting reward values (Achiam al,, 2023). This architecture effectively captures contextual information and provides robust preference predictions (Ouyang et al. 2022, Wang et al.) 2024a). In reinforcement learning, human behavioral preferences can be represented through sequential decision across various states. Similarly, transformer structure previously, the reward model in LLM alignment is trained using triplet data (æ, yor, 91), where represents a prompt, and you and yı denote the winning and losing responses, respectively. However, the scenario differs in reinforcement learning environments: instead of generating tokens based on a given prompt, the dynamics involve continuous updates of states and actions as the agent interacts with the environment (Li et al., 2023). To adapt this methodology, we train the reward model Rs for agent alignment using a similar loss function while constructing the input to consist of state- action sequences from offline human data and trajectories sampled from online agent interactions. Specifically, the loss function for training the reward model is defined as:
help capture contextual information within trajectories. As discussed
Lo = log P (Tuuman Tagent)
-E TTannan - Tura ~D [log(o(R,, (Rgg (Thuman) RR [agen)))
(4)
where Thuman = { {Shuman: Ohaman) }T represents a state-action sub-trajectory sampled from offline human data, and Tagent = {(Sagents aggent) ) is the sub-trajectory behaviors from the agent, which interacts with the environment using an opsilon-greedy strategy. Notably, unlike the environment reward function, the learned model R& does not take one single state-action pair as input but instead uses behavior sub-trajectories. This training approach is commonly referred to preference based reinforcement learning (PbRL) (Guan et al., [2022] Dong et al. 2023), building upon prior models but differing in dataset construction strategies and initial network configurations.
5

---

Published as a conference paper at ICLR 2025
4.2 CURRICULUM PREFERENCE LEARNING WITH CALIBRATED RM SCORE
After obtaining the reward model, a straightforward approach might be use it as the new reward function for fine-tuning the existing online well-trained agent. However, we find exper- imentally that this direct application often leads to sub-optimal performance, resulting in ineffective alignment with the pref- erences encoded the reward model and undesirable unlearn- ing of previously acquired behaviors (Nakamoto et al.) 2024). We believe this outcome is likely due to the significant shift in objectives for the agent, from maximizing environmental re- turns to aligning with human preferences. This motivates us
to develop method for smoother alignment of maline-trainne' agents with human preferences. From our experiments, we identify two main challenges:
Figure 3: Example of trajectories
from the online-trained agent and human achievement hunter.
The reward model pro- vides scores that reflect human preferences, but crucial pect is determining "what is preferred by humans but not yet learned by the agent." Given that the agent has undergone extensive online training, already has an understanding of environmental dynamics. Our goal is to guide the agent towards decisions that align more closely with human preferences. For instance, in Figure 3] while the reward model may assign high rewards human- preferred behaviors (indicated by the gray and blue trajectories), we aim for the preference signal used for alignment to prioritize actions that the agent has not yet learned adopt (shown by the blue trajectory) over those it has already mastered (the gray trajectory).
1) Identification of what to learn:
2) Mitigating sharp reward switching: Directly applying the trained reward model as a new re- ward function for reinforcement learning fine-tuning often leads to significant initial drop in per- formance, indicative of agent unlearning (Nakamoto et al.] 2024). This can occur due to the sharp transition from reward function that maximizes environmental returns to one that aligns with hu- man preferences, creating a substantial gap between the agent's learned Q-function and the optimal Q-function for preference alignment (Gleave et al.. 2020).
RM score calibration with initial behavior baseline. To address the first issue, we calibrate the reward signals used for preference learning to focus on aspects preferred by humans but not yet aligned by the agent. The reward for alignment should adapt to both human behavior and the style of the trained agent. We begin by evaluating the behavior of the online-trained policy using the reward model and using this assessment as baseline for calibrating the preference signal:
Realibmndd (T) = RT( - (T)()')
(5)
where 7 ~ [(s,, (s())) represents the state-action trajectory sampled from the current policy Te, and 7' is derived by replacing the actions in т with those taken under the same states by the during the alignment phase. This calibrated reward tends to assign higher rewards to human- human--referree preferred behaviors that the agent has not yet learned, while the rewards for behaviors already aligned with human preferences approach zero. Since the alignment phase's 's policy initially matches Tp. under calibrated rewards, positive rewards imply that the agent's alignment is at least not worse than that of the mnline-trained policy.
initial policy
Reward curriculum for smoothed preference learning. To address the second issue, we ob- served that even with calibrated rewards, the distribution of the reward function during the agent's online training can differ significantly, potentially leading to agent unlearning. In reinforcement learning, transitioning between different reward functions often involves transfer learning strategies to ensure smooth adaptation. It is important to note that transfer reinforcement learning typically addresses similar reward functions, such as perturbations of a single reward function (Zhu et al.) 2023). and requires careful design to prevent unlearning when faced with significantly different reward functions (Gleave et al. 2020).
Considering that the agent is already well-trained to maximize environmental returns, preference learning presents a novel and potentially more challenging task. Therefore, we can design reward curriculum that gradually transitions the task represented by the reward function from maximizing

---

Published as a conference paper at ICLR 2025
environmental returns to learning preferences:
Reurriculum = (1 - a(t) Tew + a(t) Realibrated,
(6)
where Tewv is the immediate reward feedback from the environment, Relibraed is the calibrated re- ward defined above, and a(t): / [0, balances the ratio of environmental rewards and calibrated RM rewards, which can usually grow from O to 1. For instance, a linear transition can be defined as (t) eurrent see This reward curriculum facilitates gradual transition from the familiar task lotal step of environmental reward maximization to the novel task of human preference alignment, offering general solution for the learning process. We provide analysis from theoretical perspective in Ap- pendix A.2 for the effect linear preference curriculum. In Section we explore the effectiveness of different strategies of adjusting att) for reward curriculum design.
EXPERIMENTS
In this section, aim to address the following key questions through experiments: (i) How effee- tively can our proposed method align online agents with human preferences represented with oracle rewards (Section5) 5.2 (ii) How effectively can our proposed method align with human preferences from unsupervised classified human data (Section 5.3) ? (iii) How do various components of our method, such as reward model calibration and preference curriculum, impact the overall alignment performance (Section 5.4)?
5.1 EXPERIMENTAL SETUP
Environments and human preferences. In our experiments, we evaluate various environments and constructions of human preferences. Specifically, we conduct experiments the D4RL loco- motion tasks, including HalfCheetah, Walker2D, and Hopper environments (Fu et al. 2020). These environments control-orieet -ntrol-oriented and resemble aetion-based gaming scenarios. For each environ- ment, we train online agents using the reward functions provided by the respective environments.
To construct different human preferences, design reward weights based the environment wards in the locomotion tasks. The default preference weights are set to 1,, 1]. reflecting three dimensions: maintaining health, moving forward, and controlling costs (Fu et al.) 2222)] We ex- plore four combinations of preference weights: A: [1, O, 1J. B:[1, 0J. 0), C: [1, 10, I]. and D: 11 1, 10). These combinations yield distinct reward oracles that are solely used to train human proxies and evaluate the aligned agents. After training the human proxies, we collect a small amount of human data (only 10 episodes) using the proxies for subsequent agent alignment. In Appendix A.I we provide a visual analysis of the different styles exhibited by various proxies across environments.
Additionally, we extend our experiments to Atari Pac-Man and Space Invaders games (Brockman, 2016). To assess the performance of our proposed methods against various constructions of human preferences, we utilize Google's ' Atari Replay Datasets (Agarwal et al. 2020) and conduct unsuper- vised classification on the dataset using UMAP (McInnes al. 2018). In each environment, we obtain three distinct preferences of human data, retaining only 10 episodes for each preference. The online-trainee agent is then aligned with the preferences encoded in the human data.
Comparisons, baselines, and evaluation protocol. For the D4RL locomotion tasks, we employ Soft Actor-Critic (SAC) (Haarnoja al. 2018) to train our online well-trained agents. In the Atari games, we utilize Deep Q-Networks (DQN) (Roderick et al., 2017) for online agent training. We compare our approach against a fine-tuning strategy with SAC (termed Finetune w/ SAC) or DQN (termed Finetune w/ DQN) that leverages offline data alongside the online training process. Specifi- cally, we adopt symmetric sampling, where we sample 50% from the replay buffer containing offline data and 50% from online data. This method has been shown to yield better results compared to ing only offline data (Ball et al. 2023).
Additionally, we evaluate baseline that trains the agent using both online environments and offline human data, mimicking DDPGfD (Vecerik et al.] (017)) but employing SAC (termed SACID) or DQN (termed DQNID) instead of DDPG, to match the algorithm used during the agent's online learning phase. Further details and hyperparameter settings can be found in Appendix [A.4 We also assess the effectiveness of Behavior Cloning (BC) (Torabi al. 2018) baseline, evaluating how
7

---

Published as a conference paper at ICLR 2025
Table 1: Returns for online trained agent and human performance, and differences in return to human for different alignment approaches. Results are averaged over 5 random seeds.
Tesl-Preference
Online Traned
Human Performance
Finelune SAC
Finelune
SACT>
Finetume RM
ALIGN-GAP
Halfchestabb-A
2.2+0:01
101.77e1.2
20.61=24.04
52,08401.57
4.07=0.7
-2.51=0008
20.660 31
Halfeheerah-B
Halfchoetah-C
103005
125. 3260).6
105.2743.3
-2.0841.78
3.00044
-31,2559.81
-37711.31
-100
-113-9813.14
-18.772-1
125.99:0.0
-14,9933322
30
40,4e199 197
Hulfchecttah-D
3296624855
-13.5e5.94
-109955-2.58
-11681=0.11
Walker2a.A
20. 00.7222210 Do
89.00110
2.9565.57
67.9120.5
-194561 41
2.550.03
2.2233.1
Walker2dd B
Walker2d-C
112.3300.0
122.1140.8
-1.5441.38
1.57.0.71
$1.21
50,88244766
9.5920.08
119.78-2.72
114.69.1
-74.4t11661
100
2.87226.06
-1.43a1.78
Walker2d-D
97,38=26.55
16.1400.5
10.42699,58
12
86.0665.99
-75 3..0660.19
-3.04221997
Hoppes-A
10.74e0.01
63,824 80,29
0.0146.01
79.2540.19
-25 33.3433.4)
-300000000
6.46+.556
Hoppor-al
6,7434.8
13122...83
-3.33322263
93.15.00
Hopper-C
15.1544444
.0.05+147
1.24+2.7
-7.71=11442
-23277+0.3
0.57+2.64
Hopper-D
110,0740,35
-18.57429.14
-0.22+21.58
-105.18-5.55
-78.14+0.02
-8.77=0 49
Table 2: Ranking the alignment degree of different alignment approaches by their similarity to human under oracle rewards (preferences) Results are averaged over 5 random seeds.
Tisk- Prefeereeeeeee
Online Trained
Finetune
Finetune
BC
SACID
BC
Finetune
ALIGN GAP
(Chrs)
ALIGN GAP
Calibrution
ALIGN GAP
Cariicluum
Hallchectab-A
Halfehertab-B
Rank
Rank
Rank 8
Rank
Rank
Rani
Rank
Rank
Rank
Hhllchhetab-CC
Rank
Rank
Rank 7
Rank
Rank
Rank
Rank
Rank
Rank
Rulfchent-D
Walke/25.A
Rank
Kank
Rank 5
Rank 6
Rank
Rank
Runk 2
Kank
Kank
WalkerrddB
Kank 9
Kank
Rank 6
Kank 5
Kank
Kank
Rank
Kank
Kank
Walker2d C
Rank
Kank
Rank 8
Kank
Rank
Rank
Rank
Kank
Kank
Walke:24DD
Hopper-A
Rank
Rank
Rank 5
Rank
Rank
Rank
Rank
Rank
Rank 4
Hopper B
Rank
Rank
Rank 5
Rank
Rank
Rank
Rank
Rank
Rank
Hopper-C
Hopper-D
Rank
Rank
directly applying cloning on limited human data aligns with human preferences (termed Finetune w/ BC for behavior cloning with policy initialized from online agent, and BC for behavior cloning from scratch). Moreover, we investigate the performance of agents aligned by directly using the trained RM as the reward function (termed Finetune w/ RM), comparing with our proposed approach that incorporates reward calibration and reward curriculum.
During evaluation, we first assess the effectiveness of agent alignment by examining the perfor- mance both the online-trained agents and human proxies within the context of environmental rewards. Specifically, we evaluate how well these agents perform in the environment by comparing their returns against those achieved by human players. For each alignment method, we quantify its success by measuring the performance discrepancy between the agents aligned using that method and the human proxies that serve as benchmarks for human-like behavior. This analysis enables us to evaluate the extent to which the performance of the aligned agents aligns with human preferences, as reflected in the differences in their environmental rewards.
Given that human preferences inherently subjective and relative, utilize the reward oracles from training the human proxies as representatives of human preferences for locomotion tasks. We assess the relative performance human proxy under the oracle rewards. Additionally, we visualize the trajectory styles exhibited by different methods in both locomotion and Atari tasks for subjective evaluation and comparison.
various alignment approaches by ranking their similarity the
5.2 ALIGNING AGENTS WITH PREFERENCE DATA FROM HUMAN PROXY
To assess the alignment effectiveness of the online well-trained agents with human preferences, we first focus on the performance gaps between the agents and human proxies within locomotion tasks. In [ we present the returns of both the online-trained agents (labeled "Online Trained") and the human proxies (labeled "Human Performance"). Additionally, for the baseline alignment approaches and ALIGN-GAP, the table displays the differences in returns against human under environmental rewards. Smaller differences this metric indicate that the agents perform a manner that can be more consistent with human behavior in the game environment. Our proposed method demonstrates smaller gaps across various environments and preference types compared to the baselines, highlighting its alignment capability.
8

---

Published as a conference paper at ICLR 2025
To evaluate the effectiveness of our alignment from the human perspective, that trained the human proxy to assess various agents. Table presents the rankings based on the differences in performance between the methods and the human proxy under oracle rewards. A higher ranking indicates that an agent's behavior aligns more closely with the human preferences implied by the oracle rewards. ALIGN-GAP significantly outperforms other methods, achieving top rankings across various environments and preference settings. To further validate the reliability compare agent trajectories generated by different alignment methods against
the reward oracle
of these rankings, those of the human proxy. This qualitative analysis is complemented by quantitative evaluation using Spearman's 's rank correlation coefficient (Sedgwick) 2014). Spearman's 's coefficient measures the degree of rank-order correlation between two variables, with value of 0.87 observed between rankings derived from oracle rewards and human evaluations. This high correlation substantiates the alignment results reported in Table 2 An example of different human proxy preferences and the styles learned by BLIGN-GAP is shown Figure of Appendix A.I
5.3 ALIGNING AGENTS WITH UNSUPERVISED CLASSIHED HUMAN DATA
In the Atari tasks, we focus aligning the online well-trained agents with unsupervised classified human data, processed through UMAP to represent the diverse preferences inherent within this dataset. Evaluations based on performance gaps in environmental rewards similar to those outlined in Table are listed in Appendix A.3
To further assess the manifestation of human preferences in clustered data and the alignment effi- cacy of ALIGN-GAR Figure (4a presents the UMAP clustered results for Space Invaders. Besides, conduct qualitative evaluations by visualizing both the categorized human preferences and the behaviors learned by ALIGN-GAP, with Space Invaders serving as an example (as depicted in Fig- ure 5). Observations reveal that while the online-trained agent typically adopts strategy focused on climinating the majority of invaders, human players exhibit varied approaches. For example, some may focus on attacking specific columns or strategic areas the screen. Style players might prioritize attacking the second and fourth columns of invaders, indicative of a patterned approach to game-play play, whereas Style 2 players may target invaders predominantly on the left side, perhaps reflecting& a preference for handling imminent threats specific game-play strategy.
5.4 ANALYSIS ON THE COMPONENTS OF ALIGN-GAP
Ablation Study Table shows the performance of our proposed method without RM score cal- ibration or preference curriculum. The results clearly indicate that the alignment effectiveness is significantly diminished in the absence the curriculum, demonstrating substantial impact on performance. This highlights the critical role of both the calibrated preference curriculum and RM score calibration in ensuring robust and effective alignment of agents with human preferences.
Additionally, regarding the design of (t()) in the pref- erence curriculum, we explore alternative adjustment methods beyond the linear approach. Specifically, evaluate an exponential form of at) = cxp(-X-
Table Comparing different att) for pref- erence curriculum learning on HalfChee- tah environment. The evaluation metrics
lotal sotalstep ), where À = 5 is a positive constant to en- sure that (t) follows the trend of increasing from 0 to approaching The results for the HalfCheetah envi- ronment are presented in Table The findings indi- cate that both linear and exponential adjustments yield similar performance, demonstrating the robustness of the preference curriculum method.
cureetssep))
remain consistent with Table 1
Preference
Linear
Exponential
A
-0.86+0.33
-0.95+0.48
B
-0.44+1.97
-0.93+0.76 76
C
-0.91+0.86
10.80+1.13
D
-7.340.29
-7.9=0.14
What is learned from the reward model? After training the reward model, evaluating its effectiveness in reflecting human preferences embedded in the data essential. To do this, we assess how well the reward model's predictions align with human preferences using trajectories from unseen test set. Specifically, we analyze whether the model accurately assigns preferences to these trajectories consistent with ground truth expectations. The results, illustrated in Figure demonstrate that the

---

Published as a conference paper at ICLR 2025
Style
10.0
Moan Towerd M4224
Soya2
(a) UMAP visualization of col- lected offline data for Space In- vaders.
(b) RM score ovaluations on differ- ent preference types of Walker2d.
(c) The reward distribution
leameed by the reward model.
Figure 4: Analysis of reward distribution for RM and offline human data distribution.
(a) Online Trained
(h) Style
(c) Align w/ Style
(d) Style 2
(e) Align w/ Style
Figure 5: State of remaining invaders at the end of Space Invaders game for online-trained trained agent, two styles offline behavior data, and the corresponding ALIGN-GAS LLGG-AAP aligned agents. All episodes used the same random seed initialize the episode and game engine.
model successfully prefers human trajectories over those of the online-trained agent, indicating that it effectively captures the nuances of human-like behavior.
Moreover, to visually explore the stylistic differences between human data and online-trained agent trajectories, we utilize PCA (Maćkiewiez & Ratajezak 1993) for dimension reduction both datasets. We then create heat map of the reward model's scores by querying the model with data points reconstructed using the inverse mapping from PCA. Presented in Figure 4c this visualization displays the reward model's 's distribution of scores across the reduced dimensional space. The PCA highlights clear distinctions between the human data (depicted blue points) and the online-traines agent data (shown in red points). Notably, areas of proximity in this space mainly reflect similari- ties between agent and human behaviors during the initial stages of episodes. The heat map further reveals that the reward model assigns higher rewards points closely resembling human trajectory data and lower rewards to those more aligned with the behaviors exhibited by the online- trained agent. This pattern emphasizes the reward model's ability to effectively discriminate and prioritize human-like behaviors, confirming its capacity to guide agents towards more human-aligned actions and decision-making styles.
CONCLUSION, LIMITATIONS AND FUTURE WORK
In this paper, we formulate the problem of online-to-offline RL for agent alignment and introduce ALIGN-GAP align agents with human preferences game environments. While ALIGN-GAP demonstrates strong alignment capabilities, we believe that achieving more fine-grained alignment is possible by carefully designing the calibration for specific environments. Our experiments focus on gaming due to the availability of trained agents and preference data, and future work may extend to broader domains such as autonomous driving, healthcare, and multi-agent systems, where aligning Al with human values remains crucial.
ACKNOWLEDGMENTS
The corresponding author Shuai Li is sponsored by CCF-Tencent Open Research Fund.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Nematollah Ab Azar, Aref Shahmansoorian, and Mohsen Davoudi. From inverse optimal control to inverse reinforcement learning: A historical review. Annual Reviews Control, 50:1 13-18,, 2020.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
Stephen Adams, Tyler Cody, and Peter A Beling, A survey of inverse reinforcement learning. Artificial Intelligence Review, 53(6):4307-4346 2022.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International conference machine learning, pp. 104-114. PMLR, 2020.
Alain Andres, Lukas Schäfer, Esther Villar-Rodriguez, Stefano V. Albrecht, and Javier Del Ser. Using offline data to speed-up reinforcement learning in procedurally generated environments. Neurocomputing, 2024.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, meth- ods and progress. Artificial Intelligence, 297:103500) 2021.
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep rein- forcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
G Brockman. Openai gym. arXiv preprint arXiv:1606.015400 2016
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lx, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via hehavior-custommsable diffusion model. arXiv preprint arXiv:2310.02054, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arX1v:2402.01306, 2024.
Ben Eysenbach. Xinyang Geng, Sergey Levine, and Russ R Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. Advances in neural information process- ing systems, 33: 13:14783-14799 2020.
Farshid Faal, Ketra Schmitt. and Jia Yuan Yu. Reward modeling for mitigating toxicity in transformer- based language models. Applied Intelligence, 53(7):8421-8435, 2023.
Xueman Fan, Wu, and I. Tian. A review of artificial intelligence for games. In Artificial Intelligence in China: Proceedings of the International Conference on Artificial Intelligence in China, pp- 298-303, Springer, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
lason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411-437, 2020.
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions. arXiv preprint arXiv:2006.. 13900, 2020.
Lin Guan, Karthik Valmeckam, and Subbarao Kambhampati. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences. arXiv preprint arXiv:2210. 15906, 2022.

---

Published as a conference paper at ICLR 2025
Siyuan Guo, Lixin Zou, Hechang Chen, Bohao Qu, Haotian Chi, S Yu Philip, and Yi Chang. Sample efficient offline-to-onie reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 2023.
William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: large-scale dataset of minecraft demonstrations. arXiv preprint 19t.:v:907..34444 2019.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.0590) 2018.
Philippe Hansen-Fstruch, llya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Charitha Sree Jayaramireddy, Sree Veera Venkata Sai Saran Naraharisetti, Mohamad Nassar, and Mehdi Mekni. A survey of reinforcement learning toolkits for gaming: applications, challenges and trends. In Proceedings of the Future Technologies Conference, pp. 165-184. Springer, 2022.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Pref- erence transformer: Modeling human preferences using transformers for rl. arXiv preprint arXiv:2303.00957, 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2 169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic g-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1001.0727. 2017.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi Aahi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, 2023.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pp. 3053-3062. PMLR, 2018
Guiliang Liu. Sheng Xu, Shicheng Liu, Ashish Gaurav, Sriram Ganapathi Subramanian, and Pas- cal Poupart. A survey of inverse constrained reinforcement learning: Definitions, progress and challenges. arXiv preprint (tt::24009005669 2024.
Yang Liu, Yuanshun Yao, Jean-Francodd Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: survey and guideline for evaluating large language models' alignment. arXiv preprint 23Xi:23308.0374, 2023.
Andrzej Maćkiewicz and Waldemar Ratajezak. Principal components analysis (pea). Computers & Geosciences, 19(3):303-342, 1993.
Leland McInnes, John Healy, and James Melville. Umap:
Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: /802.0040 2018.
Trevor Mclnroe, Adam Jelley, Stefano V. Albrecht, and Amos Storkey Planning go out-of- distribution in offline-to- to-oinee reinforcement learning. In Reinforcement Learning Conference, 2024.
12

---

Published as a conference paper at ICLR 2025
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36, 2024.
Thanh Nguyen-Tanne Sunil Gupta, Hung Tran- The, and Svetha Venkatesh. Sample complexity of offline reinforcement learning with deep relu networks. arXiv preprint arXiv:2103.066771 2021.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Martin I. Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Melrose Roderick, James MacGlashan, and Stefanic Tellex Implementing the deep q-network.
arXiv preprint arXiv:1711.07444 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1eXiiv1707.066. 2017.
Lukas Schäfer, Filippos Christianos, Josiah P. Hanna, and Stefano V. Albrecht. Decoupled rein- forcement learning to stabilise aurinsically-motivvtedd exploration. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2022.
Philip Sedgwick. Spearman's rank correlation coefficient. Bmj. 349, 2014.
Kun Shao, Yuanheng Zhu, and Dongbin Zhao. Starcraft mieromanagement with reinforcement learning and curriculum transfer learning. IEEE Transactions on Emerging Topics Computa- tional Intelligence, 3(1):73-84 2018.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06711, 2022.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954. 2018
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning robotics problems with sparse rewards. arXiv preprint arXiv:1707.088/1 2017.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schritwieser,, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arX8v:
2017.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint XXX::2401.06080 2024a.
Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get a family: adattve balances for dffine-to-onling reinforce- ment learning. Advances in Neural Information Processing Systems, 36, 2024b.
13

---

Published as a conference paper at ICLR 2025
Xu Wang, Sen Wang, Xingxing Liang, Dawei Zhao, Jincai Huang, Xin Xu, Bin Dai, and Qiguang Miao. Deep reinforcement learning; A survey. IEEE Transactions on Neural Networks and Learning Systems, 35(4)55044-0035 2022.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey, arXiv preprint arXiv:2307.12966 2023.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al. A comprehensive survey of Ilm alignment tech- niques: Rlhf, rlaif, ppo, dpo and more. arXiv preprimi arXiv:2407. 16216, 2024c.
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Out- racing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223- 228, 2022.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? comprehensive study. arXiv preprint arXiv:2404. 10719, 2024.
Wengian Xue, Patrik Kolaric, Jialu Fan, Bosen Lian, Tianyou Chai, and Frank Lewis. Inverse reinforcement learning in tracking control based on inverse optimal control. IEEE Transactions on Cybernetics,
10581, 2021.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint M.Xiv:2407.106711 2024.
Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-onling reinforcement learning.
In International Conference on Machine Learning, pp. 40452-40474, 40474. PMLR, 2023.
Haichao Zhang. We Xu, and Haonan Yu, Policy expansion for bridging offline-to tonine-t--onliee reinforce- ment learning. arXiv preprint 2xX1v:2332..00335 2023.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04960 049644 2023.
Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922 2024.
Rujie Zhong, Duohan Zhang, Lukas Schäfer, Stefano V. Albrecht, and Josiah P. Hanna. Robust on- policy data collection for data efficient policy evaluation. In Conference on Neural Information Processing Systems, 2022.
Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08599 2019.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 STYLES OF HUMAN PREFERENCES HALFCHERTAH, WALKER2D AND HOPPER
(a) HallCheetah Style A
(h) HalfCheetah Style B
(c) HalfCheetah Style C
(d) HalfCheetah Style
Figure 6: HalfCheetah Styles.
(a) walker2d Style A
(b) walker2d Style B
(c) walker2d Style C
(d) walker2d Style D
Figure Walker2d Styles.
In Figure 6 7 and 8 we showcase examples different styles exhibited by human proxies trained
under various reward oracles. In different environments, human styles may manifest in movement
15

---

Published as a conference paper at ICLR 2025
(a) hopper Style A
hopper- Style B
(c) hopper Style C
(d) hopper Style D
Figure 8: Hopper Styles.
speed, special postures during movement, evasive action styles in special situations, and so on. Using Walker2D and Hopper environment example, demonstrate the distinctions between different human styles in terms of specific stylistic features, as well as ALIGN-GAR learning effectiveness for these styles, and example is shown in Figure D
In Figure 9 we showcase the varied styles of human preferences in the Walker2D and Hopper envi- ronments and what is learned with offline data of those preferences by ALIGN-GAP The Walker2D examples illustrate distinct running styles: one proxy exhibits a long-stride running pattern, char- acterized by extended leg movements cover more ground per stride, while another demonstrates a forward-leaning running style, emphasizing speed and forward momentum with an aggressive stance. In the Hopper environment, contrasting jumping styles are evident; one style marked by large-amplitude jumps that prioritize vertical movement and height, whereas the other features small-step jumping, focusing on stability and control with shorter, more frequent leaps. We also conduct visualizations on the styles learned by the agent with ALIGN-GAP showing similar styles to that of human. can be seen that by using ALIGN-GAP for agent alignment, the agent could recognize the preference of human and learn from human styles in offline data improve its behav- iors.
A.2 THEORETICAL JUSTIFICATION FOR REWARD CURRICULUM OF ALIGN-GAP
In this section, provide analysis from theoretical perspective the effect of linear a(t) for preference curriculum, focusing on its effect for bridging two reward functions, the reward function Rew from the environment and the reward function Rauman for measuring the alignment degree of the agent. The analysis is first motivated from the reward function measure EPIC (Gleave et al. 2020). where we can measure the "distance" between two reward functions.
Definition A.1 (Canonically Shaped Reward). Let R S x Ax S + R be a reward function. Given distributions Ds ASS) and A E (AA) over states and actions, let and S' be random variables independently sampled from Ds and A sampled from A. Define the canonically shaped R to be:
CDs.PA (s, a, 8') = R (s, 8') + E R (8', A, R R(s, A,S') RR (S, A, 5
(7)
Based on the canonically shaped reward, we can define the EPIC pseudometric that measures the distance between reward functions (Gleave al. 2020
16

---

Published as a conference paper at ICLR 2025
ALIGN-GAP
Human Data
Style
Aligned Agent
Walker2D
Long-stride running
Walker2D
Forward-leaning
running
Hopper
Large-amplitude
jumping
Hopper
Small-step jumping
Figure 9: Different human styles in Walker2d and Hopper environment and the aligned agent by ALIGN-GAP. All episodes used the same random seed to initialise the episode and game engine.
Definition A.2 (Equivalent-Pole Policy Invariant Comparison (EPIC) pseudometric). Let be some coverage distribution over transitions s S s'. Let S, A, be random variables jointly sampled from D. Let Ds and D be some distributions over states S and A respectively. The Equivalent- Policy Invariant Comparison (EPIC) distance between reward functions RA and R is:
DEPIC (RA, RB) D, (CDs. (RA) (S, S') CDS.PA (RB) (S,),S)).
(8)
Previous work shows that the Equivalent-Polic Invariant Comparison distance is pseudometric, and that let RA, RB S x A x - IR be reward functions, then 0 < DEPIC (RA, RB) < 1. Now will prove the main result for showing the reduction in distance between the environment rewards and the alignment target. by introducing linear reward curriculum between them.
Theorem A.3. Denote the reward function for the online game environment as Renv and the re- ward function implied and learned from the reward model and calibrated in ALIGN-GAP Rmuman- Assuming the reward functions are of unit variance (which can be realized in practice by normal- ization), then the EPIC distance for the constructed curriculum reward Ralign satisfies
DEPIC (Renv: Ralign)
E
S
DEPIC (Rewy, Rhuman
3
Proof. Considering the curriculum reward learning with parameter
Ralign = aRRan + (1 - a Rhuman-
(10)
where E [0. 1] is the curriculum hyper-parameter Denoting the canonically shaped reward for environment rewards and human rewards as Cenv and Chumans respectively, since the definition of canonically shaped reward maintains lincarity, have the canonically shaped reward for Ralign (denoted as Calign) as also a linear combination of Cony and Chuman- By definition we have
p(Conv. Chuman
v pl Canys Calign)
Do (Conv, Chuman)
Dp (Cenv Calign)
(11)
2
2
17

---

Published as a conference paper at ICLR 2025
Now will first analyze the relations between p(Ceuv, Chuman) and p(Ceav, Calign P (Ceny: Catiga) = p (Cenv: aCeav (1 ch)Chu)))))
cov (Cemy, aCenv + (1 hhmmmm)))))
CC Jc (1 (1-a)Craman
ad a0
Can + (1 a1-aa a:) cov (Ceavs Chuman)
a Can + (1 -0) + 2a(1 - (x) cov (Cenvs Chuman)
Ocen
amc + (11 a:ppl (Cenv Chuman) ) OC... Thamn
Oce a Car + +(1 (1-aa) a1) + 20(11 (x)p (Cenv: Charann) TCeo Chunn
a+ alpppppp (Ceny, Chaman)
Va² +1 - a)2 + 20 0x)p (Cenv, Chuman)
2 a + (1 - a (Ceav, Chuman)
(12)
Since a, (C (Ceme Chuman) E [0, 1), firstly we have (Cenv Calign) 2 P (Cenv, Chummn). ) Furthermore, by the definition the Pearson distance, can be seen that
Dp (Cenv, Catign) D, (Ceny, Chaman)
(Cenvy Calign) (Cenv, Chuman)
1 (1 at)p (Cene Churann)
(13)
1 (Convs Chaman)
v1 a.
Therefore, by the non-negativity of the Pearson distance, have
Dp (Conv, Calign)
S/' v1 - ad
2 3
E
S LE[VI-
(14)
Dp (Cene, Chuman)
Remark A.4. This result indicates that by introducing a linear reward curriculum, we can reduce the EPIC distance between different reward functions, thereby simplifying the learning difficulty of the agent's 's alignment process. For other forms of curricula, such as the exponential curriculum we also experimented with, similar proof processes can employed evaluate their potential advantages.
Moreover, given that there is typically a significant gap between the reward objectives for alignment with humans and the inherent reward objectives of the environment, we can derive the following inference under the assumption that these two are uncorrelated.
Corollary A.5. If Rewy and Rhoman are uncorrelated (i.e., cov (Rew: Rnuman) = 0), then the curricu- lum reward learning in ALIGN-GA GAP mitigates the EPIC distance by nearly 50%.
Proof. When the environment rewards and human preference rewards learned by the reward model uncorrelated, we have
DEPIc (Rew, R V1 CeCen Chuman )
22
(15)
22
2
which is the largest EPIC we could have for two different reward functions. In this case, from the deviation in Eq. (1). the relation between the aligned reward ALIGN-GAP and the environment reward are now simpler:
P (Cenv, Calign)
for [0, 1].
(16)
V +
a)
18

---

Published as a conference paper at ICLR 2025
Therefore, we have
1 E [DEPiC (Cene, Calign)) V2 Lo
da 0.371
(17)
V
a) o
and ratio of the expected EPIC distance is
DEPIC (Reny, Ratiga)
IE [DEPIC (Renv: Raliga) 23 52.5%.
(18)
DEPIC (Heav, Rhuman)
DEPIC (Reme, Ryuman)
A.3 ATARI RESULTS
In this section, evaluate the effectiveness of different alignment methods in Atari domain (Pac- man, Space-invaders and Alien), and the result shown in Table Specifically, we analyze the performance of the online-trained agents and the performance of offline human data under various preferences. It is noteworthy that the offline data sampled within this dataset generally shows lower performance compared the agents trained online, reflecting gap between the two. For different alignment methods, we assess and record the difference in performance between the aligned agents and the human performance captured in the offline data. A smaller difference suggests that the aligned agent's behavior is more similar to human performance from a performance perspective. However, important note that similarity performance does not necessarily imply that the agent's style aligns with human preferences. In Section 5.2 we visually compare the styles of humans and the aligned agents to determine if the agents are well well-aligned with human preferences.
Table Returns for online trained agent offline data performance, and differences in return to offline data for different alignment approaches. Results are averaged over 5 random seeds.
Preference
Online Trained
Offiie Trejectory
Finetune DQN
DQNID
Finetune
ALIGN- GAP
Pac-Man-A
148112
13902175
518287
S102236
Pac:
Pac-Man- Man B
2252a320
207a30
16362808
$461246
144:56
462+197
132+25
228114
289144
192a32
174138
197240
52421
Spece Invaders
656=108
431+38
188485
152468
180662
60s38
96
Spece-Invaders-e
556+37
163452
441422
.91447
-73426
36t
Alien-A
400H33
1731#92
468261
328476
178+28
36E19
Alien-B
3069#284
740:24
1890274
565287
236120
147:50
37230
Alien:C
982476
1295484
480s46
241429
136:24
42428
A.4 HYPER- PAPEE-PARRMETERS
The hyper-parameters and values are summarized in Table 5
Table 5: Hyper-parameters for agent alignment with ALIGN-GAP and baselines.
Hyper- Hyper-Parameee Parameters
Values
Online Pre-train Steps
5e6
Offline Alignment Steps
le6
SAC Actor Learning Rate
3e-4
SAC Critic Learning Rate
3e-4
SAC Batch Size
256
Offline Trajectory Number
10
DQN Learning Rate
5e-5
DQN Target Update Interval
10000
DQN Batch Size
32
Reward Model Sequence Length
64
Reward Model Latent Dim
256
Reward Model Training Batch Size
64
Reward Model Learning Rate
le-4
19

---

Published as a conference paper at ICLR 2025
A.5 COMPARATIVE STUDY WITH RLPD-LIKE RE-TRAINING APPROACHES
To provide a comprehensive evaluation of ALIGN-GAP, we conducted comparative study against RLPD-like approaches that align agents without leveraging pre-trained models (Ball et al. 2023). In this approach, the agent is trained from scratch using a mix of offline human preference data and environmental interactions, and the rewards come from the trained reward model to encode human preferences. This comparison is particularly relevant in understanding the trade-offs between training a new agent versus fine-tuning an existing well-trained agent for human alignment.
In the RLPD framework, offline data and online data mixed according specified ratio dur- ing training. Following the recommendations in RLPD, we employed symmetric sampling, where offline and online data sampled equally (50% each). To ensure a fair comparison, evaluated RLPD under two settings:
1. le6 alignment steps: Equivalent to the number of steps used in ALIGN-GAP for post- training alignment.
2. (5e6 1e6) total steps: Adding the steps to get the pretrained agent (5e6) and alignment steps for ALIGN-GAP (le6 steps) to account for the total steps in ALIGN-GAP.
Experimental results shown in Table 6 reveal that at 1e6 steps, RLPD has not fully converged to achieve effective alignment, as must Simultaneously learn basic environmental behaviors and align with human preferences. ALIGN-GAP, by contrast, achieves alignment more efficiently by lever- aging the pre-trained agent's understanding of the environment, allowing it to focus exclusively on preference alignment. Furthermore, even when RLPD is trained for 6e6 steps, including pretraining and alignment, ALIGN-GAP still demonstrates superior alignment performance. This underscores ALIGN-GAP's efficiency and effectiveness in aligning with human preferences on the foundation of a well-trained agent. These results highlight ALIGN-GAP's practical advantages in scenarios where pre-trained agents available, such Game AI applications. In practical applications, retraining an agent from scratch can be prohibitively costly: therefore, ALIGN-GAP! approach of achieving alignment through post-tras post-training may hold greater practical value.
Table 6: Returns for online trained agent and human performance, and differences in return to human for RLPD and ALIGN-GAP. Results are averaged over random seeds.
Human
RLPD
RLPD (5e6+1e6 steps)
ALIGN-GAP (1e6 steps)
Task PeefPrefeeeecc
Performance
(1e6 steps)
Halfcheetah-A A
2.240.01
-1.9340.41
-1.35+0.38
-0.86+0.33
Halfchcetah-B
125.3220.76
-39.22+10880
-10.04+3.17
-0,44+1.97
Halfcheetah-C
105.27+3.30
-25.71+2.46
4.89+0.15
-0,9140.86
Halfcheetah-D
114.74=0.45
-13.0643.25
-6+1.27
-7.310.29
Walker2d-A
20.72+2.06
-5,00+1.79
-1.8440.96
2.2243.1
Walker2d-B
122.11#0.42
-39.5329.01
-14.4+2.03
-2.87426.06
Walker2d-C
116.53=0.74
-47.01+5.20
18.2645.78
-1.43+1.78
Walker2d-D
97.35+26.53
-32.1948.62
7.74+0.62
-304427.97
Hopper-A A
30.7420.01
-14.8+3.20
19.7740.15
6.46+1.56
Hopper-B
93.88+28.17
-13.7446.89
-12.56+2.30
-3.33+0.02
Hopper-C
31.1524.44
-18.70+2.71
-13.0621.91
0.5742.64
Hopper-D
110.07=0.35
-25.01+7.20
-14.4743.85
-8.7740.49
A.6 COMPUTATIONAL EFFICIENCY ANALYSIS
To evaluate the computational efficiency of ALIGN-GAP, we measured the training speed in itera- tions per second under different configurations, as shown in Table These results demonstrate that both reward calibration and reward curriculum introduce minimal computational overhead, affirming ALIGN-GAP's suitability for practical applications. The addition of reward calibration introduces minor computational cost, it requires additional forward pass through the reward model for each trajectory to compute calibrated scores. Despite this, the reduction training speed compared configurations without calibration only about 4.4%, highlighting the lightweight nature of this
20

---

Published as a conference paper at ICLR 2025
operation. Similarly, the reward curriculum dynamically adjusts the weighting between environmen- tal rewards and weeference-alignee rewards, but this computation is exceedingly efficient, involving only simple arithmetic operation at each training step. Consequently, the observed reduction in speed with the inclusion of curriculum is only about 0.5%, further demonstrating its computational feasibility.
The results show that ALIGN-GAP's computational cost is nearly identical to configurations omit- ting one or more of its components. Given the significant alignment improvements observed in our experiments, this slight reduction in training speed represents a reasonable trade-off. Reward calibration and reward curriculum are crucial components for guiding the agent towards human- preferred behaviors while maintaining stability in previously learned skills. Their lightweight design ensures ALIGN-GAP remains efficient even for large-scale training tasks. These findings empha- size the practicality of ALIGN-GAP in scenarios with computational constraints, such as real-world Game Al applications. Furthermore, in settings with severe resource limitations, additional opti- mization techniques like model quantization, pruning, parallelized computations can be employed to further reduce computational costs while preserving alignment quality. By prioritizing both align- ment performance and efficiency, ALIGN-GAP balances practical applicability with its innovative design.
Table 7: Iteration per second.
Configuration
Iteration Per Second
ALIGN-GAP
42.61+0.12 44.59=0.09
ALIGN-GAP w/o Reward Calibration
ALIGN-GAP w/o Reward Curriculum
42.83+0.11
21


Reviews:
[{"review_id": "JujTOCtPo9", "reviewer": "Reviewer_9gAS", "summary": "The paper proposes an online-to-offline reinforcement learning (RL) framework, ALIGN-GAP, aimed at aligning RL agents’ behavior with human preferences, specifically within Game AI environments. This framework incorporates a transformer-based reward model to encode human preferences from limited offline data and a calibrated preference curriculum that incrementally transitions the agent's learning objectives from maximizing environmental rewards to prioritizing human-aligned behaviors. The experimental results across various game environments highlight ALIGN-GAP’s effectiveness in reducing the alignment gap between agents and humans.", "questions": "- In Section 5.4, $\\lambda$ in the expression for $\\alpha(t)$ appears to be an important hyperparameter, but I couldn't find any information in the paper on how to tune $\\lambda$. Could you explain how you tuned $\\lambda$ for the experiments with the Linear / Exponential \\$\\alpha(t)$? \n- In equation (5), it seems that $R_{\\text{calibrated}}$ might need to be scaled differently depending on the environment. Does it empirically perform well with just simple subtraction, without considering such adjustments?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The approach of fine-tuning a well-trained agent to match human preferences is novel and previously unexplored. The paper presents various challenges for this process and achieves strong performance through well-designed reward calibration and curriculum learning.\n\n- Provides a theoretical justification for the reward calibration method, lending a solid foundation to the alignment strategy.\n\n- Through comprehensive testing, ALIGN-GAP shows consistent improvement in agent-human alignment, proving its practicality and effectiveness in real-world scenarios like gaming.", "weaknesses": "- I'm curious if fine-tuning a well-trained agent is necessary for aligning it with human preferences. Could training an agent offline, like AlignDiff [1], using the rollout and replay buffer data from the well-trained agent, be more advantageous?\nHere’s my perspective in more detail:\n\n  - Fine-tuning a well-trained agent to align with human preferences is challenging, particularly due to unlearning risks. While techniques the authors proposed like reward calibration and curriculum learning help, they may not fully solve the problem. Research (e.g., RLPD [2]) suggests that training a new agent using pre-collected data is often more stable and effective than fine-tuning.\n  - Thus, instead of direct fine-tuning, using the well-trained agent’s replay buffer and rollout data to train a new, preference-aligned policy—either offline or online with offline data settings like RLPD—could be more effective.\n  - The paper criticizes AlignDiff for needing manually designed attributes, but this seems inconsistent. For example, the MuJoCo experiments also rely on manually crafted attributes (such as health, movement, and cost), similar to AlignDiff. In Atari tasks, the paper relies on UMAP for unsupervised clustering of important attributes, but I think manually designing attributes might be simpler and more intuitive compared to relying on additional UMAP training.\n\n  Therefore, it would be helpful if the authors could explain the specific benefits of the proposed method compared to using AlignDiff’s offline training (or an RLPD-like setting), supported by experimental results.\n\n- Although the introduction of this paper suggests a focus on game AI settings, the experiments were primarily conducted on MuJoCo locomotion tasks and two Atari games. Expanding the range of experiments to include more varied game environments could help build a more cohesive argument.\n\n[1] Dong, Zibin, et al. \"AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model.\" The Twelfth International Conference on Learning Representations.\n\n[2] Ball, Philip J., et al. \"Efficient online reinforcement learning with offline data.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LGDAZhJJbE", "reviewer": "Reviewer_rThM", "summary": "The paper is well-written and clearly presents ALIGN-GAP, a simple method for aligning reinforcement learning (RL) agents with human preferences through an online2offline framework. ALIGN-GAP leverages a Transformer-based reward model to extract human preferences from limited offline data and utilizes a curriculum-based preference learning strategy to gradually shift the agent’s objectives from maximizing environmental rewards to aligning with human values. Extensive experiments across D4RL locomotion tasks and Atari games demonstrate that ALIGN-GAP effectively aligns agent behaviors with human preferences, outperforming several baseline methods. The thorough evaluations, including ablation studies, highlight the robustness and generalizability of the approach. However, the method lacks of theoretical foundations and also suffers from a limited range of experiments. While it performs well in the selected D4RL and Atari game environments, the scope of tasks is relatively narrow.", "questions": "1. **Data Requirements**: The method relies on high-quality offline human data to build a preference model. How sensitive is ALIGN-GAP to the quality and quantity of this data? Would noisy or sparse data significantly impact the model’s performance?\n2. **Theoretical Guarantees**: The paper lacks a rigorous theoretical analysis. Are there any formal guarantees on the stability or convergence of the agent's alignment process, especially when switching between environmental and human-aligned rewards?\n3. **Computational Cost and Efficiency**: Given the multi-component nature of ALIGN-GAP (e.g., reward calibration, curriculum learning), what is the computational cost of this method compared to simpler baselines? Could this limit its feasibility for resource-constrained applications?\n4. **Evaluation Metrics**: The paper evaluates alignment through performance similarity to human proxies. However, this measure is somewhat subjective. Are there more objective or standardized metrics that could improve the evaluation of alignment quality?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The strengths of the paper include its clear and well-organized writing, which makes the proposed approach easy to follow. Additionally, the method itself is intuitive and relatively simple, combining straightforward components like a reward model and curriculum learning for preference alignment. This simplicity makes the approach accessible and potentially easier to implement, enhancing its appeal for practical applications.", "weaknesses": "The method lacks of theoretical foundations and also suffers from a limited range of experiments. While it performs well in the selected D4RL and Atari game environments, the scope of tasks is relatively narrow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UkLexCPEBL", "reviewer": "Reviewer_AHtP", "summary": "The paper  introduces ALIGN-GAP, an innovative framework that enables alignment of high-performing RL agents trained online with specific human preferences. This framework consists of two main components: a reward model trained on limited offline human data to capture preferences and a calibrated curriculum-based learning approach to gradually adjust the agent's goals from maximizing environmental rewards to human-aligned behaviors.\nThrough experiments in diverse game environments, the ALIGN-GAP approach demonstrates effective alignment across various human styles, achieving behavior that closely resembles human preferences.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) **Significance of the Problem**: The paper addresses a crucial and emerging issue in reinforcement learning — aligning agent behavior with human preferences, especially in cases where human data is limited and offline. By tackling the alignment problem in online-to-offline RL contexts, this work directly addresses gaps in current RL applications and sets a foundation for broader, practical deployment of RL agents in human-interactive settings.\n\n2) **Well-structured and Methodologically Clear**: The paper is logically structured, with each section building effectively on the last. The clear delineation between problem formulation, related work, methodology, and experimental results enhances readability and comprehension, making it easy to follow the progression from theoretical motivation to practical implementation.\n\n3) **Robust Experimental Results**: The experimental results demonstrate the effectiveness of ALIGN-GAP across a variety of environments and human preference types, showcasing the method’s versatility. The use of both qualitative and quantitative evaluations provides a compelling case for the approach, with detailed analysis that underscores the alignment improvements made by ALIGN-GAP over baselines.\n\n4) **Rationale Behind the Proposed Method**: The approach is intuitively sensible, leveraging a transformer-based reward model and calibrated curriculum learning in a way that aligns well with recent advancements in human-preference modeling. The stepwise, calibrated shift from performance-based to human-aligned behaviors ensures that the agent maintains learned skills while adapting effectively, reinforcing the practical value and feasibility of the proposed solution.", "weaknesses": "1) **Preference Learning Paradigm Clarity**: In Section 4.1, the method constructs a preference learning framework where human trajectories are set as the “winner” samples and agent trajectories as the “loser” samples. This choice is somewhat unclear because agent trajectories may still exhibit successful behavior, albeit not fully aligned with human preferences. This setup raises concerns about whether such binary labeling could unintentionally degrade or \"unlearn\" the agent's effective skills. It would be beneficial if the authors could further clarify how the method differentiates between genuinely preferred human behavior and successful agent behavior to avoid compromising the agent's performance.\n\n2) **Impact of Agent Pretraining**: Building on the previous point, it remains unclear how essential the agent pretraining phase is to the final performance. Would the method still be effective if the agent was trained exclusively with the human preference-aligned reward model, without prior environmental training? Additionally, an analysis of how different levels of pretraining influence the results would provide insight into the dependence on the agent's initial training stage and could help clarify the method’s requirements.\n\n3) **Potential Time and Computational Costs**: The ALIGN-GAP approach may be computationally intensive due to the need for separate alignment processes for each human player’s unique preferences. This potential cost in terms of training time, especially for applications requiring frequent preference realignments, raises questions about scalability. It would be valuable for the authors to discuss potential optimizations, such as reusing alignment data across similar preferences or developing faster adaptation techniques to make this approach more time-efficient for practical use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JIsKuMImh6", "reviewer": "Reviewer_qPd3", "summary": "This paper conceptualizes a new problem setting for reinforcement learning, namely, the online-to-offline setting, which aims to align RL agents with targeted human values in offline data after extensive online training. The proposed method aims at solving challenges in this setting, e.g., agent unlearning caused by rapid reward shifting. Experiements on Game AI verify the effectiveness of the proposed method.", "questions": "- What causes the agent's failure to learn if we don't explicitly specify \"what is preferred by humans but not yet learned by the agent\" by Eq. (5)? Since when the preferences of humans and the agent are aligned, $R_{\\phi}(\\tau)$ would be similar to $R_{\\phi}(\\tau')$ and so there'll be no significant reward shifting. The motivation to use RM score calibration therefore seems a bit unclear, in contrast to the use of curriculum approach.\n- It would be interesting to look into why exactly the agent unlearns when the reward signal shifts rapidly. This reminds me of the recent studies of plasticity loss in the field. I think these work may help understanding this phenomenon.\n- Figure 9 is not very persuasive for the purpose of showcasing the effectiveness of the aligning method, since there are only 4 pairs of static images. I think it would be nicer to accompany the paper with videos that compares the performances of different alignment methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a novel problem setting, namely, online-to-offline RL. The authors clearly explain the distinction between online-to-offline and offline-to-online RL. The practical importance of the newly proposed setting is also well explained.\n- The proposed method is simple and intuitive, and it well handles the issue of sharp reward shifting in the online-to-offline setting, which is demonstrated by extensive experiments.", "weaknesses": "- The newly conceptualized setting is termed \"online-to-offline\", however the proposed method for this setting, namely, reward curriculum, needs access to the online environment in the offline phase, which seems inconsistent with the offline concept. Although, considering the object of this new setting, i.e., leveraging potentially insuficient offline data to align the online learned agent, it is somewhat reasonable to assume access to the online environment, this may still limit the application of the proposed method to some real-world scenarios that are truly offline in the second phase.\n- The work only focuses on the Game AI context. It would be nice to see if the method works on more practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)