Two-Step Offline Preference-Based Reinforcement Learning on Explicitly Constrained Policies
Authors: Yinglun Xu
Tarun Suresh
Rohan Gumaste
David Zhu
Ruirui Li
Zhengyang Wang
Haoming Jiang
Xianfeng Tang
Qingyu Yin
Monica Xiao Cheng
Qi Zeng
Chao Zhang
Gagandeep Singh
Venue: Accepted by TMLR ()

Abstract: Preference-based reinforcement learning (PBRL) in the offline setting has succeeded greatly in industrial applications such as chatbots. A two-step learning framework that learns a reward model from an offline dataset first and then optimizes a policy over the learned reward model through online reinforcement learning has been widely adopted. However, such a method faces challenges from the risk of reward hacking and the complexity of reinforcement learning. To overcome the challenge, our insight is that both challenges come from the state-actions not supported in the dataset. Such state actions are unreliable and increase the complexity of the reinforcement learning problem. Based on the insight, we develop a novel two-step learning method called PRC: preference-based reinforcement learning on explicitly constrained policies. The high-level idea is to limit the reinforcement learning agent to optimize over policies supported on an explicitly constrained action space that excludes the out-of-distribution state-actions. We empirically verify that our method has high learning efficiency on various datasets in robotic control environments.

Published in Transactions on Machine Learning Research (10/2025)
PBRL with Offline Reward Modeling on Explicitly Con strained Policies
Yinglun Xu', Tarun Suresh', Rohan Gumaste', David Zhu', Ruirui Li",
Zhengyang Wang', Haoming Jiang", Xianfeng Tang', Qingyu Yin?, Monica Xiao Cheng", Qi Zeng', Chao Zhang", Gagandeep Singh UUniveriity of Illinois Jrbana-Champaii "Amazon, GGorria institute of technology
Reviewed on OpenRevicw: https://ppaaaiieoeeee/////eoeted
Abstract
Preference-based reinforcement learning (PBRL) in the offline setting has succeeded greatly in industrial applications such as chatbots. A two-step learning framework that learns a reward model from an offline dataset first and then optimizes a policy over the learned reward model through online reinforcement learning has been widely adopted. However, such a method faces challenges from the risk of reward hacking and the complexity of reinforcement learning. To overcome the challenge, our insight is that both challenges come from the statc-actions not supported in the dataset. Such statc-actions are unreliable and increase the complexity of the reinforcement learning problem. Based on the insight, we develop a novel two-step learning method called PRC: preference-based reinforcement learning on explicitly constrained policies. The high-level idea is to limit the reinforcement learning agent to optimize over policies supported on an explicitly constrained action space that excludes the out-of-distribution state-actions. We empirically verify that our method has high learning efficiency various datasets in robotic control environments.
1 Introduction
Deep reinforcement learning (DRL) is fundamental learning paradigm for solving sequential decision- making problems and has rich real-world applications (Sutton & Barto 2018 Luong et al. 2019 Haydari
& Yilmaz 2020). However, traditional DRL approaches require numerical reward feedback during learning, which might be hard to design or obtain in practice. In contrast, binary preference foedback is usually more accessible (Wirth
Fürnkranz) 2013 Wirth et al. 2017 Fürnkranz et al.] 2012 Akrour el al. 2012 For example, to judge how well a robot dances, it is challenging rate each move in dance. However, is relatively casier to tell if piece of dance is better than another. As result, preferenco-based reinforcement learning (PBRL) that only requires preference feedback has become more realistic learning paradigm and is applied in real-world applications (Christiano et al. 2017]. Furthermore, as special instance of PBRL, reinforcement learning from human feedback (RLHF)), which utilizes human preferences, has drawn considerable attention and achieved significant success in many NLP tasks (Ouyang et al. 2022 Rafailov et al. 2023). Tn this work, we focus on PBRI. in the offline setting (Zhan et al., 2023 Zhu el al. 2023) where the agent trains on a pre-collected preference datasct on pairs of trajectorics. Offline learning is considered to be more practical as it is usually safer and more convenient to access an offline dataset (Levine)
et al. 2020 Prudencio et al. 2023 Cheng et al. 2022). Particularly, are interested in popular learning framework that is widely applied in various PBRL literature and industrial applications, which we call 'two-step learning.'
Two-step learning framework: In a typical two-step learning framework (Ibarz et al. 2018 Christiano et al.] 2017), the first lcarning step is reward modeling on an offinee preference dataset. The agent ap- proximates the reward model of the environment that can best explain the observations from the training preference dataset. The second learning step policy optimization, where the learning agent optimizes

---

Published in Transactions on Machine Learning Research (10/2025)
policy to achieve a high reward on the reward model learned in the previous step. This step is typically implemented by standard online reinforcement learning algorithms such PPO (Schulman et al. 2017 There are multiple reasons behind the vast popularity of the two-step learning paradigm: 1. similar to the motivation of inverse reinforcement learning (Arora & Doshi 2021), the learned reward model provides a succinct. and transferable definition of the lask; 2. the method modular in that each phase of learning can be implemented by existing well-developed methods. Despite its popularity, this method faces two main challenges for offline PBRL.
Reward sear--pptiiiatiio/Pesssiisss
learning: Pessimistic learning has always been the key chal- lenge in offline learning settings. Given policy, due to the distribution mismatch between the trajectory distribution of the dataset and that induced by the policy, the agent cannot accurately evaluate the policy if the mismatch is large. Reward over-optimization can happen if an agent overestimates and outputs a policy that achioves a high score on the learned reward model but has poor performance in the real environment.
Reinforcement lcarning complexity: Implementing Reinforcement lcarning efficiently is known to be challenging (Dulac-Arnodd et al., 2021). In addition, the stochastic binary preference feedback contains less information compared to the exact and accurate scalar reward feedback. This can influence the quality of the learned reward model and might increase the difficully of policy optimization. We have empirically verified the difficulty of running an online reinforcement learning process on the reward model learned from preference signals compared to reward signals in our experiments from Section 4
Most studies on offline PBRL seek other learning frameworks to circumvent the two challenges increase the learning efficiency from aspects. For example, Hejna & Sadigh (023) An et al. (2023) consider frameworks
that do not learn a reward model. Kim et al. )2023); Zhang et al. (2023) considers using more advanced
methods for reward modeling and trajectory generation, and the focus is not on solving the two challenges above. Standard two-step learning methods (Christiano et al., 2017) tackle the first challenge by applying KL regularization during the policy optimization step, but the second challenge from reinforcement learning difficulty remains. Due to the popularity and advantages of the two-step learning framework in practice and its advantages mentioned before, this work aims at increasing the learning efficiency of the two-step learning framework by solving the two challenges above at the same time.
Our contributions. In this work, we propose a novel two-slep learning PBRL algorithm called 'RRC' that simultaneously tackles the two main challenges above. Our key insight is that both challenges are induced by the same element: the state actions that are out of the datasot distribution. For pessimistic learning, the agent must avoid the policies that are more likely to visit such stale-actions cannot evaluate their performance accurately. Meanwhile, these to the complexity of the learning problem. Therefore, our insight is to constrain the action space to exclude such out-of- off-iiiribbution state-actions. In this way, our method not only follows the idea of pessimism but also reduces the complexity of the corresponding reinforcement learning problem. Tn the experiments, we apply our method to the continuous control problems, which are the standard RL benchmark. We construct offline PBRL datasets based on the offline RL benchmark DARL (Fu et al. 2020). We empirically verify that, compared to most baseline methods, our algorithm always learns a policy of better performance, When compared to a recent stato-of-the-art offline PBRL method 'fow to better' (FTB) (Zhang et al., 2023), PRC has similar performance. However, FTB requires access to additional behavior dataset that is not used by PRC. More importantly, FTB uses the technique of diffusion (Yang et al. 2023). which is computationally expensive. On our benchmark, the PRC algorithm achieves 7x speedup compared to FTB while achieving similar performance.
the agent state-actions contribute
2 Preliminary
2.1 Offline Preference Reinforcement Learning Problem
We consider preference-basee reinforcement learning with offline preference feodback (Zhan et al. 2023 The agent has access to an environment characterized by an incomplete Markov Decision Process (MDP)
2

---

Published in Transactions on Machine Learning Research (10/2025)
without reward function M (S,A, po), where S is the state space, A is the action space, P is the state transition function, and No is the initial state distribution. A policy т : S - A(A) maps a state 8 to an action stochastically drawn from a(-ss). We denote 11S.A as the class of policies supported on the state space S and action space A. A trajectory of length t is a sequence of states and actions ($1, 01,..., St, at S1+1) where a E A, 81+1 ~ P(-]&, at), Vi E [4]. A preference model F lakes a pair of trajectories T1: 72 of the same length as input, and outputs preference a E (>, <); indicating its preference over the two trajectories, i.c. 71 > T2 or Ti < 72. In this work, we assume the standard conventions and consider the Sradley-Terry (BT) family of preference models (Bradley & Terry [952). Specifically, there exists reward function R : S A + R, such that the probability distribution of the output preference of F given TI, 72 satisfies:
exp(Z R(s,a)) [11 (2) exp(E R(s,a) + exp(e exp(E(a.))Er R(s,a),
F F
(1)
We define the performance of policy π on a reward function R as the expected cumulative reward of trajectory generated by the policy. Formally, we abuse the notation and denote the policy performance as [())= == =(== "( 12) rf R(s', aa'),, where y [0, is discount factor. An offline preference dataset. D = ((7), o"),")) ,,...,
(T,"..,,''' (TN -. [ consists of multiple preference data over different trajectory pairs. We use the term 'behavior policy' to represent how the trajectories are generated in the dataset. The action distribution given by the behavior policy is the same as that of the dataset (Wu et al. 2019). A learning agent. is given access to the incomplete MDP M = ASP) of the environment and a dataset D whose preferences are generated by a preference model F. The learning goal is to learn a policy that has high performance on the reward function of the preference model F. We say a learning algorithm is efficient if it can learn such a high-performing policy.
2.2 Model-Based Step Learning Framework
Here, we introduce prevalent model-based two-step learning framework widely applied for solving PBRL problems (Christiano al. 2017) and is the framework for our algorithm in this work. The general algorithm consists of the two learning steps as follows.
1. Step 1: Offline Reward Modeling. The agent learns the optimal reward model R in R that minimizes the prediction loss on the preference dataset: At +- arg ming (%)
2. Step 2: Online Policy Optimization (Online Reinforcement Learning). The agent learns the optimal policy on the learned reward model in a constrained policy class: f <- maxxelle R(T).
Here, The first step is usually solved by standard supervised learning process. At this step, (..-)) is a prediction loss function that evaluates how well a reward model reproduces the preference labels in the dataset. A standard choice is the sum of the negative log probability of the reward model predicting the correct preference: (R) = Enn, (..r)r - log (a((), The second step is usually solved by an online reinforcement learning process on the incomplete environment MDP combined with the reward model learned from the first step. At this step, the agent specifies a constrained policy class Ie S IIS.A to avoid the risk of reward hacking. A standard choice is policies with limited KL divergence to the behavior policy from which the dataset's trajectories are generated (Christiano et al., 2017). This requires the agent to learn a behavior clone policy that approximately reproduces the action distribution in the dataset. To achieve this, the agent specifies an imitation loss function (T) that measures how well a policy imitates the behavior in the dataset. Then, the agent learns a behavior clone policy that minimizes the imitation loss Ti E arg ming (T). This is often achieved by a supervised learning process. Let KL(TI, T2) := m[(ass Erwa, E(..)er log Ra(a[s] be the definition of the KL divergence between two policies. Then the constrained policy class can be defined as I = (- (T( = S {}, where , > 0 is a constraint parameter.
In practice, it is hard to directly search for the optimal policy on II specified by the KL regularization con- straint above. practical way solve the problem is to introduce a Lagrange multiplier to the optimization goal and search for the optimal policy in the entire policy space: max, R(x))) a KT( m:) (Zhan et al.
3

---

Published in Transactions on Machine Learning Research (10/2025)
2023 which be solved by standard RL algorithm such PPO (Shhulman et al. 2017 Formally, Amgorithm below shows popular practical implementation of two-step learning. The reward model R and the imitation policy na can be learned by standard supervised fine-tuning methods.
Algorithm 1: Two-step training with KL-regularization and PPO algorithm
Inputs
Environment (8, A,),, Dataset D, prediction loss function L imitation loss function
1. Train a reward model ft arg REn (R)
2. Train an imitation policy n; arg min (T)
3. Run PPO algorithm to solve the oplimization problem â +- arg max R(x)) a (L(T,
4. Output n²
3 Preference Based Reinforcement Learning on an Explicit Constrained Policies (PRC)
3.1 Sub-Oppimaality of Policy Optimization on Regularized Learned Reward Model
In this work, we focus on the choice the constrained action space Hg in the policy optimization step. Let A be the reward function learned the reward modeling step, and R" be the true reward function. We define I ( Teet) == [=) - Rº (Fret) (f(m)) [[TTer)]l as the estimation error on policy π given by the reward model ft compared to a reference policy nret E IIS.A, We focus on relative rewards because the preference model in Eq. [] only depends on relative rewards. Intuitively, a policy will have low estimation error if is likely to visit the state actions covered by the datasot. A rigorous analysis of T+(+) for reward model R learned by deep neural networks based function approximation remains an open question. Based on the definition of R.), we can give a thcorctical bound on the performance of the policy learned at the second step in Proposition 3.1
Proposition 3.1. Let R be the true reward function, Rt be the learned reward function from the dalasel, and I' (-) be the relative estimation error of the learned reward R. Let be the learned policy to approzimate the optimal policy in R and capt max f((x')= - f(((t) be the corvesponding oplimization error. Then, the performance of the learned policy is at least
(+)
(R*(T) I (T Kref) TE
max
max T (T' TTee)) - Copt.
='elle
The proof can be found the appendix. By Proposition 3.1 we can betler understand the difficulties of reward hacking and the complexity of reinforcement learning in the two-step learning framework. Here, the last term Copt == max R(x') R(()) represents the optimization crror during the policy optimization process. Intuitively, the optimization error would be large if the policy optimization problem is hard to solve. The first term for the choice of the constrained policy class 1l.. Enlarging the constrained policy space can include poli- cies higher performance R* (T). However, the maximal estimation error in the constrained policy space min max (T Fret) also increases, which can cause reward hacking. To balance the trade-off, the constrained policy space should be limited a subset of policies with low estimation crror. This idca is closely related the concept of 'pessimism,, which requires the agent to focus the policies covered by the dataset.
(R*(TT) (ir) [n( Trat) max I [a( Rtes)) implies a trade-off
As shown in Alg 1 an intuitive and popular way to balance the tradc-off is to use KL regularization to specify the constrained policy space (Christiano et al.) 2017). However, this approach could suffer from the difficulty RL training for finding the optimal policy on the regularized reward among the full policy class и E ISA.A, resulting in significant optimization errors. This also observed our experiments in Section 4 Next, we introduce our approach balance the estimation-constrannt error while reducing the optimization error.
4

---

Published in Transactions on Machine Learning Research (10/2025)
3.2 Policy Optimization on An Explicitly Constrained Policy Space
A straightforward way to reduce the oplimization error is to reduce the complexity of the optimization problem. Our high-level idea is to ensure that the agent can directly search for the optimal policy in the constrained policy class instead of searching among all possible policies. This can be achieved by explicitly choosing specific policies for the constrained policy class. In order to avoid the problem of reward over- optimization, such policies should mostly visit. the state-actions covered by the dataset. Therefore, we define a special constrained policy space with an imitation policy T' and a pessimistic coefficient p as below:
1l == {m: a(s,a) s.t. (als) <p 0}
Here, we abuse the notation of m(]ss). If the action space is discrete, then x (a]s) represents the probability of the policy to generate the response at the state s. If the action space is continuous, the i[als) becomes the probability density. The constrained policy class He consists of policies supported on a constrained action space. Given a value of p, the constrained action space at a state A'Is includes the actions that satisfy (als)) 2 p. This choice tackles the problem of reward hacking in a way similar to the KI. regularization approach. If the imitation policy x' can represent the action distribution in the dataset, then the constrained policy class here explicitly restricts the agent to consider policies that only take the actions likely covered by the dataset. More importantly, the complexity of finding the optimal policy is reduced when the agent only focuses on a constrained policy space instead of the full policy space. Here, the agent only needs to search among the policies supported on a constrained action space that is much smaller than the original action space. This can make the reinforcement learning process easier and reduce the optimization loss. We use an intuitive example show how much optimization loss can be roduced. Consider the case of the environment being a tabular MDP where mature theoretical understandings have been developed. For the popular and efficient UCBVI algorithm (Azar et al.) 2017,, the dependency of optimization loss the size of action space |A| is O(V[Al). In the case of learning on clipped action space, if the clipped action space has size of |A/N, then the optimization loss is reduced by 1/vN times. In our practical implementations in Section 4 the size of the clipped action is 10-d times smaller than that of the full action space, where d is the number of action space dimensions. Intuitively, learning such small action space can significantly reduce the optimization error, and we further empirically verified this in Section 4
Formally, we show the general form of the PRC algorithm using the constrained policy class above in Alg 2 In the first step, the learning agent learns a behavior policy that can best imitate the behavior demonstrated in the dataset and then learns reward model that best interprets the dataset's preference label. In the second learning step, the agent learns the policy supported on a constrained action space A' that maximizes the cumulative return on the learned reward model.
Algorithm 2: Preference Based Reinforcement Learning a Constrained Action Space
Inputs : FFavironment (S, A, P), Dataset. D
1. Train a reward function R + arg mina (R).
2. Train a behavior clone policy कe + (T)
3. Construct a clipped action space A' such that at state s, the clipped action space is
A'|s {a: mu(als)= p).
4. Find the optimal policy á supported on the clipped action space a t- arg m R(T))
4. Outpul n²
3.3 Practical Implementation of PRC
Finally, we present a practical implementation of the PRC algorithm for our experiments on continuous control problems. In this case, the action space is continuous and has multiple dimensions. Our goal is to determine the clipped action space that easy to represent in practice. To achieve this, we train a deterministic imitation policy æ² that generates actions close to the ones in the dataset and then specifies the constrained action space centered at the actions given by the policy with radius r. In the context of
5

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 2 this process essentially considers stochastic behavior clone policy To (-(ss) that has uniform distribution over a constrained action space centered at (=(+()) with radius The action given by the deterministic policy can be thought of as an empirical estimation of the center of the distribution given by the behavior policy of the dataset. Since we have no prior knowledge about the shape of the distribution of the behavior policy, we adopt a reasonable assumption that an action is more likely to be sampled by the behavior policy if it is closer the center of its distribution. Then, we approximate the space of A' as a box whose center is the deterministic policy with radius v, which is a hyperparameter related to the degree of pessimism. As we show in Appendix A the method based on this approach is not sensitive to the value of r.
Formally, in Alg 3 we show our practical implementation for the PRC algorithm. Lines 1-2 learn reward model and an imitation policy, which a standard supervised learning approach can achieve. The loss functions are set Eq. 2 Here, d(::-) measures the distance between two actions. In our experiments, we consider the standard continuous control problems, and the distance mcasure is defined as the L2 distance between the two actions the action space. Lines 3 construct constrained action space for each state centered at the action given by the imitation policy. Line 5 learns the optimal policy f' S + A' supported on the constrained action space. In practice, we can solve this by applying the SAC or PPO algorithm. Given a state, the output action satisfies a = ( + +a),, where a' ~ f'(+ss) Note that this process is similar to how the batch-constrained Q learning (BCQ) (Fujimoto et al. 2019 algorithm uses the perturbation function in its implementation. While our practical implementation of PRC shares similarities with BCQ, in the appendix, we discuss the fundamental differences between BCQ and PRC in detail.
exp (s.a)ET R(s,a))
(R) = Σ
log (N..2)ED
exp( (sa(En (sajEn R(s,
+ exp(E (s,a)er R(s,a)
(2)
Cb(T)= Σ d(
(s.0)~D
Algorithm 3: PRC practical implementation
Euviroument AS, P, Mo), Datasel Parameters: Positive real number r
Inputs
1. Train reward function R <- arg min (R)
2. Train deterministic behavior clone policy x² +- (T)).
3. Construct constrained action space A' = RN where N is the dimensions of A. Each dimension is constrained on [-r,r].
4. Find the optimal policy #' that optimizes H' + arg max E. Es-po R(s, Proj (T' (s) + a'))
5. Output à (aaas) ( + d')ls)
4
Experiments
4.1 Setup
Zhang et al., 2023). we construct our offline preference dataset from D4RL benchmark (Fu et al. 2020) labeled by synthetic preference following the preference model in Section 2 The details of our dataset construction can be found in the Appendix. Note that consider the offline setting where the agent can only access a dataset of preference labels. We consider various datasets from D4RL to represent the general learning scenarios. The robot control environments we choose include 'Hopper', 'HallCheetah'; and 'Walker'. The types of trajectories we choose include 'Medium', "Medium-Replay', and 'Medium-Expert" To make the number of state-action in the dataset. aligned with that of the D4RL benchmark, the total number of trajectory pairs with a preference label is x 10
Dataset:
Following previous studies (Hejna Sadigh 2023
Baseline Algorithms: Here, we consider multiple two-step learning algorithms and state-of-the-an attacks in the related offline PBRL setting baselines, which listed below:
6

---

Published in Transactions on Machine Learning Research (10/2025)
1. Two-step learning with KL-regularization and regularization: We adopt the traditional two-step learn- ing baseline with and without KI-regularivation introduced in Section 2 the algorithms initialize their policies as the behavior clone policy.
To make the comparison fair,
2. Reward modeling followed by standard offline RL algorithms: Another simple yet efficient two-step Icarn- ing approach for offline PBRL problems is to combine reward modeling with a standard reward-based offine RL algorithm. First, a reward model is learned from the preference dataset. Then, the reward model is used to provide scalar reward labels to the stato-action in the datasot. Finally, we apply a standard offline RL algorithm IQL (Kostrikov al. 2021) training on the dataset with scalar reward
labels.
3. Inverse preference learning (TPL) (Hejna & Sadigh 2023 and flow to better (FTB) (Zhang al. 2023 IPL and FTB are the state-of-the-art learning algorithms for a related offline PBRL setting that requires preference dataset and a behavior dataset. To avoid underestimating the performance of them, these algorithms have access to the full D4RL dataset during training. Note that our algorithm has no access to the extra bchavior dataset.
4. Oracle: The oracle is trained with true rewards instead of preference labels. Here, we apply IQL training on the base D4RL dataset with true reward signals. Note that the information from the reward signals is strictly more than that from the preference signals in this case. The oracle should be considered as an upper bound on the performance of an offline PBRL algorithm.
To ensure a fair comparison, all the methods that require reward modeling share the same learned reward model during training. Additional detailed training sctups can be found in the Appendix.
4.2 Learning Efficiency Evaluation
Here, we compare the efficiency of PRC against other baseline algorithms on different datasets. To straight- forwardly compare the performance of different learning algorithms, we show the standard D4RL. score (normalized performance) of the policy learned by a method during training.
The results in Table I show that the PRC algorithm has high learning efficiency. Here, we use the SAC algorithm for the online reinforcement learning step in PRC. The baseline two-step learning methods use the PPO algorithm for the online learning step, it achicves better performance. PRC is generally more efficient than most baselines and sometimes even competes with the oracle. Our algorithun is more efficient two-step learning method as it performs much better than other two-step learning baselines initialized from the behavior clone policy. This observation indicates that sctting the initial policy as the behavior clone policy is not enough to achieve high learning efficiency, and training on the constrained action space is the key to high learning efficiency for PRC. Even with additional access to the behaviors in the D4RL. dataset, PRC algorithm outperforms the IPL algorithm. Although PRC and FTB have similar performance, we notice that FTB uses the diffusion technique (Yang et al. 2023) during training, which is very computationally expensive. In our testing, the training time required by the FTB is about 7 times as long as that required by PRC. The details can be found in the Appendix. In conclusion, the PRC algorithm has high learning efficiency while being computationally efficient.
In addition, we note that the PRC algorithm achieves high learning efficiency on the medium-expert dataset. The action distribution of this datasct is multimodal, contributed by multiple policies. Half of the trajectories in the dataset are collected by medium policy, while the rest are collected by an expert policy. In principle, single deterministic behavior clone policy is not able to well characterize a multimodal dataset action distribution, which is a limitation on the implementation. The reason for the empirical success of our method could be that at each state, the behavior clone policy learns to predict an action that is close to the action given by either policy, instead of some actions in the middle that are not close to any dataset policies. In Section G we discuss this limitation on our method and the future direction to make it more universal.
Next, we empirically analyze why PRC should be an efficient learning algorithm from the aspect of pessimistic learning and reduced reinforcement learning complexity.
7

---

Published in Transactions on Machine Learning Research (10/2025)
Datasct
Oracie
PRC
Naive two-step
KL two-step
IPL
RM
FTB
HallCheetah- Medium
0.2
0.5
40.1
0.1
42.7 0.1
43.5
HallCheetahi- -nvetal-Modhur---- Modhatn-Replay
+: 0.2
31.9
0.9
34.9
0.7
38.6
HalfChoetal-- Seatt--Moohuut----e Expert
92.1 0.7
78.4
40.9 4:
40.4 0.5
41.7
0.8
86.0
Hopper-Masn Medium
76.1 1.2
71.0 = 7.5
67.5 =
73.5 + 4.0
72.4 7.4
67.2 = 0.3
66.8 = 5.7
Hopper Hopper-Mediung Medium- Medlun-Replay
76.7 5.3
47.0 19.5
49.4 4.44774 7.4
53.1 2.2
39.5
17.5
32.2 + 0.4
62.9
9.2
Hopper-Meding Mepeit--xppert
113.1 0.4
100.2 + 9.5
6T.3 + 7.1
71.4 10.9
76.9 + 8.1
97.1 + 5.2
109
1.7
Walker2d-Mediiug
84.5 = 0.3
84.4 = 0.8
81.0 = 0.8
79.5 2.1
80.8 = 1.6
81.9 = 2.5
80.5 = 0.6
Walker2d-Me (00----iumm--eee Replay
83.1 = 2.3
87.9 = 6.1
49.8 = 7.3
45.4
49.3 = 3.5
71.8 = 6.4
79.3 = 1.9
Walker2d-Me 24-Medium-------------- Expert
111.5 + 0.4
110.4 + 0.6
93.4 4.9
92.1 = 2.3
107.5 + 3.0
105.7
= 6.9
109 + 0.3
Comparison between the performance of different learning methods. Oracle is trained on the true reward signals instead of preference signals, which is not baseline. Bolded performances within 95% of the highest performance. FTB requires much more training time than other methods.
Table 1:
4.3 Pessimism Effectiveness
Here, we empirically verify that by training on constrained action space, the PRC algorithm achieves effective pessimistic learning. In Figure we show some representative examples. For the PRC algorithm, during training, the trend of the learned policies' performance measured the learned reward model is aligned with that measured on the true reward. This suggests that the pessimistic learning in the PRC algorithm effective: it mainly considers the policies that are supported by the dataset so that the agent can evaluate their relative performance accurately. In contrast, for the KL-regularized two-step learning method, we find multiple cases where the trends can even be opposite when evaluated on the reward model and on the true reward. During training, the simulated performance of the learned policy increases while the actual performance decreases. This is result of relying on overestimated rewards during policy optimization. Since the methods use the same learned reward model, the results support that our method achieves pessimistic learning effectively by exploiting the more accurate prediction of the learned reward model.
Simulated Performance
KL Two-Step: Walker Medium-Reple
PRC: Walker Medium-Repley
KL Two-Step: Wa Medium-EXppet
PRC: Walker Medium-Expertt
-60
600
MM Min
150
450
-80
400
100
200
0
300
250
100
200
Spoch
Spoch
Epoch
Epoch
KL TWo Ste-:tep: Walker Medium Replay
PRC: Walker Medium Replay
KL Two Steo-Step: Walker Medium Expert
PRC: Walker Medium Medium-Expeerf Expert
True Performance
50
60
110
100
20
45
W
Figure 1: Comparison between the trend of the performance of the learned policies on the learned (simulated performance) and true reward models (true performance) during training. An algorithm is not pessimistic enough if the trends of the simulated and true performance over time are not aligned.
8

---

Published in Transactions on Machine Learning Research (10/2025)
4.4 Reinforcement Learning Efficiency Constrained Action Space
Here, we empirically verify that reinforcement learning is much easier the constrained action space pared to learning on the full action space. We measure the simulated performances of the policies learned by the PPO algorithm on the full or the constrained action spaces. Therefore, the baseline method is the naive two-step learning that uses PPO for policy optimization, and the PRC method also uses PPO in this case. In Figure 2
we observe that the simulated performance of the policies learned by the PRC method (optimizing over constrained action space) much higher than that learned by the naive two-step learning methods (optimizing over the full action space). While the baseline method struggles with low-performing policies, the PRC method steadily learns policies of much higher performance. Considering that the optimal policy supported on constrained action space is no higher than that supported on the full action space, the empirical results show that the optimization error of reinforcement learning on a constrained action space is much less than that on the full action space. Therefore, the empirical observation verifies our statement that the PRC method has less optimization error during reinforcement learning.
Walker ME
Cheetah M
Cheetah ME
Simulated Performance
Nnw mn Mnn
400
200
200
0-
31033
200
-200
-200
-400
-400
-600
50
100
150
50
100
150
100
150
Epoch
Epoch
Baseline
PRC
Figure 2: Comparison between the simulated performance of the policies learned by different methods. The baseline method is the naive two-step learning method. The learning elliciency is measured by the simulated performance of the learned policies.
4.5 Reinforcement Learning Complexity in PBRL
Here, we empirically show that solving the policy optimization process with a standard RL algorithm on a reward model learned from preference signals is harder than that from true reward signals. Given the same D4RL dataset, we label the dataset with preference signals, in the main experiments, and true reward signals. Then, we train two reward models based on the two kinds of signals. Finally, we train an efficient RL algorithm, PPO, on the two reward models and compare the learning efficiency in the two cases. The results in Figure g show that when learning on the reward model trained from true rewards, the RL agent can quickly learn some policies that have high performance on the reward model. In comparison, it can take much longer for the agent to find a good policy on the reward model from preference signals, and sometimes, the agent may not be able to find decent policy after lot of training epochs. This indicates that is harder to learn from reward model that is trained preference signals instead of true reward signals.
In the appendix, we have added additional experimental results for evaluating our method. In Appendix A.1 we test our method on the ollline dalaset with human preference labels based on the D4RL and Metaworld benchmark [Km et al. 2023). We show that our method can still learn policies of high performance from human preference feedback. In Appendix A.2] we conduct an ablation study on the constrained action space radius We show that our method is not sensitive the choice of this hyperparameter, and it can always learn policies of high performance for different values of
9

---

Published in Transactions on Machine Learning Research (10/2025)
True Rewards: Hopper ME
Preference Signals: Hopper ME
True Rewards: Hopper M
Preference Signals: Hopper M
Simulated Performance
1000
1000
800
56
800
600
600
Sc0ie
|
400
-50
50
100
150 50 100
100
150
50
100
150
Epach
Epoch
Eboch
Epoch
Figure 3: For each dataset, two reward models are trained on the true rewards and preference signals, respectively. The same RL algorithm, PPO, is applied to learn on both reward models. The difficulty of policy optimization on a reward model is large if the PPO algorithm takes more training epochs to find a policy of high performance.
5 Related Work
Offline Reinforcement learning: There have been many studies on the standard offline reinforcement
learning selting with reward feedback (Kumar et al. 2019 Fujimolo et al. 2019 Wu et al. 2019 Levine et al. 2020 Cheng et al.] 2022] Yu et al.) 2021 Fujimoto & Gu, 2021 Bhardwaj et al., 2023 Lyu et al,
2024), contrast to preference feodback considered in our sctting. The key idea behind offline reinforcement learning is pessimism, which encourages the learning agent to focus on the policies or actions supported by the dataset (Cheng al.] 2022). However, the gap between learning from reward feedback and preference feedback is currently unclear. Particularly, we find that the batch constrained Q learning (BCQ) algorithm (Fujimoto et al. 2019) shares some similarity with the idea and approach in our work. We highlight the similarity and key difference between PRC and BCQ in the appendix.
Online PBRL: Preference- based reinforcement learning is popular in the online learning scenario (Chris- tiano al. 2017 Ye et al., 2024 Ibarz et al. 2018 Lee et al., 2021b). Various types of preference models and labels have been considered the tabular case (Firnkranz et al. 2012 Wirth et al. 2017). Similar the works that consider the general case, in our work, we consider the preference models to be Bradlcy-Terry models and output binary preference labels (Ibarz et al.] 2018 Lee et al. 2021a). The two-phase learning approach mentioned in Section [ is prevalent in the online PBRL setting. Note that directly applying online
methods to the offline scenario is also likely to be inefficient even when possible (Van Hasselt et al. 2018 as the online learning approaches do not need to be pessimistic.
Offline PBRL: Offline preference-based reinforcement learning has attracted much attention recently. Specifically, we find three types of offline PBRL settings attract most attention. 1. Zhan al. (2023): Zhu et al. (2023) consider the offline learning setting where the training data only consists of preference labels. This setling is similar to the standard reward-based offline RL setting, where the training data only consists of reward labels. This work focuses on this setting. Zhan et al. (2023) proposes an optimization problem following the two-phase learning framework and theoretically proves that the solution to the prob- lem is a near-optimal policy with respect to the training dataset. However, how to solve the optimization problem in practice remains unknown. Concurrently, Zhuu et al. (2023) formulates a similar optimization problem under the linear assumption of the environment and proposes a way solve the problem. Still, the results cannot be extended to the general case without the linear assumption. 2. [Sadigh et al. (2117): ) Shin et al. 2021 2023) study the PBRL problem in the active learning setting where the agent has access to offline behavior datasct and can choose a subset of the datasct to be labcled with preference feedback. 3.
An el al. (2023); Kim et al. 2023);; Hejna & Sadigh (2023) ; Zhang et al. (2023); Tu et al. (2024): ) Gao et al. (2024): Kang & Oh [2025]: Choi et al. 2024); Liu et al. (2024) consider a different offline learning setting
where the agent can access both a dataset of preference labels and an additional trajectory dataset that only
contains trajectories. The fundamental difference between this setting and the active learning setting is that,
here, the preference labels are given to the trajectories before training. Note that reinforcement learning
10

---

Published in Transactions on Machine Learning Research (10/2025)
from human feedback (RLHF) for natural language processing (NLP) tasks is special instance of PBRL where humans provide the preference labels and is often studied in the standard offline setting (Oyang et al. 2022 Bai et al. 2022 Rafailov et al. 2023) which is the same as the setting considered in this work.
6
Conclusion and Limitation
In this work, we propose a novel two-step learning algorithm called PRC for the offline PBRL setting. On the standard offline RL benchmarks, we empirically verily that the PRC has high learning efficiency and provide evidence for why is more efficient two-step learning algorithm than others. Our framework is limited to the typical offline learning setting where the agent only has access to an offline preference dataset. Our experimental evaluation is limited to continuous control problems, the standard benchmark in RT studies. Our method relies on external online learning method to learn the policy. Our practical implementation strictly forces the learned policy to output actions in a small action space, which could limit the full potential of the dataset. To approximate the dataset's 's action distribution, the implementation uses a constrained action space centered at actions given by a learned deterministic imitation policy. This choice is not able well characterize all possible dataset action distributions. future extension of this work can be adopting a stronger behavior cloning technique that can better characterize the dataset's action distribution.
Acknowledgments
We sincerely thank the anonymous reviewers for their insightful comments and helpful suggestions. This work was supported by funding through NSF Grants No. CCF-2238079, CCF-2316233, CNS-2148583 anda Research Gift. from Amazon AGI Labs
References
Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: Active preference learning-based reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 28, pp. 116-131. Springer, 2012.
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung- Min Kim, and Hyun Oh Song. Direct preference-based policy optimization without reward modeling. Advances in Newral Information Processing Systems, 36:70247-70266, 2023.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement
learning. In International conference on machine learning, pp. 263 272. PMLR, 2017.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 255::2204.05863 2022.
Mohak Bhardwaj, Tengyang Xie, Byron Boots, Nan Jiang and Ching-An Cheng. Adversarial model for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 1et122--1299 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biomelrika, 39(3/4):32433M) 345, 1952.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for oflline reinforcement learning. In International Conference on Machine Learning, pp. 3852 3878. PMLR, 2022.
Heewoong Choi, Sangwon Jung, Hongjoon Ahn, and Taesup Moon. Listwise reward estimation for oflline preference-based based reinforcement learning. arXie preprint arXiv:2408.044 04100 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment. learning from human preferences. Advances in neural information processing systems, 30, 2017.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Gabriel Dulac-Arood,, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9))2419 2468, 2021.
Justin Fu, Aviral Kumar, Ofr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:204..07119, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132 20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 9052-2062. PMLR, 2019.
Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preserence-based reinforce- ment learning: formal framework and a policy iteration algorithm. Machine learning, 89:123 156, 2012.
Chen-Xiao Gao, Shengjun Fang, Chenjun Xiao, Yang Yu, and Zongzhang Zhang. Hindsight preference learning for offline preference-based reinforcement learning. arXiv preprint arXiv:2407.04451, 2024.
Ammar Haydari and Yasin Yilmaz. Deep reinforcement learning for intelligent transportation systems: A
survey. IEEE Transactions on Intelligent Trunsportation Systems, 23(1):11 32, 2020.
Joey Hejua and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. arXiv preprinl 2305 15363, 2023.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Hyungkyu Kang and Min-hwan Oh. Adversarial policy optimization for offline preference-based reinforcement learning, ar preprint arXiv:2503. (xXi::2533.0306, 2025.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for arXiv preprint arXiv:2303.00957 2023.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint orXi::2110.000 10.06169, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.055 050911, 2021a.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based rein- forcement learning. arXiv preprint arXi::2111.0000 03026, 2021b.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXie preprint 2rXim:2005.01643 2020.
Xiao- Yin Liu, Guotao Li, Xiao-Hu Zhou, and Gung-Guang Hou. Tease: Ofline preference-based reinforcement learning with high sample efficiency. arXiv preprint 1rXiv:2412.21001, 2024.
Nguyen Cong Luong, Dinh Thai Hoang, Shimin Gong, Dusit Niyato, Ping Wang, Dong In Kim. Applications of deop reinforcement lcarning in communications and networking: A survey. IEEE Communications Surveys & Tutorials, 21(4):3133-3174 2019.
Liang, and
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative qlearning for offline reinforcement learning, 2024. URL http::://awww.oo///aaaaaaaae/ wwtp:////........oo../// 04745
12

---

Published in Transactions on Machine Learning Research (10/2025)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Newral Information Processing Systems, 35:27730 27744, 2022.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline rein- forcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.1822 2023.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia, Active preference-based learning of reward functions. 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy oplimiza- tion algorithms. arXiv preprint arXiv: 06347, 2017.
Daniel Shin, Daniel S Brown, and Anca D Dragan. Offline preference-based apprenticeship learning. arXiv preprint arXiv22107.0925 2021.
Daniel Shin, Anca D Dragan, and Daniel S Brown. Benchmarks and algorithms for offiine preference-based reward learning. Xr preprint ar orXiv:2301.01300 2023.
Richard S Sutton and Andrew G Barlo. Reinforcement learning: An introduction. MIT press, 2018.
Songjun Tu, Jingbo Sun, Qichao Zhang, Yaocheng Zhang, Jia Liu, Ke Chen, and Dongbin Zhao. In- dataset trajectory return regularization for offiine preference- based reinforcement learning. arXiv preprint arXiu:241..09104, 2024.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXin:111..06644 2018.
Christian Wirth and Johannes Fürnkranz. Preference-baseed reinforcement learning: preliminary survey. In Proceedings of the ECML/PKDD-1S 13 Workshop on Reinforcement Learning from Generalized Feedback: Beyond Numeric Rewards. Citescer, 2013.
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1 46, 2017.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforccment learning. arXiv preprint 1rXi::1911.111 113661, 2019.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-33 39, 2023.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, and Tong Zhang. Online iterative rein- forcement learning from human feedback with general preference model. Advances in Neural Information Processing Systems, 11817733-18077 2024.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
Wenhao Zhan, Masatoshi Uchara, Nathan Kallus, Jason D Lec, and Wen Sun. Provable offline reinforcement learning with human leedback. arXiv preprini Xiiv:2305.14816 2023.
Zhilong Zhang, Yihao Sun, Junyin Ye, Tian-Shuo Liu, Jiaji Zhang, and Yang Yu. Flow to better: Offline preferenco-ban based reinforcement learning via preferred trajectory generation. In The Twelfth International Conference Learning Represendations, 2023.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or wise comparisons. arXiv preprint or arXiv:2301. 1 11270 2023.
14

---

Published in Transactions on Machine Learning Research (10/2025)
A Additional Experiment Results
A.1 Evaluation on Human Preference Dataset
This section evaluates the performance of our method on the offline PBRL datasets with preference labels collected from real humans, More specifically, we consider the datasets based on the D4RL and MetaWorld benchmark provided in (Kim et al. 2023). The training setup for the D4RL, datasets is the same as that of our experiments in Section [ We follow the training setup for the Meta World datasets from (Zhang et al. 2023), which demonstrates high learning efficiency. We adopt the same baselines used in (Zhang et al. 2023 including the RM (based on IQL (Kostrikov et al., (021))) OPRL (Shin et al. 2023 PT (Kim et al. 2023 IPI. (eejna & Sadigh, 2023), and FTB (Zhang et al. 2023 We use the performances reported by (Zhang et al. 2023) to represent the performance of the baseline methods. Note that the baseline methods here are developed for a different offiine PBRL setting. The baseline methods have access to the human preference
dataset and a large behavior dataset. Our method has access to the same human preference dataset and the state transition function of the environment.
In Table show the performance of the policies learned by different methods the human preference datasets. The results show that PRC outperforms the IPL, PT, and reward modeling methods by clear margin in most cases. Compared to FTB, PRC has better performance on both D4RL datasets and the "plate-liid--22' dataset, and a worse performance on 'sweep-into-v2.. This is similar to our observations in Table [ that the two methods have similar performance in general. Our results verify the learning efficiency of our method on the human preference labels and on the Meta World benchmark, which is considered more challenging than D4RL.
Dataset
PRC
RM
OPRL
PT
IPL
FTB
41.8 1.4
36.0
38.3
41.1
36.5
38.4
Baaker-MediunnRRplae plate-slide-22 sweep-into-v2
94.1 + 10.9
75.7
64.0
79.6
8.8
79.1
0.85 0.05
0.00
0.44
0.45
0.38
0.51
0.35 = 0.03
0.00
0.21
0.18
0.14
0.97
Table 2: Comparison between the performance of different learning methods datasets with human pref- erence feedback. Bolded performances are within 95% of the highest performance. FTB requires much more training time than other methods.
Datasct
r r= 0.05
r 0.1
r= 0.2
r= 0.5
RM
HalfCheetah- Medieeahh--eiiu---eeee-eee Replay
40.8 + 0.2
43.3 = 0.2
45.4 0.5
44.1 0.7
40.1 0.7
Walker-Meding Mediun-Repllyy
79.5 8.4
87.9 + 6.1
88.6 + 2.3
87.3 5.8
71.8 + 6.4
Table 3: Performance of PRC for different values of the constrained action space radius r.
A.2 Ablation Study on Constrained Action Space Radius
This section evaluates the performance of PRC for different values of the constrained action space radius r. We test on wide range of radius coefficients {0.05, 0.1, 0.2, 0.5). The other training setups the same as those of the experiments in Section 4 The results in Table 3 show that PRC can always learn policies with decent performance for different values of r, suggesting that PRC is not sensitive to the choice of r. PRC achieves a higher performance for r E [0.1, Intuitively, a smaller value of r makes the reinforcement learning process more conservative and less complicated. Therefore, we set r = 0.1 for our experiments in Scction !
A.3 Baseline Two-step learning algorithms combined with SAC or PPO
In Section we mention that for the two bascline two-step learning PBRL algorithms, using the PPO algorithm for the online RL step learns a better policy compared to using the SAC algorithm. Here, we give concrete examples in Table 4 For both baseline algorithms, the results show that the performance of
15

---

Published in Transactions on Machine Learning Research (10/2025)
Dataset
Naive-SAC
Naivo-PPO
KL-SAC
KL-PPO
MaalChheeta--Meiiu-RReee
14.7 + 1.2
31.9 + 0.8
16.9 + 0.6
32.9 + 0.9
Wallor2ddMMddum-RRphy.
22.7 + 0.2
49.8 + 7.3
25.2 = 0.8
45.4 + 0.1
Bascline Two-step learning algorithms combined with SAC or PPO. 'Naive' is the baseline mothod that directly optimizes a policy on the learned reward model. 'KL' is the baseline method that applies a KL divergence penalty during policy optimization.
Table 4:
the policy learned by the PPO algorithm exceeds that of the SAC algorithm by clear margin. Therefore, to avoid underestimating the basclinc algorithms, they always use the PPO algorithm for the online RL step.
B Proof for Proposition 3.1
Proof. For any policy π E Ic and any reference policy Tret E IISAA, the performance gap between и and the learned policy satisfies:
(*(x) (+) = (T) - R ( - (R* RR (TTeet)
= (T) R1" (Trot) (f((T) f((TTTr)) (f((T)) (((Trot))-
((I - R² ()Tret)) - (f((A)) - f(( (R(()) - f((TTat)
T TR( Tzat) + [f, Tref) + R(xx) R(AA)
< r (T. Neef) + max T R Tret) + copt
x'Elle
The first inequality follows the definition of the estimation error T. Since the bound on performance gap holds for any policy x E II given any reference policy Tref E IIS,A, the performance of A can be lower bound by
R (+)
(T) 1' I'T(T,, Fret) max L' Tret)) Copt M
max
s'Elle
C Additional Details for Experimental Setups
To learn reward model, we follow standard supervised learning framework. Specifically, perceptron (MLP) structure for the neural network to approximate the utility model. The neural network consists of 3 hidden layers, and each has 64 neurons. We use a tanh function as the output activation so that the output is bound between []1, To train a deterministic behavior clone policy, the neural network we use to represent the clone policy has the same structure as that for the utility model. To train a stochastic behavior clone policy, the network outputs the mean and standard deviation of a Gaussian distribution separately. The network is MLP with 3 hidden layers each with 64 neurons. The last layer is split into two for the two outputs. For the mean output, deviation, we use lincar function and an exp activation for the last layer. In the reinforcement lcarning step, we use either the SAC algorithm or the PPO algorithm, depending on the dataset. The structure of the neural network for the actor is identical to that of the network used for training a stochastic behavior clone policy. The structure of the neural network for the critic identical to that of the network used for training the reward model.
multilayer
use linear function for the last layer. For the standard
Dataset Construction
The construction of our preference dataset is based on the reward--bard based datasets from the standard D4RL benchmark. We generate synthetic preference labels following the preference model introduced in Section 2 ensure that the dataset construction is consistent with the problem setting. We formally show the process of data construction follows.
16

---

Published in Transactions on Machine Learning Research (10/2025)
1. Randomly sample pairs of trajectory clips from the D4RL dataset. Following previous studies (Christiano et al. 2017j, the length of the clip is set to be 20 steps.
2. For each pair of trajectory clips, compute the probability of trajectory to be preferred based on the reward signals. To ensure consistency between different datascts, the reward signals are regularized to be bound in 1].
3. For each pair of trajectory clips, randomly generate preference label through Bernoulli trial with the probability computed above.
4. Return the preference dataset consisting of the trajectory clip pairs and the corresponding preference labels.
E Comparison between BCQ and PRC
The PRC algorithm shares several similarities with the famous offline RL algorithm BCQ (Fujimoto et al. 2019) in the reward feedback sctting. Both works are based on the common idca of pessimism that constrains the learning agent to policies covered by the dalaset. In the practical implementations, both methods adopt the concept of constrained action space during training. However, here we clarify that PRC and BCQ adopt very different learning strategies.
In our practical implementation, we first train a deterministic imitation policy n¹. Then we train a policy x (als) supported on a constrained action space with radius r centered at the action given by n'. The goal of our implementation is to find a constrained action space that only includes actions with high probability of being generated by the behavior policy of the dataset. The purpose of constructing the constrained action space is to exclude the policies not covered by the dataset.
In BCQ, they train a stochastic imitation policy n' and Q function. Essentially, they want to find a policy x that has a similar distribution to T' but is also more likely to select actions of higher value on the Q function. To achieve this, at a state 8, they randomly sample n responses from the imitation policy x (-|ss,, and they want to output the action with the highest value on Q. To further enhance the diversity of the samples, they introduce a perturbation function c(s, a, b) that takes a state s and action as input, and outputs a perturbation that is in a bound space 4. In principle, the bounded space here can be arbitrary and unrelated to the imitation policy, which is completely different from our method. The perturbation function is used to find the action with the highest value on Q in a constrained action space & centered at the sampled actions. The purpose of the constrained action space here is to increase the diversity of the sampled actions. In the end, the BCQ method outputs a policy as T(s) = arg maxa,te(s.a, ((s, + E(s, a,, Φ)), {az <-Iss)]] 1- This suggests that BCQ can output any action a with n' (als) > O. The possimism in BCQ comes from sampling actions from the stochastic imilation policy and then outputting an action close to the samples. In this case, the final action distribution not very different from the imitation policy as long as the number of samples is limited.
In conclusion, BCQ achieves pessimism in a different approach. At high level, the main purpose of using a constrained action space in our method is to exclude policies not covered by the dataset. In comparison, BCQ uses the bounded action space to enhance the diversity of sampled actions. The principle in the construction of the constrained action space in our method is to include high-probability actions in the behavior policy of the dataset. The bounded action space in BCQ follows no specific principle and can be defined arbitrarily. There are also critical technical differences. 1. Our practical implementation trains a deterministic imitation policy and standard policy supported the constrained action space. BCQ trains stochastic imitation policy and a perturbation 'policy' that takes both state and action as input. 2. Our method learns a policy that can only output actions in a constrained action space with radius r centered at the action given by a deterministic imitation policy. BCQ can output any action that has a positive probability on the distribution given by the stochastic imitation policy. For example, if the imitation policy is a Gaussian policy that has a distribution over the whole action space, then the policy can output any action in the action space. Therefore, our method is very different from BCQ, especially in the usage of the constrained action space.
17

---

Published in Transactions on Machine Learning Research (10/2025)
F PRC and FTB Runtime Comparison
For the computational resource, we use single NVIDIA A100-PCI GPU with 40 GB RAM and an Int Re) Silver 4214R CPU @ 2.40GHz with 64 GB RAM. The architecture for all neural networks used in the FTB and PRC algorithms is a multilayer perceptron with 3 hidden layers, and each has 64 neurons. On the datasets we construct, on average, the training time until convergence for the PRC algorithm is 5.14 1.50 hours, while for the FTB algorithm it 34.64 + 2.96 hours.
18


Reviews:
[{"review_id": "3f8VKgphLb", "reviewer": "Reviewer_duRi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Simplicity: The proposed method is simple, easy to implement, and addressing issues in preference-based offline reinforcement learning.\n\n2. Theoretical Justification: The authors provide theoretical support showing that constraining the policy to stay within high-probability regions reduces both reward estimation and optimization errors.\n\n**Weaknesses**\n\n1. While the idea of constraining policies to high-probability actions is well-established in Offline RL [1], the main contribution lies in applying it to the preference-based setting. However, the method is evaluated solely on D4RL tasks, a benchmark that is increasingly considered outdated even for standard Offline RL. This limits the impact and relevance of the empirical validation.\n\n2. The approach assumes a reliable behavior cloning policy to define the constrained action set, which may be difficult to obtain in practice, especially in noisy or low-data regimes.\n\n3. The paper briefly mentions its limitations but does not thoroughly analyze failure modes or potential drawbacks of the proposed method.\n\n4. It is uncertain whether this simple approach can generalize to complex real-world scenarios, such as training chatbots or other industrial applications. The authors should discuss the broader applicability and potential extensions.\n\n**References**\n\n[1] Kumar A, Fu J, Soh M, Tucker G, Levine S. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems. 2019;32.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on Preference-Based Reinforcement Learning and introduces a simple formulation that constrains the action space by excluding out-of-distribution state action pairs. Experimental results on standard continuous-control benchmarks demonstrate that this approach improves performance and sample efficiency compared to existing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please, refer to the Weaknesses.", "broader_impact_concerns": "Broader impact concerns are related to the use of preference-based learning in the context of training large models. Although I believe there are no direct concerns related to the proposed method, I would have appreciated a Broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8fYItySd7z", "reviewer": "Reviewer_TFpN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed method exhibits promising empirical performance across the evaluated tasks.\n- The core concept is straightforward and grounded in intuitive reasoning.\n\nWeaknesses\n- Several sections lack clarity in presentation, which may hinder comprehension.\n- The manuscript provides limited discussion of prior work, particularly in relation to closely related approaches.\n\n\nOn page 3, the authors state: “A learning agent is given access to the incomplete MDP of the environment and a dataset.” However, based on the descriptions in Sections 4.1 and 4.2, this does not appear to be accurate. The agent seems to operate in a strictly offline setting, without access to the environment. It would be helpful to clarify whether the policy is trained using online reinforcement learning with the learned reward model, or if it is entirely offline.\n\nOn page 6, the manuscript notes that “In practice, we can solve this by applying the SAC or PPO algorithm.” However, it is not clearly specified which algorithm was used in the experiments described in Sections 4.1 and 4.2. Additionally, standard SAC implementations are generally not suitable for offline reinforcement learning settings. Recent offline RL algorithms such as Implicit Q-Learning (IQL) or TD7 may be more appropriate and could enhance reproducibility and performance.\n\nFurthermore, the proposed method shares substantial similarities with Batch-Constrained Q-learning (BCQ) [Fujimoto et al., 2018]. BCQ also involves training a behavior policy and refining it through bounded perturbations. While BCQ employs a conditional variational autoencoder (VAE) to model the behavior policy, the overall framework aligns closely with the method presented in this paper. Given that the primary contribution of the work centers on policy optimization, a more thorough comparison with BCQ, including both conceptual and empirical differences—would strengthen the manuscript.\n\n===comments after rebuttal===\n\n I think that the authors’ response satisfactorily addressed my concerns. I updated the score accordingly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an algorithm for offline preference-based reinforcement learning. The algorithm follows a widely adopted two-step approach commonly used in preference learning.\nIn the first stage, a reward model is trained using a given preference dataset. A policy is then optimized to maximize the expected return as defined by the learned reward model. The main focus of the proposed algorithm lies in the policy optimization during the second stage.\nThe core idea is to reduce the action space by constraining it to the neighborhood of the behavior policy. Specifically, the policy's action space is restricted around the deterministic behavior policy, and the constrained policy is then optimized using a standard reinforcement learning algorithm.\nThe paper also empirically demonstrates the challenges of learning from a reward model trained on preference data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In Algorithm 2, $\\mathcal{L}_{u}$ and $\\mathcal{L}_{\\pi}$ are introduced before they are properly defined in the main text. It would be advisable to revise the presentation to ensure that these terms are explained beforehand or concurrently for better clarity.\n\n- The authors are encouraged to clarify the rationale for choosing SAC in the offline setting, rather than leveraging more recent offline reinforcement learning algorithms. \n\n- The proposed method appears to share conceptual similarities with Batch-Constrained Q-learning (BCQ) [Fujimoto et al., 2018]. While one difference is the use of a deterministic behavior cloning policy rather than a conditional VAE, a more comprehensive comparison is recommended. If there are additional distinctions between the proposed approach and BCQ, those should be explicitly addressed to clarify the novelty and contributions of this work.\n\n- While the paper is focused on preference-based RL, I do not see what part of the proposed method solves specific challenges in preference-based learning. The contribution of the paper appears to lie in offline policy optimization. Please elaborate this point.\n\n- Although the paper is positioned within the domain of preference-based reinforcement learning, it is unclear which aspects of the proposed method directly address the unique challenges associated with preference-based learning. From the current presentation, the primary contribution appears to lie in offline policy optimization. The authors are encouraged to clarify and elaborate on how the method specifically tackles issues inherent to preference-based settings, such as preference elicitation, reward inference, or robustness to noise in human feedback.", "broader_impact_concerns": "I do not have any concerns regarding the broader impact of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aD35kcZOB7", "reviewer": "Reviewer_yXYF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n \nThe author proposes a method for offline reinforcement learning from preference.\n \nThe proposed method demonstrates comparable or better accuracy to existing offline methods with lower training computation time.\n \nThe author visualizes that the proposed method is robust to the learned reward model, training is stable and efficient.\n \n \nWeakness\n \nI think the idea of constraining the action space to well-observed actions is naïve and too pessimistic. Doesn’t the proposed method converge fast but the final score is smaller?\n \nThough the author formulates the proposed method with the action probability, the final algorithm in the continuous case does not obey this probabilistic framework and uses distance to the representative deterministic action. For example, we can use probability by modeling behavior policy with gaussian policy.\n \nThe experiment is conducted on synthesized preference and on continuous action case. I want to see the results on true preference task and on discrete action case.\n \nAlso, I want to see comparison to more existing offline methods (e.g. CQL, TD3+BC, Decision transformer).\n \nFurther, I think the performance of the proposed method depends on the constraint parameter $r$. I want to see the ablation study about this parameter.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for offline reinforcement learning from preference data. The proposed method constrains the policy so that outputs only the action whose probability in the behavior policy is larger than threshold. In the continuous case, the constraint is within the region whose distance to the output of deterministic behavior policy is lower than threshold. The proposed method demonstrates comparable or better performance with small training time than existing methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Major\n \nDiscussion on the technical novelty.\n \nExperimental evaluation on the case of real preference reward and on the case of discrete action.\n \nExperimental comparison to more existing offline methods.\n \nAblation study on the hyperparameter $r$.\n \n \nMinor\n \nProposed method on the continuous space using stochastic behavior policy modeling.", "broader_impact_concerns": "Since this is algorithm paper, I think the risk of ethical problem is small.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)