Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Authors: Zhilong Zhang
Tian Xu
Xinghao Du
Xingchen Cao
Yihao Sun
Yang Yu
Venue: ICML 2025 poster (2025)
Keywords: reward modeling
preference generation
preference-based reinforcement learning
adversarial imitation learning
Primary area: reinforcement_learning

Abstract: In sequential decision-making, the reward function serves as the primary supervision signal, guiding agents to acquire the desired behaviors. Traditional reward modeling methods rely heavily on human expertise, limiting their scalability. Automated preference generation from suboptimal demonstrations has emerged as a promising alternative to address this limitation. This approach first generates preference data from suboptimal demonstrations and then trains reward models based on these preferences. Despite its potential, existing methods often struggle to generate preference data with sufficient coverage, limiting the accuracy and generalizability of the resulting reward models. To overcome this limitation, we propose APEC (Automated Preference generation with Enhanced Coverage), a novel method that improves the coverage of preference data. APEC achieves this by selecting policy pairs with significantly different iteration indices from the whole adversarial imitation learning process. We provide a theoretical analysis to validate that the selected policy pairs provably hold preference relationships. Experimental results demonstrate that APEC consistently outperforms baseline methods in generating preferences with broader coverage across both vector-based and pixel-based control tasks. Consequently, the reward models trained with APEC align more closely with ground-truth rewards, deriving improved policy performance.

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Zhilong Zhang 123+ Tian Xu 123+ Xinghao Du x Xingchen Cao 123
Yihao Sun
Yang Yu 123t
Abstract
1. Introduction
In reinforcement learning (RL) (Sutton & Barto, 2018), the reward function serves as the comnerstone for defining the agent's objective, quantifying its performance, and steering it toward learning optimal behavior by maximizing cumu- lative rewards. However, designing an appropriate reward function remains a significant challenge. An ideal reward function should capture the intended behaviors while pre- venting the reward hacking issue that the policy produces artificially high rewards without achieving the desired behav- iors (Gao et al., 2023). Moreover, it should provide dense and informative feedback, offering actionable guidance to boost efficient policy learning.
In sequential decision-maken the reward func- tion serves as the primary supervision signal, guid- ing agents to acquire the desired behaviors. Tradi- tional reward modeling methods rely heavily on human expertise, limiting their scalability. Au- tomated preference generation from suboptimal demonstrations has emerged promising alter- native to address this limitation. This approach first generates preference data from suboptimal demonstrations and then trains reward models based on these preferences. Despite its potential, existing methods often struggle to generate prefer- ence data with sufficient coverage, limiting the ac- curacy and generalizability of the resulting reward models. To overcome this limitation, we propose APEC (Automated Preference generation with En- hanced Coverage), a novel method that improves the coverage preference data. APEC achieves this by selecting policy pairs with significantly dif- ferent iteration indices from the whole adversarial imitation learning process. We provide theoret- ical analysis to validate that the selected policy pairs provably hold preference relationships. Ex- perimental results demonstrate that APEC consis- tently outperforms baseline methods in generat- ing preferences with broader coverage across both vector-based and pixel-based control tasks. Con- sequently, the reward models trained with APEC align more closely with ground-truth rewards, de- riving improved policy performance. Our code is released at https//gittubcoom/221/////Z
Traditional reward modeling methods can be broadly cat- egorized into three types: manual design (Hwangbo et al., 2019; Kumar et al., 2021), inverse reinforcement learning (IRL) (Ng & Russell, 2000; Abbeel Ng. 2004; Fu et al., 2017a; Luo et al., 2022), and preference-based reward learn- ing (Bradley & Terry. 1952: Christiano et al., 2017; Wirth et al., 2017). Manual design involves crafting reward func- tions by hand, requiring extensive domain expertise and frequent adjustments. This approach is labor-intensive and often impractical for complex tasks. In contrast, IRI. at- tempts to infer reward functions from expert demonstrations, eliminating the need for manual reward design. However, it suffers from inherent ambiguity and relies optimal demonstrations (Amin & Singh, 2016; Kim et al., 2021). limiting its utility to tasks where such optimal demonstra- tions are difficult to obtain. Preference- based reward learn- ing, which learns reward models from preference feedback using the Bradley-Terry model (Bradley & Terry, 1952), has emerged as a promising alternative. Despite this, exist- ing methods typically require substantial human preference, limiting their applicability to large-scale tasks. In summary, traditional reward modeling methods rely heavily on human expertise, limiting their scalability.
Equal contribution. 1: Corresponding author. 'National Key Laboratory for Novel Sofwware Technology, Nanjing Univer- sity 'School of Artificial Intelligence, Nanjing University, China PPolixirai. Correspondence to: Yang Yu
address these challenges, researchers have been explor- ing various solutions, with automated preference generation from suboptimal demonstrations emerging as promising approach (Brown et al., 2020; Chen et al., 2021; Huo et al., 2023; Huang et al., 2023). For example, D-REX (Brown et al., 2020) starts by learning a behavior cloning (BC)
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
(Pomerleau, 1991) policy from suboptimal demonstrations
We provide a theoretical justification for APEC, demon- strating that the value gap bound of the learned policy diminishes as the iteration index increases.
and then generates trajectories with varying preference lev-
els by introducing different uniformly random noise to the
BC policy. Based on this, LERP (Huo et al., 2023) improves the utility of the preference by adding noise directly into the reward space. Despite these innovations, such methods overlook the importance of the coverage of generated pref- erences, which is proven to be crucial for learning accurate reward models (Zhu et al., 2023; Nika et al., 2024). As result, the reward models learned by these methods are not robust and thus cannot be used for subsequent policy learn- ing. The purpose of this paper is to address the following research problem.
We validate APEC across eight continuous control tasks, demonstrating that it can learn reward models that align more closely with true rewards, leading to improved policy performance.
2. Preliminaries
Markov Decision Process. In this paper, we consider finite- horizon Markov Decision Processes (MDPs), represented by the tuple MM AS, p). Here, S and A de- note the state and action spaces, respectively. H signifies the planning horizon, while p stands for the initial state distribution. Besides, P: :S x A + A(S) denotes the transition function where P(s ((s([s.a) a) determines the prob- ability of transiting to state s' given state s and action a. Similarly, r S x A [0. 1] outlines the reward function of this MDP. Without loss of generality, we assume that the reward value is bounded in [0, 1J. A non-stationary policy is denoted by π = { ) with TT : S + A(A), where A (AA) denotes the probability simplex. Here, TA (a|s) repre- sents the probability of selecting action a in state s at time step ht for h E [H]. The quality of policy π is evaluated by policy value: V EE [[>"", r"(sh,, an); P.. Be- sides, for a policy x. we define the state-action distribution d (s, = (1/H) Σ TP (sh = 8, = 0; a;
How automatically generate accurate preferences with broad coverage?
In this paper, we present APEC, an Automated Preference generation with Enhanced Coverage method. This method inspired by the key observation in adversarial imitation learning (AIL) (Ho & Ermon, 2016; Kostrikov al., 2019; Garg et al., 2021; Xu et al., 2024), the value of the learned policy gradually increases as the training progresses, sug- gesting that later policies are preferred over earlier ones. Based on existing analysis on AIL (Xu et al., 2023), we pro- vide a theoretical justification for this observation, demon- strating that the value gap bound of the learned policy di- minishes as the iteration index increases. Building on this insight, APEC chooses policy pairs with significantly differ- ent iteration indexes from the whole AIL training process and then executes them to generate the preference data. In stark contrast to previous methods (Brown et al., 2020; Huo et al., 2023), which generate preference data by merely in- jecting uniformly random noise into a fixed policy, APEC leverages a wide range of policies from the whole AIL train- ing process, resulting in preference dataset with broad coverage.
Reward Learning from Suboptimal Demonstrations. This work studies the setup of reward learning from sub- optimal demonstrations, which aims to infer reward func- tions from imperfect trajectories (Brown et al., 2019; 2020). In this setup, we typically assume the exis- tence of a suboptimal policy, T.. that interacts with the environment to generate a suboptimal dataset. For- mally, D {7o (#1, a1 82, .2...., :H; (x); ; ;a, T (-Isn, 86+1 ~ P(-|sh, ax), vh E [H]] The learner then uses Do to infer reward function To parameterized by o which is subsequently employed to derive an effective policy.
Empirically, we evaluate APEC across five vector-based and three pixel-based continuous control tasks. The empirical results demonstrate that, compared to existing baselines, our method can generate preferences with broader coverage, and the reward models learned from these preferences align more closely with the true environment rewards. In partic- ular, APEC is the only method that consistently produces policies that achieve performance comparable to or even better than demonstrations in all tasks.
Preference- baaseee Reward Learning. Preference-based reward learning (Christiano et al., 2017: Brown et al., 2019) aims to infer rewards from preference dataset. These meth- ods are typically built on the Bradley-Terry (BT) model (Bradley & Terry, 1952). In particular, for pair of trajecto- ries (7', 7?), the BT model assumes that the probability of - being preferred over 72 can be formulated as
We highlight the main contributions of our work below.
We propose an automated preference generation method, APEC, which leverages the policy pairs with different iteration indexes from the AIL training pro- cess, ensuring broad coverage in the generated prefer- ence data.
IP (T) > 72) = o (r r+*))))
Here a(x) = 1/(1 exp( is the logistic function and (*( = Σm ($t, an) denotes the return of the trajec- tory T = (81, (1,,01,.-00 ..,,.... 8H, az). Based on this probability
2

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
modeling, these methods apply maximum likelihood esti- mation learn rewards from the preference dataset Drref
the level of disturbance, and Unif denotes the uniform distribution over the action space. Finally, D-REX generates preferences by executing these disturbed policies with dif- ferent values of e, i.e. (r+, if - ~ IT' and r ~ x² with é < e'. Consequently, the resulting preference data distribution is induced by these disturbed policies. We argue that this distribution exhibits poor coverage. In particular, due to the well-known compounding errors issue (Syed Schapire, 2010), the BC policy n suffers a significant performance gap compared to 70, As result, these dis- turbed policies, interpolated between nBC and Unif fail to provide adequate coverage of high-return regions. We also empirically validate this claim. As shown in Figure 4, the preference data distributions generated by D-REX exhibit relatively poor coverage.
min -B or I-DDe [log (o(ra(Tt) - -ra(T
Here x-)
means that 7- preferred over 7-
3. Method
In this section, we introduce APEC: Automated Preference Generation with Enhanced Coverage. We first highlight the importance of preference data coverage in preference-basee reward learning, a critical issue that previous methods have largely overlooked (Section 3.1). Next, we present the key observation behind APEC: adversarial imitation learning (AIL) can produce a series of policies with increasing values. We also provide a theoretical justification for this observa- tion (Section 3.2). Finally, we describe the resulting APEC method, which incorporates additional techniques such as the Wasserstein distance criterion and random segment crop- ping to further enhance the generalization of reward models (Section 3.3).
3.2. AIL Produces Policy Pairs with Clear Preferences and Good Coverage
This work aims to enhance previous automatic preference generation methods by improving preference data cover- age. The proposed method is inspired by the following key observation: in another imitation learning method, ad- versarial imitation learning (AIL), the value of the learned policy gradually increases as training progresses, as shown in Figure We briefly introduce AlL the following. AlL mimics DO by solving the minimax optimization problem of:
3.1. The Coverage of Preference Data Crucial
This work follows the pipeline of first automatically gener- ating preference dataset and then learning rewards from it. In this preference- beaeee reward learning procedure, the coverage of the preference dataset is a pivotal factor (Zhu et al., 2023; Nika et al., 2024). Specifically, (Zhu et al., 2023, Theorem 5.2) indicates that if policy optimization is performed using the reward learned from the preference dataset, the resulting policy exhibits sub-optimality bound that depends on the following coverage coefficient.
H
H
min max E EDDDO
r(sh, an)
E
Σ -==1
rshh, as)
For the inner optimization, the variable r : 8 A [0.1] can be regarded reward function, aiming maximize the value gap between T and œ. To solve the above mini- max problem, AlL methods alternatively update the policy variable and reward variable in an iterated manner; see Algo- rithm for a standard AIL training procedure. Proposition 3.1 provides theoretical justification for the above obser- vation on the AIL training process.
1/2 Cov(Drref) = [I + AI) A1)-1/2 B(e.a)aae [o(s,a]]
Here o : S A + Rd is the feature function. For tra- jectory T, we use ((T) = (r,))er a(s,a) a) to denote the summation of features along this trajectory and E (1/[Der(()
(x(r+) - ((((t)
Proposition 3.1. Consider the AlL method shown in Algo- rithm 1. For any fixed E (0, 11, with probability at least 1- 6, we have
T(r-))T E Raxd is the covariance matrix of Deref (Dr(D))))) qualifies the coverage of Drer over the target data distribution induced by the optimal policy x* in the feature space. If Dreef exhibits good coverage in the sense that CDv(D) is small, the derived policy enjoys small sub-optimality.
2| An[(1A))
VT - -V"' S 4HH2
+ Satat-
k
However, previous automated preference generation meth- ods (Brown et al., 2020; Huo et al., 2023) often overlook the crucial role of coverage and struggle to generate a prefer- ence dataset with sufficient coverage. For instance, D-REX (Brown et al., 2020) first learns a behavioral cloning pol- icy N BCC from Do. D-REX then designs a disturbed policy of (-1s) = (1 - =)
Here t denotes the policy in the k-th iteration and Sstat 4 2V2H( AA log(1/5)) |DO - denotes the statistical er- ror due to finite samples in D
The proof is deferred to Appendix A. Proposition 3.1 demon- strates that during the AIL training process, the value gap bound of the learned policy diminishes as the iteration index
BC (-Is) + e An)f,A()), where e controls
3

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
high R
AlL with suboptimal demos
compare by policy preference
rollout
RL
D
peet
selected policy pair
preference buffer
reward model T
policy collection
low
i<
Stagel: Collection of AIL Policies
Stuge2: Generation of Diverse Preferences
Stage3: Reward Learning and Evaluation
Figure 1. Overview of APEC. There are three key stages: (1) Collecting policies from the AlL training process. (2) Generating diverse
preferences based on their corresponding iteration indexes. (3) Learning generalizable reward models from generated preferences.
Humandid-22 ./2
walker
verse preferences, and reward learning and evaluation, as illustrated in Figure
5000
Epiade Retum
4000
Collection of AIL Policies. In Section 3.2, we prove that the policy performance bound of the AlL algorithm im- proves for training epochs. For our implementation, DAC (Kostrikov et al., 2019). well-known AlL algorithm recognized for its excellent sample efficiency and conver- gence properties. Additionally, we save checkpoints of the policy model at regular intervals during training. While these policies display diverse behaviors, they consistently show a steady increase in expected cumulative return, which is then used to generate preferences for subsequent steps.
2000
300
200
1000
100
20 Training Steps (x10")
10
15
25
20
40
60
nD
100
Training Frames (x10"))
Demos.[Avg]
AIL
Figure 2.
The learning curves of All. Mujoco and DMControl tasks. The x-axis represents the number of interactions with the environment (frames for pixel-based tasks), and the y-axis repre- sents the policy return. The gray dashed line represents the average return trajectories in suboptimal demonstrations DO
Generation of Diverse Preferences. After policy collec- tion, we construct a preference buffer. Following the guid- ance of Proposition 3.1, we iteratively sample policy pairs n; and Ty from the collected policy set, ensuring that j-í > where ls is the training epoch interval between two policies. For each pair, and Ty in the environment, respectively. However, directly adding (rf, 1f,,,, >>) to the preference buffer presents two challenges: 1) In tasks with high stochasticity, the perfor- mance variance is high among trajectories sampled from the same policy. Although V(*²) expected to be greater than V (=*",, the instance-level performance R(r") < R(T') may not always hold. 2) During training, AIL may experience occasional performance spikes, where policy performance drops dramatically but recovers quickly. To address the above two issues, we introduce a Wasserstein distance crite- rion for doubly robust preference generation. Specifically, we calculate the Wasserstein distance between sampled tra- jectories and suboptimal demonstrations DO, denoted as Σ W(T', =()). We
increases. This may indicate progressive improvement in policy value over successive iterations, suggesting that later policies are preferred over earlier ones.
allow n, and πr to rollout trajectories т;
Building on this insight, we propose constructing policy pairs with distinct preferences by repeatedly selecting two policies with significantly different iteration indices from the AIL training process. These policy pairs are then used to generate the preference dataset. In our approach, the preference data distribution is induced by policies across the entire AIL training process, ensuring broad coverage for two reasons. First, both theoretical (Xu et al., 2020: Rajaraman et al., 2020) and practical (Ho Ermon, 2016; Kostrikov al., 2019) works have demonstrated that AIL effectively mitigates the compounding errors issue in BC, leading to final policies that closely match 7 and induce distribution with good coverage of high-return regions. Second, by drawing from policies throughout the entire AIL training process, the preference dataset captures a broad and representative coverage of the whole state-action space.
d, - Σ W(F', -2) d, then impose the constraint d; d, > 8 for threshold 8. Finally, only trajectory pairs that satisfy both the epoch criterion and the Wasserstein distance criterion conditions are added to the preference buffer. In Section 4.5, we vali- date the effectiveness of the Wasserstein distance criterion through ablation experiments.
3.3. APEC: Automated Preference Generation with Enhanced Coverage
Reward Learning. Reward learning from preferences aims at estimating the reward model that reflects the observed
We present our method, APEC, which consists of three components: collection of AlL. policies, generation of di-

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
preferences. Most approaches are based on the Bradley- Terry model (Bradley & 'eerry, 1952), which assumes that the probability of preferring one trajectory segment over an- other is exponentially related to the sum of the reward model rs estimates for those segments. However, we have found that when the trajectory horizon is long and preference data consist of full trajectories, directly applying the Bradley- Terry model often fails to capture the desired behavior in complex tasks. To address this, we randomly extract seg- ments from full trajectories and use these partial trajectories for training as proposed in T-REX (Brown et al., 2019). The loss function for this approach is defined in Equation (1).
for DMControl.
Experimental Challenges. Compared to previous works, our experimental setup is significantly more challeng- ing. (1) Pixel-based continuous control tasks are included. (2) Fewer demonstrations are available, for Mujoco tasks, only one suboptimal demonstration is provided, whereas for DMControl tasks, ten suboptimal demonstrations are used. (3) The suboptimal demonstrations are selected from high-return range, making it more difficult to achieve better-than-demo performance.
4.2. Reward Model Accuracy
min- Bo H-I]. [log (a(ro(Thant)) - ro (o((Thnt+))))
To address Q1,
we compare the learned reward accuracy of APEC with that of other baselines. Specifically, we evaluate performance on the test dataset using three key metrics: (1) Reward correlation, the Pearson correlation between learned and true rewards. (2) Return correlation, the Pearson correlation between learned and true returns. (3) Preference accuracy, the accuracy of preference labels given by learned returns on randomly constructed trajectory pairs from the test dataset.
x 1~D
(1)
4. Experiment
We perform comprehensive experiments to answer the fol- lowing questions. Q1: How accurate is the learned reward model? How does its performance compare to the baselines? (Section 4.2) Q2: Can the learned reward model derive effective policy? (Section 4.3) Q3: How accurate are the generated preferences? Do they offer broader coverage than baselines? (Section 4.4) Q4: How does each component in APEC affect performance? (Section 4.5)
Tables I and 2 present the reward and return correlation results across various tasks and methods, highlighting the effectiveness of APEC compared to baseline approaches. Specifically, Table 1 shows that APEC achieves the highest reward correlation scores, consistently surpassing baselines. Notably, in the Ant and HalfCheetah tasks, APEC reaches reward correlation values of 0.94 and 0.91, respectively. Table 2 presents return correlation results, further reinfore- ing APEC's superiority in deriving high-quality policies. APEC consistently achieves the highest return correlation scores, reaching near-perfect alignment (0.99-1.00) in sev- eral vector-based tasks. The method also performs excep- tionally well in pixel-based control environments, with re- turn correlation values averaging 0.96, significantly higher than competing approaches. Overall, our method surpasses the baselines in nearly all tasks, indicating that the pref- erences generated by APEC lead to more accurate reward models.
4.1. Experimental Setup
Benchmark. We evaluate our method on five tasks from the feature-based Mujoco benchmark (Todorov et al., 2012), and three tasks from the pixel-based DMControl bench- mark (Tassa et al., 2018), which are leading benchmarks in reinforcement learning and imitation learning that provide diverse set of continuous control tasks.
Suboptimal demonstrations and test dataset. We train RL agents using SAC (Haarnoja et al., 2) for MuloCo tasks and DrQ-v2 (Yarats et al.) for DMControl tasks, ensuring sufficient training epochs and treating the final policy check- point the optimal one. To generate suboptimal demon- strations, select policy checkpoints with performance ranging from 50% to 80% of the optimal. Additionally, we uniformly sample 1,000 trajectories from the replay buffers during training to create test datasets, enabling the evalua- tion of the generalization of the learned reward functions across different algorithms.
Environment
SSRR
AIRI
D-REX
LERP
APEC(ours)
Ast
0.16
0.61g0.01
-0.07
-D 00.00000
0.94:0.00
HaltCheetah
0.33
0.48
0.122000
0. 0.12:c.10
0.91 0.91
Hopper
0.37
0.77
0.00
0.00gass
0.83
Humanoid
00050mm
0,23
0:02200
0.0220000
0.04 0.04_0.04
Walker2d
0.25
0.24g9.0
0.19g00s
0.19
0.88
Average
0.23
0.47
0.05
0.05
0.72
Primary Baselines. We compare APEC with four state-of- the-art baselines: D-REX (Brown et al., 2020), AIRL (Fu et al., 2017a), SSRR (Chen et al., 2021), ), and LERP (Huo et al., 2023). To control for the impact of sample size on training results, we ensure that all methods are trained with an identical number of sampled trajectories. Specifically, we use 2000 trajectories for Mujoco and 10,000 trajectories
cheetahirun
-0.1220m
-U. 00
0.0%
U. 0.70
0.73
walker
0.06gnu
0.00=
0.600g00:
0.0220000
1.88
walker.walk
0006g009
-0. -0.172000
0.01
g.54
0.76.0.00
Average
0.00
-0.12
0.59
0.62
0.79
Table I. Reward correlation Mujoco and DMControl tasks over random seeds.
5

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Ant-v2
HalfCheetah-v2
Hopper-v2
Humanoid-v2
6000
15000
4000
5000
Epsode Return
5000
12500
3000
4000
10000
3000
7500
2000
3000
2000
2000
1000
1000
50
100
150
200
250
50
100
150
200
250
0
50
106
150
200
250
50
100
150
200
250
Training Steps (x10*)
Training Steps (x10*)
Training Steps (x10f)
Training Steps (x10*)
2d-k2erdd-22
cheetah run
walker run
wall ker walk
1000
Episode Return
4000
600
600
800
3000
400
400
2000
400
200
200
1000
250
0
50
100
150
200
250
40
80
120
160
200
0
40
80
120
160
200
40
80
120
160
200
Training Steps (x10*)
Training Frames (x10f)
Training Frames (x<10%)
Training Frames (x10f)
(Avg)
SSRR
AIRL
D-REX
LERP
APEC(ours)
Figure 3. Overall performance on policy learning evaluation on Mujeco and DMControl tasks. Here the x-axis is the number training steps (frames for pixel- pixel-based tasks) and the y-axis is the policy return. The solid lines are the mean of results while the shaded region corresponds to the standard deviation over 5 random seeds.
Enviromment
D-REX
LERP
APEC(ours)
Environment
SSRR
AIRL
D-REX
LERP
APEC(ours)
Ard
0.57=2.2
0.44500o:
0.49
01.99
0.66__.11
0.69_n.02
0.37
0.34
0.85g.002
HallChieethh
0.92g0.s
0.42go.0
0,40
HallCheetah
0.83_a.11
0.90_z.01
0.692o.10
0.69
0.97
Hopper
0.9020000
0.97=0o:
0.47
0.46ga.0o
1,00
Hopper
0.91_oo.2
0.84_o.0
0.68
0.68 1o.0
0.95g0.00
Hamonoid
2.99
-009012000
008772017
-0800g0
0.05
Humanoid
0.10
0. 0.23:30.0
0.23
0.78
Walker2d
0.98
0. 0.00
0.75
0.84ga.2
1.00
Walker2d
0.88 0..0
0.65, _o.os
0. 0.80 _0.10.1
0.81
0,95 jo.0
Average
0.89
0.42
0.06
0.05
0.99
Average
0.83
0.63
0.55
0.55
0,90
cheethh.run
-0 -0:43
0.94
0.94
0.98
walkerzun
0. 1.35
-.01
0.80 o
0.87
0.98 oc.0
chectah_run
0.39
0.44
0.87=0.00
0.87
0.92.0.00
walk
0.12
g.65
0. 0,70
0.92
walkerrrun
0. 0.55
0.55
0.78
0.78: 20.0m
0.90
Average
0.06
0.22
0.81
0.84
0.96
walker_walle
0.54
0.41
0. 2..500.00
0.05.0.00
0.76.0.00
Average
0.49
0.47
0.77
0.77
0.86
Table 2. Return correlation on Mujoco and DMControl tasks over 5 random seeds.
Table 3. Preference accuracy on Mujoco and DMControl tasks over 5 random seeds.
Furthermore, our method outperforms others in labeling preferences for unseen trajectory pairs, as shown in Table 3. APEC achieves an average labeling accuracy of 90% for Mujoco tasks and 86 % for DMControl tasks on the test dataset. In contrast, all other baselines fail achieve 80% accuracy in any task category.
trajectories, while for DMControl tasks, all available trajec- tories are used. We believe that well-trained reward model should enable the agent to achieve performance superior to suboptimal demonstrations.
The results of the policy learning evaluation are presented in Figure 3. APEC achieves better-than-demo performance on 7 out of the 8 tasks, and for the Humano dd-22, also demonstrates comparable performance, showcasing the strong generalization of our reward models. While other baselines have demonstrated effectiveness in prior works (Fu et al., 2017a; Chen et al., 2021; Huo et al., 2023), they struggle to match demonstration performance under our more challenging experimental setup, which involves fewer demonstrations, more complex tasks, and a stricter hetter-than-demo requirement. Our analysis reveals that
4.3. Policy Learning Evaulation
To answer Q2, we directly use the learned reward model for RL training and assess its reusability by comparing con- vergence performance. Specifically, we use SAC for Mu- joco tasks and DrQ-v2 for DMControl tasks. Additionally, to accelerate convergence, the training buffer was initial- ized with trajectories sampled from the AIL step (Stage in Figure 1). For Mujoco tasks, we uniformly sample 10
6

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
this is primarily caused by reward hacking in the baselines, whereas our method benefits from the broader coverage of generated data, thus avoiding this problem. Appendix C.2 provides a detailed discussion of reward hacking.
which crucial for achieving better-than-demo policies.
dproj
min SEDO
||
s'112
(2)
D
RED
4.4. Generated Preference Analysis
To answer Q3, analyze the generated preferences of D-REX, SSRR, and APEC from generation accuracy and generation coverage two perspectives.
Hopper-v2
walker run
High
5.0
2.5
x-axii A
0.0
Generation accuracy. The experimental results for genera- tion accuracy are presented in Table 10. In both vector- vector-based and pixel-based control tasks, APEC achieves impressive accuracy of 98%, consistently outperforming the baseline methods by a significant margin. SSRR and D-REX, which are based on noise injection, achieve 92% and 80% respec- tively. This improvement accuracy is critical for learning robust reward models, as it ensures that the preferences used for training are reliable.
-2.5
-5.0
10
20
50 X-axis Displacement
X-akis Displacement
5.0
X-axii Vel oellyctty
о N b
REX
2.5
Pensity
0.0
-2.
-5.0
10
20
0
50
X-axis Displacement
X-axis Displacement
Environment
SSRR
D-REX
APEC(ours)
5.0
APEC(ours) x-axii Velocity 2
Ant
0.100
0.1940.00
0.0940.00
2.5
HalfCheetah
1.54
2.13+0.02
1.20
0.0
Hopper
0.29 0.20
0.71
0.23 0.23
Humanoid
0.002
1.14400001
1. 1.0220.04
-5.0
Low
Walker2d
0.48 10.01
0.77 10.02
0.45 10.00
10
20
50 Xaaxis Displacement
Xaaxis Displacement
Average
0.68
0.99
0.60
Figure 4. Visualization of generated preference distributions on Hopper-v2 and walker run tasks. From top to bottom, the rows represent the generated samples for SSRR, D-REX, and APEC, respectively. In each plot, the x-axis represents the x-axis displace- ment, while the y-axis represents the x-axis velocity.
cheetah_run
0.90
0.8540.00
0.82+0.00
walker run
0.68
0.79
0.63
walker_walk
0.53
0.56+0.00 00
0.49
Average
0.70
0.73
0.65
Table 4. Projection distances on Mujoco and DMControl tasks over random seeds.
4.5. Ablation Study
In this section, we investigate the effect of the Wasserstein distance criterion and the number of collection policies on the generalization of the reward model. We use reward correlation mentioned Section 4.2 as the evaluation metric. The results are shown in Figure 5.
Generation coverage. To assess the coverage of the gen- crated dataset, we first calculate the projection distance dpeoj from the test dataset to the generated dataset as de- fined in Equation 2, where DO and D' are the generated dataset and test dataset respectively. A smaller projection distance typically indicates better sample coverage. The results presented in Table 4 show that APEC exhibits the smallest projection distance from the test dataset to the gen- erated samples except for the humanoid task. We believe that APEC outperforms other baselines due to its ability to generate the most well-covered samples while maintaining high production accuracy. Additionally, we visualize the sample distributions for Hopper-v2 and walker .un in Figure 4. These visualizations reveal that APEC-generated samples provide broader coverage in both the x-axis dis- placement and x-axis velocity dimensions, especially in high-return regions (the return of these locomotion tasks is generally strongly correlated with the x-axis displacement),
Wasserstein distance criterion. We observe that the Wasserstein distance criterion is crucial to improving the ac- curacy of generated preferences. Its importance is especially evident in Mujoco tasks, where the training process may exhibit instability. By combining the epoch criterion with the Wasserstein distance criterion, we substantially enhance the generalization of reward models.
Number of collection policies. For the Mujoco tasks, reduce the number of policies used for preference collection from 400 to 4, and for the DMControl tasks, from 50 to 5. Despite this reduction, the number of generated pref- erences remains constant (1000 for Mujoco and 5000 for DMControl). Our results reveal that using smaller set of
7

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
policies decreases the diversity and coverage of the gener- ated preferences, which in turn hampers the generalization of the reward function. In DMControl tasks, where the model inputs are images and the tasks are more complex, the diversity of the policies becomes even more critical for the generalization of reward models.
like IRL and AIL. preference- reinforcement learning (PBRL) leverages human preferences train reward models. The pioneering work by (Christiano et al., 2017) introduced neural networks to PBRL, demonstrating their capacity to capture complex behaviors. Building on this foundation, (Ibarz et al., 2018) improves the sample efficiency of PBRL by utilizing expert demonstrations, while (Cao al., 2024) advances the approach by relaxing the optimality constraint trajectories. Additionally, FTB (Zhang et al., 2023) uses diffusion model for trajectory generation, achieving more robust policy learning by preferred trajectory augmentation. In contrast, APEC does not require human annotations or optimal trajectories and can achieve robust reward learning from a small number of suboptimal trajectories.
Ablation Study Results
0.8
APEC
fewer policies
.a w-criterion
0.7
0.6
Reward Correlation
0.5
0.4
Reward Learning from Suboptimality. Standard PBRL methods typically require extensive manual labeling of tra- jectory pairs, posing huge burden on human annotators. Active learning methods (Eric et al., 2007) have been used to alleviate this burden by selectively presenting the most infor- mative preference pairs for labeling. addition, approaches such as (Palan et al., 2019) and (Ibarz et al., 2018) enhance the quality of preference data by incorporating constraints from demonstrations. Recently, methods have emerged to fully automate preference generation, reducing eliminat- ing the need for manual annotation. D-REX (Brown et al., 2020) was the first to inject uniformly random noise at vary- ing levels into the BC policy to generate preferences for reward learning. LERP (Huo et al., 2023) enhances reward learning from preferences in D-REX by introducing noise directly into the reward space, rather than the action space. SSRR (Chen et al., 2021) employs AIRL for policy train- ing, using sigmoid function to enable reward regression. What distinguishes APEC from these existing methods is its capacity to produce preference data with much broader coverage by leveraging the diverse policies produced the adversarial training process, enabling more robust reward learning.
0.3
0.2
0.1
0.0
Mujoco
DMControl
Environment
Figure 5. Ablation study on Mujoco and DMControl Tasks. We use reward correlation as the evaluation metric. with results averaged across all tasks over 5 random seeds. The blue bars represent APEC, the orange bars show the results without the Wasserstein distance criterion, and the red bars indicate the results obtained using fewer policies.
5. Related Works
Reward Modeling, Reward modeling a critical aspect of reinforcement learning, traditionally relying on expert knowledge, which often requires significant human input and incurs considerable costs (Hadfield-Menell et al., 2017). One of the most widely used methods for reward model- ing, inverse reinforcement learning (IRL), aims to recover reward functions from expert behaviors, thus minimizing human involvement in the reward design process. In appren- ticeship learning (Abbeel & Ng. 2004), the reward model is optimized to maximize the performance gap between expert demonstrations and the learned policy. MaxEnt IRL (Ziebart et al., 2008) formulates reward updates as maximum likeli- hood estimation problems, integrating a maximum entropy principle to promote policy diversity and better represent expert behavior. As an extension of IRL, adversarial imita- tion learning (AIL) algorithms learn rewards through a fully adversarial process inspired by GANs (Goodfellow al., 2014), achieving high data efficiency and replicating expert policies with minimal expert demonstrations. For instance, AIRL (Fu et al., 2017b) learns a reward function by training a discriminator to distinguish between expert and generated trajectories, resulting in a reward model that is robust to changes in dynamics and environmental properties. Un-
6. Discussions and Limitations
In this paper, we introduce APEC, framework for auto- mated preference generation with enhanced coverage for reward learning. Building on the theoretical and empirical evidence that AIL generates policy pairs with clear pref- erences and good coverage, APEC selects policies from different stages of the AIL training process to produce pref- erence data that is both diverse and accurate. This approach addresses key limitations in existing preference generation methods, which often struggle to generate sufficiently cover- age samples, limiting the generalizability learned reward models. Our experimental results on rector-based and pixel- based control tasks demonstrate that APEC significantly improves the coverage of generated preferences compared
8

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
baseline methods. The reward models learned from these preferences align more elosely with ground-truth rewards.
Acknowledgements
This work was supported by the NSFC (62495093), the NSFC Fundamental Research Program for Young Schol- ars (PhD Candidates) of the National Science Founda- tion of China (623B2049) and Jiangsu Science Foundation (BK20243039).
Limitations and Future Works. Although APEC repre- sents significant progress, there is still considerable room for further improvement in future work. For example, APEC still requires interaction with the environment to generate preference data, making the extension of APEC offline fully data- driven reward learning framework, APEC may still suffer from reward hacking in more complex tasks. A key chal- lenge for future work is integrating human prior knowledge into the reward learning process to mitigate such risks.
settings an interesting problem. Additionally,
Impact Statement
This paper contributes to advancing the field of reinforce- ment learning and reward modeling. While there are po- tential societal implications of this work, we believe none require specific attention or discussion here.
References
Abbeel, and Ng. A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Amin, K. and Singh, S. Towards resolving unidentifia- bility in inverse reinforcement learning. arXiv preprint 1xXv::6601.0669 2016.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method paired comparisons. Biometrika, 39(3/4):324 345- 1952.
Brown, D., Goo, W., Nagarajan, P., and Nickum, S. Extrap- olating beyond suboptimal demonstrations via inverse re- inforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Brown, D. S., Goo, W., and Niekum, S. Better-than- demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning, pp. 330-359. PMLR, 2020.
Cao, X., Luo. F.M., Ye, Xu, T., Zhang, Z., and Yu. Limited preference aided imitation learning from imper- fect demonstrations. In Forty-first International Confer- ence Machine Learning, 2024.
Chen, L., Paleja, R., and Gombolay, M. Learning from sub- optimal demonstration via self-supervised reward regres- sion. In Conference on robot learning, pp. 1262-1277. PMLR, 2021.
Christiano, P. F., Leike, J., Brown, T.. Martic, M., Legg. S., and Amodei, D. Deep reinforcement learning from human preferences. Advances neural information pro- cessing systems, 30, 2017.

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Bric, B., Freitas, N., and Ghosh, A. Active preference learning with discrete choice data. Advances in neural information processing systems, 20, 2007.
Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsou- nis, V., Koltun, V., and Hutter, M. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26)) 2019.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint ar arXiv: /2448, 2017a.
Ibarz, B., Leike, J., Pohlen, T., Irving. G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning, arXiv preprint 17I.:11700..148,, 2017b.
Kang. Y., Hsieh, C.-J., and Lee, C. M. Robust lipschitz
bandits to adversarial corruptions. Advances in Neural Information Processing Systems 37, 2023.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, pp- 10835-10866, 2023.
Kim, K., Garg, S., Shiragur, K., and Ermon, S. Reward iden- tification in inverse reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 5496-5505. PMLR, 2021.
Garg, D., Chakraborty, S., Cundy, C., Song, J., and Ermon, S. Iq-leam: Inverse soft-q learning for imitation. In Advances in Neural Information Processing Systems 34, pp. 4028 0008-4039,550 2021.
Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., and Addressing sam-
Tompson, Diseriminator-actor--c ple inefficiency and reward bias in adversarial imitation learning. In Proceedings of the 7th International Confer- ence Learning Representations, 2019.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial nets. In Advances in Neural In- formation Processing Systems 27, pp. 2672-2688 2014.
Kumar, A., Fu, Z., Pathak, D., and Malik, J. Rma: Rapid motor adaptation for legged robots. Robotics: Science and Systems XVII, 2021.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pp- 1856-1865, 2018.
Luo, F-M.,, Cao, X., Qin, R.-J,, and Yu, Y. Transferable reward learning by dynonics-agnostic discriminator en- semble, arXiv, 2206 0000000 2022.
Ng, A. Y. and Russell, S. J. Algorithms for inverse rein- forcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, pp. 663- 670, 2000.
Hadeield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Nika, A., Mandal, D., Kamalaruban, P., Tzannetos, G., Radanovic, G., and Singla, A. Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences. In Proceedings of the Forty-first International Conference on Machine Learning, 2024.
Haldar, S., Mathur, V., Yarats, D., and Pinto, L. Watch and match: Supercharging imitation with regularized optimal transport. Conference on Robot Learning. pp. 32-43, 2023.
Ho, J. and Ermon, Generative adversarial imitation learn- ing. In Advances in Neural Information Processing Sys- tems 29, pp. 4565-4573, 2016.
Palan, M., Landolfi, N. C., Shevchuk, G., and Sadigh, Learning reward functions by integrating hu- man demonstrations and preferences. arXiv preprint arXiv: DrXiv:/906.088 1906.08928,, 2019.
Huang. S. Zhao, J., Li, Y., and Wang. L. Learning prefer- ence model for Ilms via automatic preference data gen- eration. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, pp. 9187-9199, 2023.
Pomerleau, D. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1): 88 98-97,11 1991.
Huo, L., Wang. Z., and Xu, M. Learning noise-induced reward functions for surpassing demonstrations in imita- tion learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7953-7961, 2023.
Rajaraman, N., Yang, L. F., Jiao, J., and Ramchandran, K. Toward the fundamental limits of imitation learning. In Advances in Neural Information Processing Systems 33, pp. 2014-2924, 2020.
10

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Shalev-Shwartz, S. Online learning: Theory, Algorithms, and Applications. Ph.D. Thesis, The Hebrew University, 2007.
Twelfth International Conference on Learning Represen- lations, 2023.
Zhu, B., Jordan, M., and Jiao, Principled reinforeement learning with human feedback from pairwise k-wise comparisons. In International Conference Machine Learning, pp- 43037-43067. PMLR, 2023.
Shalev-Shwartz, S. Online learning and online convex opti- mization. Foundations and Trends in Machine Learning. 4(2):107-194, 2012.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT press, 2018.
Ziebart, B. D., Maas, A. L.., Bagnell, A., and Dey, A. K.
Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd AAAI Conference on Artificial Intelligence, pp. 1433-1438, 2008.
Syed, U. and Schapire, R. E. A reduction from appren- ticeship learning to classification. In Advances in Neu- ral Information Processing Systems 23, pp. 2253-2261, 2010.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv: 1801,00690, 2018.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model- based control. In 2012 IEEE/RSJ inter- national conference on intelligens robots and systems, pp- 5026-5033. IEEE, 2012.
Weissman, T., Ordentlich, E., Seroussi, G., Verdu, S., and Weinberger, M. J. Inequalities for the 11 deviation of the empirical distribution. Hewlett-Packard Labs, Techical Report, 2003.
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. A survey of preference-basee reinforcement learning meth- ods. Journal of Machine Learning Research, 18(136): 1-46, 2017.
Xu, Li, Z., and Yu, Error bounds of imitating policies and environments. In Advances in Neural Information Processing Systems 33, pp- 15737-15749 2020.
Xu, T.. Li, Z., Yu, Y. and Luo, Z.-Q. Provably efficient adversarial imitation learning with unknown transitions. In Proceedings of the 39th Conference Uncertainty in Artificial Intelligence, pp. 2367-233 2023.
Xu, T., Zhang, Z., Chen, R., Sun, Y., and Yu, Y. Provably and practically efficient adversarial imitation learning with general function approximation. In The Teirth-eigtth Annual Conference on Neural Information Processing Systems, 2024.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Master- ing visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations.
Zhang, Z., Sun, Y. Ye, J., Liu, T.SS., Zhang, J., and Yu, Y. Flow to better: Offline preference- reinforce- ment learning via preferred trajectory generation. In The

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
A. Proof of Proposition 3.1
In this part, we present the proof of Proposition 3.1. First, Algorithm outlines the procedure standard adversarial imitation learning algorithm.
Algorithm 1 Adversarial Imitation Learning
Require: Initialized reward rl, initialized policy ø1, reward step size n"" (18H)A) policy step size 1y" (2I for = 1.2....... do
Rollout H* to compute the state-action distribution d=t
Update the reward by projected gradient descent:
k+ PR (1* - ()))
where (*(r) = VR = H E
E(na)Esn r(s, (s,a)= (s, a)) and d= is the empirical state-
= action distribution based on 200 Calculate the Q-value function T0 , I for policy th
Update the policy by KL-regularized policy optimization:
(-|s) h
max Bamp()) PEA(A)
[0 #t 1
(s.0)]
1 17t
DKL (p()). (+1s))
end for
Ensure: nK sampled uniformly from (
**
Now proceed to present the proof of Proposition 3.1. First of all, we have the following error decomposition.
V*
V*
V=
v*k
max rER
K
(n) 1
yT"r ir p.atr
max PER
K
1=1
1
V
vaa
v ,t
Vx
ppt
VR ma
K
max
FER
K
k=
k=
R=1
Term
Term
Here R (r: S A + [0, 1]} is the set of reward functions. Here equation (a) follows the definition of xt K For Term I, we have that
K
1
vo jn
max
K
TER
K
K
1
P v r + or
p +
pst,l
+ V
- p
max
K
PER
k
1 K
1
p= - v
V int
2 max V= v FE
V ,
1
K
PER
K
1
max PER
Dx y ,
p&c ,
max
K
FER
Here x= (1/1DB)) ETEDE DH (T((())), the estimated policy value for policy 7o regarding reward r.
12

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
For the first term in RHS, we have that
K s
K p=
V V
V= ir -A*
min 1
V ,
k=
which is exactly the regret for solving the online optimization problem with loss functions leverage the following lemma to upper bound this regret.
(r) = (())=v'', Vr We
Lemma A.1. Consider Algorithm we have
K (*() 2H /
min rER
where (*(r)) = H
(8 - d (s.
Proof of Lemma A.I. Lemma A.1 is direct consequence of the regret bound of online gradient descent -Shalev-Shwarth
2012). To apply such regret bound, need verify that the iterate norm ||r]| has upper bound; 2) the gradient norm v (r) ll2 also has an upper bound. The first point casy to show, i.c., ||r|| S VIS|[A| by the condition that E (r: x A [0, 1]}. For the second point, we have
V. = HH
Σ
(s) (s. 2
(s,0)ESxA
I H
Σ
(s,))) + (s)
(s.a)ESxA
S V2H
Invoking Corollary 2.7 in (Shalev-Shwartz, 2012) with B VIS|AA, L V211 and of' B/(LV2)) (25]AA/(2HXK) finishes the proof.
Lemma A.1 demonstrates that
K
K x x - v**.())
pa
-
/2H
max TER
Furthermore, we have that
4 max v= _ gr
4max H FER
Σ
-d r(s,a)
(8.0)ESXA
< 4H
(..
dm
According to the concentration inequality for 4noormm (Weissman et al., 2003; Kang et al., 2023), with probability at least 1 - 8, we have
2 L ln(1/) l|(1/)
d (..)) - d00 (o)(
[Do)
13

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Consequently, for Term we have that
K
K
va
V
Vxo
int
V
int
max
PER
k=
k=1
K
K
V
p ,r
snt
1
+ 4 max VT - p= EER
S
max
K
PER
k=1
k=1
|
@|S||AA|nn(1/00
S
2H
4H
K
[Do]
For Term II, according to the policy difference lemma, have that
1 V V
1
Σ T (-1s)) (1ss)) 1 Qhe (sh. -) +)
For each fixed (s,h) h) E S x [H] we analyze the term
K Σ (1)))s). -0 -x (s..)) (-1()))) -A" (s.
R1lss) - (+1ss, 0 (s.
which is exactly the regret of online linear minimization with respect to loss function (*( = -aEd E p(a) (s, a) for probability distribution p E ((A). From the perspective of online linear minimization, we can view the policy update as the well-known mirror descent update. Based on the theory of online mirror descent (Shalev-Shward 2007), we have that
K
( (1s))
-% (s, (-1s), 2 "²"""" n (s, log(lAA)
+
17t
K
Σ
ah(a 8) (0R (0"""""(s,,) (s,
17t
2
k= aEE
In([Al)
17"
H²K
17th
2
2m
The last equation holds by choosing 17t ((2n ())) KH² Consequently, have the following upper bound Term II.
K
K
H
V
(T (-1sm) - *#(+s))), Qr 2tt (sn, (sn, .)) T0
(
K
K
1=1
k=1
2 In((AA))
S 2H
K
Combining the upper bounds on Term I and Term II yields that
|SSAAA]
|| ln(1/8)
2 1m(A
< 2H V
$2H
4H
2H2
K
|DOI
K
2|S|||A| In((A)
2|S||A ln(1/))
< 4H²
+ 4H
Do
We complete the proof.
B. Implementation Details
B.I. Implementation Details of APEC
Policy Collection. We choose DAC (Kostrikov et al., 2019), an AIL algorithm known for its sample efficiency and excellent convergence performance, to collect policies. Moreover, we use SAC (Haarnoja al., 2018) for policy updates in
14

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
vector-based tasks and DrQ-v2 (Yarats et al.) for pixel-based tasks, as outlined in Algorithm 1. Specifically, for vector-based MuJoCo tasks, train policies using single suboptimal demonstration for 400 iterations with 5000 steps per iteration. Checkpoints of the policy model are saved at the end of each iteration, labeled by the corresponding iteration number. For pixel-based DMControl tasks, we train policy with 10 suboptimal demonstrations. The training spans 4M frames, with evaluations conducted every 20,000 frames. During each evaluation phase, a checkpoint of the policy is saved. We select the official codes of PAao(Cao et al., 2024) and ROT(Haldar et al., 2023) as the codebase for Mujoco and DMControl tasks respectively and use their recommended default hyperparameters as listed Table 5 and
Hyperparameter
Value
Hyperparameter
Value
Hidden layers
2
Hidden layers
2
Feature dimension
50
Hidden dimension
128
Hidden dimension
1024
Auto Alpha
False
Activation
ReLU
Activation
ReLU
Batch size
256
Batch size
256
Buffer size
150000
Buffer size
1000000
Gradient penalty coefficient
10
Gradient penalty coefficient
0
Exploration steps
2000
Exploration steps
10000
DDPG exploration schedule
limear(1.01,5500000))
Learning rate
le-3
Learning rate Optimizer
le-4
Optimizer
Adam
Adam
Table Hyperparameters AlL. in Mujoco tasks.
Table 6. Hyperparameters of AIL in DMControl tasks.
Preference Generation. each iteration, we sample policy pairs n, and n, from the policy set and rollout trajectories, only trajectory pairs that satisfy both the epoch criterion and the Wasserstein distance criterion conditions are added the preference buffer. Table 7 lists the hyperparameters used in this process. For Mujoco tasks, we collect 1,000 preference pairs, while for more challenging DMControl tasks, we collect 5,000 pairs.
Task
epoch threshold
distance threshold ō
Mujoco
10
0.1
walker-rum run
10
0.01
walker_wall
15
0
cheetah run
15
0
Table 7. Hyperparameters of preference generation.
Reward Learning. For Mujoco tasks, the reward is trained for 100 epochs, each using 1,000 samples. For DMControl tasks, all samples in the preference buffer used in each iteration and are trained for 50 epochs. In addition, implement random shift augmentation for based data recommended in DrQ-v2 to improve the robustness of the reward by encouraging the model to learn invariant features under various shifts in the input. The hyperparameters of reward learning are listed in Table 8 and 9.
Reward Evaluation. We evaluate the reward by using it to train RL agents with the same policy learning algorithms applied in the policy collection stage (SAC for Mujoco and DrQ-v2 for DMControl). We initialize the buffer with selected trajectories from the AIL buffer the policy collection stage speed up convergence. For Mujoco tasks, 10 trajectories are chosen, while for DMControl tasks, all available trajectories are utilized. Both this stage and the Policy Collection stage share the same hyperparameters for policy learning.
C. Additional Experimental Results
C.1. Reward Accuracy
We visualized the return correlation and the reward correlation of across all tasks, as shown in Figures and Figure 7. Here, we only present the results for seed=1 due to space constraints.
15

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Hyperparameter
Value
Hyperparameter
Value
Hidden layers
Hidden layers
2
Feature dimension
50
Hidden dimension
512
Hidden dimension
1024
Activation
ReLU
Activation
ReLU
Batch size
256
Batch size
256
Weight decay
le-3
Weight decay
le-4
Learning rate
le-4
Learning rate
le-4
Optimizer
Adam
Optimizer
Adam
Table 8. Hyperparametees of reward model in Mujoco tasks.
Table 9. Hyperparameters of reward model in DMControl tasks.
C.2. Generated Preference Accuracy
The generated preference accuracy is summarized in Table 10.
Environment
SSRR
D-REX
APEC(ours)
Ant
0.89=0.03
0.96
0.94
HalfCheetah
0.00
0.94
1.00
Hopper
0.96_0.03
0.89+0.02
+0.00
Humanoid
0.95
0.31
0.99 +0000
Walker2d
0.88_0.05
0.70+0.21
0.99
Average
0.93
0.76
0.98
cheetah_run
0.91
0.94+0.00
0.98
walker-run
0.93
0.72+0.02
1.00
walker
0.89
0.89
0.9510.01
Average
0.91
0.85
0.98
Table 10. Generate accuracy on Mujoco and DMControl tasks over 5 random seeds.
C.3. Additional experiment on Atari
Table 11 shows an additional experiment on the Atari game Pong, which features discrete action space. Results show that APEC consistently outperforms both DREX and LERP in terms of task performance, demonstrating its broader applicability. Notably, for the Pong experiment, used alternative AlL method, IQ-Learn (Garg al., 2021). generate preference data, which further highlights APEC's generalizability across different AIL algorithms.
Table 11. Task performance comparison on Pong
Method
Demo Mean
Task Performance
DREX
3.7
-9.5
LERP
3.7
-20.0
APEC (ours)
3.7
-0.44
C.4. Sensitivity analysis on hyperparameters
As shown in Table 12, 13 and 14, We perform ablation studies on key hyperparameter choices, including the training epoch interval, Wasserstein distance threshold, and segment length for reward learning, across both feature- based Hal fChehhahh and pixel-based based walker.rum tasks. The "default" refers to the values used in our original experiments. The results show that APEC not sensitive different hyperparameter choices.
16

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Table 12, Ablation study training epoch interval
Task
5
10 (default)
50/20
HalfCheetah
11,957
12,232
10,019
walker.run
710
701
591
Table Ablation study on Wasserstein distance threshold (higher is better)
Task
Lower
Default
Higher
Ha fCheetah (0.05/0.110.22)
13,291
12,232
8,848
walker.run (0.005/0.01/0.022)
709
701
736
Table 14. Ablation study on segment length (higher is better)
Task
Shorter
Default
Longer
Hal FCheetah (100/500/1000)
12,658
13,434
12,232
walker.run (5/10/20)
712
701
696
Table 15. Experiments on four MuJoCo tasks under two conditions: lower quality demonstrations and more demonstrations, In each value pair "x/y", x represents the performance the demonstrations, while y the performance of the policy trained with the reward model learned from APEC.
Performance
Ant
HalfCheetah
Hopper
Walker2d
low demo
2413 1522
4232 / 7874
1325 / 3300
2205
2089
high demo
4105 5221
7313 / 12232
2275 3310
3706 5394
high 10 demo
4179 4880
7385 / 14700
2388 / 3325
3728 5390
C.5. Reward Hacking Analysis
Figure 8, 9, 10, 11, and 12 demonstrates the relationship between the predicted return and ground truth return for the baselines and APEC. The learned return of all algorithms increases with the number of training steps. However, only APEC ensures that the ground truth return increases throughout the process, while the other methods show a decline in ground truth return, indicating reward hacking. These plots show the strong robustness and generalizability of APEC's learned reward.
D. Computational Resources
All experiments were performed on an RTX 4090 GPU platform. In Stage I (Policy Collection), Mujoco tasks require approximately 24 hours to complete 2 million training steps. while DMControl tasks take about 10 hours to complete I million frames. Stage 2 (Preference Generation) does not require GPU resources. In Stage 3 (Reward Model Training), Mujoco tasks typically take around 20 minutes, while DMControl tasks require approximately hours.
17

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
SSRR
D-REX
(ours)
Prodicted Ratur
800
Prodicted Ratur
Predicted Retumm
100
correp
Prodicted Batuan
100
cormejo.18
Prudiced Raturn
6000
200
50
30
Ant
300
2000
2500
5000
0
2500
5000
2500
5000
2500
5000
2500 5000
Ground Retur
Return
Ground Return
Retum
Ground Truth Retum
SSin
AIRL
D-REX
LERP
APEC (ours)
Prodicted Ratur.
2000
com-D
5000
comY-0.97
Prediccted Reeurn
com--0.93
Predictted Reeurn
comoo
Prodicted Raturn
20000
comeo
2500
HalfCheetah
Predkctted
-200
-200
20000
2500
400
400
-2000
-5000
5000
10000
0
5000
12000
5,00
20000
5000
10000
5000 10000 Ground Truth Retur
0
Ground Truth Return
Ground Truth Retur
Ground Truch Retum
Ground Truch Retur
SSRR
AIRL
D-REX
LERP
APEC (ours)
Predicted Return
35000
comro
0.99
Predictus Return
4000
0,99
Predicted Return
200
comm-00.33
Predicted Return
100 comn--0.55 53
Predicted Return
com-- 1.00
6000
Hopper
20000
2000
8000
2000
50
2000
2000
2000
2000
0
2000
Ground Truth Rctum
Ground Truth Retum
Ground Truth Rcturn
Ground Truth Rcturn
Ground Truth Acturn
SSRR
AIRL
D REX
LERP
APEC (ours)
com-+B 9M
401m 99
com---0.6 63
cormm- 69
<orr-0 92
8cu
Prodicted Rotum
300
Humanoid
Prodictod
800
Prodicted
-2000
Predicte Ret
Predicted Biet
200
200
4000
2500
5000
2500
5000
2500
5000
2500
5000
2500
5000
Ground Truth Retum
Ground Truth Retur
Ground Truth Return
Ground Truth Retum
Ground Truth Retum
SSHIL
AIRL
D-REX
LERP
APEC (ours)
Prodicied Retum
com-0.98
98
200 corn-O
62
Prodicced Raturn
200
com-0.99
200
com+-0.99
Predicted Renure
10000
com-1.00
200
7500
Walker2d
Predicted
200
Predicted
100
5000
300
400
50
2500
2000
4000
4000
2000
4000
Ground Truth Raturn
Ground Truth Retum
Ground Tuuh Raturn
Ground Truth Ratum
Graund Truth Belurn
SSRR
DREX
LERP
APEC tours)
com- 0.82
comn 0.60
1000
comme 96
0om-0.37
comwe..e 0.0
Predeced Return
Pradided hocurn
Prediced Ratuun
1100
cheetah_run
-00
Prddcted
50c
Predictdd
200
110
2500
-20
250
250
750
500
750
250
500
750
Retum
Truth
Ground
Retur
Ground
cormo
come 81
com-0
comm0.76
Predlctdd team
Fredicttdd Retur
Prosiiood Rotur
Prediczed netur
Fredected neturn
walker_run
-200
-300
-000
500
1500
$00
-2020
-200
250
500 750
230
305
750
230
500
750
250
500
750
250
500
750
SSAR
ARL
DREX
lours)
1000
1000
dee Retum
Retum
Prudi doud Rutuun
Rtetum
Predi toul muturs
-500
walker_ walk
Prodicted
-440
-1000
-1000
-1500
11300
110
500
1800
500 Ground buch Retum
1000
500
1000
500
1000
500
1900
Ground Recum
Ground Neturn
Ground Iruhh Rutum
Ground Tuth Retur
Figure 6. Return correlation visualization. Blue dots represent trajectories generated for training reward models, and orange dots represent additional trajectories not seen during training. Here, the x-axis represents the ground truth return, while the y-axis represents the return predicted by the learned reward models.
18

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
0,28
cor--00.00
0:50
6,02
0.50
com--6
Predicted Reward
Predicued Rowwad
0.23
0.25
Prodicted Rowerd
6.00
Ant
Prodectd
G
-0 25
0.30
-10
-0.75
corni
10
comru
00 25
26
Prodicted Revord
Roooard
Mudil hoourd
Predicre Rewwrd
Producted Roward
20
HalfCheetah
10
1
-30
10
10
20
18
10
com-io 24
10
75
Predicted Reword
Producted Roweard
Prodicted Noward
Prodicted Roward
3.0
Prodectoa Reward
Hopper
0.0
-05
0.0
25
5.0
0.0
2.5
5.0
0.
2.5
5.0
8.0
2.5
5.0
0.5
2.5
SSKK
AIRL
D-REX
LER?
Prefcen (Ours) corred. D0
corn=2.03
25 corr-0.2
come-
corn
Predicted Roward
Roward
0.0
PRODICM Hawaadd
Roward
Prodicae Humaar
Humanoid
Ground Truth hewand
Cround Truth Beoard
Ground Tuch Bemwar
Ground truth heeaadd
Crourd Thuth Beward
SSAR
REX
Preffon (Ours)
SUPTW0L19
comr-O,33
-0.2
Lurre 26
comroo
Predicted Reward
Rosord
dedd Reward
Pridictes Beward
Walker2d
10
-30
6
Truch Reward
Ground T:th Reward
Ground Truth Reward
SSAA
DREX
APEC
0.28
comm-0.11
corr-D699
1.5 com--0.33
cornwo.7e P6
Roward Rowars
Predt Thearrs Rheaard
dodd Rowars Roward
cred Howward
Prodicted Reward
8:00
3.0
cheetah_run
Prodicted
0.05
8.5
0.0
-0.30
-05
Ground Tuth Roward
Ground Roward
Ground Roward
Ground Reward
Ground Truch Roward
Prodected Roward
-0.01
Predicted texeend
0.2
Producted Roward
com--0.40
Prodicted manard
57
Prodected Seaward
8.02
walker_run
18.00
0.0
0.04
0.0
055
1.0
1.3
0.8
0.3
1,5
2.5
3.0
5.5
3-0
1.5
0.0
05
1.
13
0.0
03 1.0 1.3
SSAR
DREX
LERP
APEC jour)
03 corm-o
Prodecdd Revond
Predicted Revvard
0.05
Prediced Reoord
+0.
Predered ed Reowrrt
0.00
2.0
walker_wwr paak
0.05
-0.10
2..0
6
-0.15
-25
-0.0
Ground Trish Rewerd
Ground Bith Reward
Ground Thuth Reward
Grunn Trish Reward
Ground Thuth Revard
Figure 7. Reward correlation Return correlation visualization. Orange dots represent state-action pairs the test dataset. Here, the x-axis
represents the ground truth reward, while the y-axis represents the reward predicted by the learned reward models.
19

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Ant-v2
HalfCheetah-v2
Hopper-v2
Humanoid-v2
1.0
1.0
1.-
1.0
-10
0.8
0.8 0.8-
0.8 0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
8.2
0.2
0.0 0.2
02
0.2 0.-
0.0
200
250
253 cheetah run
0.0
0.0
-0.0
50
0.0
Training Steps (x10")
Training Steps (x10")
Training Steps (x100)
Training Steps (x10*)
Wälker2d-12
walker run
walker _alk
1.0
1.0.
1214-
1.0
1.0
1.0
0.8
1.0 0.8
1.0
0.8
0.8
8.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4 0.4
04 0.4
0.4
0.4
0.
0.2
0.2
0.2
0.-
0.0
0.0
0.0
0.0
150
200
250
0.0
160 Training Frames (x10")
40
80
120
200
160 Training Frames (x10*)
40
80
120
200
40
80
120
160
200
Training Steps (x10")
Training Frames (x (x10*)
Real episode return
Predicted episode return
Figure& 8. The relationship between the predicted return and the ground truth return in SSRR. The left y-axis represents the normalized episode return, while the right y-axis also indicates the normalized episode return for comparison. The x-axis denotes the number of training steps.
Ant-v2
HalfCheetah-v2
Hopperv2
Humanold-v2
1.0
1.2
1.0
1.0
0.8
1.0..8
1.0.8
0.8
0.8
0.6
0.8
0.6-
0.8
0.6
0.6
0.6
3.6
0.4
3. 0.4
0.4
0.4
0.4
0.2
020.
02 n2
62
0.0
-00
0.0
0.0
0.0
0.0
0.0
0.0
Training Steps (x10*)
Training Steps (x10*) cheetah run
Training Steps (x10*) walker run
Training Steps (x10*)
1.0
Walker2d-42
1.0
1.0
1.0
walker walk
1.0
1.0
1.0
0.8
0.8
aa
0.8
0.
9.8
9.8
36
0.6
0.6
0.4
0.4
04 0.4
0.4
0.4
0.2
0.2
02
0.
62
02-
0.0
0.0
0.0
150
200
250
40
120
180
200
0.0
140 Training Frames (x10*)
40
120
200
0.0
160 Training Frames (x10f)
40
120
200
Training Steps (x10%)
Training Frames (x10*)
Real episode return
Predicted episode return
Figure 9. The relationship between the predicted return and the ground truth return in AIRL. The left y-axis represents the normalized episode return, while the right y-axis also indicates the normalized episode return for comparison. The x-axis denotes the number of training steps.
20

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Ant-v2
HalfCheetah-v2
Hopper-v2
Humanoid-v2
1.0
1.0
1.0
128
1.0
1.0
0.8
0.8-
1.00,8
Von
0.8
0.8
0.0
0.75
0.6
0.6 0.6-
0.6
0.6
0.6
0.50
0.4
0.4
0.25
0.2
0.2
0.2
3.002
0.0
92-
0.0
200
250
0.0
250
250
-0
50
0.0
Training Steps (x10")
Training Steps (x10%)
Training Steps (x10*)
Training Steps (x10*)
Wälker2d-42
cheetah run
walker run
wal ker walk
1.0
1.0
1.0
1.0
100
1.0
1.0
0.8
0.
0.8
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
04
0.4
0.4
0.2
0.2 0.2
9.2
6.2 0.-
0.0
100
150
200
250
0.0
40
80
120
160
200
160 Training Frames (x10*)
40
80
120
200
40
80
120
160
200
Training Steps (x10")
Training Frames (x10")
Training Frames (x (x10*)
Real episode return
Predicted episode return
Figure 10. The relationship between the predicted return and the ground truth return in D-REX. The left y-axis represents the normalized episode return, while the right y-axis also indicates the normalized episode return for comparison. The x-axis denotes the number of training steps.
Ant-v2
HalfCheetah-v2
Hopper-v2
Humanold-v2
1.0
12
1.0
0.8
a.n 0.8
0.8
0.8 0.8
0.8
3.8
0.6
0.6 0.
0.6
0.6
0.4
0.4 0.4
0.4 0.4
02
0.2
8.2
0.2
0.2 n2
0.0
0.0
0.0
0.0
0.0
0.0
Training Steps (x10*)
Training Steps (x10*) cheetah run
Training Steps (x10*) walker run
Training Steps (x10*)
1.0
Walker2d-22
1.0
1.0
1.0
wälker walk
10
1.0
1.0
0.8
1.00.8
aa 0.8
0.8
0.8
0.8
36
0.6
0.6
8.6
0.4
0.4 0.4
04 0.4
0.4
0.4
0.2
9.2 n.2-
32 n2
62
0.0
3.0
0.0
200 Training Steps (x10%)
150
250
40
120
180
200
0.0-
40
8C
120
140
200
0.0
160 Training Frames (x10f)
40
120
200
Training Frames (x10*)
Training Frames (x10*)
Real episode return
Predicted episode return
Figure 11. The relationship between the predicted return and the ground truth return in LERP. The left y-axis represents the normalized episode return, while the right y-axis also indicates the normalized episode return for comparison. The x-axis denotes the number of training steps.
21

---

Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Ant-v2
HallCheetah-v2
Hopper-v2
Humanoid-v2
-1.0
1.0
-1.0
1.0
0.8
0.8
0.8
0.8 3.8-
0.8
0.6
0.6
0.6
0.6
36
0.4
0.4
04
0.4
0.2
02 0202-
0.2 n.
0.0
0.0
3.
0.0
.0.0
0.0
0.0
Training Steps (x10%)
Training Steps (x10*) cheetah run
Training Steps (x10*) walker run
Training Steps (x10*)
1.0
Walker2dd42
1.0
1.c
1.0
wal ker walk
1.0
0.8
0.8
0.8
0.8
0.8
0.6 0.6
0.6
0.6
0.6
0.4
0.4
34 0.4
0.4 3.4
0.4
0.2
0.
0.2
9.9
100
150
200
250
160 Training Frames (x10*)
40
80
129
200
160 Training Frames (x10*)
40
80
120
200
160 Training Frames (x10f)
40
80
120
200
Training Steps (x10")
Real episode return
Predicted episode return
Figure 12. The relationship between the predicted return and the ground truth return in APEC. The left y-axis represents the normalized episode return, while the right y-axis also indicates the normalized episode return for comparison. The x-axis denotes the number of training steps.
22


Reviews:
[{"review_id": "kjZPIOgMRb", "reviewer": "Reviewer_Gguy", "summary": "APEC addresses the challenge of generating high-quality preference data for reward modeling in sequential decision-making without human expertise. Unlike previous approaches that rely on injecting random noise into fixed policies, APEC selects policy pairs with significantly different iteration indices from the adversarial imitation learning (AIL) process. The authors provide theoretical validation that later policies in the AIL process are preferred over earlier ones, ensuring that the generated preference data has broader coverage. Empirical evaluations demonstrate that APEC produces reward models that align more closely with ground-truth rewards, resulting in improved policy performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions!\n\n**Q1**: Could the authors provide a comparative analysis of computational costs between APEC and baseline methods? \n\n**A1**: The table below shows the computation costs (training time) of all methods on the MuJoCo task. We can see that APEC and the other methods require similar training time, with the majority of time spent on policy learning using the learned reward. Additionally, we present the training time of each stage in the reward learning process. We find that calculating the W-distance incurs only a slight increase in training time. \n\n\n|  | SSRR | AIRL | DREX | LERP | APEC |\n| --- | --- | --- | --- | --- | --- |\n| reward learning | 4-6h | 2h | 2-4h | 2-4h | 4-6h |\n| policy learning | 24h | 24h | 24h | 24h | 24h |\n\n|  | SSRR | AIRL | DREX | LERP | APEC |\n| --- | --- | --- | --- | --- | --- |\n| AIL/BC training | 2h | 2h | 5min | 5min | 2h |\n| collect demos  | 2-4h | / | 2-4h | 2-4h | 2-4h |\n| compute distance | / | / | / | / | ~20min |\n| PBRL training | ~5min | / | ~5min | ~5min | ~5min |\n| total | 4-6h | 2h | 2-4h | 2-4h | 4-6h |\n\n\n**Q2**: Have the authors explored APEC's performance on tasks beyond continuous control, such as discrete action spaces or goal-oriented environments? Dependence on DAC specifically might limit generalizability to other AIL algorithms. \n\n**A2**: We conducted an additional experiment on the Atari game Pong, which features a discrete action space. Results show that APEC consistently outperforms both DREX and LERP in terms of task performance, demonstrating its broader applicability. Notably, for the Pong experiment, we used an alternative AIL method, IQLearn [1], to generate preference data, which further highlights APEC's generalizability across different AIL algorithms.\n\n| task performance | Demo Mean | DREX | LERP | APEC (ours) |\n| --- | --- | --- | --- | --- |\n| pong | 3.7 | -9.5 | -20.0 | **-0.44** |\n\n[1] IQ-Learn: Inverse soft-Q Learning for Imitation. NeurIPS’21.\n\n\n**Q3**: How sensitive is APEC's performance to hyperparameter choices? \n\n**A3**: We perform ablation studies on key hyperparameter choices, including the **training epoch interval**, **Wasserstein distance threshold**, and **segment length for reward learning**, across both feature-based HalfCheetah and pixel-based walker_run tasks. The \"default\" refers to the values used in our original experiments, and we show the overall task performance here, as suggested by the reviewer. The results show that APEC is not sensitive to these hyperparameters.\n\n- Epoch Interval\n    \n    |  | 5 | 10 (default) | 50 |\n    | --- | --- | --- | --- |\n    | HalfCheetah | 11957 | **12232** | 10019 |\n    |  | 5 | 10 (default) | 20 |\n    | walker_run | **710** | 701 | 591 |\n- Distance Threshold\n    \n    |  | 0.05 | 0.1 (default) | 0.2 |\n    | --- | --- | --- | --- |\n    | HalfCheetah | **13291** | 12232 | 8848 |\n    |  | 0.005 | 0.01 (default) | 0.02 |\n    | walker_run | 709 | 701 | **736** |\n- Segment Length\n    \n    |  | 100 | 500 | 1000 (default) |\n    | --- | --- | --- | --- |\n    | HalfCheetah | 12658 | **13434** | 12232 |\n    |  | 5 | 10 (default) | 20 |\n    | walker_run | **712** | 701 | 696 |\n\n\n**Q4:** The method assumes relatively monotonic improvement during AIL training, which might not hold in highly stochastic environments.\n\n**A4**: We would like to clarify that the theoretical analysis (Proposition 3.1) does **not** assume monotonic improvement during AIL training. Instead, we indeed prove that the value gap bounds of policies learned during AIL process increase as the iteration index $k$ decreases. This suggests that the preference relationship between two policies holds if their corresponding iteration indexes differ sufficiently, which further inspired our algorithmic design.\n\nTo handle with the case where the environment is highly stochastic, APEC leverages the Wasserstein distance metric to help choose the policy pair, which enhances the robustness of preference generation. \n\n\n**Q5**: Relatively high-quality demonstrations despite claims of challenging setup and exploration of APEC's performance with varying amounts of demonstration data.\n\n**A5**: To better understand how the quality and quantity of demonstrations impact overall task performance, we conduct experiments on four MuJoCo tasks under two conditions: lower quality demonstrations and more demonstrations. In each value pair “x/y”, x represents the performance of the demonstrations, while y is the performance of the policy trained with the reward model learned from APEC. The results demonstrate that APEC can still achieve a satisfying performance in tasks with low-quality and scarce demonstrations.\n\n| **Performance** | Ant | HalfCheetah | Hopper | Walker2d |\n| --- | --- | --- | --- | --- |\n| low 1 demo | 2413 / 1522 | 4232 / 7874 | 1325 / 3300 | 2205 / 2089 |\n| high 1 demo | 4105 / 5221 | 7313 / 12232 | 2275 / 3310 | 3706 / 5394 |\n| high 10 demo | 4179 / 4830 | 7385 / 14700 | 2388 / 3325 | 3728 / 5390 |", "claims_and_evidence": "The paper's claim that later AIL policies are preferred over earlier ones is adequately supported through both empirical and theoretical evidence. Figure 2 clearly demonstrates increasing policy returns over training iterations across multiple environments. Proposition 3.1 provides mathematical validation by proving that the value gap between expert and learned policies decreases as iterations increase, with appropriate consideration for statistical error.", "methods_and_evaluation_criteria": "The proposed methods are well-structured and appropriate for automated preference generation. The three-component approach (collecting AIL policies, generating diverse preferences using both epoch differences and Wasserstein distance criteria, and adapting reward learning for trajectory segments) addresses both theoretical foundations and practical challenges. The Wasserstein distance criterion intelligently handles trajectory variance and performance spikes, while the segment-based approach to reward learning sensibly adapts the Bradley-Terry model for long-horizon tasks. \n\nHowever, some weaknesses exist: the approach relies on carefully tuned hyperparameters (k for epoch intervals and δ for Wasserstein distance); dependence on DAC specifically might limit generalizability to other AIL algorithms; the Wasserstein distance criterion adds computational overhead; the segment-based reward learning introduces another design choice that could impact performance; and the method assumes relatively monotonic improvement during AIL training, which might not hold in highly stochastic environments.", "theoretical_claims": "I examined Proposition 3.1, which claims the value gap between expert and learned policies decreases with iterations. The theoretical claim effectively supports APEC's core design principle.", "experimental_designs_or_analyses": "The experimental design is generally sound with key strengths: diverse benchmarks (MuJoCo and DMControl), clearly defined suboptimal demonstrations (50-80% of optimal), test dataset for generalization assessment, controlled sample sizes across methods, and appropriate evaluation metrics.\n\nLimitations include: restriction to continuous control tasks, relatively high-quality demonstrations despite claims of challenging setup, focus on correlation metrics rather than task performance, limited discussion of computational efficiency or training stability, and no exploration of APEC's performance with varying amounts of demonstration data.", "relation_to_broader_scientific_literature": "APEC relates to the broader scientific literature in several key ways:\n\n1. It extends reward learning from suboptimal demonstrations, building on work like AIRL while removing requirements for optimal demonstrations or direct human feedback.\n\n2. It advances preference-based reward learning by addressing coverage limitations in methods like T-REX and D-REX, replacing noise injection with a principled approach using training progression.\n\n3. It leverages theoretical properties of adversarial imitation learning (GAIL, DAC), particularly monotonic improvement during training.", "essential_references_not_discussed": "No significant omissions of relevant literature were identified that would impact understanding of the paper's contributions.", "other_strengths_and_weaknesses": "While APEC represents an advancement in automated preference generation, several weaknesses deserve attention: the method introduces significant computational overhead through full AIL training and Wasserstein distance calculations without discussing efficiency implications; experiments are limited to continuous control tasks, raising questions about generalizability to discrete actions or goal-oriented problems; there's no concrete path toward offline settings despite acknowledging this limitation; the vulnerability to reward hacking remains unaddressed for complex environments; the approach depends on hyperparameters with limited sensitivity analysis or tuning guidance; and the ablation studies could be more comprehensive to clarify each component's contribution.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "1. Could the authors provide a comparative analysis of computational costs between APEC and baseline methods? This would help assess the practical implications of the added complexity from AIL training and Wasserstein distance calculations.\n\n2. Have the authors explored APEC's performance on tasks beyond continuous control, such as discrete action spaces or goal-oriented environments? Understanding generalizability across diverse domains would strengthen the method's broader applicability.\n\n3. How sensitive is APEC's performance to hyperparameter choices, particularly the minimum training epoch interval (k) and Wasserstein distance threshold (δ)? Additional analysis showing performance across different parameter settings would clarify robustness.\n\n4. The theoretical analysis assumes monotonic improvement during AIL training. How does APEC perform when this assumption is violated due to training instabilities or high environment stochasticity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PBRhxz6UUc", "reviewer": "Reviewer_YpdB", "summary": "In this paper, an automated preference generation method, namely APEC (Automated Preference generation with Enhanced Coverage), is proposed, which claims that broad coverage in the generated preference data can be ensured by leveraging the policy pairs with different iteration indices from the AIL training process. Experimental results on eight continuous control benchmark tasks demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Whether the claim that a progressive improvement in policy value over successive iterations can be achieved is correct or not.\n\n**A1:** We apologize for this confusing claim. Here we want to claim that Proposition 3.1 can indicate that policies from the AIL process will establish a preference relationship when their corresponding iteration indexes differ significantly. To see this, on one hand, Proposition 3.1 essentially proves a lower bound on $V^{\\pi^k}$ ($V^{\\pi^k} \\geq f(k)$). On the other hand, due to the proximal update nature in AIL, it is easy to prove an upper bound on the value difference between $\\pi^k$ and the initial policy $\\pi^1$. In particular, we can prove that\n$$\nV^{\\pi^k} - V^{\\pi^1} \\leq H^2 (\\ln(|\\mathcal{A}|))^{\\frac{1}{4}} K^{\\frac{3}{4}}.\n$$\nThis provides an upper bound on $V^{\\pi^k}$ ($V^{\\pi^k} \\leq g(k)$). By combining the lower and upper bounds, we can establish that for two policies $\\pi^{k_1}$ and $\\pi^{k_2}$ whose indexes $k_1$ and $k_2$ differ sufficiently such that $f(k_1) \\geq g(k_2)$, the preference relationship $V^{\\pi^{k_1}} \\geq V^{\\pi^{k_2}}$ holds. This insight inspires us to leverage the AIL training process to generate preference data.\n\nWe appreciate your valuable question and will fix this claim in the revision.         \n\n**Q2:** Whether such a claim can theoretically guarantee a board coverage of the preference data with significantly different iteration indices.\n\n**A2:** We would like to clarify that we do **not** claim that Proposition 3.1 can theoretically guarantee a board coverage of the preference data. Rather, it establishes that policies from the AIL training process will hold a preference relationship when their corresponding iteration indexes differ sufficiently. This provides the rationale for leveraging the AIL in our preference data construction methodology.\n\nFor the coverage property, we empirically validate that the preference data constructed based on the AIL process can provide a wider coverage than previous methods; see Table 4 and Figure 4 for details. This improved coverage likely stems from AIL's inherent advantages over BC—specifically, AIL-trained policies typically achieve better values by avoiding the compounding errors that plague BC approaches. Therefore, the preference data collected by policies from the AIL could have a better coverage on high-return regions.      \n\n**Q3:** Some more recent baselines should be compared and discussed.\n\n**A3**: We add a more recent baseline DRAIL [1]. We use the recommended hyperparameters from the source code. APEC still achieves a better performance than DRAIL.\n\n|**Performance**|Ant|HalfCheetah|Hopper|Humanoid|Walker|\n|---|---|---|---|---|---|\n|DRAIL|1425|3699|3316|138|3586|\n|APEC|5221|12232|3310|4953|5394|\n\n[1] Diffusion-Reward Adversarial Imitation Learning. NeurIPS’24.\n\n**Q4**: The work is only verified on some simple benchmark.\n\n**A4:** As noted by the reviewer, most works use these benchmarks as testbeds. We emphasize that our setup is more challenging compared to prior works. Specifically, we test on pixel-based control tasks and use fewer demonstrations. APEC consistently achieves strong performance across these challenging tasks, showing the potential benefits for real-world tasks.\n\n\n**Q5**: Can only the Wasserstein distance criterion be utilized for the preference generation?\n\n**A5**: We use the L2 distance as an alternative. We find that APEC with L2 distance can still have a satisfying performance.\n\n|reward corr|Ant|HalfCheetah|Hopper|Humanoid|Walker|\n|---|---|---|---|---|---|\n|L2|0.86|0.88|0.71|0.01|0.86|\n|Wasserstein|0.94|0.91|0.83|0.04|0.88|\n\n**Q6**: How about the performance of a random policy collection with the combination of the Wasserstein distance criterion?\n\n**A6**: We conduct experiments using only the Wasserstein distance criterion for constructing preference pairs. The results show that this leads to significantly lower performance across all five tasks. This further reinforces the importance of epoch interval filtering.\n\n|**reward_corr**|Ant|HalfCheetah|Hopper|Humanoid|Walker2d|\n|---|---|---|---|---|---|\n|Wasserstein|0.70|0.72|0.39|0.00|0.57|\n|APEC|0.94|0.91|0.83|0.04|0.88|\n\n**Q7**: Does the number of the demonstrations have a high impact on the final performance?\n\n**A7**: We conduct experiments with different numbers of demonstrations. The results show that the number of demonstrations has a limited impact on the final performance.\n\n|#demo|Ant|HalfCheetah|Hopper|Walker2d|\n|---|---|---|---|---|\n|1|5221|12232|3310|5394|\n|10|4830|14700|3325|5390|\n\n**Q8**: Curious on achieving better-than-demo performance.\n\n**A8**: In certain tasks with structured true reward (e.g., linear w.r.t the state feature), PBRL could identify the underlying structure, resulting in a generalizable reward. Such a generalizable reward can guide policy learning in unseen regions, thereby achieving better-than-demo performance.", "claims_and_evidence": "Experiments are conducted to support that different iteration indexes from the AIL training process can generate broad preference data coverage. However, as Proposition 3.1 only shows that with the increase of the iterations, the value gap bound of the learned policy diminishes, I am not certain whether the claim that a progressive improvement in policy value over successive iterations can be achieved is correct or not. It is noticed that a Wasserstein distance criterion is utilized for doubly robust preference generation.", "methods_and_evaluation_criteria": "Yes, experimental results on several benchmark problems show the effectiveness of the proposed method.", "theoretical_claims": "Yes, the correctness of the Proposition 3.1 is checked. But as mentioned, it only shows with the increase of the iterations, the value gap bound of the learned policy diminishes. I am not certain whether such a claim can guarantee theoretically a board coverage of the preference data with significantly different iteration indices.", "experimental_designs_or_analyses": "Yes, the soundness/validity of the experimental designs is checked, and only several simple benchmark problems are adopted to verify the effectiveness of the proposed approach. It will be better that the proposed approach can be verified on some real-world applications.", "relation_to_broader_scientific_literature": "The key contribution of the paper is to propose an automated preference generation method, which is critical in the learning of the reward models in RL.", "essential_references_not_discussed": "Necessary references are cited and discussed.", "other_strengths_and_weaknesses": "Strengths:\nThe paper aims to address an important issue in the literature, that to increase the preference buffer coverage to make the learned reward model to align more closely to ground-truth rewards, the presented method is novel and interesting, experimental results on several benchmark problems also show the effectiveness of the proposed method. \n\nWeaknesses*:\n1. As mentioned, it only shows with the increase of the iterations, the value gap bound of the learned policy diminishes. I am not certain whether such a claim can guarantee theoretically a board coverage of the preference data with significantly different iteration indices.\n2. The baselines used are works published in 2017, 2020, 2021, and 2023 respectively, as this is a rapidly developing research area, some more recent baselines should be compared and discussed. \n3. The work is only verified on some simple benchmark problems, it is noticed in the literature, most work uses these benchmarks as the testbed, but it is still welcome to show the potential benefits of the proposed approach in some real-world applications.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Can only the Wasserstein distance criterion be utilized for the preference generation?\n2. In the experiments, it is shown APEC can even achieve better-than-demo performance on 7 out of the 8 tasks, can the authors provide further explanations of the reason behind this phenomenon?\n3. Different number of demonstrations are used in the experiments, namely, one suboptimal demonstration for Mujoco task and ten demonstrations for DMControl tasks. Does the number of the demonstrations has a high impact on the final performance? \n4. It is noticed the Wasserstein distance criterion is utilized for the preference generation, and ablation study show the importance of this criterion in generating the preference data. But no ablation study is conduct on the role of the proposed policy collection, how about the performance of a randomly policy collection but with the combination of the Wasserstein distance criterion?\n5. For other questions, please refer to the weaknesses part of the review.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nNsdWTzXQP", "reviewer": "Reviewer_aJ5F", "summary": "The approach of automatically generating preference data can reduce human intervention, but it has limited generalizability and coverage. To solve this problem, the authors propose APEC, which addresses these challenges by selecting policy pairs with significantly different iteration indices from the whole adversarial imitation learning process. As a result, the reward models trained with APEC align more closely with the ground-truth rewards and consistently generate preference data with broader coverage, demonstrating superior performance compared to existing methods.\n\n## update after rebuttal\n\nI agree that it can provide insights for researchers in this particular area, and maintain original recommendation, weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Need to check whether there are more recent and suitable baselines available\n\n**A1**: Thank you for your valuable suggestions. We add two more recent reward learning baselines, BC-IRL [1] and DRAIL [2], and evaluate them on the MuJoCo benchmark. We use the recommended hyperparameters from the source code and slightly tune them. The results are shown in the table below, where we find that APEC still achieves the best performance.\n\n| **Performance** | Ant | HalfCheetah | Hopper | Humanoid | Walker |\n| --- | --- | --- | --- | --- | --- |\n| BC-IRL | -156721 | -48835 | 1084 | 97 | -4.82 |\n| DRAIL | 1425 | 3699 | 3316 | 138 | 3586 |\n| APEC (ours) | 5221 | 12232 | 3310 | 4953 | 5394 |\n\n[1] BC-IRL: Learning Generalizable Reward Functions from Demonstrations. ICLR’23.\n\n[2] Diffusion-Reward Adversarial Imitation Learning. NeurIPS’24.\n\n**Q2**: It would be valuable to explain whether there are other autonomous pipelines that operate without relying on human preference data, and if such approaches exist, what advantages the method using suboptimal data offers in comparison. Including this discussion would provide important insights for understanding the significance of this study.\n\n**A2**: There are other autonomous methods that can operate without human preference data. Notably, LLM-based labeling [1][2] has recently emerged as a popular approach for generating preference data. However, the accuracy of LLM-generated labels is not always guaranteed due to the potential hallucinations of LLMs. Moreover, for vector-based tasks, LLMs are currently incapable of providing accurate labels because LLMs cannot understand such abstract vectors. In contrast, constructing preference pairs from suboptimal data offers a rather principled approach to generating accurate preference data. For instance, APEC builds on the theoretical property of AIL that the value gap bounds decrease as the iteration number increases.\n\n[1] Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning. arXiv preprint ,2025\n\n[2] Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model. arXiv preprint ,2025\n\n**Q3:** Regarding the graph in Figure 2, are there specific environments where the AIL process does not match the performance of the demonstrations? If so, is there an analysis of the performance loss resulting from that gap?\n\n**A3**: In our experiments, there is no environment where AIL fails to match the performance of the demonstrations.", "claims_and_evidence": "The authors argue that the proposed method, APEC, use pairs with different iteration indexes from the Adversarial IL training and provides wider coverages of preference data. The authors provide a clear explanation of the use of AIL, the criteria used in the process of sampling preference pairs, and these are well supported by both qualitative and quantitative results.", "methods_and_evaluation_criteria": "Overall methods and evaluations are plausible. For example, to avoid the errors coming from occasional performance spike, this work provides a Wasserstein distance criterion. As shown in Figure 5, these additional criteria have shown to contribute to further performance improvement. Beyond performance gains, the authors also suggest through qualitative results that the preference coverage is broader compared to previous studies, which demonstrates the effectiveness of the proposed methodology. Furthermore, the selection of benchmarks and baselines (SSRR, LERO) appears to be appropriate. (but need to check whether there are more recent and suitable baselines available.)", "theoretical_claims": "The authors provide a theoretical analysis that the selected policy pairs provably hold preference relationships. This paper also provides a theoretical justification for using the AIL process.", "experimental_designs_or_analyses": "The experimental design and analysis seem valid.", "relation_to_broader_scientific_literature": "Compared to the preference-based RL using suboptimal data, there is a significant amount of research on preference-based RL under the assumption that human preference data is available. Overall, reducing human effort is the practical direction, but it would be helpful to broaden the scope a bit. For example, it would be valuable to explain whether there are other autonomous pipelines that operate without relying on human preference data, and if such approaches exist, what advantages the method using suboptimal data offers in comparison. Including this discussion would provide important insights for understanding the significance of this study.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "The proposed study is simple yet effective. Despite its simplicity, it appears to be supported by appropriate theoretical proofs.", "other_comments_or_suggestions": ".", "questions_for_authors": "Regarding the graph in Figure 2, are there specific environments where the AIL process does not match the performance of the demonstrations? If so, is there an analysis of the performance loss resulting from that gap?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NlYtkVNzYE", "reviewer": "Reviewer_LmzP", "summary": "The paper presents APEC, a novel approach to improving reward model learning by generating diverse preference pairs from different stages of adversarial imitation learning. This ensures broader coverage of the learning process. Theoretical analysis shows that later AIL policies are generally preferred over earlier ones. Experiments on both vector-based and pixel-based control tasks demonstrate that APEC outperforms existing methods, leading to reward models that better align with ground-truth rewards and produce stronger RL policies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The method is dependent on that later policies are better than earlier ones, but this may not always hold due to training fluctuations. A discussion on how this randomness impacts learning would be valuable.\n\n**A1**: This randomness indeed has a negative impact on the accuracy of the generated preference. Nevertheless, APEC additionally leverages the Wasserstein distance metric to help choose the policy pair, which enhances the robustness of preference generation to randomness. Experimental results in Section 4.4 show that APEC can generate more accurate preference data compared to baselines.\n\n**Q2**: In Eq. 1, the loss calculation uses a section of the trajectory, but the impact of the parameter l on performance is not explored.\n\n**A2**: We apologize for missing this ablation study. Here we perform the ablation study on **segment length for reward learning**, across both feature-based HalfCheetah and pixel-based walker_run tasks. The \"default\" refers to the values used in our original experiments, and we show the overall task performance here. The findings show that APEC is **not** sensitive to this parameter.\n\n| |100|500|1000(default)|\n|---|---|---|---|\nHalfCheetah\t|12658\t|13434|\t12232|\n| |5|10(default)|20|\n|walker_run|\t712|\t701|\t696|\n\n\n**Q3**: Comparisons are limited to older methods. Additionally, including a baseline with an oracle reward would help illustrate the gap to the upper bound. Additionally, including a baseline with an oracle reward would help illustrate the gap to the upper bound.\n\n**A3**: To address your concern, we add two recent baselines, BC-IRL [1] and DRAIL [2] and the method with oracle reward. We use the recommended hyperparameters from the source code and slightly tune them. The task performance results are shown in the table below, where we find that APEC still achieves the best performance compared to BC-IRL and DRAIL.\n\n|Performance|\tAnt\t|HalfCheetah|\tHopper|\tHumanoid|\tWalker|\n|---|---|---|---|---|---|\n|BC-IRL|\t-156721\t|-48835\t|1084|\t97|\t-4.82|\n|DRAIL\t|1425\t|3699\t|3316\t|138\t|3586|\n|Oracle|\t5850|\t12138|\t3542|\t5489|\t5007|\n|APEC (ours)|\t5221|\t12232|\t3310|\t4953|\t5394|\n\n\n[1] BC-IRL: Learning Generalizable Reward Functions from Demonstrations. ICLR’23.\n\n[2] Diffusion-Reward Adversarial Imitation Learning. NeurIPS’24.\n\n**Q4**: Limited discussion with related domains: The adversarial imitation learning framework shares similarities with GAN training, yet there is minimal discussion of techniques from this domain. Furthermore, the work is closely tied to preference learning in LLMs, and a discussion of related works, which also focus on improving reward model generalization, is recommended.\n\n**A4**: Thank you for your insightful feedback. We agree that there are notable similarities between the AIL framework and GAN training because both of them involve an adversarial learning procedure. We will replenish the discussion on GAN in the revision.  \n\nBesides, while preference learning in LLM is indeed relevant to our work, it is crucial to highlight that APEC focuses on the automatic generation of diverse and accurate preference data through AIL, whereas the paper referenced by the reviewer centers on improving reward generalization by improving preference learning algorithms with fixed preference data. We will elaborate on these distinctions in the revised version.", "claims_and_evidence": "Yes, the claims in this paper are supported by theoretical analysis and empirical experiments.", "methods_and_evaluation_criteria": "Yes, Mujoco and DMControl tasks are common benchmarks in this domain.", "theoretical_claims": "I have not thoroughly verified Proposition 3.1, but it appears to be correct upon initial inspection.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "I think this paper has a broader impact on various research domains, including human preference learning and understanding (related to RLHF for LLMs), and areas like GAN training. However, the paper does not discuss these broader implications in detail. I suggest expanding on this aspect to better highlight its significance across related fields.", "essential_references_not_discussed": "The adversarial imitation learning framework is closely related to GAN training, where many techniques have been proposed to improve stability. I suggest discussing these stabilization techniques in the appendix [1][2]. In particular, [2] explores leveraging previous discriminators and generators to enhance GAN training, which share similar idea with this paper.\n\n\n[1] Stabilizing generative adversarial networks: A survey[J]. arXiv preprint arXiv:1910.00927, 2019.\n\n[2] Training gans with historical models[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 119-134.", "other_strengths_and_weaknesses": "The paper presents a meaningful contribution by improving reward generalization through enhanced preference coverage, supported by empirical and theoretical validation. \n\n**Strengths:**  \n- APEC integrates adversarial imitation learning with automated preference generation. The theoretical analysis of policy value progression in AIL further grounds the method’s design.  \n- The studied problem is a critical challenge in reward learning.\n-  The presentation is clear and well-organized.\n\n**Weaknesses:**  \n- The method is dependent on that later policies are better than earlier ones, but this may not always hold due to training fluctuations. A discussion on how this randomness impacts learning would be valuable.\n- In Eq. 1, the loss calculation uses a section of the trajectory, but the impact of the parameter l on performance is not explored.\n- Comparisons are limited to older methods (e.g., D-REX, AIRL, SSRR), and recent advances in preference learning or AIL variants after 2023 are not included. For instance, [1] is highly relevant for learning generalizable reward models. Additionally, including a baseline with an oracle reward would help illustrate the gap to the upper bound.\n- Limited discussion with related domains: The adversarial imitation learning framework shares similarities with GAN training, yet there is minimal discussion of techniques from this domain. Furthermore, the work is closely tied to preference learning in LLMs, and a discussion of related works, such as [2][3][4], which also focus on improving reward model generalization, is recommended.\n\n\n[1] BC-IRL: Learning generalizable reward functions from demonstrations. arXiv preprint arXiv:2303.16194, 2023.\n\n[2] Reward Generalization in RLHF: A Topological Perspective. arXiv preprint arXiv:2402.10184, 2024.\n\n[3] Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs. arXiv e-prints, 2024: arXiv: 2406.10216.\n\n[4] Improving Discriminative Capability of Reward Models in RLHF Using Contrastive Learning. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024: 15270-15283.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Please see the weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This was one of the most controversial papers in my AC stack. At the start of the author-reviewer discussion period, reviewers were originally [2,2,3,4], but during reviewer discussion one reviewer increased their score to 3, leading to scores that are closer to a consensus accept: [2,3,3,4]. However, reviewer Gguy still believes the experimental scope is insufficient for acceptance to ICML. 

**Summarizing reviewer comments:**

Strengths:
- “The paper's claim that later AIL policies are preferred over earlier ones is adequately supported through both empirical and theoretical evidence.” (Gguy)
- Theoretical results are correct (Gguy, YpdB)
- “The proposed methods are well-structured and appropriate for automated preference generation.” (Gguy)
- “the presented method is novel and interesting” (YpdB)
- “The experimental design is generally sound with key strengths: diverse benchmarks (MuJoCo and DMControl), clearly defined suboptimal demonstrations (50-80% of optimal), test dataset for generalization assessment, controlled sample sizes across methods, and appropriate evaluation metrics” (Gguy)
- “the selection of benchmarks and baselines (SSRR, LERO) appears to be appropriate” (aJ5F)
- “Overall methods and evaluations are plausible” (aJ5F)
- “The proposed study is simple yet effective. Despite its simplicity, it appears to be supported by appropriate theoretical proofs.” (aJ5F)
- “I think this paper has a broader impact on various research domains, including human preference learning and understanding (related to RLHF for LLMs), and areas like GAN training.” (LmzP)

Weaknesses:
- The practicality of the method, specifically the “significant computational overhead” and the fact that the method “relies on carefully tuned hyperparameters” (Gguy)
- Limited experiments
  - “restriction to continuous control tasks, relatively high-quality demonstrations despite claims of challenging setup, focus on correlation metrics rather than task performance, limited discussion of computational efficiency or training stability, and no exploration of APEC's performance with varying amounts of demonstration data.” (Gguy)
  - “only several simple benchmark problems are adopted to verify the effectiveness of the proposed approach” (YpdB)
  - requires more recent baselines (YpdB)

**Rebuttal:**

In the rebuttal, the authors provided a significant number of new experiments showing the computational time only adds 2 hours / 28, it works for discrete games (Atari), it's not sensitive to hyperparameters, the monotonicity assumption is not required, and they added an additional baseline. This appeared to me to address the main weaknesses identified during the review period. 

**Decision:**

Given the move towards a consensus accept and the strength of the new experiments in the rebuttal, I am giving this paper a weak accept. The authors should include the new rebuttal results in the revised version of the paper.
Decision: Accept (poster)