Deep Reinforcement Learning from Hierarchical Preference Design
Authors: Alexander Bukharin
Yixiao Li
Pengcheng He
Tuo Zhao
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Reward Modeling
Code Generation
Primary area: reinforcement_learning->deep_rl
TL;DR: We develop a new reward design framework, HERON, for reinforcement learning problems where the feedback signals have hierarchical structure.

Abstract: Reward design is a fundamental, yet challenging aspect of reinforcement learning (RL). Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals. This paper shows by exploiting certain structures, one can ease the reward design process. Specifically, we propose a hierarchical reward design framework -- HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning. Both scenarios allow us to design a hierarchical decision tree induced by the importance ranking of the feedback signals to compare RL trajectories. With such preference data, we can then train a reward model for policy learning. We apply HERON to several RL applications, and we find that our framework can not only train high performing agents on a variety of difficult tasks, but also provide additional benefits such as improved sample efficiency and robustness.

Deep Reinforcement Learning from Hierarchical Preference Design
Alexander Bukharin
Yixiao Li Pengcheng He Tuo Zhao
Abstract
we unable to access to the ground-truth reward, or the reward is sparse: receive zero reward most of the time. Therefore, designing a reward for the agent is necessary.
Reward design is fundamental, yet challeng- ing aspect of reinforcement learning (RI.). Re- searchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals. This paper shows by exploit- ing certain structures, one ease the reward design process. Specifically, we propose hi- erarchical reward design framework - HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning. Both scenarios allow us design hierarchical decision tree induced by the impor- tance ranking of the feedback signals to compare RL trajectories. With such preference data, we can then train a reward model for policy learning. We apply HERON to several RL applications, and we find that our framework can not only train high performing agents on a variety of difficult tasks, but also provide additional benefits such improved sample efficiency and robustness.
To construct the reward, practitioners often use multiple feedback signals 21. Ema each of which captures differ- ent facets of an agent's behavior. settings where the re- ward is inaccessible, the most common approach to utilizing these signals is the linear combination, e.g., E, w. zi (Booth al., 2023; Le et al., 2022; Zhang et al., 2019). The hyperparameters wy" 's are tuned to provide a comprehensive description of the agent's behavior, a process commonly known as reward engineering (Fu et al., 2017: Wu et al., 2021). Taking the traffic light control as an example, Zhang et al. (2019) consider a weighted combination of vehicle queue length, average waiting time and some other feed- back signals the reward. The hyperparameters W; 's are determined by extensive tuning in Van der Pol & Oliehoek (2016); Wu et al. (2017); Zhang et al. (2019). In sparse reward settings, the feedback signals are used to compose a reward surrogate, which is then combined with the sparse reward. Taking code generation as an example, Le al. (2022) design a piece-wise function where the region is divided according to the feedback signals (e.g., compila- tion error, runtime error) and values are designed by human experts.
1. Introduction
Although feedback signals may serve as useful criteria of an agent's behavior, reward engineering is not always an effective way to ensemble these signals. This is because feedback signals may have different scales as well intri- cate dependencies with other feedback signals. In this case, determining the weight by humans becomes challenging, and multiple weights must be simultaneously tuned since their respective feedhack signals are often correlated. More- over, in sparse reward settings, a piece-wist wise reward function often requires massive trials to determine the value, and dividing the region is also difficult because of the complex relation among different feedback signals.
Over the past decade, sigmificant advancements in deep learning techniques, along with unprecedented growth in computational power, have facilitated remarkable achieve- ments in the field of deep reinforcement learning (RL) across diverse domains, including finance, transportation, and auto- matic programming (Deng et al., 2016; Haydari & Yilmaz, 2020; Le et al., 2022). A key component of modern RL the reward function, which is typically predefined in benchmark environments such as the OpenAl gym or games (Mnih et al., 2013; Silver et al., 2016; Brockman et al., 2016). When dealing with complex real-world environments, however,
How to address the aforementioned issues still remains un- clear. While most existing works have focused on reward engineering for specific applications (Liu et al., 2020; Zhang et al., 2019), this paper proposes novel reward design framework for when feedback signals exhibit hierarchical relations. This relation exists in many RL problems. For
Equal contribution 'NVIDIA Georgia Institute of Technology 2 ZZoom Communications. Correspondence to: Alexander Bukharin abukharin @nvidia.com>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Deep Reinforcement Learning from Hierarchical Preference Design
example, in traffic light control the vehicle queue length significantly outweighs the average wait time and other feedback signals. Our framework is also suitable for sparse reward settings, where sparse rewards naturally have greater importance than the surrogates. For code generation, practi- tioners enhance the sparse reward (whether the code passes unit tests) with surrogates, such as the type of error.
model prompts and outputs labelled with response helpful- ness, coherence, correctness, verbosity, and complexity. By ranking these factors and applying HERON, we are able to train an aligned language model.
Robotic Control. We also evaluate HERON on robotic con- trol (Coumans & Bai, 2016; Brockman et al., 2016), where the hierarchy of the feedback signals is unclear. In these en- vironments, HERON performs better than reward engineer- ing and even achieves comparable performance compared to the ground-truth reward. This shows that HERON is able to train reasonable policy even if the hierarchy is unclear.
To leverage such hierarchical structures in the afore- mentioned scenarios, we propose HERON (Hierarchical prEference-based ReinfOrcement (eaarieinnnttaaningggggggn HERON trains a preference-basee reward model (Bradley& & Terry. 1952: Ouyang et al., 2022) through pair-wise trajectory compar- isons. Specifically, we design a decision tree, at each level of which compares trajectories based on a feedback signal. The feedback signal we use each level determined by its importance ranking, as assigned by the human annotator.
The rest of this paper is organized as follows: Section 2 introduces the related work. Section 3 introduces our pro- posed reward design framework. We conduct experiments in Section 4, We discuss the limitations of our method in Section 5.
This decision tree based on importance ranking provides a number of benefits. First, it is a more natural way to resemble the human decision process compared to reward engineering. When making decision between two choices, humans typically start with the the most important factor, then proceed to the secondary factor, and continue until the remaining less important factors. Second, ranking feedback signals is usually easier than specifying numerical weights. Third, the comparison process of HERON does not depend on the absolute value of a feedback signal, but on their rela- tive quantity. We find that this brings additional robustness in scenarios where the training environment changes. We will further discuss this in Section 4.1. Finally, HERON is able to leverage pre-trained knowledge, which allows for the creation of more powerful rewards.
2. Related Work
Besides reward engincering, there are several works that attempt to improve reward design for RL.
Reward Shaping. Reward shaping aims accelerate the convergence of RL algorithms by incorporating auxiliary reward information through shaping functions (Ng et al., 1999; Semorio-Gonzaler et al., 2010; Devlin & Kudenko, 2012). These approaches aim to mitigate the sparsity of a pre-defined reward function. While reward shaping has demonstrated success in practice, it often necessitates ex- tensive tuning. To circumvent the need for costly tuning, several methods have been proposed to automatically shape rewards by utilizing an abstract MDP (Marthi, 2007), tile coding (Grzes & Kudenko, 2008), and bi-level optimization (Fu et al., 2019; Hu et al., 2020). In contrast, our work pursues a different direction that eliminates the requirement for a pre-specified reward function and does not assume that the reward is a linear combination of auxiliary factors.
We empirically validate HERON framework through exten- sive experiments on real world applications:
Traffic Light Control. In traffic light control (Zhang et al., 2019), there are feedback signals with the hierarchy: queue length the average vehicle waiting time > other feedback signals. HERON consistently outperforms the policies trained with reward engineering techniques.
AutoRL. AutoRL (Afshar et al., 2022; Holder et al., 2022) automates various aspects of hyperparameter selec- tion in RL, including parameters related to the reward. Par- ticularly relevant to our work, Faust et al. (2019) and Chiang et al. (2019) treat reward weights as hyperparameters and optimize them using population- based training.
Code Generation. Code generation (Le et al., 2022) is a sparse reward scenario. Most of the programs gener- ated by the policy cannot pass all the unit tests and thus fail to receive the reward. Therefore, surrogates, like the type of error, are added to compose piece-wis reward function. In code generation, HERON demonstrates the ability to achieve higher Pass@ K scores compared to the hand-crafted piece-wise reward function employed in state- of-the-art approaches.
Inverse Reinforcement Learning. Inverse reinforcement learning (IRL) aims to learn reward function from ex- pert demonstrations (Ng et al., 2000; Abbeel & Ng, 2004; Boularias et al., 201 1). Although IRL enables the learning of complex behaviors without manual reward tuning, it re- quires observed, optimal behavior. These demonstrations are often costly to obtain, and in our experiments, acquir- ing them would be far more expensive than obtaining hierarchy of feedback signals. Furthermore, IRL methods
Language Model Alignment. Although language models are powerful, they are not always aligned with human prin- ciples (Brown et al., 2020). We propose to use HERON to align language models, by using public datasets of language
2

---

Deep Reinforcement Learning from Hierarchical Preference Design
typically require unstable bi-level optimization procedures, which our approach does not involve.
(1(T) (T2)
0
Reinforcement Learning from Human Feedback. Rein- forcement learning from human feedback (RLHF) (Chris- tiano et al., 2017; Ouyang et al., 2022: Bai et al., 2022) aims to train a policy model that aligns with human preference. Although both RLHF and our method involve preference- based reward model, in RLHF the preference labels come directly from the annotators. On the other hand, HERON au- tomatically compares trajectories by the importance ranking of the feedback signals. This ranking can be easily set up by a human overseer if the feedback signals have hierarchy, or in sparse reward settings. We will discuss more differences in Section
#
~
H =
{ - 22(
0
<
==1 1
20
H =
(3(Tg)
23(22)
0
1
=
Figure I. Preference elicitation: compare trajectories Ti and 72 through 3 feedback signals 21, 22, Ra.
3. Method
knowledge: rankings over the feedback signals. We assume 21...., 2n have been ordered in descending order of im- portance by an expert with domain knowledge. In sparse reward settings, 21 is always the sparse reward and the re- maining z 's are the surrogates. We then elicit preference I E {0, 1, between trajectory pairs (21,7) with a deci- sion tree induced by the given feedback signal hierarchy. A tie is denoted by μ = 0, μ = means 71 is preferred, and f 2 means 72 is preferred.
We consider a Markov decision process (S, A, P. R, 7) where an agent interacts with an environment over a series of discrete time steps. At time step t, the agent observes 8t E S. takes an action ar E A according to a policy, and receives the next state observation $t+1 E S and reward r. E IR. In most real-world applications, the reward r, is not available. Therefore, our goal is to design appropriate reward model R S x A + R such that when maximizing the objective
The decision tree is constructed as follows. We first set the current level 1 = 1. We then calculate
((0) = Br
Σ (TT
7 ('gssse
(1)
0 if (T() =x s 6
/
1 if ()()) > (T + d1
2 if (7(2)) > (T) + 51
the agent's behavior, guided by the policy model Ty : S x A + IR. meets our expectation.
where 6 is a margin hyperparameter for level l. The margin parameter is ensures that we only elicit preference using 2 if the two trajectories are significantly different according to 21. The margin dj can be used to inject further domain knowledge into the HERON algorithm, but in our experiments we set d, to the standard deviation of 2 over the collected data.
To design the reward, we utilize a set of n feedback signals 2/, ....... 2 given at each time step t. These signals serve as multiple measurements of a trajectory. We denote segments of the resulting trajectory as
T = (8t, at, ( (=())))) (SE+4, Beaky (oitk )
If μa 0, we update I 1+1++ 1 and compare the trajectories with the next most important feedback signal. If the two trajectories are not significantly different in any of the feed- back signals (i.e. 1 > n)), we discard the trajectory pair. We illustrate the algorithm in Figure 1.
and we overload the notation for 2; such that it represents the feedback signal of segment of trajectory, 24 (T) E(naier (5 ar).
Given the feedback signals of trajectories and the impor- tance ranking, HERON builds the greference-baseee re- ward model by preference elicitation and reward learning. HERON then trains a policy model through policy learning.
Reward Learning Given a labeled dataset of trajec- tories (Tw, Tu) where Tu is the trajectory preferred by the preference elicitation algorithm (i.e. م = 1), we would like to assign higher reward to the preferred trajectory (we remove all ties from the dataset, since we find includ- ing them has negligible effect on training). To accomplish this, we train a reward model RS : S x A + RR where (T(T) = -(srar)r Ro,,,, a.). To assign a higher re- ward to the preferred trajectory Twx we follow the methodol-
Preference Elicitation. We generate set of trajectory data with policy model. We can obtain the initial policy model by (i) behavior cloning from expert demonstration data, (ii) pre- preitrain it using a handcrafted reward, or (iii) purely random initialization. With the trajectory data, HERON first compares them based on an intuitive form of domain
3

---

Deep Reinforcement Learning from Hierarchical Preference Design
ogy in Ouyang et al. (2022) and optimize the loss
vehicles passing through the system (vl).
[ = -a( [log (R(R((Tw) - RT(T)))) (2)
For these experiments, we evaluate variety of reward hi- erarchies. Our reward model is parameterized by three- layer MLP that is learned by multi-stage training. We use QCOMBO (Zhang et al., 2019), a Q-learning based algo- rithm as the RL algorithm and conduct experiments using the Flow framework (Wu al., 2017). We train aggett RL policies on two-by-two grid (four agents), each pa- rameterized by a three-layer MLP. For more details on the environment and the experiment setting, see Appendix E.
We remark that this loss employs the Bradley-Terry prefer- ence model (Bradley& & Terry, 1952). Once we have trained the reward model Rg. we can assign a reward to each tra- jectory T as (T) It is important note that unlike some prior works which learn linear reward models on top of state features, HERON allows for more complicated reward mod- els parameterized by neural networks (Sadigh et al.. 2017: Bryuk et al., 2020). Therefore, it is also possible to introduce pre-trained knowledge into the reward model.
Baseline. We compare our method to reward engineering, where the reward is formulated as Σm= W22, where the Wi's 's are hyperparameters and 2 's are normalized feedback signals. To inject HERON's domain knowledge and to make hyperparameter tuning tractable, we set the W,'s i's to be geometrically decreasing such that W; 3 and select B E (0.1,0.2, ...,, 1.0). This a very realistic and competitive reward engineering baseline. Note that we also explore re- ward engineering without this prior hierarchical knowledge. We also compare to ensemble approaches, which train a separate policy on each feedback signal and then select an action at each time step by weighted combination of each policy (Brys et al., 2017).
Policy Learning. With the reward model Rg, the policy in (1) can be learned via popular reinforcement learning algo- rithms such Q-learning or Proximal Policy Optimization (Sutton& & Barto, 2018; 8; Schulman et al., 2015; 2017).
Multi-stage Training. The success of the reward learning depends on the quality of the trajectories generated by the policy model, but if the initial policy model is not optimal, e.g., pre-trained from handcrafted reward function or randomly initialized, it may introduce significant sam- pling bias to the trajectories. To address this issue, we can repeat the preference elicitation, reward learning, and policy learning for multiple rounds. In each new round, trajectories in preference elicitation are generated by the policy model from the last round, and the reward model then adapted using the new comparisons.
Optional:
In this experiment we evaluate different reward designs by comparing the ground-truth reward of the associated policy,
Results. In Figure 2, we plot the evaluation reward (as defined in (Zhang et al., 2019)) of policies in the traffic light control environment. We observe that the policy trained with HERON performs significantly better than the policies trained with the reward engineering baseline even by the ground-trum -ruth reward developed in Zhang et al. (2019). The gain of HERON over all other methods passes a t-test with < 0.005. We hypothesize that HERON can utilize each reward signal better than a linear combination does; a significant change in single feedback signal may be drowned out in the linear combination, but HERON can incorporate this information due to its hierarchical nature.
Extension: Direct Preference Optimization. Rafailov et al. (2024) showed that in contextual bandit settings, the KL-regularized preference optimization problem can be optimized directly, without a reward model. In appropriate settings, we can use DPO to simultancously train the policy and reward, reducing the computational cost of HERON.
4. Experiment
We evaluate the efficacy of our framework traffic light con- trol experiments, code generation, language model align- ment, and robotic control.
Flexibility of Hierarchical Reward Modeling, In various tasks, there is no one ideal reward, and the aspects of an agent's behavior that should be prioritized depend on the practitioner's 's preference. As a result, a crucial characteristic that reward design algorithms should possess is flexibility, In particular, modifying the domain knowledge inputted should result in corresponding changes in the behavior of the agent. To evaluate the flexibility of HERON, we examine how changing the feedback signal rankings changes agent behavior in the traffic light control environment.
4.1. Multi-Agent Traffic Light Control
Environments. In this real-world scenario, cooperative agents learn to increase the throughput and minimize the wait of cars passing through a traffic network. Due to the complexity of the traffic system, unfortunately, there is no one optimal reward, and different reward may be preferred in different scenarios. To solve this problem, Wei et al. (2018); Van der Pol & Oliehoek (2016); Zhang et al. (2019) define the ground-truth reward by balancing the following six feedback signals: queue length (g), vehicle waiting time (wt), vehicle delay (dl), number of vehicle emergency stops (em), number of light phase changes (fl), and number of
In this experiment we always set the most important signal as the number of cars passed, and then we use the queue length, wait time, or delay the second signal. The results can be seen in Figure 4. We observe that HERON is quite
4

---

Deep Reinforcement Learning from Hierarchical Preference Design
flexible, and that by changing the reward hierarchy we can significantly influence the agent's behavior: when priori- tizing certain signals the policies performance (measured according to the prioritized signal) will greatly increase.
1.1
Passerd,uuuan Delay
0.9
Num, Passed Wolt Delay Quese
Reward
0.0
0.7
44
0.6
Signal Utilization. We also show the level the decision tree induced by HERON reaches in Figure 3. This may change with different reward hierarchies, but as we can see from the figure, a relatively similar proportion of decisions are made at each level of the decision tree. This confirms the efficacy of setting & to 2 s standard deviation.
10
0.5
0.4
1000
4000
3000
Evaluation Step
Evaluation Time Step
(a) Num. Passed
(h) Wait Time
3.2
40
Ground Truth Reward
2.0
20
-20
-40
HERON
2.4
10
M
Ground Truth
-60
Ens Pref.
1000
2000
3000
4000
2000
3000
4000
-80
Evalluation Time Step
Evaluation Time Step
Ens Lin.
-100
(c) Delay
(d) Queue Length
Reward Eng.
-120
Figure 4. Evaluation reward curves with different reward hierar- chies traffic light control. The importance decreases from left the right in a label. The curve is within + one standard deviation.
-140
0
1000
2000
3000
4000
Evaluation Time Step
Figure Evaluation curves with different reward hierarchies in traffic light control. The curve is within + one standard deviation.
be misaligned with the human overseer's desires.
To evaluate HERON's robustness, we change the speed of the cars halfway into training (this a realistic setting, since many areas have time-dependent speed limits). We then evaluate each policy after training under the environ- ment, and see which algorithms were able to adapt the best. We compare the HERON-trainee policy with two policies trained with the reward engineering: one that uses the opti- mal learning rate in the unchanged environment (1 x 10-3) and one that uses a smaller, more stable learning rate of 10-5
1.0
Pansed Fmmegeerty y stage
Ddlay
10
20
howord model spoc?
From Figure 5, we can see that reward engineering is quite sensitive to changes in the environment during training. This can be combatted with smaller learning rate, but this will result in slower learning and a sub-optimal reward. On the other hand, HERON is able to attain a high reward regard- less of the environment change, supporting our hypothesis that HERON's inveriniaaiaatt design leads to increased ro- bustness.
Figure 3. Utilization of different signals.
Robustness. An advantage of HERON that unlike re- ward engineering, it does not depend on the magnitude of the different feedback signals. This is because the prefer- ence elicitation algorithm will label trajectory pairs with E {0, 1, 2}, regardless of the scale of the different sig- nals. This ccale-invariance is beneficial, since algorithms that depend on the scale of the feedback signals may be vulnerable to changes the environment during training. For example, if the scale of feedback signal suddenly dou- bles, (i.e. the traffic on highway doubles due to rush hour) then two things will happen: (1) the scale of the reward signal may sharply increase, which is similar to a sudden change in learning rate; (2) the weight vector used in reward engineering to combine the feedback signals will effectively be changed. The first phenomenon may cause training insta- bility, and the second phenomenon could cause the agent to
4.2. Code Generation
Environment. RL has recently gained considerable atten- tion for its state-of-the-art performance in various text gen- eration tasks. Therefore, we investigate if HERON can achieve similar improvements in LLM performance solely based on rankings over feedback signals. First, we consider the code generation task. In code task, the goal of the agent is to write a program that will satisfy the criteria specified in a given problem.
5

---

Deep Reinforcement Learning from Hierarchical Preference Design
each method is evaluated on I million generated programs. To evaluate the generalization ability of the policies, eval- uate each policy in zero-shot manner on the MBPP dataset, which contains 974 basic python programming questions (Austin et al., 2021)-
-40
40
n
-10
Rewwerd
-100
-10
-220
-140
HERON
Eng: Small Ln
-220
-20
Reward Eng: Optimal LR
Free
1933
1000
2005
2000 Step
4200
Post- dost-Trannng Reward Scaling, To further incorporate do- main knowledge and environment feedback into the reward, we propose to rescale the reward learnt from (2). Specifi- cally, we multiply the reward (T(T) by a shaping constant, denoted as ax F(r)) Here, is hyperparameter and tuned over [1, 2, 3}. while F(T) corresponds to piece-wise func- tion of the feedback signals. We define it for code generation as
Evalustion
(a) 25 MPH
(b) 30 MPH
-40
-50
-00
125
-150
-10
140
-200
Evaluation Time Step
Evaluation Time Seop
(c) 40 MPH
(d) 45 MPH
3 if program passes all unit tests F(r)
Figure 5. Evaluation curves with different environments: changes of vehicles' speed limit. The baseline speed limit is 35 MPH.
2 if program T fails a unit test
1 if program yields any emror.
Baselines. Recently, Le et al. (2022) demonstrated state-of- the-art performance can be achieved by training with RL. They manually design a constant piece-wise reward that is determined by feedback signals including whether the program passes all the unit test and the type of error if failed (i.e. compilation error or runtime error).
This function is motivated by the importance ranking of feedback signals. It explicitly reinforces feedback signals in policy learning according to their importance ranking and serves as the supplement of the preference- reward model. Specifically, by tuning a, we can effectively control the reward's shape and the degree of separation between the best and worst trajectories. We focus efforts exclusively on the code generation task due to its high complexity.
1.0
if program 8 fails to compile if program 8 has a runtime error
tuning
0.6
RCadeRL (s)=
0.3 if program s fails unit test
+1.0 if program s passes all unit tests.
Results. We display the results for the code generation task in Table and Table 2. HERON outperforms all other ap- proaches. For larger K in Pass@ K the gain in Pass K pass t-test with p < 0.05. This is most likely because reward engineering only gives large reward to programs that pass all the unit tests or are similar to the expert demonstrations, while HERON can give a large reward to programs that may fail some unit tests but the reward model predicts as being likely to satisfy the prompt. This means that HERON will promote more diverse set of programming strategies. In addition, HERON's smooth reward function (opposed to the discontinuous piece-wise function in sparse reward settings) may be more conducive for learning, and therefore lead to higher performance.
Shojaee al. (2023) (PPOCoder) build upon this work, inte- grating more feedback signals such as a program's abstract syntax tree (AST) similarity to expert demonstrations.
Implementation Details. Our decision tree is based on three signals: the percent of tests a program passes, the type of error a program incurs, and the AST similarity to expert demonstrations. To train policies we follow the implemen- tation of Le et al. (2022). We initialize our policies with the CodeTSSplus-lage Code (Wang et al., 2021). The policies are first trained with behavior cloning on the expert demonstrations. Next, we generate 20 samples per training program, and conduct RL training over these generated samples. We train with the policy gradient objective. We evaluate the perfor- mance of each algorithm using Pass metric, which is the number of programming problems passed with K submis- sions per problem (Chen et al., 2021). We primarily evaluate HERON on APPS, a python programming datasets contain- ing 5000 test problems (Hendrycks et al., 2021). Each question in the dataset comes with expert demonstrations and test cases the program should pass. To evaluate each algorithm, we generate 200 programs per problem. In total,
model and our reward model with
We further analyze code generation performance using the filtered Pass@ K metric, which only submits programs that pass unit tests provided in the prompt (Chen al., 2021). As seen in Table 3, HERON uniformly and significantly out- performs the baselines, confirming the efficacy of HERON.
As in Le et al. (2022), we evaluate the performance of policies trained by HERON on the MBPP dataset. The results are displayed in Table 4. HERON outperforms the other methods, confirming its efficacy.
6

---

Deep Reinforcement Learning from Hierarchical Preference Design
Table 2. Pass@55 APPS by three difficulty levels: introductory,
Table I. Raw P on APPS.
interview, competition.
Pass @1 1 @ Pass 10 Pass @ 20 Pass @5
Intro. Inter. Comp.
BC
1.59
3.82
5.19
6.74
6.74
BC
18.8
4.23
2.10
CodeRL
1.71
4.12
5.57
7.26
9.81
CodeRL
23.7
6.93
4.51
PPOCoder
1.23
3.08
4.19
5.50
7.62
PPOCoder
18.6
5.41
3.23
HERON
1.72
4.19
5.71
7.49
10.19
HERON
24.6
7.28
4.53
Table 3. Filtered @ on APPS.
Table 4. Pass@K K on MBPP
Pass@1 Pass@1 10 Pass@21 20
Pass@1 1 Pass @2 Pass@5
BC
4.70
6.36
6.44
CodeRL
6,58
10.27
16.24
CodeRL. 5.73
8.57
8.96
PPOCoder
6.58
10.09
15.85
PPOCoder 5.60
8.61
8.93
HERON 7.40
11.03
16.54
HERON 5.74
9.03
9.43
4.3. Language Model Alignment
4.4. More Experiments
Environment. Beyond code generation, we also evaluate the ability of HERON to train instruction following models that are aligned with human principles. For this experiment we employ Phi-2 (Javaheripi al.) as our base model, and train it on the HelpSteer dataset (Wang et al., 2023). The HelpSteer dataset is composed of instruction-respones pairs annotated (from 0-4) across five feedback signals: correct- ness, helpfulness, coherence, complexity, and verbosity.
To demonstrate that HERON is able to train reasonable pol- icy even if the hierarchy is unclear, we experiment on four robotic control tasks: Ant, Half-Cheetah, Hopper, and Pen- dulum. We use the PyBullet simulator, where the ground- truth reward is formulated as a linear combination of sev- eral signals such the robot's potential, the power cost, whether the joints are at their limit, and whether the robot's ' feet collide (Coumans & Bai, 2016). These factors do not necessarily display clear hierarchy. More details on this environment can be found in Appendix K. The results can be found in Table 6, where we observe that HERON can always exceed the performance of the reward engineering baseline.
Implementation. For HERON, we use the following hier- archy: correctness > helpfulness > coherence > complex- ity > verbosity. We then use HERON-DPO to optimize the policy. We consider two reward engineering baselines: REINFORCC--base finetuning with equal reward weights on all signals and REINFORCE-basse finetuning with ex- ponential decaying reward weights (Williams, 1992). Every algorithm is initialized from a version of Phi-2 that has undergone supervised finetuning on HelpSteer.
4.5. Ablation
Training Time Analysis. The main computational cost of HERON comes from reward model training, as data collec- tion is already part of most RL algorithms and preference elicitation is very fast. To accelerate reward model training in the multi-stage setting, we can use an annealed training schedule (see Appendix 1). The normalized training time of HERON, reward engincering, and ensemble- based learning are shown in Figure 6a. HERON is 25% slower than reward engineering on average. which is quite reasonable given that the tuning cost of reward engineering is usually large.
Results. To evaluate the resulting models, we use three state- of-the-art reward models (RLHF-Flow (Dong al., 2024),
PairRM (Jiang al., 2023), Eurus-7B (Yuan et al., 2024)) and Claude 3 Sonnet to compute a win-rate compared to the SFT policy on the HelpSteer test dataset. The results can be seen in Table 5. We find that HERON-DPO can significantly outperform the baselines across all judges. We hypothesize that this gain is due to the fact that HERON'S decision tree over these signals can better capture human principles compared to linear combinations of feedback signals. More details can be found in Appendix M.
Hyperparameters. We set o; to the standard deviation of 2 over the collected data in our experiments. Nonetheless, we evaluate the sensitivity of HERON to these parameters in Figure 6. We find values in [0, 2 oı] work well, where o; the standard deviation of 2,.
Tuning Cost. Finally, compare the tuning cost of
7

---

Deep Reinforcement Learning from Hierarchical Preference Design
Table Win against the SET model calculated by various LLM judges.
RLHF-Flow
PairRM
Eurus-7B
Claude-3
Avg.
Reward Eng. (Equal Weight) Reward Eng. (Decaying Weight) HERON-DPO
54.67
58.49
53.68
57.46
56.08
59.24
59.64
52.49
56.85
57.06
66.20
63.02
63.22
63.82
64.57
useful in environments where there are several feedback signals available and human overseer can rank these sig- nals. Our experiments show that in such environments (code generation and traffic light control) HERON can outperform state-of- tfateoof-the-ar the-art baselines. Moreover, HERON shows great versatility can achieve decent performance in non-ideal environments (robotic control).
Table 6. Ground-truth reward obtained in robotics environments,
Hopper
Cheetah
Pendulum
Ground-truth 0.99(0.0)
0.86(0.01)
0.94(0.10)
1.0(0.01)
Reward Eng. 0.88(0.02)
0.72(0.05)
0.61(0.04)
0.99(0.0)
HERON
1.0(0.01)
0.78().04)
(.62(0.04)
1.0(0.0)
HERON with reward engineering in the traffic light con- trol environment. For HERON we consider tuning with exact domain knowledge (the hierarchy is given) and with inexact domain knowledge (the top 3 elements are given but their order is not specified). For reward engineering we consider tuning with exact domain knowledge and with no domain knowledge (the reward weights do not have hier- archical structure). For the latter case we tune the weights with bayesian optimization. The results are shown in Fig- ure 6c. We find that both versions of HERON significantly outperform reward engineering. Although bayesian opti- mization train high performing policies, it requires 5 to 15 times the tuning iterations of HERON.
Reward Flexibilityy HERON is also capable of dealing with the feedback signals that are of nearly equal importance. One feasible solution is to flip the importance ranking of the equally important feedback signals with certain probability during the preference elicitation step described Section 3. One even design more complex preference elicita- tion algorithms that do not require strict hierarchy over feedback signals, which we leave for future work.
Low Labeling Budget Setting, RLHF is an effective method to obtain a powerful policy model. This is because humans can provide more informative insight for the reward model than feedback signals. However, involving humans to compare every pair of trajectories is often not affordable and it is difficult to create suitable instructions how to compare trajectories. We consider a separate setting, where only some feedback signals and some domain knowledge about them are available. In this case, reward engineering becomes reasonable and cheap choice, as does our method. Therefore, reward engincering the most suitable baseline.
1.0
Ensemble
Reward Eng.
1.4
O
0.5g 1a 20
0.8
HERON
1.2
0.8
0.4
Reward
0.4
0.6
0.2
02
0.0
Mountain Car Pendulum
Traffic Control
Code Gen.
Mountain Car
Pendulum
Thaffic Lights
HaifCheetah
Multi-objective RL. Our method is not designed specif- ically for multi-objective RL problems. Generic multi- objective RL is complex because some objectives adversely affect other objectives during the optimization. In this case, researchers try to find the Pareto frontier and balance among the objectives many different scenarios (Van Moffaert & Nowé, 2014; Mossalam al., 2016). In contrast, our method can be applied if and only if the feedback signals are available and have hierarchal structure.
(a) Training Time
(b) 6
-40
-60
-80
100
-12
HERON: Exact Knowledge
140
HERON: Inexact Knowledge
160
Reward Eng: Exact Knowledge
Full Tuning: Bayesian Opt.
-200
0
4
8 10 12
14 16
Number of Tuning Iterations
Future Work. Our proposed hierarchical comparison pro- cedure enjoys flexibility and can be extended. For instance, we can consider the level of the feedback in the hierarchy as the preference strength. More specifically, preference outcomes drawn based on more important feedback make stronger preferences between two RL trajectories. To exploit such preference strength, we can add rescaling or margin hyperparameters to reward learning. We leave this for future investigation.
(c) Tuning Cost
Figure 6. Training time and ablation study for HERON.
5. Diseussion
Suitable Scenarios. HERON is not intended to serve as a solution for all RL problems; however, it can perform quite well in specific settings. In particular, HERON will be most
8

---

Deep Reinforcement Learning from Hierarchical Preference Design
Impact Statement
Brys, T., Harutyunyan, A., Vrancx, P., Nowé, A., and Taylor, M. E. Multi-objectivizztion and ensembles of shapings in reinforcement learning. Neurocomputing, 263:48-59, 2017.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman,
References
G., et al. Evaluating large language models trained on code. arXiv preprint urXiv:2107.03374, 2021.
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-firsl international conference on Machine learning, pp. 1,
Chiang, H.-T. L., Faust, A., Fiser, M., and Francis, A. Learn- ing navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2):2007-2014, 2019.
2004.
Afshar, R. R., Zhang, Y., Vanschoren, J., and Kaymak, U. Automated reinforcement learning: An overview. arXiv preprint arXiv:2201.05000 2022.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg. S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Austin, J., Odena, A., Nye, M., Bosma, M.. Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le. Q.. al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733: 2021.
Coumans, E. and Bai, Y. Pybullet, a python module for physics simulation for games, robotics and machine learn- ing. 2016.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 22X::22044005862. 2022.
Deng, Y., Bao, F., Kong, Y., Ren, Z., and Dai, Q. Deep direct reinforcement learning for financial signal representation and trading. IEEE transactions on neural networks and learning systems, 28(3):653 664, 2016.
Devlin, S. M. and Kudenko, Dynamic potential- reward shaping. In Proceedings of the 11th international conference on autonomouu agents and multiagent systems, pp. 433-440. IFAAMAS, 2012.
Bıyık, E., Huynh, N., Kochenderfer, M. J., and Sadigh, D. Active preference-bare based gaussian process regression for reward learning. arXiv preprint arXiv:2005.02575. 2020.
Booth, S., Knox, B. W. Shah, J., Nickum, S., Stone, P., and Allievi, A. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifi- cations. In AAA/ Conference on Artificial Intelligence, 2023.
Dong, H., Xiong, W., Pang. B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv e-prints, pp. arXiv-2405, 2024.
Faust, A., Francis, A., and Mehta, D. Evolving rewards to automate reinforcement learning. arXiv preprint arXiv:1955.07626 2019.
Boularias, A., Kober, J., and Peters, J. Relative entropy inverse reinforcement learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 182-189. JMLR Workshop and Conference Proceedings, 2011.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv: /248, 2017.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 324344))3 1952.
Fu, Z.-Y., Zhan, D.-C., Li, X.-C., and Lu, Y.-.. Automatic successive reinforcement learning with multiple auxiliary rewards. In IJCAI, pp. 2336-2342, 2019.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J.,
Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint 1rXiv:1606.01550 2016.
Grzes, M. and Kudenko, D. Learning potential for reward shaping in reinforcement learning with tile coding. In Pro- ceedings AAMAS 2008 Workshop on Adaptive and Learn- ing Agents and Multi-Agent Systems (ALAMAS-ALAg 2008), pp. 17-23, 2008
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Haydari, A. and Yılmaz, Y. Deep reinforcement learning for intelligent transportation systems: A survey. IEEE

---

Deep Reinforcement Learning from Hierarchical Preference Design
Trunsactions on Intelligent Transportation Systems, 23 (1):11-32, 2020.
Parker-Holder, J., Rajan, R., Song, X., Biedenkapp. A., Miao, Y., Eimer, T., Zhang, B., Nguyen, V., Calandra, R., Faust, A., et al. Automated reinforcement learning (au- torl); A survey and open problems. Journal of Artificial Intelligence Research, 74:517-568, 2022.
Hendrycks, D., Basart, S., Kadavath. S., Mazeika, M., Arora, A., Guo, E., Burns, C.,, Puranik, S., He, H., Song, D., et al. Measuring coding challenge competence with apps. arXiv preprint arXiv: 105iv:2205.00 100.0995099 2021.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Hu, Y., Wang, W., Jia, H., Wang, Y., Chen, Y., Hao, J., Wu, F., and Fan, C. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931-15941, 2020.
Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A. Ac- tive preference- based learning of reward functions. 2017.
Javaheripi, M., Bubeck, S., al. Phi-2: The surprising power of small language models, 2023. URL https://www. microsoft. wwwpr//ewaarrbboo/piiii--eeeeeee nower-of-smen ssmallllnnuuage models.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference machine learning, pp. 1889 1899.1897. PMLR, 2015.
Jiang, D., Ren, X., and Lin, B. Y. Llm-blender. Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306. 02561, 2023.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: Feti:::777.06444 2017.
Le. H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Ad- vances in Neural Information Processing Systems, 35: 21314-21328, 2022.
Shojaee, P., Jain, A., Tipireni,, S., and Reddy, C. K. Execution- code generation using deep reinforce- ment learning. arXiv preprint arXlv:2301.13116 2023.
Liu, X.-Y., Yang, H., Chen, Q., Zhang. R., Yang. I.., Xiao, B., and Wang, C. D. Finrl: A deep reinforcement learning library for automated stock trading in quantitative finance. arXiv preprint ar arXiv:2011.09600 2020.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,
Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvan V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Marthi, B. Automatic shaping and decomposition of re- ward functions. In Proceedings of the 24th International Conference Machine learning, pp. 601-608, 2007.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018
Mnih, V., Kavukcuoglu, K., Silver. D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprinl arXiv:1312.5602, 2013.
Tenorio-Gonzalez, A. C., Morales, E. F., and Villasenor- Pineda, L. Dynamic reward shaping: training robot by voice. In Advances in Artificial Intellieence-IBERAMII 2010: 12th Ibero-American Conference on AI, Bahía Blanca, Argentina, November 1-5, 2010. Proceedings 12, pp. 483 433-492.6 Springer, 2010.
Mossalam, H., Assael, Y. M., Roijers, D. M., and Whiteson, S. Multi-ohjective deep reinforcement learning. arXiv preprint arXiv:1610.02707. 2016.
Van der Pol, E. and Oliehoek, F. A. Coordinated deep rein- forcement learners for traffic light control. Proceedings of learning, inference and control of multi-agent systems (at NIPS 2016), 8:21-38, 2016.
Ng. A. Y., Harada, D.,. and Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pp. 278-287. Citeseer, 1999.
Van Moffaert, K. and Nowé, A. Multi-objective reinforce- ment learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):3483- 3512, 2014.
Ng. A. Y., Russell, S., et al. Algorithms for inverse rein- forcement learning. In leml, volume 1, pp. 2, 2000.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, (-2773022774-- 2022.
Wang. Y., Wang, W., Joty, S., and Hoi, S. C. Codet5: Identifier-aware unified pre-trained encoder-decoder mod- els for code understanding and generation. arXiv preprint arXiv:2 109 09.00059, 2021.
10

---

Deep Reinforcement Learning from Hicrarchical Preference Design
Wang, Z., Dong, Y., Zeng, J., Adams, V., Sreedhar, M. N.,
Egert, D., Delalleau, O., Scowcroft, J. P., Kant, N., Swope, A., et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023.
Wei, H., Zheng, G., Yao, H., and Li, Z. Intellilight: A reinforcement learning approach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2496-2505, 2018.
Williams, R. J. Simple statistical cradient-following algo- rithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Wu, C., Kreidieh, A., Parvate, K., Vinitsky, E., and Bayen, A. M. Flow: A modular learning framework for autonomy in traffic. arXiv preprint arXiv: arXiv:/710.000: 17.0055665 2017.
Wu, Z., Lian, W., Unhelkar, V., Tomizuka, M., and Schaal, S. Learning dense rewards for contact-rich manipula- tion tasks. In 2021 IEEE International Conference Robolics and Automation (ICRA), pp. 6214-6221. IEEE, 2021.
Yuan, L., Cui, G., Wang, H., Ding, N., Wang. X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., Liu, Z., Zhou, B., Peng, H., Liu, Z., and Sun, M. Advancing Ilm reasoning generalists with preference trees, 2024.
Zhang, Z. Yang. J., and Zha, H. Integrating indepen- dent and centralized multi-agent reinforcement learning for traffic signal network optimization. arXiv preprint arXiv:1909.1663 2019.

---

Deep Reinforcement Learning from Hierarchical Preference Design
A. Appendix supplemmnnt material
B. Classic Control Experiment Details
For the classic control experiments we use the OpenAl gym (Brockman al., 2016). To train policies we use the DDPG algorithm, where the policies are parameterized by three layer MLPs with 256 hidden units per layer. We use the Adam optimizer, and search for a learning rate in [1 10-5, 1 10-3)
For mountain car we train for total of 15000 timesteps and begin training after 5000 timesteps. For pendulum, we train for a total of 50000 timesteps and begin learning after 25000 timesteps.
C. Baselines
C.1. Ensemble Baseline
Beyond the ground-truth reward, we compare the HERON algorithm with two ensemble baselines inspired by Brys et al. (2017). These ensemble baselines train separate policy each feedback signal, and then combine the policies' outputs in given state select action. In every environment we train each policy in the ensemble with the similar parameters as used for the reward engineering baseline and we again tune the learning rate in [1 x 10-5, 1 x 10-3).
As described in the main text, we consider two variants of this ensemble based algorithm: one where the action is selected according to an average over each policy (a + argmax Ek=1 1 (s, a)) and one where the preference ranking used as a
input to HERON used to combine the actions (a f- argmax Ek=1 m),, With the second variant, y selected from {0.25, 0.35, 0.45, ... 0.95, 0.99, 13.
C.2. Reward Engineering Baseline
We also examine the performance of a reward engineering baseline where the reward is formulated as E 3' 2,, where a hyperparameter selected from {0.3, 0.4, .... 0.9, 1.0} and 2; are the normalized feedback signals. The feedback signals ordered according to the HERON reward hierarchy, making this very realistic and competitive reward engineering baseline. However, we came across few challenges when trying to make this algorithm work. First, the feedback signals all need to be normalized, which either requires complex algorithms or multiple agent rollouts before training. In addition, we find that this baseline is very sensitive to 3 and therefore has higher tuning cost. In addition, it can often not beat the performance of HERON. We plot the performance of the reward engineering baseline in Figure 7. Note that this plot shows performance over all of training, and HERON typically displays larger reward (comparatively) in the last stages of training.
As we can see from Figure 7, the reward engineering baseline requires extensive tuning achieve good performance. In addition, the choice of normalization strategy is very important (Figure 7f). These results further show the benefits of HERON.
D. Robotics
All of our experiments are conducted with the PyBullet simulator (Coumans & Bai, 2016). The feedback signals in each environment are follows: for Ant, it is whether the robot alive, the progress towards the goal state, whether the joints their limits, and whether the feet colliding. For HalfCheetah, the signals are the potential and the power cost. For Hopper, the signals are the potential, an alive bonus, and the power cost.
E. Traffic Light Control
In our experiments we train four agents in a two by two grid. The length of each road segment is 400 meters and cars enter through each in-flowing lane at rate of 700 car/hour. The traffic grid can seen in Figure 8. The control frequency is Hz, i.c. we need input an action every second, The reward based on the following attributes for each agent
q": The sum of queue length in all incoming lanes.
wtt:: Sum of vehiele waiting time in all incoming lanes.
dm The sum of the delay of all vehicles in the incoming lanes.
12

---

Deep Reinforcement Learning from Hierarchical Preference Design
1.05
1.00
0.75
Reward
0.95
Reward
0.5
Reward
0.90
0.4
0.3
0.65
0.85
Reward Eng.
0.80
Ground Truth
0.60
HERON
0.75
0.3 0.4
0.5 0.6
0.7 0.8
0.9
1.0
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
(a) Ant
(b) Half-Cheetah
(c) Hopper
-12.5
-40
-15.0
-60
-17.5
Reward
-80
Reward
-20.0
Reward
-100
-22.5
-120
-25.0
-35
Reward Eng.
-140
standard weight
clip
-27.5
HERON
Ground Truth
-160
HERON
-30.0
Ground Truth
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.4
0.5
0.5
0.7
10
B
B
(d) Traffic Lights
(e) Pendulum
(1) Mountain Car
Figure 7. Ablation study of the reward engineering haseline.
em": The number of emergency stops by vehicles all incoming lanes.
fl A Boolean variable indicating whether not the light phase changed.
ol": The number of vehicles that passed through the intersection.
We can then define the reward-engineering reward as
Rd = -0.5g" 0.5wt" - 0.5dl" 0.25em" - fl" + vl
All algorithms have the same training strategy. Each agent is trained for three episodes with 3000 SUMO time steps each. At the beginning of training the agent makes random decisions to populate the road network before training begins. Each algorithm evaluated for 5000 time steps, where the first 1000 seconds are used to randomly populate the road. For adversarial regularization, we use the l2 norm bound the attacks 5.
F. RLHF Comparison
To explicitly compare RLHF with HERON, we compare the algorithms in the pendulum environment. To simulate human feedback, we rank one trajectory over another if the ground truth reward achieved by that trajectory is higher than the ground truth reward achieved by the other trajectory. We then evaluate the performance this simulated RLHFF algorithm when varying amounts feedback are given. The results can seen in Figure In this table we vary the number feedbacks in RLHF, while keeping the number feedbacks for HERON constant. In this setting HERON can perform as well as RLHF, but such good performance is not guaranteed in every environment.
G. HERON Flexibility
In this section we evaluate how the behavior of the policies trained by HERON change when we change the reward hierarchy. We plot several hierarchies in Figure 10. The reward engineering is the thick black line. We try three signals as the most important signal (num passed, wait time, and delay). We notice that all these observations can outperform the reward
13

---

Deep Reinforcement Learning from Hicrarchical Preference Design
Figure 8. Traffic light control environment.
-20
RLHF HERON
-40
Reward
-60
-100
-120
-140
-160
9
25
60
100
Percent of Trajectories Labelled
Figure 9. RLIIF comparison in the Pendulum Environment.
engineering reward, even though we measure the return with the reward engineering reward. One important deviation from this good performance is when wait time is not ranked highly. The wait time a very important signal, and when do put this variable high up in the hierarchy, the performance becomes unstable when measured according the reward engineering reward. This is because if we ignore the wait time of cars, the policy may make some wait for long time, which is not ideal. However, this can easily be accounted for in the reward design process.
-40
-60
-80
-100
wait,num pwwppd..qseeeeee queue
num pw_pppasssveee passed.walt.quue walt.queue
walt.nwal.numm passed
delay, www.,num. _acuum___ded__--- Reward Engineering
-120
num num_passed.que bassed,qqueu,ddlle
num_passe dessed,delly.wwe..
gullay.quee.nummaae paasedd
num lrin_pased,,eee delay
num_ palpppaaassaal.e.eeeeepe delay.queue
-140
1000
2000
3000
4000
Evaluation Time Step
Figure 10. Different reward hierarchies in HERON.
H. Code Generation
In this section we describe details for the code generation task.
H.1. Behavior Cloning
To train the initial behavior model we use behavior cloning (supervised fine-tuning) to adapt the pre-trained Code to the APPS task. In particular, we use train with the cross-entropy loss for 12000 iterations, using a batch size of 64, We use the
14

---

Deep Reinforcement Learning from Hierarchical Preference Design
Adam optimizer with learning rate of 2 x 10-3.
H.2. Temperature Selection
hyperparameter that can have a large impact on generation quality the temperature parameter, which essentially alters how greedy we in the next-token sampling step. In all settings we follow the implementation of Le al. (2022), using temperature of 0.6 for APPS and 1.2 for MBPP. In addition, sample tokens greedily to construct a baseline sample for each problem.
H.3. Reward Model
It has been noted that reward models often overfit the dataset (Ouyang al., 2022). Therefore of CodeT5 for reward model with only 220 million parameters. We train this model for around 40000 steps with batch size of 64. This is roughly a single epoch on the preference dataset, which is comprised of 20 samples per problem sampled from the behavior model and some expert samples provided by the APPS dataset. We use the Adam optimizer with a learning rate of 2 x 10 10-5
smaller version
H.4. Reinforcement Learning
Once we have trained the reward model, we assign a reward to each program in our preference dataset and train using reinforcement learning on this dataset. Similar to Le et al. (2022), we train on the policy gradient loss and add the cross entropy loss as a regularization term. We compare our method to two reward engineering rewards:
CodeRL reward. The first reward we compare HERON from CodeRL, which defines the reward as
1.0
if program fails to compile if program has runtime error
0.6
RCodeR1. (s)
0.3
if program fails a unit test
1.0 if program passes all unit tests.
PPOCoder reward. The second reward we compare HERON to is based PPOCoder, which has the insight to include syntactic similarity to expert samples in the reward. This effectively smooths the reward, and can therefore make the reward more informative. In particular, they compare the abstract syntax trees of the generated programs with the expert example programs. This is computed as
Rust (8, Count AST;)/
We then construct the final PPOCoder based reward Rppocuter ReaARR (s) AMEAN;(RRA Raast (s, where MEAN is the mean operator. We tune À E {0.001, 0.01, 0.1, 1}. We remark that the original PPOCoder reward contains more feedback signals, but we do not use all of them due to the large tuning cost required to tune the ourselves.
For both of these rewards and the HERON reward we tune the learning rate {3 x 10-6 5,5 x 10-6, 6 8 10-67.
H.S. Example Programs
To further analyze the performance of HERON, we examine some of the programs generated by HERON. These programs are randomly selected. We display concatenated prompts and completions in Figure 11.
I. Reward Training
In this section detail our reward model training. For the classic control tasks and the traffic light control task do not have good initial behavior policy, must train our reward model in an iterative manner. In these settings, we iteratively update the reward model using samples from the current version of the policy. In this way the reward model is trained on samples generated from progressively better policies.
As we mentioned in our discussion on the computational costs of HERON, the cost of reward model training depends on the frequency which the reward model is trained. For the classic control environments we simply use linear training
15

---

Deep Reinforcement Learning from Hierarchical Preference Design
schedule, in which the reward model is updated every 400 steps. For traffic light control train the reward model with annealed frequency, where the reward model is trained every 100v steps, where is set 1.3 and is the current time step.
We demonstrate the multi-step reward model training in Figure 12. The sharp drop in accuracy occurs at time step 1000, where the behavior model changes from random to a trained policy. This large change in accuracy indicates that multi-step reward model training is needed, as reward models trained on random behavior do not perform as well when the behavior changes.
L1. The a Hyperparameter
Formal description of shaping signal: Given a trajectory т, let us compare with n other trajectories Ti,-.., Ta- Let F(T) denote the average level the decision tree T wins at. To allow us to incorporate domain knowledge into HERON, we multiply the reward assigned by a signal a F(r), where a is hyperparameter, When the feedback signals are categorical, F(r) can capture which category lies in, and multiplying the reward by of (7) can control the reward separation between different categories.
Visual description of shaping signal:As mentioned in the main text, the hyperparameter can be used to control the shape of the rewards. In Figure 13, we show how changing a changes the reward shape in the code generation task.
J. Computational Setup
For the classic control tasks and traffic light control experiment we run experiments on Intel Xeon 6154 CPUs, For the code generation task, train with Tesla V100 32GB GPUs.
K. Robotics Learning Curves
In Figure 14 we display the learning curves in the robotics environments.
L. Limitations
The main limitation of HERON is that not every problem will contain obvious ranking over the feedback signals, some signals may be equally important. We propose to mitigate this limitation in future works by allowing for ties or using randomized decision tree in the preference elicitation procedure.
M. Alignment Experiments
Here we present more details on our language model alignment experiments. We use LoRA (Hu et al., 2020) for all experiments. For the SFT base model, we train for two epochs with learning rate 5e-5. We use batch size 32 and train for 2 epochs. For Reinforce we also use learning rate 5e-5, batch size 32, and train for 2 epochs. For DPO, use learning rate 5e-5, batch size 32, B = 0.1, and train for 2 epochs.
For evaluation, we use each reward model as specified in their respective release. For Claude based evaluation, we prompt it to select the most correct, helpful, and harmless response.
16

---

Deep Reinforcement Learning from Hierarchical Preference Design
your Write function to ninimum tength ing Lanbda function. Your code should pass these tests:
expert Python programtr,,
assert nin. Length List [0], [1, 31, [5, 71, I9, 11], [13, 15, 1711)=(11 [8]]
assert win_length__usee Serth__isttifi Listil [1, 3,,,,,,,,
1i,,5,,,,,, 13,2,3,41 21,2,3, (1,2), 4 (6,7,,,, 11, (1,21),,,111
class Solution:
def Length_listtaaee *invars):
oue
:type invars: List [List(]]]
:rtype: (int, List [int]]
index float("inf"), II for iten invars: fleat('inf"):
return 1, index
elif not res:
index index [len]iten]]
else:
index[-1] len(iten):
res,
len(iten), [ten[iten]]
else:
index index [len(iten]]
return index
You are expert Python programrer, and here is your task: Mrite function to find the nth tetrahedral number. You should tests:
assert tetrahedrainumber(S)
35.8
assert tetrahedral_nurbrre (unber(6) == 56.8
assert tetrahedral_nunberl = 84.8
class Solution:
def Letrahedra an): print(n)
def
x2, y):
y
-1.0
x, y = x1, y +1
if [1x 5e-6
Se-6) and (x x1 = x2)): return -1.0
if 5e-6 and
x2) = 5e-7 and xl x2: retur -1.0
if (x & 8): y =11
while ly = //
<= y2):
if (y &
8):
= //2
abs(y1
abssyy2 x2)):
== abs xi)
x1, x2, yl, y2 x2, x1, y2, 2 (-1)
return
w
9. q=
while Trues
=
b
= f(x,
(a 1f else -1.0
if
return y a //
if
0): y
if [a 21:
if 1.1e-6 and 5.6e-71: 1.1e-5
1.2e-6): 1.2e-6
= p
9 P
expert Python progranner, here is task: Write function the given decinal number binary equivalent.
should pass tests:
assert dec secinate decinal_to Lo _tt_t_ttabbbb_t___ binary(8) = '1888'
assert decina _tt__binary(1__ '18818'
assert decina [to_binary(())
class Solution:
def decinal_to botaytoo_barrleefe
sen
:type x: int :rtype: str
[0]=0; range(8): a[i]==ieci;
return "'.joinlmaplstr, al);
Figure 1I. Example programs generate by LLMs trained HERON.
17

---

Deep Reinforcement Learning from Hicrarchical Preference Design
Reward Model Accuracy
0.90
0.85
0.80
0.75
0.70
0
2000
4000
6000
8000
Training Epoch
Figure 12, Reward model accuracy throughout training.
Compile Error
Number of Programs
Number Programs
Runtime Error
12000
12000
Failed Test
1.0000
Pass Tests
10000
8000
8000
6000
4000
4000
2000
2000
0.5
Reward Scores
Reward Scores
(a) 1.0
(b) 2.0
Number Programs
14000
Number of Programs
12000
12000
10000
20000
8000
8000
6000
6000
4000
4000
2000
2000
Reward Scores
Reward Scores
(c) a = 3.0
(d) 4.0
Figure 13. Reward shape with different values a.
18

---

Deep Reinforcement Learning from Hicrarchical Preference Design
1.2
1.1
1.2
1.0
1.0
0.9
Reward
Reward
0.8
0.8
0.7
Ground Truth HERON
0.6
0.6
0.4
0.5
0.4
Reward Eng.
0.2
0.3
0.4
0.8
1.2
2.0
0.4
0.8
1.2
1.6
2.0
Training Episodes (Millions)
Training Episodes (Millions)
Ant
(b) Hopper
1.2
1.
0.8
0.6
Reward
0.2
0.0
-0.2
-0.4
0.2
0.4
0.6
0.8
1.0
Training Episodes (Millions)
(c) HalfCheetah
Figure 14. Training curves different robotics tasks.
19


Reviews:
[{"review_id": "h3ueZVqpNs", "reviewer": "Reviewer_uDMx", "summary": "In this work, the authors propose a novel hierarchical reward design framework tailored for environments that require the integration of multiple feedback signals. The framework is motivated by the observation that these signals often contribute unequally to the overall reward and that there is a hierarchy in how much each feedback contributes. The paper includes extensive empirical evaluations across four different applications and compares the proposed method with alternative reward design approaches.\n\n---\n\n## update after rebuttal\n\nThank you to the authors for their detailed response. After reading the rebuttal and considering the other reviews, my main concerns were addressed, and I changed my original score.\n\n---", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer uDMx,\n\nThank you for your thoughtful and detailed review of our paper. We are glad you appreciate the novelty of our approach as well as our experimental analysis. We provide our rebuttal to your critiques below. We enumerate your questions and comments to save characters.\n\n## **Experimental Design and Analysis:**\n\n### **1. Traffic Light Control:**\n\n**1.1: How many samples were used?**\n- We present results over five random seeds. Indeed we find that the baselines have high variance, but HERON exhibits very low variance. To further validate the efficacy of our method, we have run a t-test for the last 1000 evaluation time steps in the traffic light control environment. Our method is significantly better than the reward engineering baseline, with p=0.003. This gain justifies the use of our algorithm.\n\n**1.2: Ground truth reward.**\n- The ground truth reward can be found on line 693, and it has been developed over several papers. It was finalized in [Zhang 2019].  Other baselines sometimes outperform it, as we extensively tune all baselines. This tuning can be seen as a type of reward shaping, which can improve performance.\n\n**1.3: Proportion of decisions made at different levels.**\n- If too many decisions are made at a single level, that effectively means that information from other levels of the decision tree are being completely ignored, which is not ideal. Decisions being made at all levels indicate information from all feedback signals is being incorporated into the reward, which is desirable and indicates the efficacy of our reward design. We compute the proportions in traffic light control by recording which level of the tree each decision was made at throughout training.\n\n**1.4: Figure 4.**\n- We only vary the second feedback signal to keep the reward realistic, as the first feedback (traffic throughput) is almost always viewed as most important in traffic light control. We show the policy can still achieve good performance with other hierarchies in Figure 10.\n\n**1.5: Figure 5.**\n- The original speed is 35, then we change it to 25, 30, 40, 45 (See Figure 5). We conduct 5 independent trials for each experiment and report mean and standard deviation over the runs. \n\n### **2. For the code generation experiments, what is the number of trials used, and what is the variance in the results?**\n- We conduct training over 1 seed due to the large expense of these training runs, which is in line with prior works.  However, when evaluating a policy on APPS, we evaluate each policy with 1 million generated programs (there are 5000 test problems, we generate 200 for each problem) and 100000 on MBPP. Treating each set of programs as an independent Bernoulli variable, we can conduct a t-test. When considering the largest value of K in pass@k in tables 1, 2, 3, 4 we find that the a t-test comparing HERON and the best baseline has p values < 0.05, indicating that HERON indeed outperforms the baselines in a statistically significant manner.\n\n\n### **3. In the LLM alignment experiments, why was HERON-DPO compared with REINFORCE?**\n- We compare with REINFORCE as it is one of the most popular and high performing approaches for LLM alignment these days [1]. It is possible the baseline could do better with PPO, but this would require extensive tuning and PPO is shown to underperform REINFORCE for LLM alignment [1].\n\n[1] Ahmadian, Arash, et al. \"Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.\" \n\n\n### **4. Regarding the robotics experiment, could the authors clarify the details of Table 6?**\n- In table 6 we show the average ground truth reward obtained by each algorithm over the final 1000 iterations of training (training is 2 million steps total). We conduct experiments over 5 random seeds, and use the PPO algorithm to optimize the policies. The reward hierarchy is generally is-alive > moving forward > control cost > contact cost. We will make this more clear in the paper.\n\n## **Questions**\n\n**Q1**: In code generation, we use a pre-trained language model as the initialization for the reward model. This is advantageous as it means reward training is faster and more accurate. Similarly for LLM alignment we could use a pre-trained LM as a reward, but opt for DPO in our experiments due to its simplicity.\n\n**Q2**: DPO is most useful when the horizon length of the task is one, as in that case we can directly train the policy on preference comparisons, without having to train a reward model. This is the case in LLM alignment, which is where DPO is mainly used.\n\nThank you again for the detailed review. We are eager to know if your questions about our empirical evaluation have been satisfied, or if there are any more details we can give you.", "claims_and_evidence": "The paper’s claims are primarily empirical—for instance, the authors claim improvements in robustness and overall performance compared to existing techniques in specific scenarios. Although the results appear to support these claims, crucial details are missing from the empirical evaluation section, making it difficult to fully confirm that the evidence supports the claims. Please refer to the detailed comments and questions in the Empirical Designs or Analyses section of the review.", "methods_and_evaluation_criteria": "The method is generally well-described and motivated. The evaluation is also thoughtfully executed: the authors selected a variety of environments with distinct characteristics to showcase the different properties of the proposed method. For example, they demonstrate how the method performs when there is a clear hierarchy in the feedback signals versus when such a hierarchy is absent.", "theoretical_claims": "The paper does not present any theoretical claims or proofs.", "experimental_designs_or_analyses": "The experimental design is thorough, but several details require further clarification:\n- For the traffic light control experiment:\n    - How many samples were used? The plots show considerable variance, and the sample size is important for drawing reliable conclusions.\n    - What is the definition of the ground truth reward, and why does it show more variance compared to the other methods? Additionally, why do the other reward design methods appear to perform better than it?\n    - Could the authors clarify why it is important that a relatively small proportion of decisions are made at each level (as suggested by the results in Figure 3), and how these proportion values were computed?\n     - In the experiment presented in Figure 4, why was only the second feedback signal varied? What difference would it make if the number of cars passed (the first feedback) was not kept constant?\n     - Could the authors elaborate on the setup for the robustness experiment detailed in Figure 5? For instance, what was the original training speed, to which speed was it changed afterward, and how many trials were used?\n- For the code generation experiments, what is the number of trials used, and what is the variance in the results? While the paper claims that the proposed method “significantly outperforms the baselines,” some results (e.g., those in Tables 1-4 for HERON and CodeRL) are fairly close, which might challenge this claim if the variance is high.\n- In the LLM alignment experiments, why was HERON-DPO compared with REINFORCE? Could the reward engineering baseline potentially perform better if coupled with a different algorithm (e.g., PPO or TRPO)?\n- Regarding the robotics experiment, could the authors clarify the details of Table 6? For example, what do the numbers represent, how were they generated (e.g., the number of trials), what is the final hierarchy used for the feedback signals, and how was this hierarchy determined?", "relation_to_broader_scientific_literature": "This paper introduces a novel method for combining multiple feedback signals into a single reward measure. The approach offers advantages over traditional techniques, such as linear combinations with engineered weights, particularly in scenarios where the feedback signals have a clear hierarchy. Although the method addresses a somewhat specific problem, it appears to perform well in that context. I believe that the community could benefit from this approach.", "essential_references_not_discussed": "The reference list appears comprehensive, and I did not identify any essential references that were missing.", "other_strengths_and_weaknesses": "The method is clearly described and straightforward, making it easy to follow. The motivation is also well-articulated and intuitive. However, as noted in earlier sections of the review, there are important aspects of the empirical evaluation that should be further clarified in the paper.", "other_comments_or_suggestions": "Figure 6 is missing axis information.", "questions_for_authors": "1. On line 176, the authors mention that \"it is possible to introduce pre-trained knowledge into the reward model.\" Could the authors provide concrete examples of this?\n2. On line 198, the authors state that \"in appropriate settings, we can use DPO …\". Could the authors clarify what constitutes an appropriate setting in this case? For instance, are there particular types of tasks or domain characteristics that make DPO especially suitable?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6rrUZlPov6", "reviewer": "Reviewer_7gbW", "summary": "The paper proposes HERON, a hierarchical preference-based RL framework that leverages the hierarchical importance of feedback signals to design reward functions. HERON constructs a decision tree based on human-provided importance rankings of feedback signals to compare trajectories and train a preference-based reward model. The authors claim HERON improves sample efficiency, robustness, and applicability to sparse reward settings. Experiments are conducted on traffic control, code generation, and other tasks to validate these claims.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7gbW,\n\nThank you for your detailed review and suggestions to improve our work. First and foremost we would like to address your review of our claims, in which you say we claim “HERON universally eases reward design”. Nowhere in the paper do we make this claim, and in fact we consistently state that our scope is limited to problems with hierarchical structure (see line 51 right, line 62 left, and line 436 left).\n\nWe present our rebuttal of the remainder of your review below. We shorten some questions to save space.\n\n## **Experimental Design or Analyses**\n\n**The choice of baselines is relatively outdated, lacking comparisons with [1] and [2]**\n- We believe there is a misunderstanding here. HERON is not comparable to preference learning methods. HERON assumes online access to an environment which gives the agent several feedback signals. We seek to design a reward function from these feedback signals, which is a classical setting in RL. On the other hand, [1] requires access to an offline preference dataset, while [2] requires access to online preference annotation. In contrast, HERON asks a human annotator to rank the importance of feedback signals (usually there are 3-7 such signals) one time. This is completely separate from the recommended papers. We include a detailed discussion on suitable scenarios in line 436.\n\n**It remains unclear if the experiments cover a sufficiently diverse range of scenarios...**\n- We conduct extensive experiments in 7 diverse environments, which we believe is sufficient to show the benefit of HERON (multi-agent traffic light control, code generation, LLM alignment, and four robotic control experiments). In robotic control experiments the feedback signals do not all have a clear hierarchy, yet we are able to beat or match the baselines in all environments. These results demonstrate that our method performs well in a diverse set of hierarchical or roughly hierarchical environments.\n\n## **Relation To Broader Scientific Literature**\n**A more comprehensive discussion of how HERON fits into and advances the current state-of-the-art would be beneficial.**\n- Thank you for the suggestion. We will add more works on SOTA approaches in our related works section, including [1] and [2]. Please note we already have a discussion on suitable settings for HERON (line 436).\n\n## **Weaknesses**\n**Weakness 1**\n- First, we want to point out that HERON is designed for settings where there is a clear hierarchy (see our discussion on suitable scenarios) and performance in other settings is a “bonus.” However, in environments without clear hierarchy, we find HERON can still perform decently, outperforming reward engineering baselines. See section 4.4 of our paper for more details.\n\n**Weakness 2**\n- Theoretical analysis is challenging, as it is difficult to precisely characterize the relationship between the feedback signals and the ground truth reward. For example, in traffic light control, defining an optimal combination of all relevant factors is inherently ambiguous. \n- That said, we hypothesize that if theoretical guarantees were to be established, the convergence of the policy would likely depend on two key factors: (1) the statistical error in the learned reward, and (2) the distribution shift between the state visitation distributions of the sampling policy and the optimal policy.\n\n## **Questions**\n**Q1:** See our response to weakness 1.\n\n**Q2:** One approach is to learn a decision tree based on human preference data. This may work when the feedback signals have hierarchical structure and we only have a limited amount of human preference data. However, this would be a separate setting and we leave it for future work.\n\n**Q3:** See response to weakness 2.\n\n**Q4:** The main baseline we compare against is reward engineering, which has been the most popular approach for reward design over the past 20 years. We also compare against two baselines that have been used in MORL literature, the ensemble baselines. In relevant settings such as code generation, we compare to carefully designed and publicized reward functions like that of CodeRL. Finally, when available, we also compare training directly on the ground-truth reward. Again we note that those SOTA methods of [1] and [2] are not relevant in our setting, as they assume access to a large set of human preference data (either online or offline) which HERON is not designed to use.\n\n**Q5:** HERON is relatively robust to suboptimal rankings. In Figure 6, we show how HERON performs with inexact domain knowledge, i.e. only knowing which factors fall in the top 3 and which ones fall in the bottom 3. By one tuning iteration, HERON can already outperform the best-case performance of reward engineering. This indicates HERON can perform well even with slightly noisy rankings.\n\nThank you for the detailed review, and please let us know if you have any further questions or need any clarification.", "claims_and_evidence": "The claim that HERON universally eases reward design is problematic since many real-world tasks do not naturally offer clearly prioritized feedback signals. More comprehensive experimental validation and discussion are needed to convincingly support the broad claims.", "methods_and_evaluation_criteria": "**Methods**:\n\nThe use of pairwise comparisons to train a reward model is innovative, yet its dependency on accurate, human-specified importance rankings may limit its scalability and applicability in domains lacking clear hierarchy.\n\n**Evaluation Criteria**: \n\nThe benchmarks and tasks selected for evaluation appear to showcase improved sample efficiency and robustness. However, the evaluation does not sufficiently address scenarios where the hierarchical structure is ambiguous or entirely absent, raising concerns about the method’s generalizability.", "theoretical_claims": "**Theoretical Analysis**: \n\nThe paper does not provide a rigorous theoretical analysis or proofs ensuring that the learned reward function preserves the optimal policy.\n\n**Issues**: \n\nWithout theoretical guarantees, there is a risk that the policy may converge to a suboptimal local optimum, particularly in sparse reward environments where reward signals are less distinct.", "experimental_designs_or_analyses": "The choice of baselines is relatively outdated, lacking comparisons with recent advancements such as inverse preference learning approaches (e.g., Hejna & Sadigh, 2023) and Hindsight PRIORs (e.g., Verma & Metcalf, 2023).It remains unclear if the experiments cover a sufficiently diverse range of scenarios, especially those where feedback signals do not exhibit a clear hierarchical structure.", "relation_to_broader_scientific_literature": "The work builds upon prior research in preference-based reinforcement learning and reward shaping by introducing a hierarchical structure that mimics human decision-making. A more comprehensive discussion of how HERON fits into and advances the current state-of-the-art would be beneficial.", "essential_references_not_discussed": "Lack of disscussions of recent studies, such as [1], [2].\n\n[1] Hejna, J., & Sadigh, D. (2023). Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 18806-18827. \n\n[2] Verma, M., & Metcalf, K. (2024) Hindsight PRIORs for Reward Learning from Human Preferences. In The Twelfth International Conference on Learning Representations.", "other_strengths_and_weaknesses": "**Strengths**: \n\nInnovative hierarchical framework that intuitively aligns with human decision processes.\n\nEmpirical results indicating improvements in sample efficiency and robustness on the tested tasks.\n\n**Weaknesses**: \n\nReliance on a clear hierarchy in feedback signals, which may not be present in many practical applications.\n\nLack of theoretical analysis to ensure that the learned reward function leads to optimal or near-optimal policies.\n\nExperimental comparisons are limited to older baselines, missing insights from more recent literature.", "other_comments_or_suggestions": "No", "questions_for_authors": "1.\tHow does HERON perform in environments where feedback signals do not have an inherent hierarchical structure?\n\n2.\tHave you considered leveraging neural network architectures to automatically learn or approximate the hierarchical priorities when explicit human rankings are unavailable?\n\n3.\tCan you provide any theoretical analysis or formal guarantees regarding the convergence and optimality of the policies derived from the HERON framework, especially in sparse reward settings?\n\n4.\tWhat is the rationale behind the selection of baseline methods, and how do you expect HERON to compare against more recent state-of-the-art methods like those proposed by Hejna & Sadigh (2023) and Verma & Metcalf (2024)?\n\n5. How sensitive is HERON to noisy or suboptimal human rankings of feedback signals?\n\n[1] Hejna, J., & Sadigh, D. (2023). Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 18806-18827. \n\n[2] Verma, M., & Metcalf, K. (2024) Hindsight PRIORs for Reward Learning from Human Preferences. In The Twelfth International Conference on Learning Representations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHHglpmSki", "reviewer": "Reviewer_zX5S", "summary": "This paper introduces HERON, a decision-tree-based approach for reward design in reinforcement learning. In particular, the authors leverage human expertise to define a hierarchy of feedback signals. The authors employ that hierarchy to compare trajectories, collecting a dataset (with a policy model) and learn a reward model (similar to standard procedure in preference-based RL methods). The reward model is then used to train the policy, and the whole procedure can be repeated to improve the reward model. The authors extensively evaluate their approach in diverse scenarios (multi-agent traffic light control, code generation, language model alignment and standard control environments), highlighting its versatility and performance. Finally, the authors present an ablation study, focused on the training time, hyperparameter sensitivity and tuning cost.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zX5S,\n\nThank you for your detailed review and insightful suggestions! We are glad you think our work is novel. We provide our response to your review below. We shorten some questions to save space.\n\n## **Claims and Evidence**\n\n**The authors hypothesize that HERON's reward function \"may be more conducive to learning\". However no empirical support to this claim is provided.**\n- We would like to clarify that this is an intuition rather than a definitive claim. Our hypothesis—that HERON's reward function may be more conducive to learning—was meant as a possible explanation for HERON's empirical performance advantage, not a conclusive statement. This hypothesis stems from the fact that many raw feedback signals are discrete or sparse, and reward models can serve as a way to smooth out this sparsity.\n- We do find some supporting evidence in Figure 14, where HERON exhibits significantly smoother and more stable learning curves compared to reward engineering. This suggests that HERON's reward function facilitates more stable training dynamics, which aligns with our intuition.\n\n## **Experimental Designs Or Analyses**\n**However, I would like to single out HERON’s post-training reward scaling procedure in Section 4.2. This method appears to be an additional form of reward shaping, but it is not really motivated in the paper.** \n- Our post-training scaling is motivated by Figure 13, in which we found that reward modeling alone does not perfectly separate correct code from incorrect code at a global level. In order to further separate the two cases, we introduce a scaling parameter. Note that we only have a single parameter to tune, compared to the 4 used in CodeRL.\n- The pass@50 results without reward scaling can be found below. HERON still performs decently (outperforming all baselines, which are carefully tuned in prior works), but we find that post-scaling can further improve performance. We will include the results in the appendix.\n### Table: Pass@50 Scores \n| Model       | Pass@50 Score |\n|-------------|---------------|\n| HERON       | 9.85          |\n| CodeRL      | 9.81          |\n| HERON+PTS   | 10.19         |\n| PPOCoder    | 7.62          |\n| BC          | 6.74          |\n\n\n## **Relation To Broader Scientific Literature**\n\n**There is also a significant connection to multi-objective RL (MORL) literature...**\n- Thank you for pointing out reference [1], we will include it in our related work. As for MORL,we remark that MORL tries to find the pareto frontier among several reward factors. In contrast, HERON tries to find a way to combine feedback signals into a reward in a user-friendly way, such that the user can easily guide the agent’s behavior. We believe these are two orthogonal directions. We will move discussion of MORL to the related work section.\n\n## **Essential References Not Discussed**\n\n**A major missing reference and baseline is PEBBLE**.\n- PEBBLE is not directly comparable to HERON, as PEBBLE relies on human preference data for learning. HERON on the other hand is meant for settings where we try to design a reward function from some freely available feedback signals. Therefore PEBBLE cannot be used as a baseline in our experiments.\n- Moreover, PEBBLE’s contributions (unsupervised pre-training and off-policy learning) are orthogonal to HERON’s contributions. Therefore we do not include PEBBLE as a reference.\n\n## **Weaknesses**\n\n**However, a significant limitation of HERON is its assumption of an underlying ranking of feedback signals.**\n- While HERON assumes an underlying ranking of feedback signals, this assumption is well-motivated by many real-world scenarios where feedback signals naturally have a hierarchy—for example, in traffic light control, code generation, and LLM alignments. In such contexts, ranking the relative importance of feedback signals is valid. We believe good results in these environments already represent a significant contribution.\n- Moreover, as demonstrated in Section 4.4, HERON performs well even in the absence of a strict feedback hierarchy. This empirical resilience highlights its practical utility and broad applicability, even when the ranking structure is noisy or only partially specified.\n\n## **Questions**\n**Q1:** See above response to **Experimental Designs Or Analyses**.\n\n**Q2:** We find HERON can still perform well in these settings, outperforming reward engineering baselines (Table 6). HERON is relatively robust to shuffled rankings. In Figure 6, we show how HERON performs with inexact domain knowledge, i.e. only knowing which factors fall in the top 3 and which ones fall in the bottom 3. By one tuning iteration (i.e. the best of two shuffled rankings out of a possible 36), HERON can already outperform the best-case performance of reward engineering. \n\n**Q3**: See above response to **Claims and Evidence**.\n\nThank you again for your thoughtful review, and please let us know if you have any further questions.", "claims_and_evidence": "The authors mostly position their claims of HERON over: (i) the overall performance of the method in comparison with baselines, (ii) the robustness of the method to changes in magnitude of the underlying reward signals, and (iii)  the adaptability of the method across multiple domains.\n\nThe empirical evidence presented in Section 4 supports many of these claims: the authors show that HERON outperforms relevant baselines across multiple settings. In Section 4.1, the authors also show how the performance of HERON is less sensitive to changes in the dynamics of the environment. The authors also present an extensive ablation study in Section 4.5, which provides additional insights on the sensitivity of the model to hyper parameters, as well as the training and computational cost of training HERON.\n\nI would like to point out that in Section 4.2 (Results), the authors hypothesize that HERON's reward function \"may be more conducive to learning\". However no empirical support to this claim is provided, such as learning curves showing faster convergence.", "methods_and_evaluation_criteria": "The proposed method is sound and addresses the real-world RL challenge of designing effective reward functions. The use of hierarchical preferences over reward features, instead of linear combinations is an interesting contribution and is well motivated and explained in the paper. The evaluation criteria are also appropriate for each domain: reward of the agents in the evaluation environment (given by a ground-truth reward function), win rates for language model alignment and pass@K metrics for code generation.", "theoretical_claims": "There are very little theoretical claims in this paper. The loss function of HERON (Equation 2) follows standard preference-based RL formulations and appears correct. There are no proofs in the paper.", "experimental_designs_or_analyses": "The experimental design and analysis appears to be sound. In particular, I would like to highlight the impressive range of evaluation scenarios employed in this work: from multi-agent systems, to code generation, LLM alignment and control tasks. Across all of them, HERON either outperforms or performs on par with the baseline methods.\n\nHowever, I would like to single out HERON’s post-training reward scaling procedure in Section 4.2. This method appears to be an additional form of reward shaping, but it is not really motivated in the paper. The authors should experimentally evaluate if this extra step actually contributes to the performance of HERON, with an ablated version of the model.", "relation_to_broader_scientific_literature": "The paper positions HERON within the area of preference-based RL. The authors discuss in Section 2 the connections and differences between the proposed method and others in RLHF, reward shaping, and inverse RL. The authors propose a decision-tree-based approach to compare agent trajectories to build a reward function. Tree-based structures for the reward function have been explored previously explored [1], but the use of a hierarchy of feedback signals appears to be novel. There is also a significant connection to multi-objective RL (MORL) literature, yet the authors only mention this at the end of the paper. I believe that it should deserve a section in related work, especially as the feedback signals employed by the authors can be considered different objectives in MORL.\n\n\n[1] Bewley, Tom, and Freddy Lecue. \"Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions.\" Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.", "essential_references_not_discussed": "A major missing reference and baseline is PEBBLE [1], a literature-standard method in preference-based RL. A direct comparison, in particular for the control experiments of Section 4.4., would clarify whether HERON’s benefits stem from its hierarchical structure or simply from being a preference-based reward modeling approach.\n\n[1] Lee, Kimin, Laura Smith, and Pieter Abbeel. \"Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.\" arXiv preprint arXiv:2106.05091 (2021).", "other_strengths_and_weaknesses": "One of the main strengths of this paper is that it presents an easily interpretable approach to reward design: the authors already explore some interpretability in Section 4.1 (Signal Utilization). The extensive evaluation suite is also to be commended.\n\nHowever, a significant limitation of HERON is its assumption of an underlying ranking of feedback signals. While the authors evaluate HERON on control tasks where it is not clear what in the ranking, it still remains unclear if the level of performance observed would generalize to other tasks.", "other_comments_or_suggestions": "There is a typo in Page 2: \"humans typically start with the the most important factor\"", "questions_for_authors": "1 - Can the authors justify better the need for the post-training reward scaling (presented in Section 4.2)? Shouldn't the original decision tree already learn to differentiate between these different feedback signals? Does HERON still outperform baselines without this heuristic adjustment?\n\n2- How does HERON handle cases where the importance ranking of feedback signals is uncertain (for example in the experiments of Section 4.4)? Would its performance degrade significantly if the ranking were shuffled?\n\n3 - Can you provide some empirical support to the claim that HERON's reward function \"may be more conducive to learning\" in Section 4.2? For example, showing the learning curves of HERON against the baselines.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p2j1eV1jL1", "reviewer": "Reviewer_Q7xT", "summary": "This paper proposes HERON, a novel hierarchical reward design framework for reinforcement learning (RL) that leverages the hierarchical structures of feedback signals to ease the reward design process. HERON constructs a decision tree based on the importance ranking of feedback signals to compare RL trajectories and trains a reward model using these comparisons. The authors demonstrate HERON's effectiveness across various RL applications, including traffic light control, code generation, language model alignment, and robotic control. In traffic light control, HERON outperforms reward engineering techniques and achieves higher performance than the ground-truth reward. In code generation, HERON surpasses state-of-the-art methods using handcrafted piece-wise reward functions, showing improved sample efficiency and robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Q7xT,\t\n\nThank you for the insightful review! We are glad you appreciate our work, and present our rebuttal to your review below.\n\n## **Experimental Designs Or Analyses**\n\n**The experiments could benefit from additional ablation studies to further isolate the impact of specific components of the HERON framework, such as the hierarchical decision tree and the preference-based reward model, on the overall performance.**\n- Note that we conduct an extensive ablation study in section 4.5. We find that the margin parameter is quite important. \n- Furthermore, we also investigate the flexibility and robustness of HERON, finding HERON is both flexible and provides increased robustness.\n\n## **Other Strengths And Weaknesses**\n\n**A potential weakness is the limited exploration of scenarios with unclear hierarchical structures, which could restrict the framework's applicability in more complex real-world environments**\n- First, we remark that HERON is designed for settings where there is a clear hierarchy (see our discussion on suitable scenarios) and performance in other settings is a “bonus.”  In settings with unclear hierarchies like robotic control (see section 4.4), we find HERON can still outperform reward engineering. Our intuition is that in these settings a roughly correct hierarchy provides sufficient information for good performance. If the signals have equal performance, we propose in line 402 to randomly flip the ranking of equal feedback signals. However, we leave this study for future work.\n\n## **Questions**\n\n**Could the authors provide more details on how HERON handles scenarios where the hierarchy of feedback signals is not clear or where signals have equal importance?**\n- See our above response to **Other Strengths And Weaknesses**.\n\n**How does the computational cost of HERON compare to other state-of-the-art methods, especially in terms of training time and resource requirements?**\n- We show the training time of HERON versus baselines in Figure 6a. HERON is around 25% slower than reward engineering, but greatly reduces tuning cost (Figure 6c). HERON is faster than other baselines such as the ensemble baselines as well.\n\n**Can the authors discuss the potential impact of HERON on multi-objective RL problems, where objectives may conflict?**\n- Thank you for the suggestion. As we mention in Line 420, HERON is not designed for multi-objective RL (MORL). In fact, HERON and MORL are solving different problems. MORL tries to train a policy on the pareto frontier among several reward factors. In contrast, HERON tries to find a way to combine feedback signals into a reward in a user-friendly way, such that the user can easily guide the agent’s behavior. This is useful in many real-world tasks—like code generation, aligning language models, or traffic light control—where some goals are naturally more important than others. To make this clearer, we’ll move the MORL discussion to the related work section and explain how it differs from our approach.\n\n\nThank you again for the insightful review, and please let us know if you have any other questions. We look forward to further discussion!", "claims_and_evidence": "The claims made in the submission are generally supported by evidence. The authors provide extensive experimental results across multiple domains (traffic light control, code generation, language model alignment, and robotic control) to demonstrate the effectiveness of HERON compared to existing methods. However, the claim that HERON can achieve decent performance even in environments with unclear hierarchy (robotic control) is less convincing, as the results show mixed performance and the environments tested may not fully represent the complexity of real-world scenarios where hierarchy is unclear.", "methods_and_evaluation_criteria": "The proposed methods in the paper are well-suited for the problem of reward design in reinforcement learning, especially in scenarios where feedback signals have a natural hierarchy or where rewards are sparse. The evaluation criteria and benchmark datasets used, such as traffic light control, code generation, and robotic control tasks, are relevant and effectively demonstrate the versatility and effectiveness of the HERON framework. However, the evaluation could be further strengthened by including additional real-world applications with more complex hierarchical structures to better validate the robustness of HERON in diverse environments.", "theoretical_claims": "The paper does not present any formal proofs for theoretical claims. Instead, it focuses on empirical validation through extensive experiments across various applications. Therefore, there are no proofs to verify for correctness in this submission.", "experimental_designs_or_analyses": "The experimental designs and analyses presented in the paper are generally sound and well-structured. The authors conducted extensive experiments across multiple domains, including traffic light control, code generation, language model alignment, and robotic control, to validate the effectiveness of the HERON framework. However, the experiments could benefit from additional ablation studies to further isolate the impact of specific components of the HERON framework, such as the hierarchical decision tree and the preference-based reward model, on the overall performance.", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to the broader scientific literature on reinforcement learning (RL) and reward design. Specifically, the proposed HERON framework builds upon prior work in hierarchical reward modeling and preference-based learning, offering a novel approach to leverage hierarchical structures in feedback signals to simplify reward design. This work also connects to recent advancements in deep RL, particularly in applications like traffic light control and code generation, where it demonstrates significant improvements over existing methods, highlighting its relevance and potential impact in the field.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper's strengths include its original approach to reward design through hierarchical structures, which is a creative combination of existing ideas in preference learning and RL. The application of HERON to real-world use cases like traffic light control and code generation demonstrates its practical significance and potential impact. The clarity of the paper is also commendable, with well-organized sections and clear explanations of the methodology and results. However, a potential weakness is the limited exploration of scenarios with unclear hierarchical structures, which could restrict the framework's applicability in more complex real-world environments.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Could the authors provide more details on how HERON handles scenarios where the hierarchy of feedback signals is not clear or where signals have equal importance? This would help in understanding the robustness of the framework in more complex real-world applications.\n\n2. How does the computational cost of HERON compare to other state-of-the-art methods, especially in terms of training time and resource requirements? This information is crucial for assessing the practical feasibility of deploying HERON in resource-constrained environments.\n\n3. Can the authors discuss the potential impact of HERON on multi-objective RL problems, where objectives may conflict? Understanding this could highlight the broader applicability of HERON beyond the current scope of the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces HERON, which leverages the rankings of multiple reward factors to derive reward functions using a hierarchical decision tree.

Strengths

* Interpretability of proposed framework in reward modeling

* Extensive evaluation across various domains (multi-agent traffic light control, code generation, language model alignment and standard control environments)

Weaknesses

* Assumption of an underlying ranking of feedback signals

Overall, all reviewers agreed that this is a very solid submission and the authors also handled concerns from reviewers during the discussion period. I think the paper makes a nice contribution that the community will find valuable.
Decision: Accept (poster)