Reward Learning with Trees: Methods and Evaluation
Authors: Tom Bewley
Jonathan Lawry
Arthur Richards
Rachel Craddock
Ian Henderson
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
reward learning
alignment
human-agent interaction
explainable AI
XAI
interpretability
decision trees

Abstract: Recent efforts to learn reward functions from human feedback have tended to use deep neural networks, whose lack of transparency hampers our ability to explain agent behaviour or verify alignment. We explore the merits of learning intrinsically interpretable tree models instead. We develop a recently proposed method for learning reward trees from preference labels, and show it to be broadly competitive with neural networks on challenging high-dimensional tasks, with good robustness to limited or corrupted data. Having found that reward tree learning can be done effectively in complex settings, we then consider why it should be used, demonstrating that the interpretable reward structure gives significant scope for traceability, verification and explanation.

Under review as a conference paper at ICLR 2023
REWARD LEARNING WITH TREES: METHODS AND EVALUATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent efforts to learn reward functions from human feedback have tended to use deep neural networks, whose lack of transparency hampers our ability to explain agent behaviour or verify alignment. We explore the merits of learning intrinsi- cally interpretable tree models instead. We develop a recently proposed method for learning reward trees from preference labels, and show it to be broadly com- petitive with neural networks on challenging high-dimensional tasks, with good robustness to limited or corrupted data. Having found that reward tree learning can be done effectively in complex settings, we then consider why it should be used, demonstrating that the interpretable reward structure gives significant scope for traceability, verification and explanation.
INTRODUCTION
For a reinforcement learning (RL) agent to reliably achieve goal or desired behaviour, this objective must be encoded reward function. However, manual reward design is widely understood to be challenging, with risks of under-, over-, and mis-specification leading to undesirable, unsafe and variable outcomes (Pan al., 2022). For this reason, there has been growing interest in enabling RI. agents to learn reward functions from normative feedback provided by humans (Leike et al. 20188) These efforts have proven successful from a technical perspective, but an oft-unquestioned aspect of the approach creates a roadblock to practical applications: reward learning typically uses black-box neural networks (NNs), which resist human scrutiny and interpretation.
For advocates of explainable AI (XAI), this is a problematic state of affairs. The XAI community is vocal about the safety and accountability risks of opaque learning algorithms (Rudin) 2019), but an inability to interpret even the objective that an agent is optimising places us in yet murkier epistemic territory, in which an understanding of the causal origins of learnt behaviour, and their alignment with human preferences, becomes virtually unattainable. Black-box reward learning could also be learnt reward function is a tantalising object of study
seen as a missed scientific opportunity. from an XAI perspective, due to its triple status as (1) an explanatory model of revealed human preferences, (2) a normative model of agent behaviour, and (3) a causal link between the two.
The approach proposed by Bewley & Lecue (2022) provides a promising way forward. Here, human preference labels over pairs of agent behaviours are used to learn tree-structured reward functions (reward frees), which are hierarchies of local rules that admit visual and textual representation and can be leveraged monitor and debug agent learning. In this paper, we adapt and extend the method (including by integrating it with model- RL agents), and compare it to NN-based re- ward learning in challenging aircraft handling domain. We find it to broadly competitive on both quantitative metrics and qualitative assessments, with our new modification to tree growth yielding significant improvements. The resultant trees are small enough to be globally interpretable (*v 20 leaves), and we demonstrate how they can be analysed, verified, and used to generate explanations.
The primary contribution of this paper positive empirical evidence that reward learning can be done effectively using interpretable models such as trees, even in complex, high-dimensional continuous environments. We also make secondary methodological contributions: improvements proposed learning algorithm, as well as metrics and methods for reward evaluation and interpretability that may be useful to others working in what remains a somewhat preparadig- matic field. After reviewing the necessary background and related work in Sections 2 and 3 we present our refinement of reward tree learning in Section 4 and describe how we deploy online experiments and results, which con-
to the originally-proposed
with model-based agent in Section 5 Section contains sider both quantitative and qualitative aspects of learning performance, and an illustrative analysis of learnt tree structures. Finally, Section concludes and discusses avenues for future work.

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND AND RELATED WORK
Markoy Decision Processes (MDPs) In this formulation of sequential decision making, the state of system at time f, St E S, and the action of an agent, d, A, condition the successor state 8(41 according to dynamics D: : S x A (SS) (A(+) denotes the set of all probability distributions over set). A reward function R A - IR then outputs scalar reward rt-1 given St, a and 81+1- RL uses exploratory data collection to learn action-selection policies π S - ((A), with the goal of maximising the expected discounted sum of future reward, ED -n-o 7 E [0, 1].
Reward Learning In the usual MDP framing, R is an immutable property of the environment, which belies the practical fact that Al objectives originate in the uncertain goals and preferences of fallible humans [Russel] 2019). Reward learning (or modelling) (Leike et al. (0188) replaces hand- specified reward functions with models learnt from humans via revealed preference cues such as demonstrations (Ng et al. 20000, scalar evaluations (Knox & Stone 2008). aaprrvaal labels (Griffith) itt al. 2013). corrections Bajesy et al.] (2017). and rankings (Christiano et al. 2017). The default use of NNs for reward learning severely limits interpretability; reward trees provide possible solution.
XAI for RL (XRL) Surveys of XAI for RL (Puiutta & Veith 2020 Heuillet et al. 2021) divide between intrinsic approaches, which imbue agents with structure such as object-oriented representa- tions (Zhu et al. 2018) or symbolic policy primitives (Verma et al. 2018), and post hoc analyses of learnt representations (Zahavy et al. 2016). including computing feature importance/saliency (Hu- ber et al. 2019). Spatiotemporal scope varies from the local explanation of single actions (van der Waa et al. 2018 to the summary of entire policies via representative trajectories (Amir & Amir, 2018) or critical states (Huang et al. 2018 While most post hoc methods focus on fixed policies, some provide insight into the dynamics of agent learning (Dao et al. 20188) Bewley et al. 2022).
Explainable Reward Functions At the intersection of reward learning and XRL lie efforts to im- prove human understanding of reward functions and their effects on action selection. While this area is "less developed" than other XRL sub-fields (Glanois et al. 202T]. distinction has again emerged between intrinsic approaches which create rewards that decompose into semantic com- ponents (Quozapaitis et al., 2019) or optimise for sparsity (Devidze et al. [02T], and post hoc ap- proaches which apply feature importance analysis (Russell & Santos 2019), |antos|[019), counterfactual probing (Michaud et al., 2020), or simplifying transformations (Jenner & Gleave 2022). [Sanneman & Shah (022) use -urinnt metrics to compare the efficacy of reward explanation techniques. In this taxonomy, reward tree learning is an intrinsic approach, as the rule structure is inherently readable.
Trees in RL Tree models have long history in RL (Chapman & Kaelbling 1991 Džeroski al., 1998 Pyeatt 2003). Their use is increasingly given an XRL motivation. Applications again divide into intrinsic methods, where an agent's policy (Silva all 2020), value function (Liu et al. 2018 Roth al. 2019) or dynamics model (Jiang et al. 2019) is tree, and post hoc tree approximations of an existing agent's policy (Bastani al., 2018 Coppens et al. 2019) or transition statistics (Bewley et 2022). Related to our focus on human- centrai--eeticcccccc learning, Cobo et al. (2012) learn strustrudd MDP abstractions from demonstrations and "Tambwekar et al. (2021) distill a differentiable tree policy from natural language. While (Sheikh et al. (2022) use tree evolution to learn dense intrinsic rewards from sparse environment ones, Bewley & Lecue 2002) are the first to learn and use reward trees in the absence of any ground truth reward signal, and the first to do so from human feedback.
3 PREFERENCE BASSED REWARD LEARNING
We adopt the preference-bact approach to reward learning, in which a human is presented with pairs of agent trajectories (sequences of state, action, next state transitions) and expresses which of each pair they prefer as a solution to a given task of interest. A reward function is then learnt to explain the pattern of preferences. This approach is popular in the existing literature (Wirth et al. 2016 Christiano et al. 2017: Lee et al. (2221b) and has a firm psychological basis. Experimental results indicate that humans find cogmitively easier to make relative (vs. absolute) quality judge- ments (Kendall 1975; Wilde et al. 2020) and exhibit lower variance when doing so (Guo et al.) 2018). This is due part to the lack of requirement for an absolute scale be maintained in working memory, which is liable to induce bias as it shifts over time (Eric et al. 2007).
We formalise trajectory ş as sequence (xi....,''p x/ where x² 0( st) E RR repre- sents a single transition as an F-dimensional feature vector. Given N trajectories, = = ! the human provides K < N(N 1)/2 pairwise preference labels, L= ((i.)) each of which indicates that the jth trajectory is preferred to the ith (denoted by g > {'). Figure| [ (left) shows how preference dataset D (E, L) can be viewed directed graph.
2

---

Under review as a conference paper at ICLR 2023
4.
42
14
P2
8
Y1
Figure 1: Left: The input to preference-based reward learning is a directed graph over a trajectory set = { Ji=1, where each edge (i,j) represents a preference & et. Each member of E is
sequence of points in R (blue connectors show mapping). Right: Application of the four model induction stages from Sections 4.1 t.4 to this example. See Appendix A.3 for an annotated version.
To learn a reward function from D, we must assume a generative model for the preference labels. Typically, it is assumed that the human produces labels Boltzmann-ration sum of rewards (or return) output by latent reward function over the feature space, R IR + R.
accordance with the
This is formalised by adapting the classic preference model of Bradley 'eery (1952):
P(e: > (R) 1 (xp(3
(())R))) where G(g > R(x)), (I)
and B > 0 is temperature coefficient. The objective of reward learning is to approximate R within some learnable function class R. This is often formalised as minimising the negative log-likelihoo (NLL) loss over L. |Wirth et al. (2016) also use the discrete 0-1 loss, which considers only the directions of predicted preferences rather than their strengths. These two losses are defined as:
(D,,,(,, Rt)
Σ (D,))CC
log P(g² > RR); fa. (D,
Σ I[P(s' > (R) - 0.5]. (2)
(W.))C
4 REWARD TREE INDUCTION
In prior work, R is the class of linear models R(x) w (Sadigh et al. 2017), which have limited expressivity, or deep NNs (Christiano et al. 2017), which resist human interpretation. As an intermediate option, Bewleyy& & Lecue (2022) (BL) propose the reward tree model. Here, the parameter space consists node-level splitting rules and reward predictions for an axis-aligned decision tree, whose leaves induce hyperrectangular partition of IR.. While differentiable trees exist (Suárez & Lutsko| 1999). these are of the oblique (c.f. axis-aligned) kind, whose rules far harder interpret in high dimensions. Therefore, instead of optimising the losses in Equation Snatiiinnnndde end-tt--nd, we use a multi-stage induction method with a proxy objective at each stage. The four stages outlined below, and depicted in Figure [ (right), depart from BL's original method in several respects. A list of changes, and their performance implications, is given in Appendix A.
4.1 TRAJECTORY- RETURN ESTIMATION
This stage considers the N trajectories as atomic units, and uses the preference graph to construct vector of return estimates g E R.,, which should be higher for more preferred trajectories (blue in Figure (4.1), (4.1), c.f. red). This is a vanilla preference-baset ranking problem, and admits a standard solution. BL use least squares matrix method to solve for g under Thurstone's s Case V preference model (Gulliksen) 1956). For consistency with prior work, and to avoid awkward clipping step which biases preference probabilities to enable matrix inversion, we instead use a gradient method minimise the NI.I loss under the Bradley-Tenn Terry model. Concretely, the objective for this stage is
Σ -log 1+ explg - gi) subject to {
min = and std(g) = 3, (3) or max (g)
argmin BERN
(
where is the mean trajectory length in E. EE,TT/N The min-or-max constraint ensures that all return estimates have the same sign (positive or negative), which aids both policy learning and interpretability (see Appendix A.2 We first optimise the NLL loss by gradient descent with the Adam optimiser (Kingma & Ba 2014), then apply shift and scale factors to meet the two constraints.
4.2 LEAF-LEVEL REWARD PREDICTION
The vector g estimates trajectory-level returns, but the aim of reward learning is decompose these into sums of rewards for the constituent transitions, then generalise this to make reward predictions for unseen data (e.g. novel trajectories executed by learning agent). BL's contribution is to do this using a tree model T, consisting of hierarchy of rules that partition the transition- feature space R² into T.r hyperrectangular subsets called leaves. Each leaf / E {1....) is associated with
3

---

Under review as a conference paper at ICLR 2023
{1..Lr} map feature vector x E IR' to the leaf in which it resides by propagating it through the rule hierarchy. ri is defined as an average over g, weighted by the proportion of timesteps that each trajectory in E spends in I:
reward prediction ry follows. Let the function leafy R
N
TM Ileaafy (x)) =
r =
(4)
1== ا (x/)
The effect of Equation 4 is assign higher reward to leaves that contain more timesteps from trajectories with high values. Predicting the reward for an arbitrary unseen feature vector x E R. then involves simply looking up the reward of the leaf in which it resides: (x) = Teefy (x).
While ostensibly naive, BL find that this time- weighted credit assignment more robust than several more sophisticated alternatives. It reduces the number of free parameters in subsequent induction stages, permits fast implementation, and provides an intuitive interpretation of predicted reward that is traceable back g value and timestep count for each 5 E. Figure | (4.2) shows how 4, 2 and 3 timesteps from El, 5 and ç are averaged over to yield the reward prediction for one Ieaf (indicated by the orange shading). For more intuition this stage, see the annotated figure in Appendix A.3
4.3 TREE GROWTH
Recall that the objective of preference- based reward learning is to find a reward model that optimises a measure of fidelity to D, such as the losses in Equation 2 When the model is a tree, this is achieved by the discrete operations of growth (adding partitioning rules) and pruning (removing rules). Given a tree T, a new rule has the effect of splitting the Ith leaf with hyperplane at location c E Cj along the fth feature dimension (where Cj C IR is set of candidate split thresholds, e.g. all midpoints between unique values in E). Let T + [[ denote the newly-enlarged tree. Splitting recursively creates an increasingly fine partition of R Figure I (4.3) shows example with 23 leaves.
A central issue is the criterion for selecting the next rule add. BL use the proxy objective of minimising the local variance of reward predictions, which is exactly the CART algorithm (Breiman ctt al. 2017). While very fast, this criterion is only loosely aligned with fidelity the preferences in D. We propose the more direct criterion of greedily maximising the immediate reduction in lo-1
(5) In Section 6 we show that switching to this bespoke criterion consistently improves performance. Its implementation involves major reformulation of the tree growth algorithm; we provide vectorised, just-in-ting instin-time compiled code for this in the Supplementary Material. Recursive splitting stops when no reduction in lo can be achieved by any single split, or tree size limit L = Lmax is reached.
SeemaxisIgLyg ISSSF, cEC, (D, RT) fo.1 (D, RT+((ra)]]
4.4 TREE PRUNING
Growth is followed by pruning sweep which reduces the size of the tree by rule removal. Such reduction is beneficial for both performance (Tien et al.] (2022) find that limiting model capacity lowers the risk of causal confusion in preference-baset reward learning) and human comprehension (in the language of Jenner& & Gleave (2022), is a form of " "prucessing for interpretabillty)) Given tree T. one pruning operation has the effect of merging two leaves into one by removing the rule at the common parent node. Let denote the sequence of nested subtrees induced by pruning the tree recursively back its root, at each step removing the rule that minimises the next subtree's fo.1- We select the T E T that minimises fa-1. additionally regularised by a term that encourages small trees: argmin (D, RT) aL.) where 0. Note that even with a = 0 pruning may still yield a reduced tree, as unlike in traditional decision tree induction, the effect of individual rules on fo.1 depends on the order in which they are added or removed. In the example in Figure F4. pruning yields final tree with 3 leaves, for which illustrative leaf-level reward predictions are shown.
ONLINE LEARNING SETUP
5.1 ITERATED POLICY AND REWARD LEARNING
Sections 3 and 4 do not discuss the origins of the trajectories E, or how reward learning relates to the downstream objective of learning a policy for the underlying task. Following most recent work since Christiano et al. (0117). we resolve both questions with an online bootstrapped approach. Assuming an episodic MDP, the ith episode of policy learning produces new trajectory ş to add to E. We immediately connect ç to the preference graph by asking the human to compare it to Kbath random trajectories from the existing set (while Sadigh et al. (2017) and others have proposed active querying schemes, that is not our focus here, and this simple strategy performs satisfactorily). We
then update the reward tree the full preference graph via the four stages given in Section 4 We
4

---

Under review as a conference paper at ICLR 2023
find that BL's original method of starting growth from the current state of the tree causes lock-in to poor initial solutions, so instead re- re-grow from scratch on each update. The rule structure nonetheless tends to stabilise, as the enlarging preference graph becomes increasingly similar for later updates. For the (i +i+11 )th episode, the policy learning agent then attempts to optimise for the newly-updated reward. By iterating this process up to total preference budget mmax and/or episode budget Nmax. we hope to converge to both a reward tree that reflects the human's 's preferences, and an agent policy that satisfies those preferences. Appendix A.4 contains pseudocode for the online algorithm.
5.2 INTEGRATION WITH MODEL-BASED RL
Reward learning methods are generally agnostic to the structure of the policy learning agent: this modularity is hailed as an advantage over other human-agent teaching paradigms (Leike et al. 2018). In line with most recent works, BL use model-free RL agent, specifically soft actor-critic (SAC)
HHarnooa et al. 2018). However, other works (Reddy et al., 2020, Rahtz et al. 2022) use model- based RL (MBRL) agents that leverage learnt dynamics models and planning. MBRL attractive in the reward learning context because it disentangles the predictive and normative aspects of decision- making. Since (assuming no changes to the environment) dynamics remain stationary during online reward learning, the amount of re-learning required is reduced and along with it, the risk of pitfalls such as manipulation (Armstrong et al. 2020) and premature convergence. Additionally, MBRL can be very data-efficient; we find that switching from SAC to model-based algorithm called PETS (Chua et al. (2018) reduces environment interaction during reward learning by orders of magnitude, and cuts wall-clock runtime (see Appendix B). PETS selects actions by decision-time planning through a learnt dynamics model D' x A + (SS) up to a horizon H. In state s, planning searches for sequence of H future actions that maximise return under the current reward model:
DD Σ H-1 ( 1)EAA
argmax
h= 7 >>=
RT(0(sk, an s where so 8, Sh+1
D' an). (6)
The first action ag is executed, and then the agent re-plans on the next timestep. In practice, D' is an ensemble of probabilistic NNs, the expectation over D' is replaced by Monte Carlo estimate, and the optimisation approximated by the iterative cross-entropy method.
EXPERIMENTS AND RESULTS
In this section, we combine quantitative and qualitative evaluations to assess the performance of reward tree learning, specifically in comparison to the standard approach of using NNs. We also illustrate how the intrinsic interpretability of reward trees allows to analyse what they have learnt. aireraft handling domain (Figure[)) in which agent must manoeuvre
Our experiments focus an aircraft (the ego jet, EJ) in a desired manner relative to a second reference jel (RJ) whose motion, if any, is part of the environment dynamics. We consider three tasks: Follow (turn fly in formation with RJ on linear path); Chase (maintain distance/line of sight to RJ as it turns randomly); and Land (approach a runway using RJ as a reference). These tasks are useful test cases for reward learning, as each has large space of plausible reward functions, which may reflect the divergent priorities and stylistic preferences of aeronautical experts. Such expert knowledge is often tacit and difficult to codify (Sternberg & Horvath 1999), motivating a learning- based approach. Appendix C contains a broader justification of this experimental domain alongside implementation details.
State: pose information for both E] and RJ
Fellow
Chase
Represents target towdldown pose
RJ
Land
Reference
RJ Proceeds de
Task tumn
Flies
Tarkc stay close to RJ with line of sight. keen safe altitude
Taok execute a
Ego Jet
Actions pitch. roll, yaw, chrust Jemands for Ej
along lincat flight path
EJ
and mntcl speed
flight path
onto sume
subject to random
control
Ej
stable approach
path, ending
targes prss
impac
Rj
Figure 2:
State-action space of aircraft handling domain, and diagrams of Follow/Chasel/ hha///Chhae//aaada tasks.
In place of costly numan-in-the-loon evaluation, our experiments use synthetic oracle preferences with respect to nominal reward functions of varying complexity, which are given in Appendix C.3 This approach is popular (Griffith et al. 2013 Christiano et al. 2017: Reddy et al. 2020 Lindner et al. 2021) as it enables scalable systematic comparison, with the ability to quantify performance (and in our case, appraise learnt trees) in terms of reconstruction of known ground truth. However, emulating human with an oracle that responds with perfect rationality is unrealistic (Lee et al.) 2021a). For this reason, Section exammines the performance impacts of noisy and myopic oracles, and restricted data budget. Experimental details and hyperparameters are given in Appendix D
5

---

Under review as a conference paper at ICLR 2023
6.1 QUANTITATIVE PERFORMANCE
We evaluate online reward learning with PETS using trees with the fo-1 split criterion, baselined against BL's original variance criterion, well as the de facto standard of NN reward learning (see Appendix D.4 for details). We use max 1000 preferences over Nmxx 200 online trajectories, and run repeats. As a headline statistic, we report the oracle regret ratio (ORR): the median drop in oracle return of PETS agents deployed using each trained reward model compared with directly using the oracle reward, as a fraction of the drop to random policy (lower better). Below are the median (top) and minimum (bottom) ORR values across the 10 repeats for each task- pairing:
Follow
Chase
Land
NN
Tre)(O-1)
Tree (var)
NN
Tree (0-1)
Tree (var)
NN
Tree (0-1)
Tree (var)
.000
.120
288
.000
.040
.126
.014
.050
.062
-.01
.057
.158
-0051
-011
.065
.030
.011
.010
We observe that: NN reward learning is strong on all tasks; 2) switching to reward tree induces small but variable performance hit; 3) fo-1 splitting outperforms the variance-based method; and 4) both NN and tree models sometimes exceed the direct use of the oracle (negative ORR). This has been observed before (Cao et al., (2021) and may be due to improved shaping in the learnt reward.
TigureBexpante these results with more metrics, revealing subtler trends not captured by headline ORR values. Metrics are plotted as time series over the 200 learning episodes (sliding-winde dians and interquartile ranges across repeats). In the left column (a), the ORR of online trajectories shows how agent performance converges. For Follow, there is gap between the models, with fo.1 splitting clearly aiding performance but still lagging behind the NNs. The learning curves for Chase and Land are more homogeneous, and the NNs reach only slightly lower asymptotes. For the reward tree models, (b) shows how the number of leaves changes over time. The variance-based based trees tend to grow rapidly initially before stabilising or shrinking, while the f0-1 trees enlarge more conservatively, suggesting this method is less liable to overfit to small preference graphs. Trees of a readily-interpretabb interpretable size (= 20 leaves) are produced for all tasks; it is possible that performance could be improved by independently tuning the size regulariser a per task. (c) shows fa.1 over time, which tends to increase as the growing preference graph presents a harder reconstruction problem, though the shape all curves suggests convergence (note that random prediction gives 4..1 0.5). For Follow and Land, the trees that directly split on fo-1 actually perform better than the NNs; they more accurately predict the direction of preferences not translate into lower ORR indicates that the problems of learning good policy and replicating the preference dataset are not identical, point made by Lindner et al. (202T). In the final two columns, invariant comparison of the mod-
me-
the graph. The fact that this does
follow Gleave et al. (2021) in performing els by correlating their outputs with the oracle reward functions on common evaluation datasets (see Appendix D.5 for dataset creation). We compute online correlations with the oracles in terms of both transition-level rewards (d) and the ordinal ranking of trajectories by return (e), the latter via the Kendall (938) 7 coefficient. The curves subtly differ, indicating that it is possible to reconstruct trajectory rankings (and by extension, any pairwise preferences) to given accuracy with varying fidelity at the individual reward level. However, the common overall trend that fa.r- trees outperform caraance-based ones, with NNs sometimes improving again by smaller margin, and sometimes bringing no added benefit. Moving top-to-bottom down the tasks, the gap between models reduces from both sides; NN performance worsens while variance-based trees improve.
unbiased, policy-invarian
Online ORR
Tree size
loss
Reward correlation
Rank conrelation
NN
04 015
Trax (1-1) (var)
Follow
0.5
0/55
0.50
Chase
0.5
0.
0.6
0.06
10.2
01550
0.
025
Land
0.2
05
95
0.6
0225
025
Figure 3; Time series of metrics for online NN- and tree-based reward learning on all three tasks.

---

Under review as a conference paper at ICLR 2023
A potentially important factor in these experiments is that the oracle reward for Follow is simple linear function, while the other two contain progressively more terms and discontinuities (see Ap- pendix C33 A trend suggested by these results is thus that the performance gap between NNs and reward trees (on both ORR and correlation metrics) reduces as the ground truth reward becomes more complex and nonlinear. Further experiments would be needed to test this hypothesis.
6.2 VISUAL TRAJECTORY INSPECTION
While useful for benchmarking, quantitative metries provide little insight into the structure of the learnt solutions. They would also mostly be undefined when learning from humans since the ground truth reward is unknown. We therefore complement them with visual analysis of induced agent behaviour. Figure | plots 500 trajectories of PETS agents using the best repeat by ORR for each task-model combination, across range of features as well as time (see Appendix C.2 for feature definitions). Dashed curves show the trajectory with the highest predicted return according to each model. We also show trajectories for PETS agents with direct oracle access, and for random policies. The high-level trend is that all models are far closer to the oracle than random, with few examples of obviously incorrect behaviour (highlighted in red, due to colouring by ORR). While the NNs induce trajectories that are almost indistinguishable from the oracle, the -based reward trees lag not far behind. The variance-based frees produce more anomalies. Successes of the Eo-1 trees include the execution of Follow with a single banked turn before straightening up. as shown by the up error time series (a). Indeed, the trajectories for this model almost imperceptibly different from those of the NN, a result which is belied by the mediocre ORR of 0.158. This underlines the importance of joint aaantiaativ-qualitativee evaluation. For Chase (b), the f--1 tree has learnt keep the agent narrowly above the altitude threshold alt < 50, below which the oracle reward is strongly negative (see Appendix C.3). The threshold is violated in only eight of 500 trajectories (1.6%). For Land, the fo-1 tree replicates the oracle in producing gradual reduction in alt (c) while usually keeping pit t ch close to (d), although the distribution of roll values is less narrow.
In contrast, the agent using the variance-based tree for Follow sometimes fails to reach the target position (e; red trajectories), and also does not reliably straighten up to reduce up error (f). For Chase, the altitude threshold does not appear to have been learnt precisely, and lower-altitude trajectories often fail close the distance to RJ (g and h; red trajectories). For Land, the variance- based tree gives a later and less smooth descent (i), and less consistent pitch control (j), than the NN or -b.sed tree, although all models produce a somewhat higher altitude profile than the oracle. Random
Oracle
NN
Tree (0-1)
Tree (var)
eloing geed St
Righes
Pollow
353
T Timeoop
540
150
Chase
ab
50
Ias stnorr
Timeaup 20
30
155
Land
32
2,40
dis her
200
moll
Figure 4: Agent trajectories using the best models by ORR, with oracle and random for comparison.
6.3 SENSITIVITY ANALYSTS
It is important to consider how learning performance degrades with reduced or corrupted data. In Figure we evaluate the effect of varying the number of preferences Kmax (with fixed Nmux 200) and trajectories Nmax (with fixed Kmux = 1000) on reward learning with NNs and fo--spliiting trees. Following Lee et al. (2021a). we also create more human-likk preference data via two modes of oracle irrationality: preference noise (by using nonzero Boltzmann temperature 33 to give a desired error rate on the coverage datasets) and a myopic recency bias (by exponentially discounting earlier timesteps when evaluating trajectory returns). We run five repeats for all cases, and report the medians and interquartile ranges of ORR (lower is better) and rank correlation (higher is better).
Both NN and tree models exhibit good robustness with respect to all four parameters. Although NNs remain superior in most cases, the gap varies, and is often reduced compared to the base
7

---

Under review as a conference paper at ICLR 2023
NN
Follow
Chase
Land
Averge ORR
NN-Tire Gap
Teee (1-1
ORR
Rank correlation
ORR
Rank correlation
ORR
Rank correlation
Bank correlation
Preference
4000
2000
1000
500
250
Trck
Tec:
NNN
Trjtecory
buslge
200
100
50
probubility
(Oraee
0.45
Orade
myooiia
0.98
0.95
0.8
025
Figure 5: Comparative sensitivity analysis of reward learning with NNs and trees.
cases (bold labels). The budget sensitivity is low, with little improvement for Kmmx 1000 and Nmax > 200, and no major drop even with 25% of the data as the base case. For all tasks, the oracle error probability can increase around 20% before significant drops in performance are observed. This is a promising indicator of the transferability of reward tree learning to imperfect human data. Another general observation is that the trends for trees are somewhat smoother than for NNs, with fewer sharp jumps and fewer instances of very high spread across the five repeats.
In the right column (a), we summarise these results by taking the difference between the NN and tree metries, and averaging across the three tasks. In all cases aside from rank correlation with 3 > O, the NN-tree gap tends to become more favourable to the tree models as the varied parameter becomes more challenging
This sensitivity analysis thus indicates that reward trees are at least as robust to difficult learning scenarios as NNs, and may even be slightly more so.
6.4 TREE STRUCTURE ANALYSIS
Thus far we have shown that reward learning with la -based trees can be competitive with NNs. but not quite as performant overall. We now turn to a concrete advantage which may tip practical tradeoffs in its favour: the ability to interpret the learnt model, and analyse how its structure arises from the underlying preference graph. In this section we favour depth over breadth, so focus on the single best tree by ORR on the Chase task. The analysis in Figure o divided into sections (a d): (a) This reward tree has 17 leaves. The oracle reward, printed below, uses four features, all of which are used in the tree in ways that are broadly aligned (e.g. lower los error leads to leaves with higher reward). The model has learnt the crucial threshold alt 50, correctly assigning low reward when it is crossed. This explains why we observe rare violations of the altitude threshold in Figure However, has not learnt the ideal distance RJ, dist 20, with 43.3 being the lowest value used in rule. This could be because the underlying preference graph lacks sufficient preferences to make this distinction; adopting an active querying scheme may help to discover such subtleties efficiently. Other features besides those used by the oracle are present in the tree, indicat- ing some causal confusion (Tien et al., 2022). This may not necessarily harm agent performance, as it could provide beneficial shaping (c.g. penalising positive closing speed, which indicates in- creasing distance to RJ). That may indeed be the case for this model since ORR is actually negative.
(b) We plot the tree's predicted reward against the oracle reward for all timesteps in the online trajectories (correlation 0.903). The predictions for each leaf lie along horizontal line. Most leaves, including 1 and 2, are well-aligned on this data because their oracle reward distributions are tightly concentrated around low/high averages respectively (note that the absolute scale is irrelevant here). Leaf 16 has a wider oracle reward distribution, with several negative outliers. An optimal tree would likely split this leaf further, perhaps using the alt < 50 threshold. The one anomaly leaf 13, which contains just single timestep from 5?7 This trajectory is the eighth best in the dataset by oracle return, but this leaf assigns that credit to state that seemingly does not merit it, as the distance to RJ is so high (dist 73). This may be an example of suboptimal reward learning, but the fact that its origin can be pinpointed precisely is a testament to the value of interpretability.
(c) We leverage the tree structure to produce a human-readable explanation of reward predictions for single trajectory, which may be of value to an end user (e.g. pilot). We consider 119 rare
8

---

Under review as a conference paper at ICLR 2023
4.45 D1 Perdicted sowand
Ieternodiecl reaard deaf tieettep- weigltd aving
x 3.35
(1) 176
[] 0937
-500
Onsil roward
-200
-100
Dlowssh
Croik reard
fuuiid))
wids
(EG) e 1.7 tisengg
whle alx in D2
(15)
lee wrrer 2. a chruee 2.M
(1o)
abe rall 774
Cracle sewards
-(ale(ctst 20) 10 los srrer ats roll
100
07
"key wiih vi Do casarope)
dalta sat attur -2:-0
(towutep))
1 Timestep
20
350
16
Total 30 simoleps
30
HC5-241
dar
43.3
75
95.4
150
Trajerpeyy recuEo
Figure 6: Analysis of reward tree learnt for the Chase task.
case that violates the altitude threshold. The time series of reward shows that the 20 timesteps are spent in leaves 16, 15, 11 and 7. Rescaled oracle rewards are overlaid in teal, and show that the model's 's predictions are well-aligned aligned. To the right, we translate this visualisation into textual form, similar to a nested program. Read top-to-bottom, the text indicates which rules of the tree are active at each timestep, and the effect this has on predicted reward. This trajectory starts fairly positively, with reward gradually increasing over the first 16 timesteps as di st is reduced to between 43.3 and 73, but then falls dramatically when the alt < 50 threshold is crossed. We are unaware of any method that could extract such compact explanation of sequential predictions from an NN.
(d) We isolate a subtree, starting at the root node, that splits only on st and alt. We give a spatial representation of the subtree, and how it is populated by the 200 online trajectories, using a 2D partition plot analogous to those in Figure Zooming into leaf 1, which covers cases where the altitude threshold is violated, we see that it contains total of 30 timesteps across four trajectories.
By Equation 4 the low reward for this leaf results from a weighted average of the return estimates for these four trajectories, which in turn (by Equation 3 are derived from the preference graph. We can use this inverse reasoning to ask why this leaf has much lower reward than its sibling (leaf 2 of the subtree). A proximal explanation comes by filtering the graph for preferences that specifically compare trajectories that visit those two leaves. 49 such preferences exist, and in all cases, the oracle prefers the trajectory that does not visit leaf 1. Some of these preferences may be more practically salient than others. For example, we might highlight trajectories that feature more than once (e.g. 22 is preferred to both 2 and <4 or cases where trajectories with low overall return estimates are nonetheless preferred to those in leaf 1 (e.g. 5²3 > e²l and 500 > 5t7). We believe that much more could be done to extend this framework for traceable explanation of preference based reward.
CONCLUSION AND FUTURE WORK
Reward learning with trees provides a promising alternative to black-box NNs, and could enable more trustworthy and verifiable agent alignment. Through oracle experiments on high-dimensional tasks, we show that reward trees with around 20 leaves can achieve quantitative and qualitative per- formance close to that of NNs, with a more direct split criterion bringing consistent improvements. We find evidence that the NN-tree gap reduces as the ground truth reward becomes more nonlinear, and remains stable or reduces further in the presence of limited or corrupted data. While practical applications may accept some loss in performance for a gain in interpretability, further algorithmic improvements should be sought, including to move beyond locally-greedy split criteria. However, our immediate aim is to develop an end-to-end framework for explainable model-basee agents with preference-basee reward trees (roughly: planning can be reframed as comparing alternative paths through the discrete leaves of the tree). Having established this framework, we then intend to evalu- ate reward learning and explanation with real human preferences in the aircraft handling domain.
9

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Reward learning from feedback is a technique for improving the alignment of learning agents with human preferences, by replacing the rigidity of explicit reward design with a dynamic interaction with a human-in-the-lowe
As such, its successful use can benefit the performance, reliability and safety of these learning systems, which has the potential to deliver immensely positive ethical value. The contribution of reward trees is to render the process of reward learning more human- interpretable, and thus easier to explain, debug and verify. We believe this can deliver a further reduction in ethical risk through the identification and mitigation of unforeseen consequences.
Preference-basee reward tree learning has many diverse applications. The aircraft handling domain used in our evaluation was selected to provide good balance of technical complexity, task diversity, industrial relevance, intuitiveness for our intended readership, and transferability to other domains such as land and sea transportation. In addition to this wide range of related use cases, well as civilian uses of aviation itself (e.g. aerobatics), we acknowledge that the ability to interactively learn aircraft control policies may have applications in the defence sector. The three concrete tasks of Follow, Chase and Land are largely application-neutral with no implication of harm, and are concerned solely with the safe and human-like control of aircraft environments with other aircraft. Any general learning technique such as ours is, however, fundamentally dual-use, and transparency about this fact seems us the best mitigation of the ethical risk. It vital that anyone intending to use or develop our method continues to do so in an ethically responsible manner.
REFERENCES
Dan Amir and Ofra Amir. Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, 2018.
Stuart Armstrong, Jan Leike, Laurent Orseau, and Shane Legg. Pitfalls of learning reward function online. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelli- gence, IJCAI-20, 2020.
Andrea Bajesy, Dylan P Losey, Marcia K O'Malley, and Anca D Dragan. Learning robot objectives from physical human interaction. In Conference on Robot Learning, pp. 217-226. PMLR, 2017.
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in Neural Information Processing Systems, 31, 2018.
Tom Bewley and Freddy Lecue. Interpretable preference-based reinforcement learning with tree- structured reward functions. In Proceedings of the s1 International Conference on Autonomous Agents and Multiagent Systems, pp. 118-126, 2022.
Tom Bewley, Jonathan Lawry, and Arthur Richards. Summarising and comparing agent dynamics with contrastive spatiotemporal abstraction. In IJCAI/ECAI Workshop on Explainable Artificial Intelligence, 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-45,, 1952.
Leo Breiman, Jerome H Friedman, Richard A Olshen, and Charles J Stone. Classification and regression Irees. Routledge, 2017.
Zehong Cao, KaiChiu Wong, and Chin-Teng Lin. Weak human preference supervision for deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 32(12): 5369-5378, 2021.
David Chapman and Leslie Pack Kaelbling, Input generalization in delayed reinforcement learning: An algorithm and performance comparisons. In Ijcai, volume 91, pp. 726-731, 1991.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
10

---

Under review as a conference paper at ICLR 2023
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
Luis C Cobo, Charles L Isbell Jr, and Andrea Thomaz. Automatic task decomposition and state abstraction from demonstration. Georgia Institute of Technology. 2012.
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, and Ann Nowé. Distilling deep reinforce-
ment learning policies in soft decision trees. In IJCAI/ECA/ Workshop on Explainable Artificial Intelligence, 2019.
Giang Dao, Indrajeet Mishra, and Minwoo Lee. Deep reinforcement learning monitor for snapshot recording. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 591-598. IEEE, 2018.
Jan De Leeuw and Patrick Mair. Multidimensional scaling using majorization: Smacof in Journal of statistical software, 31:1-30, 2009.
Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, and Adish Singla. Explicable reward design for reinforcement learning agents. Advances Neural Information Processing Systems, 34:201 201311, 2021.
Saão Džeroski, Lue De Raedt, and Hendrik Blockeel. Relational reinforcement learning. In Inter- national Conference Inductive Logic Programming, pp. 11-22. Springer, 1998.
Brochu Eric, Nando Freitas, and Abhijeet Ghosh. Active Preference Learning with Discrete Choice Data. Advances in Neural Information Processing Systems, 20, 2007.
Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang. Jianye Hao, and Wulong Liu. A survey interpretable reinforcement learning. arXiv preprint 1rXiv:21/2.13111 2021.
Adam Gleave, Michael D Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differ- ences in reward functions. In International Conference on Learning Representations, 2021.
Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. Advances in neural information processing systems, 26, 2013.
Harold Gulliksen. least squares solution for paired comparisons with incomplete data. Psychome- trika, 21(2):125-134, 1956.
Yuan Guo, Peng Tian, Jayashree Calpathy-Cramen Susan Ostmo, Peter Campbell, Michael Chiang, Deniz Erdogmus, Jennifer G Dy. and Stratis loannidis. Experimental design under the bradley-terry model. IJCAL, pp. 2198 2204, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Alexandre Heuillet, Fabien Couthouis, and Natalia Rodrigur Explainability in deep rein- 2021.
forcement learning. Knowledge-Based Systems,
Sandy Huang, Kush Bhatia, Pieter Abbeel, and Anca D Dragan. Establishing appropriate trust via critical states. In 2018 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 9929-3936. IEEE, 2018
Tobias Huber, Dominik Schiller, and Elisabeth André. Enhancing explainability of deep reinforce- ment learning through selective layer-wise relevance propagation. In Joint German/Austrian Con- ference Artificial Intelligence (Känstliche Intelligenz), pp. 188-202. Springer, 2019.
Erik Jenner and Adam Gleave. Preprocessing reward functions for interpretability. arXiv preprint arXiv:2203. 13553, 2022.
11

---

Under review as a conference paper at ICLR 2023
Wei-Cheng Jiang, Kao-Shing Hwang, and Jin-Ling Lin. An experience replay method based tree structure for reinforcement learning. IEEE Transactions on Emerging Topics in Computing, 9(2): 972-982, 2019.
Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Brwig, and Finale Doshi Velez. Explainable rein- forcement learning via reward decomposition. In I/CA//ECAI Workshop on Explainable Artificial Intelligence, 2019.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
MG Kendall. Rank Correlation Methods; Griffin, C., Ed, 1975.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 6980, 2014.
W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference on development and learning, pp. 292-297. IEEE, 2008.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference- based reinforcement learning. Advances in Neural Information Processing Systems, 35, 2021a.
Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pp. 6152-6163, PMLR, 2021b.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. arXiv preprint 1 2018.
David Lindner, Matteo Turchetta, Sebastian Tschiatschek, Kamil Ciosek, and Andreas Krause. In- formation directed reward learning for reinforcement learning. Advances in Neural Information Processing Systems, 34:3850-3880 3862, 2021.
Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforcement learning with linear model u-trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 414 429. Springer, 2018.
Eric J Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv preprint arXiv:2012.05862. 2020.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, vol- ume 1, pp. 2, 2000.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects reward misspecification: Mapping and mitigating misaligned models. In International Conference Learning Representations, 2022.
Erika Puiutta and Eric Veith. Explainable reinforcement learning: A survey. In International cross- domain conference for machine learning and knowledge extraction, pp. 77-95. Springer, 2020.
Larry Pyeatt. Reinforcement learning with decision trees. 21 st IASTED International Multi- Conference Applied Informatics, pp. 26-31, 2003.
Matthew Rahtz, Vikrant Varma, Ramana Kumar, Zachary Kenton, Shane Legg, and Jan Leike. Safe deep rl in 3d environments using human feedback. arXiv preprint arXiv:2201.08100 2022.
Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. Learning human objec- tives by evaluating hypothetical behavior. International Conference on Machine Learning, pp- 8029. PMLR, 2020.
Aaron M Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso. Conservative q- improvement: Reinforcement learning for an interpretable decision-tree policy. arXiv preprint 190 1800, 2019.
12

---

Under review as a conference paper at ICLR 2023
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-22) 215, 2019.
Jacob Russell and Eugene Santos. Explaining reward functions in Markov decision processes. In The Thiriy-Second International Flairs Conference, 2019.
Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learn- ing of reward functions. In Proceedings of Robotics: Science and Systems (RSS), 2017.
Lindsay Sanneman and Julie A Shah. An empirical study of reward explanations with human-robot interaction applications. IEEE Robotics and Automation Letlers, 2022.
Hassam Ullah Sheikh, Shauharda Khadka, Santiago Miret, Somdeb Majumdar, and Mariano Phielipp. Learning intrinsic symbolic rewards in reinforcement learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2022.
Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, and Sung-Hyun Son. Optimiza- tion methods for interpretable differentiable decision trees applied to reinforcement learning. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pp. 1865. PMLR, 26-28 28 Aug 2020.
Robert J Sternberg and Joseph A Horvath. Tacit knowledge in professional practice: Researcher and practitioner perspectives. Psychology Press, 1999.
Alberto Suárez and James F Lutsko. Globally optimal fuzzy decision trees for classification and regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21912111297-1111, 1999.
Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan, and Matthew Gombolay. Specifying and interpreting reinforcement learning policies through simulatable machine learning. arXiv preprint arXiv:2101.07140, 2021.
Jeremy Tien, Jerry Zhi- hang He, Zackory Erickson, Anca Dragan, and Daniel Brown. A study of causal confusion in preference-baie reward learning. arXiv preprint arXiv:2204.0660) 2022.
Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerinex. Contrastive explanations for reinforcement learning in terms expected consequences. In IJCAI/ECAI Work- shop on Explainable Artificial Intelligence, 2018.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pp. 5045-5054 PMLR, 2018.
Nils Wilde, Alexandru Blidaru, Stephen L Smith, and Dana Kulić, Improving user specifications for robot behavior through active preference learning: Framework and evaluation. The International Journal of Robotics Research, (9(6):651-661) 2020.
Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann. Model-free preference-based rein- forcement learning. In Thirtieth AAAI Conference Artificial Intelligence, 2016.
Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In International conference on machine learning, pp. 1899-1908, PMLR, 2016.
Guangxiang Zhu, Zhiao Huang, and Chongjie Zhang. Object-oriented dynamics predictor. Advances in Neural Information Processing Systems, 31, 2018.
13

---

Under review as a conference paper at ICLR 2023
A
METHODOLOGICAL DETAILS
A.1 LIST OF CHANGES TO BEWLEY & LECUE'S ORIGINAL METHOD
Change from Thurstone to Bradley-Terry preference model and least squares matrix method to gradient descent on ENLL;
Motivation:
Consistency with prior work.
Performance implications: Thurstone Bradley-Terry minimally affects results. Ma- trix method - gradient descent eliminates a bias caused by preference clipping.
Computational implications: Thurstone - Bradley-Terry improves computational ef- ficiency (bypasses inverse normal CDF computation). Matrix method + gradient descent tends to slightly increase runtime, but this depends on preference dataset size.
Add scale and sign constraints to return estimates;
Motivation implications: Scaling results in leaf-level reward predictions having consistent scale with standard deviation close to 1. This is intended to aid human readability but has no effect on performance. For sign constraints, see Appendix A.2
Change from variance-based to fbr-ba split criterion during tree growth;
Hypothesised performance gain; split criterion is more directly aligned with the objective of preference reconstruction.
Motivation:
Performance implications: Significantly improves preference construction and quan- titative/qualitative agent performance on evaluation tasks; see Section
- Computational implications: More costly as variance computation has an extremely efficient iterative implementation. However, have developed optimised code for the new splitting method using just-in-time compilation; see Supplementary Material.
Deploy online with a model-based (PETS) RL agent instead model-free (SAC);
Motivation / implications: See Section 5.2 and Appendix
Always latest online trajectory in all pairs during each preference batch;
Motivation: Acts simple form of active sampling correct reward overestimation; model-based planning liable to exploit any behaviours with inappropriately high current reward, which can then be immediately corrected by negative preference. -Performance implications: Performs similarly BL's active sampling method, which up-weights trajectories in E with high predicted return.
Computational implications: Less expensive than BL's method; no need to recompute return predictions for all trajectories in E on each batch.
* Note: New method would be less effective with a model-free agent where policy updates are gradual; can rely less on agent immediately exploiting current reward.
Regrow tree from scratch on each update.
Motivation: Prevents rule structure from prematurely converging to local minima. Performance implications: Early experiments indicated that premature convergence problem mitigated by this change, resulting in more sustained improvements in reward fidelity and agent performance.
Computationall implications: Since splits are evaluated and made per update step, computation time is increased. However, when typical post-pruning tree size Ly (<o 20 our experiments) small compared with Lmax (= 100), this increase fractional, and contributes only a few percentage points to overall runtime.
14

---

Under review as a conference paper at ICLR 2023
A.2 SIGN CONSTRAINT FOR RETURN ESTIMATES
Applying a sign constraint to the trajectory-level return estimates means that rewards output by reward tree (via Equation 4 are also all either positive negative. This has effect on any measure of preference reconstruction since preferences are invariant to affine transformations of an underlying utility function. However, we find brings distinct benefits:
Enabling the prevention of pervense incentives for agents terminate elongate episodes in tasks with termination conditions (negative rewards on non-terminal transitions incen- tivise termination, while positive rewards incentivise elongation).
Simplifying the manual interpretation of tradeoffs between rewards from different leaves of a tree (understanding the relative impacts of "more of negative reward" and "less of positive reward" requires the awkward mental juggling of negatives).
For the task with termination condition in this paper (Land), 0 constraint) to disincentivise episode elongation, because termination is generally indicative of success. For the two fixed-length tasks (Follow and Chase) default to using positive rewards (min = 0 constraint). Although this arbitrary, our own experience is that positive rewards make for somewhat more intuitive interpretation of the structure, and its effect agent actions. We stress that this is purely anecdotal; the relative human interpretability of positive, negative and mixed-sign rewards would be a worthy subject for deeper empirical investigation.
negative rewards (max
A.3 ANNOTATED VERSION OF FIGURE
pairwise preferences trajectories can be visuliied as directed graph, where nodes are trajectories and edges are preferences.
Each trajectory sequence vectors feature space. In turn, each vector represents the (state, action, state) transition that particular timestep
Eacc edge points to preferred traeesorr
Feature vector for timertep t: di 12-11
Rewari cafeulation for highigghed leaf:
Preferred Ç but not to ς², intermediate estimated return
- 22 mapping
(4 3 2
2)
4.1
Thme unfasurabble prefornirry lowest extimated
4.2
- ga
Before doing anything in the
Given an existing tree structure and trjectorics with estimated define the reward for each keaf a timestep-weighted average over the trajectories that
Nature space, we listt the standard Bradley- Terry preference model estimate an overal return value for each trajectory in the dataset based preferences.
leaf.
Thn favonrable proferences; highest estimated peturn
Nai ascumption: all timestepy trajectory contribute equally reture (ve. uniforn colour)
Pruning merges lnaves back together, while retaining the moxt imporiant splits for preference predistion
Grouth rends to reparate high- and low- trajecturies into different leaves
4.4
Growing by adding recunively splits feature space into an increasing number of leaves. in improved growth method, we the d-1 loss as the criterion for selecting rules to add.
After growing maximum pruning stage recursively removes rules to minimise the 0-1 loss, with additional regularisation penalise large This yields a reduced tree for use as reward function.
"Pure" leaf containing a ringle majectory
r2 = 0.5
r1 = 0.1
values are illustrative
Figure AI: Annotated version Figure
15

---

Under review as a conference paper at ICLR 2023
A.4 PSEUDOCODE FOR ONLINE ALGORITHM
Algorithm Online preference- busse reward tree learning
Inputs: Possibly pre-trained dynamics model D', feature function é, trajectory budget Nmax. preference budget mmxx, tree size limit Lmax. tree size regularisation a
1: Initialise empty preference graph E <- 0, C 0
2: Initialise one-leaf tree T with f- [0]
3: for (L......Naar. do
4: Initialise time & <- 0 and environment state só
5: while episode not yet terminated do
Model-based trajectory generation (Sec 5.2
6:
Compute action of using PETS algorithm with D' and rewards via Equation 4
7:
Send of to environment and get next state si+1 Update D' on recent transitions
8:
May not required; see Appendix D.3
9:
x < o(s;, o/. s/41)
+-1+1
11:
end while
12:
<' (xi,..,,;y.. x.
13:
Kbatch f- min (mmxx (C1) +1-
14:
for {l,..., K do
Preference batch collection (Sec)
15:
Sample 5 from E uniformly without replacement
16:
Query human for preference Ç > gd or 5d >
c+ {(, if > 5² end for -
(G.1))
otherwise
18:
19:
E
20:
if [] then
21:
Compute g via Equation Initialise lne-leaf tree T < midpoints between per-feature unique values in while LT T.mux do for E (1...., [ do for f (1, F} do for c E Cj do Compute la11 reduction for + [Ifc] via Equation end for end for end for max1 reduction) < then break
> Trajectory- return estimation (Sec 4.1
22:
23:
24:
Tree growth (Section
25:
26:
27:
29:
30:
31:
32:
33:
Stop tree growth early
34:
end
35:
1, f.f. argmax(fos reduction)
36:
T [
37:
end while
38:
T T=0 ()
while LT >
Tree pruning (Section 4.4)
40:
for 1 E {1...., LT} do
Compute fo. reduction for T end for 1 argmax(44. reduction) T [1]
b T denotes pruning /th leaf
42:
44:
45:
Append to T
46:
end while
47:
T +- argmin (6o1 plus a-scaled tree size)
48:
end if
49: end for
16

---

Under review as a conference paper at ICLR 2023
B COMPARISON TO MODEL-FREE REINFORCEMENT LEARNING
One of the most consistently observed benefits of model-based RL is its sample efficiency, and this trend holds in our context. Running Algorithm unchanged except for the use of a soft actor-critic (SAC) agent for policy learning, we find that approximately two orders of magnitude more environ- ment interaction is required to achieve equivalent performance in terms of regret at convergence. In tur,, this increases wall-clock runtime by 10-20 times, thereby outweighing the higher per- per-timestt timestep computational cost of PETS over SAC. The caption of Figure A2 gives further details.
PETS (200 episodes) NN
SAC (200 episodes)
SAC episodes)
SAC (10k episodes)
SAC (50k episodes)
Online ORR
Tre (yor) (1-3)
Wall-clock time
(xiinines)
400
100
400
300
200
200
200
Ephods
Fpinod::
Epiaod: 1000
Epiarde 10000
Epiode 50003
Figure A2: Comparing the use of PETS (model-based) and SAC (model-free) agents on the Follow task (see Appendix C.3 for task details). The PETS results are taken directly from Figure 3 For SAC we retain the total preference budget of Kmax 1000, but for longer runs add episode trajectories to E and L at reduced frequency so that .Nmax = 200 (e.g. for 50000 episodes, only in 250 episodes networks with two 256-unit hidden layers each, learning rates of le and 1e-3 for the policy and value updates with the Adam optimiser, a discount factor of 7 = 0.99, and an interpolation factor of 0.99 for Polyak averaging of the target networks. Updates use mini-batches of 32 transitions sampled uniformly from rolling replay buffer of capacity 4
added to the graph; the rest are skipped). All SAC agents use policy and value
We find that initially running SAC for a total of 200 episodes, matching our PETS experiments, gives the model-free learning algorithm insufficient time achieve good performance terms of regret on the oracle reward function (a higher learning rate leads SAC to become unstable). We then progressively increase the length of SAC runs until regret performance matches the use of PETS, and find that this requires around 50000 episodes, increase of 250 times. It is noteworthy that the variance-based tree model seems to perform best in the short-runtime regime, but worst in the long-runtime regime. Time constraints prevented from investigating whether this holds in other tasks, but such an investigation would be worthwhile.
In terms of wall-clock time (on a single NVIDIA Tesla P100 GPU), running reward learning with SAC for 1000 episodes is roughly equivalent to 200 episodes using PETS (25-60 minutes, depending the reward model architecture). For 50000 episodes, this time increases 9 hours. This brings a very practical disadvantage if reward learning were done using human preferences instead of an oracle, that person would have dedicate more than a full working day the exercise, most of which would be spent waiting for several minutes between each successive preference batch.
Note: The PETS wall-clock times quoted here exclude the time to pre-train the dynamics models. Although this is not how MBRL sample complexity is lypically measured, we argue that is uppropriate for the reward learning context, where the key factor is the period for which a human would be required to be in in-the- loop. Regardless, pre-training in the aircraft handling domain takes around 30 minutes, which remains low compared with the 9 hours for high- tegh-eerformin SAC agents.
17

---

Under review as a conference paper at ICLR 2023
C AIRCRAFT HANDLING ENVIRONMENT
C.I MOTIVATION
Pilots of fast jet aircraft require exceptional handling abilities, acquired over years of advanced train- ing. There would be immense practical value in developing a method for distilling the knowledge and preferences of pilots and other domain experts into a software model that captures realistic handling behaviour. The scalability of such a model would make it useful for strategic planning exercises, training a range of operational roles, and development and testing of other software systems. However, as in many contexts where intuitive decision-making and rapid motor control are paramount, the preferences of experts (over the space of fast jet handling trajectories) are in large part tacit, and thus defy direct scrutiny verbal description. Put simply: experts know good handling when they see it, but cannot directly express why [1 to accurately elicit this knowledge for codification into automated system.
This makes it practically challenging
The methods presented in this paper form the basis of a possible solution to this dilemma. Given dataset of trajectories executed by artificial learning agent and labelled with pairwise expert preferences (which require only tacit knowledge to produce), we use statistical learning algorithms to construct an interpretable explanatory model of those preferences. The result is two distinct outputs that could form valuable components of future planning, training and development software:
1. A tree-structured reward function, which may be used for automated scoring of flight tra- jectories executed by human artificial pilots. We aim for this to produce evaluation that is consistent, unbiased and aligned with the judgement that the original expert would have made, alongside explanatory rationale that can be leveraged to justify, verify and improve handling behaviour.
2. A model-based RL agent capable of executing high-quality handling behaviour with respect to the reward function, for use in simulation.
should be noted that any realistic handling scenario would involve multiple experts somewhat- differing knowledge and expertise. A natural extension of our approach, which we see as valuable future work, is to learn individual reward functions for each expert, then leverage the intrinsic in- terpretability identify biases, inconsistencies and tradeoffs. This suggests third application of reward tree learning: providing a basis for evaluating and training the experts themselves.
C.2 IMPLEMENTATION
We consider a simple set-piece formulation of the aircraft handling problem, which the piloting agent is given short time window to manoeuvre their aircraft (the ego jet, EJ) in particular manner relative to a second reference jet (RJ). Special cases of this formulation create a wide variety of tasks for the pilot to solve. Options include:
RJ is a friendly aircraft which EJ should accompany in formation flight.
RJ is adversarial and EJ must outmanoeuvre to gain a tactical advantage.
Rather than being a distinct physical entity, RJ defines goal pose (position and attitude) for FJ to reach. The goal pose may be fixed or moving over time.
We developed this formulation to strike a balance between simplicity and generality; many realistic scenarios faced by fast jet pilot involve interaction with single other airborne entity. On a practical level, it provides scope for the definition of many alternative tasks given the same state and action spaces, and largely unchanged dynamics.
The state space contains the positions, attitudes, velocities and accelerations of both EJ and RJ (state dimensionality =37) and the action space consists of pitch, roll, yaw and thrust demands for
'This statement certainly underestimates the rich complexity human expertise; in reality, an expert's mental model is likely to be partly tacit and partly explicit. The general strategy of preference- based reward learning is to operate if the mental model were 100% tacit, and explore what can be achieved under such strong restriction. Real-world applications would likely henefit from combining this approach with some amount of hand-coded expert knowledge.
18

---

Under review as a conference paper at ICLR 2023
EJ only (action dimensionalityy = 4). The EJ dynamics function integrates these demands with simplified physics engine, including gravity and air resistance (we make no claim of realism here; the simulator is merely proof of concept). A new action is accepted every 25 steps of the physics engine, reducing an agent's decision frequency to approximately 1 Hzz. RJ dynamics, as well as the conditions of state initialisation and termination, vary between tasks (see Appendix
The final generic aspect of the implementation is the feature function 6. which maps the transition space S A S (total dimensionality = 37 + 37 78) into an F-dimensional space of task- relevant features. In consultation with engineers with experience of aerospace simulation and control algorithms, we devised the following set of F = 30 features that sufficiently expressive to capture the important information for all three of our target tasks, without being overly specialised to one or providing too much explicit guidance to the reward learning process. Apart from those containing "delta" or "rate", all features are computed over the successor state for each transition, 81+1-
dist closing speed alt alt error
Euclidean distance between EJ and RJ
Closing speed between EJ and RJ (negative moving closer)
Altitude of EJ
Difference altitude hetween and RJ (negative
is lower)
delta alt error dist hor delta dist hor
Change in a t. error between 8t and 8 Euclidean distance between EJ and RJ in horizontal plane
Change in dist hor between sy and 8:+11 (negative moving closer)
pitch error delta pitch error abs roll rell error delta roll error hdg error delta hdg error
Absolute difference in pitch angle between EJ and Change in pitch error between 8; and 8141
Absolute roll angle of EJ
Absolute difference in roll angle between EJ and RJ Change in roll 1 error between 8, and st+11
Absolute difference in heading angle between and RJ
Change in hdg error between 8t and 8141
fwd error
Angle between 3D vectors indicating forward axes of EJ and RJ
delta fwd error
Change in fwd error between s; and 8:41
up error delta up error right error
Angle between 3D vectors indicating upward axes EJ and
Change in up error hetween & and 8++1
Angle between 3D vectors indicating rightward axes of EJ and RJ
delta right error
Change in right error between and 8:41
los error
Angle between forward axis of EJ and vector from EJ RJ (measures whether RJ is in EJ's line of sight)
delta los error abs ir offsct
Change in los error between St and 8t411
Magnitude of projection of vector from EJ to RJ onto RJ's rightward axis (measures left left-right offset between the two aircraft in RJ's reference frame)
speed g force
Airspeed of EJ
Instantaneous g-force experienced by EJ Absolute change of EJ pitch between 8t and 8++1 Absolute change of EJ roll between and 8141 Absolute change of EJ yaw between 8t and s
pi rate roll rate yaw rate thrust delta thrust
Instantaneous thrust output by EJ engines Absolute change in thrust between 8, and $141
C.3 TASKS AND ORACLES
In this paper, consider three concrete tasks that instantiate the general EJ-RJ framework. For each, we construct a plausible oracle reward function from a subset of the 30 features, meaning that reward learning is in part feature selection problem (tree models perform feature selection explicitly whenever they add a new splitting rule). Although the precise nature of the oracle reward functions is secondary, and those given below are among many equally reasonable alternatives, we dedicated several hours of development time to ensuring they yield reasonable behaviour upon visual inspection. The difficulty and seeming arbitrariness of this manual reward design process is precisely why reward learning (ultimately from real human preferences) is an enticing proposition. Descriptions of the three tasks, along with their respective oracles, are given below:
Follow: Here RJ follows linear horizontal flight path at constant velocity, which is oriented opposite to the initial velocity of EJ. The goal is to turn onto and then maintain the path up to the episode time limit of 20 timesteps. This constitutes very simple form of
19

---

Under review as a conference paper at ICLR 2023
formation flight. The oracle reward function incentivises closing the distance to the moving target, and matching the upward axes of EJ and RJ:
(dist + 0.05 x closing speed + x up error).
Chase: Here RJ follows an erratic trajectory generated by random control inputs, and the goal to chase it without taking EJ below safe altitude of 50. Episodes terminate after 20 timesteps. The oracle reward function incentivises keeping RJ at distance of 20 and within EJ's line of sight, while keeping EJ oriented upright. also has large penalty for dropping below the safe altitude:
(abss 20) + 10 los error abs roll + { 100 if alt < 50
)
otherwise
Land:
Here the goal is execute safe approach towards landing on runway, where RJ represents the ideal landing position (central, zero altitude, slight upward pitch). EJ is initialised at random altitude, pitch, roll and offset, such that landing may be challenging but always physically possible. An episode terminates if EJ passes RJ along the axis of the runway, or after 25 timesteps otherwise. The oracle reward function for this task is far the most complex of the three, including terms that incentivise continual descent, penalise g-force and engine thrust, and punish the agent for making contact with the ground (alt < 0.6) before the start of the runway:
r= -0.05 x abs 1r offset + 0.05 x alt hdg error + abs roll +0.5 x pitch error + 0.25 x (yaw rate + roll rate pitch rate) +0.1 x g force + 0.025 x thrust + 0.05 x delta thrust
+
if delta dist hor > otherwise
+ {{
2 if delta alt > 0
0 otherwise
{ 1 if abs 1r offset > 10
{ 0 otherwise
+
+
10 if 0.6
0 otherwise
20

---

Under review as a conference paper at ICLR 2023
D IMPLEMENTATION AND EXPERIMENT DETAILS
D.1 ORACLE PREFERENCES
Oracle preferences are generated in accordance with the Bradley-Terry model given in Equation | Lee by computing the returns for the two trajectories ç and Ç and sampling from Boltzmann distribution parameterised by those returns. our main experiments, we set the temperature coeffi- cient B = O, which results in the oracle deterministically selecting the trajectory with higher return (ties broken uniform-rane raadoolll..ddmll)... In Section 6..3 we study cases with B > 0, which provide a more realistic emulation of real human preference data.
D.2 HYPERPARAMETERS FOR TREE INDUCTION
In all experiments, we use the following hyperparameters during tree induction. These were iden- tified through informal search, and we make claim optimality, but they do lead reasonable performance on the three tasks of varying complexity. This indicates a general insensitivity of the method to precise hyperparameter values, which often practically advantageous.
Trajectory return estimation using the Adam optimiser with learning of 0.1. Optimi- sation stops when the mean ENLI changes by le-5 between successive gradient steps.
Per-feature candidate split thresholds C defined as all midpoints between adjacent unique values in the trajectory set =. These are recomputed on each update.
Tree size limit Tamax = 100.
Tree size regularisation coefficient a 5e. -3
As mentioned in Appendix A.2 enforce negative rewards (max 0 constraint) for the Land
task, and positive rewards (min constraint) for Follow and Chase.
D.3 MODEL- MODEL-BASED RL IMPLEMENTATION
For conceptual details on the PETS algorithm, we refer readers the original paper by Chua et al. (211)). In our implementation, the dynamics model is an ensemble of five NNs, each with four hid- den layers 200 hidden units and ReLU activations. State vectors are pre-normalised by applying a hand-specifed scale factor to each dimension. Decision-time planning operates over time horizon of 10 and consists of 10 iterations of the cross-entropy method. Each iteration samples 20 candidate action sequences from independent Gaussian, of which the top in terms of return identified as elites, then updates the sampling Gaussian towards the clites with a learning rate 0.5. In all experiments use 7 meaning no temporal discounting applied during planning.
In our experiments, we find that the particular dynamics of the aircraft handling environment permit pre-train D' on random offline data, and accurately generalise states encountered during line reward learning. This means we perform no further updates to the model while reward learning ongoing. As well improving wall-clock speed, this avoids complexity and convergence issues arising from having two interacting learning processes (note that simultancous learning is completely unavoidable with model-free RL). pre-train, we collect le" transitions by rolling out uniform random policy, then update each of the five networks on le independently sampled mini-batches of 256 transitions, using the mean squared error loss over normalised next-state predictions.
D.4 NEURAL NETWORK REWARD LEARNING BASELINE
We baseline reward tree models against the de facto standard approach of reward learning using a NN. constructing this baseline, we aimed to retain as much of Algorithm [[] as possible, so that only the model architecture varies. The result is that replace lines 21-47 with the following:
21

---

Under review as a conference paper at ICLR 2023
21:
for m E (I,...,,M) do
22:
Lmini-batch mini-batch of B preference labels sampled from L
23:
Compute ENLI over Lmini- via Equations and Backpropagate loss and update network parameters
24:
25: end for
Fall f reward predictions for all feature vectors in E
27:
Scale network oulputs by (rai
Shift network outputs by mmin(raaa)) or -max(ran) depending on desired reward sign
The new lines 26-28 replicate the two constraints applied in Equation
In all experiments, we follow Lee et (2021b) in implementing the reward model as a three-layer network with 256 hidden units each and leaky ReLU activations, and performing the update on line 24 using the Adam optimiser (Kingma & Ba 2014) with a learning rate 3e-1, On each update, we sample M 100 mini-batches of size B 32 and take one gradient step per mini-batch.
D.5 COVERAGE DATASETS FOR POLICY- INVARIANT EVALUATION
Gleave et al. 2021 recently highlighted the importance of comparing and evaluating learnt reward functions in policy-invariant manner, by using a common evaluation distribution rather than policy data generated by agents optimising for each reward. Ideally, the offline evaluation data should have high coverage (i.e. high-entropy state distribution, both high- and low-quality trajecto- ries), in order to characterise the reward functions' outputs across spectrum of plausible policies.
In our context, we can generate data that satisfies these requirements by leveraging the known oracle reward functions and the PETS algorithm. We deploy PETS using the oracle reward, but randomise planning parameters (number of planning iterations E {1, ..., 50), number of action sequence samples E {4,...,0)) elites. This randomisation results trajectories that are sometimes near-optimal with respect the oracle, sometimes moderate in quality, and sometimes barely better than random. For all three tasks, we generate dataset of 200 evaluation trajectories in this manner.
every episode. In all cases, we take the top 25% of action sequences as
22

---

Under review as a conference paper at ICLR 2023
E VISUALING REWARD FUNCTIONS WITH SIMILARITY EMBEDDINGS
Here we briefly discuss a novel form of reward visualisation which we developed and found valu- able during our own analysis. Given some measure of similarity between reward functions, such as the EPIC metric proposed by Gleave et al. (2021), we can compute a matrix of pairwise similarities between any number of such functions (computational cost permitting). We can then produce a 2D embedding of the functions by applying multidimensional scaling (MDS). Visualising this embed- ding as scatter plot enables the discovery of salient patterns and trends in the set of functions.
In Figure A3 we use the metric of rank correlation on the coverage datasets, and the SMACOF MDS algorithm (De Leeuw & Mair 2009), to embed all model repeats and the oracle for each task. This gives an impression of the models' similarity not just to the oracle, but to each other. Aside from the Follow NNs, which form a tight cluster near the oracle, the distribution for each model indicates roughly equal consistency between repeats. The overlap of convex hulls suggests that the rankings produced by all models are broadly similar for Land, but more distinct for Chase. Shading points by ORR reveals that while models further from the oracle tend induce worse perro-enfooming policies, the trend not monotonie. This reinforces the point made elsewhere that the problems of learning good policy and exactly replicating the ground truth reward not identical.
VN
Follow
Chase
Land
Teee (9-1) Thee (vad
0000
X
Oracle
Figure A3:
Rank correlation embeddings for all model repeats from the main experiments, with the scatter point for each repeat shaded by ORR.
Populating such embedding plots more densely, perhaps by varying model hyperparameters, could provide a means of mapping the space of learnable reward functions and its relationship to policy performance. It would also be straightforward to compute similarity values for the same model repeat at multiple checkpoints during learning. This would yield a trajectory in the rank embedding space, which could aid the assessment of the stability and convergence properties of online learning with different models and hyperparameter values.
23


Reviews:
[{"review_id": "rW57h-CLbXt", "reviewer": "Reviewer_1NsX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the method of learning interpretable trees for representing reward functions by following a previously-proposed reward tree model. The paper proposes several updates to the reward tree inference algorithm, including an NLL-based objective for estimating trajectory-level returns, a new criterion facilitating tree growth, and a model-based RL method. This paper empirically studies the performance of their model on an aircraft handling environment by mainly focusing on three control tasks, including 'Follow', 'Chase' and 'Land'. The results demonstrate the performance of the proposed updates from the perspectives of quantitative performance, visual inspection, sensitivity analysis, and tree structure analysis.", "summary_of_the_review": "The authors spend considerable effort to demonstrate the performance of reward trees, which is remarkable in my mind. However, the reward tree has been proposed in previous work, and the updates mentioned in this paper are marginal and incremental without enough evidence to justify why they are critical. I also have some concerns about the argument made in the paper, and I hope the authors can respond to them. The environment picked for the empirical study is not satisfying. The author should consider including some more popular environments. I think I have provided some useful suggestions which could significantly improve the paper (I do believe so), but implementing them takes some time and effort. In conclusion, I vote for a rejection based on the current version.", "strength_and_weaknesses": "**Strength**: \n1. The paper is well-structured and easy to follow. The reward tree is clearly defined with sufficient details (in both the main paper and appendix) for supporting implementation and reproduction. \n2. The empirical results demonstrate the performance (including fidelity and interpretability) of the reward tree from multiple perspectives, although the paper studies only an aircraft handling environment which is not a common benchmark for RL.\n\n\n**Weaknesses**:\n\n1. **Contribution.** The novelty of this paper is marginal. The main methodological contribution is the reward tree, but this method was proposed by a previous paper and it is well-studied under the RL environment. The updates on the reward tree are incremental, without strong motivations or careful justification about why these updates are critical. The main argument is that the performance can be somehow improved with these updates, but this argument is fairly weak to be a part of a scientific paper, especially for the papers to be published at top-tier conferences such as ICLR. I strongly recommend the author(s) to focus on one of these updates and justify the necessity of this update with theory results and relevant experiments.\n\n2. **Evaluation Environment** This paper focuses only on an aircraft handling environment, but this environment is not a common benchmark for RL. I do recommend using some popular environments like Atari and MoJoCo. Most of the RL works are evaluated in these environments, and in this way, readers can understand how well the proposed method advances the relevant research fields/topics and whether this method can be scaled to popular RL tasks. \n\n3. **Related Works** I believe Inverse Reinforcement Learning (IRL) shares the same goal as this paper. The major difference is IRL does not require human feedback while the proposed task has to rely on such signals during reward inference. Whether this human-in-the-loop system can be generalized to solve complex tasks that require a large number of interactions and long-term planning is questionable. I won't argue for this issue since I understand the authors just follow some previous works that have the same issue, but I believe the author should at least mention the works of IRL in the paper.\n\n4. **Additional Concerns** \n\n- \"*We show it to be broadly competitive with neural networks on challenging high-dimensional tasks,*\"   \nBy claiming high-dimensional tasks in RL, we often refer to image-based games (e.g., Atari) or text-based games (e.g., TextWorld). Recently year, Deep RL has achieved significant performance in these games while the tree model can not. I think the current reward tree cannot be scaled to these tasks, but it could be a promising direction to explore or advance.\n\n- \"*These parameters are non-differentiable, making end-to-end optimization of the losses in Equation 2 computationally intractable.*\"  \nThis is a common issue for the tree model, but I have not found the corresponding solution or circumvention. I guess the issue remains unsolved, but In fact, many differentiable tree models have been proposed in recent years and I do recommend referring to these works.\n\n- \"*For consistency with prior work, we instead minimize the NLL loss under the Bradley-Terry model.*\"   \nI believe \"consistency with prior work\" is the motivation for the NLL loss. However, it is a well-known fact that the square loss can be consistent with the log-likelihood loss if we assume the targets (or returns in this work) are Gaussian distributed with unit variance. This motivation needs to be clarified.\n\n- \"*We optimise Equation 3 by unconstrained gradient descent with the Adam optimiser (Kingma & Ba, 2014), followed by post-normalisation to meet the constraints*\"   \nI recommend expanding this sentence by defining the post-normalization method. Have you projected the constrained objective to the unconstrained one in each step? If yes, which projection framework have you applied? I cannot imagine how post-normalization can achieve this projection. I question the correctness here, please explain.\n\n- \"*we find that switching from SAC to a model-based algorithm called PETS (Chua et al., 2018) reduces training steps by multiple orders of magnitude, and cuts wall-clock runtime (see Appendix B).*\"    \nOne of the main challenges of MBRL is learning the transition dynamics based on collected samples, and we often count the training steps for the transition model when we compare  MBRL's sample complexity with model-free methods. It seems in this paper the comparison is based on a well-trained transition model if I understand correctly, which makes the conclusion questionable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qfYKnOMm-ZX", "reviewer": "Reviewer_M2ur", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors work on preference-based RL setups, where they develop reward trees from the preference labels. They use four stages for the learning, first estimating the returns for trajectories, then, predicting the reward at the leaf level, and finally, they apply tree growth and tree pruning operations for the reward trees. Even though they observe performance hit when switching from neural network-estimated rewards to tree-based rewards, the authors argue that the tree-based methods can be competitive. The advantages of the tree-based method are shown in Figure 6 by easy interpretation of the reward meanings and semantic understanding. ", "summary_of_the_review": "I'm currently leaning towards rejection of the paper since it is showing a certain level of ethical concerns. The domains tested in the paper could be used for wars, weapons, etc, which I don't think it's the only application. The authors could demonstrate the effectiveness of their method in a less sensitive domain.", "strength_and_weaknesses": "Strength\n+ A complete pipeline for tree-based reward learning from preference labels where the authors show effective learning and competitive performance against commonly used neural network approaches;\n+ Detailed description of the four-stage pipeline;\n+ Competitive performance is shown in evaluation against NN approaches;\n+ Example reward trees are shown to understand the advantages of the tree-based reward methods;\n\nWeaknesses\n- The method is tested upon a not commonly used domain, i.e., aircraft handling environment. It's hard to understand the properties of the domain and the difficulty of learning just from the text description. The authors could use additional relatively standard benchmarks for their experiments;\n- The generality of the proposed method could be shown by using more diverse domains for the experiments.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Llm5yNEH518", "reviewer": "Reviewer_aaeK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper takes a published method by Bewley and Lecue (2022) named BL, which is about reward tree learning for reinforcement learning (RL) problems using preferences over trajectories as input, extends it slightly with some algorithmic tweaks and a vanilla incorporation of it inside a model-based reinforcement learning (MBRL) loop, and provides an extensive experimental evaluation with the main aim to compare against neural networks for the same task, and to conclude that these neural networks are still often the better choice but less capable of providing transparency and explainability than tree structures.", "summary_of_the_review": "Solid paper, with a very extensive evaluation (on a single domain though, and no real comparison against the original BL algorithm) and well written, and mostly well motivated and place in the related literature. However, the novelty and significance are limited, compared to the state-of-the-art.", "strength_and_weaknesses": "The strengths of this paper are \n1) it is a solid paper in terms of the text, the structure, the content, \n2) it is a solid set of experiments with clear goals, good methods, and extensive analysis, \n3) the topic is interesting as explainable models that can rival the (still) less explainable neural networks for RL tasks are interesting, \n4) the description of the method, including some tweaks and small extensions are solid.\n\nThe main weaknesses of the paper are:\n5) the novelty and significance are limited, especially compared to the state-of-the-art in general and the work by Bewley and Lecue (2022), BL, in particular.\n6) The paper is very dense, which is a good thing because it is quite informative for that matter, but for some parts I feel that more general intuitions would be better in the main tet than highly detailed descriptions of all outcomes (I mean, there are still appendices).\n7) The experimental evaluation is extensive, but has a very simple conclusion: NNs are still often better (in many ways) as expected because of the more general model class, but tree structures can help in explanations, which is not new at all.\nI'll elaborate a bit more on 5-7:\n\nAbout (5) I feel that the extension of BL is only minor. I checked the original BL paper for that, and the description is different in some ways (notationally) but overall it seems that the main difference lies in somewhat different optimization in the first step and a different criterion in the third step. This difference is not analysed/evaluated later on anyway. Another difference is the use of the method in model-based RL settings, whereas BL focuses on model-free settings. The current paper uses a very naive approach where both tree induction and planning happens constantly from scratch, hence also here no real new mechanisms are in place. I think it would be good to highlight the algorithmic and performance-wise aspects of these parts more explicitly in a kind of contributions, to make this more clear. \nThis could also be said about the related work section: it is nice and informative, and certainly substantial, but I do miss these explicit placements of the current method in the state-of-the-art. Also, I think that \"trees for explainable\" is something that has been considered before in RL for decades, although not explicitly so in terms of \"explainable\" since that is a more recent development. Tree-based models in RL have a long history, and it would be good to connect to much more older work to, to highlight that. Take, for example, the seminal work by Dzeroski, Blockeel and De Raedt on relational RL (in ICML 1998) inducing relational (Q-) trees. Around that time other work on trees, and comparisons against NNs (standard MLPs) can also be found in propositional settings (see also related seminal work by Chapman and Kaelbling in 1991 on the G-algorithm). Just to say trees have a long history in RL.\n\nAbout (6): the dense nature of the paper makes it solid on one hand, but sometimes not easy to quickly see the main outcomes (especially in the experimental evaluation). I think this can be improved by making better distinctions between main and sub results. Also, some motivations, and some explanations of parts of the technical machinery in the first half of the paper could be extended to make it slightly more self-contained (although: I admit that for a technical paper like this, it is not possible entirely). I especially think that the main step from \"preferences over trajectories\" towards \"trees of local splitting rules used to induce rewards for individual steps\" can use more explanation and intuition (both technically, and conceptually).\n\nAbout (7): there are many comparisons between trees and NNs for many tasks, RL or other (like supervised). The main conclusion that NNs are typically better but trees more explainable is not surprising. The paper does not provide too many more insights into the problem, and many other smalle results often come with phrases like \"This could be\" and \"This may indeed\" with some general ideas on why and how the results come about. I do want to emphasize the results in the paper are solid and systematically carried out, but I am looking for new insights on either the method/concepts as a whole (and trees vs NNs) or on the comparison of the new additions vs the original BL (but this last aspect is not part of the paper). So, it seems many experiments are done, but without too much to gain from it, is my feeling.\nAlso, I feel that the paper could benefit from first doing the comparison on a much simpler domain, where trees could be induce with very simple structure, to highlight aspects of the method, but also aspects of the evaluation methodology, and as a simple illustration and sanity check. With the more complex domain used, not all phenomena observed in the results can actually be fully understood/explained.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject