Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors work on preference-based RL setups, where they develop reward trees from the preference labels. They use four stages for the learning, first estimating the returns for trajectories, then, predicting the reward at the leaf level, and finally, they apply tree growth and tree pruning operations for the reward trees. Even though they observe performance hit when switching from neural network-estimated rewards to tree-based rewards, the authors argue that the tree-based methods can be competitive. The advantages of the tree-based method are shown in Figure 6 by easy interpretation of the reward meanings and semantic understanding. 
strength_and_weaknesses: Strength
+ A complete pipeline for tree-based reward learning from preference labels where the authors show effective learning and competitive performance against commonly used neural network approaches;
+ Detailed description of the four-stage pipeline;
+ Competitive performance is shown in evaluation against NN approaches;
+ Example reward trees are shown to understand the advantages of the tree-based reward methods;

Weaknesses
- The method is tested upon a not commonly used domain, i.e., aircraft handling environment. It's hard to understand the properties of the domain and the difficulty of learning just from the text description. The authors could use additional relatively standard benchmarks for their experiments;
- The generality of the proposed method could be shown by using more diverse domains for the experiments.

clarity,_quality,_novelty_and_reproducibility: The paper has decent clarity and quality.  The novelty seems a bit limited but it's a complete pipeline from preference labels to agent training. The reproducibility should be good since they provided source code.
summary_of_the_review: I'm currently leaning towards rejection of the paper since it is showing a certain level of ethical concerns. The domains tested in the paper could be used for wars, weapons, etc, which I don't think it's the only application. The authors could demonstrate the effectiveness of their method in a less sensitive domain.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper takes a published method by Bewley and Lecue (2022) named BL, which is about reward tree learning for reinforcement learning (RL) problems using preferences over trajectories as input, extends it slightly with some algorithmic tweaks and a vanilla incorporation of it inside a model-based reinforcement learning (MBRL) loop, and provides an extensive experimental evaluation with the main aim to compare against neural networks for the same task, and to conclude that these neural networks are still often the better choice but less capable of providing transparency and explainability than tree structures.
strength_and_weaknesses: The strengths of this paper are 
1) it is a solid paper in terms of the text, the structure, the content, 
2) it is a solid set of experiments with clear goals, good methods, and extensive analysis, 
3) the topic is interesting as explainable models that can rival the (still) less explainable neural networks for RL tasks are interesting, 
4) the description of the method, including some tweaks and small extensions are solid.

The main weaknesses of the paper are:
5) the novelty and significance are limited, especially compared to the state-of-the-art in general and the work by Bewley and Lecue (2022), BL, in particular.
6) The paper is very dense, which is a good thing because it is quite informative for that matter, but for some parts I feel that more general intuitions would be better in the main tet than highly detailed descriptions of all outcomes (I mean, there are still appendices).
7) The experimental evaluation is extensive, but has a very simple conclusion: NNs are still often better (in many ways) as expected because of the more general model class, but tree structures can help in explanations, which is not new at all.
I'll elaborate a bit more on 5-7:

About (5) I feel that the extension of BL is only minor. I checked the original BL paper for that, and the description is different in some ways (notationally) but overall it seems that the main difference lies in somewhat different optimization in the first step and a different criterion in the third step. This difference is not analysed/evaluated later on anyway. Another difference is the use of the method in model-based RL settings, whereas BL focuses on model-free settings. The current paper uses a very naive approach where both tree induction and planning happens constantly from scratch, hence also here no real new mechanisms are in place. I think it would be good to highlight the algorithmic and performance-wise aspects of these parts more explicitly in a kind of contributions, to make this more clear. 
This could also be said about the related work section: it is nice and informative, and certainly substantial, but I do miss these explicit placements of the current method in the state-of-the-art. Also, I think that "trees for explainable" is something that has been considered before in RL for decades, although not explicitly so in terms of "explainable" since that is a more recent development. Tree-based models in RL have a long history, and it would be good to connect to much more older work to, to highlight that. Take, for example, the seminal work by Dzeroski, Blockeel and De Raedt on relational RL (in ICML 1998) inducing relational (Q-) trees. Around that time other work on trees, and comparisons against NNs (standard MLPs) can also be found in propositional settings (see also related seminal work by Chapman and Kaelbling in 1991 on the G-algorithm). Just to say trees have a long history in RL.

About (6): the dense nature of the paper makes it solid on one hand, but sometimes not easy to quickly see the main outcomes (especially in the experimental evaluation). I think this can be improved by making better distinctions between main and sub results. Also, some motivations, and some explanations of parts of the technical machinery in the first half of the paper could be extended to make it slightly more self-contained (although: I admit that for a technical paper like this, it is not possible entirely). I especially think that the main step from "preferences over trajectories" towards "trees of local splitting rules used to induce rewards for individual steps" can use more explanation and intuition (both technically, and conceptually).

About (7): there are many comparisons between trees and NNs for many tasks, RL or other (like supervised). The main conclusion that NNs are typically better but trees more explainable is not surprising. The paper does not provide too many more insights into the problem, and many other smalle results often come with phrases like "This could be" and "This may indeed" with some general ideas on why and how the results come about. I do want to emphasize the results in the paper are solid and systematically carried out, but I am looking for new insights on either the method/concepts as a whole (and trees vs NNs) or on the comparison of the new additions vs the original BL (but this last aspect is not part of the paper). So, it seems many experiments are done, but without too much to gain from it, is my feeling.
Also, I feel that the paper could benefit from first doing the comparison on a much simpler domain, where trees could be induce with very simple structure, to highlight aspects of the method, but also aspects of the evaluation methodology, and as a simple illustration and sanity check. With the more complex domain used, not all phenomena observed in the results can actually be fully understood/explained.

clarity,_quality,_novelty_and_reproducibility: As indicated earlier, the clarity and quality of the approach are co-dependent: the paper is solid, and the authors do quite a good job in explaining all aspects (the paper is overall well written) but it can be improved. Novelty is a weak point as discussed earlier. Reproducibility should not be an issue if code would be provided, and based on the description a lot is possible too, but with some additional intepretation/implementation or help.
summary_of_the_review: Solid paper, with a very extensive evaluation (on a single domain though, and no real comparison against the original BL algorithm) and well written, and mostly well motivated and place in the related literature. However, the novelty and significance are limited, compared to the state-of-the-art.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces the method of learning interpretable trees for representing reward functions by following a previously-proposed reward tree model. The paper proposes several updates to the reward tree inference algorithm, including an NLL-based objective for estimating trajectory-level returns, a new criterion facilitating tree growth, and a model-based RL method. This paper empirically studies the performance of their model on an aircraft handling environment by mainly focusing on three control tasks, including 'Follow', 'Chase' and 'Land'. The results demonstrate the performance of the proposed updates from the perspectives of quantitative performance, visual inspection, sensitivity analysis, and tree structure analysis.
strength_and_weaknesses: **Strength**: 
1. The paper is well-structured and easy to follow. The reward tree is clearly defined with sufficient details (in both the main paper and appendix) for supporting implementation and reproduction. 
2. The empirical results demonstrate the performance (including fidelity and interpretability) of the reward tree from multiple perspectives, although the paper studies only an aircraft handling environment which is not a common benchmark for RL.


**Weaknesses**:

1. **Contribution.** The novelty of this paper is marginal. The main methodological contribution is the reward tree, but this method was proposed by a previous paper and it is well-studied under the RL environment. The updates on the reward tree are incremental, without strong motivations or careful justification about why these updates are critical. The main argument is that the performance can be somehow improved with these updates, but this argument is fairly weak to be a part of a scientific paper, especially for the papers to be published at top-tier conferences such as ICLR. I strongly recommend the author(s) to focus on one of these updates and justify the necessity of this update with theory results and relevant experiments.

2. **Evaluation Environment** This paper focuses only on an aircraft handling environment, but this environment is not a common benchmark for RL. I do recommend using some popular environments like Atari and MoJoCo. Most of the RL works are evaluated in these environments, and in this way, readers can understand how well the proposed method advances the relevant research fields/topics and whether this method can be scaled to popular RL tasks. 

3. **Related Works** I believe Inverse Reinforcement Learning (IRL) shares the same goal as this paper. The major difference is IRL does not require human feedback while the proposed task has to rely on such signals during reward inference. Whether this human-in-the-loop system can be generalized to solve complex tasks that require a large number of interactions and long-term planning is questionable. I won't argue for this issue since I understand the authors just follow some previous works that have the same issue, but I believe the author should at least mention the works of IRL in the paper.

4. **Additional Concerns** 

- "*We show it to be broadly competitive with neural networks on challenging high-dimensional tasks,*"   
By claiming high-dimensional tasks in RL, we often refer to image-based games (e.g., Atari) or text-based games (e.g., TextWorld). Recently year, Deep RL has achieved significant performance in these games while the tree model can not. I think the current reward tree cannot be scaled to these tasks, but it could be a promising direction to explore or advance.

- "*These parameters are non-differentiable, making end-to-end optimization of the losses in Equation 2 computationally intractable.*"  
This is a common issue for the tree model, but I have not found the corresponding solution or circumvention. I guess the issue remains unsolved, but In fact, many differentiable tree models have been proposed in recent years and I do recommend referring to these works.

- "*For consistency with prior work, we instead minimize the NLL loss under the Bradley-Terry model.*"   
I believe "consistency with prior work" is the motivation for the NLL loss. However, it is a well-known fact that the square loss can be consistent with the log-likelihood loss if we assume the targets (or returns in this work) are Gaussian distributed with unit variance. This motivation needs to be clarified.

- "*We optimise Equation 3 by unconstrained gradient descent with the Adam optimiser (Kingma & Ba, 2014), followed by post-normalisation to meet the constraints*"   
I recommend expanding this sentence by defining the post-normalization method. Have you projected the constrained objective to the unconstrained one in each step? If yes, which projection framework have you applied? I cannot imagine how post-normalization can achieve this projection. I question the correctness here, please explain.

- "*we find that switching from SAC to a model-based algorithm called PETS (Chua et al., 2018) reduces training steps by multiple orders of magnitude, and cuts wall-clock runtime (see Appendix B).*"    
One of the main challenges of MBRL is learning the transition dynamics based on collected samples, and we often count the training steps for the transition model when we compare  MBRL's sample complexity with model-free methods. It seems in this paper the comparison is based on a well-trained transition model if I understand correctly, which makes the conclusion questionable.
clarity,_quality,_novelty_and_reproducibility: As I mentioned above, the paper is well-structured and easy to follow, but the proposed updates on the reward tree are incremental and the novelty is not well-defined. In terms of reproducibility, I have not run the code, but after scanning through the appendix, I believe the paper has sufficient details for supporting reproduction.
summary_of_the_review: The authors spend considerable effort to demonstrate the performance of reward trees, which is remarkable in my mind. However, the reward tree has been proposed in previous work, and the updates mentioned in this paper are marginal and incremental without enough evidence to justify why they are critical. I also have some concerns about the argument made in the paper, and I hope the authors can respond to them. The environment picked for the empirical study is not satisfying. The author should consider including some more popular environments. I think I have provided some useful suggestions which could significantly improve the paper (I do believe so), but implementing them takes some time and effort. In conclusion, I vote for a rejection based on the current version.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to replace neural network-based learnt reward models (RMs) with tree-based RMs due to the interpretability properties of the latter. The algorithm is closely based on prior work of Bewley and Lecue (2022) (BL), and the approach is evaluated on a number of tasks in the aircraft handling domain. This paper runs solid empirical evaluation of the proposed method and shows that it does work in the tested domain, resulting in interpretable RMs. However, the methodological innovation proposed in this paper is seen by the reviewers as minor (the base BL algorithm is modified only slightly), the proposed approach performs worse than NNs (although the tree-based approach is more interpretable), and the method is only evaluated on a single novel domain, which makes it hard to understand the value of the approach more broadly against the existing body of literature. Although the authors did a great job participating in the discussion period, unfortunately the reviewers remained not fully convinced that the paper merits acceptance at this point. 
justification_for_why_not_higher_score: The value add of this paper is interpretability, however the interpretability of trees vs NNs is already known, and the relative performance of trees vs NNs has been investigated in other domains (e.g. in finance). Given that this paper is also targeting a very particular application domain with potential ethical implications, it does not seem to me that this paper in its current form will be a meaningful contribution to the community.
justification_for_why_not_lower_score: NA
Decision: Reject