Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a new method for training RL agents with human feedback. In prior work, agents are trained either with a fully programmed reward function (which is hard to program correctly) or are trained with low bandwidth human feedback (e.g. binary feedback on which of two trajectories was better). This work proposes learning a set of attributes that describe trajectories, then getting human feedback on how a trajectory should change, in terms which which attributes to increase / decrease. This gives a higher bandwidth method of human feedback, leading to fewer rounds of feedback before successfully learning a concept.
strength_and_weaknesses: Strengths
-----------
- Novelty : there is surprisingly little published academic work on how to design reward functions. As the manager of a team using RL for a real product, I was surprised to find that "reward function" is not in the index of the Sutton and Barto book, and not a page in Wikipedia. So this paper contributes to a very underdeveloped area.
- Usefulness: my experience working in industry is that the biggest barrier to using RL for real products is not the performance of the RL algorithm itself, it's challenges outside of core RL, like sim to real transfer and how to design the reward function and evaluate performance of the agent. Real-world products interacting with human users have much more complicated reward functions than 0/1 winning or losing Go. So this paper contributes to an area that is important for practitioners in industry.

Weaknesses
--------------
- Experimental results: the experiments here are all on synthetic data and toy tasks. I'm not super optimistic that the methods proposed here would help my RL team on a real product. However, I think this is acceptable due to the limited amount of literature on the topic / I think other papers accepted in this area share this weakness.


clarity,_quality,_novelty_and_reproducibility: I think everything seems OK, but I don't work in this area myself, so I'm not sure what kinds of details I'd want to know if I was going to try to literally reproduce the experiment myself.
summary_of_the_review: I think this paper provides a reasonable step forward in an important subfield that is highly underdeveloped.

Statement on confidence: I haven't worked on this topic or even topics all that closely related to this firsthand, so it's reasonably likely that I've missed something important, particularly in terms of related work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes "relative behavioral attributes" an approach to allow non-expert end users to quickly (i.e., with limited data or demonstrations) correct robot/autonomous behaviors according to their preferences. The central approach involves learning (and later retraining) a ranking function that yields how strongly a particular attribute appears in a trajectory or behavior. Designers provide these attributes and labeled data for a ranking of how strongly attributes appear in each trajectory. During deployment, an end user can specify which behaviors they prefer through limited interactions---involving stating that a trajectory is preferred over another---and this data is used to train a preference-based reward model that uses the learned attribute models. Because trajectories are characterized in terms of a small set of attributes, retraining happens quickly, and far fewer feedback examples are required to achieve the desired behaviors. The authors show good performance on four different domains and significantly faster behavior change compared to the recent PbRL approach.

strength_and_weaknesses: Overall, the paper is fairly strong. The authors introduce two different approaches to using their approach and do a good job discussing the relative advantages of each and provide suitable performance analysis. Experiments are well-suited to understand the problem and cover multiple domains of interest to this community. PbRL is also a solid baseline, and the comparison to PbRL does a good job at highlighting why an alternative approach is necessary.

One part of the paper that could benefit from further discussion is where the approach is limited or is unlikely to succeed. For example: what if the attributes are limited or poorly chosen? How would performance change (presumably decrease) if an attribute is omitted? Further experiments are likely unnecessary, but adding such details (at least some of which should appear in the main body of the paper, even if others appear in the appendix) would help clarify when the proposed approach(es) are or are not well-suited to the proposed work.

Could the authors comment in more depth on what a typical failure mode looks like? There is some general discussion of why failure modes occur: namely what seems to be the limited gradient of the ranking function near the extreme values. However, some additional figures would improve understanding. Is the failure that the corrections do not change behavior? Some additional (qualitative) visuals or analysis would help clarity.

Finally, though it is straightforward to understand how embedded language descriptions could be used in place of a one-hot vector representation for the attribute vectors. However, it is not clear to me why the language-input models (in most of the experiments in Table 1) is competitive with or even frequently improves upon the performance of the non-language counterparts, since the non-language versions are trained using a representation specifically chosen by the designers (in this case, the authors) for this purpose. Could the authors clarify what the source of this improvement might be?

Other smaller comments:
- The abstract is quite long, and it might be good to remove some of the high-level context in the interest of conciseness.
- The Walker column in Table 2 seems to be incorrectly highlighted: RA-Global-L seems to perform best across all metrics.

clarity,_quality,_novelty_and_reproducibility: The paper is very well written and easy to follow. The motivation is incredibly clear and the introduction does an excellent job of characterizing the existing state of the art, how it is limited, and proposing an alternative approach that overcomes the data inefficiency of competitive approaches.

Experiments are fairly comprehensive within the context of the provided benchmarks (a reasonable set that demonstrate the versatility of the approach). The approach is novel, expanding on a problem studied in the literature, but proposes a new approach (loosely inspired by recent work in recommender systems) to support preference-aligned behaviors with few demonstrations.

There is no reproducibility statement and no code is provided.

summary_of_the_review: The paper is clear and delivers on most of its promises. I have a few questions and comments that the authors should address so that the paper can be maximally useful for the community, though they are relatively small.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper takes an existing reward learning paradigm in reinforcement learning (RL) that can learn reward functions from preferences over traces (of states) and refines it by allowing to target specific "relative attributes" (properties of a trace usually) to be optimized. The incorporation of the relative attributes enables to tune some "general" properties (usually) of traces, for example the smoothness or speed of certain movements of a robotic manipulator. The approach comes with two different ways to incorporate (and tune by a user) the relative attributes: one global technique that focuses on a vector of values, and one local technique where one attribute can be optimized. The approach is tested on a couple of domains and various relative attributes.
strength_and_weaknesses: Reward functions are the core of RL and work in finding new/different ways to incorporate and specify them is interesting. The strength of the paper is that it is a quite practical idea for a particular kind of reward function learning (and specification) in RL. The approach is intuitive, and amenable to all kinds of extensions. I think that the outcomes of the experiments show that the approach works.

The paper has, as a research paper, a couple of weaknesses that are not all easy to address. 
- An overall weakness is the text of the paper itself, including the formalizations. As far as I understand the paper, the work builds on typical (and recent) approaches leveraging Bradley-Terry-style models like on page 4 that can turn datasets of traces with preferences over these traces into a reward function that can be utilized to tune behavior towards those preferences. The main idea of the paper is to place either a vector of relative attributes in that ranking, or separate attributes (in the local model) and to derive a new reward function that incorporates the intended values (or direction of values) of those attributes. In both the local and the global model, the user can adjust the value of attributes such that the behavior of the agent changes in some intended direction. This is a fairly simple story and setup, but I had a hard time reading through all the verbose texts (already starting in Section 1) to find out what was happening. I'd advise a full rewriting and restructuring of the paper to reflect this. Also, the formalizations can then be made more precise and complete, and all this would help to create a much better description of the method. Also define simple things like "skill" or "skill behaviors", which seem to be assumed to be known, but which have very specific meanings in sub-areas of RL. Also, formalize the main aspects of the "re-weighting"/"re-ranking" of the traces better. The precise setting in sections 4.2. and 4.3. stays too implicit formally. Things like "attribute encoder" are used without explanation or formalization. Maybe give a couple of good examples in the beginning to sketch te idea more precisely. Also, the paper is not polished: several small errors (like there is no Table 5 on page 9, "sample at large" on the last line of page 8, informal style for "don't" instead of "do not", double words "the the" on page 5, and ungrammatical sentences like the first sentence of the 2nd par of Section 4.2. around "than", etc). Overall, I think the core of the paper is not described well, and this hinders a good evaluation of the technical parts of the paper.
- The positioning of the work is done poorly. First of all, already in Section 1 we see many different aspects, and levels of reward function definitions, but nothing is really explained nor compared. I think, since nothing is really explicitated around this "lingua franca" (page 1), it would be much better to say something like "sometimes one has some ideas about aspects of a full behavior that should be changed, a property of the behavior, and we would like to increase or decrease it, and for this we introduce relative attributes, a little in direction of how people do style transfer in deep learning". The title promises much more than is in the paper now. There is nothing really about symbolic goal specifications as in (temporal) logic, where relative attributes as presented in this paper would be something like a maintenance goal. Furthermore, relative attributes are limited (as in the experiments) to a general number about the trace as a whole, which should be discussed. The experiments too all use these types of attributes. All these aspects should be related to the literature, which is not really happening in Section 2: we see some general references and some more specific, but it is hard to what is the state-of-the-art and what is the precies positioning of this paper, simply also because the section before it does not really define the setting well. The related work section should be much improved, and include things like reward functions (symbolic, preference rankings, but also inverse RL which is very related) but also things like style transfer in RL and other related topics. Also, some of the paper sounds too pretentious, like "lingua franca" and "extremely efficient manner" and "superior performance", etc.
- The experiments are showing viability but not much more. From the technical description I get the feeling that this approach will work in some way (since, more information goes into the ranking and that seems to be a sound idea) but not how well, nor how it "really" works. The only baseline used is PbRL but this is not explained at all. The comparison is based on a couple of assumptions on how to use both (PbRL and the new technique) but this is not motivated enough. An important choice are the amounts of feedbacks (30 and 1000) but it is unclear how they influence the results, which are based on "proximity" to some good solution, and this is the only thing we get to see in Table 1. The number of domains considered is good, but the set of experiments is very limited and the analysis very short (one paragraph). I think the experimental evaluation should be greatly extended (not necessarily in the amount of domains or experiments, but in terms of analysis and sensitivity analysis) which is possible if the first half of the paper is written more concisely. I like the general embedding approach where also language descriptions can be used, but also here I'd like to see much more information and analysis (and examples). Basically, I'd like to get more detailed (visual) analysis of "how" behaviors are tuned based on the attributes in a specific domain (how fast, how well, artifacts, visibility of changes, correspondence between attributes and measured proxies, and so on).
clarity,_quality,_novelty_and_reproducibility: As said in the above, the clarity of the approach is far from optimal, and a good rewriting of the core ideas of the paper should be done. The quality of the work is ok; I think that technically the idea is sound, but it needs a more solid evaluation (and possibly comparison to more work). Novelty is ok as well; the core idea of learning reward functions from preferences over traces is not new, but this particular way of reranking trajectories is, and it seems to be medium significant although it is not entirely clear since the related work is not done optimally. I think that novelty in terms of the title, as in "providing a new middle ground to reward functions" is a bit pretentious given what is in the paper as description of this sub-field. The reproducibility of the paper should be fine, based on this description, and if code would be available.
summary_of_the_review: Nice practical idea in the sub-area of RL to learn from user preferences. The description (both text and formalizations, related work) needs quite some work, as well as the experimental analysis which should be extended in terms of analysis and description.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a novel approach of doing RL against a reward model trained from human feedback. The authors propose using a set of attributes to describe the quality of an RL trajectory, and human feedback is collected against these attributes - i.e. does the trajectory need to be improved in any of the dimensions described by the attributes. This makes human feedback on each trajectory more precise and more informative, which leads to better performance.

The idea is novel and general with important practical applications. Saying this, the current work is seen by the reviewers as a proof of principle for the idea, and more work will be needed in the future to generalise this to the real world applications.
justification_for_why_not_higher_score: This paper proposes a novel solution to an important problem, but the reviewers agree that the current experiments are somewhat limited in their impact. They show that the idea is viable but more work will be needed to scale it up and apply to real world problems.
justification_for_why_not_lower_score: This paper proposes a novel solution to an important problem, and the experimental results support the viability of the proposed method. It will be a valuable contribution to the community and hence should be accepted to ICLR.
Decision: Accept: poster