Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences
Authors: Lin Guan
Karthik Valmeekam
Subbarao Kambhampati
Venue: ICLR 2023 poster (2023)
Keywords: Neuro-Symbolic
Human-AI Interaction

Abstract: Generating complex behaviors that satisfy the preferences of non-expert users is a crucial requirement for AI agents. Interactive reward learning from trajectory comparisons (a.k.a. RLHF) is one way to allow non-expert users to convey complex objectives by expressing preferences over short clips of agent behaviors. Even though this parametric method can encode complex tacit knowledge present in the underlying tasks, it implicitly assumes that the human is unable to provide richer feedback than binary preference labels, leading to intolerably high feedback complexity and poor user experience. While providing a detailed symbolic closed-form specification of the objectives might be tempting, it is not always feasible even for an expert user. However, in most cases, humans are aware of how the agent should change its behavior along meaningful axes to fulfill their underlying purpose, even if they are not able to fully specify task objectives symbolically. Using this as motivation, we introduce the notion of Relative Behavioral Attributes, which allows the users to tweak the agent behavior through symbolic concepts (e.g., increasing the softness or speed of agents' movement). We propose two practical methods that can learn to model any kind of behavioral attributes from ordered behavior clips. We demonstrate the effectiveness of our methods on four tasks with nine different behavioral attributes, showing that once the attributes are learned, end users can produce desirable agent behaviors relatively effortlessly, by providing feedback just around ten times. This is over an order of magnitude less than that required by the popular learning-from-human-preferences baselines. The supplementary video and source code are available at: https://guansuns.github.io/pages/rba.

Published as a conference paper at ICLR 2023
RELATIVE BEHAVIORAL ATTRIBUTES: FILLING THE GAP BETWEEN SYMBOLIC GOAL SPECIFICATION AND REWARD LEARNING FROM HUMAN PREFERENCES
Lin Guan, Karthik Valmeekam, Subbarao Kambhampati School of Computing & Al Arizona State University Tempe, AZ 85281, USA
(1guan9, 9, kva 1 lmeek, rao rao) @aoueesueedu
ABSTRACT
Generating complex behaviors that satisfy the preferences of non-expert users is a crucial requirement for AI agents. Interactive reward learning from trajec- tory comparisons (a.k.a. RLHF) is one way to allow non-expert users to convey complex objectives by expressing preferences over short clips of agent behav- iors. Even though this parametric method can encode complex tacit knowledge present in the underlying tasks, it implicitly assumes that the human is unable to provide richer feedback than binary preference labels, leading to intolerably high feedback complexity and poor user experience. While providing a detailed symbolic closed-form specification of the objectives might be tempting, it is not always feasible even for an expert user. However, most cases, humans are aware of how the agent should change its behavior along meaningful axes to fulfill their underlying purpose, even if they are not able to fully specify task objectives symbolically. Using this motivation, havioral Attributes, which allows the users to tweak the agent behavior through symbolic concepts (e.g., increasing the softness speed of agents' movement). We propose two practical methods that can learn to model any kind of behavioral attributes from ordered behavior clips. We demonstrate the effectiveness of our methods on four tasks with nine different behavioral attributes, showing that once the attributes are learned, end users can produce desirable agent behaviors rela- tively effortlessly, by providing feedback just around ten times. This is over an order of magnitude less than that required by the popular learning-from-honn preferences baselines. The supplementary video and source code are available at: https: / //tp///uannnssns//a//e github. io/ paaps/arrbbb /rba.
introduce the notion of Relative Be-
INTRODUCTION
A central problem in building versatile autonomous agents is how to specify and customize agent behaviors. Two representative ways to specify tasks include manual specification tions, and reward learning from trajectory comparisons. In the former, the user needs to provide an exact description of the objective as a suitable reward function to be used by the agent. This is often only feasible when the specification can be done at high level, e.g. in symbolic terms (Russell & Norvig, 2003) or by providing a symbolic reward machine or domain model (Yang al., 2018; Illanes et al., 2020; Guan et al., 2022; Icarte et al., 2022), or by giving a natural language instruction (Mahmoudieh al., 2022). Instead of requiring users to give precise task descriptions, reward learning from trajectory comparisons ka. reinforcement learning from human feedback or RLHF) learns a reward function from human preference labels over pairs of behavior clips (Wilson al., 2012; Christiano al., 2017: Ouyang et al., 2022; Bai et al., 2022a), or from numerical ratings on trajectory segments (Knox & Stone, 2009; MacGlashan et al., 2017; Warnell et al., 2018; Guan et al., 2021; Abramson et al., 2022) or from rankings over set of behaviors (Brown et al., 2019).
reward func-
To illustrate the distinct characteristics of the two objective specification methods, let us consider Humanoid control task in benchmark DeepMind Control Suite (Tunyasuvunakool et al., 2020).

---

Published as a conference paper at ICLR 2023
The default symbolic reward function in the benchmark is carefully-designed linear combination of multiple explicit factors such as moving speed and control force. By optimizing the reward, the agent will learn to rum at the specified speed but in unnatural way. However, to further specify the motion styles of the robot (e.g., to define a human-like walking style), non-expert users may find it hard to express such objectives symbolically since this involves tacit motion knowledge. Simi- larly, in natural language processing tasks like dialogue (Ouyang et al., 2022) and summarization (Stiennon et al., 2020), it can be extremely challenging construct reward functions purely with explicit concepts. Hence, for tacit- seriit-kwwledgee tasks, people usually resort to reward specification via pairwise trajectory comparisons and learn a parametric reward function (e.g., parameterized by deep neural networks). While pairwise comparisons are general enough to work for any setting, due to the limited information a binary label can carry, they are also an impoverished way for humans to communicate their preferences. Thus, treating every task at hand as a nnoll-nooledge one and limit- ing reward specification to binary comparisons can be unnecessarily inefficient. Moreover, since the internal representations learned within neural networks are typically inscrutable, it's unclear how learned reward model can be reused to serve multiple users and fulfill a variety of goals.
In short, symbolic goal specification is more straightforward and intuitive to use, but it offers limited expressiveness, making it more suitable for sppiiit-knowledge tasks. Reward learning from trajectory comparisons, in contrast, of- fers better expressiveness, but it is more costly and less intuitive to use. However, in many real-world scenarios, user objectives are neither completely explicit nor purely tacit. In other words, although human users may not be able to construct a reward function symbolically, they still have some idea of how the AI agent can change its behavior along certain meaning- ful axes of variation to better serve the users. In the above Humanoid example, even though the users can not fully define the motion style in closed-form manner, they may still be able to express their preferences in terms of some nameable concepts that describe certain properties in the agent behavior. For instance, the users of a household humanoid robot may want not only walk in natural way but also walk more softly and take smaller steps at night when people are sleeping.
Lane
Toger
Haar
Figure 1: Visualizing behavioral attributes of
Walker and Lane-Change The behavioral at- tributes of other domains are shown in Fig. 3 in Appendix A.
Motivated by the observations above, we introduce the notion of Relative Behavioral Attributes (RBA) to capture the relative strength of some properties' presence in the agent's 's behavior. We aim to learn an attribute-parameterizedd reward function that can encode complex task knowledge while allowing end users to frrely increase or decrease the strength of certain behavioral attributes such as "step size" and "movement softness" (Fig. 1). This new paradigm of reward specification is intended to bring the best of symbolic and pairwise trajectory comparison approaches by allowing for reward feedback in terms of conceptual attributes. The benefits of explicitly modeling RBAs are obvious: (a) it offers semantically richer and more natural mode of communication, such that the hypothesis space of user intents or preferences can be greatly reduced each time the system receives attribute- level feedback from the user, thereby improving the overall feedback complexity significantly; (b) since humans communicate at the level of symbols and concepts, the learned RBAs can be used as shared vocabulary between humans and inscrutable models (Kambhampati et al., 2022), and more importantly, such shared vocabulary can be reused to serve any future users and support diverse set of objectives.
The concept of relative attributes has been introduced earlier in other areas like computer vision (Parikh & Grauman, 2011) and recommender systems (Goenka et al., 2022). Though they share some similarities in motivations and definition to our relative behavioral attribute, the key difference is that relative attributes there only capture properties present in a single static image, while RBAs aim to capture properties over trajectories of behavior, wherein attribute may be associated with either static features at a certain timestep (e.g., step size of a walking agent) or cepporally-extendee features spanning multiple steps (e.g., the softness of movement). Additionally, we need to consider
2

---

Published as a conference paper at ICLR 2023
how to connect the captured attributes to valid reward function. In this work, we propose two generic data-driven methods to encode RBAs within a reward function. On four domains with nine attributes in total, we show that our methods can learn to accurately capture the attributes from roughly 200 labelled behavior clips. Once the attribut-pprameteriiie reward is learned, any end user can produce diverse agent behaviors with relative ease, by providing feedback just around 10 times. This offers a significant advantage over the (Christiano et al., 2017) that requires hundreds of binary labels from the user per task.
baseline
2 RELATED WORK
In general, direct reward specification is the most commonly employed approach. It is used in almost all simulators with engineered reward functions and in industrial applications such as self-driving vehicle control systems. Yet, there still some challenges associated with it, such as extensive requirement for sensory instrumentation, incomplete (sub)goal specification (Guan et al., 2022), reward exploitation (Lee et al., 2021) and the expressiveness issue mentioned in the previous sec- tion. These impediments motivate the idea of learning reward from states or trajectory comparisons (Christiano et al., 2017; Warnell et al., 2018; Zhang et al., 2019; Lee et al., 2021), which lever- ages the exceptional representational capacity of neural networks but tends to suffer from high data complexity. Another alternative symbolic reward specification is imitation learning or inverse re- inforcement learning (Schaal, 1996; Ng et al., 2000; Abbeel & Ng, 2004). However, this is usually infeasible for non-expert end-users, as it requires the user have sufficient knowledge and proper hardware setup in order to teleoperate the agent. Even if a pre-collected large-scale behavior dataset provided to the user, the process of finding the target behavior trace(s) from the large dataset can still be frustrating.
The introduction of RBAs aligns with the broader intent of building symbolic interfaces as a middle layer for humans to communicate effectively with the agent (Bobu et al., 2021; Guan et al., 2021; 2022; Silver et al., 2022; Zhang et al., 2022; Bucker et al., 2022; Cui et al., 2023) or for the agent to explain to humans (Kim et al., 2018; Sreedharan et al., 2022; Kambhampati et al., 2022). Lee et al. (2020) utilize relative-atrriutte information in robot skill learning, but their GAN-based formulation restricted to static visual attributes and is not applicable to temporall--extendee concepts.
This paper adopts a similar setup to works that learn diverse skills or motion styles from large- scale offline behavior datasets or demonstrations (Lee & Popović, 2010; Wang et al., 2017; Zhou & Dragan, 2018; Peng et al., 2018b; 8b; Luo er al., 2020; Chebotar et al., 2021; Peng et al., 2021). These works emphasize conditioned on skill latent codes. Since the latent codes are inscrutable to humans, for each new task, the user must specify the desirable agent behavior by constructing an engineered symbolic reward and use it to train a separate high-level policy that controls the low-level controller. Our methods are complemented by existing diverse-skil learning methods because skill priors (i.e., pre-trained low-level controllers) allow us to optimize the behavioral reward more efficiently. More recently, there have been works in diffusion-based text-to-motion animation generation (Tevet et al., 2022: Guo et al., 2022). They are similar to this work in the sense that we both allow humans to control the agent behavior through explicit concepts. However, they do not support fine- grained control over the strength of individual behavioral attributes, and their works are not applicable to physics- bassed character control.
modeling a variety of reusable motor skills by learning low-level controller
3 PROBLEM SETUP
Personalizing agent behaviors at skill level. We assume the users are interested in customizing agent behaviors at the level of skills (e.g., making a lane change, taking step, picking up an object). A skill is a solution (i.e., policy) to an episodic task in an indefinite soltiintte-oriioonn discounted MDP M (S, A, R. P. P,7)), where S is the set of states, A is the set of actions, R. S x A + I is the provided reward function,
x A x + [0, 1] is the transition function and y is the discount factor. An episode terminates whenever the agent reaches an absorbing terminal state or exceeds fixed number of time steps T. An agent M is supposed to find policy (σ : S + A) that maximizes E
the expected return E = <rr,), E R. When executing a policy, the agent is initialized to some initial state 8p E S at the beginning of each episode; and then at each succeeding time step the
3

---

Published as a conference paper at ICLR 2023
agent interacts with the environment & by taking action at A and receiving the next state S141 E S. In order to personalize agent behavior, rather than assuming that the reward function R is supplied by the environment, require the reward function to be specified by the end user.
Given a behavior clip or a trajectory T = {(5p, ag), ..., (s), a})}, where is the length of the trajec- AA captures the strength of the presence of certain property exhibited in r. It assumes that a mapping Ç can be established to map any (a, pair to a real value that reflects the relative strength of in т. In this work, our goal is construct reward function that allows end users to iteratively adjust attribute strengths presented in the agent's behavior.
tory, relative behavioral attribute
Learning to model RBAs from offiine behavior datasets. RBAS essentially semantically capturing different ways to carry out a task. Ideally, the training data for the reward function should contain clips of diverse skill behaviors. Although we used synthetic data in this study, there are many publicly accessible behavior corpora, such as the Waymo Open Dataset for autonomous driving tasks (Ettinger et al., 2021) and large-scale motion clips data for character control (Peng et al., 201 8a; 2022). Note that we do not expect the offline dataset to exhaustively cover all possible skill behaviors, as the reward function should generalize to unseen configurations. Also, considering that action information is not always available, to be more general, we assume the training dataset D only consists of state-only trajectories. Last but not least, as a common setting, we assume the embodiment of the agent is not significantly different from that in D.
Reward learning from trajectory comparisons. Methods for reward learning from trajectory comparisons construct a reward function by learning to infer the user's latent preference from a set of ranked trajectories {(r) (r), -²)).
Typically, the user preference is modelled according to the Bradley-Terry model (Bradley & Terry. 1952):
exp E, r (sl, (a!1) Eie(1a)
P - exp E. r (ot, [a:])
where 7² > 7² denotes the event that the user prefers 71 over 72, is a parametric learnable reward function, and [a]] means the action input is optional. A cross-entropy loss is typically used for optimization:
Lossp(r) =-
Σ log) 71 v(2) log P p|- } +]
(2)
(F)DA
or r¹ and / [11, (), (0. 1), (0.5, 0.5) are possible preference labels indicating 7| 72, 72 T! are equally preferred, respectively. The set of preference labels Dp can be collected beforehand (Brown et al., 2019) through active queries to the user in an online manner (Wilson et al., 2012; Christiano et al., 2017). The latter is often referred to as preference-based reinforcement learning, or PbRL for short. Most PbRL methods build upon the Bradley- Terry model but differ in the query strategies and how the network is initialized (Lee et al., 2021; Park et al., 2022; III & Sadigh, 2022; Ren et al., 2022).
where y
4 METHODOLOGY
4.1 PERSONALIZING AGENT BEHAVIOR VIA RELATIVE BEHAVIORAL ATTRIBUTES
Our framework involves two phases, namely, learning an attribute parameterized reward function and interacting with the end user.
Learning attribute parameterized reward function (no interaction with the end user). This phase is supposed to learn a reward function that internally learns family of rewards that correspond to behaviors with diverse attribute strengths. By varying the input attribute configurations, end users will be able to find the preferred "reward member" and obtain desired behaviors. In Section 4.2 and Section 4.3, we will present two methods that can learn such a reward function given a subset of labelled trajectories from offline behavior dataset D. We assume that the agent builders are the ones who provide the training labels (e.g., the engineers of autonomous vehicles, and the developer of virtual characters). In some extreme cases, if novel concept has to be learned. the training labels may also come from the end user. Also note that, this step can be skipped if the RBAs have already been captured by a learned reward function.
4

---

Published as a conference paper at ICLR 2023
Supporting end users in the loop. Once attribute parameterized reward function is learned, any incoming users can leverage it to personalize the agent behavior through multiple rounds of query. In each round of interaction, the agent presents the user trajectory, sampled according to the policy that optimizes the current reward function. Then the user provides a feedback on whether the current behavior is desirable; and if not satisfied, the user can express the intent to increase or decrease the strength of certain attributes. The concept-level feedback is set of attribute-feedbar pairs {(a, h)}, where is the attribute of interest, and h is a binary value indicating whether the user wants to increase or decrease a's strength. In this work, we consider two types of attribute representation, namely the index of in a list of known attributes or a natural-language description of a. Upon collecting feedback from the user, the agent will adjust the reward function and update the corresponding policy. This human-agent interaction process repeats until the user is satisfied with the latest agent behavior.
In the next two sections, we will elaborate on two candidate architectures for the attribute parame- terized reward function. We note that both architectures support the same type of user interaction as outlined above.
4.2 МETHOD I: MODELING BEHAVIORAL ATTRIBUTES BY ESTABLISHING GLOBAL RANKINGS
The learning process of Method 1 consists of two steps. In the first step, we learn an attribute strength estimator &o (parameterized by о) that can map any given attribute a and trajectory to a real-valued score that measures the relative strength of attribute a in T. Here, So is essen- tially establishing a global ranking among all possible behaviors according to any given attribute a. Hence, we denote this method as RBA-Global. Then in the second step, given a finite set of at- tributes A, we learn dense reward function r&s (s|u, = (u. ,..., of*)), where 02 is called the target attribute-score vector, v/ is the target strength of attribute a, and k |Al is the total number of attributes. Tg is expected to satisfy that, when re (-1ee) is optimized, the agent is able to sample a trajectory 7', such that for any aa E A, we have (T',,a;) ~ 2, Hence, with learned reward function re. the agent can produce diverse behaviors by varying the input target attribute-scor vec- tor 12- As example, let consider the humanoid household robot domain with two attributes foo "softness of movement", .. : "step size"). By setting the target attribute vector ie to be (u 1.5, o 2.0), we expect the agent that optimizes r3p to be able produce trajectory r' with a "softness" score Ca (T', (T',a0) approximately equal to - 1.5.. A graphical illustration of the overall architecture can be found in Fig. 4 in Appendix A. In the remainder of this section, we will explain how So and rgg can be learned from the offline behavior dataset D.
The problem of learning an attribute strength estimator Co is essentially learning-to to-rnkk problem. Specifically, we assume we are given (normally by the agent designers rather than the end users) set of state-only trajectories {r} and their orderings according to different attributes {(7 > - >
> 7 [a)), or a set of ranked trajectory pairs Dj = {(r' > r3(a)), where a E A is one of the attributes in the domain, (7' > -3(aa) represents the event that the attribute a has a stronger presence in trajectory r' than that in trajectory r³, and N < [D] is the length of the ranked trajectory sequence. We propose to employ a modified state-only version Bradley- model (Eq. 1), in which rather than assuming that the ranking is governed by the latent user preferences, we assume the ranking is determined by the given attribute a:
exp Ex fo (1sh, Eal) Po [ Eie(1e) exp E, Ja (1si,c.])
(3)
where fo is an attribute conditioned ranking function with parameters o. [.-1 is the vector con- catenation operation, and e, is the embedding of attribute a. The strength of any attribute in a trajectory is given as (T(T ar) = E.er So ((s.ea))) ea.]). Recall that we consider two types of attribute representation, namely attribute index and natural language description. Accordingly, ca can either be a one-hot vector or a sentence embedding generated by any pretrained natural language sentence encoder like Sentence-BERG (Reimers& & Gurevych, 2019). Since different behaviors may result in trajectories of varying lengths, in Eq. 3 we do not require the two trajectories to have the same size. Given the training dataset Dr, fo can be trained via a cross-entropy loss similar to the one in Eq. 2. For numerical stability, in practice, we also clip the values of E. fo ([s}, eal) (we used [-20, 20] for all the attributes in our experiment). With learned attribute strength estimator Ço and finite
5

---

Published as a conference paper at ICLR 2023
set of k attributes, the agent behavior in any trajectory T can be characterized by attribute vector ((r) =
(T 0x.)).
The problem of learning ro (s, ut]) can also be cast to a learning-to-rante or preference modeling problem, wherein the reward function is supposed to give higher cunulative rewards to trajectories that have attribute strengths closer the targets in vz- Specifically, given ut, for any two trajectories and Tj from the offline behavior dataset D, the preference label (p(r', T² vz) is given as:
if [|][(()) vella [|((7)) 0z112 & if [|]((')) v.lla [|((T)) vall2 otherwise,
b 04)
(4)
no ordering
where e, is a small slack variable. To train rø, we can randomly sample a set of triplets from D, namely {(7", [.)]. By treating 7" as the target behavior, we can generate a set of training labels (4((T), . 14 = ))3 for ry. In short, To can be trained as a standard state-only preference-baser reward function according to Eq. 1 and Eq. 2 but with preference labels given by the extracted attribute strengths (i.e., by (.). Note that, unlike the training of G. the training of ro does not require any human provvdee labels.
Once a reward function re is learned, the end user can use to specify agent behavior by simply tuning the attribute strength values in the input target attribute vector er. In our realization, we implement the process of finding the target attribute score as a process of performing binary search in real space (details can be found in Appendix A.3). The process of personalizing agent behavior through ro is highly intuitive because ro handles the complex tacit parts of the problem internally (e.g., how walk naturally and realistically with the constraints of softness and step size) and only relies on the end user to set the explicit parts. Additional discussion on alternative ways to leverage So without learning a reward function can be found Appendix A.2.
4.3 METHOD 2: MODELING BEHAVIORAL ATTRIBUTES BY CAPTURING MINIMALLY VIABLE LOCAL CHANGES
One limitation of RBA-Global is that, it requires the total number of encoded attributes to be finite because the size of the target attribute vector it grows with the number of attributes. This may limit the scalability of RBA-Global. In this section, we will introduce a more extensible method that can potentially encode an arbitrary number of attributes. The key motivation that RBA-Global, the user never directly manipulates the attribute scores. Instead, we can skip the explicit modeling of attribute strength (i.c., learning (o) and directly learn a behavior-editing reward function.
Specifically, given trajectory and the corresponding human feedback (a, h). our goal is to construct a reward function ra h, Te) that gives higher cumulative rewards to trajectories that have some minimal but noticeable change in in the direction specified by h while keeping other unmentioned attributes unchanged (or minimally changed). We refer to such minimal but noticeable changes as minimally viable local changes, and the queried trajectory Ta the anchor trajectory. Accordingly, we denote this method as RBA-Local.
To learn rg, we assume we are provided (again, normally by the agent builder and not the end user) set of trajectory pairs Dj (TT. Te, a, h)), where T is trajectory that reflects some minimally viable local changes to the anchor trajectory Te in terms of the attribute a and the direction h. ra trained to prefer Tr over other negative samples (such as trajectories that make excessive changes to T., or trajectories that are not significantly different from Te, or trajectories that make changes to unspecified attributes). In practice, we select the negative samples by randomly sampling from the behavior dataset D. Again, re can be formulated as a modified Bradley-Terr Terry model:
exp Exer, (Is, ea, h, ((Te))
Po |T. > a, exp S.pE... Ta ([5, Co. ((Ta))) exp Exer, To hi,
(5)
where Tn is a negative sample (i.e., trajectory), [-] is the vector concatenation operation, ea as in RBA-Global can be either an one-hot representation of attribute by Sentence-BERTI and Φ(T.) is sequence encoder (e.g., an LSTM (Hochreiter & Schmidhuber, 1997)) that encodes the anchor trajectory To to compact latent representation. Note that d(-) is a sub-module of ra and it's jointly optimized with rg- Since ra is essentially a preference-basee reward function, it can be optimized by employing a cross-entropy loss as in the one in Eq. 2. We
sentence embedding output

---

Published as a conference paper at ICLR 2023
note that our computational framework is similar to the Prompt-DT (Xu et al., 2022) in the sense that we both take a reference trajectory as a "prompt" to the model to obtain conditioned outputs. But reward
instead of trying to replicate the behavior in the "prompt" trajectory as in Prompt-DT. function learns to modify the prompt in a controlled way. In a more recent work (Liu et al., 2023), similar ideas have been shown be effective in refining language model outputs through sequences of local changes informed by human feedback.
Compared RBA-Global (Sec. 4.2) which requires a full specification of all the attributes' strengths. the reward function in RBA-Local only takes one attribute as input at time. This de- sign is appealing as it offers better scalability and it affords the development of big universal behavioral concept "encoder"". Nevertheless, RBA-Local still has the following shortcomings: (a) is less efficient than RBA-Global in searching for the target behavior because it can only make minimally viable changes to the presented behavior; (b) The training data for RBA-Local is harder to collect. Unlike RBA-Global, where any random subset of trajectories exhibiting distinct behav- iors can be used as the training samples, in RBA-Local, the agent builder must carefully pick pairs of trajectories that reflect local changes. Also, the judgement of how much variation constitutes a minimally viable change can be fairly subjective.
EMPIRICAL EVALUATION
As a proof of concept, we demonstrate the effectiveness of our methods in diverse set of four domains with nine behavioral attributes that are depicted Fig. and Fig. 3:
Walker. This environment corresponds to scenario that involves 2-legged home-service robot walking around the house to perform household tasks. The users may want the robot to walk more softly at night. This environment is also related to physical character control scenarios wherein we want the character to move in a sneaky way. Two attributes are considered here: (a) step size; (b) softness of movement.
We consider a virtual character control scenario wherein we want a simulated arm to mimic the ways of human lifting objects. When humans, especially elders and children, are lifting heavy objects, their movement can be unstable. Hence, we consider two attributes: (a) moving speed of the arm; (b) instability of the movement.
Manipulator.
Lane Change. We consider driving scenario wherein the rider would want to change the lane- changing behavior of the cab get a more pleasant experience. Two attributes are used for evalu- ation: (a) the sharpness of steering: this attribute corresponds to how sharp a turn the agent makes while changing lanes: (b) distance the following vehicle: this attribute about the distance be- tween our agent and the following car at the moment when our agent starts making the lane change.
Snake Concertina. In this task, the agent is supposed to control a virtual snake to imitate diverse concertina styles of real snake's ' locomotion. There are three relevant attributes in concertina locomotion: (a) width of the bend (i.e., the maximal width that the snake occupies); (b) compression (i.e., how much the snake's body is compressed when it is moving); (c) speed of movement.
More details of the evaluation domains can be found in Appendix A. In our experiments, all the behavior clips are generated either by hard-coded motions or by using reinforcement learning with sophisticated reward designs and hard-coded constraints.
Lano-Change
Manipulator
Suake
Walker
Method
(std)
(std)
AF (st)
SR
AF (std)
RBA Global
0.95
3.95 (2.43)
1.0 2.8 (1.21)
0.85
4.17 (1.85)
1.0
3.75 (1.47)
RBB--GlobalLL
1.0 3.05 (2.06)
1.0
2.5 (1.32)
0.8
6.38 (5.03)
0.95 3.78 (2.25)
RBA Local
0.7 7.07 (3.49)
0.7
12.23 (5.75)
0.55
6.45 (3.82)
0.95
5.47
RBA-Local-L
5.78 (4.04)
10.75 (6.35)
0.6
4.17 (2.11)
0.9
(3.6
PbRL
162.3 (184)
0.6
159.5 (188.87)
N/A
1.0 87.6
Table 1: SR - Success Rate; AF- Average Feedback (when success); L Language
7

---

Published as a conference paper at ICLR 2023
Caftices
Gallea 877
Safense MeT
Sefrran
Salaere S4T
.X.
...
(SHATTE ARM
Figure 2: A step-by-step visualization of the interaction process showing how the agent's be- haviors change according to sequences of user's feedback in the Walker domain. The two se- quences on the left showcase RBA-Global and the sequences on the right showcase RBA-Local. The upper and bottom rows represent the success and failure cases respectively. The attribute strength scores above each image frame are given by hard-coded proxy measures, which are in range of [0, 1]. For RBA-Global, we also show the corresponding reward function and input o (target step size, target softness). A more detailed visualization is presented in the supplemen- tary video.
5.1 BASELINE AND RESULTS
For comparisons, use the PbRL algorithm proposed in (Christiano et al., 2017; Lee et al., 2021), which learns a reward function from human preference labels collected by making active queries. Considering that methods assume the additional access to the offline behavior dataset D, made a couple of modifications to make PbRL. into stronger baseline. The most important one is that, optimize the most recent reward function and update the agent's behavior after each query, rather than applying reinforcement learning, we use the policies extracted from D. Specifically, we stochastically sample large set of rollouts by executing the policies we used synthesize dataset D, and the rollout with the highest cumulative rewards is set as the agent's 's latest behavior. In practice, these policies can also be unsupervisedly learned from This is identical to the use of skill priors as in (Peng et al., 2018b; Pertsch et al., 2020; Luo et al., 2020; Peng et al., 2022), which first learns a diverse set of natural and plausible motions from offline behavior datasets reduce online computation. Besides, for PbRL, we also experimented with different query strategies and considered reusing previously trained reward models. For the sake of simplicity, we only report the best PbRL performance achieved at different setups.
As the evaluation metric, we count the number of human feedbacks (i.c., the binary preference labels in PbRL and the attribute-level feedback in methods) needed to produce the target behavior. For each domain, we randomly sample 20 behavior configurations as targets, and the selected targets unseen in the behavior dataset D. trial is considered behavior has ground truth attribute strength or proxy score that falls within a certain range of the target value. We buckets. A trial is deemed unsuccessful if the agent fails to produce the target behavior within user-affordable number of feedbacks (we used 500 in our evaluation). The results are shown in Table 1, where we add "L." as suffix to the names of variants that use language embedding at- tribute representation. Results show that with RBAs, users can obtain desired agent behavior much more efficiently than with PbRL. The upper row in Fig. 2 showcases how the user can obtain de- sired agent behaviors through sequences of attribute feedback with both methods. The interaction processes suggest that the attribute- paabbbte-pprmmeteizzeee reward is able to modify agent behaviors mean- ingfully in directions that are informed by RBAs. For full information of the interaction processes in all domains, we encourage the reader to check out the supplementary video.
success if the generated agent
threshold that roughly divides the strength of each attribute into five ten
Analysis of failure modes. Results suggest that both methods have a lower success rate when trying generate behaviors close to the extremes (e.g., moving very softly or very recklessly). The bottom row in Fig. 2 shows two failure cases. RBA-Global, the reward function fails to
'Supplementary video https: http:////g/aauuuuuuu wtt.tth.......///eetttt
8

---

Published as a conference paper at ICLR 2023
produce behavior with larger step size when we increase the target step size score in it from 18.51 to 18.62. This disrupts the binary search process (Appendix A.3) and causes the system to get stuck. Similar failure patterns also be observed in other domains. For example, in the Lane- Change domain, the agent fails to increase the distance to the following vehicle when the sharpness (proxy) score is 0.09. Since RBA-Global is composed of two learned models, namely the attribute strength estimator So and the reward function re. we also examine them separately to see which module contributes more the failures. By visualizing the outputs of &o and the corresponding proxy ground truth (Fig. 5 in Appendix A), we observe a positively correlated or sometimes linear relationship between them, suggesting that &o can accurately capture the attribute strength even at the two extremes. This observation verifies that the ineffectiveness of RBA-Global is mainly caused by ro's 's failure to recover behaviors specified is not surprising since Co only needs to learn one global ordering per attribute while ra has to learn almost an infinite number of orderings given various imput targets U,. Future work can explore better training paradigms and more expressive model architectures for re. In terms of the failure modes of RBA-Local, it sometimes fails to edit the behavior in the anchor trajectory Te and simply produces the same behavior in Tz- As shown in the right bottom plot of Fig. 2, the agent gets stuck when the user wants to increase the softness when the agent is taking large step (proxy score: 0.954). Also, RBA-Local tends have a lower success rate than RBA-Global, indicating that its more complex formulation and architecture might affect its performance.
the target attribute-score vector et- Note that this
5.2 ADDITIONAL DISCUSSION
To get more insights into the number of labels needed to learn an accurate attribute ranking function or reward function in our methods, we conduct an extra experiment in which we train the model with different numbers of samples uniformly sampled from the training set, and evaluate each model on a held-out testing set. Results (Appendix A.5) show that to simultaneously learn two attributes, RBA- Global needs around 200 labelled trajectories (and the orderings among them) and RBA-Local needs around 200 (Te: Tr) pairs, which is a reasonable number. Also, the cost of learning a reward function is amortized when we continue to use it to support incoming users over its lifetime.
Recall that in the main experiment, we consider a trial as a successful one the difference between the agent's behavior and the target behavior is lower than threshold. One interesting thing to is whether our methods can achieve even higher-precisiin additionally experiment with threshold value that roughly divides the attribute strengths into five to ten times more buckets. As expected, a more restrictive threshold reduces the performance of all algorithms, but methods still have significant advantage in terms of feedback efficiency. Results are shown in Table 2 in Appendix A. Note that this performance degradation was expected because the control precision we wanted to achieve in this experiment is higher than what we set in the training data (e.g., the precision corresponds to changes that are smaller than the minimally viable local changes defined in the training data for RBA-Local).
control over the agent's ' behavior. Hence,
6 CONCLUSION
In this paper, we introduced the notion of relative behavioral attributes which allows users to pro- vide symbolic feedback (i.e., their intent increase/decrease attribute strength) to efficiently tweak and get desired agent behavior. We proposed two approaches and demonstrated their effectiveness through experiments in varied set of domains. For future works, apart from the limitations we discussed earlier, it would be interesting if we could develop methods that combine the strengths of the two approaches proposed in this work. Also, currently, we use the sentence embedding of each attribute only as an alternative to the one-hot vector. However, it would be beneficial to make better use of the semantic structure inside the sentence embeddings.
Furthermore, it would be useful to explore the use of RBAs outside of continuous control tasks. For instance, for AI chatbots, we may construct rewards to capture not only binary attributes like helpfuiness and harmfulness (Bai et al., 2022b) but also abstraction levels of the text (e.g., scientific concepts need to be explained differently to kids and researchers) or the tonality of the response (ranging from casual to more formal professional way). Inspired by recent attempts build reward-driven vision models (Pinto et al., 2023). it would be also interesting to investigate whether RBAs can facilitate reward construction for fine-tuning models in computer vision tasks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research is supported in part by ONR grants N00014-16-1-2892, N00014-18-1-2442, N00014-18-112840, N00014-9-1-2119, AFOSR grant FA9550-18-1-008 DARPA SAIL-ON grant W911NF19-2-0006 and a JP Morgan AI Faculty Research grant.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, pp. 2004.
Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, et al. Improving multi- modal interactive agents with reinforcement learning from human feedback. arXiv preprint arXiv:2211.11600 2022.
Junwen Bai, Weiran Wang, and Carla P Gomes. Contrastively disentangled sequential variational autoencoder. Advances Neural Information Processing Systems, 34, 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2tti::2004.08662. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073 2022b.
Andreea Bobu, Chris Paxton, Wei Yang, Balakumar Sundaralingam, Yu- Chao, Maya Cakmak, and Dieter Fox. Learning perceptual concepts by bootstrapping from human queries. arXiv preprint arXiv:2111.05555 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, and Rogerio Bonatti.
Latte: Language trajectory transformer. arXiv preprint 20XX::220080091888 2022.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749, 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, and Dorsa Sadigh. no, to the right"-onlin language corrections for robotic manipulation via shared autonomy. arXiv preprint arXhv:2301.02555, 2023.
Scott Ettinger, Shuyang Cheng. Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 17199979, 2021.
Sonam Goenka, Zhaoheng Zheng, Ayush Jaiswal, Rakesh Chada, Yue Wu, Varsha Hedau, and Pradeep Natarajan. Fashionvlp: Vision language transformer for fashion retrieval with feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14105-14115, 2022.
10

---

Published as a conference paper at ICLR 2023
Lin Guan, Mudit Verma, Suna Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human-guute guidedd reinforcement learning with explanation and context-aware data augmentation. Advances in Neural Information Processing Systems, 34:21885-218977 2021.
Lin Guan, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging approximate symbolic mod-
els for reinforcement learning via skill diversity. arXiv preprint 22XXx:22022.28886 2022.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A Mcllraith. Reward ma- chines: Exploiting reward function structure in reinforcement learning. Journal of Artificial In- telligence Research, 73:173-208, 2022.
Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. In 6th Annual Conference Robol Learning, 2022. URL https: //pp///eeee........e fest/forumg se-idddisSSS IIIITTXXLLL.
León Illanes, Xi Yan, Rodrigo Toro Icarte, and Sheila A. Mcllraith. Symbolic plans as high-level in- structions for reinforcement learning. In Proceedings of the Thirtieth International Conference Automated Planning and Scheduling, Nancy, France, October 26-30, 2020, pp. 540-550. AAAI Press, 2020.
Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In AAAI, 2022.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of fifth international conference Knowledge capture, pp. 9-16, 2009.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficientt interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2/06.05091, 2021.
Lisa Lee, Ben Eysenbach, Russ R Salakhutdinov, Shixiang Shane Gu, and Chelsea Finn. Weakly- supervised reinforcement learning for controllable behavior. Advances in Neural Information Processing Systems, 33:2661-2674 2020.
Seong Jae Lee and Zoran Popović.
Learning behavior styles with inverse reinforcement learning. ACM transactions on graphics (TOG), 29(4):1-7, 2010.
Edouard Leurent. An environment for autonomous driving eeciision-makin. https://gichie htmm/eeeurent/tiee hwwpttttwhhwhhwheeeteeeee
2018. 2018.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02670 2023.
Ying-Sheng Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen, and Wei-Chao Chen. Carl: Con- trollable agent with reinforcement learning for quadruped locomotion. ACM Transactions Graphics (TOG), 39(4):88-1, 2020.
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 2285- 2294. JMLR. org, 2017.
11

---

Published as a conference paper at ICLR 2023
Parsa Mahmoudich, Deepak Pathak, and Trevor Darrell, Zero-shot reward specification via grounded natural language. In ICML, 2022.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, vol- ume 1, pp. 2. 2000.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow 2022.
instructions with human feedback. arXiv preprint
Devi Parikh and Kristen Grauman. Relative attributes. In 201 International Conference on Com- puter Vision, pp. 503-510. IEEE, 2011.
Jongjin Park. Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint arXiv:2203. 10050, 2022.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018a.
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv: Rein- forcement learning of physical skills from videos. ACM Transactions On Graphics 1-14, 2018b.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adver- sarial motion priors for stylized physics-based character control. ACM Trans. Graph., 40 (4), July 2021. doi: 10. 01145/3450626.3455666 URL nttp:://dot aww.aacm.or.....ooooomoooooooooa 3450626. 3459670.
acm. wccm.org/1..100// 1145//
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large- scale reusable adversarial skill embeddings for physically simulated characters. arXiv preprint arXiv:2205.01900 2022.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. 1944, 2020.
André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai Tuning computer vision models with task rewards. arXiv preprint arXiv:2302.08822, 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL https::/arxxt org/abs /tabb//190 1908 10084.
Zhizhou Ren, Anji Liu, Yitao Liang Jian Peng. and Jianzhu Ma. Efficient meta reinforcement learning for preference-based fast adaptation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL nee.eeeeeweeeeeeeeeeeeeeeeeeeeee fortt/frum? dommmid===111 610wgeitnn.
https: /pppreeeeeeeeeeeeeeeeeeeeee
Stuart J. Russell and Peter Norvig. Artificial intelligence modern approach, 2nd edition. In
Prentice Hall series in artificial intelligence, 2003.
Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
Tom Silver, Ashay Athalye, Joshua B Tenenbaum, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Learning neuro-symbolic skills for bilevel planning. arXiv preprint arXiv:2206. 10680, 2022.
Sarath Sreedharan, Utkarsh Soni, Mudit Verma, Siddharth Srivastava, and Subbarao Kambhampati. Bridging the gap: Providing hos symbolic explanations for sequential decision makiiinnnmmmingggggg prob- lems with inscrutable representations. In International Conference on Learning Representations, 2022. URL https: http://oppeeevveeeeeeeeee net forum? d=o- 1 whddul
12

---

Published as a conference paper at ICLR 2023
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3024 2020.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa, dm_control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation diverse behaviors. Advances in Neural Information Processing Systems, 30, 2017.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25, 2012.
Mengdi Xu. Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Con- ference Machine Learning, pp. 24631-24645. PMLR, 2022.
Fangkai Yang, Daoming Lyu, Bo Liu, and Steven Gustafson. Peorl: Integrating symbolic planning and hierarchical reinforcement learning for robust decision-making arXiv preprint arXiv: 1804. 2018.
Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H Ballard, and Peter Stone. Leveraging human guidance for deep reinforcement learning tasks. arXiv preprint weXiv:199.099900 2019.
Ruohan Zhang, Dhruva Bansal, Yilun Hao, Ayano Hiranaka, Jialu Gao, Chen Wang. Roberto Martín-Martin, Li Fei-Fei, and Jiajun Wu. A dual representation framework for robot learning with human guidance. In 6th Annual Conference on Robot Learning, 2022.
Allan Zhou and Anca D Dragan. Cost functions for robot motion style. 2018 JEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 3632-3639. IEEE, 2018.
13

---

Published as a conference paper at ICLR 2023
Woller
Manipalator
Lane Change
Smake Cancertine
1-2
Figure 3: Visualizations of the evaluation domains and behavioral attributes.
A APPENDIX
A.1 DETAILS OF THE EVALUATION DOMAINS
Walker. Two attributes are considered in this domain: (a) step size; (b) softness of movement. To synthesize dataset and simulate human feedbacks, we use the moving speed and landing speed of the feet as proxies to measure the abstract concept softness or "sneaky". The environment is implemented based on the Walker-2d domain in the DeepMind control suite (Tunyasuvunakool et al., 2020).
Manipulator. We consider two attributes: (a) moving speed of the arm; (b) instability of the movement. The environment is implemented based on the Manipulator domain in the DeepMind control suite. We synthesize the behavior clips by hard coding the arm motion and by adding random noises in a controlled way. The purpose of conducting experiments in this domain is to demonstrate that our method can capture not only regular behavior patterns but also the irregularities. A Markov state is constructed by stacking five consecutive raw states of the environment to ensure it contains information about the irregularities.
Lane Change. Two attributes are used for evaluation: (a) the sharpness of steering: this attribute corresponds to how sharp a turn the agent makes while changing lanes; (b) distance to the following vehicle: this attribute is about the distance between our agent and the following car at the moment when our agent starts making the lane change. This environment is built on the highway environment in (Leurent, 2018), 8). Note that the environment is an image based domain, so the objective here is to verify that our methods can be scaled image inputs.
Snake Concertina. There are three relevant attributes in concertina locomotion: (a) width of the bend (i.e., the maximal width that the snake occupies); (b) compression (i.e., how much the snake's body is compressed when it is moving); (c) speed of movement.
A.2 ALTERNATIVE WAYS TO USE THE ATTRIBUTE STRENGTH ESTIMATOR FUNCTION IN METHOD 1 (RBA-GLOBAL)
Recall that in RBA-Global, given trained attribute strength estimator for and finite of attributes A, the agent behavior in any trajectory T can be represented by an attribute vector (G ag)), where k [A] is the number of attributes. If we assume the optimal policy is approximated via some parametric model (e.g., neural networks), we can actually skip the learning of the attribute parameterized reward function by viewing the attribute vectors as skill latent codes and learning a versatile policy conditioned on them. This is similar to the opera- tions in (Wang et al., 2017; Peng et al., 2022) but with a more structured latent code. Also, one might create sparse reward given at the end of episode by computing the distance between the extracted attribute vector and the target attribute vector. But such a reward is usually hard to optimize.
v(7)
The main reason choose to construct reward function is that we find it more general, since rewards can be optimized not only by RL, but also by other optimization based methods. Another consideration here is whether it is easier to learn a policy directly (e.g.. via BC IL) or to learn the reward first and then the policy (e.g., via IRL or PbRL). Prior empirical results suggest that the latter tends to be more robust solution.
14

---

Published as a conference paper at ICLR 2023
A.3 FINDING THE TARGET ATTRIBUTE SCORES WITH BINARY SEARCH
Binary search is highly efficient because it narrows down the search space by cutting it in half at each step. In order to apply binary search in the attribute space, need maintain beliefss about the upper and lower bounds of each attribute. In our case, the upper and lower bounds can be initialized the maximum and minimum attribute scores observed in the offline behavioral dataset D. where the scores are given by Ça. At each query step, the agent presents to the human the behavior corresponding to the median attribute value, i.e., Gusper-tOtoer and updates the beliefs accordingly after getting the feedback. the case of extrapolation, we can also go beyond the maximum and minimum attribute strengths, but this is no longer binary search.
A.4 ARCHITECTURES AND HYPERPARAMETERS
When one-hot representation is used to represent attributes, both fo and ra in RBA-Global employ a 3-layer fully-connected network with 512 hidden neurons the architecture. In RBA-Local, for the trajectory encoder, we use a 2-layer bi-directional LSTM with 128 as the hidden dim. The trajectory embedding, along with the input state and attribute, are fed into a 3-layer fully-connected network with 512 hidden neurons to compute the reward.
When sentence embeddings are used attribute representation, for both RBA-Global and RBA- Local, we increase the number of hidden neurons in fully-connected layers from 512 to 1024 due to the increase in the size of attribute embedding (size 768).
A.5 NUMBER OF TRAINING SAMPLES NEEDED IN OUR METHODS
Fig. 7 and Fig. 8 show the performance (on held-out testing set) of Method 1 (RBA-Global) and Method (RBA-Local) when different numbers of training samples are used. For RBA-Global, given an ordered trajectory pair (Ti >a 72), the ranking function is converted into binary classifier that predicts the ordering of the given pair. The performance is measured in terms of the accuracy of this binary classifier. Similarly for RBA-Local, the performance is measured by converting Equation 5 to binary classification problem where the function predicts whether trajectory is the target trajectory or not. Note that the sample complexity of the two methods is not directly comparable because their training samples are in different formats.
A.6 DISCUSSION ON UNSUPERVISED DISCOVERY OF BEHAVIORAL ATTRIBUTES
As a preliminary attempt in this study, we explored the possibility of unsupervised discovery of concepts or properties. We applied state-of-the-art disentangled sequential variational autoencoder method, C-DSVAE (Bai et al., 2021), to learn to encode behaviors the offline behavior dataset D. Though the behavior/motton embeddings given by C-DSVAE are able to cluster visually similar traces together, it still has difficulty capturing subtle but meaningful differences in behaviors. More importantly, it fails to establish any meaningful ordering among behaviors. As an example (Fig. 6), we visualize the variations encoded in specific dimension in the latent space of the Lane ChannChange domain. This observation is consistent with the current trend in vision-text research, which suggests unless additional supervision signals are provided, representations developed by neural networks are not guaranteed to capture semanties that make sense to humans. This preliminary study confirms the necessity to employ supervised learning for behavioral attribute modeling.
15

---

Published as a conference paper at ICLR 2023
Encodinn by establishing global rankings
attributes
Encoding behavioral attribbtees by capturing minimally vieble changes
Extahilsting nude--canttooeed glota!
Stop 2: Learn proferance based reward socording to
to Incrasan of decrnsss
e-ehe
anchoridderence trajectory
trajcctory
fa($/)) (s,elo
r(sive)
reward: r(s H
abt atriloute
attibutte strangth
tarpet attributt vector e'stpp sizal: 15.5. "softrreen"
a trajootor that makes minimall viable local changos
j..)
Figure Overview of Method (RBA-Global) and Method (RBA-Local).
Linng
Step
Smake Concertra
Amtioris sethees
Fooy Grensd
Procy Gmonc Tron
4S Soore
Figure 5: The relationship between Co and the hand-designed proxy ground truth. The attribute strength scores computed with one-hot vectors used the attribute representation. We can observe a similar pattern when language embeddings are used.
Lane-Change
Manipulator
Soake
Walker
Method
SR
AF (std)
SR AF (std)
SR
AF (std)
SR AF (std)
RBA-Global
0.60
6.75 (2.98)
0.80 7.00 (2.45)
0.55
8.00 (3.67)
0.70 7.29 (2.63)
RBA LlobaGL
0.7 6.79 (1.66)
0.75 5:47 (3.2)
0.25
10.4 (6.02)
0.75 6.00 (3.59)
Local
0.25
5.00 (0.63)
(5.57))
(3.18)
(4.15)
RBA Local-L
0.4
8.63 (2.45)
0.30
10.5
0.50
7.2 (4.62/)
0.40 6.25 (2.48)
PbRL
0.35
286.22
0.15
81.33 (66.97)
0.05 N/A
0.35 288.0 (143.45)
(167.49)
Table Results on controlling agent's hehavior with higher precision. SR Success Rate; Average Feedback (when success); I. Language
16

---

Published as a conference paper at ICLR 2023
Figure 6: A failure case of unsupervised concept discovery. Each row corresponds to a behavior trace.
17

---

Published as a conference paper at ICLR 2023
Lane Change Turning
Lane Change Sharpness
Manipulator Instability
RB4-Global REA-Global. Global-Lenguage
0.9
0.9
0.9
0.8
0.8
0.8
0.7
0.6
0.6
0.6
of ooflabelled trajectories
of trajettoiss
of labelled trajectories
0.5
0.5
0.5
10C
200
500
100
200
400
500
100
150
250
300
Manipulator Force/Speed
Bend Width
Snake Compression
0.9
0.9
0.5
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
of labelleed trajectoriis
of rrjjetooos
af labiellledd trajectories
0.5
0.5
0.5
100
150
200
250
200
100
200
300
420
500
100
200
300
400
500
Snake Moving Speed
Walker- Softness
Walker- - Step Size
0.9
0.9
0.5
0.8
0.7
0.7
0.7
0.6
0.6
0.6
I oflabelled trajectories
of labelled trojectories
of lflabelled trajectories
0.5
0.5
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Figure/ Performance of the ranking function in Method (RBA-Global) versus of training samples.
18

---

Published as a conference paper at ICLR 2023
Lane Change- Turning Dist
Lane Change Sharpness
Manipulator- Instability
RBA-Local RBA-Local- TBA--Loal-Lannuuce -Lagguage
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.2
lflabelled
cf labelled tuples
A of labelled tuples
200
400
500
200
200
400
500
200
300
400
500
Manipulator - Force/Speed
Snake Bend Width
Snake Compression
0.8
0.8
0.8
0.4
0.4
0.4
02
0.2
0.2
labebbele tuples
of Isbel led tuples
A jabelled tuples
100
200
300
400
500
2k
3k
4k
24
3k
4k
Snake - Moving Speed
Walker- Softness
Walker- Step Size
0.8
0.8
0.8
0.6
0.4
0.2
0.2
02
of lolllabbleddtn tuples
# of labelled tuplos
o abblllled tuples
1k
2k
3k
4k
200
400
600
500
1k
200
400
600
800
1k
Figure 8: Performance of Method 2 (RBA-LLcal) versus of training samples in Method 2 (RBA- Local).
19


Reviews:
[{"review_id": "YT_ypt6uqF", "reviewer": "Reviewer_mgGn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper takes an existing reward learning paradigm in reinforcement learning (RL) that can learn reward functions from preferences over traces (of states) and refines it by allowing to target specific \"relative attributes\" (properties of a trace usually) to be optimized. The incorporation of the relative attributes enables to tune some \"general\" properties (usually) of traces, for example the smoothness or speed of certain movements of a robotic manipulator. The approach comes with two different ways to incorporate (and tune by a user) the relative attributes: one global technique that focuses on a vector of values, and one local technique where one attribute can be optimized. The approach is tested on a couple of domains and various relative attributes.", "summary_of_the_review": "Nice practical idea in the sub-area of RL to learn from user preferences. The description (both text and formalizations, related work) needs quite some work, as well as the experimental analysis which should be extended in terms of analysis and description.", "strength_and_weaknesses": "Reward functions are the core of RL and work in finding new/different ways to incorporate and specify them is interesting. The strength of the paper is that it is a quite practical idea for a particular kind of reward function learning (and specification) in RL. The approach is intuitive, and amenable to all kinds of extensions. I think that the outcomes of the experiments show that the approach works.\n\nThe paper has, as a research paper, a couple of weaknesses that are not all easy to address. \n- An overall weakness is the text of the paper itself, including the formalizations. As far as I understand the paper, the work builds on typical (and recent) approaches leveraging Bradley-Terry-style models like on page 4 that can turn datasets of traces with preferences over these traces into a reward function that can be utilized to tune behavior towards those preferences. The main idea of the paper is to place either a vector of relative attributes in that ranking, or separate attributes (in the local model) and to derive a new reward function that incorporates the intended values (or direction of values) of those attributes. In both the local and the global model, the user can adjust the value of attributes such that the behavior of the agent changes in some intended direction. This is a fairly simple story and setup, but I had a hard time reading through all the verbose texts (already starting in Section 1) to find out what was happening. I'd advise a full rewriting and restructuring of the paper to reflect this. Also, the formalizations can then be made more precise and complete, and all this would help to create a much better description of the method. Also define simple things like \"skill\" or \"skill behaviors\", which seem to be assumed to be known, but which have very specific meanings in sub-areas of RL. Also, formalize the main aspects of the \"re-weighting\"/\"re-ranking\" of the traces better. The precise setting in sections 4.2. and 4.3. stays too implicit formally. Things like \"attribute encoder\" are used without explanation or formalization. Maybe give a couple of good examples in the beginning to sketch te idea more precisely. Also, the paper is not polished: several small errors (like there is no Table 5 on page 9, \"sample at large\" on the last line of page 8, informal style for \"don't\" instead of \"do not\", double words \"the the\" on page 5, and ungrammatical sentences like the first sentence of the 2nd par of Section 4.2. around \"than\", etc). Overall, I think the core of the paper is not described well, and this hinders a good evaluation of the technical parts of the paper.\n- The positioning of the work is done poorly. First of all, already in Section 1 we see many different aspects, and levels of reward function definitions, but nothing is really explained nor compared. I think, since nothing is really explicitated around this \"lingua franca\" (page 1), it would be much better to say something like \"sometimes one has some ideas about aspects of a full behavior that should be changed, a property of the behavior, and we would like to increase or decrease it, and for this we introduce relative attributes, a little in direction of how people do style transfer in deep learning\". The title promises much more than is in the paper now. There is nothing really about symbolic goal specifications as in (temporal) logic, where relative attributes as presented in this paper would be something like a maintenance goal. Furthermore, relative attributes are limited (as in the experiments) to a general number about the trace as a whole, which should be discussed. The experiments too all use these types of attributes. All these aspects should be related to the literature, which is not really happening in Section 2: we see some general references and some more specific, but it is hard to what is the state-of-the-art and what is the precies positioning of this paper, simply also because the section before it does not really define the setting well. The related work section should be much improved, and include things like reward functions (symbolic, preference rankings, but also inverse RL which is very related) but also things like style transfer in RL and other related topics. Also, some of the paper sounds too pretentious, like \"lingua franca\" and \"extremely efficient manner\" and \"superior performance\", etc.\n- The experiments are showing viability but not much more. From the technical description I get the feeling that this approach will work in some way (since, more information goes into the ranking and that seems to be a sound idea) but not how well, nor how it \"really\" works. The only baseline used is PbRL but this is not explained at all. The comparison is based on a couple of assumptions on how to use both (PbRL and the new technique) but this is not motivated enough. An important choice are the amounts of feedbacks (30 and 1000) but it is unclear how they influence the results, which are based on \"proximity\" to some good solution, and this is the only thing we get to see in Table 1. The number of domains considered is good, but the set of experiments is very limited and the analysis very short (one paragraph). I think the experimental evaluation should be greatly extended (not necessarily in the amount of domains or experiments, but in terms of analysis and sensitivity analysis) which is possible if the first half of the paper is written more concisely. I like the general embedding approach where also language descriptions can be used, but also here I'd like to see much more information and analysis (and examples). Basically, I'd like to get more detailed (visual) analysis of \"how\" behaviors are tuned based on the attributes in a specific domain (how fast, how well, artifacts, visibility of changes, correspondence between attributes and measured proxies, and so on).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NNVpI7TIjG", "reviewer": "Reviewer_Ry2j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes \"relative behavioral attributes\" an approach to allow non-expert end users to quickly (i.e., with limited data or demonstrations) correct robot/autonomous behaviors according to their preferences. The central approach involves learning (and later retraining) a ranking function that yields how strongly a particular attribute appears in a trajectory or behavior. Designers provide these attributes and labeled data for a ranking of how strongly attributes appear in each trajectory. During deployment, an end user can specify which behaviors they prefer through limited interactions---involving stating that a trajectory is preferred over another---and this data is used to train a preference-based reward model that uses the learned attribute models. Because trajectories are characterized in terms of a small set of attributes, retraining happens quickly, and far fewer feedback examples are required to achieve the desired behaviors. The authors show good performance on four different domains and significantly faster behavior change compared to the recent PbRL approach.\n", "summary_of_the_review": "The paper is clear and delivers on most of its promises. I have a few questions and comments that the authors should address so that the paper can be maximally useful for the community, though they are relatively small.\n", "strength_and_weaknesses": "Overall, the paper is fairly strong. The authors introduce two different approaches to using their approach and do a good job discussing the relative advantages of each and provide suitable performance analysis. Experiments are well-suited to understand the problem and cover multiple domains of interest to this community. PbRL is also a solid baseline, and the comparison to PbRL does a good job at highlighting why an alternative approach is necessary.\n\nOne part of the paper that could benefit from further discussion is where the approach is limited or is unlikely to succeed. For example: what if the attributes are limited or poorly chosen? How would performance change (presumably decrease) if an attribute is omitted? Further experiments are likely unnecessary, but adding such details (at least some of which should appear in the main body of the paper, even if others appear in the appendix) would help clarify when the proposed approach(es) are or are not well-suited to the proposed work.\n\nCould the authors comment in more depth on what a typical failure mode looks like? There is some general discussion of why failure modes occur: namely what seems to be the limited gradient of the ranking function near the extreme values. However, some additional figures would improve understanding. Is the failure that the corrections do not change behavior? Some additional (qualitative) visuals or analysis would help clarity.\n\nFinally, though it is straightforward to understand how embedded language descriptions could be used in place of a one-hot vector representation for the attribute vectors. However, it is not clear to me why the language-input models (in most of the experiments in Table 1) is competitive with or even frequently improves upon the performance of the non-language counterparts, since the non-language versions are trained using a representation specifically chosen by the designers (in this case, the authors) for this purpose. Could the authors clarify what the source of this improvement might be?\n\nOther smaller comments:\n- The abstract is quite long, and it might be good to remove some of the high-level context in the interest of conciseness.\n- The Walker column in Table 2 seems to be incorrectly highlighted: RA-Global-L seems to perform best across all metrics.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2-ucTauqREb", "reviewer": "Reviewer_5ZSD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new method for training RL agents with human feedback. In prior work, agents are trained either with a fully programmed reward function (which is hard to program correctly) or are trained with low bandwidth human feedback (e.g. binary feedback on which of two trajectories was better). This work proposes learning a set of attributes that describe trajectories, then getting human feedback on how a trajectory should change, in terms which which attributes to increase / decrease. This gives a higher bandwidth method of human feedback, leading to fewer rounds of feedback before successfully learning a concept.", "summary_of_the_review": "I think this paper provides a reasonable step forward in an important subfield that is highly underdeveloped.\n\nStatement on confidence: I haven't worked on this topic or even topics all that closely related to this firsthand, so it's reasonably likely that I've missed something important, particularly in terms of related work.", "strength_and_weaknesses": "Strengths\n-----------\n- Novelty : there is surprisingly little published academic work on how to design reward functions. As the manager of a team using RL for a real product, I was surprised to find that \"reward function\" is not in the index of the Sutton and Barto book, and not a page in Wikipedia. So this paper contributes to a very underdeveloped area.\n- Usefulness: my experience working in industry is that the biggest barrier to using RL for real products is not the performance of the RL algorithm itself, it's challenges outside of core RL, like sim to real transfer and how to design the reward function and evaluate performance of the agent. Real-world products interacting with human users have much more complicated reward functions than 0/1 winning or losing Go. So this paper contributes to an area that is important for practitioners in industry.\n\nWeaknesses\n--------------\n- Experimental results: the experiments here are all on synthetic data and toy tasks. I'm not super optimistic that the methods proposed here would help my RL team on a real product. However, I think this is acceptable due to the limited amount of literature on the topic / I think other papers accepted in this area share this weakness.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster