A Dual Representation Framework for Robot Learning with Human Guidance
Authors: Ruohan Zhang
Dhruva Bansal
Yilun Hao
Ayano Hiranaka
Jialu Gao
Chen Wang
Roberto Martín-Martín
Li Fei-Fei
Jiajun Wu
Venue: CoRL 2022 Poster (2022)
Keywords: Human Guidance
Evaluative Feedback
Preference Learning

Abstract: The ability to interactively learn skills from human guidance and adjust behavior according to human preference is crucial to accelerating robot learning. But human guidance is an expensive resource, calling for methods that can learn efficiently. In this work, we argue that learning is more efficient if the agent is equipped with a high-level, symbolic representation. We propose a dual representation framework for robot learning from human guidance. The dual representation used by the robotic agent includes one for learning a sensorimotor control policy, and the other, in the form of a symbolic scene graph, for encoding the task-relevant information that motivates human input. We propose two novel learning algorithms based on this framework for learning from human evaluative feedback and from preference. In five continuous control tasks in simulation and in the real world, we demonstrate that our algorithms lead to significant improvement in task performance and learning speed. Additionally, these algorithms require less human effort and are qualitatively preferred by users.

A Dual Representation Framework for Robot Learning with Human Guidance
Ruohan Zhang*1, Dhruva B Yilun Hao*! Ayano Hiranaka", Jialu Gaoi, Chen Wang',
Roberto Martín-Martin Li Fei-Fei Jiajun Wu 1.6
Departmment of Computer Science, Stanford University Department of Mechnical Engineering, Stanford University TTinghua University
DDpaarmeent of Computer Science, The University of Texas at Austin Institute for Center-Ceneered AI (HAI), Stanford University
Abstract: The ability to interactively learn skills from human guidance and adjust behavior according to human preference is crucial to accelerating robot learning. But human guidance is expensive resource, calling for methods that can learn efficiently. In this work, we argue that learning is more efficient the agent is equipped with high-level, symbolic representation. We propose dual repre- sentation framework for robot learning from human guidance. The dual repre- sentation used by the robotic agent includes one for learning sensorimotor con- trol policy, and the other, in the form of a symbolic scene graph, for encoding the task-relevant information that motivates human input. We propose two novel learning algorithms based on this framework for learning from human evalua- tive feedback and from preference. In five continuous control tasks in simulation and in the real world, we demonstrate that our algorithms lead to significant im- provement task performance and learning speed. Additionally, these algorithms require less human effort and are qualitatively preferred by users. Project website: https: //sites.//eteeeeeeeeesseeeeess teriitessgoogeeee..oooooooooeeee wwwm/viww/ddweiieeeeeeeeeeeeeeee
Keywords: Human Guidance, Evaluative Feedback, Preference Learning
Introduction
Human guidance refers to a diverse set of human training signals provided to a learning agent [1- 5]. These alternative forms of human input can be combined with the conventional reward signal in reinforcement learning (RL) [6] or demonstrations in imitation learning (IL) [7-9] to facilitate learning. Recently, the robot learning community has increased its attention human guidance as mechanism to overcome two critical challenges: 1) the low sample efficiency of learning algorithms, and 2) the effort in manually specifying the objectives for learning. Human guidance is helpful for these challenges because 1) guidance like evaluative feedback [10-27] can be used as domain knowledge to speed up learning, and 2) through their guidance, humans can define the learning objectives for robots so that the learning algorithm better infers and aligns to the underlying human goals and values, such as their preferences [28-46].
Despite its benefits, human guidance is scarce and valuable resource, and human-in seenan-in-thhe-ooe loop mechanisms strive find ways to reduce the amount of guidance required from humans to make them more broadly and practically applicable. This is only possible if the human guidance is properly represented and interpreted [47-49]. The first step to choose a representation that allows the learning agent to query humans more efficiently and learn with less human guidance [47-49].
Robots typically use a fine-grained state and action space for continuous control. However, when humans observe, evaluate, and guide robot behaviors, their representation is likely different. For example, in continuous control task such as placing an object, the robotic agent typically has access to low-level states including proprioceptive information and other objects' poses. But the guidance
$indicates equal contribution; correspondence to zharu @stanford.edu
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

provided by humans is typically based on higher-level abstract information. such as is the robot's 's end-effector to the left or the right of the goal?, or is the robot grasping the object? This observation leads to the "dual representation hypothesis" proposed in this work, inspired by cognitive science studies [50-53]. This analogical to the "fast and slow" systems proposed by Kahneman [52], in which the fast system manages intuitive, automatic, unconscious behaviors while the slow system manages logical, calculating, conscious thoughts [52]. We hypothesize that the "slow" system and its representation are useful in guiding learning agents.
In this work, we propose a dual representation framework for robot learning from human guidance: the robotic learning agent uses low-level state representation for learning control policies, but keeps a symbolic scene graph [54-57] as a high-level representation of human internal states. We show that this framework enables novel learning algorithms: Dual Representation-basee Evaluative Feed- back (DREF) and Dual Representation-bant active learning that allows the agent to estimate the uncertainty of human
Preference Learning (DRPL.. DREF is based on the
idea of uncertainty- feedback in unseen states. Such generalization ability is achieved by using scene graph representa- tion to group low-level states into abstract states. DRPL builds upon scene suph-based trajectory segmentation and selection, allows efficient reward learning from chosen trajectory segments.
In three simulation tasks and two real-robot tasks, we demonstrate that our proposed approaches lead to significant improvements in learning speed and performance. For challenging long-horizon real-world robot manipulation tasks, we show that DREF can learn to solve the task efficiently, while end-to-end RL algorithms fail to solve because of the sigh-dimensional continuous state and action space. Critically, we observe significant reduction in the amount of human guidance required for learning, and an improvement in overall user experience, making learning from human guidance methods more practical and appealing for real robot learning.
2 Related Work
Among the multiple forms of human-in-the-loyy robot learning [58-60], in recent years, the robotics community has paid increased attention to human guidance [3] because of being powerful and easy to collect, and complements standard training signals such as rewards or demonstrations.
Learning from human evaluative feedback. This is approach in which human trainers monitor the learning process of an agents, and provide a scalar signal to indicate whether the observed behav- ior is desirable [10-27]. The agent then learns a policy to maximize positive feedback from humans. This approach has the advantage of placing minimum demand on both the human trainer's ss expertise and the ability provide guidance, compared to learning from demonstrations. Significantly, hu- man evaluation is often interpreted as a value function [14, 16] or an advantage function [15, 17], not as the reward itself. Nonetheless, human evaluation can be naturally combined with environ- ment rewards so the agent learns simultaneously from both sources [18-20]. Evaluative feedback often targets individual state-action pairs. Hence one outstanding challenge is to generalize observed human feedback to unseen state-action pairs.
Learning from human preference. In this framework, the learning agent queries human trainers for their preferences over a set of exhibited behavior trajectories [28-46]. Preferences can be used to directly learn policies [28, 29], preference model [30], or reward function [31, 32]. Recent works often learn a hypothesized latent human reward function from the preference signals, and combine preference learning with deep RL to learn policy afterwards [34-37]. Notably, recent work attempts to align agent representation with human representation in preference learning [61]. While evaluative feedback targets a state-action pair, preference learning targets trajectories. Selecting optimal query trajectory length is a challenging problem in preference learning [3), we will show that the dual representation learning framework provides an intuitive way to segment lengthy trajectories.
Scene graph as abstract representation. In robot learning, abstract representation has been an ac- tive research topic, exemplified by recent works in neuro-symbolic robot learning [62-65]. Human- robot interaction tasks, such as shared autonomy [66, 67], highlight the importance of abstract rep- resentation since humans often communicate high-level goals the agent in this setting. Low- dimensional, abstract representations could potentially be learned (68-72] but sample efficiency is a major limiting factor when applying these methods physical robot learning from real humans.
scene graph, often specified by humans, a form of abstract representation for state information in which objects are represented as nodes and relations between objects are represented as edges [54-

---

Environment (Placin-RRea))
Dual Representation
Human Guidance
uncertaiitty
InHand
aware
query
trajectory
segmentattin selection
High-lewer symbolic crenn-grrahh
preference
propriaception reward
www-level states and rewards
Low-frequency
High-frequency control
query and guldance
Figure 1: Overview of the proposed dual representation framework. The robot maintains two state representations. The first one for learning a fine-grate graaggr low-level continuous control policy. The second representation is specified by an expert human in the format of symbolic scene graph. The robot uses this representation to actively query human trainers for evaluation or preference during the training process.
57]. Scene graphs store explicitly and compactly information about object geometry, placement, semantics, and relationships, making them suitable for tasks that require sophisticated reasoning about these types of information. Recently, scene graphs have started to be used in roboties [73- 75] as a state representation that facilitates planning and reasoning. We explore how a symbolic scene graph could be useful additional representation for human-in-the-logg robot learning. We hypothesize that allows the learning agent to query humans more efficiently and learn with less human guidance than using the low-level, raw state representation alone.
Method
Our method is designed overcome significant challenge in human-in-the-loop robot learning: human feedback expensive, and frequently asking for guidance infeasible in real-world robotic systems. To address this challenge, we propose a novel dual representation facilitate human-in- the-loop policy learning. Below we introduce the general human guidance learning problem and our proposed dual representation setups (Sec. 3.1). Then, we propose an algorithm to learn from evalua- tion feedback that decreases the amount of human feedback needed during policy learning (Sec. 3.2), and query generation algorithm efficiently query humans for their preference (Sec. 3.3).
3.1 Dual Representation for Learning from Human Guidance
We represent the robot learning problem as a Markov Decision Process denoted by the tuple (S, A. P. R. 7). where S is the state space, A is the action space, R S A IR is the re- ward function, P
A > 00, 1] the transition function, and y the discount factor. A policy π S x A + [0, 1] is mapping from S to probability distribution over A.
We propose dual representation framework for human-in-the-loop robot learning that consists of the common low-level state information from the environment and an additional high-level symbolic scene graph G. G is represented unary state of an object or a pairwise semantic relation between the objects and the robot. Let be an instance of G, i.e., an abstract state which contains infinite low-level. continuous states. The objects and relations in G are specified by humans based on task knowledge. The proposed dual representation framework shown Fig.
binary vector of dim(g), where each dimension represents
3.2 Learning from Evaluative Feedback with Dual Representation
Human evaluative feedback contains rich task-level knowledge that could be used to assist robot learning. We adopt an active learning setting, in which an RL agent asks humans for their evalua- tive feedback. Hypothetically, asking and receiving feedback in the right state could lead to better learning results with less human effort, but it is unclear when to ask for feedback. Our key insight

---

Algorithm Dual Representation-based Evaluative Feedback (DREF)
1: Initialize UCB1(g), Vg E G. as 0
2: Initialize as the replay buffer
3: Initialize network weights for actor A enviromment critic 0g, and human feedback critic 0zz
4: for 1 do
5: Select 0 s;;15;;AA)
6: Infer current scene graph state gr from 8t
7: if rank I k. across 9 sorted by UCB1 score then
8:
Query for evaluative feedback H,(s,, 0z)
9:
end if
10:
Execute a, observe reward r, and next state 81++1+
11:
Store transition (st, de, Tt. Hr, 8t+11 ) in D
12:
Sample random minibatch of transitions (s1, az. rt. Hi, S(+1) from D
13:
Perform gradient step according to Equations 3 4, and for 0g 0g, A
14: Update UCB1(g), Vg E G using Equations 1,2
15:
end for
is that querying for human feedback can be formulated multi armed bandit problem with sym- bolic scene graph representation. This formulation brings the opportunities to use formal methods designed for discrete state and action spaces, such Upper Confidence Bound (UCB1). Integrating UCB1 with continuous control enables efficient uncertainty-awarr active learning from humans.
TAMER+RL [18-20] is widely used framework for learning from evaluative feedback. For the RL. part, we use Soft Actor-Critic (SAC) [76]. In addition to the environment reward, human trainers provide a scalar signal H, a) { 0, +1} to indicate whether the observed state-action pair desirable or not. Our goal is to reduce the amount of total feedback while achieving the same or better task performance.
Inspired by the standard UCB1 for value function in bandit problems, we use the following equa- tion to estimate the upper confidence bound of human feedback prediction eπror (FPE), the type of uncertainty we care about, an abstract state gr:
2 log N₂ UCB1 = FPE(g)) + V Nz(g))
where N, is the total number of human feedback received at time t, and No(g)) is the number of feedback given to the abstract state gt. The constant e weighs the exploitation and exploration terms. FPE(g)) ) is the average feedback prediction error for all the low-level states encountered far that belongs to ge:
FPE(g() = (1 Σ1 E (s)))
(.) - (s.,
We estimate (s, a) using additional critic head in SAC. Assume 0H, θg A. parameterize the human feedback critic, the environment critic, and the actor, respectively, the learning objectives are:
((0#) = -E(s,,,
- a)s,a)
(3)
Lsuc (fg) = Dll.)) ((s.
(4)
((OA) = EgnD
E Bowr(1ss: log(T 0a)) - (Q(s, a; fx) + (fT(s (x))]]
(5)
In Eq. 4, Lsac(Og) the standard soft Bellman residual in SAC [76. 77]. In Eq. 5, a is the temper- ature parameter in SAC [76, 77]. Note that the actor updates the policy distribution in the direction suggested by the weighted average of both critics. The agent learns policy to maximize expected positive feedback from humans and environment reward simultaneously.
The full algorithm, DREF or Dual Representation Evaluative Feedback, is shown Algo- rithm 1. We first calculate the running mean of UCB value for each abstract state and rank them.

---

Algorithm Dual Representation-based Preference Learning (DRPL)
1: Collect a set of random trajectories T
2: Set prior of reward weights (0) randomly
3: Segment all trajectories in 7 based on abstract state changes: Teeg (51,62....
4: for i = O, dim(G) do
5: Select two segments E' Taeg. with ties broken arbitrarily
s.t. their associated abstract states g(il * g' and d(g. g') = 1 or d(g.g') ming.g d(g.9')
6: Query for preference ((,,E")
7: Update posterior p(++1 ((++)(())))) (0) x ((q))))'(0, where P(q is the probability of preference
response y given 0
8: end for
Then we query for feedback at a specific state s; its abstract state gi has a high rank (determined by hyperparameter k). As a result, the agent actively asks for feedback in abstract states with uncertainty in the human evaluation of actions, e-g., query for feedback when the gripper and the ball move into the hoop for the first time, as shown in Fig. 1. With the low-level state and action space alone, this cannot be easily done because the number states is infinite. The abstract state representation effectively groups these states together, and estimates the uncertainty in predicting feedback of a new state using the average FPE of other states its group. We hypothesize that in this way, the dual representation could lead to efficient uncertainty- active learning from human feedback, which we will demonstrate in our experiments.
3.3 Preference Learning with Dual Representation
Preference learning is important method define the objective for learning agents. While evaluative feedback targets a state-action pair, in preference learning human trainers indicate their preference over pair of trajectories, from which the agent learns a reward function R. For sim- plicity, we assume that reward function R is linear combination of state-action features [78]: r(st, 04) = 8 o6 St, at). hence the goal to infer Our goal is to reduce the amount of human guidance: the algorithm should accurately estimate 0 with minimum number of queries.
Here we address two issues in preference learning: generating and selecting meaningful queries. For query generation, selecting trajectory length is challenging [3]. Indicating preference over longer trajectories requires cognitive effort (e.g., summing all the rewards in each trajectory). Short trajectories (e.g.. random 1-2 seconds clips [34]) allow humans to provide feedhack of high granularity, but these clips may not be meaningful or comparable. After trajectories are segmented, selecting meaningful pairs to query humans is yet another challenge: tpples-to-apples comparisons are more meaningful.
We propose simple solution to these challenges: scene graph-based based trajectory segmentation and selection. The abstract state dimensions in the scene graphs naturally overlap with the reward fea- tures, since these scene graphs are designed to contain information that is critical to task success. The key observation is that in long-horizon tasks, trajectory consists of multiple abstract state sitions, and two consecutive abstract state fransitions naturally define the starting and ending points of segment, i.e., a segment corresponds only abstract state g. Then, for query selection, select pair of segments with two abstract states and g', such that d(g. g') = where d denotes the hamming distance (recall that g is a binary vector). In other words, g and g' are two abstract states that only differ in one dimension. Hence the remaining dimensions are held constant, leading controlled comparison. If we cannot find a pair of segments such that d(g. g') 1, we choose the pair that has the minimum hamming distance. The algorithm, DRPL, or Dual Representation based Preference Learning, is shown in Algorithm 2. We use a standard reward weight estima- tion algorithm (a Bayesian inverse reinforcement learning algorithm) that is based on the low-level representation [41, 78], the only difference how we segment and select trajectories for query.
Experiments
We test our algorithms in five continuous control tasks (Fig. 2) for both evaluative feedback and pref- erence learning. The following describes the dual representation (low and high level) of each task.
5

---

Figure Five continuous control tasks in our experiments: Lunar-Lander, Reaching-Sim. Placing-Sim, Baaching-Real, and Placing-Real.
[79], we control spaceship to land in the middle of two flags without crashing. The state, E RR$ includes position, velocity, and leg-contact information, and the action signals, a E R² control the engines. In Reaching-Sti -Sim and Reaching- Real, the robot's s goal is to move its gripper, to the center area marked on the table from random starting location. The state, E R2, specifies the position of the gripper in the zy- zy-plane and the actions, a E IR?, are delta movements on that plane. In Placing-S Sim and Placing-Ree Real, the robot must move initially grasped ball from random location into hoop and drop there. The state, s E RRA, includes the gripper position in 3D space and the gripper state (close/open). The actions, a IR are delta movements in 3D Cartesian space and the gripper control. The simulation tasks are implemented in the Robosuite [80]. Further details can be found in Appendix 1.
Low-level representation. Lunar-lander--Cntiiuuute
High-level representation. Fig. depicts the symbolic scene graph for Placing- tasks, where nodes correspond to task-relevant objects, and edges encode binary relations between them. Ap- pendix 1 includes further details about the tasks and their symbolic scene graphs.
4.1 Results: Evaluative Feedback
We use synthetic humans in simulated environments and real humans in real-world environments. For synthetic humans (a fully trained SAC agent, more details in Appendix 2), the learning agent chooses action a in state 8, and the oracle chooses action a". The oracle SAC computes the Q values for these actions: ((s, a) and ((s,a*)) If the learning agent chooses an action that has a Q-value close enough to Q(s, a"). it is good action and the agent should receive positive feedback. Otherwise, it should receive negative feedback: H(s, a) = +f, Q(s,a) 2 aQ(s, ); -1 otherwise. The increases over time (sce Appendix to encourage the agent to learn to choose better actions during training.
We compare method to the following baselines in simulated environments: (a) the SAC baseline without any human feedback (b) EF-100% EF-50%, and EF-25%: the agent asks for feedback with probability of 100%, 50%, or 25% every timestep. Hyperparameters of all the algorithms can be found in Appendix 2.
Results for simulation (averaged across 5 seeds) shown in Fig. 3. Our algorithm DREF (shown in blue) achieves comparable performance to EF-100% with only 6.2%, 1.6%, and 8.4% feedback in Lunar-Lander, Reaching-Sine and Placing respectively, corresponding to approximately 16x, 62x, and 11 x improvement in feedback sample efficiency.
For real robot experiments with six humans per task, we omit the 10-100% baseline since it requires step-by-step feedback from humans for a very long period of time. Humans are also allowed to choose "no feedback" if they prefer. The order of the three methods is randomized to counterbalance the ordering effect (see Appendix 4). Fig. shows the results. Reaching-Real, DREF achieves comparable performance to EF-50% with 13.8% feedback, leading to a 3x improvement in human feedback sample efficiency. Placing-Ress Reann is extremely challenging due to the long task horizon and sparsity in reward signals. DREF achieves better performance than all baselines, by large margin, with only 17% human feedback.
The human experiments are followed by survey (details in Appendix 4.1) asking humans about their overall experience training the robots (E), perceived intelligence level (I), and cognitive ease (C) (see Fig. 4). It is evident that our algorithm leads to better user experience and reported effort (additional analyses are Appendix 4.2). These results are supported by the observation that the total training time and no feedback count are lower for DREF. This may also explain why EF-50% performs poorly in the challenging Placing-Real task; the training is laborious and cognitively demanding with this amount of guidance, and humans are prone to errors in this process.

---

1.5
1e5
Luner-Lander
1s4
Reaching-
145
Placing-Sim
DREF 6.2%
DREF 1.65
DREF 8.45
Cummeete Beward
EP 17-100%
EP-100%%
15
EN-100%
1.0
5F-52%
EF-50%
EF-50%
EF 25%
EF 25%
6F-25%
SAC
SAC
SAC
2
0.5
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Ensfronment stopo
Environment stops
Ervironmont stops
Figure Cumulative rewards gained during training for Lunar-Lander, Reaching-Si Placing- Sim. The percentage corresponds to the percentage of feedback provided by the oracle during training. The proposed algorithm, DREF, achieves comparable performance with EF-1009 with much less feedback and outperforms all other baselines. Error bars indicate the standard error of the means (n 5).
and
2.5
1+3
Reaching
1.75
lef
Placing
Reaching
EF EF-259
BF-50%
DREP
E7-25%
EF 50%
DREF-13
1.50
1.25
EF-25
DASF-17
50%
NFCL
T
54.5+13.5
20:2::16
2.+003
898-2668
26.61:00
1.240.3
22:2-888
16.11007
4,2+0.3
1.5
SAC
SAC
2,5+04
3.0406
4.2+115
C7
301003
36.10.5
4.01044
1.0
Placing
FR-2558
RF-50%
DIFF
T
36.9144
46.127.5
25.0::00.2
NFCL
156.7143.8
421.5195.0
119.7..25.4
2.1004
25.103
4.2...2
0.0
27103
2.710.5
4.7.0.2
2.1006
2.700.7
4.21.033
Env
163
Environment
1w3
Figure 4: Cumulative rewards gained during training for Reaching Beachhin-RRea,, and Placing-Real The proposed algorithm, DREF, achieves comparable better performance with 50% feedback with much less feedback and outperforms all other baselines. Error bars indicate the standard error of the means (n 6). Post-completion user survey (5-point Likert scale) results indicate that DREF leads to better user experience, perceived intelligence, and less reported effort (n 6). T: total training time, NFC: number of "no feedback" responses, E: overall experience, perceived intelligence, C: cognitive ease. See Appendix 4.1 and 4.2 for survey design and additional analyses.
To conclude, the results strongly support hypothesis about evaluative feedback: asking for hu- man feedback sensibly leads to better learning outcome. DREF achieves this by implementing uncertainty-awar active learning within the dual representation framework.
4.2 Results: Preference Learning
We now present the results of preference learning experiments. Similar evaluative feedback, use synthetic humans in simulated environments and real humans real-world environments. The synthetic human has access the true reward weight 0 (more details in Appendix 3). For every pair of queries(E, E'), the oracle calculates the true reward r and r'. The oracle returns q(E, E') otherwise. The trajectories are generated by agents starting random
r(E) > r(')), and q(E, positions and taking random actions. We record such trajectories, store them, and select trajectories or segments to query synthetic or real humans. Preference learning algorithms update the posterior belief of the reward function after each query. We use cosine similarity as the alignment score to measure the distance between the estimated reward weights Ô and true weights 0 [35, 41).
We compare DRPL with three baselines: (a) full trajectory query: randomly select full trajecto- ries to query; (b) random fragment query: randomly select two full trajectories, and cut one fragment of each with length equal to the average length of DRPL queries;
DRPL-SS: version of our algorithm that selects a pair of trajectories associated with the same abstract state, instead of two abstract states that differ in one dimension. Further details can be found in Appendix
Results for simulation (averaged across seeds with different true rewards) are shown in Fig. 5. DRPL (blue) converges forms all other algorithms. DRPL-SS performs better than the other baselines but is worse than DRPL, indicating that both components of our DRPL (segmentation and selection) are important.
alignment
around or greater than 0.8 for all tasks and outper-
Results for real robot experiments with humans (six for each task) are shown Fig. We omit the full trajectory baseline due to the time required to compare long trajectories generated by the random
7

---

Reaching
Placing Sim
1.3
1.0
1.0
3.8
0.8
Algimenntooe
stgement Score
Allynmentscoo
0.6
DRPL DRPL-55
anndom/tagmoon full vejuctory
-02
20
30
40
10
30
40
50
10
20
20
50
Queries
Quorics
Quericcs
Figure 5: Alignment scores [35, 41] for Luuna--Lande Reaching- SSm, and Placing- Sim. DRPL performs the best upon convergence. Error bars indicate the standard error of the means (n = 5).
Reaching Real
Placing Real
Reaching
RF
DRPL-SS
DRPL
0.8
0.5
Alymmork 500me
nandom, Iragrens
Algmmman SCOPPO
random, fragment
B1
3:2+0.2
3.2=0.2
3.3=0.2
0.6
0.6
1,5+0.2
3.8+0.3
4.5-0033
Ct
3.340.3
3.2=0.2
4.0:0.0
0.4
3.4
Placing
RF
DRPL-SS
DRPL
B1
3.0+:0.0
3.2=0.2
3.3:0.2
17
1.51:.3
2.720.4
38:00.
Ct
3.2+0.3
33:002
33=00.
20
25
20
25
Queffes
Querles
Figure 6: Alignment scores [35, 41] for Reaching- and Placing-Real. DRPL achieves the best performance upon convergence. Error bars indicate the standard error of the means (n 6). Post-completion survey (5 (5-point Likert scale) results indicate that DRPL leads to better perceived intelligence. E: overall experience, I: perceived intelligence, C: cognitive ease. See Appendix 4.3 and 4.4 for survey design and additional analyses.
agents. Our algorithm DRPL (shown in blue) achieves better performance than both baselines. Similar to evaluative feedback, the user experience survey (Fig. 6) shows that DRPL leads to better perceived intelligence (see Appendix 4.3 and 4.4). To conclude, the results support key insight about preference learning: abstract state-based trajectory segmentation and query lead to a better learning outcome. DRPL is instantiation of this idea within the dual representation framework.
5 Discussion
To summarize, we have proposed a dual representation framework for robot learning from human guidance. In the context of robot learning, abstract representation has long been research topic. However, adopting it in decision learning often comes with price: the critical information needed learn a good policy may be lost in the process of abstraction. Researchers typically avoid this problem by having a hierarchical representation. The key difference is that we only use abstract representation as an auxiliary representation.
We show that the abstract scene graph representation allows us to utilize important heuristics that facilitate training in two popular forms of human guidance: evaluation and preference, both of which utilize human evaluations for observed agent policies. Evaluative feedback targets state-action pairs that are fine-grained, while preference learning targets trajectories that could be too coarse. Hence the former needs a grouping mechanism for generalization, and the latter needs a segmentation mechanism for efficient queries. Our proposed framework is unified approach to provide both.
We demonstrate the effectiveness of our approaches in five challenging continuous control tasks. Our algorithms show significant improvements in performance and reduction in human effort, compared to algorithms without an auxiliary scene graph representation. These improvements make learning from human guidance methods significantly more appealing for real-robot learning.
Limitations. Currently, our proposed approaches leverage expert-defined scene graphs. Although this representation should be closer to the human representation, the actual representations are likely to be different across human trainers. Adapting different abstract representations may further improve learning outcomes and user experience.

---

Acknowledgement
The work is in part supported by the Stanford Institute for Human-Centered AT (HAD), ONR MURI N000)4--221122740, ONR MURI N00014-21-1-8011, Amazon, Analog, IBM, JPMC, Meta, Sales- force, and Samsung. Ruohan Zhang supported by Wu Tsai Human Performance Alliance Fellow- ship.
References
[1] A. L. Thomaz, C. Breazeal, et al. Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In Aaai, volume 6, pages 1000-1005. Boston, MA, 2006.
[2] R. Zhang, F. Torabi, L. Guan, D. H. Ballard, and P. Stone. Leveraging human guidance for deep reinforcement learning tasks. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 6339-6346. AAAI Press, 2019.
[3] R. Zhang, F. Torabi, G. Warnell, and P. Stone. Recent advances in leveraging human guidance for sequential decision-making tasks. Autonomous Agents and Multi-Agent Systems, 35(2): 1-39, 2021.
[4] H. B. Suay and S. Chernova. Effect of human guidance and state space size on interactive reinforcement learning. In 2011 Ro-Man, pages 1-6. IEEE, 2011.
[5] C. Basu, M. Singhal, and A. D. Dragan. Learning from richer human guidance: Augment- ing comparison-basee learning with feature queries. In 2018 13th ACM/IEEE International Conference on Human-Robee Interaction (HRI), pages 132-140. IEEE, 2018.
[6] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[7] Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6):233-242, 1999.
[8] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-485 2009.
[9] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends Robotics, 7(1-2):1-177, 2018.
[10] C. Isbell, C. R. Shelton, M. Kearns, S. Singh, and P. Stone. A social reinforcement learning agent. In Proceedings of the fifth international conference Autonomous agents, pages 377- 384. ACM, 2001.
[11] A. C. Genorio-Gonzaler E. F. Morales, and L. Villaseñor- Pinedd.. Dynamic reward shaping: training a robot by voice. In Ibero-American conference on artificial intelligence, pages 483- 492. Springer, 2010.
[12] S. Griffith, K. Subramanian, Scholz, C. Isbell, and L. Thomaz. Policy shaping: In- tegrating human feedback with reinforcement learning. In Advances in neural information processing systems, pages 2625-2633 2013.
[13] T. Cederborg, I. Grover, C. L. Isbell, and A. L. Thomaz. Policy shaping with human teachers. In Proceedings of the 24th International Conference Artificial Intelligence, pages 3366- 3372. AAAI Press, 2015.
[14] W. B. Knox and P. Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pages 9-16. ACM, 2009.
[15] MacGlashan, M. K. Ho, Loftin, B. Peng, G. Wang, D. L. Roberts, M. E. Taylor, and M. L. Littman. Interactive learning from policy-dependent human feedback. In Proceedings of carning--ollume Volume 70, pages the 34th International Conference on Machine Learning-
JMLR.
org, 2017.

---

[16] G. Warnell, N. Waytowich, V. Lawhern, and P. Stone. Deep tamer: Interactive agent shap- ing in nigh-dimensional state spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[17] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint arXiv: Stti::9902.0225,, 2019.
[18] W. B. Knox and Stone. Combining manual feedback with subsequent mdp reward signals for reinforcement learning. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume I, pages 5-12. International Foundation for Autonomous Agents and Multiagent Systems, 2010.
[19] W. B. Knox and P. Stone. Reinforcement learning from simultaneous human and mdp re- ward. Proceedings of the 11th International Conference on Autonomous Agents and Mul- fiagent Systems-Volume 1, pages 475-482. International Foundation for Autonomous Agents and Multiagent Systems, 2012.
[20] R. Arakawa, Kobayashi, Y. Unno, Y. Tsuboi, and S.-i. Maeda. Dqn-tamer: Human-in- tho-loop reinforcement learning with intractable feedback. arXiv preprint arXiv:1810.11744 2018.
[21] Y. Cui, Q. Zhang, A. Allievi, P. Stone, S. Niekum, and W. B. Knox. The empathic framework for task learning from implicit human feedback. arXiv preprint arXiv:2009. 10699,3699 2020.
[22] T. Kessler Faulkner, R. A. Gutierrez, E. S. Short, G. Hoffman, and A. L. Thomaz. Active iatention-modifee policy shaping: Socially interactive agents track. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, pages 728-736, Richland, SC, 2019. International Foundation for Autonomous Agents and Multia- gent Systems. ISBN 978-1-4503-6309-9) URL http: //dl. acm. org/ www.loogggttteeeeeeeeeoeeeee/eee /et/ttiiiiion.../ cfm?id= 3306127 1336127.3331762.
[23] G. Li, S. Whiteson, W. B. Knox, and H. Hung. Using informative behavior to increase engage- ment while learning from human reward. Autonomous agents and multi-agent systems, 30(5): 826-848, 2016.
[24] J. Grizou, I. Iturrate, L. Montesano, P.-Y. Oudeyer, and M. Lopes. Interactive learning from unlabeled instructions. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, pages 290-299, 2014.
[25] R. Loftin, B. Peng, J. MacGlashan, M. L. Littman, M. E. Taylor, J. Huang, and D. L. Roberts. Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strate- gies to speed up learning. Autonomous agents and multi-agent systems, 30(1):30-55 59, 2016.
[26] L. Guan, M. Verma, S. S. Guo, R. Zhang, and S. Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. Advances in Neural Information Processing Systems,
21897, 2021.
[27] A. Najar, O. Sigaud, and M. Chetouani. Interactively shaping robot behaviour with unlabeled human instructions. Auton. Agents Multi Agent Syst., 34(2):35, 2020.
[28] A. Wilson, A. Fern, and Tadepalli. bayesian approach for policy learning from trajectory preference queries. In Advances in neural information processing systems, pages 1133-1141, 2012.
[29] R. Busa-Fekete, Szörényi, P. Weng, W. Cheng, and E. Hüllermeier. Preference-based evo- lutionary direct policy search. In ICRA Workshop on Autonomous Learning, 2013.
[30] J. Fürnkranz, F. Hüllermeier, W. Cheng, and S.-H. Park. Preference-hased reinforcement learn- ing: a formal framework and policy iteration algorithm. Machine learning, 99(1-2))123-156 2012.
[31] C. Wirth, J. Fürnkranz, and G. Neumann. Model-free preference-based reinforcement learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2222-2228, 2016.
10

---

[32] R. Akrour, M. Schoenauer, M. Sebag, and J.-C. Souplet. Programming by feedback. In Inter- national Conference on Machine Learning (ICML), volume 32, pages 1503-1511. JMLR. org, 2014.
[33] C. Wirth, R. Akrour, G. Neumann, and Fürnkranz. A survey of preference-based reinforce- ment learning methods. The Journal of Machine Learning Research, 18(1):4945 4990, 2017.
[34] Christiano, Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299 43099-4377220 2017.
[35] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.
[36] A. Bestick, R. Pandya, R. Bajesy, and A. D. Dragan. Learning human ergonomic preferences for handovers. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-9. IEEE, 2018.
[37] Y. Cui and S. Nickum. Active reward learning from critiques. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 6907-6914. IEEE, 2018
[38] M. Palan, N. C. Landolfi, G.
Shevchuk, and D. Sadigh. Learning reward functions by integrat- ing human demonstrations and preferences. In Proceedings of Robotics: Science and Systems (RSS), June 2019. doi: 1ed::1..1566771 15607/rss. 2019/xxv...2
[39] E. Bıyık, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. arXiv preprint arXiv:2006. /4091, 2020.
[40] L. M. Zintgraf, D. M. Roijers, S. Linders, C. M. Jonker, and A. Nowé, Ordered preference elicitation strategies for supporting multi-objective decision making. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 1477- 1485. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
[41] E. Biyik and D. Sadigh. Batch active preference-based learning of reward functions. In Con- ference on Robot Learning, pages 519-528, 2018.
[42] E. Biyik, D. A. Lazar, D. Sadigh, and R. Pedarsani. The green choice: Learning and influencing human decisions on shared roads. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 347-354. IEEE, 2019.
[43] E. Biyik, N. Huynh, M. J. Kochenderfer, and D. Sadigh. Active preference-based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020. 10..11.1666//ss22200...
[44] F. Memarian, Z. Xu, B. Wu, M. Wen, and U. Topcu. Active task- lask-inference-re inference-guided deep inverse reinforcement learning. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 1932-1938. IEEE, 2020.
[45] H. Sikchi, A. Saran, W. Goo, and Nickum. A ranking game for imitation learning. arXiv preprint arXiv:2202.03481, 2022.
[46] A. Jain, Sharma, Joachims, and Saxena. Learning preferences for manipulation tasks from online coactive feedback. The International Journal of Robotics Reseurch, 34(10): 1296 1313, 2015.
[47] A. I.. Thomaz and C. Breazeal. Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence, 17266-7)716--37, 2008
[48] O. Amir, E. Kamar, A. Kolobov, and B. J. Grosz. Interactive teaching strategies for agent training. In Proceedings of the Twenty- Fifth International Joint Conference on Artificial Intel- ligence, pages 804-811. AAAI Press, 2016.

---

[49] M. K. Ho, M. Littman, J. MacGlashan, F. Cushman, and J. L. Austerweil. Showing versus doing: Teaching by demonstration. In Advances neural information processing systems, pages 3027-3035, 2016.
[50] D. H. Uttal, K. O'Doherty, R. Newland, L. L. Hand, and J. DeLoache. Dual representation and the linking of concrete and symbolic representations. Child Development Perspectives, 3(3):
156-159, 2009.
[51] J. S. DeLoache. Dual representation and young children's 's use of scale models. Child develop- ment, 71(2):329-33 2000.
[52] D. Kahneman. Thinking, fast and slow. Macmillan, 2011.
[53] M. K. Ho, D. Abel, C. G. Correa, M. L. Littman, J. Cohen, and L. Griffiths. People construct simplified mental representations to plan. Nature, 606(7912):129- 136, 2022.
[54] J. Johnson, R. Krishna, M. Stark, L.JJ. Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3668-367 2015.
[55] I. Armeni, 7.-Y. He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5664-5673, 2019.
[56] Z. Ravichandran, L. Peng, N. Hughes, J. D. Griffith, and L. Carlone. Hierarchical represen- tations and explicit memory: Learning effective navigation policies on scene graphs using graph neural networks. In IEEE International Conference on Robotics and Automation (ICRA), 2022.
[57] N. Hughes, Y. Chang, and Carlone. Hydra:
real-time spatial perception engine for 3d scene graph construction and optimization. arXiv preprint arXiv:2201. 13360, 2022.
[58] S. Ross, G. J. Gordon, and D. Bagnell. A reduction of imitation learning and structured pre- diction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.
[59] M. Kelly, C. Sidrane, K. Driggs-Camphell, and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083, IEEE, 2019.
[60] A. Mandlekar, D. Xu, R. Martín-Martin, Y. Zhu, L. Fei-Fei, and S. Savarese. Human-in-the- loop imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733 2020.
[61] A. Bobu, M. Wiggert, C. Tomlin, and A. D. Dragan. Inducing structure in reward learning by learning features. The International Journal of Robotics Reseurch, page 02783344921008031,, 2022.
[62] T. Silver, R. Chitnis, J. Tenenbaum, L. Kaelbling, and T. Lozano-Pérez. Learning sym- bolic operators for task and motion planning. In 2021 IEEE/RS] International Conference Intelligent Robots and Systems (IROS), pages 3182-3189, IEEE.
[63] T. Silver, A. Athalye, J. B. Tenenbaum, T. Lozano-Perre Perez, and L. P. Kaelbling. Learning neuro- symbolic skills for bilevel planning. arXiv preprint arXiv:2206.1066 2022.
[64] R. Chitnis, T. Silver, J. B. Tenenbaum, T. Lozano-Perez, and I. P. Kaelbling. Learning neuro- symbolic relational transition models for bilevel planning. arXiv preprint arXiv:2105.14074, 2021.
[65] J. Achterhold, M. Krimmel, and J. Stueckler. Learning temporally extended skills in continu- ous domains as symbolic actions for planning. arXiv preprint rrXiv:2207.05013 2022.
[66] A. D. Dragan and S. S. Srinivasa. Formalizing assistive teleoperation. MIT Press, July, 2012.
[67] N. Amirshirzad, A. Kumru, and F. Oztop. Human adaptation to human-robot shared control. IEEE Transactions on Human-Machine Systems, (2-2))126-1333 2019.
12

---

[68] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Inter- pretable representation learning by information maximizing generative adversarial nets. Ad- vances in neural information processing systems, 29, 2016.
[69] R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[70] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
[71] D. Brown, R. Coleman, R. Srinivasan, and S. Nickum. Safe imitation learning via fast bayesian reward inference from preferences. International Conference on Machine Learning, pages 1165-1177 PMLR, 2020.
[72] D. Abel, D. Arumugam, K. Asadi, Y. Jinnai, M. L. Littman, and L.. L. Wong. State abstraction as compression in apprenticeship learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3134-3142,
[73] C. Agia, K. M. Jatavallabhula, M. Khodeir, O. Miksik, V. Vineet, M. Mukadam, L. Paull, and F. Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs In Conference Robot Learning, pages 46-58. PMLR, 2022.
[74] Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu. Hierarchical planning for long-horizon manip- ulation with geometric and symbolic scene graphs. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6541-6548 IEEE, 2021.
[75] S. Nguyen, O. S. Oguz, V. N. Hartmann, and M. Toussaint. supfvuuee learning of scene- graph representations for robotic sequential manipulation planning. In CoRL, pages 2104- 2119, 2020.
[76] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[77] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.09950 2018.
[78] E. Biyik, A. Talati, and D. Sadigh. Aprel: A library for active preference-based reward learning algorithms. arXiv preprint 108.::2/0..07259, 2021.
[79] G. Brockman, V. Cheung, I.. Pettersson, Schneider, Schulman, J. Tang, and Zaremba. Openai gym. arXiv preprint arXiv: 1606.01540, 2016.
[80] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Marttn... robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293,, 2020.
13


Reviews:
[{"review_id": "oaZB1GO2BYj", "reviewer": "Reviewer_32aJ", "summary": "The paper presents methods for learning from human guidance that leverage scene graphs to more efficiently elicit feedback. In the evaluative feedback setting, a UCB-inspired algorithm is developed to query humans based on prediction error in the preferences for given abstract states. In the learning from preferences setting, the proposed algorithm selects query trajectory pairs based on similar abstract states. It is shown that this scheme yields better asymptotic performance in inferring a reward function. On the whole, the paper demonstrates that formulating queries based on abstract states rather than low-level states is an effective mechanism for obtaining quality feedback from human users. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-motivated and studies an important problem. The proposed algorithms demonstrate clear improvements in performance over the baselines. The paper is generally well-written and clear. \n\nThe learning-from-preferences experiment is a bit lacking. The data generation scheme (random trajectories) could be more efficient (e.g. something online, like DAgger, or using human demonstrations), and it would be nice to see learning curves using the estimated reward functions rather than just the alignment scores.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I’m recommending weak acceptance - while the evaluation has some issues, I think the core idea is novel and the results demonstrate the effectiveness of the method. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "A rather similar idea was explored in the paper [Active Task-Inference-Guided Deep Inverse Reinforcement Learning](https://arxiv.org/pdf/2001.09227.pdf), though I think the novelty claim of this paper is preserved. It’s worth a read through [Memarian’s thesis](https://repositories.lib.utexas.edu/handle/2152/88568) nonetheless. \nConsider citing and/or using recent work from Scott Niekum’s group on learning from preferences - see “A Ranking Game for Imitation Learning”. \n\nI’m a bit perplexed surrounding the incorporation of the human feedback into the learning objective for the ER setting. It seems the model is trained with equation (3) to predict the scalar feedback signal $H$, not necessarily to maximize it. It is stated that the agent is trained to maximize human feedback and environment reward simultaneously, but the given loss function seems to minimize the standard SAC loss plus the difference between $\\hat H$ and $H$, minimizing the prediction error about the human feedback (rather than maximizing it’s value). TAMER+RL, DQN-TAMER etc. suggest several ways in which to incorporate the feedback into the RL objective, e.g.  incorporating into the value targets, or choosing actions from some combination of Q and H, but these do not appear to be in use here. I suppose the gradients of the feedback prediction term are providing some learning signal? \n\ntypo line 126 “receveid”"}}, {"review_id": "fWYWsHjLhj7", "reviewer": "Reviewer_M5fV", "summary": "This paper proposed and evaluated two methods that reduce the number of times the robot asks for human feedback while learning different skills. The policies are learned in the continuous sensorimotor space. Yet, the robot exploits symbolic state information when it queries feedback/preferences from the humans. The main contribution of the paper is to exploit symbolic state information in deciding when and how to ask for human guidance/feedback. The first method enables the robot to ask for human feedback considering the robot's uncertainty about the symbolic states (based on their visitation count). The second method enables the robot to select a pair of trajectories again based on symbolic representations and ask which trajectory the humans prefer. The method is validated with 3 simulated and 2 real robot experiments. The method is mainly compared against a baseline that does not use any human feedback and also compared against baselines that use human feedback %N of the time.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well-written and clear.\n- The experimental evaluation is thorough in terms of the number and diversity of both simulated and real robot experiments.\n- Linking and exploiting representations at different levels is important and have the potential to make a high impact in general. The robots need to learn low-level sensorimotor control, but for generalization and for high-level cognitive skills such as planning, reasoning, and communication, high-level symbolic knowledge becomes important. We need architectures that effectively combine different levels of representations. In this sense, the paper has the potential to make important contributions to the literature.\n- I have three major comments:\n1- The paper lacks providing a coherent and powerful message as it proposes, implements and evaluates two different human guidance methods with almost no connection in between. I can understand that the main point is that both of these human guidance methods exploit symbolic representations, still, the paper is written as two papers, i.e. with two separate methods and two separate experiments sections. This really reduces the potential impact of the paper.\n2- The methods were compared against some baselines. However, in order to strongly support the main claim (use of symbols in human guidance brings efficiency), the methods should be compared with human guidance without symbols (e.g. TAMER+RL [17,18,19]). Without such comparison, it is not possible to convince and justify the advantage of the dual representation framework in tasks where human guidance is used.\n3- The authors should drop the statements related to encoding/understanding \"human internal representations\", which is emphasized especially in the Introduction section. Some examples: Does aligning better with human internal state really correspond to encoding information that motivates human input (abstract)? Lines 31-33, related to understanding the internal representation of humans ([43-45] do not have any claim related to such understanding).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is clearly written and works on an important topic, using dual representations to speed up learning (via minimizing the human guidance queries). The methods are also sound. However, the advantage of using symbols in learning with human guidance was not vigorously shown. Additionally, the central claim of the paper is not clear (which type of guidance should we use) and the methods and related experiments parts are not connected.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Some other comments:\n- Symbolic scene graphs are emphasized. However, what I see is a binary vector that represents the environment. The full power of the graph structure is not used (e.g. object-object relations are not used). So emphasizing on the graph structure is not necessary.\n- The claim on \"humans adopting the slow system\" in teaching is not well-justified. In teaching some skills, very low-level detailed trajectories might be important, in teaching some others, high-level representations.\n- Lines 48-50: The correspondence between Evaluative Feedback and uncertainty awareness is not clear. The correspondence between Preference Learning and scene graph-based trajectory segmentation is not clear.\n- Line 54: The authors missed important recent hybrid skill learning approaches such as Silver et al., 2022; Chitnes et al., 2021; Achterhold, 2021.\n- The authors can include shared control reviews: Dragan and Srinivasa, 2012; Amirshirzad, 2019.\n- Line 91: Unjustified statement. The tasks are probably represented at different levels also in humans.\n- How \"maximization of human feedback\" is integrated into the loss in lines 131-132 is not clear.\n- Line 136 (and line 7 in Algo 1): It is not clear exactly which state is queried from the ones ranked smaller than k.\n- Line 141: Unjustified statement related to being \"meaningful\" to humans.\n- Line 154: Related to the length of the trajectories: Why does your initial human insight idea not work here? No human would sum up any reward, they would simply say this is better than the other.\n- Is IRL implemented in Section 3.3? What are p and P (also in Algo 2)?\n- Line 209: \"less effort\": Effort was not measured. use \"reported effort\"\n- What is alignment score? Where is it defined? Was it not possible to provide the results using a common metric in Sections 4.1 and 4.2?\n\n\nSilver, Tom, et al. \"Learning Neuro-Symbolic Skills for Bilevel Planning.\" arXiv preprint arXiv:2206.10680 (2022).\nChitnis, Rohan, et al. \"Learning neuro-symbolic relational transition models for bilevel planning.\" arXiv preprint arXiv:2105.14074 (2021).\nAchterhold, Jan, et al. \"Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning.\" arXiv preprint arXiv:2207.05018 (2022).\nDragan, Anca D., and Siddhartha S. Srinivasa. Formalizing assistive teleoperation. MIT Press, July, 2012.\nAmirshirzad et al. \"Human adaptation to human-robot shared control.\" IEEE Transactions on Human-Machine Systems 49.2 (2019): 126-136.\n\n"}}, {"review_id": "MJCGx72zXNI", "reviewer": "Reviewer_JYfb", "summary": "The authors present an approach for learning robot skills under human guidance. They propose two independent algorithms, one for learning from human evaluative feedback and one for learning from human preferences. The basis for their approach are two different representations, one continuous representation (states and actions) of the robot and one discrete representation of the task. While the continuous representation is used for training the RL agent, the discrete more abstract representation of the feedback is used for deciding when and how to ask the human for guidance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well written and straightforward to follow\n- Number of experiments is adequate (both simulation and real experiments)\n\nWeaknesses\n- According to my understanding the algorithms are not mutually exclusive. I do not understand why there is no experiment where it is tested if using both algorithms at the same time further improves the learning efficiency. If the algorithms can't be combined, the authors should clarify why.\n- Scientifically, it would have been interesting to evaluate how different scene graph representations of the same task effect the learning efficiency and task performance. However, I also think this could be follow-up work that does not have to be included in this paper.\n- The tasks in the experiments can be solved by an unambiguous state-action pair mapping (no history of actions is required for solving a task). Therefore, the RL agent does not need to know the scene graph which otherwise would be needed to disambiguate the continuous states. Therefore, no coupling between the two representations is necessary and they can be kept independent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think it is a solid paper without bigger weaknesses. With the dual representation, the authors shifted a lot of the complexity of the learning problem to the designer of the scene graph. Designing a good scene graph is crucial for task success, as the authors also stated in the limitations section. Therefore, I see this paper as an interesting first step in the right direction.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- From reading the paper, it is unclear how the scalar guidance signal H is genereated for the synthetic human experiments. I had to read the Appendix in order to understand how it is done. In my opinion the explanation must be in the paper and not in the Appenxid since it is crucial for understanding the learning process.\n\n- Line 126 receveid -> received\n\n- I was confused by the term \"alignment score\". Is it an established term in the preference learning / human guidance community? I guess you chose the term since it measure how well the predicted preference aligns with the true human preference. From a learning perspective, in your context, it is the error between the predicted and actual reward and therefore I would simply use a more general term like \"Avg. Prediction Error\". If you think alignment score is more fitting, I would recommend adding some explanation to the text (and maybe to Figure 5)."}}, {"review_id": "-m1qgyZj1j-", "reviewer": "Reviewer_8h9i", "summary": "This work advocates for aligning robot representations with human internal state representations. The core hypothesis of this work is that a symbolic scene graph (where objects are nodes and relationships between them are edges) is an effective representation for such alignment. With this idea, the paper proposes two learning algorithms: learning from evaluative human feedback (which is formulated as a multi-armed bandit) and learning from human preferences. The idea and algorithms are evaluated with simulated and real human data and on simulated and real robots. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n--------------\n–Interesting and timely choice of problem that is valuable to the HRI community\n\n–Interesting proposed representation of a scene graph\n\n–Experiments in both simulation and with a real robot, and with simulated and real people\n\nWeaknesses:\n-----------\n–There seems to be a disconnect between the paper’s idea / claim and the proposed evaluation. The hypothesis that scene graphs are a useful representation is very interesting, but I would expect much of the evaluations to be focused on comparing different existing representations to the proposed one. For example, one could compare the scene-graph representation vs. assuming that the feature space used in the reward function is the person’s representation vs. extracting some learned latent state representation in an unsupervised way (e.g., via autoencoders) vs. having no explicit intermediate representation. However, the comparisons focus only on different methods for querying for human feedback (e.g., SAC model with no human feedback vs. requesting for human feedback 50% of the time, etc.). While a very important comparison, I think this is not properly testing the core hypothesis of this paper: that scene-graph representations help humans teach robots more effectively as compared to alternatives (like no intermediate representation or other representations). \n\n–The paper proposes two alternative learning methods for the evaluation-based and preference-based learning settings. While it is ok to have different algorithmic approaches for different types of feedback, it was difficult to understand their connection or their need for the scene-graph itself. For example, in preference-based learning, the robot has a reward represented as a linear combination of features. However, this reward is not used in the evaluation-based learning setting – why not? Furthermore, what is the relationship between the reward function and the scene graph representation? It seems like the scene graph could be used as a “featurization”, and then existing SOTA preference-based learning methods (like the ones cited in the related work) could be leveraged. Why is it desirable to decouple the graph-based representation from the reward featurization in the proposed method? Answering these questions both through the text and through ablation studies would significantly strengthen the paper’s claim. \n\nSection 2:\n\n–”We explore how a symbolic scene graph could also be a useful intermediate representation for human-in-the loop robot learning, under the assumption that it aligns better with human representation.” What is the symbolic scene graph being compared to here? It would be helpful to explicitly mention here that the symbolic scene graph is hypothesized to do better than <insert the other approaches here> (e.g., learning without intermediate representations, learned latent-state representations, hand-crafted feature representations, etc.)\n\n–The related work could be strengthened by discussing the representation learning literature from machine learning, and the similarities / differences from the proposed work. For example:\n\n[1] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in Proceedings of the 30th International Conference on Neural Information Processing Systems,\nser. NIPS’16. \n\n[2] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerchner, “beta-vae: Learning basic visual concepts with a constrained variational framework,” in ICLR, 2017.\n\n[3] R. T. Q. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud, “Isolating sources of disentanglement in variational autoencoders,” in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018.\n\n[4] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum, “Safe imitation learning via fast Bayesian reward inference from preferences,” in Proceedings of the 37th International Conference on Machine Learning, 2020.\n\n–Some related work that would be valuable to include (the former is about aligning robot representations with human representations, the latter is work on online learning from human evaluative feedback):\n\n[1] Bobu, Andreea, et al. \"Inducing structure in reward learning by learning features.\" The International Journal of Robotics Research (2022): 02783649221078031.\n\n[2] Jain, Ashesh, et al. \"Learning preferences for manipulation tasks from online coactive feedback.\" The International Journal of Robotics Research 34.10 (2015): 1296-1313.\n\nSection 3:\n\n–The claim ”Integrating these methods with continuous control enables efficient uncertainty-aware active learning from humans” should be substantiated a bit more in the text. For example, are there prior works that achieve this? If so, please cite them. \n\n–”The agent learns a policy to maximize human feedback and environment reward simultaneously.” This objective seem counter-intuitive – it should be desirable to find a policy which *minimizes* human feedback (to decrease human effort) and *maximizes* environmental reward? Perhaps this was a typo. \n\n–”In this way, the dual representation leads to efficient uncertainty-aware active learning from human feedback.” This claim should also be substantiated with a proof, numerical results, or a citation, since the presentation of the algorithm does not immediately demonstrate that the proposed dual representation yields more efficient active learning. \n\nSection 4:\n\n–The right-most table in Figure 4 is difficult to decipher. Are these the results of the Likert Scale Survey? If so, it would be helpful to title it as such. Additionally, it would be helpful to include the specific questions that were asked in the survey, as well results of an ANOVA and the p-values for each question to indicate if the results are statistically significant. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the problem choice and high-level idea (of using a scene graph as a way to align human representations with robot representations) are relevant and interesting, I am concerned that there is a disconnect between the proposed approach and the way in which it has been evaluated. Specifically, I believe that more experiments need to be conducted to compare the scene-graph representation to alternative representations, and show how the proposed method hits the sweet spot of low-sample complexity / low perceived human effort while achieving high reward when compared to alternative representations (e.g., hand-engineered featurizations, automatically-extracted featurizations, etc.). Thus, I recommend weak reject. \n\n==================================================\nAfter discussion, I changed my recommendation to \"weak accept\".", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The following are summarized from the weaknesses mentioned above:\n\n–Please elaborate on the disconnect between the paper’s idea / claim and the proposed evaluation. The hypothesis that scene graphs are a useful representation is very interesting, but I would expect much of the evaluations to be focused on comparing different existing representations to the proposed one. For example, one could compare the scene-graph representation vs. assuming that the feature space used in the reward function is the person’s representation vs. extracting some learned latent state representation in an unsupervised way (e.g., via autoencoders) vs. having no explicit intermediate representation. I believe further evaluation is needed to properly test the core hypothesis of this paper: that scene-graph representations help humans teach robots more effectively as compared to alternatives (like no intermediate representation or other representations). \n\n–The paper proposes two alternative learning methods for the evaluation-based and preference-based learning settings. Please clarify the connection between them and the need for the scene-graph itself. For example, in preference-based learning, the robot has a reward represented as a linear combination of features. However, this reward is not used in the evaluation-based learning setting – why not? Furthermore, what is the relationship between the reward function and the scene graph representation? It seems like the scene graph could be used as a “featurization”, and then existing SOTA preference-based learning methods (like the ones cited in the related work) could be leveraged. Why is it desirable to decouple the graph-based representation from the reward featurization in the proposed method? Answering these questions both through the text and through ablation studies would significantly strengthen the paper’s claim.\n\n"}}]

Author rebuttal:

Decision: Accept (poster)