"Good Robot! Now Watch This!": Repurposing Reinforcement Learning for Task-to-Task Transfer
Authors: Andrew Hundt
Aditya Murali
Priyanka Hubli
Ran Liu
Nakul Gopalan
Matthew Gombolay
Gregory D. Hager
Venue: CoRL2021 Poster (2021)
Keywords: Deep Learning in Grasping and Manipulation
Computer Vision for Robotic Applications
Imitation Learning
Reinforcement Learning
Learning from Demonstration

Abstract: Modern Reinforcement Learning (RL) algorithms are not sample efficient to train on multi-step tasks in complex domains, impeding their wider deployment in the real world. We address this problem by leveraging the insight that RL models trained to complete one set of tasks can be repurposed to complete related tasks when given just a handful of demonstrations. Based upon this insight, we propose See-SPOT-Run (SSR), a new computational approach to robot learning that enables a robot to complete a variety of real robot tasks in novel problem domains without task-specific training. SSR uses pretrained RL models to create vectors that represent model, task, and action relevance in demonstration and test scenes. SSR then compares these vectors via our Cycle Consistency Distance (CCD) metric to determine the next action to take. SSR completes 58% more task steps and 20% more trials than a baseline few-shot learning method that requires task-specific training. SSR also achieves a four order of magnitude improvement in compute efficiency and a 20% to three order of magnitude improvement in sample efficiency compared to the baseline and to training RL models from scratch. To our knowledge, we are the first to address multi-step tasks from demonstration on a real robot without task-specific training, where both the visual input and action space output are high dimensional. Code will be made available.

"Good Robot! Now Watch This!": Repurposing Reinforcement Learning for Task-to-Task Transfer
Andrew Hundt
Aditya Murali
Priyanka Hubli
Ran Liu
Nakul Gopalan
Matthew Gombolay² Gregory D. Hager!
Johns Hopkins University Georgia Institute of Technology
Equal contribution (ahundt, amurali6) fu. thu.e. edu
A Abstract: Modern Reinforcement Learning (RI.) algorithms are not sample- efficient to train on multi-step tasks in complex domains, impeding their wider deployment in the real world. We address this problem by leveraging the insight that RL models trained to complete one set of tasks can be re-purposed to complete related tasks when given just handful of demonstrations. Based upon this insight, we propose Lee-SPOT-RAE (SSR), new computational approach to robot learn- ing that enables robot to complete variety of real robot tasks in novel problem domains without task-specific training. SSR uses pretrained RL models to cre- ate vectors that represent model, task, and action relevance in demonstration and test scenes. SSR then compares these vectors via our Cycle Consistency Distance (CCD) metric determine the next action to take. SSR completes 58% more task steps and 20% more trials than a baseline few-shot learning method that requires task-specific training. SSR also achieves four order of magnitude improvement in compute efficiency and a 20% to three order of magnitude improvement in sam- ple efficiency compared to the baseline and training RL models from scratch. To our knowledge, we are the first to address multi-step tasks from demonstration on real robot without task-fpecific training, where both the visual input and action space output are high dimensional. Code is available in the supplement.
Introduction
Before Demonstratio
Train
Demo x2 Vertical Square Tack
RL Training
Trained Models
Remove
Embreding Function
Stock Task
Last Layer
II. Collect Demonstrations of Test Task
Furceions
Demonstration
(D)...a))
Compate Embeddings
Row Task
(Staes Actions
H1 Bis)
XMN
Test Real Vertical Square Trok
Cached Demoo Embeddings lHg)
III. Text Time
Cached Functions TE1
Unstack Task
State 4
Enviromment
See-SPOT-RRm Run Policy Rar
Action
Reyest notil Tisk Success
Figure 1: Left: Reinforcement Learning Before Demonstration (RLBD) paradigm. Right: One of four folds for leave-one-our cross-validation with sim to real transfer and novel test task.
The ability to quickly adapt deep Reinforcement Learning (RL) models to new tasks with just handful of demonstrations would dramatically enhance their practicality and applicability. Repur- posing periousslytrriied models for new tasks consumes several orders of magnitude fewer com- putational resources than training from scratch. For our approach repurposing RL models, we draw inspiration from social imitation and learning in animals. Certain animals, dogs. for example, have a strong capacity to imitate human performance of novel task when trained with the "Do as Do" method [1, 2]. Imagine a dog named Spot that can be been trained with positive reinforcement
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

RI. Policies SPOT-Q XM
II + III. See See-SPOT- -uO
Simulated Demo State
Real Robot Live Test State
Meal Dono
Live Red
Enchedlings
Astion
DaneeNel PCN
Embedding Funczion
Eibedding Punetion Bu.
Emb:aling Function
Fanction
Frnction
Punction Il.
Function Byg
Fixdwise Embedding (ASEg
f corr
Foler rabion on Blst Har
Euroaee Sruup Autios Concloat
Correpaa
All HEak
Figure 2: Left: An outline of our pretrained RL policies trained with SPOT-Q [4]. Each
RL policy contains Embedding Function E that outputs pixel-wise (per action) embeddings (16x224x224x64 or HexXXY ASEs). These embeddings are turned into Q-values (16x224x224x1)
by linear layer. The action space A has the same dimension as the Q-values (16x224x224x1, or OxXxY) for each grasp and place action. Right: See-SPOT-Run (SSR) Framework architecture executing task, the grasp action is selected from the vertical square policy. SSR successfully executes novel tasks using correspondence function foorr (Eqs. 4-8) that matches the demonstrated action the live test scene using the embeddings (ASEs) derived from each pre-trate pre-trained policy.
("Good Spot!") to imitate new action shown to her, such as fetching specific object from a con- tainer, when prompted by command ("Watch This!"). When Spot imitates, "it the orientation and the shape of the action [that is] attended to and reproduced" [2, 3]. Spot encodes what she sees in an internal representation, which she may then map onto her trained skillset in order successfully imitate a new task.
We observe that some deep RL models, like Spot the dog, already encode meaningful represen- tations of a scene that can be mapped to a skillset. We hypothesize that we can leverage these encodings solve new tasks from demonstration without task-specific learning, concept we call Reinforcement Learning Before Demonstration (RLBD, Fig. 1). To evaluate our hypothesis, we propose the Sec-SPOT-Run Framework for RLBD.
See-SPOT-Run (SSR, Fig. 2) rapidly adapts an ensemble of existing RL policies to new tasks by imitating demonstrations. Intuitively, existing fully-trained and validated RL models already invariant to certain changes in the scene within the tested circumstances. We pass a known state and novel state to a preexisting model to create and then compare latent vectors in search for embedded task-invariant knowledge to use in novel cireumstances. When the known state is from novel task demonstration, and the novel state is from a live test scene, vectors that map to actions in the test environment that are likely to progress towards completing the novel demonstrated task. Therefore, we apply See-SPOT-Run to imitate demonstrations of new multi-step task ("Watch This!") by matching demonstrated actions to candidate actions in test environment. In doing so, the potential to vastly expand robotic skill sets at dramatically lower costs.
can pinpoint latent
only enable few-shot imitation of unseen tasks but also provide
In few-shot imitation experiments, demonstrate See-SPOT-RRut ability to complete set of multi-step tasks, including block stacking, row making. block unstacking, and building a vertical square structure (see Fig. 1). These tasks are order of magnitude more challenging than typical reach or limited grasp and place tasks [4] (see Sec 3). To evaluate each task, we use RI policies conflicting (stacking unstacking) tasks, and provide two expert demonstrations collected in simulation. In summary, our contributions are as follows:
trained complete different, and
1. Reinforcement Learning Before Demonstration (RLBD):
We propose concept RL policies to accelerate robotic imitation learning. We show that these policies generate Action- State Embeddings (ASEs) from robot state observations, and that these embeddings be purposed to execute unseen tasks.
pretrained
2. See-SPOT-Run (SSR): Our RLBD implementation that accomplishes both few-shot imitation and zero-shot sim-to-real domain transfer by matching successful demonstrated actions to the test environment with novel Cycle Consistency Distance (CCD) metric.

---

3. SSR achieves competitive task performance without the task-specific training required by prior work, leading to four order of magnitude improvement in compute efficiency and a 20% to three order of magnitude improvement in sample efficiency for novel tasks when compared to the baseline and to training RL models from scratch.
Related Work
Deep learning approaches have successfully completed a range of robotic tasks in visually-guided
manipulation [5, 6] and navigation with high sample complexity [7. 8, 9, 10, 11, 12, 13]. VPG [6] achieved particularly efficient RL training for decluttering tasks. "Good Robot!" [4] improved on VPG's RL sample efficiency and showed zero-shot sim-to-real transfer for multi-step block tasks with risk of progress reversal [4], e-g. toppling partially completed tower (see Sec. 3); however, it requires from-scrater task-specific training.
Another approach to tackling deep RL's high sample complexity is through multi-task and meta- learning [14, 15, 16, 17, 18, 19, 20] of general purpose task representations or policies that allow an agent quickly adapt to novel tasks by fine-tuning previously learned policies. James et al. [16] approach the few-shot learning problem by first meta-learning a task embedding network from a set of training tasks, then computing a test task embedding from few demonstrations; this task embed- ding is provided as input to a control policy. Meanwhile, Finn et al. [14] perform one-shot imitation on tasks seen during meta-training but evaluated on unseen objects. While these approaches achieve good performance on one-step tasks, they do not consider multi-step tasks or progress reversal. Al- ternately, few-shot inverse RL methods [21, 22, 23] aim to increase RL sample efficiency by learning a reward function from one or a few demonstrations, then using this reward function to supervise RL training. Yet, Liu al. [21] and Singh et al. [22] do not address learning multi-step tasks, which can require dense reward functions to learn effectively [4, 10]. Goo and Niekum [23], in turn, propose an IRL approach for multi-step tasks that requires time and compute-intenen inteesive training. Zhu et al. [24] reviews Transfer Learning, and our work can be framed as Representation Transfer.
mechanism for comparing representations across domains [25, 26, 27J. Zhang et al. [28] utilizes Cycle Consistency for cross domain dynamics in simulated locomotion tasks and sim-to-real transfer of basic robot reaching task, showing that Cycle Consistency can mitigate domain shift for the same, single-step, task. Our See-SPOT-Ruun framework, by contrast, completes both sim-to-real and known-to-novel task transfer with more complex multi-step tasks via our own cycle consistency concept.
Cycle Consistency
Behavior Cloning (BC), Imitation Learning (IL. [29, 30, 31, 33, 35), and IL from Observation (ILO) [31], are approaches to mimic the behaviors of agents. Recent reviews cover IL [36, 37] and ILO [38]. Jung and Kim [39] take hybrid BC and IL approach to grasp and place one block on another block. GAIL [40] learns to imitate with inspiration from generative adversarial networks. Others pursue distribution matching approaches [33, 25, 30J. However, these BC, II, and ILO papers variously require training on novel tasks, are evaluated only on simulated tasks, or do not consider multi-step tasks with a risk of progress reversal [4]. NTP [41] and NTG [42] learn task plans in a blocks environment by imitating videos, but require perception pipeline to act.
TransporterNetss [29] models multi-step IL tasks characterized by progress reversal as a sequence of displacements via behavior cloning, with impressive results that scale from 1-1k demonstrations per task. However, TransporterNets makes no attempt at task-to-task transfer for different tasks. Instead, it requires real-world, task-specific demonstrations and training for every unique task, so their approach is simultaneously much easier to solve (direct task transfer) and much more difficult to adapt to new scenarios (due to task-specific training). In comparison, SSR addresses the problem of few-shot imitation with task-specific training. Even with these limitations, TransporterNets is the closest available method to SSR and consequently, we chose it as our baseline for comparison.
3 Preliminaries
As in "Good Robot!" [4] and VPG [6]. we investigate multi-step tasks by framing our problem
as a Markov Decision Process (MDP) (S, A, P. R), with state space 5; action space A; transition probability function P S x S x A - IR; and reward function R: S + R.. We make an MDP- simplifying assumption where states equal observations, and that there is discrete action space. At time step I, the agent observes state 8, and chooses an action d, according to its policy π S + A. leading to new state $141 with probability P( at). Q-Learning is an RL method whose
3

---

purpose is to seek a Q function that maximizes R over time. Q-Learning policies select an action a, time t by first estimating the reward with a function Q S x A + R, then selecting the action that maximizes the expected reward: = x (st) argmax ((),,
Our See-SPOT- Framework experiments are pretrained with the Q-Leeaming--asse SPOT-Q framework introduced by "Good Robot!" [4]. SPOT-Q demonstrates the importance of prioritizing progress reversal during RL training, and also effectively integrates 'common-sense' exploration constraints into Q-learning. Our RL policies are pretrained with the same exploration constraints and reward functions Rmat Rp as the best performing models in [4]. our development, we make integer indicating the number of successful task steps; the task is complete when P((x)) = Pmax- Consequently, a progress reversal [4] occurs when an action d, a timestep & undoes previous task progress i.e. there exists E [0, such that (st-i) > P(s++1). Next, we will introduce our proposed methods.
of task progress measure, P S - P.ue
4 Methods
Demn Stote
Stacking Task
( Live Test State
Albe Mork in Gutppor
SSR has three phases: (1) Embed, (2) Corre- spond, and (3) Act.
Tagge Beee Ploc Artia
Lox Black
Dass All
Tee AR2A
In phase (1) Embed, SSR takes a set of fully- trained policies, such as Q-functions, and con- verts them into embedding functions by extract- ing the latent per-action feature vectors embed- ded in the final network layer immediately be- fore Q-values are created (Fig. 2). SSR runs these embedding functions on every demonstra- tion and test state to make a unique latent space for each (policy, demonstration) pair.
Cheakey
Emieddiin Function
Embebdins Fanction E.j
funbettionn
High Cycle Constimey Leeunee
Palicy- Denoumatiio (PDP)
Cariddin Tou Antio
Does Alla
Toe Alds
Embedding Funciion E.
Embedding Ng
Lox Cycla Cooblteary Distance
Cevisra Teua Aution
Dena ASPA
Ted ASR
Embedding Function Bag
Embadding Function Eag
Phase (2) Correspond has two steps: (2a) Match and (2b) Verify. In (2a), SSR compares the embedded demonstration state to the em- bedded test state of each latent space, searching for the test action that most closely matches the successful demonstration action. It finds this closest math by minimizing the euclidean dis- tance between the demonstration action's em- bedding (feature vector) and each potential test action's embedding. Since matches are made in the embedding space, the matched action should mirror the qualities of the demonstrated action for the new task, and need not bear any resemblance to the action the original preexist- ing policy would choose for its original task. Intuitively, the high dimensionality of latent vectors ensures dissimilar actions do not collide, a manner analogous to hash functions. For example, successfully choose reasonable action matches within policies and across states (Sec. 5), but matches across policies were random and unsuccessful in small tests. Ultimately, this matching process yields set of candidate actions, one for each (policy, demonstration) pair.
High Cycle Cumbllacy Destane:
Canciia Tou Antin
Exocate Candidate from Bes Loves
Cyele Courisssey Dartner Torget
Alter Seccisial
Figure 3: An illustration of cycle consistency correspondence for policy-demonssration pairs (each row is a pair) as See-SPOT-Run (SSR, Fig. 2, Eq. 8) chooses to place hidden blue block al- ready in the gripper onto the blue block visible in the test state image.
In (2b) Verify, SSR reverses the process, starting with each test candidate action, retrieving its em- bedding, and performing the match process in the demo scene to see how that action would perform in the demo world. The physical cartesian distance between the known idcal demo action and our new rematch demo action in the action space is our cycle consistency distance metric (Fig. 3); intu- itively, actions that minimize this distance are more likely to accomplish the demonstrated behavior.
Finally, in phase (3) Act, SSR selects the candidate action that minimizes the cycle consistency distance and acts. It repeats this entire process until task completion. SSR thus conducts a com- prehensive search over all (policy, demonstration) pairs. jointly selecting the latent space and demonstration that are most relevant to the novel task as well as the action that will most likely advance task progress. In Section 4.1, we will describe how we construct the embedding func- tions, then in Section 4.2, we will elaborate on how SSR uses these embedding functions to imitate demonstrations, and provide additional intuition for our cycle consistency distance metric.

---

4.1 Generating Embedding Functions
We begin by pretraining task-specific SPOT-Q [4] policies Tm : m = 1...M on M known tasks: Recall that Q functions are defined v. = Q(s, a); we reframe this as V = Qm()), where V contains scores for every action a E A pertaining to task m. Thus, the Q-value of taking action a in state Qm (s) Finally, from each Qm we strip off the last linear layer Wm. linear projection (dense layer) that maps feature vector to a scalar Q-valee, to obtain as many Embedding F Functions' Em:
Tim (8) argmax Qm (m, = argmax (+) argmax Em (G)[(a)))uu Wm aЕA
aEA
Evaluating each Em an arbitrary state yields latent space array H composed of one Action- State Embedding (ASE) himaa = Hm [a] for each action space coordinate a E A, as in Fig. 2.
4.2 Test-Time See-SPOT-Run Policy
In this section, we will define our test-time SSR policy, mi" which observes the test state s and selects action de for the robot to execute, as it attempts complete the demonstrated test task. It solves the imitation problem: approximating an unknown optimal live test task action oj that will complete the next step of a novel, demonstrated task. SSR uses information from N demonstrations
D [Di,.., } Each demo D,, : {Sdemo, ademo) 1, ..... (8demo, Ademm T is sequence of State- Action pairs which maximize R at each task step / T.
(1) Embed: We begin by observing the test state 8; and progress p = P(s)) and we consider the N state-action pairs ( Ademo,n) with progress p from the N demonstrations'. We use these to compute latent space representations of each demonstration state and the test state with each embedding function Em yielding M N demo latent space arrays m,, and M test latent space arrays Hm,i
Hm,n Em (5demon))
(2)
Hm.t = Em
(2) Correspond: We find candidate test actions (2a), then verify these candidate actions with our cycle consistency distance metric (2b). To find a candidate action, we match the demo action to the test state by selecting the minimum euclidean distance between the demo action embedding (ASE) and all ASE vectors in the test latent space. We repeat this comparison for each latent space, yielding M x N total candidate actions Fm,.. one for each (policy, demonstrations pair. Candidate actions are expressed as coordinates in the action space A, and can be used to index quantities such as the latent spaces H, which have the same dimension A, as detailed in the Fig. 2 caption.
Lunn min || Hmm, Hm,, (. 1ll2 (4) Fm,n argmin A
|| [m,m Hm n [Adem.. (5)
We then verify each matched candidate test action in the demo scene using our cycle consistency distance metric. We verify, or match in reverse, the candidate action's ASE H Hm,,[Fm... [m,,.] to the
corresponding demonstration latent space Hm.n. minimizing euclidean distance in the latent space as above. We expect the verification action Gremaich to be physically close to the original demonstrated action ademo,n in the action space A. Therefore, we define the cycle consistency distance Cm,, as the Cartesian distance between ademo,n and Grematch-
Orematch argmin [.H.m Hm |a]l
(6)
aeA
Cmnn ||ademo
Orematch l2
The purpose of the cycle consistency distance Cm., to consider other ASE vectors in the latent space Hm.. that might indicate that a candidate action Fm... is not an authentic match. Alternatively, the rematch step can be interpreted as an assessment of how relevant a given latent space is to the test state and task at hand.
In equations, parenthesis () are for function arguments and brackets [] index arrays or discrete coordinates. We note that, because each target demonstration action is optimal, demos satisfy the property that they have monotonically increasing progress. Our method is trivially extended to suboptimal demonstrations by considering all state-action pairs with progress p.
'While Section 4.1 refers to latent spaces as H.., we extend this notation here, using Hm.... to designate the state from the n-th demonstration embedded with Em. Similarly, Hm.t is the test state embedded with Em..
5

---

(3) Act: We select and execute the candidate action with minimal Cycle Consistency Dis- tance (Eq 8). We also evaluate a baseline SSR without cycle consistency by skipping the rematch phase and directly minimizing the 1.2 Consistency Distance (Eq. 9) Imma-
m, f argmin Cun.. m.
m, i. = argmin Imm, m.n
az Fa,a
(8)
4
sar
Fm,
This completes our definition of the SSR policy a; Tar (s.). Every time an action oi com- pletes we collect new observation S1+1, running the SSR policy saa repeatedly until task progress ((sT) reaches its maximum, Pmax, indicating that the task is complete. We evaluate SPO-SPOT- Run with 1.2 consistency distance (SSR L,2DD in Tables 1, 2, and 3) and cycle consistency dis- tance (SSR CCD Tables I, 2, and 3), as well as prior work, with few-shot experiments in Sec. 5. We will also discuss how our SSR Framework with CCD surpasses the other methods.
Novel Task Efficlency and Novel Task Progress Simulation
Experiments
10%
SSR CCT
Ouurs
We outline our assessment metrics in Sec. 5.1, provide simulation results in Sec. 5.2, then cover our sim-to-real transfer results in Sec. 5.3. We pretrain models with SPOT-Q [4]
TransporterNet
57%
Finetuned
SSR CCD
Sumple Efficeeccy -Lag Scale
SSR L3CD
79%
Robat
ThansporterNet
67%
Fimetianed
Fineluned
(Sec. 3) with the workspace, commands and action space defined in "Good Robot!" [4].
0.1%
More Eliccent
Legend
Task
Implementation and Robot details are in Fig. and Appendix C.I.
Progress 100%
0.01%
5.1 Evaluation Metrics
"Gond Robot!
Prugress, avenuee
raek eaeps complearat.
96 %
10046 shows for stale.
Our metrics quantify the broad improvements in task performance and reductions in the re- sources necessary to perform novel tasks. Most important, we consider critical efficiency mea- sures that have not previously been evaluated in the baselines, to motivate the broad range of useful applications for both RBLD and SSR.
0.001%
1%
Compute KMfieeeny
Log Seale
Figure 4: Higher is better all metrics (Sec 5.
Log Scale. Larger bubbles are better. See Tab. 1.
Test Metries evaluate how effectively the robot completes the test tasks, and higher is better: (1)
Trial Success Rate (Trials) is the percentage of multi-step tasks completed 100% successfully, and many applications completing task a prerequisite to moving on to the next task. (2) Action Efficiency (Eff.) is the ideal number of actions per trial, and more efficient models will complete tasks with fewer actions. [6] Our ideal 6 actions for all tasks except for rows, which is actions. [4] (3) Progress (Prog.) is each trial's 's maximum proportional within-task progress averaged over all trials, e.g. a block stack is 75%, to show capabilities that nearly complete trials. (4) Recoveries (Recov.) is the percentage of trials in which there was mistake such progress reversal that the agent able to complete with trial success, i.e. tril successes containing progress reversal trials containing progress reversol rates reflect better robustness to perturbations and uncommon situations.
actal
Higher recovery
Cost Metries delineate resources spent, and lower is better: (5) Train Steps is the number of neural network batch steps performed prior to executing on novel test task. Each individual experiment run on one NVIDIA GeForce RTX 2080Ti GPU. (6) Annotated Actions (Ann. Actions) is the number of robot actions a, that have been annotated by either a human scripted observer, which are each fairly high cost activitics in robot or human time [43, 44].
Efficiency Metries evaluate trial success benefits with respect to the cost metrics, and higher is bet- Trials is the amortized percentage of trials that can be completed ter. (7) Compute Efficiency Train Steps++1
for every training batch step when completing a novel task. We add one to the train steps denomi- Trials
nator to prevent dividing by 0 with SSR. (8) Sample Efficiency Ann. measures the increase in trial success rate amortized over annotated actions on a test task.
5.2 Simulation Experiments
We pretrain RI. policies on four tasks as in Fig. 1: stacking, row-making, unstacking, and 2x2 vertical square. At test time perform four-fold cross- coos-validatiin where the test task is given two

---

demonstrations but no pretrained model, then the remaining three models predict actions for the un- trained test task. For the finetuned case, we pretrain each of the three models on two demonstrations of the untrained test task for 333 steps each, for a total of 1K steps. We then apply SSR as described in Sec. We additionally investigate "Good Robot" finetuning baseline where we use one of the finetuned models directly as the test-time policy, without applying SSR.
To obtain our TransporterNet [29] baselines on their less challenging scenarios (Sec. 2), we col- lect two demos of each test task in their ravens framework and train TransporterNet on these de- mos for 40k iterations; meanwhile, for the pretrained case we fine-tune their vertical block triangle model [29] on the novel task for Ik iterations (thus matching the overall training cost of our pre- trained method). We will examine overall performance on the test metrics, specifpcc test metrics of particular interest, and then our efficiency metrics.
Simulation Task
Average Test Metries
Costs
Efficiency
Trials
Artion Fificirny
Prog.
Recov.
Thin Stepe
Amnotatal Actions
Compute
Sample
SSR CCD Eq. 8 (ours)
36%
41
79 19
19%
0
12
36%
3.0%
SSR L2CD By.
13%
30%
1%%
12
13%
1.1%
TransporierNet [29]*
30%
35%
501.3%
8%
40k
12
0.00075%
2.5%
SSR CCD Eq. 8 Finetuned (ours)*
51%
35%
86%
30%
1k
12
0.051 %
4.3%
TransporterNet [29] Finctuned"
41%
36%
57%
12%
Ik
12
0.041 %
3.4%
"Good Robot" [4] Finetuned
34%
43%
80+11 1%
16%
1k
12
0.034%
2.83%
"Good Robot." [4]
91%%
57%
96:
90%*
120k
40k
0.00076%
0.0023%
Simulation task performance on the metrics detailed in Sec. 5.1, averaged over all four cross-var pross-validttio.. "Average Test Metrics" averages Table 2 values. Bold indicates the best performing model. Higher is better for all metrics except costs. The progress range, e.g. in 50+33% the 3 is standard error. * Starred methods address the simpler problems de- scribed in Sec. 2, so comparisons should carefully consider this context. TransporterNets [29] trains on robot demos for each novel task with no task-to-task transfer. SPOT-Q [4], the SSR pretrain- ing step, tests on the train task, provides cost and efficiency baseline plus a test metries ceiling; finetuned SPOT-Q is Ik steps of tuning from a random task to the novel task on two demonstrations.
Table 1:
folds of Srave-one-model--ee
Simolation Task
Stack
Unstack Prog. Recov. Trials
Row
Vertical Square
Trials
Dif
Prug.
Recov.
Trials
Prog.
Recov.
Trials
Prog. Revov.
SSR CCD Eq. 8 (ours)
24%
22%
75%
16%
666
77%
82%
30%
38%
81%
28%
24%
2855
79%
14%
SSR L2CD Eg.
51%
0%
38%
60%
766
73%
7%
34%
67%
TransporteNNet [29]
27%
51%
12%
24%
32%
SSR CCD Eq. 8 Finetuned (ours)*
58%
29%
90%
42%
72%
64%
87%
40%
19%
84%
24%
34%
28%
83%
26%
TransporterNel [29] Finetunzd*
541%
34%
74%
12%
56%
34%
64%
24%
26%
44%
16%
28%
25%
48%
8%
"Good Robot" [4] Finctuned*
64%
33%
80%
38%
59%
86%
82%
20%
24%
90%
11%
0%
28%
68%
0%
Table 2:
cross- validdaiin of our SPOT-PoTTRR (SSR) framework for 50 sim- ulation trials. Bold indicates key best metrics (Sec. 5.1). Unstacking has no notion of recovery.
Our results the overall Test Metries in Table 1 show SSR with CCD achieving 36% trial com- pletion, 79% average progress, 41% action efficiency, and 29% recovery rate; L2 Correspondence L2CD gets 13%, 67%, 30%, and 5%, respectively, which demonstrates the benefit of evaluating demo embeddings Ha with Cycle Consistency Correspondence CCD (Alg. 2); and TransporterNet gets 30%, 50%, 35%, and 8%, respectively, which highlights our method's overall improvement over prior fhw-shot imitation work.
Our task-specific simulation results in Table 2 show SSR with CCD performs better than L2CD in all cases, and better than TransporterNets in all cases except unstacking. The SSR worst case trial success rate of 24% is 12x better than the TransporterNets worst case of 2%. SSR completes 66% of unstacking trials vs 86% for TransporterNets, and SSR completes 30% of rows vs 2% for TransporterNets. This highlights shortcoming of TransporterNets' feature template displacement matching approach which is not present in SSR. A more detailed per-task simulation breakdown is in Appendix C.3.
Lastly, the finetuning experiments show that SSR with CCD effective not only in a zero-training setting, but also as an extension to few-shot model finetuning. Compared to the baseline finetuned "Good Robot", our SSR CCD Finetuned achieves 86% vs 80% average progress, 51% vs 35% trial completion, 35% vs 43% action efficiency, and 30% vs 16% recoveries. This highlights the mipact of the SSR Test Time policy (Sec. B.3), as SSR with Finetuning and "Good Robot" finetuning use the same base models and finetuning method. Moreover, SSR with Finetuning also outperforms
7

---

TransporterNetss with Pre- Treininnnn achieving 86% vs 57% average progress, 51% vs 41 41% trial com- pletion, 35% vs 36% action efficiency, and 30% vs 12% recoveries.
Efficiency Metrics: With CCD, SSR achieves 36 % compute efficiency, which is roughly four orders of magnitude better than TransporterNet and "Good Robot!", which both have 8e-4% com- pute efficiency. This is triple the 13% compute efficiency of L2CD, fürther underlining the effective- ness of our CCD metric. SSR also achieves three orders of magnitude better sample efficiency with CCD than 'Good Robot!". at 3% vs. 2e-3%, while maintaining a much smaller, but still signifi- cant, lead over TransporterNet 2.5% sample efficiency, and L2CD at 1.1% % sample efficiency. The smaller gap is because, as an imitation method, TransporterNets is able to replicate behaviors with very just few annotated actions. Even so, SSR maintains its lead over TransporterNets due to its superior trial completion rate. Next we will examine our performance on the real robot experiments.
5.3 Real Robot Experiments
Real Task
Trials
Action Efficiency
Prog.
Recov,
Stack
30%
25%
80 5%
22%
We transfer SSR to a real robot using our mod- els pretrained in simulation from Section 5.2 with results in Tab. 3. All other aspects of the method remain the same except task progress is recorded by a different observer, since in simu- lation read internal simulator states. Trans- porterNets is not designed for sim to real trans- fer, and is thus not included here.
Unstack
90% %
86%
97
Row
30 %
28%
77
66%
Square
10%
27%
7514%%
10%
Average
40% %
42%
82 +-3%
33%
Table 3: Real See-SPOT-Ruu framework with Cycle Consistency, SSR CCD Eq. 8, performance on Sim- To-Rall transfer to novel tasks from sim- ulated demos during leave-one-model-oww cross- validation. Bold entries are for ease of reading and progress range (+) is standard error.
In real experiments SSR has an average of 40% of trials complete, 42% action efficiency, 84% progress, 35% rate of recoveries, 40% compute; which is very similar to results in simula- tion at 36%, 41%, 79%, 29%, and 36% respectively. For reference, "Good Robot" completed 100% of trials after 20k RL training actions and 60k training steps for both stacks and rows, resulting in 0.0008% compute and 0.003% sample efficiency; but it scored on the train task, and is neither de- signed for capable of completing new test tasks its own. We examine per-task real SSR test metrics in Appendix C.4.
6 Conclusion
See-SPOT-Run' three to four order of magnitude sample efficiency and compute efficiency im- provements show can immediately and effectively begin making progress on novel task in a novel domain with scarce demonstration examples. By contrast, RL models such "Good Robot!" typically start out with highly randomized exploration and no ability to complete a novel task, re- quiring inefficient sampling and compute-intensive training to first match and then beat SSR given task. Furthermore, although SSR performs 12x better than TransporterNets [29] in the worst case sim trials, TransporterNets may scale better than SSR some metrics demonstrations added. Even so, it would ultimately be more efficient in the long run to develop methods where demonstrations
multi-step task also improve performance and efficiency on the next task.
We expect that our proposed methods will be generally applicable to enhancing other methods across broad contexts such as Reinforcement Learning [4, 6). Behavior Cloning, Imitation Learning, and Meta-Learning; as well as in specific contexts such as TransporterNets. Future work should investi- gate such applications. Further improvements to methods might include better correspondence metries, automatically assessing task progress, mitigating the need for dense reward [23, 26], and an extension to continuous domains.
In summary, we have introduced the concept of Reinforcement Learning Before Demonstration, implemented via the See-SPOT-Run framework. Our framework achieves simultaneous few-shot imitation and sim-to-real transfer for multi-step tasks, such as block stacking and row-making, that are prone to progress reversal. See-SPOT-Run improves compute and sample efficiency by three to four orders of magnitude vs. prior work by leveraging pretrained RI. models to solve the imitation problem novel tasks.
8

---

References
[1] C. Fugazza and Ádám Miklósi. Should old dog trainers learn new tricks? the efficiency of the do as do method and shaping/clicker training method to train dogs. Applied Animal Behaviour Science, 153:53-61, 2014. ISSN 1168-11911 doi.httpss//doi..rra
wpppppppp.iimm..mmmmmmmm 2014.01. 440004444
009. URL
[2] C. Fugazza, Petro, Ádám Miklósi, and Ákos Pogány. Social learning of goal-directed actions in dogs (canis familiaris): Imitation or emulation? Journal of Comparative Psychology, 133 (2):244 251, 2019.
[3] V. Horner and A. Whiten, Causal knowledge and imitation/emulationn switching in chim- panzees (pan troglodytes) and children (homo sapiens). Animal Cognition, 8(3):164-111, 2005.
[4] A. Hundt, B. Killeen, N. Greene, H. Wu, H. Kwon, C. Paxton, and G. D. Hager. "Good Robot!": Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer. IEEE Robotics and Automation Letters (RA-L), 5(4))6724-6731, 2020. URL
https//axii..rr/ab/119...111
[5] O. Kroemer, S. Niekum, and G. D. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. Journal of Machine Learning Research, 22(30): 1-82, 2021.
[6] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In IROS, pages 4238-4245 IEEE, 2018.
[7] S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373 2016.
[8] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly,
M. Kalakrishnan, V. Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, 2018.
[9] J. Mahler and K. Goldberg, Learning deep policies for robot bin picking by simulating robust grasping sequences. Conference on Robot Learning, pages 515-524, 2017.
[10] Mermandez-Garcia and R. Sutton. Understanding multi-step deep reinforcement learn- ing: A systematic study of the dqn target. arXiv preprint arXiv: 2019.
[11] M. Q. Mohammed, K. L. Chung, and C. S. Chyi. Review of deep reinforcement learning- based object grasping: Techniques, open challenges, and recommendations. IEEE Access, 8: 178450-178481, 2020.
[12] C. R. Garrett, C. Paxton, T. Lozano-Perez, L. P. Kaelbling, and D. Fox. Online replanning in belief space for partially observable task and motion problems. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5678-5684, 2020.
[13] A. K. Tanwani. Domain invariant representation learning for sim-to-real transfer. In Confer- ence on Robot Learning (CoRL), 2020.
[14] C. Finn, T. Yu, T. Zhang. P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. 2017.
[15] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML'I7 Proceedings the 34th International Conference Machine Learning Volume 70, pages 1126 1135, 2017.
[16] S. James, M. Bloesch, and A. Davison. Task-embedded control networks for few-shot imi- tation learning. Conference on Robot Learning, pages 783-795, 2018.
[17] R. Rahmatizadeh, P. Abolghasemi, L. Boloni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3758-3765 2018.

---

[18] A. Zhou, E. Jang, D. Kappler. A. Herzog, M. Khansari, P. Wohlhart, Y. Bai, M. Kalakrishnan, Levine, and C. Finn. Watch, try, learn: Meta-learning from demonstrations and rewards. In ICLR 2020 : Eighth International Conference on Learning Representations, 2020.
[19] T. Z. Zhao, A. Nagabandi, K. Rakelly, C. Finn, and S. Levine. Meld: Meta-reinforcement learning from images via latent state models. arXiv preprini arXiv:2010. 13355, 2020.
[20] D. Kalashnkov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and
K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv,
2021.
[21] Y. Liu, A. Gupta, P. Abbeel, and Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1118-1125, 2018.
[22] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering, arXiv preprint arXiv: 07854, 2019.
[23] W. Goo and S. Niekum. One-shot learning of multi-step tasks from observation via activity localization in auxiliary video. In 2019 International Conference on Robotics and Automation (ICRA), pages 7755-7761, 2019.
[24] Z. Zhu, K. Lin, and J. Zhou. Transfer learning in deep reinforcement learning: A survey, arXiv preprint 20Xi::2009,,7888 2020.
[25] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251, 2017.
[26] Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas. Playing hard exploration games by watching youtube. In Advances in NeurlPS, pages 2935-2945, 2018.
[27] K. Rao, C. Harris, A. Irpan, S. Levine, Ibarz, and M. Khansari. Rl-cyelegan: Reinforcement learning aware simulation-to-ret relll.... In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11 11157-11166. 2020.
[28] Q. Zhang, T. Xiao, A. A. Efros, L. Pinto, and X. Wang. Learning cross-domain correspon- dence for control with dynamics cycle-consistency. In International Conference on Learning Representations, 2021. URL http:////ccmeeeeeeeeee//ooooonss
[29] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong. V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
[30] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Diseriminator-actor: critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2018.
[31] Z. Zhu, K. Lin, B. Dai, and J. Zhou. Off-policy imitation learning from observations. In Advances in Neural Information Processing Systems, volume 33, pages 12402- 12413, 2020.
[32] S. Reddy, A. D. Dragan, and S. Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. In ICLR 2020 Eighth International Conference on Learning Represen- tations, 2020.
[33] I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off-policy distribution match- ing. In ICLR 2020: Eighth International Conference on Learning Representations, 2020.
[34] R. Jena, C. Liu, and K. Sycara. Augmenting gail with be for sample efficient imitation learning. In Conference on Robot Learning (CoRL), 2021.
[35] A. Z. Ren, S. Veer, and A. Majumdar. Generalization guarantees for imitation learning. Con- ference on Robot Learning (CoRL), 2020.
10

---

[36] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and F. Sun. Survey of imitation learning for robotic manipulation. In International Journal of Intelligent Robotics and Applications, volume 3, pages 362-369, 2019.
[37] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys, 50(2):21, 2017.
[38] F. Torabi, G. Warnell, and Stone. Recent advances in imitation learning from observation. In Proceedings of the Eiwhtt-Eighth International Joint Conference Artificial Intelligence, pages 6325-6331, 2019.
[39] E. Jung and I. Kim. Hybrid imitation learning framework for robotic manipulation tasks. Sensors, 21(10):3409, 2021.
[40] and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Infor- 2016.
mation Processing Systems, volume 29, pages
[41] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese. Neural task programming: Learning to generalize across hicrarchical tasks. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3795-3802, 2018. 1o1:1001109/CCRA.000 201....6699...
[42] D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and C. Niebles. Neural task graphs: Generalizing to unseen tasks from single video demonstration. In 2019 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 8557-
8566, 2019. doi: 10. 101.1109.V........000
[43] M. L. Gray and S. Suri. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. 2019.
[44] K. Hara, A. Adams, K. Milland, S. Savage, C. Callison-Burch, and P. Bigham. A data- driven analysis of workers' earnings on amazon mechanical turk. In Proceedings of the 2018 CHIl Conference on Human Factors in Computing Systems, page 449, 2018.


Reviews:
[{"review_id": "grK2ac8c0XC", "reviewer": "Reviewer_tTpU", "summary": "\nThe paper proposes a novel algorithm for imitation learning using multi-task reinforcement learning pretraining. First, a set of policies is learned on the training tasks, together with Action-State Representations. The Action-State Representations are produced from a state using a neural network encoder, one for each action. At the imitation learning stage, the algorithm is always given the demonstration state-action corresponding to the current progress on the task. The algorithm finds the best action for the current state by maximizing the match between the Action-State Representation of the demo state-action pair and the current state-action pair. The match is evaluated using a cycle consistency loss. The performance of this method is slightly better than direct imitation learning.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n- The paper proposes a new system that is claimed to perform imitation learning, and empirically performs comparably to an imitation learning baseline (although using different assumptions).\n- The paper proposes a new concept of state-action representations, which might be useful for future work.\n\nWeaknesses:\n1. The abstract contains numerous misleading statements. \n1.1. For instance, it is claimed that SSR completes 20% more trials than the baseline. While technically a correct statement, this refers to a 6% improvement from 30% to 36%, which is the normal way of quantifying improvement and is perhaps less impressive. \n1.2. Further, it is claimed that the method produces up to 3 orders of magnitude sample efficiency improvement. This is again correct: one of the baselines has the exact same data efficiency as the proposed method, but another baseline is 1000x worse. Another correct statement is that the proposed method produces between zero and 3 orders of magnitude sample efficiency improvement.\n2. The proposed method only slightly outperforms the direct imitation learning baseline (Transporter). Given that the proposed method is significantly more complex, it is hard to justify this complexity. This baseline is also not included in the real world experiments. Since there is nothing special the proposed method does for real world experiments, the baseline should be also run the same exact way.\n3. It is assumed that the policy observes its progress \u0005(i.e. how close to task completion it currently is) at test time. This is non-standard, and further a rather unrealistic assumption. Training the task completion estimator is technically as hard as training the policy to solve the task. Since the baseline methods don’t have access to task progress, the comparison is unfair.\n4. The proposed imitation algorithm is simply presented mechanistically and it is entirely unclear what the design principles are. \n5. The algorithm is specific to discrete actions\n6. The system assumes the demonstrations are optimal (l151), which is often unrealistic.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nThe proposed method only slightly outperforms the direct imitation learning baseline (Transporter). Given that it is significantly more complex, it is hard to justify this complexity. The paper also suffers from many other issues with methodology, experiments, and writing, as outlined in my review. Overall, I am not convinced the proposed system represents a significant contribution to SOTA and leaning towards reject.\n\n--- Update --- \n\nIn light of the otherwise positive updates to the paper (improved writing, improved performance, and the additional finetuned Transporter and SPOT-Q baselines), I switch my vote to weak accept. While the paper still suffers from certain writing issues, the experimental evaluation introduces extra assumptions, and the proposed method is rather cryptic, the updated results present a reasonable contribution for CORL. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- How was the number of training steps for the Transporter tuned? What is the batch size? It appears that Transporter was trained on 12 data points for 40K iterations. Assuming a batch size of 12, that is 40K epochs. Is this training for this long really necessary?\n- A lower bound baseline is missing. For instance, a very naive baseline would be executing random actions, or random actions taken from the demonstration trajectory. Another naive baseline would be repeatedly executing the demonstration trajectory until episode ends."}}, {"review_id": "dd1cZwi2j-v", "reviewer": "Reviewer_FEvJ", "summary": "The paper proposes a few-shot learning methods for rapidly adapting to a new task using an ensemble of policies (and demonstrations) from old tasks. The essential idea is to consider already-acquired policies (Q-functions) to itself be themselves embedding functions which generate a score function E(s) -> H(a). To perform on a novel task given a couple demonstrations, we can embed the demonstration states using the policies for already-acquired tasks. We embed the test-time state in the same manner, and then choose new actions to most closely match (in the latent space) successful actions in from demonstration (which are presumed to be optimal). This search is exhaustive over `(policy, demonstration)` pairs.\n\nIn the variation presented here, the Q-functions are specifically learned to generate an a pixel-wise action-value function, to make computing the distance metric easier and more appropriate for the state space.\n\nIf policies (Q-functions) encode task semantics which are invariant to the irrelevant state of the environment, this should allow us to make forward progress on new tasks, by always choosing the \"best\" action which we already know works well on other tasks in a new task.\n\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\n* Novel approach for sim2real/few-shot learning which bravely eschews end-to-end approaches\n* Addresses difficult problems often avoided by this community, such as task-task adaptation and long-horizon tasks\n* Compelling real and simulation experiments on challenging tasks\n* Formulation appears very sample efficient, and extensions are easy to imagine (not a dead-end method)\n\nWeaknesses\n---\n* Presentation issues make this difficult for readers to decipher, e.g. over-reliance on concepts for related works, prodigious use of bespoke acronyms and backronyms for many concepts (ASE, RBLD, SSR, CCD, PDP), complex diagrams demonstrations multiple concepts, tons of notation\n* The paper struggles to help readers develop conceptual intuition for the methods before diving into immense amounts of detail", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an approach to the few-shot learning problem which diverges quite markedly from the current vogue, which centers on end-to-end and high-data techniques. End-to-end research tends to ignore long-horizon tasks and task-task adaptation, because it's very difficult to exploit the strengths of end-to-end learning to solve those (I have tried!). \n\nAfter some laborious parsing, I believe the technique is technically sound and a novel, clever, and scalable approach to these neglected problems. I would love to see more research like this in the future, and I think original approaches like this one have a very high potential for impact.\n\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "So why the Weak Accept? Because I believe this paper has major presentation issues which will severely abridge its potential impact. If the authors revise the writing the paper for far greater intuition, easier reading, and conceptional clarity, this paper is clearly a strong accept.\n\nFirstly, it relies heavily on a prior work for conceptual clarity -- it would be difficult to understand this paper without reading [1,2].\n\nSecond, within the body of this paper itself, the exposition of the motivation is compelling, but the authors fail to develop much intuition for their design choices, either referring to previous papers, or simply stating the design choices without motivating them with intuition. This makes it difficult for most readers to understand a paper with algorithmic designs which are much more complicated than a simple training loop.\n\nThird, the exposition of the method itself is dripping with novel acronyms and custom-named concepts, only a fraction of which seems necessary for understanding the core ideas of the paper. These make it extremely difficult to move through the paper as a ready looking to answer her own questions, because each answer requires referring back in the text several times to understand the meaning and context of each named concept.\n\nFourth and more minorly, the diagrams can be quite complex. I don't think this would be an issue if the text contained more clear exposition of the method. "}}, {"review_id": "A-Xq8AZJ8hj", "reviewer": "Reviewer_YJsz", "summary": "This paper proposes the Reinforcement Learning Before Demonstration (RLBD) paradigm for transfering agent policies to novel tasks given a few demonstrations but without task-specific training. The key idea is that 1) learning some embeding functions E by RL training; 2) compute the embeddings of demonstrations H_d = E(d) and testing state  H_l = E(s); 3) find the most ''relevant'' H_d for H_l and take the action corresponding to H_d under the current testing state. The authors propose two methods (i.e., L2 Consistency Distance and Cycle Consistency Distance) to measure the relevance of H_d and H_l. The evaluation results are much better than baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) a reasonable transfer framework to handle novel tasks.\n2) good evaluation results.\n\n\nWeaknesses:\n1) poor writing. The method can only be understood with very careful reading, not because the method itself is complex, but due to the writing is poor. For example,  there are many short notes like 'SSR, Fig.2, Alg. 1, PDP, CTA ...'. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea is good and the results are encouraging, especially for the zero-shot sim-to-real domain transfer and real robot testing. Thus, I prefer to accept this paper.\n\nHowever, my major concerns are that 1) the writting is poor and 2) the experiments are relatively simple. \n\nOverall, I give a weak accept for this paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "1) What if the best action in demonstration cannot cover the testing scenarios? could your methods deal with this?\n\n\n2) How to train the embedding function E? Because there are totally M number of embedding functions, how to make them diverse enough? Or diversity is necessary? Training a large number of  embedding functions is also computational extensive."}}, {"review_id": "2YrrH-THf9m", "reviewer": "Reviewer_Hszk", "summary": "This work proposes the SSR (See-SPOT-Run) algorithm. Given a set of k pre-trained policies, SSR can generalize to a new robotic manipulation task given a small number of demonstrations and no fine-tuning. SSR does this by measuring the cycle-consistency distance between the demonstrations and new task in multiple learned embedding spaces. SSR is evaluated on block arrangement and stacking tasks in simulation and on a physical robot.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* Tackles an important problem in robot learning - how to generalize to new tasks with minimal new data and training requirements.\n* SSR is interesting approach that is an original application of cycle consistency techniques. To my knowledge this is the first time it has been applied to task generalization in robotics, although I am not very familiar with the cycle consistency literature.\n* Strong results demonstrating zero shot transfer to new tasks\n     1. These are demonstrated both in simulation and on a real robot system, with comparable performance between simulation and the real world.\n     2. The transfer from sim-to-real is also impressive.\n* Related work is thorough.\n* The cycle consistency approach is potentially quite general, although I have some questions given the current presentation (discussed below).\n\n**Weaknesses**\n\nGenerality\n* SSR is applied only to the SPOT framework which is a particular reward shaping methodology aimed at incorporating progress reversal into policy rewards. Consequently I question generality of the method.\n* It would be great to see SSR applied to policies that are pre-trained differently (for example using TransporterNets or a Q-Learning method). Demonstrating that SSR yields strong performance across multiple methods for training the task-specific policies would be valuable.\n* Similarly, it would be great to see how SSR performs for tasks not characterized by policy reversal.\n    1. The method as proposed does not appear restricted to policy reversal tasks. The paper would be stronger if evidence was provided for the effectiveness of SSR in a robotics domain with different characteristics. For example, navigation which typically involves long episodic time horizons and continuous control. This would offer a good contrast short time horizon (~4 - 6 steps (optimally)) discrete action space problems considered.\n* Reliance on a predefined task progress metric to identify the relevant demonstration action to use also limits the generality of SSR\n    1. A discussion of to what extent this is a limitation and how this might be overcome would be a valuable addition to the paper.\n\nLimited assessment of SSR compared with TransporterNets\n* Given that TransporterNets are the only other method that SSR is compared with, it would have been nice to see a more extensive comparison of the two methods. Especially since the average success rates of the two methods are quite similar, at 30% and 36%.\n* In particular, an ablation study which explores the effect of the number of demonstrations / annotated actions on both TransporterNet and SSR’s performance would be very valuable.\n    1. As-is with the results presented using just 2 demonstrations, I question whether the lower performance of TransporterNets is simply due to the very low data regime.\n    2. The paper would be stronger if SSR demonstrated consistently better performance across a range of data regimes, for example from 1 - 100 or 1000 annotated actions. The authors do mention this as future work, but I think it is important to include in this paper, especially since a large enough dataset appears to be already available.\n\nThe lack of pre-training for TransporterNets limits the fairness of the comparison.\n* The SSR policies shown in Table 1 have seen 40k actions (albeit from different tasks), whereas the TransporterNets policies have only seen 12. Given the importance of data for training any neural network, it is possible that pretraining the TransporterNets could significantly improve performance. Please consider adding the following results or discussing why they are not relevant.\n    1. TransporterNets, zero -shot: A baseline in which TransporterNets were pre-trained in the same way as SSR with 40k actions for 120k steps.\n    2. TransporterNets, fine-tuned: A baseline in which TransporterNets were pre-trained in the same way as SSR with 40k actions for 120k steps, then fine-tuned on the 12 task-specific new actions. \n\nRobustness\n* No discussion is given on policy variance for a particular task or across training runs.\n    1. At a minimum stating the number of trials per task that each policy was evaluated on and including the standard deviation of performance over these trials is important to include.\n\nMinor:\n* It would be nice to see how a Q-Learning method (e.g. Rainbow) performs on this task without the SPOT reward shaping. Similarly for an on-policy method. However I recognize time and compute constraints so do not view this as critical.\n* The method is a little hard to understand, especially in sections 4.2 and 4.3\n    1. Section 4.2: The notation is confusing, especially the re-framing of the Q function given that the Q function has a formal definition. Additionally the use of V is confusing given the Value function in RL: V^pi(s_t).\n    2. Section 4.2:  E_m (s_t) → H_m. Isn’t the embedding also a function of the action?\n    3. Section 4.3: Line 173: What does (4) refer to? Link between all the phases listed is lines 159 - 161 and the text in the section is hard to follow. \n* Inaccuracies in Q function description\n     1. Line 108: The Q function does not maximize R (reward) over time. It is defined as the expected cumulative discounted future rewards from state s_t given action a_t under policy pi\n    2. General: The Q function is defined with respect to a particular policy, i.e. Q^pi\n* Typos:\n    1. Line 107: “seeks” → “learns”\n    2. Line 267: “Transporternets” → “TransporterNets”\n* Line 217: Which prior work?\n* Line 233: Please reference that the Efficiency Metric was defined in [4]\n* Line 281: Please give a little more detail as to which robot you are using.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Thank you for your submission. This work proposes a promising approach to the important problem in robotic learning of task generalization. The application of cycle-consistency to zero-shot task generalization is especially interesting. The zero-shot results are promising in simulation and in the real world, as is the sim-to-real transfer. The paper would be even stronger if the issues regarding method generality and the comparison with TransporterNets were addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Please address the points raised above on generality, the comparison with TransporterNets and robustness, as well as the minor issues.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for learning from demonstrations that can transfer to new tasks, and includes experiments on challenging tasks, including on a real robot. Further, the related work section is thorough and the proposed ideas are interesting. These strengths are significant; however, the paper also has significant weaknesses:
* The proposed method assumes access to a task completion metric. This assumption is both non-standard and unrealistic, and is not justified in the paper. It is essentially akin to having access to a shaped reward function at test time. This assumption should be convincingly justified.
* The reviewers brought up multiple concerns about the fairness of the comparisons:
   * The TransporterNet comparison does not use the substantial pre-training data that is made available to the proposed method
   * None of the comparisons have access to the task completion metric, making it difficult to assess how much this assumption is helping the proposed method vs. other aspects of the proposed method
* Generally, the proposed method is fairly complex, and the improvement over TransporterNets is relatively small (30% success vs 36% success), making it unclear whether the complexity is worth it and whether future works will build upon this work in light of the complexity. It is strange that there is not a comparison to TransporterNets in the real world experiments.
* The clarity of the writing can be improved significantly to make it easier to follow and provide better intuition for the paper's design choices.

In the balance, while the paper has significant promise, the weaknesses of the paper currently outweigh the strengths.

-----------------------------------

-----------------------------------

After the response period, multiple of the concerns above have been addressed. There are still some important weaknesses, including (a) certain issues with writing / the description of the method and (b) being more upfront about assumptions like the task progress metric which does not come up until page 4 (or ideally experiments that learn this metric!). Further, I think the below statement in the abstract is not accurate:
> To our knowledge, we are the first to address multi-step tasks from demonstration on a real robot without task-specific training, where both the visual input and action space output are high dimensional

I believe that papers [1,2] also performed multi-step tasks from demos on a real robot without task specific training with visual inputs and high-dim actions, so this statement should be revised and these papers should potentially be referenced. I also believe that neither of these papers assume access to a task progress metric.

However, the strengths generally outweigh the weaknesses (despite these prior papers, this is still a very difficult problem setting!) and I recommend accept. Nonetheless, I strongly encourage the authors to make the changes listed above for the final version of the paper.


[1] Yu et al. IROS 2019 (https://arxiv.org/abs/1810.11043)

[2] Mandlekar et al. RSS 2020 (https://arxiv.org/abs/2003.06085)

Decision: Accept (Poster)