Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes an imitation learning framework that extracts temporally-extended sensorimotor skills from prior data and subsequently
learns a policy for the target task with respect to these learned skills.
The contributions of this manuscript are the temporal consistency term in the VAE loss function, a
representational learning objective to enable more predictable and consistent skill representations,and the data retrieval based on the ensuing encoder,
to increase the scope of supervision for the policy.
There are both simulated and real robot results.
strengths_and_weaknesses: STRENGTHS

The contributions of this manuscript are the temporal consistency term
in the VAE, and the data retrieval based on the ensuing encoder.

Experiments involve simulation and hardware, and compare the method to
other baselines.
Good set of ablation studies, which ties in nicely with the baseline
 comparison (like against BC-RNN without D_prior, as the authors'
 method without D_prior still performs better)

Data retrieval based on skill matching is interesting

ISSUES and CONCERNS

The real robot tasks could be much more compelling.
The tasks seem limited to pick and place. It would be easy to
use traditional methods to achieve the same results, so why bother with
the complexity and risk of learning approaches?
From a perceptual point of view, for the real robot there is a set of
fixed objects - no variation in size, shape, or appearance. It would have
been better to do the experiments with real vegetables and fruit, which
have natural variation.
There is generalization in terms of adjusting to object and goal locations.
There is no evidence for generalization in terms of objects.
77% performance on these tasks is not impressive.
It is not surprising that the ABC prior data vs.
D task ablation works well (around
line 433) as all the tasks are similar.
What about using the utensils to actually manipulate something else,
pouring the milk,
actually using a real stove (turning knobs and pressing buttons),
spreading butter on toast or something else for the breakfast task,
cutting the bread, sausage, or fish, or 
peeling the banana?

This paper presents a simplistic view of what a skill is.
1) Only position, yaw, and gripper state are controlled in the real robot
experiments. This is a reflection of the limited nature of the tasks
explored in this paper.
2) Skills are limited to trajectories (open loop sequence of actions) only, and trajectory-only skills
didn't get robotics very far over the past decades. 
That is why robotics  adopted:
 - force control, impedance control, and other control laws
   (including nonlinear control laws)
 - timing and transition rules for finite state machines
 - error handling
In situations with rigid objects and rigid environments, something must be
compliant or infinite (large) forces result. That means either some
mechanical compliance in the robot or some form of force control.
I don't see how the proposed system would be extended to handle these
aspects of skills.

For many tasks a solution has to be a policy u=pi(x). An open loop sequence of
tasks cannot solve unstable tasks, for example.
This is a mistake made in good old fashioned AI.
It is true that demonstrations or any instance of a task execution
are sequences of actions, but this does not imply that a controller
that just emits previously successful sequences of actions will succeed.
Consider riding a bicycle. A bicycle rider can't just play back the
steering and balance motions used in a previous ride, they will quickly
fall over.

I had to go to the supplementary document to figure out what the
actions were. It turned out they were different in the 3 cases:
452: agent operates the robot via joint torque control resulting in a
9-dimensional action space
(joint torque control is probably a bad choice)
474: agent operates the robot via inverse kinematics control resulting
in a 7-dimensional action space.
507: OSC, position and yaw of the end effector, gripper state.

It is not clear what the gripper state is (open/close, or finger
positions).
How is gripper closing controlled? Do the gripper fingers
go to a fixed position, or is a closing force specified?
If positions are controlled, what happens if a position is too small
for the object being gripped? The gripper outputs a maximum force?
All of this should be clearly explained in the paper itself.

It is not clear how goals are specified to the system? Desired final
positions for objects? An optimization criterion? Some measure of
similarity to a demonstration?

The network architectures chosen were not justified or compared to
alternative architectures.

The paper does not make clear which trajectory segments (sub-trajectories)
are used from D_target or D_prior. Are all possible trajectory segments
used, in which case adjacent segments of length H share H-1 values
(which is implied by the use of the term "all" in line 160 and 162)?
Or are trajectory segments distinct, with no overlap between adjacent
segments?
Or are trajectory segments initiated or terminated at significant events
like making or breaking contacts?
Prior work (6, 8) is described as "parsing" the demonstrations (line 122),
but how does the parse/segmentation work in this paper?

In line 163 similar sub-trajectories are retrieved. If sub-trajectories
overlap, this will choose many from the same demonstration. Is that good
or bad?

How is H picked? I would imagine it would have a huge possible range
(some skills are quite short, while others (like steering a car) last
a long time (as long as one is driving the car)).

It is not obvious to this reader that predicting temporal differences
is useful, but let's see what happens in the ablations section.
One could always decode the latent representations producing trajectory
segments and then explicitly correlate the trajectory segments.
How is that a constraint on the latent representation, if the
encoder/decoder pair already work well?

Is F, the number of observations in the argument to the policy, a constant?
How it chosen? Wouldn't it make more sense to map observations into a
observation-latent-space (this is what state estimators like Kalman
filters do), and use the latent variable (the "estimated state") as
the first component of the training points as well as the index into
the policy? Perhaps the output of the perception network is such an
observation latent space.
[I see this is addressed starting at line 547 in the supplementary materials.
Perhaps it should be discussed in the paper.]

How is it decided where to start the readout of a trajectory? Always
at the start?

How is it decided when a new/different skill should be executed?
When the readout of the trajectory is finished (which is implied by
the paper's reference to "H timesteps" on line 159)?
There is no preemption of the current trajectory if things do
not go as expected? This leads to robots doing stupid things like trying
to close an automatic door when the person going through is having trouble
with their luggage, and the door starts swatting them.

In the discussion of "retrieval-based", what is retrieved?
The actual trajectory segment? The latent representation?

The policy is trained using instances (input: o_fs, output: tau).
The retrieval-based policy learning focuses on similarities of tau.
Why not focus on similarities of o_fs? It would seem to this reader
that for a policy similar inputs should lead to similar outputs
(except at a policy discontinuity), but I would not expect that
similar outputs are caused by similar inputs (Many (most?) policies
are many to one mappings, in that the dimension of the state or
observation space is usually much larger than the dimension of the
action space, and by replacing actions with skills, the dimension
of the policy output space is reduced further.)

I don't understand the purpose of the binary input bit saying whether
the training point is from the prior or target demonstrations.

How are initial state and goal conditioning of trajectories done (how
are the trajectories morphed to start and stop in the right places)?
How are grasping trajectories modified to be appropriate for an object.

The real robot video is impressive. I wouldn't bother presenting the
simulation results, and just focus on the real results.  You give 9
lines to real results, and more than 10 times that to simulation
results. Do the comparisons and ablations on the real robot!  Much
more convincing!  Though simulation is helpful to speed up
development and evaluation, the reviewer believes that
it is ultimately the real robot experiment that demonstrates the value
of the authors' method. The reviewer would suggest the authors put
more emphasis on the real robot experiment and conduct quantitative
analyses similar to the ones completed in simulation on the real robot
setup. This change could make the readers better appreciate the value
of the authors' method and its performance.

The authors have used only "success rate" as the performance metric in
the real robot experiment.  However, for the authors' method to be
practical in the real world, the reviewer believes that performance
metrics are not limited to success rate---other, even more important,
performance metrics exist.  Since the task in the real robot
experiment is setting up a breakfast table and cooking a meal, one of
the relevant performance metrics could be the total time for
completing the entire meal preparation process.  Obviously, if such a
robot cooks a meal in a real household, the users would tend to prefer
the robot the finish the meal within a reasonable and predictable
amount of time. In this regard, the reviewer would suggest the authors
to put more emphasis on practical performance metrics for the task
chosen for the real robot experiment, instead of a single "success
rate".

How do we know whether the skills failed (the output action sequence
is not good enough)? It appears the simulators simulate physics as joint
torques are generated, but there is little description of them.
In lines 577-584 the words "we evaluate" are used, but I have no
idea what those words actually mean. What is a checkpoint? Who chose
them? How are they evaluated?

It’s not clear exactly how this paper fits into the broader context of
existing research in the field. A major shortcoming is the discussion
of prior approaches for skill learning - the authors seem to only be
referencing works that use variational auto-encoders.  While the
proposed method does demonstrate relatively good performance in the
evaluation domains, the discussion of the results does not provide a
lot of intuition about why other baselines failed to perform as
well. I think this paper would be much improved by more explicitly
describing how this work compares to prior works and the
baselines. Specifically, what mechanisms can we attribute to the
success of the method?

The distinction between a skill and a policy is confusing. The terms
skill and policy are used interchangeably in many contexts. I think by
skills the authors are referring to the latent embedding.

The authors claim “A major limitation of existing approaches is that
they only focus on using the prior data for skill learning but not
policy learning.” This needs to be better supported. Among works that
don’t make a distinction between “skill” learning (i.e. learning
latent embeddings) and policy learning, prior data has been used to
update the
policy. (http://proceedings.mlr.press/v139/wulfmeier21a/wulfmeier21a.pdf
is just one example)

Figure 2: I think the components of the system should be better
justified. Why make a distinction between skill learning and policy
learning? Why not just learn low level policies that are “skills” and
a high-level policy that selects the skills? What is the advantage of
continuous skill embeddings vs. having a discrete high-level policy
that learns to select skills?

Section 5.1: Quality of the prior and target datasets - Because
imitation learning is used and there is no explicit representation of
task goals or reward, the quality of learned policy is dependent on
the quality of the data in the prior and target datasets. It would be
helpful to have more detail about how each dataset affects the learned
policy. For instance, I would imagine that it is more important for
the target dataset to have high quality demonstrations, since the
samples from the prior dataset are chosen to be close to the
target. What would happen if the target had poor demonstrations and
but the prior had higher quality demonstrations and vice versa?

What is the relationship between the number of sub-trajectories
sampled from the prior dataset and policy performance? What would
happen if you thresholded the distance instead of selecting a fixed
number? If the prior was poor quality or not particularly relevant to
the target, it would probably be more beneficial to use threshold for
selecting sub-trajectories. Depending on the data, it may be better to
not take any samples from the prior.

Line 225: What is the form of the non-parametric policy used in the
FIST baseline? And what would the performance of this method be if
data retrieval was used with the non-parametric policy?

How could knowledge of task goal/reward be incorporated into this
framework? It seems like indexing the most similar trajectories would
not always be sufficient to filter out bad examples. There can be poor
demonstrations that are closer to the target than other samples, but
still would not result in reaching the goal state. Relabeling the
sub-trajectories with rewards for the target task is a common approach
used in multi-task reinforcement learning. Is there a benefit to
framing this as an imitation learning problem without using task
reward signals?

Every single task needs re-learning: there's no obvious way of
changing the order of parts of the task for example: if the agent has
learnt to do A-->B-->C, it has to re-learn to do C-->B-->A. (that
said, if I open a cabinet drawer, I can take a fork then a knife, or
take a knife then a fork without a problem. But if I learn to play a
music sheet, even just a few bars, I can't directly play it in
reverse. What is the fundamental difference? and if I learn to play it
in reverse, there's probably going to be catastrophic forgetting for
the normal play)

The temporal consistency for skill embedding seems only suitable for
restricted domains with highly repetitive and ordered action
sequences. Not really suitable for improvisation and
problem-solving. Discuss.

The distinction between a skill and a policy is confusing. The terms
skill and policy are used interchangeably in many contexts. I think by
skills the authors are referring to the latent embedding.

Comments on the experiments:
- for the baselines: any training on D_target alone is unfair as
  only the target task dataset is used. That said, It makes more sense
  after the ablation study when comparing to the "non-retrieval"
  study.
- What is meant by "struggles to find appropriate goal observations to reach"\
 ?
- Why does FIST perform better for no-microwave?
- Could the authors provide a comment on why BC-RNN (FT) performs so poorly o\
n hardware experiments?
- There is no real inherent generalizability, the training and
  execution domains are the same. The no-microwave in D_prior was
  interesting though, showing that the encoder still made enough sense
  of this new skill to supervise the policy learning. However, other
  methods also seem pretty robust against no-microwave.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I would like to see more challenging tasks done on the real robot.

Performance on the tasks in simulation (77%) was not good.

I would like to see a richer notion of what a skill is, ideally a policy mapping low level states to low level actions.

Since information on whether trajectory segments was not clear, it is difficult to assess the role of temporal consistency.

I would focus on real robot results.

POST REBUTTAL COMMENT

I am still in favor of accepting this paper.

issues: Do a wider range of tasks on the real robot, beyond pick and place.

Improve performance on in the simulation and on the real robot.

LEFTOVERS FROM MAIN REVIEW (MOVED HERE BECAUSE OF STUPID CHARACTER LIMIT)

Small comments on paper:
- sub-trajectories -> trajectory segments - throughout the paper
- 41: pace -> space
- 73: to a specific types of tasks -> to a specific type of tasks
- 93: complimentary -> complementary
- 94: Jargon alert: you are assuming your reader knows what VAE means.
    - spell out (as in line 134) and provide a reference to make your
      paper more accessible.
- 136, 153: Jargon! LSTM has not been explained, spelled out, or had a
  reference provided.
- Equation 2: you might explicitly says that mu() is E() or mean(),
or just use the standard notation E()
- 171: train the policy the aggregate -> train the policy with the aggregate
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The work introduces a new method for skill-based imitation learning which can exploit the availability of prior data. It proposes a two-step method. In the first step a "latent skill spaced" is learned using a variational objective augmented with a temporal consistency term which enforces the predictability of the temporal distance of sub-trajectories. Second, a policy is learned by behavioral cloning on skill level using both data from the target task at hand as well as a subset of the prior data which is built based on the distances of embedded skills. 
The resulting method is evaluated on two simulated and one real-world experiment and ablations are performed on one of the settings. 
strengths_and_weaknesses: I think the paper proposes an interesting approach to an important problem in robotics. The main algorithmic contributions, adding the temporal consistency loss and distance-based retrieval of relevant prior data for policy learning are novel and, despite their rather heuristic nature, seem to be effective as shown by the ablation. 

In general. the experimental evaluation shows the method's capabilities in several challenging tasks and indicates the method's potential impact. While the presented experiments are sufficient, for an even better assessment of this impact further ablations on the sizes of prior, retrieval, and target data-set would be useful. 

While the main ideas are clear and the paper is generally easy to follow, I think there is some rigor missing in the methods section, where the objective is simply stated without context and several details have to be pieced together from the supplement (for more details see issue section)
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes an interesting, novel idea and sufficiently demonstrates the potential impact on a set of challenging experiments. While the key ideas are easy to follow and clear, the paper could be improved with more rigor in the method section 
issues: I believe the paper would heavily benefit from more rigor and details in the method section. To be clear, I don't think any of this is "wrong" or unreasonable but figuring that out and piecing it together should not be left to the reader but be more clearly stated:

- VAE loss objective (Equation 1): What exactly is lower bounded ($\sum_{t=0}^{H-1} log p(a_t|o_t)$, what are the independence assumptions/graphical models underlying the generative and inference model and why $-\mathcal{L}_{VAE}$ is a proper lower bound.
- From the supplement, it seems a modified version of the bound (MSE instead of gaussian ll and scaled KL term) is actually used (Algorithm 1), again a detail that should be mentioned. 
- Is the policy deterministic or stochastic and which loss is used for training? While that information is available in the appendix, I again think it should be stated here for clarity. In particular, the loss, which seems to be down-weighted for the "retrieval" data in the real-world experiment (i.e. there is a factor beta in line 15 of Algorithm 2 in the supplement, which according to table 5 is considerably lower than 1 for the real world data). This seems like an important detail and a method to address the shift between prior and task data. 

Questions / further remarks: 
- Why is the distance between the means used for ranking during retrieval and not something that considers the standard deviation/variance of the inference distribution? 
- Is there any further intuition/ analysis it is sufficient to enforce predictability of the temporal difference (especially as $m_\omega$ seems to be a rather large network for such a presumably simple task) to get the shown performance gains? Some analysis how the learned representations differ would be helpful 

Typos: 
- "on" missing in line 171 
- punctuation missing after equations.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of imitation learning. Specifically, this work assumes access to a prior dataset that contains task-agnostic play data or demonstrations solving different tasks and extracts temporally-extended skills from it. Then, a small sef of demonstrations for the target task are provided for learning the target tasks. To this end, the paper proposes a framework that is built upon existing skill-based imitation learning methods, while introducing two techniques: (1) when learning skills, additionally optimizes a temporal consistency term that encourages the learned skill space, and (2) when learning policy, leverages the prior demonstrations which are similar to the target demonstrations. The experiments on two robot manipulation domains (Franka Kitchen and CALVIN) show that the proposed framework outperforms baselines, including BC, IQL, and FIST. Ablation studies justify the effectiveness of the two proposed techniques.
strengths_and_weaknesses: ## Paper strengths and contributions
**Motivation and intuition**
The motivation for leveraging prior datasets to accelerate target task learning is convincing.

**Technical contribution**
- Encouraging the learned skill space to capture temporal difference seems effective and this paper presents a reasonable implementation to achieve it.
- Augmenting the target dataset using similar demonstrations from the prior dataset seems simple yet helpful.

**Clarity**
The overall writing is clear. The authors utilize figures well to illustrate the ideas.

**Ablation study**
Ablation studies are comprehensive. The proposed framework incorporates two techniques and the provided ablation studies are helpful for analyzing the effectiveness of each of them.

**Experimental results**
- The presentation of the experimental results is clear.
- The experimental results on two robot manipulation domains (Franka Kitchen and CALVIN) show that the proposed framework outperforms baselines, including BC, IQL, and FIST.

**Reproducibility**
Given the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible. 

## Paper weaknesses and questions

**Navigation and locomotion domains**
This work evaluates the proposed framework in two robot manipulation domains. It would be more convincing if the experiments were also conducted in navigation or locomotion domains. I believe the proposed temporal consistency term would yield an even larger improvement in navigation domains.

**Implicitly predicting time step difference**
This work proposes to predict the time step difference between two sub-trajectories for encouraging temporal consistency. I wonder if implicitly learning this (such as performing contrastive learning where the model select which sub-trajectories are closer in terms of temporal difference, see "Time-Contrastive Networks: Self-Supervised Learning from Video" by Sermanet et al.) would also yield improvement.

**Kitchen-No Microwave vs. Kitchen-All**
I wonder why FIST and IQL (UDS) perform better on Kitchen-No Microwave, which is supposed to be harder compared to Kitchen-All.

**Similarity between the prior dataset and the target dataset**
This work assumes that the prior dataset and the target dataset are similar enough so that learning from some of the prior demonstrations is helpful. I found this assumption a little too strong.

**Similarity between trajectories**
This work proposes to use $L_2$ in the learned skill space to measure the similarity between two trajectories. At first glance, it does not look like a good idea. Yet, the improved performance backs it up. I feel I would get more convinced if prior demonstrations with high and low similarity are qualitatively presented.

**$\mu$ in eq(2)**
I was a bit confused reading Eq. (2) since $\mu$ in Eq. (2) is not introduced when it is first mentioned there. It is only mentioned in L157.

====== After rebuttal ======

I thank the authors for the response and the revision, which addresses many of my concerns. I have carefully read other reviews and decided to keep my original rating (weak accept).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe this work studies an important and active research direction (i.e. skill-based learning in RL) and proposes two interesting reasonable techniques. The experimental results justify the effectiveness of the proposed techniques. Yet, I have a few concerns (stated above) such as the limited scope of the evaluation (only in robot manipulation), I am leaning toward accepting this paper.
issues: Described in the Strengths And Weaknesses section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presented Skill-Augmented Imitation Learning with prior Retrieval
(SAILOR), for improving task learning using skill learning from prior data and a small amount of target task data. The two key additions of this paper are: latent skill learning with an auxiliary task that encourages skills that are temporally similar to be close in latent skill space, and data retrieval that compares (in the latent space) the skills of the prior data to the target task dataset and limits the sampling of skills during downstream policy learning to those close to demonstrations on the target task. 

strengths_and_weaknesses: This paper is well presented, clear, and easy to follow. The contributions are minor practical enhancements of existing approaches, however, the tasks are well selected to show a significant improvement over existing methods. The comparisons seem relevant, and an ablation study shows the effect of each component of the method. Finally, the real-world experiments round off this paper nicely.

I think the main thing that is missing is a deeper evaluation of the two contributions. For example, it would be nice to show that regions in the latent space are indeed close in temporal space, and that behaviours retrieved are indeed close to behaviours from the target task dataset. The ablation study does enough to show these contributions are effective, though it would be a nice addition to see the effect of these methods practically, and also beneficial to the community that may adopt these approaches. This could be as simple as showing samples in each case and discussing the edge cases.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: As mentioned, a deeper analysis of the effectiveness of each of the contributions would enhance the paper.
issues: No figures are referenced in the text. It is not known when the figures back up what is written, which breaks up readability. 

The symbols are confusing in parts. 
- “p” is used as transition probably, initial state distribution, skill decoder, and learned prior.
- Figure 2 could be clearer. Is the “temporal consistency” block the same as “m_ω”? Is “phi” the skill encoder reported as “q_phi” in the paper? Is “psi” the same as “p_psi”? 
- It might be worth changing the font in figure 2 to match what is in the text to improve first-glance readability. 

The reference list often shows the arxiv URL, but not the publication venue if the paper was accepted. For example: “LASER: Learning a Latent Action Space for Efficient Reinforcement Learning” was accepted at ICRA 2021. At first glance the reader may think many of the references are not from peer-reviewed sources. 

Spelling / Grammar / Typos:

“to a specific types of tasks”

“These includes task-conditioned”

“our goal is learn a policy”

“representations of behavior is critical”

“train the policy the aggregated set”

“prior dataset that similar”

Very minor, (to improve readability):
“In contrast to other skill-based imitation learning (see
Sec. 3) approaches,” -> “In contrast to other skill-based imitation learning approaches (see
Sec. 3),”

“Yet despite their appeal, after experimenting with one such approach, R3M [28], we found that it can sometimes hinder downstream performance.”  -> as this was encountered later in this paper, maybe change the wording to “for the tasks explored in this paper we found that it can sometimes hinder downstream performance.”
review_title: null

Meta-review: metareview: Phase 1:

Strengths:
The submission targets a relevant problem and is written clearly and intuitive in its argument. The proposed method effectively combines known techniques with convincing empirical results and useful ablations which demonstrate the contributions of individual components.
The real world experiments find particular appreciation among reviewers.

Weaknesses:
While the presented experiments are helpful, multiple reviewers ask for further ablations to improve understanding (distance in embedding space wrt temporal aspects, impact of different data sizes for retrieval, comparison of resulting behaviors to retrieval dataset). 
Further criticism applies to the limits of the used manipulation tasks and reviewers propose different paths to extend them. Finally, additional clarifications would also help to connect the method section to existing prior work. 

Phase 2:

The feedback has been generally positive with some criticism. The submission targets a relevant problem and is clearly written and intuitive, convincing empirical results, real world experiments and useful ablations. Multiple reviewers ask for further ablations and clarifications to improve the understanding. I agree with the reviewers and recommend acceptance. Please take the remaining points from the review process seriously and follow up with improvements on open points and promised changes.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)