Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a solution to long-horizon manipulation problem through chaining skills. The main contribution of the paper is an adversarial learning pipeline to bring the terminal states of one subtask close to the initial state of the upcoming one. The authors show that the proposed framework can significantly outperform IL, RL, IL+RL baselines, as well as an existing algorithm for skill chaining on two simulated IKEA furniture assemble tasks. 
strengths_and_weaknesses: Strengths:
- The presentation of the paper is very clear. The authors also provide well-made Figures to facilitate the understanding of the proposed solution, in particular, Figures 1 and 2. 
- The research problem is well-motivated. The authors not only accurately point out the limitation of existing approaches, but also validate these intuitions in the experiments (Figure 4). 
- The proposed solution of using adversarial learning is clean and neat. The algorithm proposed by the authors also seem very practical. 
- The authors compare the proposed algorithms with various baselines including IL, RL, IL+RL, and anther skill-chaining approach. The authors also provide explanations on when and how the baseline fails. 

Weaknesses:
- The authors did a thorough literature survey in the field of RL, IL, hierarchical RL, and skill chaining. However, it would be great if the authors can include existing work on learning for long-horizon robotic tasks, which is also relevant to the paper. A few examples here:
  - Driess et al. Learning Geometric Reasoning and Control for Long-Horizon Tasks from Visual Input. ICRA 2021
  - Kase et al. Transferable task execution from pixels through deep planning domain learning. ICRA 2020
- It would be great if the authors can expand the technical section (Section 3) a little to provide more technical details. In particular, it would be great if the authors can write out the loss function for $f_\phi^I$ rather than referring readers to [46]. I personally think it is important to also point out that the output range of $f_\phi^I$ is actually $[-1, 1]$ (rather than $[0,1]$ in more common formulations, including what the authors use in Eq. (2)). It actually took me a while to understand why there is a $0.25$ in for $R_{GAIL}$. 
- Related to my previous point, it is interesting that the authors choose to use different adversarial formulations for the policy (IL) and the terminal states (TSR). The ranges of the discriminators are different ($[-1,1]$ for IL and $[0,1]$ for TSR); How the rewards are related to the discriminator is also different (quadratic reshaping for IL, while linear reshaping for TSR, both of them are different from the original formulation of $\log(\mathcal{D})$ in the original GAN/GAIL papers). Although the authors mentioned in the appendix that they observe the quadratic reward [46] works better. It would be great if the authors can provide more thorough discussion on these design choices and/or theoretical comparison of them. 
- In the algorithm, the authors choose to pre-train the subtask policies (line 3-7) with a fixed set of initial states before the adversarial training. How does this pre-training stage help? It would be great if the authors can provide some intuitions and/or experimental results.
- Also in the algorithm, line 17-18, why is $\mathcal{D}_\omega^I$ updated after the policy $\pi_\theta^I$?
- Line 220, "Subtasks are independently trained on the initial states sampled from the environment with random noise ranging from [−2cm, 2cm] and [−3°, 3°] in the (x, y)-plane." Are the "states" here referred to the states of the furnitures? Why is the perturbation only on the (x,y)-plane? 
- The fact that the authors reset the robot arm "back to its initial pose in the center of the workplace" after each subtask seems to shrink the set of terminal states quite a bit. Would the proposed method also generalize to when we don't reset the state of the robot after each subtask completion?
summary_of_recommendation: The paper solves a well-motivated problem for skill-chaining for long-horizon robot manipulation tasks. The authors accurately point out the limitation of the existing approaches, and also empirically validate these limitations in the experiments. The proposed solution seem technically sound and practical. The experiment validation is thorough. Though it would be great if the authors can justify some decision choices (see "Strengths and Weaknesses" part). 
issues: See the "Strengths and Weaknesses" part.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes the algorithm T-STAR which approaches the problem of learning complex extended time-horizon skills through subtask skill chaining. T-STAR combines ideas from RL adversarial imitation learning to regularize the termination state distribution of a subtask policy to be close to the start state distribution of the next subtask. T-STAR builds on ideas in [6] by addressing some limitations in that method. The authors evaluate T-STAR on two furniture assembly tasks in simulation and show that a model-free RL method (T-STAR) is able to solve furniture assembly tasks without manual engineering.
strengths_and_weaknesses: **Strengths**

* The problem of learning complex long time-horizon skills efficiently is important and relevant to robot learning. The challenges in doing so are clearly identified and well explained. Finally the motivation for the proposed algorithm, T-STAR, is well grounded in the limitations of prior work (see section 3.3 which is especially clear).
* The proposed algorithm T-STAR is interesting and a novel approach to the skill chaining problem as far as I am aware. It also has the benefit of not requiring any additional policies to be trained over the sub-task policies.
* The qualitative results (section 4.4 and Figure 4) are a nice way of assessing if T-STAR is behaving as intended. It also visualizes the advantage of T-STAR over policy sequencing [6]. However I do have a question about the choice of time steps (see below)
* Since the authors are very clear about the differences between T-STAR and PS [6], the difference between PS and T-STAR in Table 1 quantifies the effectiveness of terminal state regularization. However I have a question about the robustness of the results (see below).
* Clarity of presentation is excellent. The paper is lucid and easy to follow. In particular, Figures 1 and 2 are very clear and do a great job of summarizing the main ideas and algorithm. The use of color highlighting in Algorithm 1 to highlight the differences with prior work is also useful.
* Inclusion of a mostly comprehensive set of baselines covering behavioral cloning, RL, imitation learning (adversarial), hierarchical RL and policy sequencing through initial state expansion.
* The implementation details (appendix), algorithm description (algorithm 1), and task description (section 4.2) appear to contain sufficient information for implementing T-STAR and reproducing the results.

**Weaknesses**

Lack of robustness in results
* Only a single score for each algorithm (both for task completion - Table 1 and success rate - lines 266-267) is presented in the main paper, and this appears to be from a single run. There is no discussion of variance in performance or why only a single result is presented.
* The appendix does present an additional result over 2 or 3 random seeds depending on the task, however there is a discrepancy between the results presented in the main paper (Table 1) and the results presented in the appendix (Table 2) with the results in the appendix on 3 of 4 key results being lower in the appendix.

Table 1:

PS 0.82 | 0.7

T-STAR 0.95 | 0.8

Table 2

PS 0.75 +/- 0.06 | 0.68 +/- 0.09

T-STAR 0.86 +/- 0.09 | 0.87 +/- 0.07

* It is not clear if Table 2 shows results with new seeds different to the seed used in Table 1 or if Table 2 includes the results in Table 1.
* It is also not clear why only 2 seeds are selected for TABLE_LACK but 3 for CHAIR_INGOLF
* There is a discrepancy between the caption for Table 2 (which implies that 3 seeds have been used for all tasks) and the text.
Looking at the results across Table 1 and Table 2, it appears that T-STAR performs significantly better on CHAIR_INGOLF but not on TABLE_LACK. However for the reasons described above this is difficult to assess.
* To address these robustness issues I suggest
    1. Unifying Table 1 from the main paper and Table 2 from the appendix and presenting the mean and std deviation across all random seeds in a single table in the main paper.
     2. Presenting the mean and standard deviation of the task success rate across all random seeds.
     3. Being clear about how many seeds were used in total for each task and reporting results averaged over all of these.
     4. Using the same number of seeds for both TABLE_LACK and CHAIR_INGOLF
     5. Not bolding results if the distributions overlap (e.g. Table 2: TABLE_LACK).

Single policy baselines do not appear to be a fair comparison with PS and T-STAR
* The single policy baselines BC, PPO, GAIL, GAIL+PPO are trained for 100M environment interactions, sPiRL is trained for 5M, whilst the PS and T-STAR results are trained for at least 200M interactions (not including the sub-task pre-training).

Whilst there is good coverage in baselines, two alternative approaches that are highlighted and cited by the authors as relevant are not included as baselines. One baseline is an important omission, the other is more minor.
* Transition policies from [7]. The authors reference this work frequently and it would be valuable to know how T-STAR compares to this approach. This approach directly addresses the policy sequencing problem, so without the inclusion of this baseline it is hard to assess the advantages of T-STAR over this method.
    1. The authors note that [7] is brittle to external state changes and can fail if the end state of a prior subtask policy is too far away from the initiation set of the following task. It would be valuable to see this assessed and compared with T-STAR experimentally on the furniture assembly task. 
    2. [minor] The second approach is perhaps less directly relevant, so it is understandable why it is not included. However, a comparison with one modulated skills method e.g. one from [11 - 14, 9] would be interesting to see.

Minor
* Figure 4
     1. Why were timesteps 36M, 39M, and 42M steps chosen? Without any explanation it seems like these could have been cherry picked.
     2. It would be useful to see the same visualization evenly spaced throughout the fine tuning part of training, from 0 - 50M steps inclusive.
* Algorithm 1
    1. Lines 3 - 7: Is this repeated until convergence for each subtask or only carried out once?
     2. Line 5: Are the expert demonstrations (sampled from D^e) subtask specific or demonstrate the whole task. If the former then it appears that the subtask index i is missing. If the latter, then how are the relevant parts of the demonstration identified per subtask? 
    3. Line10: What is the sampling ratio and how was it chosen?
* Table 1: Average progress over how many trials?
* Typos
    1. Line 28: “requires to adapting the policies to be suitable” -> “requires adapting the policies to make them suitable”
    2. Line 200: “an” -> “a”


summary_of_recommendation: Thank you for this submission. T-STAR is an interesting approach to an important problem in robot learning. T-STAR is well motivated, the paper is very clearly written and the main points are easy to understand. As-is however, it is not clear that T-STAR represents a significant improvement over PS or over the other relevant baseline from [7]. If the questions around robustness and baselines (described above) are addressed this would make a strong submission. In the future it would also be great to see results on a physical robot.
issues: * Lack of robustness in results.
* Fair comparison of single policy baselines.
* Missing baseline from [7] 
* Please also address the minor issues
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a novel learning objective for skill learning in the context of skill chaining. 
Existing approaches to executing a sequence of skills fail when a subtask skill terminates outside a subsequent skill's initiation set. 
To remedy this, existing approaches attempt to iteratively widen initiation sets to account for broad termination sets, but as such are limited to short skill sequences due to compounding error. 
The paper proposes an adversarial learning objective for the termination condition such that the size of the termination set is regularized.  
A discriminator is proposed which is trained to distinguish states from the termination set of one skill and the initiation set of the subsequent skill. 
The discriminator is used as an additional term in the reward function to encourage subtask policies to reach the subsequent initiation set. 

strengths_and_weaknesses: - The adversarial formulation of learning the initiation set is clever. 
- The main evaluation against existing methods is thorough, though the main table of results needs to be run across multiple seeds. 
	Especially since the run reported in the main paper was (apparently) relatively lucky on the first task. 
	Moreover, the improvement is relatively marginal. 

- Relevant previous work is missing; esp. the original skill chaining paper. 
- The theory is given as it pertains to skill chains, though skill trees also warrant consideration. Does the theory handle this case? 

- Supplementary experimentation is lacking. 
	It would be worth adding the initiation set to the visualization shown in Figure 4. 
	This would provide compelling justification to the hypothesis presented regarding the mismatch between termination and subsequent initiation. 
	I am also curious as to how something simple like (approximate) L2 regularization would do as a baseline.  

- Lacks hardware results; understandable given COVID. 
summary_of_recommendation: I'm recommending weak rejection. The idea has merit but isn't entirely original; the evaluation is a bit lacking. In particular, the main result should have been presented across multiple seeds, as in the appendix. Moreover, additional experimentation is warranted in the form of ablation to better understand the behavior of the proposed method, and the overview of existing work is lacking. 
issues: - Please consider citing the original skill chaining paper from Konidaris and Barto. Termination and subsequent initiation sets are designed to be identical in this work. 
- Are the subtask rewards distinct from the environment reward? If so, consider denoting subtask reward with R_{ENV}^i. 
- Line 18 in the algorithm may have a typo? The loss in Equation (2) takes samples from the i-th initiation set and the (i-1)-th termination set, whereas it is written here with termination samples and initiation samples from the same skill execution.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper assumes that a segmentation for a long horizon task is given with individual demonstrations and reward function for each subskill, and aims to learn corresponding policies that, when executed sequentially, succeed in learning the overall task. To make sequential execution successful, the paper proposes to store for each subtask the initial and terminal states for each subskill, if it succeeded. A discriminator is then trained to classify between terminal states of the preceding subskill and successful initial states of the current subskill. The corresponding probability is provided as additional reward (on top of an RL and IL reward term) of the terminal state of the preceding subskill. The policies should thus learn to terminate in states where the subsequent skill succeeds. The approach is tested on a furniture assembly task and compared with a prior method [6] that instead of aiming to reach a good initial state for the next skill, aims to deal with the initial states of the previous skill.
strengths_and_weaknesses: Strengths
------------
* The paper addresses an important problem, namely, how to compose lower-level policies to solve long-horizon tasks.

* The presentation of the approach is mostly clear.

Weaknesses
------------------
* The discussion of prior work is insufficient. Namely, the paper completely misses the branch of research based on skill chaining by Konidaris et al., e.g. [A], [B]. These approaches also train an initiation set classifier and use them to train subskills to reach the initiation set of the next subskill. Whereas [A] considered a reinforcement learning setting, [B] considered learning from demonstrations.

* Especially, when relating the current work to these prior papers, I can not find a significant contribution. Providing additional reward to subskills for reaching initial states from which the subsequent policies succeed, seems to the main idea of the current submission, but this is idea is not novel ([A], [B]). The problem setting is a bit different from those works, because the current submission assumes that the overall task is already segmented and that subskill-specific reward functions are given, but I don't think that these simplifications increase the contribution.

* There is also no theoretical contribution as the presented approach is not based on any proofs or derivations.

* It is not clear how the proposed approach determines whether a subskill succeeds. Is this based on prior knowledge? Is this decision made on the respective state only, or is it maybe even non-Markovian?

* The experimental evaluation seems to be based on only one seed. The appendix contains a table (I assumed based on experiments that were performed after the paper submission deadline) for three seeds, which also contains confidence intervals. However, based on these confidence intervals, T-STAR might actually perform worse than Policy Sequencing (0.86-0.09 < 0.75+0.06). Also, even with three seeds, the experimental results seem insufficient.

* The "GAIL-reward" given in line 139 is very different from the reward used by GAIL. This difference needs to be discussed.

* The paper also uses a rather uncommon "AMP"-discriminator objectives and also an uncommon reward function for the terminal state regularization, which makes it difficult to interpret the effect of the additional reward term. Typical AIL reward functions would aim to minimize a divergence between the terminal state of subskill $i$ and the initiation set of the next subskill. I see that the chosen reward function in the range $[0,1]$ is easier to combine with the RL and IL reward objectives, but at least a short motivation for these design choices would be very important.

Minor
-------
* Why is the one-hot representation of the phase 8-dimensional, even though there are only four phases per task?

* Wouldn't it make more sense to train the classifier between successful and unsuccessful initial states?

* The paper is too repetitive regarding the difference (and benefits) between the current approach and [6]. Namely line 30-43, the caption of Fig. 1, 94-99, 153-161, are mainly repeating the same motivation. I would at least remove the paragraph 153-161.

* Caption should be more descriptive.

* in Figure 2, it would be good to show the "initial state buffer"-box also for subtask 1, and the "Terminal state buffer"-box also for subtask 3. For example, in the current Figure, it seems like "initiation Set Discriminator 3" is based on different types of data compared to "Initation Set Discriminator 2".

References
---------------

[A] Konidaris, George, and Andrew Barto. "Skill discovery in continuous reinforcement learning domains using skill chaining." Advances in neural information processing systems 22 (2009): 1015-1023.

[B] Konidaris, George, et al. "Robot learning from demonstration by constructing skill trees." The International Journal of Robotics Research 31.3 (2012): 360-375.

summary_of_recommendation: My recommendation is mainly based on the perceived lack of contribution. To me it seems like the paper proposes a hack (since no theoretical justification was given) for fine-tuning chained skills, that is not even significantly different from previous approaches. I also think that the discussion of prior work and experimental evaluation is insufficient.
issues: * The revision should discuss differences to prior work [A] and [B].

* Rebuttal and revision should make clear how the approach determines whether a subskill succeeded.

* Rebuttal and revision should clarify why 8 dimensions are used for encoding the phase of the skill.

* Rebuttal (at least) should explain why the classifier discriminates between terminal states of the last skill and successful initial states of the current skill, instead of discriminating between unsuccessful and successful initial states of the current skill.
review_title: null

Meta-review: The paper tackles the problem of long-horizon skill changing for manipulation tasks and proposes an adversarial learning algorithm - T-STAR - which brings the terminal states of one subtask close to the initial state of the upcoming one (avoiding the need for large initial state distributions required for baseline methods). The method is demonstrated to outperform various RL and IL baselines on two simulated IKEA furniture assembly tasks.

During the rebuttal phase, the authors greatly improved the quality of exposition and addressed the majority of reviewer concerns. All missing citations were added. The addition of std estimates (thanks to 5 seeded runs) shows that the empirical improvements are statistically significant. The experimental details were also made clearer. A majority of reviewers increased their recommended score and I agree with this sentiment. The paper is an interesting contribution to the thread of skill chaining in robot manipulation problems with strong empirical results and would be of benefit to the CoRL community.
Decision: Accept (Poster)