Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents Self Imitation via Reduction (SIR) an approach to learning long-horizon tasks by successively reducing it to easy to solve tasks, generating solutions to these easier tasks and self-imitation on successful task solutions. This is done by training a goal-conditioned policy together with a universal value function. When given a task that cannot be solved via the current policy, SIR searches for an intermediate state that divides the task into two easily solvable sub-tasks; this search is carried out by maximising the (composed) value of the sub-trajectories under the current policy. The policy is then executed for these sub-tasks in sequence; success in both these sub-tasks means a solution for the full task is now found. This solution is used as a demonstration that the policy can use for self-imitation via an advantage weighted behavioural cloning objective. This is combined with the policy loss of standard actor-critic algorithms such as SAC and PPO in both the off-policy and on-policy settings respectively and shows significant performance improvements compared to baselines on several long-horizon tasks such as robotic pushing, stacking 3 blocks and a multi-room maze task. 

A few points:
1. This approach tackles an important problem of effectively learning policies for long-horizon tasks taking advantage of compositional structure of sub-problems. It nicely combines several ideas from prior work such as self-imitation and universal value functions to present a method that is conceptual simple but performs well empirically on complex tasks.
2. SIR seems quite related to the two-level architecture in standard hierarchical RL approaches — the planner for task reduction is the top level and the low level being the policy. There is a few key difference though: SIR uses this structure primarily for learning and over time the knowledge in this bi-level structure is distilled into the low level policy. It would be interesting if this is discussed a bit further in the paper.
3. Unlike traditional sub-goal selection methods which recursively decompose the problem into sub-problems, SIR does a single reduction step to decompose the task into two sub-tasks. This works well under the assumption that the goal-conditioned policy has sufficient representational capacity to capture the variety of tasks in the environment. Is it possible to extend the current approach to a recursive decomposition for harder tasks?
4. The paper is very well written. There is a clear motivation, contributions and a thorough overview of the related work in this area. The discussions are well structured and together with the appendix a lot of detail is provided on the experiments and methods.
5. A crucial component of the proposed approach is the search for possible reductions. In the proposed approach this is highly structured and limited to very few dimensions of the actual observation space (e.g. only considering object translations in the pushing task, only considering moving unstacked blocks in the stacking task). This provides a key advantage to SIR compared to baselines as it significantly reduces the branching factor of search and consequently can lead to many successful reductions early on during training. This somewhat reduces the strength of the proposed results. As an additional baseline, it would be good to see the performance achieved by SIR when search is not structured and allowed to explore all dimensions of the state space. Does this reduce the performance and/or learning speed of SIR?
6. The paper presents initial results on a vision-based task where a VAE representation is used as state. What is the dimensionality of this representation? As mentioned above, this can have a significant impact on the learning performance (while CEM should do better compared to random search it is not clear if this can mitigate the issue by itself).
7. Another key limitation of the proposed approach is the reliance on arbitrary resets which is not feasible in the real world. While this is briefly discussed in the paper it is not clear how this can be mitigated easily. A more detailed discussion would be useful.

Overall, the approach is quite nice and the initial results are encouraging. I would suggest a weak accept.
review_title: Task reduction and self imitiation; simple approach with nice empirical results

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Summary 
This paper proposes a new method that combines task reduction and self-imitation learning for goal-based reinforcement learning problems. The idea is to decompose a hard task into two subtasks (subgoals) such that the solution to one of them is already known. Self-imitation learning is used to quickly learn to reproduce such successful trajectories. The experimental result shows that the proposed method outperforms the baseline SAC + HER and SAC + SIL as well as hierarchical architectures such as HIRO and DSC.

## Pros
* The idea is interesting and novel.
* The empirical results are good.

## Cons
* Limited to (a subset of) goal-based RL problems.

# Novelty
The proposed idea of decomposing a task into two easy tasks is novel and interesting. It would be worth citing and discussing a relevant prior work [1], which also proposes such a decomposition for goal-based RL. 

# Quality
* The empirical results are good. Specifically, it is interesting that the proposed method outperforms hierarchical RL methods without being explicitly hierarchical.
* At the same time, the proposed method seems very specific to a subset of goal-based RL problems, where searching the goal space is computationally tractable. 
* Though I appreciate the extension to visual domains using $\beta$-VAE to remedy the aforementioned limitation, the U-Wall maze doesn’t seem like visually complex, and the result (Figure 8) is not very strong. Either showing much better results on Figure 8 or showing results on complex visual domains would strengthen the claim. 
* It would be more convincing and interesting to show that the proposed method can do deeper planning by applying task reduction recursively.

# Clarity
* The paper is easy to follow, and the figures are well-presented.  
* What is the rationale behind $V(s_a, s_b, g_a) = V(s_a, s_b) * V(s_b, g_a)$? This seems quite specific to “goal-reaching” 1-or-0 reward structures. Do you have an idea how to generalize this to more general reward structures? 
* In Algorithm 1, do you generate a new trajectory to get $\tau’$? If this is the case, is this taken into account as the number of steps (x-axis) in the learning curves? Otherwise, they are not fair comparisons. 
* Just to check if they are apples-to-apples comparisons, do you use HER across all methods (yours and baselines)?
It would be good to mention that this paper considers goal-based RL problems early in the paper (in abstract or introduction). 

[1] Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals, Vikas Dhiman et al.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The submission proposes an intuitive curriculum learning method which focuses on sparse reward tasks in RL and uses universal value function approximators. 
It has 3 explicit steps: 
1. identifying a state to decompose the one task into two 
2. solve these new tasks 
3. solve the complete task by imitating the trajectories from both subtasks.
 
On the positive side, the paper is overall clearly written and easy to follow for most parts and performs commensurate or better to baselines on a set of simulated manipulation and locomotion domains.

On the negative side, the method often only performs commensurate or close to the baselines while introducing significant added complexity and additional hyperparameters. The stacking task, which demonstrates the strongest benefit for SIR, requires strong domain knowledge as the search for intermediate states is highly constrained.  Constraining the search for intermediate states to positions with blocks under the required stacking position is significant domain knowledge unavailable to the other methods. The minimum requirement for a fairer comparison would be to include a version of SIR without this constraint. 

More generally, aspects regarding the specifics of the space in which we search for intermediate states and the baselines remain unclear (e.g. in terms of the search space since according to the appendix the space for states and goals is not the same and e.g. for stacking other constraints exist).

The final problem regarding the evaluation is that while presenting essentially a curriculum learning method, the paper does not compare against other work in curriculum learning as baseline (e.g. [1,2]).

Other questions remain such as the surprising statement that off-policy SAC underperforms on-policy PPO on the navigation task. Statements that are counter to intuition and existing comparisons between SAC and PPO should be supported with experimental results.

Overall, the introduced method follows a valuable direction for curriculum learning in RL but the submission demonstrates significant weaknesses regarding fair evaluation.

[1] Florensa, Carlos, et al. Automatic goal generation  for reinforcement learning agents. In International Conference on Machine Learning 2018
[2] Racaniere, Sebastien et al. Automated curriculum generation through setter-solver interactions. In International Conference on Learning Representations 2020.


(Disclaimer: I have reviewed a previously submitted version of this work and a big share of critical points remains the same between both reviews including domain knowledge unavailable to baselines and comparison to other curriculum learning methods.)

review_title: Relevant method, good performance, very limited baselines, and some question remain open.

final_score: rating: 3: Clear rejection
review_content: #######################################################################

Summary:

In this paper the authors propose a method for solving compositional tasks in the RL setting. The method (Self-Imitation via Reduction), is a 2-step method in which the agent first reduces the target task into two simpler tasks, and then solves the full task using as a demonstration the composite task uncovered in step-1.

#######################################################################

Reasons for score:

I am currently voting for a rejection based on what seem to be two key omissions from the paper:

1. The way in which the set of 'library' tasks is selected, and
2. A measure of how expensive this offline component of the algorithm is

The experimental comparisons could also have been stronger.

#######################################################################
Pros:

1. I think the paper is reasonable well written. I found just a few typos, and I think the key conceptual ideas are reasonably easy to follow. 

2. The 2-step implementation of some sort of "policy reduction" followed by an imitation learning step would seem to be a good in principle idea that merits further exploration

#######################################################################

Cons:

1. In Sec 3 the authors talk explicitly about a multi-task RL learning setting, and indeed a central part of the method is a search step over interim states $s_\beta$ , but I do not see explicitly how this set is constructed. This is of course a critical consideration for a number of reasons:

(1)  If this set of tasks is large, then you are likely to find a good $s_\beta$, but the search space increases (as does the memory footprint)

(2) If this set is small (and perhaps curated), then the subsequent result is weak (if the agent need only search over a small handful of interim tasks which already include moving the elongated box say, then of course the subsequent learning is rapid). 

2. The experimental results would be more compelling if comparisons were made to methods that similarly had some access to interim policies. The comparisons to SAC for example are valid in that they provide a lower bound, but there is additional information available to SIR. Similar it is unclear what the comparison to SAC with a dense rewards shows exactly.

3. The authors mention that the method is extensible even though "... tasks reduction only performs 1-step planning... SIR still retains the capability of learning an arbitrarily complex policy by alternating between imitation and reduction: as more tasks are solved, these learned tasks can recursive further serve as new reductions..." - this is a nice idea, but I saw no evidence of this implementation in the paper.

#######################################################################

Questions during rebuttal period:

Please address the concerns above. Also, you have some comparison with hierarchical policy algorithms in 9.a/b - is it possible to extend these to the other domains?

#######################################################################

Some general comments:

- I think the results for these multi-step methods are often easier to digest and understand if the phases are presented separately:
    - Phase 1, reduction:
        - for each experiment, which task reduction was uncovered by the agent?
        - as you know, in many cases there are multiple valid reductions, which does your algorithm find and why?
        - since your reduction phase is 1-step and greedy, in which situations might it not work so well
        - etc.
    - Phase 2, imitation:
        - comparison to other methods
        - effect of parameter choices
        - etc.
- A little more justification for you particular experimental comparisons can be helpful. It seems like there are a few potential avenues you might have liked to explore:
    - comparison to an information impoverished baseline (like SAC)
    - comparison to different task distributions
    - comparison to other subtask methods
    - etc.
review_title: Review: Solving Compositional RL problems via task reduction: Some key omissions

Meta-review: The paper proposes a method for solving challenging sparse reward problems by performing task reduction followed by self-imitation learning from solution trajectories to the reduced tasks.  The core innovation seems to me to be the uses of the reduction search, which is essentially a form of recursive subgoal selection, but where the subgoals are sure to be achievable as assessed by leveraging the learned value function.  This idea seems rather general, though its use is strongly facilitated in this paper by definition of the space (i.e. that object target is the space, is pre-specified; there is only one, rather limited result on a pixel-based task). 

Note: Another submission to this conference also explores a quite similar idea to the task reduction proposed in this paper -- see "Divide-and-Conquer Monte Carlo Tree Search".  The breaking down of the problem into sub-problems using the value function is similar, but the details of how the papers proceed from there is quite distinct.

This is a difficult meta-review decision due to the fairly mixed reviews, coupled with limited engagement in the discussion phase.  Two reviewers felt the paper was solid and could be accepted (R1 and R2 with scores 7 and 6 respectively).  R3 gave a borderline review that leaned towards reject (score 5).  R3 replied to the initial author response, which provided helpful feedback to the authors. Ultimately, in my assessment, the authors did a fairly thorough job of addressing some of the points raised by R3, including by adding an additional comparison even where they didn't agree with the reviewer. R4 assigned the paper the lowest score of 3.  The authors provided a lengthy reply to this review asserting that the review may have reflected misunderstanding of paper details, but the reviewer did not respond to the authors.    

Two core issues raised about this paper relate to the definition of the space for subgoals and the limited difficulty of the tasks. However, this method does not claim to be entirely ignorant of the task space so I don't see the fact that they do include some domain knowledge in designing the goal space to be totally undermining of the method.  They focus on the complementary issue of how to break down difficult problems into sub-problems.  While it would be considerably more impressive if the goal space were learned, I think this harder version of the problem remains a fundamental and deep problem within AI, so it seems to me too much to ask of the present paper (especially given that it was not the stated focus of the paper).  And while the tasks explored in the paper are a little contrived (some repetitive motifs and designed with a relatively small search space over subtasks), these problems do have some complex structure.  Compared to many works in this field, I applaud the authors for engaging with problems with both long-horizon task structure as well as complex high-DoF continuous control component.

While I agree with some of the concerns raised, my overall assessment is that I find the contributions sufficiently innovative and substantial to justify acceptance.  The authors proposed a specific innovation and evaluated that innovation.  Insofar as their innovation is somewhat general, I don't think this paper can be the last word on how well it compares with the diverse approaches it could be set against.  And while the experiments are not definitive, I do think they do constitute a fairly ambitious initial validation of the core idea.  

Decision: Accept (Poster)