Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the challenge of improving sample efficiency of RUDDER-style algorithms in sparse MDPs. Building on prior work by Arjona-Medina et al [1], the authors incorporate demonstrations of optimal trajectories from an expert in the training pipeline. Additionally, to improve the sample efficiency and stability, the authors replace LSTM-model of RUDDER with an alignment based profile model. 

The approach is evaluated on two synthetic grid-world based environments and a MineCraft based environment. On both benchmarks the proposed algorithm works better than baseline RUDDER. 
main_review: Strengths:
+ The empirical evaluation is well laid out and the experiments are described with necessary details. The authors perform careful evaluation on multiple (synthetic) environments to build intuition for and motivate the changes in the training algorithm.

Weakness:
+ The paper writing could be improved to better explain the prior work and segregating the core contributions of the work (e.g reward redistribution builds on RUDDER). 
+ The experiments are restricted to navigation based grid-world style environments. It would be useful to have more comparison on other benchmark tasks like locomotion.
summary_of_the_review: The authors consider the challenge of improving sample efficient HRL in sparse environments. They identify several drawbacks in RUDDER and propose effective modifications to the learning algorithm to improve performance. While the experiments are promising they are restricted to navigation style experiments, it would be useful to have comparison to more imitation learning based baselines with diversity in learning environments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose Align-RUDDER, which aligns given task demonstrations (sharing a common strategy) using a profile model, to identify common events as subgoals and redistibute reward to realize more efficient and effective RL. Align-RUDDER replaces RUDDER's replay buffer with task demonstrations and RUDDER's LSTM with a simpler profile model to maximally leverage high reward but scarce demonstrations in tasks characterized by sparse and delayed reward signals, which are notoriously difficult to explore effectively.
main_review: Strengths:

- Simple, effective and general technique for identifying subgoals with few expert demonstrations sharing a common strategy

- Large gains over RUDDER and sota imitation learning methods on 2 gridworld tasks

- The only technique that was able to mine a diamond in the minecraft challenge with automatically learned subgoals (infrequently)

Current Limitations:

- As shown in figure 6, the success rate of Align-RUDDER is high, but then degrades rapidly in the last quarter of the minecraft task. An analysis of what elements of Align-RUDDER contributed to that degradation would be highly instructive in understanding the current limitations of the approach, and future research directions.

- Align-RUDDER's "five steps" are described completely enough, but I feel that the presentation needs to be further improved for those not already familiar with profile models (many). Expanding figure 3 and better connecting the description with it would improve the paper substantially.

- The assumption of a single underling strategy in Align-RUDDER seems like a significant limitation. A clustering step to identify multi-strategy profiles will be required in many situations. How common such situations are and how they would be detected and mitigated with e.g. with multi-strategy profiles is not adequately discussed.
- Figure 1, which describes the basic intuitions behind the RUDDER approach, distributes all of the reward to earlier subtasks, leaving none for the end-goal, which seems like an issue...

Post-rebuttal comments:

Thank you to the authors for their response. I have looked at the other reviews and re-read parts of the updated paper, and I tend to agree with the more critical reviews wrt the following: 

1) The explanation of RUDDER is still not complete or detailed enough to appreciate Align-RUDDER's relative strengths (and weaknesses), and thus the contributions of the paper. 

2) The explanation of Align-RUDDER needs significant work wrt both the layering and quality of explanation, it is just not clear enough. Read the first sentence in (V) of section 3, does it make sense? There are many more examples of poorly crafted sentences. Perhaps more significantly, it just doesn't come together to explain the technique clearly. In my opinion it still needs to be completely revised.

3) In addition, following up on my concern with figure 1, I also take issue with the "Q-functions are step-functions statement", and the characterization around figure 1 that RUDDER makes all EFRs zero... These are not the correct characterizations. RUDDER is representing the EFR with a (sparse) set of reward advantage events through reward redistribution, and as such, the details around RUDDER become very important--perhaps with better "event detection" regularization, the LSTM approach is far more effective than the Align approach, which is constructed in a somewhat ad-hoc manner with a diverse set of tools. 

Based on these considerations, the paper is still in need of substantial revision, and I have lowered my score.
summary_of_the_review: A solid approach to the difficult and important problem of learning from scarce expert demonstrations. An analysis of the success rate degradation in the last part of the minecraft task, and an improved explanation of Align-RUDDER's "five steps" would strengthen the paper significantly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper uses Sequence Alignment technique to redistribute rewards, to a similar effect as with LSTM in RUDDER. The hierarchical agent is trained with behavioral cloning and fine-tuned with RL (tabular in Rooms environment / PPO in MineRL). Tasks are automatically divided into subtasks and specialized agents used for the subtasks. The main contributions seem to be: a) introduction of the Sequence Alignment technique which works well with few expert demonstrations; b) experimental demonstration in Rooms / MineRL.
main_review: While the results in Minecraft are presented as impressive, the paper contains several shortcomings from the strict scientific point of view. The key question - is the new method working better than the original RUDDER and why? - is poorly addressed. The comparison is made only on two different GridWorld problems, and it is unclear whether the benefit is from the new reward redistribution technique or division to subtasks. An ablation study and experiments on different domains, perhaps those used in the original RUDDER paper, are required to find the answer.

Moreover, some key concepts of the paper are poorly explained. Section "Defining Events" does not explain how the clustering is exactly done. A function "f(s, a)" remain undefined. I don't agree to the authors' claim that "Q-function of an optimal policy resembles a step function" and "is mostly constant". Presentation of the paper can be improved: It is not clear why the biological sequences in Fig. 2 left is included. Fig. 6 include letters on x-axis without any explanation about their meaning. Grammar and typography can be improved.

summary_of_the_review: Given all the drawbacks, the paper is not ready for publication. I would recommend the authors to focus more on the in-depth comparison with Rudder, save some space by moving the figures regarding Minecraft to the appendix, rewrite the key concepts in more depth, verify the key assumptions and demonstrate WHY the new method is better.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an improvement over RUDDER to allow for RUDDER-style reward redistribution when given limited sets of demonstrations by using a biological sequence alignment model.

main_review: ## **Paper Strengths**

**Motivation**: The authors make very clear the motivation. Improving RUDDER, a reward redistribution method, so that it can work on tasks in which demonstrations of high-reward trajectories are given because the task is too hard for exploration alone. 

**Method Novelty:** Applying a sequence alignment mechanism seems relevant and a novel contribution, also leading to good empirical results.  

**Experiment Results:** Align-RUDDER seems to have great performance improvements over RUDDER, especially with a very limited number of demonstrations. The compared baselines also seem to be relevant, demonstrating Align-RUDDER's good performance. Furthermore, the MineCraft diamond mining demonstrates impressive performance gains. The authors also perform hypothesis testing, further validating the results of their experiments.

**Supplementary Material:** The authors provide a detailed set of background knowledge for RUDDER, sequence alignment, and extra figures/examples in the appendix. Furthermore, the supplementary video was nice and informative.

 

## **Paper Weaknesses**

**Grammatical Issues:** A few grammatical issues throughout, it doesn't detract too much but occasionally causes a hiccup in the flow when reading, so please fix these.

**Method Clarity**: 

- The paper figures do not explain the method too well. Figure 2 is a decent figure, but the caption does not really explain the right half, nor does it draw any parallel between the two parts of the figures. I think this figure should be simplified, labelled, and used to better show how the reward redistribution using conservation score mechanism works in the biological sequence alignment case, and in the reward redistribution case. Figure 3 is nice, but it should have some text explaining things better in the caption. As the actual explanation for how the method + alignment strategy works is very complicated, these figures are critical for showing it clearly at a higher level.
- Furthermore, there should be some more intuition presented earlier about *why* sequence alignment is easier than RUDDER's LSTM.
- In Section 3, "Reward Redistribution by Sequence Alignment," to someone unfamiliar with biological alignment techniques, you should explain why alignment will intuitively help first, before diving into details.
- In fact, in general, I think at least to me the amount of detail given to the alignment explanation should be redistributed for a conference like ICLR. The authors should rewrite the method section to have far fewer details (except those explicitly needed) about the alignment algorithm itself, this should all be in the appendix. Instead, the entire reward redistribution scheme should be used to explain both intuitively and in detail how this 1) encourages alignment of similar trajectories in terms of states and actions, 2) better alignment scores  result in a better reward redistribution scheme, and 3) is far easier to learn with few demonstrations than RUDDER's LSTM. Of course, some details should be kept but in general I think there is too much detail placed in the nitty-gritty of *how* alignment is done in the main paper.

**Experiments:** To me, the experimental setup seems valid. But along the same lines as what was stated above, there should be a bit more analysis in the main text about *why* Align-RUDDER performs the way it does in comparison to the other methods.

## **Questions**

- Appendix page 33: "We...transform images into feature vectors using a standard pre-trained network," what standard pre-trained network was used?

## Minor Issues

- Contribution 2 and 4 in the intro are saying basically the same thing
- "Sub-tasks" in page 5, is this meant to have a new line before and be bolded instead?
- Page 7, describing the hierarchical setup, "more details can be found in the appendix," please link an appendix section for easier referencing here
- The reference to learning methods should be a clickable link back to that subsection describing the learning method (i had forgotten the learning methods by the time I reached page 6 where it's mentioned in the experiments)

summary_of_the_review: In summary, I think this paper presents strong results, and is a valid technical contribution, but the main reason I am currently voting for reject is that it needs some rewriting to address my clarity concerns. I am willing to adjust my score if the authors can improve on this.
review_title: null

Meta-review: ## A Brief Summary
This paper proposes two critical modifications to the original RUDDER algorithm:
1. Proposes the Align-RUDDER method that assumes that the episodes with high rewards can be used as demonstrations.
2. Uses a profile model from the Multiple sequence alignment approach to align the demonstrations and redistribute the rewards according to how frequently events in the demos are shared across different demonstrators. MSA is being used as a profile model instead of LSTM.

The paper uses successor features to represent state-action pairs, which is then used to compute the similarity matrix used for MSA afterward. 
The paper shows promising results in the Minecraft environment (ObtainDiamond task,) as well as synthetic grid-world environments. 

## Reviewer bJbP
*Strengths:* 
- Empirical evaluation is well-done.
*Weaknesses:*
- The writing requires more work.
- Limited experiments: Mostly on toy-grid world/navigation environments, it is not clear if the results will generalize to the control problems.

## Reviewer mK3T
*Strengths:* 
- Simple and effective technique for identifying sub-goals.
- Large improvements over original RUDDER.
- Impressive results on Minecraft.
*Weaknesses:*
- More through ablations on the importance of different elements of Align-RUDDER.
- Presentation and writing need more improvements.
- Assumption of a single underlying successful strategy is an important limitation.
- Figure 1 is problematic and confusing because of the way it explains the RUDDER algorithm. 

## Reviewer nk2L
*Strengths:* 
- Impressive results on Minecraft.
*Weaknesses:*
- Poor justification and motivation.
- RUDDER vs Align-RUDDER comparisons are only done on two grid-world environments.
- More ablations are required to justify the approach.
- Writing requires more work, some important concepts require more clarity. Some undefined concepts...
- Incorrect claims such as:
> Q-function of an optimal policy resembles a step function

## Reviewer YcqX
*Strengths:* 
- Strong motivation.
- MSA for demos is novel.
- Strong experimental results.
*Weaknesses:* 
- Several grammatical errors.
- The method is not explained well in the paper, the writing needs more work to improve the clarity.
- Lack of sufficient analysis and ablations on the Align-RUDDER approach.

## Key Takeaways and Thoughts
Overall, the result provided in this paper in the Minecraft environment is impressive. The motivation for the Align-RUDDER is clear for me. I like the paper; in particular, the application of the MSA for the alignments across the demos is novel. However, as all the reviewers of this paper agreed that the paper is unclear, especially the method description requires more work. The paper needs to present more ablations and analysis to justify which components of Align-RUDDER algorithm are essential. I agree with both insights, the authors have made improvements in the paper to improve the exposition of the algorithms, but still, the paper feels a bit rushed. I would recommend the authors reconsider the paper's current structure and improve the writing further, especially the description of the method can be further improved. I would recommend that the authors fix those essential issues with the paper and the other comments reviewers made in a future resubmission.
Decision: Reject