Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: What is the specific question/problem tackled by the paper?
This paper addresses the problem of learning and using useful skills through unsupervised RL. While past work (VIC, DIAYN) found skills by maximizing the mutual information between skills and states, this paper (LTS) aims to find transitional skills to move between the primitive skills.

The proposed method is, if the primitive skills found by DIAYN are labeled w_i, to find skills z_[i,j,k] which corresponds to the kth state while transitioning from w_i to w_j. These transition skills are learned jointly with the primitive skills using the mutual information objective.

The results in the paper are on mountain car, cart pole, and pendulum. The plots show that distinct primitive skills can be learned in these environments, and that the transition skill transitions between them. The results do not show the actual performance of using these skills for any specific task, which was the motivation presented in the introduction. The 

Is the approach well motivated?
The approach is not well motivated. The paper claims that transitional skills are necessary, but does not show any evidence of an agents behavior improving by using transitional skills. 

I argue to reject the paper in its current state; Sections 3 and 4 need to be rewritten to be more easily understandable, and the need for transitional skills should be more empirically motivated.


Clarity Issues:
- The method is presented in a very confusing way. I am not confident that I understand how the transitional skills are defined. 
- Figure 2: What is the “statistical characteristic” and “feature value”? The vague terms are not defined. Which value? What statistic?
- Figure 5: I see that LTS transitions more smoothly between skills, but why is DIAYN’s fast transition bad? Clearly it still succeeds at reaching the other state.

--------------------------------
Update
Due to the additional experiments that show the benefit of LTS, I have increased my score to weak reject.
Section 6.5: Thank you for adding this additional experiment. This section lacks details about how the experiment was run. How was the meta-policy trained? In “3) LTS-master: Transfer policy with optimal weights learned by meta-policy”, why is there transfer? Transfer from what to what? 
“Moreover, we conduct hierarchical framework to weight (or choose) the action modeled by different primitive skills.” What does this sentence mean?

Reading over the paper again, the paper needs several editing passes to fix the grammar, as a number of sections (like above) are not understandable. While the paper has improved and is on a good track, I do not think it is yet ready for publication. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper aims to learn transitional skills in addition to creating diverse primitive skills without a reward function by using an information theoretic objective, borrowing ideas from instrinsic motivation. The paper addresses the challenge of learning diverse primitives and also learning to transition between any pair of them, which is valuable for real-world complex problem solving with sparse rewards. The proposed LTS approach is compared againt a state-of-the-art DIAYN approach on several benchmark control tasks.

I have several comments/concerns regarding the experimental results.

1. How would the feature engineering used (Appendix E) affect the performance of tasks solved using the primitive skils learned using LTS and how `would policies learned with this framework perform compared to standard RL policies learned individually with external rewards? This is to establish whether the learned skills and transitions between them are useful for any down-stream tasks. In the DIAYN paper, experiments on down-stream tasks were provided.

2. What are these learned primitive tasks for each benchmark task? A visualization (or some interpretation of the meaning of the skills learned) would prove useful in understanding the primitive skills learned. Again, the DIAYN paper visualizes the primitive skills learned in their tasks.

3. I don't understand how the results in 6.3 demonstrate the generalization ability of the method. The authors clearly state that using 3 transtional skills fails when evaluated on 50 transitional skills. Can this be elaborated on? 

4. Related to 1 above, I found it difficult to relate the features used for the tasks with the actual tasks themselves. More background information and motivation for the features is required (as opposed to saying it makes distinguishing between skills better -- what does better mean here?).

5. Lastly, the paper is full of silly typos, such as the incorrect usage of singular and plural nouns. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: General description
The paper tackles the problem of transition between different skills in hierarchical reinforcement learning. 
In particular, they follow the work of VIC and DIAYN and define an information-theoretic objective function that maximizes the mutual information between the future states and options, given the initial state, while minimizing the distance between the options and the so-called transitional states. The algorithm, LTS, is compared to DIAYN on three environments, namely CartPole, MontainCar, and Pendulum.

General remarks,
The problem tackled and the proposed idea are interesting; however, I am not fully convinced by the derivation and the experiments of the paper. The writing is in general not particularly clear and the notations are hard to follow, the symbols are often bloated with superscripts that are not clearly defined, and mixing capital and small letters for random variables and their realization.

On the derivation, Eq 3. How is it possible to replace p(s^p | p^t) with p(omega|s_t)? I understand the connection between the two but what guarantees the mutual information is still maximized? (the whole derivation depends on that)

The experiments and the plots are interesting, showing a smoother transition between skills than DIAYN, however, it is still not clear how that can help solve the task at hand. Could the author elaborate on that please?

Some details:
- Page 2 \in should be \sim. In general, the notation does not clearly distinguish a random variable (in capital) from a realization (in small letters). For instance, page 3, big \Omega (the random variable I presume) is written with a subscript i to indicate the ith skill. 
- Footnote 1 page 2, a log is missing in the MI definition. 
- Page 3, what does t refer to in S^t_{i ,j, 1}?
- Page 3, muture -> mutual. 
- the paper mentions the experiments are conducted on MuJoCo but the appendix mentions the classical OpenAI Gym experiments. 
review_title: Official Blind Review #3

Meta-review: The submission has two issues, identified by the reviewers; (1) the description of the proposed method was found to be confusing at times and could be improved, and (2) the proposed transitional skills were not well motivated/justified as a solution to the problem the authors propose to solve.
Decision: Reject