Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes an interesting method to construct a world graph of helpful exploration nodes to provide “structured exploration”. This graph is used in an HRL structure based on the feudal net structure. While the method is very interesting the proposed method is designed to help learn good policies via a better exploration structure. This is a very important problem but I find that the environments this method is tested on in the paper can be easily solved using normal RL methods. It would be very important to evaluate the methods progress on more interesting problems with complex temporal structure. Potentially some of the tasks from the HIRO paper or better yet an assembly tasks or version of the clevr objects environment where multiple items need to be rearranged into a goal. One of the more advanced tasks from the Hierarchical Actor-Critic paper would also be a good option. It is also important to include more analysis of the amount of data needed to train the VAE and create the graph. This amount should be included in the evaluation results for the method.

More detailed comments:
-	The last paragraph of the introduction that begins to explain the method is a bit confusing. More detail here would be helpful. How frequently do binary latent variables need to be selected for them to become nodes? Similar for adding edges.
-	In Figure 1, there are many terms that have not been defined yet, "pivotal state", "world graph traversal"... It would help in understanding the figure if these were explained beforehand. The figure text is also very small.
-	This work seems very similar to Search on the replay buffer (Eysenbach et al 2019). That work created a graph over the data in the replay buffer based on the Q value of different states. These act as waypoints in planning. Could this method not be used to also construct a more sparse waypoint graph to use such as what is described in this work?
-	It is said that the primary goal of the graph is to accelerate downstream tasks. Yet, the graph is constructed with states that are most critical in recovering action sequences.  Is there some guarantee that this selection criterion will help downstream tasks?
-	The method also seems to have a similarity to the GoExplore paper that keeps around an exploration frontier, that is similar to the world graph, of states as it is making progress on the task. This paper should be discussed in more detail in the related work.
-	The VAE is trained over data that is collected from the policy during exploration. Is there an issue with collecting data that will extrapolate to explore areas of the state space that are outside of the data collected for training the VAE.
-	More detail should be included in the use of \mu_0. As it is written now it is difficult for the reader to understand how the method works without some of the additional information in the appendix.
-	The method to collect enough data to learn and represent a graph the covers the state space well. How well does this method work? Essentially this method is making progress on the exploration problem. Is there some analysis on how well this method is at collecting enough data to use on downstream tasks?
-	The method uses curiosity to help explore the state space by using the reconstruction error from the VAE as an intrinsic reward. Is a version of this intrinsic reward use for the baseline A2C method in the paper? 
-	It is said that the world graph helps accelerate learning via structured exploration. However, there is there a significant amount of compute and environment interaction to compute the world graph? This should be taken into consideration when performing any comparisons.
-	Can the graph be updated during the policy training phase? Also, in the first phase where data is collected to fit the VAE, can this data be used to train an off-policy method? It seems like this data would work very well for training a policy.
-	Table 3 does not seem to be referenced in the paper. They could also use some additional explination as to what the values represent that are being presented.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Summary: This paper proposes an approach to identifying important waypoint states in RL domains in an unsupervised fashion, and then for using these states within a hierarchical RL approach.  Specifically, the authors propose to use a binary latent variable VAE to identify waypoint states, then an HRL algorithm uses these waypoints as intermediate goals to better decompose large RL domains. The authors show that on several grid world tasks, the resulting policies substantially outperform baseline approaches.

Comments: I have mixed opinions on this paper, though ultimately feel that it is below the bar for acceptance.  There are a lot of aspects to the proposed approach, and overall I'm left with the impression that there are some interesting and worthwhile aspects to the proposed method, but ultimately it is hard to disentangle precisely what is the effect of each contribution.

For example, let's consider the waypoint discovery method.  The basic approach is to use a binary latent variable VAE, using a recently proposed Hard Kumaraswamy distribution to model the latent state.  This seems like a reasonable approach, but there's no real analysis of the actual waypoints that are discovered in the target domains, whether they indeed correspond to intuitively important waypoints in a domain, or whether they are just producing some arbitrary segmentation of the proposed task.

The other elements of the paper have similar issues for me.  The whole HRL process, using these waypoint states as intermediate goals, seems reasonable, but it's hard to disentangle the performance of this particular approach versus the performance of any approach that would use (any) intermediate states as goals within an HRL approach.  And the impression I'm left with, given the level of detail included I the paper, is that I would have no idea how to apply or extend this process to any other RL domains.

I looked at the provided code hoping it would help to clarify some of the implementation details, but the code is not at all a complete collection of routines that could re-create the experiments.  Rather, the code just includes a few of the model architectures, which aren't really the important aspects of this work.  

Thus, I'm overall left with the impression that it's quite difficult to assess the contribution of this approach, and determine precisely which of the different proposed aspects is really contributing most to the improved performance.  I know there is some ablative analysis in the paper comparing the pi_g-init and G_w-traversal independently and together, but I'm more questioning the basic question of what each portion of the network is really learning.

I'd be curious if the authors are able to clarify any of these points during their rebuttal.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel approach to hierarchical reinforcement learning approach by first learning a graph decomposition of the state space through a recurrent VAE and then use the learned graph to efficiently explore the environment. The algorithm is separated into 2 stages where in the first stage random walk and goal conditioned policy is used to explore the environment and simultaneous use a recurrent binary VAE to compress the trajectory. The inference network is given the observation and action and the reconstruction is to, given the hidden state or hidden state+observation, reconstruct the action taken. The approximate posterior takes on the form of a hard Kumaraswamy distribution which can differentiably approximate a binary variable; when the approximate posterior is 0, the decoder must reconstruct the action using the hidden state alone. The nodes of the world graph are roughly states that are used to reconstruct the trajectories in the environment. After the graph is constructed, the agent can use a combination of high-level policy and classical planning to solve tasks with sparse reward.

Personally, I quite like the idea of decomposing the world into important states -- it is closely related to the concept of empowerment [1] which the authors might want to take a further look into. I believe extracting meaningful abstraction from the environment will be a key component for general purpose RL agent. One concept I really like in the paper is using the reconstruction error as the reward for the RL agent, which has some flavors of adversarial representation learning. Further, I also really like the idea of doing structured exploration in the world graph and I believe doing so can help efficiently solve difficult tasks.

However, I cannot recommend accepting this paper in its current draft as there might be potential major technical flaw and I also have worries about the generality of the algorithm. My main concerns are the following:
    1. The ELBO given in the paper is wrong -- the KL divergence should be negative. I want to give the paper the benefit of doubts since this could be just a typo and some (very rare) researchers use a reverse convention; however, this sign is wrong everywhere in the paper including the appendix yet the KL between Kuma and beta distributions uses the regular convention. I tried to check the source code provided by the authors but the code only contains architecture but not the training objective, training loops or environments. As such, I have to assume that the ELBO was wrongfully implemented, unless the author can provide the full source code, or, if the ELBO is indeed incorrectly implemented, rerun the experiments with the correct implementation.

    2. The proposed method for learning the graph does not have to be a VAE at all. The appendix shows that the paper uses a 0.01 coefficient on the KL, which is an extremely small value for VAE (in fact, most VAE’s have beta larger than 1 for disentanglement). Thus, I suspect the KL term is not actually doing anything; instead, the main reason why the model worked might be due to the sparsity constraints L_0 and L_T. In other words, the model is simply behaving like a sequence autoencoder with some sort of hard attention mechanism on the hidden code, which might explain why the model still worked well even with the wrong ELBO. To clarify, I think this is a perfectly acceptable approach for learning the graph and it would still be very novel, but the manuscript should be revised accordingly to reflect this. If the (fixed) VAE is important, then this comparison (0 KL regularization) would be a nice ablation regardless.

    3. Algorithm 1 requires navigating the agent to the key points from \mathcal{V}_p. This assumption is quite strong. When the transition dynamic is deterministic and fully reversible like the ones considered in the paper, using the reverse of replay buffer can indeed take the agent back to s_p, but in settings where the transitions are stochastic or the transitions are non-linear or non-reversible, how should the algorithm be used?

    4. It is not clear how \mathcal{V}_p are maintained. If multiple new nodes are added every iteration, wouldn't there be more than necessary nodes in \mathcal{V}_p? It seems to me some pruning criteria were used unless the model converged within small number of iterations? Are the older ones are discarded in favor of newer ones?

    5. How are the actions sequences “normalized”?

    6. In what way are the Door-Key environment stochastic? It seems like the other environments also have randomness, so is the only difference the lava pool?

I believe the propose method is sound, so if the revision can address either 1 or 2, I am willing to raise my score to weakly accept. If the revision in addition addresses 3, 4, 5, 6 in a reasonable manner, I am willing to raise my score to accept.

=======================================================================
Minor comments that did not affect my decision:
    - I think mentioning the names of the environment in the abstract might be uninformative since the readers do not know what they are a priori.

Reference:
[1] Empowerment -- An Introduction, Salge et al. 2014

review_title: Official Blind Review #2

Meta-review: This paper introduces an approach for structured exploration based on graph-based representations.  While a number of the ideas in the paper are quite interesting and relevant to the ICLR community, the reviewers were generally in agreement about several concerns, which were discussed after the author response. These concerns include the ad-hoc nature of the approach, the limited technical novelty, and the difficulty of the experimental domains (and whether the approach could be applied to a more general class of challenging long-horizon problems such as those in prior works). Overall, the paper is not quite ready for publication at ICLR.
Decision: Reject