Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addressed a valuable problem: what should the agent learn in the absence of extrinsic rewards? Instead of sampling a single goal to be reached within the duration of the episode, The proposed method also samples a sequence of states from global goal coordinates. This paper is well organized and easy to follow. 
main_review: Strengthens:
Compared to prior methods, the globally consistent codes of the proposed method allow it to be far more exploratory. And the corresponding experimental results also demonstrated that the proposed method could promote the state coverage and increase unsupervised performance

Weakness:
The experimental result part is not sufficient. 

It seems that there are some errors in Fig.2 and Fig.3. Inserting problem? 

I am not an expert in this research field. Even my current score is 5, I will change my score in the rebuttal stage and refer to other reviews. I will check the response carefully and give my final rating in the rebuttal stage.
limitations_and_societal_impact: As the author mentioned, general purpose function approximators (e.g. neural networks) could be used to specify more general dynamics, but makes learning tractable remains an important open question.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a novel extension of VIC, which incorporates a linear latent dynamics model. This addresses the difficulty of training VIC on large code spaces by latent decomposition of goals into sequences of subgoals. Empirical results show that this leads to a novel intrinsic control algorithm that outperforms alternative VIC variants on challenging exploration problems in terms of state coverage, as well as exhibiting improvements to the interpretability of the learned latent space.
main_review: This paper clearly motivates and constructs EDDICT, making a compelling argument for the usefulness of imposing a latent dynamics model on the latents used for VIC, by showing such a formulation improves over standard VIC in state coverage. The results of the paper provide a useful contribution in demonstrating the benefits of learning relative latent goals in the framework of VIC. 

I believe the paper could be strengthened by providing more understanding of how EDDICT performs as an exploration method, as well as how it compares to VIC. Specifically: 
- The authors frame the problem of VIC as a form of empowerment-based exploration, however, in the process ignore one of the original motivations of VIC, namely as a means for unsupervised option discovery. I believe it is fair to focus on the exploration aspect of VIC, but if the aim is to present EDDICT as an exploration method, comparison to at least one or two standard exploration baselines (e.g. RND, RIDE) should be included in the experiments section.
- It would be useful to see how the state coverage performance compares to DIAYN, which claims to improve over VIC in terms of diversity of sampled goals.
- Further, as the focus is on EDDICT as an exploration method, it would be valuable to include experiments showing learning curves for EDDICT in combination with extrinsic rewards. 
- On the other hand, while not the focus of the paper's narrative around exploration, it could be interesting to understand how the latent, relative options learned by EDDICT can be used by a manager policy, similar to the HRL experiments performed in DIAYN. This would help us understand the relative merits of the relative goal representation in terms of discovering useful options, which was an original motivation of VIC, which the paper focuses on as the primary baseline.

Additional feedback 
- The paper claims to have solved the "noisy TV problem," however none of the environments in the experiments clearly feature a noisy TV.  Providing results on an environment with such a noisy TV (e.g. adding a noisy TV to the Point Mass environment) would provide more support for this claim.
- Presentation of results for "Code norm as difficulty" section could be improved. For example, showing a plot of code norm vs distance to the center frame of the starting room in Montezuma's Revenge. The current qualitative results are a bit hard to parse.
- In regards to the above point, what exactly is being shown in the right side of Figure 2? Is it the final state of several latent codes for each quartile, all overlaid on top of each other? If so, this should be clarified.
- Results in Table 1 are only over 3 seeds and the number of seeds is not detailed for the other experimental results. Error bars would allow for a more principled comparison of the methods.
- Related to the previous point on including exploration baselines, the conclusion claims "But in terms of pure exploration, EDDICT falls short of state of the art methods that learn a single policy", however, none of the results presented in the experiments section actually seem to directly support this claim, as the performance for these alternative SoTA exploration methods are not included in the reported results.
limitations_and_societal_impact: - The limitations of EDDICT with respect to standard exploration methods could be better showcased in the paper.
- The extent to which EDDICT's learned options can be used for HRL or transfer to downstream tasks should be compared to that of VIC (or variants like DAIYN) to better understand whether learning relative options induces an additional trade-off in terms of the usefulness of the learned options for downstream HRL or transfer to new environments.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a new pure exploration algorithm, EDDICT, in the thread of the methods that seek to maximize the mutual information between a latent code z and the states reached by a policy conditioned on z. Especially, it considers an additive dynamics on the latent codes as opposed to a fixed sampling distribution in prior works. The method is evaluated on a Point Mass domain and a set of Atari games, which includes Montezuma's Revenge.
main_review: Post-rebuttal

Having read other reviews and after further exchanges with the authors, I am now getting a clearer perspective on the paper and its motivations. The approach might be interesting, and the changes proposed by the authors might strengthen the submission. However, I think that a complete assessment of its value would require a re-evaluation of the updated manuscript. My evaluation on the work as it is remains borderline.

---

To my understanding, the crucial idea of this paper is to consider a temporal correlation between the latent codes that are sampled along a trajectory, so that to induce a deeper exploration of the environment w.r.t. previous methods that just sample the latent code randomly. However, the general goal of those prior works seem slightly different from the one pursued here: They seek to learn a set of diverse (i.e., discriminable) skills instead of temporally-correlated skills that are combined in a sequence.

Beside this point, two other factors are making me skeptical on the actual value of the paper:
1) I think the paper fails to convey strong motivations or intuitions behind the algorithmic choices, which are mostly presented as empirical tricks. Of the three ingredients incorporated in the VIC methodology, i.e., greedy optimization, hindsight correction, and latent dynamics, only the latter is clearly motivated in the text;
2) Even if some results are promising (such as the average return in Montezuma), I found the experimental analysis not sufficiently convincing for an essentially empirical work. On the one hand, the method is only compared against VIC, RVIC and a reduced version of EDDICT without the latent dynamics (which is quite competitive), whereas other methods in the same thread, e.g., DIAYN [14], DISCERN [46], VISR [20], VALOR [1], are not considered. On the other hand, it is not even clear if those methods are the most relevant comparison, as opposed to deep exploration algorithms, such as [10,11] that are mentioned in the text, or hierarchical reinforcement learning.

For the reasons described above, I am currently providing a negative evaluation to this work. I report below some additional comments.

PRESENTATION

Unfortunately, I found this paper not really easy to read. Several sentences are too convoluted, the terminology and notation are sometimes confusing. The text does not provide much intuition on what is going on with the method.

RELATED WORKS

The paper by (Campos et al., Explore discover learn: Unsupervised discovery of state-covering skills, 2020) also identified state-covering as the main limitation of mutual information methods. Even if they seem to address the problem from a fairly different perspective, I think it should be mentioned in the paper.

Another concurrent work (Liu & Abbeel, APS: Active pretraining with successor features, 2021) combines a mutual information method (e.g., VISR) with a specific state-covering incentive, which might also be a relevant comparison to EDDICT.

EXPERIMENTS RESULTS

The actual significance of the performance reported in Table 1 is unclear without providing confidence intervals around the average of the three runs.
limitations_and_societal_impact: Limitations and societal impact are discussed in the paper checklist and through the text.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The submission extends variational intrinsic control by adding a temporal dynamics to the prior distribution of latent skills. This allows using multiple skills in the same training trajectory more easily as the skill dynamics ensure that only the skills relevant to the corresponding part of the state space are used for training. Crucially, the temporal skill prior evolves by simply adding Gaussian noise with no learned parameters, which avoids the need to train this prior. Experiments demonstrate improved skill discovery on atari. To my knowledge, the method is novel.

main_review: The paper proposes an interesting and promising method, and appears to fix an important issue with VIC. Further, the method is adequately evaluated and achieves good performance, outperforming some prior work. However, the method appears to have a hidden assumption that the initial state distribution is narrow, as I detail below. This is an important issue since in many practical environments, or in those considered by Eysenbach’18, the initial state distribution is wide, and the method might need to be modified to account for that. Because this issue can be addressed in the rebuttal, I am not providing my final decision at this time.

The major issue with the proposed approach is the treatment of the initial skill $z_0$. It appears from the text below l114 that its prior distribution is unconditional p(z_0). This distribution does not seem to be specified in the paper. However, it seems to me that any unconditional distribution will be inadequate. Consider an initial state distribution $p(s_0)$ wide enough so that navigating between two states sampled from it is hard. In the extreme case, the initial state distribution can be the entire state space (as commonly done e.g. for some navigation tasks). Since the skills have global meaning and correspond to states, the agent will often observe an initial skill that is too hard to execute, which is exactly the issue that the current submission attempted to address! This is because the prior $p(z_0)$ distribution is too wide, and the reverse predictor distribution will also learn to match it. Perhaps the agent should be evaluated on environments with more diverse initial state distributions to confirm the agent is robust to this.

There are also a number of more minor issues with clarity
- What is O in Eq 2? On l133 Eq 2 is stated to be the agent’s objective. However, Eq 2 drops action entropy terms. Why are they dropped? l90 states “In practice, we optimize the above objective using a value-based reinforcement learning algorithm and -greedy policies (in lieu of a Boltzmann policy), and thus drop these terms altogether.” It is rather unclear to me what this means. Eq 1 is the objective for the policy, it does not change with the chosen exploration strategy. If the proposed method is not derived from control as inference, why is the background section in 3.1 necessary?
- There is no difference between Eq 2 and Eq 3. The $O^{(i)}$ term is expanded in one and compressed in the other. The text implies that Eq 3 does not consider long-term consequences, but it does, up to M-1.
- l142, “we thus condition on the code most likely to have yielded state si, under the reverse predictor.” This is not what Eq 4 says. In Eq 4, the code is sampled from the reverse predictor instead of taking the mode. Which one is actually used?
- l174 is confusing. Is the function $f$ trained through f(s_i+1), or both f(s_i+1) and f(s_i)? This sentence implies that both, but algorithm 1 says that only f(s_i+1). 
- The choice of notation is extremely confusing. Why is prior over actions $\pi^0$ but prior over latents $q$? Why is the forward distribution denoted as $p^z(\tau)$ instead of the conditional distributions $p(\tau|z)$ as used elsewhere in the paper? All 4 distributions in Eq (1) are denoted differently.

Finally, the experimental evaluation omits some important baselines such as DISCERN, which was also evaluated on Atari. Since the proposed method is motivated by learning hierarchical skills, it would also be instructive to provide a comparison on downstream hierarchical RL, as done in DIAYN and DADS.
limitations_and_societal_impact: Limitations and societal impact are addressed adequately.
review_title: null

Meta-review: To reliably reach a large number of different states in the environment requires AI agents to be able to manipulate the environment effectively. 
Practically, this type of exploration reward commonly aims to disambiguate between a set of latent codes using short horizons of future states, which limits the “exploration depth” of the agent.

Instead, this work proposes to generate conditional latent variables which effectively let the agent explore states that cannot be reached from a single unrolling.

The paper presents compelling motivation and theory for the method. However, as pointed out by multiple reviewers, the experimental results are not very compelling. In particular, the number of seeds evaluated is small and the EDDICT results are obtained by maximizing the performance over the size of the codebook, which clearly will bias these numbers. 


One important topic of discussion was the experimental validation of the method. The authors ran additional seeds for some of the experiments in the paper and found that EDDICT still significantly beats a baseline (RVIC) and an ablation (EDDICT-Delta). 

This line in the paper is concerning: “For EDDICT-based agents, we pick the best metric across code sizes.” 
This sounds like the  _best_ code size for EDDICT was reported for each metric. If so, that will both introduce bias and make the results incomparable to the baselines. 

As pointed out by the reviewers, the current results fail to provide convincing experimental evidence that EDDICT works better than the baselines. This issue is compounded by the potentially biased evaluation maximizing over different code-sizes. Given these concerns, I do not believe the paper is ready for acceptance at this point. 

Decision: Accept (Poster)