Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Summary:
This paper proposes an algorithm to combine the ideas of unsupervised skill/option discovery (Eysenbach et. al., 2018, Gregor et. al., 2016, referred to as “BMI” in the paper) with successor features “SFs” (Barreto et. al., 2017, 2018). While unsupervised skill/option discovery algorithms employ mutual information maximization of visited states and the latent variables corresponding to options (typically discrete), this paper adds a restriction that this latent variable (now continuous) should be the task vector specified by some learnt successor features.

With such a restriction, the algorithm can now be used in an unsupervised pre-training stage to learn conditional policies corresponding to several different task vectors and can be used to directly infer (without training or fine-tuning) a good policy for a supervised phase where external reward is present (i.e. via GPI from Barreto et. al., 2018) by simply regressing to the best task vector.

Such unsupervised pre-training is shown to outperform DIAYN (Eysenbach et. al., 2018) in 3 different Atari suites (including the full 57 game suite) and also ablations to the proposed model where GPI and SFs are excluded individually.

Decision:
I vote for accept as this paper proposes a novel technique to combine mutual information based intrinsic control objectives with successor features, which allow for combining the benefits of both in a complementary way. An unsupervised phase can now discover good conditional policies with successor features which can be used to infer a good policy to solve an external reward task in a supervised phase, with such a policy capable of attaining human level performance in several Atari games and outperforming several baselines such as DQNs in limited data regimes.

Other comments:
- The technique for enforcing the restriction in Eq. 10, as well as being able to use it with generalized policy improvement is a good novel contribution in the paper.

- The detailed comparison with baselines on the full Atari suite is sufficient to back the claims in the paper that the strengths of BMI and SFs do complement each other.

- The fact that fast task inference is sufficient to get good performance is impressive i.e. without the need to fine-tune the best inferred policy.


Minor typos:
- In section 5 para 5, “UFVA” -> “UVFA”, “UFSA” -> “USFA”.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors address the problem of finding optimal policies in reinforcement learning problems after an initial unsupervised phase in which the agent can interact with the environment without receiving rewards.  After this initial phase, the agent can again interact with the environment while having access to the reward function. To address this specific setting, the authors propose to use the successor feature representation of policies and combine it with methods that estimate policies in the unsupervised setting (without a reward function) by maximizing the mutual information of a policy-conditioning variable and the agent behaviour. The result is a method called Variational Intrinsic Successor Features (VISF) which obtains significant performance gains on the full Atari task suite in a setting of few-step RL with unsupervised pre-training. The main contribution seems to parameterize the successor features in terms of a variable specifying the policy. This variable will be the same as the linear weights in the linear model for the reward assume by the successor features representation. Finally, a discriminator aims to predict the linear weights of the policy from the observed state-feature representation.

Clarity:

I think this is one of the weaknesses of the paper. The writing and clarity could be significantly improved. How do you go from equation 8 to equation 9? How does q lower boun p? In the paragraph after equation 9 the authors mention the score function estimator. But very few details are given. After reading the paper, my impression is that the reproducibility of the results could be very hard, because of the lack of details of the specific implementation. The authors also do not mention that the code will be publicly available after acceptance.

I feel that, to better understand the method, the authors should include experiments in simple and easy to understand synthetic environments which can be more illustrative than ATARI.

What is the difference between the policy parameters theta and the conditioning variable z?

Novelty:

The proposed approach is novel up to my knowledge. I find the idea of parameterizing the successor features in terms of the policy parameters very innovative. 

Quality:

The proposed approach seems well justified and the experiments performed indicate that the method can be useful in practice. However, I think it would be very useful to have results on other environments besides ATARI. For example,  DIAYN contains experiments on a wide range of tasks, including gridworld style tabular experiments to illustrate what their method does. This work would benefit from similar simpler and easier to understand synthetic environments (unlike ATARI).

Significance:

The proposed contribution seems significant as illustrated by the experimental results and the novel methodological contributions. However, the lack of clarity and the difficulty in the reproduction of the results limit this.

Some minor comments:

I recommend the authors to remove references in the abstract.

Update after the authors' rebuttal:

After looking at the response from the authors, I believe that they have successfully addressed my concerns. 
Therefore, I have decided to update my rating and vote for acceptance. I am looking forward to seeing the python notebook with the implementation of the VISR algorithm.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper builds upon the idea of Successor Features; this is a principle used to facilitate generalization beyond the finite set of behaviors being explicitly learned by an MDP. The proposed paper ameliorates the need of defining the reward function as linear in some grounded feature space by resorting to variational autoencoder arguments. The derivations are correct, the motivation adequate, the experiments diverse and convincing. The literature review us up to date and the comparisons proper. This is a valuable contribution to the field. 

review_title: Official Blind Review #2

Meta-review: This work uses a variational autoencoder-based approach to combine the benefits of recent methods that learn policies with behavioral diversity with the advantages of successor representations, addressing the generalization and slow inference problems of competing methods such as DIAYN.  After discussion of the author rebuttal, the reviewers all agreed on the significant contribution of the paper and that concerns about clarity were sufficiently addressed.  Thus, I recommend this paper for acceptance.
Decision: Accept (Talk)