Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes an new algorithm to simultaneously estimate and maximise empowerment for achieving unsupervised stabilization. The method relies on the formulation of a dynamic system as a linear Gaussian channel. In this formulation, empowerment can be efficiently estimated by solving a line search problem. The authors propose to learn the channel matrix G(s) from samples and exploit then linear formulation to learn a corresponding policy maximising empowerment. These two steps are done in an alternating fashion until convergence. In the experiments the method is compared to previous approaches for unsupervised control via empowerment. The authors show convergence close to the true empowerment landscape and prove the sample efficiency and low variance of their method. Additionally, their method is capable of unsupervised control based on image observations.

Overall, I am torn between accepting and rejecting the paper and I lean towards rejection. I think it represents a good step  toward intrinsically motivated agents which can learn useful behaviour solely based on interaction with their environment, however, my major concern is about the clarity and the form of the paper. I think the description of the method is missing important details (see cons below) and I would like to see additional evaluations to support the claims of the paper. Overall I would recommend to polish especially the method and the experimental section before accepting it for publication.

Pros:
1. The method is well placed in the existing literature on empowerment and provides a thorough comparisons of past approaches.

2. The algorithm is novel and at the same time simple and very easy to understand. It relies on previous work for empowerment of linear Gaussian channels but frames the problem in a data-driven setting by learning the system dynamics from raw data.

3. The paper provides good experimental results which compare the method to existing approaches and show advantages when it comes to sample efficiency and convergence to the true empowerment landscape for the pendulum. Additionally, the method allows empowerment-based control from image observations for the first time.

Cons:
1. I miss important details in the paper:
    
    a) How is the parametrisation of G(s) with a neural network being done? Especially in the case of image based observations I would like more details.

    b) Connected to (a), please make it clear how the sequence of actions a_t^{H-1} is handled.

    c) You present an alternative to the SVD, that requires constraints. How do they enter the cost function or do you perform constraint optimization?

2. I think the claims of the paper are a bit bold. The method requires that the system dynamics can be formulated as a linear Gaussian channel, which requires a control affine system. Nonetheless the paper claims that the approach can handle arbitrary dynamics. Can you clarify how the approach would tackle non-affine systems (dx = f(x, u)). There is also no support for this claim in the experiments. Specifically, except for the ball-in-box, all experiments are on pendulum-like systems.

3. The subfigures 3g and 3h only show that the std of average empowerment is very low for the proposed method. I am missing an evaluation of the reward (empowerment) over time itself.

review_title: I lean towards rejection due to the clarity and form of the paper, however I welcome the authors to polish the paper during the rebuttal period.

final_score: rating: 6: Marginally above acceptance threshold
review_content: First of all, the paper is well organized, provides a very clear description of the method. 

Reproducing results
Will code be available? It is not mentioned that code is made available.

Correctness
Authors mention that training with empowerment converges to an unstable equilibrium (3rd paragraph in Introduction). This is not always the case but depends on the system. 
The agent does not maximize the MI, but maximizes the potential MI, as Empowerment is not MI but channel capacity (4th paragraph Introduction).

Novelty/Relevance
The relevance is that they use a representation for the dynamical system (i.e. Gaussian channel), that should improve the quality of the Empowerment estimation. 
They mention that they for the first time estimate empowerment from images. Karl et al. 2017 also use a DVBF, which also allows a transition in latent space from images observed. 

Unclear
They estimate empowerment for a state s_t, which now is used to train a policy. However, normally the reward is a value for s_t, a_t pairs, so essentially a function of s_t+1. Should empowerment also be computed for s_t+1?

Small notes
Maximilian et al. should be Karl et al

I am uncertain whether the Gaussian channel is a contribution to the research community. Is it only useful for research done on empowerment? Can it be used for other systems that use a VLB?


review_title: Great work, unsure about the impact and relevance

final_score: rating: 7: Good paper, accept
review_content: EDIT: the authors have removed the claims I was concerned by, as well as adding a requested baseline. The new quantitative results on stabilization also make their method much more compelling. The new hopper results are a bit noisy (would it stay stabilized if training continued?), but are still quite promising in that they show an ability to handle more complex dynamics.
4-->7, thanks for the great revision effort!

This work tackles the generally intractable problem of calculating channel capacity by learning a dynamics model with a highly constrained latent space (linear with Gaussian noise) that permits an efficient solution. This allows for using the exact empowerment (of the approximate dynamics) as the reward function for a reinforcement learning agent. This method is demonstrated on a few simple control problems, with the most difficult between a pendulum from pixel observations, which results in unsupervised stabilization.

The fundamental idea is sound; generalizing a special-case algorithm by forcing its constraints to hold in latent space has a long line of success (e.g. "Embed to Control"). And the model-based approach used here might allow for completely off-policy learning of empowerment. DADS is also model-based and could be adapted to optimize empowerment, but since its not a true lower-bound the case in favor of your approach over variational methods would be quite strong.

But the experiment baselines are not what they should be. Neither DADS nor DIAYN are measuring the same quantity as this method, namely empowerment: I(A^k; s_{t+k} | s_t). DADS is (almost) a lower bound on I(z ; s' | s) and DIAYN lower bounds I(z; s)+H(a|s, z). If there were some performance metric they all aim to achieve then this might be fine, but the only comparisons are to the quality of empowerment estimation (which they aren't estimating). I'd suggest switching to VIM[1] and VIC which lower bound I(A^k; s_{t+k} | s_t) and I(z; s_{t+k} | s_t) respectively.

A larger problem that is more easily remedied is the use of overstated claims:

"our method has a lower sample complexity" -- this is not shown. Yes, the empowerment of the modelled channel can be computed without additional interaction, but is building the model more sample efficient than getting a model-free estimate using a variational lower-bound? Maybe, but this isn't demonstrated.  

"allows, for the first time, to estimate empowerment from images" -- this can only be true in the most contrived sense. While it's true that DIAYN and DADS both used explicit state, only DADS really exploits that decision architecturally. Indeed [2] includes a pixel-based version of DIAYN as a baseline without comment, and while that version of DIAYN doesn't perform that well, the method introduced there (VISR) tackles full Atari games, which constitute a much harder perceptual challenge than a pendulum on a blank background. Now, you could argue that none of these methods actually lower-bound empowerment (indeed, I just did in the previous paragraph). But adapting them lower-bound empowerment requires trivial changes e.g. for VISR you'd just have to evaluate the reverse predictor at the end of a trajectory rather than at every state. But one can look back far earlier. VIM [1] calculated empowerment from pixels in 2015. You could argue the pendulum environment is more complicated than the toy navigation tasks they used, but its hard to argue that it wouldn't work for pendulum since you didn't try it out.

I am not willing to see this work published in its current state, but I promise I'll support publication if 1) VIM is added as a baseline for all (including pendulum from pixels) experiments 2) the overstated claim are dropped.

[1] Mohamed, S., & Rezende, D. J. (2015). Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems (pp. 2125-2133).
[2] Hansen, S., Dabney, W., Barreto, A., Van de Wiele, T., Warde-Farley, D., & Mnih, V. (2019). Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:1906.05030.
review_title: Interesting work in light of the revisions.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary

The paper studies reward-free reinforcement learning (RL) methods based on empowerment. The authors propose a technique for empowerment estimation under the assumption that a state after H timesteps can be factorized as a product of the current action and a matrix G(s) that depends on the current state. In contrast to the existing methods that rely on the optimization of variational lower bounds on mutual information for empowerment estimation, the technique allows having an almost closed-form expression for empowerment. The authors qualitatively demonstrate the convergence of their method to the true empowerment function on simple environments such as 2D ball-in-box as well on image-based Pendulum environment.

Strengths
- Under the proposed factorization, the empowerment estimation requires only calculating SVD of the G(s) matrix and line search for satisfying constraints.
- The plots for 2D ball-in-box environment demonstrate that the proposed method estimates empowerment more accurately than other algorithms that have empowerment estimation as a component.

Weaknesses
- The experimental results could be demonstrated on harder environments such as MuJoCo. For example, DIAYN [1], one of the methods the authors compare with, provides results on Ant and Cheetah environments.
- The proposed factorization, which is a key component for empowerment estimation, might be too restrictive and not scalable. Moreover, the reviewer did not find the details on how exactly G(s) is calculated except that it is the output of a neural network.
- The positioning of the paper is unclear. If the main contribution is the improvement over existing reward-free RL methods, then it is more appropriate to demonstrate that the agent that uses estimated empowerment achieves high returns in the environment or has an interesting emergent behavior. If the main contribution is the improvement in empowerment estimation, it is more appropriate to compare with methods designed for mutual information estimation.
- The overall clarity of the paper could be significantly improved.

Recommendation

The reviewer votes for rejection. The methods the authors compare with are not designed for empowerment estimation, they use empowerment only as a proxy reward e.g. for overcoming exploration or learning skills. Moreover, it is unclear whether the method will scale beyond simple environments. Addressing the outlined weaknesses might increase the final score.

**

Post-rebuttal update: the score is increased from 3 to 5. See the response to authors' comments.

**

Notes:
1. The connection between equations (1) and (3) which both define empowerment is unclear. 
2. It would be helpful to provide more experimental details. For example, the authors state that their method requires only training only two neural networks compared to three networks for variational lower bound methods. However, the benefit of this is unclear without providing the architectures of the neural networks and the total number of parameters.
3. The code and additional materials were not available in the Google Drive folder indicated in Supplementary Material at the moment of submitting the review. However, this did not influence the final score.

[1] Eysenbach, Benjamin, Abhishek Gupta, Julian Ibarz, and Sergey Levine. "Diversity is all you need: Learning skills without a reward function." arXiv preprint arXiv:1802.06070 (2018).

review_title: Almost closed form empowerment estimation, but the assumptions are unrealistic

Meta-review: The paper advocates empowerment for stabilising dynamical systems, the dynamics of which is estimated with Gaussian channels. Baseline comparisons have improved and that makes the experimental section good. While initial versions of the paper were problematic, all reviewer issues have been addressed and acceptance is almost unanimous.
Decision: Accept (Poster)