Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to address an important question in reinforcement learning: policy learning from high-dimensional sensory observations. The authors propose an algorithm for Learning Controllable Embedding (LCE) based on policy iteration in the latent space. The authors provide a theorem to show how the policy performance in latent-space policy improvement depends on the learned representation and develop three algorithmic variations that attempt to maximize the theoretical lower bounds. In the experiments, the proposed algorithm CARL shows improved performance when compared with other LCE baseline algorithms. 

While I'm not particularly familiar with the field of LCE, I think the idea of learning a representation that is suitable for policy improvement is an interesting idea. The readability of this paper is also pretty good, which can be difficult to get right because the of the correspondence between the original space and the latent space. Overall the paper is easy to follow. 

While I do think Algorithm 1 is reasonable, I found its theoretical foundation, namely Theorem 1, is incorrect. In the proof of Theorem 7 on p15 in the appendix, I do not think the implication T^2 VE(x) < T VE(x) + \gamma Delta(x) for all x, would hold. Because Bellman operator contracts in the L-inf norm, a basic inequality would rather take a form of  T^2 VE(x) < T VE(x) + \gamma sup_y Delta(y). In addition to this, another minor error happens in the first equation on pg 16, where I believe the correct right hand side would be 1/(1-gamma) sup_y Delta(y), without the gamma dependency.

However, a bound that depends on L-inf norm would be quite bad for Theorem 1, and current data collection process in Alg 1 is not sufficient for minimizing it. I think it might be possible not using an L-inf bound but using an expected error based on the policy's rollout distribution. However, this change would largely change the theoretical results, and perhaps the motivation or details of the algorithm design. Therefore, I do not think the paper is ready for acceptance at the current stage without a large revision. If the authors can address this question properly, I would raise my score.

Beyond the flaw in the theory, there are some parts which can benefit from some clarification: 
1. In the offline CARL, how does the algorithm address the issue of out of distribution error due to using a batch dataset? 
2. The authors argue that the loss here is different from PCC many times in the paper, but they never explain whether the choice here is better (or in which way). 
3. In line 4 of Alg 1, how do we ensure such pi would exist?
4. What is the definition of "compatible reward function" in the last paragraph on p4?
5. For completeness of presentation, please include the definition of curvature loss.






review_title: The algorithm is promising but the theoretical foundation is inaccurate

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new representation learning + RL algorithm called CARL, with a specific objective for learning a latent representation and dynamics model coupled with SAC policy learning in the latent space. Experiments on a few domains show CARL outperforming previous algorithms such as DREAMER and PCC.

Pros:

+ The key points of the paper are relatively well organized and motivated properly.
+ The experimental results succinctly demonstrate the promise of the proposed approach.

Cons:

- It is difficult to follow important details about the operation of this relatively complicated method.
- The experimental results are not sufficient for this largely empirical work.

With these pros and cons in mind, I am recommending a weak reject. See below for additional detailed comments.

EDIT: After discussion, I have increased my score and am recommending weak accept. See the discussion with the authors for details.


Quality
---

The paper studies an important problem, proposes a novel solution, and has promising experimental results. However, the main drawback in terms of the quality of the work is that the results are not complete enough. For work that is empirically driven, I do not view the current results as sufficient for publication.

In particular, DREAMER appears to be competitive with CARL on a few domains. But DREAMER was also evaluated much more broadly across many tasks from the DeepMind control suite, indicating a level of robustness and performance that is, at best, hinted at in this work for CARL. A wider suite of experiments, for example using the same tasks as DREAMER, would go a long way in better shaping the reader's understanding of the proposed method.

Clarity
---

As mentioned, the main points of the paper are presented well. The problem is properly motivated, and a central theorem gives rise to the proposed representation learning method. I did not check the proof for this theorem, but it appears sensible.

However, the finer points in the paper, which are also very important, are difficult to follow. For example, what is "model-based SAC"? There does not appear to be a proper explanation or citation for this. Is the learned dynamics model F used in some way to learn the Q-function? Is this novel, or is it from prior work?

Considering the proposition that replacing other control algorithms in the latent space with model-based SAC is important for the overall performance improvement, a description of model-based SAC is important. Furthermore, an ablation study would be helpful in terms of understanding the relative importance of this change vs the proposed representation learning approach, which seems to be the novel part.

Some other minor concerns about the methodological sections: there are many hyperparameters and not much guidance as to how to pick these; more discussion of why there are different versions of CARL and what are their respective strengths and weaknesses would be useful, especially for VCARL; I personally found the last paragraph of the VCARL description almost impossible to follow.

Originality
---

To the best of my knowledge, the representation learning algorithm itself is novel. Perhaps a related work that is overlooked is https://arxiv.org/abs/1907.00953, which apparently has been accepted to NeurIPS 2020 but has been out for some time. At a high level, this work also incorporates representation learning into SAC, though the underlying details are different. Still, this approach seems actually more closely related than some of the current citations and comparisons, e.g., SOLAR and DREAMER. At least a citation seems to be in order, and preferably a comparison. Indeed, this prior work also carries out a more comprehensive evaluation on more tasks than the current work.

Significance
---

This work has the potential to be significant, as many researchers and practitioners are currently interested in how to make deep RL more efficient and performative, in particular in visual settings. However, without a more comprehensive evaluation, it is difficult to judge for sure.
review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper examines the problem of learning controllable embedding (LCE), with the goal of learning good representations (usually achieved using variational inference algorithms) such that the maximum cumulative reward can be achieved. The main difference lies in the simultaneous learning of both the low-dimensional latent space as well as the action policy.

One of the main strengths of this paper is found in Theorem 1. The authors devise a simple policy iteration approach in the low dimensional learned space. Then, using mostly qualitative analysis, a bound on the policy improvement error is formulated. This error combines several intuitive and straightforward factors, which are then extracted to form more involved offline and online reinforcement learning algorithms. I have read through the proofs, and they seems correct.

While I appreciate the quality of the theoretical work, the paper had some drawbacks that brought me to my current score :
1. The loss function consists of many hyper-parameters. The authors should provide some guidelines for choosing these hyper-parameters due to the large number of possible combinations, and clearly state how the scalings affect performance.
2. Experimentation is lacking. While the authors conducted experiments mostly on toy problems, I expect them to compare against more involved environments which are harder to model. Their comparison with state of the art algorithms (e.g. Dreamer) which were also tested on such environments is thus not fair.
3. Minor comment: There is a newer version of Dreamer that the authors can compared against: https://arxiv.org/pdf/2010.02193.pdf 
4. Minor comment: How would CARL compare against model-free offline RL methods, or generally to algorithms that are not SAC?

Question to authors:
Would there be a benefit in removing F altogether and learning a mapping X -> Z -> X’ without transitioning in the latent space? (i.e., errors III and IV in Theorem 1)

Finally, it would be beneficial if the authors could include code for their work. If the authors can't supply the complete code base, even code snippets with clarifying explanations to demonstrate their main ideas would be beneficial. This would greatly improve the quality and credibility of their work as well as the reviews.

To conclude, the paper provides strong theoretical intuition, which is a significant value-add of the paper. Nevertheless, its lack of experimentation and large number of hyper-parameters limit its overall quality. If the authors provide substantial improvement in the experimentation I will increase my score.

review_title: Good theoretical intuition but lacking experimentation

Meta-review: This paper addresses the question of RL in high-dimensional spaces by learning lower-dimensional representations for control purposes. The work contains both theoretical and empirical results that shows the promise of the proposed approach.

While the reviewers had initial concerns, including with a problem in a proof and questions around the contributions, after robust responses and discussions this paper is now in good shape.
Decision: Accept (Poster)