Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel approach to learning parameterized policies. The way I understand the work: it jointly optimizes a policy objective, a probabilistic dynamic model objective, and a compression objective. A bottleneck encoding distribution is key here, which is subject to both model predictions (given previous action and encoding) and compression (given current observation). In spirit, it is similar to variational information bottleneck (VIB) but can be seen as an extension of that to RL. So, the interpretation of this optimization is that the agent is aiming for encoding distribution that is maximally predictive of the next observation and maximally compressive of the observation while the agent is maximizing the future aggregate reward. The idea is neat. The optimization can also be easily put together. Some theoretical and empirical results are given.

main_review: I am generally accepting of the work. I would appreciate discussing the following issues.

The authors admit that being compressive will necessarily lead to lower rewards in complex situations. But then it felt like you took it as a goal because it by itself is still a good thing to achieve. Later you show that they have better generalization power. But can you motivate a bit more why aiming for compressed policies is a good goal? You gave some specific examples where compressed policies will naturally be safer in some cases. But if we are trying to build a general learning agent, where the agent needs to learn many things and address more and more complex situations over time, should we aim for compressed policies?

In equation 3, we have phi(z|s) and m(z’|z,a) in pairs. However, in the rest of the paper, phi(z’|s’) and m(z’|z,a) come in pairs. Seems like there is a mismatch. Can you explain the mismatch?

Clarification questions:
A large part of the paper initially talks about learning a “simple policy,” which is defined as a simple input-output relationship. It will be good if you specify directly that by simple you mean compressive.

You claimed that “Q value needs at least as many bits as a representation sufficient for predicting the action.” This argument may make sense for value-based methods such as Q-learning where the policy is derived from learned values. But in your case, the policy is parameterized directly and not through the values function. Can you clarify why this argument still makes sense for parameterized policies?

When you compare with VIB variants, did you make sure that they use the same amount of computation and memory (e.g. network size) as RPC? If not, how did you make sure that this comparison is fair?

In Figure 5, the x-axis of the plots has episodes. Are they of the same length? If not, how is it a sensible choice to align in terms of episodes?

And if those tasks have constant episode length, then it is a fixed horizon problem. Did you pass the remaining time steps as part of the observation? Otherwise, is the task from the agent’s perspective really well defined that we can describe coherently?

Why action repeat is a reasonable or interesting baseline?

Few edit suggestions:
Line 32: “minimizing bits” -> “minimizing the number of bits”

Line 157: both Q and y are functions of psi. This means both will be differentiated. Is that what you did because that’s not what temporal difference learning does.

limitations_and_societal_impact: N/AA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a new reinforcement learning method, called robust predictable control, that optimizes a policy under limited representational resources. Similar to previous methods, they formalize the problem in a fully observable MDP and learn a compressed representation of the state space using a variational information bottleneck constraint. Crucially, they apply the bottleneck on sequences of states. Through a clever factorization of the variational prior, they obtain a prior dynamics model in the compressed state space. This dynamics model is predictive and the agent only needs additional representational resources if the encoded states are not predicted by the prior dynamics already. Using a set of standard RL benchmarks, the authors show that this set up improves compression and enforces predictable and robust behaviors .

main_review: All agents face a fundamental problem: how much effort should they invest in order to achieve a certain outcome? In RL, the quality of an outcome can be described by the collected rewards, whereas the effort can be described by the informational expenses. The authors tackle this problem by considering an important informational expense, namely the complexity of the representation on which an agent bases its decisions. Low complexity representations are desirable because they are cheap (e.g. in terms of storage) but also because they tend to avoid overfitting. If agents don’t overfit they are more robust and can generalize better. The authors thus address a fundamental issue of decision-making and show that their method actually achieves improvements in compression, robustness, and generalization.

While previous work has addressed the same problem in a similar setups (e.g. ref 25), the authors here propose a valuable extension. Instead of compressing states one-by-one, they compress state sequences as a whole and thus get a step closer to the actual problem faced by the agent. Furthermore, the predictive prior this approach yields is useful on its own as something akin to a default dynamics model and the authors showcase one application of this in hierarchical RL (fig. 5). Most importantly, their approach also enforces the agent to learn behaviors that are inherently predictive. I think this is a conceptually important point and they nicely show evidence for this in fig. 3. 

The only issues I have are with clarity of exposition:

- Eq. 2: It is unclear whether there is really equality here. In my understanding every sequence, or partially truncated sequence, has a certain mutual information associated. When taking this MI as a cost, we can sum them up similar to rewards and take into account the discount factor in the sense of a probability of a sequence continuing. When doing this we get the RHS, but this is not equal to the LHS. Furthermore, the prior that is factored autoregressively should read $m(z_{1:\inty}|a_{1:\infty})$, i.e. $m(z_{1:\inty}|a_{1:\infty}) = \prod_t m(z_{t+1}|z_t,a_t)$, or otherwise one would have to sum over probability of actions.

- Eq. 5: It is unclear where this objective is coming from and how it is optimized (I also couldn’t find the code on the provided website to check). The Q function is learned online by TD learning minimizing $L(\psi)$. Since $L(\theta)$ is not considering whole sequences, is it really a function depending on $s_{t-1}$, i.e. $L(\theta,s_{t-1})$? And then this function is optimized for all $s_{t-1}$? It is unclear to me how this optimization procedure works, and how does it relate to a policy gradient approach as in ref. 25 or as in actor critic methods?

- Some sloppy notation or typos. E.g. on bottom of page 4, the high-level policy is denoted $\pi(a|s)$, but shouldn’t it be $\pi(a|z)$ as in fig. 1? Similarly, in the variables over which the expectation is taken in eq. 5 it reads $z_{t-1} \sim \phi(z_{t-1})$ instead of $z_{t-1} \sim \phi(s_{t-1})$. On page 3 the prior is suddenly called $r$. In eq. 4 the regularized reward function should depend on $z_t, z_{t-1},a_{t-1}$ as well. These are just some I found.

- It would be interesting to see some visualizations of the representations and dynamics that RPC finds. E.g. for the hierarchical tasks, $z$-space is three dimensional and could be visualized I assume. Visualizations would give a better intuition for how RPC exactly works.

limitations_and_societal_impact: Technical limitations are pointed out briefly. Although exactly how much performance in terms of return decays with compression is never really shown, but this might be beyond the scope of the publication. I could not find any discussion of societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes the method "robust predictable control (RPC)". It proposes to use a information
bottleneck between inputs and actions. However, in contrast to previous works, it does so not on a
per-observation basis, but on a per-sequence basis using a latent model. Similar to some of the
prior work, the encoding cost is not added to the loss, but subtracted from the reward.

main_review: Overall, I think this is an interesting and valuable paper which is well written and easy to read. 
While the idea itself does not seem like
a too big departure from previous work (although it is novel to my knowledge), the paper provides a
lot of analysis (both experimental and theoretical) as well as an intuitive explanation of how the
newly proposed reward affects the behavior of the agent - which I believe is a valuable contribution. 

While I'm in favor of acceptance, I'm hesitant to endorse it stronger at the moment as it does not
provide clear information about the absolute achievable return for some of the experiments (at least
as far as I can tell).  And while it is expected that the provided information-regularisation
decreases the training performance, the extend to which this happens is important to judge the
validity of the proposed method. In particular, it would be important to know whether the shown
advantages of RPC hold even with (somewhat) competitive performance on the task or if they are only
realised in the very low information & return regime.
To some extend the information is provided (e.g. Fig 6/8) but those plots are cut off and don't show
the performance of the unperturbed agent and also seem to use some different bitrates, all on the
very low end of the spectrum. In short: It would be helpful to have a overview table how RPC
compares to SAC for different bitrates on all used environments.

On a positive note, I think section 5 is great. I did have a look at the proofs and
they seem reasonable, although it's possible I overlooked something.

Minor comments:
* Fig 3b-top) Why is the variance of the blue line so much higher?
* Appenidx line 695: This should be a `- log q(...)` not `- log p(...)`?
* Eq. 10: Should be `phi(z_{t+1} | s_{t+1})` ?


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper "Robust Predictable Control" describes a method to learn policies with an information bottleneck between environment observations and a latent representation which is used for decision-making.
The bottleneck is posed between *sequences* of observation and latent states, in contrast to prior work, where a bottleneck is posed between individual states and latents independently.
A crucial component of the proposed method is an action-conditioned dynamics model in the latent space.
The proposed training objective is jointly minimized by training an encoder, latent dynamics model, and policy. 
First, this encourages the encoder and dynamics model to yield good predictions of encoded environment observations. 
Furthermore, by training the policy, the expected task return is maximized while ensuring that the behaviour of the agent is well-predictable. 
The latent dynamics model formulation can be exploited to derive theoretical guarantees between open-loop and information-constrained closed-loop control. 
Several experiments show that the information-constrained agent performs well in the nominal case but also on distorted environments and offers smooth transitions between different behaviours depending on the tightness of the information constraint.
main_review: **Originality**
 * The paper clearly states the difference to prior work and adequately cites it.
 
 * In my opinion, the paper has close connection to Rate-Distortion-Theory, or,
   to be more precise, "bounded rational decision-making". 
   Especially the experiment in Figure 3a shows this behaviour quite nicely. 
   I think the paper would benefit from establishing the connection and referring to this domain of research.
  

**Quality**

 * I perceive the paper as of very high quality.
 It is a consistent paper containing a sound theoretical motivation of the
 followed approach, interesting and well-executed theoretical derivations
 and insightful experiments. 
 Prior work is adequately cited and compared to in the experimental evaluation.


**Clarity**

 * In general, the paper is very clearly written and good to follow. I have the following comments:
   * The term "states" and "observations" seem to be used interchangeably. 
     I would appreciate a clarification of that fact at the beginning of section 3.1, eventually
     including the statement that you assume the environment state to be fully observable, if this is the case.
   * Related to the first point, in l. 95 you define the policy to be conditioned on 
     a *sequence* of past states. If $s_t$ are *states* of an MDP (and not observations of a POMDP)
     then conditioning on the current state should be sufficient.
   * The joint distribution $m_{1:\infty}(z_{1:\infty})$ as defined in l. 127 misses a factor to be a proper joint distribution
     in my opinion, e.g. $m_{1:\infty}(z_{1:\infty}) = p(z_1) \prod_{t \geq 1} m(z_{t+1} | z_t, a_t)$
   * The definition of the probabilistic bound in l. 242 is not aligned with the introducing text. 
     Due to the absolute difference it is rather the "probability that the policy's expected
     return on an observed (deterministic) MDP is much *different* to the policy's expected return ..."
   * Appendix, l. 694/695: Introducing $p$ and $q$ here as done in l. 710 would definitely help
     to understand the following derivations. Furthermore, I think that l. 695 contains a typo and it should
     be $\log p - \log q$ in the expectation.
   * Appendix, l. 698-699: I think the statement should be slightly changed to clarify that
     "expectations over a mixture of finite-horizon episodes are equivalent 
     to expectations over a discounted infinite-length episode"

  * Minor comments:
    * l. 133: Typo, you probably mean "prior $m$"?
    * l. 166: Typo, $z_{t-1} \sim \Phi(s_{t-1})$
    * eq. 4 / l.139: time-index in prior $m$ not consistent
    * l. 241: Introduce the "observed MDP" as $\hat{\mathcal{M}}$


**Significance**
In my opinion, the paper is of very high significance for the RL community as it contains insightful theoretical and experimental results on applying ideas of compression / information-regularization to reinforcement learning. 
It establishes interesting connections between open-loop and closed-loop control and model-free and model-based RL.


**Conclusion**:
To me, this paper is a clear accept. It is well-written and contains a well-motivated, novel approach with interesting theoretical and experimental results for the reinforcement learning community.
limitations_and_societal_impact: * The authors list as limitation that information-regularized policies receive lower reward
on the training tasks. I do not see this as a limitation per se, but just as an inevitable
consequence of rate-distortion theory (see above).
* One not yet mentioned limitation might be that the presented approach requires full
observability of the environments, if this is the case.
* Societal impacts are not explicitly addressed. In my opinion, this work does not exhibit extraordinary societal impacts  in addition to those induced by basic reasearch on reinforcement learning methods in general.
review_title: null

Meta-review: All 4 reviewers suggested acceptance of the paper and the authors clarified all open questions and concerns in their rebuttal. Therefore I am recommending acceptance of the paper.
Decision: Accept (Spotlight)