Robust Predictable Control
Authors: Benjamin Eysenbach
Ruslan Salakhutdinov
Sergey Levine
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: reinforcement learning
information bottleneck

Abstract: Many of the challenges facing today's reinforcement learning (RL) algorithms, such as robustness, generalization, transfer, and computational efficiency are closely related to compression.  Prior work has convincingly argued why minimizing information is useful in the supervised learning setting, but standard RL algorithms lack an explicit mechanism for compression. The RL setting is unique because (1) its sequential nature allows an agent to use past information to avoid looking at future observations and (2) the agent can optimize its behavior to prefer states where decision making requires few bits. We take advantage of these properties to propose a method (RPC) for learning simple policies. This method brings together ideas from information bottlenecks, model-based RL, and bits-back coding into a simple and theoretically-justified algorithm. Our method jointly optimizes a latent-space model and policy to be self-consistent, such that the policy avoids states where the model is inaccurate. We demonstrate that our method achieves much tighter compression than prior methods, achieving up to 5$\times$ higher reward than a standard information bottleneck when constrained to use just 0.3 bits per observation. We also demonstrate that our method learns policies that are more robust and generalize better to new tasks.

Robust Predictable Control
Benjamin Eysenbach" 2
Ruslan Salakhutdinov'
Sergey Levine
'Carnegie Mellon University,
Google Brain, beysenbaccs. cmu. edu
UC Berkeley
Abstract
Many of the challenges facing today's reinforcement learning (RL) algorithms, such robustness, generalization, transfer, and computational efficiency, are closely related to compression. Prior work has convincingly argued why minimizing information is useful in the supervised learning setting, but standard RL algorithms lack an explicit mechanism for compression. The RL setting is unique because (1) its sequential nature allows agent to past information to avoid looking at future observations and (2) the agent can optimize its behavior to prefer states where decision making requires few bits. We take advantage of these properties propose a method (RPC) for learning simple policies. This method brings together ideas from information bottlenecks, model-based RI., and bits-back coding into a simple and theoretically-jjustfiee algorithm. Our method jointly optimizes a latent-space model and policy to be self-comsistent, such that the policy avoids states where the model is inaccurate. We demonstrate that our method achieves much tighter compression than prior methods, yielding up to 5x higher reward than a standard information bottleneck. As result of this compression, the policies learned by our method are robust and generalize well new tasks.'
1 Introduction
Many areas of reinforcement learning (RL) research focus on specialized problems, such as learning invariant representations, improving robustness to adversarial attacks, improving generalization, or building better world models. These problems are often symptoms of deeper underlying problem: autonomous agents use too many bits from their environment. For the purpose of decision making, most information about the world is irrelevant. For example, a lane keeping feature on car may take as input high-resolution camera input (millions of bits). but only needs to extract a few bits of information about the relative orientation of the car in the lane. Agents that rely on more bits of information run the risk of overfitting to the training task.
Agents that few bits of information enjoy number of appealing properties. These agents can better cope with high-dimensional sensory inputs (e.g., dozens of cameras on self-driving car) and will be forced learn representations that are more broadly applicable. Agents that throw away most information will be agnostic to idiosyncrasies in observations, providing robustness to missing corrupted observations and allowing for transfer to different scenarios. For example, if an agent ignores 99.9% of bits, then corrupting random bit is unlikely to change the agent's 's behavior. Moreover, an agent that minimizes bits will prefer states where the dynamics are easy predict, meaning that the agent's 's resulting behavior will be easier to model. Thus, compression not only changes an agent's representation, but also changes its behavior: an agent that can only use a limited number of bits will avoid risky behaviors that require more bits to execute (see Fig. 3a).
The generalization and robustness of machine learning model is directly related to the complexity of that model. Indeed, standard techniques for reducing complexity, such as the information bottle- neck [1, 43], can be directly applied the RL setting [14, 21, 31, 42]. While these approaches make
Project site with videos and code: https: http::://ben-eeyenbbale srathbbach.ggghtiti github.io/rpe io/rpc
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the policy's ss action simple function of the state, they ignore the temporal dimension of decision making, Instead, we will focus on learning policies whose temporally-extente -ettended behavior simple, where we use compression to measure simplicity. Our key observation is that policy' 's behavior is simple if it is predictable.
Our method improves upon prior methods that apply an information bottleneck to RL [14, 21, 31] by recognizing two important properties of the decision making setting. First, because agents make a sequence of decisions, they can use salient information at one time step to predict salient information at the next time step. These predictions can decrease the amount of information that the agent needs to sense from the environment. We will show that learning a predictive model is not an ad-hoo heuristic, but rather direct consequence of minimizing information using bits-back coding [12. 19]. Second, unlike supervised learning, the agent can change the distribution over states, choosing behaviors that visit states that are easier to compress. For example, imagine driving on a crowded road. Aggressively passing and tailgating other cars may result in reaching the destination faster, but requires careful attention to other vehicles and fast reactions. In contrast, a policy optimized for using few bits would not pass other cars and would leave a larger following distance (see Fig. 3b). Combined, these two capabilities result in a method that jointly trains a latent space model and a control policy, with the policy being rewarded for visiting states where that model is accurate. Unlike typical model-based methods, our method explicitly optimizes for the accuracy of open-loop planning and results in a model and policy that are self-consistent.
The main contribution of this paper is an RL algorithm, robust predictable control (RPC), for learning policies that use few bits of information. We will refer to such policies as compressed policies. RPC brings together ideas from information bottlenecks, model-based RI., and bits-back coding into simple and theoretically-jutifee algorithm. RPC attains up to 5x higher return than a standard information bottleneck when compared at the same bitrate. Experiments demonstrate that the compressed policies learned by RPC are more robust than those learned by alternative approaches, generalize well to new tasks, and learn behaviors that can be composed for hierarchical RL.
2 Related Work
Our work builds on a large body of prior work, both in supervised learning and reinforcement learning, that argues that compression is a good objective for learning representations and improving robustness [4, 19, 27, 30, 38, 39, 42, 44]. We measure compression by the mutual information between its inputs and outputs [4, 43], metric that has been used to study the generalization properties [43] and representations learned by neural networks [1]. Our work extends these results to the RL setting by observing that the agent can change its behavior (i.e., the data distribution) to be more easily compressed. In the RL community, prior work has used the variational information bottleneck (VIB) [2] minimize communication between agents in multi-agent setting [45] and to improve exploration in a goal-reaching setting [14]. The most related RL. methods are those that an information bottleneck in RL to improve generalization [21, 31). Whereas these prior methods compress observations individually, we will compress sequences of observations, similar to sequential VAE [8. 23]. This difference, which corresponds to learning latent-space model, improves compression and increases robustness on downstream tasks.
Prior work in RL has sought robustness, transfer, and generalization in many ways. For example, prior RL methods often use explicit representation learning objectives to accelerate learning of masge- tasks [13, 26, 28, 36). The presentations learned by our method, like those learned by prior contrastive learning methods [26, 34, 36), do not require reconstructing observations. Prior work has also studied numerous strategies for learning policies that resilient to perturbations in the environment, a problem known as robust RL [20, 22, 33, 41]. While prior robust RL. methods typically involve solving two-player game, show that compression is simpler mechanism for achieving some robustness benefits. Finally, the problem of learning RL policies that generalize has been studied by many prior papers [9, 10].
3 Reinforcement Learning with Fewer Bits
This section introduces the idea that predicting the future allows RL policies to operate with fewer bits. We derive this idea from first principles, develop it into a complete RL method, then discuss connections with model-based RL, bits-back coding, and other related topics.

---

3.1 Notation and Preliminaries
An agent interacts in an MDP defined by states s, and actions a, we will use "states" and "obser- vations" interchangeably. The agent samples actions from policy To(a | St, St,1, 1,-2,, 2e..,...). We will construct this policy by learning an encoder Φ(zz 8e) (which encodes state st into a representation ze). and a high-level policy Tÿ(ay | Zt), which decodes representation 24 into an action at. The environment dynamics are defined by an initial state $1 ~ pi (si) and a transition function p(St41 SSt, at). The RL objective is to maximize the expected 7-discounted sum of rewards: max, E 1-t= 7'r(Se; at)]. The discount factor can be viewed as saying that the episode terminates at every time step with probability (1 1-77 7), an interpretation we will use in Sec. 3.2.
A model is simpler if it expresses simpler input-output relationship [4, 43]. We will measure the complexity of function using an information bottleneck, which is the mutual information / yx between an input x and an output y [1, 43). The variational information bottleneck provides a tractable upper bound on mutual information [1, 2]:
I(x:y)= S Ep(cy) logg
ply x) m(())
where mly) is an arbitrary prior distribution. Applying the information bottleneck to an intermediate layer (xx is sufficient for bounding the mutual information between input x and output y.
Following prior work on compression in RL [21, 31J, we aim to maximize rewards while minimizing the number of bits. While there are many ways to apply compression to RL (e.g.. compressing actions, goals, or individual observations), we will focus on compressing sequences of states. The input is
sequence of states, $1:00 & ($1, S2, ); the output is sequence of actions, a1:00 ف (a,, a2....).
The objective is learn representation Φe (t)) and policy wo (a, Zzz) that maximize reward, subject to the constraint that the policy uses on average C' > O bits of information per episode:
max BT
(SE, at)
s.t. Ex[7(51:00 21:3)) S C,
(1)
While prior work on compression in RI (21, 31] has applied the VIB to states independently, we will aim compress entire sequences of observations. Applying the VIB to sequences allows to use more expressive choices of the prior m() Our prior will use previous representations to predict the prior for the next representation, allowing us obtain tighter bound on mutual information.
3.2 Using Fewer Bits by Predicting the Future
Predicting the future allows agents to observe fewer bits from future observations. To further decrease the number of bits used, the agent can change its behavior visit states that are more easily compressed. States where the dynamics are hard to predict will require more bits, so the agent will prefer visiting states where its learned model can accurately predict the next state.
This intuition corresponds solving the optimization problem in Eq. I with a prior that is factored autoregressively: m1 (Zi:0 a1:s0) = m(11) II. m0(2++1 Ze; a.). Note that the prior has learnable parameters 0. We apply the VIB to obtain an upper bound on the constraint in Eq. 1:
E. [=((((1:00:21 Z1:00) < Ex
log p(Z11:0
$1100)
]
E Σ "log (o | (t) - logmo(Zet1 Ze. ac))
m(Z1:00)
(2)
This objective is different from prior work that applies the VIB to RL [14, 31] because the prior mo(z;) is predicted, rather than fixed to be unit Normal distribution. The discount factor reflects the assumption from Sec. 3.1 that the episode terminates with probability (1 7) each time step; of course, no bits are used after the episode terminates. Our final objective optimizes the policy n. the encoder e and the prior mn to maximize reward and minimizing information:
max Eng. dig ma 0
Σ yr'r(st ac)
s.t.
Bio .0g .m
Σ r (log@o(
st) - log mo Ze. at))
S C.
(3)
For the encoder, this objective looks like modification of the information bottlenck where the prior is predicted from the previous representation and action. Of course, we cannot predict the first
We condition the policy on sequences of states because, even if the task is Markovian, the agent's choice to omit certain bits from the representation can turn otherwise oblly-observed task into partially oabteelly-observed task.
3

---

representation, so we substitute use a unit Normal distribution for the prior at initial time step. For the policy, this objective looks like the standard RL objective but with different reward function. one that includes an additional information cost:
Fx (st. at, st-1) f r(st. ax) + A (log ma ma Ze, at) log 6e (Zt+1 st+1)),
(4)
(uegative of the) information
where À is the cost per bit. The term log (Zt st) corresponds to the number of bits required to represent the representation Zt. The term log mo (z Ze, a4) reflects how well the agent can predict the next representation. In effect, the agent has to "pay" for bits of the observation, but gets "refund" on bits that it predicted from the previous time step, analogous bits-back coding [12, 19]. Said in other words, we want to minimize the excess bits required to infer 2, from the current state, relative the number of bits required predict Ze from the previous (2 ( Note that the standard VIB approaches to RL [14, 21, 31J do not receive this refund because they compress observations independently, rather than compressing sequences of observations.
Note that the policy, encoder, and model are all jointly optimized with respect to the same objective, so updates to any component increases the objective for all components. Importantly, the agent optimizes not only its representation but also its behavior to minimize this information cost: the agent learns representation that is easily predictable and learns to visit states where that representation is easily predictable. We therefore call our method robust predictable control (RPC).
4 A Practical Algorithm
Our method for optimizing (Eq. 3) is actor-critic method applied to the information- augmented reward function Eq. 4. We intro- duce a Q function,
a1
a2
03
(High-level) policy
(
22
23
Encoder
Latent-space model (Lee. predicted prior)
(st, ac) = E Σ Ffst, at, st.)))))
)
5 d(z
~ m(4 2-
S
S2
S3
corresponding to the augmented reward function A. We optimize this Q function using standard temporal difference learning:
Figure 1: Robust Predictable Control (RPC): Our method learns three components: an encoder ((ze ss). a latent-space model m(2: Zt-1. at.)). and a policy (at zz). conventional VIB [21, 31] omits the blue arrows.
((v) = (Qt (Q v1)²
(5)
where Yt = [ + overall objective (Eq. 1) only entails compressing the policy, not the Q function, we condition the Q function directly on the state. To optimize the encoder, prior, and policy, we reexpress our objective (Eq. 1) in terms of the immediate reward plus the Q function (see derivation in Appendix A.1):
at+1)) le and 1. is the stop-gradion operator [6, 11]. Since our
[(0; s = Ez, 10 (st (.sspp(ssll)) 1.8t a) 11Q (st. a+) (log ma 1, at-1) - log Ó0 (s)) ss))] (rr-)))ss). sem (m.)
(6)
Since the encoder is stochastic, we compute gradients using the reparametrization trick [25]. The fact that three components are jointly optimized with respect to the same objective makes implementation of RPC surprisingly simple. Unlike sequential VAEs and RNNs, training agents with RPC does not require sampling from m& m (Z++1 Zt a₂) and does not require backpropagation through time. In our implementation. we instantiate the encoder Φn (sx), the prior my (Z++1 Z, a4). and the high-level policy (a, ze) as neural networks with parameters O. The Q function ( at) is likewise represented as neural network with parameters é. We update the dual parameter À 2 0 using dual gradient descent. We use standard tricks such as target networks and taking the minimum over two Q values. We refer the reader to Appendix B and the open-sourced code for details.
5 Connections and Analysis
In this section we discuss the connections between robust predictable control and other areas of RL. We then prove that RPC enjoys certain robustness guarantees.

---

5.1 Connections
This section explains connections between RPC and related ideas in the RI. literature, with the aim of building intuition into how RPC works and providing an explanation for why RPC should learn robust policies and useful representations. We include a further discussion of the relationship to MaxEnt RL, the expression for the optimal encoder, and the value of information in Appendix A.3.
Model-Based RL. The prior mz(Zt 1t-1..a-11) learned by RPC can be viewed as a dynamics model. Rather than predicting what the next state will be, this model predicts what the representation of the next state will be. While the model is trained to make accurate predictions, the policy is also trained to visit states and actions where the model will be more accurate.
Representation learning. What, precisely, does the representation 2t represent? Given prior m(2++1 Zt, a²)) and latent-conditioned policy x (a, 2e), we can use the current representation Zt predict good actions both the current time step and (by unrolling the prior) at future time steps. Thus, the representation can be thought of as a compact representation of open-loop action sequences. Our experiments demonstrate that this compact representation of action sequences, once leamed on one task, can be used to quickly learn range of downstream tasks.
We can also view RPC learning new action space for the MDP, where the reward function is now (Eq. 4). What we previously called the encoder, ØZz, Se), is now the policy for selecting the actions 2z in this new MDP. The log d(2 8t)) term in from F is the entropy of this encoder, so the encoder is performing MaxEnt RI. on this new MDP. This new MDP encodes a strong prior for open-loop policies: sampling actions from the prior yields high reward.
Open-loop control. RPC learns model m(2t 1 a that predicts the state representation at the next time step. Thus, we can unroll our policy in an open-loop manner, without observing transitions from the true system dynamics. Because the model and policy are trained to be self- consistent, we expect that the highly compressed policies learned by RPC will perform well in this open-loop setting, as compared to uncompressed policies (see experiments in Sec. 6).
5.2 Theoretical Guarantees
We now present analysis showing that the compressed policies learned by RPC generalize better. We also discuss a close connection between compression and open-loop control, proving that RPC learns policies and models that are self-consistent in way that guarantees good performance when even when the agent cannot observe new observations. We do not intend to to analyze all benefits deriving from model compression (see, e.g., [3, 4J). All proofs are in Appendix C.
well known that compression results in models that generalize better from the training set to the testing set. Our main result applies this same reasoning to RL. We assume that the policy has been trained on an empirical distribution of MDPs, and will be evaluated on new MDP sampled from that sample distribution. For the following result, we assume that the given stochastic MDP a mixture of deterministie MDPs, each described by a b-bit random string (see Ng and Jordan [35]).
In supervised learning,
Lemma 5.1. Let stochastic MDP M and policy π be given. Define RD (M) to be the expected reward on MDP M. Then the probability that the policy's 's expected return on an observed (deterministic) MDP M is much different than the policy's 's expected return on the stochastic MDP is bounded by the
policy's bitrate:
C+1
PAil (M) RT(M) > e
262--1
The proof is a direct application of Bassily et al. [4, Theorem 8]. This result may be of interest in the offline RL setting, where observed trajectories effectively constitute deterministic MDP; that policies that use fewer bits will be less likely to overfit to the offline dataset.
Not only do compressed policies generalize to new MDPs, they are also robust to missing observations. Intuitively, policy that uses zero bits of information will perform identically the open- setting. The following result shows that our model compression objective corresponds to maximizing a lower bound on the expected return of the open-loop policy. Let n open be the open-loop policy corresponding to the composition of the prior m mZt+11 Zt. at) and the high-level policy л² (a, zz). Let x reactive
5

---

0.3
observation
Normalized Mallzzded Roturn
1.00
0.75
0.5
RPC (ours)
0.30
0.4
VIB reward
RNN + VIB
Hopperred
HatiCheethh
Andoet
(imageo)
run
watler oxal
(magees)
Hopplened
Waiter
hattheeeahh
Cheetat ruo (mmaged)
watterr werle comqees)
Figure 2: Learning Compressed Policies. We measure the return achieved by policies constrained to have fixed bit rate. Especially at low bit rates, RPC achieves higher return than alternative methods.
be the reactive policy corresponding to the composition of the encoder Φz (t)) and the high-level policy (a, z.).
Lemma 5.2. Let encoder Φ(z, St), policy (a 1t). prior m(zt Zt-1, ae-1,ar-1). and reward function r(st, ar) > 0 be given. Then our model compression objective (Eq. 3) with reward function (1 (1-7) (og ((), ar) a lower bound on the expected return of the open-loop policy.
Σ (s)) 2 (E t=1
Σr' E
log ((s, mlg Z4-1, 8c-1) log oizz st)) )))
where f(x) = - is a monotone increasing function of x.
Moreover, the policies learned by RPC will receive similar returns in settings where they can and cannot observe observations, result that may be useful for quantifying the regret incurred when using the learned representation
an action space of cemporally-xtendde behaviors. Let Raax max, R(T) be the maximum return of any trajectory.
Lemma 5.3.
The expected return of the open-loop policy opeen is at most Rmax V //2 worse than the
expected return of the reactive policy a reacivve
Exysyon
Σ rr at) 2 Beemactrr
Σ En'r(se,ap) ac)
Peaax //22.
In summary, our theoretical results suggest that compressed policies generalize well and, by virtue of their open-loop guarantees, may be useful for planning. We emphasize that the theoretical benefits of model compression have been well studied in the supervised learning literature. As our method likewise performing model compression, we expect that will inherit a wide range of additional guarantees, such as guarantees about sample complexity.
Experiments
Our experiments have two aims. First, we will demonstrate that RPC achieves better compression than alternative approaches, obtaining higher reward for the same number of bits. Second, we will study the empirical properties of compressed policies learned by our method, such as their robustness and ability to learn representations suitable for hierarchical RI.. We do not intend this section to exhaustively demonstrate every possible benefit from compression; we acknowledge that there are many purported benefits of compression, such as exploration and sample efficiency, which we do not attempt to study here. We include additional experiments in Appendix B.
6.1 Evaluating Compression
Our first experiment studies whether RPC out- performs alternative approaches to compression, which we summarize in the inline table. We compare against standard VIB [21] and extension that adds the information cost to the reward, "VIB+reward" [31]. This baseline can be viewed as a special case of RPC where the blue arrows Fig. are removed. Finally, since the predictive prior RPC is similar a recurrent network, we compare against an extension of the VIB [21] that uses an LSTM ("RNN + VIB"). Unlike RPC, this baseline requires training on entire trajectories (rather than individual transitions) and requires backpropagating gradients through time. We evaluate all methods on four tasks from
feedforward
predicted prior
augmented
architecture
reward
RPC (ours)
VIB [21]
X
VIB+rreward [31] VIB+RNN
XX

---

Contormeted Pillry roum) Standard hollcy
For an couavatton
Thre seeesseesh
cont Infomattoon Dest)
C
550 201
145
(a) Driving with Traffic
(b) Active Cruise Control
Representations: (Top) Compressed policies ob- serve more bits from observations that have large value of informa- tion. (Bottom) Compressed poli- cies learn sparse representations.
Figure 4:
Figure 3: Behavior of compressed policies: On two driving tasks, we observe that highly compressed policies (leeft) avoid passing other cars and (Right) leave a larger following distance between cars. The passing and tailgating that our method forgoes would require more bits of information about the precise locations of the other cars.
OpenAl-Gym [5] and two image-based tasks from dm-control [40]. Because of computational constraints, we omit the RNN+VIB baseline on the image-based tasks.
We plot results in Fig. 2. To make the rewards comparable across tasks, we normalize the total return by the median return of the best method. On almost all tasks, RPC achieves higher returns than prior methods for the same bitrate. For example, when learning the Walker task using 0.3 bits per observation, RPC achieves return of 3,562, the VIB+RNN baseline achieves return of 1,978 (-44%), and the other VIB baselines achieve a return of around 530 (-85%). We include full results on wider range of bitrates in Appendix Fig. 10. While, in theory, the VIB+RNN baseline could implement RPC internally, in practice it achieved lower returns, perhaps because of optimization challenges associated with training LSTMs [7]. Even if the VIB+RNN baseline could implement the strategy learned by RPC, RPC is simpler (it does not require training on trajectories) and trains about 25% times faster (it does not require backpropagation through time).
6.2 Visualizing Compressed Policies
Behavior of compressed policies. To visualize how compression changes behavior, we applied RPC to two simulated driving tasks shown Fig. 3a (top left), which are based on prior work [29]. In the first task, the agent can pass cars by driving into the lane for oncoming traffic; the second task uses the same simulated environment but restricts the agent to remain in its own lane to study active cruise control in isolation. The rewards for these tasks corresponds to driving to the right as quickly as possible, without colliding with any other vehicles. In the first task, we observe that compressed policies passed fewer cars than uncompressed policies. Intuitively, passing maneuver requires many bits of information about the relative positions of other cars, so we expect that compressed policies to engage in fewer passing maneuvers. Fig. 3a (bottom) shows that the bitrate of RPC is directly correlated with the number of cars passed. In the second task, we observe that compressed policies leave larger following distance from the leading car (Fig. 3b (top)). Fig. 3b (hollom) shows that the number of bits used per observation increases when the car is within 15m of another car. Thus, by maintaining a following distance of more than 30m, the compressed policy can avoid these situations where it would have to use more bits. See the project website for vidcos.
Representations of compressed policies. A policy learned by compression must trade off between maximizing reward and paying to receive bits of information. Using the plot the value of information versus cost of information for many sampled states. See Appendices A.4 and B.5 for details. As shown in Fig. 4 (top), the RPC uses more bits from observations that are more valuable for maximizing future returns. Then, visualize the learned representation Zg, we sample the representation from randomly sampled states, and plot the numbers of bits used for each coordinate. Fig. 4 (bottom) shows that RPC learns sparse representations. Whereas the uncompressed policy uses all coordinates, a policy compressed with bitrate 10 uses only 10/50 coordinates and policy compressed with bitrate 0.3 uses only 2/50 coordinates.
task, we
Intuitively, the ways in which compression changes the behavior and representations of policies should make these policies more robust to disturbances. Our next experiments explicitly test this hypothesis, which would validate the analysis in Sec. 5.2.
Project site with videos and code: https: //ben/////eeeeeee wwwenbach.cc........ github. 1o/rpc
7

---

6.3 Robustness
The connection between compression and robustness has been well established in the literature (e.g., [16, 47]). Our next set of experiments test the robustness of compressed policies to different types of disturbances: missing observations (i.e., open-loop control), adversarial perturbations the observations, and perturbations to the dynamics (i.e., robust RL). We emphasize that, because these experiments focus on robustness, the policy is trained and tested in different environments.
Robustness to missing observations and open-loop control. Since compressed policies rely fewer bits of input from the observations, we expect that they not only will be less sensitive missing ob- servations, but also will actively modify their behavior to adopt strategies that quire fewer bits from the observation. In this experiment, we drop each observation independently with probability E [0, 1]. where p = corresponds to using a fully open-loop policy. For RPC. we handle missing observations by predicting the representation from the previous time step. Our two main baselines take a policy used by standard RL and learn either a latent-space model or a state-space model. When the observation is missing, these baselines make predictions using the learned model. We also compare against RNN+VIB, the strongest baseline from Fig. 2. When observations are missing, the LSTM's input for that time step is sampled from the prior. Fig. 5 shows that all methods perform similarly when no observations are dropped, but RPC achieves higher reward than baselines when a larger fraction of observations are dropped. This experiment shows that more effective compression, as done by RPC, yields more robust policies. This robustness may prove useful in real-world environments where sensor measurements are missing or corrupted.
haal Boong
Fracdie Mixding Ooseratione
Figure 5: Robustness to missing observations: RPC is more robust to missing observations than prior methods, including those that learn dynamics models. We show HalfCheetah--2 on left and Walker2d-v2 on right.
Adversarial Robustness. Compressed policies extract fewer bits from each observation, so expect that compressed policies will be more robust to adversarial perturbations to the observation. While prior work has proposed purpose-desinnes investigate whether compression is simple yet effective means for achieving a modicum of robustness. We do not claim that compression is the best method for learning robust policies.
methods for achieving robustness [33, 411, here
We first study adversarial perturbations to the dynamics on the Ant-v2 environment. Given a policy T(a (s)) and the current state 8t, the adversary aims to apply small perturbation to that state to make the policy perform as poorly as possible. We implement the adversary using projected gradient descent [32]; see Appendix B for details. Fig. 6 (left) shows the expected re- turn as we increase the magnitude of the attack. The compressed policy is more resilient to larger attacks than the uncompressed policy. Our next experiment looks perturbations to observations. Unlike the previous experiment, we let the adversary perturb every step in an episode; see Appendix B for full details. Fig. 6 (right) shows that policies that fewer bits achieve higher returns in this ad- versarial setting. When the Ant-v2 agent uses too many bits (3 or more), the adversarial perturbations the dynamics flip the agent over, whereas agents that use fewer bits remain upright.
Randed by mord
Figure 6: Adversarial robustness: Compressed poli- cies are more robust against adversarial attacks to the (Left) dynamics and (Right) observations.
Robust RI. Our final set of experiments look higher-level perturbations to the dynamics, as are typically studied in the robust RL commu- nity [33, 41]. Using the same Ant-v2 environ- ment as before, we (/) increase the mass of each body element by a fixed multiplier, or (2) de- crease the friction of each body geometry by fixed multiplier. These experiments test whether the learned policies are robust to more massive robots or more "slippery" settings. Fig. 7 shows that compressed policies generalize to larger masses and smaller frictions more effectively than an uncompressed policy. Comparing the policies learned
Mass
triction
2000
BPC (0.3 bit0
1500
RFC (3.2 bita)
Ratumn
Star Policy
1000
500
22
Figure 7: Robust RL: Compressed policies are more robust to increases in mass and decreases in friction.
8

---

50.00
M
1096
276
12:0.00
return
40030
BPC (ourd
AcCot hipeal
300
200
600
600
2003
opisades
opisodas
Figure 8: Hierarchical RL: We apply RPC to the pushing task shown in the top-left. We then use the learned representation of action sequences as the action space for solving three new tasks. On all tasks, the representations learned by RPC accelerate learning. All episodes have a fixed length.
by RPC with bit rate of 0.3 bits versus 3.0 bits, we observe that the bit rate effectively balances performance versus robustness. See Appendix Fig. for a larger version of this plot with error bars.
6.4 Hierarchical RL using the Learned Representation
RPC learns a representation of temporally-exter that these representations can accelerate the learning of new tasks in hierarchical setting. Our goal not to propose complete hierarchical RL system, but rather evaluate whether these action repre- sentations are suitable for high-level control. During training, we apply RPC to a coal-conditioned object pushing task, shown in Fig. (top-left). The initial position the object and the goal position are randomized, so we expect that different representations 25 will correspond to high-level behaviors of moving the end effector and object to different positions. At test-time, the agent is presented with task. The agent will attempt to solve the task by commanding one two behaviors Z;. See Appendix B for details and pscudocode.
action sequences (see Sec. 5.1). We hypothesize
We compare the action representations learned by RPC to three baselines. The first baseline is "Action Repeat", which constantly outputs the same action, effectively allowing the agent to command desired poses. To test whether RPC has learned a prior over useful action sequences, we use a variant of RPC ("RPC (randomly initialized))) where the policy and model are randomly initialized. Finally, "RL from scratch" applies a state-of-the-art off-policy RL algorithm (SAC [17]) the task. Taken together, these baselines allow us to study whether RPC not only learns behaviors that do more than move to particular poses, but also learns behaviors for obstacle avoidance and object manipulation. We apply all methods to four tasks and present results in Fig. 8. First, as a sanity check, we apply all methods to the training task, finding that RPC quickly finds single zt that solves the task. On the remaining three tasks, we likewise observe that the representations learned by RPC allow for much faster learning than the baselines. The final task, "pusher wall" requires chaining together multiple representations z to solve. While the "RL from scratch" baseline eventually matches and then surpasses the performance of RPC, RPC accelerates learning in the low-data regime.
7 Conclusion
In this paper, presented method for learning robust and predictable policies. Our objective differs from prior work by compressing sequences observations, resulting in a method that jointly trains a policy and model to be self-consistent. Not only does approach achieve better compression than prior methods, it also learns policies that are more robust (Fig. 6). We also demonstrate that our method learns representations that suitable for use in hierarchical RL.
Our work suggests a few areas for future work. Ideas based on RPC may be useful for model-based RL, training the policy to be self-consistent with the model may help prevent model exploitation. Similarly, because the skills learned by RPC are optimized to be predictable, we expect that they will perform well when used for open-loop planning. As compression is closely related the energy required to implement a function on an ideal physical system [37], RPC might be useful to building energy-efficieet control policies.
The main limitation of this work is that policies that use few bits will often receive lower reward on the training tasks. Also, for the purpose of exploration, the most informative states may be those that are hardest to compress. While the first limitation is likely irreconcilable, the second might be lifted by maximizing information collected during exploration but minimizing information for policy optimization.
Limitations.

---

Acknowledgments. We thank Dibya Ghosh and the members of the Salakhutdinov lab for feedback through- the project. We thank Oscar Ramirez for help setting up the image-based experiments and thank Rico Jonschkowski introductions to potential collaborators. We thank Ryan Julian, Vincent Vanhoucke, and anonymous reviewers for feedback on the paper, and thank Abhishek Gupta for discussions during the carly stages of this project.
This material is supported by the Fannic and John Hertz Foundation and the NSF GRFP (DGE1745016).
References
[1] Achille, A, and Soatto, (2018). Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 1947:1997
[2] Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy. K. (2017). Deep variational information bottleneck. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
[3] Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. (2018). Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pages 254-263. PMLR.
[4] Bassily, R., Moran, S., Nachum, I., Shafer, J., and Yehudayoff, A. (2018). Learners that use little information.
In Algorithmic Learning Theory, pages 25-55. PMLR.
[S] Broekman, G.. Cheung, V., Pettersson, L.. Schneider, J., Schulman, J., Tang. J., and Zaremba, W. (2016). Openai gym. arXiv preprint 10Xiv:1606.01540.
[6] Chen, X. and He, K. (2021). Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15750-155 15758.
[7] Cho, K., van Merrienboer, B., Gülçchre, Ç., Bahdanau, D.,, Bougares, Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Moschitti, A., Pang, B., and Daelemans, W., editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 1724-1734. ACL.
[8] Chung. J., Kastner, K., Dinh, L., Gocl, K., Courville, A. C., and Bengio, Y. (2015). A recurrent latent variable model for sequential data, Advances in neural information processing systems, 28:2980 2988.
[9] Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. (2019). Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-1289. PMIR.
[10] Farebrother, J., Machado, M. C., and Bowling, M. (2018). Generalization and regularization in dgn, arXiv preprint 1.0.0018...00123.
[11] Foerster, J., Farquhar, G., Al-Shedivat, M., Rocktäschel, T., Xing. E., and Whiteson, (2018). Dice: The infinitely differentiable monte carlo estimator. In International Conference on Machine Learning, pages 1529- 1538, PMLR.
[12] Frey, B. and Hinton, G. F. (1997). Efficient stochastic source coding and application to bayesian network source model. The Computer Journal, 40:157-165.
[13] Gelada, C., Kumar, S., Buckman, J., Nachum, O., Bellemare, M. G. (2019). Deepmdp: Learning continuous latent space models for representation learning. In Intermational Conference on Machine Learning, pages 2170-2179. PMLR.
[14] Goyal, A., Islam, R.. Strouse, D., Ahmed, Z., Larochelle, H., Botvinick, M., Levine, S., and Bengio, Y. (2019). Transfer and exploration via the information bottleneck. International Conference on Learning Representations.
[15] Guadarrama, S., Korattikara, A., Ramirez, O., Castro, P., Holly, E., Fishman, S., Wang, K., Gonina, E., Wu, N., Kokiopoulou. E., Sbaiz, L., Smith, J., Bartók, G., Berent, J., Harris, C., Vanhoucke, V., and Brevdo, E. (2018). TF-Agents: A library for reinforcement learning in tensorflow. https:// http://ggtttbb.ccc com/ ammsorflow/agente. [Online; accessed 25-June-2019]
[16] Gui. S., Wang, H. N., Yang. H., Yu, C., Wang, Z., and Liu, (2019). Model compression with adversarial robustness: A unified optimization framework. Advances in Neural Information Processing Systems, 32:1285- 1296.
10

---

[17] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor: In International Conference Machine Learning, pages 1861-1870. PMLR.
[18] Hansen, N. (2006). The ema evolution strategy: a comparing review. Towards a new evolutionary computation, pages 75-102.
[19] Hinton, G. E. and Van Camp, D. (1993). Keeping the neural networks simple by minimizing the description length of the weights, In Proceedings of the sixth annual conference on Computational learning theory; pages 5-13.
[20] Huang, H., Papernot, N., Goodfellow, J., Duan, Y., and Abbeel, P. (2017). Adversarial attacks on neural network policies. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net.
[21] Igl. M., Ciosek, K., Li, Y., Tschiatschek, S.. Zhang, C., Devlin, S., and Hofmann, K. (2019). Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in Neural Information Processing Systems, 32:1 1977--1990..
[22] Kamalaruban, P., Huang, Y., Hsieh, Y., Rolland, P., Shi, C., and Cevher, V. (2020). Robust reinforcement learning via adversarial training with langevin dynamics. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.. and Lin. H., editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual.
[23] Ke, N. R., Singh, A., Touati, A., Goyal, A., Bengio, Y., Parikh, D., and Batra, D. (2019). Learning dynamics model in reinforcement learning by incorporating the long term future. arXiv preprint arXiv:1903.00 010.3.0150
[24] Kingma, D. P. and Ba, (2015). Adam: method for stochastic optimization. ICLR (Poster).
[25] Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y., editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings.
[26] Laskin, M., Srinivas, A., and Abbeel, P. (2020). Curl: Contrastive unsupervised representations for reinforcement learning. International Conference on Machine Learning, pages 5639 5650. PMLR.
[27] LeCun, Y., Denker, J. S., and Solla, A. (1990). Optimal brain damage. Advances in neural information processing systems, pages 598-605.
[28] Lec, A., Nagabandi, A., Abbeel, P., and Levine, S. (2020). Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33.
[29] Leurent, E. (2018), An environment for autonomous driving decision-making making. https://githe: com/ eleurent/highway-ee env.
[30] Littlestone, N. and Warmuth, M. (1986). Relating data compression and learnability.
[31] Lu, X., Lec, K., Abbeel, P., and Tiomkin, (2020), Dynamics generalization via information bottleneck in deep reinforcement learning. arXiv preprint arXiv: 00614.
[32] Madry, A., Makelov, A.. Schmidt, L.. Tsipras. D.., and Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
[33] Morimoto, J. and Doya, K. (2005). Robust reinforcement learning. Neural computation, 17(2):335-359,
[34] Nachum, O., Gu, S., Lee, H., and Levine, S. (2019). Near-optimal representation learning hierarchical reinforcement learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net.
[35] Ng, A. Y. and Jordan, M. I. (2000), PEGASUS: A policy search method for large mdps and pomdps. In Boutilier, C. and Goldszmidt, M., editors, UAI '00: Proceedings of the 16th Conference Uncertainty in Artificial Intelligence, Stanford University, Stanford, California, USA, June 30 July 3, 2000, pages 406-415. Morgan Kaufmann.
[36] Oord, A. v. d., Li, Y. and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03745
11

---

[37] Ortega, P. A. and Braun, D. A. (2013). Thermodynamics as theory of decision- making with information- processing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 469021533:20120683.
[38] Schmidhuber, J. (1992). Learning complex, extended sequences using the principle history compression. Neural Computation, 4(2):234-24 242.
[39] Shamir, O., Sabato, S., and Tishby, N. (2010). Learning and generalization with the information bottleneck. Theoretical Computer Science, 41 (099-33)::66627717
[40] Tassa, Y., Doron, Y.. Muldal, A., Erez, T., Li, Y., Casas, D. L.. Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018). Deepmind control suite. arXiv preprint wxXiv1881.00699
[41] Tessler, C., Efroni, Y., and Mannor, S. (2019). Action robust reinforcement learning and applications in continuous control. International Conference on Machine Learning, pages 2224-6224. PMLR.
[42] Tishby, N. Polani, (2011). Information theory of decisions and actions. In Perception cycle, pages 601-636. Springer.
[43] Tishby, N. and Zaslavsky, N. (2015). Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pages 1-5, IEEE.
[44] Veness, J., Bellemare, M. G., Hutter, M., Chua, A., and Desjardins, G. (2015). Compress and control. In Twenty-Ninth AAAI Conference Artificial Intelligence.
[45] Wang, R., He, X., Yu, R., Qiu, W. An, B., and Rabinovich, Z. (2020). Learning efficient multi-agent communication: An information bottleneck approach. In International Conference on Machine Leurning. pages 9908-9918. PMLR.
[46] Yarats, D., Kostrikov, I,, and Fergus, R. (2021). Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations.
[47] Ye. S., Xu, K., Liu, S., Cheng. H., Lambrechts, J.-H., Zhang, H., Zhou, A., Ma. K., Wang. Y., and Lin. X. (2019). Adversarial robustness vs. model compression, or both? In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 111-120.
[48] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. (2020). Meta-world: benchmark and evaluation for multi task and meta reinforcement leaming. In Conference on Robot Learning. pages 1094-1100 1100. PMLR.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Limitations in Sec. 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] All experiments were run for approximately one day. The image-hased experiments used half of a NVIDIA p100 or v100 GPU. The state-based experiments did not use GPU.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "pGCVvyhRbx", "reviewer": "Reviewer_evKc", "summary": "The paper \"Robust Predictable Control\" describes a method to learn policies with an information bottleneck between environment observations and a latent representation which is used for decision-making.\nThe bottleneck is posed between *sequences* of observation and latent states, in contrast to prior work, where a bottleneck is posed between individual states and latents independently.\nA crucial component of the proposed method is an action-conditioned dynamics model in the latent space.\nThe proposed training objective is jointly minimized by training an encoder, latent dynamics model, and policy. \nFirst, this encourages the encoder and dynamics model to yield good predictions of encoded environment observations. \nFurthermore, by training the policy, the expected task return is maximized while ensuring that the behaviour of the agent is well-predictable. \nThe latent dynamics model formulation can be exploited to derive theoretical guarantees between open-loop and information-constrained closed-loop control. \nSeveral experiments show that the information-constrained agent performs well in the nominal case but also on distorted environments and offers smooth transitions between different behaviours depending on the tightness of the information constraint.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n * The paper clearly states the difference to prior work and adequately cites it.\n \n * In my opinion, the paper has close connection to Rate-Distortion-Theory, or,\n   to be more precise, \"bounded rational decision-making\". \n   Especially the experiment in Figure 3a shows this behaviour quite nicely. \n   I think the paper would benefit from establishing the connection and referring to this domain of research.\n  \n\n**Quality**\n\n * I perceive the paper as of very high quality.\n It is a consistent paper containing a sound theoretical motivation of the\n followed approach, interesting and well-executed theoretical derivations\n and insightful experiments. \n Prior work is adequately cited and compared to in the experimental evaluation.\n\n\n**Clarity**\n\n * In general, the paper is very clearly written and good to follow. I have the following comments:\n   * The term \"states\" and \"observations\" seem to be used interchangeably. \n     I would appreciate a clarification of that fact at the beginning of section 3.1, eventually\n     including the statement that you assume the environment state to be fully observable, if this is the case.\n   * Related to the first point, in l. 95 you define the policy to be conditioned on \n     a *sequence* of past states. If $s_t$ are *states* of an MDP (and not observations of a POMDP)\n     then conditioning on the current state should be sufficient.\n   * The joint distribution $m_{1:\\infty}(z_{1:\\infty})$ as defined in l. 127 misses a factor to be a proper joint distribution\n     in my opinion, e.g. $m_{1:\\infty}(z_{1:\\infty}) = p(z_1) \\prod_{t \\geq 1} m(z_{t+1} | z_t, a_t)$\n   * The definition of the probabilistic bound in l. 242 is not aligned with the introducing text. \n     Due to the absolute difference it is rather the \"probability that the policy's expected\n     return on an observed (deterministic) MDP is much *different* to the policy's expected return ...\"\n   * Appendix, l. 694/695: Introducing $p$ and $q$ here as done in l. 710 would definitely help\n     to understand the following derivations. Furthermore, I think that l. 695 contains a typo and it should\n     be $\\log p - \\log q$ in the expectation.\n   * Appendix, l. 698-699: I think the statement should be slightly changed to clarify that\n     \"expectations over a mixture of finite-horizon episodes are equivalent \n     to expectations over a discounted infinite-length episode\"\n\n  * Minor comments:\n    * l. 133: Typo, you probably mean \"prior $m$\"?\n    * l. 166: Typo, $z_{t-1} \\sim \\Phi(s_{t-1})$\n    * eq. 4 / l.139: time-index in prior $m$ not consistent\n    * l. 241: Introduce the \"observed MDP\" as $\\hat{\\mathcal{M}}$\n\n\n**Significance**\nIn my opinion, the paper is of very high significance for the RL community as it contains insightful theoretical and experimental results on applying ideas of compression / information-regularization to reinforcement learning. \nIt establishes interesting connections between open-loop and closed-loop control and model-free and model-based RL.\n\n\n**Conclusion**:\nTo me, this paper is a clear accept. It is well-written and contains a well-motivated, novel approach with interesting theoretical and experimental results for the reinforcement learning community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* The authors list as limitation that information-regularized policies receive lower reward\non the training tasks. I do not see this as a limitation per se, but just as an inevitable\nconsequence of rate-distortion theory (see above).\n* One not yet mentioned limitation might be that the presented approach requires full\nobservability of the environments, if this is the case.\n* Societal impacts are not explicitly addressed. In my opinion, this work does not exhibit extraordinary societal impacts  in addition to those induced by basic reasearch on reinforcement learning methods in general.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "haOmuPpGPBU", "reviewer": "Reviewer_8Hiw", "summary": "This paper proposes the method \"robust predictable control (RPC)\". It proposes to use a information\nbottleneck between inputs and actions. However, in contrast to previous works, it does so not on a\nper-observation basis, but on a per-sequence basis using a latent model. Similar to some of the\nprior work, the encoding cost is not added to the loss, but subtracted from the reward.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this is an interesting and valuable paper which is well written and easy to read. \nWhile the idea itself does not seem like\na too big departure from previous work (although it is novel to my knowledge), the paper provides a\nlot of analysis (both experimental and theoretical) as well as an intuitive explanation of how the\nnewly proposed reward affects the behavior of the agent - which I believe is a valuable contribution. \n\nWhile I'm in favor of acceptance, I'm hesitant to endorse it stronger at the moment as it does not\nprovide clear information about the absolute achievable return for some of the experiments (at least\nas far as I can tell).  And while it is expected that the provided information-regularisation\ndecreases the training performance, the extend to which this happens is important to judge the\nvalidity of the proposed method. In particular, it would be important to know whether the shown\nadvantages of RPC hold even with (somewhat) competitive performance on the task or if they are only\nrealised in the very low information & return regime.\nTo some extend the information is provided (e.g. Fig 6/8) but those plots are cut off and don't show\nthe performance of the unperturbed agent and also seem to use some different bitrates, all on the\nvery low end of the spectrum. In short: It would be helpful to have a overview table how RPC\ncompares to SAC for different bitrates on all used environments.\n\nOn a positive note, I think section 5 is great. I did have a look at the proofs and\nthey seem reasonable, although it's possible I overlooked something.\n\nMinor comments:\n* Fig 3b-top) Why is the variance of the blue line so much higher?\n* Appenidx line 695: This should be a `- log q(...)` not `- log p(...)`?\n* Eq. 10: Should be `phi(z_{t+1} | s_{t+1})` ?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Oty48FAfVKF", "reviewer": "Reviewer_wQud", "summary": "This paper introduces a novel approach to learning parameterized policies. The way I understand the work: it jointly optimizes a policy objective, a probabilistic dynamic model objective, and a compression objective. A bottleneck encoding distribution is key here, which is subject to both model predictions (given previous action and encoding) and compression (given current observation). In spirit, it is similar to variational information bottleneck (VIB) but can be seen as an extension of that to RL. So, the interpretation of this optimization is that the agent is aiming for encoding distribution that is maximally predictive of the next observation and maximally compressive of the observation while the agent is maximizing the future aggregate reward. The idea is neat. The optimization can also be easily put together. Some theoretical and empirical results are given.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am generally accepting of the work. I would appreciate discussing the following issues.\n\nThe authors admit that being compressive will necessarily lead to lower rewards in complex situations. But then it felt like you took it as a goal because it by itself is still a good thing to achieve. Later you show that they have better generalization power. But can you motivate a bit more why aiming for compressed policies is a good goal? You gave some specific examples where compressed policies will naturally be safer in some cases. But if we are trying to build a general learning agent, where the agent needs to learn many things and address more and more complex situations over time, should we aim for compressed policies?\n\nIn equation 3, we have phi(z|s) and m(z’|z,a) in pairs. However, in the rest of the paper, phi(z’|s’) and m(z’|z,a) come in pairs. Seems like there is a mismatch. Can you explain the mismatch?\n\nClarification questions:\nA large part of the paper initially talks about learning a “simple policy,” which is defined as a simple input-output relationship. It will be good if you specify directly that by simple you mean compressive.\n\nYou claimed that “Q value needs at least as many bits as a representation sufficient for predicting the action.” This argument may make sense for value-based methods such as Q-learning where the policy is derived from learned values. But in your case, the policy is parameterized directly and not through the values function. Can you clarify why this argument still makes sense for parameterized policies?\n\nWhen you compare with VIB variants, did you make sure that they use the same amount of computation and memory (e.g. network size) as RPC? If not, how did you make sure that this comparison is fair?\n\nIn Figure 5, the x-axis of the plots has episodes. Are they of the same length? If not, how is it a sensible choice to align in terms of episodes?\n\nAnd if those tasks have constant episode length, then it is a fixed horizon problem. Did you pass the remaining time steps as part of the observation? Otherwise, is the task from the agent’s perspective really well defined that we can describe coherently?\n\nWhy action repeat is a reasonable or interesting baseline?\n\nFew edit suggestions:\nLine 32: “minimizing bits” -> “minimizing the number of bits”\n\nLine 157: both Q and y are functions of psi. This means both will be differentiated. Is that what you did because that’s not what temporal difference learning does.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/AA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Many hours"}, "extra_text": {}}, {"review_id": "D6CLNdnh8Qg", "reviewer": "Reviewer_SVEk", "summary": "In this paper, the authors propose a new reinforcement learning method, called robust predictable control, that optimizes a policy under limited representational resources. Similar to previous methods, they formalize the problem in a fully observable MDP and learn a compressed representation of the state space using a variational information bottleneck constraint. Crucially, they apply the bottleneck on sequences of states. Through a clever factorization of the variational prior, they obtain a prior dynamics model in the compressed state space. This dynamics model is predictive and the agent only needs additional representational resources if the encoded states are not predicted by the prior dynamics already. Using a set of standard RL benchmarks, the authors show that this set up improves compression and enforces predictable and robust behaviors .\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "All agents face a fundamental problem: how much effort should they invest in order to achieve a certain outcome? In RL, the quality of an outcome can be described by the collected rewards, whereas the effort can be described by the informational expenses. The authors tackle this problem by considering an important informational expense, namely the complexity of the representation on which an agent bases its decisions. Low complexity representations are desirable because they are cheap (e.g. in terms of storage) but also because they tend to avoid overfitting. If agents don’t overfit they are more robust and can generalize better. The authors thus address a fundamental issue of decision-making and show that their method actually achieves improvements in compression, robustness, and generalization.\n\nWhile previous work has addressed the same problem in a similar setups (e.g. ref 25), the authors here propose a valuable extension. Instead of compressing states one-by-one, they compress state sequences as a whole and thus get a step closer to the actual problem faced by the agent. Furthermore, the predictive prior this approach yields is useful on its own as something akin to a default dynamics model and the authors showcase one application of this in hierarchical RL (fig. 5). Most importantly, their approach also enforces the agent to learn behaviors that are inherently predictive. I think this is a conceptually important point and they nicely show evidence for this in fig. 3. \n\nThe only issues I have are with clarity of exposition:\n\n- Eq. 2: It is unclear whether there is really equality here. In my understanding every sequence, or partially truncated sequence, has a certain mutual information associated. When taking this MI as a cost, we can sum them up similar to rewards and take into account the discount factor in the sense of a probability of a sequence continuing. When doing this we get the RHS, but this is not equal to the LHS. Furthermore, the prior that is factored autoregressively should read $m(z_{1:\\inty}|a_{1:\\infty})$, i.e. $m(z_{1:\\inty}|a_{1:\\infty}) = \\prod_t m(z_{t+1}|z_t,a_t)$, or otherwise one would have to sum over probability of actions.\n\n- Eq. 5: It is unclear where this objective is coming from and how it is optimized (I also couldn’t find the code on the provided website to check). The Q function is learned online by TD learning minimizing $L(\\psi)$. Since $L(\\theta)$ is not considering whole sequences, is it really a function depending on $s_{t-1}$, i.e. $L(\\theta,s_{t-1})$? And then this function is optimized for all $s_{t-1}$? It is unclear to me how this optimization procedure works, and how does it relate to a policy gradient approach as in ref. 25 or as in actor critic methods?\n\n- Some sloppy notation or typos. E.g. on bottom of page 4, the high-level policy is denoted $\\pi(a|s)$, but shouldn’t it be $\\pi(a|z)$ as in fig. 1? Similarly, in the variables over which the expectation is taken in eq. 5 it reads $z_{t-1} \\sim \\phi(z_{t-1})$ instead of $z_{t-1} \\sim \\phi(s_{t-1})$. On page 3 the prior is suddenly called $r$. In eq. 4 the regularized reward function should depend on $z_t, z_{t-1},a_{t-1}$ as well. These are just some I found.\n\n- It would be interesting to see some visualizations of the representations and dynamics that RPC finds. E.g. for the hierarchical tasks, $z$-space is three dimensional and could be visualized I assume. Visualizations would give a better intuition for how RPC exactly works.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Technical limitations are pointed out briefly. Although exactly how much performance in terms of return decays with compression is never really shown, but this might be beyond the scope of the publication. I could not find any discussion of societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12h  (joint review with a graduate student in the lab)"}, "extra_text": {"ethical_concerns": "I could not find any discussion of ethical concerns. While I do not see any particular ethical or societal concerns with this work, I think a small discussion would be useful, be it the authors just stating that these concerns don’t apply to their work and why."}}]

Author rebuttal:

Meta-review: All 4 reviewers suggested acceptance of the paper and the authors clarified all open questions and concerns in their rebuttal. Therefore I am recommending acceptance of the paper.
Decision: Accept (Spotlight)