Control-Aware Representations for Model-based Reinforcement Learning
Authors: Brandon Cui
Yinlam Chow
Mohammad Ghavamzadeh
Venue: ICLR 2021 Poster (2021)

Abstract: A major challenge in modern reinforcement learning (RL) is efficient control of dynamical systems from high-dimensional sensory observations.   Learning controllable embedding (LCE) is a promising approach that addresses this challenge by embedding the observations into a lower-dimensional latent space, estimating the latent dynamics, and utilizing it to perform control in the latent space.  Two important questions in this area are how to learn a representation that is amenable to the control problem at hand, and how to achieve an end-to-end framework for representation learning and control.  In this paper, we take a few steps towards addressing these questions. We first formulate a LCE model to learn representations that are suitable to be used by a policy iteration style algorithm in the latent space.We call this model control-aware representation learning(CARL). We derive a loss function and three implementations for CARL. In the offline implementation, we replace the locally-linear control algorithm (e.g., iLQR) used by the existing LCE methods with a RL algorithm, namely model-based soft actor-critic, and show that it results in significant improvement. In online CARL, we interleave representation learning and control, and demonstrate further gain in performance.  Finally, we propose value-guided CARL, a variation in which we optimize a weighted version of the CARL loss function, where the weights depend on the TD-error of the current policy. We evaluate the proposed algorithms by extensive experiments on benchmark tasks and compare them with several LCE baselines.

Published as a conference paper at ICLR 2021
CONTROL-AWARE REPRESENTATIONS FOR MODEL- BASED REINFORCEMENT LEARNING
Brandon Cui Facebook AI Research Dcui@fb ftw.ffbbcoo. com
Yinlam Chow Google Research yin wen.llmmmh@@@ogge...oo com
Mohammad Ghavanzadeh Google Research
ghavamza@gooolee com
ABSTRACT
A major challenge in modern reinforcement learning (RL) is efficient control of dynamical systems from high- dittemmiiiii sensory observations. Learning controllable embedding (LCE) is promising approach that addresses this challenge by embedding the observations into a power-dimensions latent space, estimating the latent dynamics, and utilizing it to perform control in the latent space. Two important questions this area are how to learn representation that amenable to the control problem at hand, and how to achieve an end-to-end framework for representation learning and control. In this paper, take few steps towards addressing these questions. We first formulate a LCE model to learn representations that are suitable to be used by policy iteration style algorithm in the latent space. We call this model control-awan representation learning (CARL). We derive loss function and three implementations for CARL. In the offline implementation, we replace the locally-linear control algorithm (c.g.. iLQR) used by the existing LCE methods with a RI. algorithm, namely model-based soft actor-critic, and show that it results in significant improvement. In online CARL, we interleave representation learning and control, and demonstrate further gain in performance. Finally, we propose value-guided CARL, a variation in which we optimize weighted version of the CARL loss function, where the weights depend on the TD-error of the current policy. We evaluate the proposed algorithms by extensive experiments benchmark tasks and compare them with several LCE basclines.
INTRODUCTION
Control of non-linear dynamical systems is a key problem in control theory. Many methods have been developed with different levels of success in different classes of such problems. The majority of these methods assume that a model of the system is known and its underlying state is low-dimensional and observable. These requirements limit the usage of these techniques in controlling dynamical systems from high-dimensional raw sensory data (e.g., image), where the system dynamics is unknown, scenario often seen in modern reinforcement learning (RL).
Recent years have witnessed rapid development of large arsenal of model-free RL algorithms, such as DON (Mnih et al.) 2013). TRPO (Schulman et al., 2015), PPO (Schulman et al.) 2017). and SAC (Haarnoj et al. 2018), with impressive success in solving high- digh-dimensional control problems. However, most of this success has been limited to simulated environments (e.g., computer games), mainly due to the fact that these algorithms often require a large number of samples from the environment. This restricts their applicability in real-world physical systems, for which data collection often a difficult process. On the other hand, model-based RL algorithms, such PILCO (Deisenroth & Rasmussen| 2011). MBPO (Janner et al.) 2019), and Visual Foresight (Ebert et al., 2018), despite their success, still face difficulties in learning model (dynamics) in a dine-dimensionnl (pixel) space.
To address the problems faced by model-free and model-based RL algorithms in solving high- dimensional control problems, a class of algorithms have been developed, whose main idea is to first learn din-eeenniionll latent (embedding) space and a latent model (dynamics), and then use this model to control the system in the latent space. This class has been referred as learning controllable embedding (L.CE) and includes algorithms, such as E22C (Watter et al., 2015). RCE (Banijamali et al.) (220), Dreamer (Hafner al. 2020a b). | (020)). The tollowing two properties are extremely important in designing LCE models and algorithms. First, to learn a representation that is the most
2018), SOLAR (Zhang et al. 2019), PCC (Levine
PC3 (Shu et al., 2020), and SLAC (ee al.
suitable for the control problem hand. This suggests incorporating the control algorithm in the

---

Published as a conference paper at ICLR 2021
process of learning representation. This view of learning control-aware representations is aligned with the value-aware and policy-aware model learning, VAML (Farahmand] (018) and PAML (Abachi et al. 2020), frameworks that have been recently proposed in model-based RL. Second, to interleave the representation learning and control, and to update them both, using a unifying objective function. This allows to have an end-to-end framework for representation learning and control.
LCE methods, such as SOLAR. Dreamer, and SLAC, have taken steps towards the second objective by performing representation learning and control in online fashion. This is in contrast to offline methods like E2C, RCE, PCC, and PC3 that leam a representation once and then use it in the entire control process. On the other hand, methods like PCC and PC3 address the first objective by adding term to their representation learning loss function that accounts for the curvature of the latent dynamics. This term regularizes the representation towards smoother latent dynamics, which are suitable for the locally-linear controllers, e.g., iI.QR (Li & Todorov, 2004), used by these methods.
In this paper, we take a few steps towards the above two objectives. We first formulate a LCE model to learn representations that are suitable to be used by a policy iteration (PI) style algorithm in the latent space. We call this model control-aware representation learning (CARL) and derive loss function for it that exhibits a close connection to the prediction, consistency, and curvature (PCC) principle for representation learning (Levine et al.) (020). We derive three implementations of CARL:: offline, online, and value- value-guided. Similar to offline LCE methods, such E2C, RCE, PCC, and PC3, in offline CARL, we first learn representation and then use it in the entire control process. However, in offline CARL, we replace the locally-lineal control algorithm (c.g., iLQR) used by these LCE methods with Pl-style (actor-critic) RL. algorithm. Our choice of RL algorithm is the model-based implementation of soft actor-critic (SAC) (Haarnoja et al. 2018). Our experiments show significant performance improvement by replacing iLQR with SAC. Online CARL is an iterative algorithm in which at each iteration, we first learn a latent representation by minimizing the CARL loss, and then perform several policy updates using SAC in this latent space. Our experiments with online CARL show further performance gain over its offline version. Finally, in we optimize a weighted version of the CARL loss function, in which the weights depend on the TD-error of the current policy. This would help to further incorporate the control algorithm in the representation learning process. We evaluate the proposed algorithms by extensive experiments on benchmark tasks and compare them with several LCE baselines: PCC, SOLAR, and Dreamer.
CARL (V-CARL),
2 PROBLEM FORMULATION
We are interested in learning control policies for non-linear dynamical systems, where the states E S C R not fully observed and we only have access their high-dimensional observations r E C R== nT >> nl. This scenario captures many practical applications in which we interact with a system only through high-dimensional sensory signals, such as image and audio. We assume that the observations x have been selected such that we can model the system in the observation space using a Markov decision process (MDP Mx MM = A, P. 7), where X' and A are observation and action spaces; r X x A + R is the reward function with maximum value Rmaxs defined by the designer of the system to achieve the control objective P P::a X' x A -> PP(2') is the unknown transition kernel; and y E (0, 1) is the discount factor. Our goal IS to find a mapping from observations to control signals, μ X + P(A), with maximum expected return, i.e., (1) E E 7'r( ax) P. nl)
2
Since the observations x are gigh--imensional and the observation dynamics P is unknown, solving the control problem in the observation space may not be efficient. As discussed in Section [] the class of learning controllable embedding (LCE) algorithms addresses this by learning low-dimensional latent (embedding) space Z C R" n2 & nu. together with a latent dynamics, and controlling the system there. The main idea behind LCE is to learn an encoder E X' - P(Z). a latent space dynamics F Z x A + PZZ), and a decoder D : Z + P(X) such that a good or optimal controller (policy) in Z performs well in the observation space X. This means that if we model the control problem in as a MDP Mz (Z, A. F, F, 7) and solve it using a model-based RL al algorithm to obtain a policy x 2 + P(A), the image of π back in the observation space, i.e.,
'A method to ensure ohservations are Markovian is to buffer them for several time steps Mnih et al. 2013 "For example, in a goal tracking problem in which the agent (rohot) aims finding the shortest path reach the observation goal Z.g (the observation corresponding to the goal state 8g); we may define the reward for each observation x as the negative of its distance to x... i.c.. || xy1 1!
'Some recent ICE models, such as PC3 (Shu et al. 2020), are advocating latent models without decoder. Although we are aware of the merits of such approach, we use decoder in the models proposed in this paper.
2

---

Published as a conference paper at ICLR 2021
Algorithm Latent Space Learning with Policy Iteration (LSLPI)
1: Inputs: _ _r(0) (
2: Initialization: ,0(0) random policy; D <- samples generated from ju (0;
3: for 0,1, de
4: Compute 7()) as the projection of ( in the latent space w.n.t. DKL (T
p): # 1 =s 7. a E36)
5: Compute the value function of (0)) and set V(a) = Vami:
policy evaluation (critic) policy improvement (actor)
6: Compute the greedy policy w.r.t. yl and set 7 = =]
7: Set (1=1)) = = o E
# projjctt the improved policy 4
(6) back into the observation space
8: Learn (E(+-1)
p((+1), ((64)
D(++1), ,,(+t) from D, (()) and =
# representation learning
9: Generate samples D(++1) {(11, at, nt, 1141)] from (141);
D DUDUD
10:
end for
(T (alx)) S2 dB(s)x)<(al2)) should have high expected return. Thus, the loss function to learn 2 and DE) from observations {(Ie, qz 1(++1) should be designed to comply with this goal.
This is why in this paper, we propose LCE framework that tries incorporate the control algorithm used in the latent space in the representation learning process. We call this model, control-aware rep- resentation learning (CARL). In CARL, we set the class of control (RI.) algorithms used in the latent space approximate policy iteration (PI), and more specifically soft actor-critic (SAC) (Haarnoja ctt al. 2018). Before describing CARL in details in the following sections, we present a number of useful definitions and notations here.
For any policy μ in X, we define its value function U₂ and Bellman operator Tj as
(2(2) = E (xt) Per To z],,
TxU](() = (oor-
(I)
for all a X and U X' RR where (x)) du(a x)r(x, a) and Pa (x((x))= du(aa(z) P(z',>
anypolicy T the reward function and dynamics induced by μ.. Similarly, for any policy т in Z, we de-
dr Sa dr (((22) a). We also define its value function Va and Bellman operator TT as
fine its induced reward function and dynamics as (z) (d(aa)=)(Z, and (=(2'(z) =
(2) E y (z)) Fx, 20
2],
= -E. (.
(2)
For any policy π and value function V in the latent space Z, we denote by o E and V E, their image in the observation space X, given encoder E, and define them as
(T ()(lz) = / d((z)))m(alz)
(Vo E) = (E(2)=)V(s).
(3)
3 CARL MODEL: A CONTROL PERSPECTIVE
In this section, we formulate our LCE model, which we refer to as control-aware representation learning (CARL). As described in Section CARL is a model for learning low-dimensional latent space Z and the latent dynamics, from data generated representation is suitable to be used by a policy iteration (PI) style algorithm in Z. In order to derive the loss function used by CARL to learn Z and its dynamics, i.e., (E, F. D.F). F), first describe how the representation learning can be interleaved with Pl in Z. Algorithm Caaiummmoooiii contains the pseudo-code of the resulting algorithm, which we refer to as latent space learning policy iteration (LSLPI).
the observation space X, such that this
Each iteration LSLPI starts with a policy (e(() in the observation space X, which the mapping
of the improved policy in Z in iteration i - 1, i.e., x (i-1) back in X' through the encoder ( (Lines (and We then compute (() the current policy in Z, as the image of (6) in Z through the encoder E (L. Note that E(+) is the encoder learned the end of iteration i 1 (Line
We then use the latent space dynamics F(i) learned at the end of iteration i (Line and
first compute the value function of (1) in the policy evaluation or critic step, i.c., v(t V
(6) (Line 5). and then use (0) compute the improved policy n the greedy policy w.r.t. V()
3

---

Published as a conference paper at ICLR 2021
Using the samples in the buffer D, together with the current policies in Z, i... x00) and # we learn the new representation F(++))))((++)),. p(++1), 1(++1) (Line 8p. Finally, we generate samples D(++1) by following ((++1, the image of the improved policy TT back in X' using the old encoder E (Line 7). and add it the buffer D (Line 19). and the algorithm iterates. is important to note that both critic and actor operate in the low-dimensional latent space Z.
Le., m((11) G[V in the policy improvement actor step (Line 6p.
LSLPI is a PI algorithm in Z. However, what desired is that also acts as PI algorithm in X, i.e., it results in (monotonic) policy improvement in X, i.e., U,(++ 2 Uy,00 Therefore, we define the representation learning loss function for CARL, such that ensures LSLPI also results in policy improvement in X. The following theorem, whose proof is reported in Appendix [A] shows the relationship between the value functions of two consecutive polices generated by LSLPI SnLPlin4 X.
Theorem 1. Let μs p+,, n, T4, and (E, F,DF) be the policies ,l), plit1) T(0,, ((0) and the learned latent representation (F(++1), p(++1), D(++1), fli+1) iteration i of the LSLPI algorithm (Algorithm Z.. Then, the following holds for the value functions of μ and p4:
UP ( Un(2)
Σ (E{
E D,F,
2 Rmaxx
V DKLl(T E)(()) II (((1))) (mo =t), noe
(4)
Edy
1
(F
for all x E X, where di L (1'10)) = (1-7) -
P(re = ao: π o E) is the y-stationary
distribution induced by policy E, and the errur term A for policy is given by
(E.,.)
(R
Riaax F,,,,,,,
V
log D(rls) (TTE(())
(E(2|z)r+<2))
(5)
+ Rmax
(Vara Dec(Pnoe(( (I - + v Dia((B (-0 (Fr B)+2z)
2211 7))
(IV)
It is easy on the RHS of 4 are zero. We now describe these terms. The last term on the RHS of (4) is the KL between x(²)
that LSLPI guarantees (policy) improvement in X, if the terms in the parentheses
and (0) (() E((), This term can be seen as a regularizer keep the new encoder E close to the old one ( The four terms in (5) are: (I) The incoding-decoding error to ensure x R (D o (z); (II) The error that measures the mismatch between the reward of taking action according to policy π E at E X, and the reward of taking action according to policy æ at the image of and Z. This is the error between x' and i' shown in Fig. [ [a): and (IV) The error in predicting the next latent state through paths in X' and Z. This is the error between z' and E' shown in Fig, [b).
Z under E: (III) The error in predicting the next observation through paths X'
F
Fr
E
D
②
a
Prof
ProE
(a)
(b)
Figure 1: (a) Paths from the current observation to the next one, (left) in X' and (right) through 2. (b) Paths from the current observation to the next latent state, (left) through X' followed by encoding and (right) starting with encoding and then through Z.
Representation Learning in CARL Theorem provides us with a recipe (loss function) to learn the latent space Z and (E, D) In CARL, we propose to learn a representation for which the terms in the parentheses on the RHS of are small. As mentioned earlier, the second term,
4

---

Published as a conference paper at ICLR 2021
Leeg (E,g(E,H f4, n. 22), can be considered regularizer to keep the new encoder E close to the old one E_, when the policy μ is given by π E_. Term (I) minimizes the reconstruction error between encoder and decoder, which is standard for training auto-encoders (Kingma & Welling 2013). Term (II) that measures the mismatch between rewards can be kept small, or even zero, if the designer of the system selects the rewards in a compatible way f Although CARL allows us to learn a reward function in the latent space, similar to several other LCC works (Watter et al. 2015; Banijamali et al.) 2018 Levine itt al., 2020; Shu et al. 2020), in this paper, we assume that a compatible latent reward function is given. Terms (II)) and (IV) are the equivalent of the prediction and convistency terms in PCC (Levine) ctt al,, 2020) for particular latent space policy т. Since PCC has been designed for an offline setting (i.e., one-shot representation learning and control), its prediction and consistency terms are independent of a particular policy and are defined for state-action pairs. While CARL is designed for an online setting (i.e., interleaving representation learning and control), and thus, its loss function at each iteration depends on the current latent space policies and T As we will see in Section in our offline implementation of CARL., these two ferms are similar to prediction and consistency terms in PCC. Note that (IV) is slightly different than the consistency term PCC. However, we upper-bound it using Jensen inequality: (IV) I Le(E, F. 2) == Jarex dl's (x' [z)) Dra. (E() II (F' o (+xr)) the resulted loss, Le (E, F, T, would be similar to the consistency term in PCC. Similar to PCC, we also add curvature loss to the loss function of CARL to encourage having smoother latent
space dynamics F.. Putting all these terms together, we obtain the following loss function for CARL:
min E,F,D
Σ Naa (D,,)) Xp F, A.
(6)
(Fuu x) + rrgg H.
where (Aod, Ap. de: dear A are 1 of the algorithm, (Led. Lp) are the encoding- decoding and prediction losses defined in (5), L is the consistency loss defined above, Towr Ez,.. [IS, [fz(= + c., (.. f2(2, u) (V.fz(z, u) + V..fa(z,u))..
E]] is the curvature loss
that regulates the 2nd derivative of fz. the mean of latent dynamies F, in which Eg, are standard Gaussian noise, and Treg is the regularizer that ensures the new encoder remains close to the old one.
DIFFERENT IMPLEMENTATIONS OF CARL
The CARL loss function in (6) introduces optimization problem that takes a policy in Z as input and learns a representation suitable for its evaluation and improvement. To optimize this loss
in practice, similar to the PCC model (Levine et al., 2020), we define DD o E as latent
variable model that is factorized as P(r 24, 2++1 T4. T) P(z. P(2.41/2,,
and use variational approximation to the interactable negative log-likeldhood of the loss terms in (6).
The variational bounds for these terms can be obtained similar to Eqs. and in Levine et al. (020))
Below we describe three instantiations of the CARL model in practice. Implementation details can be found in Algorithm 2 in Appendix D Although CARL is compatible with most Pl-style (actor-critic)
RL algorithms, we choose soft actor-critic (SAC) (Haarnoja et al. 2018 as its control algorithm. Since most actor-critic algorithms are based on first-order gradient updates, as discussed in Section 3 we regularize the curvature of the latent dynamics F (see Eqs. 8 and 9 in Levine et al. 2020) in CARL to improve its empirical stability and performance in policy learning.
1. Offline CARL We first implement CARL in an offline setting, where we generate a (relatively)
large batch of observation samples ( dz, Tt, Zx+ J using an exploratory (c.g., random) policy.
We then use this batch to optimize the CARL's loss function (6) via the variational approximation
scheme described above, and Iearn a latent representation Z and (E, F, D). Finally, we solve the decision problem in Z using a model-based RL algorithm, which in our case The learned policy #* in 2 is then used to control the system from observations as Or ~ (#* o E (- I
modell-baed SACE
This is the setting that has been used in several recent LCE works, such as E2C (Watter et al. 2015
RCE (Banijamali et al. 2018). PCC (Levine et al. 2020), and PC3 (Shu et al. 2020). Our offinee
$ 'For example, in goal- based RL problems, a compatible reward function can be the one that measures the negative distance between a latent state and the image of the goal in the latent space. 'Theorem
| provides high-level guideline for selecting the hyper- parameters of the loss function: Aet
2 Rmaax/(1 (1 ) X = >o = Rm2y - -))², and Areg = 2 Rmax/(1 - -7).
'By model-based SAC, we refer to learning latent policy with SAC using synthetic trajectories generated
by unrolling the learned latent dynamics model F, similar the MBPO algorithm (Janner et al. 2019).
5

---

Published as a conference paper at ICLR 2021
implementation is different than those in which 1) we replace their locally-linear control algorithm, namely iterative LQR (iLQR) (Li & Todorov] 2004). with model-based SAC, which results in significant performance improvement, as shown in Section [ and 2) we optimize the CARL loss function, that despite close connection, is still different than the one used by PCC.
The CARL. loss function presented in Section 3 has been designed for an online setting in which at each iteration, it takes policy as input and learns a representation that is suitable for evaluating and improving this policy. However, in the offline setting, the learned representation should be good for any policy generated in the course of running the Pl-style control algorithm. Therefore, we marginalize out the policy from the (online) CARL'S loss function and use the RHS of the following corollary (proof in Appendix BD to construct the CARL loss function used in our offline experiments.
Corollary 2. Let μs and p.+ be two conseculive policies in X generated by Pl-style control algorithm in the latent space constructed by (E.F,D.)). Then, the following holds for the value functions of μ and pt., where A is defined by (5) (in modulo replacing sampled action a To E with action a):
(x) (x) 1-Y
max EEE,
D,F,F,D, Ve X.
2. Online CARL In the online implementation of CARL, at each iteration i, the current policy (() is the improved policy of the last iteration, .r (1-1) We first generate relatively (to offline CARL) small batch of samples using the image of the current policy in X, i.e., 11()) x(()) o E(i-1), and then learn a representation (E()) F(),, DG1) suitable for evaluating and improving the image of (ممر in Z under the new encoder E00, This means that with the new representation, the current policy that was the image of ( in Z under EG-"), should be replaced by its image (9)) under the new encoder, i... (()) E se (() In online CARL. address this by the following policy distillation step in which we minimize the following loss
n E arg min
Σ aND
DKL ((TT a (-)))))))) Il (+4-1)) o ((i=)))((=))-
(8)
After the current policy (1) is set, we perform multiple steps of (model- SAC in 2 using the current model, (F()), r()), and then send the resulting policy T. to the next iteration.
(i)
3. Value-Guided CARL (V-CARL) While Theorem D shows that minimizing the loss in 66 guar- antees performance improvement, this loss does not contain any information about the performance of the current policy jr, and thus, the LCE model trained with this loss may have low accuracy in regions of the latent space that are crucial for learning good RL policies. In V-CARL, we tackle this issue by modifying the loss function in a way that the resulted LCE model has more accuracy in regions with higher anticipated future returns.
To derive the V-CARL's loss function, (VMBPO) framework by [Cow et al.] (2020) in which the optimal dynamics for model-based RL can be expressed in closed-form
use the variational model-based policy optimization
(= Wy((,,))) where (x)x)) := 1 log E [exp (7 (r
P(x [z, a) exp (f(r(z, a) 70,((a)
[Pa, xo = x]] and Wa(2,, a) = r(x, a) +
log Et.mp())nn)) [exp(TU,, (xx))) are the optimistic value and action-value functions of policy μ. and 7 > 0 is a temperature parameter. Note that in the VMBPO framework, the optimal dynamics p' is value-aware, because it re-weighs P with an exponential-wwisting weight exp( [u(2 a, x')).
where
= r(z,a) + (a)) - W, a) is the temporal difference (TD) error.
In V-CARL, we use the VMBPO framework to modify the CARL's prediction loss L, F. D. n. r). Since the regularizer loss Leg (E, u4, x) in CARL forces policies я o E and p to be close to each other, we may replace the transition dynamics Preg with P, in Lp- This makes minimizing Lp equivalent to maximizing the log-likelihood f,, d.P,. (x' -log(D [ o F, (x(((x)) Finally, we replace P, with Pt in this log-likelihood and obtain J. du(alz) [ (1 a) exp(? w(2, a, x')) log Do F. o E2 ()). which is a weighted (by the exponential TD ((x,,,x") )) log-likelihood function (w.r.t. P). Note
show that adding distillation improves the performance in online CARL. Thus, all our results for online CARL and V-CARL. unless mentioned, are with policy distillation. We refer to U2 as the optimistic value function RRuszzyyński & Shapiro 2006). because it models the right tail of the return via the exponential utility pr (U(()
Our experiments reported in Appendix show
[z, x.a) i log E (exp(T- | U(=')))

---

Published as a conference paper at ICLR 2021
that this weight depends on the optimistic value functions Un, and WH. When > 0 is small (see Appendix [ more details), these value functions can be approximated by their standard counterparts, ie., U,, (a)) Rs U₁, (a) and
(x) a) [ (zU,,)"), which can be further approximated by their latent-space counterparts, i.e., U,(2) R (V₂ o Ex)x) and (1, a) R (Q+ (2, a). according to Lemma [] Appendix A.I]] Since the latent reward function r is defined such that r3z, a) # (F o (2, a), we may write the TD-error waa a, in terms of the encoder E and the latent value functions as ((x a,x') := Sa.2' dE(z)æ) dE(2' (e)) (r(2,a) Q- + +V, (:))..
5 EXPERIMENTAL RESULTS
In this section, we experiment with the following continuous control domains: (i) Planar System, (ii) Inverted Pendulum (Swingup), (iii) Cartpole, (iv) Three-link Manipulator (3-Pole), and compare the performance of our CARL algorithms with three LCE baselines: PCC (Levine et al.) 2020). SOLAR (Zhang et al. 2019). SLAC (Lee et al. 2020), and two implementations of Dreamer (Halner ct al. 2020a) (described below) PI These tasks have underlying start and goal states that are "not" observable, instead, the algorithms only have access to the start and goal observations. We report the detailed setup of the experiments in Appendix [L] in particular, the description of the domains in Appendix E. and the implementation of the algorithms in Appendix
E.3
To evaluate the performance of the algorithms. similar to Levine et al. (2020), we report the %- time spent in the goal. The initial policy that is used for data generation is uniformly random (see Appendix E.2 for more details). To mcasure performance reproducibility for each experiment, we (i) train 25 models, and (ii) perform 10 control tasks for each model. For SOLAR, due to its high computation cost, we only train and evaluate 10 different models. Besides the average results, we also report the results from the best LCE models, averaged over the 10 control tasks.
General Results Table Talleeshaee shows the means and standard errors of %-time spent in goal, averaged over all models and control tasks, and averaged over all control tasks for the best model. To compare data efficiency, we also report the number of samples required to train the latent space and controller in each algorithm. We also show the training curves (performance vs. number of samples) of the algorithms in Fig. 2 We report more experiments and ablation studies in Appendix F
Below summarizes our main observations of the experiments. First, offline CARL that uses model- based SAC as its control algorithm achieves significantly better performance than PCC that uses iLQR in all tasks. This can be attributed the advantage that SAC is more robust and effective in non- (locally)-linea environments. We report more detailed comparison between PCC and offline CARL in Appendix F.3] where we explicitly compare their control performance and latent representation maps. Second, in all tasks, online CARL is more data-efficient than its offline counterpart, i.e., achieves similar or better performance with fewer samples. In particular, online CARL is notably superior in Planar, Cartpole, and Swingup, in which it achieves similar performance to offline CARL with 2, 2.5, and 4 times less samples, respectively (see Fig. 2i.. In Appendix F.3 we show how the latent representation of online CARL progressively improves through the iterations of the algorithm (in particular, see Fig. L... Third, in the simpler tasks (Planar, Swingup. Cartpole), V-CARL performs even better than online CARL. This corroborates our hypothesis that CARL can achieve extra improvement when its LCE model is more accurate in the regions of the latent space with higher temporal difference (regions with higher anticipated future return). In 3-pole, the performance of V-CARL is worse than online CARL. This is likely due to the instability in representation learning resulted from sample variance amplification by the exponential- TD weight. Fourth, SOLAR requires significantly more samples to learn reasonable latent space for control, and with limited data fails to converge to a good policy. Even with the fine-tuned latent space from Zhang et al. (2019). its performance is incomparable to those of CARL variants and Dreamer. We report more experiments with SOLAR in Appendix F.3] in which we show that SOLAR can perform better, especially in Planar when we fix the start and goal locations. However, the improved performance is still incomparable with those of CARL and Dreamer. Fifth, include ablation study in Appendix F2 to demonstrate how each term of the CARL's loss function impacts policy learning. shows the importance of the prediction and consistency terms, without which the resulting algorithms struggle, and the (relatively) minor role of the curvature and encoder-decoder terms in the performance of the algorithms.
We did not include E2C and RCE in our experiments, because Levine al. 2020 has previously shown that PCC outperforms them.
7

---

Published as a conference paper at ICLR 2021
Dreamer As described in Section 2 most LCE algorithms, including E2C, PCC, and CARL variants, assume the observation space X' is selected such that the system Markovian there. In contrast, Dreamer does not make this assumption and has been designed for more general class of control problems that can be modeled as POMDPs. Thus, it is expected that it performs inferior (requires more samples to achieve the same performance) to CARL when the system is Markov in the observation space. Moreover, CARL and other LCE methods define the reward as the negative distance the goal in the latent space. This cannot be done in Dreamer, where the encoder is an RNN that takes am entire observation trajectory as imput. To address this, we propose two methods to train the Dreamer's reward function in the latent space, which we refer to as Dreamer Pixel and Dreamer Oracle. While Dreamer Pixel uses the negative distance the goal in the observation space X' as the signal to train the reward function, Dreamer Oracle uses the negative distance in the (unobserved) underlying state space S. Thus, it is more fair to compare the CARL algorithms with Dreamer Pixel than Dreamer Oracle that has the advantage of having access to the underlying state space (see Appendix F.6 for more details). As it was expected, our results show that although both Dreamer's s implementations learn reasonably-pertie -prforming policies for most tasks (except Planar), they require twice to 100-times more samples to achieve the same performance as the CARL algorithms. We report longer (more samples) experiments with Dreamer on all tasks in Appendix F.6 (Fig.
103
103
70
Ooling CAB
100
93
14b0
S000
Nimber dherrrmment Semples
Nimber Covranment Samples
Mambera Envrenment Sampres
Number Cevenomentt Samplex
(a) Planar
(b) Swingup
(c) Cartpole
(d) Three-pole
Figure 2: Training curves offline CARL, online CARL, V-CARL, and two implementations of Dreamer. The shaded region represents mean = standard error.
Environment
Algorithm
Number of Samples
Avg Go-Goa
Best %-Go
Planar
PCC
5000
38.85 1 2.45
62.5 1 10.42
Planar
Offline CARL
5000
63.43 = 2.78
79.51 0.38
Planar
Online CARL
3072
68.03 = 1.69
79.02 + 0.38
Planar
V-CARL
3200
71.05 1 1.46
79.51 +: 0.38
Planar
SOLAR
5000 (VAE) 16000 (Control)
5.82 2.50
9.13 1 3.54
Swingup
PCC
5000
86.60 = 1.00
97.40 - 0.61
Swingup
Offline CARL
5000
88.43 1: 2.02
98.50
0.0
Swingup
Online CARL
1408
95.04 = 0.96
98.50
0.0
Swingup
V-CARL
1408
96.50 = 0.25
98.50 = 0.0
Swingup
SOLAR
5200 (VAE) + 40000 (Control)
16.1 = 0.69
22.45 + 1.96
Swingup
Dreamer Pixel
180895
70.35 + 0.62
98.5 + 0.0
Swingup
Dreamer Oracle
183084
94.65 = 0.20
98.25 0.0
Cartpole
PCC
10000
83.64 = 0.63
100,0 1: 0.0
Cartpole
Offline CARL
10000
91.11 = 1.50
100.0 = 0.0
Cartpole
Online CARL
5120
95.34 = 1.17
100.0
0.0
Cartpole
V-CARI
5T20
95.79 + 1.06
100.0
0.0
Cartpole
SOLAR
(VAE) + 40000 (Control)
10.61 = 2.58
12.33
2.96
Cartpole
Dreamer Pixel
96941
95.59 = 3.77
100.0 1: 0.0
Cartpole
Dreamer Oracle
14474
97.77 = 1.525
100.0 + 0.0
Three-pole
PCC
4096
4.41 = 0.75
36.20 + 7.06
Three-pole
Offline CARL
4096
63.20 = 1.77
88.55
0.0
Threc-pole
Online CARI.
2944
62.17 - 2.28
90.05
0.0
Three-pole
V-CARL
2816
55.06 1: 2.42
89.05 0.0
Three-pole
SOLAR
2000 (VAE) + 20000 (Control)
011
010
Threc-pole
Dreamer Pixel
6245
61.93 = 2.30
90.00 + 0.0
Three-pole
Dreamer Oracie
6245
71.07 = 2.45
88.40 0.0
Table 1: Mean + standard error results (%-goal) and samples used for different LCE algorithms.
Results with dnvironment-biased Sampling In the previous experiments, all the online LCE algorithms are warm-started with data collected by uniformly random policy the entire
8

---

Published as a conference paper at ICLR 2021
environment. With sufficient data the latent dynamics is accurate enough most parts of the state space for control, therefore we do not observe a significant difference between online CARL and V- CARL. To further illustrate the advantage of V-CARL over online CARL, we modify the experimental setting by gathering initial samples only from specific region of the environment (see Appendix [E.]] more details). Fig. showws the learning curves of online CARL and V-CARL this case. As expected, with biased data, both algorithms experience certain level of performance degradation, yet, V-CARL clearly outperforms online CARL this verifies our conjecture that control-aware LCE models are more robust to initial data distribution and superior in policy optimization.
Pecceraap of steps Encilal
2500 3250 10 soon bocal
Guibet
Perorragp
10
Yatao
2000 2250 2550 2750 1000 3250
1005
1520
2002
2502
3500
6000
5000
1033 1350
2020
Number cormmneentt Semples
Number Etvrrmment Samples
Number Erchroment Samples
Number Envirammnt Samples
(a) Planar
(b) Swingup
(c) Cartpole
(d) Three-pole
Figure 3: Training curves of Online CARL and V-CARL with environment-biasee initial samples.
6 CONCLUSIONS
In this paper, we argued for incorporating control representation learning process and for the interaction between control and representation learning in learning controllable embedding (LCE) algorithms. We proposed LCE model called control-aware representation learning (CARL) that learns representations suitable for policy iteration (PI) style control algorithms. We proposed three implementations of CARL that combine representation learning with model-based soft actor-critic (SAC), as the controller, in offline and online fashions. In the third implementation, called value- guided CARL, we further included the control process in representation learning by optimizing a weighted version of the CARL loss function, in which the weights depend on the TD-error of the current policy. We evaluated the proposed algorithms on benchmark tasks and compared them with several LCE baselines. The experiments show the importance of SAC as the controller and of the online implementation. Future directions include 1) investigating other Pl-style algorithms in place of SAC, 2) developing LCE models suitable value iteration style algorithms, and 3) identifying other forms of bias for learning an effective embedding and latent dynamics.
REFERENCES
R. Abachi, M. Ghavamzadeh, and A. Farahmand. Policy-aware model learning for policy gradient methods. preprint UrXiv:2003.000300 2020.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. arXiv preprint arXiv: 705. 10528, 2017.
E. Banijamali, R. Shu, M. Ghavamzadeh, H. Bui, and A. Ghodsi. Robust locally-linear controllable embedding. In Proceedings of the Twenty First International Conference on Artificial Intelligence and Statistics, pp. 1751-1759, 2018.
V. Borkar. Q-learning for risk-sensitive control. Mathematics of operations research, 27(2):294-311, 2002.
M. Breivik and Fossen. Principles of guidance-based path following and 3D. Proceedings of the 44th IEEE Conference on Decision and Control, pp. 627-634, 2005.
Y. Chow, B. Cui, M. Ryu, and M. Ghavamzadeh. Variational model-based policy optimization. In arXiv, 2020.
M. Deisenroth and C. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on Machine Learning, pp. 465-46444 472, 2011.
F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual Foresight: Model-based deep reinforcement learning for dision-based robotic control. preprint arXiv: 18/2. 00568, 2018.

---

Published as a conference paper at ICLR 2021
A. Farahmand. Iterative value-aware model learning. In Advances in Neural Information Processing Systems 31, Pp- 9072-9083, 2018.
A. Farahmand, A. Barreto, and D. Nikovski. Value-aware loss function for model- reinforcement learning, Artificial Intelligence and Statistics, pp. 1486-1494, 2017.
K. Furuta, M. Yamakita, and S. Kobayashi. Swing up control of inverted pendulum. In Proceedings of International Conference Industrial Electronics, Control and Instrumentation, pp. 2193 2198, 1991.
S. Geva and J. Sitte. A cartpole experiment benchmark for trainable controllers. IEEE Control Systems Magazine, 13(5):40-51) 51, 1993.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning. pp. 1861-1870, 2018.
D. Hafner, T. Lillicrap. J. Ba, and M. Norouzi. Dream to control:
Learning behaviors by latent imagination. International Conference on Learning Representations, 2020a.
D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02199 2020b.
M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimiza- tion. In Advances in Neural Information Processing Systems 32, pp.
2019.
D. Kingma and J. Ba. Adam: A method for stochastic optimization, 2014.
D. Kingma and M. Welling. Auto-encoding variational bayes, 2013.
X. Lai, A. Zhang, M. Wu, and She. Eegularity-avoidiin swing-up control for underactuated three-link gymnast robot using virtual coupling between control torques. International Journal of Robust and Nonlinear Control, 25(2):207-221, 2015.
A. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic; Deep reinforcement learning with latent variable model. Advances in Neural Information Processing Systems, 33, 2020.
N. Levine, Y. Chow, R. Shu, A. Li, M. Ghavamzadeh, and H. Bui. Prediction, consistency, curva- ture: Representation learning for locally-lines control. Proceedings of the 8th International Conference on Learning Representations, 2020.
W. Li and E. Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO (I), pp. 222-229,20 2004.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. preprint 1311v:1312.5000 2013.
A. Ruszczyúski and A. Shapiro. Optimization of convex risk functions. Mathematics of operations research, 31(3):433-455 2006.
J. Schulman, Proceedings of the 32nd International Conference Machine Learning, pp. 1889-1897, 2015.
S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. preprint arXiv:1707 06177,2017.
R. Shu, T. Nguyen, Y. Chow, T. Pham, K. Than, M. Ghavamzadeh, S. Ermon, and H. Bui. Predictive coding for locally-linear control. preprint arXiv:2003.0000 00006, 2020.
M. Spong. The swing up control problem for the acrobot. IEEE Control Systems Magazine, 15(1): 49-55, 1995.
M. Watter, Springenberg, J. Boedecker, and M. Riedmiller. Embed control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems 28, pp. 2746-2754. 2015.
M. Zhang, Vikram, Smith, Abbeel, M. Johnson, and Levine. SOLAR: Deep structured representations for model- based reinforcement learning. In Proceedings of the 36th International Conference Machine Learning, pp. 7444-7453. 2019.
10


Reviews:
[{"review_id": "okBn6BHrzZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines the problem of learning controllable embedding (LCE), with the goal of learning good representations (usually achieved using variational inference algorithms) such that the maximum cumulative reward can be achieved. The main difference lies in the simultaneous learning of both the low-dimensional latent space as well as the action policy.\n\nOne of the main strengths of this paper is found in Theorem 1. The authors devise a simple policy iteration approach in the low dimensional learned space. Then, using mostly qualitative analysis, a bound on the policy improvement error is formulated. This error combines several intuitive and straightforward factors, which are then extracted to form more involved offline and online reinforcement learning algorithms. I have read through the proofs, and they seems correct.\n\nWhile I appreciate the quality of the theoretical work, the paper had some drawbacks that brought me to my current score :\n1. The loss function consists of many hyper-parameters. The authors should provide some guidelines for choosing these hyper-parameters due to the large number of possible combinations, and clearly state how the scalings affect performance.\n2. Experimentation is lacking. While the authors conducted experiments mostly on toy problems, I expect them to compare against more involved environments which are harder to model. Their comparison with state of the art algorithms (e.g. Dreamer) which were also tested on such environments is thus not fair.\n3. Minor comment: There is a newer version of Dreamer that the authors can compared against: https://arxiv.org/pdf/2010.02193.pdf \n4. Minor comment: How would CARL compare against model-free offline RL methods, or generally to algorithms that are not SAC?\n\nQuestion to authors:\nWould there be a benefit in removing F altogether and learning a mapping X -> Z -> X’ without transitioning in the latent space? (i.e., errors III and IV in Theorem 1)\n\nFinally, it would be beneficial if the authors could include code for their work. If the authors can't supply the complete code base, even code snippets with clarifying explanations to demonstrate their main ideas would be beneficial. This would greatly improve the quality and credibility of their work as well as the reviews.\n\nTo conclude, the paper provides strong theoretical intuition, which is a significant value-add of the paper. Nevertheless, its lack of experimentation and large number of hyper-parameters limit its overall quality. If the authors provide substantial improvement in the experimentation I will increase my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "o67CDKifq89", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to address an important question in reinforcement learning: policy learning from high-dimensional sensory observations. The authors propose an algorithm for Learning Controllable Embedding (LCE) based on policy iteration in the latent space. The authors provide a theorem to show how the policy performance in latent-space policy improvement depends on the learned representation and develop three algorithmic variations that attempt to maximize the theoretical lower bounds. In the experiments, the proposed algorithm CARL shows improved performance when compared with other LCE baseline algorithms. \n\nWhile I'm not particularly familiar with the field of LCE, I think the idea of learning a representation that is suitable for policy improvement is an interesting idea. The readability of this paper is also pretty good, which can be difficult to get right because the of the correspondence between the original space and the latent space. Overall the paper is easy to follow. \n\nWhile I do think Algorithm 1 is reasonable, I found its theoretical foundation, namely Theorem 1, is incorrect. In the proof of Theorem 7 on p15 in the appendix, I do not think the implication T^2 VE(x) < T VE(x) + \\gamma Delta(x) for all x, would hold. Because Bellman operator contracts in the L-inf norm, a basic inequality would rather take a form of  T^2 VE(x) < T VE(x) + \\gamma sup_y Delta(y). In addition to this, another minor error happens in the first equation on pg 16, where I believe the correct right hand side would be 1/(1-gamma) sup_y Delta(y), without the gamma dependency.\n\nHowever, a bound that depends on L-inf norm would be quite bad for Theorem 1, and current data collection process in Alg 1 is not sufficient for minimizing it. I think it might be possible not using an L-inf bound but using an expected error based on the policy's rollout distribution. However, this change would largely change the theoretical results, and perhaps the motivation or details of the algorithm design. Therefore, I do not think the paper is ready for acceptance at the current stage without a large revision. If the authors can address this question properly, I would raise my score.\n\nBeyond the flaw in the theory, there are some parts which can benefit from some clarification: \n1. In the offline CARL, how does the algorithm address the issue of out of distribution error due to using a batch dataset? \n2. The authors argue that the loss here is different from PCC many times in the paper, but they never explain whether the choice here is better (or in which way). \n3. In line 4 of Alg 1, how do we ensure such pi would exist?\n4. What is the definition of \"compatible reward function\" in the last paragraph on p4?\n5. For completeness of presentation, please include the definition of curvature loss.\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hyW1G_q-8aI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new representation learning + RL algorithm called CARL, with a specific objective for learning a latent representation and dynamics model coupled with SAC policy learning in the latent space. Experiments on a few domains show CARL outperforming previous algorithms such as DREAMER and PCC.\n\nPros:\n\n+ The key points of the paper are relatively well organized and motivated properly.\n+ The experimental results succinctly demonstrate the promise of the proposed approach.\n\nCons:\n\n- It is difficult to follow important details about the operation of this relatively complicated method.\n- The experimental results are not sufficient for this largely empirical work.\n\nWith these pros and cons in mind, I am recommending a weak reject. See below for additional detailed comments.\n\nEDIT: After discussion, I have increased my score and am recommending weak accept. See the discussion with the authors for details.\n\n\nQuality\n---\n\nThe paper studies an important problem, proposes a novel solution, and has promising experimental results. However, the main drawback in terms of the quality of the work is that the results are not complete enough. For work that is empirically driven, I do not view the current results as sufficient for publication.\n\nIn particular, DREAMER appears to be competitive with CARL on a few domains. But DREAMER was also evaluated much more broadly across many tasks from the DeepMind control suite, indicating a level of robustness and performance that is, at best, hinted at in this work for CARL. A wider suite of experiments, for example using the same tasks as DREAMER, would go a long way in better shaping the reader's understanding of the proposed method.\n\nClarity\n---\n\nAs mentioned, the main points of the paper are presented well. The problem is properly motivated, and a central theorem gives rise to the proposed representation learning method. I did not check the proof for this theorem, but it appears sensible.\n\nHowever, the finer points in the paper, which are also very important, are difficult to follow. For example, what is \"model-based SAC\"? There does not appear to be a proper explanation or citation for this. Is the learned dynamics model F used in some way to learn the Q-function? Is this novel, or is it from prior work?\n\nConsidering the proposition that replacing other control algorithms in the latent space with model-based SAC is important for the overall performance improvement, a description of model-based SAC is important. Furthermore, an ablation study would be helpful in terms of understanding the relative importance of this change vs the proposed representation learning approach, which seems to be the novel part.\n\nSome other minor concerns about the methodological sections: there are many hyperparameters and not much guidance as to how to pick these; more discussion of why there are different versions of CARL and what are their respective strengths and weaknesses would be useful, especially for VCARL; I personally found the last paragraph of the VCARL description almost impossible to follow.\n\nOriginality\n---\n\nTo the best of my knowledge, the representation learning algorithm itself is novel. Perhaps a related work that is overlooked is https://arxiv.org/abs/1907.00953, which apparently has been accepted to NeurIPS 2020 but has been out for some time. At a high level, this work also incorporates representation learning into SAC, though the underlying details are different. Still, this approach seems actually more closely related than some of the current citations and comparisons, e.g., SOLAR and DREAMER. At least a citation seems to be in order, and preferably a comparison. Indeed, this prior work also carries out a more comprehensive evaluation on more tasks than the current work.\n\nSignificance\n---\n\nThis work has the potential to be significant, as many researchers and practitioners are currently interested in how to make deep RL more efficient and performative, in particular in visual settings. However, without a more comprehensive evaluation, it is difficult to judge for sure.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the question of RL in high-dimensional spaces by learning lower-dimensional representations for control purposes. The work contains both theoretical and empirical results that shows the promise of the proposed approach.

While the reviewers had initial concerns, including with a problem in a proof and questions around the contributions, after robust responses and discussions this paper is now in good shape.
Decision: Accept (Poster)