Provable Representation Learning for Imitation with Contrastive Fourier Features
Authors: Ofir Nachum
Mengjiao Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representation Learning
Imitation Learning
Contrastive Learning
Reinforcement Learning

Abstract: In imitation learning, it is common to learn a behavior policy to match an unknown target policy via max-likelihood training on a collected set of target demonstrations. In this work, we consider using offline experience datasets -- potentially far from the target distribution -- to learn low-dimensional state representations that provably accelerate the sample-efficiency of downstream imitation learning. A central challenge in this setting is that the unknown target policy itself may not exhibit low-dimensional behavior, and so there is a potential for the representation learning objective to alias states in which the target policy acts differently. Circumventing this challenge, we derive a representation learning objective that provides an upper bound on the performance difference between the target policy and a low-dimensional policy trained with max-likelihood, and this bound is tight regardless of whether the target policy itself exhibits low-dimensional structure. Moving to the practicality of our method, we show that our objective can be implemented as contrastive learning, in which the transition dynamics are approximated by either an implicit energy-based model or, in some special cases, an implicit linear model with representations given by random Fourier features. Experiments on both tabular environments and high-dimensional Atari games provide quantitative evidence for the practical benefits of our proposed objective.

Provable Representation Learning for Imitation with Contrastive Fourier Features
Ofir Nachum Google Brain of cirnachuu@googge... com
Mengjiao Yang Google Brain sherryy@google.com com com
Abstract
In imitation learning, is common to learn behavior policy to match an unknown target policy via max-likelihood training on collected set of target demonstrations. In this work, consider using offline experience datasets potentially far from the target distribution to learn low-dimensional state representations that provably accelerate the sample-effciency of downstream imitation learning. central challenge in this setting is that the unknown target policy itself may not exhibit ewwddimesiionll behavior, and so there is potential for the representation learning objective to alias states in which the target policy acts differently. Circumventing this challenge, we derive a representation learning objective that provides an upper bound on the performance difference between the target policy and a low- dimensional policy trained with max-likelihood, and this bound is tight regardless of whether the target policy itself exhibits low-dimensional structure. Moving the practicality our method, we show that our objective can be implemented as contrastive learning, in which the transition dynamics are approximated by either an implicit energy-based model or, in some special cases, an implicit linear model with representations given by random Fourier features. Experiments both tabular environments and high-dimensional Atari games provide quantitative evidence for the practical benefits of our proposed objective.'
Introduction
In the field of sequential decision making one aims learn behavior policy to act in an environment to optimize some criteria. The well-known field of reinforcement learning (RL) corresponds to one aspect of sequential decision making, where the aim is learn how to act in the environment to maximize cumulative returns via trial-and-error experience [38]. In this work, we focus on imitation learning, where the aim is learn how to act in the environment match the behavior of some unknown target policy [25]. This focus puts us closer to the supervised learning regime, and, indeed, common approach to imitation learning known behavioral cloning (BC) is perform max-liked max-likelihood training on collected a set of target demonstrations composed of state-action pairs sampled from the target policy [31, 34).
Since the learned behavior policy produces predictions (actions) conditioned on observations (states), the amount of demonstrations needed to accurately match the target policy typically scales with the state dimension, and this can limit the applicability of imitation learning to settings where collecting large amounts of demonstrations is expensive, in health [18] and roboties [24] applications. The limited availability of target demonstrations stands in contrast to the recent proliferation of large offline datasets for sequential decision making [28. 19, 7, 21]. These datasets may exhibit behavior far from the target policy and so are not directly relevant to imitation learning via max likelihood
Find experimental code at attps://githin htt.aa..coooooooglaeoooo rew.aleeeeeeeeeeeeeeeeeeeee www.rlleerrrerrrrrreeeeeeeeeeeee maater/rl_repr.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

training, Nevertheless, the offline datasets provide information about the unknown environment, presenting samples of environment reward and transition dynamics. It is therefore natural wonder, is it possible to use such offline datasets to improve the sample efficiency of imitation learning?
Recent empirical work suggests that this is possible [40, 10), by using the offline datasets to learn a low-dimensional state representation via unsupervised training objectives. While these empirical successes are clear, the theoretical foundation for these results is less obvious. The main challenge in providing theoretical guarantees for such techniques is that of aliasing. Namely, even if environment rewards or dynamics exhibit a oow-dimensional structure, the target policy and its demonstrations may not. If the target policy acts differently in states which the representation learning objective maps to the same oww-imeesiooal representation. the downstream behavioral cloning objective may end up learning a policy which "averages" between these different states in unpredictable ways.
In this work, we aim to bridge the gap between practical objectives and theoretical understanding. We derive an offline objective that learns low-dimensional representations of the environment dynamics and, if available, rewards. We show that minimizing this objective in conjunction with downstream behavioral cloning objective corresponds to minimizing an upper bound on the performance difference between the learned low-dimensional BC policy and the unknown and possibly high-dimensional target policy. The form of our bound immediately makes clear that, as long as the learned policy is sufficiently expressive on top of the -iw-iieesional representations, the implicit "averaging" occurring in the BC objective due to any aliasing is irrelevant, and a learned policy can match the target regardless of whether the target policy itself is now-dimensional.
Extending our results to policies with limited expressivity, we consider the commonly used parameter- ization of setting the learned policy be log-linear with respect to the representations (, softmax of linear transformation). In this setting, we show that it is enough to use the same offline representation learning objective, but with linearly parameterized dynamics and rewards, and this again leads to an upper bound showing that the downstream BC policy can match the target policy regardless of whether the target is low-dimensional or log-linear itself. We compare the form of our representation learning objective to "latent space model" approaches based on bisimulation principles, popular in the RL literature [20, 41, 22. 13), and show that these objectives are, in contrast, very liable to aliasing issues even in simple scenarios, explaining their poor performance in recent empirical studies [40].
We continue the practicality of our own objective, and show that it be implemented contrastive learning objective that implicitly learns an energy based model, which, in many common cases, corresponds to linear model with respect to representations given by random Fourier features [33]. We evaluate our objective in both tabular synthetic domains and high-dimensional Atari game environments [11]. We find that our representation learning objective effectively leverages offline datasets to dramatically improve performance of behavioral cloning.
2 Related Work
Representation learning in sequential decision making has traditionally focused on learning repre- sentations for improved RL rather than imitation. While some works have proposed learning action representations [8, 30), our work focuses on state representation learning, which more common in the literature, and whose aim is generally to distill aspects of the observation relevant to control from those relevant only to measurement [3]: see [27] for a review. Of these approaches, bisimulation is the most theoretically mature [17, 13], and several recent works apply bisimulation principles to derive practical representation learning objectives [20, 41, 6). However, the existing theoretical results for bisimulation fall short of the guarantees we provide. For one, many of the bisimulation results rely on defining a representation error which holds globally on all states and actions [1]. On the other hand, theoretical bisimulation results that define representation error in terms of an expectation are inapplicable to imitation learning, as they only provide guarantees bounding the performance difference between policies that are "close" (, LLiischitzz in the representation space and say nothing regarding whether an arbitrary target policy in the true MDP can be represented in the latent MDP [20]. In Section 4.4, we will show that these shortcomings of bisimulation fundamentally limit its applicability to an imitation learning setting.
In contrast to RL, there are comparatively fewer theoretical works on representation learning for imitation learning. One previous line of research this vein is given by [9], which considers learning state representation using a dataset of multiple demonstrations from multiple target policies.

---

Accordingly, this approach requires that each target policy admits a low-dimensional representation. In contrast, our own work makes no assumption on the form of the target policy, and, in fact, this is one of the central challenges of representation learning in this setting.
As imitation learning is close to supervised learning, it is an interesting avenue for future work to extend our results to more common supervised learning domains. We emphasize that our own contrastive objectives are distinct from typical approaches in image domains [15] and popular in image-based RL [26, 37, 36), which use prior knowledge to generate pairs of similar images (, via random cropping). We avoid any such prior knowledge of the task, and our losses are closer to temporal contrastive learning, more common in NLP [29].
3 Background
We begin by introducing the notation and concepts we will build upon in the later sections.
MDP Notation We consider the standard MDP framework [32], in which the environment is given by a tuple M ;= (S, A, R, P, μt, 7). where S is the state space, the action space, R. : S x A + [- -maax Rmax] is the reward function, P S x A + (S)) is the transition function," 2 μ E ( is the initial state distribution, and 7 E [0, 1) is the discount factor. In this work, we restrict our attention
to finite action spaces, |A| E N. A stationary policy in this MDP is function ग S + A(A). A policy acts in the environment by starting at an initial state sp ~ f and then at time 2 0 sampling an action de (st). The environment then produces reward r. R(s,, a.) and stochastically transitions to a state 1 P(s a.). While we consider deterministic rewards, all of our results readily generalize to stochastic rewards, in which case the same bounds hold for R(s, a) denoting the expected value of the reward at (8, a).
The performance associated with policy ı is its expected future discounted reward when acting in the manner described above: Jpec((M) E
n.M]
Σ RR(s a.)
(1)
The visitation distribution of ग is the state distribution induced by the sequential process:
d= (o) =1
(2)
t=
Behavioral Cloning (BC) In imitation learning, one wishes to recover an unknown targel policy T, with access to demonstrations of n. acting in the environment. More formally, the demonstrations given by dataset DV {(s, a; a)),," where 8; ~ d aa; az n. (s.). A popular approach to imitation learning is behavioral cloning (BC), which suggests to learn a policy ı to approximate
T. via mix-likel optimization. That is, one wishes to use the N samples to approximately minimize the objective
(T)(T) := ((s,))((m,,.=))) log (a]s)).
(3)
In this work, we consider using state representation function to simplify this objective. Namely, we consider function S + Z. Given this representation, one no longer learns policy π S + A(A), but rather a policy πW 2 + A(A). The BC loss with representation é becomes
(4) smaller representation space 7 can help reduce the hypothesis space for æ compared to n, and this in turn reduces the number of demonstrations N needed to achieve small error in JBC, However, whether a small error in JBC, translates to a small error in JBC depends on the nature of o, and thus how to determine good Φ is central challenge.
JBC, == ((s,a)-(dT=,
.-)|- (1))))
log (c)osss))
Offline Data In this work, we consider learning ø via offline objectives. We assume access to a dataset of transition tuples Doff = [(8i, 04, ri, x() sampled independently according to
8; dfff 0. Unif R(s,, a:), s ~ P(si, a;):
(5)
We use A(X') to denote the simplex over set X.
3

---

where dd is some unknown offline state distribution. We assume that the support of d'iff includes the support of d= d -s)
+ d'ff (s) O. The uniform sampling of actions in D follows similar settings in related work [5] and in principle can be replaced with any distribution uniformly bounded from below by 17 > 0 and scaling our derived bounds by [A]ii At times we will abuse notation and write samples of these sub-tuples (s, a) doft or (s, 0, r, 8') doft
Learning Goal Similar to related work [9], we will measure the discrepancy between a candidate and the target n. via the performance difference:
Per n.) = (Peet(TT) JPerf(
(6)
At times we will also use the notation PerfDif((TZ: n.), and this is understood to mean PerfDiff (Tz Φ, n. While we focus on the performance difference, all of our results may be easily modified to alternative evaluation metrics based on distributional divergences, Drv (d [(d*) or DxL(d (d= (d=- where Dry is the total variation Also note that we make no assumption that T, is an optimal or near-optimal policy in M.
(TV) divergence and DEL is the Kullback Leibler (KL) divergence.
In the case of vanilla behavioral cloning, we have the following relationship between JBC and the performance difference, which is variant of Theorem 2.1 in [34].
Lemma 1. For any n, Tu, the performance difference may be bounded as
Bmax PerIDifT(TT,T) S (1- 2 (DKI((T.))]IT(())
Rmax
(const(T.) + 2JBC(TT. 7)2
See Appendices and C for all proofs.
Remark (Quadratic dependence on horizon). Notice that the guarantee above for vanilla BC includes quadratic dependence horizon in the form of (1-7)-² and this quadratic dependence is maintained in all our subsequent bounds. While there exists a number of imitation learning works that aim to reduce this dependence, the specific problem our paper focuses on aliasing in the context of learning state representations- is an orthogonal problem to quadratic dependence on horizon. Indeed, if some representation maps two very different raw observations to the same lalent state, no downstream imitation learning algorithm (regardless of sample complexity) will be able to learn g good policy. Still, extending our representation learning bounds more sophisticated algorithms with potentially smaller dependence on horizon, like DAgger [35], is promising direction for future work.
4 Representation Learning with Performance Bounds
We now continue to our contributions, beginning by presenting performance difference bounds analogous to Lemma but with respect to specific representation ø. The bounds will necessarily depend on quantities which correspond to how "good" the representation is, and these quantities then form the representation learning objective for learning Φ; ideally these quantities are independent of Tx, which is unknown.
Intuitively, we need ò to encapsulate the important aspects of the environment. To this end, we consider representation-bue based models Pz Z + ASS) and Rz Z x - [-Rmax: Rmax] of the environment transitions and rewards. We define the error incurred by these models on the offfine distribution as,
JR )R) E (x, ((((.)))RR(((s))))))))))))))
(8)
1
Jr Jr(P2, = 2 ( (Pu.(P((s
(9)
We will elaborate on how these errors are translated to representation learning objectives in practice in Section 5, but for now we note that the expectation over d already leads to a connection between theory and practice much closer than exists in other works, which often resort to supremums over state-action errors [30, 41. 1] or zero errors globally [17].

---

4.1 General Policies
We now relate the representation errors above to the performance difference, analogous to Lemma 1 but with JB We emphasize that the performance difference below measures the difference in returns in the true MDP M, rather than, as commonly seen in model-based RL [23]. the difference in the latent MDP defined by Pz.
Theorem 2. Consider a representation function o: 8 Z and models Rz, Pz as defined above. Denote the representation error as
|A|
CR.T == 1-7 JR (1 (1-7)² Jr (r.
(10)
Then the performance difference M between n. and a latent policy az : Z : A(A) may be bounded as,
PerfDiff TZ,
(TZ,, (Z, T+) S (1+ 1+ ER.T -C Bandy [DLL (T= (2)) (11)
offline pretraining
const (T,, Φ) + JBC (TT)
downstroam behaviond clowing
where C 2Rmaax and dz x,,2 are the marginalization of d= TT onto Z according d: (1-7)² dz (z) = Pr[z o(s) s ~ d T+,2 := E (s) s ~~ -~dd ,=====
(12)
Proof in Appendix B.2.
We thus have a guarantee showing that TH can match the performance of n. regardless of the form of (. whether itself possesses low-dimensional parameterizzaion) Indeed, as long is is learned well enough (En.T + 0) and the space of candidate Tz is expressive enough, optimizing JBC.s achieve near-zero performance difference, since the latent policy optimizing JBC., is T2 = T, and this setting of тz zeros out the second term of the bound (11). Note that, in general n, F T. 6, yet the performance difference of these two distinct policies is nevertheless zero when ER,T = 0. The bound in Theorem 2 also clearly exhibits the trade-off due to the offline distribution, encapsulated by the Pearson x² divergence (x= (d
Remark (Representations agnostic to environment rewards). In some imitation learning settings, rewards are unobserved and so optimizing Jn is infeasible. In these settings, one can consider setting Rz to (unobserved) constant function E E [R(s, a)), ensuring JR (Rz, (2 Ruuxx- Furthermore, in environments where the reward does not depend on the action. R(s,, ) R(s, a2) V2) a2 E A, the bound in Theorem 2 can be modified remove Jn altogether (see Appendix B for details).
4.2 Log-linear Policies
Theorem establishes connection between the performance difference and behavioral cloning over representations given by Φ. The optimal latent policy for BC is T. and this is the same policy which achieves minimal performance difference. Whether we can find πz T..2 depends on how we parameterize our latent policy. If тz is tabular or if zz is represented as sufficiently expressive neural network, then the approximation error is effectively zero. But what about other cases?
In this subsection, we consider Z C T and log-linear policies of the form
exp((0222 Σa cxp(0)
ne (z) softmax(0 2)
(13)
aEA
where 0 E Rd In general, T+,2 cannot be expressed as my for some O. Nevertheless, we can still derive strong bounds for this scenario, by considering factored linear [4] models R2, Pz:
Theorem 3. Consider C Rd,, a representation function á 5 - Z, and linear models Rz, Pz: Rz(z, a) == r(a)r= 2; Pz(8'|z, a) = v6(5', .a)T= for some r A + RR; S S x A Rd. Denote the representation error ÉR,T as in Theorem 2. Then the performance difference between n. and a latent policy mo(=) == softmaxx(1" 2) may be bounded as,
θ 08 ag JBC /BC.
PerfDiff n.) S (1 + Dia (1 CN.T + C
(14)
5

---

where C= 1- 11 + ( 1101
Proof in Appendix B.3.
The statement of Theorem 3 makes it clear that realizability of nT is irrelevant for log-linear policies. It is enough to only have the gradient with respect to learned 0 be close to zero, which is guarantee of virtually all gradient-based algorithms. Thus, in these settings performing BC on top of learned representations is provably optimal regardless of both the form of т. and the form of T.
Remark (Kernel-basee based models). is possible to extend the statement of Theorem to generalized linear dynamics and reward models based kernels by replacing the gradient in the bound with the functional gradient with respect to the kernel [16].
4.3 Sample Efficiency
The previous theorems show that we can reduce imitation learning to (1) representation learning on an offline dataset, and (2) behavioral cloning on target demonstrations with the learned representation. How does this compare to performing BC on the target demonstrations directly? Intuitively, repre- sentation learning should help when the learned representation is "good" (, ER,T is small) and the complexity of the representation space Z is low (. |Z] is small or Z IR for small d). In this subsec- tion, we formalize this intuition for a simple setting. We consider finite S, Z. For the representation o, we assume access to an oracle ΦM OPT & (Dof) which yields an error ER, (@oM), ). For BC we consider learning tabular on the finite demonstration set Di We have the following theorem, which characterizes the expected performance difference when using representation learning.
Theorem 4. Consider the setting described above. Let Φm OPT T (D resulting from BC with respect to ΦM. Then we have,
and NNZ be the policy
Z|A
ED
S (1+ (x= (d (R.I((N)) +C
(15)
N
where C as in Theorem
Proof in Appendix C.I.
Note that application of vanilla BC to this setting would achieve a similar bound but with ER,T and |Z| = [S]. Thus, an improvement from representation learning is expected when ER, (ΦM)) and are small.
4.4 Comparison to Bisimulation
The form of our representation learning objectives - learning & to be predictive of rewards and next state dynamics recalls similar ideas in the bisimulation literature [17, 20, 41, 13]. However, key difference that in bisimulation the divergence over next state dynamics is measured in the latent representation space; divergence between do (ss, a) and / [<)(s), a) for some "latent space model" f, whereas our proposed representation error is between ((s, a) and Pa (){(),), a). We find that this difference is crucial, and in fact there exist no theoretical guarantees for bisimulation similar those in Theorems 2 and 3. Indeed, one can construct a simple example where the use of latent space models leads a complete failure, see Figure
5 Learning the Representations in Practice
The bounds presented in the previous section suggest that good representation & should be learned minimize JR, JT in (8) and (9). To this end we propose learn ò in conjunction with auxiliary models of reward Rz and dynamics Pz. The offline representation learning objective is given by,
Jeep (R RRz
Pz, Φ) = B (aaa. [aR (r Rz (z(((s). at log Pz(s' ((s)..)) (16)
our implementation we choose oR 1, aT = (1 -) to roughly match the coefficients of the bounds in Theorems 2 and 3. Once one chooses parameterizations of RR, Pz, this objective may be optimized using any stochastic sample-based solver, SGD.
where aR, aT are appropriately chosen hyperparameters:

---

MDP
Latent MDP: bisioulaatio
MDP
Ris, joft) 11
Ris, lefty =
RIx, left =1
((3,) 3x
22 %
d dir
Figure 1: An example where latent space model (bisimulation) approach would fail. The MDP has six states and two actions ('left" and 'right"); ); arrows denote action dynamics and the green-colored arrows denote the action selection of π. (left); rewards are zero everywhere except for (s4,, left) 1. In this example, representation Φ given by ((*s) = Zk transitions (ø(s), a) + o(s') on (s, a) (d Unif A). ensuring zero bisimulation error. Imitation learning on this representation yields policy zz which exactly matches n,,2 (orange-colored arrows in the middle) but which achieves significantly worse performance compared to т. on the original MDP (right). Unlike latent space model approaches, approach measures the dynamics error (é(s), a) + 8', and so would rightfully reject the á presented here.
consider an offline distribution doff d= thus there is distribution shift. The perfectly preserves rewards (Ø(s), a) - r and latent
5.1 Contrastive Learning
One may recover contrastive learning objective by parameterizing Pz as an energy- based model. Namely, consider parameterizing Pz as
Pz(s'|z. a) x p(s') exp {-1|= a(''s)))1
(17)
where p is a fixed (untrainable) distribution over S (typically set to the distribution of s' in d and g is learnable function S x A - a neural network). Then Egor log Pz(s' a)] yields contrastive loss:
1 E || ( a)ll (exp---- a)]l
E |- log Pz (z(s')o(s) a)
Similar contrastive learning objectives have appeared related works [10, 40), and so our theoretical bounds can be used explain these previous empirical successes.
5.2 Linear Models with Contrastive Fourier Features
While the connection between temporal contrastive learning and approximate dynamics models has appeared in previous works [30], it is not immediately clear how one should learn the approximate linear dynamics required by Theorem 3. In this section, we show how the same contrastive learning objective can be used to learn approximate linear models, thus illuminating a new connection between contrastive learning and near-optimal sequential decision making; see Appendix A for pseudocode.
We propose to learn dynamics model P(s'Is, a) & p(s') exp(-115(s) g(s', a) l²/ for some functions f.g (, neural networks), which admits a similar contrastive learning objective as mentioned above. Note that this parameterization does not involve d. To recover á, may leverage random Fourier features from the kernel literature [33]. Namely, for k-dimensional vectors x, y we can
approximate expi-l|a /2 se pz(z) ((y), where (xx) == cos W x + b) for W a
matrix with entries sampled from standard Gaussian and b vector with entries sampled uniformly from [0, 2T.. We can therefore approximate P as follows, using E(s, a) = (expp g(3',a)) a)]///2]
p() p(s')
E(s,a) exp(-11f(6) - g(s', a2l//)) = 2p(8))
d. E(s,a) (((((s))T(((((s.)))))))) ((g(s',a)) (18)
Finally, we recover o S MI4
M4d : S A R Ald as
@(s) := (o((s))
(65 a) -_ال 2p(s') a) (19)
7

---

Vanilla BC
SVD feature
Fourier feature
performatcon
2.0
purformence
8.8
8.5
0.9
0.9
0.7
10
20 50
50
100
200
500
164
126
206
513
target demos IN /33
offlnee
data (M /3)
statess ([S) ()
embed ((2)
(N/S
(M/3- 500. 1np4
(N/3
Figure 2: Advantages of representation learning over vanilla behavioral cloning in the tree environ- ment across different N, M. [S). Each subplot shows the average performance and standard error across five seeds. Black dotted line shows the performance of the target policy. Representation learning consistently yields significant performance gains.
ensuring ()) (ss) Pz(s'(z,,) ())) as required for Theorem 3.² Notably, during learning explicit computation of (b(s', a) is never needed, while E(s, a) is straight- forward to estimate from data when p(s') d (s'), thus making the whole learning procedure both the offline representation learning and downstream behavioral cloning practical and easy to implement with deep neural network function approximators for f. R2; in fact, one can interpret as simply adding an additional untrainable neural network layer on top of J, and so these deriva- tions may partially explain why previous works in supervised learning found benefit from adding non-linear layer on top of representations [15].
Experiments
We now empirically verify the performance benefits of the proposed representation learning objective in both tabular and Atari game environments. See environment details Appendix D.
6.1 Tree Environments with Low-Rank Transitions
For tabular evaluation, construct decision tree environment whose transitions exhibit low-rank structures. To achieve this, we first construct "canonical" three-level binary tree where each node is associated with a stochastic reward for taking left or right branch and stochastic transition matrix indicating the probability of landing at either child node. We then duplicate this canonical tree the state space that agent walks down the duplicated tree while the MDP transitions determined by the canonical tree, thus it is possible to achieve cR.T = with | = 8. We collect the offline data using a uniform random policy and the target demonstrations using return- policy.
We learn contrastive Fourier features as described Section 5.2 using tabular f, 9. We then these representations and train a log-linear policy on the target demonstrations. For the baseline, we learn vanilla BC policy with tabular parametrization directly on target demonstrations.
We also experiment with representations given by singular value decomposition (SVD) of the empirical transition matrix, which is another form of learning factored linear dynamics. Figure shows the performance achieved by the learned policy with and without representation learning. Representation learning consistently yields significant performance gains, especially with few target demonstrations. SVD performs similar to contrastive Fourier features when the offline data is abundant with respect to the state space size, but degrades as the offline data size reduces or the state space grows.
6.2 Atari 2600 with Deep Neural Networks
We now study the practical benefit of the proposed contrastive learning objective to imitation learning on 60 Atari 2600 games [11], taking for the offline dataset the DQN Replay Dataset [7], which for
random Fourier features with the radial basis kernel, it is clear that similar derivation can be used with other approximate featurization schemes and other kernlls.
While
'In our experiments, we ignore the scaling E'ss, a) altogether and simply set Φ(s) = *(f(s)). We found this simplification to have little effect results.

---

Normalized improvement
120
80
60
40
20
-20
-40
Contrastive Fourler
Energy-based model
DeepMDP
DeepMDP
end2end
DBC
end2end
Figure 3: Performance improvements of contrastive Fourier features (setting Theorem 3), energy- based model (setting of Theorem 2), and bisimulation DDepMMDP [20] and DBC [41] over vanilla BC. In the baselines, 'end2end" refers allowing gradients to pass into the representation during downstream imitation learning; by default, the representation is fixed during this phase. Each box and whisker shows the percentiles of normalized improvements (see details in Appendix D) among the 60 Atari games.
each game provides 50M steps collected during DQN training. For the target demonstrations, take 10k single-step transitions from the last 1M steps of each dataset, corresponding to the data collected the end of the DQN training. For the offline data, we use all 50M transitions of each dataset.
For our learning agents, we extend the implementations found in Dopamine [14]. We use the standard Atari CNN architecture embed image-based inputs to vectors of dimension 256. In the case of vanilla BC, we pass this embedding to log-linear policy and learn the whole network end-to-end with behavioral cloning. For contrastive Fourier features, the objective Section 5.2, and the representation & == yp o f is given by the random Fourier feature procedure described in Section 5.2. A lin-linear policy is then trained on top of this representation. but without passing any BC gradients through Φ This corresponds to the setting of Theorem 3. We also experiment with the setting of Theorem 2; in this case the setup is same as for contrastive Fourier features, only that we define Φ == f (, Pz is an energy-based dynamics model) and we parameterize Tz expressive singll-hidden-layeee softmax policy on top of cb
separate CNNs to parameterize g. f in
We compare contrastive learning with Fourier features and energy- based models to two latent space models, DeepMDP [20] and Deep Bisimulation for Control (DBC) [41]. in Figure 3. Both linear (Fourier features) and energy-basee based parametrization of contrastive learning achieve dramatic performance gains (> 40%) over half of the games. DeepMDP and DBC, achieve little improvement over vanilla BC when presented as separate loss from behavioral cloning. Enabling end-to-end learning of the latent space models as auxiliary loss to behavioral cloning leads to better performance, but DeepMDP and DBC still underperform contrastive learning. See Appendix D for further ablations.
the other hand,
7 Conclusion
We have derived an offline representation learning objective which, when combined with BC, provably minimizes an upper bound on the performance difference from the target policy. We further showed that the proposed objective can be implemented as contrastive learning with an optional projection to Fourier features. Interesting avenues for future work include (1) extending our theory to multi-step contrastive learning, popular in practice [40, 10]. (2) deriving similar results for policy learning in offline and online RL settings, and (3) reducing the effect of offline distribution shifts. We also note that our use of contrastive Fourier features for learning a linear dynamics model may be of independent interest, especially considering that number of theoretical RL. works rely on such an approximation [4, 39]), while to our knowledge previous work has demonstrated practical

---

and scalable learning algorithm for linear dynamics approximation. Determining if the technique of learning contrastive Fourier features works well for these settings offers another interesting direction to explore.
Limitations One of the main limitations inherent in our theoretical derivations the dependence on distribution shift, the form of (d=( (|dt.) ) in all bounds. Arguably, some dependence on the offline distribution is unavoidable: Certainly, if specific state does not appear in the offline distribution, then there is no way to learn a good representation of it (without further assumptions on the MDP). In practice one potential remedy is to make sure that the offline dataset sufficiently covers the whole state space; the inequality + Dx² (pllg) & llp/qlle will ensure this limits the dependence on distribution shift.
Acknowledgments and Disclosure of Funding
We thank Bo Dai, Rishabh Agarwal, Mohammad Norouzi, Pablo Castro, Marlos Machado, Marc Bellemare, and the rest of the Google Brain team for fruitful discussions and valuable feedback.
References
[1] David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate state abstraction. In International Conference on Machine Learning, pages 2915-2923, PMLR, 2016.
[2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference Machine Learning, pages 22-31. PMLR, 2017.
[3] Alessandro Achille and Stefano Soatto. A separation principle for control in the age of deep learning. Annual Review of Control, Robotics, and Autonomous Systems, 1:287-307, 2018.
[4] Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. arXiv preprint arXiv:2006. 10814, 2020.
[5] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. arXiv preprint wwXiv:1908.00261 2019.
[6] Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265. 2021.
[7] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning, 2020.
[8] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Of- fline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 13611, 2020.
[9] Sanjeev Arora, Simon Du, Sham Kakade, Yuping Luo, and Nikunj Saunshi. Provable represen- tation learning for imitation learning via bi-level optimization. In International Conference on Machine Learning, pages 367-376. PMLR, 2020.
[10] Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. arXiv preprint 1805 2018.
[11] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[12] Daniel Berend and Aryeh Kontorovich. On the convergence of the empirical distribution, arXiv preprint 1rXiv:1205.6711, 2012.
10

---

[13] Pablo Samuel Castro.
Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10069-10076, 2020.
[14] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint
1rXiv:1812.06110, 2018.
[15] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[16] Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina Balcan, and Le Song. Scalable kernel methods via doubly stochastic gradients. arXiv preprint arXiv: 1407.559 2014.
[17] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes, In UAI, volume 4, pages 162-169, 2004.
[18] Raphael Fonteneau, Susan A Murphy, Louis Wehenkel, and Damien Ernst. Batch mode reinforcement learning based on the synthesis of artificial trajectories. Annals of operations research, (08(1))383-416 2013.
[19] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0722 2020.
[20] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. In International Conference Machine Learning, pages 2170-2179. PMLR, 2019.
[21] Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning. arXiv preprint arXiv:2006. 2020.
[22] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1rXfv:1112.0160: 2019.
[23] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. arXiv preprint arXiv: 190..0890 2019.
[24] Jens Kober and Jan Peters. Imitation and reinforcement learning. IEEE Robotics & Automation Magazine, 17(2):55-66 62, 2010.
[25] Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off- off-policy distribu- 2019.
tion matching. arXiv preprini
[26] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint (tti::2004/33449 2020.
[27] Timothée Lesort, Natalia Doaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108:379-392, 2018
[28] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0/6433 2020.
[29] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3788 2013
[30] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arX::: 2018.
[31] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.

---

[32]
Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
[33] Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In NIPS, volume 3, page 5. Citeseer, 2007.
[34]
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[35] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-633 JMLR Workshop and Conference Proceedings, 2011.
[36] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141. IEEE, 2018.
[37] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning, 2020.
[38] Richard Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press, 2018.
[39] Lin Yang and Mengdi Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pages 10746-10756. PMLR, 2020.
[40] Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiv:2102.05815, 2021.
[41] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine, Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint 1rX1v 2020.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yss], [No], or [N/A]. . You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/]]
Please do not modify the questions and only use the provided macros for your answers. Note that the
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See "future work"in Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
12

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 6.
(b) Did you mention the license of the assets? [Yes] The license is Apache License 2.0.
(c) Did you include any new assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "xDU3eW-nSoe", "reviewer": "Reviewer_K8Hd", "summary": "This paper tackles the problem of learning representations for imitation learning, in the context of offline data (with demonstrations). Firstly, given some low dimensional paramterization of the state, the authors provide a performance bound on the learnt policy over the representation matching the ground truth policy, regardless of the structure of the ground truth, as well as bound on the sample efficiency. The authors show that it is possible to learn the model and representations using a contrastive learning objective, and the final representation is output by applying random fourier features to the features learnt via contrastive learning. The authors provide some toy experiments in tabular decision tree settings to verify this hypothesis as well as on offline Atari dataset. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This paper tackles representation learning in offline data and 1) presents a thorough proof of the optimal representations needed for imitation learning, 2) the contrastive objective based representation learning insights seem very useful as well. The bounds derived seem quite thorough and the experiments provided are informative. The paper is overall clear and well written. \n\n\nWeaknesses: While the paper provides potentially useful insights, the fact that most cases consider linear dynamics makes it hard to be applicable to realistic settings. It would be good for the authors to discuss which aspects/assumptions make it difficult to scale to the real world, and potential down-stream uses of the learnt representations/policies, especially since the authors provide experiments on a non-standard Atari setting. It would also be good if the authors provided an exact training procedure used to learn the representation from the contrastive objective.\n\nClarifications: \n\nWhat term in theorem 3 shows the dependency on the quality of the representation? \n\nWhy is theorem 4 only in the tabular case? How does learning happen in tabular cases? \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "the authors do not address limitations of their work in detail and do not have a section on potential negative societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "pxQ19eigZQv", "reviewer": "Reviewer_QKhN", "summary": "This paper considers using offline datasets for improving representation learning for subsequent imitation learning. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is in general well written and the theoretical contributions are nicely explained with important differences from the similar field of bisimulation metric based methods. The authors also validate the approach on toy environment and Atari domain, although I would suggest the authors to also report percentile performance improvements for individual Atari games which seem to be missing in the report. Overall the paper is a decent contribution to the field of imitation learning.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors need to better address the limitation of the work so that its more informative for practical scenarios, this could for instance include some paragraphs about the specific shortcomings of the method complemented with the theoretical results.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "fPQLJKHGGHW", "reviewer": "Reviewer_wHHg", "summary": "The authors study the impact of representation learning when applied to behavioral cloning (BC), and present a series of interesting theoretical results that establish its utility in this context. In particular, the authors show that there is a direct relationship between the quality of the state representation and the performance of the policy derived using BC over that representation. The authors then present a practical representation learning algorithm designed using the insight gained from their theoretical results and show that using it in combination with BC results in an improvement over comparison approaches.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n\n(S1) The overall thrust of the work, i.e., the studying the connection between representation learning and imitation learning, is excellent and extremely important to the community.\n\n(S2) The paper contains interesting and seemingly well-founded theoretical insights that have practical implications.\n\n(S3) The authors have proposed a practical algorithm based on their analysis that appears to achieve good empirical results.\n    \nWEAKNESSES:\n\n(W1) My major concern with the paper is that it seems to ignore a major portion of the literature that has provided the field with an understanding of behavioral cloning (BC) for the better part of a decade, i.e., the work of Stephane Ross and Drew Bagnell as far back as 2010 (e..g, [Ross and Bagnell, AISTATS 2010]). In the submission under review here, the authors seem to identify the major problem with BC to be that of aliasing, whereas in the referenced work above—and throughout the community, to the best of my own knowledge—the major issue is commonly understood to be the quadratic growth in error with respect to the task duration. While both issues seem to be intrinsically linked to a lack of demonstration data, it’s not clear the extent to which the ignorance of this work impacts the presented technique. Is it simply a hole in the related work which the authors can easily address, or is it a more fundamental issue? For example, in the error bounds derived by the authors, I do not see any hint of an error term that grows quadratically with the duration of the task—where have these terms gone? I am very curious to read the authors’ response.\n\n\nPOST-DISCUSSION COMMENTS:\n\nThanks to the authors for addressing (W1) above. As mentioned in the discussion, I’d like to see some text added to the next version of the paper acknowledging this connection and explicitly drawing the distinction between distribution shift and aliasing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I found the authors’ treatment of these topics in the paper to be sufficient.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Gmz0W0w3E5Q", "reviewer": "Reviewer_pRgN", "summary": "This paper proposes an approach for learning a low-dimensional state representation that favors the training procedure of a policy with imitation learning. One of the main challenges is that the representation learning strategy is run on offline data, and is possibly not informed about the expert policy to imitate. The paper presents a theoretical framework to bound the performance of the student policy in function of the expressiveness of the representation and the distance to the distance between expert and student on low-dimensional states. The theoretical findings are then put into practice by deriving an algorithm to learn representations via contrastive learning and Fourier features. The algorithm is then evaluated on both toy tabular examples and on more complex high-dimensional tasks (Atari).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main findings of this work are, to the best of my knowledge, novel. The paper does a very good job of covering the (large) field of representation learning for control. The most important related work is adequately cited, and some interesting connections are drawn between prior empirical evidence and the proposed method. However, it would have been nice to also mention and discuss existing work on learning representation by maximizing information for the task [1].\n\nI find the paper presentation very good, well organized, and easy to follow (except for some minor comments that I report below). However, I am a bit concerned that very few technical details are given for the key experiments of the paper, i.e., the one on Atari games. Indeed, only there learning a representation for the states (i.e., images) actually makes sense (the other demonstrator is just a toy tabular MDP). No mention is made of the architecture, the learning rate, the training time, etc., which makes the experiment basically not reproducible. I have checked in the supplementary code, but I only found code related to the toy tabular MDP and not of the Atari games.\n\nRegarding the technical quality of the submission, I think that the derivations are sound and supported by empirical evidence. On the negative side, I believe this work's biggest weakness is the lack of a clear discussion about the limitations of the proposed method. More specifically, Eq. 11, which presents the central finding of the paper, shows how the student performance is bound in terms of the representation power and the distance expert-student in representation space. However, it is completely unclear what are, in the general settings, the dominant terms of this bound. My concern is that the state observations probability is pretty different between an expert and offline demonstrations, giving a large  $D_{\\chi^2}(d^{\\pi} || d^{off})$. Such component will probably also be amplified by the loss of representations $\\epsilon_{R, T}$ (which I would doubt to be below one). This probably becomes even worse as the representation and policies are forced to be log-linear. As a result, I feel that the bound is completely dominated by its first term and de facto loose for most cases. Note that such a remark does not diminish the derived practical algorithm, which is interesting and with good performance. However, I think that the paper should be more open about this potential issue and discuss how to address it. On a similar note, it would be nice to clarify the effect of realizability of the expert policy on the low-level representation z (the so-called $const(\\pi_*, \\phi)$, which could actually also pretty large if the demonstrations and expert policy are very different. \n\nOther minor comments to improve the presentation and strengthen the contribution:\n\n1. Include stronger baselines for representation learning (e.g. [39]  or several other methods mentioned in the related work).\n2. After each lemma/theorem report in which part of the appendix the proof can be found.\n3. Section 4.3 is titled sample efficiency, but it has nothing to do with the number of samples required to training the policy. It only specializes the bound of Eq.11 to a tabular setting, as far as I understood.\n4. It would be nice to clarify that the low-level representation needs to encode enough information to solve the downstream task. Otherwise, some trivial solutions could be found (e.g., putting everything to zero).\n\nOverall, I think this is a nice and interesting paper. The findings could be useful to the control learning community and open new ways to learn policies from high-level observations in a simple and (possibly) provable fashion. However, the contribution could be strongly enhanced by openly discussing the limitations of the theory and algorithm, ensuring reproducibility, and empirically comparing with stronger (or simply more related) baselines. \n\n[1] A Separation Principle for Control in the Age of Deep Learning, Achille, et al.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No remark at all was done about the possible negative societal impacts of this work. I think this is acceptable since the contribution is mainly theoretical, but I would have appreciated it if the paper would have at least mentioned it (for example, can such work be used for military applications?).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After discussion, all reviewers believe that the rebuttal has addressed their concerns and agree on an acceptance. 
Decision: Accept (Poster)