Bridging State and History Representations: Understanding Self-Predictive RL
Authors: Tianwei Ni
Benjamin Eysenbach
Erfan SeyedSalehi
Michel Ma
Clement Gehring
Aditya Mahajan
Pierre-Luc Bacon
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Representation Learning
POMDPs
Information States
Self-supervised Learning
Primary area: reinforcement learning
TL;DR: We offer theoretical insights into learning self-predictive representations in POMDPs and validate our theories with our simplified algorithm across several benchmarks.

Abstract: Representations are at the core of all deep reinforcement learning (RL) methods for both Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). Many representation learning methods and theoretical frameworks have been developed to understand what constitutes an effective representation. However, the relationships between these methods and the shared properties among them remain unclear. In this paper, we show that many of these seemingly distinct methods and frameworks for state and history abstractions are, in fact, based on a common idea of self-predictive abstraction. Furthermore, we provide theoretical insights into the widely adopted objectives and optimization, such as the stop-gradient technique, in learning self-predictive representations. These findings together yield a minimalist algorithm to learn self-predictive representations for states and histories. We validate our theories by applying our algorithm to standard MDPs, MDPs with distractors, and POMDPs with sparse rewards. These findings culminate in a set of preliminary guidelines for RL practitioners.

Published as a conference paper at ICLR 2024
BRIDGING STATE AND HISTORY REPRESENTATIONS: UNDERSTANDING SELF-PREDICTIVE RL
Tianwei Ni*, Benjamin Eysenbach Erfan Seyedsalehi Michel Ma*, Clement Gehring* Aditya Mahajan Pierre-Lue Bacon
Mila, Université de Montréal, Princeton University, Mila, McGill University
(tianwel. michel. ma, clement - gehring, pierre- bacon 8mi quebee..
eysenbbacherrinceton edu, erfan. seyedsaledddeeeeeee www.aalleeeemlailemeeemmaaaallla mogi illl 11 ca, aditya.maha fanemggg 11.0a
ABSTRACT
Representations are at the core of all deep reinforcement learning (RL) methods for both Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). Many representation learning methods and theoretical frameworks have been developed to understand what constitutes representation. However, the relationships between these methods and the shared properties among them remain unclear. In this paper, we show that many of these seemingly distinct methods and frameworks for state and history abstractions are, in fact, based on common idea of self-predictive abstraction. Furthermore, we pro- vide theoretical insights into the widely adopted objectives and optimization, such the stop-gradient technique, in learning self-predictive representations. These findings together yield a minimalist algorithm to learn self- eeff-rrdiittive representa- tions for states and histories. We validate our theories by applying our algorithm standard MDPs, MDPs with distractors, and POMDPs with sparse rewards. These findings culminate in a set of preliminary guidelines for RL practitioners.
effective
INTRODUCTION
Reinforcement learning holds great potential to automatically Iearn optimal policies, mapping obser- vations to return- return-maximizing maximizing actions. However, the application of RL in the real world encounters challenges when observations are high-dimensional and/or noisy. These challenges become even more severe in partially observable environments, where the observation (history) dimension grows over time. In fact, current RL algorithms are often brittle and sample inefficient in these settings (Wang al., 2019; Stone et al., 2021; Tomar et al., 2021; Morad et al., 2023).
To address the curse of dimensionality, a substantial body of work has focused on compressing observations into latent state space, known state abstraction in MDPs (Dayan, 1993; Dean & Givan, 1997; Li et al., 2006), history abstraction in POMDPs (Littman et al., 2001; Castro et al., 2009), and sufficient statistics or information states in stochastic control (Striebel, 1965; Kwakernaak, 1965; Bohlin, 1970; Kumar& & Varaiya, 1986). Traditionally, this compression has been achieved through hand-crafted feature extractors (Sutton, 1995; Konidaris et al., 201 1) or with the discovery of set of core tests sufficient for predicting future observations (Littman et al., 2001; Singh et al., 2003). Modern approaches learn the latent state space using an encoder to automatically filter out irrelevant parts of observations (Lange & Riedmiller, 2010; Watter et al., 2015; Munk et al., 2016). Furthermore, deep RL enables end-to-end and online learning of compact state or history representations alongside policy training. As result, numerous representation learning techniques for RL have surfaced (refer to Table 1), drawing inspiration from diverse fields within ML and RL. However, this abundance of methods may have inadvertently presented practitioners with "paradox of choice", hindering their ability to identify the best approach for their specific RL problem.
This paper aims to offer systematic guidance regarding the essential characteristics that good repre- sentations should possess in RL (the "what") and effective strategies for learning such representations (the "how"). We begin our analysis from first principles by comparing and connecting various repre- sentations proposed in prior works for MDPs and POMDPs, resulting in a unified view. Remarkably, these representations are all connected by a self preficrre condition the encoder can predict its next latent state (Subramanian et al., 2022). Next, we examine how to learn such self-predictiv condition
Work done while ES was at McGill University.
Please refer to httpp:://rrxvv..og//bbb//
008888888.88888 for the 10-main-page version of this paper.

---

Published as a conference paper at ICLR 2024
in RL, a difficult subtask due to the bootstrapping effect (Gelada et al., 2019; Schwarzer et al., 2020; Tang et al., 2022). We provide fresh insights on why the popular the parameters of the encoder do not update when used target, has the promise of learning the desired condition without representational collapse in POMDPs. Building on our new theoretical findings, we introduce minimalist RI. algorithm that learns self-predictive representations end- to-end with a single auxiliary loss, without the need for reward model learning (thereby removing planning) (François-Lavee
technique, in which
al., 2019; Gelada al., 2019; Tomar et al., 2021; Hansen al., 2022; Ghugare et al., 2022; Ye et al., 2021; Subramanian et al., 2022), reward regularization (Eysenbach et al., 2021), multi-step predictions and projections (Schwarzer et al., 2020; Guo et al., 2020), and metric learning (Zhang et al., 2020; Castro et al., 2021). Furthermore, the simplicity of our approach allows us to investigate the role of representation learning RL, in isolation from policy optimization.
The core contributions of this paper are as follows. We establish unified view of state and history representations with novel connections (Sec. 3), revealing that many prior methods optimize a collec- tion of closely interconnected properties, each representing a different facet of the same fundamental concept. Moreover, we enhance the understanding of self-predictive learning in RL regarding the choice of the objective and its impact the optimization dynamics (Sec. 4). Our theory results in a simplified and novel RL algorithm designed to learn self prediceeee representations fully end-to-ent (Sec. 4.3). Through extensive experimentation across three benchmarks (Sec. 5), we provide empirical evidence substantiating all our theoretical predictions using our simple algorithm. Finally, we offer our recommendations for RL practitioners in Sec. 6. Taken together, we believe that our work potentially aids in addressing the longstanding challenge of learning representations in MDPs and POMDPs.
2 BACKGROUND
agent receives an observation Og E O at time step t, selects an action Gr E A based on the observed history ht := (Ie-1, -,,,, 0z) H,3, and obtains reward r, R(he; de) along with the subsequent observation 0++1 ~ P(- he, az). The initial observation /1 := is sampled from the distribution P(o1). The total time horizon denoted as T E U [++o]. and the discount factor is y E [0, [0.1] (less than 1 for infinite horizon). To maintain brevity, we employ the "prime" symbol to represent the next time step, for example writing h' = (h, a, o'). Under the above assumptions, our agent acts according to a policy x xaa h) with action-valie Q* a). Furthermore, it can be shown that there exists an optimal value function Q" (h, a) such that Q" (h, a) = E[r h, a] + >'Eo'' [max.' Q (h'))aa)), and a deterministic optimal policy '' (h) = argmax, (h, a). In an MDP Ms = AS, P. R, T), the observation and history h4 are replaced by the state s; E S*.
MDPs and POMDPs. In the context of POMDP Mo = (0, TT)
State and history representations. In POMDP, an encoder is function Φ : H, - Z that produces a history representation = = <(h) E 2. Similarly, in an MDP, we replace h with 8, resulting in a state encoder & S + 2 and state representation 2 = (ss) E Z. This representation known as an "abstraction" (Li et al., 2006) or "latent state" (Gelada et al., 2019). Such encoders are sometimes shared and simultaneously updated by downstream components (e.g. policy, value, world model) of RL system (Hafner et al., 2020a; Hansen al., 2022). In this paper, we are interested in such shared encoder, or the encoder learned for the value function if the encoders are separately learned.
Below, present the key abstractions that are central to this paper, along with their established connections. We will highlight the conditions met by each abstraction. We defer additional common abstractions and related concepts to Sec. A.2.
1. Q" -r-ireleevanee abstraction. An encoder ΦQ- provides a Q" Q"-irrelevannce abstraction (Li et al., 2006) if it contains the necessary information for predicting the return. Formally, if Φo- (ha) = de- (hy), then Q* a) = Q" (hy, a). Va. A Q' Q'-irrelevance abstraction can be achieved as a by- product of learning encoder @ through value function ((d(h), a) end-to-end using model-free RL. If the optimal values match, then (0q- (h), a) = Q" (h, a), Vh, a.
2. Sell-predictive (model-irrelevance) abstraction. We view the model-irrelevance concept (Li et al., 2006) from a self-predictive standpoint. Specifically, a model-inreevant encoder di fulfills two
Whiile the classic definition of POMDP (Cassandra et al., 1994) features a state space, we assume it to be unknown, thus our view of POMDP is black-box input-output system (Subramanian et al., 2022).
'n general, h := (hę-1, (he-1,0--1,Pt-2 7t-1,, ot) (Izadi & Precup, 2005). this study, we assume rewards are inaccessible during policy inference.
In finite-horizon MDPs, we assume includes the time step
2

---

Published as a conference paper at ICLR 2024
conditions: expected reward prediction (RP) and next latent state distribution prediction (ZP)², ensuring that the encoder can be used to predict expected reward and the next latent state distribution.
S.l. Eir h, (2r a), vh, a, 3P= 2x A - (2), s.t. P(=' h, Pz(z' (h))),)), wh, ah,, E|z' h, = E (h), Vh, a.
3Rg 2 x A + R,
(RP)
(ZP)
(EZP)
A weak version of ZP is the expected next latent state 2 prediction (EZP) condition. ZP can be interpreted
sufficient statistics condition ΦL: the next latent state 2' is conditionally
independent of the history h when (h) and a is known, symbolized as 2' 11 h dg (h), a. Satisfying ZP only is trivial and can be achieved by employing constant representation (bh)) c, where e is a fixed constant. Therefore, ZP must be used in conjunction with other conditions (e.g.. RP) to avoid such degeneration. The ΦL is known as a bisimulation generator (Givan et al., 2003) in MDPs and an information state generator (Subramanian et al., 2022) in POMDPs.
3. Observation-preddicti (belief) abstraction. This abstraction is implicitly introduced by Sub- ramanian et al. (2022), which we denote by Φo, and satisfies three conditions: expected reward prediction RP, recurrent encoder (Rec) and next observation distribution prediction (OP)
Z, S.L. ((k') th ()o a',", Vh,a,o' Eoth 2 x A
(Rec)
3P, -6: A(O), s.t. P(o' a) Polo @o(h),a), Vh,a,o' s.t. o vo(oo(())), Vh.
(OP)
Reb Z + O, Similarly, the OP condition is equivalent o' T1 h éo(h), a, and OP is closely related to observation reconstruction (OR). widely used in practice (Yarats et al., 2021). The recurrent encoder (Rec) condition is satisfied for encoders parameterized with feedforward or recurrent neural networks (Elman, 1990; Hochreiter & Schmidhuber, 1997), but not Transformers (Vaswani et al., 2017). In this paper, we assume the Rec condition is always satisfied. In POMDPs, Φo is well-known as belief state generator (Kaelbling et al., 1998).
(OR)
We extend the relations between these abstractions known in MDPs (Li et al., 2006) to POMDPs.
Theorem 1 (Relationships between common abstractions (informal)). An encoder satisfying do also belongs to Φτ; an encoder satisfying Φt, also belongs to óq=; the reverse is not necessarily true.
A UNIFIED VIEW ON STATE AND HISTORY REPRESENTATIONS
3.1 AN IMPLICATION GRAPH OF REPRESENTATIONS IN RI.
OP
Using the taxonomy of state and history abstractions, it becomes possible to establish theoretical links among dif- ferent representations and their respective conditions dis- cussed earlier. These connections are succinctly illustrated in directed graph, as shown in Fig. 1. In this section, we highlight the most significantly novel finding, while postponing the other propositions and proofs to Sec. A.
RP
OR
ZP
ΦQ'
Rec
The definition of self- predictve and observation predicctvve abstractions suggests the classic phased training frame- work. In phased training, we alternatively train an encoder to predict expected rewards (RP) and predict next latent states (ZP) or next observations (OP), and also train an RL or planning agent on the latent space with the encoder "detached" from downstream components. On the other hand, we show in our Thm. 2 that if we learn an encoder end-to-end in model-free fashion but using 7P (or OP) as an auxiliary task, then the ground-truft expected reward can be induced by the latent Q-value and latent transition. Thus, the encoder also satisfies RP and generates dl (or do) representation already.
Figure 1: An implication graph showing the relations between the conditions on his- tory representations. The source nodes of the edges with the same color together imply the target node. The dashed edge means only applics to MDPs. All the connections are dis- covered in this work, except for (1) OP + Rec implying Z.P. (2) ZP + RP implying Φor-.
Theorem (7.P + &Q imply RP). If an encoder di satisfies ZP, and (()(h), a) = Q" a), Yh, a, then we can construct a latent reward function function Rz(2,, a) == ((2,a)-7By P [max. Q(=',a') (e),
such that R2 (@(h), a) = [[r h, a). Vh, a.
RP and 7P are labeled as (P1) and (P2), respectively, in Subramanian et al. (2022).
'OP and Rec are labeled as (P2a) and (P2b), respectively, in Subramanian al. (2022).
3

---

Published as a conference paper at ICLR 2024
Table I: Which optimal representation will be learned by the value function in prior works? The "PO" column shows if the approach applies to POMDPs. The "Conditions" column shows the conditions that the encoder of the optimal value satisfies (see the appendix for the "metric" and "regularization" conditions). The ZP loss shows the loss function they use to learn ZP condition. The ZP target shows whether they use online or stop-gradient (including detached and EMA) encoder target. Due to the space limit, we omit the citations for recurrent model-free RI. (Hausknecht & Stone, 2015; Kanturowski al., 2018: Ni et al., 2022) and helief-based methods (Wayne et al., 2018; Hafner et al., 2019; Han et al., 2020; Lee et al.. 2020).
Work
PO? Abstraction
Conditions
ZP loss
ZP target
Model-Free & Classic Model-Base Boael--Bsedd RL
X
ÓQ-
ΦQ- @q- + RP
N/A
N/A
MuZero (Schrittwieser ct al., 2020)
X
unknown
N/A
N/A
MICo (Castro et al., 2021)
X
unknown
ΦQ- metric
N/A
N/A
CRAR (François-Lape Lavet et al., 2019)
X
ΦL
ΦQ' + RP + 7P reg.
62
online
Dee (Gelada al., 2019)
X
Φ
ΦQ- + RP 7P
W (62)
online
SPR (Schwarzer et al., 2020)
X
ØL
ΦQ- + ZP
COS
EMA
DBC (Zhang et al., 2020)
X
ØL
ΦQ RP ZP + metric
FKL
detached
LSFM (Lehnert & Littman, 2020)
X
Ø1
94' + RP EZP
SF
detached
Baseline in (Tomar al., 2021)
X
ΦQ' + RP
62
detached
EfficientZero (Ye al., 2021)
ΦQ- + RP
COS
detached
TD-MPC (Hansen et al., 2022)
X
61.
6Q" + ZPP
Eg
EMA
ALM (Ghugare et al., 2022)
01.
ΦQ- + ZPP
RKL
EMA
TCRL (Zhao et al., 2023)
41.
RP +ZP
COS
EMA
OFENet (Ota ct al., 2020)
X
60
6q OP
N/A
N/A
Recurrent Model-Free RT.
0Q"
9Q- ΦQ- 7P RP ZP OP
N/A
N/A
PBI. (Guo et al., 2020)
ΦL
6z
detached
AIS (Subramanian et al., 2022)
1T. 90
L. FKL
detached
Belief-Based Methods
60
RP + ZP OR
FKL
online
Causal States (Zhang al., 2019)
60
RP OP
N/A
N/A
Minimalist Φe (this work)
ΦL
ΦQ- + ZP
82, KL
stop-grad
This result holds significance, it suggests that end-to-end approaches to learning σq. + +ZP (OP) have similar theoretical justification as classic phased approaches.
3.2 WIICH REPRESENTATIONS Do PRIOR METHOODS LEARN?
With the unified view of state and history representations, we can categorize prior works based on the conditions satisfied by the optimal encoders of their value functions. Table shows representative examples. The unified view enables to draw interesting connections between prior works, even though they may differ in RL or planning algorithms and the encoder objectives. Here we highlight some important connections and provide more detailed discussion of all prior works in Sec. C.
To begin with, it is important to recognize that classic model-based RL actually learns ΦQ- in value function. Model-based RL trains a policy and value by rolling out on the learned model. However, the policy and value do not share representations with the model (Sutton, 1990; Sutton et al., 2012; Chua al., 2018; Kaiser et al., 2019; Janner et al., 2019), learn their representations from maximizing returns (Tamar et al., 2016; Oh et al., 2017; Silver et al., 2017). Secondly, as shown in Table 1, there a wealth of prior work on approximating ΦL. stemming from different perspectives. These include bisimulation (Gelada et al., 2019), information states (Subramanian et al., 2022), variational infer- ence (Eysenbach al., 2021: Ghugare al., 2022), successor features (Barreto al., 2017: Lehnert Littman, 2020), and self- self-supervised learning (Schwarzer et al., 2020; Guo et al., 2020). The primary differences between these approaches in their selection of (1) architecture (whether learning RP, Φq-, or both), (2) ZP objectives (such as fz, cosine, forward or reverse KL, discussed in Sec. 4.1), and (3) ZP targets for optimization (including online, detached, EMA, as detailed Sec. 4.2). Finally, observation- predictive Deservation-predicttve representations are typically studied in POMDPs. where they are known as belief states (Kaclbling et al., 1998) and predictive state representations (Littman et al., 2001).
4 ON LEARNING SELF-PREDICTIV REPRESSENATIIONS IN RL
The implication graph (Fig. 1) establishes the theoretical connections among various representations in RI. yet it does not address the core learning problems. This section aims to give some theoretical answers to how to learn self-predictive representations. While self-predictive representation holds promise, it poses significant learning challenges compared to grounded model-free and observation- predictive representations. The bootstrapping effect, where ó appears in both sides of ZP (since s' also relies on ((h'),, contributes to this difficulty. We present detailed analyses of the objectives in Sec. 4.1 and optimization in Sec. 4.2, with proofs deferred to Sec. B. Building on these analyses, we propose simple representation learning algorithm for dr. in RL. in Sec. 4.3.
4

---

Published as a conference paper at ICLR 2024
4.1 ARE PRACTICAL ZP OBJECTIVES BIASED?
Thm. 2 suggests that can learn OL by simply training an auxiliary task of ZP on model-free agent. Prior works have proposed several auxiliary losses, summarized in Table I's ZP loss column. Formally, we parametrize (probabilisttic case)'. The latent transition function Pa | 2, a) is parameterized by ge Z x A
encoder with fa He 2 (deterministic case) ʃa Hr 4(2)
(deterministic case) or ge : 2 x A - A(Z) (probabilisttic case). We use (² h)) and Po( 2 to represent the encoder and latent transition, respectively. The self- seff-predictive metric for ZP is ideally:
Lzp,D(0, a) B. (DD(P 2, 2, || Po(=' h,a))],
where P=(='| |,, a) = Eo (= }.). D(- 11 .) E IR20 compares two distributions. When Eq. reaches minimum, then for any = ~ ( (1 h), the ZP condition is satisfied.
When designing practical ZP loss, prior works are mainly divided into deterministic l ap- proach (Gelada et al., 2019; Schwarzer et al., 2020; Tomar et al., 2021; Hansen et al., 2022; Ye et al., 2021)* or probabilistic --divergence approach (Zhang al., 2020; Ghugare et al., 2022; Hafner et al., 2019) that includes forward and reverse KL divergences (in short, FKL and RKL):
Je(Ø, 0, 0; h a) = Eg' P((h,a) |190 (Fs().) (8(1)11))
De (Py(2' (N) Po [2.))]]
JD, (0, 0 ø; a) PP-))
where 6, called 7P target, can be online (exact that allows gradient backpropagation), or the stop-gradient version @ (detached from the computation graph and using copy or exponential moving average (EMA) of Φ). The update rule is Φ +- TΦ + (1 T)Ø, with T 0 for detached and T E 1) for generic EMA. We summarize the choices of ZP targets in one column of Table 1.
We first investigate the relationship between the ideal objective Eq. 1 and practical objec- tives Eq. 2 and Eq. 3 to better understand their implications
Proposition 1 (The practical & objective Eq. 2 is an upper bound of the ideal objective Eq. 1 ( 0; h, a) that targets EZP condition. The equality holds in deterministic environments.), Proposition 2 (The practical f-divergence objective Eq. is an upper bound of the ideal ob- jective Eq. 1 LZP.D,(e (1, 0; h, a) that targets ZP condition. The equality holds in deterministic environments.).
These propositions show that environment stochasticity (P(o') | h, a)) affects both the practical l2 and E-divergence objectives. While unbiased in deterministic tasks to learn the ZP condition", they problematic in stochastic tasks (e.g. with data augmentation or noisy distractors) due double sampling issue (Baird, 1995), as the ideal objective Eq. 1 cannot be used (see Sec. B for discussion).
4.2 WHY Do STOP- GRADIENTS WORK FOR ZP OPTIMIZATION?
In this subsection, we further discuss optimizing the practical ZP objective Eq. 2. Specifically, we aim to justify that stop-gradien gradieett (detached or EMA) ZP targets, widely used in practice (Schwarzer et al., 2020: Zhang et al., 2020; Ghugare al., 2022), play an important role in optimization. We find that they may lead to EZP condition in stochastic environments (Prop. 3) and can avoid representational collapse under some linear assumptions (Thm. 3). Meanwhile, online ZP targets lack these properties.
Before introducing the results, we want to clarify the discrepancy between learning ZP and the well-known TD learning (Sutton, 1988). At first glance, Eq. 2 (stop-gradient version) reminiscent of mean- -quured TD error, both having the bootstrapping structures. However, Eq. 2 has an extra challenge due to the missing reward, leading to trivial solution of constant representation, known as complete representational collapse (Jing et al., 2021). Moreover, ZP requires distribution matching, while the Bellman equations that TD learning aims to optimize only require matching expectations.
Proposition (The & objective Eq. with stop gradients (Je(0, 9, ōe h, a)) ensures stationary points that satisfy EZP, but the &g objective with online targets lacks this guarantee.).
Despite being special case of probabilistic encoders, deterministic encoders deserve distinct discussion because they can he optimal in POMDPs and have been frequently used in prior works. In addition, it should be noted that probabilistic one may help it smooths the objective.
Coosiee distance is an fa distance on the normalized vector space Z = { 2 E R
||=||2 1}.
"The EZP condition is equivalent to the 7P condition in deterministic tasks.
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Minimalist oz. learning self- pref-preeettiee representations RL
Require: Encoder fo H4- + Z, Actor TTy Z - A, Critic Qu : Z x A + R. Latent Transition Model go Σ x A + 2. Learning Rate >> 0 and Loss Coefficient À >> 0.
1: procedure UPDATE(h, a, d',r)
2: Compute any model-free RL loss LRI. (based on DDPG (Lillicrap et al., 2016) here) let Qui(h'r)) : : 70=( (T(s)
[o, wollo,,,,hh ,, r) = (1)), (M) Q2(s(()) ( (To(h)))
(4)
3: Compute the auxiliary ZP loss Lu (0, 0;h²)= = 1190 (f (h), - (h)]l
4: Optimize all parameters using the sum of losses:
10, 0, v, < [0, 0, v,
V (fRL w,
,), w,, + ALrux (0, 0 h))
(5)
Prop. 3 suggests the adoption of stop- hoop-gradient targets in Eq. 2 to preserve the stationary points of EZP in both deterministic and stochastic tasks.
Theorem 3 (Stop-gradient provably avoids representational collapse in linear models). Assume a linear encoder (h) == h-k E TR with parameters & E RA(OO+[A))x which always operates on h a recent-k truncation of history h. Assume linear deterministic latent transition go(2,a) a) == 0_ 0T a E Rd with parameters 0g E IRddd and fg E Rlllxdd we train 6 O using the hop-gradient lg objective En,a [Jelo, 0 o: h, a)] without RL loss, and 0 relies on ó by reaching the stationary point with VglEn, [J(( 0 o; h a = 0, then the matrix multiplication ΦJ will retain its initial value over continuous-time training dynamics.
Thm. 3 extends the results of (Tang et al., 2022, Theorem 1) to action-dependent latent transition, POMDP, and EMA settings. This theorem also implies that é will keep full-rank during training if the initialized & is full-rank
Mountaon Car
Load-Uniioe
ale. cootre endfonty
Similar to Tang et al. (2022), we illustrate our theoretical contribution by examining the be- havior of the learned encoder over time when starting from random orthogonal initialization. We extend these results by considering both the MDP and the POMDP setting and consider two classical domains, mountain car (Moore, 1990) (MDP) and load-unload (Meuleau et al., 2013)
FMA
2001
Figure 2: The absolute normalized inner product of the two column vectors in the leamned encoder when using online, detached, EMA ZP target in an MDP (left) and a POMDP (right). We plot the results for 100 different seeds, which controls the rollouts used to sample tran- sition and the initialization of the representation. The (POMDP), where we fit an encoder & with bold lines represent the median of the seeds.
latent state dimension of 2. Fig. 2 shows the iaatogoonaiityypreevviii comparing the cosine similarity between columns of the learned Φ. As expected by Thm. 3, we see this similarity stay several orders of magnitude smaller when using stop-gra moop-gradient (detached or EMA) compared to the online case. Note that although our theory discusses the continuous-ting dynamics, we can approximate them with gradient steps with a small learning rate, as was done for these results.
effect of the stop-gra gradgradienttt by
4.3 A MINIMALIST RL ALGORITHM FOR LEARNING SELF-PREDICTIVE REPRESENTATIONS
Our theory leads to straightforward RL algorithm that can target or., Essentially, integrates single auxiliary task into any model-free RL algorithm (e.g.. DDPG (Lillicrap et al., 2016) and R2D2 (Kap- turowski et al., 2018)), as indicated by Thm. 2. Algo. 1 provides the pseudocode for the update rule of all parameters in our algorithm given tuple of transition data, with PyTorch code included in Appendix. The E ZP loss can be replaced with KL. objective with probabilistic encoder and latent model, especially in stochastic environments, suggested by Prop. 2. The (h') in ZP loss stops the gradient from the encoder, following Sec. 4.2. The actor loss (h). Tp (f)()) freezes the parameters of the critic and encoder. suggested by recent work on memory- based RL (Ni et al., 2023).
Our algorithm greatly simplifies prior self-predictive methods and enables a fair comparison spanning iuservation-predictin representation learning. It characterized minimalist
from model-free by removing reward learning, planning, multi-step predictions, projections, and metric learning. It is also novel by being the first to learn seef-predictive representations and-to-end in POMDP literature.
Our algorithm also bridges model-free and observation -rretict-- representation learning. We derive learning Φg. by setting the coefficient to O, and learning do by replacing ZP loss with OP loss like OFENet (Ota et al., 2020). As a by-pruduct, by comparing (ØQ-, ΦL, Φo} derived from
This is due to the fact that rank(A' (A A) = rank (A) for any real-valued matrix A.

---

Published as a conference paper at ICLR 2024
X85
X16f
X110
Aun-12
100
Homandde222
1.6
splodd peum
petur
spiaods oduuss
Qioek idium
2.0
0.6
0.0
coriecnment
x10
ALM(3) OP-12
OP-FKL TD3 ZP- (ours)
ZP-FKL (ouzs)
2P-RKL (ours)
Figure 3: Decoupling representation learning from policy optimization using our algorithm based on ALM(3) (Ghugare et al., 2022). Comparison between Φo- (TD3), ΦL (our algorithm (ZP-12, ZP-FKI, ZP- RKL) and ALM(3)), do (OP-&2, OP-FKL), in the standard MuloCo benchmark for 500k steps, averaged over 12 seeds. The observation dimension increases from left figure to right figure (17, 17, 111,376),
our algorithm, we can disentangle representation learning from policy optimization because all representations are learned by the same RL. algorithm. This is rarely seen in prior works, as learning Φo or ΦI typically involves planning, while model-free RL does not. Such disentanglement allows us to examine the sample efficiency benefits derived purely from representation learning.
EXPERIMENTS
We conduct experiments to compare RL agents learning the three representations {őq. LL. 6o }. respectively. To decouple representation learning from policy optimization, we follow our minimalist algorithm (Algo. I) to learn Φr,, and instantiate Φ2- and @o by setting À = 0 and replacing ZP loss with OP loss, as we discuss in Sec. 4.3. We evaluate the algorithms in standard MDPs, distracting MDPs and sparse-reward POMDPs. The experimental details are shown in Sec. E. Through the subsequent experiments, we aim to validate five hypotheses based our theoretical insights in Sec. 3 and Sec. 4, with their motivation shown in Sec. D.1.
Sample efficiency hypothesis: do the extra OP and ZP signals help do and Φr have better sample- efficiency than ΦQ' in standard MDPs (Sec. 5.1) and especially in sparse-reward tasks (Sec. 5.3)?
Distraction hypothesis (Sec. 5.2): since learning Φo may struggle with predicting distracting observations, is it less sample-efficiens than learning ΦL?
End-to-end hypothesis (Sec. 5.3): as predicted by Thm. 2, is training an encoder end-to-end with an auxiliary task of ZP (OP) comparable to the phased training with RP and ZP (OP)?
ZP objective hypothesis (Sec. 5.1, Sec. 5.2): as predicted by Prop. I and Prop. 2, is using & loss ZP objective similar to KL loss in deterministic tasks, but not necessarily stochastic tasks?
7P stop-gradient hypothesis (See. 5, Sec. 5.3): as predicted by Thm. 3, does stop-gradient on ZP targets mitigate representational collapse compared to online ZP targets?
5.1 STATE REPRESENTATION LEARNING IN STANDARD MDPs
We first evaluate our algorithm in the relatively low-dimensional MuJoCo benchmark (Todorov et al., 2012). We implement it by simplifying ALM(3) (Ghugare et al., 2022), a state-of-the-ard algorithm on MuJoCo. ALM(3) aims to learn ΦL end-to-end with EMA ZP target and reverse KL, shown by Thm. 2. ALM(3) requires a reward model in the encoder objective and optimizes the actor via SVG (Heess et al., 2015) with planning for 3 steps. Following our Algo. 1, we remove the reward model and multi-step predictions; instead, we train the encoder using our loss (Eq. 2 or Eq. 3) and the actor-critic is conditioned on representations and trained using model-free TD3 (Fujimoto al., 2018), while keeping the other hyperparameters the same. These simplifications result in a 50% faster speed than ALM(3). Please see Sec. E.2 for a detailed comparison between our algorithm and ALM(3).
Ago
Figure 4: Representation collapse with online targets. On four benchmark tasks, observe that using the online ZP target in & objectives results in lower returns (top) and low-rank representations (bottom). In line with our theory, using a detached or EMA ZP target mitigates the representational collapse and yields higher returns.
Validation of sample efficiency and ZP objective hypotheses. Fig. 3 shows that our minimalist Φ using EMA ZP targets can attain similar (in Ant) or even better (in HalfCheetah and Walker2d)
" Distracting MDPs refers to MDPs with distracting observations inrelevant to optimal control in this work.
7

---

Published as a conference paper at ICLR 2024
x10t
Heppee22
xtgt
2.0
opincdde eetumn
2.3
8.2
0.5
0.
21
Number of
Number
Number
die
OP-Fz
OP-FKI
TD3
7P-rz (ours)
ZP-FKI (ours)
ZP-RKL (aurs)
Figure 5: Self-predictive representations are more robust. Comparison between di- (TD3), Φr. (ZP-L2. ZP-FKL, ZP-RKL) using our algorithm, do (OP-l,, OP-FKL) in the distracting MuloCo benchmark, varying the distractor dimension from 2n to 2", averaged over 12 seeds. The y-axis is final performance at 1.5M steps.
sample efficiency 500k steps compared to ALM(3), and greatly outperforms dQ-, across the entire benchmark except for Humanoid task. This suggests that the primary advantage ALM(3) brings to model-free RL in these MuJoCo tasks, lies in state representation rather than policy optimization. This benefit could be further enhanced by streamlining the algorithmic design. In Humanoid task, ALM(3)'s superior performance is likely due to SVG policy optimization's use of first-order gradient information from latent dynamics, which is particularly beneficial in high-dimensional tasks like this. In line with Prop. 1 and Prop. 2, different ZP objectives (f2, FKL, RKL) perform similarly on these tasks, which are nearly deterministic. Thus, these results support our sample efficiency and ZP objective hypotheses.
Validation of ZP stop-gradient hypothesis. We observe a significant performance degradation when switching from the stop- goadggrdient ZP targets to online ones in all MuJoCo tasks for all ZP objectives (l2, FKI., RKL.). Fig. 4 (top) shows the results for the l2 objective Eq. 2. To estimate the rank of the associated linearized operator for the MIP encoder, we compute the matrix rank of latent states given a batch of inputs 12, where the batch size is 512 and the latent state dimension is 50. The MLP encoders are orthogonally initialized (Saxe et al., 2013) with a full rank of 50. Fig. 4 (bottom) shows that the estimated rank of lg objective with online targets collapses from full rank to low rank, phenomenon known as dimensional collapse in sulf--uper learning (Jing et al., 2021). Interestingly, the high dimensionality of a task worsens the rank collapse by comparing from left to right figures. In contrast, hoo-ggradient targets suffer less from rank collapse, in support to our ZP stop-gradient hypothesis. For KI. objectives, switching to online targets also decreases the rank, though less severely (see Fig. 11).
5.2 STATE REPRESENTATION LEARNING IN DISTRACTING MDPs
We then evaluate the robustness of representation learning by augmenting states with distractor dimen- sions in the MuJoCo benchmark. The distractors are i.i.d. standard isotropic Gaussians, varying the number of dimensions from 2a to 28, following the practice in Nikishin et al. (2022). The distracting task has the same optimal return the original one and can be challenging to RL algorithms, even model-based RL could perfectly model Gaussians. We use the same code in standard MDPs. Fig. shows the final averaged returns of each algorithm (variant) in the distracting MuJoCo benchmark.
Validation of distraction hypothesis. By comparing ZP with OP objectives, with higher- dimensional distractors, learning do degrades much faster than learning ΦL. verifying our distraction hypothesis. Surprisingly, model-free RL (ΦQ.) ) performs worse than do. as ΦQ- does not need to predict the distractors. However, we still observe severe degradation in Ant for all methods when there 128 distractors, which we will study in the future work.
Extending ZP objective hypothesis to stochastic tasks. In stochastic tasks, Prop. and Prop. 2 tell us about the (strict) upper bounds for learning EZP and ZP conditions with the practical &z and KL objectives. respectively. Yet, they do not indicate which objective is better for stochastic tasks based on these bounds. In fact, we observe that both &g and reverse KL perform better than forward KL in the distracting tasks, possibly because the entropy term in reverse KL smooths the training objective, and l2 objective simplifies the learning.
5.3 HISTORY REPRESENTATION LEARNING IN SPARSE-REWHH POMDPs
Finally, we perform an extensive empirical study 20 MiniGrid tasks -hevvalir--Boiseert et al., 2018). These tasks, featuring partial observability and sparse rewards, serve as a rigorous best-bed for history representation learning. The rewards are only non-zero upon successful task completion. Episode returns are between 0 and 1, with higher returns indicating faster completion. Each task has an observation space of 7 7 x 3 147 dimensions and discrete action space of 7 options. Our
The rank of m matrix A is the dimension of the image of mapping f RR" R., where f(z) = Ar.
8

---

Published as a conference paper at ICLR 2024
MiniGrid benchnark (20 tasks)
MiniGrid benchmark (20 tasks)
1.00
episode retumn (IQM)
OP
125
GP
0.75
R2D2
matrix rank (IQM)
(Docachol)
RP+OP
100
(EMA)
0.50
RP+ZP
ZP (Oaline)
ZP
75
0.25
0.00
0
4
0
2
euvironment steps x10
environment stops
x10"
Figure 6: End-to-end learned self- self-predictive and invervation-predictiyy representations stand out in sparse-reward tasks. We show the interquartile mean (IQM) (Agarwal al., 2021) across 20 MiniGrid tasks, computed over 9 seeds per task with 95% stratified bootstrap confidence intervals. The individual task plots are shown Appendix Fig. 14 and Fig. 15. Left: comparison of episode returns between dg. (R2D2), &r (ZP, RP + ZP), Po (OP, RP + OP). RP + ZP and RP OP methods are phased, while ZP and OP methads are end-to-end. Right: comparison of estimated matrix rank between ZP targets (online, detached, E.MA), R2D2, and OP. The maximal achievable rank is 128.
model-free baseline is R2D2 (Kapturowski al., 2018), composed of double Q-learning (Van Hasselt et al., 2016) with LSTMs (Hochreiter & Schmidhuber, 1997) as history encoders. We implement based on recent work (Seyedsalchi et al., 2023). R2D2 uses stored LSTM hidden states for initialization, a 50-step burn-in period, and a 10-step history rollout during training. This result in high dimensionality of (50 + 10) (147 + 7) = 9240 on histories.
Based on R2D2, we implement our algorithm by adding an auxiliary task of ZP (or OP) under l objectives, which sufficient for solving these deterministic tasks (see Prop. 1). To compare the phased training (Seyedsalehi et al., 2023), we further implement methods (RP + OP, RP + ZP) that also predict rewards and freeze the encoders during Q-learning. Due to the space limit, we show the aggregated plots in Fig. 6 and defer the individual task plots to Sec. G.
Validation of sample efficiency hypothesis. By examining the aggregated learning curves of end-to- end methods in Fig. 6 left, we find that minimalist &z (ZP) significantly outperforms dq. (R2D2) on average. It, however, fall shorts of do (OP), which aligns with our expectations given that MiniGrid tasks are deterministic with medium- dimensionnnl clean observations. The enhanced performance of Φ1, and Φo over ΦQ- provides empirical validation of our hypothesis in sparse-reward tasks.
Validation of end-to-end hypothesis. By comparing the aggregated learning curves between end- to-end methods and phased methods (i.e., OP vs RP + OP, and ZP vs RP + ZP) in Fig. 6 left, we observe that end-to-end training (OP or ZP) yields equal or superior sample efficiency relative to phased training (RP+ + OP or RP + ZP) when learning observation-prec predicccveeee or self-predictive representations. This supports our hypothesis, and is particularly noticeable when end-to-end learning Ø (ZP) markedly excels over its phased learning counterpart (RP + +ZP).
Validation of ZP stop-gradient hypothesis. We extend our rank analysis from MuJoCo MiniGrid using the same estimation metric. As depicted in Fig. 6 right, our finding averaged across the benchmark indicates that EMA ZP targets are able to preserve their rank, while both detached and online ZP targets degrade the rank during training. Notably, while our theory does not distinguish detached and EMA targets, the observed lower rank of online targets relative to both detached and EMA targets is in line with our hypothesis. Finally, without any auxiliary task such ZP and OP, R2D2 degrades the rank, which is predictable in the spwrse-reward setting (Lyle et al., 2021).
DISCUSSION
Recommendations. Based on our theoretical and empirical results, we suggest the following preliminary guidance to RL practitioners:
1. Analyze your task first. For example, in noisy or distracting tasks, consider using self- eelf-predietive representations. In sparse-reward tasks, consider using abservation-prediccin representations. In deterministic tasks, choose the deterministic &g objectives for representation learning.
2. Use our minimalist algorithm as your baseline. Our algorithm allows for an independent evaluation of representation learning and policy optimization effects. Start with end-to-end learning and model-free RL for policy optimization.
3. Implementation tips. For minimalist algorithm, recommend adopting the & objective with EMA ZP targets first. When tackling POMDPs, start with recurrent networks as the encoder.
Please refer Sec. D.2 for our discussion on limitations and conclusion.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS AND DISCLOSURE OF FUNDING
We thank Pierluca D'Oro and Zhixuan Lin for their technical help. We thank Amit Sinha, David Kanaa, David Yu-Tung Hui, Dinghuai Zhang, Doina Precup, Léo Gagnon, Pablo Samuel Castro, Raj Ghugare, Shreyas Chaudhari, and Ziyan Luo for the constructive discussion. This work was enabled by the computational resources provided by the Calcul Québec (www.. calculquebes ca) and the Digital Research Alliance of Canada (https: from NVIDIA Corporation. This work was funded by IBM Research and Google DeepMind.
ca/), with material support
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Proc. of NeurlPS, 2021. 9, 9,39
Cameron Allen, Neev Parikh, Omer Gottesman, and George Konidaris. Learning markov state abstractions for deep reinforcement lcarning. Proc. of NeurIPS, 2021. 17, 33
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 2016. 37
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995. Elsevier, 1995. 5, 26
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Proc. of NeurlPS, 2017. 4, 28, 29
Torsten Bohlin. Information pattern for linear discrete-time models with stochastic coefficients. IEEE Transac- tions on Automatic Control, 1970.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1rXiv:1606.012 10..60000000.... 2016. 36
Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the 12th National Conference on Artificial Intelligence, Seattle, WA, USA, July 3/- August 4, 1994, Volume 2, 1994. 2
Pablo Samuel Castro, Prakash Panangaden, and Doina Precup. Equivalence relations in fully and partially observable markov decision processes. In Proc. of IJCAI, 2009. 1, 18, 19, 23, 31
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Improved representations via sampling-basse based state similarity for markov decision processes. Proc. of NeurlPS, 2021. 2, 4, 33
Maxime Chevalier- Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https::tt/tt. //gith/gittt coon/mxximech. htreee///yy//eeeee h////yymmmm woym-minigrid. 2018. 3.
Maxime Chevalier-Boisverr, Bolun Dai, Mark Towers, Rodrigo de I Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, 2023. 37
Era Choshen and Aviv Tamar. Contrabar: Contrastive bayes- deep rl. arXiv preprint arXiv:2306.02416 2023. 33
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Proc. of NeurIPS, 2018.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter, Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint ar11v:1511.07289, 2015. 37
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 1993. 1,29 2,29
Thomas Dean and Robert Givan. Model minimization markov decision processes. Proc. of AAAI, 1997. 1
Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-fres model-based reinforcement learning with prototypical representations. Proc. of ICML. PMLR, 2022. 34
Jeffrey I. Elman. Finding structure in time. Cognitive science, 1990.
10

---

Published as a conference paper at ICLR 2024
Benjamin Eysenbach, Russ Salakhutdinov,, and Sergey Levine. Robust predictable control. Proc. of NeurIPS. 2021. 2, 4,
Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, 2004.
Vincent François-Lavet, Yoshua Bengio, Doina Precup, and Joelle Pineau, Combined reinforcement learning via abstract representations. In Proc. of AAAI, 2019. 2, 4,
Xiang Fu, Ge Yang, Pulkit Agrawal, and Tommi Jaakkola. Learning task informed abstractions. In Proc. of ICML. PMLR, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation in actor-critic methods. Proc. of ICML,, 2018. 7
Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforeement learning. arXiv preprint vrXiv:2306.02455 2023.
33
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In Proc. of ICML., PMLR, 2019. 2, 4,5, 5, 28
Raj Ghugare, Homanga Bharadhwaj, Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. Simplity- ing model- based rl: Learning representations, latent-space models, and policies with one objective. arXiv preprint arXiv:2209.0844. 08466 2022. 2, 4, 5, 7, 30, 34, 36, 37
Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial Intelligence, 2003. 3, 18
Zhaohan Daniel Guo, Bernardo Ávila Pires, Bilal Piot, Jean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent preettpredietive representations for multitask reinforcement learning. In Proc. of ICML, 2020. 2, 4, 28
Zhaohan Daniel Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastien Grill, Yunhao Tang, et al. Byol-explore: Exploration by bootstrapped prediction. arXiv preprint 28330220,,,,33,, 2022. 28
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Proc. of NeurlPS, 2018. 32
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proc. of ICML, 2019. 4, 31
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 2020a. 2
Danijar Hafner, Timothy P. Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In Proc. of ICIR, 2020b. 31
Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. In Proc. ICLR, 2020. 4, 41
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.0499: 2022. 2, 4, 30
Matthe J. Hausknecht and Peter Stone. Deep recurrent Q-learning for partially observable mdps. In Proc. of AAAI, 2015.
Nicolas Heess, Gregory Wayne, David Silver. Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. Proc. of NewrIPS, 2015. 7, 30
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997. 3,9
Masoumeh T. Izadi and Doina Precup. Using rewards for belief state updates in partially observable Markov decision processes. In Proc. of ECML, 2005.
Max Jaderberg, Volodymyr Mnih Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Ko- ray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks, arXiv preprint or Xiv: /61 1,05397, 2016. 32
11

---

Published as a conference paper at ICLR 2024
Michael R James, Satinder Singh, and Michael Littman. Planning with predictive state representations. In Proc. of ICML. IEEE, 2004.
Michael Janner, Justin Fu, Marvin Zhang. and Sergey Levine. When trust your model: Model-based policy optimization. Proc. of NeurlPS, 2019. 4
Nan Jiang. Notes on state abstractions.
http://nanjiane web www.ee.nnnnnnn....oooo
cs598/ 11598/note4. pdf, 2018. 19
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:21 10.09348, 10.0944,, 2021. 5,
Lestie Pack Kaclbling, Michael L. Littman, and Anthony R. Cassandra, Planning and acting in partially observable stochastic domains. Artif. Intell., 1998. 3, 4,31
Lukasz Kaiser, Mohammad Bahacizadch, Piotr Milos, Blazej Osinski, Roy H Camphell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. arXiv preprint (=XXixv993.00777. 2019. 4
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In Proc. of ICLR, 2018. 4, 6, 9, 38
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015. 37
George Konidaris. Sarah Osentoski, and Philip Thomas. Value function approximation in reinforcement learning using the fourier basis. In Proc. of AAAI, 2011. 1
Tejas D Kulkarni, Ardavan Sacedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint 1rXiv: 02396. 2016.
Kumar and Pravin Varaiya. Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice- Hall, Inc., Upper Saddle River, NJ, USA, 1986.
H. Kwakernaak. Theory of Self-Adaptive Control Systems. Springer, 1965.
Moritz Lange. Noah Krystiniak, Raphael Engelhardt, Wolfgang Konen, and Laurenz Wiskott. Comparing auxiliary tasks for learning representations for reinforcement learning, 2023. http:/tp:///// /opppreei///peeeiiiieee net/ forum? ? d-7 P b-b7q. 32
Sascha Lange and Martin Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In Proc. of IJCNN. IEEE, 2010. 1
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In Proc. of ICML. PMLR, 2020.
Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Proc. of NeurlPS, 2020. 4, 31
Lucas Lehnert and Michael L Littman. Successor features combine elements of model-free and model-based reinforcement learning. The Journal of Machine Learning Research, 2020. 4, 28, 29
Timothée Lesort, Natalia Diaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 2018. 27
Lihong Li. Thomas J Walsh, and Michael L Littman. Towards unified theory of state abstraction for mdps. In AlandM, 2006. 1, 2, 3,
Xiang Li, Jinghuan Shang. Srijan Das, and Michael Ryoo. Does self-supervised learning really improve reinforcement learning from pixels? Proc. of NeurlPS, 2022. 34
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa. David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. In Proc. of ICLR, 2016.
Michael L. Littman, Richard S. Sutton, and Satinder P. Singh. Predictive representations of state. In Proc. of NewrIPS, 2001. 1, 31
Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks representation dynamics. In Proc. of AISTATS. PMI.R, 2021. 9
12

---

Published as a conference paper at ICLR 2024
Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. Proc. of NeurlPS, 2020.
Nicolas Meuleau, Kee-Fung Kim, Leslie Pack Kaelbling, and Anthony Cassandra. Solving pomdps by searching the space of finite policies, arXiv preprint arXiv:1301. 2013. 36
Aditya Mohan, Amy Zhang, and Marius Lindauer. Structure in reinforcement learning: A survey and open problems. arXiv preprint arXiv:2306. 16021, 2023. 34
Andrew William Moore. Efficient memory- based learning for robot control. Technical report, 1990. 35
Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning. arXiv preprint arXiv:2303.08859, 2023.
Jelle Munk, Jens Koher, and Rohert Babuška. Learning state representation for deep actor-critic control. In 2016 IEEE 55th Conference on Decision and Control (CDC). IEEE, 2016.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be strong baseline for many pomdps. In Proc. of ICML. PMLR, 2022. 4, 33,
Tianwei Ni. Michel Ma, Benjamin Eysenbach, and Pierre-Lue Bacon. When transformers shine in rI?
decoupling memory from credit assignment. arXiv preprint FrXXi:2307.03866 2023. 6
Evgenii Nikishin, Romina Abachi, Rishabh Agarwal, and Pierre-Luc Bacon, Control-oriented model-based reinforcement learning with implicit differentiation. In Proc. of AAAI, 2022. 8, 36
Junhyuk Oh, Satinder Singh, and Honglak Lec. Value prediction network. Proc. of NeurlPS, 2017. 4, 32
Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. In Proc. of ICRA. IEEE, 2021. 34
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.
arXiv preprint arXiv: /000007770 2018. 33
Kei Ota. Tomoaki Oiki, Devesh Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimen- sionality improve deep reinforcement lcarning? In Proc. of ICML. PMLR, 2020. 4, 6, 32
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Proc. of NeurlPS, 2019. 38, 40
Gandharv Patil, Aditya Mahajan, and Doina Precup. On learning history based policies for controlling markov decision processes. arXiv preprint arXiv:2211.33011, 2022. 30
Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann, 1988. 20
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Proc. of ICML. PMLR, 2019,
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programmmingg John Wiley and Sons, 1994. 16
Kate Rakelly, Abhishek Gupta, Carlos Florensa, and Sergey Levine. Which -intuaaiiifoomationn representation learning objectives are sufficient for control? Proc. of NeurlPS, 2021. 33
Andrew Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep lincar neural networks. arXiv preprint 13Xiv:1312.6120 2013. 8
Julian Schrittiieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 2020. 4, 22.32
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman, Data- efficient reinforcement learning with self- predictvee representations. arXiv preprint arXiv:2007.05929, 2020. 28, 29, 34,35
Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu. Equivalence of distance-based and rkhs-hased statistics in hypothesis testing. The annals of statisties, 2013. 30
13

---

Published as a conference paper at ICLR 2024
Erfan Seyedsalehi, Nima Akbarzadeh, Amit Sinha, and Aditya Mahajan. Approximate information state based convergence analysis of recurrent q-leaming. arXiv preprint arXiv:2306.0599) 0599), 2023. 30, 37, 38
Cosma Rohilla Shalizi and James P Crutchfield, Computational mechanics: Pattern and prediction, structure and
simplicity. Journal of statistical physics, 2001, 31
Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell, Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint 11.007761..073077 2016. 32
David Silver. Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnoid, David P. Reichert, Neil C. Rabinowitz, André Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. In Proc. of ICML, 2017.
Satinder P Singh, Michael I. Littman, Nicholas K Jong, David Pardoe, and Peter Stone. Learning predictive state representations. In Proc. of ICMI. 2003.
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02222, 2021. 1
Charlotte Striebel. Sufficient statistics in the optimum control of stochastic systems. Journal of Mathematicul Analysis and Applications, 1965.
Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. J. Mach. Learn. Res., 2022. 1, 2, 3, 4, 16, 18, 20, 23, 24, 25, 30, 34
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 1988. 5
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990. Elsevier, 1990. 4
Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. Proc. of NewrlPS, 1995.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. 35
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforeement learning with function approximation. Proc. of NeurlPS, 1999. 17
Richard Sutton, Csaba Szepesvári, Alborz Geramifard, and Michael P Bowling. Dyna-style planning with linear function approximation and prioritized sweeping. arXiv preprint arXiv:1206.3285. 2012. 4
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. Proc. of NewrlPS, 2016.
Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Ávila Pires, Yash Chandak, Rémi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, et al. Understanding self-predictive leamning for reinforcement leamning arXiv preprint arXiv:22/2.033/1 2022, 2, 6, 27
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proc. of IROS, 2012.7
Manan Tomar, Utkarsh A Mishra, Amy Zhang, and Matthew E Taylor. Learning representations for pixel-based control: What matters and why? arXiv preprint arXiv:2111.07775. 2021. 1, 2, 4,
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-leaming. In Proc. of AAAI, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Proc. of NeurIPS, 2017. 3
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.00 02057, 2019. 1,
Tongzhou Wang, Simon S Du, Antonio Torrala,, Phillip Isola, Amy Zhang, and Yuandong Tian. Denoised mdps: Learning world models better than the world itself. arXiv preprint arXiv:2206. 15477, 2022.
Manuel Watter, Jost Tobias Springenberg. Joschka Boedecker, and Martin A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Proc. of NewrIPS, 2015. 1,32
14

---

Published as a conference paper at ICLR 2024
Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Rac, Piotr Mirowski, Joel Leibo, Adam Santoro, al. Unsupervised predictive memory goal-directed agent, arXiv preprint arx: 10760, 2018. 4
Jack
Lujie Yang, Kaiqing Zhang. Alexandre Amice. Yunzhu and Russ Tedrake. Discrete approximate information states in partially observable environments. In 2022 American Control Conference (ACC). IEEE, 2022. 30
Denis Yarats, Amy Zhang, llya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model free reinforcement learning from images. In Proc. of AAAI, 2021. 3, 32
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Proc. of NeurlPS, 2021. 2, 4, 5, 29
Amy Zhang, Zachary Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandkumar, Laurent Itti, Joelle Pineau, and Tommaso Furlanello, Learning causal state representations of partially observable environments. arXiv preprint arXiv: 1906. 10437, 2019. 31
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representa- tions for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742 2020. 2, 4, 5, 28, 34
Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang. Metacure: Meta reinforcement learning with empowerment-drrven exploration. In Proc. of ICML, 2021. 34
Yi Zhao, Wenshuai Zhao, Rinu Boney, Juho Kannala, and Joni Pajarinen. Simplified temporal consistency reinforcement learning. arXiv preprint arXiv:2306.09466 094666 2023. 31
Luisa M. Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In Proc. of ICML, 2021. 34
15


Reviews:
[{"review_id": "GN9Sjhlp1W", "reviewer": "Reviewer_fv1F", "summary": "The paper attempts at providing a unified view at self-predicting reinforcement learning. Different self-predicting representation targets are described, and the prior work is classified according to the target learned. Based on the unified view, a minimalist algorithm learning a self-predicting representation is learned. The algorithm is evaluated on a set of benchmarks.", "questions": "In the introduction, you are writing \"However, this abundance of\nmethods may have inadvertently presented practitioners with a “paradox of choice”, hindering their\nability to identify the best approach for their specific RL problem.\" \n\nHow does your paper help practitioners identify and use the best approach for their specific RL problem? Can you give an example of application? For example, suggest (and describe) a simple RL problem, and show how your result help choose the best solution.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper encompasses a broad range of recent and ongoing work on self-predicting RL, providing a unifying view. Theoretical results are presented, with proofs in the appendix. An algorithm proposed in the paper is evaluated on benchmarks.", "weaknesses": "The results in the paper are either trivial or indecisive. The paper is built around an insight that different works do similar things trying to optimize self-prediction of certain features, but this is, in my opinion, trivial. A classification of things that can be optimized for self-prediction is worth a survey, but this paper is not a survey. The paper uses a lot of abbreviations, the proofs a sketchy and uncommented, and veryfiying or even following the proofs takes tremendous effort.\n\nThe empirical evaluation is unconvincing. According to the plots presented in the paper, the proposed unified algorithm does not outperform (and does not always perform comparably) to algorithms from the literature. Looking at the algorithm pseudocode and implementation, this is not surprising, given that the 'minimalist algorithm' is more of a boilerplate, which, when filled with details, reduces to one of the earlier published algorithms. However, any practical implementation requires attending to details, which the unifying minimalist algorithms fails to achieve. \n\nThe paper would be extremely hard to follow, in my opinion, for an outsider, or for someone less familiar with the slang of a particular research group. For example, the paper discusses (and presents theoretical results) wrt to \"stop-gradient\" technique without formally defining the technique (which is described in passing and requires referring to the cited sources to understand the paper).\n\nI believe that his research may have a potential, but for a publication, I would suggest deciding on a small subset of ideas among those sketched in this submission, presenting them thoroughly and rigorously, with proofs that are possible to follow, and accompanied by an implementation that brings competitive results, in some form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xREkuhGMJS", "reviewer": "Reviewer_5DUm", "summary": "The paper proposes a conceptual framework for unifying many existing techniques for decision-making-focused representation learning based on the concept of \"self-prediction\". Within this framework, the paper then proposes a simple and seemingly novel \"minimalist\" approach to construct representations simply by training a model-free agent and simultaneously learning an abstract forward model while preventing gradients from flowing backward through the encoder from the targets. The framework allows a large number of representation learning methods to be evaluated side-by-side while controlling for other factors, which enables the paper to present and evaluate several hypotheses about the various learning objectives. The experiments suggest that the proposed \"minimalist\" approach works well.", "questions": "Do the authors have a clear recommendation on when to use OP vs ZP vs RP+[*]? It would be nice to have some clear takeaways after all this analysis.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this is a nice contribution!\n\nThe shared framework is helpful for comparing a seemingly endless number of decision-making-focused representation learning methods. It distills these methods down to their core representation learning ideas, which allows for comparing those ideas without needing to worry about the remaining complexity (such as the particular RL algorithm employed). I suspect the community will find this quite valuable.\n\nThe paper also offers interesting insights, such as the fact that online targets do not guarantee the same fixed points as the objectives with stop-gradients.\n\nThe proposed \"minimalist\" approach is extremely simple and clean. It's shocking that this hasn't been tried before, yet I cannot think of an example where it has. Time will tell if this approach generalizes to other problems, but if it does, it will greatly simplify many projects that rely on representation learning.\n\nThe experiments seem well designed, the baselines seem well chosen, and the results look promising. I particularly liked the experiments measuring the change in matrix rank over time---nice job. That is clear evidence that the method avoids representation collapse.", "weaknesses": "My concerns are minor.\n\n- I'm not familiar with using the phrase \"distracted MDPs\" to mean MDPs with distracting elements.\n- Maybe I'm wrong, but I feel like we already know that ZP + $\\phi_{Q*}$ implies RP.\n- Top of p5: The use of $\\mathbb{P}$ and $\\mathbb{Q}$ is confusing, given that $P$ and $Q$ are often overloaded in RL.\n- Towards end of sec 4.2, last para: \"...cosine similarity between columns of the learned $\\phi$. As expected by Thm. 3, [...] stay several orders of magnitude smaller when using stop-gradient.\" How does Thm 3 predict low absolute cosine similarity? Because we start with full rank?\n- Fig 3. Kind of hard to distinguish lines.\n- Sec 5, first para: would be helpful to summarize the findings when introducing the hypotheses.\n- p7, penultimate para: I don't know about similar sample efficiency in ant. It seems a lot worse. I also don't know if I agree with the conclusion that \"the primary advantage ALM(3) brings to model-free RL, lies in the state representation rather than policy optimization, except for Humanoid.\" I feel like it's too soon to conclude something as sweeping as that.\n- Fig 5. A little difficult to read. See if maybe log return would help?\n- Top of p9, minigrid experiment. This feels like a minor bait-and-switch. Normally minigrid uses pixels, no? And why does detached rank drop on minigrid when theory suggests it should remain high?\n- \"Validation of end-to-end hypothesis\" (and throughout). It's a bit hard to keep track of which is which. Text uses $\\phi_o$, $\\phi_L$, but fig uses OP, ZP, RP+OP, RP+ZP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7eE24YTuEu", "reviewer": "Reviewer_kS5Z", "summary": "The paper proposes a unification of various state representation learning algorithms under the umbrella of self-predictive representations. They draw relations between different algorithms under their terminology and even suggest why stop-gradients are useful when learning self-predictive representations.", "questions": "1. While the main message of the paper seems clear, I am struggling to understand the core message of the empirical section. Few questions:\n  - Should we be using minimalist over everything else? If so, ALM outperforming minimalist in Figure 3 on multiple occasions would not support that.\n  - Is minimalist better than $\\phi_{Q^{*}}$? I think this is somewhat obvious, which is the reason we have all these works on representation learning (this is not a criticism, just a remark).\n  - Is it broadly to categorize the entire zoo of representation learning algorithms into their essential components i.e. $\\phi_{Q^*}$, $\\phi_L$, $\\phi_O$ and just study how they perform empirically?\n  - Misc point: \n     - I understand the point of the different loss function and representation collapse experiments, and those are useful.\n     - It would be very useful to include $\\textit{why}$ it is important to test the questions that are posed in Section 5, and what the final algorithm recommendation should be.\n     - The paper discusses this paradox of choice, but I still find it unclear how to make a choice at the end of the paper.\n2. The sample efficiency claims in Section 5.1 are unclear to me. It appears that ALM(3) outperforms $\\phi_L$ in all cases. It is true that $\\phi_L$ does better than $\\phi_{Q^*}$, but the former does not seem to be. I would suspect that ALM(3) will do better because it is explicitly learning a reward model, whereas $\\phi_L$ is suggesting that the reward model/representations are implicitly learned based on their implication graph.\n3. Is there intuition for why $\\phi_O$ may struggle with distractors? The question is posed in the empirical section, but it’s a bit unclear what motivates this question.\n\nI should note that I do like the paper, but the above things are confusing. I would be willing to re-evaluate the score based on the response to the above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I agree very much with the paradox of choice that is stated in the paper. I think a unification is much needed, and this paper is valuable from that point of view.\n2. The paper is generally well-written.\n3. Table 1 and Figure 1 are useful to understand the framework.", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M37yeWXV7l", "reviewer": "Reviewer_sXU8", "summary": "The work overviews existing abstractions for state/history representations and their \"conditions\", and, with an implication graph, shows how are they connected (supported by proofs). It overviews various RL algorithms and classifies them by which abstraction and conditions they use. Finally, based on their theoretical findings, the authors suggests a minimalist algorithm (and its variants), which they empirically evaluate and compare, from various angles.", "questions": "- In Sec. 5.2, you write: \"Surprisingly, model-free RL (ϕ_{Q∗}) performs worse than ϕ_O.\" Isn't that expected?\n- Given the scope of the article, why did you preferer conference vs. journal? (not answering this question can be understandable)\n- Please elaborate on \"This suggests that the primary advantage ALM(3) brings to model-free RL, lies in state representation rather than policy optimization\" from Sec. 5.1. Isn't your model based on ALM? How does the stripped-down version of ALM (that you use as a basis for your augmentations) compare to ϕ_Q∗ (TD3)? Is it the same?\n\nSuggestions:\n- define FKL / RKL abbr. - good place good be on p. 5 after \"includes forward and reverse KL\".\n- p. 5: detached from the computation graph [OR] using a copy ...\n- in Fig. 6 caption, mention that RP+OP / RP+ZP are phased; it is not obvious what the difference is and why they should perform worse\n\nNitpicking:\n- legend in Figure 5 is slightly different from the graphs", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is concise and clearly written. It is noteworthy that the studied topic is very large and requires lot of details, and that the Appendix contains some interesting details and even novel contributions (e.g., details regarding the implication graph), which implies that a journal format would suit this work better. Nevertheless, the authors manged to fit the most interesting information in the page limit, and hence I will focus only on the main text. \n\nGiven by the number of works focusing on representation learning (Table 1), it is clear that it is an important topic. Hence the theoretical connections given in the paper between the individual parts of the abstractions is very significant to avoid double and superfluous work. Also, the contribution about stop-gradients, connection to representation collapse and the experiments showing the rank of the weight matrices is very interesting. The theoretical part of the paper is very well done and would alone justify my rating of the paper. \n\nIn the experimental section, the authors verify their theoretical contributions. The experiments about stop-gradients are good. The experiments showing different variants of the minimalistic algorithm (Figure 3) are somewhat inconclusive, or require further discussion (e.g., claims about better/similar performance compared to ALM(3), or no justification for superior performance of ALM on Humanoid). I appreciate the negative result in 5.2 for ZP objective hypothesis.", "weaknesses": "As mentioned, the paper sometimes outsources interesting details into the Appendix and the experiment in 5.1 is inconclusive and requires unbiased and fair discussion (also see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)