Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information
Authors: Riashat Islam
Manan Tomar
Alex Lamb
Hongyu Zang
Yonathan Efroni
Dipendra Misra
Xin Li
Harm van Seijen
Remi Tachet des Combes
John Langford
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline RL
exogenous information
representation learning
latent state recovery
robustness

Abstract: Learning to control an agent from data collected offline in a rich pixel-based visual observation space is vital for real-world applications of reinforcement learning (RL). A major challenge in this setting is the presence of input information that is hard to model and irrelevant to controlling the agent. This problem has been approached by the theoretical RL community through the lens of exogenous information, i.e, any control-irrelevant information contained in observations. For example, a robot navigating in busy streets needs to ignore irrelevant information, such as other people walking in the background, textures of objects, or birds in the sky. In this paper, we focus on the setting with visually detailed exogenous information, and introduce new offline RL benchmarks offering the ability to study this problem. We find that contemporary representation learning techniques can fail on datasets where the noise is a complex and time dependent process, which is prevalent in practical applications. To address these, we propose to use multi-step inverse models, which have seen a great deal of interest in the RL theory community, to learn Agent-Controller Representations for Offline-RL (ACRO). Despite being simple and requiring no reward, we show theoretically and empirically that the representation created by this objective greatly outperforms baselines.  

Under review as a conference paper at ICLR 2023
AGENT-CONTROLLES REPRESENTATIONS: PRINCIPLED OFFLINE RL WITH RICH EXOGENOUS INFORMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Learning to control an agent from data collected offline in a rich pixel-based vi- sual observation space is vital for real-world applications of reinforcement learn- ing (RL). major challenge in this setting is the presence of input information that is hard to model and irrelevant to controlling the agent. This problem has been approached by the theoretical RL community through the lens of exogenous information, i.e, any control-irrelevane information contained in observations. For example, a robot navigating in busy streets needs to ignore irrelevant information, such other people walking in the background, textures of objects, birds in the sky. In this paper, we focus on the setting with visually detailed exogenous in- formation, and introduce new offline RL benchmarks offering the ability to study this problem. We find that contemporary representation learning techniques can fail on datasets where the noise a complex and time dependent process, which is prevalent in practical applications. To address these, we propose to use multi-step inverse models, which have seen great deal of interest the RI. theory commu- nity, to learn Agent-Controller Representations for Offline- RL (ACRO), Despite being simple and requiring no reward, we show theoretically and empirically that the representation created by this objective greatly outperforms baselines.
INTRODUCTION
Effective real-world applications of reinforcement learning or sequential maaiing must cope with exogenous information in sensory data. For example, visual datasets of robot navigating in busy city streets might contain information such as advertisement billboards, birds in the sky or other people crossing the road walks. Parts of the observation (such birds in the sky) irrelevant for controlling the agent, while other parts (such as people crossing along the navigation route) are extremely relevant. How can we effectively learn representation of the world which extracts just the information relevant for controlling the agent while igmoring irrelevant information?
Real world tasks are often more easily solved with fixed offline datasets since operating from offline data enables thorough testing before deployment which can ensure safety, reliability, and quality in the deployed policy (Lange et al., 2012; Ebert et al., 2018; Kumar al., 2019; Jaques et al., 2019; Levine et al., 2020). The Rf setting also eliminates the need to address exploration and planning which comes into play during data collection. Although approaches from representation learning have been studied in the online-RL case, yielding improvements, exogenous information has proved to be empirically challenging. A benchmark for learning from offline pixel- pixel-based data (Lu et al., 2022a) formalizes this challenge empirically. Combining these challenges, is it possible to learn distraction-invariie representations with rich observations in offline RL.?
Approaches for discovering small tabular-MDPs (5500 discrete latent states) linear control prob- lems invariant to exogenous information have been introduced (Dietterich et al., 2018; Efroni et al., 2021; 2022b;a; Lamb al., 2022) before. However, the planning and exploration techniques in these algorithms are difficult to scale. A key insight that Efroni et al. (2021); I; Lamb et al. (2022) uncovered the usefulness of multi-step action prediction for learning exogenous-invan invariant representation.
This climination however can make offline RI. more difficult if the wrong data is collected.

---

Under review as a conference paper at ICLR 2023
1-Ste--rreene
Beageesn Obsarittio
ACRO
Narmazzed Returns
Latend on
100
Offine Algoriti
Normalized Returne
at,y F't
15
Zt; dt
Oftine Datowet
Committed Imaga
Change Videa
Background Agant
Figure 1: Left: Representation Learning for Visual Offline RL in Presence of Exogenous In- formation. We propose ACRO, that recovers the controller latent representations from visual data which includes uncontrollable irrelevant information, such as observations of other agents acting in the same environment. Right: Results Summary. ACRO learns to ignore the observations of task irrelevant agents, while baselines tend capture such exogenous information. We use differ- ent offline datasets with varying levels of exogenous information (Section 5) and find that baseline methods consistently under-perfome w.r.t. ACRO, supported by our theoretical analysis.
Following these, we propose to learn Agent-Contrenn multi-step inverse models, which predict actions given current and future observations as in Figure 2. ACRO avoids the problem of learning distractors, because they are not predictive of the agent's 's ac- tions. This property even holds for temporally-corred time, multi-step inverse models capture all the information that is sufficient for controlling the agent (Efroni et al., 2021; Lamb et al., 2022), which we refer to as the agent-controller representation. ACRO is learned in entirely reward-free fashion. Our first contribution is to show that ACRO outperforms current baselines on datasets from policies varying quality and stochasticity. Fig- ure gives illustration of ACRO, with summary of our experimental findings.
Representations for Offtine-Ri RL (ACRO) using
exogenous information. At the same
A second core contribution of this work is to develop and release several new benchmarks for offline- RL designed to have especially challenging exogenous information. In particular, we focus diverse temporally-correla
exogenous information, with datasets where (1) every episode has different video playing in the background, (2) the same STL-10 image is placed to the side or corner of the observation throughout the episode, and (3) the observation consists of views of nine independent agents but the actions only control one of them (see Fig. 1). Task (3) is particularly challenging because which agent is controllable must be learned from data.
Finally, also introduce a new theoretical analysis (Section 3) which explores the connection between exogenous noise in the learned representation and the success of Offline-RL. In particular, we show that Bellman completeness is achieved from the agent-controller representation of ACRO while representations which include exogenous noise may not verify it. Bellman completeness has been previously shown to sufficient condition for the convergence of offline RL methods based on Bellman error minimzation (Munos, 2003; Munos & Szepesvári, 2008; Antos al., 2008).
2 ACRO: AGENN-CONTTOLLEE REPRESENTATIONS FOR OFFLINE-RL
2.1 PRELIMINARIES
We consider Markov Decision Process (MDP) setting for modeling systems with both relevant and irrelevant components (also referred as exogenous block MDP in Efroni et al. (2021)). This MDP consists of a set of observations, X: a set of latent states, Z: a set of actions, A; transition distribution, T(='] 2, a); emission distribution q(z z); reward function It X x A - R; and a start state distribution (2)(2). We also assume that the support of the emission distributions of any two latent states disjoint. The latent state is decoupled into two parts
(s, e) where e S the agent-controller state and e E is the exogenous state. For 2, E Z, a E A the transition function is decoupled T(=' a) T(s' I s,a)Te(e' e), and the reward only depends
2

---

Under review as a conference paper at ICLR 2023
(s, a). These definitions imply that there exist mappings 6. X S and Ó..e X - E from observations to the corresponding controller and exogenous and uncontrollable latent states. The agent interacts with the environment, generating latent state, observation and action sequence, (21, #1, 01, 22, 42, ....) .) where 21 ~ ja(-) and I2 ~ q(- zt). The agent does not observe the latent states (21; 22, ), instead receiving only the observations (21, ). The agent chooses actions using a policy distribution (a x). A policy is an exo-free policy if it is not function of
the exogenous noise. Formally, for any 21 and æ2, if &. (x1) = (12), then T(: x1) = T(- x2).
2.2 PROPOSED METHOD
We consider learning representations from an offline dataset D = (A) consisting of sequences of N ob- servations X' (I1, 22, I3, .... NN) and the correspond- ing actions A = (41, 02, as...., ag) We are in the rich- observation setting, Le., observation F; E R is sufficient to decode žz. Our focus is on pre-training an encoder : IR. R2d on DD such that the frozen representation = ((x,) is suitable for offlin policy optimization.
0
St
a+
D t-+k
To learn representations that discard exogenous infor- mation, we leverage prior works from the theoretical RL community and train multi-step inverse action pre- diction model, which captures rongrrange dependencies thanks to its conditioning on distant future observations. This leads to the ACRO objective, namely to predict the action conditioning on (b(xze) and (x+++)) Note that even though we are conditioning on the future observation, we
sttk
Figure 2: ACRO a multi-step inverse model that predicts the first action con- ditioned on the current state and the fu- ture state. denotes concatenation.
only predict the first action instead of the sequence of ac-
tions up the k-th timestep, as the former is easier to learn.
Our proposed method, which call optimizes the following objective based on a multi-step inverse model:
Representations for Offfine-RL RL (ACRO),
de E arg max EE
E log (IP(a; I d(zx), ((x++))))
(1)
de
(0, K-)
This approach motivated by two desiderata: (i) ignoring exogenous information and (ii) capturing the latent state that is necessary for control. The following invariance lemma (Lemma 1, see Ap- pendix A for proof) states that optimal action predictor models can be obtained without dependence on exogenous noise, when the data-collection policy is assumed not to depend on it either.
Lemma 1 (Invariance Lemma: Multi-step inverse model is invariant to exogenous information, Lamb et al. (2022)). For any exo-free policy π : X A. for all at A. and (I1,
supp Px (X,,,, Xx+k):
(a Z1, T2+k) P., (a I Φ. (2x), Φ. (21+k))
At the same time, prior works have shown that single step inverse models of action prediction can fail to capture the full controller latent states (Efroni al., 2021; Lamb et al., 2022; Hutter & Hansen, 2022). One type of counter-example for single-step inverse models stems from failure to capture long-range dependencies. For example, in an empty gridworld, a pair of positions which are two or more spaces apart can be mapped to the same representation without increasing the loss of a one-step inverse model. Another simple counter-example involves a problem where the last action the agent took is recorded in the observation, in which case the encoder can simply retrieve that action directly while ignoring all other information (although recording all recent actions in the observation is an issue for multi-step inverse models). The use of multi-step inverse models resolves both of these counter-examples and is able to learn the full agent-controller state (Lamb et al., 2022).
We emphasize here that even though inverse models of action prediction have appeared in past lit- erature (as discussed in related works), they are often proposed for the purposes of exploration and reward bonus. In contrast, we propose learn the multi-step inverse model to explicitly uncover
3

---

Under review as a conference paper at ICLR 2023
representation that contains only the controller, endogenous part of the state. Recently, Lamb et al. (2022) proposed a multi-step inverse model where the learnt representation o(-) is regularized, so that o(-) discards irrelevant details from observations This was accomplished by using vector- quantization on the encoder's output, forcing discrete latent states to be learnt for constructing a tabular MDP for latent recovery. In contrast, ACRO learns the continuous endogenous latent state, without a bottleneck, and the learnt pre-trained representation 6(-) is later used for policy optimiza- tion in offline RL. More details of our algorithm are discussed Appendix I.
BENEFITS OF EXOGENOUS INVARIANT REPRESENTATION IN OFFLINE RL
Due to its importance to practical applications, the offline RL setting has been extensively stud- ied by the theoretical community. The majority of provable malue-based offline RL algorithms follow a Bellman error minimization approach (Munos, 2003; Munos & Szepesvári, 2008; Antos et al., 2008), in line with the techniques used in practice. The common representational assump- tions needed to derive these results are: (A1) the function class contains the optimal Q function (realizabilitt). (A2) the data distribution is sufficiently diverse (concentrabilityy, and (A3) Bellman completeness (Munos & Szepesvári, 2008). This last condition is the most subtle one, it states that the function class can properly represent the Bellman backup of any function it contains.
Definition 2 (Bellman Completeness). We say that a function class F is Bellman complete if it closed under the Bellman operator. That is, for any S E F it holds that TJ E F, where (Tf) a) = R(x, a) + [e maxx f'(x', for all (x,, a) E X x A.
Chen & Jiang (2019) conjectured that (A1) and (A2) alone are not sufficient for sample efficient offline RI, and, recently, Foster et al. (2021) established a lower bound proving this claim. Thus, the representational requirements needed for offlin RI. are more intricate than in supervised learning.
With these observations in mind, we highlight a key advantage of the agent-controller representation é. relatively to other representations in the offline RL setting. Namely, show one construct a Bellman complete function class on top of o., while some representations that include exoge- nous information provably violate Bellman completeness. To formalize these claims, we denote by 2s {s, a) + [0 (s.a) E S x A} the set of Q-functions defined over S, and for given representation d. we let F(Ø) {(s, a) + Q()(s),) Q E Qs, (s.a) a) E S A} denote the set of Q-functions defined on top of co The following proposition states that the Agent-Controlles rep- resentation leads to a Bellman complete function class (all proofs in Appendix D.1/ Appendix D.2). Proposition 3 (ACRO Representation is Bellman Complete). F(ø.) is Bellman complete.
Next, we show that there exists representation strictly more expressive than ACRO (i.e., one that includes exogenous information) which, surprisingly, violates the Bellman completeness property. Proposition 4 (Exogenous Information May Violate Bellman Completeness). There exists é which is a refinement of Φ. such that F(Ø) is not Bellman complete.
This proposition implies that exogenous information being included in the representation may break the Bellman completeness assumption, which is a requirement for establishing the convergence of offline RL algorithms based on Bellman error minimization. From this perspective, additional information in the representation may deteriorate the performance of offline RL. Conversely, a coarser representation may trivially violate the realizability assumption A1: such representation may merge states on which the optimal Q-function differs, preventing it from being realized.
Together, these observations motivate the experimental pipeline used this work: learn the agent- controller representation by optimizing Equation 20, then perform offline RL on top of it. In doing so, we obtain representation that sufficient for optimal performance, and yet filters the exogenous information which can (i) be impossible to exactly model, and (ii) hurt the offline RL performance.
4 RELATED WORK
In Table 1, we list prior works and whether they verify various properties, in particular invariance to exogenous information. An extended discussion on related works provided in Appendix E.
Lett X' he finite set of elements. Given partition P. of X let its induced equivalence relation be denoted by rpp A partition P; is finer than P3 if for any 22, X such that xi ruP, 2%2 it also holds that zi rep $2-
4

---

Under review as a conference paper at ICLR 2023
Table 1: Overview of Properties prior works on representation learning RL, in particular their robustness to exogenous information. The comparison to ACRO aims to be as generous as possible the baselines. X is used to indicate a known counterexample for given property.
TD3 (DiQ)
1-Step Inverse
Behavior Cloning
BYOL Explore
ACRO
Algorithms
CURL
DRIML
DBC
AE
(Ours)
Time-Ind. Invarintri
V
Reward Free
X
X
V
/
Exogenous Invuriant
/
V
?
Non- Non-Expert Policy
/
V
V
X
V
Rep.
X
X
X
/
Inverse Dynamics Models. One-Step Inverse Models predict the action taken conditioned on the previous and resulting observations. This is invariant exogenous noise but fails capture the agent-controlerr latent state (Efroni et al., 2021), as previously discussed in Section 2.2. This can result from inability to capture long-range dependencies (Lamb et al., 2022) or could result from trivial prediction of actions using dashboard displaying the last action taken, such as the brakelight which turns on after the break is applied on a car (De Haan al., 2019). Behavior Cloning predicts actions given current state and may also condition on future returns. This is invariant to exogenous noise but can struggle with non-expert policies and generally fails to learn agent-controller latent state. Inverse models predicting sequences of actions, like GLAMOR (Paster al., 2020) considers an online setting where they learn an action sequence as a sequential multi-step inverse model and rollout via random shooting and re-scoring, using both the inverse-model accuracies and an action- prior distribution. On the other hand, we learn a representation fully offline with a multi-step inverse model and then do policy optimization over the learnt representation, given fixed dataset.
Contrastive Methods. CURL (Augmentation Contrastive, Laskin et al. (2020)) learns a repre- sentation which is invariant to class of data augmentations while heing different across random example pairs. Depending on what augmentations and datasets are used, the learnt representations would generally learn exogenous noise and also fail capture agent-controller latent states (which could be removed by some augmentations). HOMER and DRIML (Time Contrastive, Misra et al. (2020), Mazoure et al. (2020)) learns representations which can discriminate between adjacent ob- servations in rollout and pairs of random observations. This has been proven to not be invariant to exogenous information and neither can capture the
latent state (Efroni al., 2021).
Predictive Models. Autoencoders learn to reconstruct an observation through representation bot- tleneck. Generative modeling approaches usually capture all information in the input space which includes both exogenous noise and the agent-controller latent state (Hafner et al., 2019). Wang et al. (2022a;b) showed that a generative model of transition in the observation space can decom- pose the space into agent-controller state and exogenous information. While this does, in principle, eventually achieve an Agent-Controlle nous representation and its dynamics before discarding the information. et al., 2022) achieved impressive empirical results in online exploration by predicting future rep- resentations based past representations and actions. While this approach can ignore exogenous information, there is no guarantee that it will do so, nor that it will learn the full agent-controller state.
representation, it comes at the cost of learning the exoge-
-OL-EXXLOOE (Guo
Neward-Dependevv Methods. DRQV2 (Kostrikov et al., 2020) learns a value function from offline tuples of observations, rewards, and actions. This could feasibly ignore exogenous noise given suitable data-collection policy, but will not generally learn the full agent-controller latent state due a heavy dependence on the reward structure. DBC (Bisimulation, Zhang al. (2020) learns representations which have similar values under learned value function. In general, bisimulation an overly restrictive state abstraction that fails to transfer different tasks.
RL with Exogenous Information. Several prior works study the RL with exogenous information problem. In Dietterich et al. (2018); Efroni et al. (2022a;b) the authors consider specific represen- tational assumptions on the underlying model, such as linear dynamics or factorized representation of the exogenous information in observations. Our work focuses on the rich observation setting where the representation itself should be learned. Efroni et al. (2021) proposes deterministic path planning algorithm for being invariant to exogenous noise. Unlike their approach which requires interaction with the environment using a tabular-MDP, ACRO is purely offline algorithm. Lastly,
5

---

Under review as a conference paper at ICLR 2023
Table EASY-ExO. Comparison of different representation methods on the standard v-d4rl bench- mark, without additional exogenous information. ACRO consistently outperforms baseline methods in visual offline data. Performance plots in Appendix Figure 10. 10 seeds and std. dev. reported.
ENVIRONMENT
DATASET
ACRO
DRIMI
HOMER
DRQV2
CURL
1-STEP INVERSE:
Expert
451.0 3.9
330.2 29
2278 = 1.6
256.9 + 2.2
213.0 = 0.6
239.9 + 0,4
CHIRTAIL-RUN
Expeet
466.0 + 3.2
399.2 4:
390.7 = 1.3
388.1 + 3.5
328.4 2.1
299.3 4: 0.6
Mediumm
528.7 + 0.8
508.5 0.7
518.1 04
188.3 1: 0.5
377.0
100.3 1: 0.4
Modium Moplun-Replay
233.3
333.2
Expert
485.10
2.5
831.5
WALKER- WALK
Medium-Exxpet
=
+
Medium
125.6
389.4
Medium-Replay
227.8
204.3 3.4
318.9
308.5
234.8
146.7
Expert
17.5
21.6:=044
0.3
28.5-+00.2
25.4
HUMANOm-WALL
Medium-Exppe
142.4
26.8 4:
31.8 0.1
70.8 0.5
63.2
56.3 0.5
Mediuum
103.8 4:
35.1 0.3
53.8 0.4
96.1 1: 0.9
40.6 0.4
46,7 0.1
Medium-Replay
197.8 + 0.5
92.6 + 0.3
102.7 = 0.6
121.0 0.4
77,8
100.7 + 1.1
AVERAGE
415.8
270.0
327
363.9
209.7
28%
the work of Lamb et al. (2022) suggests an endogenous latent state recovery algorithm through the use of a discretization bottleneck to construct a small tabular-MDP. In contrast, ACRO recovers the continuous counterpart of the endogenous latent state space directly, without the need construct tabular-MDP. Moreover, here we focus on reward optimization, and not only latent state discovery.
5 EXPERIMENTS: OFFLINE RL WITH EXOGENOUS INFORMATION
This section provides extensive analysis of rep- resentation learning from visual offline data un- der rich exogenous information (Figure 3). Our experiments aim to understand the effect of ex- ogenous information and if ACRO can truly learn the agent controller state and thus improve performance in visual offline RL. To this end,
Timesteps
Episode
Episode
L+0
t-1
1-2
t-0
t+1
1+2
Distractor Difficulty
evaluate ACRO against several state of the art representation learning baselines across two axes of added exogenous information: Tempo-
Medium Exo a)
ral Correlation and Diversity, hence character- izing the level of difficulty systematically. We find that under exogenous information in offline
Hard Exo a]
RL, the performance of several state of the art representation learning objectives can degrade dramatically.
Hard Exo
Two particular challenges in the datasets we ex- plore are the temporal correlation and diversity in the exogenous noise. Temporal Correlation: Exogenous noise which lacks temporal correla- tion hime-inneeendent noise) is relatively easy to filter out the representation, especially in tasks where the agent-controller latent state has strong temporal correlation. Diversity: Similarly for the other axis, if exogenous noise is more diverse, it has greater impact the complexity of the subsequently learned representation. For example, if there are only two possible distracting background images, in the worst case the cardinal- ity of a discrete representation is only doubled. On the other hand if there are thousands of possible distracting background images, then the effect on the complexity of representation would be far greater. We primarily categorize our novel visual offline datasets into three categories (Figure 3 in appendix provides observations under different exogenous distractors):
Figure 3: Examples of Different Categories of Exogenous Information. Further details of dif- ferent exogenous information in offline datasets, with visual examples, are provided in Ap- pendix F.I.
EASY-ExO. Exogenous noise with low-diversity and no time correlation. a) Visual offline datasets from v-d4rl benchmark (Lu et al., 2022b) without any background distractors; b) Dis- tractor setting (Lu al., 2022a) with single fixed exogenous image in the background.

---

Under review as a conference paper at ICLR 2023
HOMER
Comner
Side
Fixed Video
Performance
600
Export
Medium-Experi
Medium
Expert
Mecium-Exppot
Medium
Expert
Medium-Expert
Medium
Figure 4: MEDIUM-ESE Results. Performance comparison of ACRO with several other baselines, with varying levels of exogenous information settings, either from STL10 dataset (Coates et al., 2011) or fixed video distractors in background during offline data collection.
MEDIUM-EXO. Exogenous noise with either low-diversity or simple time-correlation. a) Exoge- nous image placed in the corner of agent observations, changes per episode; b) Exogenous image placed on the side of agent observations, changes per episode; A single fixed exogenous video playing in the background.
HARD-EXO Exogenous noise with both high-diversity and rich temporal correlation. a) Exoge- nous image in the background which changes per episode; b) Exogenous video in the background which changes per episode; c) Exogenous observations of nine agents placed in grid, but the ac- tions only control one of the agents (see Figure 1).
Experiment Setup. We provide details of each EXOGENOUS DATASETS in Appendix F.I, along with descriptions for the data collection process in Appendix G. Following Fu et al. (2020); Lu et al. (2022b), we release these datasets for future use by the RL community. All experiments involve pre-training the representation, and then freezing it for use in offline RI algorithm. We use TD3 + BC as the downstream RI. algorithm, along with data augmentations (Kostrikov et al., 2020), combination which has been shown to be a reasonable baseline for visual offline RL (Lu et al., 2022b). Experiment setup and implementation details are discussed in Appendix I.
Baselines. We compare five other baselines, which are standard for learning representa- tions of visual data. The baselines we con- sider are: (i) two temporal contrastive learn- ing methods, DRIMI. (Mazoure et al., 2020) and HOMER (Misra et al., 2020); (ii) a data augmentation method, DRQ Kostrikov et al. (2020), and spatial contrastive approach, CURL Laskin et al. (2020); and (iii) inverse dynamics model learning, i.e., 1-step inverse action prediction (Pathak et al., 2017). We do not consider baselines such SPR (Schwarzer et al.. 2020) and SGI (Schwarzer et al., 2021) which work well on the ALE Atari bench- mark but not on continuous control bench- marks (Tomar al., 2021). We also include preliminary Atari results in Appendix Section H where the representations are pre-trained using ACRO and used over a Decision Transformer architecture (Chen et al., 2021b).
ACRO
DRIML HOMER
Performance
1 Stop Inverso DrOv2
80
Every-Sill
Meduu-Shil
Hard--Sal
Figure 5: Normalized results across two domains from the v-d4rl distractor suite with varying levels of data shift severity. The easy, medium and hard categories are within the v-d4rl distractor suite of varying shift severity.
5.1 EASY-EXOGENOUS INFORMATION OFFLINE DATASETS
Table 2 summarizes results from the v-d4rl benchmark with visual offline data (Lu et al., 2022a). We label this as EASY-ExO since the dataset only contains blank background without any additional exogenous noise being added. We find that ACRO learns a good agent-controller latent representa- tion from pixel data with apparent noise in observations, and can lead to effective performance improvements through pre-training representations. Extending results of EASY- Exo with static un- correlated image background distractors from the v-d4rl benchmark, we see that the performance significantly decreases for all methods, while ACRO can strongly outperform all baselines, with the smallest drop in performance. Figure 5 shows normalized results across two different datasets and domains from v-rl.. The distractors in this case belong to varying degree of shifts in the data distribution, according (Lu al., 2022a).

---

Under review as a conference paper at ICLR 2023
ACRO
DRIML
Homer
CURL
One-Step-Inverse
DrQv2
Correlated-Image
Video
Background-Agent
500
Walker
250
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
1.0
0.0
0.2
0.4
0.8
1,0
Cumulative Returns
Humanoid
50
10
0.0
02
0.4
0.6
0.8
170
0.0
0.2
0/4
0.6
0.8
1.0
0.0
0.2
0.
0.5
0.8
1.0
200
Cheetah
200
0.0
0.2
0.5
0.8
1.0
00.0
074
0.6
0.8
1.0
000 0.2
0.4
035
0.0
Iterations
Figure 6: HARD-EXO Results Normalized performance across three datasets: medium, expert. and medium-expert. First Column. Time correlated exogenous distractor in background: Second Column. Video distractors that changes per episode background; Third Column. Multiple background agent observations distractors are placed in grid of agent observation space
5.2 MEDIUM- EXOUUN-EX INFORMATION OFFLINE DATASETS
Figure 4 shows normalized results across three domains (cheetah-run run, walker walker-walk,, humanoid-walk, for the MEDIUM Exx setting. Among these, the fixed background video is the hardest task. As ex- pected, most methods underperform expert and expert policies. However, ACRO consistently outperforms all methods across all datasets and distractor settings. Note the high variability in performance of baselines when changing the type of exogenous information (from corner, to side, to fixed video). Conversely, with the exception of Corner + Medium policy, ACRO performs similarly for all three settings. This suggests that base- line methods leam representations that are affected by the exogenous information, while ACRO remains rather impervious
data collected from medium policy, compared to medium-
5.3 BARD-EXOGENOUS INFORMATION OFFLINE DATASETS
With correlated exogenous noise in the form of either images video. observe that baseline representation objectives can be remarkably broken. Figure 6 shows normalized performance com- parisons across different types of datasets (expert, medium-expert, medium) for three different types of HARD-EXO settings. Comparatively, ACRO can be more robust to the hard exogenous distrac- tors, even though the HARD-EXO types increase in difficulty, the maximum performance reached by all methods can degrade. Among the three HARD-EXO settings, changing video distractors in background during data collection seems to be the hardest, leading to performance drops for most methods. This suggests there is strong correlation issue between the representation and the video pixels, which breaks when the episode changes, hence leading to the worst scores across the three settings. However, ACRO remains comparatively robust and outperforms all baselines across all the HARD-EXO settings.
5.4 METHOD ABLATIONS AND ANALYSIS
We compare ACRO with three of its variations, 1) when = 1, i.e. a standard one-step inverse model; 2) with X++ not provided as input, i.e. simply training the representation with behavior cloning loss; and 3) when m(k) the timestep embedding, is additionally provided as input. Ablations shown over three different policies: random, medium-replay and expert in Table 3.
8

---

Under review as a conference paper at ICLR 2023
For both random and medium-replas policies, & 1 leads to similar results than when is randomly cho- from 1 to 15. ACRO performs much better der an expert policy. We conjecture that the bene- fits of larger l can only be realized when the pol- icy is of a high enough quality to preserve informa- tion over long time horizons. Additionally training a behavior cloning loss performs similarly to ACRO for the medium-replay and expert datasets. How- ever, when the actions come from a random pol- icy, ACRO performs much better while the behavior cloning ablation collapses completely. This result is analyzed theoretically in Appendix B, which shows that ACRO is equivalent to behavior cloning under deterministic and fixed expert policy, but should be much better otherwise. Adding a embedding generally degrades performance, although the effect is inconsistent. These results suggest that ACRO is more well rounded and robust objective than other variants.
Table 3: Ablations for different policies. The highlighted cells indicate where each variant fails to match ACRO's performance, hence showing that each component of ACRO is essential for consistently good per- formance. seeds and std. dev. reported.
EXVIRONMENT
RANDOM
MEDIUM-REPLAA
EXPERT
ACRO
$2.9 5.5
228.8 + 50.1
525.8 89.0
94.7
241.0 9.9
21
WITH
43.1
153
302.2
Visualizing Reconstructions. Having learnt representation, we can train a decoder over it to minimize the reconstruction loss given the orig- inal observation. Such reconstructions would therefore measure how much information in the original observation is preserved in the repre- sentation, and thus act as metric for evaluat- ing the quality of representations. We compare such reconstructions tah domain where the exogenous noise comes from a video playing in the background. No- tably, ACRO is able to remove most back- ground information while keeping the relevant body pose information intact. On the other hand, DRIML performs contrastive compar- isons between states in a given trajectory and is not able to remove exogenous information quite as well. DRQ is able remove exoge- nous noise but is unable to learn the controller state.
Original
ACRO Reconstruction
Figure 7 for the chee-
DRIML Reconstruction
DRQ Reconstruction
Figure 7: Reconstructions from decoder with a static background image per episode: Top- Bottom: Original, ACRO, DRIML, DRQ.
DISCUSSION
In this work, we introduced offline RI. datasets with varying difficulties of exogenous information in the observations. Our results show that existing representation learning methods can significantly drop in performance for certain types of exogenous noise. We presented ACRO, a pre-training objective for offline RL based on multi-step inverse prediction model, and showed it is far more robust to exogenous information, both theoretically and empirically. We hope this work will drive future interests in offline RL under different definitions of exogenous information.
Limitations and Future Work. Since ACRO does not require reward information for learning representations, it is natural to wonder if data from multiple datasets (e.g. combining random and reciium-replay) or different domains (c.g. a transfer learning setting) can be used to train a stronger representation than when using single dataset. Additionally, under varying transition dynamics across tasks, model-based counterpart of ACRO would be interesting to study. For the domains and tasks considered in this work, it would be interesting to quantify how accurately ACRO recovers the underlying relevant or endogenous latent states. We also studied variations of ACRO through ablation studies, and believe that with carefully designed information bottlenecks, ability of ACRO to further recover relevant latent states for empirical improvement can be significantly improved.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
The experiment details and use of different exogenous offline datasets are discussed in section 5. We provide further architecture and algorithm details in appendix I. For implementation, open source codebase from the visual D4RL benchmark (Lu et al., 2022a) and implement ACRO and other baseline representation objectives using the same structure and pipeline from the code- base. All the representation learning objectives use the same encoder architecture and optimization specifications. the main draft, we provide normalized performance plots for comparison. where we average across different types of datasets (expert, medium, mediium-experty and we provide individual performance plots for comparison in the appendix. We also provide code for our im- plementation along with supplementary materials. The use of different exogenous offlin datasets hopefully introduces new benchmarks to be considered in the offline RI. community, and we plan to release these benchmark datasets for future use. The proof for propositions and lemmas are included in details in the appendix.
the
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline re- inforcement learning with diversified q-ensemble. In Marc' Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pp. 7436-7447, 2021.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman- residual minimization based fitted policy iteration and single sample path. Machine Learning. (1(1):89-129, 2008
Jacob Buckman, Carles
Gelada, and Mare G. Bellemare. The importance of pessimism in fixed- dataset policy optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Cynthia Chen, Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Kuang-Huei Lee, Neel Alex, Steven H. Wang, Ping Luo, Stuart Russell, Pieter Abbeel, and Rohin Shah. An em- pirical investigation of representation learning for imitation. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 202 la.
Jinglin Chen and Nan Jiang. Informatiin--heorrtic considerations in batch reinforcement learning. In International Conference on Machine Learning, pp. 1042-1051. PMLR, 2019.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15084-15097. Cur- neurips htcp/aaper/222///// pdf.
ran Associates, Inc., b02 URL http::::/ttocceeettthth httl:/EE888862222220011022222223
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215-223, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https: http:///pooeeeieeee/eee/ee mlr prwss/ vttp:/v1/////// coates 1a html.
Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
Thomas Dietterich, George Trimponias, and Zhitang Chen. Discovering and removing exogenous state variables and rewards for reinforcement learning. In International Conference on Machine Learning, pp. 1262-1270. PMLR, 2018
10

---

Under review as a conference paper at ICLR 2023
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pp. 1665-1674. PMLR, 2019.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex X. Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. CoRR, abs/1812.00568,, 2018. URL http: //tr://aaiiii .rgg httg//bbs/e abs/ 00568.
Yonathan Efroni, Dipendra Misra, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Provably filtering exogenous distractors using multistep inverse dynamics. In International Conference on Learning Representations, 2021.
Yonathan Efroni, Dylan Foster, Dipendra Misra, Akshay Krishnamurthy, and John Langford. Sample-cfficient reinforcement learning in the presence of exogenous information. In Po-Ling Loh and Maxim Raginsky (eds.), Conference on Learning Theory, 2-5 July 2022, London, UK, volume 178 of Proceedings of Machine Learning Research, pp. 5062- 5127. PMLR, 2022a. URL https: / http::/ppoccceeeeee proceedings mlr press vvv.v/v/////// htt.7///ffroin2222 htw.rroonni.a..........
Yonathan Efroni, Sham Kakade, Akshay Krishnamurthy, and Cyril Zhang. Sparsity in partially controllable linear systems. International Conference Machine Learning, pp. 5851-5860. PMLR, 2022b.
Dylan Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline rein- forcement learning: Fundamental barriers for value function approximation. arXiv preprint arXiv:2111.1091 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07211 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 20132-20145, 2021a.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 20132-20145 2021b. URL https: http:///ooeeennnnce///ee hash/ 0dw.66da005555000 www.:550000000022444444444 www.444188865e-bbeeeeee
hww../ceeedddnn...neee.o.o.oo.pe neurips cc/ pccpppppr//22/// html.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning with- out exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2052 2062. PMLR, 2019.
Dibya Ghosh and Mare G. Bellemare, Representations for stable off-policy reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3556- 3565. PMLR, 2020.
Zhaohan Daniel Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Saan-Battien Grill, Yunhao Tang, Michal Valko, Rémi Munos, Mohammad Gheshlaghi Azar, and Bilal Piot. Byol-explore: Exploration by bootstrapped prediction. arXiv preprint arXiv: Arxiv-2206.08333 2022.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:191 1912.01603, 2019.
Marcus Hutter and Steven Hansen. Uniqueness and complexity of inverse mdp models. arXiv preprint ar arXiv:2206.01 192, 2022.
11

---

Under review as a conference paper at ICLR 2023
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR. abx/1907.00056, 2019. URL http://arxiv org/abs / 9077 00456.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- -ien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint wxXvv:200.133449 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report University of Toronto, Toronto, Ontario, 2009.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Alché-Buc, Emily B. Fox, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11761-11771, 2019. URL https: httpss///ccccceeieeeeeooo neurips. cc/ paper/2019/ www.ttassttt.eeeee.e...eaaaaat html.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, Virtual, 2020.
Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra, Dylan Foster, Lekan Molu, Rajan Chari, Akshay Krishnamurthy, and John Langford. Guaranteed discovery of con- trollable latent states with multi-step inverse models. arXiv preprint arXiv:2207.08229 2022.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforcement Learning. 2012.
Romain Laroche, Paul Trichelair, and Remi Tachet des Combes. Safe policy improvement with baseline bootstrapping. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3652- 3661. PMLR, 2019.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020. URL
https: //tp:///aavvvv/aa//////eeeaaaava
org/ alos /20...000... 01643.
Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. CoRR, 0etp/2206604779, 2022a. doi: 10.48550/axXiv.2200007777 URL https: //doi./////// wwlddiioorg.. .rr// 10. 40.48550/arxiv. 2206. 04779.
Cong Lu, Philip Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations.
arXiv preprint arXiv:2206. 04779, 2022b.
Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. Advances in Neural Information Processing Systems, 33:3686-3698, 2020.
12

---

Under review as a conference paper at ICLR 2023
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. abstraction and provably efficient rich-observation reinforcement learning. conference machine learning, pp. 6961-6971. PMLR, 2020.
Kinematic state In International
Rémi Munos. Error bounds for approximate policy iteration. In ICML, volume 3, pp. 560-567, 2003.
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008.
Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference
Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 30100-30112, 2021. URL https: http:::ss///ppceeednnnegee wwwrinurrrs.ooo paper/2021/ httpe//hhhhhhhaaaaaaaaaaa
/ f400d34 7255
.tt.artttatttttllll.... htmll.
Kimia Nadjahi, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with soft baseline bootstrapping. In Proceedings of the 17th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery Databases (BCML-PKDD), 2019.
Keiran Paster, Sheila Mcllraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02419 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity driven exploration by self-supervised prediction. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney. NSW, Australia, 6- August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2787. PMLR, 2017. URL http: /ttpp/eeeeinn....e/ee//p mlr. srress/v700 0ttv000pppp /tt70//paaall1111 haw.haalaa.... atmm..
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforce- ment learning and demonstrations. In Hadas Kress-Gazit, Siddhartha S. Srinivasa, Tom Howard, and Nikolay Atanasov (eds.), Robotics: Science and Systems XIV, Carnegie Mellon University. Pittsburgh, Pennsylvania, USA, June 26-30, 2018 2018
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929 2020.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Char- lin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representa- tions for data-efficient reinforcement learning. In Marc'Aurela Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 12686- 12699, 2021. URL https::/proceeee new.eeeeeennn...........ee.....p cc/pappr/202111111/ 22t0//hhhhh//// 69eba 3467333e66eeee wweeefefffffeee 2eeee388eeaaeeeeeee ab Abstract. html.
Thiago D. Simão, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with estimated baseline bootstrapping. In Proceedings of the 19th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. et al. Deepmind control suite. arXiv preprint arXiv: 18011000000000 2018
Manan Tomar, Utkarsh A Mishra, Amy Zhang, and Matthew E Taylor. Learning representations for pixel-based control: What matters and why? arXiv preprint arXiv:2111.07773 2021.
Masatoshi Uchara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline RL in low-rank mdps. CoRR, abs/21 10.04652, 2021. URL httpa: /tpp::///aaae aaxivv wtt./bo///////a////o/o/a abs//110.. 0 652.
13

---

Under review as a conference paper at ICLR 2023
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Tongzhou Wang, Simon Du, Antonio Torralba, Phillip Isola, Amy Zhang, and Yuandong Tian. De- noised mdps: Learning world models better than the world itself. In International Conference on Machine Learning, pp. 22591-22612. PMLR, 2022a.
Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task-independent state abstraction. arXiv preprint arXiv:2206.1344 13452, 2022b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint MrXiv:200.100742 2020.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL https:////// http///ppeeeeeeeeeeeeeeeeeeeeeee net for.//ooummdm--- 2FWD==ERR....
Chi Zhang, Sanmukh R. Kuppannagari, and Viktor K. Prasanna. BRAC+: improved behavior regularized actor critic for offline reinforcement learning. Vineeth N. Balasubramanian and Ivor W. Tsang (eds.), Asian Conference on Machine Learning, ACML 2021, 17-19 November 2021, Virtual Event, volume 157 of Proceedings of Machine Learning Research, pp. 204-219 PMLR, 2021b.
14

---

Under review as a conference paper at ICLR 2023
Appendix
A INVARIANCE LEMMA PROOF
For any k > O, consider x' E X' such that a and x' are separated by k steps. Both proof' first use bayes theorem, then apply the factorized transition dynamics, and then eliminate terms shared in the numerator and denominator. This proof is essentially the same as lemmas found in Lamb et al. (2022); Efroni et al. (2021), but is presented here for clarity.
Lemma states that the multi-step inverse model is invariant to exogenous noise. For any exo-free policy æ X + A, for all O E A, and De+k) E supp (X, X+++):
Px | 24, F(+k) P- (az (xx), &- (z++k))
(3)
Proof.
P (a x',
(a) Pa Paal(' I, Ir,a)F a
Enr Payll' x.aa))
(
Pria(x' I., a2,a)((a (x())))
Prilll 2, a'a') (.(2))
alx') (+(x)) (x (.(x) ((() (n((z))<(a (x(x))
En g(x"| (2"') (e))) (z.(), a (z)()))(( o.(x))
Pa.ald,()) (x) (. a)) (x)
En Pr, (2') (1), (x))
Relation (a) holds by Bayes' theorem. Relation (b) holds by the assumption that π is uniformly random (in the first proof) exo-free (in the second proof). Relation (c) holds by the factorization property. Thus, F (a x', .) PT (- &x (x)), and is constant upon changing the observation while fixing the agent controller state.
B CONNECTION BETWEEN ACRO AND BEHAVIOR CLONING
In the special case where all data is collected under a fixed deterministic, exogenous-fre free policy, ACRO and behavior cloning become equivalent. This case can still be non-trivial, the start state of the episode is stochastic or if the environment dynamics are stochastic.
Lemma 5. Under fixed, deterministic, and exo-free policy शt : X - A, multi-step inverse model is equivalent to behavior cloning. For all A. and (It, Iz+*) such that P 21, X1+b
kt-k) > 0 we have:
(a (x Pa(ar (xx)))
The proof this claim is simply that behavior cloning already able to predict actions perfectly in this special case, so there can be no benefit to conditioning future observations.
Proof. By the assumption of the deterministic exo-free policy, we have that Pa (a = (x))) (x.)) where à + A is function mapping the latent state the action.
Using bayes theorem we write:
Ps Po (1))) (x(2)))
(a (z.)))
(5)
En. Ps (Teek ( (a" ( (xx)
15

---

Under review as a conference paper at ICLR 2023
For any examples in the dataset and for all d' A:
Case 1: d' = á(Ø.(z)) (x:)). It holds that
Pa(o, (x (e) (x((1)))
(a I (x+)), Px (1++1))
En Ps + (+(2)), aa) (x()))
(e (Ie+* Φ. (xt),, ar =a)
Pa(ae (x.)), o (x+++))
PA (o (Fe+h (x())), at = a')
(a(ae (xr), (1e+1)) 1.
On the other hand, it holds that (a(a: a' o. (xx)) since a' a(d+(1))). Hence, for this case, the claim holds true.
Case 2: d' + (())(x))) It holds that
Ps (1(0,(I14A (().:))))PPt.). (_(2)))
(a (x.)). 0.
En Pa "0( (.(()) a") (a" (x1)))
0
(a I (x.)) d. (I++1)) = P+ (10 (
(x(x)),
a(p+(x:)))
(a (x1). (T(+1)) =.
On the other hand, it holds that Pa = d' a (x+)) = 0 since a' * (())(x))). Hence, for this case the claim also holds true. This concludes the proof since the two distributions are equal in both cases.
C PREDICTING THE FIRST ACTION VS. PREDICTING ACTION SEQUENCES
In ACRO we only predict the first action from 2, Zi+k rather than predicting the entire action sequence. In an environment with deterministic dynamics, we will prove that these two approaches are asymptotically equivalent. For the proof we will also make stronger assumption that the dy- namics are deterministic in the learned latent space, i.e. for the learned encoder Φ, there exists function f such that: o(z,) = f()(z;), ar.y). This assumption will hold for the optimal o, and is also likely to be empirically true since is high-dimensional continuous latent state, thus two points are likely to have exactly the same representation. a stochastic environment, the two approaches different, but will provide counter-example make the case against predicting action sequences.
The ACRO objective optimizes the following
E arg max I de
E log (PP(,, d(æx), ((x++))))
(6)
(O,N) k-)
The k step action sequence prediction approach optimizes:
Φe E arg max Φes
E
log (IP(a..... .r.k 6(xx), o(x++))))
L~U(O,N) k~)(0.))
C.1 DETERMINISTIC DYNAMICS
P(ant*k (((x)) ((T++))
I[ d(xx), ((T+k) ong)
(8)
By the assumption of deterministic dynamics in the latent space:
1+
P(anetk ((xz), o(x++)) = II
P(a; ((xz). ong, ang, M(Iz), ((x++)))
j=t
16

---

Under review as a conference paper at ICLR 2023
After applying the markov assumption have assumed an MDP:
1+1
P(a ((x)); (xx++)))
II
Paa ((x,), (x+++))
(10)
j=t
Now can put this back into the k-step action sequence prediction problem:
L+
Φx arg max
E (-
E k0)
log
II
P(ay d(x)), ((s)))((re)))))
(11)
dEΦ
j=1
b+
@+ E arg max dep
E
E
Σ
log (P(a,) d(r,), (olr,),,(a=+t) ((x++))))
(12)
(-U(0, k-,
Φ. E arg max dE
E
E
E log (P(a) I ((zz) o(
(13)
(0 (0.K) (1.Ckk)
When > k the distributions of t and converge, and we can write:
Φ.
E arg max PE
E
E
log (PP(a, ((xr)), ((T++))))
(14)
(0,( (0,K)
which we can see is the same as the first-action prediction objective that ACRO optimizes.
C.2 STOCHASTIC DYNAMICS
If the environment has stochastic dynamics, predicting future actions reach goal state without conditioning on the preceding observations is very difficult. This is because what action needs to be taken may depend on what actually happened in the environment. For example, if I'm playing a video game, and there is small chance that the game pauses for one minute, the actions will need depend on whether the pause occurred. We can construct stochastic environment which every action following the first action is completely unpredictable. We can imagine an environment where there some information in the observation space which is set randomly on every step and indicates how the agent's 's controls are randomly permuted for that step. In principle, the first-action predictor can easily use this information to adapt what actions it predicts, and can obtain its orig- inal accuracy given sufficient model capacity. On the other hand, the action-sequence predictor (P(a,,.. a++* o(2,), ((x++))) will only be able to predict the first action well, and can have no better than uniformly random accuracy at predicting the remaining actions. This is because only ((x)) contains the information about what has happened in the environment which is necessary for control, the history of past actions do not contain the necessary information. In this example, pre- dicting the sequence of actions makes the task much noisier, while providing no additional signal for the model.
D BENEFITS OF EXOGENOUS INVARIANT REPRESENTATION IN OFFLINE RL
D.1 PROOF OF PROPOSITION 3.
We need show that for any f E F(b.) and x, a, it holds that R(2, a) + Eg'-T( s.a) [max f(x', a')] a'
(15)
contained in F(o.). Since the reward a function of the agent controller representation only, and since f E F(O)). equation 15 can be written
R(z, a) + (. [max f(z', a')]
= R(ob (x), + Ex-T(aao Imax s(, (x'), )')))
= ((),(x), a) + (erTT(( |max (x)))'),, a())]
(16)
17

---

Under review as a conference paper at ICLR 2023
The first relation holds since F().) and by the assumption the reward function (that it is function of the endogenous states). The second relation holds by
-T max f(o, (x.),)))
Σ
Σ
g(ø' (. (0. (=.c(=))
sf Eoupp((e'(ese)
I T(' ()())) d))
Σ Tale' (z,()))
Σ T(' (o.
where (a) holds by the Ex-BMDP transition model assumption,
T(x' | (.,a)= g(x (,(x), dn (z)))T(ó=xx")) (x(x),
dx,c))),
(b) and (c) hold by marginalizing over x' and e'. This establishes equation 16 and the proposition: the function ()), + (o), (())(a)), a')) is contained within F(Ø+) since only depends on Φ..
D.2 PROOF OF PROPOSITION 4.
Consider Ex-BMDP with one action where the agent controller representation is trivial and has a single fixed state (the agent has no ability to affect the dynamics). We will establish a counter- example by constructing a tabular-MDP. Because tabular-MDP special case of more general MDP with continuous states, this will also establish a counterexample for the more general non- tabular setting considered the paper.
Let the observations and dynamics be given has follows. The observation is 2-dimensional vector (Ir)),, x2)) where (11,, x(2) {0, 13. The dynamics deterministic and its time evoluation is given as follows:
T+ (1) x1(1) (2)
x++1(2) (2) x, (2),
where @ is the XOR operation. In this case, the transition model is given by T'x' x, xx = T(x' x))
and ø. Su) where Sq is a single state; since the observations are not controllable the controller representation maps all observations to unique state. Further, assume that the reward function is 0 for all observations.
Assume that (xx) = (x1), i.e., the representation ignores the second feature This representation more refined than øe since the latter maps all observations into the same state. Consider the tabular Q function class on top of this representation QN=2, and consider Q E QN=2 given as follows
Q = 1) =
Q(x: 0.
We now show that TQ is not contained in QN=2. According to the construction of the transition model. it holds that
(TQ)(z: 1,x2 1) = 0 (TQ)(z: = 0, = 1) = 1
(TQ)(z1 1, 0) 1
(TQ) = 0, 0, = =0) = 0. This function cannot be represented by function from QN=2: cannot represent (TQ) since it is not a mapping of the form Ti + IR by the fact that, c.g.,
(TQ) = 1, x2 = 1) + (TQ)(=) = 1, 22 = 0).
Meaning, it depends on the value of 222-
18

---

Under review as a conference paper at ICLR 2023
E EXTENDED RELATED WORK
Representation learning in Offline RL. Representation learning offers an exciting avenue to ad- dress the demands of learning compact feature for state by incorporating the auxiliary task of the state feature within the learning task. Empirical studies on representation learning in Offline RL have been first addressed by ?, which evaluate the ability of broad set of representation learn- ing objectives in the offline dataset and propose Attentive Contrastive Learning (ACL) to improve downstream policy performance. After that, Chen et al. (2021a) investigate whether the auxiliary representation learning objectives that broadly used in NI.P CV domains can help for imitation al. (2022h) further explores the existing challenges for
across different Offline RI. tasks. visual observation input with the Offline RL dataset, meanwhile providing simple modifications several state-of-the-art Offline RL algorithms to establish a competitive baseline. Another branch of representation learning in Offline RL is theoretical side. Uehara et al. (2021) studies the repre- sentation learning in low-rank MDPs with Offline settings and proposes an algorithm that leverages pessimism learn under a partial coverage condition, Nachum & Yang (2021) develops a repre- sentation objective that provably accelerate the sample-effiiincy of downstream Offine RL tasks, Ghosh & Bellemare (2020) theoretically shows that the stability of the policy is tightly connected with the geometry of the transition matrix, which can provide stability conditions for algorithms that learn features from the transition matrix of policy and rewards.
Offine RL. The predominant approach to train offline RL agent is regularizing the learned policy be close to the behavior policy of the offline dataset. This can be implemented by generating the actions that similar to the dataset and restricting the output of the learned policy close to the generated actions (Fujimoto et al., 2019), penalizing the distance between the learned policy and behavior of the dataset (Kumar et al., 2019; Zhang et al., 2021b). or introducing pessimism term to regularize the Q function for avoiding high Q value of the out- distrriibtttoon actions (Kumar et al., 2020; Buckman al., 2021). Some approaches utilize BC as reference for policy optimization with the baseline methods (Fujimoto & Gu, 2021a; Laroche et al., 2019; Nadjahi et al., 2019; Simão et al., 2020; Rajeswaran al., 2018). Some other approaches improve the performance by measuring the uncertainty of the model's prediction (Yu et al., 2020; Kidambi et al., 2020; An et al., 2021).
19

---

Under review as a conference paper at ICLR 2023
F EXOGENOUS INFORMATION DATASETS
In this section, we provide a detailed summary of the different types of exogenous information based datasets demonstrated in Figure 3. In Figure 8 show samples from all of the datasets explored. We further provide details for how each dataset is collected in Appendix G.
F.1 DATASET DETAILS
Background
Agent Observations
Fixed
Video
Distractors
Diverse
Video
Distractors
Image Distractor
(Side)
Image
Distractor
(Corner)
Diverse
Image
Distractors
Figure 8: Summary of Different Exogenous Information Offline Datasets. In this work, we collected new datasets under different exogenous observations for offline RL. Top to Bottom: (a) Background Agent Observations as Exogenous Information (first row) showing endogenous troller agent (in red) and other exogenous agent observations taking random actions from other environments. (b) Fixed and Changing Video Distractors (second and third row respectively) where background video distractor changes per episode during offline data collection. (c) Uncor- related Image Distractors (fourth and fifth row) showing that exogenous distactors can either be on the background, on the side or in the corner of the agent's observation space. (d) Correlated Background Image Distractors (sixth row) where background image distractor remains fixed per episode during data collection, and only changes per episode, to introduce time correlated exogenous image distractors. We find the type of exogenous distractor becomes harder, from uncorrelated to correlated exogenous noise, the ability for baselines to learn good policies significantly breaks, as seen from the performance evaluations; whereas ACRO can still be robust the exogenous infor- mation.
Casy-Exogenos Information (EASY-EXO). A visual RL offline benchmark has been recently proposed in (Lu et al., 2022a), where the authors provided paxel-based offline datasets collected using varying degrees of a soft actor-critic (SAC) policy. Furthermore, (Lu et al., 2022a) proposed a suite of distractor based datasets with different levels of severity in distractor shift, ranging from easy-shift, medium-shift hard-shift. In the EASY-Ex Exo setting, we first consider exee-based offline
20

---

Under review as a conference paper at ICLR 2023
Exogenous Observations
Exagenous Observations
ASRd
DRVI
700
AGRO
HOVER
649
HOVER
120
400
Exegenous Distractor
Exegenous Distratorr
Figure 9: Summary of Experiment Results on Walker Domain with Expert Dataset: We vary the type of exogenous distractors present in offline datasets, and evaluate the ability of ACRO for policy learning from provably robust exogenous- representations, while baseline methods can be prone to the exogenous information present in datasets. From easiest distractor (uncorrelated static images placed in corner or on the side), to corrrelated background exogenous images, and then to fixed or changing video distractors playing in the background, to finally the hardest exogenous information of other random agent information in the agent observation space; we show that ACRO can consistently learn good policies for downstream control tasks, while the ability of baselines to ignore exogenous information dramatically degrades, as we move to hard exogenous information settings. Appendix F.5 provides further ablation studies on the different LARD-EXc offline settings, and performance difference for different domains and datasets.
data without and with visual distractors, as shown in Table 2. Experimental results in Table 2 show that even without any exogenous noise, ACRO learns controller latent state representations more accurately than the different state-of-the- thr- baselines, such that by efficiently decoupling the endogenous state from the exogenous states, policy learning during the offline RL algorithm can lead to significantly better evaluation performance compared to several other baselines.
Medium -Eiiinoo-s Information (MEDIUM-EXO). We then consider three different types of medium exogenous information that might appear in visual offline data. To that end, consider exogenous uncorrelated images from STL-10 image dataset (Coates et al., 201 that appear on the corner or the side of the agent observation, and the goal of ACRO is to filter out the exogenous information while recovering only the controller part of the agent state. We consider three different types of MEDIUM- REDIUM-EEXO information:
The exogenous image from STL10 dataset appears in the corner of the agent observation. This does not change the observation size of the agent; and we simply add the exogenous image in one corner, which is fixed during an entire episode during the offline data collec- tion. Figure 12 summarizes the result with different exogenous images placed in the corner. ACRO consistently outperforms several other baselines for range of different datasets, since can suitably filter out the exogenous part the agent state.
A slightly more difficult setting where now the STL10 exogenous image appears on the side of the agent observation space. This augments the agent observation space from 84 84 x 3 to 84 84 x 2 x since we consider downsampled STL10 images. Figure 13 summarizes this result comparing ACRO with the baseline representation objectives.
Finally we consider the distractor setting that has appeared in prior works in online RL (Zhang al., 2021a) with fixed video distractors playing in the background of agent obser- vation space. For this setting, we specifically re-collect the dataset following the procedure in (Lu et al., 2022a) where the SAC data collecting agent also sees fixed video distractor playing in the background. Figure 14 summarizes the result and shows performance plots where ACRO can significantly outperform all the baselines across all different types of exogenous datasets.
Hard Exogenous Information (HARD-EXO). We finally consider three sets different hard ex- ogenous information settings, and find that these HARD-EXO can remarkably make it difficult for existing state of the art representation objectives to learnt underlying agent controller states. This set- ting provides evidence that under suitably constructed exogenous information, which appear highly time correlated during the offline data collection, the baseline methods can fail to capture underly- ing controller latent representation of states. In contrast, the objective consider in ACRO, along
21

---

Under review as a conference paper at ICLR 2023
with the theoretical guarantees for learning suitable encoder recover the endogenous states, shows that policy optimization based on the endogenous controller latent states can lead to efficient learning in these control tasks. We consider three different rypes of HARD- Exo information:
We first consider time correlated static images appearing in the background of the agent observation. For this setting, during data collection, the agent sees a fixed image in the background for entire episode, and it changes per episode of data collection. This time correlated exogenous information ensures that the baselines can remarkably get distracted, while ACRO can still be robust to the static image background. Figure 15 summarizes the results and shows that several existing representation baselines can fail due to time correlated static image distractors.
We then consider an even more difficult HARD-ES setting where now the video distrac- tors playing in background also changes per episode during data collection. This is novel setting with video distractors in RI., since we explicitly consider diverse set of background videos which also changes per episode of data collection. Similar the above, Figure 16 summarizing the results with diverse video distractors in background per episode, shows that this setting can also break the baseline representation learners to recover the controller latent states, while performance of ACRO remains robust to it, since the ACRO objective learns encoder to recover the endogenous controller latent states accurately.
Finally, we consider the most challenging HARD-EXO where now in addition to the envi- ronment observation, the agent additionally other random action agent observations. Here, the goal of the agent is to learn representations to identify the controllable environ- ment, while other random-action observations are uncontrollable exogenous to the agent. This is quite a difficult task since the agent we are tring to control also sees observations from the same domain, of other agents playing with random actions. The controller agent observation now consists of other agents placed in 3 x 3 grid. Figure 18 summarizes the experiment results showing that ACRO significantly outperforms all baseline representa- tion learners.
F.2 EASY-Exo: PIXFL-BASED OFFLINE RL FROM V-D4RL. BENCHMARKS
Visual Offline Control (V-D4RL) without Distractors. We first verify the effectiveness of learning representations with ACRO without any additional exogenous distractors, and compare with several baselines for learning representations. Figure 10 provides detailed performance curves for Table 2.
V-D4RL with Varying Severity of Distractor Data Shift. We then consider the distractor setting in v-d4rl benehmark (Lu al., 2022a) with varying levels of distractor difficulty. Here, the ex- ogenous noise is based on background static image distractors inducing distribution shift in the dataset, depending on the level of difficulty from easy, to medium to hard distractors. As shown in Figure 11, we consider two different domains and find that with varying difficulty levels, ACRO can consistently outperform several state of the art baselines, learning directly from pixel data.
F.3 MEDIUM-EXO: STL1O EXOGENOUS IMAGES OR FIXED VIDEO DISTRACTORS
We extend our experimental results with different types of exogenous image distractors in the ob- servation space of the agent. Detailed description of the dataset collection process is provided in Appendix G.
Exogenous Image Distractors Placed on the Corner or Side of Agent Observation. We consider two different settings where the agent environment observation is augmented with STL-10 image Coates al. (2011) distractors, either placed in the corner or on the side of the environment ob- servation. Here, we consider adding uncorrelated exogenous images where for each pre-training of representations update, the environment observation has image distractors. When placed on the side, it extends the observation space of the agent. Figure 12 and Figure 13 shows results with exogenous images placed in the corner or on the side of the agent observations respectively.
Fixed Video Distractor. We first consider setting where the exogenous distractor in the back- ground is fixed with a single type of video distraction. Figure 14 shows results with fixed video distractor showing that across several datasets, ACRO consistently outperform baselines.
22

---

Under review as a conference paper at ICLR 2023
ACRO
DRIML
Homer
CURL
One-Step-Invee levelliveese
DrQv2
Waker walk Aeper
cheeton, nun exped
Numancd, www.EnEEEnR...
cheesann
Cumulative Returss
iilir
Waler 2nk, nedium rapley
turannel walk medumm pectly
cheeten nun medkum, replay
Iterations
Figure 10: EAsy-Exo-No Distractors Full Results. Experiments over 6 random seeds. For these experiments, we use the visual offline benchmark from (Lu et al., 2022a) and compare ACRO with several state of the representation learning objectives. We find that across tasks, ACRO either outperforms or equally performs well as the best performing baseline method.
Performance Different Distraciors (Cheelah Run)
53 Performance Different Distrsclors (Waker Walk)
175
ADRO
ACRO
CRR
GRI
1-0ky- verio
Cumulative
100
Or0kz
Cumutatte
SrON2
Casy
Vedur
Hare
Cary
Vedur
Hare
Figure 11: EASY- Distractors Full Results. Comparison of ACRO with baselines using the distractor suite data shift severity from v-d4rl (Lu al., 2022a) benchmark. We compare results with the two domains and datasets that were released in the v-d4rl benchmark distractor suite.
F.4 HARD-EXO: TIME CORRELATED AND MOST DIVERSE EXOGENOUS DISTRACTORS
Static Background Image that Changes Per Episode. We further experiment with time correlated exogenous distractors in the background, where during every episode of data collection, we provide a background image to the data collecting policy. We find that in presence of exogenous background distractors, ACRO can still be robust to the exogenous noise, while the existing baselines learning representations are more likely to fail, as shown in Figure 15.
We then consider more difficult setting where the type video distractor playing background changes during every episode. Further details the data collection with fixed and changing video distractor in background is provided in the appendix. Figure 16 shows results with changing video distractor, where ACRO can consistently outperform baselines representation learning methods.
Exogenous Video Distractors that Changes Per Episode.
Multi-Envine Colti-Ennirooment Agent Observations as Exogenous Information. We then consider a setting where the observation space of the agent is augmented with other random agents moving in the
23

---

Under review as a conference paper at ICLR 2023
ACRO
DRIML
Homer
CURL
One-Step-
DrQv2
waker Walk Aered
cheetar, Jun expert
walter walk mes.mm seper
Sumance, www.nneennnccc
cheesaan aapert
Cumulative Returns
medim
Walle 20k, medium, rapbey
turancet mwd.m Peciay
cheeten nun medkum, replay
Iterations
Figure 12: MEDIUM-EXO-Cornet Full Results. Performance comparison of ACRO with several other baselines.
environment. For this experiment, take the observations from the environment the agent wants to control, while the other observations of other agents come from a mudom-pplicy dataset, either from the same domain, from different domains. We do this since the random observations from other agents can be treated as exogenous information to the representation encoder; our experiment results show that ACRO can ignore the exogenous information from other agents, leading to signif- icant performance improvements from the offline RL algorithm based on the exogenous observation dataset. Figure 18 summarizes the results.
F.5 ABLATION STUDIES HARD-EXO OFFLINE RL
Figure 17 provides a summary of comparison between different datasets in the HARD-EXO noise setting.
G DATA COLLECTION FOR OFFLINE RL WITH EXOGENOUS INFORMATION
EASY-Exo Datasets. For the EASY-ExO setting with exogenous information, consider uncor- related visual distractors in the background of the observation space. For this setting, we extensively the datasets released from the v-d4rl benchmark (Lu al., 2022a) for offline RL. We note that the data shift severity in v-d4rl benchmark are only limited to two different domains and two data distributions (medium-expert and random). We experiment with both these datasets, and addition- ally consider the setting with no uncorrelated static images in the background.
MEdiUM-EXo Datasets. For the MEDIUM-EXO datasets, we collect new offline datasets using SAC policy, following the same data collection procedure as in the literature from d4rl benchmark (Fu et al., 2020). The main difference is that when collecting new datasets with the SAC policy, consider variations of different exogenous noise types in the dataset. As discussed earlier, for the MEDIUM-EXT Exo setting, we collect three different exo-types of datasets (a) Exogenous stl10 images placed in the corner of the agent observation space. For this setting, during episode of data collection, at each time step. we sample a new STL-10 image and place in the corner of the agent observations. (b) In the side exogenous information setting, instead of the corner, place
24

---

Under review as a conference paper at ICLR 2023
ACRO
DRIML
Homer
CURL
One-Step-
DrQv2
Waker Walk Agre
Parmaoodd MOOHT
cheetar, nun exper
walter www.mmesummmmmmmmmmmmmmmmmmm.e.s sepert
Sumance, www.mn.eerrrr...
cheeaan expert
Cumulative Returns
Waler 200, medium papley
turannet walk mwdummmtmmtn. replay
cheeten, Jun, meddkuT, replay
Iterations
Figure 13: MEDIUM-EXO-Side Full Results ACRO can be quite robust the exogenous images when the exogenous images appear to be similar to the agent in the environment. For example, con- sider the Cheetah-Ree Run environment with a dog run image on the side, which can be quite distracting to the baseline methods.
ACRO
DRIML
Homer
CURL
One-Steg-inversa
DrQv2
Tumonodd
cheetah snger
Cumulative Returns
sopert
Iterations
Figure 14: MEDIUM-EXO-Fixed Video Full Results. ACRO can outperform baselines with fixed exogenous distractors in background, which is time-correlated in nature. The fixed video distractor setting have often been studied in online RL literature. In this work, we study fixed video distractors in offline RL, where the distractors were present during the offline data collection.
the exogenous image the side of the environment observation. This augments the entire agent observaton space. A major difference with (a), however that, in this setting we consider time correlated exogenous images where each step of SAC policy during episode sees the same ogenous image, which only changes per episode of data collection. consider this to harder setting compared to changing the image each time-step, since this induced time correlation
25

---

Under review as a conference paper at ICLR 2023
ACRO
DRIML
Homer
CURL
Ono-Step-Invenn heep-invers
DrQv2
Tumano wilk
cheeesh
Cumulative Returns
cheetah won.mediun.oo.
wakk
Iterations
Figure 15: HARD-ExO-Static Image Full Results. We find that ACRO can significantly outper- form baselines in presence of correlated exogenous static images playing in the background
ACRO
DRIML
CURL
One-Step-inverss
WAllo wilk Diport
Turrno d, walkooooon
cheotah sun, ongoo
Curmulative Returns
walter MAL www.nnnduuuoooooo
numana d. weR un..mndddmooooo eeper
cheotah BoreanLNnummeeee www.mmedummoo..
walk
humarasc madim
mediu
lterations
Figure 16: HARD-Exo-Videw Full Results Across all datasets, ACRO can consistently outper- form baseline methods for all types of datasets. When learning representations for offline policy optimization, the ability of ACRO to ignore exogenous information makes outperform baseine representation objectives in almost all cases. The changing and time correlated video distractors are often hard for baseline methods to ignore, leading to significant performance drops depending on the offline data distribution
can make it harder for the representation objectives be completely robust the side exogenous information. (c) Finally, we conisder a fixed video distractor setting, which has been extensively studied in the online control benchmark (Tassa al., 2018). Prior works have experimented in the online setting with fixed video distractors. We use the same procedure and fixed video distractor as here, except we use a SAC policy for data collection to be used the offline setting. All these cat- egories, cumulatively are denoted MEDIUM Exo in this work. We release datasets and detailed experiment details for all these settings for MEDIUM Exo based offline RL datasets.
HARD- HARD-EXO Datasets. We consider this be the hardest of the exogenous distractor setting. For this setting, we introduce several new offline benchmarks with different types of exogenous infor- mation. (a) Time Correlated Exogenous Image in the Background This is hard distractor type
26

---

Under review as a conference paper at ICLR 2023
Hard Exo (chaeebani pun, scpwr))
Hard Eso (chewtahh medium, expart)
Peffomance
Changer Bep
Drstractor
Exogendus Distractor
Hard -Bhumansid
Hard (humanoid medium
AD12 Hnaee
12
Hrares
Partummni
Porforancc
Crangen de
Sestractor
Sc-gen
Do-gee
Wooker exper1)
modium expert)
AORO
AGRO
Peroomanc
Videe
Dirige Yalu
Real Valle
Figure 17: Summary and Performance Difference between ACRO and baselines in the HARD- Exo offline RL setting
ACRO
DRIML
Homer
CURL
One-Step-inverse
DrQv2
wodteer sunft papprtt
Pamanod walk eeperi
cheersh sun, peperd
Cumulaatvve Returss
watker medurn_oppt
humano if walk wwlkk/eddunnn eoper
cheetah sun, medium_opoot
wake: work mockum
humorolc melkn
Iterations
Figure 18: BORD-ExO-Backennn Agent Full Results. Random agent observations placed on the grid, of the entire agent observation space.
where most existing representation learning baselines can fail. For this, during the data collection with SAC agent, we place static background image from the STL10 dataset in the background. This image remains fixed for all timesteps within an episode, and we only sample a new exogenous background image at every episode. This makes the exogenous noise to be highly time correlated, where baseline representations are likely to capture the background image addition to learning an embedding of the environment. (b) Changing Video Distractors In this setting, like (a), at every new episode we change the rype of video distractor that we use. During an episode for different timesteps, the agent already correlated noise from the video frames playing in the background. However, since the type video sequence that plays in background changes, this makes the offline datasets difficult learn from. This setting is inspired by real world application, where
27

---

Under review as a conference paper at ICLR 2023
for example, the agent perceiving the world, can see background data from different data distribu- tions (e.g background moving cars compared to people walking in the background). (c) Finally, we consider another HLARD-EXO dataset, where now during every timestep of data collection, the agent sees other agents playing randomly. Here, we place several other agents, taking random actions, in grid, and the goal of the agent is to recover only the controllable agent, while being able to ignore the other uncontrollable agents taking random actions, within its observation space. For this setting, the other uncontrollable agents can either be from the same domain (e.g using a Humanoid walker agent for both the endogenous, controllable part and the exogenous, uncontrollable part), or a dif- ferent setting where the exogenous agents can be from other domains (e.g using Humanoid agent for the controllable part, while the uncontrollable agents are from a Cheetah agent). We demon- strate both these types of observations either same-exogenous or different-eNe -xxoeenus Figure 21 and Figure 20 respectively. We also release these datasets as an additional contribution to this work, and hope that future works in offline RL will use these datasets as benchmarks, for learning robust representations.
H ADDITIONAL EXPERIMENT RESULTS: ATARI MEDIUM-EXO
We also consider the setting of Atari. We build on the setup introduced in decision transformer (Chen et al., 2021b). While decision transformer focuses on framing the reinforcement learning problem as a sequence model- ing problem, we mainly focus on learning representations which can learn to ignore exogenous noise. We use the same 4 games used in Chen et al. (2021b)- Pong, Qbert, Breakout, and Seaquest. We consider the MEDIUM-Ex Exo setting where a different image is used in each episode and concatenated to the side of each observation. We add randomly sampled images from the CIFARIO dataset (Krizhevsky & Hinton, 2009) as exogenous noise. Figure 19 shows example observation from breakout with exogenous noise.
Figure 19: Example of an observation from Breakout with a CIFARIO image on the side.
Decision Transformer uses the DON-Replay dataset (Agar- wal et al., 2020) for training. The model is trained us- ing a sequence modeling objective to predict the next action given the past states, actions, and returns-to-ge R. = Eee Te. where e denotes the timesteps. This results in the following trajectory representation: т (R1, $1,01, 01, R2,,82 82, 02, Rg, $3, ag....). where a, denotes the actions and 8, denotes the states. At test time, the start state 81 and desired return Ri is fed into the model and it autore- gressively generates the rest of the trajectory.
Table 4: Atari (MEDIUM-EX0)
Here we compare ACRO to One- Step-Inverse model and DRIML on various games from the atari bench- mark with exo-noise. We can see that ACRO consistently outper- forms both the baselines. Results averaged across 5 seeds.
Game
1-Step Inv.
DRIML
ACRO
Decision Transformer uses convolutional encoder to encode the observations. We first pretrain this encoder using the pro- posed ACRO objective. We use the 1-step inverse objective and DRIML as our baselines. After pretraining, we train the decision transformer using the sequence modeling objective keeping the encoder fixed. We present results in Table 4. We can see that ACRO outperforms both the baselines in all games further showing the effectiveness of the proposed approach.
Breakout
3.8
1,0gn0
20.6g33
Pong
8.6g1a
-000g00
11.8
Qbert
277.8
mm.
Seaquest
274.0
94.4
972.4_
Hyperparameter Details. We keep most of the hyperparameter details same as used in Chen et al. (2021b). They use episodes of fixed length during training also referred to as the context length. We use a context length of 30 for Seaquest and Breakout and 50 for Pong and Qbert. Similar to Chen et al. (2021b), we consider one observation to be a stack of atari frames. To implement the ACRO objective, we sample 8 different values for & and calculate the objective for each value of k obtaining the final loss by taking the sum across all the sampled values of k. We do not feed the embedding for & in the MLP that predicts the action while computing the ACRO objective.
28

---

Under review as a conference paper at ICLR 2023
I EXPERIMENT SETUP AND DETAILS
We describe our experiment setup in details. We use the visual d4rl (v-d4rl) benchmark Lu et al. (2022a) and additionally add exogenous noise to visual datasets, as described earlier. For all our experiments, comparing ACRO with other baseline representation objectives, we pre-train the rep- resentations for 100K pre-training steps. Given pixel based visual offline data, we use a simple CNN+MLP architecture for encoding obeservations and predicting the ACRO actions. We also use cropping-basee based data augmentation as in DrQv2 while pre- pre-training the representations for all meth- ods. Specifically, the ACRO encoder uses 4-layers of convolutions, each with a kernel size of 3 and 32 channels. The original observation of 84 x 84 x 9, corresponding to a channel-observation and frame stacking of 3. The final encoder layer is an MLP which maps the convolutional output to a representation dimension of 256, giving the output Φ(z). This followed by a 2-layer MLP (hidden dim-256) that is used to predict the action given 5 512 input corresponding to concate- nated St and &t+k representations. For ACRO, we sample k from to 15 uniformly. We use ReLU non-linearity and ADAM for optimization all throughout.
For our experiments, we build off from the open source code base accompanying the v-d4rl bench- mark (Lu et al., 2022a). We implement the pre-trained representation objectives in model-free setting, where for the baseline offline RL algorithm we use TD3 + BC (Fujimoto & Gu, 2021b) since it achieves state of the art performance raw state based offline RL benchmarks (Fu al., 2020) and is a reasonably performing algorithm for visual offline RL as well Lu et al. (2022a). The policy improvement objective for the baseline TD3 + BC algorithm thus:
arg max Es a.)-aa 10(s,,7(8) - (T().) m.) - )
(17)
x
where the critic (ss, π (s)) is evaluated by TD loss, and we use re- pa-aamtteizzed gradients through the critic for policy improvement step. For pixel based visual observations, recent work Lu et al. (2022a) proposed the DrQ BC algorithm, which is essentially the TD3 BC algorithm, except additionally applies the data augmentations on pixel based inputs. In DrQv2, data augmentation is applied only to the images sampled from the replay buffer, and not during the sample collection procedure. Given the pixel based control tasks, where the images are 84 x 84, DrQ pads each side by 4 pixels (repeating boundary pixels) and then selects random 84 x 84 crop. yielding the original image, shifted by 4 pixels. This procedure is repeated every time an image is sampled from the replay buffer; and makes DrQ data augmentation quite effective based the random shifts alone, without the need for any additional auxiliary losses. More concretely, denoting policy as Tæ, the policy network trained with the following loss:
(D) EEND (e))
(18)
where z, = f. (aug(2+)) the encoded augmented visual observation, ā. (z() + (action with clipped noise to smooth targets, c ~ clip(N 0²), -c, c)). Overall loss for policy improvement
LD( -B,,A, XQ 1 (Zi, a.) (To a..²²
(19)
where DrQ passes the gradients of the critic to learn the encoder, and there are no separate or explicit representation losses other than the critic estimation. for training the encoder in DrQ. For all our experiments, in the EASY-EXO setting, we use datasets provided from v-d4rl benchmark (Lu et al., 2022a). For the MEDIUM-EXX and HARD-EXO settings, we follow the procedure from (Fu et al., 2020; Lu et al., 2022a) to collect new datasets using a soft cctor-critic (SAC) policy, under different variations of exogenous information that we considered in this work.
J BACKGROUND AGENT CONSECUTIVE FRAMES VISUALIZATION
29

---

Under review as a conference paper at ICLR 2023
Figure 20: Consecutive timesteps from three different episodes (each row); where the controllable agent and the background agents are from different domains
Figure 21: Consecutive timesteps from three different episodes (each row); where the controllable agent and the background agents are from same domains
30

---

Under review as a conference paper at ICLR 2023
K VISUALIZING RECONSTRUCTIONS FROM THE DECODER
We show additional reconstructions for ACRO, DrQ, behavior cloning, and random features on different types of exogenous noise in Figure 22, Figure 23, and Figure 24.
Figure 22: Reconstructions for Fixed Background Distractor from decoder learnt of over two kinds of representations: Top-Bottom: Random features, DrQ, Behavior cloning, and ACRO. Left Column: Original observation, Right column: Reconstruction.
31

---

Under review as a conference paper at ICLR 2023
Figure 23: Reconstructions with CIFAR images in background from a decoder learnt of two kinds of representations: Top-Bottom: ACRO, DrQ. Left Column: Original observation, Right column: Reconstruction.
Figure 24: Reconstructions with a distractor video playing in the background from a decoder learnt of two kinds of representations: Top-Bottom: BC, ACRO. Left Column: Original observa- tion, Right column: Reconstruction.
32

---

Under review as a conference paper at ICLR 2023
Author Rebuttal
We thank all the reviewers for helpful and detailed feedback, which we have already taken advantage of to improve the paper. We thank the reviewers for praising the strong experimental results, new theory that this paper provides, along with an important contribution to the community by providing new offline RL datasets containing varying levels of exogenous information in the datasets. We plan to release these datasets, along with our implementation, learning robust representations in offline RI., which we hope will be significant to the RI community.
Note : After the rebuttal period, we will merge the responses and figures from the sections below into the main draft. Currently, we include these as separate sections to address the reviewer comments explicitly.
L COMMON RESPONSE TO ALL REVIEWERS
We first provide a common response all reviewers, since some of this feedback is useful for overall improvement and understanding of the paper.
L.1 COMPARISON WITH AC-STATE (LAMB ET AL. (2022)) AND ACRO, NOVELTY OF USING INVERSE MODELS
We would like to thank the reviewers for highlighting the need to produce comparisons between our proposed method and the AC-State algorithm (Lamb et al., 2022). The reviewers rightly pointed out, as already addressed in the paper, the similarity with the AC-State objective from Lamb et al. (2022) and ACRO. We would like to highlight here the main differences between AC-State and ACRO and what makes ACRO critical for learning robust representations that can be used for any RI. algorithm. The AC-State objective uses a multi-step inverse model with an additional con- ditioning on the k-th time-step (E, (o K) log (La(a, ((2+), ó(xc+k), k)), unlike ACRO objective as in equation 1. We would also point out that the theoretical support for multi-step in- verse models for representation learning largely comes from Efroni al. (2021), ) an ICLR2022 paper which predates the AC-State paper.
The work of Lamb et al. (2022) primarily has different purpose and scope, where the goal is to recover latents with perfect accuracy. for then using it for exact planning in tabular MDP, an approach which only scales to very small discrete systems. Most importantly, Lamb al. (2022) learns 100 discrete latent states using controlled exploration, by the use of discretization bottleneck Van Den Oord et al. (2017) and primarily addresses recovering discrete ground truth latent states. Lamb et al. (2022) considers only environments with well under 100 discrete endogenous states. In comparison, ACRO considers offline RL with general continuous hidden states, which is much more general. Lamb et al. (2022) does not address how the learnt representation can be used in any RL algorithm, and the controlled exploration is mostly required to recover the ground truth states. The discrete bottleneck along with the requirement for exploration (Lamb al., 2022) applicable for any deep RL algorithm to be used off-the-shelf with Iearnt representations. Lamb et al. (2022) considered moderate state space sizes and used planning on top of the latents. In contrast, we consider larger scale problems and train an RL algorithm on top of the representation. Hence, unlike prior works, we study larger scale problems in offline deep RL.
generally
In contrast, ACRO addresses how to effectively use the multi-step inverse objective in situations where there are infinite continuous latent states and uncontrolled exploration. ACRO shows the capability of learning robust representations in presence of exogenous noise, that can be used for learning continuous representations, and does not require any use of a discretization bottleneck. While the objective remains the same, the data and representations here differ in critical ways, greatly broadening potential applications. Together these changes provide results in regimes the AC-state paper does not address. In ACRO, the focus shifts from doing exact planning (DP or Dijkstra) in a small tabular-MDP to extracting a representation that enables learning a policy via Offline- RL with learned value function.
In figure 25 and 26, we empirically compare ACRO and a version of ACRO with a discretized bottleneck representation, an extension of Lamb et al. (2022) that can be used in an offline RI.
33

---

Under review as a conference paper at ICLR 2023
(uoper
Auralorns
Tacttons
Partilin
(a) Halfcheetab-exppee expert
(b) Humanoid-Wall-experr Walk-expert
(c) Walker- expe-t
dhveden nun www.://///mnno. cneree)
Iwesaned vokk jostire sxjeet (isnea))
valiow wolk (mediar paper decsoo))
Aeralina
64
Parsdow
(d) slalicheetah-med-eee exxeet
(e)
Humanoid Walk mex-exp
(1) Walker- Walk-med-ent
V2
cheeth (madumm dedena))
humanolld vak (naduar datssal)
voller volk: /wwddu darmast)
1.0
Cumsatiie Rotara
1000
1000
(g) Halfchettah-mediine
(h) Humanood- Walk-medium
ii Walker- Walk-medium
Figure 25: Comprisons between ACRO and AC-State Lamb et al. (2022) in time-correlated HARD-
Exo offline datasets of different data distributions (expert, medium-expert and medium)
algorithm. We compare both these methods under two of the hardest HARD-EXO offline datasets of varying data distributions. The primary difference is the use of the small discrete representation required by AC-State. In most cases, ACRO works significantly better in the offline RL task with HARD-EXC noise.
We see our work as a bridge between earlier theoretical work on multi-step inverse models and the Offline-RL literature. In terms of the novelty of theory, it is true that the proof in Appendix A is the same these papers, which we already stated very clearly in our paper: "This proof is essentially the same as lemmas found Lamb et al. (2022), Efroni et al. (2021), but is presented here for clarity!". The proofs in Appendix B and Appendix C are novel and specifically related to conditions for Offline-RL RL to succeed. To better explain the differences, we added table with these prior works, and we added clearer explanations of novelty to the main text.
Theoretical Claims
We established new results for the setting of RL with exogenous information. Namely, that including some exogenous information as part of the representation may violate Bellman completeness, i.e., the ability to optimally represent any possible Q function that should be learned during the training process. Additionally, we showed that the agent controller representation is sufficient represent the optimal Q function and induces Bellman complete class. We first showcase the challenge of the offline RI. problem with exogenous noise, and then show the advantage of the agent controller representation.
Novelty of Using Inverse Models We highlight here that even though inverse dynamics mod- els have been used previously in the literature, they are primarily used for either exploration solely (Pathak et al., 2017) or for representation learning in an online task that requires exploration (Misra et al., 2020; Efroni al., 2021; Lamb et al., 2022). In contrast, in this work, we first use inverse dynamics models in exogenous block MDPs to show the capability of learning robust representa- tions that then be integrated to an offline RL algorithm. As such, ACRO does not require any
34

---

Under review as a conference paper at ICLR 2023
Aunalons
Partitin
(a) Halfcheetab-exppee expert
(b) Humanoid-Walk-exppe Walk-expert
(c) Walker- expe-
dhwedonn run pm../mm in sapert cnerre))
Iwesaned wokk jostire sope munwl))
voliow walk (mediars woper decsoe)
Arsalons
Parsdor
(d) Gaalfcheetah-meedeee expeexperr
(e)
Humanoid Walk-med mex-ex
(i) Walker- Walk- Walk-med-expe -ee-ee
V2
cheethh dedasa))
humanold vnk (nadiar datssal)
voller voli:
Cumsaiee Potume
10:0
6000
10000
05
24
(g) Halfcheetah-mediine
(h) Humaaood- Walk-medium
ii Walker- Walk-medium
Figure 26: Comprisons between ACRO and AC-State Lamb al. (2022) changing video HARD-
Exo offline datasets of different data distributions (expert, medium-expert and medium)
additional exploration and can work with any data distribution the offline datasets. We already show results comparing ACRO to one-step inverse models (Pathak al., 2017).
1.2 SIGNIFICANCE MULTI-STEP ACTION PREDICTION vs. SINGLE ACTION PREDICTION
We thank the reviewers for asking insightful questions about how the ACRO algorithm compares if we use multi-action prediction up to the k-th timestep, in comparison to only predicting a single next step action. Since ACRO already conditions on Φ(x++*) for k-th step in the future, it natural to ask how the performance varies if we predict multiple future actions compared to single action. Concretely, the ACRO objective optimizes the following
Φ, E arg max de
E (O,N) k~U(O,
E
log (Pa(, d(xx), ((x++))))
(20)
whereas, we could instead predict the step action sequence:
Φ+ E arg (0.(). (0,K)
E
E
log (PP(a,,.... .44k ((xx), o(1++)))
(21)
dE
where equation 21 is implemented using an LSTM that outputs the i actions. One reason prefer predicting just the first action is that it is simpler model and is computationally cheaper (as it requires just classifier over actions and not an autoregressive model over sequences like the L.ST)). Intuitively, we also felt that predicting the first action to reach goal would be sufficient, because ultimately every action along the trajectory is still predicted, but conditioned on the observation prior to the action being taken. In environment with stochastic dynamics, principle, because the best action to take at a given step is dependent on what has happened in the environment. In deterministic environment, both approaches valid in principle.
this being better in
Nonetheless, we agree that it is important to also answer this question experimentally. Figures 27 and 28 show comparison between ACRO and variation of ACRO predicting multiple actions in
35

---

Under review as a conference paper at ICLR 2023
the future. We use the same training setup, where all the representations are pre-trained in presence of HARD-EXO noise in observations.
chestil run Lnt.eteeeddlaaee
hurnnoati valk (uspet dilend)
vaiber with /espet deleae
Aaratons
Paraboon
(a) Halfcheetab-expert
(b) Humanoid- exper
(c) Walker- Waakker-all-exxpert
Figure 27: Comparions between ACRO, ACRO with multiple action predicton (ie, predicting an action sequence) and also with AC-State (Lamb al., 2022) time-correlated HARD-EXe offline datasets
cheeth (ettaad
harowatd (nppet daaan)
undle
Ouruaatee Regura
A030
(a) Halfcheetah expert
(b) Humanoid- Walk-expert
(c) Walker- Walk-expert
Figure 28: Comparions between ACRO, ACRO with multiple action predicton (ie, predicting an action sequence) and also with AC-State (Lamb et al., 2022) in changing video HARD-EXO offline datasets
L.3 DOES ACRO ALSO REMOVE TASK RELEVANT INFORMATION?
Various reviewers expressed concerns about the agent-controller representation ignoring informa- tion that is vital to the task (pedestrians for reviewer GMBs and objects / blocks in robot hand manipulation task for reviewers GMBs and yx3D). However, the notion of what in the agent- controller representation is actually more extensive than might be imagined at first glance. Anything that influences the actions taken by the policy needs to be captured by the representation in order to predict the first action it took from pair of representations. Going back to the examples ahove, in the case pedestrians, the policy used to collect the data (hopefully) avoided the pedestrians to reach its goal. In order to properly predict this avoidance behavior (compared to presumably moving in straight line in the absence of pedestrians), the representation needs to encode the presence of these pedestrians. Similarly, for the example of robot hand grabbing block, let us consider a pair of states (st, sy) ). where š, corresponds to the state before the object was grasped and sy after. Then, the agent-contre contrllert representation needs to include information that pertains to the block's position and orientation, otherwise, it would be impossible to predict the actions governing the robot's 's mo- tion (presumably in the direction of the object), or how its joints are adjusting to grab the object. Note that this would not be required for simple one-step inverse model (which justifies the use of a multi-step onc) as the robot hand's joint positions in successive states are sufficient to infer what action was taken.
Additionally, in ACRO we also consider expert policies as a source of offline training data. In the case of expert policy, the knowledge about what parts of the world matter will be captured even by a behavioral cloning policy. ACRO inherits this strength of behavioral cloning, while dramatically outperforming it when the data comes from random policy.
36

---

Under review as a conference paper at ICLR 2023
L.4 EXPLANATION OF NORMALIZED RESULT FIGURES
We apologize that the results presented in our paper, given we had too many results to present, might have been confusing for reviewers to interpret. In offine RL, since our experiments are across sev- eral domains (cheetah, walker, humanoid), and for each domain, we have different dataset distribu- tions (expert, eexeer, medium) as generally present in offline benchmarks (Lu et al., 2022a; Fu et al., 2020), we averaged or normalized results to present the significance of ACRO across the experiments. Given the experiments, whether we are considering MEDIUM- Exo or HARD-EXO, either average the results across domains or datasets. All individual experiments are run for 5 different seeds across the board. We emphasize that the normalization or averaging done so we could present all our experimental results, explaining significance of ACRO in precise way, across the board. All individual results showing improvements that ACRO achieves are included in the appendix.
Figure 4 for example, presents results where averaging is being made across domains. We further include individual results for each domain in the appendix. We show that ACRO achieves improve- ments across all domains in general, in addition to improvements across all dataset types, as shown in figure 4.
Figure 6 shows results where the normalization is being made across the dataset types, instead of domains, in figure 4. Figure 6 shows results for the domains (cheetah, humanoid, walker) for different HARD-EXO types, and averaged across expert, medium-expert and medium datasets. We show that ACRO can achieve significantly better performance compared to 5 other baselines considered in this work.
L.5 IMPACT IN OFFLINE RL AND SIGNIFICANCE OF THE WORK
We would like to emphasize and repeat here, to the reviewers, the significance of our work and the impact it can have in the community. Firstly, we propose an approach for learning robust represen- tations, where we considered many diverse set of exogenous noise that can be present in the obser- vations. We emphasize that we are the first to study, both empirically and theoretically, presenting a practically feasible algorithm, capable of working under different exogenous noises. We propose an algorithm based on the multi-step inverse dynamics objective, a variation of which has already been studied theoretically (Efroni et al., 2021; Lamb et al., 2022), and show that when pre-trained under exogenous noises in observations, it can lead to significant performance during fine-tuning. Additionally,
releasing all exogenous noise based offline datasets such that it can be used later in the community. We believe, for practical scalability and significance of RL in the real world, offline RL can play significant role, and demonstrating ability to learn from such datasets, that are likely to appear in practice, is an important step in the future.
M RESPONSE TO INDIVIDUAL REVIEWERS
M.1 RESPONSE TO REVIEWER GMBS
M.1.1 EXOGENOUS INFORMATION AND DISTRACTOR SETTINGS FROM PRIOR WORKS
Question: Similarity of D4RL with some exogenous images or videos incorporated as noise in observations
Response: : We would like to emphasize, as briefly mentioned in L.5 that this work studies pixel- based offline RL with exogenous noises in observations. In comparison. Fu et al. (2020) provides benchmark datasets for raw state- offline RL, whereas recently Lu et al. (2022a) provided pixel- based offline RL. benchmarks. We extended from Lu et (2022a) to additionally add exogenous noises in observations, with different levels of difficulty : EASY-EXO is what we call the existing Lu et al. (2022a) benchmark; we then extend and re-collect datasets using the same pipeline as Lu et al. (2022a); Fu et al. (2020), but now additionally add MEDIUM-EXC or HARD-EXO exogenous noise during data collection with SAC policy. For example, during MEDIUM- -ExO, we place random STL10 images in corner side, or place fixed video in background. The fixed video background distractor has been studied in online RL before (Zhang et al., 2020), and we provide an extension of it to offline RL. In the HARD-EXc Exo setting, we consider novel datasets and distractors, where
37

---

Under review as a conference paper at ICLR 2023
now have either changing video distractors in background, have multiple agent observations that the agent sees, in addition to the controllable environment observation. Please see figure 8 for samples of what the agent exogenous observation looks like. Such datasets and distractors have not been studied in the literature hefore, and we provide a well capable representation learning approach to robustly learn representations under such difficult distractors.
M.1.2 CONCERN ON REPRESENTATION CONTAINING ONLY TASK RELEVANT INFORMATION.
Feedback: Weaknesses: The major concern is about the limitation of the representation only con- taining control-relevant information. How about there are other moving entities in the observations, which are not controllable by the agent but are still very important for the success of the policy? For example, robot navigates busy streets. Pedestrians should consider the status of the pedestrian to avoid collisions. Will the proposed representation learning approaches fully ignore pedestrians and make errors in the decision? Another example is robolics manipulation tasks. The robol hand should manipulate the objects, so the object position and orientations are important information for the control policy. But the proposed method may tend to ignore the object information. This is problematic and limits the scope of the proposed method
not controllable by the robot, but the robot
Response: Thank you for the comments and feedback, which are playing a major role in improving the impact of the paper. Please see section L.3 where we provide generic response to all reviewers.
M.1.3 IMPORTANCE OF BELLMAN COMPLETENESS
Feedback: : It will be great to explain the importance of the Bellman completeness more clearly and intuitively
Response and Comment on Bellman Completeness As we elaborated on in the paper, the im- portance of the Bellman completness assumption was recently demonstrated in the theoretical RL community. Intuitively, this assumption says the following: for any function [(x, a) in the function class it is possible represent its Bellman backup r(x: a) F[max,,' f('', | ar, That is, it is possible to optimally learn the Bellman backup at each point of the training process. We will highlight this fact better in the text. Bellamn completeness overall means the ability to represent the optimal solution that mininimizes the bellman error of any function. For example, if the expected bellman error is small then completeness holds, however, vice versa does not necessarily hold. That is, the bellman error might be large because of stochasticisty and not because completeness doesn't 't hold.
M.1.4 ADDITIONAL EXPERIMENT RESULTS ON D4RL
Feedback: : The quality is good with the technically solid method and impressive experiment results. However, the experiments are only conducted on locomotion tasks. It will be great to show results in D4RL kitchen lasks, especially because these lasks require object manipulation, and the proposed representation approach may fail in this scenario. I I'm curious whether the proposed method can be modified or extended the manipulation tasks.
Thank you for the feedback on the impressive experimental results that ACRO achieves. We agree that all our experiments are on the locomotion tasks, based on the available offline benchmark (Lu et al., 2022a) and our newly proposed benchmark datasets. We definitely agree that it would be interesting to show whether ACRO kind of objectives can also perform well object manipulation based tasks.
For the D4RI. kitchen task, we implemented ACRO on top of the existing CQL and TD3 + BC baselines for testing. However, unfortunately find that, as also reported in past works, that none of the baselines actually work well on the kitchen tasks. For ACRO to work well, we would still require well performing offline RL algorithm that solve the kitchen tasks based on the learnt representations, and we found that the existing baselines are not significantly well performing on this task.
However, we also ran ACRO on the D4RL benchmark (Fu et al., 2020). We would like to first mention that D4RL is only raw state based benchmark and not pixel based observations. As such,
38

---

Under review as a conference paper at ICLR 2023
even though past works have shown results for learning representations in D4RL, learning in raw state based environments does not play a significant role for learning representations. Morcover, ex- primarily for rich observation MDPs with pixels. Despite that,
ogenous noise in observations implemented ACRO on top of the TD3 Baseline Fujimoto & Gu (2021b) which has been shown to work well raw state based D4RI. benchmark. In figure 29 show results on two different domains with two different datasets, where on top of the existing baseline, we learn representa- tions with ACRO. Experimental results show that we can still achieve some marginal improvements over baselines by learning with ACRO. We would like to mention here that even though there are marginal improvements, this result may not be significant in the context of our work, since there is no exogenous noise present here, and ACRO is primarily proposed for pixel based RL tasks, to demonstrate ability of learning robust representations by ignoring the irrelevant information.
Figure 29: Experimental results on the D4RL benchmark comparing ACRO with the baseline TD3 + BC method. We mention here that previously we do not do experiments with D4RL since the environments are primarily raw state based, instead of pixel based observations. In this work, we primarily propose ACRO for learning robust representations that can ignore exogenous information from pixel observations. We include this result since it is asked by the reviewer on why we do not do experiments with D4RL. We include this preliminary result, and demonstrate that ACRO can work well in raw state D4RL too, even though that is not the primary contribution of our work.
M.1.5 NOVELTY OF WORK AND RELEASE OF OFFLINE DATASETS
Feedback: :
The novelty is okay but not very great, because the multi-step inverse model has been studied for a well in the RL area, and this work is just to adapt it from standard RL to offline RL.
Thank you for your feedback on this, and on raising the question on the novelty of the work. We provided several generic responses in section clarifying how this work is different from what appeared in the past literature. Most importantly, please see section L.5 where we highlight how this work can be significant to the community. We understand the issue that inverse dynamics models have appeared in the past literature. We emphasize the context of which such objectives have been used (mostly for online RI. and exploration), whereas we are primarily focused on learning in presence of exogenous noise (of varying difficulty) and found an instantiation of multi-step inverse models that be very significant in learning from such datasets in offline RL. Furthermore, theoretical works have studied variations of the ACRO objective theoretically (Efroni et al., 2021; Lamb al., 2022), while we provide a practical algorithm based on pre- training representations that can be used off the shelf in any existing deep RL based offline RL pipeline. Our primary focus is to provide such findings, along with the novel datasets, which we believe would be significant for stepping towards practical scalability of RL in the real world.
Response:
M.2 RESPONSE TO REVIEWER M6QW
M.2.1 KEY ASSUMPTIONS AND CLARIFICATION
Feedback: 2) there is a lack of in-depth discussion on the assumplion and limitation of the cur-
rent method, including requirement for offline dataset. I may missed some information in the paper and appendix. I would consider increase the ratings if the authors can provide convincing analy- sis/results for the above points.
Thank you for the feedback, and apologies if the key assumptions made in the work were not clear from the main text. We repeat here some of the key yet minor assumptions:
1. We assume that the reward function is free of any exogenous noise; ie, even observations consist of exogenous noise, the rewards depend the raw endogenous states, r(s,a).
39

---

Under review as a conference paper at ICLR 2023
2. We clarify that purely random policy can be free of any exogenous noise in observations, since the action selection of a random policy does not depend on the observations. Any other non-random polciy would depend on the exgeonous noises (T(a)z) where consists of both exogenous and endogenous part of the state space. However, when we pre-train rep- resentations, and then in RI. algorithm, we indeed learning policy (<aaz) and depending on the representation being learnt, the resulting optimal policy would depend on the extent which depends on the exogenous noise. We show that the latents recov- ered from ACRO are robust and filters out exogenous noise compared to other baselines considered.
3.
There are no assumptions being made on the requirement of the dataset. The offline datasets are collected with SAC policy in presence of exogenous noise, and can have different data distributions depending on the data collecting policy (expert, medium-expert or medium datasets). We emphasize and repeat here that the offinn datasets consist of all pixel-based observations.
4.
We assume we are in an exogenous block MDP setting Efroni et al. (2021); Du et al. (2019) that has been studied the theoretical RL community before. In this work, we provide practical algorithm that can be scaled easily to existing deep RL algorithm pipeline, where learning representations play a key role. The most important contribution of the work is that ACRO can be generally applicable in settings with different types of exogenous information, under block MDP assumptions Du et al. (2019)
M.2.2 COMPARISONS WITH PRIOR WORK LAMB ET AL. (2022)
Question:
Novelty of the work compared to Lamb et al. (2022) Following bullet 1, I am wonder- ing whether the authors can comment more on the theoretical comparison with Lamb et al. (2022). Without the bottleneck constraint, will the learned representation be guaranteed to discard exoge- nous information? If not, the claims of Section 3 do not hold for the proposed algorithm, right?
Thank you for the question about comparisons with prior work Lamb et al. (2022). We empirically compare ACRO with the AC-State objective Lamb et al. (2022) along with details on how ACRO is different compared previous works (also mentioned in related works). Please see the generic response to all reviewers about this in section L.I, since we believe clarifying these differences would be helpful for all reviewers.
M.2.3 DATA COVERAGE AND ROLE OF EXPLORATION OFFLINE RL
Question: : The algorithu adopts a process of offline representation pretraining and policy fine- tuning with frozen representation. However, such representation learning may highly depend on coverage of the offline dataset. I hope there could be more discussion on the requirement of datalexploration, both in theory and in experiments.
Response: Role of data coverage in offline representation learning: we agree that lack of coverage would degrade the algorithm. This, however, is well known fact in the offline RL setting. The fact that the concentrability coefficient is finite implies there is a good dataset coverage. We do agree that studying the theoretical aspect of this question is needed. In this work, we mostly focused on this problem from its empirical side and established the failure of existing approaches, as well as offering fix (that works under the dataset coverage assumption.
Further, the online RL problem with exogenous noise is also of interest. As of now, there is no algorithm with provable guarantees for the general RL problem with exogenous noise (there are some solutions under different sets of assumptions we elaborated on in the related work section). Our work does not tackle this challenging problem, but study the offline aspects of its; a prevalent problem from practical perspective.
Experiments We varied the amount of data coverage in the offline datasets, by taking the of times, the data collecting policy takes random actions. We vary the % of random actions from 10% to 50% where we assume that more randomness in actions taken by an expert policy means higher state space coverage. We follow the same experiment setup before, and now show how the performance of ACRO (and two other baselines) varies as we have lower to higher coverage in the datasets, in figure 30.
40

---

Under review as a conference paper at ICLR 2023
ACRO
Datasets
State Action Datasets
140
Cumutalive Retums
120
Cumuuallive Relume
100
Random Acdor Daissal
Acdde Dalaee
Randon Actor Datasal
Random Dalaoe
20
40
llerations
16e
Itorations
164
(a) ACRO (ours)
(b) AC-State Lamb et al. (2022)
DRIML with Rendom Action Datasels
ACRO with Sequence Prediction with Random Action Datasets
17.5
No Ramson
1.20
Rardiam Actora
Randon Acdon Dateoal
Random Acdon Dalaves
Currulative Relums
Rlandom Actin
Carmulattvve Riaurree
1:33
Rarntomykzttnn Daboor
12.5
flandom Actin Datecel
flarstom Action Datior
50%
Random Acdon Dataset
50%
Rande Acdoo Dataser
12.0
2.5
5.0
2.5
6.0
lteralions
1et
Iterations
Ted
(c) DRIML Mazoure et al. (2020)
(d) ACRO with Sequence Prediction
Figure 30: Experimental results comparing different representation learning methods as we have varying amounts of coverage on the datasets. We make the assumption that adding more random actions to the datasets means the dataset has higher coverage of state and action space. We show that the % of random actions datasets increases, the performance of each method degrates, especially the ones like DRIML that are independent of action prediction. In contrast, all other methods that rely on action prediction, suffer less depending on the amount of random actions that may exist in the datasets. This shows potential for representation objectives based on action prediction that the performance of these methods degrades least even if the dataset quality poor (where in our case, higher coverage due to more random actions means the dataset is degrading from an expert dataset to more of a random dataset).
M.2.4 ACRO IN ONLINE RL FOR REPRESENTATION LEARNING AND EXPLORATION
Feedback: : The representation is only tested with offline RL finetunin while it is not clear whether the learned representation is sufficient for learning optimal policies in an online manner. Intuitively, the latter would be harder and thus more interesting (the representation should be not only good for modeling optimal behaviors, but also good for exploration). If the claims hold that the learned representation is controllable, it should be able learn policy by interaction. Combined with bullet 3 above, am worried that if the offline dataset does not have good coverage, downstream online policy learning will be hard.
Thank you for the comment and providing suggestions on how we can extend the paper. We would like to note here that the problem considering the online setting, and extending beyond our current focus on offline RL is that the online setting is dependent on exploration too (since representation learning in online setting is inherently coupled with the amount of exploration) and this something, while interesting, is perhaps beyond the scope of our current work. In future, we would definitely consider extending ACRO to the online setting, which might additionally require mechanism for exploration while ignoring exogenous noise. In our current work, we mostly studying robust- ness to exogenous information given datasets, and we study the exploration question there through assumptions on the role of coverage in the datasets (as you noted in the previous question). As pointed out rightly, in offline RL this is tested based on data coverage in the data distributions, and response and results above is an attempt towards addressing this with ACRO.
41

---

Under review as a conference paper at ICLR 2023
M.3 RESPONSE TO REVIEWER YX3D
M.3.1 INVERSE DYNAMICS IN RL
It would have been nice to include in the introduction some indication of how this work innovates over existing representation learning methods since inverse dynamics models have been heavily used in RL and model-based RL.
Feedback: :
Response: : We would like to thank you for bring this question, and highlight that inverse dynamics models have appeared in the literature in the past Pathak et al. (2017); Efroni et al. (2021); Lamb et al. (2022). Please see how response on this in section L.1 where we compare how the ACRO is different compared to past works in terms of novelty, and significance in offline RL. We emphasize that even though objective-wiss ACRO is a variation of past works, the context to which we study this in presence of exogenous noise, and to show the capability of learning generic robust representations that can be used in any RL algorithm, can be of practical importance. In addition to discussions around this in generic responses in L, we would be happy to answer further questions on novelty and significance of ACRO.
M.3.2 POMDP AND EXOGENOUS BLOCK MDP
Question: : The description of an MDP actually describes a PO-MDP, with an exogenous block context. In addition, the typical formulation of the agent carries T(a | F1..a), or the history of past observations. Policies conditioned on the current observation require the "rich observation" assumption, which should be stated in 2.1, not 2.2.
The decision process we consider here is an MDP in the sense that the observations are Markovian; the observation time step dir does not depend the history conditioning on -t-1 and dz-1- We will highlight that in section 2.2 to avoid confusions and miss-nnderss in the future. Indeed, in this work do not study the problem of representation learning in POMDPs (in the presence of high-dimensional observation data). This problem is of great interest and challenge.
M.3.3 ABOUT MULTI-STER INVERSE DYNAMICS OBJECTIVE
The work claims that the multi-step objective allows for more diverse representations because it forces the representation to encode long onngrrange dependencies. However, only the first action in the sequence is added, so it is entively unintuitive as to why the representation would have be different.
Question:
What is the specific reference on the claim that ACRO allows for "more diverse representations"? We talk about our newly proposed datasets having diverse exogenous distractors, but I'm not of us claiming that the representations are more "diverse".
Feedback: In the example of the action being stored in the observation, this would still be the case even if 1+k is given.
Suppose that 7. = (It, at-1): 1), the observation stores an original observation along with the most recent action. Then consider the 1-step inverse model: P(a,)ē,, The optimal solution can thus completely ignore æ. Note that you can't use H, (zt, as the observation for a counter-example because it violates causality. You are right that recording many recent actions will cause issues for multi-step inverse models, so we updated the text to reflect this.
P(a Zr, at-1, F141, az)
"Furthermore, this representation introduces its own issues, since the problem itself now ill-defined suppose that there are multiple action sequences between St and st+1- In this case, the inverse dynamics model would have to ouipul an ambiguous probability which would be dependent on the frequency one path was taken over another the dataset. It is not clear what representation this would encode."
Feedback:
You are right that there are multiple action sequences for travelling from one state to another. Even in the case of simple empty gridworld, there are two equally valid actions for going from the top- to the bottom-right in as few steps as possible. We can either go down, or go right on the first step. As another example, if the agent starts in the top-left corner, going left or going up will both keep the agent the same position. Thus the NLL of the optimal multi-step inverse model will indeed not be zero for most environments. Can you explain further why you believe this is problem?
42

---

an Under review as a conference paper at ICLR 2023
M.3.4 TASK RELEVANT INFORMATION
Question: A clear weakness of this work is that though it equates exogenous with irrelevant, actually likely to remove information that could be task-vital. This is because the inverse dynamics model only needs capture sufficient information predict the first action between pair of states, and can lose any information once it can make that prediction. Suppose that we are in an object manipulation domain where the agent
grasp block. The representation has reason to capture the block because only the arm is necessary to predict the action. None of the experiments appear to capture these kinds of relationships
We would like to thank you for such insightful comments and feedback, on whether ACRO objective can ignore task relevant information. If this is true, then we agree this would definitely be key weakness of our work. We provided a generic response to this in section L.3 since this would also provide better insights to other reviews. Please refer to section L..3.
Response:
M.3.5 COMPARISONS TO PRIOR WORK LAMB ET AL. (2022)
Question This work repeatedly cites Guaranteed Discovery of Controllable Latent States with Multi-Step Inverse Models (Lamb et al 2022), which appears to approach the same problem with the same solution. If this work is by the authors, then this is disingenuous since the authors' claims are being supported by citations of the author's 's own work. Furthermore, the claims being supported, such the resolving of single-step issues by using multi-step predictor, appear be made without strong support.
Response: The question about comparisons prior work Lamb et al. (2022) have been brought up by other reviewers too. We provided a generic response, along with additional experimental results, and detailed comparison between ACRO and Lamb al. (2022) in section L. We emphasize that there are few major differences between the two works, and how Lamb et al. (2022) can be practi- only recovers a small underlying tabular MDP for planning and does show any
cally limited significance in practical deep RL setting. In contrast, ACRO pre-trains robust representations that can then be used in any existing deep RL algorithm, which makes the approach more generic, and not requiring any specific discrete bottlenecks as in Lamb et al. (2022).
M.3.6 PREVIOUS WORKS WITH DISTRACTORS IN RL ZIIANG ET AL. (2020)
Question: The proposed offline RL datasets are the same (at least in conceit) as those described
by Learning invariant representations for reinforcement learning without reconstruction (Zhang et. al. 2020)
Response: We would like to emphasize how the exogenous distractors considered in this work is quite different than what appeared in past literature. In section L.5 we provided a generic response to all reviewers about the significance and impact this work can have, towards practical scalability of RL in real world. Prior works in the online setting, as in Zhang et al. (2020) have only considered time uncorrelatred distractors whereas build primarily the theoretical studies of exogenous RL Efroni et al. (2022b: 2021) to further provide an offline dataset benchmark with exogenous information. We would like to note that we will be releasing these datasets so they can be used as benchmarks, for future research and reproducibility, beyond the novelty of specific exogenous noise that we considered. We also provided a similar response to another reviewer question about this in section M.1.1.
M.3.7 ASSUMPTIONS ON DATASETS
Please see response in section M.2.1 where we listed the key assumptions and clarified them.
On the issue of assumptions in datasets: There no assumptions being made the requirement of the dataset. The offline datasets are collected with SAC policy in presence of exogenous noise, and can have different data distributions depending the data collecting policy (expert, medium- expert or medium datasets). We emphasize and repeat here that the offline datasets consist of all pixel-based observations.
43

---

Under review as a conference paper at ICLR 2023
M.4 RESPONSE TO REVIEWER GC23
M.4.1 EMPIRICAL COMPARISON TO PRIOR WORK LAMB ET AL. (2022)
Question: No empirical comparison the closely related prior work Lamb et al. (2022). I highly encourage the authors to move the majority part of Section 2.2 (Proposed Method) to Section 2.1 Lamb al.
(Preliminaries) and only highlight new ideas in Section 2.2...... comparison (2022) should be included, because the paper claims that learning a continuous representation is the main distinction (hence novelty) to Lamb et al. (2022). Without this, the use of continuous representation is not well-justified.
Response: : We included generic response in section L.1 comparing the ACRO objective with Lamb et al. (2022), both empirically and theoretically. We included details on how ACRO differs from prior works, and why an approach based on ACRO can be generally applicable, compared to the limitations in Lamb et al. (2022).
M.4.2 EXPERIMENTS WITHOUT ANY EXOGENOUS NOISE; ACRO WORKS BETTER EVEN WITHOUT ANY EXOGENOUS INFORMATION
Question: It would be more comprehensive to include a noise/dissrractorffee setting to get an idea of how much each method suffers visual distractors.
Response: We actually included results in table 2 where we do not consider any exogenous noise distractors all. All the results provided in 2 are based the no-distractor setting from the Lu et al. (2022a) pixel-based offline benchmark. We label this as the EASY-ExO setting, with other more difficult exogenous noise datasets as in MEDIUM-EXO and HARD-EXo based results, that are extended, and provided as new benchmarks in this work.
We would like to emphasize here that even though the main focus of the work is on exogenous information in offline datasets, our results as in table 2 also highlights that ACRO can be generally applicable for representation learning. While we focus mostly on exogenous information in the description of our setting, we should have emphasized the scope of the ACRO technique is more broadly applicable in the representation learning literature for RL.
M.4.3 NORMALIZED PERFORMANCE FIGURES
We apologize that some of the normalized or averaged results that were presented in the paper were not clear. We originally included details of the averaging the caption of figures. For furhter clarification, and since it would be helpful for other reviewers as well, we provided an explanation of the normalized or averaged results figures section L.4.
Response: :
M.4.4 EXPLANATION OF RECONSTRUCTION FIGURES
Feedback: Although I appreciate the analysis through reconstruction (Figure 7), it's unclear whether the proposed method is clearly better than DRIML.
Response: : The reconstruction for ACRO removes exogenous information more than DRIML. Note that blurring of the controllable part does not necessarily about the controllable part in the encoder. Specifically for such locomotion tasks, only capturing certain pixels for each joint is actually sufficient to predict the actions accurately. Therefore, in principle the controllable part can be even more noisy while still encoding sufficient information for control. This combined with the fact that ACRO achieves better downstream performance on both with and without distractors case suggests that it is able to remove exogenous information and retain endogenous information better than DRIML. Another case in point here is the DrQ reconstructions, which blur out most of the agent yet achieve similar performance to DRIML.
that there is less information
M.5 RESPONSE TO REVIEWER MOWF
We would like to thank the reviewer for the high score in the paper, and greatly appreciate the insightful comments and feedback. The other reviewers raised some concerns about the scope of this work too, and provided generic response all reviewers, in section L.. We hope the
44

---

Under review as a conference paper at ICLR 2023
responses to other reviewers would also help in fully understanding the significance and impact of our work.
M.5.1 ASSUMPTIONS AND LIMITATIONS
Feedback: : Limitation on exo-free policy that the learning problem requires data from an exo-free policy. The experimental results are both extensive and in support of the proposed method. The main drawbacks see are the requirement for an exo-free policy to learn from and the block assumption, which excludes POMDPs.
Response: Thank you for the feedback, and we apologize that some of the assumptions made in this work were not clear. We list below some of the key assumptions made in this work
1. We assume that the reward function free of any exogenous noise; ie, even observations consist of exogenous noise, the rewards depend the raw endogenous states, r(s.a).
2. We clarify that purely random policy can be free of any exogenous noise in observations, since the action selection of random policy does not depend on the observations. Any other non-random polciy would depend on the exgeonous noises I (alar) where consists of both exogenous and endogenous part the state space. However, when we pre-train rep- resentations, and then use it in an RI. algorithm, we are indeed learning policy (a) and depending on the representation being learnt, the resulting optimal policy would depend on the extent to which depends the exogenous noise. We show that the latents recov- ered from ACRO are robust and filters out exogenous noise compared to other baselines considered.
3. There are no assumptions being made on the requirement of the dataset. The offline datasets are collected with SAC policy in presence of exogenous noise, and can have different data distributions depending on the data collecting policy (expert, medium-exper expert or medium datasets). We emphasize and repeat here that the offline datasets consist of all pixel-based observations.
4.
We assume we are in an exogenous block MDP setting Efroni et al. (2021); Du et al. (2019) that has been studied in the theoretical RL community before. In this work, provide practical algorithm that can be scaled easily to existing deep RI. algorithm pipeline, where learning representations play a key role. The most important contribution of the work is that ACRO can be generally applicable in settings with different types of exogenous information, under block MDP assumptions Du al. (2019)
M.5.2 ACRO PERFORMS BETTER EVEN WITHOUT ANY EXOGENOUS INFORMATION
Question: is interesting that ACRO outperforms other method even when there are distractors (if I understand Table 2). This would imply that the representation learned by ACRO is a better representation than alternatives, even when there is no exogenous information. This is interesting, since the theory used to derive ACRO is all based on the existence of exogenous information. Do the authors have some thoughts here?
Respone We would like to emphasize here that even though the main focus of the work is on exogenous information in offline datasets, our results as in table also highlights that ACRO can be generally applicable for representation learning. While we focus mostly on exogenous information the description of our setting, we should have emphasized the scope of the ACRO technique is more broadly applicable in the representation learning literature for RL.
Furthermore, since the offline datasets from Lu et al. (2022a) which we label as EASY-ExO are pixel- based data, we want to emphasize that technically there is still some level of exogenous information present here for example, the background, floor and colors the control tasks may contain some exogenous info from the observations that are not relevant for solving the task. This is why, in the main text, we refer to the results of 2 as EASY-EXO. However, as the reviewer has rightly pointed ACRO is more generic as representation learning approach for offline RL. Even though inverse dynamics models have appeared in the literature in past, in other context such as exploration in online RL, emphasize that ACRO objective based on multi-step inverse model can be particularly suited for offline representation learning, which has not been well studied, empirically, in the past literatre.
45

---

Under review as a conference paper at ICLR 2023
Q1
40
d0
00
80
$1
$3
0o
02
02
02
85
Figure 31: An MDP where one-step models can fail.
M.5.3 EXPLANATION OF RESULTS
Question:
What is the difference in setup between Table 2 and Figure 5? As far as understood, Table 2 contains no distractors. while Figure 5 contains the distractors used by Lu et al. [1]. What I don't I understand is what "easy", "medium" and "hard" refer to in Figure 5.
Response: Thank you for asking for the clarification. You correct that 2 contains results without any exogenous distractors. It contains results from using the Lu et al. (2022a) benchmark it is, without distractors. Figure contains results from the Lu al. (2022a) benchmark too, except now we have the distractor suite from V-D4RL benchmark. In their datasets, they have different type of distractors, which induces distribution shift in the dataset, which is referred to asy easy-shift, medium- shift and hard-shift. Figure 5 is therefore what we refer to as the results for EASY-ExO our setting. In this work, mainly extend from Lu al. (2022a) and collect new datasets where we vary the type of exogenous distractors, not considered in Lu et al. (2022a) referred as MEDIUM-EXO and HARD-EXO settings.
Question: Normalized result figures + Where are the numbers in Table 3 coming from? Are these averuges across the tasks? In which distractor setting?
Response: We provide clarification for the normalized or averaged results included in main text, in section L.4. We aplogize that even though we mentioned it in the appendix, this was not clear to few reviewers to what the normalized results mean in this context.
Table contains results of ablation studies with the ACRO objective. For this, we consider only the Cheetah-Run task with different dataset types (random, medium-replay and expert) with the HARD-EXO exogenous distractor. As described Section 5.4, we perform ablations with the ACRO objective where we added the timestep embedding, used only single-step future observation (instead of a varied value of horizon k), and removed the conditioning on future observations.
Additional Results for Rebuttal
We include some additional results the rebuttal, which were primarily not raised by reviewers, but are still useful for overall understanding and quality of work.
N COUNTEREXAMPLE FOR OTE-STEP MODELS
In this section, we build an MDP and dataset of trajectories in that MDP where a one-step model will fail to allow the learning of the optimal policy. The MDP can be seen in Figure 3 1, all transitions are deterministic, and yield 0 reward, apart for action 02 in state 82, which gives a reward of 1 (bold arrow in the graph). The final state 8 denotes the termination of the trajectory
46

---

Under review as a conference paper at ICLR 2023
We consider the dataset comprised of the following five trajectories, formatted as (31, ae, Tt, )..
D = (s0, ao, 0, su, ao. O, su. a2 0,sy)
(22)
(su, a1 0, $1 ao 0. $1 a2 0,
(23)
(50, 1,.01.0,00 81, 41 0, 22 40 0. 82, 42 1,5
(24)
(50. a1 81, a1 0, 82 a1 0,83 ao. 8, a2 0,5
(25)
(so, a1 0,81, a1 0, 82, a1 0, 83, a1 0, s0 02,00,0000 0,
(26)
loops twice in So and then terminates. The other four trajectories reach $1. 82, 83 and so by taking a1 a minimal number times, then loop once (except Ts), and terminate. We note that this dataset covers the full state and action space.
possible to reach a 0 loss with a one-step inverse model pr built on top of the following rep- resentation é from S to X' (1, ((so) ((s2) To. ((s1) Φ(53) I1 and ((ss) = If:
Pi (a)xro, Ho) = 5amaa
(27)
Pi (a) [o, F1 Sa=ay
(28)
pi (a/ da=az'
(29)
Now, once projected onto X, the dataset becomes:
D (wo, ao, 0, to, an, O, ro, a2, 0, 0,
(30)
(2o, a1 0, Fi, do, 0, 21, a2 0,z
(31)
(oo. 01, O, æ1, a1, ro, ao. O, vo. 02 1,
(32)
(oo, an, O, 01,,,,1,400 01, O, ro, ai, 0, 21 ao 0, 21 02,
(33)
(wo, an 0,,1,,111 0, vo, a 0, 01 0, 02 a2, 0,
(34)
76
We that action 02 in state 2g has expected reward of 1/3, and it is the only state-action pair with a non-zero expected reward. Any reasonable offline RI. algorithm applied to this dataset will output policy with non-zero probability assigned to that action in zg- However, executing that policy, i.e., taking action 0g in state So terminates the episode with O reward, which is suboptimal.
On the other hand, an optimal two-step model (and by extension an n-step one) will not collapse states su and 82 as this would prevent distinguishing the first action taken in from the first action taken T3 (corresponding respectively to pairs (sp. sy) and $2) after two timesteps). Conse- quently, an offline RL applied in representation can still learn an optimal policy.
O END TO END FINE-TUNING EXPERIMENTS
In all our current experiments, primarily presented in the paper. we only pre-train the representa- tions and then keep fixed during later fine-tuning of the offline RL algorithm. In this section,
47

---

Under review as a conference paper at ICLR 2023
perform variation where instead of only pre-training, learning where the representations are simultaneously updated along with the RL algorithm. Ex- perimental results in this section, with different exogenous distractor types, shows that even in the end to end learning setting, ACRO can significantly outperform the baselines by learning robust representations.
perform fine-tuning end to end
Due to the questions about relevancy of task information we also perform additional experiments also predict the rewards using reward function
where in addition to the ACRO objective, predictor. We show that even when trying to predict task relevant reward information, the perfor- mance of ACRO with and without reward prediction are almost similar. This further verifies that ACRO does not ignore any task relevant information, such as information present in the reward function.
O.1 EXPERIMENT RESULTS CORRELATED AND UNCORRELATED IMAGE DISTRACTORS
Uncorrelated Exogenous Image Distractors Figure 12 shows results with exogenous image distractors placed the comeer of the agent observation space. Figure 13 and Figure 33 addition- ally shows results with STL10 exogenous distractors placed the side of the agent observations, augmenting the observation space of the agent. We find that ACRO can significantly outperform baselines irrespective of whether pre-train representations only, in Figure 13 or do end end fine tuning of representations with uncorrelated exogenous observations. as shown in Figure 33.
48

---

Under review as a conference paper at ICLR 2023
DCAL
200A
16
04
(a) exxpc
(b) Humanoid-expert--2
(c) Walker-expert-v2
1:50
(d) Hallcheetah-mediune medium-expprt--22 (e) Humanoid -emmanooiimddiuu---xert--22222 (1)
D00A
15
12
(g) Halfcheetah-mmedune
(h) Humamoid-medium--22
(i) Walker2d-medium--2
100
K003
10
CPN
(i) mailineetal-mmdiuu--eeeaaaaa
(k) Humanoid-meddun--eelly
(1) Walker2d-mediun---ppay
Figure 32: Performance comparison with uncorrelated exogenous observations on the side; pre-training representations only In this experiment, we place an exogenous image in the corner of the observation space of the agent; fine-tuning or end to end learning of representations, comparing ACRO, ACRO + Reward Prediction with several other baselines in presence of cifar exogenous image distractor.
49

---

Under review as a conference paper at ICLR 2023
3.0
1000
2002
(a) Halfchectah-expertt222
(h) Humanoid-expent-22
(c) Walker-expert-v2
2:00
DC14
CRAA
13
Halfchoetah- wwwssshh---i----mme -2 Humanoid-mcdium- mwdiim-cxpeet--22222
(f)
320
(g) Calfchectah-mediung
(h) Humanoid Humanoid-medium--2
(i) Walker2d- 2-liurr2-medium-2222
DOAA
CMA
44
Hallchetal-mmediun--elaaaa
(k) Humanoiddmeddiun--ellayy
()) Walkerrd-meddium--relayy
Figure 33: Performance comparison with uncorrelated exogenous observations the side; end to end, fine-tuning of representations Comparisons with baselines for ACRO and ACRO with additional reward prediction
50


Reviews:
[{"review_id": "cv3kQ7qvXP", "reviewer": "Reviewer_M6qw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-step inverse model to learn the representation for RL problems with offline data. The proposed method ACRO predicts the current action based on the current state and a future state, in a reward-free manner. ACRO can learn controllable states while keep invariant to exogenous information. Experiments in various settings show that ACRO outperforms prior methods. Theoretical analysis shows the benefits of ACRO. The paper also provides several benchmarks to evaluate representation learning in RL.", "summary_of_the_review": "This is an interesting paper with good empirical results. But I am hesitating to recommend it for acceptance, because 1) the method is a direction extension from prior work (and in the meanwhile it may lose the theoretical guarantees of prior work), and 2) there is a lack of in-depth discussion on the assumption and limitation of the current method, including requirement for offline dataset. I may missed some information in the paper and appendix. I would consider increase the ratings if the authors can provide convincing analysis/results for the above points. ", "strength_and_weaknesses": "### Strengths\n\n1. The idea is neat and easy to implement, intuitive and theoretical grounded.\n2. The empirical results are promising, covering multiple scenarios.\n3. Good presentation and visualization of the ideas and results.\n4. Interesting theoretical insights.\n\n### Weaknesses and Questions:\n\n1. The novelty of the proposed method is relatively limited, as a continuous-control extension of prior work (Lamb et al. 2022). \n2. Following bullet 1, I am wondering whether the authors can comment more on the theoretical comparison with Lamb et al. 2022. Without the bottleneck constraint, will the learned representation be guaranteed to discard exogenous information? If not, the claims of Section 3 do not hold for the proposed algorithm, right?\n3. The algorithm adopts a process of offline representation pretraining + policy finetuning with frozen representation. However, such representation learning may highly depend on coverage of the offline dataset. I hope there could be more discussion on the requiremenet of data/exploration, both in theory and in experiments.\n4. The representation is only tested with offline RL finetuning, while it is not clear whether the learned representation is sufficient for learning optimal policies in an online manner. Intuitively, the latter would be harder and thus more interesting (the representation should be not only good for modeling optimal behaviors, but also good for exploration). If the claims hold that the learned representation is controllable, it should be able to learn a policy by interaction. Combined with bullet 3 above, I am worried that if the offline dataset does not have good coverage, downstream online policy learning will be hard.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XD1yyiELSa", "reviewer": "Reviewer_GC23", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn a multi-step inverse model [Lamb et al.] to learn a representation that ignores exogenous information (i.e., uncontrollable information) for offline RL. In addition, this paper introduces several temporally-correlated and diverse visual distractors on top of the v-d4rl dataset to investigate the quality of the representation in RL. The results show that the proposed method outperforms several self-supervised learning methods including CURL and DRIML.", "summary_of_the_review": "Although this paper demonstrates good results on offline RL with exogenous information, the main idea seems to mostly come from the prior work [Lamb et al.], which is not clearly stated in the current manuscript. The paper would benefit from making a clear distinction from the prior work and adding a comparison to the prior work.", "strength_and_weaknesses": "[Strength]\n* The idea is technically sound.\n* The results look good. \n* Introduces interesting visual distractors.\n\n\n[Weakness]\n* The technical novelty is not significant, as the main objective is essentially the same as Actor-Controller State from [Lamb et al.].\n* The novelty compared to the prior work [Lamb et al.] is not clearly explained.  \n* No empirical comparison to the closely related prior work [Lamb et al.]. \n* Some of the empirical results are not convincing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VeTKGSBVJ74", "reviewer": "Reviewer_GMBs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers learning a good representation for offline reinforcement learning algorithms with pixel-based visual observation space. The authors aim to extract the representation, which ignores any control-irrelevant information. The authors choose multi-step inverse models to learn the observation representation because the multi-step action prediction for learning exogenous-invariant representations has been shown useful in  RL theory community.\n\nThe proposed method ACRO first trains the image encoders so that the features can be used to predict multiple actions between two observations. With the well-trained encoder, the downstream offline RL algorithm TD3+BC takes in the learned representation as policy input to output actions. \n\nIn order to evaluate the robustness of the proposed method for pixel observations, the authors introduce new datasets with temporally-correlated noise and diverse noise in the observations. The dataset is similar to D4RL but with some exogenous images or videos incorporated in the observations as noise.\n\nIn comparison with other representation learning methods, the proposed one significantly outperforms the baselines.", "summary_of_the_review": "This paper presented a well-motivated representation learning method for offline RL, with great experiment performance and some theoretical foundation. But the proposed method may be only suitable for locomotion tasks, due to the limitation of the learned representation. This seems not general and widely useful.", "strength_and_weaknesses": "Strengths:\nThe authors introduced the multi-step inverse model in offline RL problems and proved that Bellman completeness can be achieved via the representations without exogenous noise.\nThe experimental results support the claims well. In the datasets of noisy observations, this paper demonstrates the significant advantage of the proposed representation learning approach.\nThe newly introduced datasets for offline RL can be useful for the community.\n\nWeaknesses:\nThe major concern is about the limitation of the representation only containing control-relevant information. How about there are other moving entities in the observations, which are not controllable by the agent but are still very important for the success of the policy? For example, a robot navigates busy streets. Pedestrians are not controllable by the robot, but the robot should consider the status of the pedestrian to avoid collisions. Will the proposed representation learning approaches fully ignore pedestrians and make errors in the decision?\n\nAnother example is robotics manipulation tasks. The robot hand should manipulate the objects, so the object position and orientations are important information for the control policy. But the proposed method may tend to ignore the object information. This is problematic and limits the scope of the proposed method.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FSvO1CLs7T", "reviewer": "Reviewer_moWf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for learning representations that are  invariant to task-unrelated information in the offline RL setting.  Theoretical results are presented which show that such representations  allow accurate value function approximation and that using  representations which are not invariant to task-unrelated information  can cause value function approximation to fail. Experiments and ablation studies support these claims.\n", "summary_of_the_review": "The paper makes solid experimental and theoretic contributions to the problem of discarding irrelevant information in RL. The experimental results are both extensive and in support of the proposed method. The main drawbacks I see are the requirement for an exo-free policy to learn from and the block assumption, which excludes POMDPs.", "strength_and_weaknesses": "**strengths**\n\n* The paper's about a relevant question (how do we ignore unimportant/distracting information in RL?).\n\n* The paper is rigorous and argues its case convincingly. The approach is motivated by theoretical arguments. The experiments are extensive and show an edge over strong baselines. Ablations empirically motivate the decisions that the authors make.\n\n**weaknesses**\n\n* The block assumption (that the emission distributions of different states are disjoint) is limiting. It amounts to a full-observability assumption.\n\n* Another limiting factor is that the learning problem requires data from an exo-free policy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3pOCV8qSpN", "reviewer": "Reviewer_yx3d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "By defining exogeneous information as information irrelevant for control, this work works to learn a representation that removes these features in the context of offline RL. These representations are learned by taking a latent space learned through inverse dynamics modeling. It also provides a set of benchmarks for offline RL by adding background videos to Mujoco tasks. The key insight of this method is to predict the first action of the sequence connecting the first state and a state k steps in the future. \n", "summary_of_the_review": "I propose to reject this work due to lack of novelty since inverse dynamics models are a common choice for state representation learning. It also has some concerns with the blind format, since it repeatedly cites works that are likely by the same authors. Last, the writing and definitions are not entirely consistent with the motivation. ", "strength_and_weaknesses": "Strengths: \n\nThis work investigates an important question of how to construct representations based on controllability and introduces a method for doing so.\n\nThis work implements a large number of baselines, which would be useful if released as code. \nIt would have been nice to include in the introduction some indication of how this work innovates over existing representation learning methods since inverse dynamics models have been heavily used in RL and model-based RL.\n\nWeaknesses: \nThe description of an MDP actually describes a PO-MDP, with an exogenous block context. In addition, the typical formulation of the agent carries \\pi(a|x_{1,...t}), or the history of past observations. Policies conditioned on the current observation require the \"rich observation\" assumption, which should be stated in 2.1, not 2.2.\n\nThere is an unstated assumption that the policies from which the data is collected are also invariant to the exogenous information, otherwise, information about the action between two states would result in improved performance. This is actually a significant issue in the offline RL setting since we don't have control over the collected data.\n\nThe work claims that the multi-step objective allows for more diverse representations because it forces the representation to encode long-range dependencies. However, only the first action in the sequence is added, so it is entirely unintuitive as to why the representation would have to be different. In the example of the action being stored in the observation, this would still be the case even if t+k is given. Furthermore, this representation introduces its own issues, since the problem itself is now ill-defined: suppose that there are multiple action sequences between s_t and s_{t+1}. In this case, the inverse dynamics model would have to output an ambiguous probability which would be dependent on the frequency one path was taken over another in the dataset. It is not clear what representation this would encode. Figure 2 also shows x_t and x_{t+k} with the same image.\n\n\nA clear weakness of this work is that even though it equates exogenous with irrelevant, it is actually likely to remove information that could be task-vital. This is because the inverse dynamics model only needs to capture sufficient information to predict the first action between a pair of states, and can lose any information once it can make that prediction. Suppose that we are in an object manipulation domain where the agent uses an arm to grasp a block. The representation has no reason to capture the block because only the arm is necessary to predict the action. None of the experiments appear to capture these kinds of relationships", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject