Learning to Act without Actions
Authors: Dominik Schmidt
Minqi Jiang
Venue: ICLR 2024 spotlight (2024)
Keywords: reinforcement learning
world models
inverse dynamics models
imitation learning
representation learning
Primary area: reinforcement learning
TL;DR: We introduce Latent Action Policies (LAPO), a method for recovering latent action information—and thereby latent-action policies, world models, and inverse dynamics models—purely from videos.

Abstract: Pre-training large models on vast amounts of web data has proven to be an effective approach for obtaining powerful, general models in domains such as language and vision. However, this paradigm has not yet taken hold in reinforcement learning. This is because videos, the most abundant form of embodied behavioral data on the web, lack the action labels required by existing methods for imitating behavior from demonstrations. We introduce **Latent Action Policies** (LAPO), a method for recovering latent action information—and thereby latent-action policies, world models, and inverse dynamics models—purely from videos. LAPO is the first method able to recover the structure of the true action space just from observed dynamics, even in challenging procedurally-generated environments. LAPO enables training latent-action policies that can be rapidly fine-tuned into expert-level policies, either offline using a small action-labeled dataset, or online with rewards. LAPO takes a first step towards pre-training powerful, generalist policies and world models on the vast amounts of videos readily available on the web. Our code is available here: 
[https://github.com/schmidtdominik/LAPO](https://github.com/schmidtdominik/LAPO).

Published as a conference paper at ICLR 2024
LEARNING TO ACT WITHOUT ACTIONS
Dominik Schmidt"
Minqi Jiang FAIR Meta A1
Weco AI
ABSTRACT
Pre-training large models on vast amounts of web data has proven be ef- fective approach for obtaining powerful, general models in domains such as lan- guage and vision. However, this paradigm has not yet taken hold in reinforcement learning. This is because videos, the most abundant form of embodied behavioral data on the web, lack the action labels required by existing methods for imitating behavior from demonstrations. We introduce Latent Action Policies (LAPO), a method for recovering latent action information- thereby latent-action poli- cies, world models, and inverse dynamics models-purely from videos. LAPO is the first method able to recover the structure of the true action space just from observed dynamics, even in challenging LAPO enables training latent-action policies that can be rapidly fine-tuned into expert-level policies, either offline using a small action-labeled dataset, online with rewards. LAPO takes a first step towards pre-training powerful, generalist policies and world models on the vast amounts of vidcos readily available the web. Our code is available here:
pnnceuurally-generatee
environments.
INTRODUCTION
Training on web-scale data has shown to be an effective approach for obtaining powerful models with broad capabilities in domains including language and vision (Radford et al., 2019; Caron al., 2021). Much recent work has thus sought to apply the same paradigm in deep reinforcement learn- ing (RL, Sutton & Barto, 2018), in order learn generalist policies from massive amounts of web data (Baker et al., 2022; Reed et al., 2022). However, common methods for learning policies from offline demonstrations, such as imitation learning (Pomerleau, 1988; Ross & Bagnell, 2010) and offline RL (Kumar et al., 2020; Levine al., 2020), typically require action reward labels, which are missing from purely observational data, such as vidcos found on the web.
In this work introduce Latent Action Policies (LAPO), method for recovering latent-action information from videos. In addition to inferring the structure of the underlying action space purely from observed dynamics, LAPO produces latent-action versions of useful RL models including forward -ewaadd-ynamics models (world models), inverse inverseddnammiss models, and importantly, policies.
LAPO is founded the key insights that (1) some notion of latent action that explains vironment transition can be inferred from observations alone, and (2) given such inferred latent actions per transition, latent-action policy can be obtained using standard imitation learning meth- ods. Our experiments provide strong evidence that such latent policies accurately capture the ob- served expert's behavior, by showing that they can be efficiently fine-tuned into expert-level policies in the true action space. This results makes LAPO significant step toward pre-training general, rapidly-adaptane policies on massive action-free demonstration datasets, such as the vast quantities of videos available on the web.
Crucially, LAPO learns to infer latent actions, and consequently, obtain latent action policies a fully unsupervised manner. LAPO is similar to prior work (Torabi et al., 2018; 8; Schmeckpeper et al., 2020; Baker et al., 2022; Zheng al., 2023) in that we first train an inverse dynamics model (IDM) that predicts the action taken between two consecutive observations, and then use this IDM to add action labels to a large action-free dataset. However, unlike these prior works, which rely on some amount of ground-truth action-labelled data for training the IDM, LAPO does not make use of any labels and infers latent action information purely from observed environment dynamics. To do so,
"Correspondence dominik. dominik.schmidr. 22 @2@ucl.cc.uk

---

Published as a conference paper at ICLR 2024
DOWN
UP
RIGHT
Figure 1: UMAP projection of the learned latent action space for Mi niner alongside illustrative next-state pre- dictions generated by the FDM for each cluster of latent actions. Fach point represents continuous latent action generated by the IDM for transition in the video dataset. Each point color-coded by the action taken by the agent at that transition. For clarity, NOOP actions are omitted. The structure the latent action space is highly interpretable and closely corresponds to the true action space, even though no ground-truth action labels were used during training.
the IDM in LAPO learns to predict latent actions rather than true actions. These latent actions take the form of a learned representation that explains an observed transition. To learn this representation, LAPO makes use of simple unsupervised objective that seeks to establish predictive consistency between the IDM and separate forward dynamics model (FDM). The FDM trained on transitions consisting of two consecutive observations (ot, ot+1) to predict the future observation O++1, given the past observation Or and latent action. This latent action generated by the IDM given both past and future observations. Thus, unlike the FDM which sees only the past, the IDM has access to both the past and future and learns to pass useful information about the future to the FDM through the latent action. By making the latent action information bottleneck (Tishby et al., 2000), prevent the IDM from simply forwarding the entire future observation to the FDM, thereby forcing learn a highly compressed encoding of state transitions. We refer this encoding latent action, as it captures the observable effects of the agent's 's true actions.
In our experiments in Section 6, we train latent IDM via LAPO on large expert-level action-fit free offline datasets for each of the 16 games of the Procgen Benchmark (Cobbe et al., 2019; 2020). We observe that the structure the learned latent action spaces are highly interpretable, often exhibiting clusters that closely align with the true action space (see Figure 1). This is remarkable result, as our approach uses zero ground-tre -arnhhhhhhhhh action labels and instead recovers all action information purely from observed dynamics. To go beyond merely recovering action information, we then show how we can leverage the learned latent representation to quickly obtain powerful expert-level policies in downstream tasks: We first the IDM to assign latent action labels to each transition in the same video dataset on which it was trained. We then perform behavior cloning on the resulting action-observation dataset to learn latent-action policy, which imitates the latent actions predicted by the IDM. Finally, to deploy the policy in the online environment, we seek to turn it from a latent action policy into policy in the true action space. We show that if tiny, action-labeled dataset is available, a latent + true action decoder can be learned in a supervised manner and used to decode the outputs of the latent policy. This approach is extremely data-efficient, requiring only ~200 labeled transitions to exceed the performance of 4M steps of regular RL via PPO. Alternatively, an online environment is available, we show that simply fine-tuning the last layers of the latent policy with ceneral-purpose RI. algorithm allows the policy to rapidly adapt to the true action space and recover (or times exceed) expert performance. These results indicate that latent policies produced by LAPO capture meaningful behavior the latent action space.
While we take a first step in demonstrating the potential of these latent representations for obtain- ing powerful policies in Section 6, the fundamental contribution of this work is simply showing that comprehensive action information can be recovered from pure video through unsupervised ob- jectives, such the one proposed herein. This result opens the door to future work that employs representation learning methods like LAPO to train powerful, generalist policies and world models on massive video datasets from the web.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Most closely related to our approach, ILPO (Edwards et al., 2019) aims to infer latent actions via a dynamics modelling objective. However, unlike LAPO, ILPO learns latent policy rather than an IDM jointly with the FDM, and uses discrete rather than a continuous latent action space. For each FDM update, ILPO identifies the discrete latent action that minimizes the next-state prediction error, and only optimizes the FDM for that specific action. In practice, this objective can lead to mode collapse, where a feedback loop causes only a small number of discrete latents to ever be selected (Struckmeier & Kyrki, 2023). By using an IDM rather than discrete latent policy, LAPO avoids enumerating all hypothetically possible transitions and instead learns the latent action corresponding to the actual observed transition. To train the policy, ILPO minimizes the difference between the true next state and the expectation of the next state under the policy. This loss is ill-conditioned, as even when it is minimized, the next-state predictions for individual latent actions do not have to align with the environment. Moreover, LAPO's use of continuous rather than finite discrete latents may allow better modeling in complex and partially-observabl environments, as any information useful for prediction can be captured within the latent representation. We provide evidence for this in Appendix A.5, where we show that ILPO breaks down when applied to more than single Procgen level at a time. Finally, ILPO's use of discrete latents makes its computational complexity linear rather than constant in the latent dimensionality, due to the required enumeration of all actions.
In the model-based setting, FICC (Ye et al., 2023) pre-trains a world model from observation-only demonstrations. FICC thus seeks to distill dynamics from demonstrations, rather than behaviors, as done by LAPO. Moreover, while the cyclic consistency loss used by FICC closely resembles the LAPO objective, the adaptation phase of FICC must operate reverse, mapping true actions to latent actions in order to finetune the world model. Unlike the approach taken by LAPO, this adap- tation strategy cannot make direct use of online, ground-trute action labels, and requires continued training and application of world model. In contrast, LAPO directly produces latent action poli- cies imitating the expert behavior, which can be rapidly adapted online to expert-level performance. Outside of RL, Playable Video Generation (PVG, Menapace et al., 2021) similarly focuses world model learning with a similar objective for the purpose of controllable video generation. Like ILPO, PVG uses a more limiting, small set of discrete latent actions.
A related set of semi-supervised approaches first train an IDM using a smaller dataset containing ground-truth action labels, and then use this IDM to label larger action-free dataset, which can subsequently be used for behavior cloning. VPT (Baker et al., 2022) and ACO (Zhang et al., 2022) follow this approach, training an IDM on action-labeled data which is then used generate pseudo labels for training policies on unlabeled footage gathered from the web. RLV (Schmeckpeper et al., 2020) uses an observation-oni -anll dataset within an online RL loop, where action and reward labels are provided by an IDM trained on action-labelled data and a hand-crafted reward function respectively. Similarly, BCO (Torabi et al., 2018) trains an IDM through environment interaction, then uses the IDM to label action-free demonstrations for behavior cloning. However, relying interactions for labels can be inefficient, as finding useful labels online may itself be a difficult RL exploration problem. SS-ORL (Zheng et al., 2023) is similar to VPT but performs offline RL instead of imitation learning on the IDMM-labelled data and requires reward-labels. Unlike these methods, LAPO avoids the need for significant amounts of action-labeled data train an IDM, by directly inferring latent actions and latent policies which can easily be decoded into true actions.
LAPO differs from previous methods for imitation learning from observation (IfO; Torabi et al., 2019a;b; Yang et al., 2019), which typically require the imitating policy to have access to the true action space when training on action-free demonstrations. Crucially, unlike these prior approaches, LAPO does not require access to the ground-truth action space to learn from action-free demon- strations. Other methods leverage observation-oni. demonstrations for purposes other than learning useful behaviors. Intention-Condee Conditioned Value Functions (ICVF; Ghosh et al., 2023) uses temporal difference learning objective based on observation-only demonstrations to learn state representations useful for downstream RL tasks. Aytar et al. (2018) propose to use expert demonstrations during online RL to guide the agent along the expert's trajectories. For this, they propose two auxiliary classification tasks for learning state representations based on observation-only demonstrations.
3

---

Published as a conference paper at ICLR 2024
3 BACKGROUND
3.1 REINFORCEMENT LEARNING
Throughout this paper, we consider RL under partial observability, using the framework of the partially-obseraabee Markov decision process (POMDP, Aström, 1965; Kaelbling al., 1998). A POMDP consists of tuple (S, A,,A, R. O, P, where S is the state space, A is the action space, O is the observation space, and y is the discount factor. At each timestep t, the RL agent receives an observation or derived from the state St. according to the observation function S S + O expected discounted
and takes an action according to its policy [ (or), ) in order to maximize return, 2k=t "k- The environment then transitions to the next state S1+11 according to the tran- sition function, T S x A H S, and agent receives a reward 7, based on the reward function R 8 x A x S + IR. This work considers first learning policy from offline demonstrations, followed by further fine-tuning the policy online as the agent interacts with its environment.
3.2 LEARNING FROM OBSERVATIONS
Often, we have access recordings of performant policy, e.g. human expert, performing task of interest. When the dataset includes the action labels for transitions, a supervised learning approach called behavior cloning (BC, Pomerleau, 1988) can be used to train an RL policc to directly imitate the expert. Consider a dataset D of such expert trajectories within the environment, where
each trajectory т consists of a list of all transition tuples (oo, ag. 01),.... [0)+1- "/ 0/z)) in a complete episode within the environment. BC then trains policy TaC to imitate the expert by
minimizing the cross-entropy loss between the policy 's action distribution and the expert's 's action a"
for each observation in D: LBc = [D] ΣTED Elrl log()(a))o:)).
Unfortunately, most demonstration data in the wild, e.g. videos, do not contain action labels. In this case, the demonstration data simply consists continuous stream observations taking the form (oo, .I.............................. 01=1). This setting, sometimes called imitation learning from observations (IfO, Torabi et al., 2019a), poses a more difficult (and practical) challenge. IfO methods often seek to learn a model that predicts the missing action labels, typically trained on separate dataset with ground- truth actions. Once labeled this way, the previously action-free dataset can then be used for BC. In this work, we likewise convert the IfO problem into the BC problem. However, instead of relying on access to a dataset with ground-truth action labels, our method directly infers latent actions 2, that explain each observed transition (oe, 0t41), with which we train latent action policies, मे (22
3.3 DYNAMICS MODELS
LAPO employs two kinds of dynamics models: The first is the inverse dynamics model, Pael(aroo; Oz, 0t-1), which predicts which action a, was taken by the agent between consecutive obser- vations o, and 01+1- An IDM can be used to label sequence of observations with the corresponding sequence of actions. The second is the forward dynamics model, PFDM 00+1|os, az), which predicts the next observation O++-1 given the previous observation o, and the action de taken by the agent after observing Og. The FDM can be used as an approximation of the environment' 's transition function. In this work, the IDM and FDM are deterministic and implemented deep neural networks. Unlike a standard IDM, the IDM used by LAPO predicts continuous latent actions, E Z, where 2 = RR.
3.4 VECTOR-QUANTIRR QUAATIZATNNIN
Vector-quantization (Gray, 1984, VQ) a method for learning discrete features by quantizing an underlying continuous representation. This has been shown particularly useful in deep learning, where VQ enables learning discrete representations while allowing gradients to flow through the quantization step. VQ has been used effectively in many domains including vision (van den Oord et al., 2017; Ramesh et al., 2021b; Huh et al., 2023), audio (Dhariwal et al., 2020; Zeghidour et al., 2022), and model-based RL (Micheli et al., 2023). To quantize continuous vector VQ maintains a codebook {c), C .., J and maps z to the closest codebook vector c, The straight- -throuh gradient estimator used pass gradients through the quantization step (Bengio et al., 2013).
4

---

Published as a conference paper at ICLR 2024
4 LATENT ACTION POLICIES
4.1 LEARNING A LATENT ACTION REPRESENTATION
We now describe our approach for learning a latent IDM via forward dynamics modelling objective (see Fig. 2). First, to learn about the action at time i, we sample a sequence of observations or- ,t, Oz++1 from our observation-only dataset. Here, on and 0++1 are the observations before and after the action of inter- additional con- text controlled by hyperparameter & 2 O. The IDM then predicts the latent action 2 based on the full sequence of observations.
FDM
1-1
1 2
est is taken and observations Ot-k:-.. ................................ Ot-1
IDM
PPDM P-iaa-lot-kk......... ot, 06-1)
O
Next, the FDM predicts the post-transition observation o++1 based only the past observations Oz-k:-r-; Or and the latent action 2z.
1-1
Figure 2: LAPO architecture. Both IDM and FDM observe or, but only the IDM observes Ot-1. To en- able accurate predictions of oc+1, the IDM must pass useful transi- tion information through the quan- tized information bottleneck 2, to the FDM.
ōc+1 ~ p Or, 2x)
Both models are trained jointly via gradient descent to minimize the next state prediction error [[x++1 0s+1ll?.
This method, as described so far, would likely result in the IDM learning to simply copy 0++1 into at, and the FDM learning to forward म5 is. To remedy this, we make the latent action information bottleneck. This forces the IDM to compress any information to be passed to the FDM. Since both the FDM and IDM have access to past observations but only the IDM observes the post- transition observation o++1, the IDM is learns to encode only the difference between 0++1 and into 2,, rather than full information about Ot+1- Naturally, one highly efficient encoding of the differences of two consecutive observations, at least in deterministic environments, is simply the agent's true action. Our hypothesis is thus, that forcing the IDM to heavily compress transition information as described, may allow us to learn a latent action representation with a structure closely corresponding to the true action space.
As both the IDM and FDM have access past observations, the learned latent actions may become conditional on these observations. Intuitively, this means that some latent action z could correspond to different true actions when performed in different states. While this is not necessarily biasing the method toward simpler representations is likely preferrable (Solmonoff, 1964; Schmid- huber, 1997: Hutter, 2003). Consequently, we apply vector quantization (VQ) each latent action before passing it to the FDM, thus forcing the IDM to reuse the limited number of discrete latents across different parts of the state-space, leading disentangled representations.
issue,
4.2 BEHAVIOR CLONING LATENT ACTION POLICY
Using the trained latent IDM, we now seek to obtain policy. For this, we initialize a latent policy :: O - and perform behavior cloning on the same observation-only dataset that the IDM was trained on, with the required action labels generated by the IDM. This is done via gradient descent with respect to policy parameters on the loss (o+( - =e|22 where 21 ~ PIDM(-lO
Ot, o(+1).
4.3 DECODING LATENT ACTIONS
The policy æ,, obtained via BC, produces actions in the latent action space. We now consider different ways to adapt to produce outputs in the true action space, depending on the kinds of data available.
Small -etiee-labeled dataset. If a small dataset of true-action-labeled transitions is available, we can train an action decoder, d: Z + A, to map the IDM-predicted latent action for each transition to the known ground-truth action. The trained decoder can then be composed with the frozen latent policy and the resulting decoded latent policy, d O A, can be deployed online.
5

---

Published as a conference paper at ICLR 2024
Online environment. By interacting with the online environment get both action-labeled tran- sitions, which can be used to train a decoder in a supervised manner as described above, and a reward signal, which can be used for online RL. By doing both simultaneously, crude, initial decoder via supervised learning, and then fine-tune the entire composed policy o a by directly optimizing for the environment's reward function via RI. Through RI., the performance of the policy can be potentially be improved beyond that of the data-generating policy. In particular, when the data was generated by a mixture of policies of different skill levels or pursuing different goals (as is the case with data from the web), this step can extract the data mata-generating policies more closely aligned with the reward function used for fine-tuning.
quickly bootstrap
EXPERIMENTAL SETTING
Our experiments center on the Procgen Benchmark (Cobbe et al., 2020), as it features a wide variety of tasks that present different challenges for our method. Compared to other classic benchmarks such as Atari (Bellemare al., 2013), the procedural generation underlying Proegen introduces a difficult generalization problem and results in greater visual complexity which makes dynamics modeling at pixel-level challenging. Moreover, several Proegen environments feature partial observability, which along with stochasticity, presents issues for methods attempting to infer actions purely from observation by making it ambiguous which parts of an environment transition are due to the agent and which are due to stochastic effects unobserved information. Our observation-only dataset consists of approximately 8M frames sampled from an expert policy that was trained with PPO for 50M frames. The use of this synthetic dataset, rather than video data from the web, allows to better evaluate our method, as we can directly access the expert policy's 's true actions, as well as metadata such as episodic returns for evaluation purposes.
We use the IMPALA-CNN (Espeholt et al., 2018) to implement both our policy and IDM with 4x channel multiplier as used by Cobbe al. (2020), and U-Net (Ronneberger al., 2015) based on a ResNet backbone (He et al., 2016) with approximately 8M parameters for the FDM. The latent action decoder is fully-comnected network with hidden sizes (128. 128). We use an EMA-based update (Polyak & Juditsky, 1992) for the vector quantization embedding. We use single observa- tion of additional pre-transition context, i.e. = 1. When decoding the latent policy in the online environment, we have two decoder heads on top of the latent policy whose logits are averaged per action before applying softmax. One head is trained in a supervised manner on (a,, 22) tuples from historical data, the other is trained via Proximal Policy Optimization (PPO, Schulman et al., 2017). We keep all convolutional layers frozen and found that a much larger learning rate of 0.01 can be stably used when only training these final few layers. We similarly tuned the learning rate for train- ing the full network from scratch, but found no improvement compared to the original value of 5e-4. Other hyperparameters are given in Appendix A.4.
RESULTS AND DISCUSSION
6.1 DECODING THE LATENT POLICY ONLINE
We now discuss our results for decoding the latent policy in the online environment. Recall, that this is the third stage of our approach, after first training a latent IDM, followed by obtaining a latent policy via behavior cloning. In this experiment, we compare how decoding the latent policy (via supervised learning and RL) compares to learning a policy from scratch purely from environment interaction via PPO. Since our work focuses the setting where large amounts of observation-only data are freely available, but environment interaction is limited or expensive, we run PPO for 4M steps rather than the 25M steps suggested by Cobbe et al. (2020). Experiments are performed sepa- rately per environment. As can be seen in Figure 3, using LAPO's latent policy we are able to fully recover expert performance within only 4M frames, while PPO from scratch reaches only 44% of expert performance in the same period. We highlight that in 9 of 16 tasks, our approach exceeds the performance of the expert (which was trained for 25M frames) within only 4M frames. We also pro- vide results for two ablations. The "RL, no-SL" ablation trains the decoder only using reinforcement learning. In this setting. LAPO can be viewed as an RL-pretraining step that produces an initial pol- icy representation that is useful for downstream RL. The "RL, no-SL, no- VQ" ablation additionally removes vector- vector-quantion quantization, showing that VQ is an important component of our method.

---

Published as a conference paper at ICLR 2024
bightsh
bosstight
cavetyer
Expert polkyy
LAPO lalen
Leing
10
LPO sdert palicy
Climber
codgeball
frubbot
16
0.8-
0.8-
helet
jumpeer
leaper
maze
0.4-
miner
plunder
sappk
IM
Onllnne cnvironert stops
Online environmen staps
Figure 3: Left: Mean episodic returns (over the course of training) for decoding LAPO's latent policy and PPO from scratch (averaged across seeds). Right: Mean test returns relative to per-environment expert policies averaged across all 16 Proegen environments. Error bars indicate standard deviation across seeds.
primary baseline, but found that its policy immediately collapses in several Procgen tasks. When does not collapse, online decoding of the ILPO policy did not perform better than PPO from scratch. While Edwards et al. (2019) do provide positive results for Co nuu (which we reproduce Appendix A.5), their experiments target only individual levels of this single task. We thus hypothesize that ILPO's policy collapse is in part due the modeling challenge posed by the visually diverse levels in the full, procedurally-generaa than policy, plus a high-dimensional, continuous latent space, LAPO can capture stochasticity and unobserved information (such as off-screen parts of the game). In contrast, it is likely difficult for ILPO model the full distribution of possible transitions using only a few discrete latents.
We use ILPO
setting. By using an IDM rather
6.2 DECODING THE LATENT POLICY OFFLINE
Next, consider training latent action decoder fully-offline using only a small action-labeled dataset and no access the online environment. The trained decoder is then composed with the la- tent policy to map the policy's actions into the true action space. As we will see in Section 6.3, the latent and true action spaces share similar struc- ture. This similarity allows an effective decoder to be trained on only a miniscule amount of action- labeled data. Figure 4 shows that decoder trained on less than 256 labeled transitions matches the performance of policy trained from scratch for 4M steps. However, we observe that with increas- ing dataset size, performance eventually plateaus below the level of online decoder training. This is likely because, as previously noted, latent actions are not necessarily state-invariant. A decoder that latent action, but the state
Export pollty
PPO LAPO latent policy
PPO from LAPO istent policy (RL, no no-SL)
PPO from scresch
Latent policy head (oftinne
Relative episodie return
-0.2
16
32
64
128
256
512
1k
26
4k
8c
only has access in which that action is being taken, may not always be able to successfully decode it. We provide per- environment results in Appendix A.
Number action-labeled fremes for decoder treining
Figure 4: Offline decoding performance vs. la- beled transitions (Mean and std across 3 seeds).
7

---

Published as a conference paper at ICLR 2024
bigfish
boasfight
cavertyer
climber
dodgeball
fruitbot
heist
jumper
leaper
$;
ninja
plunder
starpilot
P - W A
Figure 5: UMAP projection of the learned latent action space for all 16 proegen games. Each point represents the continuous (pre-quantization) latent action generated by the IDM for transition in the observation-only dataset. Each point color-coded by the true action taken by the agent that transition (true action labels only for visualization, not used for training). Arrows in legend correspond movement directions. NOOP actions are omitted for clarity.
6.3 INSPECTING THE LATENT ACTION SPACE
To better understand what LAPO learns, we generate UMAP projection (McInnes et al., 2018) of the latent action space for each of the 16 Procgen environments, shown Figure 5. For most games, the structure of the latent action space is highly interpretable, with distinct clusters of latent actions closely corresponding to the true discrete actions. We also observe that the structure of the latent space varies greatly across environments. In BigFiss if,sh, Chaser, Dodgeba 1 1, Ma ze, Miner, P Plunder, and StarPilos latent actions aggregate together in well-defined clusters aligned with the true actions. In other environments, including C1 Climber, Coin.un. Frui Bot, Jumper r and Ninja the latent action space exhibits more fragmented structure. This split roughly corresponds to environments with higher lower degrees of partial observability. In the latter group of environments, the pixel observation shows only part of the level, cropped locally to the player character. Thus, when the player is moving, most observations will show parts of the level that were
8

---

Published as a conference paper at ICLR 2024
hidden in previous observations. In order for the FDM to accurately predict the next observation, the IDM needs to encode this off-screen information into the latent action, which we hypothesize leads to more complex structures in the latent space. However, demonstrated in Section 6.1, the latent-space policy still performs well downstream even on environments with greater degrees of partial observability. Moreso, we find that vector quantization has large impact in simplifying the structure of the latent action space. We generate the same UMAP projection for an ablation of our method without vector quantization Appendix A.2. We note that for several environments, including those with and without partial observability, latent action clusters corresponding to distinct true actions are at times duplicated. For example, BigFi seFiist sh, each true action sees four latent action stusters-likel) reflecting how taking the same true action in different parts of the state-space can have differing effects. These results strongly support our hypothesis that vector quantization leads to simpler latent representations with less fragmentation within actions.
6.4 LIMITATIONS
A few factors can adversely impact the performance of LAPO. First, actions that have a delayed ef- feet in observations will be predicted to take place with the same delay, i.e. the latent policy actually models the visible effects of an action, not the action itself. Nevertheless, in most environments, actions that have any impact on the state of the environment will elicit some degree of immediate change in the observation. Moreover, delayed actions can be partially addressed by extending the IDM and FDM architecture to consider multiple timesteps into the past and future, e.g. by using a Transformer-bon based architecture (Vaswani et al., 2017). Second, significant stochasticity can make difficult for the IDM to compress the useful bits of information among the noise, degrading the quality of the latent representation. This issue can potentially be mitigated by training much larger datasets. Lastly, training on much larger datasets as would be required for modeling more complex domains like web-scale video introduces new challenges in balancing the strength of the FDM and the capacity of latent actions representations, as is often the case in autoencoding architectures (Chen et al., 2016).
require scaling up the model architecture, which
CONCLUSION
This work introduced LAPO, method for training policies over learned latent action space, in- ferred from purely observational data. Unlike prior work on imitation learning from observation, LAPO does not rely on access to the true action space or a predefined set of discrete latent actions to learn a useful, pretrained policy. Instead, LAPO learns a latent action space end-to-end, by opti- mizing an unsupervised objective based on predictive consistency between an inverse and a forward dynamics model. Vector quantization of the continuous latent actions induces an information bot- tleneck that forces the quantized actions to capture state-invariant transition information. Across all 16 games of the challenging Procgen Benchmark, we show that this approach can learn latent action spaces that reflect the structure of the true action spaces, despite LAPO never having access to the latter. We then demonstrate that latent action policies, obtained through behavior cloning of latent actions, can serve as useful pretrained models that can be rapidly adapted to recover or even exceed the performance of the expert that generated the original action-free dataset.
Our results thus suggest that LAPO can serve as a useful approach for obtaining pretrained policies from web-scale action-free demonstration data, e.g. in the form of videos. We believe LAPO could serve as an important step in unlocking the web-scale unsupervised pretraining paradigm that has proven effective in other domains like language and vision (Brown et al., 2020; Radford et al., 2021; Ramesh et al., 2021a). Toward this goal, we are excited about future work aimed scaling up the world model and policy to more powerful architectures that can consider multiple timesteps and richer observations. Such models may enable efficient learning of general- ist latent action policies in complex, multi-task settings, all within a single LAPO instance, while enabling more effective generalization to tasks unseen during training.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Karl Johan Aström. Optimal control of Markov processes with incomplete state information. Jour- nal of Mathematical Analysis and Applications, 10(1):174-22) 205, 1965.
Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang. and Nando de Fre- itas. Playing hard exploration games by watching youtube. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Desa-Bianchi, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Mon- tréal, Canada, pp. 55509926eh4ee3366c22233322222233 Absrrcctttbtttttttmm..m
2018. URL
8//has//
Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): learning to act by watching unlabeled online videos. In NeurlPS, 2022. URL http://paapers.miss.com wwwsscc/paapprrrcccccppp/ htwpppppeeeillsspppr/0/000 -22ada0-Absttactttt bistract-Conferenee.tttl
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Arlificial Intelligence Research, 47: 253-279, 2013.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3433 34332, 2013. URL http: //arxiv.org/abs/13000 30tps///330333333333 33432.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada. October 10- 17, 2021, pp. 9630-9640. IEEE, 2021. doi: 10.1109/CCCV48922..00 100911. URL https://doi. org/10.1 109//CCV48922.201.11
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. International Conference on Learning Representations, 2016.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. International Conference on Machine Learning, pp. 1282-1289 1229. PMLR, 2019,
Karl Cobbe. Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 2048 2056. PMLR, 2020. URL http://oeeedinns....pr.ss.. v1 1119/bbbe20a.html..
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. CoRR, abs/2005.00341, 2020. URL htts://arxiv.org/
abs/2005.00341.
Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, and Charles L. Isbell Jr. Imitating latent policies from observation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1755- 1763. PMLR, 2019. URL http://proceedingg. htt.ig/s.mmmress//////waarr.. h9a.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: scalable distributed deep-rl with importance weighted actor-leamer architectures. In Jen- nifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on
10

---

Published as a conference paper at ICLR 2024
Machine Learning. ICML 2018, Stockholmssässaan, Stockholm, Sweden, July 10-15, 2018, vol- ume 80 of Proceedings of Machine Learning Research, pp. 1406-1415. PMLR, 2018. URL http://pwoeccinnng....ppss/////s Sa..se 8a.html.
Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 11321- 11321-11333 11339. PMIR, 2023. URL. http:://prroeedding..rrrrrperee ghosh22a.html.
R. Gray. Vector quantization. IEEE ASSP Magazine, 1(2):4-99, 1984. doi: 10.1109//ASSP11984 1162229.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight- through estimator: Overcoming optimization challenges in vector quantized networks. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 14096- 14113. PMLR, 2023. URL https:/pproceedigg..nm www...mm..p........... ww..gsss//..p../// 200ess//2222//h2222.ammmm
Marcus Hutter. On the existence and convergence of computable universal priors. In International Conference on Algorithmic Learning Theory, pp. 292-312. Springer, 2003.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Plamning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-13, 1998.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 13:1179-1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems, arXiv preprint arXiv:2005.01663 2020.
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: /802.0342, 2018.
Willi Menapace, Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video generation. In IEEE Conference Computer Vision and Pattern Recogni- fion, CVPR 2021, virtual, June 19-25, 2021, pp. 10061-10070. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437 CVPR2021/hum//Meappcc. Playable Video_ SeatoeGeeeeeeennCeCee Cennrrain__CVRR_DDDpppppo.....
URL
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representationss, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URI. https://opemeeeeww.tee///// vcbolxc
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4):838-88)) -5585588888 1992.
Dean Pomerleau. ALVINN: autonomous land vehicle in neural network. In David S. Touretzky (ed.), Advances in Neural Information Processing Systems 1, [NIPS Conference, Denver, Col- orado, USA, 1988], pp. 305-313. Morgan Kaufmann, 1988.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
11

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-imme text-to-image generation. In International Conference on Machine Learning. pp. 3921-8831. PMLR, 2021a.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8821-883) PMLR, 2021b. URL http://procceedings.ll:: press// 33/rramss22 1a a.html.
Scott E. Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth- Marth,Maron, Mai Gimenez, Yury Sulsky, Jackie Kay. Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Trans. Mach. Learn. Res., 2022, 2022. URL
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Conputer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234 231.241.5 Springer, 2015.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668. JMLR Workshop and Conference Proceedings, 2010.
Karl Schmeckpeper, Oleh Rybkin, Kostas Danillidis, Sergey Levine, and Chelsea Finn. Reinforce- ment learning with videos: Combining offline observations with interaction. In Jens Kober, Fabio Ramos, and Claire J. Tomlin (eds.), 4th Conference on Robot Learning, CoRL 2020, 16- 18 November 2020. Virtual Event / Cambridge, MA, USA, volume 155 of Proceedings of Ma- chine Learning Research, pp. 339-354. PMLR, 2020. URL http://pnocedingg..lr..essss 155/ schmeckpeper2la..tmi
Jürgen Schmidhuber. Discovering neural nets with low kolmogorov complexity and high general- ization capability. Neural Networks, 10(5):857-873, 1997.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, /e//1707006447, 2017. URL http://axiiv.or/abbs. wwtggass//70....on 707.06347.
RJ Solmonoff. A formal theory of inductive inference. i. Il Information and Control, 7:224 1964.
Oliver Struckmeier and Ville Kyrki. Preventing mode collapse when imitating latent policies from observations, 2023. URL
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Jérôme Lang (ed.), Proceedings of the Seenth-Seventh International Joint Conference on Artificial Intel- ligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp. 4950-4957. ijcai.org, 2018. doi: 10.24963/jjcai.201 20188/877 URL http:///ooooog//... 042...93//..........0. 2018/887.
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from obser- vation. In Sarit Kraus (ed.), Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 6325-6331. ijcai.org. 2019a. doi: 10. 2244999///aai.201198888 URL http:://bo.orr///022446.ee.e...
12

---

Published as a conference paper at ICLR 2024
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation.
ICML Workshop on Imitation, Intent, and Interaction (I3), 2019b.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. December 4-9, 2017, Long Beach, CA, USA, pp. 6306 6315, 2017. URL https://poceedigg.nerrp....ppppp Abstract.html.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. Ad- vances in neural information processing systems, 32, 2019.
Weirui Ye, Yunsheng Zhang, Pieter Abbeel, and Yang Gao. Become proficient player with lim- ited data through watching pure videos. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process. 30: 495-507, 2022. doi: 10.1 1.110//AASLPP2201111999999 URL https://doi.org/10.. 109/ASSLP2021. 3129994.
Qihang Zhang. Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022
17th European Conference, Tel Aviv, Isruel, October 23-27, 2022, Proceedings, Part XXVI, volume 13686 of Lecture Notes in Computer Science pp. 111-128. Springer, 2022. doi: 10.1007/978-3-000 1-77788---01--980893
7. URL https://oooor///0.0007777-7--111
Qinging Zheng, Mikael Henaff, Brandon Amos, and Aditya Grover. Semi-supervised offline rein- forcement learning with action-free trajectories. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA volume 202 of Proceedings of Machine Learning Research, pp. 42339 42362. PMLR, 2023. URL http::/prooceeding..mil mir. http://wwwcc.......mee////////// www.reesssw22222hng22.....
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
OFFLINE DECODING RESULTS
bigfish
bossfight
caveflyer
chaser
10
6-
10
2-
climber
coinrun
dodgeball
fruitbot
8-
5-
10-
5-
Episodic return
D-
heist
jumper
leaper
maze
8-
6-
6-
4-
2-
4-
miner
ninja
plunder
starpilot
10
1.0
6-
20
0-
Number action-labeled frames for decoder training
Expert policy
PPO from LAPO latent policy (RL + SL)
PPO from LAPO lanent policy (RL, no-SL.) PPO from scratch
LAPO latent policy decoder (oMtine)
Figure 6: Test performance of the latent policy combined with a latent action decoder that is trained on an action-labelled offline dataset of certain size, consisting of (zs, a.) tuples. An effective decoder can be trained with only a few hundred samples although performance generally plateaus before reaching the performance of our proposed online (RL) decoding approach.
14

---

Published as a conference paper at ICLR 2024
A.2
LATENT SPACE ANALYSIS FOR NO- VQ ABLATION
biglish
bossfight
cavellyer
chaser
climber
coinrun
dodgeball
Truibbot
heist
jumper
leaper
rlss
miner
plunder
starpilot
5
Figure 7: UMAP projection of the learned latent action space for all 16 procgen games, generated by IDMs trained without vector-quantization. Each point represents the continuous latent action generated by the IDM for a transition the observation-only dataset, Each point is color-coded by the true action taken by the agent at that transition (true action labels are only used for visualization, not for training). Arrows in the legend correspond to movement directions. NOOF actions omitted for clarity.
We note that although the No-VQ ablation performs significantly lower. This indicates that VQ indeed acts a bottleneck that constrains the amount of information passed from the IDM to the FDM. By forcing the IDM to compress transition information, this leads to better latent representation for downstream policy learning.
in terms of online performance, its FDM loss
15

---

Published as a conference paper at ICLR 2024
A.3
CONTINUOUS CONTROL EXPERIMENTS
cheetah_run
walker walk
40-
100
35
00
Episodic return
25-
20
15-
60
10-
0.2M
0.4M
0.8M
0.2M
0.4M
0.6M
0.0M
Online environment steps
policy
PPO from scrstch
Figure 8: Mean returns the course of training for online decoding of a LAPO latent policy compared to PPO from scratch (averaged across 2 seeds) on continuous control tasks. These experiments used the same hyperparameters for training the latent IDM and for behavior cloning as in Procgen experiments.
16

---

Published as a conference paper at ICLR 2024
A.4 HYPERPARAMETERS
Table I: Hyperparameters the three stages our method. Latent actions are 128-dimensional continuous vectors and are split and quantized into discrete latents with 16-dimensional embeddings. We use Procgen distribution mode easy.
Stage
Parameter
Value
Pre-transition additional context &t
VQ # of codebooks
2
VQ # of discrete latents per codebook
VQ # of embeddings
64
VQ embedding dimension
16
Latent IDM training
VQ commitment cost
0.05
VQ EMA decay
0.999
Learning rate
2e-4
Batch size
128
Total update steps
70,000
IMPALA-CNN channel multiplier
Learning rate
2c-4
Batch size
128
Latent behavior cloning
Total update steps
60,000
IMPALA-CNN channel multiplier
4
Total environment interactions
4,000,000
RL online fine-tuning
PPO hyperparameters
As in (Cobbe al., 2020).
A.5 ILPO REPRODUCTION
Export policy
ILPO discrete latent policy decoder
Episodic
2-
T
# action-labeled frames for decoder training
Figure 9: Reproduction of Figure S.a from Edwards al. (2019) for single level from coinrun. When applied to data from the distribution of levels, rather than just single level, ILPO FDM consistently collapsed in terms which discrete latent achieves the minimum of Lmin leading to collapse of the policy too. Results here are not exactly comparable to results from Edwards et al. (2019) since it is unknown to us which specific coinrun level was used by the authors.
17


Reviews:
[{"review_id": "94slfgU9nv", "reviewer": "Reviewer_ATYE", "summary": "The paper proposes an approach (LAPO) to learn latent-action policies from action-free demonstrations. The approach involves training a inverse dynamics model (IDM) along with a Forward Dynamics Model (FDM) using an unsupervised objective. The IDM learns to predict the latent action given the past and future observation. FDM on the other hand uses the predicted latent action, and past action to predict the future observation. Training these two models together for predictive consistency helps learn latent actions that can reliably explain observed transitions. To prevent the model from compressing all of past observation’s information into the latent action, the authors propose using vector quantization which acts as an information bottleneck. Finally, the authors show that latent actions can be mapped to real actions using online RL policy fine-tuning, or by using an offline dataset of action-labeled transitions. The authors show results on the Procgen benchmark. They show that the approach exceeds expert demonstrations by fine-tuning a policy pretrained with LAPO using significantly fewer number of steps. The authors also show through many visualisations that the structure of the latent space is highly interpretable", "questions": "Please see weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Learning policies from unlabeled (lacking action annotations) demonstrations is an important problem because it allows to leverage large collection of videos available on the web. The proposed approach shows how we can leverage this data to train policies in an unsupervised way, even when no labeled demonstrations are available. \n2. I really appreciate the UMAP projection visualisations in the paper! The visualisations clearly demonstrate the effectiveness of the approach to learn disentangled latent space for actions, that can be meaningfully mapped to real actions corresponding to the transitions. Comparing Figure 5 and Figure 7, also clearly demonstrate the usefulness of Vector Quantization in their method. \n3. The paper is well-written! I specially appreciate a well-written related works section that did a great job putting the paper in context with other related works!", "weaknesses": "### Weaknesses\n\n1. The experiments assume that the underlying action space in the videos used for pre-training is the same as the action space of the agent. In other words, the videos used during pre-training are of the same agent. I think this assumption is limiting. To leverage large-scale video data available on the internet, one of the key ingredients is learning from videos that doesn’t necessarily match the action space of the agent. Imagine learning from Ego4D like ego-centric observations, and using it to execute tabletop rearrangement tasks. Or using RealEstate10K videos to learn how to navigate in indoor environments. \n2. Secondly, it would have also been nice to consider experiments on continuous control tasks. The approach uses a vector quantised latent space to model actions. Will such an approach also work for actions that are continuous control? \n3. Finally, the paper doesn’t fully justify using vector quanitization approach to learn disentagled latent actions. While the empirical results do show the efficacy of this approach, I don’t fully grasp why vector quantization based information bottleneck works. Additionally, did the author consider an approach like maximizing the conditional mutual information I(o_t+1 | a_t) while minimizing  I(o_t+1 | o_t)  (as in  Deep Variational Information Bottleneck, Alemi et al, 2016) to learn a latent action space that is consistent with the transition observed without over-relying on the past observations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G1RFnYjK6r", "reviewer": "Reviewer_Z13u", "summary": "In this paper, the authors focus on the problem of inferring and leveraging latent actions to train RL models from unlabeled demonstrations. To this end, they propose Latent Action Policies from Observation, where an Inverse Dynamics Model infers an aggressively bottlenecked latent from two observations adjacent in time. These latent actions can then be used to train a behavioral cloning model. They evaluate their approach on the Procgen dataset, show large performance margins over baselines, and analyse the learned action space visually through UMAP projection.", "questions": "1. Do the results of LAPO generalize to other environments, especially in continuous control settings? (such as MoJoCo, DMControl, Meta-World, etc.)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unsupervised learning has been successfully applied to domains such as language understanding and computer vision, but this is still a frontier problem in the reinforcment learning community. This paper is a promising step in that direction.\n\n2. The approach is simple and makes intuitive sense. A latent action could well be considered the \"difference\" between two observations.\n\n3. Experimental results are strong, and the analysis of learned latent space is insightful.", "weaknesses": "1. The approach is only evaluated in one environment, Procgen. It is unclear if this approach will generalize across different domains (i.e. continuous control/Robotics). Additional experiments in other environments could strengthen the case of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YLZWhF6Smh", "reviewer": "Reviewer_BaGY", "summary": "In this paper, authors present a method to learn from pure observation-only data (e.g., videos demonstrating a control task without explicit action labels). The proposed algorithm LAPO aims to infer actions taking place between two consecutive observations of such data. LAPO models latent representations for the actions instead of trying to infer the true ground-truth action. It does so using two components -- inverse dynamics model (IDM) and forward dynamics model (FDM). The IDM's task is to generate a latent representation of the action that takes place between a given history of observations (including current observation) and the next observation. Consequently, the FDM's task is to generate the next observation given the current observation and a latent representation generated by IDM. Both these components are trained in conjugation to learn latent representations of unknown actions. These representations are further finetuned using true actions via either online rollout or matching actions in an offline dataset. The paper presents results on 16 discrete-action ProcGen environments.", "questions": "In the context of the aforementioned weaknesses, I have the following questions:\n\n1. How can we advantageously use the LAPO latent representations for large-scale pretraining on unlabelled videos of control tasks?\n2. Do authors observe that NOOP IDM representation does not affect FDM's transition? \n3. Do NOOP IDM representations get clustered similarly to other actions? \n4. How does NOOP compare with actions with delayed effects?\n5. How will VQ-based LAPO fare in continuous action environments? \n6. What are the possible extensions to the current architecture that will allow us to use LAPO seamlessly on real-world continuous action control tasks?\n\nGiven these questions, I am presently inclined to borderline reject the work. But with clarifications provided, I would definitely like to increase the score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "I find the following strengths of the paper:\n\nThe paper does a superb job in terms of the writing and the clarity of the presentation. The algorithm design seems logical from the description. The UMAP plots on 16 different environments are very well-organized and exciting to go through. They convincingly substantiate the algorithm's usefulness in inferring unknown actions.\n\nThe problem statement of the work, inferring actions from observation-only data, is quite relevant for current RL research where there is a need to learn control aspects from unlabelled video demonstrations.", "weaknesses": "I find the following weaknesses in the paper:\n1. The utility of learned latent representations for large-scale pretraining: It is evident from the plots that the actions are meaningfully clustered. However, if one has enough compute, to me, it seems plausible to train a transformer autoregressively to generate the next observations in observation-only data, ensuring that the latent representation for actions is also learned. I raise this point because if such a pretraining is possible directly with transformers, the utility of LAPO reduces. Plus, if someone wants to apply the LAPO (IDM + FDM) approach, it could be inflexible compared to a transformer extension. Anyway, I do acknowledge that LAPO does provide evidence that it is possible to meaningfully identify actions in observation-only data.\n\n2. Issue with NOOP: From the IDM-FDM perspective, the NOOP being a null action, its latent representation should produce no transition when sent to FDM. It was unclear from the writing if there is any experimental validation of the same. Also, there is an issue with NOOP and actions with delayed effects. The present IDM-FDM model will fall short in modeling the two separately. (Authors briefly touch up on modeling the delayed effect actions, but I did not find their mention contrasting them with NOOP's.)\n\n3. LAPO with continuous action environments: The current implementation of LAPO involves using vector quantization (VQ). However, using VQ would, in principle, limit the actions to be chosen from a finite set of discrete codes. This, in turn, creates issues while scaling to continuous action environments. The ProcGen games, environments used in the experiments, are discrete action environments, too, and I find it difficult to see how the approach would scale to non-discrete real-world control tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LuNABew8Zd", "reviewer": "Reviewer_NDab", "summary": "This paper proposes a novel method to pretrain for RL with only observation sequences without action labels. It first infers latent actions, then learns a policy by behavior cloning on the inferred latent actions, and subsequently refining the policy via RL. What sets this research apart from previous studies is its unique strategy in latent action inference, which features a VQ latent action space and a combined optimization target for IDM and FDM.  Experiments on Procgen demonstrate the effectiveness of the proposed method.", "questions": "1. Do you think this method can generalize to domains with continuous action space?\n2. Does the number of discrete tokens in VQ matter? I would appreciate a ablation on this.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well written. Motivation, methodology, and experiments are presented clearly. The idea of optimizing forward modelling objectives using a constrained action latent space is coherent and logical. The provided UMAP projections greatly support the effectiveness of the proposed method.", "weaknesses": "1. Toy experimental setup: The paper motivates from learning (pretraining) with large scale web data. However, during experiments, the author uses the data from a simulator (Procgen). The pretrain data contains less visual complexity compared to real world data. I encourage author to explore pretraining with real world data (like MC YouTube videos [1] and first-view driving videos [2]). Or, the authors should at least make the pretraining data different from the RL observation data to simulate the difference between pretrain and finetune scenario. One way could be using different backgrounds in Procgen.\n2. In my opinoin, this paper could benefit from including references to a relevant work [2] which uses IDM to label the dataset and conduct contrastive learning for policy pretraining. \n3. I think this paper could benefit from additional comparison with [1] and [2]. It could provide additional insight of how methods that use extra data for IDM training perform compared to the proposed method. I am particularly interested in understanding whether training IDM  yields performance improvements or if the proposed method (without IDM) is already capable of achieving comparable, or even superior, results to the aforementioned prior works [1][2].\n\n[1] Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. https://github.com/openai/Video-Pre-Training\n\n[2] Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. https://github.com/metadriverse/ACO", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)