What Do Latent Action Models Actually Learn?
Authors: Chuheng Zhang
Tim Pearce
Pushi Zhang
Kaixin Wang
Xiaoyu Chen
Wei Shen
Li Zhao
Jiang Bian
Venue: NeurIPS 2025 poster ()
Keywords: latent action model
embodied ai
Primary area: deep_learning
TL;DR: We provide a theretical analysis on a linear version of LAM, which is recently popularized in embodied AI.

Abstract: Latent action models (LAMs) aim to learn action-relevant changes from unlabeled videos by compressing changes between frames as latents. However, differences between video frames can be caused by \textit{controllable changes} as well as exogenous noise, leading to an important concern -- do latents capture the changes caused by actions or irrelevant noise? This paper studies this issue analytically, presenting a linear model that encapsulates the essence of LAM learning, while being tractable. This provides several insights, including connections between LAM and principal component analysis (PCA), desiderata of the data-generating policy, and justification of strategies to encourage learning controllable changes using data augmentation, data cleaning, and auxiliary action-prediction. We also provide illustrative results based on numerical simulation, shedding light on the specific structure of observations, actions, and noise in data that influence LAM learning.

What Do Latent Action Models Actually Learn?
Chuheng Zhang", Tim Pearce" Pushi Zhang', Kaixin Wang', Xiaoyu Chen?,
Wei Shen', Li Jiang Bian"
Microsoft Research Tsinghua University IIndependent Researcher
'Equal contribution: thangchuhengt123011w com, timpearce@mi cow.
Abstract
Latent action models (LAMs) aim to learn action-relevant changes from unlabeled videos by compressing changes between frames latents. However, differences between video frames can be caused by controllable changes as well as exogenous noise. leading to an important concern do latents capture the changes caused by actions or irrelevant noise? This paper studies this issue analytically, present- ing a linear model that encapsulates the essence of LAM learning, while being tractable. This provides several insights, including connections between LAM and principal component analysis (PCA), desiderata of the data- generating policy, and justification of strategies to encourage learning controllable changes using data augmentation, data cleaning, and auxiliary action-prediction. These findings are validated through numerical simulations, as well as experiments in more realistic settings. This investigation is the first to rigorously investigate how the structure observations, actions, and noise influence LAM learning.
1 Introduction
Latent action models (LAMs) aim to infer controllable action changes from streams of image observations in an unsupervised manner (Rybkin al. 2018 Menapace et al. (02T). This is valuable because action-labeled data is typically expensive to source, while unlabeled videos are abundant. Hence, it offers route for embodied Al systems to learn from large unlabeled datasets, for instance using the inferred latent actions as targets for pre-training a policy, while small amount of labeled data can be used learn a mapping from latent real action controls (Ye al., 2024). This has proven effective in learning from videos of 2D video games, robotics. and even broadcast tennis
footage (Menapace et al. 2021 Schmidt and Jiang 2023 Bruce et al. 2024 Chen al. 2024b Ye
et al. 2024 Sun et al.) 2024 Cui et al. 2025 Gao et al. 2025).
The success of such LAM-based recipes relies the inferred latent action labels mapping to the real control action signals of interest. However, there is concern that this may not always be the case. For example, there is an intuition that LAM's inferred latent 'actions" simply compress differences between consecutive frames, even when control actions are not the cause of those differences (McCarthy et al. 2024). As such, LAMs may only succeed domains where the cause of changes between observations can be fully attributed to the control action. A further point of dispute whether bottleneck is required (Schmidt and Jiang 2023).
To study these issues and more, this paper conducts a theoretical analysis of a linear version of LAM. By retaining the architecture of recent LAM models, but swapping deep neural network components with simpler linear layers, we preserve the fundamental challenge of LAM training in an analytically tractable form. Our analysis of linear LAM firstly provides precise insights into when inferred latent actions capture true control signals compared to noise, as well as what information is captured by different components within LAM. Surprisingly, our analysis also reveals additional issues currently known to the LAM community related to the over parametrization property of LAM and the randomness of data- gettaagenertiinn policy. Finally, we propose and study potential solutions to ameliorate these issues data augmentation and predicting action as an auxiliary task.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Learning Act without Actions (LAPO)
Latent Action Pretraining from Vidcos (LAPA)
Token
Linear LAM (ours)
0
Genie: Generative Interactivo Environments
Ada World: Adaptive World
Models with Latent Actions
AgiBot World Go-1
IDM
FDM
Figure I: Linear LAM is an abstraction of the LAMs used in previous work. Inputting consecutive observation pairs (o, o'), the LAMs output the second observation via reconstruction loss, | o''-o'113 An information bottleneck tries to stop the direct copying of o', with the expectation the latent will correspond to the control action a. Linear LAM captures the essence of LAM training whilst being analytically tractable. The diagrams of previous LAMs are copied from their original papers:
LAPO (Schmidt and Jiang 2023), LAPA (Ye al. 2024 Moto (Chen et al. 2024c), Genie (Bruce et al. 2024), Adalo (Gao et al. 2025). and Go- (AgiBot-Worke 2025).
Concretely, this paper makes several key contributions.
1. Section presents linear LAM, a tractable model preserving the essence LAMs used in practice.
Section 4.1 shows linear LAM reduces to principal component analysis (PCA) on a mixture of controllable changes and exogenous noise, under certain assumptions. Our analysis justifies the practical use of LAM when the controllable action signals cause larger changes to observations than the exogenous noise.
3. Section 4.2 shows correlation between observations and actions decreases LAM's focus on learning controllable changes. This suggests that higher randomness in data- eetta-gnnraatng policies benefits LAM's learning.
4. Section 43 validates that performing data augmentation during LAM training can mitigate the over-parametrization issue and thus improve the semantics of the latent.
5. Section|4.4] finds that adding an action- piriiiiireiition head encourages LAM to prioritize the learning of controllable changes for the latent.
findds
6. Section verifies that the main findings based on linear LAM still hold on more realistic LAMs.
2 Related Work
The study of learning representations of real actions in reinforcement learning (RL) has a long history. For instance, PG-RA (Chandak et al., 2019) clusters actions based on the similarity of their impact on the state to improve generalization in the action space. LASER (Allshire et al. (021) learns latent actions through an encoder-decoded architecture trained reconstruct real actions, resulting in higher learning efficiency in RL. TAP (Jiang et al. 2022) learns the latent action that can help reconstruct the full trajectory (with state, action, and reward) condition on the state. EAR (Hua et al.) 2022) finds that the latent task embedding resulting from the training of multi-task policies turn out to be good action representations with a geometrically and semantically meaningful structure. AD3 (Wang al. 2024) adopts inverse dynamics model and forward dynamics model to extract latent action, similar to popular LAMs, but conditions these models on the real actions.
While the above papers learn action representations based on real actions, our paper focuses on learning latent actions without access the real action labels. Removing the need for action labels during training is advantageous as it allows leveraging internet-scal video datasets in the pre-training stage (Miech al.) 2019 Chen et al., 2024a Pei al.) 2025) Wang et al., (023) for example unlabeled demonstrations of humans completing diverse tasks. While high-quality robotic datasets with action annotations exist (Vuong et al. 2023 Fang et al. 2023 Khazatsky et al.) 2024. AggBBotWWrll 2025), they remain limited in scale. Such datasets can be integrated into the

---

semi- aumi-supervieed learning framework to extract latent actions NNiuulin et al. 2025) or the policy fine-tuning stage (Schmidt and Jiang 2023) Ye et al. 2024). An alternative approach aims to extract pre-defined actions from observations using computer vision techniques (Mendonca et al., 2023).
Beginning from Rybkin al. 2018). LAMs have featured information bottleneck or auto-encoder
to allow learning in an unsupervised manner. ILPO (Edwards et al., 2019) learns latent actions along with the training of policy that outputs the latent action and a world model that conditions on the latent action. Menapace et al. (202T) proposes a probabilistic action network that extracts a discrete action label and continuous action variability embedding from consecutive observations. This network is trained jointly with an action decoder to generate video controlled by extracted actions. While these works adopt a bottleneck in the learning of latent actions, their training losses are complicated by involving policy learning or recurrent networks. LAPO (Schmidt and Jiang 2023) proposes a I.A design with an inverse dynamies model extracting a latent action and forward dynamics model that reconstructs the next observation based on the latent action. This forms the template of the modern LAM. Many papers follow a similar architectures to LAPO with a discrete latent action space such as FICC (Ye et al., 2022), Genie (Bruce al.] 2024), LAPA YYe et al. 2024), Moto (Chen et al., 2024c), IGOR (Chen et al. 2024b), Go-1 [AgiBot-Worlu 2025), and GROOIT NI [ 2025). However, there is a debate whether discrete latent actions are better than continuous latent actions. For example, Nikulin et al. (2025) finds that using continuous latent actions with a larger bottleneck can not only result in better predictability lead to better performance for downstream policies. We consider continuous latent actions in our linear LAM analysis while vector quantized latents are tested empirically in Section [
real actions but also
In contrast to the popularity of LAMSs recent foundation models for embodied AI, issues around the objective, learnability, and robustness of LAM have received limited attention. Consequently, our paper aims to investigate these issues.
3 Setup
This section broadly introduces the problem setting, goal and model used in recent LAM work in practice. We then more formally detail these for the linear model to be used in subsequent analysis. Finally, we outline the details of our simulation setup to be used in support of our later analysis.
3.1 LAMs in Practice
Setting. The setting tackled by recent LAM work assumes access to a large dataset of pairs of observations and next observations D {(o,, of) J.N and only small subset of has action labels DA = {(o, of a.)) N, with := / << 1. Note we drop the i subscript when we do not need to refer to specific samples. In practice, o and o' could be feature vectors extracted from the original observations (Cui et al.] (025)), and o could be a stack of historical observations to account for partial observability (Bruce et 20024). The two datasets are assumed to come from the same Yc et al.
distribution in most LAM work (Menapace et al. 2021
Bruce et al. 2024
2024
Model. Recent LAM designs (Schmidt and Jiang 2023
Bruce et al. 2024 Ye et al.
2024 Chen
al. 2024b) AgiBottWorrld 2025) (see Figure ! take pair of consecutive observations (o, o') as
input, and output latent well a prediction of the next observation o'. (We subsequently avoid calling= z 'latent action' to avoid confusion with the real action.)
LAMs are typically decomposed into inverse dynamics model (IDM) and forward dynamics model (FDM), implemented as deep neural networks, and trained via a reconstruction loss,
t'DM (., o'), o' 4'DDM (o, 2), L:= min ViDe- VTDM
ED [|o' o'
where wlDu contains bottleneck so the dimension of is smaller than that of o.
Use cases. We identify two primary downstream use cases of LAMs.
The LAM latents can be used as imput to a world model T trained to generate future frames, T(o'(o,)) z) (Sun et al. 2024 Chen et al. 2024b Bruce et al. 2024 Menapace et al. 2021). These
'Certain work explores mismatch case, e.g., D is human videos and D. is robotics data Chen al. 2024b
3

---

0
+K
C
+ H
A
Rd
0 R%
0' 4 K2
0' Ed..
D
B
IDM
Rd
FDM
Figure 2: Overview of linear LAM. Grey blocks represent learnable parameter matrices, giving rise the predictive model o' Ao + B(Co + Do'). Green illustrates linear LAM with data augmentation to reduce the amount of information the latent contains about the observation (in Section 4.3 Pink illustrate linear LAM with auxiliary action prediction to encourage the latent to focus the controllable actions and suppress the noise signal (in Section 4.4).
world models can generate higher-quality frames than the FDM in LAM, but have been used only qualitatively interpret the meaning of the learned latents.
The LAM latents can be used as labels in the pre-training of latent policy Tinteat (2|o) (similar behavior cloning on actions). This policy may later be mapped real actions, maap (a ) (Bruce et al., 20244; Schmidt and Jiang, 2023). 3, or be followed by a fine-tuning phase on real actions (Schmidt and Jiang 2023 Ye et al. 2024 Chen al. 2024b).
In both cases, is hoped that the learned latents can be aligned with the true actions closely as possible. As in Schmidt and Jiang (2023), "our hypothesis is ... [this] may allow us to learn latent action representation with structure closely corresponding to the true action space".
3.2 Linear LAM
Setting, We conduct analysis in the controlled Markov process (CMP) framework (Puterman, 2014) with (O, A, T). At given timestep, the agent receives observation O and takes action A. The transition function describes the probability distribution over the next observation, Too o, a))
Rdd (which could be thought of as image observations processed with pre-trained image encoder as in Cui et al. (2025); Chen et al. (2024b)). The action space A Rd has lower dimensionality than O, i.c., du < d. These actions are mapped to create controllable changes in observation space via action effect matrix X xd.,d.. q Xa.
consider vector observations,
We generally assume linear LAM cannot access the action a during the training, but will make use of it to evaluate the learned LAM in simulations. We choose the transition function to be an additive combination of controllable changes q E Rdd (the state changes caused by the ego agent's actions) and exogenous noise Rd (representing environmental stochasticity other agent's actions), i.e. with the next state generated via, o' + q +
Model. For linear LAM, the IDM and FDM consist of linear mappings. Summarized in Figure 2 the IDM and FDM are given by,
= Lincar (o, 0') = Co Do'
(2)
(0,z) = Ao + Bz = Ao B(Co DD'), ò' = FDM
I Linear
(3)
where Id with d & do is the latent. All matrices are learnable parameters, including FDM parameters A E Rdoxdd B Rdoxd, and IDM parameters C, D E Rd..
As for practical LAM (Eq. [. the linear LAM is trained via reconstruction loss,
B, D) E [i0ō] [[
Appendix A summarizes the gap between linear and practical LAM.
2 Discrete actions can be represented one-ho hot vectors within de While most of analysis (except that on bottleneck) should also apply discrete actions, confine analysis in continuous actions.

---

Goal. Following our discussion of use-cases of LAM cases in Section 3.1 we interpret the desired latents contain as much information as possible about the real action, while minimizing the amount of information about the exogenous noise and the observation. This can be formalized as,
LAM Objective := max [T(z; I(=; - [[=
(5)
where I is mutual information. Hence, an ideal latent should contain all information about the action label, and no other parts of the environment, consistent with intuitions of previous work "biasing [the latent] towards simpler representations is likely preferrable" (Schmidt and Jiang 2023).
Whilst mutual information is typically a challenging quantity to measure, in our linear model we can approximate this straightforwardly. After training the linear LAM via Eq. and freezing the parameters, we fit three additional linear layers predicting (g,6,0) from 2, to give (q, é, ó) respectively. Note that, under the assumption that the variables x and y are independent multivariate
where ý is the least squares estimate (LSB) of based on x, and Var(-) indicates the total variance (see e.g., Chapter of (Johnson] et al., 2002 ) of multivariate random variable. Hence, we can define an evaluation metric that captures the objective of training linear LAM,
Gaussian variables, the mutual information Z(x. y) = - log
Ly Var(y)
2
1 Var(q)
J 6-
1- Var(c)
1 21
Linear LAM Objective (LL.O) = max
(6)
Var
Note that we get rid of the log(-) function wrapped around the MSE loss to better calculate the values for this objective since the value range for the mutual information [0, +o0) is unbounded. This
objective is maximized when z perfectly predicts q while containing no information about e and o,
resulting in the optimal value for LLO equal to 0 + 1+++=2 2.
4 Analysis
At times, will present numerical simulations of linear LAM to visually communicate later analysis. See the details of simulation in Appendix and the code in supplementary.
4.1 Analysis 1: Linear LAM is PCA
This section first shows that training linear LAM is equivalent to performing PCA on the mixture of controllable changes and exogenous noise. This requires that the controllable changes and exogenous noise q: 6 are uncorrelated with the observation o (Section relaxes this assumption).
We discuss the insight that this connection provides, followed by an analysis of several important cases covering specific settings of controllable changes and exogenous noise.
Proposition 4.1 (Linear LAM is PCA). Under the linear LAM model and setup defined in Section 3.2 and additionally assuming Elo(q + e)² = O, the objective of linear LAM is equivalent to performing PCA on mixture of controllable changes q and exogenous noise e.
= E [I[ () + <)][
Note, BD is a low-rank matrix to capture the main components of q + e. See proof in Appendix C.I
Given that we have transformed the optimization problem of linear LAM into a PCA problem (which can also be viewed as linear auto-encoder), PCA's property applies to linear LAM.
Proposition 4.2 (Linear LAM tries to capture q + E). Denote the covariance matrix of q + as Eg+t = El[q + () 4 + e)" and its eigenvalue decomposition Eg = UAU with eigenvalues
X1 d2
Ad.- Under the same conditions of Proposition 4.1 L optimized when
1. B = Ua, spans the subspace of the top d- principal components of Egtt where Ud. contains the first d- columns Of
2. = UT, such that the reconstruction BD(g + e) = Ua, U3, (q + 6) projects q + 6 onto the subspace spanned by Ua.-
5

---

Unzer LAM obyective
Unear Lich cojective
Useer LAM Onjective
38
23
hol. NCSC
14
14
3.50
ICT HSE
Fook, NSEE
4,"2
1.2
Optimini
130
6,41
4,-10
12
0-11
6--12
13
Opt ma
Optmal
dG
Figure 3: LLO (Linear LAM objective 6). higher better) measured in three noise settings. (Left) 0. (Middle) is i.i.d. noise. (Right) contains the effect of other agents. Action MSE, noise MSE, and observation MSE are the three terms in (6) We set real action dimension da = 8 and exogenous action dimension d, = 8 unless otherwise stated, and ensure q has unit variance.
The minimum loss given by the sum of the eigenvalues of corresponding to the discarded principal components - Dizd,+1 Ag.
This proposition is equivalent to the Eckart- Young Mirsky theorem (Eckart and Young. 1936 whose proof can be found in Chapter 2.4 of Golub and Van Loan (2013).
Over-parameteriizaion issue. We show that linear LAM is over-parameterized, with multiple solutions of (A, B, C, D) able minimize the reconstruction loss objective. Specifically, the latent Co + Do', in addition to capturing information about q and may further contain information about o. There is no detrimental effect provided the A matrix compensates to 'knock out' o's information in z. Concretely, there exists family of solutions BCCo + Do') = (q + e) + ao and A = (1 / for any E R such that o' B(Co - Do')= + Ao (a +)++) + - a-o ao o'.
We will revisit this issue in Section
showing that data augmentation handles this over- parameterization issue. For the purpose of our immediate analysis, we predict (q. é, O) from a surrogate latent 2 := B- (o' - o) when calculating LLO to get around this issue. This surrogate la- tent is the same as the original Hence, we have C -D which indicates that the semantic meaning of the latent 2 = Co + Do' remains the same (no movement) when of across different observations.
when A I, which is the when data augmentation is adopted.
0. In the absence of exogenous noise e, linear LAM does capture the true action a in the latent when the bottleneck dimension is set equal or larger than the action dimension.
Case
When 0 and q = Xa. the covariance matrix of Zig+e = Σn aaa X] only has d, non-zero cigenvalues. Following Proposition 4.2
we can make the following conclusions.
When d, 2 du, the capacity of the latent is large enough to capture all the information about the controllable change q, the minimum loss - = 0, and LLO achieves the optimal.
Specifically, when d₂ = da the subspace spanned by the columns of B = Ua, is the same to the subspace spanned by the columns of action effect matrix X (by noting that Ua, AUT, Egtt " XE[aa"" xT)) ). In this case, the learned latent z (whose effect is interpreted by B) fully captures the information of a (whose effect is X), and LLO is maximized. In other words, for this ideal case linear LAM's latent perfectly captures the information of the true action a without access to it.
When d2 < da. L > and this lincar auto-encoder captures the first d, components in q.
We illustrate how LLO varies across different d. and d. through numerical simulation in Figure (left). The simulation validates that linear LAAM is optimized terms of LLO when d, 2 d..
Case 2: e is i.i.d. noise. We consider i.i.d. noise (independent and identically distributed), which may be a realistic assumption in the case sensors or image encoders. We assume e is i.i.d. with zero- mean [|E] = 0 and isotropic covariance Ejee"] ]= L. Considering that q and are independent, the covariance matrix Eg can be eigenvalue decomposed as Egtt UT,UT Uo UA TUU where Un and Ag are the eigenvectors and eigenvalues of En respectively (i.e., when
0). Combining the results in Proposition 4.2 we conclude that, when there is i..d. noise, 1) the FDM

---

parameter B (and therefore the semantics of the latent since B interpret the latent) remains the same, and 2) the loss increases since the eigenvalues increase. This conclusion consistent with the conclusion on the robustness of PCA (Anderson 1963. Johnstone 2001).
Figure 3 (middle) shows how LLO changes for Gaussian noise of differing variance. We observe that linear LAM is robust to i.i.d. noise up to around azd = 0.5 (when the noise intensity is half that of the signal), linear LAM still succeeds with negligible gap the optimal case.
Case 3: e contains the effect of other agents. In many real-world datasets (such as Ego4d (Grauman et al. 2) the change between two observations may not only be caused by the control action of the ego-agent (e.g. joints of a robot arm), but additionally can be effected by exogenous noise, such as 'other agents' (e.g. a person walking in the background, camera shake). Compared with i.i.d. noise, this noise is structured and would he predictable if the other agent's 's control actions were available.
Here, we assume action taken by other agents. In this case, analysis similar to that of Case I concludes that linear LAM will learn to capture effects with largest variances, no matter if they result from the controllable action or other agents. Moreover, when the columns in Y are not orthogonal to that of X, the FDM parameter B will be impacted by the exogenous noise.
Yb where Y E I&d. is the exogenous effect matrix and b represents the
from
Simulation results in Figure 3 (right) illustrate which part of information (the controllable q or the noise E) enters the latent when the latent dimension d₂ is increased. We observe that: 1) When the variance of exogenous noise is smaller than q (i.e., Tg > Tezo = 0.5), linear LAM learns to fit the controllable part first, and d= = du is still the optimal configuration. 2) When the variance of noise exceeds that the signal q (i.e., og < aexo = 2.0), linear LAM will first fit the noise, and the latent fails to exclude the information of the noise no matter how we set dg. To alleviate this issue, an obvious solution in practice is to preprocess training data to reduce significant noise components (such stabilizing camera shake).
4.2 Analysis 2: Effects of Data Collection Policy
considered the case where both the controllable changes and the exogenous noise independent of the observation, i.e., E[o(q + c) O. However, practical LAMs are usually trained on data created by expert policies, e.g., robotic datasets often consist of human teleoperation. Within such datasets, the observation is correlated with the action a, and thus the controllable change q. resulting in Elo(q + +)²] 7 O. This section analyzes this case in linear LAM.
Section 4.1
Specifically, we assume the data generating policy acts via a = o1 + n, where 11 E RR xd. represents the deterministic part of the policy and π. E Rdd represents the stochastic part of the policy. We assume that o, c, and n, are uncorrelated, but there will be correlation between and a.
In this case, since R Bloq") A O, the cross term in the expansion of this loss cannot be ignored, and no longer reduces to PCA. Letting E. == Ejoo"] be the covariance matrix of the observation, we can further solve for A by setting the partial derivative of the loss function w.r.t. A to zero, obtaining
A= 1 - (BC + BD) -
(8)
This differs from the previous result for the random policy where A = I - (BC + BD) by the last term E X E2- The intuitive interpretation for this term is that A will cupture changes caused by the deterministic part of the policy (noting that BD - / can project the vectors onto the orthogonal complement of the column space of B). This is problematic for LAM, since the FDM can implicitly absorb deterministic parts of the policy, leaving capacity in the latent available to absorb exogenous noise. Intuitively, if action a' is always selected for observation o* this always leads to o The FDM can simply learn that the mapping o - o²*, without the need for a latent
In our simulation, we control the randomness of the policy using a coefficient X. and let a X + (1 x)H, where II is a random projection matrix and т. is standard Gaussian vector. The simulation results Figure 4 (left) indicate that the more deterministic the data collection policy is, the less information about a is captured by z. Therefore, the analysis in this section suggests data collection policies with higher randomness are preferred for LAM training. Trajectories collected by deterministic expert policies may lead to poor quality LAMs.
7

---

Unerr LAN Oejective
Linear Lidh colecsie
Liner LAM objezove
Ikl m0.
Ikl 0.43
125
3.4
5-3
14
Dg 0.2
Mos LU
Ogo 20 lade
12
Optin
0.5 19: booe
Tabel
2.200
Optimal
33
Opcimal
Palkyy
Bottleneck
Figure 4: LLO 6 in three settings. (Left) Data generated from policies ranging from fully random to fully deterministic. (Middle) Linear LAM trained w/ varying strength data augmentation. (Right) Linear LAM trained w/ and w/o action prediction. We set da = 8 the experiments.
4.3 Analysis 3: Improvements via Data Augmentation
This section analyzes the augmentation scheme used in Chen et al. (2024b). which will turn out to resolve a key issue encountered in linear LAM (see the discussion on over wer-parrametrrizatin in Section 4.1 where information about the observation o enters into the latent z.
The IGOR model (Chen et al., 2024b) applies one shared random crop to both inputs of the IDM, and a second shared random crop applied to the FDM and reconstruction target. The intuition is that "by using different croppings [for IDM and FDM), the model is encouraged to learn a more semantically invariant latent action". Our subsequent analysis shows this intuition is provably correct in linear LAM, resulting in new terms to the loss that encourage removal of the latent of any semantic information about the observation. [Sun et al. (2024) also apply similar scheme, with single action preserving crop applied to the IDM which makes it harder "to copy the appearance information directly from the [IDM]".
Data augmentation in linear LAM. We extend our linear LAM setup to include a data augmentation operator Aog, = + K; that takes observation input and adds some random vector R. Aug,E applies the same i-th operator to different variables, say Aug, (o) and Aug, [o'] will apply consistent random variable Ki each observation.
Proposition 4.3 (Data augmentation addresses over-paraaeteerizatoon) With data augmentation,
Linear
( (Aug, (o], Aug, (o'1))))) C(o k1) + D(o' + tx1)
(9)
ó'
FDM (Aug2lo), = A(o k2) Bz.
(10)
and assuming Elo(q c)²) 0, optimizing the loss defined in results in A = I and C D 0.
See Appendix C.2 for proof. What is the effect of encouraging A 7 and C + = 0? If, A is the identity, all observation information flows directly through this matrix, and need not carry any additional information about o (given our additive transition function). Equivalently, setting C' + D = 0 cancels out information in 2.
z z= = K1)= + D(o' + kn) = - k1) + ++++q++++ K1) c
(11)
This condition improves the semantic meaning of the latent action across different observations. For example, the zero latent 0 has the semantic meaning "the frame does not change from o'". which is consistent across different observations. Hence, this augmentation scheme is a mechanism to remove information about o from 2, explicitly minimizing I[z; o) in the original objective (5).
Simulation. We show the effect of data augmentation under different noise levels in Figure (middle). Starting from here, the simulations calculate LLO based the true latent instead of the pseudo-latent. Due to this change, even when there is no noise o = 0, the learned latent z cannot perfectly predict the real action without other information. However, adding an augmentation vector with 0.1 magnitude variance (10% of the observation variance), greatly improves the latent learned by linear LAM, achieving close the ideal L.L..
Designing data augmentation. In our linear setting, we have designed the data augmentation k: as additive and i.i.d. across its elements. This design is based our knowledge that the semantic

---

meaning for the frame change should be invariant to this data augmentation, i.e., o' (o' - x) - (o + k),, since the dynamics are also additive. For real images. Chen et al. (2024b) adopt random crops, reflecting prior that the action is invariant to position (e.g., a robot closing its gripper should produce the same latent action regardless of its location in the image). Further, the variance of different augmentations determines how important this term is a larger augmentation variance enforces this constraint more strictly. In summary, our analysis justifies the practice of designing data augmentations that capture a model designer's domain knowledge about desired invariances to improve the semantics the learned latent.
4.4 Analysis 4: Improvements via Auxiliary Action Prediction
This section analyzes the setting when a small dataset of action labeled data Da is available during training of the LAM. This can be used help guide the latents to represent controllable changes rather than exogenous noise. Specifically we consider the a simple auxiliary loss, with latents input, predicting the action labels when available. Niiullnn al. (2025) also provide empirically evidence to show that this is promising strategy to avoiding focus on 'distractors' present in the observations.
Action prediction in linear LAM. Consider linear prediction head E E Rd,xd, at the latent bottleneck, â = Ez, and corresponding action reconstruction loss || a 12.. we optimize this objective, L. := Ep [1l6' - o' [ AED [ll a [ and where À = DDal/DD)
Proposition 4.4 (Action prediction can denoise). Following the conditions in Proposition 4.3 and assuming E[q"* <] = 0 and d, 2 du. optimizing L. biases the encoder parameter perpendicular to the noise. For an artificial case where X +00, we obtain perfect LAM with De 0 and Bz q.
We provide the proof in Appendix C.3 Note that we assume that e X = O, meaning that the action effect q Xa and the noise are independent. For example, this would hold in the case of table-top manipulation with passers-by occasionally walking by in the background as the source of E (cf. Case 3 in Section 4.1. The conclusion D) indicates that noise will not enter the latent (noting that - in this case).
Proposition 4.4 indicates that auxiliary action prediction helps linear LAM reduce the noise in its latents. Figure 4 (right) present our simulation, we apply different levels of 'other agents' noise (case 3 in Section 4.1) with the same settings in Figure 3 (right). Unlike Figure 3 (right), where Tezo 2.0 caused linear LAM to fail, now 1 1% of action labels in auxiliary action prediction leads to successful learning. The simulation results indicate that only small proportion of action labels are needed to encourage the latent to focus on encoding real actions, and not noise.
5 Beyond Linear LAM Experiments
To test whether the theoretical insights from our linear LAM analysis hold in more realistic settings, we ran empirical experiments using more realistic LAM setup, on a carefully designed synthetic dataset, which allows us to carefully measures proxies for LLO. We use small 4x4 images as input (instead of vectors), non-linear CNNs in IDM and FDM (instead of linear layers), and vector quantization (instead of using a reduced linear dimension as a bottleneck). We will show that the main conclusions in our paper still hold in this more complex setting.
Dataset. We designed 4 x 4 grid-world world style synthetic dataset. The top x grid of the observation contains a square (intensit==1.0) 10) that can be controlled with five actions (up. down, left, right, and stay still). The bottom 1 x 4 grid of the observation contains random Bernoulli noise (with prob 0.5). An intensity parameter controls the noise magnitude (none=0.0, low=1.0, .0, high=2.0).
Policy. By default, we use uniform policy, where each action is equally probable. For one experiment, we also use a correlated policy, where state and action are correlated. With 95% prob, the action moves the square on a fixed snaking pattern through the grid, and with 5% chance a random action is selected.
Model. For the IDM, we use a small CNN to encode o and o'. followed by a VQ bottleneck with codebook size of 5 outputting the latent. Finally, for the FDM, separate UNet takes the latent and previous observation o to output the predicted o' When predicting actions, codes are preassigned to actions, and latents are trained to minimize 1.2 distance to their true action code. For data augmentation, we shift the 4 x 4 image left/rigtt for one grid with periodic padding. Models

---

Setting
Controllable loss (L)
Stochastic loss (1)
No noise
0.624 = 0.087
Low noise
0.781 0.079
0.739 + 0.047
High noise
1.046 = 0.017
0.607 + 0.021
Uniform policy
0.781 = 0,079
0.739 + 0.047
Correlated policy
1.997 = 0.022
0.599 + 0.036
data augmentation
0.781 0.079
0.739 1 0.047
Data augmentation
0,415 1 0.020
0.898 1 0.049
No action prediction
0.781 + 0.079
0.739 1 0.047
1% action prediction
0.295 0.011
0.986 + 0.002
Table 1: Ablations of performance of practical LAM under different settings. We present the mean and standard errors of controllable loss (measures prediction loss of true action from latent, lower means more action information) and stochastic loss (measures prediction loss of noise from latents, higher means less noise information) over five seeds. Each group corresponds the analysis of one sub-section in Section and we bold the losses of the better variant in each group.
were trained for 16k updates with the Adam optimizer. Unless specified, we use low stochastic noise, no action prediction, no data augmentation, and five codebook vectors.
Evaluation. Not being linear, is not possible to measure the mutual information between latent and quantities of interest exactly. Instead, since by design the observation separates the controllable (top half) from stochastic (bottom half) changes. we measure the reconstruction loss on the relevant portion of the observation to assess how what information the latent has captured following training. lower controllable loss means the latent contains more information about the action, and lower stochastic loss means more noise has been captured. Similar to LLO,, this is normalized by the variance of the signal. Note there unfortunately no straightforward way to measure the information about the observation o in the latent for this non-linear case.
Results. We present experiment results containing controllable loss and stochastic loss in Table Firstly, we train vanilla LAM on different noise levels. Results show that as noise is increased from none to high, the action information in the latent decreases (increasing controllable loss). At the same time, the information about the stochastic noise increases (decreasing stochastic loss). This is consistent with our linear LAM theory in Section 4. that latents encode whatever leads to most variance. Secondly, we see that switching from uniform to a correlated policy reduces the information about actions in the latents, while increasing the information about stochastic noise. This is predicted by our linear LAM theory in Section 4.2 that deterministic data collection policy can lead to degenerated performance. Thirdly, we see that data augmentation improves LAM learning. This is consistent with the conclusion in Section 4.3 that data augmentation can help LAM learning. Finally, we show that incorporating 1% of actions labels into the training process improves over vanilla LAM by increasing information about actions, and reducing information about noise. This is consistent with the conclusion in Section 4.4, that predicting true actions improves LAM learning.
Conclusion
Latent action models (LAMs) are increasingly used in the pre-training phase of embodied Al models. However, they have so far been intuitively and empirically motivated, without thorough analysis into their learnability. This paper bridges this gap by proposing study LAMs through a simple model capturing the essence of LAM training while remaining mathematically tractable. Through our analysis of linear LAM, we made several observations. including showing that under certain assumptions, it corresponds precisely to PCA. In several noise settings, linear LAM leads to recovery of the true action this supports the use of real-world LAM in low noise settings, as well as the application of preprocessing steps to reduce noise in dataset. However, we also showed that when the main cause of variation between consecutive observations not the controllable signal, LAM latents will capture noise rather than action information. We observed further danger so far undocumented in the literature- a lack of randomization latents learned by LAM. Finally, we provided analytical results justifying techniques emerging in the LAM literature improve the quality of latents learned data augmentation and auxiliary action prediction.
data collection policy can harm
10

---

References
AgiBot-Worde World, Agibot world colosseo: Large-scale manipulation platform for scalable and intelligent embodied systems. agibot-world.com, 2025.
Allshire, A. al. Laser: Learning latent action space for efficient reinforcement learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 6650-6656. IEEE, 2021.
Anderson, T.W. Asymptotic theory for principal component analysis. The Annals of Muthematical Statistics, (4(1):122-148) 1963.
Bruce, J. et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
Chandak, Y. et al. Learning action representations for reinforcement learning. In International conference on machine learning, pages 941-950. PMLR, 2019.
Chen, T.S. al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13320-13331, 2024a.
Chen, X. et al. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai. arXiv preprint arXiv:2411.00788 2024b.
Chen, Y. et al. Moto: Latent motion token as the bridging language for robot manipulation. arXiv preprint arXiv:2412.04445 2024c.
Cui, Z. et al. Dynamo: In-domain dynamics pretraining for visuo-motor control. Advances in Neural Information Processing Systems, 37:33933-33961, 2025.
Eckart, C. and Young, G. The approximation of one matrix by another of lower rank. Psychometrika, 133)2211-218, 1936. doi: 10. 10.10077BF02288367.
Edwards, et al. Imitating latent policies from observation. In International conference on machine learning, pages 1755-1763 1763. PMLR, 2019.
Fang, H.S. et al. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. arXiv preprint arXiv:2307.00595 2023.
Gao, S. et al. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503. 18938, 2025.
Golub, G.H. and Van Loan, C.F. Matrix computations. JHU press, 2013.
Grauman, et Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 18995-19012, 2022.
Hua, P., Chen, Y. and Xu, H. Simple emergent action representations from multi-task policy training. arXiv preprint arXiv:2210.09566. 2022.
Jiang, Z. et al. Efficient planning in compact latent action space. arXiv preprint arXiv:2208.1022 10291, 2022.
Johnson, R.A., Wichern, D.W. et al. Applied multivariate statistical analysis. Prentice hall Upper Saddle River, NJ, 2002.
Johnstone, LMM. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, 29(2):295-321 322))222022,,0 2001.
Khazatsky, A. et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.1 12945. 2024.
McCarthy, R. et al. Towards generalist robot learning from internet video: A survey, arXiv preprint arX1v:2404.1964,, 2024.

---

Menapace, W. et al. Playable video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10061-100780 2021.
Mendonca, R., Bahl, S. and Pathak, D. Structured world models from human videos. arXiv preprint arXiv:2308. 10901, 2023.
Miech, A. et al. Howto100m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630-2640) 2019.
Nikulin, al. Latent action learning requires supervision in the presence of distractors. arXiv preprint arXiv:2502.0033 2025.
Nvidia. GrOOt nl: An open foundation model for generalist humanoid robot. Nvidia Release, 2025.
Pei, B. et al. Modeling fine-grained hand-object dynamics for egocentric video representation learning. arXiv preprint arXiv:2503.0090 00986, 2025.
Puterman, M.I. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Rybkin, O. al. Learning what you do before doing anything. arXiv preprint a0X1v:/806.006555 2018.
Schmidt, D. and Jiang, M. Learning to act without actions. arXiv preprint 2rX11:2312.10812, 2023.
Sun, Y. et al. Video creation by demonstration. arXiv preprint arXiv:2412.09551, 2024.
Van Den Oord, A., Vinyals, O. et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Vuong, Q. et al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@@ CoRI.2023, 2023.
Wang, Y. al. Internvid: large- video-text dataset for multimodal understanding and generation. arXiv preprint (rXivv2307.06944/ 2023.
Wang, Y. et al. Ad3: implicit action is the key for world models to distinguish the diverse visual distractors. arXiv preprint arXiv:2403.09976 2024.
Yc, S. et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11752 2024.
Ye. W. et al. Become proficient player with limited data through watching pure videos. In The Eleventh International Conference Learning Representations, 2022.
12

---

A From practical LAM to linear LAM.
Our goal in designing linear LAM was to preserve the key features of LAMs in practice, while making the resulting model as simple as possible. Here we remark on similarities and differences.
Function approximation. Linear LAM uses linear layers, while practical LAM uses deep neural networks. However, linear LAM is compatible with input vectors processed by other non-linear pre-trained image encoders.
Bottleneck. Both models have an information bottleneck between in the IDM but this is implemented in different ways. Practical LAMs usually adopt vector quantization Van Den Oord et al. (2017) while linear LAM uses low continuous dimension d, < de.
Additive changes. We formulate changes between observations additions of controllable changes and exogenous noise. This additive structure provides the simplest combination of two elements which we believe
primary concern of whether LAM''s latents actually represent actions.
Noise. While our model constrains the noise as additive, it does not make further assumptions. We later analyze realistic cases corresponding real world scenarios, such as when action and noise are correlated, and action and observations are correlated.
B Simulation Set-ups
We adopt the following setting unless otherwise stated. We provide the source code in the appendix.
Observations. Observations are sampled from the standard normal distribution, o -N(0,, I) with I as the identity matrix. Note that Var(o) = (i.e., each element in o has unit variance). We set the dimension of the observations d,- 128.
Actions. We continuous actions also sampled from standard normal distribution with di- mensionality da, so a I). Note that Var(a) = 1. We set da = unless otherwise stated.
Controllable changes. The action effect matrix X that maps the action a to the controllable changes q. chosen as random orthogonal matrix (using QR decomposition), subsequently normalized to ensure Var(q) = 1.
i.i.d. noise. In our simulation we use isotropic Gaussian e (N, ?? with variance on as the i..d. noise. For this noise, Var(e) = oi
Exogenous noise. We also consider the noise induced by other agents Yb Geno Yob where b is other agents' action, Y is the action effect matrix of other agents, and Yo is the normalized matrix of Yo. Similarly, Yo is chosen as a random orthogonal matrix using QR dccomposition to ensure that V Var(E) = =
Optimization. We implement our system in PyTorch, optimizing trainable parameters via stochastic gradient descent with the Adam optimizer with batch size 128. We use the default learning rate and run for 4,000 steps to ensure convergence.
Evaluation. Our the quantity defined in (6) as the default evaluation metric. For the experiments that do not involve noise, we set normalized MSE for noise to 1.
Data augmentation. Data augmentation is a trick that may improve the learnability of LAM
mentioned in several previous papers (Chen et al. 2024b Sun al. 2024). We implement data augmentation by adding a Gaussian noise K ~ (O, K I)) for linear LAM. Note that
Var(k) |x|2 By default data augmentation is turned off, except for Figure 4 (middle and right).
Action prediction. Action prediction is another trick proposed in the previous paper (Nikulin et al.) 2025). We implement action prediction by predicting the true action label based on the latent with a learnable linear transformation for a small proportion of the data samples. We denote the ratio of the samples that we access their action labels as X. We find that setting À 1% enough. By default action prediction is turned off, except for Figure 4 (right).
13

---

C Proofs
C.1 Proof of Proposition 4.1
Proof. Expand the loss function in with the definitions of o' and o' (o' =0+ q c), then
rearrange (ignoring the expectation outside the RHS).
L ==llo' 8'112
(12)
=Ilo' (Ao BCo+ BDo)
(13)
=Ilo + q c) (Ao BCo+ (D q c))l
(14)
=I[AA BC BD 1)0 (BD e)
(15)
=I[AA BC + -D-
- + + c)l
(A+ 34++BC++ + B--BD-- - [B)++((++)))
(16)
=II(A + BC + BD + || - -I)( +
2 ((A+ + BC BD e))
(17)
Recall that this loss C is found within expectation in (4). By assumption E|o(q c) 0 and the final term can be ignored (since both expectation and trace are additive).
L = E (I + BC + BD 1)ol + [I( - 1) c)
(18)
Regarding the first term, note that A is full rank do while BC and BD are of rank d.. Since A only appears this and it is of greater or equal rank than BD + BD, it's optimal value is A = 1 - BCC + D), setting the first term is zero. Hence we are left with the middle term.
[I - + e)ll2
(19)
Note that, under the condition of Proposition adopting data augmentation Proposition 4.3 the condition A = BBC + D) is satisfied.
4.1 (A, B, C, D) are over-parameterized. When we
we obtain and C + 0, which refines how
C.2 Proof of Proposition 4.3
We start by expanding the loss defined in ① with the data augmentation scheme, and unpack (ignoring the expectation outside the RHS).
L = |Augg[o'] | 0'112
(20)
I|o' K2 (A(o + K2) + B(C +ld x1))
(21)
(A(o +((o+k2)) k2) + B(C m)) + D + +++
(22)
[[AA+ BC+ BD 1)o (BD- 10) (BC + BD) + A)
(23)
[IBD +)(q+))13 - +1(A+ + BC BD l)ol
+IB[CC + D)x, || [I(1 - A)x211
(24)
Where the last line follows since K2 and K2 are sampled independently of all other terms and E[o" (q E)] Hence, we left with the vanilla linear LAM loss in (13) plus two additional terms.
Minimizing this function can be achieved by exactly setting A I and C DD which zeros the last three terms simultaneously. In this case, the optimization problem again reduces to PCA [|BD (q 1)
Discussion. the correlated case EEo (q + <)) 0, when the third term (the cross term) in 17 cannot be ignored, nevertheless there is encouragement to reach A = I and C' -D.
14

---

C.3 Proof of Proposition 4.4
Based on the conclusion in Proposition optimizing for the first term in La results in A = I and C' = D. We consider the case where D and Da come from the same distribution and the availability of action labels are uniformly random. Therefore, we can ignore the expectation outside the RHS (for simplicity) and re-write the loss as,
La = IIBD l) + + + AII (q - alı
(25)
= -I(BD - 1) + |IBBI
- 1) + AI all + DDE
(26)
= -I(BDX - -x)al² + II(BD r)cl2 2 + AII - Ia. )all 2 ED
(27)
The second line leverage the fact that Elq" E] U E Rd.xd (which spans the column space of X), E Rd xd, and V Rd xd,
We decompose X X UEVT with
We will show that, when À + too, where indicates arbitrary entries and vanishes when d₂ da-
B [U, and E [VE minimizes Ca
U
First, this solution zeros the last two terms. For the last term, De = 0
== since ef xx=0
=0
and U spans the column space of X. For the third term, EDX = [VE U
UEVI = Id.
Then, since we have determined D, the second term becomes
(BD - r)el3 = |El/2. We can choose
U
B [U, to zero the first term since BDX U *]
X. In this way, we obtain
minimal loss - = ||= We can also verify that Bz = BCCo+ Do') = BDq = UD"
Though the À + +oo setting is artificial, the analysis show that positive À will bias the encoder D capture less about the noise and more about the signal q (or a).
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] pprovided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We propose and analyze linear LAM to provide insight for its practical usage
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: our appendix, we discuss there exists gap between linear LAM and its practical version.
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We have stated the assumptions and provide proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We describe the experiment setting in the appendix and provide the code as supplementary.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We de not need external data and provide the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guideeee http://oeeuuiiissooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
We describe the experiment details in the main text and the appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We results are mostly qualitative and reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experiments only need a minimal computation set-up (e.g.. laptop).
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is a theoretical work and does not apply for discussion on immediate societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No model data is released.
Guidelines:
20

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does existing
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
21

---

Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy htte:e/r/eerrppppecc//ooeeereeee for what should or should not be described.
22


Reviews:
[{"review_id": "B18LPTYoNT", "reviewer": "Reviewer_VEZh", "summary": "This paper proposes a simplified version of latent action models (LAM), called linear LAM, and develops a mathematical theory around it, supported by synthetic experiments. Using this simplified framework, the authors justify the standard training objective for LAM and highlight the risks of over-parameterization. They also theoretically show that correlations between observations and actions can degrade performance. Finally, the paper provides theoretical explanations for how two practical techniques—data augmentation and semi-supervised learning with a small amount of labeled data—can help mitigate the effects of over-parameterization and exogenous noise.", "questions": "How well does the simplified linear LAM model capture the behavior of real-world latent action models? Could you discuss the gap between this abstraction and practical architectures, and its impact on the applicability of your theoretical results?", "limitations": "I do not see any major ethical or societal concerns raised by this work. The paper acknowledges some limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- To the best of my knowledge, this is one of the first works to provide a theoretical analysis of the learnability of latent action models (LAM).\n- The mathematical derivation is thorough and complemented by intuitive explanations. The paper discusses a range of scenarios that help clarify the theoretical insights.\n- The synthetic experiments are well designed, align closely with the theoretical results, and are easy to interpret.\n- Although the analysis is conducted in a simplified setting, it may offer useful insights into training behaviors and challenges that arise in practical applications of LAMs, such as imitation learning or offline reinforcement learning.\n\n**Weaknesses**:\n- While the simplified linear LAM enables a more tractable analysis, it is unclear how well this abstraction reflects the behavior of LAMs used in practice. It would strengthen the paper if the authors could discuss the gap between the simplified model and real-world architectures in more detail.\n- There are minor typos, such as using “a” instead of “an” in some cases. A careful proofreading pass is recommended.\n- Results from Section 4.3 are used earlier in Section 4.1. While this is noted in the text, clearer referencing (e.g., citing the relevant proposition) would reduce reader confusion.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are grateful that the reviewer acknowledged our paper as “one of the first works to provide a theoretical analysis of the learnability of LAM” with “thorough mathematical derivation” and “intuitive explanations”. We address the concerns as below, and hope these might be worthy of an upgrade in score.\n\n***[Simplified-to-real gap] Discuss the gap between simplified model and real-world architecture. How well does the simplified linear LAM model capture the behavior of real-world latent action models? Could you discuss the gap between this abstraction and practical architectures, and its impact on the applicability of your theoretical results?***\n\nThank you for this important point – in order to better understand how linear LAM captures the real-world LAM, we have conducted further experiments to validate that the main conclusions we obtained from linear LAM also applies to the setting that bridges the three gaps between linear and practical LAM: 1) linear to nonlinear model architecture, 2) continuous to sometimes vector quantized bottleneck; and 3) linear to complex world dynamics (also see Appendix A). We present the new experiment results in the end. \n\n***[Paper writing] There are minor typos and the referencing issue.***\n\nApologies for allowing these to pass in the first version. We have fixed these issues in the next version.\n\n---\n\n# New Experiment Results\n\nWe provide a set of experiments using practical LAM to understand how our insights from linear LAM analysis transfer to real settings. Specifically, we use image as the input, non-linear CNNs in IDM and FDM, and vector quantization. We show that the main conclusions in our paper still hold in this complex setting.\n\n**Dataset**\n\nWe designed a 4x4 grid-world style synthetic dataset. The top half of the observation contains a square (intensity=1.0) that can be controlled with 5 actions (up, down, left, right, and stay still). The bottom half of the observation contains random Bernoulli noise (with prob 0.5). An intensity parameter controls the noise magnitude (none=0.0, low=1.0, high=2.0).\n\n**Policy**\n\nBy default, we use a uniform policy, where each action is equally probable. For one experiment, we also use a correlated policy, where state and action are correlated. With 95% prob, the action moves the square on a fixed snaking pattern through the grid, and with 5% chance a random action is selected.\n\n**Model**\n\nFor the IDM, we use a small CNN to process $o$ and $o’$, followed by a VQ bottleneck with codebook size of 5 outputting the latent. Finally, for the FDM, a separate UNet takes the latent and previous observation $o$ to output the predicted $\\hat{o}’$. When predicting actions, codes are preassigned to actions, and latents are trained to minimize L2 distance to their true action code. For data augmentation, we shift the 4x4 image left/right for one grid with periodic padding. Models were trained for 16k updates with the Adam optimizer.\n\n**Default setting**\n\nUnless specified, we use low stochastic noise, no action prediction, no data augmentation, and 5 codebook vectors.\n\n**Evaluation**\n\nNot being linear, it is not possible to measure the mutual information between latent and quantities of interest exactly. Instead, since by design the observation separates the controllable (top half) from stochastic (bottom half) changes, we measure the reconstruction loss on the relevant portion of the observation to assess how what information the latent has captured following training. A lower controllable loss means the latent contains more information about the action, and lower stochastic loss means more noise has been captured. Similar to LLO, this is normalized by the variance of the signal. Note there is unfortunately no straightforward way to measure the information about the observation $o$ in the latent for this non-linear case.\n\n**Results**\n\nFirstly, we train vanilla LAM on different noise levels. Results show that as noise is increased from none to high, the action information in the latent decreases (increasing controllable loss). At the same time, the information about the stochastic noise increases (decreasing stochastic loss). **This is consistent with our linear LAM theory in Section 4.1, that latents encode whatever leads to most variance.**\n\n|  | Controllable loss | Stochastic loss |\n|--|--|--|\n|No noise |  0.520   | 0.000  |\n|Low noise |  0.951 | 0.694 |\n|High noise |  1.013  | 0.539   |\n\nSecondly, we see that switching from a uniform to a correlated policy reduces the information about actions in the latents, while increasing the information about stochastic noise. **This is predicted by our linear LAM theory in Section 4.2, that deterministic data collection policy can lead to degenerated performance.**\n\n| | Controllable loss | Stochastic loss |\n|--|--|--|\n| Uniform policy  |  0.951 | 0.694 |\n| Correlated policy |  2.022 | 0.579 |\n\nThirdly, we see that using data augmentation can improve LAM learning. **This is consistent with the conclusion in Section 4.3, that data augmentation can help LAM learning.**\n\n| | Controllable loss | Stochastic loss |\n|--|--|--|\n| No data augmentation |  0.951 | 0.694 |\n| Data augmentation | 0.450 | 0.891 |\n\nFinally, we show that incorporating 1% of actions labels into the training process improves over vanilla LAM by increasing information about actions, and reducing information about noise. **This is consistent with the conclusion in Section 4.4, that predicting true action can improve LAM learning.**\n\n| | Controllable loss | Stochastic loss |\n|--|--|--|\n| No action prediction |  0.951 | 0.694 |\n| 1% action prediction |  0.317 | 0.980 |"}, {"review_id": "LhijJAlOXv", "reviewer": "Reviewer_EmH8", "summary": "This paper presents a theoretical analysis of a linear latent action model, offering several insights into its fundamental characteristics and the impact of various data manipulation strategies. In particular, the analysis is based on a bottleneck structure and focuses on the relationship to controllable changes, noise, and observations. The theoretical findings are supported by numerical simulations.", "questions": "1. Is the analysis applicable when observations consist of multiple frames? Intuitively, capturing actions based on only two adjacent frames seems limiting.\n2. All analysis relies on the additive transition function in Line 139. Is this assumption valid in practice?\n3.  I understand in the ideal case when $z$ contains no information about the noise and observation, the LLO value is 2 in Line162, but why this is the optimal value of LLO.\n4. Some typos. Line 195, should be $A=(1-\\alpha)o$; Line 323, missing $\\lambda$ in the objective.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper is clearly written and well-organized.\n2. It provides new proofs and theoretical analysis for latent action models, with findings consistent with common practices in the literature.\n3. The analysis helps bridge the gap between theory and practice, offering valuable insights for future work.\n\n\nWeaknesses:\n1. The paper is mathematically heavy, making it challenging to fully grasp the details.\n2. The analysis is based on simplified linear models with strong assumptions(Line 170), limiting the generalizability and practical applicability of the results to more complex or real-world models.\n3. The paper validates its findings solely through numerical simulations, lacking empirical evaluations on real-world datasets to support the theoretical claims.", "was_revised": true, "final_justification": "Though the experiments are too simple to capture the complexity of practical applications, the paper presents valuable insights from theoretical perspective, and lays a solid foundation for future research. I will keep my original score, and would like to support this paper.", "rebuttal": "Thank you for taking the time to review our paper. We hope our responses, and **a new set of experiments addressing the reviewers' concern about the gap from linear LAM to real (see the results in the response to the first reviewer)** might justify an uplift in score.\n\n***[W1] The paper is mathematically heavy.***\n\nThanks for this comment, since we would like to make the paper accessible to many readers. If the reviewer could mention any specific areas they thought were tricky, we’d be happy to add more intuitive explanations.\n\n***[W2] The analysis is based on simplified linear model with assumptions (Line 170)***\n\nWe have studied the effect of this assumption carefully. Only the analysis in Section 4.1 is based on the assumption in Line 170 (noise and action are uncorrelated with observation). This assumption leads to perfect learnability of LAM. We later relax this assumption in Section 4.2 to show that stochastic data collection policy matters in reducing the correlation between action and observation. Also note that the noise correlated with the observation can be regarded as the signal. \n\n***[W3] The paper lacks empirical evaluations on real-world datasets***\n\nThanks for this comment. At the request of the review, to provide evidence on whether our conclusions still hold on real-world datasets which are mainly image-based, we conducted a new set of experiments aimed at bridging the gap between linear LAM and real-world LAM. Please see the new experiment results in the response to the first reviewer (Reviewer VEZh).\n\n***[Q1] Is the analysis applicable when observations consist of multiple frames? Intuitively, capturing actions based on only two adjacent frames seems limiting.***\n\nOur analysis still applies when we simply redefine the vector observation $\\bf{o}$ to contain the information about the history, corresponding to the setting in Genie and LAPO. For example, the learned latent can still capture $\\bf{q} + \\bf{\\epsilon}$ in the idea case (cf. Proposition 4.1). The only difference is that the semantics of $\\bf{q} + \\bf{\\epsilon}$ may be different. Intuitively, the latent may capture the “acceleration” instead of “velocity” when an object moves across multiple frames. \n\nMoreover, even Genie and LAPO still aim at capturing movement between two frames (with additional input about the history). We find this may not be very limiting. Consider another option that extracts a latent to encode the change across multiple frames instead of extracting a latent between each two frames. However, we find that this formulation is not common in practice, maybe because such multi-frame movement can be easily represented as the concatenation of compact representations of two-frame movements or just the latent of two skip frames. \n\n***[Q2] All analysis relies on the additive transition function in Line 139. Is this assumption valid in practice?***\n\nThis linear assumption is a simplification required for tractability. Since our linear LAM architecture is not as complex as practical LAMs, it can only handle additive transition dynamics. We believe that when LAM is more capable (as in practice), it can handle more complex dynamics. Our setting (linear LAM + linear dynamics) is a good approximation of the real case (nonlinear LAM + complex dynamics). Our new set of experiments give evidence that our insights hold in settings when dynamics are not linear.\n\n***[Q3] I understand in the ideal case when z contains no information about the noise and observation, the LLO value is 2 in Line162, but why this is the optimal value of LLO.***\n\nFor a random variable $x$, $||\\hat{x} – x||^2 / \\text{Var}(x)$ is no smaller than 0 due to the non-negativeness of norm. It equals 0 when $\\hat{x}$ is a perfect prediction of $x$. Another notable condition is that $\\hat{x}$ is trained to predict $x$ to minimize the MSE loss appearing in the numerator. Since the worst prediction for $\\hat{x}$ would be predicting the mean, this term will be no larger than 1. We will make it clearer in our revision.\n\n***[Q4] Some typos. Line 195, should be A=(1-alpha)o ; Line 323, missing lambda in the objective.***\n\nThank you, we will fix the typo in Line 323. After checking we believe Line 195 is correct as stands, it should be $A=(1-\\alpha) I$ where $I$ is the identity matrix, and thus $A \\bf{o} = (1-\\alpha) \\bf{o}$ which can be used to cancel out the information $\\alpha \\bf{o}$ mixed in the latent."}, {"review_id": "JD1c826szY", "reviewer": "Reviewer_oyh9", "summary": "LAMs are becoming a models of choice, after LAPO paper, for pre-training embodied AI models. But a lot is not known about them, such as does learned latent really encode actions or just noise? Authors answer this question and others via linear LAM model that is easy to analyze theorerically. Having two different random variables, controllable changes q and exogenous noise \\eps gives possibilty to understand better what is going on in action prediction. Aurthors also experimentally analyze, via a set of simulations, how different conditions affect the action prediction capability.", "questions": "- In line 198, authors invert matrix B, but matrix B is not a square matrix. So you actually use pseudoinverse instead?\n-Section 4.1 Case 3, you only state that similar analysis of Case 1 can be conducted and what it results to. But I do not see reference to an appendix wheres such an analysis can be found. \n- Section 4.1 constains there cases, where Case 1 has no noise, Case 2 has stochastic noise and Case 3 has a systematic noise. Wouldn't it make sense to have fourth case, which has both stochastic and systematic noise? \n- Simulation results seem to produced with one seed, as there is stochasticity in the experimental procedure it is natural to ask whether different seeds to lead to different numbers? I am guessing that trends and conclusions would still hold.", "limitations": "Mostly yes, but I feel that authors should discuss more about how their results can carry over to practical LAMs.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Theorerical framework to analyze different aspects of LAMs. Analysis is from multiple perspectives, no exogenous noise to adding data augmentation. \n- Proofs in the appendices are easy to read and seem to contain all details. \n- I find results to be interesting and illuminating. \n\nWeaknesses: \n- It is not clear how results shown here carry over to practical LAMs. I think that is anyways the main point and motivation of this work. Text contains references to practical LAM papers in certain places where obtained results are similar to those in previous studies. This is good, but more effort would be needed here to make results practically useful. Appendix A is also too weak in this respect. Real difference is that dynamics model is not linear in practical LAM. About actions being Gaussian is a desing choice by authors. \n\n- How does linear LAM compare empirically to some practical LAM? Considering that paper already has quite a bit of simulation results it would not be a huge leap to add such a study to appendices. \n\n- Some results are just mentioned in the text, but no complete rationale is given in appendices, such as the difference between Eq. (5) and Eq. (6). Can Eq. (6) be interpreted as mutual information?", "was_revised": true, "final_justification": "Paper gives insight into LAMs by obtaining theoretical insights from the linear LAM. They answered my most critical issue by performing experiments where they show that with classic LAM they will obtain similar conclusions as with the linear LAM theory.", "rebuttal": "Thanks for your review. We hope **our extra work in providing new experiments that our results do carry over to practical LAMs (see the results in the response to the first reviewer)** may warrant an increase in score.\n\n***[W1] It is not clear how results shown here carry over to practical LAMs. I think that is anyways the main point and motivation of this work. Text contains references to practical LAM papers in certain places where obtained results are similar to those in previous studies. This is good, but more effort would be needed here to make results practically useful. Appendix A is also too weak in this respect. Real difference is that dynamics model is not linear in practical LAM. About actions being Gaussian is a design choice by authors.***\n\nWe agree that many real-world dynamics are not linear. This choice has been made to allow for tractable analysis and be compatible with linear LAM architecture (i.e., linear LAM can capture linear dynamics, and nonlinear LAM can capture more complex dynamics). Linear LAM design has two benefits: 1) Providing theoretical justification for tricks currently found to work well in practice (such as data augmentation and auxiliary action prediction). 2) Produces new insights into details affecting the learning of LAM (namely we noted the importance of data collection policy, which has not been mentioned in prior literature). \n\nTo evaluate how linear LAM captures the real-world LAM, we conduct further experiments as follows to validate that the main conclusion we obtained from linear LAM also applies to the setting where the dynamics is not linear and the action is not Gaussian. Please see the new experiment results in the response to the first reviewer (Reviewer VEZh).\n\n***[W2] How does linear LAM compare empirically to some practical LAM? Considering that paper already has quite a bit of simulation results it would not be a huge leap to add such a study to appendices.***\n\nThe experiment results presented in the response to [W1] indicate that our main arguments still hold for practical LAMs. \n\n***[W3] Some results are just mentioned in the text, but no complete rationale is given in appendices, such as the difference between Eq. (5) and Eq. (6). Can Eq. (6) be interpreted as mutual information?***\n\nEq. (6) approximates Eq. (5) by exploiting the connection between mutual information and the MSE loss between the target variable and the least squares estimate (LSE) of the target variable.\n\n***[Q1] In line 198, authors invert matrix B, but matrix B is not a square matrix. So you actually use pseudoinverse instead? -Section 4.1 Case 3, you only state that similar analysis of Case 1 can be conducted and what it results to. But I do not see reference to an appendix wheres such an analysis can be found.***\n\nYes, we use pseudoinverse in Line 198. For Section 4.1 Case 3, we can define $\\bar{X} = [X|Y] \\in \\mathbb{R}^{d_o \\times (d_a + d_b)}$ (column concatenation) and $\\bar{a} := [a^T|b^T]^T$ (row concatenation). Consequently, the results in Case 1 about $X$ and $a$ can be applied directly to $\\bar{X}$ and $\\bar{a}$ in Case 3. We will add the deduction to the revised appendix.\n\n***[Q2] Section 4.1 contains three cases, where Case 1 has no noise, Case 2 has stochastic noise and Case 3 has a systematic noise. Wouldn't it make sense to have fourth case, which has both stochastic and systematic noise?***\n\nThis is a very good question. Stochastic noise increases the reconstruction loss but does not change the semantics of the learned latent. Systematic noise, when intense enough, can change the semantics of the learned latent. It is a bit hard for us to obtain a clear analytical solution when combining them together, but intuitively they will change both the loss and the latent semantics. \n\n***[Q3] Simulation results seem to produced with one seed, as there is stochasticity in the experimental procedure it is natural to ask whether different seeds to lead to different numbers? I am guessing that trends and conclusions would still hold.***\n\nWe find that seed does not influence the results too much. To show this, we run vanilla linear LAM with $d_a = d_z = 8$ with 10 seeds and find that the standard deviation for linear LAM objective (LLO) is only 0.0173 (compared with the range for LLO [-2, +2]).\n\n***[F1] Line 226 \"Gaussion\" -> \"Gaussian\" -Grammatically sentences do not always parse. A bit more work in this respect is needed.***\n\nWe will further improve the paper based on the reviewer’s suggestion."}, {"review_id": "6BT3cjHcYq", "reviewer": "Reviewer_ojQq", "summary": "The authors introduce a linear abstraction of LAM (linear LAM) that keeps the standard inverse/forward-dynamics architecture but replaces the neural networks with linear maps. This preserves the training signal (reconstruction of observations), while making the model fully analysable. Using the linear form, they deliver theoretical results on the properties of a linear LAM and substantiate them using light-weight simulations that demonstrate the effects of data collection, over-parameterisation, data augmentation and weak supervision. The paper provides an insightful account of the things a linear LAM will learn under different conditions, contributing to the general understanding of the sort of conditions we should create for linear LAMs to learn effectively.", "questions": "1. I would appreciate if the authors could improve the writing in the paper; this would be essential for making it more accessible to the readers. (e.g. it is unclear to me what 'practical LAM' means when it is first introduced). \n\n2. L147: It seems that minimising mutual information with an observation should be argued more. Isn't it possible to imagine the case where certain actions are only taken in specific states?\n\n3. L230: The formulation \"of other agents\" is a bit confusing. Wouldn't this better be phrased as predictable changes that are not being caused by agent's actions?\n\n4. L279: \"Therefore, the analysis in this section suggest us to use the data collected by the policies with higher randomness whenever possible.\" Although the theoretical component is somewhat insightful, this point seems rather trivial at first glance. If the policy is deterministic, a world model would naturally be capable of implicitly modelling the behaviour of the agent?\n\n5. As mentioned in the Strenghts & Weaknesses section, I believe the authors should more clearly address the apparent gap between the presented results and the settings in which LAMs would most likely be applied (complex, high-dimensional settings; large-scale video data). If addressed convincingly, or if the authors can argue their case in the rebuttal, I would be happy to increase my score.\n\n6. IGOR paper claims: \"In practice, we found that the video quality has a big impact on the model performance. We exclude low-quality videos characterized by excessive shakiness or rapid camera movement, and apply stabilization techniques to the remaining videos.\" I may be mistaken, but this slightly contradicts the conclusions of your paper? Could this be initial evidence that your claims do not translate to more complex settings?", "limitations": "The aforementioned gap is recognised, but is not discussed sufficiently, in my opinion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "1. Quality:\n- (+) The work is technically sound, providing proofs where necessary.\n- (+/-) The theoretical claims about the properties of a linear LAM are well-supported using a number of light-weight experiments. Within the broader scope of LAM usage (mostly large-scale video data), it is unclear whether these claims remain impactful. No thorough investigation into this has been provided. \n- (+) The analysis of linear LAM is convincing and looks sufficient. \n- (-) The afore-mentioned weakness of the paper remains largely unaddressed in the paper. It would be great to see more justification for how this work may translate to more complex domains. \n\n2. Clarity: \n- (-) The writing could be substantially improved. It is often difficult to understand what is being said in the paper. \n- (+) The paper is well-organised and follows a good and clear structure. \n- (+) The paper contains a sufficient amount of information and details to reproduce the results. \n- (-) Figures could be improved for better readability. The y-axis should be labelled. \n\n3. Significance:\n- (-) At this point, it is unclear to me whether the analysis presented in the paper may be impactful for the community. My major concern is the vast oversimplification of the underlying model. Although the motivation for this oversimplification is clear (the model becomes analysable), it nevertheless generates a large gap between the results presented in the paper and the actionable significance of these results on the existing LAM methods. Primarily, can we trust these results to be applicable to complex, high-dimensional domains? To be clear, I do not necessarily think that they are not, rather it seems that the paper does little to address this concern in general. As such, it would be great to hear whether the authors believe these concerns could be addressed with more experiments in more complex settings, without taking away from the main claims of the paper.\n- (+) The paper does a good job of studying the linear LAM, demonstrating insightful results into its properties. \n\n4. Originality:\n- (+/-) While it seems conceivable that the results presented in the paper may be insightful with respect to the existing LAM methods, the point about applicability of these results to the existing LAM models remains unchanged. The authors should provide more details / justifications as to how their analysis may be applicable. What are the limitations? Where, if at all, do the assumptions break down? \n- (+) The work presents a first account of theoretical analysis of a linear LAM. The relevant citations are provided.", "was_revised": true, "final_justification": "I believe the paper has its merits. The theoretical analysis may be useful for future work in LAMs. \n\nMy main issue with the paper, lack of experiments showcasing that their results translate to complex and relevant datasets for LAM models, was not fully resolved with the new experiment. Nevertheless, the authors cited relevant work that justifies some of their theoretical results. The authors promised to fix the formating and writing issues. \n\nAll in all, I believe it's a borderline accept.", "rebuttal": "Many thanks for your detailed review. We have aimed to address your concerns, including **a new set of experiments bridging the gap between linear and practical LAM (see the results in the response to the first reviewer)**. We hope you might view this as sufficient to update the paper’s score.\n\n***[W1] Within the broader scope of LAM usage (mostly large-scale video data), it is unclear whether these claims remain impactful. No thorough investigation into this has been provided. The afore-mentioned weakness of the paper remains largely unaddressed in the paper. It would be great to see more justification for how this work may translate to more complex domains.***\n\nThank you for this, which we agree was an important point not fully addressed in the initial paper. The gap between linear LAM and LAM usage in practice can be summarized as 1) linear to nonlinear model architecture, 2) continuous to sometimes vector quantized bottleneck; and 3) linear to complex world dynamics in video data (cf. Appendix A). To justify how our conclusions can be translated to complex domains, we conducted an additional set of experiments. Please see the new experiment results in the response to the first reviewer (Reviewer VEZh).\n\n***[W2] The writing could be substantially improved. It is often difficult to understand what is being said in the paper.***\n\nApologies for not having had time to thoroughly edit the paper before submission. We commit to doing a full edit in the next version.\n\n***[W3] Figures could be improved for better readability. The y-axis should be labelled.***\n\nWe agree. Currently the y-axis is only labeled in the title, we will revise.\n\n***[W4] At this point, it is unclear to me whether the analysis presented in the paper may be impactful for the community. My major concern is the vast oversimplification of the underlying model. Although the motivation for this oversimplification is clear (the model becomes analysable), it nevertheless generates a large gap between the results presented in the paper and the actionable significance of these results on the existing LAM methods. Primarily, can we trust these results to be applicable to complex, high-dimensional domains? To be clear, I do not necessarily think that they are not, rather it seems that the paper does little to address this concern in general. As such, it would be great to hear whether the authors believe these concerns could be addressed with more experiments in more complex settings, without taking away from the main claims of the paper.***\n\nThank you, hopefully our experiments, shared above in W1 response, using nonlinear model architecture, vector quantized bottleneck, and image-based input, will help alleviate this concern.\n\n***[W5] The authors should provide more details / justifications as to how their analysis may be applicable. What are the limitations? Where, if at all, do the assumptions break down?***\n\nApart from the new experiment results, the reason why linear LAM would catch the essence of LAM in practice is because it captures the key designs including IDM, FDM, bottleneck and reconstruction loss. The assumption that the dynamics is linear is made to match the model capacity of linear LAM, and we believe this can scale to the scenario where both are nonlinear. The limitation of linear LAM is that it cannot shed light on the LAM research on deep learning techniques (e.g., cross-attention vs self-attention in IDM), empirical data selection (e.g., the best proportion for each real dataset), or optimization (e.g., the optimizer or the learning rate schedule). We argue that the value of our paper is not we can obtain results based on assumptions that may not hold. Instead, our value is that the conclusions drawn from our analysis can be applied to the practice (e.g., using more exploratory data to train LAM) or explain the observations in practice (e.g., the data augmentation technique).\n\n***[Q1] I would appreciate if the authors could improve the writing in the paper; this would be essential for making it more accessible to the readers. (e.g. it is unclear to me what 'practical LAM' means when it is first introduced).***\n\nSee W2 response. We will add a practical LAM definition.\n\n***[Q2] L147: It seems that minimising mutual information with an observation should be argued more. Isn't it possible to imagine the case where certain actions are only taken in specific states?***\n\nMinimizing mutual information with observation is still a proper objective, even if there are specific states where only certain actions can be taken. In this case, it is true that the mutual information between latent and state cannot be zero since we can infer the state from the latent. However, minimizing this mutual information still helps LAM to discover action representations to be as independent with states as possible so that it can be shared across states. Note that the ultimate objective for LAM is to learn shared action representation with consistent semantics (e.g., the latent could be “moving the arm left if not blocked”).\n\n***[Q3] L230: The formulation \"of other agents\" is a bit confusing. Wouldn't this better be phrased as predictable changes that are not being caused by agent's actions?***\n\nWe agree this term could be changed to improve clarity. We propose to use use “predictable noise” in our revision – please let us know if this satisfies your concern.\n\n***[Q4] L279: \"Therefore, the analysis in this section suggest us to use the data collected by the policies with higher randomness whenever possible.\" Although the theoretical component is somewhat insightful, this point seems rather trivial at first glance. If the policy is deterministic, a world model would naturally be capable of implicitly modelling the behavior of the agent?***\n\nOur main observation is that, when the action is easily predicted from the current state, there is less incentive for the latent to encode the action, with the FDM (or a world model as the reviewer mentioned) predicting the next state directly from the current state. Hence, we should prefer a higher entropy policy for the purpose of training a LAM. Like all good insights, as you note, this is intuitive, but we only made this discovery through our analysis, and do not believe related literature has made this observation.\n\n***[Q5] As mentioned in the Strenghts & Weaknesses section, I believe the authors should more clearly address the apparent gap between the presented results and the settings in which LAMs would most likely be applied (complex, high-dimensional settings; large-scale video data). If addressed convincingly, or if the authors can argue their case in the rebuttal, I would be happy to increase my score.***\n\nIn response to the reviewer’s concern about the gap, we have provided additional experiment results in the response to the first reviewer (Reviewer VEZh)..\n\n***[Q6] IGOR paper claims: \"In practice, we found that the video quality has a big impact on the model performance. We exclude low-quality videos characterized by excessive shakiness or rapid camera movement, and apply stabilization techniques to the remaining videos.\" I may be mistaken, but this slightly contradicts the conclusions of your paper? Could this be initial evidence that your claims do not translate to more complex settings?***\n\nNo. Filtering data is an orthogonal technique from data’s perspective. Combining with the information that IGOR also uses data augmentation (advocated by our paper), we conclude that camera shakiness could be an issue and both the data augmentation (to help LAM robust to gentle shakiness) and data filtering (to avoid LAM suffer from excessive shakiness) are needed."}]

Author rebuttal:

Meta-review: The paper presents a theoretical analysis of Latent Action Models (LAMs) using a linear framework. The analysis shows that LAMs learn representations of high-variance changes between video frames. This provides a basis for understanding why techniques such as data augmentation and auxiliary action prediction are effective.

The work is one of the first theoretical studies of LAMs. The primary concern from all reviewers was whether findings from the linear model would apply to the non-linear LAMs used in practice. During the rebuttal, the authors provided new experiments using a non-linear, CNN-based LAM. The results were consistent with the conclusions from the linear analysis. 

The AC agrees with the consensus from reviewers, and recommends acceptance.
Decision: Accept (poster)