ICML 2025

Latent Action Learning Requires Supervision in the Presence of Distractors
Alexander Nikulin 1 2 Ilya Zisman 1 3 4 Denis Tarasov 1 Nikita Lyubaykin 1 5 Andrei Polubarov 1 3 4
Igor Kiselev 6 Vladislav Kurenkov 1 5
Abstract
Recently, latent action learning, pioneered by
Latent Action Policies (LAPO), have shown re-
markable pre-training efficiency on observation-
only data, offering potential for leveraging vast
amounts of video available on the web for em-
bodied AI. However, prior work has focused on
distractor-free data, where changes between obser-
vations are primarily explained by ground-truth
actions. Unfortunately, real-world videos contain
action-correlated distractors that may hinder la-
tent action learning. Using Distracting Control
Suite (DCS) we empirically investigate the ef-
fect of distractors on latent action learning and
demonstrate that LAPO struggle in such scenario.
We propose LAOM, a simple LAPO modification
that improves the quality of latent actions by 8x,
as measured by linear probing. Importantly, we
show that providing supervision with ground-truth
actions, as few as 2.5% of the full dataset, dur-
ing latent action learning improves downstream
performance by 4.2x on average. Our findings
suggest that integrating supervision during Latent
Action Models (LAM) training is critical in the
presence of distractors, challenging the conven-
tional pipeline of first learning LAM and only
then decoding from latent to ground-truth actions.
1. Introduction
Recently, a new wave of approaches based on latent action
learning has emerged (Edwards et al., 2019), demonstrating
superior pre-training efficiency on datasets without action
labels in large-scale robotics (Ye et al., 2024; Chen et al.,
2024b;a; Cui et al., 2024; Bruce et al., 2024) and reinforce-
ment learning (Schmidt & Jiang, 2023). Latent Action Mod-
1AIRI 2MIPT 3Skoltech 4Research Center for Trusted Artificial
Intelligence, ISP RAS 5Innopolis University 6Accenture. Corre-
spondence to: Alexand...