Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations
Authors: Xin Liu
Haoran Li
Dongbin Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Imitation learning from videos
Online policy learning
Behavior cloning
Unsuerpvised learning
Visual reinforcement learning
Primary area: reinforcement_learning
TL;DR: We achieve sample-efficient visual policy learning under the supervisions of only expert videos, without accessing rewards or expert actions.

Abstract: Humans can efficiently extract knowledge and learn skills from the videos within only a few trials and errors. However, it poses a big challenge to replicate this learning process for autonomous agents, due to the complexity of visual input, the absence of action or reward signals, and the limitations of interaction steps. In this paper, we propose a novel, unsupervised, and sample-efficient framework to achieve imitation learning from videos (ILV), named Behavior Cloning from Videos via Latent Representations (BCV-LR). BCV-LR extracts action-related latent features from high-dimensional video inputs through self-supervised tasks, and then leverages a dynamics-based unsupervised objective to predict latent actions between consecutive frames. The pre-trained latent actions are fine-tuned and efficiently aligned to the real action space online (with collected interactions) for policy behavior cloning. The cloned policy in turn enriches the agent experience for further latent action finetuning, resulting in an iterative policy improvement that is highly sample-efficient.
  We conduct extensive experiments on a set of challenging visual tasks, including both discrete control and continuous control. BCV-LR enables effective (even expert-level on some tasks) policy performance with only a few interactions, surpassing state-of-the-art ILV baselines and reinforcement learning methods (provided with environmental rewards) in terms of sample efficiency across 24/28 tasks.  To the best of our knowledge, this work for the first time demonstrates that videos can support extremely sample-efficient visual policy learning, without the need to access any other expert supervision.

Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations
Xin Liu 1.2, Haoran Lil.2,, Donghin Zhao
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences (liuxin2021, lihaoran2015 dongbin chao]@ia ac. cn
Abstract
Humans can efficiently extract knowledge and learn skills from the videos within only few trials and errors. However, it poses a big challenge replicate this learning process for autonomous agents, due to the complexity of visual input, the absence of action or reward signals, and the limitations of interaction steps. In this paper, propose novel, unsupervised, and sample-efficient framework to achieve imitation learning from videos (ILV), named Behavior Cloning from Videos via Latent Representations (BCV-LR), BCV-LR extracts action-relate la- tent features from high-ding high-dimensional video inputs through self-supervised tasks, and then leverages dynamics- based unsupervised objective predict latent actions between consecutive frames. The pre-trained latent actions are fine-tuned and efficiently aligned the real action space online (with collected interactions) for policy behavior cloning. The cloned policy in turn enriches the agent experience for further latent action finetuning, resulting in iterative policy improvement that is highly sample-efficient. We conduct extensive experiments on a set of challenging visual tasks, including both discrete control and continuous control. BCV-LR enables effective (even expert-level on some tasks) policy performance with only few interactions, surpassing state-of-the-arr ILV baselines and reinforce- ment learning methods (provided with environmental rewards) in terms of sample efficiency across 24/28 tasks. To the best of our knowledge, this work for the first time demonstrates that videos can support extremely sample-efficient visual policy learning, without the need access any other expert supervision.
Introduction
Reinforcement learning (RL) 1
3 has demonstrated powerful capabilities in solving decision-
5 7 However, the stringent training conditions, including well-designed rewards and substantial environmental interactions, have greatly restricted the appli- cation scenarios of RL 8 9 [O]. Behavior cloning from action-labeled expert trajectories [2]
making tasks across different fields
or offline RL from exploratory training experience [ 14 provide solutions for policy learning without any environmental interactions, which means the highest sample efficiency. Nevertheless, the required offline supervision of high quality generally not easily accessible.
Compared with well-designed rewards or qualified offline datasets, videos are a kind of supervisory information that much easier obtain. Imitating skills from expert videos is also a natural and efficient learning method for humans, e.g., playing games by watching tutorial videos on the Internet. However, in contrast to humans' ease learning from videos, it no simple feat to replicate this
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

efficient watch-and-lean process for autonomous agents, which are mainly attributed to the complex visual input and missed action labels. Although numerous studies have already utilized videos as a
valuable supplement to boost policy learning from expert rewards [5 16 17 18 or expert actions
19 20 211, imitating entirely from videos without any other supervision is more natural and ideal
learning mode.
Formally, this problem is named Imitation Learn- ing from Videos (ILV). Currently, the mainstream
Discrete control 1Bosstght
approach to solving ILV inverse RL [2 23 24
eopert videos
Environment rewards)
These methods aim to extract reward signals from videos that are highly consistent with expert poli- cies, and then perform imitation through RL. How- ever, the training of an extra reward prediction network and the RL value network both necessi- tate extensive exploration of the observation space, which inevitably leads to lower sample efficiency than traditional RL methods (based on steady ex- Another cate- gory of approaches 226 27 aims to predict the missed expert actions based on environmental in-
ILV
RL
11 BCV-LR (sample-efliclent
BOULE JAEWN
BDC
Continucu: ceniral hand-1000)
ecpert viteos
ILV
RL
Videos (no actions)
pert rewards) in many cases 25
BCH44 6WW
800
1MF0
160
Figure 1: BCV-LR achieves sample-efficient video-based imitation learning without access- ing expert actions rewards. It achieves expert- level policy performance on discrete task "Boss- fight" and continuous task "reacher hard"' with only 100k interactions allowed, surpassing state- of-the-art ILV and RL baselines.
teractions, conducting supervised learning (i.c., be-
havioral cloning) to imitate policies (8 [29] Such
methods require less knowledge of environment dynamics [25]. demonstrating the potential re- duce reliance on environmental samples [26] (77 in state-based imitation. However, when it comes to video imitation, both the difficulty of interpret- ing observations and predicting expert actions in-
crease sharply. Existing supervised imitation methods often encounter performance bottlenecks and are unable to learn policies close to the expert level (28
The above discussion naturally leads to a question: Is it possible to balance the effectiveness and sample efficiency in visual policy learning, where only videos are accessible supervisions? In this paper, we try to answer this question through Behavior Cloning from Videos via Latent Representations (BCV-LR), which conducts offline latent pre-training extract rich knowledge from videos for efficient adaptation to real environments online. Concretely, BCV-LR contains an offline pre-training stage and an online finetuning stage. In offline pre-training, BCV-LR first pre-trains self-supervised visual encoder over the video, aiming to extract the action-related information from raw pixels and thus alleviating the learning difficulty in subsequent training. Based on the latent features, BCV-LR employs another trainable world model. optimizing dynamics-based objective in an unsupervised manner, to obtain the latent actions between consecutive video frames. In the online stage, BCV-LR fine-tunes the latent actions with the pretrained world model over the collected reward-free transitions, simultaneously aligning these implicit actions to the real action space for behavior cloning of a policy. The cloned policy in turn enriches the collected experience, leading to better latent action finetuning and decoding, which finally results in an iterative policy improvement that is extremely sample-efficient
We conduct extensive experiments on a of challenging visual control tasks, including 16 discrete control tasks from the Procgen benchmark (30 and 12 continuous control tasks from the Deepmind Control suite (DMControl) (31] and Metaworld [32) Even with only a few environmental interactions permitted, BCV-LR can still enable effective and even expert-level policy learning performance on many tasks, which both recent advanced ILV basclines (provided with expert videos) and RL methods (provided with expert rewards) cannot achieve, as shown in Figure I art sample efficiency of the proposed BCV-LR. To the best of our knowledge, the proposed BCV-LR for the first time demonstrates the feasibility of using videos as the only expert supervisory signal to guide extremely sample-efficient visual policy learning. We provide the implementation of BCV-LR at /bttpp:://ithub
These indicate the state-of-the
We summarize the contributions of this paper as follows:

---

We propose novel framework for sample-efficient ILV, named Behavior Cloning from Videos via Latent Representations (BCV-LR). To the best of our knowledge, our work for the first time demonstrates that videos can support extremely sample-efficient visual policy learning, without the need for expert actions or rewards.
BCV-LR extracts action-related latent features from nigh-dimensional video frames through self-supervised tasks, and then leverages dynamics- based unsupervised objective to predict latent actions between consecutive images. With online interactions, the pre-trained latent actions are fine-tuned and aligned to the real action space for behavior cloning. The cloned policy in turn enriches the agent experience for better latent action finetuning, resulting in an iterative policy improvement that is highly sample-efficient
We conduct extensive experiments on a set of challenging visual control tasks, including both discrete control and continuous control. The results demonstrate that the proposed BCV-LR exhibits state-of-the-art sample efficiency, surpassing recent advanced online visual policy learning baselines, including both ILV methods and RL methods.
2 Related Works
2.1
Sample-Efficien Visual Policy Learning
Interacting with the environment to collect data is an expensive and dangerous process in many scenarios 33 &. which makes sample efficiency an important metric for policy learning methods 38) employs extra auxiliary tasks in addition to the RL objective, accelerating downstream policy learning by improving visual understanding capabílities in a target manner. Model-based RL methods [39 40 [1 42 (3) train extra world models that augment the RL experience, thereby reducing the requirement of real interactions 44] 45 46]. These methods indeed achieve higher sample efficiency, but they are inherently limited by the non-trivial hand-craft rewards [47] At the same time, some approaches try completely avoid accessing environments, such as conducting offline RL over the training experience [14] 13 [5] and behavior cloning with
34 28 Self-supervised RL [35 36 37
action-labeled expert demonstrations [12 ПI. However, these offline expert datasets of high quality can be expensive and even unavailable in many scenarios.
2.2 Utilizing Videos as Supervisions
With the continuous development of the Internet, obtaining vast number of diverse videos has become extremely convenient nowadays. Many of these videos contain expert-level demonstrations, which serve as valuable source of knowledge for both humans and machines. However, due to the lack of action information, it poses a significant challenge to well utilize these videos in machine learning. One popular way is to improve existing strategy learning methods with the help of videos, such as video-based pre-training for RL 48 21,, video-based intrinsic reward for RL [17 55). and video-based behavior cloning with expert actions [19] 20 In addition, FICC (5T) pre-trains world models on action-free agent experience, accelerating model-based RL on Atari games. JPET (52) utilizes mixed dataset containing videos achieve one-shot visual imitation. Although these methods have achieved certain success, they still require dependence on other expert supervisions. In
contrast, ILV (22 47 50 28, which aims to acquire skills from only videos in a manner similar to humans, has a much broader range of application scenarios.
2.3 Imitation Learning from Videos (ILV)
Imitation Learning from Observations (ILO) 23 26
is proposed to recover the expert policies hidden in the frreon-free demonstrations. As an extension of ILO to the visual domain, ILV poses a greater challenge due to the more complex visual inputs. The mainstream approaches solving the ILV problem are inverse RL [50 47 53
55), which tries to extract and optimize the reward signals contained in the videos. They achieve considerable results on imitation performance but usually can't 't well balance sample efficiency [25 28). This conforms to our intuition, because both reward prediction and RL require advanced understanding of the environments, which cannot be achieved without sufficient interactions. At the same time, some researchers give up reward engineering, trying to predict expert actions from videos
54
26 56 27 for supervised II... They achieve considerable success but suffer from severe performance drops when transferred into more
3

---

complex visual continuous control [29 28 In this situation, BCV-LR for the first time (i) taps into the potential of supervised ILV and (ii) exhibits the feasibility of using videos as the only supervisory signal to guide highly sample-efficient visual policy learning.
3 Behavior Cloning from Videos via Latent Representations
Problem Definition of ILV The goal of ILV is to imitate the policy contained in the expert videos by interacting with a reward-free environment. Concretely, the expert videos can be regarded as dataset containing action-free expert transitions {(of, 1f+1)) The observation o can represent a single frame or stacked historical frames, depending on the partial observability of the specific task. The interactive environment can be denoted as reward-free free Markov Decision Process (MDP) M = (O, A, P. .o),, where O is the visual observation space, AA is the action space, P is the observation transition function, and dg the distribution of the initial observation. Based on the reward-free environmental transitions {of af of+1) and the videos {(o", of }} the ILV agents can achieve policy learning by estimating the expert rewards or expert actions [2 50].
Framework of BCV-LR BCV-LR addresses ILV problem by estimating the expert actions con- tained in the videos. The predicted actions used to obtain a policy through behavior cloning. BCV-LR contains two stages: an offline pre-training stage and an online finetuning stage.
In offline pre-training, BCV-LR first pre-trains a self-supervised visual encoder f over the videos, aiming to extract the action-related information from raw pixels and thus alleviating the learning difficulty for both action prediction and policy cloning. Based on the pre-traines latent features, BCV- LR employs another trainable world model w along with the latent action predictor p. optimizing sramics-based objective in an unsupervised manner. This aims obtain the latent actions between consecutive video frames. In the online stage, BCV-LR fine-tunes the latent actions with the pretrained world model w over the collected reward-free transitions, efficiently aligning these pseudo actions to the real action space via a latent action decoder d. The transitions are collected by imitating latent policy æ that clones the latent actions based latent features from expert videos. is combined with latent feature encoder f and latent action decoder d to interact with the environment. As latent actions are finetuned, the cloned latent policy improves simultaneously, which in turn collects transitions at higher performance level for better latent action finetuning, resulting in an iterative policy improvement. After the online stage. f. π, and d together form the final policy of BCV-LR. We provide diagram in Figure 2 and pseudocode in Appendix A.
3.1 Offline Pretraining on Action-free Videos
To facilitate the utilization of environmental interactions in online learning, BCV-LR firstly extracts knowledge from videos offline. We describe how to pre-train a feature encoder f to extract useful information from high-dimension videos in Section 3.1.1, and how to train latent action predictor p and a world model w jointly through an unsupervised dynamics-basee objective in Section 3.1.2.
3.1.1 Learning Latent Features through Self-Supervised Tasks
To accurately predict the missed actions from the expert videos and support effective downstream policy, the feature encoder should be able to extract information related decision-makinn from complex high-dimensional visual inputs, which is consistent with the requirements in visual RL. Inspired by the recent success of self-supervised RL [35] 57 34, , BCV-LR learns its feature encoder I' by optimizing self-supervised objectives defined on the expert videos. Note that the proposed BCV- LR is compatible with any action -fre self-supervised tasks. By choosing appropriate self-supervised objectives, it can adapt different types of visual control tasks casily. In addition, BCV-LR can also be combined with a well-trained, off the-hlf encoder, which enables it to retain the potential for handling tasks with much more complex visual inputs.
For tasks with relatively simple dynamics but involving complex visual information (e.g., Proegen video games [30), BCV-LR optimizes contrastive learning objective to align two different randomly shifted images of the same observation together. This makes f focus more on relative position differ- ence that is highly related to actions. In practice, we empirically find that letting / simultaneously achieve a self-reconstruction task yields further improvement in Procgen. Concretely, a batch of observations {o N Ji=1 are sampled from videos. Each of is randomly shifted twice to obtain two

---

Oftine pre-training
Output
ladent (update pud)
claning [update
(update
(updatee P.W)
42
0%
BCV-LR policy
The training objectives of different stages, BCV-LR first pre-trains a self-supervised feature encoder f over the videos. Based on the latent features, BCV-LR employs another trainable world model w along with the latent action predictor p. optimizing a dynamics- unsupervised manner obtain the latent actions between consecutive video frames. In the online stage, BCV-LR fine-tunes the latent actions with the pretrained world model w over the collected reward-free transitions, aligning latent actions to the real action space via latent action decoder d. Simultaneously, BCV-LR trains a latent policy æ that clones the latent actions, which shares the latent feature encoder and latent action decoder d to interact with the environment. This enriches collected data for further latent action finetuning, resulting in an iterative improvement. Note that f, æ,, and d together form the final policy of BCV-LR.
Figure 2:
objective in an
augmented images aug(o,) and aug' (o They then separately encoded by f and its momentum encoder I' (updated via Exponential Moving Average (EMA) (58) to obtain two latent features, s and syr The self-supervised target contains contrastive learning loss and self-reconstruuctinn loss:
exp(u(s)) Wsy') "xp(u(s")TTs"") + allu(s)) aug(o,))
Cif = ---l0p log
Σ Txp(u(s))TT))) Wsy))
where W is a trainable matrix employed in contrastive learning, a is a coefficient scaling two losses, u(-) is Multi-Layer Perceptron (MLP) set to introduce asymmetry for avoiding collapse to trivial solutions, and v(-) is a trainable reconstruction decoder.
As previously mentioned, BCV-LR can select self-supervised tasks tailored to different domains to achieve better latent feature extraction. For partially observable domains with complex dynamics (e.g., continuous DMControl [31),, understanding the temporal information is proven crucial for the representation module to support effective downstream policy learning 59
34
60 This motivates
us to employ recent advanced prototype-based temporal association task (57 43 based on the
Sinkhorn- Knopp [61] algorithm, which aligns two temporally neighboring observations together in the latent space. For the sake of fluency of the main text, put the details of this self-supervised temporal association task in Appendix A.
3.1.2 Extracting Latent Actions over Latent Features
With the pre-trained encoder f. each pixel observation of is encoded into the latent feature sy. BCV- LR then trains a predictor p to extract the latent action z between neighboring feature pairs (sf, 1) To obtain 2! in an unsupervised manner, another world model is employed to reconstruct the next feature + based on the current feature s and the predicted z!". The motivation intuitive: the latent action should contain information that describes the changes between frames. To avoid the trivial solution that p learns to naively copy the next latent features into latent actions, the continuous z! is discretized via vector quantization (VQ) [62] before entering the world model 28 19]. Concretely, z is mapped to its nearest vector 2," defined in a limited codebook, which forces the model to find commonalities among numerous transition pairs. The 2;" will replace z achieve forward propagation in world model w, and the reconstruction loss can be defined as the following:
Lia = [| 2,"1) 11111
(2)
In hackpropagation, the gradient of z! is copied to z directly. p and w are optimized jointly by Lia- In addition, p is also updated along with the codebook in the VQ training process [21].
5

---

3.2 Online Finetuning with Reward-free Interactions
In the online learning stage, BCV-LR is able to quickly adapt the pre-trained knowledge the real environment, thus achieving sample-efficient policy learning. In Section 3.2.1, we describe how to finetune and align the latent actions to real action space with the latent action decoder d. based on agent experience. By sharing ʃ and d, BCV-LR clones the latent actions via latent policy to interact with the environments, shown in Section 3.2.2. This policy in turn enriches the agent experience, resulting in an iterative improvement of both latent action predictor and latent policy After the online stage, f, n, and d together form the final policy of BCV-LR.
3.2.1 Finetuning and Decoding Latent Actions with Pre-trained World Model
BCV-LR interacts with the environment to collect environmental transitions with real actions {(of, af, 1)1) By letting latent action predictor p and decoder predict real actions over these environmental transitions, BCV-LR further finetunes while aligning the predicted latent actions to the real action space with decoder d. Considering that there are distribution differences between the collected transitions and expert videos, BCV-LR simultaneously utilizes the pre-trained world model wr to provide expert dynamics constraints. This forces the prediction model p to maintain its understanding of environmental dynamics during adaptation to non-expert action labels, enhancing the finetuning robustness and achieving better results in practice. Concretely, each environmental observation pair (of, o(+1) is encoded by the mre-trained f into the latent features ( ). They are further passed into p to generate the predicted latent action z The learning objective contains (i) the action prediction loss and (ii) the future reconstruction loss over the collected transitions:
e'' -a, (=ffstaaax((((23)) + B1|w(s 21 11
(3)
where the a coefficient scaling two losses and =;" is the codebook vector nearest to zf. BCV-LR finetunes p and learns d simultaneously through backpropagation. p and the codebook are also updated through VQ training. Finetuning w is optional, leading to better performance on some tasks in practice. The cross-entropy loss is replaced with Mean Square Error (MSE) in continuous control.
3.2.2 Learning Latent Policy via Behavior Cloning
With the predicted actions aligned into the real action space, BCV-LR can imitate a policy directly executed in the environment from expert vidcos. By sharing the pre-trained feature encoder and the latent action decoder d, BCV-LR only needs to train a latent policy æ that maps the latent features into the latent actions. Concretely, feature encoder f to produce (s%, 1). and the latent action predictor p to produce the predicted latent action 2 the label. The behavior cloning objective is correspondingly defined as:
expert video transition 0 is successively fed into the
Lb ( =
(4)
The ouutppt latent action is then decoded the real action space by d for execution. After the online stage, f, n, and d together form the final policy of BCV-LR.
Experiments
In this section, we provide comprehensive evaluation of the proposed BCV-LR. We first briefly introduce the experimental settings (Section 4. I). then provide the results of discrete control and continuous control (Section 4.2 & 4.3), conduct ablation studies (Section 4.4), and provide analytical experiments on video data efficiency and multi-task adaptation (Section 4.5 & 4.6). You can refer to Appendix B for more results and analysis, and Appendix for detailed experimental settings.
4.1 Experimental Settings
4.1.1 Environments
We test the performance of BCV-LR in a set of challenging domains with complex visual inputs and environment dynamics, including 16 discrete control tasks from Procgen benchmark [30] and 8

---

Table 1: The interaction liminte policy learning in Procgen. Italics indicate using expert videos, and underlines denote using environmental rewards. Bold text indicates the highest score excluding video experts. BCV-LR exhibits the highest sample efficiency, utilizing only videos.
Task
BCV-LR (ours)
UPESV 28
BCO 26
ILPO 27
LAIFO 50
LAPO 21
PPO 63
Export Vidcos
Bigfish
38.9
30.5 1.6
3.7
0.8
0.8
2.7
0.9
36.3
Maze
9.9 0.1
0.2
2.4
0.3
4.4
9.6
5.0
10.0
Heist
9.1
0.3
76
6.7
5.4066
9.40.3
3.7
9.7
6.7
Plunder
4.2
3.7
4.2
2.2
32
4.1
4,4
11.3
Dodgehall
12.4
9.1
54+11
0.6+0.1
08
5.9
1.1
13.5
Jumper
7.5
66=022
6.4033
3.1
39
73+02
8.5
Climber
9.6
6.6
3.3
3.40.5
27
4.7
2.2
10.2
Fruitbot
1.5
1.66
3.5
-.70
-2.5
0.3
-19
29.9
Starpilot
54.8
15.0
1288
0.5
2.7
1.6
2.6
67.0
Ninja
0.3
4.211
9.5
Miner
11.6
9.3+
5.8
1.2
1.2
6.7
1.2
11.9
Cavellyer
4.6
0.6
28
3.2
24
3.9
3.0
9.2
Leaper
Chaser
3.5
0.1
0.
0.7
0.6
0.8
0.4
10.0
Bossfight
10.3 0.3
2.0
0.3
0.1
0.3
0.3
0.1
11.6
Mean
13.8
9.0
4.8
2.1
2.2
6.8
2.3
16.6
Video norm Mean
0.79
0.58
0.38
0.22
0.22
0.48
0.22
1.00
continuous control tasks from Deepmind Control suite (DMControl) [31] Procgen 130 provides diverse set of procedurally generated video game environments, each with unique challenges, different dynamics, and especially changing visual styles. DMControl [31] provides series of challenging robot control tasks. Although the visual changes are not as complex as those in Procgen, the continuity of the observation and action spaces, well as the partial observability of the states, greatly increase the difficulty of understanding the environmental dynamics. The complex visual inputs and environmental dynamics make these two benchmarks extremely challenging for ILV problems, which recognized and widely employed by recent advanced studies [0] [4 28 47). In addition, we
also provide results 4 robotic manipulation tasks from Metaworld 32 demonstrate wider
application of BCV-LR, which is detailed in Appendix B.
4.1.2 Baselines
We employ several popular and advanced ILV baselines: UPESV 228 LAIFO 50
ILPO 27
and BCO [26]. LAIFO is a recent advanced inverse RL approach that derives rewards from the expert videos through adversarial imitation techniques. The other three methods all try recover the missed actions from the expert observations (videos) and obtain the imitated policies via behavior cloning, which is similar to the proposed BCV-LR. Among them, UPESV and LAIFO have achieved state-of-the-art ILV performance discrete Procgen and continuous DMControl, respectively.
To provide comprehensive comparison, further employ several online RL baselines provided with environmental rewards. The advanced RL algorithms vary across different benchmarks. For discrete tasks, LAPO 21 and PPO (63) are employed. PPO is the most popular RL baseline in Procgen, while LAPO further leverages expert videos to improve PPO by pre-training latent policy, For continuous tasks, employ DrQv2 3 and TACO [60]. DrQv2, which combines several image-oriented tricks with DDPG, currently the most popular RL method for continuous visual control. TACO is state-of-the-arr supervud RL method, which improves DrQv2 via several visual auxiliary objectives. All of these baselines are recognized as leading RL methods in the corresponding benchmarks.
Note that the ILV methods, including the proposed BCV-LR and four baselines (UPESV, BCO, ILPO, and LAIFO), can only access expert vidcos. We directly compare the RL methods and the ILV methods in terms of online sample efficiency, but in fact, they rely on different supervisory information.
4.2 Discrete Control
In this section, we test the proposed BCV-LR in 16 discrete control tasks from Procgen. We choose "casy" mode and "full" distributions of levels across all tasks. On each task, only 100k environmental

---

point mass cady
J0CO 230 dght
Tinger Spin
1020
100
610
400
29032
100007
370000
ball CaRch
reach bottom iff
cheetah run backward
Teacher easy
000
125
233
600
600
203
150
400
200
50
103
53
20033 40302 ROSDE 83000 100000
20300 40000 62020 00000 IE00C
20000 46000
50000
00000 220000
20000 43000 GcEGO GOOCE LO3R00 Enwicemena zapo
En staps
BOVAUR (ourS)
LAIPO
BCD
LIFEV
Figure 3: Online training curves of ILV methods in DMControl. BCV-LR can efficiently utilize environmental samples and learn effective strategies at 50k steps (even 20k on some tasks).
Table 2: Comparison with both ILV and RL baselines on DMControl. Italics indicate using expert videos, and underlines denote using environmental rewards. Bold text indicates the highest score excluding video experts. BCV-LR exhibits the leading sample efficiency under supervisions of only videos.
Task
BCV-LR (ours)
LAIFO
BCO 26
UPESV
TACO 60
DrQv2
Experi Videos
point mass reacher haad hard jacn, reach, rightight
800 25
0
8 12
0
712 96
525 189
885
900 31
110 71
262 8:
8
154 68
92 + 98
967
finger_spin
942 48
108 96
881 1:39
0
618
374 264
981
ball_in_up catch
807 57
350 + 271
404 363
100 81
321 + 166
246 137
986
jaco recccreahh_ao Jottom_
123 39
93
7+ 10
12 16
4
23
203
cheetah backward
285
253
0
304 175
332
389
reacher easy
897 38
215 47
198 + 189
0
244 98
225 + 91
975
Mean Video-norm Mean
60M
158
310
232
698
0.78
0,20
0.31
0.03
0.45
0.34
1,00
steps are allowed. This number is much smaller than 4M or 25M employed by advanced RL works 30 64 211, requiring extremely high sample efficiency. The expert video dataset containing 8M steps is generated by well-trained RL agents, provided by (2T) For RL methods, we allow them to access the expert reward provided by Procgen environments. These all follow recent advanced works settings.
21 28
Refer Appendix C for detailed
The results are shown Table Compared with ILV methods based on behavior cloning (UPESV, BCO, and ILPO), the proposed BCV-LR achieves consistent leadership across all 16 tasks, indicating a more accurate and robust action prediction for better behavior cloning. By contrast, the inverse RL method LAIFO shows less satisfactory performance. Given that the diverse visual styles of the Procgen pose significant challenges for reward prediction [54] (33, we directly make comparisons with state-of-thh-an RL methods accessing ground-truth environmental rewards, and BCV-LR still demonstrates a clear advantage. The results fully demonstrate the superiority of BCV-LR in terms of sample efficiency, against different kinds of online policy learning approaches. Additionally, BCV-LR achieved an average of 79% of expert performance across all tasks, reaching expert levels in many tasks such "Maze", "Bigfish", "Fruitbot", and "Starpilot", These for the first time demonstrate that videos can support extremely sample-efficiee visual policy learning, without the need of accessing expert actions rewards.
4.3 Continuous Control
In this section, we provide the comparison on 8 continuous visual tasks from DMControl. On each task, only 100k environmental steps (50k interactions with action repeat set to 2) are allowed. For ILV methods, employed well-trained RL agent (IM steps) to collect 100k transitions expert

---

50
32
22
poltt
1000
205 rECC Sotcom
3000
TLaC Sel Gasy
630
380
201
20030 RAGD 4E8X) 100062
200E 40203 SASE BOORO
J5000 SLICE KOOD SLECE 100683
2008 42003 GUDR 12000
BOVAR CourS) Wile
Wile
Figure 4: Ablation study on both discrete control and continuous control.
videos. The RL methods permitted to access the environmental rewards. Refer Appendix C for detailed hyper-parameter settings.
We first provide the comparison between ILV methods and show their training curves in Figure 3 For the inverse RI method LAIFO, both reward prediction and value estimation require advanced understanding of the environments. It can deal well with continuous control when given enough interactions but can't perform well with limited samples. Compared with the other supervised ILV methods, BCV-LR still exhibits huge advantages, which is similar to results in discrete control. In Table 5 we further provide the numerical evaluation results sample-efficient visual RL baselines that are provided with ground-truth rewards. Compared with these advanced RL methods, BCV-LR is still better on 6/8 tasks at 100 steps. In addition, BCV-LR able to learn effective policies across all tasks at only 50k environmental steps (even 20k for some tasks), which RL baselines cannot achieve (refer to Appendix B). In summary, the results in this section further verify (i) the superiority of the proposed BCV-LR and (ii) the potential of videos solitary supervision.
In addition to DMControl, we also conduct experiments on continuous manipulation tasks from the Metaworld benchmark. Refer to Appendix B for results and details.
4.4 Ablation Study
In this section, we ablate each key component of BCV-LR to show the effect on the ILV perfor- mance, as shown in Figure 4 In "BCV-LR w/o If", we give up self-supervised visual pre-training (corresponding to Section 3.1.1), unfreezing the feature encoder f and updating it via substantial training objectives. This leads consistent performance drops, especially in continuous control, demonstrating that steady visual representation is crucial for understanding complex dynamics. In "BCV-LR w/o la", we don't 't pre-train latent actions (corresponding to Section 3.1.2) but learn predictor p and world model w in the online stage, observing clear sample efficiency differences. effective learning was unattainable without latent actions, high-
Particularly for "point_mm _a___maas________eee lighting the significance of pre-trained knowledge. Finally, we fix the pre-trained latent actions, only updating the decoder d align the latent action space and real action space online (corresponding to Section 3,2.1), denoted as "BCV-LR w/o ft". The drops not large in Procgen but extremely huge in continuous control with more complex dynamics, which is consistent with the failure of baseline UPESV that freezes the pre-trained knowledge. In conclusion, the results demonstrate that each module is necessary for BCV-LR to achieve sample-efficient ILV performance in both discrete control and continuous control. In Appendix B, further provide more ablation results about different aub-optimization objectives introduced in Section 1.1.1 and 3.2.1.
4.5 Video Data Efficiency
In this chapter, we give BCV-LR different numbers of demonstration videos to explore its data efficiency, and the results are shown in Figure 50k transitions are enough for BCV-LR to achieve performance next the expert level across both tasks. When given only 20k transitions, BCV-LR can still learn the expert-level policy "finger_spii and an effective policy "reacher hard". While

---

BCV-LR has demonstrated certain degree of data efficiency, it exhibits performance bottlenecks when the amount of video data is further reduced 5k. On the basis of high sample efficiency, how to further improve the video data efficiency of BCV-LR is another issue worthy of research.
4.6 Multi-task Pre-training and Adaptation
In previous experiments, BCV-LR's work- flow is completed on a single task: offline pre-training on expert videos of one single task, followed by online finetuning and pol- icy cloning the reward-free environment corresponding to the task. In this chapter, we attempt explore the multi-task poten- tial of BCV-LR. Following the multi-task pre-training settings employed in previous works [51 34], we pre-train one BCV-LR model on mixed videos of three diverse tasks (' ('Bigfish', 'Maze', and 'Starpilot') and then achieve online finetuning in these tasks and two unseen tasks ( "'Bosfiihh"' and 'Dodgbball") separately. The results in Ta- ble demonstraeee that the multi-task BCV- LR enables effective policy imitation on all tasks, achieving robust multi-tase pre- pre-tngining and cross-domain adaptation. The pre-trained knowl- edge can be shared across both seen and unseen domains, demonstrating the potential of BCV-LR to leverage large-scale cross-domain video data from the internet.
machhr hard
1000
8.0
600
800
400
400
200
20000 40000 99005 90000 100003
20000 40000 60000 80000 100000
Envrommertal steas
steps
100k
50k
20k
SK
Figure 5: The training curves of BCV-LR when given different numbers of action-free video transitions. 50k video transitions are enough for BCV-LR to learn effective policy.
5 Limitation & Insight
Despite the remarkable results across sev- eral challenging tasks, BCV-LR faces the same issues as all methods based on be- havior cloning: covariate shift (25). This issue restricts the ability to handle sequen- tial decision-mann maniingnmmainggg tasks, such as com- plex robot locomotion. We provide addi- tional experimental analysis in Appendix B. Some recent works have demonstrated the feasibility of combining inverse RL and be- havior cloning address based ILO problems [65]. To this end, integrating BCV-LR with existing inverse rewards designing inverse rewards based on BCV-
Table 3: Multi-task pre-training and adaptation of BCV- LR. BCV-LR-M denotes the variant of BCV-LR with multi-task pre-training. PPO-S and BCV-LR-S denote training under the default single-task setting, i.e., they are the same as those in Section 4.2.
Task
BCV.LRM
BCV
PPO-5
Expert Videos
Biglish Maze
32
35.9 2.0
0.1
9.6
9.9
5.7
10.0
Starpilot
44.3 1.9
54.8 1.4
2.6
67.0
Bosslight
5.3
10.3
0.1
11.6
Dodgeball
9.3
12.4
1.1
13.5
LR latent representations are both viable improvement directions, which we leave for future work. Additionally, directions such as attempting to further improve BCV-LR's 's video data efficiency or exploring its ability leverage large-scale videos from the internet also represent future research avenues.
6 Conclusion
In this paper, we propose novel framework, named BCV-LR, efficiently derive policies from action-free videos without accessing rewards any other expert supervision. Extensive experiments demonstrate the huge efficiency advantages of BCV-LR against both advanced ILV methods and RL methods. BCV-LR for the first time demonstrates the feasibility of using videos as the only expert supervisory signal to guide extremely sample-efficient policy learning, and we believe this work can serve key stepping stone towards sample-efficient ILV in more scenarios (c.g., real-world robot manipulation) where both traditional supervision and environmental interactions are expensive.
10

---

Acknowledgments and Disclosure of Funding
This work is partly supported by the National Natural Science Foundation of China (NSFC) under Grants No. 62173324 and No. 62206281, and in part by the CAS for Grand Challenges under Grant I04GJH22022013GG and in part by the Suzhou Innovation and Entrepreneurship Leading Talents Programme Innovation Leading Talent in Universities and Research Institutes with Grant No. 2XL2025310.
The authors would like to thank Yaran Chen (XITLU), Boyu Li (CASIA), and Zhennan Jiang (CASIA) for their help and discussions on this work.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-535 2015.
[2] Timothy P. Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[3] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representationss 2022.
[4] Xin Liu, Yaran Chen, Guixing Chen, Haoran Li, and Dongbin Zhao. Balancing state explo- ration and skill diversity in unsupervised skill discovery. IEEE Transactions on Cybernetics, (5(5):2234-22444 2025.
[5] Zijie Zhao, Honglei Guo, Shengqian Chen, Kaixuan Xu, Bo Jiang, Yuanheng Zhu, and Dongbin Zhao. Empowering multi-robot cooperation via sequential world models. arXiv preprint arXiv:2509.13095, 2025.
[6] Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, and Huajin Tang. Human-level con- trol through directly trained deep spiking g-networks. IEEE Transactions on Cybernetics, 53(11):7187-7199 2023.
[7] Jingbo Sun, Songjun Tu, Qichao Zhang, Xin Liu. Haoran Li, Yaran Chen, Ke Chen, and Dongbin Zhao. Unsupervised zero-shot reinforcement learning via dual-value representation. In The Thirteenth International Conference on Learning Representations, 2025,
[8] Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent Trunsportation Systems, 23(6):499-44928 2021.
[9] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories, and applications. IEEE Transactions 2024.
on Pattern Analysis and Machine Intelligence, 46(12):11216-11)
[10] Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. ConRFT: A Reinforced Fine-tuning Method for VI.A Models via Consistency Policy. In Proceedings of Robotics: Science and Systems, LosAngeles, CA, USA, June 2025.
[11] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on robot learning, pages 158-168. PMLR, 2022.
[12] Shang-Fu Chen, Hsiang-Chhi Chun Wang, Ming- Hao Hsu, Chun-Mao Lai, and Shao-Hua Sun. Diffusion madel--ug behavioral cloning. In Proceedings of the 41st International Conference Machine Learning, pages 7486-75100 2024.
[13] Jinxin Liu, Hongyin Zhang. Zifeng Zhuang, Yachen Kang, Donglin Wang, and Bin Wang. De- sign from policies: Conservative test-time adaptation for offline policy optimization. Advances in Neural Information Processing Systems, 36, 2023.

---

[14] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2023.
[15] Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh. Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline rl from internet videos via value- function learning. 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16977-16984, 2024.
[16] Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In International Conference on Machine Learning, pages 11321-11339, PMLR, 2023.
[17] Tao Huang, Guangqi Jiang, Yanjie 7e, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. In European Conference on Computer Vision, pages 478-494 Springer, 2024.
[18] Hao Luo, Bohan Zhou, and Zongqing Lu. Pre-trained visual dynamics representations for efficient policy learning. In European Conference on Computer Vision, pages 249-267 Springer, 2024.
[19] Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Se June Joo, Jianwei Yang. Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos. In The Thirteenth International Conference on Learning Representations, 2025.
[20] Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai. arXiv preprini arXiv:2411.00785 2024.
[21] Dominík Schmidt and Minqi Jiang. Learning to act without actions. In The Twelfth International Conference Learning Representations, 2024.
[22] Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
[23] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint /007.0/800061111 2018
[24] Haresh Karnan, Faraz Torabi, Garrett Warnell, and Peter Stone. Adversarial imitation learning from video using a state observer. 2022 International Conference on Robotics and Automation (ICRA), pages 2452-2458. IEEE, 2022.
[25] Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi. survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions Cybernetics, 2024.
[26] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4950- 4957, 2018.
[27] Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International Conference on Machine Learning, pages 1755-1763. PMLR, 2019.
[28) Xin Liu, Yaran Chen, and Haoran Li. Sample-efficient unsupervised policy cloning from ensemble self-supervised labeled videos. 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 3632-3639. IEEE, 2025.
[29] Oliver Struckmeier and Ville Kyrki. Upoo-mp: Mode priors prevent mode collapse when imitating latent policies from observations. Transactions Machine Learning Research, 2023.
12

---

[30] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation benchmark reinforcement learning. In International Conference on Machine Learning, pages PMLR, 2020.
[31] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801000600, 2018.
[32] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pages 1094-1100. PMLR, 2020.
[33] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR, 2018.
[34] Xin Liu, Yaran Chen, Haoran Li, Boyu Li, and Dongbin Zhao. Cross-domain random pre- training with prototypes for reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 55(5):3601-36133 2025.
[35] Michael Laskin, Aravind Srinivas, and Pieter Abbeel, Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[36] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06177 2022.
[37] Yang Yue, Bingyi Kang, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Value-consistent representation learning for data-efficient reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11099-11077, 2023.
[38] Xin Liu, Yaran Chen, Haoran Li, and Dongbin Zhao. Learning future representation with synthetic observations for sample-efficient reinforcement learning. Science China Information Sciences, 150202, 2025.
[39] Eukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blažej Osiński, Roy Campbell, Konrad
Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2020.
[40] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- lions, 2020.
[41] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021.
[42] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 104, 2023.
[43] Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 4936-4975. PMLR, 2022.
[44] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pages 1332-1344. PMIR, 2023.
[45] Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, and Pieter Abbeel. Multi- view masked world models for visual robotic manipulation. In International Conference on Machine Learning, pages 30613-30633 PMLR, 2023.
13

---

[46] Fan-Ming Luo, Tian Xu, Hang Lai, Xiong- Hui Chen, Weinan Zhang, and Yang Yu. A survey on model-basee reinforcement learning. Science China Information Sciences, 67(2):12110, 2024.
[47] Minghuan Liu, Tairan He, Weinan Zhang, Shuicheng YAN, and Zhongwen Xu. Visual im- itation learning with patch rewards. In The Eleventh International Conference on Learning Representations, 2023.
[48] Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. International Conference on Machine Learning, pages 19561-19579. PMLR, 2022.
[49] Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
[50] Vittorio Giammarino, James Queeney, and loannis Paschalidis. Adversarial imitation learning from visual observations using latent information. Transactions on Machine Learning Research, 2024.
[51] Weirui Ye, Yunsheng Zhang, Pieter Abbeel, and Yang Gao. Become proficient player with limited data through watching pure videos. In The Eleventh International Conference on Learning Representations, 2022.
[52] Hao Luo and Zongqing Lu. Learning video-conditioned policy on unlabelled data with joint embedding predictive transformer. In The Thirteenth International Conference on Learning Representations, 2025.
[53] Siyuan Li, Xun Wang, Rongchang Zuo, Kewu Sun, Lingfei Cui, Jishiyu Ding, Peng Liu, and Zhe Ma. Robust visual imitation learning with inverse dynamics representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages I 13609-13618, 2024.
[54] Vittorio Giammarino, James Queeney, and Ioannis Ch Paschalidis. Visually robust adversarial imitation learning from videos with contrastive learning. arXiv preprint 2tt.:2407.1292.. 2024.
[55] Yuhui Chen, Haoran Li, Zhennan Jiang, Haowei Wen, and Dongbin Zhao. Tevir: Text- to-video reward with diffusion models for efficient reinforcement learning. arXiv preprint arXiv:2505.19765 2025.
[56] Deepak Pathak, Parsa Mahmoudich, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2050-2053, 2018.
[57] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11933 11911. PMLR, 2021.
(58) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2020.
[59] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
[60] Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. Taco: T'emporal latent action-driven contrastive loss for visual reinforcement learning. Advances in Neural Information Processing Systems, 36:48203-48225, 2023.
[61] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in
Neural Information Processing Systems, 26, 2013.
14

---

[62] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
[63] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint crXiv:1707.06347 2017.
[64] Matthew Aitchison, Penny Sweetser, Gregoire Deletang, and Marcus Hutter. Policy gradient without boostrapping via truncated value learning, 2024.
[65] Joao Candido Ramos, Lionel Blondé, Naoya Takeishi, and Alexandros Kalousis. Mimicking better by matching the approximate action distribution. In Forty-first International Conference on Machine Learning, 2024.
15

---

APPENDIX
A Supplemented Details of BCV-LR Methodology
A.1 Pseudo Code
Algorithm The pseudo code of the proposed BCV-LR.
The action-free free video dataset B", the reward-free environment E. the latent feature veeitraining update times Uty. latent action pre pre-training steps Ula. the online environmental interaction steps L the online finetuning frequency F Ft. the finetuning update times U/, the behavior cloning update times Ube- Initialize: The latent feature encoder f. the latent action predictor p. the world model w, the latent action decoder d, the latent policy n, and the environmental replay buffer B
Require:
1: # Offllie stage #
2: ## Pre-training latent features ##
3: for inder = 1,.... Uif do
4: Sample batch of vidco observations from video dataset B". Calculate the self-supervised objective Liy via Equation (1),
6: Update the latent feature encoder f by backpropagation.
7: ## Pre-training latent actions #i
8:
for index 1, ..., Ula do
Sample batch vidco observations from video dataset B".
10: Calculate the dynamics- synamiic-based objective Lia via Equation (2).
11: Update the latent action predictor and the world model w by backpropagation.
12: # Online stage #
13: for index
F do
Optional preparations
15:
(Optional, letting latent policy fully imitate the pre-trained latent actions before interactions.)
16:
if indeæ == 1
17:
Conduct latent behavior cloning via Equation (4) until convergence. ## Collect environmental interactions ##
18:
19:
for index' = .., F/z do
20:
if index = 1
21:
Interact with environment E to collect data with random policy. Save the data into the buffer B
22:
clse:
24:
Interact with environment E to collect data with the learned policy (consisting of f., and d). Save the data into the buffer B ## Finetuning and decoding the latent actions ##
25:
26:
27:
for index' 1,... Use do
28:
Sample batch of environmental transitions from huffer B"
29:
Calculate the latent action finetuning objective L via Equation (3).
30:
Update the latent action predictor and the latent action decoder d by backpropagation.
31:
## Behavior cloning the latent policy ##ff
32:
for inder' 1.... Uhe do
33:
Sample batch of action-free observations from vidcos B".
34:
Obtain the predicted latent actions through and p.
35:
Calculate the latent behavior cloning objective Lie via Equation (4).
36:
Update the latent policy æ.
Output: The
visual control policy (f. d).
16

---

A.2 Prototype-based Temporal Association Task
As shown in Section 3.1.1, for partially observable domains with complex dynamics, such as DMControl benchmark [31] in this paper, we employ recent advanced prototype-basee based temporal association task [57 4 34 which aligns two temporally neighboring observations together the latent space. For the sake of fluency, we put the details of this self- seff-supervised task here instead of in the main text.
Concretely, M observation pairs {[of, (11) are sampled from videos, augmented by the random shift, and encoded by and f' to obtain latent features {s and {8]+1}. J. Each current latent feature of is further processed by MLP u, which is set to introduce asymmetry for avoiding collapse to trivial solutions. Then, we take softmax over the dot product between u(s,) and M trainable prototypes {=
(u(s?)-
01
u(s)) CM
2 softmax
(5)
where 7 denotes temperature hyper-parametes To calculate the association target, the Sinkhorn- Knopp [61] algorithm is employed on the whole batch {8/41} and prototypes {cj to obtain batch- clustering target labels {v"+1} for all training feature pairs. Concretely, the Sinkhorn-Knoph algorithm begins with the square matrix C, whose elements are computed by the dot product over each 8 and prototype cj:
Ci - 811 Cy-
(6)
Then it employs several times of doubly-none noablyyoomalizztionn on the matrix C' to obtain the clustering target matrix T. constraining every column and row have the same sum with little change of original C as possible. One doubly-normalizztion consists of row normalization and column normalization. The row normalization and column normalization are formulated as the following:
NormRow(C) ( C,
(7)
M
NormColumn(C) C' ddag(SumCoonmm(())
(8)
M
where SumRow(-) denotes the row addition, SumColumn() denotes the column addition, and diag(-) denotes the diagonalization of a matrix. The noulyy-oomalliztion, NormDouble()). is correspondingly defined as:
NormDouble(C)) NornCColum(XorrRoow(()))
Several times of doubly-normalizann normalization are applied to C' to obtain the clustering target matrix T. The i-th row of T is the clustering- based target v of the z².. The self-supervised objective is formulated as:
Ly = ve4 log x.-
(10)
The clustering-based temporal association task actually makes each observation access its own tem- porally neighboring observation in the self-supervised latent space, thus enabling the representation module f to obtain the temporal information.
17

---

B Additional Results
B.1 Numerical Results Fewer Steps DMControl
In Section 4.3, we provide the results at 100k environmental steps, where the proposed BCV-LR leads across 6/8 tasks. Here demonstrate the advantages of BCV-LR when interactions are further constrained (50k and 20k steps). As shown in Table and Table 9 BCV-LR performs much better than both ILV and RL baselines, enabling effective policy learning given only 20k steps.
Table 4: Numerical results at 50k environmental steps. Italics indicate using expert videos, and underlines denote using environmental rewards. Bold text indicates the highest score excluding video experts.
Task (50k steps)
BCV-LR (ours)
LAIFO 50
8CO
20
UPESV 28
TACO h0
DrQv2 X
Expert Videos
point_mmss_coo mass_masssss casy reacher_hard
76
23
44
967
pco_reach__op_rrgeee
73
14 14
0
3
10
22
198
fingor_spin
912 38
166 122
855 28
0
416+41
168 121
981
ball_in_up_com
683 + 106
23
208 227
33
126 115
176
986
jaco reachh heoorrech_battom_kee Jeft
101 32
20
0
8:4
8
12
203
cheetah run_ backward reachor_casy
1+0
207 27
272 157
220 158
389
101
80
110
152
975
Table 5: Numerical results at 20k environmental steps. Italies indicate using expert videos, and underlines denote using environmental rewards. Bold text indicates the highest score excluding video experts.
Task (20k steps)
BCV-LR (ours)
LAIFO 550
BCO 26
UPESY 28
TACO (60)
DrQv2 S
Expert Videos
point mass, easy reacher_har ssu_reach__uee righright
219
46
70 :
5
6
4
2
3
198
finger_spin spin
944 25
84 84
799 #:82
0
51
43
981
vall_in_up catch
459 + 139
96.1 121
196 2:5
99
99
33
986
juco reach, bettom_left
jaco_reach,
45
1+1
0
8
343
2
203
cheetah, run, backward reacher_easy
206
77.2:22
38
18

---

B.2 Experiments in Metaworld Manipulation
In this section, further conduct some extra experiments in the Metaworld manipulation benchmark [32,, which may demonstrate wider application of BCV-LR. For each Metaworld task. only 50k environmental steps are allowed. The remaining settings are similar that of DMControl experiments except for the self-supervised latent feature objective Lif employed in BCV-LR. Concretely, we employ only contrastive learning loss (described Section 3.1.1) in this benchmark because it seems to perform better than other objectives used in this paper. Results (success rate) are shown in Table 6 In this interaction-limited situation, BCV-LR can still derive effective manipulation skills from expert videos without accessing expert actions and rewards, which demonstrates its wider range of applications and potential for generalizing to real-world manipulation tasks.
Table 6:
The results (success rate) on 4 manipulation tasks from Metaworid. Italics indicate using expert videos, and underlines denote using environmental rewards. Bold text indicates the highest score excluding video experts.
Metaworld-5he
BCV-LR (ours)
BCO 26
DrQv2 3
Expert Videos
Faucet-open
0.82 = 0.20
0.13 0.19
= 0.00
Reach
0.63 0.25
0.03 0.05
0.13 0.12
1.00
Drawer-open
0.92 0.12
0.13 + 0.09
0.00 + 0.00
1.00
Faucet-close
0.98 = 0.04
0.00 + 0.00
0.50 0.28
1.00
Mean Success Rate
0.84
0.07
0.16
1.00
B.3 Additional Comparison with LAPO's BC Variant
In Section 4, we use LAPO [2] as reinforcement learning baseline, which follows its original paper and official implementation. In this section, we replace LAPO's s online RL loss with behavior cloning (BC) loss, obtaining its reward-free BC variant, and then compare it with BCV-LR. The results in Table 7 show that LAPO-BB BC works well in some tasks, but our BCV-LR still performs better.
Table 7:
The comparison between BCV-LR and LAPO's BC variant. Italics indicate using expert videos, and underlines denote using environmental rewards. Bold text indicates the highest success rate excluding video experts.
Task
BCV-LR (ours)
LAPO-BC
PPO 63
Expert Videos
Fruitbot
27.5 + 1.5
6.2 1.9
+1.9
29.9
Heist
9.3 0.1
9.2
3.2
9.7
Bossfight
10.3
0.0
0.1
11.6
Chaser
3.1 0.5
0.6
0.2
10.0
19

---

B.4 AbaatonncHyerrrppraneeeeer Sensitivity: Self-supervised Reconstruction Loss
In this section, we further provide the ablation study of the extra reconstruction loss employed in Section 3.1.1. In video games, useful visual information is abundant and scattered. Unlike contrastive learning, which understands the image as whole, the reconstruction task forces the feature encoder to focus on these key pixels. The results demonstrate that this extra objective can indeed improve the performance, as long as not set too large, which is shown Figure
Bigfish
Starpilot
50
36
24
Scores
12
100
28
20
Figure 6: Ablation and hhper-paramedee parameter sensitivity analysis of the extra reconstruction loss in self- supervised latent feature pre-training.
B.5
Ablation&Hyyer-prrammaee Sensitivity: Finetuning Latent Actions with World Model
In this section, we demonstrate whether finetuning the latent actions online with the world model yields better performance. The results in Figure that constraining the latent action finetuning with the pre-trained world model is helpful. It is consistent with our intuition, because we aim to extract the expert actions for policy cloning, while the online collection is not expert-level in most of the time. The pre-trained world model can provide the expert dynamics-based knowledge to alleviate effect of the distribution difference.
Starpilot
dodgeball
10
14
Scores
Scores
8
8-2
8-
3-
9--0
-
95
6
Figure 7: Ablation and hyper-parameter sensitivity analysis of the dynamics-constrained loss in online latent action finetuning.
20

---

B.6 Ablation: Freezing Latent Features
In this section, we provide ablation experiments to demonstrate whether it necessary to further fine- tune the pre-trained self supervveed latent features with the other objectives in BCV-LR. We finetune the sulf-supe encoder with Lia and Ltt, respectively. The results in Table g demonstrate that whether finetuning self-supervised visual features doesn't yield an apparent effect on policy performance. This phenomenon has also been observed in self-supervised RI. [59 leading some works to fine-tune self-supervised features while others opt to freeze them.
Table 8: Comparison of policy learning performance between freezing and fine-tuning BCV-LR self-supervised latent features.
Task
Finetuned via Lia
Finetuned via Ltt
Frozen
Expert vidcos
reacher_hard
876 + 15
906 + 65
900 = 31
967 942 48
finger_spin
937 26
920 57
981
B.7 Schedule of Environment Interactions
In the online stage of BCV-LR, a) we allow the agent to interact with the environment for a fixed number of times using policy and collect transitions to enrich the experience buffer. Immediately after that, b) we perform finetuning of the latent action and training of the action decoder on the experience buffer. Then, c) we train the latent policy to imitate the finetuned latent action predicted from expert videos. After this, the BCV-LR policy is improved, and we return to part a) collect better training data, which forms a cyclic online policy learning. Under the default experimental settings, set the number of interactions for each cycle at relatively large value from start to finish (for example, we fixed the number of collected transitions in part a) as 1000 in DMControl). In this section, we try much smaller interaction number (set to 2) and correspondingly reduce the number of update times for latent actions (set to 1) and latent policies (set to 2) in each cycle. We denote this variant as DEV--LR(1000-22) The results in Table show that our approach can still achieve effective policy learning, demonstrating its robustness to the schedule of environment interactions.
Table 9: Comparison of BCV-LR's performance perfomaee under different schedules of environment interactions.
Task
BCV-LR (1000-52)
BCV-LR
967 DrQv2
Expert videos
reacher_hand
875 = 65
900 + 31
92 + 98
finger_spin
956 + 20
942 + 48
374 + 264
981
B.8
Offfine Latent Action Decoding
In this section, let BCV-LR perform latent action finetuning and policy imitation with few offline action-labeled expert transitions, in a fashion akin to LAPA [1). Concretely, we maintain the original pre-training stage and use 10k offline action-labeled expert transitions achieve offline latent action finetuning and policy cloning with the original losses. The results in Table TO (RI. denotes DrQv2 3 for DMControl tasks and PPO [63] for Procgen tasks) demonstrate that BCV-LR can also achieve offline imitation learning well if expert actions are provided.
Table 10: BCV-LR can also accomplish latent action decoding and fine-tuning using offline action- labeled expert data.
Task
BCV-LR-offline
BCV-LR
RI.
Expert videos
reacher_hare hard finger_spin
938 + 44
900 +
92
967
978 7
942 1 48
374 264
981 -1.9 1.0
Fruitbot
27.7 + 0.4
27.5 1.5
29.9
21

---

B.9 Limitation Analysis
Despite the remarkable results across different kinds of control tasks (including both discrete control and continuous control), BCV-LR faces covariate shift [25], the same issues as all methods based on behavior cloning. This issue restricts the ability handle sequential as complex robot locomotion. In this section, we provide the comparison between the IRI. method LAIFO (0) and our BCV-LR on two continuous tasks: the balance control "reacher hard" and the locomotion task "walker_war waak". As shown in Figure [ when the environmental interactions are limited (only 100k steps), BCV-LR exhibits significant advantages on both tasks.
meessiinmmaiing tasks, such
walker walk(100k)
reacher hard(100k)
1000
250
2000
Reward
600
150
100
200
50
20000 40000 50000 100000
20000 40000 60000 80000 100000 Environmental steps
Environmental steps
BCV-LR (ours)
LAIFO
Figure The training curves of BCV-LR and LAIFO when 100k steps are allowed. BCV-LR exhibits advantages on both locomotion and balance control.
However, if enough environmental steps (500k) are permitted, behavior cloning-based BCV- LR's performance growth stagnates after efficient learning in the sequential decision-making task "walker walk", while RL-based LAIFO can continue to improve.
walker walk[50ok)
reacher hard(5 (00)
Video Expert
Video Experi
800
Reward
6:00
400
400
200
Environmental steps
x10>
Environmental steps
x10*
BCV-LR (ours)
LAIFO
The training curves of BCV-LR and LAIFO when 500k steps are allowed. BCV-LR is limited in sequential decision-making task, which is due to the covariate shift of the behavior cloning-
Figure 9:
Some recent works have demonstrated the feasibility of combining inverse RL and behavior cloning to address state-based ILO problems [65]. To this end, integrating BCV-LR with existing inverse rewards designing inverse rewards based on BCV-LR latent representations are both viable improvement directions, which we leave for future work.
22

---

C Experimental Details
C.1 Introductions of Environments
In this paper, we employ diverse and challenging visual control tasks to provide a comprehensive evaluation of the proposed BCV-LR, as shown in Figure 10 The top two rows are discrete Procgen tasks [30], the third row contains continuous DMControl tasks [3], while the bottom row contains Metaworld manipulation tasks [32]
The first row from left to right is Coinrun, Starpilot, Caveflyer, Dodgeball, Leaper, Maze, Bigfish, and Heist.
The second row from left right Fruitbot, Chaser, Miner, Jumper, Climber, Plunder, Ninja, and Bossfight.
The third row from left right is jacoaaaooaooooooo reach bottom left, point mass easa,ass, finger_spil reacher_eaat _asy, cheetah_ ren__uu___ae_____ backwadddd, ball in cup_cp____ cup_ccccccccc___ catch, jaco cecoo_reach_tooo teach_top_right, right, and reacher_hare
The bottom from left to right is Faucet-open, Reach, Drawer-open, and Faucet-close. See their paper for the detailed task description.
Figure 10: Screenshot of diverse and challenging visual control tasks employed in this paper.
23

---

C.2 Hyper-parameter Settings
Table 11: Default hyper-paramenn settings discrete Procgen. The "Ir" denotes "learning rate". while the "ft" denotes "finctuning". There are additionally 60k latent behavior cloning update times before the first round of the online stage to ensure that the latent policy fully imitates the pre-trained latent actions. For Miner, Climber, and Chaser, the "online latent action predictor ft Ir" is le-3 and the is set I. For Plunder, the a is 1. The experiments of BCV-LR are conducted using V100 or A800 GPUs, and the complete workflow for each task can be finished within five hours on a single GPU. Setting
Frame shape
64 64
Frame stack
1
Action repeat
1
Action type
Discrete
Action dimension
15
Video steps
8M
Video expert training steps
50M
Latent feature pt times
20000
Feature encoder Ir
3e
Random shift padding upper bound
1
Reconstruction coefficient a
le
EMA update frequency
2
EMA Momentum
0.05
VQ codebook number
2
VQ dicrete latent number
4
VQ latent embedding dimension
16
VQ latent embedding number
64
Latent action dimension
128
Latent action pt times Latent action predictor pt Ir
10000
3e
World model pt Ir
3e 4
Online steps
100000
Online env numbers Online update frequency Latent action ft update times Latent action decoder Ir Latent action predictor ft Ir
64
64
60
1e -
2e
World model loss coefficient /
15
World model ft
Frozen
Behavior cloning update times
500
Latent policy Ir
2e
24

---

Table 12: Default parammer settings in DMControl. The "Ir" denotes "learning rate". while the "ft" denotes "finetuning". There are additionally 60k latent behavior cloning update times before the first round of the online stage to ensure that the latent policy fully imitates actions. For "point_mpp poiitt_mass___ mass_ eaay', the random shift padding is to 1. For the domain jaco and reacher, the update time latent action is set to the fixed value of 100 while the / and the world model finetuning learning rate are both 1e 3. For Metaworld tasks, they share hyper-parametent with DMControl, except for the action dimension set to 4, the online interaction frames set to 50k, and the latent feature self-supervised task changed to contrastive learning. The experiments of BCV-LR are conducted using V100 or A800 GPUs, and the complete workflow for each task can be finished within five hours single GPU.
pre-trained latent
Hyper-parameter
Setting
Frame shape
64
64
Frame stack
3
Action repeat
2
Action type
Continuous
Action dimension
Refer to [31
Video frames
200k
Video expert training frames
1M
Temperature Hyper-parameter
0.1
Latent feature pt times
50000
Feature encoder Ir
le -
Random shift padding upper bound
4
Numbers of Houbly-normalizze
3
EMA update frequency
2
EMA Momentum
0.05
VQ codebook number
2
VQ dicrete latent number
4
VQ latent embedding dimension
16
VQ latent embedding number
64
Latent action dimension
128
Latent action pt times
50000
Latent action predictor pt Ir
3e
World model pt
3e
Online interaction frames Online env numbers
100000
1 1000
Online update frequency Latent action update epoch Latent action decoder
2
le -
Latent action predictor ft Ir World model loss coefficient 3 World model ft Ir
1e 3
le 5
1e
Behavior cloning update times
1000
Latent policy
le -
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The content can match the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have a seperate limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
This paper doesn't 't include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details Appendix and will release the implementation after acceptance.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
We release the data (videos) and codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiiicgguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These information is provided in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results include error bars in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We provide details in Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tp:::////rrpps// httpr//s...cpullltttttccccseeeee
Answer: [Yes]
Justification: We've reviewed it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the papers corresponding to the used data and environments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
30

---

Answer: [Yes]
Justification:
We release the videos used in experiments and provide documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
31

---

Justification:
We employ Al for grammar checker.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httess/s//eerrpppoccc///o/rreerp for what should or should not be described.
32


Reviews:
[{"review_id": "4n2xuuUxE1", "reviewer": "Reviewer_oQMP", "summary": "The work proposes BCV-LR to conduct behavior cloning with expert videos and several online interactions. BCV-LR learns the dynamics model and inverse dynamics model on latent actions during the offline phase. In the online phase, it achieves behavior cloning of expert videos by aligning latent actions to true actions through interaction. Without action labels, experiments conducted on Procgen and DMControl demonstrate BCV-LR  with good sample efficiency.", "questions": "1.  Does the phrase \"without the need to access any other supervision\" include the supervision of actions during the online phase? The supervision on the actions directly injects the environment's inverse dynamics knowledge and potentially adjusts the distribution of latent actions through alignment with ground-truth actions. If the phrase refers to merely using the latent reward information implicit in expert videos without online expert rewards for video behavior cloning, JEPT[1] may be an uncited but related work. Similarly, without using an explicit reward signal, JEPT can achieve generalization of one-shot visual imitation in some tasks with a mixture dataset.\n2.  The design of BCV-LR is very similar to FICC[2] (mainly in the offline part), while FICC is not properly referenced. FICC should be a comparable baseline in the discrete setting.\n3.  I am curious whether the effectiveness of direct behavior cloning in the online phase is related to the single-task setting or the quality of expert data. The offline phase of BCV-LR is quite similar to LAPO and FICC, while both LAPO and FICC avoid direct behavior cloning in the online phase and choose to use online rewards for policy adjustment in the multitask setting. It would be better if a direct online BC variant of LAPO were involved in the comparison, which means the online reward will also be excluded in this setting, and may further illustrate this aspect.\n4.  The paper lacks experiments in a multi-task setting or a discussion on the relationship between the expert video scale and the performance in the single-task setting. Can BCV-LR generalize to new tasks through multi-task data? Or, for new tasks lacking sufficient data, how much offline data is required for BCV-LR to fulfill behavior cloning? Otherwise, the prerequisite of obtaining sufficient data often implies having a well-performing in-domain policy on the task, reducing the significance of behavior cloning from expert videos.\n5.  Adding some ablation experiments would be more helpful in explaining what factors make BCV-LR more effective than previous works. For example, a) the impact on performance of whether the representation learning part is directly frozen after pre-training or updated with $L_{la}$ or $L_{ft}$; b) the impact on performance of not updating the latent action predictor in the online phase.\n\n[1] Learning Video-Conditioned Policy on Unlabelled Data with Joint Embedding Predictive Transformer. Hao Luo, Zongqing Lu. ICLR 2025.\n\n[2] Become a proficient player with limited data through watching pure videos. Weirui Ye, Yunsheng Zhang, Pieter Abbeel, and Yang Gao. ICLR 2023.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe presentation of the paper is quite clear. The experimental performance of BCV-LR is excellent, especially considering that only a small number of online steps are used.\n\nWeaknesses:\nI am not sure whether the offline stage is individually conducted on each task or jointly conducted on a multi-task dataset. The practical significance of video behavior cloning in a single-task setting is limited. The LAPO in the baseline is designed for the pre-training on multi-task data, which may affect comparisons. Additionally, incorporating extra ablation studies and baselines could further solidify this work. Please refer to the Questions part.", "was_revised": true, "final_justification": "The author's rebuttal addresses most of my concerns. Importantly, the weakened version of the claim mitigates my concerns about overclaiming.", "rebuttal": "We appreciate your careful reading, professional comments, and affirmation of our paper! In response to your feedback, we have supplemented the missing related works and further conducted several extra experiments, which greatly improves the paper quality. We hope these revisions will meet your expectations, and we would be most grateful if you could consider raising your score!\n___\n>**W1:I am not sure whether the offline stage is individually conducted on each task or jointly conducted on a multi-task dataset. The practical significance of video behavior cloning in a single-task setting is limited.**\n\nIn the submitted version, all experiments are conducted under single-task settings. This is mainly due to two reasons: (a) whether it is possible to balance video imitation performance and sampling efficiency in single-task settings remained an open problem before our paper; and (b) recent advanced pure ILV (no reward) works mainly achieve evaluation under single-task settings. We agree with you that achieving video cloning under multi-task settings is more meaningful and we supplement multi-task experiments as you suggested, as shown in answer to Q4.\n___\n>**W2:The LAPO in the baseline is designed for the pre-training on multi-task data, which may affect comparisons.**\n\nLAPO indeed inspires many works on multitask pre-training. However, according to the final published version of LAPO and its open-source code, it uses a 'full' distribution for each task in Procgen to ensure intra-task diversity (e.g., the 'maze' task includes mazes of different sizes, colors, and layouts) while doesn't perform cross-task 'multi-task learning' (i.e., it achieves pre-training on 'maze' videos and then performs online learning in the 'maze' task). The Procgen experiments of BCV-LR have strictly followed LAPO's setup in terms of environment configuration, and we use the same video dataset as that of LAPO, thus ensuring a fair comparison. Please feel free to let us know if you have any further questions!\n___\n**Due to space constraints, we can't list your questions here. Please refer to your reviews. We apologize for the inconvenience.**\n>**To Q1: We supplement the reference and discussion of JPET.**\n\nThank you for your careful reading! BCV-LR utilizes information from environmental actions. Our intention is to emphasize that it doesn't rely on 'expert' actions. We have replaced \"without the need to access any other supervision\" with \"without the need to access any other expert supervision\" to eliminate ambiguity.\n\nIn addition, JPET[1] is very relevant to our work and we missed it. We have added the citation and discussion of JPET in the related work (line 120): \"JPET[46] utilizes a mixed dataset containing videos to achieve one-shot visual imitation.\"\n___\n>**To Q2: We add the reference and discussion of FICC and explain why it has not been included in the comparison for the time being.**\n\nThanks for your careful reading, and we now have discussed FICC [2] in the related work (line 121): \"FICC[47] pre-trains world models on action-free agent experience, accelerating MBRL on Atari.\" That said, due to the following reasons, we have not yet included FICC in the experiments on ProcGen:\n\n1. FICC has not been evaluated on the Procgen and none of our baselines evaluates FICC on Procgen. Moreover, whether its MCTS-based backbone EfficientZero (EZ)[3] is applicable to procedurally generated environments remains an open question. This lack of information—such as hyperparameter settings and EZ framework in Procgen—leaves us unable to ensure the validity of our reproductions.\n2. We have already compared BCV-LR against 6 popular baselines on Procgen, including the state-of-the-art RL and ILV algorithms. FICC is not able to perform policy learning without relying on rewards, in the way that BCV-LR can. Its training paradigm is more similar to that of LAPO. Given that LAPO itself has achieved state-of-the-art performance on ProcGen, we believe direct comparisons with LAPO may be more persuasive.\n3. FICC provides code for its pre-training part while doesn't offer code for fine-tuning the pre-trained model and combining it with its MCTS-based backbone online. This makes it difficult for us to reproduce this work within the short timeframe of the rebuttal period.\n4. The backbone of FICC, EZ[3], is built on MCTS and thus requires considerable training time. As noted in EZ, it takes 28 GPU hours for 100k steps in discrete control, whereas BCV-LR online stage requires only around 1 GPU hour on both discrete and continuous domains. This high computational cost has led many subsequent works [4,5] to refrain from direct comparisons with EZ, even in Atari experiments. In addition, this consumption also means it's hard to obtain its results within the limited timeframe available.\n___\n>**To Q3: We explain that the single-task setting and training data don't cause unfairness. We conduct extra experiments on LAPO's BC variant.**\n\nAs we explain in our answer to W2, LAPO also focuses on single-task settings. FICC includes multi-task pre-training experiments, but its main results are also single-task.​\nWhat’s more, the Procgen video we use is the same as LAPO’s original data, as stated in Sec.4.2 (line286). This means all methods have expert data of the same quality.​\n\nAs you asked, we conducted extra experiments: we replaced LAPO’s online RL loss with BC loss, then compared it with BCV-LR. The results show that LAPO-BC works well in some tasks, but our BCV-LR still performs better.​\nWe also show that BCV-LR can be used for multi-task pre-training in a fashion akin to FICC. Please refer to answer to Q4.\n|Task|BCV-LR|LAPO-BC|PPO|/|Expert video|\n|-|-|-|-|-|-| \n|fruitbot|**27.5 ± 1.5**|6.2 ± 1.9|-1.9 ± 1.0||29.9|\n|heist|**9.3 ± 0.1**|9.2 ± 0.3|3.7 ± 0.2||9.7|\n|bossfight|**10.3 ± 0.3**|0.0 ± 0.0|0.1 ± 0.1||11.6|\n|chaser|**3.1 ± 0.5**|0.6 ± 0.0|0.4 ± 0.2||10.0|\n___\n>**To Q4: We conduct extra experiments on multi-task ability of BCV-LR and provide video data efficiency analysis.**\n\nAccording to your suggestions, we conducted extra multitask pre-training experiments for BCV-LR, following the settings of that in FICC. We pre-train one BCV-LR model on mixed videos of 'bigfish', 'maze', and 'starpilot', then finetune it in these tasks separately. Different from FICC, we further employ two unseen tasks. The results show that BCV-LR enables effective policy imitation on all tasks. It achieves robust multitask pre-training, where the pre-trained knowledge can be shared across both seen and unseen domains.\n||Task|BCV-LR(multi)|PPO(single)|/|BCV-LR(single)|\n|-|-|-|-|-|-| \n|seen|bigfish|**32.2 ± 1.0**|0.9 ± 0.1||35.9|\n||maze|**9.6 ± 0.1**|5.0 ± 0.7||9.9|\n||starpilot|**44.3 ± 1.9**|2.6 ± 0.9||54.8|\n|unseen|bossfight|**5.5 ± 0.3**|0.1 ± 0.1||10.3|\n||dodgeball|**9.5 ± 0.3**|1.1 ± 0.2||12.4|\n\nThen, we provide the video data efficiency experiments. We test BCV-LR with 5k, 20k, 50k, and 100k (default setting) video transitions. Results demonstrate that BCV-LR enables effective policy learning with only 20k transitions. 50k transitions can support near-expert performance. Refer to Appendix C.5 for more details.\n|Video data of BCV-LR|5k|20k|50k|100k|/|Expert video|\n|-|-|-|-|-|-|-| \n|reacher_hard|0 ± 0|384 ± 153|799 ± 34|**900 ± 31**||967|\n|finger_spin|596 ± 17|901 ± 33|905 ± 70|**942 ± 48**||981|\n___\n>**To Q5: We conduct extra ablation experiments.**\n\nAs per your suggestions, we first conducted additional experiments, where we finetuned the self-supervised encoder with $L_{la}$ and $L_{ft}$ respectively. The results in Table IV demonstrate that whether finetuning self-supervised visual representation doesn't yield apprent effct on policy performance (curves are also similar). This phenomenon has also been observed in self-supervised RL[7], leading some works to fine-tune self-supervised features while others opt to freeze them.\n|Task|Finetuning via $L_{la}$|Finetuning via $L_{ft}$|No visual finetuning|\n|-|-|-|-|\n|reacher_hard|876 ± 15|**906 ± 65**|900 ± 31|\n|finger_spin|937 ± 26|920 ± 57|**942 ± 48**|\n\nThen, we provide the experiments to show the impact on performance of not updating the latent action predictor with $L_{ft}$ in the online phase (denoted as 'BCV-LR w/o ft'). Partial results are below, demonstrating that utilizing the environmental actions to finetune pre-trained latent actions is a crucial step in BCV-LR, especially in DMControl tasks. Please refer to Sec.4.4 (main paper) for more results.\n||Task|BCV-LR|BCV-LR w/o ft|\n|-|-|-|-|\n|DMControl|point_mass_easy|**800 ± 25**|22 ± 2|\n||jaco_reach_bottom_left|**123 ± 39**|15 ± 10|\n|Procgen|starpilot|**54.8 ± 1.4**|29.2 ± 10.6|\n||fruitbot|**27.5 ± 1.5**|24.2 ± 2.0|\n___\n>**Finally, we further conduct additional experiments in Metaworld.**\n\nOnly 50k steps are allowed for each task, with other settings similar to that of DMControl. Results (success rate) demonstrate BCV-LR's wider range of applications and potential for generalizing to real-world manipulation tasks.\n|Metaworld-50k|BCV-LR|BCO|DrQv2|/|Expert video|\n|-|-|-|-|-|-| \n|Faucet-open|**0.82 ± 0.20**|0.13 ± 0.19|0.00 ± 0.00||1|\n|Reach|**0.63 ± 0.25**|0.03 ± 0.05|0.13 ± 0.12||1|\n|Drawer-open|**0.92 ± 0.12**|0.13 ± 0.09|0.00 ± 0.00||1|\n|Faucet-close|**0.98 ± 0.04**|0.00 ± 0.00|0.50 ± 0.28||1|\n|Mean SR|**0.84**|0.07|0.16||1|\n___\n**We hope these revisions can meet your expectations, and we would be most grateful if you consider raising your score. Thanks!**\n\n[1]Learning Video-Conditioned Policy on Unlabelled Data with Joint Embedding Predictive Transformer.ICLR25\n\n[2]Become a Proficient Player with Limited Data through Watching Pure Videos.ICLR23\n\n[3]Mastering Atari Games with Limited Data.Neurips21\n\n[4]Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning.AAAI23\n\n[5]Mask-based Latent Reconstruction for Reinforcement Learning.Neurips22\n\n[6]Learning to act without actions.ICLR24\n\n[7]Decoupling representation learning from reinforcement learning.ICML21"}, {"review_id": "y7aSdhhoPi", "reviewer": "Reviewer_vqWk", "summary": "In this paper, it presents a new framework, Behavior Cloning from Videos via Latent Representations (**BCV-LR**), that performs imitation learning directly from expert videos without access to rewards or action labels. The method works in two stages: (1) an offline pretraining stage that learns a latent representation of video frames via first self-supervised representation learning and then extracts latent actions through a dynamics-based objective, and (2) an online finetuning phase where latent actions are aligned with real actions using minimal interactions in a reward-free environment. Experimental results demonstrate that BCV-LR achieves superior sample efficiency and outperforms state-of-the-art imitation learning and reinforcement learning baselines across 24 visual control tasks, including both discrete (Procgen) and continuous (DMControl) settings.", "questions": "1. For continuous control tasks, they are relatively easy with low-dimensional action spaces. I'm curious how would the proposed methods work for environments with high-dimensional action spaces. (For example, take humanoid & dog from Deepmind Control Suite)\n2. The author mention that during self-supervised representation learning phase, it also learns some temporal dynamics by aligning representation of $o_t$ with $o_{t+1}$, but this seems not appearing in Equation 1. So what's the overall objective for this stage. Is it Equation 1 (contrastive loss with image reconstruction ) plus temporal loss?\n3. The reconstruction loss of feature learning seems suboptimal especially for environments with distracting backgrounds. The authors mention that they get better performance with this loss added. But is it because the visual observations of the environments it tests on are too simple?\n4. A very relevant work [1] is missing.\n\n[1] Ye et al. Become a Proficient Player with Limited Data through Watching Pure Videos, ICLR 2023", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n1. The paper presentation is good and easy to follow\n2. Good motivation to learn from action-free videos\n3. Good empirical performance with great sample efficiency: the proposed BCV-LR framework demonstrates impressive sample efficiency in both discrete and continuous control tasks. It performs competitively with or surpasses methods that have access to ground-truth rewards or expert actions.\n\n**Weaknesses**:\nThe scope of the experiments are relatively limited. All experiments are conducted in simulation/\"toy\" environments (Procgen and DMControl). While these are standard benchmarks, it is unclear how well BCV-LR generalizes to internet-scale human videos and real-world robotics applications.", "was_revised": true, "final_justification": "This paper is well-motivated with great empirical performance demonstrated in its experiments. During the rebuttal process, the author has addressed my concern of low-dimensional action space and they have also added experiments of Metaworld to demonstrate the method's broader applicability. Therefore, I recommend accept for this paper.", "rebuttal": "We greatly appreciate the reviewer’s careful reading, detailed feedback, and recommendation of our paper! We have carefully reviewed your comments and further refined our paper based on your constructive reviews!\n\n___\n>**Weakness: The scope of the experiments are relatively limited. All experiments are conducted in simulation/\"toy\" environments (Procgen and DMControl). While these are standard benchmarks, it is unclear how well BCV-LR generalizes to internet-scale human videos and real-world robotics applications.**\n\nWe understand your concern! Since it is highly challenging to balance video imitation performance and efficiency without access to expert actions or expert rewards, our experiments have primarily been conducted in relatively standard visual RL environments to answer the open question \"is sample efficient ILV available\" and have not been extended to real-world tasks or internet-scale pre-training, which are of greater significance. Based on your feedback, we further conduct some extra experiments in Metaworld manipulation benchmark, which may demonstrate a wider application of BCV-LR.\n\nFor each Metaworld task, only 50k environmental steps are allowed, with remaining settings similar to that of DMControl. Results (success rate) are shown in Table I. In this interaction-limited situation, BCV-LR can still derive effective manipulation skills from expert videos without accessing expert actions and rewards, which demonstrates its wider range of applications and potential for generalizing to real-world manipulation tasks. \n\n\n\n**Table I**\n|Metaworld-50k|BCV-LR|BCO|DrQv2|/|Expert video|\n|-|-|-|-|-|-| \n|Faucet-open|**0.82 ± 0.20**|0.13 ± 0.19|0.00 ± 0.00||1|\n|Reach|**0.63 ± 0.25**|0.03 ± 0.05|0.13 ± 0.12||1|\n|Drawer-open|**0.92 ± 0.12**|0.13 ± 0.09|0.00 ± 0.00||1|\n|Faucet-close|**0.98 ± 0.04**|0.00 ± 0.00|0.50 ± 0.28 ||1|\n|Mean SR|**0.84**|0.07|0.16||1|\n\nFor internet-scale cross-domain video data, one challenge BCV-LR may face stems from the difficulty of transferring visual knowledge. Considering that BCV-LR is designed to be easily compatible with any action-free self-supervised tasks, this issue could potentially be alleviated by involving more advanced self-supervised objectives or using off-the-shelf pre-trained encoders. We have added the above discussion in the future work section to inspire more thinking.\n\n___\n>**Q1:For continuous control tasks, they are relatively easy with low-dimensional action spaces. I'm curious how would the proposed methods work for environments with high-dimensional action spaces. (For example, take humanoid & dog from Deepmind Control Suite)**\n\nFollowing your suggestion, we attempt experiments on the 'dog' and 'humanoid' domains. We find that these two domains are highly challenging even for advanced RL methods that use expert rewards. Specifically, we first try training agents to collect expert data via RL, but observe that both TACO and DrQ fail within 5M steps (obtaining scores lower than 5). Furthermore, we run DrQv2 on the 'humanoid_walk' task for 20M steps across 4 seeds, with only 1 run successfully learning a policy. We use this 20M-step policy to collect data, and then train BCV-LR and all baselines under the 100k-step setting—all of which failed. (Table II) This indicates that complex dynamics and action spaces remain significant challenges for current policy learning algorithms, even when expert rewards are provided. Meanwhile, BCV-LR, which is designed to balance sample efficiency under more challenging settings (without access to rewards or expert actions), is not yet able to handle such tasks effectively. \n\n**Table II**\n| Task-100k | BCV-LR  | LAIFO   | BCO  |UPESV |TACO  | DrQv2| / |Expert video|\n| - | - | - | - | - | - |  - | - | - |\n| humanoid_walk |1.3 ± 0.1  | 1.1 ± 0.0 | 1.1 ± 0.1 | 1.0 ± 0.1 | 2.0 ± 0.2 | 1.9 ± 0.2| |529|\n\n___\n>**Q2:The author mention that during self-supervised representation learning phase, it also learns some temporal dynamics by aligning representation of $o_t$ with $o_{t+1}$, but this seems not appearing in Equation 1. So what's the overall objective for this stage. Is it Equation 1 (contrastive loss with image reconstruction ) plus temporal loss?**\n\nAs shown in Sec.3.1.1 (line 174, main paper), BCV-LR is designed to be easily compatible with any action-free self-supervised tasks (it completely decouples visual learning from subsequent training), and it can adapt to different types of tasks by choosing appropriate self-supervised objectives. To this end, we apply different latent feature training losses for Procgen and DMControl. Concretely, we employ the 'contrastive loss with image reconstruction' (Eq.1, line 185, main paper) for Procgen video games, because it has been proven effective in these kinds of tasks [1,2]. For DMControl environments where the temporal understanding is crucial, we choose another prototype-based temporal association loss (Eq.10, line 549, Appendix) because involving temporal information into self-supervised objectives have been proven necessary in DMControl [3]. In summary, BCV-LR can involve more advanced self-supervised objectives for more challenging tasks if necessary, which makes its potential not limited to specific environments.\n\nWe have further improved the presentation in Section 3.1.1 based on the above details to enhance clarity for readers.\n\n___\n>**Q3:The reconstruction loss of feature learning seems suboptimal especially for environments with distracting backgrounds. The authors mention that they get better performance with this loss added. But is it because the visual observations of the environments it tests on are too simple?**\n\nWe agree with your opinion on reconstruction loss, and some previous works [6] have demonstrated the limitation of reconstruction when faced with visual distraction. As we explain in our answer to Q2, BCV-LR employs the reconstruction loss in Procgen because of its effectiveness in video games [1], but it is not limited to this reconstruction loss. BCV-LR is designed to be easily compatible with any action-free self-supervised tasks (it completely decouples visual learning from subsequent training), and it can adapt to different types of tasks by choosing appropriate self-supervised objectives. To this end, BCV-LR can involve more advanced self-supervised objectives (e.g., ViT-based masked reconstruction [4,5] or distraction-robust prototypical representation [6]) for more challenging tasks if necessary. In addition, BCV-LR can also be combined with an off-the-shelf, well-trained encoder, which makes its potential not limited to both tasks and experimental settings. \n\n___\n>**Q4:A very relevant work [1] is missing.**\n\nWe appreciate your careful reading, and now have supplemented the discussion of FICC [1] in the related work (line 121): \"FICC[47] pre-trains world models on action-free agent experience, accelerating MBRL on Atari.\" Thank you!\n\n\n\n___\n\n**We hope these answers can meet your expectations, and we would be most grateful if you maintain your recommendation or raise your score. Thanks!**\n\n[1]Become a Proficient Player with Limited Data through Watching Pure Videos. ICLR 2023\n\n[2]Curl: Contrastive unsupervised representations for reinforcement learning. ICML 2020\n\n[3]Reinforcement learning with prototypical representations. ICML 2021\n\n[4]Masked world models for visual control. CoRL 2023\n\n[5]Mask-based Latent Reconstruction for Reinforcement Learning. Neurips 2022\n\n[6]Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representations. ICML 2022"}, {"review_id": "6Anl6xwmRI", "reviewer": "Reviewer_HHys", "summary": "The paper proposes a method for Imitation Learning from Videos, that is, the problem of learning a policy from videos of expert demonstrations with no action information. To do this, the authors propose \"Behavior Cloning from Videos via Latent Representations\" (BCV-LR).\n\nThe method: 1. learns latent features from video through self-supervision over video frames; 2. then learns a separate world model on the latent features, together with a latent action model; 3. online finetunes the latent action model and a learns a latent action decoder; 4. simultaneously to (3.), trains a latent policy that uses the latent action decoder to act.\n\nThe authors conduct experiments across discrete control and continuous control benchmarks, comparing their method to other ILV and RL based baselines.", "questions": "* As above: how exactly are phases 3 and 4 optimally alternated? Does this matter?\n* How does the method compare to performing the steps in a fashion akin to LAPA? (No online learning, only imitation learning alignment on ground truth expert action?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* The paper focuses on an important and timely problem, as imitation learning becomes widespread state of the art for many real world robotics applications in manipulation and beyond. Many are attempting to learn from large scale video, as it promises to be an extremely cheap and scalable data source of scarce robotics data.\n* The paper operates with state of the art methods in representation learning (self-supervision, latent actions, etc) and provides additional evidence towards a \"representation-learning first\" approach to solving traditional RL tasks and benchmarks. Specifically, the method's performance rivals or beats that of pure RL methods that have access to environmental rewards.\n* The comparisons with state of the art methods are extensive (within the chosen eval suite) and prove that the method beats alternatives.\n\nWeaknesses:\n* The overall method is complex, consisting of many stages that all have to work for the method to achieve its performance. The ablation study does however demonstrate that each component is required.\n* Stage 3. (as per the summary above), in which the latent action model is finetuned and the action decoder is trained, requires real actions to be observed via environment interaction. It is unclear what is the impact of the schedule of environment interactions, how exactly the phases are optimally alternated so as to sample iteratively from the policy learned online via the concurrent step 4. Does this matter?\n* The method as requested does require a source of ground truth actions from the videos, it is thus not strictly suitable for solving pressing hard problems for real world robotics, e.g. learning from real human videos and then transferring the policies to robots. This is also related to the benchmarks being limited to classic \"toy RL\" simulation environments.\n* How does the method compare to performing the steps in a fashion akin to LAPA? (No online learning, only imitation learning alignment on ground truth expert action?", "was_revised": true, "final_justification": "The in-depth rebuttal further strengthens the paper's claim, I will thus maintain my acceptance score.", "rebuttal": "We are deeply grateful to the reviewer for your careful reading, constructive reviews, and strong recommendation of our paper! We have further enhanced our paper in light of your feedback by supplementing more experiments and discussions!\n\n___\n>**Weakness1：The overall method is complex, consisting of many stages that all have to work for the method to achieve its performance. The ablation study does however demonstrate that each component is required.**\n\nThank you for your recognition of our experimental section. We agree with your opinion that the methodology section of BCV-LR is relatively complex. This complexity arises because achieving sample-efficient online policy learning from videos without access to expert actions or rewards is an extremely challenging task. To address this, BCV-LR incorporates multiple training stages, which allow it to obtain as much useful information as possible from both expert videos and environmental samples, which ultimately contributes to its promising results.\n\n___\n>**Weakness2&Q1：Stage 3. (as per the summary above), in which the latent action model is finetuned and the action decoder is trained, requires real actions to be observed via environment interaction. It is unclear what is the impact of the schedule of environment interactions, how exactly the phases are optimally alternated so as to sample iteratively from the policy learned online via the concurrent step 4. Does this matter?**\n\nThe BCV-LR online stage contains three parts. First, we (a) allow the agent to interact with the environment for a fixed number of times using its policy and collect transitions to enrich the experience buffer. Immediately after that, (b) we perform finetuning of the latent action and training of the action decoder on the experience buffer. Then, (c) we train the latent policy to imitate the finetuned latent action predicted from expert videos. After this, the BCV-LR policy is improved and we return to part a) to collect better training data, which forms a cyclic online policy learning. This alternation is intuitive, that is, first collect new data of higher quality, then use the better data to obtain the better latent actions from expert videos, and finally let the policy learn the better latent actions to achieve performance improvement. You can refer to Sec.A.1 (Appendix) for concrete pseudo code of BCV-LR and we will further clarify the descriptions in the methodology section to make them more comprehensible to readers.\n\n**For the impact of the schedule of environment interactions.** In previous experiments, we did not adjust the number of interactions for each cycle, but kept it fixed at a relatively large value from start to finish (for example, we fixed the number of collected transitions in part (a) as 1000 in DMControl) and ultimately achieved satisfactory results. To answer your question, we conduct additional experiments, where we try a much smaller interaction number (1000->2) and correspondingly reduce the number of update times for latent actions (100->1) and latent policies (1000->2) in each interaction, making BCV-LR perform in a fashion akin to off-policy RL. We denote this variant as 'BCV-LR(1000->2)'. The results in Table I show that BCV-LR can still achieve effective learning, which demonstrates that BCV-LR is robust to the schedule of environment interactions.\n\n**Table I**\n| task  | BCV-LR(1000->2) |  BCV-LR |   DrQv2| / |Expert video|\n| - | - | - | - | - | - | \n| reacher_hard |  875 ± 65 |  **900 ± 31**  | 92 ± 98  | | 967 |\n| finger_spin  | **956 ± 20**  | 942 ± 48  | 374 ± 264 || 981|\n\n\n\n___\n>**Weakness3: The method as requested does require a source of ground truth actions from the videos, it is thus not strictly suitable for solving pressing hard problems for real world robotics, e.g. learning from real human videos and then transferring the policies to robots. This is also related to the benchmarks being limited to classic \"toy RL\" simulation environments.**\n\nWe understand your concerns! Since it is highly challenging to balance video imitation performance and efficiency without access to expert actions or expert rewards, our experiments have primarily been conducted in relatively standard visual RL environments to answer the open question \"is sample efficient ILV available\" and have not been extended to real-world tasks or internet-scale pre-training, which are of greater significance. Taking your comment into account, we further conduct some extra experiments in Metaworld manipulation benchmark, which may demonstrate a wider application of BCV-LR for robots. \n\nFor each Metaworld task, only 50k environmental steps are allowed, with remaining settings similar to that of DMControl. Results (success rate) are shown in Table II. In this interaction-limited situation, BCV-LR can still derive effective manipulation skills from expert videos without accessing expert actions and rewards, which demonstrates its wider range of applications and potential for generalizing to real-world manipulation tasks.\n\n**Table II**\n|Metaworld-50k|BCV-LR|BCO|DrQv2|/|Expert video|\n|-|-|-|-|-|-| \n|Faucet-open|**0.82 ± 0.20**|0.13 ± 0.19|0.00 ± 0.00||1|\n|Reach|**0.63 ± 0.25**|0.03 ± 0.05|0.13 ± 0.12||1|\n|Drawer-open|**0.92 ± 0.12**|0.13 ± 0.09|0.00 ± 0.00||1|\n|Faucet-close|**0.98 ± 0.04**|0.00 ± 0.00|0.50 ± 0.28 ||1|\n|Mean SR|**0.84**|0.07|0.16||1|\n\nFor cross-domain video data (e.g., human videos), one challenge BCV-LR may face stems from the difficulty of transferring visual knowledge. Considering that BCV-LR is designed to be easily compatible with any action-free self-supervised tasks, this issue could potentially be alleviated by involving more advanced self-supervised objectives or using off-the-shelf pre-trained encoders. We have added the above discussion in the future work section to inspire more thinking.\n\n\n___\n>**Weakness4&Q2: How does the method compare to performing the steps in a fashion akin to LAPA? (No online learning, only imitation learning alignment on ground truth expert action?**\n\nTo answer your questions, we further conduct additional experiments, where we let BCV-LR perform latent action finetuning and policy imitation with a few action-labeled expert transitions. Concretely, we maintain the original pre-training stage and use 10k offline action-labeled expert transitions to achieve offline latent action finetuning and policy cloning with the original losses. The results in Table III (RL denotes DrQv2 for DMControl and PPO for procgen) demonstrate that BCV-LR can also achieve offline imitation learning well if expert actions are provided. \n\n**Table III**\n|| task  | BCV-LR-offline |  BCV-LR |  RL | / |Expert video|\n|-| - | - | - | - | - | - | \n|DMControl| reacher_hard |  **938 ± 44** |  900 ± 31  | 92 ± 98  | | 967 |\n|| finger_spin  | **978 ± 7**  | 942 ± 48  | 374 ± 264 || 981|\n|Procgen| fruitbot  | **27.7 ± 0.4**  | 27.5 ± 1.5  | -1.9 ± 1.0 || 29.9|\n\n\nOf course, we would like to say that BCV-LR is designed for the ILV (imitation learning from videos) problem, a much harder variant of the classical ILO (imitation learning from observation only) problem, where the employment of unsupervised online policy training without accessing expert actions adheres to the norms of this field.\n\n___\n**Thanks for your careful reading and for recommending our paper!**"}, {"review_id": "816B2EHifJ", "reviewer": "Reviewer_TLPv", "summary": "This paper introduces a method for sample-efficient imitation learning from videos. This works by first training an encoder on videos by using a contrastive loss between the frame and a shifted version of the frame, as well as a temporal loss for the next frame feature similarity. This paper also uses a world model to learn a latent action representation, which is then mapped to a real world action using a few expert trajectories. The paper evaluates on video-game tasks and compares to many prior works.", "questions": "How general is the learned encoder? Is the shift image + contrastive loss learn useful features for non-video games, say real world robotics tasks? Or will this only work for relatively simplistic videos/environments?\n\nHow well does the mapping from latent action to action work in more complex environments, e.g., robot controls rather than just video game controls?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is well written, easy to understand, and presents an approach to a important problem. The results are good, showing more sample efficient learning than prior works. The idea is interesting, and learning from videos without requiring action annotations is a hard problem.\n\n\nThere are a few weaknesses, mostly in that the training of the encoder seems limiting, and I have doubts it will generalize to more complex and realistic environments. Same for the mapping from the latent actions to real actions. The main novelty of the approach is in taking these parts and making them work for the task, ie. using the contrastive pre-training + world model + latent action + action mapping. So if these methods don't generalize to real world tasks, it will greatly limit the impact of the paper.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns. I think it is a good paper with meaningful contributions.", "rebuttal": "We greatly appreciate the reviewer's careful reading, constructive reviews, and strong recognition of our work! We have carefully reviewed your comments, addressed your questions, and further improved our paper based on your feedback!\n___\n>**Q1: How general is the learned encoder? Is the shift image + contrastive loss learn useful features for non-video games, say real world robotics tasks? Or will this only work for relatively simplistic videos/environments?**\n\nWe apply this ‘shift image + contrastive loss’ with reconstruction (Eq.1, line 185, main paper) for Procgen video games because it has been proven effective in these kinds of tasks [1,2]. As shown in Sec.3.1.1 (line 174, main paper), BCV-LR is easily compatible with any action-free self-supervised tasks, and it can adapt to different types of domains by choosing appropriate self-supervised objectives. This motivates us to choose another prototype-based temporal association loss (Eq.10, line 549, Appendix) for DMControl environments where the temporal understanding is crucial and this loss has been proven better than ‘shift image + contrastive loss’ in DMControl [3]. \n\nTo this end, BCV-LR can involve more advanced self-supervised objectives (e.g., ViT-based masked reconstruction [4,5]) for more challenging tasks (e.g., real-world manipulation) if necessary. In addition, BCV-LR can also be combined with an off-the-shelf, well-trained encoder, which makes its potential not limited to only video game environments.\n\n\n___\n>**Q2: How well does the mapping from latent action to action work in more complex environments, e.g., robot controls rather than just video game controls?**\n\nIn addition to the video games, we also demonstrate the advantages of BCV-LR on DMControl benchmark which consists of several continuous robotic control tasks. We summarize the average results below. More details are provided in Sec.4.3 (main paper) and Sec.C.1 (Appendix).\n\n**Table I**\n| DMControl-8-tasks-100k  | BCV-LR  | LAIFO   | BCO  |UPESV |TACO  | DrQv2| / |Expert video|\n| - | - | - | - | - | - |  - | - | - |\n| Mean Score | **604**   | 158  | 336  | 18     | 310  | 232 | | 698|\n| Video-norm Mean Score  | **0.78**   | 0.20  | 0.31  | 0.03     | 0.45  | 0.34 | | 1.00|\n\n\nIn addition, we further conduct additional experiments in the Metaworld manipulation benchmark. Only 50k environmental steps are allowed for each Metaworld task, with remaining settings similar to that of DMControl. Results (success rate) are shown in Table II. In this interaction-limited situation, BCV-LR can still derive effective manipulation skills from expert videos without accessing expert actions and rewards, which demonstrates its wider range of applications and potential for generalizing to real-world manipulation tasks. \n\n**Table II**\n|Metaworld-50k|BCV-LR|BCO|DrQv2|/|Expert video|\n|-|-|-|-|-|-| \n|Faucet-open|**0.82 ± 0.20**|0.13 ± 0.19|0.00 ± 0.00||1|\n|Reach|**0.63 ± 0.25**|0.03 ± 0.05|0.13 ± 0.12||1|\n|Drawer-open|**0.92 ± 0.12**|0.13 ± 0.09|0.00 ± 0.00||1|\n|Faucet-close|**0.98 ± 0.04**|0.00 ± 0.00|0.50 ± 0.28 ||1|\n|Mean SR|**0.84**|0.07|0.16||1|\n\n\n\n___\n\n**Thanks for your careful reading and for recommending our paper!**\n\n\n[1]Become a Proficient Player with Limited Data through Watching Pure Videos. ICLR 2023\n\n[2]Curl: Contrastive unsupervised representations for reinforcement learning. ICML 2020\n\n[3]Reinforcement learning with prototypical representations. ICML 2021\n\n[4]Masked world models for visual control. CoRL 2023\n\n[5]Mask-based Latent Reconstruction for Reinforcement Learning. Neurips 2022"}]

Author rebuttal:

Meta-review: The paper proposes BCV-LR, a novel framework for imitation learning from videos (without expert actions but with online interaction) that extracts latent action representations through self-supervised pretraining and aligns them to real action spaces via online interaction. The method demonstrates highly sample-efficient learning, in some cases reaching expert-level performance.

The main concerns are the limited evaluation on relatively simple, short-horizon environments and the complexity of the multi-stage pipeline. The paper also omits discussion of and comparison to a highly relevant prior work [1]. Reviewers found the contribution promising but not fully validated at scale. The rebuttal clarified design choices and provided MetaWorld experiments, but did not address the experimental limitations. 

Overall, I lean toward accept: the approach is interesting and results are strong, but more exhaustive ablations and evaluation on realistic tasks and videos would substantially strengthen the paper.


[1] Escontrela et al. "Video Prediction Models as Rewards for Reinforcement Learning" NeurIPS 2023
Decision: Accept (poster)