Watch Less, Do More: Implicit Skill Discovery for Video-Conditioned Policy
Authors: Jiangxing Wang
Zongqing Lu
Venue: ICLR 2025 Poster (2025)
Keywords: video-conditioned policy
compositional generalization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: In this paper, we study the problem of video-conditioned policy learning. While previous works mostly focus on learning policies that perform a single skill specified by the given video, we take a step further and aim to learn a policy that can perform multiple skills according to the given video, and generalize to unseen videos by recombining these skills. To solve this problem, we propose our algorithm, Watch-Less-Do-More, an information bottleneck-based imitation learning framework for implicit skill discovery and video-conditioned policy learning. In our method, an information bottleneck objective is employed to control the information contained in the video representation, ensuring that it only encodes information relevant to the current skill (Watch-Less). By discovering potential skills from training videos, the learned policy is able to recombine them and generalize to unseen videos to achieve compositional generalization (Do-More). To evaluate our method, we perform extensive experiments in various environments and show that our algorithm substantially outperforms baselines (up to 2x) in terms of compositional generalization ability.

Published as a conference paper at ICLR 2025
WATCH LESS, Do MORE: IMPLICIT SKILL DISCOVERY FOR VIDEO-CONDITIONED POLICY
Jiangxing Wang
Zongging Lu"
School of Computer Science
School of Computer Science Peking University, BAAI zongqing. luêpku edu.cn cn
Peking University
j angxi i wê wts.u.. wwtp:pkku.duu...... edu.cn
ABSTRACT
In this paper, we study the problem of video-conditioned policy learning. While previous works mostly focus on learning policies that perform a single skill spec- ified by the given video, take step further and aim to learn a policy that perform multiple skills according to the given video, and generalize to unseen videos by recombining these skills. To solve this problem, we propose our algo- rithm, Watch-Less, Less-Do-More an information bottleneck-basee imitation learning framework for implicit skill discovery and video-conditioned policy learning. our method, an information bottleneck objective is employed to control the infor- mation contained the video representation, ensuring that it only encodes infor- mation relevant to the current skill (Watch-Leess) By discovering potential skills from training videos, the learned policy is able to recombine them and generalize to unseen videos to achieve compositional generalization (Do-More). To evalu- ate our method, we perform extensive experiments in various environments and show that our algorithm substantially outperforms baselines (up to 2x) in terms of compositional generalization ability.
INTRODUCTION
As large language models (LLMs) have demonstrated remarkable zero-shot and few-shot general- ization abilities (Brown et al., 2020; Ouyang et al., 2022), the research focus of decision-making policies has also shifted from addressing specific task, such mastering environment via inforcement learning (Sutton, 2018) or replicating a dataset via imitation learning (Hussein et al., 2017), to completing diverse tasks based on given instructions. These instructions can be treated as goals for the decision-making models, and encompass modalities such as text (Nair et al., 2022; Carta al., 2023), goal image (Yadav al., 2023b;a), or future state (Cui et al., 2022: Lee al., 2024). To achieve such conditiioon policies, a variety of methods have been proposed and achieved great success across multiple domains (Liu et al., 2022). However, the aforementioned goal specifications often overlook dynamic information, such as the ordering of task completion or the method of task completion (if there are many). In contrast, video offers natural way to represent these details, thereby leading to line of research exploring video-conditioned policy learning (Eze & Crick, 2024b).
Existing methods for video-conditioned policy learning have been applied to various scenarios, in- cluding robotic manipulation (Chane-Saa Sane et al., 2023; Shin et al., 2023; Jiang et al., 2023), navi- gation (Zhou et al., 2024), open-world agent (Cai et al., 2024), and autonomous driving (Shin al., 2024). Taking different videos as input, the learned policy can be deployed to perform different skills to solve corresponding tasks. However, these methods often consider only the video demonstration of single task (Chane-Sane et al., 2023) and only the object-level generalization (Jiang et al., 2023). In real-world applications, we often want the learned policy perform of different skills to achieve a combination of multiple tasks.
When the video demonstration of a task combination is given, an ideal policy should directly per- form skills as demonstrated in the given video. To train such a policy, researchers have explored
*Corresooniing author.

---

Published as a conference paper at ICLR 2025
skill-based imitation learning methods (Xu et al., 2023; Wang al., 2023; Shin et al., 2023; 2024). However, these methods often require explicit video segmentation annotations, or videos of another embodiment train a process. Therefore, in this paper, we consider whether it is possible to learn a video-conditioned policy that can perform multiple skills without these requirements. Moreover, as we consider videos of different task combinations, we also expect the learned policy to achieve compositional general- ization (Lin et al., 2023), that is, it can still perform well on task combinations that have not been during training.
policy, which greatly increases the difficulty of the data collection
To fulfill such an expectation, we propose our algorithm, Watch-Less Leacc-Less--D-MMeee Do-More (WL-DM), an in- formation bottleneck-basc conditioned policy learning. For a video-conditioned policy, the given video can be considered as a sequence of tasks. As the policy can only work on one task at a time, it should be able to perform well by focusing only on the current task, instead of the entire task sequence. Based on this intuition, WL-DM employs the information bottleneck method (Tishby et al., 2000) to control the information contained in the video representation. This is accomplished by 1) minimizing the mutual informa- tion (Cover, 1999) between video and its representation to reduce the information contained in the video representation and 2) maximizing the mutual information between video representation and the current skill preserve enough information related to the current task. To better understand the effect of this information bottleneck method, we further build a theoretical connection between the proposed method and the intuition behind it. Using this method, WL-DM makes the learned policy only to consider the current task, which achieves the implicit video segmentation without requiring explicit video segmentation annotations. The advantage of considering only the current task can be related to the compositional generalization ability. When an unseen video is given, the conei-condi policy learned by WL-DM can implicitly decompose the unseen video into seen tasks, and perform corresponding skills, thus facilitating the compositional generalization ability of the learned video-conditioned policy. To further validate our algorithm, we propose practical implementation of our method and conduct various empirical evaluations across diverse environ- ments. The experimental results indicate that WL-DM achieves substantially better compositional generalization ability than baselines, demonstrating the effectiveness of method.
imitation learning framework for implicit skill discovery and video-
Our contributions can be summarized as follows:
We propose our method, Watch-Less-DD -o-soe (WL-DM), an information bottleneck- based imitation learning framework for implicit skill discovery and video-conditioned pol- icy learning, where two different mutual information terms work together ensure the video representation contains only information related to the current task.
The intuition behind WL-DM is that the optimal policy should behave similarly when con- ditioned on all tasks and when conditioned on only the current task. To better explain our method, we further build a theoretical connection between WL-DM and this intuition.
We propose practical implementation of our algorithm and perform empirical evaluations in Frank Kitchen (Gupta et al., 2020) and Meta world (Yu et al., 2020) to demonstrate the effectiveness of WL-DM. The experimental results indicate that WL-DM achieves (up to 2x) better compositional generalization ability compared to baselines.
2 RELATED WORK
2.1 LEARNING FROM VIDEOS
Using massive Internet data to train language models has been proven to be successful and has re- sulted in trend of research on large language models (Brown et al., 2020; Touvron et al., 2023). Inspired by this success, researchers have begun to pay attention to another type of data wildly avail- able on the Internet, video data, and produced series of studies on learning from videos (MeCarthy et al., 2024; Eze & Crick, 2024a). For decision-making ways, such as reward function learning (Escontrela et al., 2023; Sermanet et al., 2018; Chen et al., 2021a), dynamic model learning (Baker et al., 2022), representation learning (Nair et al., 2023), and policy learning (Jang et al., 2022; Jiang et al., 2023; Chane-Sate Sane et al., 2023; Shin et al., 2023; 2024). Our paper belongs to the last category. that is, using video demonstrations as instructions to learn video-conditioned policy. It is worth noting that previous work in this category often focuses
models, video data can be used in various
2

---

Published as a conference paper at ICLR 2025
only demonstration videos containing single task (Chane-Sane et al., 2023), requires aligned data of other modalities (Jang et al., 2022; Shin et al., 2023; 2024). This can be attributed to the lack of clear goal labels in demonstration videos (McCarthy et al.. 2024). Therefore, when dealing with videos containing multiple tasks, we often need to introduce information in other modalities to provide segmentation annotations for the video, to distinguish the tasks to be completed each stage (Shin et al., 2023; 2024). Unlike previous work, in this paper, we attempt to directly learn a video-conditioned policy capable of handling videos containing multiple tasks, without introducing additional segmentation annotations.
2.2 ONE-SHOT IMITATION LEARNING
One-shot imitation learning was originally introduced in Duan et al. (2017), where the goal of this problem is learn policy that can quickly adapt to a new task given single demonstration. For one-shot imitation learning, we can achieve it through different learning methods such as meta- learning (Duan et al., 2017; Finn et al., 2017), supervipe learning (Wu et al., 2024), and imitation learning (Jang et al., 2022; Cui et al., 2022; Jiang et al., 2023). Specifically, our method falls into the last category: we assume the existence of an imitation learning dataset paired with video demonstrations, such that we can use this dataset to train a video-conditioned policy.
One-shot demonstrations can be presented in various formats, such as trajectories (Cui et al., 2022; Lee et al., 2024), videos (Dasari & Gupta, 2021; Jain et al., 2024; Wang et al., 2023; Xu et al., 2023; Chane-Sane et al., 2023), multimodal information (Jiang et al., 2023; Shin et al., 2023; 2024), etc. In this paper, consider adapting to new tasks through video demonstrations, that is, one-shot video imitation learning. In previous work, video demonstrations often only include a single task, and the adaptation to new tasks mainly focuses differences at the embodiment and object level (locations, textures, etc.) (Dasari & Gupta, 2021; Mandi et al., 2022; Chane-Sane et al., 2023). Unlike these studies, we consider video demonstrations containing multiple tasks and focus on adaptation the level of task combination. For this setting, previous work generally assumes the existence of data corresponding to another embodiment (Wang et al., 2023; Xu et al., 2023) or assumes information in other modalities to provide video segmentation annotations (Shin et al., 2023; 2024). Unlike these works, we do not assume additional data and learn a video-conditioned policy that can finish multiple tasks solely through the information contained in the videos.
2.3 COMPOSITIONAL GENERALIZATION
Due to the compositional nature of natural language, most previous work considers the composi- tional generalization problem over language instructions. For example, Oh et al. (2017) proposed a method based on hierarchical reinforcement learning that enables the policy to generalize to unseen command combinations and longer command sequences at test time. Stengel-Eskin et al. (2022) combined the transformer model and the masking mechanism to obtain generalization over object combinations. The attention mechanism for compositional generalization was further investigated by Spilsbury & Ilin (2022), and method utilizing sparse factored attention for goal identifica- tion was proposed. Modular architecture is another way to induce compositional generalization. Carvalho et al. (2023) proposed modular successor features to enhance the compositional general- ization ability, and Logeswaran et al. (2023) directly considered an additive decomposition of the state-action value function obtain the generalization ability over language instructions.
Unlike these studies, we consider the generalization across different task combinations based on video demonstrations. During training, we only have access to a subset of task combinations and their corresponding video demonstrations. Our goal is to enable the policy to decompose different tasks from the videos and acquire skills to solve these tasks. reproduce an unseen video demonstration by combining a set of skills learned in the training set. This setting has been studied by Wang et al. (2023); Xu et al. (2023); Shin et al. (2023; 2024). How- ever, Wang et al. (2023); Xu et al. (2023) focused on the cross-embodiment scenario, thus requiring video data from another embodiment, and Shin et al. (2023; 2024) required language information to provide segmentation annotations for videos. Unlike them, our method incorporates an information bottleneck based objective to achieve implicit video segmentation and skill discovery, without the need for other sources of information.
test time, the policy expected to
3

---

Published as a conference paper at ICLR 2025
3 PROBLEM FORMULATION
In this paper, we consider the vidco-conditioned policy learning problem. This problem can be formulated as special case of the goal-conditioned Markov Decision Process (MDP) (Nasiriany et al., 2019) and defined by a tuple (S, A, P, R. Po. 7). Similar to the general MDP, S is the set of states, is the set of actions, P( at) the transition probability, Pe is the initial state distribution and is the discount factor. Additionally, we have Ç as the set of goals, which will also affect the reward function R(s,, de: (e: 9). For a goal-conditioned policy π (8e 9) with given goal g, we want it to maximize the following objective:
[ = Bap-po-aamm
Σ
(r(ss, 0t, 0t,9)]-
As we focus on the video-conditioned policy learning, we assume our goals to be videos G V. such that goal-conditioned policy π (a, 9).g) becomes a video-conditioned policy T v). Moreover, we consider the case where each video u = (kg. ........y kyy contains multiple tasks k E T, where N is the number of tasks and T is the set of all possible tasks. To evaluate the compositional generalization ability of the video-conditioned policy, we assume two video sets Virain and Veest. such that there and both video sets contain all possible tasks UvEVenins k = UsEViun k = T. The video- conditioned policy will be trained in Vtrain to maximize EnoPPtrai J(TT) and will be tested in Veest terms of PeeP [(TT), where Ptrain and Prest are uniform distributions across Verain and Veest respectively.
overlapping between the train video and test video set Verain O Vestt 2
4 METHOD
In this section, introduce method, Watch-Less-Do-Mer (WL-DM), The intuition behind our method is that we want the cideoccondttione policy to make decisions relying not on the entire video, but only information related the current task, thereby achieving implicit video segmen- tation and skill discovery. To achieve this, we propose an information bottleneck-based objective and theoretically establish the connection between this objective and our intuition. By decomposing training videos into combination of different skills, the video-conditioned policy can handle un- seen videos by recombining these skills to complete the required task combinations demonstrated in the unseen video.
4.1 INTUITION: FOCUSING ON THE CURRENT TASK
As formulated in Section 3, we assume that each video u contains N tasks lkg.... .ky] that need to be completed and the completion of these tasks is independent. In this case, further assume training set D = [Ti, ) where = (sn, ay ... 88, az)); is the expert trajectory corresponding to the video v (Jo.-.. (r) and fa is the video frame at each timestep. Given such dataset, can easily Iearn a coneiioondiionned policy π T(a,ss v) through imitation learning (Hussein et al., 2017) that can complete different task combinations given different videos, at least within the coverage of the training set. For example, the policy can be trained via the following behavior-cloning loss:
LBC(f, Φ) = -Eg,,; anD
log : (ailss, m)
- Eag DnD
log la (a (1.))))))))))))))))))))))))))))) 8o(1))
(1)
where Es is the video encoder and lo is the action decoder.
potential problem with this training method is that, when the size of our training set limited, the learned policy can easily overfit (Ying, 2019) to videos in the training set. This problem causes the learned policy to focus too much on the details of these videos to distinguish them completely, while ignoring the fact that these videos are composed through elements of the same task set. In such a case, when an unseen video is given, i.e., an unseen combination of tasks, the performance of the learned policy may decrease dramatically due to the overfitting issue. To address this problem, we need to focus on the fact that all videos are composed through elements of the same task set T. Even for those unseen videos, although the corresponding task combinations not included in the
4

---

Published as a conference paper at ICLR 2025
training set, each task that constitutes them has already been covered in the training set. Therefore, we can decompose vidcos into individual tasks and train the policy based on the decomposed tasks, such that π (a)//5, v) =Taa Se: Veur). where Veur is the video segment corresponding to keur and u = (Veur, Vother), the policy can then handle unseen videos as all the skills demonstrated in the videos have been covered and trained in the training set, which is commonly known as compositional generalization (Lin et al., 2023). However, such a task-level video segmentation annotation could be inaccessible in many cases. In this paper, we do not assume this kind of annotations as in previous work (Shin et al., 2023; 2024). To achieve similar effect, we propose an information bottleneck- based method, allowing the policy to implicitly decompose demonstration videos, enabling to rely only on the information related to the current skill when making decisions, thereby achieving the compositional generalization ability.
4.2 INFORMATION BOTTLENECK FOR VIDEO-CONDITIN
POLICY LEARNING
As described in Section 4.1, when all tasks can be completed independently, video-conditioned policy T [0)]88,, v) = π (ate ( Veur) can achieve the compositional generalization ability. However, video contains information about not only the current task keur but also other tasks kother- There- fore, we need an additional objective to train the video encoder ga, such that it produces a similar representation for v and Veur. and we can then ensure ( $t, v) ( (st, Veur). ). To achieve this, we need to reduce the mutual information between video representations h, and the video segments of other tasks Vothers and the reason can be seen from the following theorem:
Theorem 1. If we have Mi(he Vother (s, Veur) then DKL (1 (als,, (alss Vear)) 0 for all state-video pairs (8, v) E 8 x V with non-zero probability P(s,0) v) > O.
Proof. See Appendix B.
This theorem suggests the necessity of reducing the information of other tasks contained in the video representation. However, since we do not assume any video segmentation annotation, we cannot directly obtain segments corresponding to the current task and other tasks from the video, and therefore cannot directly manipulate the mutual information. Hence, we use constructive method to manipulate the information in the video representation indirectly. Specifically, we first minimize the mutual information between the video representation h. and the entire video minimize the amount of information contained in the video representation. At the same time, we maximize the mutual information between the video representation and some approximation of the current skill (which will be discussed later). Since different skills are required for performing different tasks, we can in this way indirectly ensure that the video representation still retains certain amount of information about the current task. Putting these two terms together, we can construct the following objective, which often referred to as the information bottleneck (Tishby al., 2000):
LiB MI(h;; v|s) MI(h;; 2|s),
where 2 is an approximated representation of the current skill, and a is the coefficient for the trade- off between two mutual information terms. As discussed in Tishby & Zaslavsky (2015), the infor- mation bottleneck is often used to Iearn a compact representation, which in our case is to dismiss the irrelevant part kother and retain the relevant part keur- the following two sections, we discuss how to compute this objective in practice.
4.3 MINIMIZING MUTUAL INFORMATION WITH VIDEO
The first term in Equation (2) is to minimize the mutual information between video representation h, and the entire video v. By expanding this term, we have:
Mi(ha; v|s) = Er()) [KL (gs (hols, e)]l
(3)
where P(s) and P(v) represent the state and video distribution, respectively. P(h, is marginal distribution P(h s) = Ep(w) Bo (hyls, s, v). As estimating this marginal distribution could be in- tractable in practice, previous work (Goyal et al., 2018; Eysenbach et al., 2021) commonly ap- proximates it with some prior ss) As we can see from Equation (3), the goal of this objec- tive is to minimize the distance between the video representation produced by the video encoder
5

---

Published as a conference paper at ICLR 2025
ko
1
k-2
K3
WL-DM
CMIT
MHA
MHA
B$
B.p.
LMI
Decoder
Decoder falill
Decoder
F
fg
ag
a2
aly
Gg
Luc
Figure 1: Overall Framework of WL-DM. We introduce an information bottleneckk-aat based objective achieve implicit video segmentation and skill discovery. Blocks with different colors represent different tasks. MHA stands for Multi-head Attention.
h = to (hylss v) and some prior ((h,(s) that does not consider video v at all. As shown later in the experiment, such target for distance minimization is undesirable as it induces too much loss of video information. To solve this problem, we need to find a better alternative for g(h,ls) such that the loss of video information can be controlled proper level.
Recall the intuition in Scction 4.1, we want the representation to be only related to the video segment of the current task Veur- Although cannot access the precise Veur, do have access to the video v = (Jo. ... fr), which allows us to approximate Veur using a future video segment Veur = (fe, fi+1, ft+)) for state st, where L is randomly sampled window size. Therefore, can use prior perior (hw]s, Veur), which leads to the minimization Vour as the input and get a prior video encoder b
of the following equation:
Er(g Er(w) DKL go (hols, v)ll e5 g (huls, Veur))
(4)
With this prior encoder, we can get the final objective for mutual information minimization:
CMr- E. DKL. go (h./ r) e) prior (h,ls, Veur))
(5)
In addition to Equation (5), the prior encoder prior is also trained via behavior cloning similar to Equation (1) with another action decoder Ig attached after it.
g
4.4 MAXIMIZING MUTUAL INFORMATION WITH SKILL APPROXIMATION
Another term in Equation (2) is to maximize the mutual information between the video represen- tation h, and some skill approximation z. We follow Yuan et al. (2024) and use the short-term behavior (a,, d++1, ... aa+-)) the representation of skills for state St, where M is domly sampled window size. To enhance the level of abstraction of the skill representation, we further propose first cluster all actions in the training dataset D and then the cluster id x, of each action to improve the skill representation, such that = = (Ie, #t+1, ... , As the mutual variables, maximize MI(he; 2), we can
information to measure the dependency between simply maximize log ((,)).)) As we have 2 = (Iz, Zt+1, M++M), similar to Yuan et al. (2024), we can decompose the above maximization into each timestep and get the final objective for mutual information maximization:
LMT -E,,
log gakill (xr|s
(x.lst {o(1))

---

Published as a conference paper at ICLR 2025
where we introduce the skill decoder fi to enhance the dependency between h4 and 2.
4.5 SUMMARY
Putting Equations (1), (5) and (6) together, now have the total loss for WL-DM:
LWL-DM = LBC +a1 CMI- +a2 LMIt,
where we have two coefficients αi and a2 to balance the scale of these three terms. The overall framework of our algorithm is illustrated in Figure 1. We multiple self-attention layers as the encoder Eø to process video tokens and state tokens and then use the action decoder fo to predict action labels at The joint optimization of LMr and LMI+ ensures the video representation contains only information related to the current task. The pseudocode of our algorithm is summarized in Algorithm 1. It is worth noting that the skill decoder fekill, the prior video encoder 8%
prlor and the prior action decoder fg will only be used during training, we will keep only the video encoder go and the action decoder fg for execution.
Algorithm WL-DM
1: Initialize video encoder gş. action decoder fa and skill decoder fakill
2: Initialize prior video encoder prior and prior action decoder fō
Initialize training dataset D
4: for i = 1 to I do
5: Sample data (st, az, v) from D
6: Construct approximation of current video segment Veur
7: Construct approximation
current skill
8:
Update Ke and fo by Equation (1) with (st, az,
Update prior and fg by Equation (1) with (st, ai Veur) &
9:
10:
Update Be and prior by Equation (5) with (s,, U, Veur) g
11: Update ge and pskill by Equation (6) with (st, 2,)
12:
end for
EXPERIMENTS
5.1 EXPERIMENT SETUP
To validate our method, we conduct empirical evaluations two different robotic environments, Franka Kitchen (Gupta et al., 2020) and Meta World (Yu al., 2020). The visualization these two environments is presented in Figure 2.
In Franka Kitchen (FK), we control a Franka Panda robot in the kitchen envi- ronment to perform seven possible tasks: mierowave (M), kettle (K), bottom burner (B), top burner (T), light switch (L), slide cabinet (S) and hinge cabinet (H). The dataset from the original paper (Gupta al., 2020) contains 566 trajectories cor- responding to 24 different task combina- tions. To enable video-conditioned policy training, we train expert policy using the original dataset to collect trajectories and corresponding video demonstrations, To evaluate the one- shot imitation learning ability, split the dataset into training dataset and test dataset, where the training dataset contains 17 different task combinations and the test dataset contains different task combinations, and there is overlap of task combinations between the training set and the test set. During testing, we sample different video demonstrations for each task combination, run the evaluation 10 times, and report the average performance.
(a) Frank Kitchen
(b) Meta World
Figure 2: Visualization of Experiment Environments

---

Published as a conference paper at ICLR 2025
In Meta world (MW), we modify the original environment et 2020) to perform multiple tasks within a single episode. In this newly devised environment, we control a Sawyer robot to perform four possible tasks: close drawer (D), open door (0), push button (B) and open window (W). We use expert policy to collect the dataset for all 24 different tasks. It worth noting that, as the dataset contains all possible task combinations, the task orders presented in the video demonstration bring additional difficulties for policy learning. To evaluate the one-shot imitation learning ability, we split the dataset into training dataset and a test dataset, where the training dataset contains 17 different task combinations and the test dataset contains different task combinations, and there is no overlap of task combinations between the training set and the test set. During testing, we sample 3 different video demonstrations for each task combination, run the evaluation 10 times, and report the average performance.
We include several challenging imitation learning algorithms our baselines: C-bet (Cui et al., 2022), decision transformer (Chen et al., 2021b), and VIMA (Jiang et al., 2023). As C-bet and decision transformer were not proposed for video-conditioned policy learning, additionally take videos as input and get baselines V-BET and V-DT. For VIMA, it was originally proposed for multimodal prompts. However, we do not assume data of other modalities, train VIMA on our video-only dataset and serve as our baseline VIMA. More details of experiments can be found in Appendix A.
modify them to
5.2 EXPERIMENT: MAIN
Table I: The performance of WL-DM and baselines on all FK and MW tasks.
Tasks
Env
Methods
Avg
MBTH
MBLS
MBTL
KBTH
MTLH
KBLS
KBTS
WL-DM
2.30
2.57
2.37
2.10
1.83
1.97
3.17
2.33
V-BET
1.37
2.47
0.83
2.27
1.73
1.80
3.10
1.94
FK
V-DT
1.00
2.33
1.70
1.33
1.47
1.70
2.10
1.66
VIMA
0.77
0.50
0.50
1,27
0.20
1.67
1.70
0.94
ODWB
DOBW
DBWO
WBOD
BDOW
BDWO
BWDO
WL-DM
3.33
2.00
2.00
2.00
2.67
2.00
4.00
2.57
V-BET
1.87
2.00
0.73
1,33
0.33
0.00
1.97
1.182085
MW
V-DT
1.33
2.13
1.23
1.93
0.37
0.83
0.83
1.86
VIMA
1.80
1.00
0.37
0.37
1.17
0.57
0.83
0.87
As shown in Table 1, our method achieves better average performance in both environments. In Franka Kitchen, our method achieves improvement of approximately 20.1% compared to the best baseline (V-BET). In Meta World, the improvement even more sigmificant, with our method achieving 100.1% improvement compared to the best baseline (V-DT). Although sistently outperforms all baselines, we note that there large gap terms of the degree of im- provement between the environments. This because, in Franka Kitchen, task combinations in the dataset are not diverse enough, and there is no variation in terms of the task orders (A, B vs B, be considered to have strong correlation within the task combinations. There-
method con-
A). Therefore, fore, even without considering the segmentation of tasks in video demonstrations, we can still utilize this correlation to achieve policy that performs well during testing. However, in Meta World, our dataset includes all task combinations and considers different task orders, making task segmenta- tion in videos even more critical, which explains the gap in improvement of our algorithm in the two environments. More specifically, out of a total of 14 test tasks, method achieves the best performance in 12 of them. Such overall performance validates the effectiveness of our algorithm.
For the performance of baselines, we found that V-BET and V-DT perform at similar level. This because the main difference between V-BET and V-DT in our implementation is whether or not action is used as part of the trajectory. For the robotic environments can often be directly inferred from changes in the state the robot. Therefore, the advantage of using action information is significant. VIMA, the other hand, does not perform well in
used, this action information
8

---

Published as a conference paper at ICLR 2025
both environments. One potential reason is that VIMA was originally proposed for multimodal scenarios. Although its training process can be transferred to the pure video scenario, this direct transfer is clearly not effective. Moreover, in terms of one-shot video imitation. VIMA mainly considers object-level variations in a single task rather than variations in task combinations, so we believe its performance decline is acceptable.
5.3 EXPERIMENT: ABLATION
Table 2: The performance of WL-DM and ablation baselines all MW tasks.
Tasks
Env
Methods
Avg
ODWB
DOBW
DBWO
WBOD
BDOW
BDWO
BWDO
WL-DM
3.33
2.00
2.00
2.00
2.67
2.00
4.00
2.57
WL-DM w/o S
1.00
2.00
1.83
1.67
1.67
1.67
2.00
1.69
MW
WL-DM w/o LMI-
2.00
2.00
2.00
2.00
1.67
2.00
3.33
24
WL-DMM Ø
2.00
2.13
1.93
2.00
2.00
1.33
2.00
1.31
As our objective function contains two different mutual information terms, we conduct ablation stud- ies in this section to verify the contribution of these two components to our method. We construct two ablation baselines, WL-DMM w/o C and WL-DM w/o LMI- The ablation baselines are iden- tical to our algorithm in all aspects, except that WL-DM w/o LMI+ does not use Equation (6) and WL-DM w/o LNT- does not use Equation (5). We validate these two ablation baselines in the Meta World environment and compare them with our method. As shown in Table 2, both ablation base- lines achieve worse performance compared to WL-DM, thereby verifying that both Equation (5) and Equation (6) contribute to algorithm. Notably. even with only Equation (5), the ablation baseline still achieves better performance than the baselines in Section 5.2, which further validates the effectiveness of Theorem in practice.
As mentioned in Section 4.3, directly minimizing the mutual information using prior g(hpls) can lead to excessive loss of video information in practice, thereby affecting the performance of the algorithm. To verify this point, we construct another ablation baseline WL-DM w Ø. This baseline is again identical to our method in all aspects, except that it does not use 8 gprio (h [s, Vour) but instead uses prior g(,,,s). From Table 2, we can see that using (h,,,1)) indeed leads to a decline in the performance, thus verifying the prior vidco encoder constructed in Section 4.3. Additionally, it is worth noting that WL-DM w/o ExI- demonstrates that when we do not minimize mutual information all, that is, do not control the information provided by the video, the algorithm cannot achieve its best performance. In contrast, WL-DM w Ø indicates that excessively reducing the information of the video also leads decline in the final performance. This phenomenon further demonstrates the importance of a proper approximation for Veur and validates our statements in Section 4.3.
CONCLUSION
In this paper, we investigate the problem of one-shot video imitation for video-conditioned policies. To enhance the compositional generalization ability of the learned policy, we propose an imita- tion learning framework, Watch-Less-Do-Moon (WL-DM). Our method introduces an information bottleneck- based objective, which leads to implicit skill discovery for video-conditioned policies. The intuition behind this method is that by segmenting the video into different tasks, the policy learns diverse skills corresponding to these tasks. When faced with unseen videos, the policy can also decompose them into combinations of previously encountered tasks, thereby completing these tasks through the learned skills. To better explain our method, we build theoretical connection between our method and this intuition using information theory. We also present practical imple- mentation of our algorithm and evaluate it a variety of tasks across multiple environments. The experimental results indicate that our algorithm outperforms baselines in terms of the compositional generalization ability, which verifies the effectiveness of our algorithm.

---

Published as a conference paper at ICLR 2025
7 LIMITATIONS AND FUTURE WORK
The limitation of this work is that our approximations the current task keur and the current skill remain naive, which just the information from the immediate future. Although similar approx- imations have been used in many previous studies (Pertsch et al., 2021; Liu et al., 2021; Xie et al., 2023; Yuan al., 2024), one issue with this approach is the need for video data be aligned with trajectory data in timesteps (for a state St, we can access its corresponding video frame fi), which can be costly collect in many cases.
Our future work will primarily focus on extending our algorithm to real-world scenarios, such as real-world robots, thereby broadening the application scope of our algorithm. Additionally, we will consider extending our algorithm to multimodal scenarios, utilizing multimodal information to obtain better approximations of tasks and skills, thereby not only enhancing the performance of the algorithm but also expanding the range of instruction formats it can process.
ACKNOWLEDGMENTT
This work was supported by NSFC under Grant 62450001 and 62476008. The authors would like thank the anonymous reviewers for their valuable comments and advice.
REFERENCES
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (ypt) learning to act by watching unlabeled online vidcos. In Proceedings of the 36th International Conference Neural Infor- mation Processing Systems, pp. 24639-2466 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 1877-1901, 2020.
Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. GROOT: Learning to follow instructions by watching gameplay videos. In The Twelfth International Conference on Learning Representations, 2024.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Y Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference Machine Learning, pp. (61763 PMLR, 2023.
Wilka Carvalho, Angelos Filos, Richard L Lewis, Satinder Singh, et al. Composing task knowledge with modular successor feature approximators. arXiv preprint 23X3v:2301.12305, 2023.
Elliot Chane-Sanc Cordelia Schmid, and Ivan Laptev. Learning video-conditioned policies for unseen manipulation tasks. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 909-916. IEEE, 2023.
Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103.16817. 202
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: reinforcement learning via sequence modeling. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 15084-15097, 2021b.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210. 10047, 2022.
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. In Conference on Robot Learning, pp.
PMLR, 2021.
10

---

Published as a conference paper at ICLR 2025
Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 1087-1098, 2017.
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Young- woon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforce- ment learning. In Proceedings of the 37th International Conference on Neural Information Pro- cessing Systems, pp. 687603 2023.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Robust predictable control. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp- 27813-27825, 2021.
Chrisantus Eze and Christopher Crick. Learning by watching: A review of video-based learning approaches for robot manipulation. arXiv preprint arXlv:2402.07117, 2024a.
Chrisantus Eze and Christopher Crick. Learning by watching: review of video-based learning approaches for robot manipulation. arXiv preprint arXiv:2402.07127, 2024b.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning. In Conference on robot learning. pp. 357-368. PMLR, 2017.
Anirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottle- neck. In International Conference on Learning Representations, 2018.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning. pp. 1025-1037. PMLR, 2020.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne, Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2) 2017. 2017
Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess. Quan Vuong, Pannag Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, et al. Vid2robot: End-to-end video- conditioned policy learning with cross-attention transformers. arXiv preprint arXiv:2403. 12943, 2024.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Confer- ence on Robot Learning, pp. 991-1002. PMLR, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: robot manipulation with multimodal prompts. In Proceedings of the 40th International Conference on Machine Learning, pp. 14975- 15022, 2023.
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03181, 2024.
Baihan Lin, Djallel Bouneffouf, and Irina Rish. A survey on compositional generalization in appli- cations. arXiv preprint arXiv:2302.01067. 2023.
Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. Coach-player multi-agent reinforcement learning for dynamic team composition. International Conference on Machine Learning, pp. 6860-6870, PMLR, 2021.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.088: 08299, 2022.
Lajanugen Logeswaran, Wilka Carvalho, and Honglak Lee. Learning compositional tasks from lan- guage instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 13300- 13308, 2023.
11

---

Published as a conference paper at ICLR 2025
Zhao Mandi, Fangchen Liu, Kimin Lee, and Pieter Abbeel. Towards more generalizable one-shot visual imitation learning. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2434-2444. IEEE, 2022.
Robert McCarthy, Daniel CH Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G Thuruthel, and Zhibin Li. Towards generalist robot learning from internet video: A survey. arXiv preprint M.Xxx:2404199660 2024.
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference Robot Learning, pp. 1303-1315. PMLR, 2022.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: univer- sal visual representation for robot manipulation. In Conference on Robot Learning, pp. 892-909. PMLR, 2023.
Soroush Nasiriany, Vitchyr H Pong. Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 14843- 14854, 2019.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning, pp- 2661-2670. PMLR, 2017.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 27730-27744, 2022.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188 204. PMLR, 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automation (ICRA), pp. 113--114. IEEE, 2018.
Sangwoo Shin, Dachee Lee, Minjong Yoo, Woo Kyung Kim, and Honguk Woo. One-shot imitation in a non-stationary environment via multi-modaa skill. In Proceedings of the 40th International Conference on Machine Learning, pp. 111622315772 2023.
Sangwoo Shin, Minjong Yoo, Jeongwoo Lee, and Honguk Woo. Semtra: A semantic skill translator for cross-domain zero-shot policy adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 15000-15008, 2024.
Sam Spilsbury and Alexander Ilin. Compositional generalization in grounded language learning via induced model sparsity. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pp. 143 143-155, 2022.
Elias Stengel-Eskin, Andrew Hundt, Zhuohong He, Aditya Murali, Nakul Gopalan, Matthew Gom- bolay, and Gregory Hager. Guiding multi-step rearrangement tasks with natural language instruc- tions. In Conference Robot Learning, pp. 1486-1501. PMLR, 2022.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
12

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXhv:230..3397, 2023.
Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play, In Confer- ence on Robot Learning, pp. 201-221. PMLR, 2023.
Philipp Wu, Kourosh Hakhamaneshi, Yuqing Du, Igor Mordatch, Aravind Rajeswaran, and Pieter Abbeel. Semi-supervised one-shot imitation learning. arXiv preprint arXiv:2408.05283 2024.
Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Yang Wei, and Shuai Li. Future-conditioned unsu- pervised pretraining for decision transformer. International Conference on Machine Learning, pp. 3011888182000 PMLR, 2023.
Mengda Xu. Zhenjia Xu, Cheng Chi. Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, pp. 3536-3555, PMLR, 2023.
Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Oleksandr Maksymets, and Dhruv Batra. Ovrl-v2: simple state-of-art baseline for imagenav and objectnav. arXiv preprint arXiv:2303.077900 2023a.
Kanmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Piere Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for em- bodied navigation. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023b.
Xue Ying. An overview of overfitting and its solutions. In Journal of physics: Conference series, volume 1 1168, pp. 022022. IOP Publishing, 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pp. 1094-1100 PMLR, 2020.
Haoqi Yuan, Zhancun Mu, Feiyang Xie, and Zongqing Lu. Pre-training goal-based models for sample-efficient reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
Bohan Zhou, Jiangxing Wang, and Zongqing Lu. Nolo: Navigate only look once. arXiv preprint arXiv:2408.01384 2024.
13

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
We implement our algorithm and all baselines based on the codebase of C-bet (Cui et al., 2022). For WL-DM and V-BET, we consider only observations in the trajectory. and for V-DT and VIMA, we consider both observations and actions in the trajectory. which aligns with the implementation stated in the paper of C-bet (Cui et al., 2022), DT Chen et al. (2021b) and VIMA (Jiang et al., 2023). For WL-DM, V-BET, and V-DT, we use the same transformer model as stated in C-bet, which contains multiple self-attention layers to process video information and trajectory information the same time. For VIMA, we use alternating cross-attention and self-attention layers described in its paper (Jiang et al., 2023).
For all experiments, to be 20 (for V-DT and VIMA, means 20 observation-action pairs). For WL-DM, the window size of future video segments is sampled from [20, 40). As use the same action decoder, where we set the number bins for action discretization to 32, and the id of each cluster will also be used for the representation of skills for WL-DM. For the Franka Kitchen environment (Gupta et al., 2020), we use decoders with 3 layers, and 3 heads and set the hidden dimension be 60 (for VIMA, it means in total self-attention layers and cross-attention layers). We train all methods for 10 epochs. For WL-DM, a is fixed to be 1 x 10-2 and is fixed be 10-1 during the training process. For the Meta World environment (Yu al., 2020), use decoders with 6 layers, and 6 heads and set the hidden dimension to be 120 (for VIMA, it means in total 6 self-attention layers and cross-attentin layers). We train all methods for 30 epochs. For WL-DM, is set to be 0 in the beginning and fixed to be 1 x 10-3 after 10 epochs, and is fixed to be during the training process.
set the learning rate to be 3 x 10 and set the window size for the trajectory
the codebase of C-bet, all methods
B PROOF OF THEOREM
If we have MI(he: (s, Veuc) = O, then DKL (I(als, ()ll=(als, veur)) = 0 for all siate-video pairs (s,0) v) E S x with non-zero probability P(s, u) > 0.
Theorem 1.
Proof. By expanding the mutual information MI(Votheri hys als, Veur), we can have the following equality:
MI(Vother; by. a/8, Veur)
P(yother: Bay a a|s,, Veur)
p( Ep(yother a venr))
log P (8, Veur P a 88, Veur)
-Ep(eveur)) E Ep hara )eve))
log P(Itus als, Veur, Vother) log P(ha, als, Veur)
log P(tvlss (s, Veurs Vother) + P(a bys 8, Veur Vother)
-E -Ep(eveer) Pi hara s.we))
- log P(h. Is, Veur) log P(a hos 8, Veur)
- P( Vother))
DKL((Phulss Veur, Vother) P(he ((ho)ssVecu)))
Veur))]
DKTT. (Rrt.P(ee (P(alhu, 8, Veur, Vother ) Vother)]l
8,
Mlihhi Vother ,s, Veur) + MI(a; Vother (s, Veur, ht).
Similarly, we can also have:
MI(Vother i hy, als, Veur) = MI;; Vother [s, Veur) + ( Vother |s, Veur: a.-
Combining these two equality, we can have:
MI(h:: Vother |s, Veur) + MI(a; Vother (s, Veur (br) = MI Vother |s, Veur) + MI(h,; Vother [s, Veury a).
As a and Vother become independent with each other when h2 is given, we have
MI(a; Yother (s, Veur, hw) 0. As we also have MT(h;; Vother |s, Veur, a) 0. can have the
14

---

Published as a conference paper at ICLR 2025
following inequality, which basically gives us the conditional version of data processing inequal- ity (Cover, 1999):
(h; Vother |s, Veur) 2 Ma(;; Vother 1s, Veur).
Since Mī Vother 18, Veur) 2 0, if we also have MT(ho; Vother 18, Veur) then conclude that:
MI(a; Vother (s, Veur) = O.
By expanding this mutual information term, we have:
MI(a; Vother (s, Veur)
P(s.ver Vother)
DKL DIL (H Veur, (ather)lla(aaas Veur))
-p((.
(RI(( u)ll (a|s, Vour))
=0.
Since the KL divergence is non-negative, for the above expectation to be zero, there must be for all siate-video pairs (s, v) with non-zero probability P(s, v) >> 0, we have the KL divergence to be zero, DKL (T v) (als,, Veur)) = 0, and conclude our proof.
C VISUALIZATION
126
140
127
10
129
130
120
13 B3
194
148
149446148
100
147449
139
150
Timestep
140
80
81
-5
60
103
102
10 100
108
107
109
40
114
11B
9
20
-15
120
1 12 125
Figure 3: Visualization h. over timesteps.
In this section, we present the visualization result of our method. We visualize how hú of WL-DL changes over timesteps. As shown in Figure 3, we can observe that h, of WL-DM tends to converge at adjacent timesteps. It is worth noting that since we use a GPT-like transformer architecture as the mixed together in hy. Furthermore,
encoder, the information of video tokens and obs tokens do not introduce any task-level information (such as task-level video segmentation annotations), so the clustering results of h, do not fully correspond to the task.
15

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTS
D.1 ABLATION: TYPES OF TASK COMBINATIONS
We further include an experiment about the effect of the number of task combinations in the training
set in the Meta World environment. In Section 5.2, we included 17 task combinations (7/3 split) in the training set, and here we further consider cases where we have 15 task combinations (6/4 split) and task combinations the training set. As shown Table 3, experimental results, WL-DMM still outperforms other baselines, further demonstrating the effectiveness of our method.
WL-DM
V-BET
V-DT
VIMA
6/4
1.93
1.34
0.86#084
0.43
7/3 (main exp)
2.57
1.18
1.24
0,87
8/2
1.88
0.63
0.94
0.81
Table The performance of methods with different number of task combinations the training set on MW tasks.
D.2 ABLATION: NUMBER OF VIDEOS FOR EACH TASK COMBINATION
We also include an experiment about the number of videos corresponding to each task combination the Meta World environment. Section 5.2 we considered 20 different videos for each task combination, and here we further consider cases with 40 different videos for each task combination. As shown Table 4, experimental results, WL-DM still outperforms other baselines, which again demonstrates the effectiivnness of WL-DM.
WL-DM
V-BET
V-DT
VIMA
20 (main exp)
2.57
1.18
1.24
0.87
40
2.211081
1.71
1.33
1.09
Table The performance of all methods with different number of videos for each task combination on MW tasks.
D.3
MORE BASELINE: VIP
Tasks
Env
Methods
Avg
ODWB
DOBW
DBWO
WBOD
BDOW
BDWO
BWDO
WL-DM
3.33
2.00
2.00
2.00
2.67
2.00
4.00
2.57:1000
V-BET
1.87
2.00
0.73
1.33
0.33
0.00
1.97
1.18
MW
V-DT
1.33
2.13
1.23
1.93
0.37
0.83
0.83
1.244
VIMA
1.80
1.00
0.37
0.37
1.17
0.57
0.83
0.87.1084
ViP
2.80
2.20
2.00
1.87
1.63
1.10
1.80
1.91
Table 5: The performance of all methods including ViP on all MW tasks.
We have added new video-conditioned baseline: ViP (Chane-Sana et al., 2023). Although the purpose of ViP is to learn iideo-conditionel policy, has different setting from method. Thus, we have made the following modifications to adapt it to our setting:
Since we do not consider human videos as input, we have removed the part that uses human videos for pre-training
16

---

Published as a conference paper at ICLR 2025
Since do not assume access to video labels, we have changed its supervised contrastive learning part to unsupervised contrastive learning on robot videos.
We used the same codebase as WL.-DM to implement ViP with minimal modifications, and con- ducted experiments in the MetaWorld environment. The experimental results shown in Table 5. ViP outperformed other baselines in this environment, demonstrating its effectiveness as video- conditioned policy. However, its performance still lags behind WL-DM, which further demonstrates the effectiveness of WL-DM.
D.4 MORE DATASET
To further evaluate our method, we construct script in a similar way of Lee et al. (2024) to convert state-hase observations of the original dataset of the Franka Kitchen environment into image-based observation, and train all methods on this dataset. For WL-DM, we use a linear schedule for dn, where ctef_start is be 0 and coef_end is set be 1 10 and for a2, fix it to be x 10-1 during the training process, As shown in Table 6, WL-DM still outperforms other methods, which further demonstrates the effectiveness of our method.
Tasks
Env
Methods
Avg
BTLS
BTSH
MBTS
MBTH
MLSH
MBTL
MKBH
WL-DM
2.43
1,63
2.63
2.57
2.27
2.27
2.5
2.33
V-BET
1.23
1.43
2.33
1.53
2.10
1.70
2.17
1.79
FK(new)
V-DT
1.63
2.20
1.40
1.80
1.47
1.73
2.20
1.78
VIMA
0.87
1.27
2.20
1.80
1.80
1.43
2.23
1.66:0.80
Table 6: The performance of all methods on new FK dataset.
D.5 MORE BASE ALGORITHM
As WL-DM can be seen as a method using information bottleneck-based loss on top of V-BET. To further validate our approach, applied the information bottleneck-based loss of WL-DM both V-DT and VIMA and conducted experiments in the Meta World environment. As shown in Table WL-DM+V-DT and WL-DM+VIMA both outperform its base algorithm, which further validates the effectiveness of the proposed information bottleneck-bast loss.
WL-DM
V-BET |
WL-DM+V-DT V-DT WL-DM+VIMA
VIMA
MW 2.57
1.18
1.722066
1.24
1.84
0.87+084
Table The performance of all methods on MW tasks, we applied the information bottleneck-based loss different base algorithms and compare their performance.
D.6 COEFFICIENT SELECTION
022
0.1
10
011
0.1
0.95 #0.77
2.09 0.69
2.16+0.71
0.01
0.82
1.83
2.05
+0.62
0.001
1.03
1.79-0.4
2.57++0.0
Table The performance of WL-DM with different coefficients on MW tasks.
17

---

Published as a conference paper at ICLR 2025
Coefficients for mutual information loss should be adjusted according to the environment. Specifi- cally, we conducted grid search to select optimal coefficients. Taking experiments in Meta World environment example, the performance of different coefficients is shown in Table 8.
18


Reviews:
[{"review_id": "tW5iM78pNg", "reviewer": "Reviewer_zDKg", "summary": "This paper presents Watch-Less-Do-More (WL-DM), an imitation learning framework for video-conditioned policy learning, to enable an agent to learn multiple skills from videos and generalize to unseen task combinations. The method uses an information bottleneck to implicitly discover skills and decompose video demonstrations into tasks without requiring explicit video segmentation annotations. WL-DM is evaluated in environments like Frank Kitchen and MetaWorld, demonstrating its capacity to achieve compositional generalization in unseen task combinations, outperforming baseline methods.", "questions": "1. Is there any ablation about the number of videos in the training set? This includes how many types of task combinations there are, and how many videos with different initializations are there for each combination.\n2. How similar do two video clips from different training videos need to be to be considered as the same task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The article is clearly written and proposes an effective solution to the problem of skill discovery without relying on language information or manual annotation.", "weaknesses": "1. There lack of comparative experiments on learning directly from the segmented sub-task videos, so it is hard to see whether the method achieves its intuition of *Focusing on the current task*.\n2. In potential real-world applications, the various steps in multi-step tasks are often causally linked, and the appearance of the same task in different videos may be different. Additionally, the accessible training videos are not guaranteed to cover all elements in downstream tasks, without including elements out of a certain task set. As a result, the generalizability of the method appears to be somewhat limited for now.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2yzeF3M6N3", "reviewer": "Reviewer_DKw6", "summary": "This paper proposes a new method to train video-conditioned policies, which employ an information bottleneck-based objective to learn a video encoder for implicit skill discovery. The proposed method is novel, and the experiments on both Frank Kitchen and MetaWorld demonstrate its performance outperforms baselines.", "questions": "- What is the generalization ability of your proposed method? Can it generalize to unseen tasks or unseen visual backgrounds? This is an important factor which should be investigated.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Learning an executable policy from videos is a good topic, since video is a general interface across different domains and there are widely available video data. This paper further proposes an information bottleneck-based imitation learning framework for implicit skill discovery and video-conditioned policy learning, which is novel and sound.\n- Good writing and clear motivation. This paper derives its final objective from both skill discovery and information bottleneck perspectives.\n- Experiments on both  Frank Kitchen and MetaWorld demonstrate its superior performance.", "weaknesses": "- The baselines compared in this paper are not originally video-conditioned. DT is proposed to be conditioned on states, and VIMA is conditioned on texts and images. I encourage the authors to include a video-based baseline to strengthen this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rvQVLdmUy5", "reviewer": "Reviewer_88xK", "summary": "The paper proposes an information bottleneck-based imitation learning framework, WL-DM (Watch-Less-Do-More), for implicit skill discovery and video-conditioned policy learning. This strategy can generalize to unseen video task combinations, demonstrating strong compositional generalization ability. The idea of implicit skill decomposition is innovative, especially in achieving task segmentation within videos without requiring explicit video segmentation annotations.", "questions": "1. what is the practical significance of compositional generalization?\n\n2. Can the author analyze the performance of the above paper methods in the experiment?\n\n3. Can the author provide more experimental results to verify the effectiveness of the model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes that the WL-DM framework innovatively employs an information bottleneck approach for implicit skill discovery, allowing effective task segmentation without explicit annotations. The method demonstrates strong compositional generalization, successfully adapting to unseen video task combinations.", "weaknesses": "1. The compositional generalization seems limited to combining different tasks, while in practice, different task combination sequences will not have different effects on similar outcomes. Could the authors clarify the practical significance of this generalization with examples of real-world scenarios where it proves valuable or challenging?\n\n2. The paper claims that video-conditioned policies can achieve combinatorial generalization when tasks can be performed independently. Could you explicitly compare WL-DM to single-task learning methods and clarify which mechanisms in WL-DM enable combinatorial generalization beyond what single-task approaches offer?\n\n3. The paper lacks an experimental comparison with skill-based imitation learning methods (e.g., Xu et al., 2023; Wang et al., 2023; Shin et al., 2023; 2024) and single-task video demonstration methods (e.g., Chane-Sane et al., 2023). Could the authors analyze specific performance metrics, such as generalization ability or data efficiency, to provide a more detailed comparison with these methods?\n\n4. The experimental results are relatively limited, such as lacking empirical support for the advantages of implicit segmentation, as well as necessary visualizations or other forms of demonstrations to validate the effects of implicit segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bp3hKE1ct0", "reviewer": "Reviewer_wcsD", "summary": "The paper introduces Watch-Less-Do-More (WL-DM), an imitation learning framework designed for video-conditioned policy learning that aims to enhance compositional generalization. WL-DM uses an information bottleneck method to identify relevant skills from videos, allowing the policy to handle complex, unseen video task combinations by focusing only on current tasks. Experimental results across two robotic environments (Franka Kitchen and Meta World) show WL-DM surpassing baseline models in generalization ability. The authors also highlight the potential for broader applications, though they note limitations regarding video data alignment with task segmentation.", "questions": "- For mutual information loss, the importance might vary depending on the diversity and length of skills in the environment. Should it be adjusted according to each environment? If so, how should it be approached?\n- Could you show how the skills were separated in the experiments conducted? Do similar z-values actually appear consecutively, allowing the skills to be segmented?\n- Additionally, providing a visual representation of whether the skill space is discretely separated into a minimal set of skills when using this type of loss would aid the reader’s understanding.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The framework is sound, well-structured, and novel.\n- The overall content is easy to understand and well-written.", "weaknesses": "- It would be beneficial to clearly explain the difference between the proposed method and various existing approaches that use mutual information for skill extraction [1, 2, 3].\n- There are questions regarding the appropriateness of the chosen baselines. The authors used VIMA as the SOTA baseline, but VIMA showed very low performance. They suggest that this may be due to VIMA's reliance on multi-modal data, which could degrade performance when using only pure video data. However, a comparison with other one-shot imitation methods that use only video data would provide a more relevant evaluation.\n\n[1] Jiang, Yiding, et al. \"Learning options via compression.\" *Advances in Neural Information Processing Systems* 35 (2022): 21184-21199.\n\n[2] Ju, Zhaoxun, et al. \"Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning.\" *Proceedings of the International Conference on Automated Planning and Scheduling*. Vol. 34. 2024.\n\n[3] Yu, Xuehui, et al. \"Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learning.\" *arXiv preprint arXiv:2406.04815* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)