Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents
Authors: Zhizhen Zhang
Lei Zhu
Zhen Fang
Zi Huang
Yadan Luo
Venue: NeurIPS 2025 poster ()
Keywords: multimodal pretraining
embodied AI
robot manipulation
Primary area: reinforcement_learning
TL;DR: A novel vision-language pretraining method that explores ordering and continuity of videos for robot manipulation

Abstract: Pre-training vision-language representations on human action videos has emerged
as a promising approach to reduce reliance on large-scale expert demonstrations
for training embodied agents. However, prior methods often employ time con-
trastive learning based on goal-reaching heuristics, progressively aligning language
instructions from the initial to the final frame. This overemphasis on future frames
can result in erroneous vision-language associations, as actions may terminate
early or include irrelevant moments in the end. To address this issue, we propose
Action Temporal Coherence Learning (AcTOL) to learn ordered and continuous
vision-language representations without rigid goal-based constraint. AcTOL treats
a video as a continuous trajectory where it (1) contrasts semantic differences be-
tween frames to reflect their natural ordering, and (2) imposes a local Brownian
bridge constraint to ensure smooth transitions across intermediate frames. Exten-
sive imitation learning experiments on both simulated and real robots show that the
pretrained features significantly enhance downstream manipulation tasks with high
robustness to different linguistic styles of instructions, offering a viable pathway
toward generalized embodied agents. Our project page is at https://actol-pretrain.github.io/.

Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents
Zhizhen Zhang
Lei Zhu Tongji University 1eizhu06080gmai1. com
Zhen Fang
The University of Queensland uqzzha390uq edu. au
University of Technology Sydney zhen. fang@uts edu.au au
Zi Huang
Yadan Luo
The University of Queensland helen. huangSuq edu. au
The University of Queensland y luouuqu edu. au
Abstract
Pre-training vision-language representations on human action videos has emerged promising approach reduce reliance on large-scale expert demonstrations for training embodied agents. However, prior methods often employ time contrastive learning based goal-reaching heuristics, progressively aligning language instruc- tions from the initial to the final frame. This overemphasis on future frames can result in erroneous rision-language include irrelevant moments in the end. To address this issue, we propose Action Temporal Coherence Learning (AcTOL) to learn ordered and continoous vision- language representations without rigid goal-based constraint. AcTOL treats video continuous trajectory where it (1) contrasts semantic differences between frames to reflect their natural ordering, and (2) imposes a local Brownian bridge constraint ensure smooth transitions across intermediate frames. Extensive imita- tion learning experiments on both simulated and real robots show that the pretrained features significantly enhance downstream manipulation tasks with high robust- ness to different linguistic styles of instructions, offering a viable pathway toward generalized embodied agents. Our project page is at httpss//actool http://accoo--peeraiin pretrain.github.on github.io/
associations, as actions may terminate early
Robot learns action by watching Internet videos
How well does robot understand action?
ACTAL
UV
DLF
Instruction: "open fridge"
Vision-L anguaae Pre training
Action start
Action end
Inrelative frames
Action
Figure 1: Pretraining on Internet human action videos for robot control, where the video-instruction
pairs are noisy and often include irrelevant frames. The red vision-language reward curve demon- strates AcTOL learns correctly align instruction with action, outperforming previous goal-reaching methods in the presence of distracting content.
Introduction
The long-term vision for embodied intelligence 29 25
is create systems that seamlessly perceive and interact with the world around them. Achieving this requires agents that integrate vision and language understand their surroundings, interpret human instructions, and autonomously plan
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Goal-reaching Pretraining
AcTOL Pretraining
AcTOL Applications
Global Ordering
Language-Conndtiined Imitation Learning
Language Instruction
dist... dist.y dist..; dist.,
Vision-I L Language Representations
MUP
Video Frame
Alignment Strength
Singuage-Conditionn Visual Reward
Smmianty
Local Continuity
Robustness under Language Perturbation
Align with later frame Sensitive to video noise!
Rew Instruction: open cupboard you pull open cupboard?
variance
Brownion Bridge
Ineed to grab something Inside.
Constraint
Figure 2: Comparison of existing goal-reaching pre-tra ve--trannng strategies and the proposed AcTOL approach. Our learned multi-modal representations can be effectively transferred to downstream Linguage-condttions robot manipulation tasks, exhibiting robustness to diverse instruction and linguistic variations.
actions for complex tasks. Current end-to-end approaches achieve policy learning through direct assion-lagguae--ctinn mapping [48] [3 21, However, the inherent unpredictability of physical environments, including unseen scenarios and dynamic object interactions, constrains these solutions by requiring massive, high-quality robotic trajectories with action annotations, which are costly to collect. To mitigate this, recent research has leveraged large-scale, readily available egocentric human action videos [14] IO, [5] for pre- pre-trainng.. Although these out-of-domand videos often lack low- level action details and contain noise, their diverse human-object interactions and task instructions provide valuable prior knowledge. This enables the pre-trained representations to be more effectively transferred to novel tasks with fewer demonstrations, reducing reliance on large-scale robotic datasets while preserving strong generalization capabilities.
promising approach for dision-languag pre-training from human action videos leverages the concept of time contrastive learning [37] to capture temporally consistent visual representations, where language serves as the guiding goal, with semantic alignment between the language and chronologically later frames in the video [30 26 23 However, this goal-reaching semantic alignment approach relies on a rigid assumption that action videos adhere specific principle: actions progressively approach the target instruction from the initial frame to the final one. Such assumption can be easily violated egocentric human action videos, which are typically annotated at a coarse- grained level and riddled with noise. Figure | shows an example video-instruction pair, where the end of the video clip does not correspond to the actual end of the action. As result, existing methods suffer from mislcading semantic alignment, which hampers their ability to learn accurate vision-lange Asiounlanguage relationships.
Given the challenges outlined above, a more natural and flexible pre-training strategy without rigid assumptions is needed enhance vision- laiiuaee representations for better policy learning. Building solely on the intrinsic temporal consistency of human action videos, we argue that the ordering and continuity of pre-trained languanlanguage representations play crucial role in ensuring the effectiveness of policy learning. Ordering refers to the need for visual features to align with the underlying action logic required by the language instruction. For instance, as the task progresses, visual representations closer to the completion of the action should exhibit stronger alignment with the language instruction. This ensures that each step in the sequence is meaningfully associated with the corresponding instruction, enabling the model effectively capture the dynamic progression of the task. Continuity, on the other hand, emphasizes that both visual features and their alignment with the language should evolve smoothly over time, with gradual transitions rather than abrupt changes. This is crucial because actions in the real world are not discrete but unfold continuously in time. Moreover, the alignment between visual and instruction should also be fluid, ensuring that as the action progresses, the visual representations consistently align with the target language instruction.
To address the aforementioned issues, illustrated in Figure 2 we propose Action Temporal Coherence Learning (AcTOL.) a novel approach designed to implicitly capture the ordering and continuity of video actions without relying on rigid assumptions, while providing strong theoretical

---

guarantees. Unlike previous approaches that focus on goal-directed semantic alignment, AcTOL introduces a Vision-Language Ordering (VLO) loss. This loss leverages the intrinsic temporal coherence of videos, contrasting frames against each other based on their relative temporal distance, theoretically ensuring that the semantic alignment between frames reflects their temporal ordering and continuity throughout the entire sequence. However, the VLO loss does not explicitly enforce the continuity of the visual features themselves, and under conditions with variations in frame content and noise, it can lead to suboptimal local consistency of the visual features. To address this, AcTOL introduces a Brownian bridge constraint over the video, treating video frames as Brownian bridge process. This approach imposes structured, continuous flow the visual representations, ensuring that the model learns more consistent and stable intermediate states, further enhancing the continuity of the visual representations and improving the stability of their alignment with language instruction. Further theoretical analysis suggests that these properties also contribute to the model's resilience language perturbations, a crucial trait for real-world applications. To evaluate the generalization ability of AcTOL on embodied agents, we conducted extensive language-conditiong imitation learning experiments using both the weal-world Unitree D1 robotic arm and two simulation environments. The results demonstrate that AcTOL significantly outperforms prior methods with limited number of expert demonstrations. Additionally, AcTOL can generate language-conditon visual rewards from real-world robot videos and remains robust to complex linguistic perturbations, highlighting its potential as a generalizable solution for real-world embodied agents.
2 Preliminaries
We first set up notations and mathematically formulate tasks.
Language-Conditoons Imitation Learning (I.C-IL). The task of LC-II. aims to train an agent to
mimic expert behaviors from a given robot demonstration set Doobut {T; 4;) 1)); where I; E L represents a task-specific language instruction. Each trajectory тÜ E T consists of a sequence of state-action pairs т; {(S,, ax) } of the horizon length T. In robot manipulation tasks, action
a, E A corresponds to the control commands executed by the agent and state s, = [pr; o. E S records proprioceptive data P. (e.g.. joint positions, velocities) and visual inputs o, (e.g., camera policy
images) at the time step t. The objective of LC-IL is to find an optimal language-conditiong r" (als, S x L A via solving the supervised optimization as follows,
x² E arg min B(r.d)
(T, Σ (Nr,a)))NT
T
where ((:, .) is task-specific loss, such mean squared error cross-entropy. Training the policy Ty in an end-to-end fashion may require hundreds of high-quality expert demonstrations to converge, primarily due to the high variance of visual inputs and language instructions l.
Vision-language Pre-training. Address such scalability issues can be achieved by leveraging large-scale, easily accessible human action video datasets [10 [5] D = [((O,,4) 4) where
O = {or (o=) represents a video clip with T frames and I, the corresponding description. Pretraining on such datasets enables policies to rapidly learn visual-language correspondences with minimal expert demonstrations. Mainstream pretraining methods employ time contrastive learning (37) to fine-tune a visual encoder & and text encoder yp, which project frames and descriptions into a shared d-dimensional embedding space, i.e., Φ(o:) E RRd and 1 (l(4) E Rd. To provide a unified perspective on various pretraining approaches, we formulate them within the objective LINCE(o, ()::
exp(Pi(v+.I)
LICE -REt.P((.)) log E. ~N(0))) exp(Pivv-.))) 1,))
where v 0(0+/-). v=(-==))
Different pretraining strategies differ in their selection of (1) the positive frame set (O))). (2) negative frame set A (O,;;; and (3) the semantic alignment scoring function Ri, 1.) measuring the gap of VI. similarities.
As motivated by goal-conditioned RL. ①. current approaches explicitly select future frames (e.g., R3M [30), DecisionNCE (23) the last frame (e.g., LIV (26) the goal within the positive frame
3

---

set, enforcing their visual embedding to align with the semantics. Likewise, the scoring functions Di are often designed to maximize this transition direction. However, the pretraining action videos are noisy as actions may terminate early or include irrelevant subsequent actions, which may mislead the encoders and result in inaccurate vision-language association. As detecting precise action boundaries is non-trivial, we argue for a more flexible approach that leverages intrinsic characteristies of actions to guide pertaining.
3 Our Approach: AcTOL
We introduce an action temporal coherence learning (AcTOL) to capture two temporal properties of video actions: ordering and continuity. Ordering was ensured in the vision-language ordering loss (Section 3.1). where the semantic difference between frames reflects their temporal distance, with closer frames exhibiting smaller differences than those further apart. Continuity requires smooth visual transitions between adjacent frames, avoiding abrupt changes and high variance. To achieve this, we model sampled frame intervals as Brownian bridge process (Section 3.2). penalizing deviations from the expected trajectories. Different from prior works that relies on setting explicit goal frames, the proposed approach implicitly explore the global and local structure of actions without imposing rigid constraints.
3.1 Visual-I Language Ordering
To capture the temporal coherence of video actions, we first propose vision-language ordering (VLO) loss that ensures the semantic alignment between frames reflects their temporal order. Since the VLO loss is applied within each video individually, we henceforth write O,, 4 as O, I for simplicity. Consider an anchor frame O, O with an index n(i) corresponding to its position in the original video. For any given frame pair (o,, 0j), we first define the semantic alignment score R to quantify differences in their VL similarities w.r.l a language description / as:
R(v,, vj, 1) = - sisssmsiv. 1) sim(v,, 1)112.
where v, = (0 (1)). The function sim(-: .) computes the VL similarity using cosine similarity. To ensure the proposed Di adhere to the temporal ordering of frames, we construct negative set Na. by selecting o e O correspond frames that are temporally more distant from og than oy:
Ni. = for i (n()) m(k)) 2 (n(1) |n(i)-n(jj}>: n(jil):
This formulation allows us reformulate LINCE by enforcing that the VL similarity difference between frames i and j should be smaller than that between frame i and any negative frame k within the video O:
exp (R(v,, Vy, 1))
LVLO = (B(oo,,))~ ) log Σ or ENs, exp (M(V.,, I))
Notably, our VLO loss does strictly require 0j to be from a future timestep for goal-reaching. Instead, we leverage the inherent temporal dynamics in videos, allowing the model to learn the natural ordering in an unsupervised manner.
3.2 Vision-Language Continuity
While the VLO property provides a strong global constraint on the structural alignment of VL pretraining, optimizing triplet relationships alone be unstable. Variations in frame content and noise often lead to suboptimal local consistency. To mitigate this, we introduce an additional local continuity constraint inspired by the Brownian bridge [36). This stochastic process models transitions between two fixed endpoints over by any sampled local video interval [n()), n (i)]. For any time step I (n(i), ni()) within this interval, the transition density of Brownian Bridge process (() follows time-dependent Gaussian distribution:
N Nvv: t-n(i) n(j) n(i) (v, v.). Un((i) (o(()-n(i)) n()) -(i)) n(j) nii
where Vi, Vy E Rdd are the visual embeddings of the first and last frames in the sampled interval. The mean trajectory B[[((t) linearly interpolates between the two endpoints, while the variance

---

Var [(t) provides uncertainty modeling that peaks in the middle of the interval. To enforce this local continuity, the Brownian bridge loss LBB formulated as,
LB
1 T 1=1 2Var[B(t]] Iv. - -
E
(2)
This loss encourages local consistency by penalizing deviations from expected trajectories, ensuring consistency across short temporal spans.
Overall Objective. The final training objective integrates both global and local constraints to achieve temporal coherence simultancously:
LACTOL = Lvv. + ALBB,
where À is empirically set to balance two components.
Theoretical Analysis
In this section, we theoretically prove the vision- language ordering and continuity, as well as extend the robustness of linguistic perturbations of representations learned by AcTOL, All proofs are provided in Appendix C for reference.
Vision-Languaae Ordering. Ordering and sorting properties are well-established in self-supervised
learning 39 18 45). Building upon these insights, formalize the concept of vision-language
ordering (VLO) below.
Definition 1 (VLO Representations). Let {oi be sequence of video frames and / the corre- sponding language description. The representations of the frames are said to satisfy the VLO property for any 0 < d < if E [Z'], and distinct frames j, k E [T]] the following conditions hold:
{
R > Ra.k a + 1/6,
if d < dg
di, di,b. R..ht < Ri - 1/6, if dij > di,k.
[RR.,1 R. 8.
where Ri denotes R (v:, Vj, 1) and dij denotes (n(i) n(j))-
Implications the VLO Property. The VLO property enforces structured representation of video frames, ensuring that temporally adjacent frames have consistent and predictable semantic differences. When two frames have equal temporal distances from an anchor frame, their semantic gaps should be similar, fostering smooth transitions. In contrast, frames that are farther apart should exhibit larger semantic gaps, thus preserving the chronological order.
To formalize the temporal ordering constraints, we define the unique sorted set of frame distances from frame as [Da1 < Di.2 < .... < D3,M, ), where each Di,m m E [M;] is obtained by sorting the set [di.j j E [7] (i)) Additionally, we define the count of frames at each distance level as:
72,m == Ki de,j = Di.m= j E (T) {i))
which denotes the number frames whose temporal distance from frame i equals Di. The VLO property is satisfied when the proposed LVLO approaches its theoretical lower bound, which is given by:
T
M
-T=
nimm log nl,m-
(5)
T1
i=1 m==1
This bound characterizes the optimal alignment of VL similarities, ensuring that the learned repre- sentations preserve the inherent temporal structure within the video sequence, as guaranteed by the following theorem:
Theorem Vision-Language Ordering). L is light lower bound of Lv i.e., L 2 L", and for any > 0, there exists feature embeddings such that LVLO L + e. Furthermore, for any 0 < < 1, there exist > O such that if Lvvo <. L + 6, the learned representations satisfy the VLO property.
Continuity. We establish the following theoretical result to rigorously describe continuity preservation in languaee representations:
5

---

right
Franka Kitchen
Slide cabinet
Open
Open microwave
light
left
right
Metaworld
Pick and Place
Press button
Hommer
Assembly
Open Drawer
camera
Real Robot
Pick cup
Open [X] drawer
Close [X] drawer
Policy learning environments, including tasks with a real- Unitree D1 robot arm and 5 tasks each in two simulation environments, i.e., Franka Kitchen and Metaworld.
Figure 3:
Theorem 2 (Vision-Languaal Continuity). Let Vk, Vi be visual representations at arbitrary time steps within a Brownian Bridge- interval [n(i), n(j),, and let E C be language embedding. the VL similarity function sim(-) (-) Lipschitz continuous with constant then for any e 0, there exists 6 > 0 such that:
Vv
v1ll2
PR(v v1. <.
This result follows from two key observations: (i) Brownian Bridge regularization constrains each embedding to remain close to a linear interpolation between anchor frames, with deviations governed by time-dependent variance; and (ii) under this constraint, the distance between temporally close frames admits an explicit upper bound. Combining this with the Lipschitz continuity of the vision- language similarity function ensures that small changes in frame embeddings lead to proportionally bounded changes in alignment scores.
Building upon the continuity result, further demonstrate that the semantic alignment score remains stable under small perturbations language input:
Theorem 3 (Robustness to Language Variations), Let l' be perturbed language embedding such that || - l'll < 8t. Then the semantic alignment score R satisfies:
R(v,, 1') R(v,, 1), 208,
This second result guarantees that small shifts in the language representation (e.g., synonym substitu- tion or phrasing variation) lead to bounded changes in the alignment score. Together, Theorems [ 3 the local stability of semantic grounding across both time and modality, providing theoretical basis for continuity-aware vision-language learning.
Experiment
In our experiments, we aim to evaluate the effectiveness of ordered and continuous vision- Langu-l representations for robotic control. First, we conduct extensive Cloning (LCBC) experiments on both real and simulated robots to validate the importance of ordering and continuity for imitation learning. Second, we assess the utility of the learned representations as reward functions on multiple real-world action videos. The results demonstrate that the ordered and continuous representations enable our method to accurately identify action boundaries and generate dense rewards aligned with the given instructions. Finally, we evaluate the robustness of our method under language perturbations, showcasing its strong generalization capability for application in real-world daily scenarios.
Behavior
Experimental Setups. Figure shows the experimental environments. For real-world robot evalua- tion, we deploy the Unitree DI robot arm to perform three challenging manipulation tasks: pick cup, open [X] drawer and close [X] drawer, where [X] is the drawer index specified by the instruction. The pick cup task requires the model to accurately identify the cup handle, while the open/close [X] drawer tasks demand grounding of language instructions to visual observations, enabling the

---

Table 1: Comparison in simulation environments with varying amounts of demonstrations. Each result reports the success rate over 50 roll-outs, averaged across 2 camera views and 3 random seeds. We also report the relative performance gain in green compared to the strongest baseline.
FRANKA KITCHEN
METAWORLD 15 demos
Method
demos
13 demos
25 demos
5 demos
25 demos
CLIP
11.67 = 0.95
27.47 = 1.01
31.20 + 2.62
42.29 + 2.65
60.33 + 1.32
62.54 + 4.36
R3M LIV
28.60 + 1.39
42,20 1.00)
46.83 * 3.85
56.50 + 5.20
60.08 # 3.62
23.40 0.78
42.73 1.17
51.93 + 0.95
46.95 + 2.07
64,33 + 3.63
66.67 + 1.49
DecisionNCE
25.33 + 1.30
43.20 + 2.25
50.87 + 2.95
44.58 * 2.79
59.08 + 1.77
69.75 # 3.90
ACTOL w/o BB
32.80 + 1.23
54,20
1 0.85
60.80 + 0.87
50.29 + 4.05
70.83 + 4.21
73.33 # 2.83
AcTOL
42.60 = 0.53
61.80 # 2.54
64,60 + 0.57
53.81 3.89
74.13 1.59
81.13 1.59
(+48.955%)
(+43.06%)
(+24.40%%)
(+14.61%)
(+15.23%)
(+16.32%)
model to interact with the correct drawer. To isolate manipulation performance, the Unitree Go2 quadruped remains lying down and stationary throughout the evaluation. We use a web camera to capture third-person view visual observation. The action space consists of 6-DoF end-effector displacement vector and gripper state, executed at a control frequency of 20 Hz. For each task, collect 60 demonstrations via remote control using the Unitree Go app, which is significantly fewer than the 100 trajectories typically used in prior work 23). For simulation, we choose two widely used simulation environments for evaluation: Franka Kitchen [16] [2] and Metaworld [43] For Franka Kitchen, we evaluate five tasks; sliding a cabinet, opening the left door, opening the microwave, turning on the stove, and switching on the light. For Metaworld, we focus on learning five tasks: hammering a nail, pressing a button, picking and placing block, assembling a ring onto a peg, and opening a drawer. Detailed environment setup can be found at Appendix [.1
a state-of-the-art image-text representation it is natural choice to include CLIIP vanilla baseline for comparison. Our primary baselines are LIV (26 and DecisionNCE: (23), as we use the same model architecture and dataset for pre-training. We also compare against R3M (30 pre- trained on Ego4D (15), a dataset containing roughly 36x longer videos than EPIC-KITCHEN-IOS We also include an ablation variant of AcTOL where the Brownian Bridge loss removed, referred to as AcTOL w/o BB.
Baselines. Since our model is initialized with CLIP
widely applied in various embodied tasks p 20 38
Implementation Details. We initialize our model with the weights of CLIP 34 with ResNet-50 vision backbone and further pre-train it on human action video dataset SPIC-KITCHEN-100 [T] For hyperparameter selection, we uniformly sample 10 frames of each video per batch. The loss weight À is 0.1. Other hyperparameters, such as temperature,s follow the default value used in CLIP 34
More details of pre-training and hyperparameter sensitivity can be found in Appendix A
5.1
Landunge-Conditione Behavior Cloning
For LCBC policy learning, we keep the pre-trained vision-language encoders frozen and feed their output representations into lightweight MLP, which is trained as a policy network.
Simulation results, In simulation, each task is performed from two camera viewpoints (left and right), with varying numbers of demonstrations [5, 15, 25) (i.e., dataset size) for training, and evaluated under three different random seeds. We report the success rate across different environments and dataset sizes, averaged over camera views and seeds. Detailed comparison results for each task can [ presents the comparison results, demonstrating that AcTOL achieves significantly enhanced performance relative to baseline methods across all evaluated datasets and environments. This superiority is particularly pronounced in the complex Franka Kitchen setting, especially under data constraints, where ACTOL with fewer demonstrations (e.g., 5/15) often matches or surpasses other methods using more data (e.g., 15/25), indicating its high data efficiency and robust low-resource generalization capabilities. Furthermore, ablation studies confirm the integral role of the Brownian Bridge (BB) constraint; its removal (AcTOL w/o BB) results in a significant performance decrease, validating its contribution to improving representation quality for effective policy optimization via behavior cloning.
be referred to Appendix [B.] Table
Robustness under visual shifts. To further as- sess the ability of the model to handle visual distribution shifts, we conduct experiments in
the Franka Kitchen environment following the protocol of [5). Specifically, compare Ac-
S1
53
S4
S5
Figure 4: Visual shifts applied in Franka Kitchen.
7

---

ACTOL
DecisionNCE
LIV
R3M
CLIP
1.0
Normalized Reward
Normalized Reward
Noralized Reward
0.8-
0.8
0.8
0.6
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.0
50
100
350
20
40
60
80
Action: Qoen cupboard Put cup into cupboard Instruction: Opan cupboard
Action: Open Close can
Action: Open dishwasher + Clase dishwashar Instruction: Close dishwasher
Instruction: Open.con can
In-distribution reward
(b) Zero-shot rewerd human action
(c) Zero-shot reward robot action
Figure 5: Visualization of the normalized learned reward corresponding different actions. Our representations effectively help capture the correct temporal order of actions in the instruction. For more results, please refer to Appendix B.6
TOL with the strongest baseline, DecisionNCE, under visual changes absent from training. These include: (1) object distractors of increasing difficulty: easy (S1), medium (S2), and hard (S3), corre- sponding to scenes with 1, 3, and 9 additional YCB objects [6]. respectively; and (2) background
texture variations with marble hinge (S4) and metal slide (S5). All shifts shown in Figure
Policies are trained with 15 demonstrations per task, and success rates averaged over five tasks are reported in Table 2 While performance drops under visual shifts, which is expected, Ac- TOL continues to outperform DecisionNCE in all available test conditions. This suggests that
Table Success rate comparison across different visual shifts in Franka Kitchen.
Method
No shift
S1
S2
S3
S4
S5
DecisionNCE AcTOL
43.2
27.2
25.6
4.8
0.0
8.8
61.8
43.2
328
9.2
4.4
38.4
the learned representation maintains useful generalization ability even without any specific adaptation for visual domain shift.
Real Robot results. Table shows the real robot comparison results. AcTOL consistently outperforms all baseline models across the three tasks. Among them, the pick cup task yields relatively lower performance, as it requires the model to precisely identify and grasp the cup handle, demanding stronger spatial perception capabilities. For the open/close [X] drawer tasks, AcTOL is able to accurately interpret the drawer number specified the language instruction, align it with the corresponding location in the visual observation, and execute continuous actions the correct drawer to complete the task. These results highlight the effectiveness of AcTOL's learned visual-language representations in real-world manipulation tasks.
Table Performance comparison on Unitree
arm. Success rates are reported over 10 trials.
Method
Pick Cup
Open [X] Drawer
Close [X] Drawer
CLIP
20
R3M
40
LIV DecisionNCE
30
40
ACTOL
80
5.2 Conduage-Condittoned Visual Rewards
Open cupboard
Open fridge
By learning semantically smooth visual repre- sentations, our model further enables the use of semantic trajectories as effective task rewards. To illustrate this, first demonstrate the con- tinuity of purely visual representations. In Fig- ure 6 we visualize the learned visual represen- tation trajectories for three tasks, each with ten video clips, using t-SNE. The results show that AcTOL significantly improves the temporal con- tinuity of video feature trajectories while retain- ing CLIP's s discriminative ability to distinguish
75
50
25
-25
-50
-50
-50
CUP
ACTOL
Figure 6: Visual trajectory visualization.

---

5-
0
Success Rote Change (%)
-5
-10
-20
Instruction Veriants: (action)
-2
L2: Help me (action).
DecisionNCE
L3: Verb/Noun change
-30
ACTOL
GPT-4o generation
L1
L2 Slide Cabinet
L3
L4
L1
12
L3
L4
L1
12
L3
L4
L1
L2
L3
L4
L1
L2
L3
L4
Open Door
Open Microwave
Turn on Stove
Switch Light
Average
Figure 7: Success rate fluctuation across tasks in Franka Kitchen for different instruction variants.
between actions associated with different instructions. As discussed in Section 3.2 the visual continuity can stabilize learning ordered vision- langualanguuag alignment. Building on this foundation, we define dense reward signal based on the semantic alignment between the current visual state and the language goal. Specifically, at each time step i, we define the reward cosine( v 1) as the similarity between the current visual state and the language goal. While prior work 26 23 focused
primarily single-action video clips, we evaluate reward quality on three clips, each containing two consecutive actions, to assess whether the model can reliably capture fine- one-grained action semantics. Figure (a) presents an in-distribution evaluation using video from model produces a clear reward peak aligned with the completion of the "open cupboard" action, followed by a decline, indicating successful temporal localization of the instructed behavior. In contrast, R3M and DecisionNCE rewards continue increasing beyond the relevant action segment. Figures 5 (b) and (c) show results on real-works videos from (2). where human and robot actors perform opposite actions. Only our method consistently produces symmetric and instruction -alleie reward curves, accurately identifying both action boundaries and semantics.
Our
5.3 Robustness Study under Linguistic Perturbations
In the EPIC-KITCHEN-IU 100 dataset, textual annotations are often concise, such as "'oeen cupboard'' In the default setting of LCBC, we employ similarly structured simple instructions. In this experiment, to validate the robustness of the representations scenarios, we introduce several modifications to the language instructions. Specifically, we transform each original instruction into four conversational variants by varying lexical choices (e.g.. verbs and nouns) and incorporating ChatGPT-44 Appendix B.4 We then evaluate the imitation learning performance conditioned instructions the Franka Kitchen environment. For comparison, we select LIV and DecisionNCF, which are also pre-trained and DecisionNCE dropped by 11 11.9% and 2.7% on average, respectively, while our method main- tained a success rate comparable to that before language perturbation. This result demonstrates the robustness of our learned representations, which generalize more effectively to real-world scenarios.
method learns in real-world
32 generated complex instructions. Details can be found in
these modified
EPIC-KITCHENN-10.. As shown in Figure 7 the success rates of LIV
5.4 Mitigating the Human-to- Robot Gap via Fine-Tuning
Although pretraining human videos provides general- izable knowledge, bridging the human-to-robot domain gap remains persistent challenge 47
Table Fine-tuning AcTOL encoders efficiently improves the success rate in Franka Kitchen.
Since the AcTOL objectives capture the inherent temporal or- dering of videos in an umbodiment-agnoste manner, we can fine-tune the vision-language encoders with the same objectives used in pretraining during downstream behav- ior cloning. Notably, we find that fine-tuning with only
22 RT
Franka Kitchen
Frozen
Finetune
ACTOL
61.8
86.4
small number of robot demonstrations is sufficient substantially mitigate the domain gap. We take 25 in-domain demonstrations (5 per task) in Franka Kitchen to fine-tune the pre-trained encoders using the AcTOL objectives. Then, as in 5.1. we freeze the fine-tuned encoders and train policy networks on top using behavior cloning. We report the comparisons when using 15 demos for LCBC. As shown in Table the success rate improvement demonstrates that the learned temporal inductive bias can be effectively adapted to the robot domain with limited supervision.

---

Related Work
Given the success of large-scale pre-training in the vision and language research communities many studies have attempted to extend this paradigm to the field of robotics. Some work leverage massive robotic trajectory data [8] for pre-training. aiming to establish unified rision-anggaug--aee models [48] 7 [1 B [3 40
33.. However, collecting large amounts of high-quality robot trajectory data extremely costly and time-consuming. Consequently, many studies have begun to explore the use of large-scale, readily available, out-of-domain human action video data to learn generalizable representations that can be transferred to robotic tasks 337
23 Among these, TCN [37], VIP 127,. MVP [35], and VC-1 [28] focus solely on studying unimodal visual representations, limiting their performance when understanding language instructions is required. R3M [30] employs language and reward models to shape progressive visual representations, while Voltron 19] and MPI A model the transition from the current state to the goal state conditioned on language. However, during training, these approaches freeze the language encoder, using it only to aid in the training of visual representations. As a result, they do not effectively achieve multi-modal representation learning. LIV [26] and DecisionNCE 23 have attempted to leverage CLIP 34 to train embodied multi-modal representations. LIV treats language as the goal of video actions, aligning it with the final frame, while DecisionNCE. aligns language with the transition from the initial to final frame. Both rely on a goal-reaching assumption, which can lead to suboptimal results in noisy real-world videos. In contrast, our approach avoids rigid assumptions by enforeing semantic alignment that follows the intrinsic temporal continuity of videos, leading to more robust and generalizable vision-language representations. This property also benefits methods like UVD 46 which rely on pretrained visual features to detect phase changes and decompose long-horizon tasks. Our method more reliably identifies action phases, enabling stronger progress rewards and improving suitability for such goal-conditioned downstream tasks.
27
35
30
19
Conclusion and Limitations
We present Action Temporal Coherence Learning (AcTOL) as a promising vision-language pre- training solution for generalizable embodied agents. By learning action consistency from large corpus of human action videos, AcTOL theoretically ensures the ordering and continuity of vision- language representations, well as robustness language perturbations. Extensive experiments across various environments demonstrate that AcTOI. effectively generalizes to complex robotic manipulation tasks. While the temporal ordering of actions provides strong inductive bias for many goal-directed tasks, it may not align well with tasks that involve ambiguous, repetitive, or cyclic behaviors. In such cases, the assumption of coherent progression might break down, potentially affecting the reliability of the model. Future work could explore adapting AcTOL to handle such repetitive action sequences.
Acknowledgments and Disclosure of Funding
This work was partially supported by ARC DE240100105, DP240101814, BP230101196 DP230101753, BA24006, DE250100363, and ARC Industrial Transformation Research Hubs IH230100013.
References
[1] Marcin Andrychowicz. Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbecl, and Wojciech Zaremba. Hindsight experience replay. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems, (NeurlPS), pages 5048- 5058, 2017.
[2] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. In Kris Hauser, Dylan A. Shell, and Shoudong Huang, editors, Robotics: Science and Systems XVIII. New York City, NY, USA, June 27 July 1, 2022, 2022.
10

---

[3] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang. and Ury Zhilinsky. T0: A vision-language-action flow model for general robot control. CoRR, abs/2410.22114 2024.
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss: Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, (NeurIPS), 2020.
[5] Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation?. 2023.
[6] Berk Calli, Arjum Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference on Advanced Robotics (ICAR), pages 5 510-517, 2015.
[7] Chilam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, and Minzhao Zhu. GR-2: A genera- tive video-language-action model with web-scale knowledge for robot manipulation. CoRR, abs/2410.06158, 2024.
[8] Open X.-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alexander Herzog, Alex Irpan, Alexander Khazatsky, Anant Raj, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Gregory Kahn, Hao Su, Haoshu Fang. Haochen Shi, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mordatch, Ilija Radosavovic, and et al. Open x-embodiment: Robotic learning datasets and RT-X models. CoRR, 0bss2330.00864, 2023.
[9] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Roya Firoozi, Negar Mehr, Esen Yel, Rika Antonova, Jeannette Bohg, Mac Schwager, and Mykel J. Kochenderfer, editors, Learning for Dynamics and Control Conference, (LADC), volume 168 of Proceedings of Machine Learning Research, pages 893-905. PMLR, 2022.
[10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The EPIC-KITCHENT dataset. CoRR, bbs//804400274 2018.
[11] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision. CoRR, abs/2006. 13256, 2020.
[12] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
[13] Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong, Ted Xiao, Pannag R. Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Dana Kulic, Gentiane Venture, Kostas E. Bekris, and Enrique Coronado, editors, Robotics: Science and Systems XX, Delft The Netherlands, July 15-19, 2024, 2024.
11

---

[14] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video datahase for learning and evaluating visual common sense. In IEEE Internaiional Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5843- 5851. IEEE Computer Society, 2017.
[15] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Ro- hit Girdhar, Jackson Hamburger. Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Iijaa Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo. Jáchym Kolár, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhengiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang. Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li. Richard A. Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3, 000 hours of egocentric video. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, (CVPR), pages 11973-189901 IEEE, 2022.
[16] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving Long-horizon tasks via imitation and reinforcement learning. In Leslic Pack Kaclbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, (CoRL), volume 100 of Proceedings of Machine Learning Research, pages 1025-1037 PMLR, 2019.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), pages 770-778. IEEE Computer Society, 2016.
[18] Kai Hu, Jie Shao, Yuan Liu, Bhiksha Raj, Marios Savvides, and Zhiqiang Shen. Contrast and order representations for video self-super learning. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 7919-7929. IEEE, 2021.
[19] Siddharth Karamcheti, Suraj Nair, Annie S. Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu, editors, Robotics: Science and Systems XIX, (RSS), 2023.
[20] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: CLIP embeddings for embodied AI. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, (CVPR), pages 14809-14818 18. IEEE, 2022.
[21] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Paul Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-actioo model. CoRR, abs/2406.09246,, 2024.
[22] Guangrun Li, Yaoxu Lyu, Zhuoyang Liu, Chengkai Hou, Jieyu Zhang, and Shanghang Zhang. H2r: A human-to-robot data augmentation for robot pre-training from videos, 2025.
[23] Jianxiong Li, Jinliang Zheng, Yinan Zheng, Liyuan Mao, Xiao Hu, Sijie Cheng, Haoyi Niu, Jihao Liu, Yu Liu, Jingjing Liu, Ya-Qin Zhang, and Xianyuan Zhan. Decisionnce: Embod- ied multimodal representations via implicit preference learning. In Forty-first International Conference Machine Learning, (ICML). OpenReview.net, 2024.
12

---

[24] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, (NeurIPS), 2023.
[25] Yang Liu, Weixing Chen, Yongjie Bai, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied AI. CoRR, abs/2407.06888 2024.
[26] Yecheng Jason Ma. Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: language-image representations and rewards for robotic control. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhhrddt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, (ICML), volume 202 of Proceedings of Machine Learning Research, pages
PMLR, 2023.
[27] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: towards universal visual reward and representation via value-implicit pre- training. In The Eleventh International Conference on Learning Representations, (ICLR). OpenReview.net, 2023.
[28] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for artificial visual cortex for embodied intelligence? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, (NeurIPS), 2023.
[29] Yao Mu, Qinglong Zhang. Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision- pre-training via embodied chain of thought. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems (NeurIPS), 2023.
[30] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In Karen Liu, Dana Kulic. and Jeffrey Ichnowski, editors, Conference on Robot Learning, (CoRL), volume 205 of Proceedings of Machine Learning Research, pages 892-909, PMLR, 2022.
[31] Nghia Nguyen, Minh Nhat Vu, Tung D Ta, Baoru Huang, Thieu Vo, Ngan Le, and Anh Nguyen. Robotic-clip: Fine-tuning clip on action data for robotic applications. In ICRA, 2025.
[32] OpenAl. Gpt-4o technical report, 2024.
[33] Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. Spatialvla: Exploring spatial representations for visual-language-en action model. CoRR, abs/2501.15830, 2025.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Ma- rina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, (ICMI), volume 139 of Proceedings of Machine Learning Research, pages 8748-8766 PMLR, 2021.
[35] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski, editors, Conference on Robot Learning, (CoRL.), volume 205 of Proceedings of Machine Learning Research, pages 416-426. PMLR, 2022.
[36] Daniel Revuz and Mare Yor. Continuous martingales and Brownian motion, volume 293. Springer Science & Business Media, 2013.
13

---

[37] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Tome-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation, (ICRA), pages 1 1134-1141 IEEE, 201 8.
[38] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Conference on Robot Learning, (CoRL), volume 164 of Proceedings of Machine Learning Research, pages 894-906. PMLR, 2021.
(CoRL),
[39] Nina Shvetsova, Felix Petersen, Anna Kukleva, Bert Schiele, and Hilde Kuehne. Learning by sorting: Self-supervised learning with group ordering constraints. IEEE/CVE International Conference Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 16407- 16417. IEEE, 2023.
[40] Andrew Szot, Bogdan Mazoure, Harsh Agrawal, R. Devon Hjelm, Zsolt Kira, and Alexander Toshev. Grounding multimodal large language models in actions. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[41] Allison C. Tam, Neil C. Rabinowitz, Andrew K. Lampinen, Nicholas A. Roy, Stephanie C. Y. Chan, DJ Strouse, Jane Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, (NeurIPS), 2022.
[42] Seonghyeon Ye. Joel Jang, Byeongguk Jeon, Se June Joo, Jianwei Yang. Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos. CoRR, abs/2410.11758, 2024.
[43] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, (CoRL), volume 100 of Proceedings of Machine Learning Research, pages 1094-1100, PMLR, 2019.
[44] Jia Zeng, Qingwen Bu, Bangiun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, Heming Cui, Bin Zhao, Xuelong Li, Yu Qiao, and Hongyang Li. Learning manipulation by predicting interaction. CoRR, abs/2406.00439, 2024.
[45] Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-n-contrast: Learning continuous representations for regression. In Alice Oh, Tristan Naumann. Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, (NeurIPS), LA, USA, December 10- 16, 2023, 2023.
[46] Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta. Dinesh Jayaraman, Yecheng Ja- son Ma, and Luca Weihs. Universal visual decomposer: Long-horizon manipulation made easy, In IEEE International Conference on Roboties and Automation, ICRA 2024, Yokohama, Japan, May 13-17, 2024, pages 6973-6980. IEFF, 2024.
[47] Jiaming Zhou, Teli Ma, Kun- Yu Lin, Ronghe Qiu, Zifan Wang, and Junwei Liang. Mitigating the human-robwr domain discrepancy in visual pre-training for robotic manipulation. arXiv preprint arXiv:2406. 14235, 2024.
[48] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao
14

---

Lu, Sergey Levine, Lisa Lee. Tsang- Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishhan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marein Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: vision-language-actionn
models transfer web knowledge to robotic control. In Jie Tan, Marc Toussaint, and Kourosh Darvish, editors, Conference Robot Learning, (CoRL), volume 229 Proceedings of Machine Learning Research, pages 2165- 2183. PMLR, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction consistently articulate the problem, the proposed AcTOL solution, its key contributions regarding ordered and continuous representation learn- ing, and its demonstrated benefits and scope in vision-language pre-training for embodied agents.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations have been discussed in Section and Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: Full set of assumptions and a complete (and correct) proof is provided in Appendix [
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details have been provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The dataset used for pre-training EPIC-KITCHENS-100 which is publicly available, and the code is included in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiiieeeep httii///uiidd/soeeeuuuiiis SroodeSubiissi.e ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setting/details are discussed in Section 5 and Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Errors are provided for all language-conditioned behavior cloning experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiments compute resources are discussed in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Authors have read NeurlPS Code of Ethics and confirm to preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The EPIC-KITCHENS-IO0 dataset used in this paper has been explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur//: /00p://ccc...cc.eeeee/e////////0 for what should or should not be described.
22

---

Appendix
A Pre-training Details
we use modified ResNet-50 [17] from CLIP [34] for the vision encoder and CLIP transformer for the language encoder. We initialize model with CLIP and train them on
Following [26 23
EPIG-KITCHEN-100 TT [T]. The training hyperparameters used during the pre-training are listed NVIDIA A800 GPUs taking approximately 30 hours. For hyperparameter sensitivity, we report the model performance under varying numbers of sampled frames and different values of the loss weight A. As shown Figure 8 increasing the number of sampled frames leads to higher success rates, likely because it better preserves the temporal ordering and continuity in the video sequence. The model shows low sensitivity to X, as we observe that LB converges much faster than LVLo due to its unimodal nature. As result, LB primarily serves as constraint during training rather than dominant optimization objective.
in Table The training was conducted
Table 5: Hyper- parameerrssstrrs for pre-training.
80
Config
Value
70-
Training epochs
1000
60
Optimizer
Adam
40
40
Learning rate
1 x 10
Batch size
128
Frames per video
10
weight
0.1
10
Weight decay
0.001
10
0.01
Momentum (8,, 82)
0.9, 0.999
S.ST frames
Augmentation
RandomCropResize
Figure 8: Hyper parameeassseters sensitivity.
B Evaluation Details
B.1 Simulation Environment
We follow for the specific simulation environment setup and code details.
Franka Kitchen. The Franka Kitchen environment [T6 12 is based on the 9 degrees freedom Franka robot. The Franka robot is placed in kitchen environment containing several common household items: a microwave, a kettle, an overhead light, cabinets, and an oven. Following [30], the Franka Kitchen environments used in this paper are modified from their original design. Specifically, we introduce additional randomization to the scene by randomly altering the kitchen's 's position between episodes. This modification makes the tasks significantly perception and control.
challenging in terms of both
The Metaworld environment is an open-source simulated benchmark for robot learning. In our settings, the target object position randomized between episodes in all tasks.
Metaworld.
43
We present the specific default language instructions for each tasks Table 6
B.2 Real Robot Environment
Our real robot environment is a real-world office scene where the Unitree DI robot arm can interact with cup and drawer. The pick cup task requires the robot accurately identify the handle of the cup, while the open/close [X] drawer task requires the robot to understand the drawer index specified in the language instruction and align it with the visual observation. As shown in Figure 9 we use the Unitree Go app interface to remotely control the robotic arm for action data collection. Visual observations are collected using -hird-person perspective web camera in a same frequency (20Hz) with action. During control, the whole system, including AcTOL and the policy MLP, runs on GeForce GTX 880M GPU.
23

---

Table 6: Language Instructions for tasks in Franka Kitchen and Metaworld.
Environment ID
Language Instruction
Muchen__microoppen-V
open microwave
suchen_sdoor_opent
slide cabinet open left door
kitchen aitchen_ldoo 1ilor_opee-vv3
kitchen knobl_one on-v3
turn on stove switch on light
kitchen Sttchen_light_toee on-v3
hammme--2-ggool--bbervvaaab
hammer nail press button
bi.-pickinggx2-ggallooeevaaa
pick and place the block between bins assemble the ring onto peg open drawer
drawer-open-v2-gun -oaaen---gggall-oberaabbblllll
joet
Robot Arm
Jeset
Trejectory
JONTZ 90*
Figure 9: Action space of Unitree D1 arm and the remote control interface on Unitree Go app.
B.3 Language-Conditionel Behavior Cloning Hyperparameters
We present the LCBC imitation learning hyperparameters Table 7 For each distinct task in simulation, we run an evaluation episode every 1,000 gradient steps by running 50 roll-outs and computing their average success rate. Over a total of 10,000 gradient steps, we conduct this evaluation 10 times. The highest success rate among these 10 evaluations is reported the final result. To ensure robustness, we average the results across two different camera viewpoints and three independent random seeds. In total, run: 9 (tasks) +2 (views)*3 (demosizes) #3 (seeds) *6 (models)=972 (episodes), each episode takes approximately 2 hours on our workstation with a 24-core CPU, resulting in total of roughly 1, 944 hours for the simulated LCBC experiments. For each task the real robot, we use the final checkpoint and perform 10 evaluation runs with fixed random seed, due to the cost of real-world policy evaluation.
Table 7: Hyper- Hyper-parameters for LCBC.
Franka Kitchen
Metaworld
Real robot
MI.P achitecture
(256,256)
[256,256]
[256,256]
Non-linear activation
ReLU
ReLU
ReLU
Optimizer
Adam
Adam
Adam
Gradient Steps
10K
10K
50K
Learning rate
1 x 10-3
x 10-3
x 10
Batch size
32
32
32
Horizon
50
100
100
Proprioception
9
4
No
B.4 Linguistic Perturbation Results
To assess the robustness of AcTOL under language perturbations, we perform extensive experiments across four instruction variants. Instructions 1 and 2 transform the original action into more con-
24

---

Table 8: Success rate fluctuation across tasks in Franka Kitchen for different instruction variants.
Task
Instruction
LIV
DecisionNCE
AcTOL
1. Please slide cabinet for me.
32
8
-1
2. Help slide cabinet.
26
Slide Cahinet
3. Push open the right cuphoard door.
32
4. Mind pushing open the right cup- board cabinet door? need to grab the cups inside.
-32
Average
30.5 + 2.6
-5.8 + 2.9
0.5 + 2.2
1. Please open left door for me. Help open left door.
-3
-3
Open Left Door
3. Pull open the left cabinct door.
-3
4. Can you pull open the left cabinet door? I need to grab something inside. Average
-1
3.4
-1. + 1.1
0.8 + 1.9
1. Please open microwave for
-5
2. Help open microwave.
-4
Open Microwave
3. Pop open the microwave oven door.
-5
4. Would you mind helping me pop open the microwave oven door so I can heat up my lunch?
-5
-2
Average
4.8 - 0.4
1.0 +
2.5
Please
for
-8
2. Help me turn on stove.
8
-5
Turn on Stove
3. Rotate the control knob to activate
the stove.
4. Let rotate the control knob
-2
activate the stove cooking dinner. Average
-.8 = 0.4
5..0 + 3.1
005 + 1.5
Please switch on light for me.
2. Help me switch on light.
13
Switch on Light
3. Flip the light switch.
- 12
5
3
4. Could you reach and flip the light switch to brighten the kitchen area?
12
-3
-6
Average
-12.3 - 0.4
-2.5 2
2.7
1.. +
3.0
Average
- -11.9 + 0.5 22.7 +
1.2 0.9 1
1.7
versational forms. Instruction 3 introduces vocabulary diversity by varying the verbs and nouns used. Instruction further extends Instruction by incorporating linguistically complex expressions generated using ChatGPT-4o. We present the comparison results obtained from experiments in the Franka Kitchen environment, with data size of 5. As shown in Table & AcTOL outperforms the baselines in most instruction perturbation scenarios, thereby validating its robustness.
B.5 Condutgge Behavior Cloning Results
In Table 9- 14 we report detailed Language-Coman Conuage-Conditioned Behavior Cloning results for different task and dataset The results demonstrate that our method achieves significant improvements across different simulation environments, varying dataset sizes, and diverse robotic manipulation tasks.
B.6 Conditge-Conddttonee Visual Reward Results
As shown in Figure I0 we present more visualizations of Language Coniitiion-niiiinneeeeee Visual Reward on real-world robot manipulation videos from (2) In Figure 10 a), the robot performs two consecutive and opposing actions. Our method effectively identifies the action boundaries and generates the correct reward sequence, increasing first and then decreasing, alignment with the given instructions. In Figures 10 b)-)),, where the robot performs a single action, the robot initially moves slowly
25

---

Table 9: LCBC results when dataset size= on Franka Kitchen.
Method
Slide Cabinct
Open Door
Open Microwave
Turn Stove
Switch Light
Average
CLIP
38.7 5.1
2.0 = 1.0
3.0
2.6
7.7
11.7
0.9
R3M
68.7
18.3 4.0
7.7
19.3
29.0
25.6
LIV
56.0 1.0
6.0 2.9
7.0
13.0 0.6
22.0 2.6
20.6 0.7
DecisionNCE
59.3 6.8
9.7 1.5
7.0
26.3 4.5
24.3 2.5
25.3 1.3
ACTOL w/o BB AcTOL
7.5
11.5
10.5
23.5
47.0=
32.8
8
20.0 2.1
18.3
24.7
62.3 2.8
42.6 0.3
Table 10: LCBC results when dataset size=
15 on Franka Kitchen.
Slide Cabinct
Open Left Door
Open Microwave
Turn Stove
Switch
Light
Average
CLIP
7.003..
8.0 2.0
15.7
11.7
25.0
27.5
R3M
81.0 1.0
3.7
22.0
19.3 4.7
57.7 = 3.8
42.2 1.0
LJV
85.0+5.6 5.6
19.0 3.0
28.3 2.9
29.7 3.5
51.7 = 2.3
42.7 = 1.2
DecisionNCE
92.0 6.6
18.7 4.5
27.0
33.3
45.0 =
43.2
ACTOL w/o BB
3.5
2.1
54.0
AcTOL
0.7
37.5 5.6
37.0 - 4.2
53.5 3.5
81.5 2.1
61.8 2.5
Table 11: LCBC results when dataset size= 25 Franka Kitchen.
Method
Slide Cabinct
Open Left Door
Open Microwave
Turn Stove
Switch
Light
Average
CLIP
66.3 + 7.5
8.7 =
18.7 1.5
23.7 =
38.7
2.3
31.2 = 2.6
R3M
84.7 + 6.8
35.3 + 4.0
40.0 1.0
34.0 + 5.3
61.7 + 10.7
51.1 = 2.8
LIV
91.7 5.9
26.0 2.6
3.6
45.3 1.:3::00 0.6
61.7 332
51.9 0.9
DecisionNCE
91.7+
10.4
37.0
47.3
ACTOL w/o BB
92.0 2.4
37.0 + 5.4
40.0
57.0 1.5
78.0 6.2
60.8 1.3
AcTOL
100.0 00.0:00.. 0.0
37.0 + 7.1
42.5 +
62.5 + 2.1
81.0 = 4.2
64.6 0.6
Table 12: LCBC results when dataset size= 5 on Metaworld.
Method
Assembly
Pick bin
Press button
Hammer
Open drawer
Average
CLIP
48.3 1.3550 5.7
35.3 = 2.3
34.3 + 4.9
51.2 + 2.8
91.0 +1.0 1.0
52.0 = 2.7
R3M
5.6
33.3 + 5.1
27.3 5.1
63.2 1 7.1
92.3 0.6
55.9 3.9
LIV
61.8 6.5
32.3 9.0
32.7 3.5
61.0 6.1
100.0 0.0
47.7+22.1
DecisionNCE
3.6
3.6
2.5
65.7
100.0 0.0
55.7 2.8
ACTOL w/o BB
66.8 1 1.4
39.0 + 16.8
20.7 11.7110 1.5
74.7 + 1.5
100.0
60.2 5.1
AcTOL
62.8 + 6.0
41.0 6.3
42.0 + 4.5
69.5 + 0.7
100.0 + 0.0
63.1 3.9
Table 13: LCBC results when dataset size= 15 on Metaworld.
Method
Assembly
Pick bin
Press button
Hammer
Open drawer
Average
CLIP
73.0 7.8
40.3 5.5
52.0 = 7.9
76.0 + 5.0
96.7 = 0.6
67.6 + 1.5
R3M
80.7 7.6
17.0 12.3
45.0 4.6
83.3 + 4.5
94.0
64.0
5.2 5.2
LIV
2..
37.0 8.7
54.7 1
81.3 5.9
100.0
71.4
DecisionNCE
73.3 10.8
36.7 5.0
43.3 1 2.1
83.0 6.0
100.0
67.3
ACTOL w/o BB
94.033.0
50.3 = 18.6
48.3 = 1.5
90.7 = 1.2
100.0
76.7 (6.7558
AcTOL
82.5 0.7
64.5 + 3.2
65.5
3.9
81.0 + 2.1
100.0 0.0
79.3 + 1.6
Table 14: LCBC results when dataset size= 25 on Metaworld.
Method
Assembly
Pick bin
Press button
Hammer
Open drawer
Average
CLIP
69.3
36.0
11.8
66.0
78.8
99.3 0.6
69.9 1
R3M
87.7
14.7 11.6
48.3 2.1
89.7 - 3.5
100.0
+ 0.0
68.1
LIV
+
+
66.0 + 6.8
89.7+ 2.5
100.0 0.0
73.3 +
DecisionNCE
4.99
47.0 + 12.8
58.0 1.0
88.3 + 6.7
100.0 0.0
75.8 1 3.9
ACTOL w/o BB
93.7
117:100
51.7
55.0 1..0:..5 3.5
93.0 = 1.0
100.0 0.0
78.7 = 3.5
AcTOL
93.5
66.0
76.5
88.5 3.9
100.0 0.0
84.9
1.6
searches for the target. Correspondingly, the reward grows gradually. Once the robot interacts with the object and completes the task, our method captures the distinct semantic changes in the action, leading to rapid reward increase. In Figures 10 (f-(), we test two complex actions and instructions
26

---

1.0
1.0
Reward
Normalized Reward
0.8
0.5
0.6
0.6
0.4
0.4
0.0
0.0
20
60
80
20
40
Frame
Frame
Action: fridge Close fridge
Action: Open faucet
Instruction: Open fridge
Instruction: Open faucet
(a)
(b)
1.0
1.0
Normalized Reward
0.8
Nornalized Reward
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
20
40
0
20
60
Frame
Frame
Action: Push chair
Action: Unplug socket
Instruction: Push chair
Instruction: Unplug socket
(c)
(d)
10
1.0
Normalized Rowerd
Normalized Rowerd
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
20
140
60
20
40
60:
Frame
Put blue cup into red cup Instruction:
Action:
Action: Stack dice
Put blue cup into red cup
Instruction: Stack dice with 6-face facing you top of dice with 4- 4-cace facing you (f)
(e)
Figure 10: Reward plots for exemplar robot action videos.
to explore the limits of our method. In Figure e0]e), the model is required to accurately distinguish between the blue and cups to complete the task. Figure 10 the model needs to differentiate the orientation and face values of two dice. These scenarios impose high demands the model's s visual and semantic understanding. Our method successfully produces the correct rewards both tasks, showcasing its potential for application in real- wallworld, complex scenarios.
27

---

C
Proofs
C.1
Proofs Theorem
For the proof of Theorem 1 closely follow the approaches presented in 45 and adapted to triplet case. We prove the theorem three steps:
(1) T(
nim log fi.m is lower bound TvLo, i.e., LVLO >
(2) L is tight, i.e., for any <> 0, there exists representations such that LVLo < L +
(3) For any 0 < 8 1, there exist e O, such that if LVLO cc, then the learned representations satisfy VLO property.
T Σ
(1) Recall that LVLO
log
exp( exp( where Wh.re.N.s (vvl + i, i,dis
T-1
$=1,$$
v kke
), we rewrite it as
exp (Ht.j.))
LVLO
log
T(T - 1) JE(RM
Σ Ae
exp (Ri,,.)
M;
exp (R.y.t)
Σ
log Σ
T(T - 1) i=1 m=
exp (R,
JE Di.m.
M
Σ
log
T(T
1)
Σ
exp (Ri,k.1 - R(
(7) Dram
T M
Σ
Σ
log
T(T
1)
Σ
exp (R. R...))
ME +
Σ
exp (Hi Righ.)
T
Σ
log
kE|TIV() D
T(T 1)
Σ
exp (9t H4y1.
:=1 m=1 dd
AE 2
M;
exp (R. (
Σ
log
T(T 1)
Σ
exp (Rt
1=1
[7],), Diom
NE(TTMM D
exp (R..k. Re...)
T M. T(T 1)
Σ
log
1+ Me D
Σ
exp RRi.k.! Ri...)
m
jE di
exp (R
Σ
log
T(T
Σ
exp (R)
i=1 m=1
(6)
Vi E [T], [M.]. from Jensen's Inequality have
exp (Rag.)
Σ
log
Σ
exp (R.
exp (Ht
2 -n log
Mi.m log mimm-
Ni.m
Σ
exp (H,,..))
JELTMM JD.. D -
28

---

Thus, by plugging Eq.
into Eq. (6) we have
M. Ni.m log fi,m 1.*
LVLO T(T
(8)
i=1 m=1
(2) We will show for Ve >> O, there is set of representations where
Raji Ra,. + y if di,j di Rijh Ra,, if d = di
and y log
Vi E k E [T] such that LVLo < L' + c. For such a set c an
T
min
[Ac]]
of representations, Vi E
( m E [M,] .J E { d. = Di
exp (R....)
log
log ni,m
(9)
Σ
exp (R.,..))
since R. Righ for all & such that di Dim di.j. and
E 1+ ACTTMG) Di.m
log
exp(9A. R....1
Ne(TivAtid Dr.m T Texp(->)) aism
(10)
log ( +1+
T Texp(->) I
nim
As R. Ri.j < - for all i such that di, Da.m di,s and Ri,ke Res, 0 for all & such that di,b = Di. = di.j- From Eq. (6) we have
T
exp (Ri.
LVL.
log
T(T 1)
exp (RR,,.1)
xm mi
(T)
MC d D..
de, Di.m
exp (Ri Rig..)
(11)
T
(+) d. Dam
log
1+
T(T
exp (Hi Ra
1=1
ME
( dr D.
d D.m
By plugging Eq.
9 and Eq.
into Eq.
we have
T
M.
LVLO < T(T
nym log nimm + L +e
(12)
12
(3) We will show 40 < 8 < 1, there is
1+ (1
1 T(T-
+ exp(o)
min
min iE[T].mE[M.
log
2 log
-
0,
Ni,m exp (6 !)
2
such that when LVLO I + e, the representations satisfy VLO property. We first show that [R Ry,A.l < 8 d = d. [T),j E [T] when Lyno < L From Eq. 6 we have
29

---

T M,
exp (Hi.).)
LVLO >
Σ
log
(13)
T(T 1) = (E) D..
E
exp (R,,),))
kE Di.m
Let Pim
arg min (E[
Righh fim
arg max (
Rijdd Si.m
Ri Pi.ps. Ti.m
Sio have
di Vi E [7], m E [M,], by splitting out the maximum term and the minimum term
exp (Si.m)
LVLO
log Σ
T(T 1) 1=1
exp (Ri,k.))
RE (i).d, Dim
(14)
E
Risd
exp (Si.m + 7m
MELT J.i., =Dum
+ log
log
Σ
exp (R)
Σ
exp (R,,
k
Let fm
Σ
exp (R....t - Si,))s we have
jELT PiPs dd D
exp (Gim)
- log
log (1 + exp (mr.) + (nim - 2) mm
(15)
Σ
exp (R.,k.))
and
exp
(Gim + 1),m)
-log
cxp (R. = log (1 + exp (7i.m + (ns.m 2) (2 - Bim
(16)
Σ
kE (i)::),))
Then, from Jensen's 's inequality, we know
ni,m
exp
Σ
Ridd
S
1
Σ
exp ()p(
(17)
ni,m
2
(f. di.=Dimm
jE(T] (1.P), dimm di., Di...
thus
exp
EsErT Mipp migmm Righ. di,; DD.m
log
nmm
(18)
E EveTTy exp(PRx...)) di Di.
(Mi.m
2) log exp(1i.m) + (1ii, -2)0i.m) (nim - 2) log log(Oi.m)
By plugging Eq. 15 Eq.
16 and Eq.
18 into Eq. (14). we have
T Mi
LVLO T(T 1)
14.m log (1 + exp(1)mm) (n m)
m=1
(19)
n. - (num - 2) log(Os,m)
30

---

Let (()) == n,m log (1 + exp (7t.m) + (nam 2) -77mm (ni.m 1+xxp(m..) and increases monotoni
7m.m
2) log(0). From derivative
analysis we know h(0) decreases monotonically when 0 E [1, 2 cally when 0 E [++ep((n.m) exp (1b.m)
thus
h(0)) 2 1 + exp (1i,m)
++ exp (7im)
=
nimm log Ni,m + 2 2og
(20)
2
2
By plugging Eq.
20
into Eq.
19
we have
T
+ exp (1i.m)
LVLO
Nim logni,m log
1i
T(T
1)
2
1m m=l
(21)
M,
+ exp (7..)
L' T(T 1)
2
T,.m
Then, since 7/.m 2 0, we have 2 log
+exp((e..))
Bim 2 Thus, Vi E [T], m E [M.]
2
1 + exp (16.m) 17.m
LV > I+ T(T
- 1) 22log
(22)
2
If LVLO < <L+ L* T(T-1] (2log 1+exp(5) - -0). then
1 + exp (m).m)
1 + exp(() 6
2 log
M1mm < 2 log
(23)
2
2
Since y(x) 2 log 1+exp(x) increases monotonically when 0, have 7i.m 8. Hence
Vi E j, E (T)) d.,j = d. = Di,m, [Rij. - R. < li.m < 5. Next, we show
Hg 9 R 08fd di when LV < L From Eq. we have
1 T(T 1)
exp (R
LVLo
log Σ TT
exp(R....)
T
DD
de D
Σ
exp(R - -Rij.)
(24)
M
kE TT
1
Σ
Σ
log 1+
d.
D.m
T(T 1)
Σ
exp(R - -Ra.s.)
(
ke K di.
D
and combining it with Eq. have
Σ kE[T]V
exp(Ri... Righ.))
M
LVIO 2 + T(T 1)
log
1+
>
exp(Ri. Rigg.)
m=
TE D D.m
kE d. Di.m
(25)
exp(Ri.t. R(y.)
I + T(T 11)
log 1+
Σ
exp(Dir,h.. Ri..)
d, d.n=di.
31

---

vi E [T].j E k E E [T] day < dr When LVLo < L + we al-
ready have [R Riji 6, vd,.,. di.. which derives R.N. Ri and thus exp b.. Ra.j.l) exp(6). By putting this into Eq. (24). we have vi E [Th E [7]
{A E (+) dis )
LvLo > L + T(T - log 1+ exp (R Ri
1
(26)
nirr. exp(o)
where ri. E [M.]] is the index such that Dirrus = di.s-
Further, given LVlo < L +e L + T(T-13 log 11 airey exp(8+?]
we have
exp kR. k.l - Risdi)
log 1+
log 1+
(27)
Birs exp(o)
Dim exp (5+
which derives Righ > R. + 1. Vi E
[T): E [7]][], e (TR das < d. Finally,
vi E [T].jk E [TM,O), Risi < R. if di > di, directly follows from Ri.j > Re +
d < di
C.2 Proofs of Theorem
Setup and Assumptions. To provide the vision-language continuity, we first assume that the frame embeddings [vr]. where [1, are regularized under Brownian Bridge process (t) as discussed in Section 3.2 where the transition density for any intermediate time t E [m(i), m()]
within a sampled interval is given as:
B(I) ~ W (B|(()), Var(B(()),
(28)
with:
nij) m(i) (v, v.), Var[B([] (L - ((()) -)
1 (()
B[B(()) V.
(29)
n(j) (i)
All time steps [1, covered by at least one sampled interval, ensuring the entire video sequence satisfies the Brownian Bridge regularization. Now, let VA. E Rd be arbitrary embeddings, not necessarily the endpoints v, and v, of sampled interval. These embeddings fall within the union Jl of all sampled local intervals. Without loss of generality, here we can identify the interval (n(i), n(j)] E I from the union containing Va and Vı.
Bounding Local Continuity. Recall that semantic alignment score R(vk. Vi, 1) is defined as:
R(V,, Vi, 1) -I| simmsvi.aa 1) sim (vi, 1)ll2,
where sim(-) is Lipschitz continuous with constant C > O when embeddings are normalized as unit vectors. By the Lipschitz continuity of sim(-), we have:
sismmvi: -imm(v, 1)1la
- ville.
To ensure the continuity of R, must bound |v vrll2. Under the Brownian Bridge regularization, the embeddings are aligned with the mean trajectory [t)], and deviations are constrained by the variance Var (ar B(t)). Specifically:
V E[B(O]I A
where > depends on the strength of the Brownian Bridge loss LB. Below we omit for simplicty. Substituting the variance:
(()(( Var = n²()) - -n())
Bounding Pairwise Distance. The total pairwise distance between and vi can be expressed as: villa < |E](() E[B()]] [ar]] + (Vr]BB()
live
32

---

Since the mean trajectory [t) is linear within the interval [n()), r2()), we have:
E [k)) E[B()]l2 S | -
n(1) n(i) |V; vill2-
Combining these bounds, now we can rewrite into the following inequality:
VIv ville - k k- VV, vella + (k (n))) - k)
(1 (1- (l=n(i)(n()) -
+
n(j) n(()
n(j) n(i)
n(i) ni)
For the variance terms, the Brownian Bridge process achieves its maximum variance at the midpoint t m()++(() This gives us,
2
n(i) - nl()) ||v vella S "n()) nii) + v (m(i) n(i))
1--1
Var Var(B(4maxi)
4
Bounding Semantic Alignment Score. Finally, by substituting this bound into the Lipschitz continuity of sim, we obtain,
(n2 1 n(() n(i)
v((()) m(1))
RR(v,, 1I
v(n(i)-n(i))
To ensure [Pt(vs, 1) < e, we require:
k c1
C
+ vn(j) (n())-n(i)) n((i) <e.
(nj) m())
Here, we consider these two terms respectively:
k 1|
2C n(i) - n(i)
<
2. Cvn(j) n18)) < <²2¹
which gives:
e. (n(n))) -n(i)
n(j) - -n(1)- ( E
1k - 1| <
4C
20
Combining these conditions, we choose:
(n(j) - n(i))
62
8 min
4C
4C
Final Conclusion. For any given e > 0, setting
= min
(=((())n(())
40
10
ensures:
vell2 < piiv,, V1, <e.
C.3 Proofs of Theorem
From the definition of the semantic alignment score, we have:
R(v,, v,, !) = =-Isim(ve (v, !) svm !)1, A(v,, vj, I') = - sim (n,, - sim(v;, 1')
The difference in scores can be bounded using the reverse triangle incquality:
R(v, vy, 1') - R(v, v,, l) S [ssim(vs, I') sim(v,, I')) (sim(v., sim(s, (an(v,,))))
Simplifying the incqualities above, it gives us: R[(;,, v,, I') R(v,, Vj, 1)] S siim,, I') sim 1)1 + simmmv, 1') - sim 1))-
By the Lipschitz continuity of sim, we have: for some constant C > 0,
sim((,, I') - sim 1)I S C|IL - 11|2, simm(v, I') - sim (v,, 1)1 S CIIII 1/ Substituting these bounds and considering |||' 1|I2
RVv,, 1') R(v,, vy, 1)] < 2C||11 - 1112 s 2C8.
(30)
33

---

D Broader Impacts
We introduce Action Temporal Coherence Learning (AcTOL), vision-language pretraining frame- work aimed at improving the generalization capabilities of embodied agents in a variety of ma- nipulation tasks. By learning from large-scale human action videos, AcTOL helps agents acquire temporally consistent representations aligned with natural language, which can support more flexible and data-efficient robotic learning. However, some potential risks should be acknowledged. AcTOL is trained on video data that contains societal biases stereotypes, those patterns may be reflected in the model's behavior. For instance, if certain groups actions are underrepresented or portrayed inaccurately, the resulting agents could behave in ways that inappropriate unreliable in diverse real-world settings. While these challenges are common across many data-driven systems in robotics and vision-language learning, we believe future work should explore strategies such as dataset auditing, fairness-aware aware training, and improved transparency to support more responsible and robust deployment.
34


Reviews:
[{"review_id": "obSzUhgeOk", "reviewer": "Reviewer_h3Cm", "summary": "This paper proposes AcTOL, a vision-language pretraining framework that leverages temporal ordering (via a Vision-Language Ordering loss) and temporal continuity (via a Brownian bridge constraint) to learn more robust and transferable visual-language representations from egocentric human action videos. The motivation is to reduce the dependence on expensive, fully-annotated robot trajectories for imitation learning. AcTOL is validated through language-conditioned behavior cloning on both real and simulated robot tasks, along with reward learning experiments and language perturbation tests. The results demonstrate improved data efficiency and robustness compared to prior methods.", "questions": "1. How do you mitigate the domain gap between egocentric pretraining data and third-person robot observations?  \n2. Does the method remain robust if significant occlusions, distractors, or dynamic environment changes are present?  \n3. The hyperparameter λ controlling the trade-off between the ordering and continuity terms is fixed at 0.1, but no sensitivity analysis is presented. Did the authors examine how varying λ impacts training stability and performance? \n4. The Brownian bridge assumes smooth transitions across video frames, yet real-world tasks often exhibit clear multi-stage or discrete transitions (e.g., “grasp → lift → place”). Could the bridge oversmooth critical semantic boundaries? Have the authors considered segment-wise or adaptive bridge constraints to better capture such multi-phase actions?  \n5. The VLO loss defines negative samples purely by temporal distance. In videos containing repeated or cyclic actions (e.g., repeatedly opening/closing a drawer), temporally distant frames might share nearly identical semantics. Could the authors clarify how reliable temporal distance is in such repetitive scenarios, and whether any alternative negative sampling strategies were explored?", "limitations": "Overall, the authors have not sufficiently addressed the broader limitations of their method or the potential societal risks, such as domain transfer challenges and biases in human demonstration data, which warrant more thorough discussion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**  \n1. Experiments span real and simulated robots, with reasonable baselines and ablation comparisons.  \n2. The mathematical analysis provides insights into the expected temporal consistency properties of the learned features.\n\n**Weakness**\n1. Real-robot experiments are limited to three fairly simple tasks in controlled conditions, without diverse backgrounds, occlusion, or dynamic disturbances.  \n2. There is a significant domain gap between egocentric pretraining data and third-person real-robot evaluation, which is insufficiently analyzed.  \n3. The language perturbation coverage is narrow and focuses on short phrases, lacking complex compositional instructions.", "was_revised": true, "final_justification": "The extra analyses and experiments have effectively addressed most of the concerns I brought up, so I have decided to raise the score.", "rebuttal": "Thank you for your thoughtful feedback! Below, we provide point-by-point responses to each of your questions.\n\n> **W1/Q2. Robustness under visual shifts.**\n\nWe appreciate the reviewer’s concern regarding the limited diversity in our real-world experiments. Due to limited time, we were unable to collect additional data and complete a full evaluation of visual shift robustness in the real-world setting. To partially address this concern, we have conducted **additional experiments in the Franka Kitchen environment by introducing visual distribution shifts** following the setup in reference [1]. In particular, we evaluate AcTOL and the strongest baseline method, DecisionNCE, under various visual changes that are not present in the training data. These changes include:\n* **Object distractors** of increasing difficulty: easy, medium and hard level, corresponding to scenes containing 1, 3, and 9 distracting objects from YCB object set, respectively.\n* **Texture variations** in the background: marble hinge texture and metal slide texture.\n    \nWe use 15 demonstrations per task for policy training. The success rates averaged over 5 tasks under each setting are presented in the following table:\n|Method|D(easy)|D(medium)|D(hard)|T(marble hinge)|T(metal slide)|No shift|\n|-|-|-|-|-|-|-|\n|DecisionNCE|27.2|25.6|4.8 |0 |8.8| 43.2|\n|AcTOL|**43.2**|**32.8**|**9.2**|**4.4**|**38.4**|61.8|\n\nWhile performance drops under visual shifts, which is expected, AcTOL continues to **outperform DecisionNCE in all available test conditions**. This suggests that the learned representation maintains useful generalization ability even without any specific adaptation for visual domain shift.\n\nFor future work, we plan to improve visual robustness by incorporating stronger image backbones such as Vision Transformers, applying more extensive data augmentation during pretraining, and exploring domain randomization techniques to enhance performance under visual out-of-distribution conditions.\n\n[1] WHAT MAKES PRE-TRAINED VISUAL REPRESENTATIONS SUCCESSFUL FOR ROBUST MANIPULATION? Arxiv'23\n\n\n> **W2/Q1. Domain gap between egocentric pretraining and third-person deployment.**\n\nThank you for raising this important point. There is indeed a visual domain gap between human-centric videos and third-person robot observations. AcTOL can addresses this challenge in belowing ways. \n* AcTOL learns a **general** inductive bias of temporal ordering and continuity from vision-language pretraining. This bias focuses on the progression of visual states over time, **rather than on specific visual details**. As a result, the learned features can generalize across different embodiments. For example, although a robot gripper may look different from a human hand, both follow similar patterns of interaction when performing tasks like opening a drawer or picking up an object. This generalization ability is demonstrated in Figures 4 and 9, where AcTOL successfully **produces meaningful reward signals from real-world robot videos, even when trained on human videos**.\n* During downstream policy learning such as behavior cloning, we find only **fine-tune the model using a small number of robot demonstrations** can largely mitigate the domain gap. To prove this, we first take 25 in-domain demonstrations (5 per task) in Franka Kitchen to **fine-tune** the pre-trained encoders using the AcTOL objective. Then, as before, we freeze the fine-tuned encoders and train policy networks on top using behavior cloning. We report the comparisons when using 15 demos for policy training. As shown in the table below, the success rate improvement also demonstrate that the learned temporal inductive bias can be **effectively adapted to the robot domain with limited supervision**.\n\n| Franka Kitchen | Frozen SR| Finetune SR|\n| -------- | -------- | -------- |\n| AcTOL     |   61.8   |  **86.4**  |\n\n> **W3. Narrow language perturbation coverage.**\n\n\nWe appreciate the reviewer’s concern regarding the evaluation of language diversity. As detailed in **Appendix B.4 (line 609)**, we have included an experiment specifically designed to assess this aspect. In this setting, we provide four language variations per task, including those generated by GPT-4o. These instructions are approximately **15** words long and feature **diverse verbs, richer noun phrases, and extended contexts**. For example:\n\n> *\"Mind pushing open the right cupboard cabinet door? I need to grab the cups inside.\"*\n\nDespite the increased linguistic complexity, AcTOL experiences only a modest **3% drop** in success rate (see Appendix B.4) while still outperforming baseline methods. This experiment is intended to emulate the variability of natural household instructions, and we believe it offers a meaningful measure of robustness under diverse language conditions.\n\n\n> **Q3. Hyperparameter sensitivity analysis ($\\lambda$).**\n\nDue to space constraints, we included the sensitivity analysis of the hyperparameter $\\lambda$ in **Appendix A (line 570)**. As shown in the results, our strategy is verified agnostic to the choices of $\\lambda$ varying from ${0.01}$ to ${1}$. We chose $0.1$ to ensure a reasonable scaling between the ordering and continuity losses.\n\n> **Q4. Brownian Bridge might oversmooth critical semantic boundaries.**\n\nWhile real-world tasks often consist of multiple discrete semantic stages (e.g., “grasp → lift → place”), these stages typically unfold in **a continuous and temporally ordered manner**. As a result, visual features near stage boundaries usually exhibit gradual transitions rather than abrupt changes. As such, visual features near stage boundaries tend to exhibit gradual transitions rather than abrupt discontinuities. \n\nTo capture this structure, our approach employs two complementary components: **VLO** captures **long-term visual-semantic ordering** and helps **recognize discrete semantic boundaries for multi-stage actions**, while Brownian bridge act at a **short-term frame level**, promoting **smooth local transitions in visual representations**. This division of roles ensures that high-level semantic stages are preserved, while still allowing dense visual reward learning at a finer timescale. \n\nAs illustrated in Figure 4, AcTOL successfully distinguishes between different action phases while maintaining temporal continuity. This continuity is beneficial for producing **dense reward signals that vary smoothly with task progress**, leading to more effective learning in complex scenarios.\n\n> **Q5. Limitations of VLO loss with repetitive actions.**\n\nThanks. The datasets we use for pretraining, such as EPIC-KITCHEN, consist of clips that focus on **a single activity segment**, and therefore do not contain cyclic or repetitive actions like repeatedly opening and closing a drawer.\n\nHowever, when using noisy web videos that may include cyclic behaviors, we acknowledge that temporally distant frames can occasionally share similar semantics, potentially affecting the training performance as **we have mentioned in Limitations (line 736)**. \n\nIn practice, this issue is somewhat alleviated since **chance of sampling semantically overlapping frames as negatives is relatively low**.\n\nTo further reduce this risk, we propose a few simple but effective strategies:\n\n* **Similarity filtering**: discard candidate negatives that are too similar to the positive.\n* **Action-boundary segmentation**: segment actions before training to avoid sampling within the same repeated behavior.\n\nFor future work, we plan to explore how noisy videos with cyclic or repetitive actions can be better incorporated into AcTOL pretraining."}, {"review_id": "ig7ou0L80B", "reviewer": "Reviewer_7Q3R", "summary": "This paper introduces **Action Temporal Coherence Learning (AcTOL)**, a vision-language pre-training framework designed for embodied agents. AcTOL tackles the traditional gap in rigid goal-based constraints from traditional video pretraining.\n\n1. **Vision-Language Ordering (VLO) loss**: contrasts pairs of frames based on their temporal distance, ensuring semantic alignment respects natural ordering.\n2. **Brownian bridge continuity constraint**: models local frame intervals as a Brownian bridge, enforcing smooth transitions in the learned visual feature space.\n\nThe authors provide theoretical guarantees for both ordering and continuity, and demonstrate via imitation-learning on simulated (Franka Kitchen, MetaWorld) and real (Unitree D1) robotic tasks that AcTOL outperforms existing methods (e.g., LIV, DecisionNCE, R3M), especially in low-data regimes", "questions": "I do wonder, tho, does the algorithm assume that the video is temporally progressive? Like an action is executed from start to finish, instead o,f say, reversed or temporally inconsistent? If such a video is feeded to the algorithm, what would happen?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strength\n\n- Interesting formulation of the problem, aiming at a meaningful gap in the community\n- Rigorous theoretical analysis\n- Exhaustive empirical study\n\n## Weakness\n- No obvious weakness.\n\nI will admit that due to the overwhelming review responsibility this year, I have not had enough time to check all the math in the paper. I will continue to check them after the review deadline.", "was_revised": true, "final_justification": "I will maintain my original judgment after the authors provide answers to my questions", "rebuttal": "Thank you for your thoughtful feedback!\n> **Q1. Assumption of temporal progression in videos.**\n\n Yes, our method assumes that input videos are temporally progressive. If temporally inconsistent or reversed videos are introduced during pretraining, they could potentially degrade the model’s learning, since the visual transitions would no longer reflect meaningful action progressions. However, we believe such cases are rare in naturally collected human demonstration videos.\n \nAt evaluation time, our trained model is capable of identifying inconsistencies between the video content and the given instruction. For example, when prompted with an instruction like \"open the drawer\", if a **reversed video** (i.e., a drawer being closed) is provided, the model tends to assign **a decreasing reward**, as shown in Figure 4. If a video contains **unrelated content** (e.g., washing dishes) or **shuffled frames**, AcTOL produces **fluctuating reward curves** that reflect poor alignment with the intended task. We appreciate this suggestion and will include additional examples of such cases in the revised version of the paper to further illustrate the model's behavior."}, {"review_id": "FXGI285Qfk", "reviewer": "Reviewer_Mtcd", "summary": "This paper proposes Action Temporal Coherence Learning (AcTOL), a vision-language pretraining method for embodied agents. AcTOL addresses limitations in prior goal-reaching alignment approaches (e.g., misalignment due to irrelevant video frames) by enforcing temporal ordering via a novel Vision-Language Ordering loss and continuity via a Brownian Bridge constraint.\nTheoretical guarantees for ordering, continuity, and language robustness are provided. Experiments on real and simulated robots show significant improvements in language-conditioned imitation learning, especially with limited demonstrations. AcTOL also enables accurate language-conditioned visual rewards and exhibits strong robustness to linguistic perturbations.", "questions": "- How does AcTOL handle long-horizon tasks with multiple subgoals (e.g., \"make coffee\")? Does the Brownian Bridge scale to such sequences?\n\n- Why was the Brownian Bridge loss weight fixed to 0.1? Was sensitivity tested for different tasks?\n\n- For real-world deployment: How does AcTOL address visual domain gaps (e.g., human hands vs. robot grippers) when pretrained on human videos?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The VLO loss and Brownian Bridge constraint offer a principled solution to noisy action boundaries, avoiding rigid goal-reaching assumptions.\n\n- Provable guarantees for ordering (Theorem 1), continuity (Theorem 2), and language-perturbation robustness (Theorem 3) strengthen the methodology.\n\n-  The proposed method significantly outperforms baselines (LIV, R3M, DecisionNCE) in simulation and real-robot tasks; excels in low-data regimes; generates dense instruction-aligned visual rewards; and maintains robustness under diverse linguistic instructions.\n\nWeaknesses\n- Assumption of linear temporal progression fundamentally conflicts with cyclic/repetitive actions (e.g., dishwashing \"scrub - rinse - scrub\" cycles), limiting applicability to such tasks, with no experimental validation provided.\n\n- Spatial perception remains a critical bottleneck, evidenced by the 50% real-robot success rate for geometrically complex tasks (cup-picking vs. ~80% for drawers; Table 2), as CLIP-based frozen encoders fail to capture fine-grained affordances like handle orientation.", "was_revised": true, "final_justification": "The clarifications have satisfactorily addressed my concerns, and I have no further questions. I will raise my rating to 5.", "rebuttal": "Thank you for your thoughtful feedback! Below, we provide point-by-point responses to each of your questions.\n> **W1. Limitation with cyclic or repetitive actions.**\n\nOur method relies on the assumption that actions progress in a temporally ordered manner. **As we have mentioned in Limitations (line 736)**, repetitive, such as dishwashing (e.g., scrub, rinse, scrub), may partially violate this assumption and could pose challenges for our current formulation. \n\nHowever, most real-world tasks, particularly in household robotics, tend to follow a primarily ordered structure. Our approach is therefore well-suited for such scenarios, as demonstrated by the empirical results. We appreciate the suggestion and consider handling repetitive or cyclic behaviors an important direction for future work. \n\n> **W2. Spatial perception and fine-grained affordance issues.**\n\nWe agree that spatial perception plays a significant role in geometrically complex tasks. While CLIP-based encoders offer strong semantic priors, they may lack the fine-grained spatial precision needed for tasks like cup-picking. \n\nHowever, our work primarily focuses on improving the temporal ordering and continuity of visual-language representation, **rather than object- or pixel-level spatial understanding**. That said, enhancing spatial perception is a promising direction to complement our approach. For instance, some **feature-level super-resolution techniques** like Featup [1] could be explored to improve the spatial granularity of CLIP embeddings, combining our method to achieve temporal and spatial optimization. \n\n[1] FeatUp: A Model-Agnostic Framework for Features at Any Resolution, ICLR'24\n\n\n> **Q1. Scalability to long-horizon, multi-subgoal tasks.**\n\nAcTOL is not specifically designed for long-horizon tasks with multiple subgoals, but it can be naturally extended to such settings by leveraging a Large Language Model (**LLM**) as **a high-level planner**, like [2]. The LLM can decompose a complex instruction (e.g., \"make coffee\") into a sequence of shorter sub-tasks (e.g., \"pick up kettle\" → \"fill water\" → \"boil\" ...), each of which can then be independently handled by AcTOL.\n\nRegarding the scalability of the Brownian Bridge loss, Figure 4 demonstrates that AcTOL **maintains smooth transitions even between distinct sub-tasks**. This continuity property is expected to generalize to longer sequences of consecutively executed tasks. We will include additional visualizations in the revised version to further support this point.\n\n[2] Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents, ICML'22\n\n\n> **Q2. Hyperparameter sensitivity analysis ($\\lambda$).**\n\nDue to space constraints, we included the sensitivity analysis of the hyperparameter $\\lambda$ in **Appendix A (line 570)**. As shown in the results, our strategy is verified agnostic to the choices of $\\lambda$ varying from ${0.01}$ to ${1}$. We chose $0.1$ to ensure a reasonable scaling between the ordering and continuity losses.\n\n> **Q3. Visual domain gaps between human videos and real-world deployment.**\n\nThank you for raising this important point. There is indeed a visual domain gap between human-centric videos and third-person robot observations. AcTOL can addresses this challenge in belowing ways. \n* AcTOL learns a **general** inductive bias of temporal ordering and continuity from vision-language pretraining. This bias focuses on the progression of visual states over time, **rather than on specific visual details**. As a result, the learned features can generalize across different embodiments. For example, although a robot gripper may look different from a human hand, both follow similar patterns of interaction when performing tasks like opening a drawer or picking up an object. This generalization ability is demonstrated in Figures 4 and 9, where AcTOL successfully produces **meaningful reward signals from real-world robot videos, even when trained on human videos**.\n* During downstream policy learning such as behavior cloning, we find only **fine-tune the model using a small number of robot demonstrations** can largely mitigate the domain gap. To prove this, we first take 25 in-domain demonstrations (5 per task) in Franka Kitchen to **fine-tune** the pre-trained encoders using the AcTOL objective. Then, as before, we freeze the fine-tuned encoders and train policies on top using behavior cloning. We report the comparisons when using 15 demos for policy training. As shown in the table below, the success rate improvement also demonstrate that the learned temporal inductive bias can be **effectively adapted to the robot domain with limited supervision**.\n\n\n| Franka Kitchen | Frozen SR| Finetune SR|\n| -------- | -------- | -------- |\n| AcTOL     |   61.8   |  **86.4**  |"}, {"review_id": "uaUGE6LXV3", "reviewer": "Reviewer_7kXX", "summary": "This paper introduces Action Temporal Coherence Learning (AcTOL) to learn ordered and continuous vision-language representation. AcTOL enforces vision-language similarity differences between closer frames to be smaller than those frames that are far apart. In addition, it uses Brownian bridge to ensure consistency and smoothness between the representations of intermediate frames. Results show that the proposed method learns representation that performs well in language-conditioned behavior cloning. In addition, the learned vision-language representation can be used as a reward model and is robust to linguistic perturbations.", "questions": "See weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. AcTOL does not rely on explicit future frame selection which may be noisy when the video terminates early or include irrelevant actions. It captures the temporal semantic via ordering, emphasizing that the semantic differences between closer frames are smaller than those far apart.\n2. The paper introduces Brownian bridge to address the continuity of the representations between frames.\n3. The paper provides theoretical analysis and proofs to support the vision-language ordering and continuity.\n4. Results show that the proposed method outperforms other representation methods in Franka Kitchen, Meteworld, and a real-robot experiment.\n\nWeakness:\n1. The paper shows that through vision-language ordering and continuity, the model is able to learn representation that can serve as a visual reward for tasks. However, I am not sure why these two conditions would lead to such outcome. The supervision for the representation seems to have no constraints on the progress towards the goal specified by the instruction. It would be great to clarify this in the paper.\n2. The paper claims that AcTOL retains CLIP's capability to distinguish between actions associated with different instructions. However, the evaluation is qualitative. It would be better to perform a quantitative study on more samples to support this claim.\n3. The paper perform experiments on Franka Kitchen, Metaworld, and a real robot arm with the vision-language encoder frozen. However, the zero-shot results do not include the visualization for these experiments. It would be beneficial to include qualitative visualization and quantitative analysis on the zero-shot visual reward of AcTOL on these experiments. This would help readers better understand the mechanisms that enable AcTOL to outperforms baseline methods.\n4. The real-world experiment is relatively simple. It would be great to verify the zero-shot capability of AcTOL on more complex tasks. For example, is AcTOL able to handle pick-and-place of multiple objects? Is it able to generalize to unseen settings, including unseen backgrounds and objects?", "was_revised": true, "final_justification": "The rebuttal has addressed all my concerns. There are no other major concerns from my side. I raise my rating to 4.", "rebuttal": "Thank you for your thoughtful feedback! Below, we provide point-by-point responses to each of your questions.\n\n> W1. Unclear justification for ordering and continuity as effective supervision.\n\nThank you for raising this point. We attribute AcTOL’s ability to provide meaningful visual reward signals to two key factors: the **pretrained CLIP embeddings** and the **self-supervised** nature of our objectives. As shown in Figure 5, CLIP features already encode meaningful distinctions between different states in a visual trajectory, **especially the initial and goal states**. This is due to large-scale vision-language pretraining. However, these features are not naturally ordered or continuous throughout the entire trajectory. \n\nBy enforcing temporal ordering and continuity, our **self-supervised objectives structure the CLIP feature space** to better reflect the progression of task execution. These constraints guide the representations to unfold in an ordered and smooth manner between the already well-separated initial and goal states，**without relying on strong goal-conditioned supervision** as in prior methods. As a result, the learned representation aligns well with task progression and can serve as a dense visual reward signal for downstream control tasks. We will clarify this point more thoroughly in the revised version of the paper.\n\n> W2. Lack of quantitative support for action-instruction alignment.\n\nThank you for the helpful suggestion. To quantitatively assess whether AcTOL retains CLIP’s ability to distinguish between actions associated with different instructions, we evaluate the clustering quality of visual features on the EPIC-Kitchens dataset. Specifically, we selected videos from **50 different action classes**, each associated with a unique instruction. For every video, we extracted frame-level features using both CLIP and AcTOL, and aggregated them using temporal mean pooling to obtain a single feature per video. Then we adopt two widely used unsupervised clustering metrics: the **Davies–Bouldin score** (lower is better) and the **Calinski–Harabasz score** (higher is better). As shown below, AcTOL achieves better scores than CLIP on both metrics, indicating improved intra-class compactness and inter-class separation:\n\n|Methods|davies_bouldin_score($\\downarrow$)|calinski_harabasz_score($\\uparrow$)|\n|-|-|-|\n|AcTOL|**2.3796**|**3.4125**|\n|CLIP|2.5071|3.1580|\n\n\nThese results suggest that AcTOL **preserves the discriminative structure** of CLIP representations with respect to action-instruction alignment.\n\n> W3. Missing zero-shot visualization and analysis on main benchmarks.\n\nThank you for the helpful suggestion. Since we are currently limited to text responses during the rebuttal phase, we will include more visualizations in the revised version. To partially address this concern here, we provide a representative table below that shows the zero-shot visual reward produced by AcTOL on **our real-world robot data**. \n\nIn this experiment, we evaluate the temporal evolution of visual reward for the \"open first drawer\" task, using two instructions: one **correct** (\"Open first drawer\") and one **distractor** (\"Open second drawer\"). The **normalized reward** is measured at different timepoints (20% to 100% of the trajectory).\n\n| Instruction | 20% | 40% | 60% | 80% | 100% |\n|-|-|-|-|-|-|\n| Open first drawer (✅)| 0.21 | 0.46 | 0.71 | 0.85 | **1.0 (peak)** |\n| Open second drawer (❌)| 0.18 | 0.59 | **1.0 (peak)** | 0.67 | 0.11 |\n\nAs seen in the table, the reward for the **correct instruction** (1st row) steadily **increases as the task progresses, peaking near completion**. In contrast, the reward for **distractor instruction** (2nd row) **initially grow due to visual similarity in the early motion but diverges in later stages**, as the action clearly corresponds to a different drawer. This trend highlights AcTOL’s ability to assign fine-grained, temporally-aware rewards that distinguish between correct and incorrect tasks over time, even in a zero-shot setting without fine-tuning.\n\n> W4. Limited complexity in real-world evaluation.\n\nAcTOL is theoretically capable of handling more complex tasks; however, doing so requires **significantly more demonstrations with high quality**. As we describe in **Appendix B.2 (line 591)**, collecting demonstrations with our D1 arm is expensive because it currently relies solely on remote control. To address this limitation, we are exploring low-cost teleoperation methods to enable more scalable data collection, which would allow us to explore more diverse and complex tasks in future work. \n\nIt's also worth noting that our current experiments **require fine-grained understanding of the scence layout**. For example, in the \"open [X] drawer\" task, we use a drawer setup with a complex layout. We collect 60 demonstrations in total, 15 for each drawer where X ∈ {\"first\", \"second\", \"third\", \"fourth\"}. During evaluation, the model is **given a randomly sampled drawer instruction** and must roll out accordingly. This setup demands fine-grained semantic understanding of the target drawer, linking language (e.g., \"third drawer\") to the corresponding visual context.\n\n Additionally, we have already **added experiments under visual domain shift conditions in the Franka Kitchen environment**, please refer to **our response to Reviewer h3Cm’s W1** for details. We hope this can partially address your concern."}, {"review_id": "hMwkiEhgki", "reviewer": "Reviewer_LtsP", "summary": "This paper introduces Action Temporal Coherence Learning (AcTOL), a novel vision-language pre-training framework for generalizable embodied agents. Addressing limitations of prior time contrastive learning methods that overemphasize goal-reaching heuristics, AcTOL learns ordered and continuous vision-language representations without rigid goal-based constraints. It achieves this by simultaneously contrasting semantic differences between video frames to reflect their natural ordering and imposing a local Brownian bridge constraint to ensure smooth transitions across intermediate frames. Extensive imitation learning experiments on simulated (Meta-World, Franka Kitchen) and real (Mobile ALOHA) robots demonstrate that AcTOL's pretrained features significantly enhance downstream manipulation tasks, showing robustness to linguistic styles and offering a viable pathway to generalized embodied agents.", "questions": "Briefly discussing the potential applicability of AcTOL's learned features for other embodied AI tasks (e.g., navigation, planning) beyond manipulation would broaden its perceived impact.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n-  The paper clearly identifies and addresses a critical limitation of existing vision-language pre-training methods for embodied agents, specifically the \"overemphasis on future frames\" and its detrimental effects on learned representations. The problem of learning intrinsic temporal coherence (ordering and continuity) without rigid goal-based constraints is well-motivated.\n-  AcTOL proposes a genuinely novel approach by combining two distinct yet complementary learning objectives: semantic contrast for temporal ordering and a local Brownian bridge constraint for continuity.\n-  Experiments are extensive, spanning diverse simulated environments (Meta-World, Franka Kitchen) and a real-world robot (Mobile ALOHA), which greatly enhances the generalizability claims. The method significantly outperforms strong, recent baselines in embodied AI VLP.\n-  The paper is exceptionally well-written, clear, and easy to follow.\n\nWeaknesses\n- While the title claims \"Provable Ordering and Continuity,\" and Appendix A provides mathematical theorems, the connection between these formal proofs and the practical implications or guarantees on the learned representations in the main text could be stronger. The proofs are primarily existence/property proofs rather than guarantees of real-world \"provability\" in the context of learned features. More discussion on this link would strengthen the claim.\n- While the \"local\" Brownian bridge constraint is a clever way to handle continuity, the paper doesn't explicitly discuss how AcTOL would scale to extremely long or hierarchical video sequences, where global temporal coherence might become more challenging to model purely through local constraints. The computational cost of pairwise comparisons for L_order might also increase significantly with very long sequences.", "was_revised": true, "final_justification": "I have read the author response and found that it addresses the concerns I previously raised. There are no remaining major issues from my perspective. I will keep my original rating.", "rebuttal": "Thank you for your thoughtful feedback! Below, we provide point-by-point responses to each of your questions.\n> **W1. Weak connection between theoretical proofs and practical guarantees.**\n\nThanks for your suggestions. Our theoretical analysis aims to offer insight into how our self-supervised objectives induce ordered and continuous representations, moving beyond traditional goal-reaching assumptions. While these proofs do not provide direct guarantees on model behavior in practical deployments, **our empirical results demonstrate that the theoretical properties are indeed reflected in real-world performance**. \n\nSpecifically, **Theorems 1 and 2** establish that the learned feature space is ordered and continuous, which directly supports real-world imitation learning by enabling more **accurate and data-efficient behavior cloning** in continuous action spaces. This structure also supports generating **dense progress rewards** that can effectively identify task boundaries, benefiting real-world reinforcement learning. Additionally, **Theorem 3** guarantees robustness to linguistic perturbations, an important property for **real-world robotic applications involving natural language instructions**.  We will further clarify these connections and provide additional discussions in the revised version.\n\n> **W2. Scalability to long or hierarchical sequences.**\n\nThanks for raising this meaningful concern. The global temporal coherence in AcTOL is primarily maintained through the Vision-Language Ordering (VLO) loss applied during pretraining. This loss samples frames randomly from each video within a batch and encourages consistent temporal ordering. For very long videos, it is true that more batches may be required to cover most frame pairs. However, it is **neither necessary nor efficient to optimize over every possible frame pair**. Instead, training should be viewed as optimizing the **expected** VLO loss across all possible random batches. Once this expectation is sufficiently minimized, Markov's inequality ensures that the loss for **any** batches will also be low enough with high probability. This statistical property allows AcTOL to scale gracefully even as sequence length grows. \n\nRegarding hierarchical or multi-task videos, such sequences can be naturally decomposed into shorter sub-trajectories using **a high-level planner**, such as **LLMs/VLMs**. Each sub-trajectory can then be modeled effectively by AcTOL using the local continuity and ordering constraints. This modular design makes AcTOL compatible with hierarchical planning and scalable to longer video sequences in practice.\n\n> **Q1. Limited discussion of broader applicability.**\n\nThanks for the suggestion. While AcTOL is primarily demonstrated on manipulation tasks, its vision-language-aligned representations are **general** and could support other embodied AI domains such as navigation and planning. To apply AcTOL in these settings, the model would need to be **pretrained on video-instruction pairs that are specific to navigation behaviors or planning scenarios**. This ensures that the learned representations capture the appropriate domain-specific semantics and temporal patterns.\n\nFor **navigation**, AcTOL could be trained on egocentric video sequences paired with spatial instructions such as “go to the sink” or “walk to the hallway.”  For **planning**, AcTOL can support high-level reasoning by evaluating the semantic progress of proposed visual plans or subgoals. We will discuss such potentials in revised version and view this as a promising direction for future exploration."}]

Author rebuttal:

Meta-review: **Summarization** This paper proposes Action Temporal Coherence Learning (AcTOL), which treats videos as continuous trajectories. AcTOL contrasts semantic differences between frames to capture temporal order and applies a Brownian bridge constraint to ensure smooth transitions. Experiments on simulated and real robots show that AcTOL-pretrained features enhance manipulation tasks and remain robust to diverse linguistic instructions, paving the way toward generalized embodied agents.

**Strengths** 
1. The mathematical analysis provides deep insights into the temporal consistency properties.
2. The VLO loss and Brownian Bridge constraint offer a principled solution to noisy action boundaries
3. Very good performance, showing the promising manner for affordable pre-training for robots.

**Weakness**

The main concerns are about long-horizon tasks and the visual domain gap. But the authors have well addressed these concerns in rebuttal, such as both the proposed continuity constraint and the combination with existing task planners can mitigate the long-horizon challenges. Regarding the domain gap, the proposed method can learn a general representation, and experimental results show very good generalization with a few data finetuning.

**Justification for Decision**

The proposed method is technically novel and demonstrated to be very effective, which makes a significant contribution in the direction of video-based robot pre-training. Furthermore, the theoretical analysis provides deep insights. All the raised concerns have been well addressed by the authors. Therefore, I made the decision to accept. The authors should incorporate the rebuttal into their final version.
Decision: Accept (poster)