Imitation Learning from a Single Temporally Misaligned Video
Authors: William Huey
Huaxiaoyue Wang
Anne Wu
Yoav Artzi
Sanjiban Choudhury
Venue: ICML 2025 poster (2025)
Keywords: Learning from Videos
Inverse Reinforcement Learning
Reward Formulation
Primary area: reinforcement_learning->inverse
TL;DR: Learning sequential tasks from a temporally misaligned video requires a reward function that measures subgoal ordering and coverage.

Abstract: We examine the problem of learning sequential tasks from a single visual demonstration.
A key challenge arises when demonstrations are temporally misaligned due to variations in timing, differences in embodiment, or inconsistencies in execution. Existing approaches treat imitation as a distribution-matching problem, aligning individual frames between the agent and the demonstration. However, we show that such frame-level matching fails to enforce temporal ordering or ensure consistent progress.
Our key insight is that matching should instead be defined at the level of sequences. 
We propose that perfect matching occurs when one sequence successfully covers all the subgoals in the same order as the other sequence. 
We present ORCA (ORdered Coverage Alignment), a dense per-timestep reward function that measures the probability of the agent covering demonstration frames in the correct order. 
On temporally misaligned demonstrations, we show that agents trained with the ORCA reward achieve $4.5$x improvement ($0.11 \rightarrow 0.50$ average normalized returns) for Meta-world tasks and $6.6$x improvement ($6.55 \rightarrow 43.3$ average returns) for Humanoid-v4 tasks compared to the best frame-level matching algorithms. 
We also provide empirical analysis showing that ORCA is robust to varying levels of temporal misalignment. The project website is at https://portal-cornell.github.io/orca/

Imitation Learning from a Single Temporally Misaligned Video
William Huey" Huaxiaoyue Wang"
Anne Wu
Yoav Artzi Sanjiban Choudhury'
Abstract
learning from videos is challenging because demonstrations are often temporally bissalinned-variattionn in timing, bodiment, or execution mean that frame-level matching fails to enforce correct sequencing. To follow a demonstration, an agent must not only match states but also make consistent progress through the subgoals in the right order.
We examine the problem of learning sequential tasks from single visual demonstration. A key challenge arises when demonstrations are tem- porally misaligned due to variations in timing, differences in embodiment, or inconsistencies in execution. Existing approaches treat imitation as distribution -aaerbbttoonmaatiinnng problem, aligning individ- ual frames between the agent and the demonstra- tion. However, show that such frame-level matching fails to enforce temporal ordering or en- sure consistent progress. Our key insight is that matching should instead be defined at the level of sequences. We propose that perfect matching occurs when one sequence successfully covers all the subgoals in the same order the other sequence. We present ORCA (ORdered Coverage Alignment), a dense per-timestep reward func- tion that measures the probability of the agent covering demonstration frames in the correct or- der. On temporally misaligned demonstrations, we show that agents trained with the ORCA re- ward achieve 4.5x improvement (0.11 + 0.50 average normalized returns) for Meta-world tasks and 6.6x improvement (6.55 + 43.3 average re- turns) for Humancica 14- tasks compared to the best frame-level matching algorithms. We also provide empirical analysis showing that ORCA is robust to varying levels of temporal misalign- ment. The project website is at https:/ // wortal-cornell.githee ootroiiiooaatbbb.oocccallll
Inverse reinforcement learning (IRL) (Abbeel & Ng. 2004; Ziebart al., 2008) provides principled way to infer re- wards by matching the learner's trajectory expert demon- strations. Recent IRL approaches apply optimal transport (OT) (Peyré & Cuturi, 2020) to align visual embeddings between frames (Cohen et al., 2022; Fu et al., 2024c; Guzey et al., 2024; Haldar et al., 2023a;b; Liu et al., 2024; Tian et al., 2024). However, these methods operate at the frame level, ignoring temporal dependencies: an agent can revisit subgoals, skip ahead, stall without penalty. As re- sult, frame-matchin approaches fail on sequence-matching tasks, where the correct subgoal order is crucial for success.
Our key insight is that matching should be defined at the sequence level instead of the frame level when learning rewards for sequence- matching tasks. Under temporal mis- alignment, claim that two sequences only match if one sequence aligns with all of the other sequence's 's subgoals in the same order. This leads to an intuitive and principled re- ward function for sequence-mate -attecinn tasks: an agent should rewarded for covering all subgoals in order, without re- quiring precise timing alignment.
We propose CRCA (ORdered Coverage Alignment), which computes dense rewards given single video demonstration and visual learner trajectory, as shown in Fig. 1. Con- cretely, the reward function is recursively defined as the probability that (1) the learner currently occupies the sub- goal specified by the final vidco frame, and (2) the learner has already covered all prior frames in the correct order. In practice, we train an ORCA policy via RL in two stages, First, initialize the policy by training with rewards that assume temporally aligned demonstration. Second, we re- fine this policy with ORCA rewards to significantly improve its efficiency and performance. Our key contributions are:
1. Introduction
Designing reward functions for reinforcement learning (RL) is tedious (Eschmann, 2021), especially for tasks that require completing subgoals in a strict order. A more scalable way is to learn rewards from video demonstration. However,
Equaa contribution 'Comell University. Correspondence to: William Hucy <wph52@ comeel.edu>, Huaxiaoyue (Yuki) Wang
1. A novel, principled reward function class ORCA that formulates the reward as ordered coverage problem.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
2. Analysis on the weakness of rewards based on optimal transport and other frame-level matching algorithms.

---

Imitation Learning from Single Temporally Misaligned Video
Ø10
0yo
850
0g0
070
Qsp
Temporally
Contains
Misaligned
Pauses
Demonstration
Per- Per-Timesten Timestep ORCA Reward
enc
Trame-Level
Ordered Coverage Matrix
Distance Matrix
R(o,
Learner Trajectory
Ou
R(oy 5
020
QuB
ORCA
Reward
Oyu
aua
62
0.4
0.0
0.0
D[r. = d(enc(o)), en)(
C = Probability of covering
euc
to the j-th demo
Distance between two frame's ss embeddings
R(o 3)
frames by time
Figure ORCA overview. The expert video demonstrates the Stick-Push task, where robot must grasp water bottle. However, this demonstration is lemporally misaligned because it contains long pauses before picking up the tool. To learn from this demonstration, ORCA provides per-timestep reward the visual learner trajectory & {or fo (1) Each frame is passed through off-the-shelf visual encoder hefore calculating its distance with respect to the other frames. (2) The ORCA reward calculates the probability that the learner has covered all the frames in the correct order.
before pushing the
3. Experiments showing that the ORCA reward can effee- tively and efficiently train RL agents to achieve 4.5x im- provement (0.11 - 0.50 average normalized return) for Meta-world tasks and 6.6x improvement (6.55 - 43.3 average return) for Humanoid d --v4 tasks compared to the best frame-level matching approach.
Instead, we approach this as an inverse reinforcement learn- ing (IRL) problem, where the reward R(o {) is defined as a function of the visual demonstration & and the learner's observations. The goal to learn a policy x (a(as) that maximizes the expected discounted sum of rewards:
T Σ
π = arg max EN
2. Problem Formulation
eequence-matching problems focus on tasks where it is critical to follow the entire sequence in the correct order: e.g., for assembly tasks, robots must assemble all parts in Markov
3. Desiderata of Sequence-Matching Rewards
IRL can be formulated distribution matching problem between the learner and the demonstrator "moments". i.e., expectations of reward basis functions (Swamy et al., 2021). Hence, solving IRL is equivalent to optimizing In- tegral Probability Metrics (IPMs) between the learner and demonstrator distributions (Sun et al., 2019). Recent works use optimal transport (OT) by minimizing the Wasserstein distance (an IPM) between embeddings of the frames in the learner and demonstration trajectories. They work with Markovian moments that depend on only single frame.
the demonstrated order. We model the problem Decision Process (MDP) (S,A 7,),7. At time t, the agent at state E S receives an image observation o, O of the state. On taking action e A, it transitions to new state $++1 E S and gets a new observation 0++1 E O.
We assume the reward function for this task is unknown. Instead, we assume access single visual demonstration of the task, & {ō1,.... ²r). The robot policy must follow all subgoals in the same order as the demonstration. Im- portantly, this demonstration may be temporally misaligned with the learner's trajectory and have a different execution speed. Additionally, the demonstration lacks state and action labels, rendering classical imitation learning inapplicable.
However, the true reward function for sequence-matching task depends on trajectory history to determine whether the agent has followed the subgoals order, so is not the span of Markovian moments. Instead of matching the distri-
reward,function
2

---

Imitation Learning from Single Temporally Misaligned Video
OT fails to enforce temporal ordering.
metric d(:--) that measures the distance between the em- beddings of learner and demonstration frame. Given {or]i= and a video demon- stration = { they define corresponding learner and demonstration distributions that are uniform in time, Specifically, p 1 Σi= S0,, where 6o, is Dirac distri- bution centered on ot- The demonstration similarly has a distribution of أ = 7 プー= ōā,- Consequently, the matching matrices are evenly weighted, defined as the set M = {μ E RT** #1l T fh 71. With an en- tropy regularizer H(-), OT solves for the optimal matching matrix:
Demonstration Traj
learner trajectory
Optimal Learner Traj
Suboptimal Learner Traj
0
05
0g
0;
8
A
0
PT
OT optimal matching matrix
matching matrix
QT optimal
1:00
0.00
0.00
1.00
T
T
0:50
0.50
H¹ = arg min NEM
d(a,, oz)ررة - -H(())
(1)
0.00
1.00
j=1
Compuis Ranard
Compute Romard
The OT reward is:
Cumulative OT Reward
Σ Ror
Cumulative Reward
Σ
Rorter A-
f Σ
Ror(or:E.))
d(or
1.00
1.00
The suboptimal learner trajectory 5- has equally high cumulative OT reward as the optimal learner trajectory
Although this is non-Markovian reward function, it still matches Markovian moments. This is clear in its failure to enforce subgoal ordering.
Figure Failure cases for OT reward. The suboptimal learner trajectory 5 moves in the counter-clockwise direction while opti- mal one 5r moves clockwise. Unlike learner, the agent the demonstration trajectory move multiple cells per timestep.
Counterexample 3.1. There exists an MDP (Fig. 2) where OT fails to penalize violations of temporal ordering.
The OT reward uses a Markovian distance based on two frame embeddings, and there are no temporal constraints on μ in (1). Thus, OT does not penalize trajectories that com- plete subgoals in the wrong order. Fig. 2 shows example where a suboptimal trajectory that visits the subgoals in the reversed order has the same OT reward as the optimal one.
bution over frames, we should be matching the distribution over trajectories. We propose that the corresponding reward should be measure of:
1. Subgoal Ordering:
A learner trajectory that completes the subgoals in the correct order should receive higher cumulative reward than one that completes the sub- goals in the wrong order.
3.2. Dynamic Time Warping Fails to Cover All Subgoals
Dynamic Time Warping (DTW) (Sakoe & Chiba, 1978) overcomes the subgoal ordering problem of OT by tem- porally aligning the learner and demonstration trajectories, while allowing for flexible distribution of the assignment over each demonstration frame. Specifically, each matching matrix μ must align the beginning and the end of two tra- jectories (i.e., |411 and MT, 1), and the indices of its positive entries must be non-decreasing in time. Subject to these constraints, DTW solves for the optimal matching: n* arg min, Σ E do,, o,).
2. Subgoal Coverage: A learner trajectory that completes more of the subgoals should receive a higher cumula- tive reward than one that completes less of them.
Rewards that use frame-level matching fail to satisfy these desiderata: OT fails to enforce subgoal covering (Sec. 3.1), and mitigations to OT's problem fail to enforce full subgoal coverage (Sec. 3.2 and 3.3).
3.1. Optimal Transport Fails to Respect Ordering
Given μ², the DTW reward is the same (2). DTW's temporal constraints allow it to avoid the example failure of OT (shown in Fig. 7 of Appendix A.2), but they lead to new failure mode.
Prior works in imitation learning use Optimal Transport (OT) (Peyré & Cuturi, 2020), specifically the Wasser- stein distance, to measure how closely a learner trajec- tory matches a demonstration trajectory (Fu et al., 2024c; Kedia et al., 2025; Papagiannis & Li, 2022; Tian et al., 2024). These approaches assume a Markovian distance
Counterexample 3.2. There exists an MDP (Fig. 8, App. A.2) where DTW fails to penalize incomplete subgoal coverage.
3

---

Imitation Learning from a Single Temporally Misaligned Video
Although DTW constrains the order of assignment, it does not limit the number of learner frames matched with each subgoal. Once the learner reaches an intermediate subgoal, it can achieve high DTW reward by remaining in that sub- goal until the last few frames of the trajectory. Fig. 8 of Appendix A.2 shows an example where a trajectory stuck in the second subgoal achieves the same DTW reward as a tra- jectory that completes all subgoals. This is local minimum that could cause RL agents to stall at earlier subgoals.
tasks. In Sec. 4.2, we theoretically analyze and prove that ORCA satisfies the desiderata proposed in Sec. 3.
4.1. ORCA: Ordered Coverage Reward Function
We define the ordered coverage for two sequences as the probability that one sequence covers all the subgoals in the same order as the other sequence. Ordered coverage shares the same recursive nature as the true reward function of sequence-matching tasks: it depends on how well the final subgoal is covered and how well previous subgoals have already been covered. We next present how to calculate ordered coverage and use it to compute rewards.
3.3. TemporalOT Fails Cover All Subgoals under Temporal Misalignment
Recognizing the limitations of OT, et al. (2024c) propose TemporalOT, which alters the OT objective to optimize over a temporally masked cost matrix. Specifically, the mask is a variant of the diagonal matrix with hyperparameter kw to define the width of the diagonal:
Calculate ordered coverage via dynamic programming. Given video demonstration and learner trajectory, we can calculate ordered coverage between the two sequences by computing the matrix Ct.j. which is the probability that the segment of the learner trajectory from time 1 to has covered the first to the j-th subgoals in the correct order. By the recursive definition of ordered coverage. Ctj can be expressed with two components: (1) the ordered coverage until the (j- - 1)-th subgoal and (2) how well the learner is currently occupying the j-th subgoal. In addition, ordered coverage should be nondecreasing over time: i.c., once the learner has covered a subgoal at a timestep. is always at least equally successful future timesteps. Let Gt,, denote the event that the learner occupies the j-th demonstration subgoal at timestep t. We assume that the probability for this event is proportional to the negative exponent distance: P(Gr,) α exp(-Xd(or,0;)) oz)), where À is a temperature hyperparameter. For simplicity, we substitute P(Gr.) with Pt.. Thus, we recursively calculate ordered coverage:
Wis = { jf (t kw t kwl,
(3)
otherwise.
Consequently, TemporalOT solves for the optimal matching matrix n² E .M with the same constraints as (I):
f Σ
p arg min MEM
We,d(or, - cH(W @
(4)
Given vt," the TemporalOT reward is the same as (2).
Counterexample 3.3. There exists an MDP (Fig. 9, App. A.2) where TemporalOT fails to penalize trajectories that do not reach every subgoal given a temporally mis- aligned demonstration.
Cr. = max(Cr-1,55 Ptj.
(5)
To define the mask window, TemporalOT makes the strong assumption that the demonstration is temporally aligned with the learner trajectory. Regardless of how temporally misaligned a demonstration is, TemporalOT's matching ma- trix always approximates diagonal matrix, matching an equal proportion of learner frames to each subgoal. Fig. 9 of Appendix A.2 shows an example where suboptimal, slower policy that does not reach later subgoals has higher TemporalOT reward than an optimal policy that makes con- sistent progress to complete the task.
Algorithm 1 shows the entire computation.
ORCA reward function. In most robotics tasks, the learner should stay in the final demonstration subgoal after covering all previous subgoals. A reward function that directly uses the ordered coverage for the final subgoal does not satisfy this requirement: even if the learner does not remain occupy- ing the final subgoal, as long as it has covered the subgoal at one timestep, the maximum operator in (5) keeps the reward high for remaining timesteps. Instead of simply equating the reward to C Cr the final ORCA reward at timestep t is:
4. Approach
RORCA (. = C C.T--IP.
(6)
We introduce ORCA (ORdered Coverage Alignment), re- ward function that measures, at each time step, the prob- ability that (1) the learner currently occupies the subgoal specified by the final video frame, and (2) it has already cov- ered all prior frames in the correct order. Since the ORCA reward at time t depends on the learner trajectory up until t, models the non- Man-Maroviinn nature of requence-matching
Runtime complexity. ORCA's runtime complexity is O(T- T). This is the lower bound time complexity of any method that relies on a frame-level distance matrix, including all OT-based methods. To validate this, we experimentally tested the latency of ORCA against a variety of common baselines, finding that ORCA is faster than TemporalOT and comparable to OT. See Appendix C.6 for details.
4

---

Imitation Learning from a Single Temporally Misaligned Video
Algorithm 1 ORCA Rewards.
time L. A formal proof is given in Appendix A.I. This overcomes the failure modes of DTW and TemporalOT in Sec. 3.2 and 3.3. Appendix A.2 visualizes how the CRCA reward avoids the example failures of these frame-level matching algorithms.
Input: learner traj 5 = (0.)11 demo traj {
distance function -), temperature term
// Cale probability matrix
Pt, = exp(-Ad(o o,)) for [[T]], [[]] // coverage at learner time t=1
= -11
4.3. Pretraining
CL Ci.j-IfI1 for j E 32.
We observe that in practice, ORCA can have multiple local minima. This is because the agent is trying to achieve high coverage for many subgoals. has a trade-off between covering all subgoals equally well, which is often slower, or quickly achieving high coverage for most subgoals while a small portion of them get lower (but still nonzero) coverage. Some of these minima are undesirable, e.g. if the small portion of subgoals that the agent only partially covers are vital to the task and require the agents to match them more perfectly, the agent is more likely to fail.
// coverage of first demo frame =1
C₁, = max } for / E 32, Solve recurrence relation (5)
for 1 = to 7 do
for to T do
Cr, max (C Ctj-i Pt.s)
end for end for
// Compute the final ORCA rewards (6)
Return Rewards RORCA = (C E [[7]}
initialize the agent a better basin, we first bias the agent towards spending an equal amount of time attempting to cover each subgoal, and then train with the ORCA reward. Specifically, we pretrain the agent on reward function that assumes the video demonstration is temporally aligned (e.g..
4.2. Analysis
The ORCA reward satisfies the two desiderata for sequence- matching reward function (Sec. 3), overcoming the failure modes of frame-level matching algorithms. In the following analysis, define subgoal oj to be occupied if Pt. Re 1. Proposition 4.1 (ORCA enforces subgoal ordering). Let S- be a trajectory that is out of order: specifically, there exists a subgoal ōj such that öj is occupied at time but 0j-1 is not yet occupied. Let g+ be a trajectory that is identical to E, exxcept that it occupies 0j-1 before time Then, RORCA(Ot (tt,,+ > RoRcalO, ..)
T Temporalo1 rewards). In Section 5, we empirically show the importance of pretraining.
5. Experiments
5.1. Experimental Setup
Environments. We evaluate approach across two envi- romments (details in Appendix B):
Meta-World (Yu et al., 2020). Following Fu et al. (2024c), we use ten tasks from the Meta-world environ- ment to evaluate the effectiveness of ORCA reward the robotic manipulation domain. On each rollout, the loca- tions of task-relevant objects are randomized according to the default setup of Metaworld. The training and evalu- ation seeds are also different. We classify the tasks into three difficulty levels based on the number of types of mo- tions required and whether the task requires precision. Vi- sual demonstrations are generated using hand hand-engineered policies provided by the environment.
By (5), at a timestep, the ordered coverage of the current subgoal can be no greater than the coverage of the previous subgoal. Since &+ occupies 0 before and & does not, E+ achieves greater coverage of the subgoal 0j-1 than &". The trajectories are otherwise equivalent, so 5 must achieve a higher ORCA reward at time f. A formal proof is given in Appendix A.I. This overcomes OT's failure mode (Sec. 3.1).
Proposition 4.2 (ORCA enforces subgoal coverage). Let E- be trajectory that occupies 0y-1 at time - and continues to occupy 0j-1 at time i, instead of progressing towards oj. Let E+ be an identical trajectory that pro- gresses towards 0j al l, and assume that neither trajectory has been closer to oj before. Then, Roncalo ,otccatottttt _.+,) Ro
Humanoid. We define four tasks in the MuJoCo Humanoid-ao v environment (Todorov et al., 2012) to examine how well ORCA works with precise motion. Be- cause there is no predefined expert, we obtain visual demonstrations by rendering an interpolation between the initial and goal joint state.
Both trajectories achieve the same coverage of subgoals up to 0 Since ++ moves closer to the next subgoal ö, than g- it achieves a higher probability of occupying o, and gets higher coverage of oj. The trajectories are otherwise equivalent, so g+ must achieve a higher ORCA reward at
RL Policy, For Meta-world, we follow the RL setup in Fu et al. (2024c). We train DrQ-v2 (Yarats al., 2021) with state-based input for IM steps and evaluate the policy every 10k steps on 10 randomly seeded environments. For
5

---

Imitation Learning from a Single Temporally Misaligned Video
Table Meta-world results on temporally misaligned demonstrations. We report the mean expert- -ormal-nore returns with standard error, and we highlight the cop-perfrming approaches. Multiple are included if they are within the standard error of the top score, Agents trained with ORCA consistently outperform other frame-level matching approaches. RoboCLIP omitted because fails for all tasks.
Category
Environment
Threshold
DTW
OT
TemporalOT
ORCA (NP)
ORCA
Button-press
0.30 (0.10)
0.00 (0,00)
0.00 (0.00)
0.10 (0.02)
0.45 (0.11)
0.62 (0.11)
Easy
Door-close
0.34 (0.07)
0.00 (0.00)
0.00 (0.00)
0.19 (0.01)
0.86 (0.01)
0.88 (0.01)
Door-open
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.08 (0.01)
1.60 (0.09)
0.89 (0.13)
Window-open
0.72 (0.14)
0.00 (0.00)
0.19 (0.06)
(0.05)
0.86 (0.17)
0.85 (0.16)
Medium
Lever-pull
0.07 (0.02)
0.00 (0.00)
0.00 (0.00)
0.07 (0.03)
0.27 (0.08)
0.28 (0.09)
Hand-insert
0.00 (0.00)
0.00 (0.00)
0.03 (0.02)
(0.00)
0.08 (0.08)
0.04 (0.04)
Push
0.07 (0.05)
0.00 (0.00)
0.03 (0.01)
0.01 (0.01)
0.02 (0.02)
0.00 (0.00)
Basketball
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.01 (0.01)
0.07 (0.03)
0.01 (0.00)
Hard
Stick-push
0.12 (0.04)
0.00 (0.00)
0.07 (0.02)
0.36 (0.00)
0.46 (0.13)
1.25 (0.04)
Door-lock
0.00 (0.00)
0.05 (0.02)
0.04 (0.02)
0.00 (0.00)
0.23 (0.09)
0.19 (0.08)
Average
0.16 (0.02) 0.01 (0.00) 0.04 (0.01)
0.11 (0.01)
0.49 (0.04) 0.50 (0.04)
baseline on a subset of the tasks in Appendix C.3.
Table Humanoid results temporally misaligned demon- strations. Results are presented as the mean returns with standard error. TemporalOT is abbreviated to TOT. For results of all base-
Metrics. We evaluate the final checkpoints of all approaches on cumulative binary rewards, or returns, so a policy that succeeds quickly and remains successful is better. In Meta- world, we use the ground-truth sparse rewards and report the normalized return, which is the return as fraction of the expert's 's return on the same task, given the same number of timesteps. We define a success metric for Humanoid, using privileged states, which no approaches have access to. An agent is successful if it can remain standing (torso height above 1.1) and its arm joint position is close to the goal joint position (Fuclidean distance less than 1). There is no expert in for Humanoid tasks, so we report unnormalized returns.
lines, see Table 4 in Appendix C.
Task
TOT
ORCA (NP)
ORCA
Arm up (L)
5.29 (2.22)
65.9 (8.25)
81.6 (3.65)
Arm up (R)
7.67 (2.88)
92.5 (4.71)
49.6 (5.00)
Arms out
1.62 (0.75)
72.7 (10.1)
8.50 (2.60)
Arms down
11.6 (3.56)
19.7 (5.03)
33.4 (7.20)
Average
6.55 (2.35)
62.9 (7.02)
43.3 (4.61)
the Humanoid environment, we train SAC (Haarnoja al., 2018) for 2M steps and evaluate the policy every 20k steps on 8 environments. All policies use state-based input, and in metaworld we include an additional feature that represents the percentage of total timesteps passed. Appendix B.4 contains RL training details and hyperparameters.
5.2. How well does ORCA perform given temporally misaligned demonstrations?
Metaworld. For each original demonstration, we subsample it such that the first one-fifth retains the original execution speed, but the rest is sped up by five to ten times. In Ta- ble 4.3, the ORCA reward significantly outperforms other baselines and trains agents to acquire the highest average normalized return of 0.50. The Push task is difficult for all approaches and the only one where ORCA or ORCA, (NP) does not achieve top performance. As analyzed by Fu et al. (2024d), the distance matrix is generally noisier for the Push task due to the small size of the target object, which benefits sparser reward function like Threshold. d. Mcanwhile, rewards based on frame-level matching algorithms perform poorly on all tasks, revealing their weakness when encoun- tering temporal misalignment.
Baselines. We compare ORCA against baselines that use frame-level matching algorithms: OT (Tian al., 2024), Temporal0T (Fu et al., 2024c), and DTW (Sakoe & Chiba, 1978). We also compare ORCA, which is pretrained on Tem- poralOT rewards for half of the total timesteps and ORCA rewards for the remaining timesteps, against ORCA (NP) ) which fully trains on ORCA rewards without any initial- ization. All approaches use the pretrained ResNet50 (He et al., 2016) to extract visual features and cosine similar- ity as the distance function. We include a simple baseline Threshole ld, which tracks the subgoals completed based on a threshold distance, and a transformer- based approach RoboCLIP (Sontakke et al., 2024), which directly encodes an entire video. Details are in Appendix B.5. We also inves- tigate a language-conditiones baseline and a traditional IRI.
Humanoid. Because there does not exist an expert policy for the Humanoid tasks, we generate demonstrations by interpolating ten frames between the initial and final joint
6

---

Imitation Learning from Single Temporally Misaligned Video
Video Demo
dip
TemporalOT Trajectory
TemporalOT
Demo
Trajectory
Moved
Distance Matrix
Matching
Quickly
Matrix
TemperalOO (TOT)
TOT Fails Because Too Slow.
ORCA Opens the Door Quickly.
ORCA
TemporalOT's approximately diagonal matching matrix causes it to spend too long reaching for the handle.
Figure 3. Qualitative example TemporaloT failing to encourage full subgoal coverage. The vidco demonstration shows how to open door latching the door handle, but it speeds through the movement after latching. Temporaloy trains slow agent that fails to complete the task due to its diagonal-like matching matrix, but ORCA trains a successful agent that completes the task efficiently.
positions. These demonstrations are naturally temporally misaligned because the environment is unstable, making it impossible to follow the subgoals at the same speed. Ta- ble shows that agents trained with ORCA (NP) achieve the highest average cumulative return of 62.9. Due to poor Temporalon performance, ORCA does not benefit from pretraining, and ORCA (NP) thus performs better because it is trained on the ordered coverage reward for more steps. We further investigate the failure of Temporalo7 in Sec. 5.4. Meanwhile, Fig. 15 in Appendix C shows how the ORCA reward satisfies Prop. 4.1 and 4.2, covering the subgoals as quickly as possible and successfully completing the task.
Temporalon OO's poor performance, we observe that when ORCA agents fail, they successfully follow the general mo- tions of the demonstration, but they miss details (e.g., align- ing the gripper with the target object). We hypothesize that this behavior is caused by the frame-level distance metric, which pays more attention the general robot arm motions than details, allowing most subgoals to achieve relatively good coverage.
5.3. How important is enforcing subgoal ordering?
Fig. 17 in Appendix C.1 shows a key failure point for OT: its matching matrix can match later subgoals to earlier learner frames and vice versa. The optimal matching matrix min- imizes the transport cost regardless of the order in which subgoals are completed, thereby giving higher OT rewards to trajectories that violate the temporal ordering. In both Meta- world and Humanoid environments, the OT rewards create difficult optimization landscape that causes the agents to learn undesirable behaviors. Fig. 17 also demonstrates that T Temporalor can violate temporal ordering depending on the demonstration length and the mask window size. Tuning this value is major drawback of Tempor roT because requires prior knowledge of the temporal alignment.
Varying Misalignment Level. We identify two types of temporal misalignment: either slower demonstration that contains pauses, or a faster demonstration that accelerates through segment. For each misalignment type, we ran- domly perturb the original demonstrations of three Meta- world tasks (Door-open, Window-open, Lever-pull), where the misalignment level controls how varied and nonlinear speed changes are. See appendix B.6 for details.
In Fig. 4, ORCA consistently maintains higher return com- pared to Temporal0T as the demonstrations become more misaligned. Meanwhile, Tempora OT's performance sig- nificantly deteriorates when there is any level of misalign- ment. When the demonstrations are sped up, because Temporalon aloo encourages agents spend an equal amount of time each subgoal (in-depth discussion in Sec 5.4), Tempora10s agents often cannot finish the task in time. This problem is further exacerbated when the demonstra- tions are slowed down and longer than the learner trajec- tory. ORCA's 's performance also worsens more given slower demonstrations compared to faster ones.
5.4. How important enforcing full subgoal coverage?
Fig. shows that ORCA successfully trains an efficient agent, while the Tempora agent opens the door much more slowly and fails. The oT/Temporalor formulation as- sumes that the learner and demonstration distributions uniform in time. However, given temporally misaligned demonstration, different portions of learner frames should be matched to different subgoals, which is impossible us- ing this formulation. Empirically, successful trajectories
In addition to being affected by poor initialization due

---

Imitation Learning from a Single Temporally Misaligned Video
TOT
ORCA
0.85
0.82
average normalized return of 0.57 compared ORCA (NP) (0.33) on aligned demonstrations.
0.78
Normalize Normalized lized Return
ORCA (NP) fails due to undesirable local minima. Consider the Stick-Push task in Fig. 19 of Appendix C.I, where the robot arm needs to grasp the stick before pushing the water bottle with that stick. The ORCA (NP) policy directly moves to push the water bottle without the stick. Because the robot arm initially seems close to the stick, the ORCA (NP) policy could get partial coverage earlier subgoals while collect- ing high coverage on later subgoals for pushing the bot- tle. Meanwhile, ORCA is initialized with the Tempora lOT policy that fails the task but spends equal time attempting each subgoal. ORCA is able to refine the policy, finding the middle ground between ORCA, (NP) and Tempo ra 1 10aand quickly grasping the stick before pushing the water bottle with it. Overall, pretraining initializes ORCA in better basin, allowing it to train successful and efficient policies.
0.39
0.31
0.25
I
0.10
0.08
Slow (H)
Slow (L)
Aligned
Fast (L.)
Fast (H)
Figure 4. Results given varying levels of temporal misalignment. We report the mean expert-normalized returns with standard er- ror across Meta-world tasks (Door open, Window open, Lever pull). We generate 3 perturbed demonstrations per task per mis- alignment level (L=Low, H=High), training on each demonstration separately.
0.7
0.6
Normalized Returns
5.6. Does ORCA scale with more demonstrations?
0.5
0.43
Following the strategy of Fu et al. (2024c), ORCA can be adapted to multiple demonstration videos _ (E.....,")) by calculating the ORCA reward at each timestep with respect to every video and max-pooling:
0.4
0.33
0.3
I
0.21
T
0.1
0.05
0.0
RCLIP
DTW
(NP) ORCA
(or,,A max RORCA
SEE
Figure 5. Results given temporally aligned demonstrations. We report mean expert-normalized returns with standard error across all 10 Meta-world tasks.
To study how the number of demonstrations affects perfor- mance, we first investigate the setting where they all have the same speed, and the demonstrations only vary due to randomly initialized object locations. Figure shows how ORCA's performance improves the number of demon- strations increases. In contrast, although Temporalor handles multiple videos using the same method, and it ini- tially benefits from having more than one demonstration, its performance starts to degrade given a large number of demonstrations.
produce coupling matrices that approximate the diagonal matrix, as shown in Fig. 3. The subsequent rewards teach the agent to spend an equal amount of time at each subgoal instead of following the demonstration fast as possible, which can cause the agent to exhaust the timesteps before completing the task. Temporal0T also exhibits poor per- formance for Humanoid tasks. We hypothesize that the rewards force the agent
agent fails because Temporales to spend an equal amount of time in each intermediate sub- goal, which is difficult in a highly unstable environment. We show DTW's failure case in Fig. 18 of Appendix C.I.
We additionally train policies using demonstrations with different speeds by randomly sampling one video from each of the four temporal misalignment categories de- scribed in Figure 4. ORCA outperforms Temporalon in this scenario, as shown in figure 6. ORCA policies ad- ditionally perform better when trained on 4 demonstra- tions with different speeds than just a single demonstra- tion (0.67 + 0.08 + 0.77 + 0.08). although they are still worse than 4 same speed demonstrations. Mcanwhile, Tempora10T shows minimal gains with different speed demonstrations (0. 0..02 - 0.16 0.02), demonstrating its inability to handle temporal misalignment at scale. Over- all, ORCA consistently benefits from training with multiple demonstrations and exhibits greater robustness compared to baselines when these demonstrations have different speeds.
5.5. How does pretraining affect ORCA's performance?
Pretraining leads to better ORCA performance when the pretraining strategy is able obtain some success. In Ta- ble. 4.3, ORCA equal or better than ORCA (NP) on tem- porally misaligned demonstrations. In Humanoid tasks, ORCA (NP) achieves higher overall performance because Temporal0 almost entirely fails on every task. The ef- fect of pretraining is most apparent when the demonstrations are temporally aligned because this setting satisfies the core assumption of Temporall 1OT, I Fig. 5, CRCA achieves an
8

---

Imitation Learning from a Single Temporally Misaligned Video
reward model that takes videos input and outputs scalar rewards for RL (Sontakke et al., 2024; Yang al., 2024), but these models often require fine-tuning on task-specific data to improve performance (Fu et al., 2024b). We follow IRL's formulation where we aim to match the learner and demonstration distributions, which is equivalent to optimiz- ing the Integral Probability Metrics (IPMs) (Sun et al., 2019; Swamy al., 2021).
1.2
TOT ORCA
1.08
1-20
1.02
Normalized Returns
1.0
0.77
0.8
0.67
0.6-
0.36
0.4
0.30
0.25
0.2-
0.14
0.16
0.0
(Same)
(Dir))
4 !Samee)
10 (Same)
20 (Seme)
Optimal Transport Used In IRL. Recent works leverage OT (Peyré & Cuturi, 2020) to optimize IPMs. In the imi- tation learning setting, where a telcoperated demonstration dataset contains state-action labels, prior works can directly minimize the Wasserstein distance between the learner and the demonstration's 's state-action distributions (Bobrin et al., 2024; Dadashi et al., 2021; Luo et al., 2023; Papagiannis & Li, 2022; Xiao et al., 2019). These approaches do not work given only a single visual demonstration. Without ac- cess to privileged states, recent works instead use distance function that measures the transport cost between two visual embeddings (Cohen et al., 2022: Fu et al., 2024c; Guzey et al., 2024; Haldar et al., 2023a;b; Kedia et al., 2025; Liu et al., 2024; Tian et al., 2024). Both types of approaches assume that Markovian function measuring transport cost between two timesteps is sufficient. However, our work tack- les sequence- mathhing problems, where agents must follow the subgoals from a temporally misaligned demonstration in the correct order. OT with a Markovian distance function fails because the true reward function depends on the entire trajectory, a limitation that our approach addresses.
Number Demonstratoos (Spaad)
Figure 6. Results given more demonstrations. Same indicates that all demonstrations in the set are the same speed, whereas Diff indicates that they are different speeds. We report mean expert- normalized returns with standard error across Meta-world tasks (Door open, Window open, Lever pull).
5.7. How does ORCA perform given a policy conditioned on image observations?
ORCA'S goal is estimate rewards from a single demon- stration in the observation space before learning a policy, making no assumptions about the policy itself. We addi- tionally evaluate ORCA when the policy is conditioned on image observations instead of ground-truth states, follow- ing the DrQv2 setup of (Fu et al., 2024c). The results in Appendix C.5 demonstrate that the state- and image- based policies trained with ORCA have similar performance (average normalized return: 0. 7.7042 + 0.10 + 0.76 + 0.06). In contrast, policies trained with Tempora10T perform poorly, regardless of their input.
7. Discussion
5.8. Visual Encoder Ablation
We investigate sequence- mateehing tasks, where the learner must follow a expert video demonstration that may be tem- porally misaligned. We analyze how algorithms that match the learner and expert distribution at the frame level (e.g.. optimal transport) result in reward functions that fail in this setting. Following our key insight that matching should be defined at the sequence level, we present ORCA: a prin- cipled reward function that computes the probability that the learner has covered every subgoal in the correct order. Experiments on Meta-world and Humanoid tasks show that ORCA rewards train agents that complete the tasks efficiently regardless of the level of temporal misalignment.
ORCA works with any visual encoder that can act as a dis- tance metric between frames. We include in Appendix C.4
an additional ablation of ORCA with LIV (Ma et al., 2023), robotics-specific visual encoder, and DINOv2 (Oquab et al., 2023), a standard vision model. The Resnet50 used for the main result in this paper achieves the best performance. However, there high variability, substantiating the find- ings of prior work that evaluates different visual encoders for RL training (Hu et al., 2023). Ultimately, the best visual encoder is a practical and task-dependent choice.
6. Related Works
We recognize few limitations of ORCA: (1) it relies on good visual distance metric that measures the similarity between two frames. Future work will explore using online finetuning to improve the encoder (Fu et al., 2024b) and solve embodimeb tasks where temporal misalignment is common. (2) video-following may result unexpected failures due to task misspecification. We plan explore how ORCA can be applied to language subgoals in addition to video frames.
Learning From Few Video Demonstrations. Prior works in imitation learning (Duan et al., 2017; Fu et al., 2024a; Jain et al., 2024; Palo & Johns, 2024; Xu et al., 2022) train a policy that, at inference time, takes video-action demon- stration and the current robot state to output actions. This is different from our policy formulation because ORCA and its baselines do not have access to demonstrations with action labels. Closer to our setting, some works focus on training
9

---

Imitation Learning from a Single Temporally Misaligned Video
Acknowledgments
Foundation, F. Humanoid-v4, 2024. URL https:
/ //gymnasiume farama www.aaa..rggg//vee wtw.//iiooooeooeettte.eo.oo/// mujoco/ nuajoco/humanoid// humanoid/.
This work was supported in part by the National Science Foundation FRR (#2327973).
Frey, J., Mattamala, M., Chebrolu, N., Cadena, C., Fallon, M., and Hutter, M. Fast Traversability Estimation for Wild Visual Navigation. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi: 10. 2.15607/RSS.2233X.00
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fu, L., Huang. H., Datta, G., Chen, L. Y., Panitch, W. C.-HH. Liu, E., Li, H., and Goldberg, K. In-context imitation learning via next-token prediction, 2024a. URL https: / //arxiv. //aaaxivorg//ablal...ea htt///b//2201.59/8/0 15980.
References
Abbeel, P. and Ng, A. Apprenticeship learning via inverse reinforcement learning. Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Fu, Y. Zhang, H., Wu, D., Xu, W., and Boulet, B. Furl: visual-language models fuzzy rewards for reinforce- ment learning. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JML.R. JMLR.org. 2024b.
Andrews, J., Morton, E.,, and Griffin, L. Detecting anoma- lous data using auto-encoders. International Journal of Machine Learning and Computing, 6:21, 01 2016.
Fu, Y., Zhang, H., Wu, D., Xu, W., and Boulet, Robot policy learning with temporal optimal transport reward.
Bobrin, M., Buzun, N., Krylov, D., and Dylov, D. V. Align your intents: Offline imitation learning via optimal trans- port, 2024. URL https:/: //tp:i/.///ii////a http:///rr/aab//00/000 13037.
In The Thirty-eighth Annual Conference Neural In- formation Processing Systems, 2024c. URL https: / //openrevi ew net/ / forum? ded.eeee55S5oo
Cohen, S., Amos, B., Deisenroth, M. P., Henaff, M., Vinit- sky, E., and Yarats, D. Imitation learning from pixel observations for continuous control, 2022. URL https: // //openreview net. forum? htt.mmmmdddd.......ccccccco
Fu, Y., Zhang, H., Wu, D., Xu, W., and Boulet, B. Rebuttal: Robot policy learning with temporal optimal transport reward, 2024d. URL https: /tpptttp::///pp//////tt forum? htt.://eeeeeeeeeeeaaoooeeoraaaea
net/
Dadashi, R., Hussenot, L., Geist, M., and Pietquin, O. Primal wasserstein imitation learning. In In- ternational Conference on Learning Representations, wwwperrvieww..ee/oo.....eeeeeee net//frunrt
Guzey, I., Dai, Y., Evans, B., Chintala, S., and Pinto, L. See to touch: Learning tactile dexterity through visual incen- tives. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 11825-13832. IEEE, 2024.
2021. URL
id-TtXSU292gR.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei,
Haarnoja, Zhou, A., Abbeel, P., and Levine, Soft actor- critic: Off- Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Dy. J. and Krause, A. (eds.). Proceedings of the 35th International Confer- ence on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URIp /ttpttt::://rreeeeeeee /pwceedd ngs .mlr. press/v80 htess//80/haarroja1110 html..
L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Duan, Y., Andrychowicz, M., Stadie, B., Jonathan Ho, O.,
Schneider, J., Sutskever, I., Abbeel, P., and Zaremba, W. One-shot imitation learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan. S., and Garnett, R. (eds.), Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:/ neurips......................... ec cc/ hee/pape__fte [haper__fileesppppr///////// ba336600:550551c4e5575521111 pdf.
Haldar, S., Mathur, V., Yarats, D., and Pinto, L. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning, pp. 32-43. PMLR, 2023a.
Haldar, S., Pari, J., Rai, A., and Pinto, L. Teach a robot to fish: Versatile imitation from one minute of demonstra- tions. In Proceedings of Robotics: Science and Systems, 2023b.
Eschmann, J. Reward function design in reinforcement learning. Reinforcement Learning Algorithms: Analysis and Applications, pp. 25-33, 2021.
10

---

Imitation Learning from a Single Temporally Misaligned Video
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10. 10.1109/CVPR.2016.90
Peyré, G. and Cuturi, M. Computational optimal trans- port, 2020. URL https:/: http:p:://axxivoooo org. haagg//bs/lae 00567.
Rocamonde, J., Montesinos, V., Nava, E., Perez, E., and Lindner, D. Vision-language models are zero-shot re- ward models for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: http//pppppeew.eeee/ee net forum? id=NOI2RED8je.
Hu, Y., Wang, R., Li, L. E., and Gao, Y. For pre-trained vision models in motor control, not all policy learning methods are created equal. In Proceedings of the 40th In- ternational Conference on Machine Learning, ICML'23. Jrg, 2023.
Jain, V., Attarian, M., Joshi, N. J., Wahid, A., Driess, D., Vuong, Q. Sanketi, P. R., Sermanet, P., Welker, S., Chan, C., Gilitschenski, I., Bisk, Y., and Dwibedi, D. Vid2Robot: End-to-end Video-conditioned Policy Learn ing with Cross-Attention Transformers. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 20000X0885..222XXX05.22.
Sakoe, H. and Chiba, S. Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans- actions on Acoustics, Speech, and Signal Processing, 26 (1):43-49, 1978, doi: 10.1 10..109/AASP..999 1978. 1163055.
Sontakke, S., Zhang, J., Arnold, S., Pertsch, K., Biyik, E., Sadigh, D., Finn, C., and Itti, L. Roboclip: One demonstration is enough to learn robot policies. Advances Neural Information Processing Systems, 36, 2024.
July 2024. doi:
Kedia, K., Dan, P., Chao, A., Pace, M. A., and Choudhury, S. One-shot imitation under mismatched execution. In 2025 IEEE International Conference on Robotics and Au- tomation (ICRA), 2025. URL https: http///xxxvvvorg// abs/ 2409. 06615.
Sun, W., Vemula, A., Boots, B., and Bagnell, D. Provably efficient imitation learning from observation alone. In International conference on machine learning, pp. 6036- 6045. PMLR, 2019.
Liu, Y., Dong, W., Hu, Y., Wen, C., Yin. Z.-H., Zhang, C., and Gao, Y. Imitation learning from observation with automatic discount scheduling. In The Twelfth International Conference on Learning Representations, 2024. URL https:///// http://pppereeeeeeee/e htt/ id=pPJIQYCPNI.
Swamy, G., Choudhury, S., BBaneell J. A., and Wu, S. Of moments and matching: A game-theoretic framework for closing the imitation gap. International Conference on Machine Learning, pp. 10022-10032. PMLR, 2021.
Tian, T., Xu, C., Tomizuka, M., Malik, J., and Bajesy, A. What matters to you? towards visual representa- tion alignment for robot learning. In The Twelfth In- ternational Conference Learning Representations, 2024. URL id=CT] [d=CTTUHTKF71, UHIKF71.
Luo. Y., zhengyao jiang, Cohen, S., Grefenstette, E., and Deisenroth, M. P. Optimal transport for offline imita- tion learning. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: / //openreview. net / forum? id-MhuFFFsrfvi.t
//penree////eeeeeeee net/ forum?
Ma, Y.J., Kumar, V., Zhang, A., Bastani, O., and Jayaraman, D. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pp. 23301-23320. PMLR, 2023.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems, pp. 5026-50333 2012. doi: 20.100//RSS.0012263666
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V. Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Torabi, F., Warnell, G., and Stone, P. Genera- tive adversarial imitation from observation. In Im- itation, Intent, and Interaction (I3) Workshop at ICML 2019, Long Beach, California, USA, June 2019. URL http: /tt::///wwwww www. cs www.ua.eesse..... edu/ users/af https//ailabbbMMMM. 9a- torabi.
Palo, N. D. and Johns, E. Keypoint Action Tokens Enable In- Context Imitation Learning in Robotics. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024. doi:
Xiao, H., Herman, M., Wagner, J., Ziesche, S., Etesami, J., and Linh, T. H. Wasserstein adversarial imitation learning, 2019. URL 1906. 08113.
Papagiannis, G. and Li, Y. Imitation learning with sinkhorn distances. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 116-131. Springer, 2022.
Xie, S., Sun, C., Huang, J., Tu, Z., and Murphy, K. Re- thinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the

---

Imitation Learning from Single Temporally Misaligned Video
European conference compuler vision (ECCV), pp. 305-321, 2018.
Xu, M., Shen, Y., Zhang, S., Lu, Y. Zhao, D., Tenenbaum, J., and Gan, C. Prompting decision transformer for few- shot policy generalization. In Chaudhuri, K., Jegelka, S., Song, I.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learming, volume 162 of Proceedings of Machine Learning Research, pp. 24631-24645. PMLR, 17-23 Jul 2022. URL htRp:/ http::/:///pccceigggs..... mlr. press pieas/vll. /vlllllel 62/ xu22g. html.
Yang, D., Tjia, D., Berg, J., Damen, D., Agrawal, P., and Gupta, A. Rank2reward: Learning shaped reward functions from passive video. In 2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 2806 2813, 2024. doi: N0.1109/CCRA57147.22244 10610873.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, Mastering visual continuous control: Improved data-nugmented rein- forcement learning. Deep RL Workshop NearlPS 2021, 2021. URL https: wwwpprewewww.eeeeeeeeeeeeeeeeeee id=L5HKN IsdSE.
Yu. T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Kael- bling, L. P., Kragic, D., and Sugiura, K. (eds.), Proceed- ings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1094 1100. PMLR, 30 Oct-01 Nov 2020. URL https:// proceedings mlr wtw.s//oeeee.oooooooooooooo http://1000yy22..hh..ll
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. 11975-11986, October 2023.
Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
12

---

Imitation Learning from Single Temporally Misaligned Vidco
Appendix
Table of Contents
Detailed Analysis of ORCA
A.1 Proofs of ORCA Desiderata
13
Toy Examples CRCA Overcoming Failure Cases of Existing Approaches
B Environment Details
16
B.I. Visual Encoder
18
B.2 Meta-world
18
B.3 Humanoid
19
B.4 RL Policy.
................................
22
B.5 Approach and Baselines.
22
B.6 Varying Temporal Alignment Level Experiment Details.
23
C Additional Experiment Results
23
C.I Frame-Level Failure Modes in Meta-world
................................
25
C.2 The importance of pretraining for ORCA
25
C,3 Additional Baselines
................................
25
C,4 Visual Encoder Ablation...
................................
................................
27
C.5 Inage-Conditiones Policy
................................
27
Runtime Comparison of Approaches
28
A. Detailed Analysis of ORCA
We prove propositions 4.1 and 4.2, showing that the ORCA reward encourages the agent to complete all subgoals in the correct order, thus addressing the limitations of baselines that use frame-level matching. First, we derive basic properties of the ORCA coverage matrix. Then, we restate the propositions and prove them using these properties.
A.1. Proofs of ORCA Desiderata
Lemma A.1. For all 1, Ct = maxi=1 Ci.j-1P.5
Proof. We prove this by induction on
Base case: For = 1, C1,5 = C1,5-1PI.5 by (5), which satisfies the statement.
Inductive step: Assume Ct,, max Cij-1Pa, holds for For 1, by the recursive definition (5):
C441,5 max(Cr,, C
Substituting the inductive hypothesis:
C++1, = max max(max Ci C4415-1 max Ct.j-1 Pa.. 1=1
t+1
1=1
Thus, the statement holds for 1. By induction, the lemma is proven.
Corollary A.2. If at time t, max Pi.j = Pt.g, then Org = Cr.j-1 Pt.j-
13

---

Imitation Learning from Single Temporally Misaligned Video
Proof. By the non-decreasing property of coverage along the learner axis (5),
max Ct - Ct,5-1-
(8)
Am
By lemma A.I.
Ct. Ci.-iPPas
(9)
Corollary A.3. Iww trajectories ç+ and g are identical, except a subgoal oj is covered at time in E+ and is not covered in E then U > Oz5
Proof. 5* achieves coverage of oj at time t, so by corollary A.2:
City = -T Pts > Ci-1s"
(10)
Since the trajectories identical prior to f, we have:
Ct-1 =Cūj-1
(11)
Moreover, Pit PT, implies
-ij-1 Pt. iij-1
(12)
It follows that
-t max Cax P
(13)
We conclude from the coverage definition (5):
CT. > Cū.
(14)
Proposition A.4 (ORCA enforces subgoal ordering (Restating Prop. 4.1)). Let be trajectory that is out of order; specifically, there exists subgoal oj such that oj is occupied at time I and 0j-1 is not yet covered. Let g+ be trajectory that is identical { except that it covers 05-1 before time Then, RoRca x+, :
Proof. Because oj is occupied at time t in E+ (max'a P.j -Pty), by corollary A.2,
Ct,y -
(15)
According to the DP recurrence relation (5), there are two cases for C
Case 1. Cūy CT-1, By lemma A.1,
C
max i=1
(16)
By corollary A.3, and the fact that coverage is nondecreasing along the demonstration,
T CT.-1 2 máx Cū
(17)
Because + and - both occupy subgoal oj at time l:
P = max P im
(18)
14

---

Imitation Learning from Single Temporally Misaligned Video
Multiplying (17) and (18) lets us establish bound on (16):
1 max Cū, 12--1 max P. 2 max C 1..- ==1 ==1
1-1
(19)
Substituting (15); get:
Crj CT.
(20)
Case 2. Ci. = 11j-1 Pt.
Because Pt, P(,z and by corollary A.3,
C - > CTj-IPTs
(21)
Substituting (15), get:
T Ci.
(22)
Since O7 > Cr. in both cases, and the trajectories are otherwise identical,
RORcAlOt.E (t,E) R .))-
(23)
Proposition A.5 (ORCA enforces subgoal coverage. (Restating Prop. 4.2)). Let 5- be trajectory that occupies 0z-1 at time I and continues to occupy 0j-1 at time I, instead of progressing towards 0z. Let E+ be an identical trajectory that progresses towards oj at and assume that neither trajectory has been closer to oj before. Then, RORCA ,.,,) > RR ,.)
Proof. At time 1, E+ moves closer 0, than it has previously been. Thus, by corollary A.2:
C = - > Ct-1s
(24)
There are two cases for CT,
Case 1. Cigy CT-1
Because &+ is identical E prior
Ct CT-1s
(25)
Thus, by (24),
City Ci CT- Cī.
(26)
Case 2. CT5 = PT.
Since + is identical to = prior to f, and time neither improves its coverage of subgoals prior to oz:
= Cū.-r
(27)
However, because + moves closer to ōy than &- at time t,
Pt. Pi.j
(28)
We conclude that
C = CT. PT, Since City > Cry both cases, and the trajectories are otherwise identical,
(29)
Roncalot, .5t RoncAloT.E_.55
(30)
15

---

Imitation Learning from a Single Temporally Misaligned Video
OT fails to enforce temporal ordering.
DTW enforces temporal ordering.
ORCA enforces temporal ordering.
Demosstration Trai
2
Demerierntion Tra
Demonstration Taj
Optimal Leamer Tral
Suboptinal Learner Trai
Optimal Learner The
Suboptimal Learner Tra
Optimal Learner Traj
Subsptimal Leamer Traj
45
45
45
Muric
Disrccc Maric
Irates
Mates
DTW
DRCA
ORCA
OT optinal matcling
votime 00 Teething matci
D' optimaal
algimmnt matrls
1,00
ORCA Ondored
Coverage Mazrik
1,00
0:14
Bild
0.14
ORCA Ordoved Coverage Mariis
0.50
050
8,50
2.50
X%
2.00
0.00
EXC
A
1,00
6:37
2:37
0.44
A.
1.00
3.00
1.00
014
Cumulative OT Reword
Cumalative OT Reward
Re
Cumulative DTW Reward
Rorv
E)
DTW Revard Ran-A
Comulttive
a)
Comalatise ORCA Reward
P9 F)
Cumulation ORCA Revard
Moncolo
1.00
1.00
1.00
5.00
1.55
0.41
The suboptimaal learner trajectory has equaally high cumulatiun the optimal trajectory
The subopiimol leamnr trajectory S has lower cumulative
The saboptimal learner trajectory 6 has lower curmulative ORCA reward optimal leamer trajectory [
DTW optimal trajectory
Figure 7. Failure cases for OT reward in the 2D-Navigation environment. Both DTW and ORCA overcomes OT's limitation.
A.2. Toy Examples of ORCA Overcoming Failure Cases of Existing Approaches
We present complete figures showing how ORCA overcomes OT's failure to enforce subgoal ordering and DTW/Tempo- ralOT's failure to enforce full subgoal coverage. The distance function between each state is the Manhattan distance, which Markovian.
In Fig 7, the suboptimal learner trajectory completes the demonstration subgoals in the wrong order compared to the optimal learner trajectory that completes the task in the correct order. Because OT treats the learner and trajectory distribution as two unordered sets, and the distance function does not encode any temporal information, it fails to penalize the suboptimal trajectory, giving both equally high rewards. In contrast, because DTW enforces temporal constraint in its alignment, the final alignment is the same for both trajectories, resulting in a lower DTW reward for the suboptimal trajectory. Similarly, ORCA measures the probability that all subgoals are covered in the correct order. Although the suboptimal learner trajectory perfectly occupies the third subgoal time 3, because has not occupied the second subgoal, its overall ordered coverage still low, thereby penalizing the suboptimal learner trajectory for covering out of order.
In Fig. 8, the suboptimal learner trajectory stalls at the second subgoal while the optimal trajectory makes consistent progress towards covering all subgoals. DTW fails because it does not constrain the number of learner frames that can be matched with each subgoal. Consequently, its alignment matrix matches most learner frames to the second subgoal, which has no cost since they perfectly occupy it, resulting in equally high DTW rewards for both trajectories. In contrast, because CRCA rewards depends on all subgoals be covered, the suboptimal trajectory receives low rewards for most timestep since has not covered the final subgoal.
In Fig. 9, the video demonstration is temporally misaligned because the agent can move one cell at time, and the subgoals would require the agent to take multiple timesteps to reach. The suboptimal learner trajectory is slow, spending 2 timesteps at earlier subgoals and failing to solve the tasks in time. Meanwhile, the optimal learner trajectory makes consistent progress and succeeds. Because TemporalOT assumes that the learner and demonstration demonstrations are temporally aligned, the mask window that then defines causes the coupling matrix to also approximate a diagonal matrix. Such coupling matrix would encourage the agent to spend equal amount of time matching each subgoal, thereby rewarding the suboptimal trajectory that does so perfectly even though they did not finish the task. In contrast, ORCA's 's rewards depend on all subgoals to be covered, so the suboptimal trajectory receives low rewards for all timesteps since it has not covered the final subgoal.
B. Environment Details
We run experiments in two different environments: Meta-world (Yu et al., 2020), a manipulation environment, and Humanoid-v4 (Todoroy et al., 2012), a more difficult control environment. Following prior work (Fu et al., 2024c), we
16

---

Imitation Learning from a Single Temporally Misaligned Video
DTW fails to encourage progress.
ORCA encourages consistent progress.
02
6,
Demonstration Traj E
Demonstration Traj
Optimal Learner Traj
Suboptimal Learner Traj
Optimal Learner Traj
Suboptimal Learner Traj
Reaching all the Frames the same
Got stuck the 2nd frame
the frames the same
Reaching
Got stuck at the 2nd of
9
! %
order as
order as
97
Diennoe Metric
Distace Matric Fni
Distrro Metticc
Distenca Maui Fni
ORCA
ORCA
DTW optimal alignment path
DTW optimal a aignment path
ORCA ordered covered matrik
ORCA ordered covered matrix
d2
6,
61
8z
d,
OT
1.00
0.00
1.00
0.05
1.00
0.05
1.00
0.00
1.00
0.14
1.C0
0.05
0.00
1.00
1.00
0.37
1.00
0.05
1.00
0.00
1.00
1.00
1.00
1.00
0.14
Comcure Reward
Rowardd
Compute Reward
Compute Rona
Cumulative DTW
Cumulative DTW Reward
Cumulative ORCA Reward
Cumulative ORCA Reward
Σ Rprw co
Rorw
Ro 1o.
RORcA
Reward
2.00
2.00
1.55
0.28
The suboptimal learner trajectory 5- has equally high cumulative DTW reward the optimal learner trajectory
The suboptimal learner trajectory 5 has lower cumulative ORCA reward than the optimal learner trajectory g+
Figure 8. Failure cases for DTW reward in the 2D-Navigation environment. ORCA overcomes DTW's limitation. The suboptimal learner trajectory& { gets stuck at the second frame of the demonstration, while the optimal one { makes consistent progress.
TemporalOT fails to encourage progress.
ORCA encourages consistent progress.
Temporally Misligned
Temporally Misal gied Demoratration
Demonstration
Tra
Subeptimal
Optimal
Tra
Suboptimal
Teppal07
00CA
06GA
0.00
1.00
0.00
0.00
1.00
0.00
0.00
0.00
1:00
1:00
0.37
0.00
55
1.00
0.37
0.05
0.00
0.00
0.53
0.67
0.00
0.00
1:00
0.00
0.00
1.00
1:30
1.00
0.030
02
1.00
1.000
87
0.00
0.00
0.00
0.00
1.00
0.00
0.00
3.00
0.00
1.00
1.00
1.00
0.14
42
1.00
1.00
1.00
0.0067
1.00
1.00
0:00
0.00
0.00
5.00
0.00
0.00
09
00
1.00
1.00
1,00
5.00
1.00
1.00
0.05
Commistive
TongoraOOT Reward
Camulatve
Ryco
6.97
Roncy E
158
Comulative ORCA Revard
6.08
The sbopt rel kasmer trajessor has higher cumulative
The suboptirral srjjcttry
cummuiation
TemponalOT
trajectory
ORCA reword tixn optinal barter trajectory
Figure 9. Failure cases for TemporalOT reward given a temporally misaligned demonstration the 2D-Navigation environment. ORCA overcomes TemporalOT's limitation. the video demonstration is temporally misaligned because the agent can move one cell a time, and the subgoals would require the agent to take multiple timesteps to reach. The suboptimal learner trajectory & that moves slowly and fails to complete the task. In contrast, optimal learner trajectory { makes steady progress and completes the task successfully.
17

---

Imitation Learning from Single Temporally Misaligned Video
Test 10 different tasks in Meta-world. To show that ORCA works in more general domains, we additionally design 4 tasks in the humanoid environment that require moving the arms of the humanoid.
B.1. Visual Encoder
ORCA, as well as all baselines except RoboCLIP, requires distance function defined on the space of images. We use visual encoder to obtain embeddings for each image in the learner and expert trajectory, and find the pairwise distances between them obtain the distance matrix.
In Meta-world, we follow prior work (Fu et al., 2024c) in using an off-the-shelff Resnet50 (He et al., 2016) as the visual encoder. We use the cosine distance between embeddings to produce a cost matrix.
In Humanoid, we find that off-the-shelf visual encoders do not capture the fine-grained details necessary for the more difficult control task. Instead, we train a model predict the joint positions of the humanoid, and use the Euclidean distance between joint predictions the distance function. To address -iitri-diitiitttoonn samples, train separate network to predict the model confidence, which is used to scale the final rewards. For more details on the joint predictor, see B.3.2.
B.2. Meta-world
B.2.1. TASKS
We run experiments on 10 different tasks in the Meta Meta-world (Yu et al., 2020) environment. In addition the 9 tasks in Fu et al. (2024c), added Door-close. We classify them into easy, medium, and hard based factors like their visual difficulty, necessity for precise motor control, and interaction with other objects. For further information on the tasks, we refer the reader to (Fu et al., 2024c). Below is a brief description of the objective for each task:
Task
Difficulty
Success Criteria
Button-press
Easy
The button is pushed.
Door-close
Easy
The door is fully closed.
Door-open
Medium
The door is fully open.
Window-open
Medium
The window slid fully open.
Lever-pull
Medium
The lever is pulled up.
Hand-insert
Medium
The brown block is inserted into the hole in the table.
Push
Medium
The cylinder is moved to the target location.
Basketball
Hard
The basketball is in the hoop.
Stick-push
Hard
The bottle is pushed to the target location using the stick.
Door-lock
Hard
The locking mechanism engaged (pushed down).
B.2.2. TRAINING DETAILS
In order to improve performance across all methods, we employ two training strategies on top of the reward model and RL algorithm:
1) Context embedding: We use the context embedding-based cost matrix proposed in (Fu et al., 2024c), which can be interpreted as a diagonal smoothing kernel. Specifically, the distance between two frames is expressed as the average distance over the next cu learner and demonstration frames (where ow refers to the context window):
dyiindow (of .09
oft
Car
We choose a context window of length 3, which resulted in the best performance in (Fu et al., 2024c). Although longer context window could damage performance on extremely mismatched tasks, we find that small window helps regularize the noisiness of the visual distance metric.
2) Timestep in agent state: By nature of the sequence- following task, the reward at a given time step depends on the states visited by the learner in previous time steps. Thus, if the policy or value estimator cannot observe the entire trajectory, then
18

---

Imitation Learning from Single Temporally Misaligned Video
Humanoid Expert Demonstrations
Arm Up (L)
Arm Up (R)
Arms out
Arms down
Figure 10. Visual Demonstrations of the tasks in the Humanoid Environment. These were generated by selecting the target final joint state, interpolating from the start joint state, and rendering the intermediate frames.
does not have enough information to model the reward. In practice, we find that including the current time step (as percentage of the episode length) in the state observation of the agent allows reasonably estimate the value function. We emphasize that, although the agent has access to the ground truth state and time step, the reward model only sees the visual learner rollout and demonstration. Because the purpose of this paper is to examine specific sequence-matching reward functions, we choose to use this empirical trick for our experiments, leaving further investigations into RI. algorithms given non-Markovian rewards as future work.
B.3. Humanoid
B.3.1. TASKS
We use the MuJoCo Humanoid-v4 environment (Foundation, 2024; Todorov et al., 2012). In order to improve visual encoder performance, we modify the environment textures and camera angle, as described in (Rocamonde et al., 2024). At the beginning of an episode, the humanoid spawned upright, slightly above the ground, with its arms curled towards its chest.
The humanoid's 's goal is to follow the motion of demonstration trajectory within maximum of 120 timesteps. We define motions, corresponding demonstration trajectories. The humanoid must remain standing while doing all tasks. The visual demonstrations for each task are shown in B. These demonstration trajectories each have a length of 10 and are
Task
Success Criteria
Arm up (L.)
The left arm is raised above the head and the right arm is down. The right arm is raised above the head and the left arm is down. Both arms are raised to shoulder height. (T-pose)
Arm up (R)
Arms out
Arms down
Both arms are lowered the side.
generated by interpolating between the initial and final poses. Fig. 10 shows snapshots of these trajectories.
19

---

Imitation Learning from Single Temporally Misaligned Video
Step 0
Step 30
Step 60
Step 90
Goal
Pretrained Vision Transformer Episode Rewards
Joint Prediction Episode Rewards
1.0
1.-
Ground Truth
Joint Prediction
Normalized Reward
0.6
0.4
0.4
Ground Truth
SigLIP
DINOv2
0.0
20
60
100
120
20
60
100 120
Rollout Time Step
Time Step
Figure 11. Goal-reaching rewards of (left) two pretrained models and (right) our joint prediction model on an example learner trajectory, with the goal of raising the left arm the side. To emphasize their shape. all rewards normalized along the trajectory dimension. Rewards using the pretrained SigLIP- ViT-B-16 16 (Zhai et al., 2023) and DINOV2-ViT-B-144-0 calculated the cosine similarity hetween the learner and demonstration embeddings. The fine-tuned joint prediction model provides smoother reward curve. The trajectories are in the MuloCo Humanoid-vá environment (Foundation, 2024; Todorov et al., 2012), which visually modified mimic the setup (Rocamonde et al., 2024).
(Oquah et al., 2023) models are
B.3.2. CONFIDENCE- SCALED VISUAL REWARDS.
Empirically, we found that off-the-shelf visual encoders produced noisy rewards the Mujoco environment, as shown in Fig. 11. This resulted training failure regardless of the distribution-maachhin this problem, train network that predicts the joint positions of the humanoid given image observation. To address distribution shift during RI. training, we use an autoencoder to predict model confidence score, which we use to scale the final rewards. We additionally assume access to stability reward function, which includes a control cost and reward for remaining standing:
eequence-matchinn function. To solve
Retabsisty =
1.3)2) Cetri
(31)
where htors is the height of the humanoid torso, and Catrl control cost provided by the environment. Given learner observation of, demonstration subgoal of. visual backbone á, and joint predictor f. we let a( of) (If(d(ott) - s((o
(32)
This distance metric is used by ORCA to obtain Rorea Then, we compute the confidence of the learner observation embedding c(p(of)) according to 34. This results in the final reward function:
R = c( Rorealot, .D) + XR
(33)
B.3.3. JOINT PREDICTOR TRAINING DETAILS
Dataset: We collect a dataset D {(o, j.) containing MuJoCo images o, of various poses and corresponding joint positions 7:- The dataset includes in total 9,038 samples. To build the dataset, we utilize a set rollout trajectories covering a of goal reaching tasks (such as different hand poses, doing splits, etc.), We include both successful and unsuccessful
20

---

Imitation Learning from Single Temporally Misaligned Video
trajectories. To ensure diversity among samples representing different stages of trajectory, we select frame every k frames (here & = 5), encouraging the network differentiate between similar images. Given the similarity of initial trajectories, we retain the first four frames only 25% of the time, and in those cases, select random frame from a five-frame interval.
Training: To train our joint predictor f d, we fully fine-tune a ResNet50 backbone (He et al., 2016) pre-trained on ImageNet-IK (Deng et al., 2009) with 3-layer MLP head that projects to the joint dimension. The MLP head has layers of shape (2048, 1024). (1024, 1024). and (1024, 54), where 54 represents the number of joints (18) multiplied by the dimension per joint (3). Optimization is performed over 100 epochs using SGD with learning rate .008, batch size 16, and momentum 0.875. After training the joint predictor, we freeze the backbone weights, and train shallow autoencoder architecture with two linear layers of shapes (dz, 32) and (32, da) using the same parameters, where dø is the dimension of the backbone (2048 in this case). This provides the reconstruction loss that is used for confidence estimation, as described in B.3.4
Step
Step 30
Step 60
Step 90
Confidence-Scaled Rewards on a Falled Episode
0.40
Rollout
0.15
Truth
0.10
Predicted
Confidence Scaled
0.05
Joint Prediction
20
60
80
200
120
Timestep
Figure 12. The impact of confidence scaling rewards for a trajectory where the robot falls down. The ground truth trajectory is shown on top and renders of model's 's joint predictions given the top frames are shown on the bottom. The rewards are computed with respect the final reference image of the left arm wave task. Once the robot starts falling, there is distribution shift in the images, and the model predicts that the robot is upright. This leads to high reward predictions, but also high uncertainty estimates. When corrected using the confidence function, the reward shape improves significantly.
B.3.4. CONFIDENCE ESTIMATE AS REWARD SCALING
The use of joint-position predictor results in additional challenge: in environment with unstable dynamics, there is large space of image observations with very different joint positions, many of which are difficult to reach through robot play. During RL training, policy can reach a state outside of DD resulting in noisy joint predictions and rewards. To solve this problem, prior work has estimated the epistemic uncertainty of the visual model using an autoencoder architecture (Andrews et al., 2016; Frey et al., 2023). Given that the training converges, an embedding z is in the domain, then the autoencoder will be able to achieve low reconstruction loss. Contrapositively, it has high reconstruction loss on an embedding z', then must not be in domain.
First, we train autoencoder to reconstruct image embeddings the offline dataset. After training coverages, we compute the final mean Preco and standard deviation Treco of the loss on this dataset. Then, we use formulation inspired by Frey et al. (2023) to compute confidence score u(z)):
if (2)) Preco
c(zi)
exp (- (Crecel1)) -
(34)
otherwise
where KG is a hyperparameter that controls the spread of the confidence function. In our experiments, we set hi = 2.
We observe that the reconstruction losses over a set of trajectories sampled from partially trained policies are skewed towards low confidence. In-domain images tend to have low reconstruction loss that is tightly clustered around the average offline loss, while of-doinn images tend to have higher and more spread out loss. Figure 12 shows a qualitative example of
21

---

Imitation Learning from a Single Temporally Misaligned Video
how uncertainty scaling fixes incorrect reward predictions on out-of-distribution images. The shape of the uncertainty scaled reward curve better matches the ground truth.
B.4. RL Policy.
In the Meta-world environment, we use an identical setup and hyperparameters Fu et al. (2024c), training the policy with DrQ-v2 over million steps. We also follow Fu et al. (2024c) to repeat a policy's 's predicted action 2 times, which effectively shortens the episode length by half. This effect is applied both in RL training and generating demonstrations from hand-engineered policies. In MuJoCo, the policy is trained with SAC over 2 million steps, and we slightly modify the parameters from Rocamonde et al. (2024) to adapt to the shorter training period (2 million steps vs 10 million steps). All policies are trained with ground truth state observations, but we emphasize that the reward function only sees visual observations.
Table 3. Training hyperparameters used for experiments both environments.
Parameter
Meta-world (DrQ-v2)
Humanoid (SAC)
Total environment steps
1, 1,000,000
2,000,000
Learning rate
lc-4
le-3
Batch size
512
256
Gamma (7)
0.9
0.99
Learning starts
500
6000
Soft update coefficient
5e-3
5e-3
Actor/Critic architecture
(256, 256)
(256, 256)
Episode length
125 or 175
120
B.5. Approach and Baselines.
We describe the hyperparameter details for each baseline as needed.
ORCA. We use À for temperature turn. For Meta-world tasks, we initialize the policy by loading Tempora10T's checkpoint at 500k steps before training on ORCA rewards. For Humanoid tasks, we load the checkpoint at 1M steps because the total training steps is 2M.
OT. We solve the entropic regularized optimal transport problem shown in (1). In Meta-world, the weight on the entropic regularization term is c = 1, and in Humanoid, it is 0.
Tempora10T. TemporalOT (Fu et al., 2024c) was originally designed for learner and reference sequences of the same length. In this paper, we compare against a slightly stronger version of TemporalOT, which allows for a learner that is linear the speed of the expert (either faster or slower). Specifically, we mask with a windowed diagonal matrix stretched along the longer of the learner and demonstration axis. Because TemporalOT also formulates the entropic regularized OT problem (4), the entropic weight is also 1. We varying mask window size, depending on the reference length. Fu al. (2024c) used km. = 10 for metaworld matched demos. We use the same for matched demos, and for mismatched demos, we let km RJ
TO
Threshold. d. This is a hand-engineered reward function with simple conditionals. Threshold d contains two terms: the number of subgoals completed and the reward with respect to the current subgoal to follow. It initializes the subgoal to follow as the first subgoal, and it starts tracking the next subgoal, when the current subgoal's reward is above a predefined threshold. We set the threshold as 0.90 for all Meta-world tasks and 0.70 for all Humanoid tasks because it is more challenging environment where it is diffieult achieve high rewards.
RoboCLIP. RoboCLIP (Sontakke et al., 2024) uses a pretrained iileo-and-langunn model (Xie et al., 2018) to directly encode the video. For videos with more than 32 frames, RoboCLIP downsample them to 32 frames before passing them to the encoder. It defines the reward for the last timestep as the cosine similarity between the learner video's 's and the demonstration video's 's embeddings, while all previous timesteps have zero as the reward. Due to Python version issues, we train all RoboCLIP policies using SAC (Haarnoja et al., 2018), following the setup and code base from Sontakke et al. (2024).
22

---

Imitation Learning from Single Temporally Misaligned Video
B.6. Varying Temporal Alignment Level Experiment Details.
We study the effect of how increasingly misaligned demonstrations would affect ORCA and Tempora performances. We identify two types of temporal misalignment: either the demonstration contains pauses and is slower, or the demonstration accelerates through segment and is faster. We randomly perturb the original demonstrations of three Meta-worke tasks (Door-open, Window-open, Lever-pull), generating 6 demonstrations per task per speed type (i.e, slow or fast).
Concretely, we first evenly split the original demonstration into five segments. To get the first three randomly perturbed demonstrations, we randomly select one segment and randomly change their speed. We can speed up a segment by 2, 4, 6, 8, or 10 times, and we can slow down a segment by 2, 3, 4, 5, or 6 times. To get the rest of the randomly perturbed demonstrations, we randomly select three segments and randomly change their speed. Then, for each task and speed type, categorize 3 of the perturbed demonstrations as having "Low" level of misalignment and the other level of misalignment by ranking the demonstrations based on the mean absolute deviation between the segment length. The more varied the segment lengths are with respect to each other, the more temporally misaligned this demonstration is. Each perturbed demonstration is trained on a random seed, and we ensure that ORCA and Temporal IOT are trained on that same seed for fair comparison.
having "High"
C. Additional Experiment Results
Temporally Misaligned Experiment.
Given Temporally Misaligned Demonstrations
Threshold
RoboCLIP
TemporalOT
ORCA
ORCA
Medium
Figure 13. Training Curves for All Meta-world tasks Given Temporally Misaligned Demonstrations We compute the mean and standard error across the training runs, each evaluated on 10 random seeds.
Fig. 13 shows training curves of all methods given temporally misaligned demonstrations. The ORCA plot diverges directly from Tempora loT at 500k steps because it is initialized with these Tempora lOT policies, and trained for additional 500k stops. In tasks where Tempora107 achieves some success by this point steps, ORCA performs at least well as ORCA (NP), because it can take advantage of the better initialization. In tasks like basketball and door-open, where Temporalon 10T is unsuccessful after 500k steps, ORCA (NP) performs better because it is trained with the ORCA reward for more steps. All methods fail to learn the push task, which we hypothesize is due to the visual encoder.
Given Humanoid Interpolated Demonstrations
Threshold
RoboCLIP
DTW
OT
TemporalOT
ORCA (MP)
ORCA
Inp-teee 1000
Arma tee
Enterment
Step
Figure Training curves for all humanoid tasks. We compute the mean standard error across the 3 training runs, cach evaluated on 8 random seeds. Because Tempora a0OT 1OT achieves poor performance, ORCA does not benefit from pretraining, and ORCA (NF) is the most successful.
23

---

Imitation Learning from Single Temporally Misaligned Video
Table Performance comparison across all tasks baselines in the Humanoid environment. The best value within standard deviation for each row is highlighted.
Task
Threshold
RoboCLIP
DTW
OT
TOT
ORCA (NP)
ORCA
Arm up (L)
7.21 (2.98)
5.17 (2.25)
11.38 (3.49)
5.42 (2.75)
5.29 (2.22)
65.88 (8.25)
81.62 (3.65)
Arm up (R)
13.58 (2.70)
34.17 (7.26)
1.12 (0.69)
19.12 (4.10)
7.67 (2.88)
92.46 (4.71)
49.58 (5.00)
Arms out
20.79 (4.47)
7.75 (5.00)
4.75 (2.36)
3.75 (1.84)
1.62 (0.75)
72.67 (10.09)
8.50 (2.60)
Arms down
0.00 (0.00)
0.67 (0.56)
3.17 (2.31)
30.38 (7.19)
11.62 (3.56)
19.71 (5.03)
33,42 (7.20)
Average
10.40 (2.54)
11.94 (3.77)
5.10 (2.22)
14.67 (3.97)
6.55 (2.35)
62.68 (7.02)
43.28 (4.61)
Fig. 14 shows training curves of all methods given interpolated demonstrations in the humanoid environment, which naturally temporally misaligned. Tab. 4 shows the final cumulative reward of all methods. Across all tasks, ORCA (NP) performs the best. In general, ORCA (NP) performs better than ORCA because Temporalon OT achieves near performance, and ORCA cannot reap the benefits of better initialization. All methods perform poorly on arms down because it is an extremely unstable position.
Fig. 15 shows qualitative comparison between frame-level matching approaches and ORCA on the arm up task. ORCA quickly solves the task, while the other approaches have the failure modes described in Sec.
8,
Video Demo
Interpolated
Demo (Misaligned)
Initial
Distance Matrix
Matching Matrix
OT matchus later leamer frames
OT moves towards the goal, then goes
OT
backwarde progress
DTW
DTW
frame the subgoaa and the trajectory
every leanner
stuck in an early subgoal
7OV (4, 1)
TOT (A, forces the learner to wait each subgool, making unstable
TOT slow and unstable
DRCA reaches final subgoal under 20 frames
ORCA
ORCA quickly completes the task and stays the final
Figure 155. Qualitative comparison between the frame-level matching approaches and ORCA when solving the Mujoco Arm Up (I.)
task. The ORCA agent completes the task quickly, while the other methods exhibit the failure cases described in Sec.
Temporally Aligned Experiment.
Fig. 16 shows the training curves for all methods in Meta-world with temporally aligned demonstrations, and Tab, shows the final cumulative reward of all methods. Across all tasks, ORCA performs the best. Since Temporalo7 achieves good baseline performance on most these tasks, ORCA is able to take advantage of pretraining, and outperforms ORCA (NP) on most tasks.
24

---

Imitation Learning from Single Temporally Misaligned Video
Given Temporally Aligned Demonstrations
Threshold
RoboCUP
DTW
TemporaloT
ORCA (NP)
ORCA
Fay
Medium
Hard
Figure Training Curves for All Meta-world tasks Given Temporally Aligned Demonstrations We compute mean and standard error across the 3 training runs, each evaluated on 10 random seeds.
Table Meta-world per-task results on temporally aligned demonstrations. We report the mean expert-normalize return and the
standard error; each task is run three random seeds.
Environment
RoboCLIP
Threshold
DTW
OT
TemporalOT
ORCA (NP)
ORCA
Button-press
0.00 (0.00)
0.17 (0.07)
0.16 (0.05)
0.45 (0.09)
0.61 (0.09)
0.86 (0.11)
0.71 (0.12)
Door-close
0.47 (0.38)
0.72 (0.02)
0.30 (0.01)
0.75 (0.01)
0.74 (0.01)
0.92 (0.01)
0.81 (0.01)
Door-open
0.00 (0.00)
0.91 (0.09)
0.40 (0.05)
1.22 (0.03)
1.20 (0.03)
0.00 (0.00)
1.45 (0.11)
Window-open
0.00 (0.00)
0.61 (0.09)
0.01 (0.01)
0.61 (0.09)
0.13 (0.06)
0.07 (0.07)
0.64 (0.17)
Lever-pull
0.00 (0.00)
0.03 (0.02)
0.14 (0.05)
0.15 (0.06)
0.37 (0.08)
0.19 (0.08)
0.35 (0.09)
Hand-insert
0.00 (0.00)
0.11 (0.05)
0.59 (0.11)
0.73 (0.09)
0.56 (0.09)
0.30 (0.10)
0.64 (0.12)
Push
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.25 (0.09)
0.10 (0.06)
0.21 (0.09)
0.12 (0.07)
Basketball
0.00 (0.00)
0.00 (0.00)
0.11 (0.05)
0.06 (0.03)
0.08 (0.08)
0.00 (0.00)
0.00 (0.00)
Stick-push
0.00 (0.00)
0.01 (0.01)
0.07 (0.02)
0.31 (0.08)
0.30 (0.08)
0.00 (0.00)
0.48 (0.13)
Door-lock
0.00 (0.00)
0.00 (0.00)
0.30 (0.08)
0.16 (0.06)
0.18 (0.07)
0.75 (0.13)
0.53 (0.13)
Average
0.05 (0.05) 0.26 (0.02) 0.21 (0.02)
0.47 (0.03)
0.43 (0.03)
0.33 (0.03)
0.57 (0.04)
C.1. Frame-Level Failure Modes in Meta-world
We show qualitative examples of the failure modes of OT, Tempora 1 OT, and DTW described in Sec. 3 occuring during Meta-world policy training.
OT and TemporalOT Fail to Enforce Ordering: Fig. 17 shows how OT and Tempora10r with large km both fail to enforce subgoal ordering, thus rewarding trajectories that do not complete subgoals in the correct order.
DTW Fails to Enforce Subgoal Coverage: Fig. 18 shows how DTW fails to enforce full subgoal coverage, getting stuck in an intermediate subgoal.
C.2. The importance of pretraining for ORCA
Fig. 19 shows a qualitative example of how pretraining on Temporal0T reward helps initialize ORCA in the correct basin. Without pretraining, ORCA (NP) gets partial credit for earlier subgoals, and gets stuck in a local minimum of immediately pushing the bottle without picking up the stick. Mcanwhile, Temporale ralOT on its own moves slowly and does not pick up the stick. ORCA picks up the stick and completes the task quickly.
C.3. Additional Baselines
In addition to the baselines described the main paper, we include a classic IRL algorithm and a conditindd reward function. We test these baselines on all easy and medium Meta-world tasks. The results are shown in Table C.3.
25

---

Imitation Learning from Single Temporally Misaligned Video
Video Demo
Demo Moved Quickly
Distance Matrix
OT/TOT Matching Matrix
10
OT fail enforce mutching
or
towards the then goes back.
looner fromes to sarferr subgoals.
TOT (2,22) TOT (4= 10)
TOT 10) risi temioral ordering dus its barge mask window.
TOT (b, 10) fails to leam meaningful behavior.
(R, moves slowly details.
has small sociions that fals to enforce tamperal ordering,
un
Figure 17. Example of how OT and TemporaloT (depending the mask window size kw)) fail to enforee subgoal ordering when solving Meta-worid task (Stick-Push).
Ay
Demo Moved Quickly
Distance Matrix
DTW Matching Matrix ORCA Coverage Matrix
30
DTW remains at inttal poatiio the
large portion
leamer frames
subgoal.
causling the agent
jest stack.
ORCA micouraage
ORCA moves quickly and succeeds.
consistent
progress,
agert succeeds quickly.
Figure 18. Example of how DTW fails to enforce full subgoal coverage when solving Meta-world task (Lever-pull).
81o
850
dm
Vidluo Demo
Temporally Aligned Demo
Distance
Matching Matrix
Matrix
= 60
ORCA (NP)
(NP) gets partial credit for subgoals. blasing to move matcl subgeals
(NP) directly puehes the battle.
TOT (A, 10)
TOT's dingonal-ikke
TOT &t, 10) slawts
matching matrix
agent
spend
each subagui.
ORCA
suconnfully pick the stick and quickly puales the
finds
betwoes ORCA
(NP) and TOT.
Figure Example of pretraining improves ORCA's performance.
26

---

Imitation Learning from a Single Temporally Misaligned Video
GAIfO (Torabi et al., 2019) is traditional IRL. algorithm. jointly trains an RL policy and discriminator between state transitions from the learner and demonstration distributions. The discriminator is used the reward function for the policy, encouraging the agent to stay within the expert state distribution. Due to computation limitations, we used state-based demonstration instead of a video. In contrast, ORCA and all existing baselines use video demonstrations. GAIfO outperforms TemporalOT; but it is still significantly worse than ORCA. We hypothesize that, because there is only one demonstration, GAIfO could fixate on inconsequential details instead of estimating the task reward.
LIV (Ma al., 2023) vision language model that can be used to calculate the rewards of video that teaches given a description of text. For this baseline, we follow the implementation FuRL [3], which closely resembles our IRL setup, While it outperforms TemporalOT, LIV is significantly worse than ORCA. LIV uses single text goal describe the task, whereas ORCA is conditioned on a sequence of images, making ORCA denser and better at capturing details. Future work should explore how ORCA can be applied to sequences of text subgoals.
Table 6. Meta-worid per-task results on additional baselines LIV and GAISO. We report the mean expert-normalized return and the standard error across different methods and task difficulties.
Category
Task
GAIfO (state-based demo)
LIV (text goal)
TemporalOT
ORCA
Easy
Button-press
0.12 (0.04)
0.00 (0.00)
0.10 (0.02)
0.62 (0.11)
Easy
Door-close
0.34 (0.04)
0.34 (0.09)
0.19 (0.01)
0.88 (0.01)
Medium
Door-open
0.22 (0.05)
0.00 (0.00)
0.08 (0.01)
0.89 (0.13)
Medium
Window-open
0.27 (0.09)
0.72 (0.19)
0.26 (0.05)
0.85 (0.16)
Medium
Lever-pull
0.10 (0.03)
0.00 (0.00)
0.07 (0.03)
0.28 (0.09)
Total
0.21 (0.03)
0.21 (0.05)
0.14 (0.01)
0.70 (0.05)
C.4. Visual Encoder Ablation
The choice of encoder is a practical, task-dependent choice, and not the main focus of this work. In Metaworld, we followed Fu et al. (2024c) and used a pretrained Resnet50. In Humanoid, we finetuned the encoder on a set of images from the simulation environment.
We include C.4 an ablation of ORCA with LIV (Ma al., 2023), robotics-specific visual encoder, and DINOv2 (Oquab al., 2023), standard vision model. Overall, Resnet50 achieves the best performance, although there is high variability.
Table Comparison of different visual backbones with ORCA. We report the mean expert-normalized return and the standard error across different backbone architectures with their parameter counts.
Task
RRCA+Resnet50 (26M)
ORCA+LIV (100M)
ORCA#DINOV24.
Door-open
1.71 (0.08)
1.10 (0.06)
0.44 (0.12)
Window-open
0.50 (0.14)
1.17 (0.15)
0.65 (0.10)
Lever-pull
0.28 (0.09)
0.04 (0.01)
0.16 (0.06)
Total
0.83 (0.09)
0.77 (0.08)
0.42 (0.06)
C.5. Image-Conditionns Policy
ORCA's goal is to estimate rewards from single demonstration in the observation space before learning policy, and we pose no assumptions on the policy itself. In this paper, we choose to provide the policy with ground truth states, for better comparisons with prior work. To investigate an image-conditioned policy, we trained an agent using DrQv2 [10] on a subset of the Metaworld tasks. Table C.5 shows the results. The state and image-based policies trained with ORCA have similar performance (average normalized return: 0.704 = 0.10 - 0.76 = 0.06). In contrast, policies trained with TemporalOT perform poorly, regardless of their input.
27

---

Imitation Learning from Single Temporally Misaligned Video
Table Comparison of policies trained on ground truth states and image observations. We report the mean expert-normalized return and the standard enror across different policy types.
Task
TemporalOT (state)
TemporalOT (image)
ORCA (state)
ORCA (image)
Button-press
0.10 (0.02)
0.00 (0.00)
0.62 (0.11)
0.60 (0.11)
Door-close
0.19 (0.01)
0.12 (0.01)
0.88 (0.01)
0.88 (0.02)
Door-open
0.08 (0.01)
0.00 (0.00)
0.89 (0.13)
1.31 (0.12)
Window-open
0.26 (0.05)
0.28 (0,05)
0.85 (0.16)
0.79 (0.15)
Lever-pull
0.07 (0.03)
0.00 (0.00)
0.28 (0.09)
0.19 (0.08)
Total
0.14 (0.02)
0.08 (0.01)
0.71 (0.10)
0.76 (0.06)
C.6. Runtime Comparison of Approaches
We experimentally tested the reward calculation latency to validate that ORCA is similar in runtime to its baselines. For each method, we evaluated 100 rollouts with demonstration of length 100 and learner rollouts of length 100 and 300.
Table C.6 shows the average latency (ms) of different reward computation methods. ORCA 24% faster than TemporalOT and 52% faster than RoboCLIP given rollouts of length 100, and is comparable threshold, OT, and DTW. LIV the fastest method because it only considers text goal and not sequence of subgoals, but this leads poor performance most tasks. The ordering of the methods is the same for 100 and 300 frames, demonstrating that ORCA efficiently scale to longer horizon tasks.
Table 9. Latency comparison of ORCA and its baselines. Methods are sorted by their latency in increasing order from left to right. We report the mean performance and standard error given learner rollouts of length 100 and 300.
Learner Length
LIV
OT
Threshold
ORCA
DTW
TemporalOT
Roboclip
100 frames
38.0 (4.7)
54.0 (0.4)
54.7 (3.8)
56.9 (0.3)
58.8 (0.4)
75.1 (1.2)
118.2 (17.9)
300 frames
130.0 (3.3)
170.2 (0.4)
171.0 (4.2)
179.9 (0.6)
185.3 (0.4)
228.5 (2.0)
300.2 (31.3)
28


Reviews:
[{"review_id": "RhPIbHgNtI", "reviewer": "Reviewer_SsGe", "summary": "This paper focuses on learning sequential tasks from a single temporally misaligned video, which belongs to the imitation learning paradigm. They propose a novel reward function - ORCA, which measures the matching at the sequence level to ensure that the agent covers all subgoals in the correct order. Experiments show that compared with the best frame by frame-level method, ORCA outperforms in several tasks from Meta-world and Humanoid.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s deep engagement with our paper, and we thank them for feedback that strengthens its clarity. We will update the paper to unify our references to tables. Please find below our responses to questions and concerns.\n# Questions\n## Q: \"When should pre-training be performed with ORCA?\"\nWe agree that there is a balance between initializing the policy in a good basin via pretraining and directly optimizing ORCA rewards. We recommend using pretraining because it results in the best and most consistent performance.\n\nGiven a temporally misaligned demonstration, ORCA outperforms all baselines on all Metaworld and Humanoid tasks, not including the ablation ORCA (No Pretraining). Additionally, given a temporally aligned demonstration, ORCA still consistently outperforms baselines on 8/10 tasks, not including ORCA (No Pretraining). The 2 tasks that ORCA does not work well on (push and basketball) are difficult for all methods. Overall, ORCA is a consistent choice for good performance, given both temporally misaligned and aligned demonstrations. \n## Q: \"Will more demonstrations lead to better results? What happens when different demonstrations have different speeds?\"\nWe show that having access to more demonstrations leads to better ORCA performance, even when these demonstrations have significantly different speeds. We kindly refer the reviewer to our reply to reviewer YcsA for the experiment setup and detailed results. \n## Q: \"What's the training wall-clock time of ORCA?\"\nEmpirically, the latency of ORCA reward calculation is significantly less than both TemporalOT [2] and RoboCLIP [3], and is almost identical to optimal transport, DTW, and threshold. The table below shows the average latency (ms) of different reward computation methods. We evaluated 100 rollouts with a demonstration of length 100 and learner rollouts of length 100 and 300, respectively. Experiments were performed on an NVIDIA RTX A6000 GPU. \n\nORCA is 24% faster than TemporalOT and 52% faster than RoboCLIP given rollouts of length 100, and it is comparable to threshold, OT, and DTW. LIV is the fastest method because it does not consider demonstration frames, but this leads to poor performance on most tasks. \n\n| method | LIV | OT | Threshold | ORCA | DTW | TemporalOT | Roboclip |\n|---|---|---|---|---|---|---|---|\n| 100 rollout frames | 38.03 ± 4.69 | 54.03 ± 0.35 | 54.69 ± 3.81 | 56.93 ± 0.33 | 58.82 ± 0.37 | 75.12 ± 1.15 | 118.21 ± 17.94 |\n| 300 rollout frames | 129.96 ± 3.29 | 170.17 ± 0.44 | 170.95 ± 4.16 | 179.85 ± 0.62 | 185.27 ± 0.42 | 228.46 ± 1.95 | 300.17 ± 31.30 |\n\nORCA’s time complexity is the learner rollout horizon multiplied by the length of the demonstration, which is also the lower bound time complexity of any method that relies on a frame-level distance matrix (including all OT-based methods). \n## Q: \"How does ORCA perform with different visual encoders?\"\nThe choice of encoder is a practical, task-dependent choice, and not the main focus of this work. In Metaworld, we followed prior work [2] and used a pretrained Resnet50. In Humanoid, we finetuned the encoder on a set of images from the simulation environment. \n\nWe include below an additional ablation of ORCA with LIV [4], a robotics-specific visual encoder, and DINOv2 [5], a standard vision model. Overall, Resnet50 achieves the best performance, although there is high variability. This substantiates the observations in [1]: \"evaluation [of visual encoders] based on RL methods is highly variable.\" \n\n| Task | ORCA+Resnet50 (26M) | ORCA+LIV (100M) | ORCA+DINOv2-L (300M) |\n|---|---|---|---|\n| Door-open | **1.71 (0.08)** | 1.10 (0.06) | 0.44 (0.12) |\n| Window-open | 0.50 (0.14) | **1.17 (0.15)** | 0.65 (0.10) |\n| Lever-pull | **0.28 (0.09)** | 0.04 (0.01) | 0.16 (0.06) |\n| Total | **0.83 (0.09)** | **0.77 (0.08)** | 0.42 (0.06) |\n## Q: \"Do slow demonstrations affect learning and lead to a slow policy?\"\nGiven slow demonstrations, TemporalOT performs poorly because it distributes the assignment over more frames. Thus, it misses important details, which causes ORCA to have similar failure modes because it is initialized in a worse basin.\n\nNevertheless, there are no temporal constraints in the ORCA formulation, so ORCA policies trained on slow demonstrations complete tasks at a similar speed compared to fast demonstrations ([as shown in the figure from an anonymized link](https://raw.githubusercontent.com/orcaicml/ICML_Rebuttal_2025/refs/heads/main/image.png)).\n\n[1] For pre-trained vision models in motor control, not all policy learning methods are created equal. ICML, 2023.\n\n[2] Robot policy learning with temporal optimal transport reward. NeurIPS, 2024.\n\n[3] Roboclip: One demonstration is enough to learn robot policies. NeurIPS, 2023.\n\n[4] LIV: Language-image representations and rewards for robotic control. 2023.\n\n[5] DINOv2: Learning Robust Visual Features without Supervision. TMLR, 2024.", "claims_and_evidence": "clear", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "none", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Paper Strengths\n\tThis paper proposes a novel reward function in which it provides rewards by calculating the probability that the agent covers all subgoals in the correct order, enhancing the performance of imitation learning in a time-shifted demonstration.\n\tThrough rigorous mathematical proof, this paper shows that ORCA can overcome the failure cases of traditional methods in subgoal ordering and coverage.\n\tThis paper achieved superior results on different tasks and the selection of tasks is sufficient(Easy-Medium-Hard).\nPaper Weaknesses\n\tThe effectiveness of ORCA can be further clarified. ORCA is a dense reward function and implemented by dynamic programming. Thus, how about the training wall-clock time of ORCA? Is it time-consuming?\n\tThe importance of pretraining is confusing. In Sec. 4.3, the authors explain that they pretrain the agent with TemporalOT and then train with the proposed ORCA. However, Table1 and Table 2 show that ORCA without pretraining might outperform ORCA in some tasks, which is inconsistent. Does this mean that ORCA has higher quality requirements for pretraining? It's necessary to analyze when it's better to introduce pretraining, otherwise it may indicate that ORCA would be difficult to deploy in real-world.\n\tThe acquisition of the distance function d(∙,∙) can be further explored. In the paper, the authors utilize ResNet50 to extract visual features and the compute cosine similarity. In Fig. 10, it's found that off-the-shelf visual encoders produce noisy rewards in Mujoco and they clarify that this may require further online fine-tuning. Will different structures or scales of visual encoders (such as CLIP, R3M or MOCO) have an impact on ORCA in Meta-world or Humanoid? Please refer to R1 for details.\nR1: Hu Y, Wang R, Li L E, et al. For pre-trained vision models in motor control, not all policy learning methods are created equal[C]//International Conference on Machine Learning. PMLR, 2023: 13628-13651.\n\tThe applicability of ORCA may need further clarification. In Fig. 4, the authors show that ORCA performs worsen when the demonstrations are slowed down and longer than the learner trajectory. Does this mean that slow demonstrations would affect the quality of skill learning and lead to slow speed during inference?\n\tORCA could reach SOTA with only one video. So, will more demonstrations lead to better results? As mentioned in R2, the proposed method fails when multiple demonstrations are given. Thus, when more demonstrations are provided, especially when the speeds of different demonstrations are different (slow and fast), will the performance of ORCA be affected?\nR2: Sontakke S, Zhang J, Arnold S, et al. Roboclip: One demonstration is enough to learn robot policies[J]. Advances in Neural Information Processing Systems, 2023, 36: 55681-55693.\nThe writing can be improved. For example, the reference of 'Table 4.3' in Line 329 and 'Table 2' in Line 311 should be unified.", "other_comments_or_suggestions": "see Strengths And Weaknesses", "questions_for_authors": "see Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVS0BF0Xy6", "reviewer": "Reviewer_HynH", "summary": "This paper studies how to provide a policy with rewards using a single video. The paper argues that temporal misalignment may occur due to pauses in the video. The authors design an algorithm that calculates the probability that the policy's frame at time t corresponds to the video's frame at time j, thereby providing better rewards. Experiments on MetaWorld validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback on how we can strengthen our work. We answer the questions and address the concerns below.\n# Questions\n## Q: “Can ORCA improve with multiple videos?”\nORCA's performance improves with more demonstration videos. We kindly refer the reviewer to our reply to reviewer YcsA for the experiment setup and detailed results. \n## Q: “What is the reward calculation time?”\nORCA is significantly faster than TemporalOT and Roboclip, and is comparable to all other frame-level matching baselines, even for very long videos. A detailed analysis of ORCA’s runtime is in our reply to reviewer SsGe.\n## Q: \"Can ORCA enhance a pre-trained diffusion policy via RL?\"\nYes, ORCA rewards can be applied to any pretrained policy, as shown by its effectiveness at improving policies pretrained with TemporalOT. Prior work [1] has explored using an OT reward to finetune a behavior cloning policy, and ORCA can easily be extended to this framework. However, as we focus on evaluating different reward formulations, we leave different pretraining strategies as interesting future work. \n# Concerns\n## Clarification on existing baselines' performance\nOur baselines have high normalized return given temporally aligned demonstrations (OT: 0.47, TemporalOT: 0.43). However, these baselines degrade when the demonstration is misaligned. In contrast, ORCA performs the best on both aligned (0.57) and misaligned (0.50) demonstrations. We include additional baselines below, which are more competitive under misalignment. \n## Additional Baselines\nWe emphasize that we focus on learning a policy from a single video demonstration, and we consider text-based guidance to be an interesting problem for future work. Based on the reviewer’s suggestions, we introduce two new baselines.\n\n### Text-Conditioned Reward Formulation: LIV [2]\nLIV is a vision-language model that can be used to calculate rewards for a learner video given a text description. For this baseline, we follow the implementation of FuRL [3], which closely resembles our IRL setup. The table below shows the average normalized return of LIV on all easy and medium Metaworld tasks. \n\nWhile outperforming TemporalOT, LIV performs significantly worse compared to ORCA. LIV uses a single text goal to describe the task, whereas ORCA is conditioned on a sequence of images, making ORCA denser and better at capturing details.\n\n| Category | Task | GAIfO [4] (state-based obs)  | LIV [2] (same setup as [3]) | TemporalOT | **ORCA** |\n|---|---|---|---|---|---|\n| Easy | Button-press | 0.12 (0.04) | 0.00 (0.00) | 0.10 (0.02) | **0.62 (0.11)** |\n| Easy | Door-close | 0.34 (0.04) | 0.34 (0.09) | 0.19 (0.01) | **0.88 (0.01)** |\n| Medium | Door-open | 0.22 (0.05) | 0.00 (0.00) | 0.08 (0.01) | **0.89 (0.13)** |\n| Medium | Window-open | 0.27 (0.09) | 0.72 (0.19) | 0.26 (0.05) | **0.85 (0.16)** |\n| Medium | Lever-pull | 0.10 (0.03) | 0.00 (0.00) | 0.07 (0.03) | **0.28 (0.09)** |\n|  | Total | 0.21 (0.03) | 0.21 (0.05) | 0.14 (0.01) | **0.70 (0.05)** |\n\nWe also point out ORCA's strength: it works with any visual encoder, including LIV. In our reply to reviewer SsGe, we evaluate ORCA with LIV as the encoder. \n\n### IRL Baseline: GAIfO [4]\nWe include GAIfO, a classical IRL algorithm. It trains a discriminator, alongside the policy, to differentiate state transitions between the learner and demonstration distributions and uses it as the reward function. Due to compute limitations, we use a state-based demonstration instead of a video. In contrast, ORCA and all existing baselines use video demonstrations. \n\nWhile GAIfO is a competitive baseline (outperforming TemporalOT), it still performs significantly worse compared to ORCA. We hypothesize that, because there is only one demonstration, GAIfO could fixate on inconsequential details instead of estimating the task reward. \n## The experiments are not sufficiently comprehensive\nPrior works use Metaworld for all experiments [3, 5, 6], and we test all of the tasks in [5], the most closely related work. We additionally include 4 more difficult control tasks in the Humanoid environment to demonstrate the viability of ORCA in a variety of environments and conditions.\n## Related works that learn from text-video pairs\nWe thank the reviewer for pointing out related work in learning from text-video pairs, and we will update the related work to include LIV, as well as other video learning methods suggested by reviewer xJms. \n\n[1] Watch and match: Supercharging imitation with regularized optimal\ntransport. CoRL, 2024.\n\n[2] LIV: Language-image representations and rewards for robotic control. ICML, 2023.\n\n[3] FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning. ICML, 2024.\n\n[4] Generative Adversarial Imitation from Observation. ICML, 2019.\n\n[5] Robot policy learning with temporal optimal transport reward. NeurIPS, 2024.\n\n[6] Imitation learning from observation with automatic discount scheduling. ICLR, 2024.", "claims_and_evidence": "For the results of Metaworld, it seems that none of the baseline methods are effective, and the significant improvement of OCRA appears to be due to the weakness of the baselines.", "methods_and_evaluation_criteria": "In many cases, we have more than one video. The article can be extended to multiple events and situations and compared with a better baseline, such as LIV: Language-Image Representations and Rewards for Robotic Control.", "theoretical_claims": "No theory in this paper.", "experimental_designs_or_analyses": "I have reviewed Table 1 and Table 2, and both seem to have a significant advantage over the baseline ToT. However, I am not sure whether the reward proposed in the paper can be widely applied.", "relation_to_broader_scientific_literature": "The paper may propose a highly general reward function that can provide rewards based on a video.", "essential_references_not_discussed": "Some works that generate rewards based on text-video pairs rather than a single video have not been discussed. In comparison, such works seem to be more scalable.\n\nLIV: Language-Image Representations and Rewards for Robotic Control", "other_strengths_and_weaknesses": "Strengths\n1. The motivation of the paper is well-developed, with thorough and thoughtful reasoning.  \n\nWeaknesses\n1. The experiments in the paper are not sufficiently comprehensive. I would expect to see the application of the reward model in more realistic scenarios. For example, in the Libero benchmark, can the reward model be used to perform rl to enhance a pre-trained diffusion policy? \nYou can refer to PARL for preforming rl for diffusion policy.\n\nPARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback", "other_comments_or_suggestions": "No", "questions_for_authors": "This method does not seem to require training. Is it scalable? How does it handle multiple videos? Will having more videos enhance the effect?  \n\nHow is the reward calculation time? For very long videos exceeding 300 frames, will computing the distance matrix each time be very time-consuming? How much time does it take?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yLCaJ35y5Q", "reviewer": "Reviewer_YcsA", "summary": "This paper tries to address the challenge of learning sequential tasks from a single visual demonstration, particularly when the demonstration is temporally misaligned with the learner's execution. This misalignment can arise from variations in timing, differences in embodiment, or inconsistencies in task execution.\n\nThe authors argue that existing imitation learning methods, which treat imitation as a frame-level distribution-matching problem, fail to enforce the correct temporal ordering of subgoals or ensure consistent progress. They propose a novel approach ORCA that defines matching at the sequence level. \n\nThe key idea is that successful matching occurs when one sequence covers all subgoals in the same order as the other sequence. The ORCA reward function is recursively defined, considering both the learner's current state and whether it has covered previous subgoals correctly.\n\nExperiments on Meta-world and Humanoid-v4 tasks demonstrate that agents trained with the ORCA reward significantly outperform existing frame-level matching algorithms.\n\n## update after rebuttal\nI still recommend acceptance since I don't have major concerns regarding this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work clear, intuitive, and engaging. Thank you for the feedback that helps further strengthen it. We will update the paper to fix minor formatting issues, and we address the reviewer's questions and concerns below. \n# Questions\n## Q1: \"Could the authors clarify the rationale for the chosen tasks?\"\nFor Metaworld, we evaluated ORCA and baselines on all nine tasks shown in TemporalOT [1], the most closely related work. We also added one more task, Door-close, because originally there was only one task (Button-press) in the Easy category.\n\nTo evaluate ORCA's effectiveness on difficult control tasks, we include the MuJoCo Humanoid Environment, following the setup in prior work [2]. Because the original tasks (walking, standing up) lack demonstrations, we define tasks where we can generate demonstrations via interpolation from the initial to the goal joint state. \n# Concerns\n## Multiple Demonstration Videos\nSame as prior work TemporalOT [1], ORCA can handle multiple videos by calculating the ORCA reward with respect to each demonstration and max-pooling to obtain the final reward.\n\nTo study how the number of demonstrations affects performance, we ran TemporalOT and ORCA with 4, 10, and 20 temporally misaligned videos on a subset of Metaworld tasks (Door-open, Window-open, Lever-pull). These demonstrations have variations due to randomly initialized object locations, but they have the same speed. The table below reports the average normalized return with standard error. \n\nAs the number of demonstrations increases, ORCA's performance also improves. In contrast, although TemporalOT handles multiple videos in the same way and benefits from having more than one demonstration, its performance soon starts decreasing as the number of demonstrations further increases. \n\n| # Demos\t| 1           \t| 4           \t| 10          \t| 20          \t|\n|------------|-----------------|-----------------|-----------------|-----------------|\n| TemporalOT | 0.14 (0.02) \t| 0.36 (0.02) \t| 0.30 (0.02) \t| 0.25 (0.02) \t|\n| **ORCA**   | **0.67 (0.08)** \t| **1.02 (0.07)** | **1.08 (0.07)** | **1.10 (0.08)** |\n\nIn addition, per reviewer SsGe's request, we study how multiple videos of different speeds affect ORCA performance. For each task, we randomly sample one video demonstration from each category of temporal misalignment Slow (H), Slow (L), Fast (L), Fast (H), as shown in Fig. 4. Policies trained with the ORCA reward and 4 demonstrations with different speed are still able to improve performance compared to having only 1 video (0.67 -> 0.77), although demonstration videos with different speeds have slightly worse performance than videos with the same speed. \n\n| # Demos\t| 1           \t| 4 (Different Speed) | 4 (Same Speed)  |\n|------------|-----------------|---------------------|-----------------|\n| TemporalOT | 0.14 (0.02) \t| 0.16 (0.02)     \t| 0.36 (0.02) \t|\n| **ORCA**   | **0.67 (0.08)** \t| **0.77 (0.08)** \t| **1.02 (0.07)** |\n## RL Training for Non-Markovian Tasks\nWe study sequence-matching tasks where it is critical to follow the entire sequence in the correct order, thereby making the true task objective non-Markovian. ORCA and all baselines face this challenge during RL training. \n\nPrior works have explored learning policies conditioned on a belief or sequential context [3,4], and we will include these in our related work. We found it sufficient to add an additional feature to the policy input that represents the percentage of total timesteps passed. We leave exploring policy input spaces as interesting future works.\n## Clarification about applications of ORCA\nWe clarify that our focus is on learning from temporally misaligned data. This is a common problem in teleoperation data with the same embodiment, and we mention the cross embodiment setting simply to provide more context. Recent works handle temporal misalignment in teleoperation data by filtering trajectories [5,6] and manually removing pauses [7]. In contrast, we focus on designing an algorithm that can directly address issues of temporal misalignment without relying on post-processing tricks. \n\n[1] Fu et al. Robot policy learning with temporal optimal transport reward. NeurIPS, 2024.\n\n[2] Rocamonde et al. Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning. ICLR, 2024.\n\n[3] Igl et al. Deep Variational Reinforcement Learning for POMDPs. ICML, 2018.\n\n[4] Qin et al. Learning non-Markovian Decision-Making from State-only Sequences. NeurIPS, 2023.\n\n[5] Chi et al. Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. RSS, 2024.\n\n[6] Ghosh et al. Octo: An Open-Source Generalist Robot Policy. RSS, 2024.\n\n[7] Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model. CoRL, 2025.", "claims_and_evidence": "I think most of the claims in this submission are supported by evidence.", "methods_and_evaluation_criteria": "Yes, they make sense to me.", "theoretical_claims": "I briefly checked the proofs in Appendix A.1 and did not identify any significant errors.", "experimental_designs_or_analyses": "I went through all the experiments, and most of them make sense to me.", "relation_to_broader_scientific_literature": "Prior inverse RL methods, especially those leveraging optimal transport primarily focus on frame-level matching. These methods often rely on Markovian distance metrics. Other works use Dynamic Time Warping to align trajectories temporally. ORCA shifts the focus from frame-level to sequence-level matching.  This is important for tasks where the order of subgoals is critical.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths\n\n- I appreciate the structure of this paper. It begins by explaining why OT and DTW are insufficient, followed by introducing the proposed method. The examples are intuitive and effectively demonstrate scenarios where OT and DTW fail. This paper is highly engaging to read.\n- The key insight of this paper is clear and intuitive: matching should be defined at the sequence level instead of  the frame level when learning rewards for sequence-matching tasks. This idea is highly logical and resonates well.\n- The paper provides a theoretical analysis of both the limitations of existing frame-level matching approaches (OT, DTW, TemporalOT) and the properties of ORCA. Though the proofs are actually quite simple and straightforward, they still strengthen the paper's claims.\n  \n\n### Weaknesses\n\n- The abstract and introduction mention variations in embodiment, but no experiments are conducted to evaluate performance under this setting.  \n- As noted in Section 4, the ORCA reward is non-Markovian, which violates some assumptions in standard RL settings (MDP). It remains unclear if this could make RL training harder.  \n- Compared to learning from a single temporally misaligned video demonstration, a more common scenario involves multiple demonstrations for a task. Extending the proposed ORCA reward to handle multiple demonstrations appears non-trivial.", "other_comments_or_suggestions": "- The formatting at line 141 is very weird.", "questions_for_authors": "- The tasks used in the experiments are from Meta-World and MuJoCo, which include many tasks that are not utilized in this paper. Could the authors clarify the rationale for excluding those tasks? Is it because ORCA does not perform well on them? This is not a criticism, as I understand that no method excels across all tasks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c1G6XuJWgw", "reviewer": "Reviewer_xJms", "summary": "Reinforcement learning and imitation learning from a single visual demonstration is a challenging problem because the demonstration may be temporally misaligned: the demonstration and online trajectories may differ in timing. Frame-level matching is not adequate because it does not enforce the correct ordering of subgoal completion. OCRA presents a dense reward function that encourages the agent to cover all subgoals from the demonstration in the correct order. Empirical results show that ORCA significantly outperforms existing frame-level alignment methods on meta-world tasks and a simulation humanoid setup.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful that the reviewer engages closely with our ideas. We thank the reviewer for the detailed suggestions to make our paper better, and we address the concerns below.\n# Concerns\n## Clarification about paper title\nWe agree that a portion of imitation learning literature assumes access to action data. However, imitation learning is a broad category of algorithms that includes inverse reinforcement learning. IRL first estimates \"the hidden objectives of desired behavior from demonstrations\" before learning a policy to maximize the expected return [1]. Our work specifically studies imitation learning from observation, where demonstrations are just visual observations [6,7]. Without action labels, IRL is a more feasible choice. \n\nOur approach closely resembles the IRL framework because the ORCA reward is estimated from demonstrations. Unlike traditional IRL methods, ORCA does not directly learn a reward function (e.g., a linear combination of features) [2,3] or a discriminator [4,5]. Instead, it is similar to works that leverage OT [6,7] to match the learner and demonstration distributions. \n## Recent works on learning from video demonstrations\nWe thank the reviewer for sharing works outside of the IRL paradigm. We will update our related works section to discuss them. \n\nWe also clarify that all the mentioned works require demonstrations with action data, which ORCA and its baselines do not have access to. Notably, all of them except KAT [11] require more than 10k trajectories of observations and robot actions to train their policy model. KAT requires 10 robot trajectories with action labels as in context examples. In contrast, ORCA and baselines can estimate the rewards and train the policy with just one action-free demonstration.\n## RL Policy Input\nORCA's goal is to estimate rewards from a single demonstration in the observation space before learning a policy. We pose no assumptions on the policy itself. As reviewer gxJg points out, by following prior work's standard RL implementation and using a state-based policy input [7-9], we can focus on evaluating ORCA’s reward formulation.\n\nWe also trained visual policies using DrQv2 [10] on a subset of the Metaworld tasks. The state and image-based policies trained with ORCA have similar performance (average normalized return: 0.704 ± 0.10 → 0.76 ± 0.06). In contrast, policies trained with TemporalOT perform poorly, regardless of their input. \n\n| Task | TemporalOT (state-based policy) | TemporalOT (image-based policy) | ORCA (state-based policy) | ORCA (image-based policy) |\n|---|---|---|---|---|\n| Button-press | 0.10 (0.02) | 0.00 (0.00) | **0.62 (0.11)** | **0.60 (0.11)** |\n| Door-close | 0.19 (0.01) | 0.12 (0.01) | **0.88 (0.01)** | **0.88 (0.02)** |\n| Door-open | 0.08 (0.01) | 0.00 (0.00) | 0.89 (0.13) | **1.31 (0.12)** |\n| Window-open | 0.26 (0.05) | 0.28 (0.05) | **0.85 (0.16)** | **0.79 (0.15)** |\n| Lever-pull | 0.07 (0.03) | 0.00 (0.00) | **0.28 (0.09)** | **0.19 (0.08)** |\n| Total | 0.14 (0.02) | 0.08 (0.01) | **0.71 (0.10)** | **0.76 (0.06)** |\n## Clarification about environment randomization\nFollowing prior works [7,13], we use the default setup of Metaworld [12], which randomizes the object of interest locations on each rollout (see page 18 of [12] for exact randomization details). For Humanoid, we use the environment's default randomization scheme, where the initial positions and velocities of the joints are randomly initialized. \n\nWe evaluate the RL policies on 10 randomly seeded environments, and the training environment has a different seed as well. \n\n[1 An algorithmic perspective on imitation learning. Foundations and Trends® in Robotics, 2018.\n\n[2] Apprenticeship learning via inverse reinforcement learning. ICML, 2004.\n\n[3] Maximum entropy inverse reinforcement learning. AAAI, 2008.\n\n[4] A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. 2016.\n\n[5] Generative adversarial imitation learning. NeurIPS, 2016.\n\n[6] Imitation learning from pixel observations for continuous control, 2022.\n\n[7] Robot policy learning with temporal optimal transport reward. NeurIPS, 2024.\n\n[8] Graph Inverse Reinforcement Learning from Diverse Videos. CoRL, 2023.\n\n[9] What Matters to You? Towards Visual Representation Alignment for Robot Learning. ICLR, 2024.\n\n[10] Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning. ICLR, 2022.\n\n[11] Keypoint action tokens enable in-context imitation learning in robotics. 2024.\n\n[12]  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. CoRL, 2020.", "claims_and_evidence": "The three key contributions seem to be verified* and supported* (* because of the questions raised in the Methods And Evaluation Criteria and Experimental Designs Or Analyses sections).", "methods_and_evaluation_criteria": "See Experimental Designs Or Analyses for the main concern. In addition, while the paper emphasizes imitation learning from a single temporally misaligned video demonstration, the policy implementation appears to rely exclusively on state-conditioned inputs, rather than directly incorporating video-based conditioning. Given the nature of the proposed ORCA reward function—which explicitly leverages visual alignment at the sequence level—it seems more natural and potentially beneficial for the policy to be conditioned jointly on both state and visual information from the demonstration.", "theoretical_claims": "The proof seems correct. However, I think the more critical questions are raised in the following section.", "experimental_designs_or_analyses": "Related to the Q3 below: based on figure 3, section 5.1, it seems that the environment for RL training is not randomized (i.e. for the door-open task, the location of the door isn’t randomized). I am not sure if this assessment is correct. The key challenge, as identified by other visual imitation learning works, is that the condition only provides a signal of the task definition, and the test time environment configuration may be quite different. It is unclear if the current experiment setup correctly accesses the model’s capability, especially since the reward function is dependent on explicit state comparisons.", "relation_to_broader_scientific_literature": "It is broadly related to visual imitation learning, one or few-shot imitation learning literature.", "essential_references_not_discussed": "In the robot learning domain, there are a few recent works that worked on learning from a few video demonstrations. It is not necessarily the case where visual imitation learning has to be framed as online RL or IRL. See below: \n\n[1] Jain, Vidhi, Maria Attarian, Nikhil J. Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R. Sanketi et al. \"Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers.\" arXiv preprint arXiv:2403.12943 (2024).\n\n[2] Fu, Letian, Huang Huang, Gaurav Datta, Lawrence Yunliang Chen, William Chung-Ho Panitch, Fangchen Liu, Hui Li, and Ken Goldberg. \"In-context imitation learning via next-token prediction.\" arXiv preprint arXiv:2408.15980 (2024).\n\n[3] Xu, Mengdi, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. \"Prompting decision transformer for few-shot policy generalization.\" In international conference on machine learning, pp. 24631-24645. PMLR, 2022.\n\n[4] Y. Duan et al., “One-shot imitation learning,” Advances in neural information processing systems, vol. 30, 2017.\n\n[5] Di Palo, Norman, and Edward Johns. \"Keypoint action tokens enable in-context imitation learning in robotics.\" arXiv preprint arXiv:2403.19578 (2024).", "other_strengths_and_weaknesses": "N.A. See other sections for a more detailed explanation.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "Q1. Concerns regarding the title of the paper. Imitation learning has a special connotation (supervised learning from (visual)-proprio-action data), where the policy is not updated via online-RL. It seems to be the case that “online RL from single temporally misaligned video” seems more reasonable. If it is purely imitation learning, then a comparison with Vid2Robot [1] seems necessary.\n\nQ2: There’s a mismatch between the demonstration and the policy learning. It is quite weird that while the demonstration only includes visual observation, the learned RL policy has access to full state information and does not take in visual observation: “All policies use state-based input, and in metaworld we include an additional feature that represents the percentage of total timesteps passed” (pg 5). This design seems counterintuitive because state information is quite hard to extract in real-world RL/robotics applications (see Vid2Robot [1], where in certain timesteps object states may be partially/not observable). \n\nQ3: Concerns on generalization: in the setting that is being presented in the paper (figure 3, section 5.1) it seems that the environment for RL training is not randomized (i.e. for the door-open task, the location of the door isn’t randomized). Please let me know if this understanding is correct. I think the key challenge, as identified by other visual imitation learning works that are video, video-action, or state-action conditioned [1,2,3], the condition only provides a signal of the task definition, and the test time environment configuration may be quite different. It is unclear how the current version of the method can be applied to those settings. \n\n[1] Jain, Vidhi, Maria Attarian, Nikhil J. Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R. Sanketi et al. \"Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers.\" arXiv preprint arXiv:2403.12943 (2024).\n\n[2] Fu, Letian, Huang Huang, Gaurav Datta, Lawrence Yunliang Chen, William Chung-Ho Panitch, Fangchen Liu, Hui Li, and Ken Goldberg. \"In-context imitation learning via next-token prediction.\" arXiv preprint arXiv:2408.15980 (2024).\n\n[3] Xu, Mengdi, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. \"Prompting decision transformer for few-shot policy generalization.\" In international conference on machine learning, pp. 24631-24645. PMLR, 2022.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gocm95e0Vo", "reviewer": "Reviewer_gxJg", "summary": "The paper tackles the problem of imitation learning from a single video demonstration that may be temporally misaligned with the learner’s execution (e.g., inconsistent execution speed, pauses, etc.). The authors show that severe misalignments such as those that were created in their experiments (long pauses or x5 or x10 speedups in parts of the trajectory) can render prior imitation-learning techniques like Optimal Transport (OT), and Dynamic Time Warping (DTW) ineffective.\nSpecifically, they show that OT can match subgoals in the wrong order and DWT can skip goals or not complete the task in these situations..\n\nthe proposed ORdered Coverage Alignment (ORCA) computes a non-Markovian reward that measures the probability that the learner has “covered” each demonstration subgoal in the correct order. At every timestep 𝑡, the reward depends on whether the learner occupies (or closely matches) the final subgoal in the demonstration and how well it has covered all preceding subgoals.\nensuring the agent stays with the last subgoal once it is covered.\n\nEmpirically, ORCA can get stuck in local minima if the policy does not have a decent initialization. Hence, the authors first pretrain the agent with a simpler, frame-level alignment reward (specifically TemporalOT) and then switch to ORCA. This yields better coverage of all subgoals in the correct order.\n\nExperiments are performed on a set of simulated robotic arm tasks and a set of humanoid movement-to-pose tasks.\nIn all experiments, intentional severe time warping is introduced.\nComparisons are performed to:\nOT (Optimal Transport with entropic regularization)\nTemporalOT (OT with a diagonal “mask” for partial temporal alignment)\nDTW (Dynamic Time Warping, which enforces monotonic time alignment but not full coverage)\nThreshold (a simpler hand-engineered approach that tracks subgoals via a distance threshold)\nRoboCLIP (a transformer-based model encoding entire demonstration videos)\n\nResults show robustness of the ORCA approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are excited that the reviewer finds our claims well-supported and that our experiments highlight the strength of ORCA. Below, we address the reviewer's concern. \n# Concerns\n## Clarification on applications of ORCA\nWe clarify that our focus is on learning from temporally misaligned data. In robotics, this is a common problem even when collecting teleoperated demonstrations. Recent works handle the temporal misalignment by filtering trajectories [1,2] and manually removing pauses [3]. \n\nInstead of relying on these data collection tricks, we focus on designing an algorithm that directly addresses issues of temporal misalignment. And we find the results exciting: ORCA is the most performant approach given either a temporally aligned (Fig. 4 in the paper) or a misaligned demonstration (Table 1 in the paper). \n\nORCA can also handle multiple demonstrations by calculating the ORCA reward with respect to each demonstration and then performing a max-pool. We show that ORCA improves its performance as the number of demonstrations increases, even when the videos have different levels of temporal misalignment. For details and results, we kindly refer the reviewer to our replies to reviewer YcsA. \n\n[1] Chi et al. Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. RSS, 2024.\n\n[2] Ghosh et al. Octo: An Open-Source Generalist Robot Policy. RSS, 2024.\n\n[3] Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model. CoRL, 2024.", "claims_and_evidence": "The claims are well supported with toy-problem demostrations of the failure modes of the other methods.", "methods_and_evaluation_criteria": "The methods that are compared against are good representatives of methods in use and halp clarify the difficulties that the proposed method can tackle.", "theoretical_claims": "The proofs in the appendix where read briefly and did not raise any red flags at the level in which they were examined.", "experimental_designs_or_analyses": "The experiments heightlight the strength of the method (covering all goals in the correct order when the demostration data is very warped))", "relation_to_broader_scientific_literature": "The method compared against ar good candidate methods to compare against though there are certainly additional methods in this field.", "essential_references_not_discussed": "I am not familiat with particular studies should have been mentioned in this context.", "other_strengths_and_weaknesses": "The situation in which training is based on a single example trajectory with sever timewarping seems very unnatural. In most real life situations there would be several to large amounts of demonstrations and time-warping would not be so sever (especially with tele-operated robots where robot dynamics are similar in demonstration and operation). Therefore it is hard to extrapolate from the situation tackled by this paper to the more commonly encountered use cases.\n\nThe paper does well to make use of image (as opposed to state) data in calculating the reward. It also does well to perform RL training with standard implementations using a state-based representation for the agent. In this way - the focus of the evaluation is on the fitness of the reward generating mechanism.", "other_comments_or_suggestions": "None", "questions_for_authors": "The paper is very clear as it is.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}]

Author rebuttal:

Meta-review: The paper proposes a novel way to compute a "tracking score" for imitating a (possibly) temporally misaligned demonstration (e.g., a video).

There is general consensus among the reviewers about the importance of the topic, the soundness of the proposed methods, and its empirical effectiveness. For this reason I propose acceptance for the paper.

I strongly encourage the authors to integrate the discussion from the rebuttal in the final revision of the paper, in particular the new comparisons and the discussion on language-vision models
Decision: Accept (poster)