Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Models
Authors: Xingyuan Zhang
Philip Becker-Ehmck
Patrick van der Smagt
Maximilian Karl
Venue: Accepted by TMLR ()

Abstract: Pretraining and finetuning models has become increasingly popular in decision-making. But there are still serious impediments in Imitation Learning from Observation (ILfO) with pretrained models. This study identifies two primary obstacles: the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB). The EKB emerges due to the pretrained models' limitations in handling novel observations, which leads to inaccurate action inference. Conversely, the DKB stems from the reliance on limited demonstration datasets, restricting the model's adaptability across diverse scenarios. We propose separate solutions to overcome each barrier and apply them to Action Inference by Maximising Evidence (AIME), a state-of-the-art algorithm. This new algorithm, AIME-NoB, integrates online interactions and a data-driven regulariser to mitigate the EKB. Additionally, it uses a surrogate reward function to broaden the policy's supported states, addressing the DKB. Our experiments on vision-based control tasks from the DeepMind Control Suite and MetaWorld benchmarks show that AIME-NoB significantly improves sample efficiency and converged performance, presenting a robust framework for overcoming the challenges in ILfO with pretrained models. Code available at https://github.com/IcarusWizard/AIME-NoB.

Published in Transactions on Machine Learning Research (04/2025)
Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Models
Xingyuan Zhang*: 2,, Philip Becker-Ehmcke Patrick van der Smagt Maximilian Karl'
'Volswwagen Group, Technical University of Munich, Vötvvö Loránd University Budapest xingyuan. wairannnhhnnnutummcm pallap.bbcarrrehmhhoooooosssss..
Reviewed OpenReview: http::///ppaaaass....rrrrrrrrrta
Abstract
Pretraining and linetuning models has become increasingly popular in decision-making. But there are still serious impediments in Imitation Learning from Observation (ILfO) with pretrained models. This study identifies two primary obstacles: the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB). The EKB emerges due to the pretrained models' limitations in handling novel observations, which leads to inaccurate action inference. Conversely, the DKB stems from the reliance limited demonstration datasets, restricting the model's adaptability across diverse scenarios. We propose separate solutions to overcome each barrier and apply them to Action Inference by Maximising Evidence (AIME), a state-of-the-art algorithm. This new algorithm, AIME-NoB, integrates online interactions and a data-driven regulariser to mitigate the EKB. Additionally, it uses a surrogate reward function to broaden the policy's supported states, addressing the DKB. Our experiments vision-based control tasks from the DeepMind Control Suite and MetaWorld benchmarks show that AIME-NoB significantly improves sample efficiency and converged performance, presenting robust framework for overcoming the challenges in ILIO with pretrained models. Code available at https:///iithu/ http://///atttooooo//r/aaaaaiaaa -MM-aardd//MM--ooOoo
1 Introduction
We have been going through a paradigm shift from learning from scratch to pretraining and finetuning, in
particular in Computer Vision (CV) (He et al.) 2016 Radford et al.] 2021 He et al. 2022) and Natural
Language Processing (NLP) [Dvlin et al.) 2019 Radford et al.) Ouyang et al. 2022 Touvron et al. 2023a
fields due to the increasing availability of foundation models (Bommasani et al. 2021 and ever-growing
datasets. However, it is still unclear how to adapt this new paradigm into decision-makingg
particular what
type of models we need to pretrain and how these models can be adapted solve rownstream lasks. Recent.
work (Zhang et al. 2023 DeMoss et al.
2023 Sekar et al. 2020 Rajeswar et al.
2023
Hansen et al.
2023a
showed that pretrained latent space world models enable successful and efficient transfer to new tasks with
either reinforcement learning (Sekar et al. 2020
Rajeswar et al.] 2023 Hansen et al. 2023a) ) or Imitation
Learning from Observation (ILfO) Zhang et al. 2023 DeMoss et al. 2023). p. [LF (ooabi et al. 2018 2019
Baker et al. 2022 Zhang et al. 2023 DeMoss et al. 2023: Liu et al. 2022a), especially from videos (Baker
ct al. 2022 Zhang al. 2023 Liu al.) 2022a DeMoss et al.
2023 is more promising approach
this new paradigm since it does not require handcrafted reward function which is hard to define for many real-world tasks.
But there are challenges when using pretrained models in ILfO To quantify these, we introduce two new barriers, which we call the Embodiment Knowledge Barrier (EKB) and the [Demonstration Knowledge Barrier [DKB] The EKB describes the limitation of a pretrained model when confronted with novel observations and actions beyond its training experience. The DKB describes the generalisation from a limited number of
* Cooreepoondngg author.

---

Published in Transactions on Machine Learning Research (04/2025)
DKB on AIME
100
Normalised Returns
AIME
EKB
80
MBSC rediel
Model Learning
Doody [0piagi og
40-
DKB
Mix Sampling
20-
9.6
[dly Ri w..
Imilation Learning
EKB and DKB on BCO(0)
D onlline (op ag
Env
Agent
Datmnn [mg
100
Normalised Returns
BCG00)
DC (oracle)
60
Roward Labeling
Reinfarcemnnt Leerning
40-
DKB
EKB
Surrogate
Derive
90²
101
10a
10a
Reward
Model
Number of Demonstrations
Figure 1: Main idea of this paper. On the left, we plot the performance of BCO and AIME logether with their oracle versions, which remove the EKB] w.r.t. different number of demonstrations on walker-run task. For cach sctting with the same number of the demonstrations, ic. cach column, the value difference between the oracle version and the expert is the Demonstration Knowledge Barrier (DKB) while the value difference between the algorithm and its oracle version represents the Embodiment Knowledge Barrier (FKB). On the right, we present the solutions proposed in this paper to overcome the two barriers. The blue parts represent the original version of the algorithms that suffer from the knowledge barriers. Orange parts demonstrate the solution for [KB] where the agent is allowed to interact with the environment and use Donline together with Dody to update the world model. Purple parts show the solution for DKB where surrogate reward model is derived from Ddemo and used to label Dontine and then used as an RL signal for policy learning.
expert demonstrations in imitation learning (Ho & Ermon 2016). State-of-the-art approaches such as BCO (Torabi et al., 20188) and AIME (Zhang et al. 2023) typically suffer from these two knowledge barriers. First, these algorithms depend on the pretrained model to infer missing actions from observation sequences. Thus, when the model has not
specific observation before, it may not know enough about the embodiment to infer the correct action. Second, if the policy optimisation is only guided by limited demonstrations, it can lead to policy that generalises poorly, working well in some states but not in others.
To better showcase the two barriers, Figure 1 we evaluate both AIME (Zhang ot al., 2023 and BCO(O) Torabi et al. 2018 and their oracle versions w.r.t. different number of demonstrations on walker-run task. Both algorithms pretrain a model from large embodiment dataset and use that to infer the actions for the observation-only demonstrations. The oracle versions remove the need to infer the missing actions by providing the algorithm with the true actions, thus removing the EKB As can see from the figure, for each setting with the same number of demonstrations, the two algorithms are always upper-bounded by the corresponding oracle version, and the difference between them represents the EKB On the other hand, even if given the true actions of the expert, imitation performance may still be impacted by a limited number of demonstrations providing insufficient coverage of the state space. Thus, the difference between the oracle version and the expert performance represents the DKB
In this paper, we study how lo overcome these barriers to improve the performance of [LIO approaches with pretrained models, in particular of AIME. For the EKB extend the setting from offline to online by allowing the agent to further interact with the environment to gather more data to train the world model. While for the DKB we introduce a surrogate reward function to allow the policy to essentially train on more data. We demonstrate that. the proposed modifications ively overcome the two barriers and significantly improve the performance on nine tasks in Deep Control Suite (DMC) Tunyasuvunkool et al. 2020 and six tasks World YYu al. 2021
We summarise contributions as follows:
2

---

Published in Transactions on Machine Learning Research (04/2025)
We identify and thoroughly analyse the two knowledge barriers, namely EKB and DKB in the current pretraineddmodee-bssee I1.00 methods.
We propose AIME-NoB as an extension of the state-of-the art AIME algorithm by resolving the two knowledge barriers. Specifically, AIME-NoB uses online interaction with a data-driven regulariser to overcome the EKB and learn a surrogate reward function enlarging state coverage to overcome the DKB
We evaluate AIME-NoB on 15 tasks from two vision-based benchmarks and the results demonstrate AIME-NoB significantly outperforms previous tate-of-theeart methods both in terms of final perfor- mance and sample efficiency. We also conduct thorough ablation studies to show how the FKB and the DKB are overcome by the proposed modifications and how different design choices influence the performance.
2
Preliminary
We mostly follow the problem setup as described in Zhang et al. (2023). We consider a POMDP problem defined by the tuple {S, A, T,, R, O,R], where S is the state space, A is the action space, T: 5 x - S is the dynamic function, R: S + R is the reward function, O is the observation space, and Ω: S + O is the emission function. The goal is to find policy æ S + A which maximises the expected accumulated reward, or return, i.e. R(T)= = Fa-= E>== re]- Since in this work we focus on imitation learning, this oracle reward is not available to the agent. We mainly use this reward to quantify the performance of the learnt policies.
We presume the existence of three datasets of the same embodiment available our agent. The embodiment dalaset Doody contains trajectories {on, dg; 01, 41 ...} that represent past experiences of interacting with the environment. This dataset provides information about the embodiment for the algorithm to learn world model. In addition, we also allow the agent to interact with the environment to collect new data in a replay buffer Donline. Note that, although the simulator will give us the reward information, the agent is not allowed to use them, and we only use the reward for evaluation purposes. The demonstration dataset Daamo contains a few expert trajectories {oo, 01, 01,00,,000 0 of the embodiment solving a certain task defined by Raemo The crucial difference between this dataset and the other two datasets is that the actions are not provided anymore since they not observable from third-person perspective. The goal of our agent is to learn policy x from Deemo which can solve the task defined by Rdemo well as the expert Tdemo who generated Ddemo-
2.1 World Models
A World Model (Ha & Schmidhuber 2018) is a generative model which models probability distribution over sequences of observations, i.e. p(OU.T). The model can be either unconditioned conditioned on other factors, such previous observations actions. When the actions taken are known, they can be considered as the condition, i.e. Mo11r//00:--1 and the model is called embodied (Zhang et al. 2023). In this paper, we consider variational latent world models where the observation is governed by a Markovian hidden state. This type of model is also referred to as a Slate-Space Model (SSM) Karl el. al. 2017 Hafner el. al. 2019b a Beeckr-Ehmcck
et al. 2019 Klushyn et al. 2021). Such a variational latent world model involves four components, namely
encoder 2, fo(or).
posterior 8, ~ (00(s))01-1,,
decoder ot po(oo)ss),
prior St ~ p0(ss)s--110)0111))
fo(o:) is the encoder to extract the features from the observation; g0(s(18t-1,44-111 and po(ss|s:-1,0 are the posterior and the prior of the latent state variable; while (o,))) is the decoder that decodes the observation distribution from the state. & and 0 represent the parameters of the inference model and the generative model respectively.
Typically, the model is trained by maximising the Evidence Lower Bound (ELBO) which is a lower bound of the likeiioi,,oo,, evidence, of the observation sequence, i.e. log 20(01:77 00. -1). Given a sequence of
3

---

Published in Transactions on Machine Learning Research (04/2025)
observations, actions, and states, the objective function can be computed
ELBO
Jt Jk log po(oo)ss) DKL kollopl
(1)
1=1
The objective function is composed of two terms: the first term J is the likelihood of the observation under the inferred stale, which is usually called the reconstruction loss; while the second term JKL is the KL divergence between the posterior and the prior distributions of the latent state. To compute the objective function, we use the re-paramctorisation trick (Kingma & Welling 2022 Rezende al. 2014)
autoregressively sample the inferred states from the observation and action sequence.
In summary, world model is trained by solving the optimisation problem
0 argmax [(o,a)~Duode a [EBBO]
(2)
0.0
2.2 Action Inference by Maximising Evidence (AIME)
AIME is state-of-the-art algorithm that uses pretrained world model to solve [LfO in an offline setting. Specifically, it uses the pretrained world model as an implicit inference model by solving for the best action sequence that makes the demonstration most likely under the trained world model. The imitation can be done jointly with the action inference using amortised inference and the ro-paramcterisation trick by solving the following optimisation problem
argmax Eg Dammo cammo..rrgg Brugge oa aa ELBOO,
3)
where s is the parameter for policy (arls)) The resulting objective is very similar to Equation (2) with subtle difference of the sampling path. That is in the new objective, only the observations are sampled from the dataset and both states and actions are sampled iteratively from the learned model and the policy, respectively.
3 Methodology
In this section we will analyse the EKB and DKB for AIME. Based on the analysis we introduce a solution for each knowledge barrier and combine them into AIME-NoB, where NoB stands for No Barriers. The general framework of the solutions shown in Figure [ and the pseudocode of AIME-NoB provided in Algorithm 1 Before diving into the analysis, we first formally define EKB and DKB with
EKB = R( - R(TT))); )+ DKB R(Tdemo) R(TTw)).
(4)
(5)
Let D :== (Ddemo: Doody, Donline), then
~x = argmax Jpolicy (Prammo (aclo1.)), Tu (arloss), represents the optimal policy parameters for maximising Jpolley with the oracle.
x = a Jpolicy (go- (o.lo1:)), ne (atlos:), D) represents the optimal policy parameters for max- imising Jpolicy with the learned model.
Jpolicy (g(aa((((()))) (oge), D) is the learning objective for the policy depending action-inference model g(ar)) For behaviour cloning based methods like BCO and AIME, it is essentially equivalent to E D
Damno Σ, (wt..gglllloeeo [ ((o))os))) [
6- argmaxx Jmodel Dbody, D Douline: Φ) is the optimal parameter for maximising the model-learning objective J Jmadet-
4

---

Published in Transactions on Machine Learning Research (04/2025)
Prammo (a1(01:)) is the ground truth of empirical distribution of the demonstration data that serves an oracle.
Based the definitions, can clearly see that EKB is caused by the imperfect inference of the unknown actions of the observation-ons demonstrations, which can be characterized by (Preoo (41:--1(91-r) 9 (#1:=1199:r)); while DKB is caused by the limited learning signal that Jpolicy offers as it is only defined on the data points in the demonstration dataset. In Appendix J we propose some upper bounds for both [EBB and [KB] based on these insights. In the rest of this section, we will focus on practical solutions for both of the barriers.
3.1 Overcoming the EKB
To reduce the EKB, we need to bring the learned inference model closer to the oracle, i.e. to minimise Eammo D (01:T-1/01:r) 90- ((:r-1199:r))- The most natural way to do so is to allow the agent to further interact with the environment, similar to how we humans practice for a novel skill. New experiences can minimise the error in the pretrained model near the policy Ty and enhance task-specific embodiment knowledge. Torabi et al. (2018) proposed modified version of BCO(0) called BCO(a) introducing such an interaction phase. However, empirical results show did not overcome the EKB as a gap remains with the BC oracle when the environment is complex. In fact, as we will show in the following, the idea of adding online interactions is not straightforward to successfully implement in practice.
As shown in recent works in Offline RL, continuing training of an actor-critic from the offline phase in the online phase requires certain measures to combat. the shift of objective (Lee et al. 2022 Ball et al. 2023
Nakamoto et al. 2023 A similar story also applies when extending AIME from purely offine to online. The most dominant problem we found overfitting to the newly collected dataset.
As the training progresses alternating between data collection, model training and policy training, in the early phase of training there are only very few new trajectories available for training the model. Because the world model is highly expressive, it may overly favour similar trajectories, especially the action sequence, in the new data, leading to a high ELBO, Normally, this may not be a big problem since, eventually, more and more data will be collected to combat this overfitting. But since AIME also depends on the ELBO to train the policy, it quickly causes the policy training to diverge. That is to say, when the model is extensively trained on a small amount of data, not only maximises the conditional likelihood (gg()))((0TT-1) but also maximises the marginal likelihood log(-lao:r-1; which diverges the likelihood- based action inference process.
In order to address the overfitting issue, we need regulariser for model learning. Instead of designing ad-hoc methods to regularise the model in the parameter space, we adopt data-driven approach. From the model's perspective, the overfitting is caused by a sudden shift of the training data from large and diverse pretraining dataset to a small and narrow replay buffer. To avoid this sudden shift, the most straightforward method to just append new data to a replay buffer that is pre-filled with the pretraining dataset. However, this causes data efficiency problems since the newly collected data relatively little compared to the big pretraining dataset. Uniformly sampling from the joint replay buffer hence overly limits usage of the new data. Instead, suggest sampling separately from both datasets. We modify the model-learning objective Jmodel in Equation (2 to
6², 0 = argmax On (body))DDood.oeaa [ELBO] + (1 - a.)Baoaaa Dontmo orggo ELBOJ.
(6)
0.0
The amount of data sample from the pretraining dataset is controlled by dyper-paranen a, which represents how much regularisation we put upon the model. Here we mainly consider setting a 0.5, so that we sample the data evenly from both datasets.
This finding contradicts Rajeswar et al. (023) and Hansen et al. 2023a). where the pretrained world models do not need such a data-driven regulariser. We conjecture that unlike AIME, these approaches mainly use their world models purely as generative models to predict states and rewards given action sequences, which is only indirectly influenced by overfitting the EL.O
5

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm AIME-NoB
1: Input: Embodiment dataset Dhody, Demonstration dataset D Diees Pretrained world model parameters
o, 0, Surrogate reward model Ry. Regulariser ratio a, Value gradient weight 3, Batch size B
2: Initialise policy and critic parameters t randomly.
3: for 1 to policy pretraining iterations do Draw a batch of demonstrations 01 Demoo Update policy parameters da with Equation 3
end for
7: Initialize Donline + 0.
8: for i= 1 to Environment Interaction budget do
9: Collect a new episode {o1:r, d1:r} with the current policy Tg.
10: Update the surrogate reward model R2 if needed, e.g. for AIL. Estimate reward using surrogate reward model rif Ro(ow.)
11:
12:
Append {OLTT 01:7, rT.T to Donline
13:
# Update world model
14:
Draw a B samples boody ~ Doody
15:
Draw (1-a) B samples bouline Donliue
16:
Define combined batch = bbody U bontine
17:
Finetune model with batch using Equation
6
18:
# Update policy
19:
Sample batch from Ddemo
20:
Updale policy parameters da with Equation 9
21:
Update value function parameters with Equation S
22:
end for
3.2
Overcoming the DKB
Based on the discussion from the previous sections, the straightforward way of overcoming the DKB is also to increase the number of demonstrations available to the agent. However, expert demonstrations are difficult. and expensive to collect. Increasing the size of the demonstration datasel not always feasible real-world applications. Alternatively, we can modify the policy learning objective Jpolicy in Equation (3) to reduce the
generalisation gap. In order to propose practical solution, first need to look deeper into what is the real cause of the DKB
The policy-learning part of the AIME algorithm is essentially behaviour cloning, and it is only conducted on the demonstration dataset. So for the states covered in the demonstration dataset, the policy is given clear guidance about what to do, while for other states, the behaviour is undefined. AIME solely relies on the generalisation abilities of the learned latent state and the trained policy network to extrapolate the correct behaviour. In particular for small demonstration datasets, this can be unreliable or even impossible. Therefore, if we were able to enlarge the space of the covered states, we should reduce the DKB Ross et al.
2011
Based on these insights, we propose to introduce a surrogate reward providing a guiding signal for the agent on the replay bufler dataset, iee. ror = (on:)). Using this reward, we train the policy with a dreamer-slyle actor-critic algorithm based on imagination in the latent space of the world model (Hafner et al.) 2019a). order to do this, we first need to modify the reconstruction term in Equation (1) by adding an extra term for decoding the surrogate reward, i.e. log log Then, we further train value estimator Ve using (D(A)-return estimates, i.e.
(8) = (1 A) Σ
x
(st)
(7)
E
nm1
with (8x) =
sur rs'
+ + Ve Ve(
#=

---

Published in Transactions on Machine Learning Research (04/2025)
Using this estimate, optimise our value function by minimising the MSE, i...
<" = argmin((e(ss) (+:)).
(8)
As is common practice, we use a target value network with parameters ç to stabilise training, whose parameters are updated using Polyak averaging with a learning rate T in every itcration.
Using this value estimate, we extend the policy-learing objective Spolicy of Equation
3
argmax Bow Ddamo, 4/0..,...
[EBBO] + (o.a) Poanaaaaaan ao wag.s (V (V
where B is a hyper--praane parameeree for balancing the two terms. We set B = 1.0 by default in this paper.
There could be many choices to derive this surrogate reward model. In this paper, we consider three different types of surrogate reward, namely AIL, OT and VIPER. AIL (Ho & Ermon 2016 Torabi et al., 2019 uses adversarial training to learn a discriminator to tell whether an observations is generated by the expert, and uses the score from the discriminator as the reward. OT (Papagiannis & Li 2023 Haldar et al. 2022 uses optimal transport theory to measure the distance between given trajectory and group of expert trajectories, and uses the negative distance as the reward. VIPER (Fscontrela et al. 2023 learns video prediction model from the demonstration datasets and uses the likelihood from the trained model as the reward. The detailed explanation of the three variants are in Appendix A We use AIL as the default variant. for AIME-NoB.
4 Experiments
the experiments, we aim to answer the following questions: Q1: How does the proposed ALME-NoB compare with statc-of-the-art methods on common benchmarks? Q2: How well do the proposed modifications overcome the EKB and the [KB] Q3: How do different design choices, components and typer-parameten influence the results? In order to answer these questions, we design our experiments on [DMC and Meta World benchmarks. The main results and the setup are described in the following sections, while some additional interesting results are presents in Appendix !
4.1 Datasets and Tasks
For the DMC benchmark, we choose nine lasks across six embodiments lollowing Liu et al. (2022ab and use the same published dataset (Haldar et al.) 2022) as the demonstration datasets. Each dataset contains only 10 trajectorios to reflect the scarcity of expert demonstrations. For the embodiment dataset, in order not to leak the task information from the pretraining phase, we follow Rajeswar et al. (2023) and run a Plan2Explore (Sekar et al. 2020) agent for each embodiment with 2M environments steps and use its replay buffer as the embodiment dataset. Different them taking the model directly from the Plan2Explore agent the pretrained model, we follow Zhang al. (2023) to retrain the model for 200k gradient steps to get better model. When evaluating the performance of the learned policy on each task, we rollout the policy 10 times with the environment, and report the mean return.
World benchmark, we use the data from Hansen et al. 2023a). The embodiment dataset was created from the replay buffer datasets. The open-sourced replay buffer datasets contain 40k trajectories for each of the 50 tasks with only state information. In order fit to our image obscrvation setup, we render the images by resetting the environment to the initial state of each trajectory and then executing the action sequence. The details can be found in Appendix
For vision-
B
With respect to the embodiment dataset, following the idca of not lcaking too much about the task information, inspired by the common practice in oflline RL benchmarks (Fu el al. 2021), we use the first 200 trajectories from each replay buffer and form a dataset with 10k trajectories in total. We call this dataset MW-mt50 and we use it for the benchmark on MoraW to compare AIME-NoB with other algorithms. To further study the mut-of-distributinn
transfer ability of the pretrained model, we follow the difficulty classification of the tasks from Seo et al. 2022a) and only the 39 easy and medium difficulty tasks to generate the datasets
7

---

Published in Transactions on Machine Learning Research (04/2025)
Aggregated results on DMC
Aggregated results on MetaWorld
1.0
1.0
AIME-NoB (Ours)
AIME
Normalised Score
0.8
0.8
BCO(a)
AIME-NoB (Ours)
Success Rate
0.6
AIME
0.6
PatchAIL
BCO(e)
0.4
OT
0.4
PatchAIL
0.2
0.2
0.0
0.0
8
10
1
4
Env steps (x10²)
Env steps (x10²)
Figure/ 2: Comparing AIME-NoB with other algorithms. The figures show aggregated IQM scores on DMC tasks and 6 MetaWorld tasks. All the algorithms evaluated with 5 seeds each task and the shaded region representing 95% CI.
and use the 11 hard and very hard tasks as hold-out tasks. We uniformly sample 250 trajectories from the first 10k trajectories from each of the 39 tasks and form a dataset with 9750 trajectories in total. We refer this dataset MW-mtt99
For evaluating the algorithms, choose four hard or very hard tasks, namely disassemble, assembly, hand- insert and push; and two medium difficult tasks, namely sweep and hammer. As for the demonstration datasets, we use the single-task policies open-sourced by TD-MPC2 and collect 50 trajectories for each task. We ensure that every trajectory in the demonstration dataset is successful. For evaluation on each Meta World task, due to the noisy nature of the task, we rollout the policy 100 times with the environments, and report success only if the very last time stop of an episode marked successful by the environment (following Hansen al. (202aa)
4.2
Implementation
For the world model, we use the RSSM architecture (Hafner al. 2019b) with the Syper-parameters in Hafner et al. (2019a) for DMC tasks. In addition, to make the training more stable. For Meta World, since the visual scene is more complex, size model from Hafner et al. (002). but still with the continuous latent variable to be aligned with other models used in this paper. The policy network is implemented with two-layer MLP, with 128 neurons for
the KT. Balancing trick from Hafner et al. (2020)
use the M
each hidden layer. All the models are trained with Adam optimiser (Kingma & Ba 2017 More details on hyper-parameters are in Appendix
4.3 Results
Q1: How does AIME-NoB compare with state-of-the-art methods on common benchmarks? We compare the ATL. variant. of ATMF-NoB with the representative state-of-the-art algorithms:
AIME [Zhngg et al.
2023 represents the base algorithm that we improve upon. The algorithm uses pretrained world model offline, and suffers from both EKB and DKB
BCO(a) Torabi et al. 2018 an online extension of the popular BCO(O) algorithm, which represents another line of methods that use pretrained models. The online interactions in BCO(a) potentially overcome EKB from BBC(()).
PatchAII. [Luu et al. 2022a
represents the Generative Adversarial Imitation Learning (GAIL.) styles
of algorithms (Ho & Ermon
2016
Torabi al. 2019
OT Haldar al.
2022
represents the trajectory matching based algorithms.
8

---

Published in Transactions on Machine Learning Research (04/2025)
Efficiency of the Barrier Solutions
Training Curves of Return
800
800
600
AIME-NoB
AIME-NoEKB
Return
Return
MBBC (oracle)
AIME
400
N-1
N=
expert
N=5
200
random
200
N-10
N=20
expert
0
10
10
10
103
2
10
Number of Demonstrations (N)
Env Steps (x10*)
Figure 3: Performance of AIME-NoB, AIME-NOEKB, MBBC, AIME w.r.t. different number of demonstrations on the walker-run task. For AIME-NoB, we do not show the result for more than 20 demonstrations since it is already saturated to the expert. All results are averaged across 5 seeds with the shaded region representing a 95% CT.
The results of the benchmark are shown in Figure 2 score and 95% CI when aggregating over all the tasks in the same suite. For each DMC task, the score is normalised using the average return of the expert, while for each World task directly report the success rate. From the results, can clearly see AIME-NoB achieves better sample-efficinncy and final performance on both of the benchmark suites comparing with all other algorithms. Benefiting from the pretrained world model, AIME-NoB lypically can reach near expert performance within 100k environment steps on DMC tasks. This even matches the performance in Rajeswar et al. (2023) where the true rewards are available. For the complete results of each individual task please refer to Appendix
We follow Agarwal et al. (2022) to report the IQM
Q2: How well do the proposed methods overcome knowledge barriers? In order to show how well AIME-NoB overcomes the two knowledge barriers, conduct the same experiment in Figure by providing the agent with different numbers of demonstrations on walker-run. We also run additional variant coined ALME-NoEKB where we only apply the solution for the EKB The result is shown in Figure 3 As we discussed before, MBBC as an oracle method that circumvents the EKB is an upper bound for AIME. AIME-NoEKB matches and even slightly outperforms MBBC, which implies the proposed solution completely overcomes the EKB] The fact that slightly outperforms MBBC is a bonus of the model choice fine-tuning the latent variable world model improves the generalization of the latent space which mitigates the [KB AIME-NoB, which further addresses the DKB matches the expert performance even when given only demonstration. This showeases that the DKB has also been completely overcome using the surrogate reward. The difference between different number of demonstrations is mainly on the sample-effciieny side, where the more demonstrations we have the less online interactions we need to attain the expert performance.
Q3.1: Which variant of AIME-NoB performs the best? We run all the three AIME-NoB variants with different surrogate rewards, i.. AIL,, OT and VIPER, on the two benchmark suites and aggregate the results. From Figure 4a and 4b] we can see the AIL variant of AIME-NoB generally performs the best on both suites. On the DMC suite, all the three variants managed to converge to the same final performance, but the AlL variant is slightly more sample efficient. On the MetaWorld suite, the distinction is larger between different variants, which highlights the supremacy of the AIL variant. We hypothesise that AIL is the only variant that directly adapts the surrogate rewards online by training the discrininator. For the other two variants: VIPER has zero adaptivity since the VIPER model is fixed during the online training; OT has an indirect adaptivity since it depends on the image encoder from the world model which get finctuned during online learning. This adaptivity enables AIL to capture more pertinent signals during the imitation process. Based on these results, we choose the AIL. variant. the default implementation for ATME-NoB.
9

---

Published in Transactions on Machine Learning Research (04/2025)
Different variants on DMC
Different variants on MetaWorld
Different datasets on Meta World
1.0
1.0
1.0
AIME-NoB w/ AIL
MEME-NoB w/ ml50
Normalised Score
0.8
0.8
AIME-NoB W/ OT
0.8
AI NoB mt39
AIME-NoB w/ VIPER
Success Rate
0.6-
0.6
0.6
Success
0.4
0.4
0.4
AIME-NOP NoB
0.2
AIME- NoB w/ OT
0.2
0.2
AIME-No8
0.0
0.0
0.0
0
2
6 Env steps (x10%)
4
8
10
1
2
2
Eny steps (x10²)
Env steps (x10²)
(a) Different variants on DMC
(b) Different variants on Meta World.
(c) Different datasets on Meta World.
Figure 4: Ablations of different variants of AIME-NoB and choices of the embodiment datasets. All the algorithms are evaluated with 5 seeds with the shaded region representing 95% CI.
Ablation on AIME-NoB
Ablation on AIME-NOEKB
800
0.8
1.0
600
Action MSE
Return
e 0.0.
0.6
Return
600
8- 0.0
400
0.25
Action
0.5
0.4
== 0.5
200
0075
200
append
0.2
append export
export
0,00
0.25
0.50
0.75
1.00
10
0.00
0.25
0.50
0.75
1.00
Env Steps (x10²)
Env Steps (x109)
Env Steps (x10F)
Env Steps (x10F)
Figure 5: Ablations of replay ratio a on walker-run task. AIME-NoB is running with 10 demonstrations, while ATMF-NoFKB is with 100 demonstrations. Action MSE is only shown for the first 105 env steps. All results are avcraged across 5 seeds with the shaded region representing 95% CI.
Q3.2: Which dataset can pretrain the better world model for AIME-NoB? The quality of the pretrained models naturally depends on the quality of the pretraining datasets. In order to understand what characteristics of the datasets influence the performance, we train all the AIME-NoB variants with two different models pretrained separately on the MW-mt39 datasets and MW-mt50 datasets. The aggregated result with each dataset is shown in Figure Ac The result demonstrates that although the size of the datasets is roughly the same, the model pretrained on MW-mt50 offers better results. This may imply covering diverse behaviours and objects is more valuable than knowing the expert, for example the mt50 dataset contains objects to be assembled while the mt39 does not.
Q3.3: How does different data regulariser ratio a influence the performance? We ablate the regulariser ratio a from [0.0, 0.25, 0.5, 0.75). Further, we compare to a simple append version where the online dataset is appended to the embodiment dataset and treated as a singular dataset for sampling. The append version can be also understood in this experiment as having an inverse proportional schedule of a from 1.0 to 0.66 during the course of training. To isolate the effect on the EKB we train both AIME-NoB and AIME-NOEKB. We show both the training curves of the return and action MSE, which is the MSE between the inferred actions and the true actions, in Figure 5 For AIME-NoEKB, regulariser, i.e. set > 0, we get reliable improvements of returns over the course of training. But if we disable the regulariscr by setting = 0, the return exhibits high variance. The cause is clear by looking at the action MSE. For a = O the action MSE diverges in the beginning and cannot recover. For AIME-NoB, the story is more complicated. While the action MSE still diverges in the beginning when a < 0.5, the surrogate reward can guide the policy back on track and even achieves lower action MSE after recovery. In this case, a small a helps the algorithm make more use of the online dataset, resulting in higher sample efficiency. Although we keep the choice of a with fixed value for ATMF-NoB in the light of simplicity, there should be
long we enable the
10

---

Published in Transactions on Machine Learning Research (04/2025)
Ablation the Pretrained Model
Ablation of Removing AIME
Ablation on the scale B
800
AIME-NoB
700
800
Return
600
AIME-NoB mm
AIME-NoB
Retur
600
E
600
500
S-0.1
=
400
expert
400
AIME-NoB
400
1=10
8 10.0
200
AIME-NoB if
300
10000.0
AIME-NoB
200
expert
AIME-NoB -p-f
100
expert
2
4 Env Steps (x10%)
8
10
2
8
10
8
10
Env Steps (x10F)
Env Steps (x10²)
(a) Effect of the pretrained world (b) Ablation of removing AIME loss (c) Ablation for the choice of the model. -m means removing the pre- from LoME-NoB. -p means removing weight the value gradient loss 3. trained world model while -d means from pretraining while -f means remov- removing the pretrained dataset.
ing from finetuning.
Figure Ablations of pretrained models, AIME loss and the weight of the value gradient loss B. All the algorithms are evaluated with 5 seeds with the shaded region representing 95% CT.
a smart scheduling for each task to strike the balance between stability and sample-efficiene. However, we think finding this optimal strategy beyond the scope of this paper.
Q3.4: How much benefit do we get from the pretrained world model and the dataset for pretraining? One of the advantages of ATME-NoB over popular II.10 algorithms is that it can make of pretrained models and pre-collected datascts. Thus, want to investigate how much AIME-NoB benefits from having a pretrained model and pre-collected dataset. We rerun AIME-NoB on walker-run without the embodiment dataset and without the pretrained world model. As we can see the result from Figure 6a without the pretrained the world model, the sample-officiency is largely affected. Between the two components, the model is more important than the dataset.
Q3.5: How do different values of the gradient loss weight influence the performance? We set the weight from 10.0, 0.1, 1.0, 10.0, 100.0] and plot the results in Figure [c] As the result shows, without the surrogate reward, i.c. 3 0, the agent cannot reach expert performance due to the DKB Having small 3 slows learning progress toward convergence. On the other hand, setting / to much larger value will improve the sample-efficieccy without causing instability. For the sample efficiency, since we only have 10 demonstrations, the DKE dominates over the EKB as shown in Figure 3 Thus, having a larger 3 will speed up learning. terms of stability, as we discussed in 3.2 ATME loss and the value gradient loss operate on different regions of the environment states. This could make their influence on the policy independent of each other.
Q3.6: Is surrogate reward all you need? Given the results above that AIME-Nob can work well even when we lower the effect of the AIME loss by making either worse action inference, i.c. set to low value, or strengthen the value gradient loss, i.e. set 3 to high value, a natural question to ask is whether the AIME loss is still needed in AIME-NoB or whether surrogate reward only is enough to solve the imitation learning task. The AIME loss used in two places the AIMÉ-NoB algorithm both for pretraining the policy offline and finetuning the policy online. We compare AIME-NoB with variants that removes AIME from these parts. Form the results shown in Figure 6b removing AIME loss from either pretraining finetuning will lower the sample efficiency. Removing from both phases causes instability and convergence issues during training. Thus, AIME loss is still crucial and cannot simply be replaced by surrogate rewards.
5 Related Work
DeMoss et al. 2023 Li et al., Baker al. 2022. Zhang et al. 2023 Liu al. 2022a) bocomes more popular recent years due their potential to utilise internet-scale videos for behaviour learning. Most of the previous works Torabi et al.,
Imitation Learning from Observation. ILfO
Torabi et al. 2018 2019
2023
2018
2019:
Li et al. 2023: Kidambi et al. 2021 study the problem only with the true state observation.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Recent works (DeMoss et al. 2023 Baker et al. 2022 Zhang al. 2023 Liu et al. 2022a have started to
shift toward image observations more general setting. Only few works (hhagg et al. 2023 Torabi et al.)
2018) can leverage pretrained models. Our work is continuation of this journey and further emphasise the performance benefit from pretrained models.
Pretrained Models for Decision-Makinn. Inspired by the tremendous progress made in recent. years CV and NLP fields with the power of pretrained models, the decision- community is also trying to follow the trend. Most recent works focus the use of Large Language Model (LLM) for decision-making.
A prompted model is used for producing trajectories and plans (Chen et al.
2024
Huang et al.) 2022
Ahn
ct al.
2022
Di Palo et al.
2023). code (Vemprala et al. 2023 Liang ct al. 2023
Singh et al.; 2022
Chen
et al.
2023
Huang et al.
2023) or for modifying the reward
Ma et al. 2023 Mahmoudieh et al. 2022
There are also other people studying the benefit of pretrained visual models for visuomotor tasks (Shah
& Kumar 2021 Majumdar et al. 2023 Hansen et al. 2023b Parisi et al. 2022) while others try to train
large policy networks directly with transformors (Vaswani et al. 2017) and huge datasets (Brohan et al.
2023: Brohan et al.] Reed el. al. 2022). However, there only little attention being put on pretrained world
models (Zhang al. 2023 Rajeswar et al., 2023 Sekar et al.] 2020), which are natively developed by the model-based decision-making community and perfectly fit into the pretraining and finctuning paradigm. Our
work explores this overlooked domain and showcases its potential.
6 Discussion
In this paper, we identify two knowledge barriers, namely the EKB and the DKB which limit the performance of stato-of-the-art ILfO methods using pretrained models. We thoroughly analysc the underlying cause of each barrier and propose practical solutions. Specifically, we propose to use online interaction with a
data-driven regulariser to overcome the EKB and surrogate reward labelling to reduce the DKB Combining these solutions, we propose AIME-NoB and showcase its supreme efficiency compared to SOTA [LfOO methods. Our ablation studies show how each knowledge barrier is addressed by the proposed solution and how different design choices influence the performance.
However, there are still limitations for the scope of this study. First, this work is mainly an empirical study. Some theoretical results could enhance the understanding of the knowledge barriers. Second, as suggested by the ablation study, a careful design for a schedule of a could further improve the sample efficiency. Third, due to the high demand computing resources, we only study the pretrained world model on a rather small scale, i.c. the biggest model has only 20M paramcters. It will be interesting to study how well the algorithm scales larger models. Lastly, although we have shown a single pretrained world model can be used for multiple tasks, the power of pretrained world models are not fully realized. It would be interesting to how world models can be used to train a multi-tasks policy. These limitations provide directions for future works.
We hope our work can shed some light on the future development of LLfO method and bring more attention to the great potential of pretrained world models.
Acknowledgments
We would like to thank Botond Cseke for his help in reviewing the mathematical aspects of this paper.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice, Jamuary 2022. URL http://arxiv.org/
abs/2108. 13264
Michael Alm, Authony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishhan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Tharz, Brian Ichter, Alex Trpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes,
12

---

Published in Transactions on Machine Learning Research (04/2025)
Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do As Can, Not As Say: Grounding Language in Robotic Affordances, August 2022. URL http://arxiv www.xx...gg/ab//.. org/abs/2204.000m 01699
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos, June 2022. URL
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient Online Reinforcement Learning with Offline Data. In ICML, May 2023. URL htttp:://axxiv.orgabbbba 2302bo223822029
Philip Becker-Ehmcle Jan Peters, and Patrick van der Smagt. Switching Linear Dynamics for Variational Bayes Filtering. In Proceedings of the 36th Internalional Conference on Machine Learning, pp. -622. PMLR, May 2019. URL htttps://proceeddnngg mlr, http://////eeeeeeeeeeeeeeeeeeeee
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bolg,, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsca Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel F. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Gcoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Lí, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitrion, Joon Sung Park, Chris Piech, Fva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models, August 2021. URL http://arxii http///axvvv..o//o org/ 0ttp//218.0700
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Tchter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang- Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Portsch, Kanishka Rao, Krista Reymaun, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision Viston-Languuge-AAtte Action Models Transfer Web Knowledge to Robotic Control.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmouth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang- Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Claylon Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics Transformer for World Control at Scale. In Robotics: Science and Systems XIX. Robotics: Science and Systems Foundation, July 2023. ISBN 978-0-9923747-9922 doi: (..5507/RRS.200 X22.025. http://///www...ooooooooooeeeee/
13

---

Published in Transactions on Machine Learning Research (04/2025)
Yongchao Chen, Rujul Gandhi, Yang Zhang, and Chuchu Fan. NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Netural Language Processing, pp. 15880 15903, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 20.18633/v1/2223 http://222.eemmmmmmmee emnlp-main 985. URL htttps://aclantheen .rgellog.oorr//// 2023. emnlp-maing malppmmai...ee 985
Yongehao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, and Chuchu Fan. AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers, March 2024. URL
http://arxiv.orgg/ httg/aab/23300 06531
Branton DeMoss, Paul Duckworth, Nick Hawes, and Ingmar Posner. DITTO: Offline Imitation Learning with World Models, February 2023. URL http:://axivy org/ wwwg/abs/2300... 230000030080
Jacob Devlin, Ming- Mei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirec- tional Transformers for Language Understanding, May 2019. URL
04805
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller, Towards Unified Agent with Foundation Models, July 2023. URL http::/arxiv.orr org/ org/abs/ abs/ 2307.09663
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video Prediction Models as Rewards for Reinforcement Learning. Tn Thirth-Seventh Conference Neural Information. Processing Systems, November 2023. URL https
//openreview.oo //openreviee.net//foooooe wwt.mn//ooorrmiiici/M1A1AAA.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for High-Resolution Image Synthesis. In 2021 IEEE/CVE /CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12868 12878, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-68544-509-2. doi: 10009/CVP444437.7.
URL https://ieeexpllrrs ieee. httg//ocumenn/95779111
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, February 2021. URL http://arxiv.oggabb/////a 07219
David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cosa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL htttps://proceeding:
Paper pdf
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learning Behaviors by Latent Imagination. In ICLR 2020, September 2019a. URL https://openrevier. net/
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, pp. 2555 2565. PMLR, May 2019b. URL https://prroceeding.. wwwpaammmmmpprreeee.e www.mml......//.....m.m hafner19a, html
Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with Discrete World Models. In International Conference on Learning Representations, September 2020. URL https
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through World Models, Jamuary 2023. URL http:://rxxiy wttp://wwwwwg////........ 04104
Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and Match: Supercharging Imitation with Regularized Optimal Transport. In 6th Annual Conference on Robot Learning, August 2022.
URL http::://ppnneeriee..com http://ppaaaaiieeeeec/commmmataa
14

---

Published in Transactions on Machine Learning Research (04/2025)
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, Robust World Models for Continuous Control. In TCLR 2024, October 2023a. URL
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongshou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On Pre- Pre-Training for Visuo-Motor Control: Revisiting from-intttom Baseline, June 2023b. URL
05749
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770 778, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-8855 http://iesexplors leeee.rreeeeee..ee.e.eee.eeeee.. org/ wtt..e..o..oocmmeeet////o0//.... document//7800559/
doi: 10.1199/CVPR.20166 URL
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 15979 15988, New Orleans, LA, USA, June 2022. IEEE. ISBN 978-1-66546-94 doi: 10.1109/CVPR52088..22 202...015.... URL https://1eeexplort ieee. orgg/dccument/88799006
Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. In Advances in Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://prroceeeiinn
files/ http:///pwpp///ooocccccccc/ccaaa
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models, July 2022. URL http://arxiv. wwwpxiivorrg/bba.. 2tt.//2277.00000
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. In 7th Annual Conference on Robot Learning, August 2023. URL https://openreview net/forum?id=9 20pr__008_00
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data. In International Conference Learning Represenlations, 2017. URL https::/ httpap////pppeeeeeeeeee////r///a
Rahul Kidambi, Jonathan Chang, and Wen Sun. MobILE: Model-Based Imitation Learning From Observation Alone. In Advances in Newral Information Processing Systems, volume 34, pp. 28598- 28611. Curran Associates, Inc., 2021. URL
.tml
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd International Conference for Learning Representations, San Diego, January 2017. doi: 10. 20.48850//a2ivv.1122..m URL
httpp///axivv. wrgg abs/ 111..112.65011 6980
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes, December 2022. URL http:
//arxiv.org// hatp//aaa///1/..e 6114
Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cseke, and Patrick van der Smagt. Latent Matters: Learning Deep State-Space Models. In Advances in Neural Information Processing Systems, volume 34, pp. 10234 10245. Curran Associates, Inc., 2021. URL httpps:/pproceedii http://orcceiiing...uiir...o.. www.nneu.cc.oo htw.:staaataaaa--tttttttt..a... html
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offine-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble. In Proceedings of the 5th Conference on Robot
Learning, pp. 1702 1712. PMLR, January 2022. URL https::/proceeddigg mwrpiiinnnmmmmmm.mmmmmmmem wwr.mmmlrppp.... presss//164/1ee222 html
Anqi Li, Byron Boots, and Ching-An Cheng. MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations. In Proceedings of the 40th International Conference on Machine Learning, ICML/23. JM..Rorg, August 2023. URL
15

---

doi: 10.1145/3440022.3445558 Published in Transactions on Machine Learning Research (04/2025)
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control, February 2023. URL. http:
//arxiv.orge hwt.:/orggbbbb/sge httg//bb/2209..0772/ 0775
Minghuan Liu, Tairan He, Weinan Zhang, Shuicheng Yan, and Zhongwen Xu. Visual Imitation Learning with Patch Rewards. In The Eleventh International Conference on Learning Representations, September 2022a. CRL https:///pperrew/e wwt.:attteetooooomooononnaee.
Yu-Ren Liu, Yi-Qi Hu, Hong Qian, Chao Qian, and Yang Yu. Z00pt: Toolbox for Derivative-Frs Optimization. Science China Information Sciences, 25(10))207101, s11432-021-3416 October 2022b. ISSN 1674-733X, 1869-1919. doi:
URL.
orggabs//801.0003330
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar, Eureka: Human-Level Reward Design via Coding Large Language Models. In The Twelfth International Conference on Learning Representations, October 2023.
URL https://openreviee wttp://oooonnnnnn-aaaaaoOO
Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-Shot Reward Specification via Grounded Natural Language. In Proceedings of the 39th International Conference on Machine Learning, pp. 14743-14752.
PMLR, June 2022. URL https::/pproceedingg .ei.igs..lr....e.... wttp::/pw..////////ma/oooeeeeeee
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we the scarch for an Artificial Visual Cortex for Embodied Intelligence? In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, March 2023. URL https://openreview.
Mitsuhiko Nakamoto, Yucxiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated Offiin RL Pre- Training for Efficient Online Fine- Tuning. In Thirty-Sewenth Conference on Neural Information Processing Systems, November 2023. URL https:
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkinn Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda. Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, March 2022. URL http://arxiv.on org.oog///aoo.. /
9203.02155
Georgios Papagiannis and Yunpeng Li. Imitation Learning with Sinkhorn Distances. In Massih-Roza Amini, Stéphane Canu, Asja Fischer, Tias Guns, Petra Kralj Novak, and Grigorios Tsounakas (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 116-131, Cham, 2023. Springer Nature Switzerland. ISBN 778-3-031-26412-- doi: 10.1007 07007/977-3-03111 3-1----1----22222-- 8.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The Unsurprising Effective- ness of Pre- Fre-Trained Vision Models for Control, March 2022. URL http::///aaxivvooo/aa//////a//// 03580
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics, 40(4):1-20, August 2021. ISSN 0730-0301, 1557-7368. 3450626 34596670
URL htttt:://ll.... acm. acm. wtt..ooo///o///.1114555555551
Alcc Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Iyya Sutskever. Language Models are Unsupervised Multitask Learners. pp. 24.
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. 1748-8766 PMLR, July 2021. URL https://proceeddnnn mlr, wtt....rses/s.////// radford21a. html
16

---

Published in Transactions on Machine Learning Research (04/2025)
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Visual Adversarial Imitation Learning using Variational Models, June 2022. CRL
08829
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexandre Lacoste. Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels. In Proceedings of the 40th International Conference on Machine Learning, pp. 28598 28617. PMLR, July 2023. URL https://prrceedding mlr. wwwppp/sss...... pttps///222/aaaeeaa333333 htmm
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A Generalist Agent, May 2022. URL
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagatton and Approximate Inference in Deep Generative Models, May 2014. URL
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In Proceedings of the Fourleenth International Conference on Artificial Intelligence and Statistics, pp. URL httppp//ppooedlggg.....eesssssss hatmmmm.ssl...llllll1.....
JMLR. Workshop and Conference Proceedings, June 2011.
mlr. wttp://....irr....ss/ssssse/e
Ramanan Sekar, Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to Explore via cole-Suud World Models. In Proccedings of the 37th International Conference Machine Learning, pp. 8583 8592. PMLR, November 2020. URL
mlr press//1/s///
Bekar20a. html
Younggyo Seo, Danijar Hafner, Hao Liu, Fangehen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked World Models for Visual Control. In 6th Conference on Robot Learning, June 2022a. URL http://arxiv.orrg /20.. 14244
Younggyo Seo, Kimin Lee, Stephen James, and Pieter Abbeel. Reinforcement Learning with Action-Fres Pre-Training from Videos. In ICML, volume 162, pp. 19561-19579. PMLR, 2022b. doi: 10.48550/ar aivvv.. 2203.13880
Rutav Shah and Vikash Kumar, RRL: Resnet as representation for Reinforcement Learning, November 2021. URL httpp::/aaxw...r//aaaaas.e argg/bas/1000/00///
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models, September 2022. URL
11302
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral Cloning from Observation, May 2018. URL http:/awwww.../b/////..... 01955
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative Adversarial Imitation from Observation, June 2019. URL http://arxiv.orr http://aaxiiv..rr/b///1///// 06158
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023a.
URL http://arxix http:///wxxvvvog//////// 230.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein,
17

---

Published in Transactions on Machine Learning Research (04/2025)
Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Finc- Tuned Chat Modols, July 2023b. URL http://arxiv. org/ www.aaaa..2.922222
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. Dm control: Software and tasks for continuous control. URL
Software Impucts, 6:100022, November 2020. ISSN 2665-9638, doi: stt.::0///iimaaa.20....... https://www. www.sseiiieenneee.e..eee wwwm//ccenceeeiiicccccocooeoeeee htt.://cww.....o.oooo....ooooo/a
http:s//www.cceeeedrrett..o.m
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http/tttts///orcediingge wwt.sseeerrrs.cceeoooo 20177aaaa/feee233777ee1115553622
files/paper/
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. ChatGPT for Robotics: Design Principles and Model Abilities, July 2023. URL
17582
Tian Xu, Ziniu Li, and Yang Yu. Error Bounds of Imitating Policies and Environments. In Advances in Neural Information Processing Systems, volume 33, pp. 15737-15749. Curran Associates, Inc., 2020. URL 350153000170444000504111110----3
neurips. hetp:/ingg.eerrrpppppppppppprr filess/paerr/2220/asss
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering Visual Contimuous Control:
Improved Data- Augmented Reinforcement Learning, July 2021. htt.a/pp///aadi..ro/aabb////////
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- World: A Benchmark and Evaluation for Multi- Task and Meta Reinforcement Learning, June 2021. URL http://arxiv: ataaxxi..rr//ab/19100..00
Xingyuan Zhang, Philip Beeker-Ehmed Patrick van der Smagt, and Maximilian Karl. Action Inference by Maximising Evidence: Zero-Shot Imitation from Observation with World Models. In Thirly-Sevenll Conference on Neural Information Processing Systems, November 2023. URL /tttpe://ppenrvvee..o net/ forum?id=Wj1(lxppxx
A Implementation details of AIME-NoB variants
AIME-NoB with AIL rewards The idea of AIL (Ho & Ermon 2016 Torabi et al., 2019) is to train discriminator D, to classify whether the obscrvation is from the expert demonstration or the agent's rollout.
As the observation is an image in our case, we follow previous works (Haldar et al.) 2022 Liu et al. 2022a to apply the discriminator upon the feature of an image encoder, for which we naturally reuse the image encoder fo from the world model. In this way, the discriminator is trained by
v² argmin E. ~Drody o ant
( -DDD- (11-))(((0011
(10)
The first two terins in Equation 10 train the discriminalor to put higher score on the observations from the demonstrations while lower score on the observations generated by the agent. The last term is regulariser to smooth out the landscape of discriminator. The regulariser weight À is set to 10. Following previvous works (Haldar et al. 2022 Liu et al., 2022a), the discriminator is implemented as a 3-layers MLP with 1024 hidden units for each layer and trained with Adam optimiser with learning rate of 1e-4. With a trained discriminator, the AlL reward can be computed as
AIL log (fo((o))) log(1 (ffloo))) r7
(11)
18

---

Published in Transactions on Machine Learning Research (04/2025)
Since the value of AIL rewards also depends the image encoder fe. which is changing during the course of training, previous works (Liu et al., 2022a Haldar et al., 2022) maintain a slow-update version of the image encoder to mitigate the non-stationary of the reward. But since our image encoder is pretrained as part of the world model and finetuned with the data-driven regulariser, we find its weight is stable during the course of training. Thus, we don't apply this additional slow encoder. In the experiments, we find the adverserial training is stable with the help of the world model, as also shown Rafailov et al. (2022).
introduced to imitation learning to alleviate the non-stationary reward and sensitive to pappe--rrameters problems in AlL. The idea is to use the optimal transport to measure the minimal effort of moving any trajectory T = (o1r) to a demonstration trajectory T = (oy:ra), where the length of the trajectory T and Ta can be different. The effort of moving the trajectory is measure by a Wasserstein distance, i...
AIME-NoB with OT rewards OT PPapagiannis & Li 2023 Haldar et al. 2022
TM -E =lp==
Pee (f(oo)
(12)
where j RTXT" is the transportation matrix that satisfy p1 #1 and /f = # So is the process function that map the raw observation to metric space for which reuse the image encoder as in AII, is the cost function that measure the distance between two vectors in the metric space for which we use the cosine distance. With the cost measure g, the optimal transport solve for the optimal transportation matrix with
n gly fo(T), aaggnin((u-aa()))ff((())..
(13)
With the optimal transportation matrix, the OT reward can be defined as
Td
TT
OT
Σ pr pi.olfs(orr
(14)
The À is hyper-parameter to scale the OT reward to be casier for the agent to learn with. We follow Haldar et al. (2022) to apply an adapted normalisation scheme where the scale factor is based on the cost measure of the first trajectory we evaluate, i.e.
4
X = ((ur,f((Tm)) (T
(15)
When multiple demonstrations are available, we take the OT reward from the trajectory with the lowest total transportation cost.
It may worth note that OT is the only variant that doesn't requires train separate model for the reward labelling. However, it still changing during the course of training since the image encoder fo gets finetuned.
AIME-NoB with VIPER rewards VIPER (Escontrela et al.) 2023) trains video prediction model on the demonstration datasets and treats the likelihood of the video prediction model as the reward for policy learning, i.c.
VIPER IT
(or(ooorr
(16)
In the original paper, the authors first pretrain VQ-GAN (Esse et al. 2021) from a multi-tasks expert datasct, and then train GPT-style auto-regressive model in the quantised space for prediction. For fair comparison with other variant, we consider to only train the VIPER model on the single demonstration dataset for the task. For simplicity of the implementation, in this paper, we consider training an unconditioned latent world model as in [Sco et al. (2022b) to model the VIPER reward. We use the same RSSM architecture of the model learning for DMC only removing the condition of the actions, and we train the VIPER model for each task separately. Especially during training, we find training such a powerful model from scratch
19

---

Published in Transactions on Machine Learning Research (04/2025)
small dataset can easily result in over-fitting. Thus, empirically choose train the model only for models and 1000 gradient steps for MetaWorke models. We show evidence of overfitting in Appendix G.1 Due to the large scale of the ELBO we also apply symlog (Hafner et al. 2023 when computing the VIPER reward. Another difference with the original VIPER paper is that we do not use intrinsic motivation as the exploration bonus as the authors suggested, since the AIME loss for policy learning already provides task-related guidance for exploration. We only apply an entropy regulariser to the policy is common practice. We further show the synergy between AIME and VIPER in Appendix
500 gradient steps for DMC
B Compute Resources
All the experiments are run on a local cluster with few A100 and RTX8000 instances. All the experiments are tuned to use less than 10GB of GPU memory so that they can run A100 MIG. World models pretraining requires about 24 GPU hours, while VIPER models require negligible time for training. Each DMC experiment requires about 40 GPU hours while each Meta World experiment requires about 20 GPU hours. And for reference, the baseline algorithm AIME requires about 7 GPU hours for each task.
C Hyper-parameters
Here, we document the detailed
for all the trained models in Table
Source of Datasets
We use the expert trajectories from
Haldar et al. 22022 at https://osf 2o/4w69f/2viens only==
The authors didn't provide a License for their dataset. Further, we use the replay buffer dataset from Hansen et al. (2023a) at htthttss: /ttps:///uuggggg htt.ggiiinaaae..ccaattttttaaatae wtt.c//ttteetssiiiccaaaaaaaaaaaa ccmpc2/tree/main//
a2bb9lceea74433385522247778
The authors provide the dataset under the MIT License. Moreover, we use the ro-
htt///imeeeeeeeeeee play buffer dataset from /Zhang et al. (2023) hattps://githube com/ http:///aaaaaaeeeee////aaaaaaaaa
The authors provide the dataset under the CC BY 4.0 License.
E
Details for Resetting MetaWorld Tasks
generale the image observation datasets from the TD-MPC2 replay buller (Hansen et al. 2023a) ) we modify the Meta World codebase reset the environment the initial state of the trajectory from the first observation. Luckily, the starting position of the robot arm is always the same for each task, that do not need to apply inverse kinematics to solve for the initial pose of the robot arm. For the object and the target position, for most of the tasks, the internal reset position can be computed by making constant shift, on the object position and the target position in the observations. There are, however, also a few edge cases which we handle differently.
In buttoon-press-toodown and nutton-pressttopdow--- observation upon the second time step, presumably due to some simulator delay in the resetting process. So for these two tasks, the initial state is reset by the second observation.
the object's true position only appears the
For basketball and box-close, it seems like there is some internal collision detection that will alter the object and robot position after the task is reset,, so computing the exact reset value from the observation is not possible. For these two Lasks, we instead resort to a search-based method. To be specific, we use a gradient-fres optimiser from (Lu et al.) 2022bb to search over the resetting space of the object and find the reset position that minimiscs the L2 distance with the true obscrvation.
More details of the implementation can be found in the code.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Table AIME-NoB use for each benchmark.
DMC
META WORLD
World Model
CNN STRUCTURE
HA SCIMIDIUBER 2018
HAFNER ET AL.
2023
CNN WIDTH
32
48
MLP HIDDEN SIZE
512
640
MLP HIDDEN LAYER
2
MLP ACTIVATIONS
LAYERNORM + SWISH
DETERMINISTIC LATENT SIZE
512
1024
STOCHASTIC LATENT SIZE
30
FREE NATS
1.0
KL BALANCING
0.8
MODEL SIZE
8M
20M
Policy
HIDDEN SIZE
128
HIDDEN LAYER
2 ELU
ACTIVATION
DISTRIBCTION
TANH- GAUSSAN
Value network
HIDDEN SIZE
128
HIDDEN LAYER
2
ACTIVATION
ELU
TARGET EMA DECAY
0.95
Training
BATCH SIZE
HORIZON
50
64
TOTAL ENV STEPS
1M
500k
UPDATE RATIO
0.1
GRADIENT CLIP
100
POLICY ENTROPY RECULARISER WEIGHT
1e-4
MODEL LEARNING RATE
3e-1
POLICY LEARNING RATE
3e-4
VALUE NETWORK LEARNING RATE
8e-5
DISCOUNT FACTOR Y
0.99
TD-LAMBDA PARAMETER À
0.95
IMAGINE HORIZON
15
AIME-NoB specific
POLICY PRETRAINING ITERATIONS
2000
DATA-DRIVEN REGULARISER RATIO
0.5
VALUE GRADIENT LOSS WEIGHT 3
1.0
F Difference between compounding error and DKB
For DKB most literature Ho & Ermon 2016 Peng al. 2021 Torabi et al. 2019 Liu et al. 2022a
understand it as compounding error, in which the error is understood as when placing the policy with the same inl states as the demonstration, the learned policy will gradually diverge from the demonstrations due to error accumulation. Our [KB] is a broader concept than compounding error. DKB attributes the gap of perform to the lack of learning signal on unsupported space of the demonstrations. In this way, [KK can explain more empirical successes in the literature than compounding error can. For example, recent work MAHALOO [Li et. al. 2023) shows evidence of the importance of the size of the covered space. The
21

---

Published in Transactions on Machine Learning Research (04/2025)
AlL Returns on walker-run
Returns on walker-run
VIPER Returns on walker-run
walleerrun-buffer
patchall walker- recpeerrun-eeeeee expert
petchail wa wulker-un expert
Linear Regression
VIPER Return
-1500
Lincar Regression
AIL Return
(2.6pe=3.26, R*=0.96)
OT Retum
-500
(slope-0.59, R7=0.75)
17750
11000
-1000
-200
walker buffer patchail weeeerree
-2000
-2250
-3000
-1500
Linear Regression
(slope=1 R*=0.90)
-2500
200
400
600
800
200
400
600
800
0
200
400
600
800
True Return
True Retum
True Return
Figure 7: Correlation of surrogate rewards and true rewards on the walker-run lask.
authors studied similar ILfO setup with embodiment and demonstration datasets. They compared two variants: for one they train an Inverse Dynamics Model (IDM) from the embodiment dataset and use it to label the rationntration dataset, while for another they train a reward model from the demonstration dataset. by labelling all time steps with a reward of 1, and then use it to label the embodiment dataset. Finally, they run the same offline RL algorithm on both labelled datasets. The results show the second variant attains a much better performance even though the labelling from the reward model is not as meaningful as the actions from the IDM
G Landscape of the surrogate rewards
To better understand the different types of surrogate reward and why works better than the others, we investigate the relevance between the surrogate reward and the true reward. We take the one seed of the final model from each of the variant on walker-run task and evaluate the surrogate rewards on both the expert dataset from PatchAlI. where the surrogate reward model is based on, and the replay buffer dataset from Zhang et al. (2023).
As we can see from the results in Figure all the surrogate rewards manage to put the expert demonstrations with higher reward than the trajectories in the replay buffer. Among them, ATT, which performs the best, has more linear correlation with the true reward and higher slow for the lincar regression.
G.1 Overfitting of the VIPER model
To better illustrate the overfitting problem for VIPER models and justify our choice of training fewer iterations, train the VIPER models for varying number of gradient steps and evaluate the correlations between the VIPER reward and the true reward. Specifically, we train the same VIPER model with (100, 500, 1000, 2000, 5000, 100000} gradient steps and plot the result in Figure R As we can clearly see, when training with less than or equal to 1000 gradient steps, VIPER reward has a very nice correlation with the true reward, with the middle-range performance even like a linear correlation. The best model could be selected from 500 and 1000 gradient steps. However, as we train the model for longer, the VIPER reward for the export trajectories is boosted even higher, and a side effect, it also relatively boosts up the VIPER reward for low- oow-performance trajectories. This is because, when overfitting the expert trajectories, the model increases the marginal likelihood of all the observations in the expert trajectories to a higher value, which also includes few frames of the robot lying on the ground at the very beginning of each trajectory after reset. For these low-performance trajectories, the robot remains mainly stuck around the initial position and struggles on the ground. This artifact the overfitted VIPER reward creates a sharp local maximum in the low-performance region that the agent can hardly get away from.
H Full results of the benchmark
In this section, we show the full results of all the variants on the tasks on the two benchmark suites. The aggregated results of all the variants are shown in Figure g and results each individual tasks shown in
22

---

Published in Transactions on Machine Learning Research (04/2025)
100 gradient steps
500 gradient steps
1000 gradient steps
-1250
-1000
-2000
-1250
-1500
VIPER Return
1500
-2200
1750
11550
-2000
-2400
-200
-2250
-2600
2500
-2500
200
400
600
200
400
200
600
800
2000
gradient steps
5000
gradient steps
100000
gradient steps
-1000
550
AIME walker-run buffer
-100
PatchAlL walkerrun walkerrun-expert
1250
1000
VIPER Return
-1500
15500
1500
1750
2000
2000
-2000
2500
2500
-2250
3300
2500
-3000
-3500
200
400
600
200
200
600
800
True Return
True Retum
True Return
Figure 8: Correlation of the VIPER reward and the real reward with models trained with different numbers of gradient steps. Each point represents one trajectory. We can clearly see the model gradually overfitiigg and losing the correlation with the real reward when training for more than 1000 gradient steps.
Aggregated results DMC
Aggregated results MetaWorid
1.0
AIME w/ mt33
0.8
0.8
AIME NeB wt AlL w/ mt39 (Durs)
Normalized Score
AIME NoB wt AlL w/ mL50 (Ours)
AIME
(Ours)
AIME NoB
AIME NoB wi OT wi mi50 (Durs)
AIME-NoB OT (Qurs)
JIME-NoBl VIPER yf 39 (Durs)
AIME-NOC w/ VIPER (Ours)
AIME-NoB VIPER
(Ours)
(C01a) mi mt50
0.4
QT
0.4
PatchAIL
PatchWiL
0.2
0.0
8
10
1
2
3
4
Env steps (x10²)
Env steps (x10=)
Figure 9: Comparing different variants of AIME-NoB with other algorithms. The figures show aggregated IQM scores on 9 DMC tasks and MetaWorld tasks, All the algorithms are evaluated with 5 seeds with the shaded region representing 95% CI.
Figure 10 and Figure II respectively for DMC and Morld We also summarised the final performance after the interaction budget at Table and Table respectively for DMC and Morl.
Additional Experiments
Do we need the whole embodiment datasets to establish the regulariser? Although the data-driven regulariser is efficient, it requires to keep the entire embodiment datasets to build the regulariser. This can be challenging when the world model pretrained on an internet-scale dataset. Therefore, we try to study if it is possible to use only a small portion of the dataset to establish the regulariser. We randomly sample a subset. of the embodiment dataset the regularisor and the result. is shown in Figure Surprisingly, the
23

---

Published in Transactions on Machine Learning Research (04/2025)
cartpole-swingup
cheetah-run
finger-spin
1000
800
1000
750
600
Retum
500
400
250
200
0-
hopper-stand
hopper-hop
quadruped-run
1000
300
750
400
Return
500
200
250
200
100
0
-250
wa walker-stand
walker-walk
wa walker-run
1000
1000
750
600
600
400
400
200
250
200
Env Steps (x10²)
Env Steps (x10²)
Env Steps (x10*)
AIME
AIME-NoB w/ OT (Ours)
BCO(a)
PatchAIL expert
AIME-NoB VIPER
Figure 10: Benchmark results 9 DMC tasks. Return calculated by running the policy 10 times with the environment and taking the average return. The results are averaged across 5 seeds with the shaded region representing 95% CT.
Table Final returns on DMC tasks 1M environment steps. The best performance on each task markod bold.
cartpole-twingup
chectali-ruz
fugee-apin
hopper-staud
hogpper-hopp
quadzuppol-run
walker-stand
walker-walk
walker-rmn
Expest
546
518 67
127
BCO())
206 +:
104 1:11:31
154 258
133 113
7
148 + 106
786 58
00
25:t:5
795 10
645
976 1.2
340 562
1 29
133
830 1 200
836 0.61.8 311
375
ADME-NoB w/ AlL (Ous)
177 1922
859 +
976 #1
874
308 ::
422 # 114
981 #
964 ::
771 5
AIME-NoR w/ QT (Orns)
100 + 23
858 +
968 + 13
807
302 +-9
336 + 164
983 +
062 +-4
758
NolB (Ours)
results show the action inference is even more stable when we only use 0.1% of the embodiment dataset, i.e. trajectories, to establish the regulariser.
Improving AIME-NoB on cartpolc-swingup. As shown in Figure 10 although ALME-NoB solves out of tasks, it still straggles cartpole-swingup. We further investigate the cause of the low performance. A ffter examination, find issues in both of the datasets. For the demonstration dataset, show the first.
24

---

Published in Transactions on Machine Learning Research (04/2025)
disassemble
assembly
hand-insert
1.0
1.0
0.8
0.8
Success Rate
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0
2
5
0
5
0
2
5
sweep
1.0
1.0
0.8
0.8
0.8
Success Rate
0.4
0.4
0.4
0.2
0.2
0.2
Env Steps (x10²)
Env Steps (x10%)
Env Steps (x10²)
AIME mt39
AIME NoB w/ AlL mt50 (Ours)
AIME NoB VIPER mt33 (Ours)
OT PatchAIL
AIME w/ mi50
A/ME-NoB w/ OT w/ mt39 (Ours)
AIME-NoB w/ VIPER w/ mL50 (Ours)
AIME NoB w/ AlL w/ mt39 (Ours)
AIME NoB w/ OT w/ mt50 (Ours)
BCO(al w/ mt50
Figure 11: Benchmark results on 6 MctaWorld tasks. Trajectorics are only counted as success when it success at the last time steps and the success rates are calculated with 100 policy rollouts. The results are averaged across 5 seeds with the shaded region representing 95% CI.
Table 3: Final success rate on Meta World tasks at 500K environment steps. The best performance on each task is marked as bold.
disassemble
assembly
hand-insert
push
sweep
hammer
AIME w/ mt39
0.00 = 0.01
0.01 0.01
0.00 0.01
0.07 0.06
0.06 0.03
0.53 0.13
AIME w/ mt50
0.32 0.13
0.09
0.08 + 0.03
0.34 0.05
0.18 0.09
0.42 0.07
BCO(a) w/ m:50
0.00 + 0.00
0.03
+ 0.03
0.06 0.06
0.14 0.09
0.57 = 0.25
OT
0.00 + 0.00
0.00 0.00
0.05 + 0.08
0.10 0.05
0.04 + 0.12
0.09 0.10
PatchAIL
0.00 + 0.00
0.00 0.00
0.00 + 0.00
0.00 -: 0.00
0.00 + 0.00
0.00 0.00
AIME-NoB AIL w/ mt39 (Ours)
0.94 = 0.03
0.41 0.29
0.10 0.14
0.42 0.48
0.77 = 0.16
0.96 0.07
AIME-NoB AIL w/ mt50 (Ours)
0.95 = 0.03
0.68 0.03
0.34 0.08
0.52 0.38
0.88 + 0.09
0.99 0.02
AIME-NoB OT w/ mt39 (Ouss)
0.82 1 0.11
0.06 00611.112
0.05
0.22
0.07 0.03
0.69 0.52
NOM w/ OT w/ mt50 (Our))
0.79 0.12
0.60 +
0.15
0.24+
0.58 0.21
0.02
AIME-NoB w/ VIPER w/ mt39 (Ours)
0.59 0.32
0.35 0.25
0.04
0.24
0.55 0.18
0.66
AIME-NoB w/ VIPER w/ mt50 (Ours)
0.53 = 0.30
0.52 0.20
0.22 0.07
0.49 0.12
0.66 = 0.09
0.97 0.03
100 frames from demonstration in Figure 13 As we can see, the initial position of the cart from the center of the image with the pole pointing downward. The expert demonstration directly drive the cart to the left and then back to the right to swingup the pole and in the end balance the pole in the middle. Due to the carmera setup in cartpole, when the cart continuously to the left, the cart move out of the scene, which makes the most important swing up process happen outside of the scene. This posc a severe challenge to the action inference and results in poor performance of AIME-NoB. We observe that all the policies from AIME-NoB learned to move left to go outside the scene in the beginning but comming back with an angle either not enough to push the pole to the up-right position or too much that the pole swing down from the right. For the embodiment dataset, it mainly contains data with high speed rotating behaviour of the pole. Learning with imagination from these states will not lead to helpful signal for swingup.
25

---

Published in Transactions on Machine Learning Research (04/2025)
Ablation AIME-NoB
Ablation AIME-NoEKB
800
0.6
0.5
Return
600
Action MSE
Return
600
Action MSE
0.4
400
0.4
(0.1%)
N-20 (1%)
N- (1%)
200
N=2005 (100%)
N-200 (10%)
0.2
200
N-200 (10%)
N==005 (100%)
expert
expert
0.1
0
0
0.00
0.25
0.50
0.75
1.00
10
0.00
0.25
0.50
0.75
1.00
Steps (x105)
Env Steps (x10f)
Env Steps (x105)
Steps (x109)
Figure 12: Ablations of the size of the data-driven regulariscr N on walker-run task. AIME-NoB is rumning with 10 demonstrations, while AIME-NoEKB is with 100 deemonttrations Action MSE is only shown for the first 10a env steps. All results are averaged across 5 seeds with the shaded region representing a 95% CL.
10
20
30
35
40
45
50
55
60
70
80
90
100
Move the left
Swing up
Move back to the middle and balance
Figure 13: The first 100 frames of a cartpole swingup demonstration. The most important swingup behaviour is happened outside the scene.
cartpole-swingup
1000
AIME-NoB
AIME-NoB True reward
AoME tuned.
800
OT
expert
Return
600
400
200
0
2
6
10
Env Steps (x105)
Figure 14: Results on cartpole-swingup with additional variants of AIME-NoB with the true reward and AIME-NoB with tuned hyper-parametors. Return are caleulated by running the policy 10 times with the environment and taking the average return. The results are averaged across seeds with the shaded region representing 95% CT.
In order to improve the performance, we need to have better emphasis on the role of the surrogate rewards. We remove the data-driven regulariser, i.e. setting = O, since we know AIME loss won't help too much on this case and we want better utilisation the online data. We further weaken the effect AIME loss with a weight of 0.01. In Figure 14 we show the tuned version of AIME-NoB improve over the default. paprr-parameters and can solve cartpole matching the best OT baseline. But still both of them have large variance during training. In order to further understand the upper bound of this task, also include
26

---

Published in Transactions on Machine Learning Research (04/2025)
pick-place
shelf-place
1.0
1.0
0.8-
0.8
Success Rate
0.6
0.6
0.4-
0.4-
0.2
0.2
0.0
0.0
1
4
2
4
stick-pull
stick-push
1.0
1.0
0.8
0.8
Success Rate
0.6
0.6
0.4
0.4
0.2
0.0
0.0
i
Env Steps (x10²)
Env Steps (x10²)
AIME
AIME-NoB (Ours)
Figure 15: Benchmark results on additional 4 hard and very hard Meta World tasks. Trajectories are only counted as success when it success at the last time steps and the success rates are calculated with 100 policy rollouts. The results are averaged across seeds with the shaded region representing 95% Cl.
variant of AIME-NoB with true reward from the environment. We can see that with the true reward the task can be solved reliably. This motivates a better design of surrogate rewards in future works.
More challenging tasks. We extends our benchmarks with more challenging tasks. As from previous results in Figure 10 and Figure 11 basclines BCO(a), OT and PatchAIL are not well-performed, are not expecting them to be good on these even harder tasks. Thus, mainly compare between AIME-NoB with AIME.
For World, we include four additional tasks, namely pick-place, shelf-place, stick-pull and stick-pun pusk, According to Seo et al. (2022a), pick-place is a hard task and the other three are very hard tasks. The results are shown in Figure 15 From the results, AIME-NoB reliably outperforms AIME. Even in the very hard tasks stick-pull and stick-push, AIME-NoB manages around 60% success rates. However, AIME-NoB docsn't 't performs well pick-place and shelf-place. We conjecture it is due to the visual difficulties of the tasks. It is known that the world models based on reconstruction loss struggle at modeling small objects, which is the small cube we need to pick-up in these two tasks. Improving the world models' ability of modeling small objects or increase the resolutions of the observations will likely improve the performance.
For DMC, we conduct additional experiments on the humanoid embodiment. Since the humanoid embodiment is not in the initial list of the study, we essentially need to recollect the datasets on the embodiment and pretrain the world model. For the embodiment dataset, to run Plan2Explore for 2M environment steps and use the 2000 trajectories in the replay buffer the embodiment dataset. The world model is pretrained on the embodiment dataset for 200k gradient steps. For the demonstration dataset, we use the state-based policy from TD-MPC2 to collect. 100 jrajectoiees, and render the images to fit to our dision-based setting. Since humanoid is a very complex task, especially when using images as the observations, existing works (Yarats al.) 2021 Hafner et al., 2020) run the tasks for at least 30M environment steps. Due to the heavy computation load, we only run AIME-NoB for 10M environment steps. We show the results in Figure 16 We can clearly ATMF-NoB outperforms ATME and
use the same setting in the main benchmark
27

---

Published in Transactions on Machine Learning Research (04/2025)
humanoid-stand
humanoid-walk
humanoid-run
600
400
Return
300
400
40
200
200
100
20
40
60
80
100
20
40
60
100
20
40
60
100
Env Steps (x10*)
Env Steps (x10%)
Env Steps (x10%)
AIME
AIME-NoB (Ours)
DrQ (officia))
Figure 16: Results of AIME-NoB on 3 humanoid tasks. Returns are calculated by running the policy 10 times with the environment and laking the average return. AIME and AIME-NoB are averaged over seeds; DrQ-v2 over 10 seeds (Results from official repo https://gittni httph//cwoo.ccccooooeeeeaaa////a
main/curves) curves). The shaded region representing 95% CI.
it has the potential to further improve if training for more environment steps. Since there are no existing vision-based ILFO algorithms have been tested on humanoid, we add an additional comparison with the official results in DrQ-v2 which is a state-of-the-art vision-based model-free RL algorithm. We can see ALME-NoB is more sample-efficiett than DrQ-v2 on all three tasks, but was caught up by DrQ-v2 around 10M steps on humanoid- walk and humanoid-run. We hypothesise that it is due to the limit capacity of the policy and value networks, which implemented with 2-layers MLP with 128 units, cannot handle thus complex tasks. To the best of our knowledge, this is the first time that ILSO algorithm shows progress on vision-based humanoid tasks.
Comparing with Dreamer. We further compare AIME-NoB with Dreamer which has access to the true reward provided by the environments. We compare with two versions, one is training Dreamer from scratch, one is training Dreamer but initialise the world model with the pretrained one, which we denote Dreamer w/ pt. One thing worth noting is although the pretrained world model has most of the components required for a new task, the reward decoder still needs be trained from scratch since it is task-specific.
For DMC, report the results at Figure 17 As we sample efficiency on 8 out of 9 tasks except the problematic environment cartpole-swingup discussed in Appendix ! On hopper-hop and quadruped-rum, AIME-NoB is surpassed by Dreamer in the end. This is because as an imitation learning algorithm, AIME-NoB's performance is limited by the quality of the expert.
from the results, AIME-NoB achieves better
For Meta World, we report the results at Figure 18 As the manipulation tasks typically impose challenges for exploration, we see that Dreamer with or without the help of the pretrained model struggles to accomplish the task within the 500k environment steps. In the same time, with the help of the demonstrations, is much easier for AIME-NoB to explore the related regions in the observation which results in a better sample efficient. This marks advantage of using demonstrations over using scalar reward to define the task.
Performance gain of AIME-NoB regarding the same computation time. Arguably, due to the additional online interactions with the environment, fintuning the model and training the actor-critic, on our implementation ATME-NoB needs approximately x2.86 times per gradient step comparing with the base algorithm AIME. In this perspective, it is reasonable ask how compute efficient is ATME-NoB comparing with AIME. We redraw the benchmark results from Figure 2 with respect to the compute time in Figure 19 From the results, over AIME even when the compute budget is tight.
conclude that, if the online interaction is allowed, we should always use AIME-NoB
Presence of EKB and DKB on more tasks. In the main texl, we only show the presence of EKB and DKB on walker-run with Figure ! and Figure 3 Here we show more evidence on hopper-hop from the DMC suite and the disassemble task from the World suite in Figure 20 As we can soe, different tasks can have different distributions of the two barriers. For hopper-hop, AIME and the MBBC oracle almost overlap with each other, meaning the FKB is negligible in this case, and the performance gap mainly attributed to DKB
28

---

Published in Transactions on Machine Learning Research (04/2025)
cartpole-swingup
cheetah-run
finger-spin
1000
1000
800
750
Return
600
600
500
400
400
250
200
200
0
0
0-
250
10
10
hopper-stand
hopper-hop
quadruped-run
1000
800
400
600
Return
500
300
400
200
250
100
200
-250
10
10
10
walker-stand
walker-walk
walker-run
1000
1000
800
800
600
400
400
400
200
200
200
0
10
10
10
Env Steps (x105)
Env Steps (x105)
Env Steps (x10=)
AIME-No8
Dreamerv3 (official)
expert
Figure 17: Additional comparison between AIME-NoB and Dreamer 9 DMC tasks. Returns are calculated by running the policy 10 times with the environment and taking the average return. The results are averaged across 5 seeds (10 seeds for Dreamerv3 (official) from official repo hppohttps:/// /ttp///gtttt..tt///eee com danii jardrramrr// tree/main/scoren with the shaded region representing 95% CI.
This is because the Plan2Explore embodiment dataset alrcady contains hopping behavior, and the pretrainod model has enough knowledge to infer the correct actions. For the disassemble task from Meta World, the trend is more similar to Walker Run, where both barriers are present. When the number of demonstrations is low, μKB is dominant, and as the number of demonstrations increases, EKB and DKB exhibit more equal strength.
J Upper bounds for EKB and DKB
In the section, propose upper bounds for both of the barriers based intuition. Without losing any generality, we assume that the reward at each step is bounded by scalar Tmaxi i.e. Irx] S Tmax-
Upper bound for EKB Our analysis is inspired by Xu et al. (020).. We assume the policy æ is expressive enough to fit the learning objective, then we have Twa = Pracmo and Tg- = 9ø-. Thus, we have
DRL(TT. (aclosa) Tg. (aclo1a)) DRL (KI(P (Prremm (ac(01.T) qo (a, (1.T))
(17)
29

---

Published in Transactions on Machine Learning Research (04/2025)
disassemble
assembly
hand-insert
1.0
0.8
0.8
0.8
Success Rate
0.6
0.6
0.6
0.4
0.4
0.4
push
sweep
hammer
0.8
Success Rate
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
2
2
3
Env Steps (x10²)
Env Steps (x10²)
Env Steps (x10²)
AIME NoB (Ours)
Dreamner w/ pt
Dreamer w/o pt
Figure 18: Additional comparison between AIME-NoB and Dreamer on MetaWorld tasks. Trajectorics are only counted success when it is marked successful at the last time steps and the success rates are calculated with 100 policy rollouts. The results are averaged across seeds with the shaded region representing 95% CI.
Aggregated results DMC
Aggregated results MetaWorid
1.0
AIME
AIME
(Ours)
(Ours)
0.8
0.8
Normalizen Score
0.6
0.6
Success
0.4
0.4
0.2
0.0
0.0
0
1
4
5
2
6
7
Compute Time (hours)
Compute Time (hours)
Figure 19: Comparing AIME-NoB with AIME the same compute budget. The figures show aggregated TQM scores on 9 DMC tasks and 6 World tasks. All the algorithms are evaluated with 5 seeds on each task and the shaded region representing 95% CL. AIME-NoB is not starting from 0 since the first point is evaluate after the 2000 pretraining steps with AIME.
S.c. Then, We assume the inference error bounded by c, i.c. En [DKL (Praomo (ay ( 46- (a.(o1:))) S. according to Theorem in Xu et al. 22200, ; have the upper bound of EKB
EKB = R(TT.) R(TT=) S 2v mi + 1)c.
(18)
The term 1/(1 - 7)2 from the original thcorem is replacod by T(T 1 as we are considering a finite horizon problem in this paper. The proof can be done by redoing their proof on the finite horizon case without the discount factor 7, i.e. for where 7 serves the normalizing factor have ? T-1 while for
30

---

Published in Transactions on Machine Learning Research (04/2025)
EKB and DKB on hopper-hop
EKB and DKB on disassemble
100
1.0
MBBC (oracle)
MBBC (oracle)
Normalised Returns
AIME
AIME
80
0.8
Success Rate
60
0.6
40
0.4
20
0.2
0.0
10"
100
10a Number of Demonstrations
Number of Demonstrations
Figure 20: Additional evidence of the presence of EKB and DKB on hopper-hop from DMC and disassemble from MetaWorkd The results averaged across seeds with the shaded region representing 95% CI.
case where it is used as discount factor we have 7 1. The bound is tightened when the inference error decreases and EKB is completely overcome when reduce the inference error to 0.
Upper bound for DKB Without losing generality, assume the initial state of each trajectory is sampled from a discrete distribution p(s). Then, the expected return can be rewritten as R(T) = Σ, P(s)R(())))) (== Based on this definition, we can rewrite DKB with a factorization based on the support the demonstration dataset, i.c.
DKB R( R(Tdemo) R((T.))=
Σ
(R))(R( )00=== -R((T-- ([(m==))s==a)))
Ddemo
Σ
(Rs) (Tdemo)) 1=0== (((w.)) e0=s)
(19)
Da
Given the policy expressiveness assumption, the learned policy тu* can match the performance of Wdemo on the support of the demonstrations. Therefore, the first term Equation (19) becomes 0. Thus, DKB only depends on the performance on the initial states out of support of the demonstration dataset. Without any assumption on the generalization of the learned policy, it can perform arbitrarily bad the OOD states and in the worst case we have R(m=)) - Trmax- But in practice,
assume the worst performance of the learned policy can be bounded by R i.e. for all & Ddemo have R(T )1m0= 2 R We denote the support ratio of the demonstrations as Exe p(s)/ E. P(s) Exe p(s), then we have the upper bound for DKB as
DKB S (1 (TT
R < 2(1 T)
(20)
It can be clearly seen from the upper bound that when we increase the support ratio the bound is tightened. If increasing support ratio is not possible as the demonstrations are expensive, we can also increase the performance on the OOD states, i.c. R by providing extra learning signals.
31


Reviews:
[{"review_id": "0gZKrYIPju", "reviewer": "Reviewer_WXaM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well-written, and tackles an important problem which is finetuning pretrained policy on novel tasks.\n- Extensive ablation study\n\nWeaknesses\n- It looks like technical wise the method is somewhat \"finetuning a policy and world model with dreamer and surrogate reward\", yet motivated in a relatively convoluted way.\n- My main concern is that all experiments are run with simulation / toy examples. How well does the method work in real-world robotics tasks? \n- For figure 1, how does the figure look with other tasks / domains? What makes a task having more DKB / EKB? Does the method and baseline perform differently with different observation space?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an algorithm that improves existing methods for finetuning a pretrained policy on novel ILfO tasks. The paper first proposes to decomposes the performance gap between policy and oracle with 1. DKB, which measures the gap between oracle with methods having access to GT action, and 2. EKB, which measures the performance gap between the . The paper then proposes modifications to existing ILfO algorithms that are motivated to addressing such gaps (\"knowledge barriers\"). To address EKB, the paper proposes to finetune the pretrained world model on new interaction data. To address DKB, the paper proposes to use surrogate reward function and trains the policy in dreamer style.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Same as the weaknesses section. Mainly, real-world robotics experiments, and analysis on questions raised above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gUg5tvRpzd", "reviewer": "Reviewer_Cwx7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The introduction of EKB and DKB provides a well-motivated framework for analyzing limitations in ILfO.\n+ The integration of online interactions and surrogate rewards effectively tackles the identified barriers.\n+ The experimental results span many tasks across two benchmarks, demonstrating superior performance compared to baselines.\n+ Various ablations offer insight into the contributions of different components of AIME-NoB.\n\nWeaknesses\n- Limited Theoretical Justification: While the empirical findings are strong, a more rigorous theoretical analysis of the knowledge barriers and proposed solutions would enhance the work, but may be out of scope.\n- Complexity of Implementation: AIME-NoB involves multiple interdependent components (regularization, surrogate reward, online interaction), which may make it harder to reproduce. It would be great if the authors can open source all their code and experimental harnesses so others can more easily build on this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses fundamental challenges in Imitation Learning from Observation. It identifies two primary barriers: the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB). The EKB is caused when pretrained models encounter novel observations and struggle to infer correct actions, while the DKB results from the limited generalization of policies trained on sparse expert demonstrations. The authors propose AIME-NoB, an extension of the AIME algorithm, which uses online interactions and a data-driven regularizer to mitigate the EKB and a surrogate reward function to expand state-space coverage and address the DKB. The paper provides a thorough empirical evaluation, demonstrating significant improvements in sample efficiency and final performance on the DeepMind Control Suite and MetaWorld benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Surrogate rewards: It would be nice to provide more justification or explanation for why AIL seems to perform better than OT or VIPER. Are there possible theoretical reasons for this? Is the benefit of AIL particular to this problem setting or do you think it more broadly superior as a surrogate reward?\n\nDiscuss computational overhead: Since AIME-NoB incorporates online interactions, surrogate rewards, and regularization, a more explicit comparison of computational costs vs. performance gains would be valuable.\n\nConsider alternative online training schedules: The current method balances pretraining data and online interactions via a static α parameter. Could an adaptive scheduling approach improve stability?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oXCQCrFNj3", "reviewer": "Reviewer_N6FK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper draws attention to the overlooked issues of knowledge barriers in ILfO. The distinction between EKB (the inability of pretrained models to generalize to unseen embodiments) and DKB (the shortcomings stemming from limited demonstrations) is intuitively appealing and addresses an important gap in current methods.\n\n- The study is backed by extensive experiments on two benchmark suites. The authors provide ablation studies that reveal how varying key hyperparameters ($\\alpha$ and $\\beta$) impact performance. This empirical analysis lends support to the claim that their modifications yield improvements in both sample efficiency and final performance.\n\n**Weaknesses:**\n\n- The definitions of the Embodiment Knowledge Barrier (EKB) and Demonstration Knowledge Barrier (DKB) remain largely informal. While the authors offer intuitive descriptions, the lack of a rigorous, formal definition—and quantitative metrics that measure these barriers—limits the clarity of the paper. As a result, it is difficult to assess precisely how much each modification reduces the corresponding gap.\n\n- The methodological contribution appears largely incremental. The proposed solutions for both barriers are essentially straightforward extensions of known techniques by including more information. For EKB, the solution is to allow online interactions with a data-driven regularizer—a sensible but not conceptually novel modification. This is compounded by the observation that Figure 1 does not clearly illustrate how these additions overcome the EKB beyond merely providing additional “oracle” information.\n\n- The contributions of the paper are more in the realm of effective engineering, i.e., combining online fine-tuning with surrogate rewards. While the empirical improvements are significant in some benchmarks, the overall methodological novelty remains modest, and the practical implications for real-world systems (e.g., robotics) are somewhat speculative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper targets two critical issues in imitation learning from observation (ILfO) when leveraging pretrained world models—namely, the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB). In response to these challenges, the authors propose AIME-NoB, an extension of the earlier AIME algorithm. The key contributions include:\n\n- EKB Mitigation: The paper proposes an online interaction phase combined with a data-driven regularizer to continuously update the world model. The rationale is that by collecting new data online, the pretrained model can better handle out-of-distribution observations and actions that were not encountered during pretraining.\n- DKB Mitigation: To address the limited coverage of expert demonstrations, the authors introduce a surrogate reward function derived from a VIPER-style video prediction model. This reward is then used in a dreamer-style actor-critic framework to extend the policy’s guidance beyond the sparse demonstration data.\n- Experiments are carried out on DMControl and MetaWorld benchmarks. Detailed ablation studies are presented to analyze the effect of the regularizer ratio (α) and the value gradient weight (β) on performance, with additional discussion of challenging tasks such as cartpole swingup and quadruped-run.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors should provide clear, formal definitions of both the Embodiment Knowledge Barrier and the Demonstration Knowledge Barrier. This could include mathematical formulations or metrics that quantify the gap between the oracle and learned models. A rigorous treatment would help readers better appreciate the extent to which each barrier is mitigated by the proposed modifications.\n\n2. It is important for the authors to more clearly differentiate their approach from prior work. For the online interaction and surrogate rewards, the paper should emphasize what is non-trivial about this integration.\n\n3. Figure 1 and the accompanying discussion should more explicitly show how the introduction of online interactions, coupled with the data-driven regularizer, reduces the EKB. If possible, providing quantitative or visual evidence would help substantiate the claim.\n\n4. The paper would benefit from experiments on more challenging tasks or from comparisons with alternative methods that use hand-designed or learned surrogate rewards. Furthermore, a more detailed discussion on how the proposed method could be adapted or scaled to address real-world robotics problems—where domain shifts and action space mismatches are significant—would enhance the practical impact of the work.", "broader_impact_concerns": "No Broader Impact Concern required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for the detailed rebuttals for the reviewers' comments and revision of the manuscript. The authors addressed the concerns raised by the reviewers appropriately and all reviewers are satisfied with the revised manuscript. Therefore, I recommend its acceptance as it is. Congratulations!
Decision: Accept (poster)