Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Models

Authors: Xingyuan Zhang, Philip Becker-Ehmck, Patrick van der Smagt, Maximilian Karl
Primary area: reinforcement learning
Keywords: World Models, Foundation Models, Pretraining, Imitation Learning from Observation, Decision-making

Abstract:
Pretraining and finetuning models has become increasingly popular in decision-making. But there are still serious impediments in Imitation Learning from Observation (ILfO) with pretrained models. This study identifies two primary obstacles: the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB). The EKB emerges due to the pretrained models' limitations in handling novel observations, which leads to inaccurate action inference. Conversely, the DKB stems from the reliance on limited demonstration datasets, restricting the model's adaptability across diverse scenarios. 
We propose separate solutions to overcome each barrier and apply them to Action Inference by Maximising Evidence (AIME), a state-of-the-art algorithm.
This new algorithm, AIME-NoB, integrates online interactions and a data-driven regulariser to mitigate the EKB. Additionally, it uses a surrogate reward function to broaden the policy's supported states, addressing the DKB. Our experiments on vision-based control tasks from the DeepMind Control Suite and MetaWorld benchmarks show that AIME-NoB significantly improves sample efficiency and converged performance, presenting a robust framework for overcoming the challenges in ILfO with pretrained models.

Decision: reject
Reviewer scores: 6, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729976800355

thread_type:
official_review

<thread_object>
creator: reviewer_vbjw

<content>
summary:
**Problem setting**: have access to a demonstration data set without actions, an offline data set of suboptimal experience, and the ability to sample experience online without reward feedback

**Proposed approach**: (1) balance the offline and online datasets when updating the model, (2) combine AIME and AIL objectives (though also considers other alternatives to AIL)

**Experiments**: comparisons to relevant prior methods on DMC and MetaWorld tasks with visual observations

strengths:
- Writing is fairly straightforward to understand 
- Results show clear improvements over relevant methods
- The paper includes fairly extensive ablations/empirical analysis

weaknesses:
Overall, the technical contribution seems like a fairly basic combination of prior works. While many methods build upon prior works, this particular combination seems closer to an implementation choice than a significant technical contribution. The method is also more complex than prior methods as it introduces one hyperparameters and combines multiple objectives. 

Other weaknesses
- It’s a bit unclear how this connects to a real-world problem. In robotics, you typically either collected data from a robot, where you can get actions or something close to them, or videos of humans/animals, which lacks actions but also has significant domain shift. Connecting the problem statement to a real world problem would be helpful to motivate the paper’s significance.
- EKB and DKB are defined quite informally and then are used extensively, sometimes in a hand-wavy way. I’m not sure if introducing them is helpful for understanding. I think it would be better to either remove them or define them more formally and measure the extent to which they contribute to poor performance.
- I appreciate that the experiments are on visual observations, though they would be stronger if they included more complex tasks such as dextrous tasks, longer horizon tasks, or tasks with greater diversity/generalization

Some relevant related works to consider citing/discussing:
- Mobile (https://arxiv.org/abs/2102.10769), though uses low-dim observations
- VMAIL (https://arxiv.org/abs/2107.08829), though assumes actions in demos

questions:
See suggestions in the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730514180971

thread_type:
official_review

<thread_object>
creator: reviewer_skmk

<content>
summary:
This paper studies the problem of imitation learning (IL) from (visual) observation, i.e., demonstrations do not contain action information using pretrained world models. The paper identifies two key bottlenecks in current IL algorithms, namely (1) OOD observations, and (2) OOD task configurations, which the authors refer to as the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB), respectively. The key technical contribution is an algorithmic extension of the method Action Inference by Maximizing Evidence (AIME); the core idea is to finetune the learned IL policy via limited online interaction and a RL objective, with surrogate rewards derived from the demonstration dataset. Experiments are conducted on DMControl and Meta-World from visual observations, and the proposed method compares favorably to both AIME without finetuning as well as other IL methods that can be finetuned online (BCO, OT, PatchAIL).

strengths:
- The paper is generally well written and easy to follow. Illustrations (Figure 1 in particular) are useful for understanding the proposed method, and Section 2 provides sufficient background for an unfamiliar reader to appreciate the contributions.
- Experiments are reasonably extensive, covering a variety of tasks from DMControl and Meta-World, as well as several (what appears to be) baselines appropriate for the problem setting. Empirical results are strong compared to baselines.
- It is a purely empirical paper, but there are sufficient ablations to understand how the algorithm behaves in different settings (e.g. number of demos), and which components are the main drivers of performance (e.g. choice of surrogate reward).

weaknesses:
- I appreciate that the authors conduct experiments on both tasks from DMControl and Meta-World, and consider tasks with varying difficulty in the case of Meta-World. However, I would have liked to see a few examples of tasks that are more challenging, i.e., where the proposed method (along with baselines) struggle a bit more. For DMControl this could be e.g. any of the Humanoid or Dog tasks, and for Meta-World this could be e.g. Stick Pull / Push or Pick Place (Shelf).
- I find that some of the claims / conclusions are a bit exaggerated relative to what the experimental results show. For example, the authors claim that " the model pretrained on MW-mt50 offers much better results" (L424) while the results in Figure 4(c) show a fairly small difference between the two curves (absolute ~15% increase in success rate on avg it seems). I would prefer claims to accurately reflect the evidence.

questions:
I would like the authors to address my comments listed in "weaknesses" above during the rebuttal. I have one additional question:

- Figure 10 indicates that the proposed method succeeds in all of the considered tasks except Cartpole Swingup. I would have expected this task to be the easiest of them all; can the authors please comment on why the method fails on this particular task?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693464479

thread_type:
official_review

<thread_object>
creator: reviewer_4pzy

<content>
summary:
This paper extends AIME to AIME-NoBarries, which addresses EKB with online interaction and DKB with online trajectories labeled with surrogate rewards. The authors evaluate the effectiveness of AIME-NoB in DMC and MetaWorld.

strengths:
- AIME-NoB alleviates EKB through online interaction. To address DKB, the most natural way is to collect more expert demonstrations, which is expensive in terms of robot manipulation. Therefore, AIME-NoB circumvents this burden by providing reward signals using surrogate models and optimizes the policy with reward-labeled online trajectories.
- Experiments in DMC and MetaWorld show that AIME-NoB can bring significant improvement to AIME.

weaknesses:
- The logic is confusing. The authors state that in order to alleviate EDK, they additionally use online interactions. It is obvious that online trajectories will bring more benefits since the behavior policy to collect offline datasets differs from the current policy. So the improvement compared to AIME might be largely caused by the setting difference rather than algorithm improvement.
- The setting is very complicated. To my understanding, AIME's setting includes an offline dataset $(s_{\mathbf{off}},a_{\mathbf{off}})$ for world model training, an expert dataset $(s_{\mathrm{expert}})$ for imitation learning. And AIME-NoB additionally assumes access to the environment to collect an online dataset $(s_{\mathrm{on}},a_\mathrm{on})$. The complexity of the setting makes it hard to make fair experimental comparisons as baselines might only assume access to a part of the datasets.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730937186220

thread_type:
official_review

<thread_object>
creator: reviewer_yddj

<content>
summary:
This paper studies two barriers in imitation learning from observations, namely the Embodiment Knowledge Barrier (EKB) and Demonstration Knowledge Barrier (DKB). EKB refers to the gap when generalizing to new observations and DKB refers to the gap when generalizing from limited number of expert demonstrations.

It proposes to use online interaction to reduce EKB and introduces a weighted loss between new interactions and pre=training data. For DKB, it proposes to use a surrogate reward, such as a discriminator in AIL.

Experimental results suggest AIME-NoB outperforms baselines with significant margin in DMC and MetaWorld.

strengths:
- Clearly identify the shortcomings of existing IFO methods and propose reasonable solutions to reduce the gaps.
- Extensive experiments on the different design choices, e.g. reward functions, and comparisons with baselines demonstrate solid improvement.
- Paper writing is clear and easy to follow.

weaknesses:
- The ideas to reduce EKB and DKB seems a simple combination of previous methods, such as using online interaction with weighted sampling, adversarial training, etc. It's unclear the distinctions from previous methods (such as AIL, e.g.) is significant enough.
- These environments aren't hard to come up with a hand-designed reward or learning a surrogate reward. Lack of comparisons with Dreamer-like methods using hand-designed or surrogate reward.

questions:
- How do you differentiate from previous work that uses online interactions and surrogate rewards?
- How does it compare with RL with some hand-designed reward or a learned surrogate reward?
- Does the method generalize to real-world environments, such as learning from videos?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732191083367

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
We sincerely thank all reviewers for their time and constructive feedback. We are pleased to note that the reviewers consistently acknowledge several key strengths of our paper:

- The clear and accessible writing style (Reviewers Yddj, SkmK, vbjw)
- The comprehensive experimental evaluation (Reviewers Yddj, SkmK, vbjw)
- The significant performance improvements achieved by AIME-NoB compared to existing methods (All reviewers)

In response to the reviewers' suggestions, we have made the following major enhancements to the paper:

1. Theoretical Foundation:
    - As suggested by Reviewer vbjw, we have provided formal definitions of the EKB and DKB concepts in Appendix K, strengthening the theoretical underpinning of our work.
2. Extended Experimental Validation:
    - Following recommendations from Reviewers Yddj, SkmK, and vbjw, we have included additional experiments in Appendix L that demonstrate: a) AIME-NoB's capability to make progress on and potentially solve complex tasks like Humanoid - a first in ILfO to the best of our knowledge b) more sample efficiency compared to Dreamer, a state-of-the-art model-based RL algorithm, even when Dreamer has access to true environmental rewards
3. Additional Improvements:
    - Soften some claims based on Reviewer SkmK's suggestions
    - Expanded citation coverage as recommended by Reviewer vbjw

All modifications are highlighted in blue in the revised manuscript. We believe these enhancements substantially strengthen our paper by providing stronger theoretical foundations and broader experimental validation. Detailed responses to individual reviewer comments are provided below. We kindly request reviewers to consider raising their scores if our responses adequately address their concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732725749848

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New Revision

comment:
We would like to extend our gratitude for all the constructive reviews, with special thanks to reviewers SkmK and vbjw for their participation in the discussion. We have updated the revised paper to a new version, incorporating the following changes:

- Moved the formal definitions of EKB and DKB to the main text, as suggested by reviewer vbjw.
- Added more results on the humanoid experiment, including additional results on humanoid-walk and humanoid-run and more seeds.

We believe these revisions enhance the clarity of the paper and better demonstrate the potential of AIME-NoB in complex tasks. We kindly request reviewers, especially Yddj and 4pZy, to engage in the discussion and raise any additional questions if something remains unclear. We also ask you to consider raising your scores if our responses have adequately addressed your concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313098069

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
summary of the discussion phase

comment:
Dear reviewers and AC,

We would like to thank you again to spend time to review our paper and join the discussion phase. As the discussion phase comes to a close, we would like to summarize the key points to facilitate the decision-making process.

Reviewers generally agree on the strength of the paper:
- The clear and accessible writing style (Reviewers Yddj, SkmK, vbjw)
- The extensive experimental evaluation (Reviewers Yddj, SkmK, vbjw)
- The significant performance improvements achieved by AIME-NoB compared to existing methods (All reviewers)

During the rebuttal, three new contents have been added:
- The formal definition of EKB and DKB, as requested by reviewer vbjw.
- Experiments on more challenging tasks, e.g. Humanoid, as requested by reviewer SkmK and reviewer vbjw.
- Experiments to compare AIME-NoB with Dreamer, as requested by reviewer Yddj.

These additions were acknowledged by the reviewers as helpful and further strengthened the paper.

After the rebuttal:
- While maintaining the score, reviewer SkmK has no concerns and lean toward acceptance.
- Reviewer Yddj only has concerns regarding the limit novelty of the method, but also lean toward acceptance.
- Reviewer vbjw has two remaining concerns: 1) the limit novelty of the method, but also acknowledged as a subjective opinion; 2) the direct applicability on robotics problem. Although we offer some examples of use cases, they did not changes the reviewer's mind. 
- Reviewer 4pZy has a contradictory opinion with all other reviewers, concerning the clarity of the paper while the paper is acknowledged by all the other reviewers to be well-written and easy to follow. Although we have submitted the rebuttal **18 days** ago to clarify the misunderstandings, unfortunately we didn't get any response from reviewer 4pZy in the entire discussion phase.

We still believe with the clear intuition and the strong performance, the paper offers valuable insights to the field. 

We hope this summary assists in the decision-making process. Thank you once again for your time and efforts in reviewing and discussing our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) The paper introduces AIME-NoB, an algorithm designed to improve Imitation Learning from Observation (ILfO) using pre-trained world models. The authors define two barriers in imitation learning from observation (ILfO) methods: the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB).  The EKB describes the limitation of a pretrained model when confronted
with novel observations and actions beyond its training experience. The DKB describes the generalization from a limited number of expert demonstrations in imitation learning. The proposed approaches to address these limitations are: (1) balance the offline and online datasets when updating the model, (2) combine AIME and AIL objectives. 

(b) Strengths:
- Problem Definition: The paper identifies and analyzes the EKB and DKB limitations in current ILfO methods using pretrained models. The clarity improved during rebuttal.
- Effective Solutions: The authors propose practical solutions for each knowledge barrier that show significant performance improvements.
- Ablation Studies: The authors perform thorough ablation studies and sensitivity tests to analyze the contribution of different components, such as the surrogate reward and data regularizer, to the overall performance.
- Clear Presentation: Multiple reviewers praised the paper's clear and accessible writing style, which helped in understanding the proposed approach. The use of figures (e.g. Figure 1) was also cited as being helpful.
- Strong results on DMC and meta-world.

(c) Weaknesses:
- Limited novelty: Several reviewers questioned the novelty of the proposed approach. The core method combines online interaction, a data-driven regularizer, and a surrogate reward, all of which have been previously used in some form.
- Incremental Contribution: Some reviewers felt that combining existing techniques, although effective, was closer to an implementation choice than a significant technical contribution.
- Realistic impact: The reviewers questioned the direct applicability to real-world robotics problems since actions are typically available, or can be estimated in robotics, or real-world videos often have a large domain shift.

(d) My decision is to reject the paper. 

I think the paper does a good job at motivating the design choices above, and has thorough experiments with strong results, especially with the new additions during the rebuttal, which show that the design choices significantly improve over prior methods. However, in terms of whether this paper makes significant contributions at least in (1) enough novel contributions in method or, (2) significant realistic impact. As the reviewers pointed out, it seems to be a weaker case here.