Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration

Authors: Heyang Zhao, Xingrui Yu, David Mark Bossens, Ivor Tsang, Quanquan Gu
Primary area: reinforcement learning
Keywords: Optimistic exploration, Curiosity, Imitation learning, Inverse reinforcement learning, Reinforcement learning

Abstract:
Imitation learning is a central problem in reinforcement learning where the goal is to learn a policy that mimics the expert's behavior. In practice, it is often challenging to learn the expert policy from a limited number of demonstrations accurately due to the complexity of the state space. Moreover, it is essential to explore the environment and collect data to achieve beyond-expert performance. To overcome these challenges, we propose a novel imitation learning algorithm called Imitation Learning with Double Exploration (ILDE), which implements exploration in two aspects: (1) optimistic policy optimization via an exploration bonus that rewards state-action pairs with high uncertainty to potentially improve the convergence to the expert policy, and (2) curiosity-driven exploration of the states that deviate from the demonstration trajectories to potentially yield beyond-expert performance. Empirically, we demonstrate that ILDE outperforms the state-of-the-art imitation learning algorithms in terms of sample efficiency and achieves beyond-expert performance on Atari and MuJoCo tasks with fewer demonstrations than in previous work. We also provide a theoretical justification of ILDE as an uncertainty-regularized policy optimization method with optimistic exploration, leading to a regret growing sublinearly in the number of episodes.

Decision: accept (poster)
Reviewer scores: 5, 8, 3, 8

Reviews and discussion:
<thread>
creation date:
1729627356318

thread_type:
official_review

<thread_object>
creator: reviewer_4bf6

<content>
summary:
This paper proposes a new framework for imitation learning, which succinctly, leverage informed interaction to improve policy performance in the absence of an abundance of expert demonstrations. In particular, two exploration strategies are considered: optimistic policy optimization with an exploration bonus, and curiosity-driven exploration of states beyond the demonstration trajectories. The authors provide extensive theoretical justification and prove performance bounds to support their algorithm, and also provide empirical results to support their method, testing metrics in both sample efficiency and final performance.

strengths:
* The proposed method seems novel and to be well-grounded in theoretical derivations. While learning from limited expert demonstrations has been explored in prior work, this work introduces a unique combination of two exploration strategies to tackle this regime. 
* The paper provides solid theoretical foundations for ILDR, and is thorough in stating their assumptions. As noted, Theorem 4.7 is the first theoretical guarantee for imitation learning in MDPs with nonlinear
function approximation.
* The problem is well motivated, and algorithms are well presented. Particularly, the integration of optimistic policy optimization and curiosity-driven exploration are clearly labeled. The empirical experiments also do a good job ablating on these two exploration models, highlighting both are needed for best performance.
* Relatedly, the empirical results are promising, demonstrating that ILDE achieves better performance over baselines on metrics such as final reward, # games better than expert, and average performance against experts over the considered environments.

weaknesses:
* There are more modern baselines that come to mind when looking at the few expert-demonstration regime, such as IQLearn [1], which also demonstrate reasonable performance in the low expert-data regime, or the model-free and/or model-based inverse RL algorithms in Hybrid Inverse Reinforcement Learning [2], which also prove competitive sample complexity bounds. The experiments section could benefit from slightly more baselines.
* Although the theoretical framework is well-explained, the practical implementation details could use a bit more clarity. In particular, how are exploration bonuses computed in practice, and how much work is involved in hyperparameter search (i.e. the original ranges considered, type of search used, and how the authors recommend searching over these hyperparameters when faced with a new environment/dataset)? (Relatedly, see Question 2 below.)
* Relatedly, I personally felt the tradeoff between exploration and imitation could have been explored a bit more thoroughly in the paper. For example, I'm curious how algorithms may perform in situations where the demonstrations may not be fully optimal. (Relatedly, see Question 3 below.)
* The ILDR-NPG variant as I understood it seems susceptible to incur large computational costs due to using natural policy gradients for policy optimization. The paper doesn't seem to fully address this issue/potential ways to mitigate it.


[1] Garg, Divyansh, et al. "Iq-learn: Inverse soft-q learning for imitation." Advances in Neural Information Processing Systems 34 (2021): 4028-4039.

[2] Ren, Juntao, et al. "Hybrid inverse reinforcement learning." International Conference on Machine Learning (2024).

questions:
1. While the authors provide a practical instantiation, I am under the impression that proposed ILDE framework should be flexible, and can be generalized to other empirical instantiations. If this is true, have the authors explored applying these exploration bonuses on top of existing methods and seeing how they fair?
2. How sensitive is ILDE to the choice of hyperparameters, such as the exploration bonus weight and the learning rate for curiosity-driven exploration? What is the amount of hyperparameter tuning that one had to search over in order to reach the optimal performance for these exploration bonus trainings? 
3. How does ILDE perform in environments where the expert demonstrations are noisy or imperfect? What is the consequence of data quality on the two exploration bonuses, and how much additional tuning may be needed to stabilize performance? This would provide insight into ILDE’s robustness in real-world settings.
4. I'm wondering whether ILDE could be combined with techniques in model-based (inverse) reinforcement learning to further improve sample efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730212592939

thread_type:
official_review

<thread_object>
creator: reviewer_fejp

<content>
summary:
This paper introduces a novel imitation learning algorithm, ILDE (Imitation Learning with Double Exploration). The goal of ILDE is to achieve beyond-expert performance by improving sample efficiency and addressing the challenges posed by limited demonstrations. ILDE implements two exploration strategies: (1) optimistic policy optimization with an exploration bonus that encourages the exploration of high-uncertainty state-action pairs, and (2) curiosity-driven exploration that rewards states deviating from the expert’s demonstration trajectory. Empirical results show that ILDE outperforms state-of-the-art algorithms in terms of sample efficiency and return on Atari and MuJoCo tasks. The authors also provide a theoretical framework, showing that ILDE’s regret grows sublinearly with the number of episodes.

strengths:
1. This paper proposes a novel ILDE framework, which combines optimistic exploration with curiosity-driven exploration for imitation learning.

weaknesses:
1. **Theoretical Limitations**
    1. First, the assumptions made in this paper are overly strong. In Assumption 4.2, the authors directly assume that the statistical error resulting from using a finite set of expert demonstrations is small. This assumption is both strong and unreasonable, as a primary goal in imitation learning theory is to examine the relationship between the number of expert trajectories and the error bound [1]. Additionally, Assumption 4.1 presumes that the loss function is convex with respect to the reward parameters, making the reward optimization a straightforward online convex optimization problem. In practice, however, the reward optimization problem may be highly non-convex due to the use of neural networks.
    2. Second, the proposed method in Algorithm 1 requires knowledge of the entire MDP transition function. In line 7 of Algorithm 2, computing the intrinsic reward $\mathcal{L}_{\tau^{E}, h}$ depends on the true transition function. However, if the true transition function were known, online exploration would no longer be necessary.
2. **Experimental Limitations**
    
    In the experiments, this paper does not compare its approach with current state-of-the-art (SOTA) imitation learning (IL) methods. In Atari games, the SOTA method is IQ-Learn [2]. In MuJoCo environments, the SOTA methods include IQ-Learn and HyPE [3]. The absence of these methods in the experiments makes it unclear whether the proposed approach outperforms current methods.
    
3. **Unsubstantiated Conjecture**
    
    The goal of this paper is to develop an IL approach capable of achieving performance beyond the expert level. To this end, the paper is based on the conjecture that optimizing the uncertainty-regularized loss shown in Eq. (3.6) enables performance surpassing the expert’s. However, no evidence or justification is provided to support this conjecture.
    

References:

[1] Nived Rajaraman et al., Toward the fundamental limits of imitation learning.

[2] Divyansh Garg et al., Iq-learn: Inverse soft-q learning for imitation.

[3] Juntao Ren et al., Hybrid inverse reinforcement learning.

questions:
Typos:

1. In line 41, the reference of GAIL is wrong.
2. In line 256, “desccent” should be descent.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730380606673

thread_type:
official_review

<thread_object>
creator: reviewer_hcfx

<content>
summary:
This paper focuses on imitation learning from limited demonstrations, where the goal is to learn a policy that either imitates the expert directly or learns a policy that is better than the expert. Prior techniques focused on exploration-based approaches for this problem, with the goal of designing useful intrinsic motivation bonuses to incentivize the agent to explore its (large) state space to ultimately learn a better-than-expert policy. This paper proposes the use of two bonuses: an exploration bonus that encourages the agent to visit novel state-action pairs in the environment, and another bonus that incentivizes the agent to deviate from the expert trajectories themselves. The authors propose theoretical justification of their method, achieving a poly(H, T) regret w.r.t. the true expert.

strengths:
I am not the best theoretician by any means, but this paper is very theoretically sound. I think the regret analysis was done well, which is the main theoretical contribution of this work.

Experimental results were quite expansive and good. The authors seemed to ablate the right variables (e.g. what happens if we turn our state-action based exploration bonus off, what happens if we turn our imitation reward off) to analyze exactly what is different about their method compared to a standard baseline (e.g. GIRIL). In terms of domains of interest, the authors also covered a variety of benchmarks across a variety of RL benchmarks, including Atari and MuJoCo tasks.

weaknesses:
It would be interesting to see how just the state-action exploration bonus does as well (e.g. the state entropy bonus stays, but the GIRIL-like intrinsic reward is not used). To me, both of these methods seem to be doing the same thing (one on the state-action space as a whole, and one focusing on the expert trajectories), so it would be good to see an experiment that keeps the state entropy bonus, doesn't use the demonstration-based intrinsic reward, and keeps the standard imitation learning reward, which I understand is different from GIRIL (where their intrinsic reward bakes in both the true demonstration actions and the demonstration-based intrinsic reward). This is pretty minor, but would love to see this myself.

questions:
I don't have any additional questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730579546671

thread_type:
official_review

<thread_object>
creator: reviewer_4jou

<content>
summary:
This paper addresses the challenge of surpassing expert performance through imitation learning. The authors propose ILDE which modifies the Generative Adversarial Imitation Learning (GAIL) framework by incorporating an intrinsic reward term to encourage exploration. The method is supported by theoretical analysis quantifying the algorithm's regret bounds. Empirical evaluation on 6 Atari games and 4 MuJoCo tasks demonstrates ILDE's improvements over baseline methods.

strengths:
- This paper studies a seemingly interesting problem of learning a policy with imitation learning to be much better than the demonstration.
- The paper offer some theoretical analysis of the proposed method.

weaknesses:
- The problem formulation of optimizing the GAIL loss + intrinsic reward to achieve a better performance than the expert is not convincing.
- The proposed method is only evaluated on limited tasks and the results lack explanation.

questions:
Majors:
- As mentioned in the weakness part, the biggest concern from me is the formulation. Could you please elaborate why, intuitively, adding an intrinsic reward term to the GAIL objective (Eq. 3.3) can help the policy to achieve a better performance than the expert? I think to achieve a better performance than the expert, the agent must understand the intention of the expert and extrapolate it. I cannot see how an intrinsic reward guarantee to help for this. Maybe there are some hidden assumptions about the task or the demonstrations?
- In the introduction, you list the formulation of the problem itself as one of the contribution, but you write the formulation in the Preliminary section. Could you elaborate what is the difference between the formulation in this paper with previous works, for example [1]?
- From the results in Table 1, ILDE w/o $r^k$ is much better than ILDE w/o $b$. Is this a desired behaviour that using only the intrinsic reward allows achieving a better performance than the imitation reward? Does this not imply that the selected 6 Atari tasks can be solved by only doing exploration?

Minors:
- In line 41, your citation of GAIL is wrong.
- No number is bold in the Qbert line of Table 1.
- In the result section, you only include Table 1 in the main text but spend the majority of the text discussing the results on Tables 9 and 10 in the appendix. Can you reorganize the paper to make the text and table better align with each other?

[1] Yu *et al.*, Intrinsic Reward Driven Imitation Learning via Generative Model, ICML 2020
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper talks about imitation learning with an additional exploration bonus and claims that it can learn a policy that outperforms the expert. The strength of the paper is that it empirically demonstrates that adding both types of bonus is actually quite effective for the imitation learning algorithm. 

The weakness comes from the application scope of the proposed approach, particularly under what situations using an exploration bonus can allow the IL agent to perform better than the expert. We encourage the authors to explicitly address this concern in the revised version since, in general, without any additional information (e.g., ground truth reward), it is hopeless for IL to outperform the expert, and an exploration bonus can, in worst case guide the learner to a low-reward region instead of the high-reward region.