RILe: Reinforced Imitation Learning

Authors: Berat Mert Albaba, Sammy Christen, Thomas Langerak, Christoph Gebhardt, Otmar Hilliges, Michael J. Black
Primary area: reinforcement learning
Keywords: Imitation Learning, Inverse Reinforcement Learning, Reinforcement Learning

Abstract:
Reinforcement Learning has achieved significant success in generating complex behavior but often requires extensive reward function engineering. Adversarial variants of Imitation Learning and Inverse Reinforcement Learning offer an alternative by learning policies from expert demonstrations via a discriminator. However, these methods struggle in complex tasks where randomly sampling expert-like behaviors is challenging. This limitation stems from their reliance on policy-agnostic discriminators, which provide insufficient guidance for agent improvement, especially as task complexity increases and expert behavior becomes more distinct. We introduce RILe (Reinforced Imitation Learning environment), a novel trainer-student system that learns a dynamic reward function based on the student's performance and alignment with expert demonstrations. In RILe, the student learns an action policy while the trainer, using reinforcement learning, continuously updates itself via the discriminator's feedback to optimize the alignment between the student and the expert. The trainer optimizes for long-term cumulative rewards from the discriminator, enabling it to provide nuanced feedback that accounts for the complexity of the task and the student's current capabilities. This approach allows for greater exploration of agent actions by providing graduated feedback rather than binary expert/non-expert classifications. By reducing dependence on policy-agnostic discriminators, RILe enables better performance in complex settings where traditional methods falter, outperforming existing methods by 2x in complex simulated robot-locomotion tasks.

Decision: reject
Reviewer scores: 8, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729049307354

thread_type:
official_review

<thread_object>
creator: reviewer_rivh

<content>
summary:
The paper proposed Reinforced Imitation Learning (RILe) to learn the reward function for policy learning with AIL and teacher-student framework. The method is evaluated on LocoMujoco benchmark and it outperforms GAIL and AIRL.

strengths:
- The idea of Decoupled Reward-function Learning with AIL is novel.

- RILe demonstrates better performance on the LocoMujoco benchmark than baselines (GAIL and AIRL).

weaknesses:
- The connection between motivation and the proposed method is weak. More intuition or theoretical proofs are needed.
- While the paper highlights some issues with existing methods, it lacks detailed references and experiments to convincingly demonstrate how RILe addresses these issues. More ablation studies or theoretical analysis could strengthen the argument.
- The scope of the experiment is limited to comparing only GAIL and AIRL on one benchmark. More baselines [1,2,3,4] and tasks should be evaluated.

questions:
- In line 241, the authors mention "the trainer’s RL-based approach optimizes for long-term performance rather than immediate feedback".  If the benefit comes from RL, all AIL methods also share it since they all use RL for policy learning. How does the RILe handle this?

- In Section 5.1 and Figure 3, it’s stated that RILe offers better rewards for guiding student policies, but in the second and third columns, GAIL appears to learn faster. The reward contours for GAIL and AIRL also seem closer to the ground truth. Moreover, learning with a fixed reward function is the most common approach in typical RL. Could the authors elaborate more on how the evolving feature benefits policy learning?

- Why did the authors choose LocoMujoco, which lacks action labels, rather than more widely used Mujoco tasks for Learning from Demonstration (LfD)? I would recommend limiting the scope to Learning from Demonstration (LfD), where action labels are available. Besides, IQLearn did provide learning with state-only rewards version in the paper. If the authors want to demonstrate the application on Learning from Observation (LfO), I think it will be better to conduct experiments separately with a different set of baselines [5,6,7,8].

- In Figure 4, how are normalized scores and steps defined? The result seems to be applicable to many online imitation learning methods. Could the authors provide more details on the purpose of these metrics in the context of RILe?

- The authors mentioned RILe improves the computational efficiency compared to other IRL works. However, RILe needs to train an extra trainer network compared to GAIL. Is there a quantitative analysis on this?

References:

[1] Pomerleau, D. A. (1991). Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1), 88-97.

[2] Papagiannis, G., & Li, Y. (2022, September). Imitation learning with sinkhorn distances. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 116-131). Cham: Springer Nature Switzerland.

[3] Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., ... & Song, S. (2023). Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 02783649241273668.

[4] Lai, C. M., Wang, H. C., Hsieh, P. C., Wang, Y. C. F., Chen, M. H., & Sun, S. H. (2024). Diffusion-Reward Adversarial Imitation Learning. arXiv preprint arXiv:2405.16194.

[5] Torabi, F., Warnell, G., & Stone, P. (2018). Behavioral cloning from observation. arXiv preprint arXiv:1805.01954.

[6] Liu, M., Zhu, Z., Zhuang, Y., Zhang, W., Hao, J., Yu, Y., & Wang, J. (2022). Plan your target and learn your skills: Transferable state-only imitation learning via decoupled policy optimization. arXiv preprint arXiv:2203.02214.

[7] Torabi, F., Warnell, G., & Stone, P. (2018). Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158.

[8] Huang, B. R., Yang, C. K., Lai, C. M., Wu, D. J., & Sun, S. H. (2024). Diffusion Imitation from Observation. arXiv preprint arXiv:2410.05429.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730375511195

thread_type:
official_review

<thread_object>
creator: reviewer_d3w2

<content>
summary:
RILe (Reinforced Imitation Learning environment) is a framework that enhances reinforcement and imitation learning by introducing a dynamic, adaptive reward function responsive to agent performance. Addressing limitations of traditional IL methods like GAIL and AIRL, RILe’s trainer-student model continuously optimizes rewards through reinforcement learning, effectively guiding the student agent to mimic expert demonstrations in complex tasks. Experiments in maze and humanoid locomotion tasks demonstrate RILe's advantages in scalability, robustness, and adaptability, highlighting its effectiveness for high-dimensional control environments.

strengths:
1. **Motivation and Intuition**: The motivation for reducing reliance on static discriminators is compelling, as complex tasks often necessitate adaptive guidance.

2. **Novelty**: The framework’s use of a trainer agent that dynamically learns a reward function to align with student policy is an innovative approach that enhances traditional adversarial learning strategies.

3. **Technical Contribution**: RILe’s decoupling of the reward function learning from the student’s policy learning shows clear benefit, especially in tasks requiring extensive exploration, as the framework enables more progressive feedback than binary classifications.

4. **Clarity**: The paper is well-structured, with clear theoretical explanations and effective visualizations. Figures, such as Figure 3, successfully illustrate the evolution of reward functions across training stages.

weaknesses:
1. **Limited Baselines** (important): The experiment lacks more recent baselines (BC from 2010, GAIL from 2016, and AIRL from 2018). It would strengthen the work to include comparisons with more recent AIL methods, such as DRAIL [1], V-MAIL [2], or ARC [3]. Especially, they should cite DRAIL [1], which also focus on improving learning efficiency, particularly in complex, high-dimensional environments like RILe does.

2. **Lack of Ablation Study**: The paper would benefit from additional ablation studies to better analyze the contributions of individual components within the RILe framework.

3. **Limited Task Diversity**: RILe is tested only on LocoMujoco (locomotion tasks). Expanding experiments to include navigation or manipulation tasks would improve the generalizability of the framework.

[1] Lai, Chun-Mao, et al. "Diffusion-Reward Adversarial Imitation Learning." arXiv preprint arXiv:2405.16194 (2024).

[2] Rafailov, Rafael, et al. "Visual adversarial imitation learning using variational models." Advances in Neural Information Processing Systems 34 (2021): 3016-3028.

[3] Deka, Ankur, Changliu Liu, and Katia P. Sycara. "ARC-Actor Residual Critic for Adversarial Imitation Learning." Conference on Robot Learning. PMLR, 2023.

questions:
1. The experimental setup relies primarily on older baselines (e.g., BC from 2010, GAIL from 2016, and AIRL from 2018). Including comparisons with more recent adversarial imitation learning (AIL) methods, such as DRAIL [1], would provide a more comprehensive assessment of RILe’s performance, especially given that DRAIL also address challenges related to learning efficiency in complex, high-dimensional environments. The paper needs to include DRAIL [1]. Doing so would significantly enhance the paper’s impact. I would be willing to score it higher with this addition.

2. How does the dynamic reward adjustment by the trainer agent handle rapidly changing or highly stochastic environments, where optimal behavior may vary significantly within episodes?

3. RILe involves multiple components with different objectives (trainer, student, and discriminator). How sensitive is RILe to hyperparameter choices across these agents, and are there any recommended tuning strategies?

4.  The paper shows RILe’s ability to perform well with noisy expert data, but how robust is this approach if the noise distribution changes dynamically during training, or if the expert data is sparse?

[1] Lai, Chun-Mao, et al. "Diffusion-Reward Adversarial Imitation Learning." arXiv preprint arXiv:2405.16194 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714826150

thread_type:
official_review

<thread_object>
creator: reviewer_bwr3

<content>
summary:
This paper proposes a novel method for adversarial imitation learning. Unlike established methods such as GAIL and AIRL, which learn a policy using reinforcement learning (RL) under a reward function derived directly from the discriminator, the proposed method, RILe, introduces an additional RL component called the trainer agent. This trainer agent generates the reward function for policy learning, effectively serving as an intermediary layer between the discriminator and the policy learning component (referred to as the student agent). The paper claims that this additional trainer component, as an RL agent, is able to dynamically provide an adaptive reward function throughout the student agent's policy learning process, resulting in superior imitation learning performance in complex settings.

strengths:
1. This paper aims to address some of the challenges faced by existing imitation learning methods in complex settings, a valuable problem to study with a wide range of potential practical applications.

2. The introduction of an additional trainer agent in the proposed method, RILe, is a novel design.

3. The discussion of related literature is clear and effectively situates the proposed method within the context of existing research.

weaknesses:
1. My major concern with the paper is that several major claims regarding the capabilities of the proposed method, RILe, are not sufficiently supported by rigorous analysis or adequate experimental evidence.
* Claim of RILe's adaptability: For instance, in Section 5.1, it is stated that "RILe demonstrates a more adaptive reward function that evolves with the student's progress ...". However, this claim is inadequately supported by the limited results in Figure 3, which only includes a single, simple setting and is open to subjective interpretation. Similar claims in the Introduction and Discussion suggest that RILe’s adaptability leads to more efficient policy learning for the student agent, but there is insufficient experimental evidence to validate this connection.
* Claim of RILe outperforming state-of-the-art adversarial IL methods: Although the main results in Section 5.2 show that RILe outperforms the baseline methods, GAIL and AIRL, on several LocoMujuco tasks, these results lack sufficient details on the experimental setup, such as the settings of tasks used for evaluation and the hyperparameter values of the compared methods.
* Additionally, the proof of the theoretical argument, Lemma 1, is neither rigorously presented nor complete.

2. Another concern I have with the proposed RILe method is that it stacks one RL agent (the trainer agent) on top of another (the student agent). The trainer agent's state transition depends on the student agent's policy since its states are represented by the state-action pairs generated by the student agent. Meanwhile, the student agent's policy is influenced by the trainer agent's policy through the reward function generation. This structure creates a dependency in the trainer's RL environment (i.e. the state transition of the RL environment faced by the trainer agent is not exogonous to its policy), making the MDP ill-defined and potentially introducing extra instability to the training framework.
* In fact, the paper briefly addresses training instability in the Discussion section and mentions a workaround of freezing the trainer midway through training.

3. The proposed optimization objective of the trainer agent, Equation 8, lacks a clear explanation of the rationale behind this particular choice. (See Question 1 below)

4. Overall, the paper is not well-organized and includes errors or inaccuracies in some parts.
* In Section 3.2, entropy regularization is introduced as part of the RL problem formulation in this work, defined in Equation 1. However, in Section 3.3, the RL formulation is presented without entropy regularization, which is then added again in Equation 2.
* In Section 4, the first term on the right-hand side of Equation 14 should be $v(D_{\phi}(s_t))a_t$ instead of $D_{\phi}(s_t)$ to remain consistent with the reward definition in Equation 11.
* The workaround for the training instability (freezing the trainer after meeting certain criteria) is only mentioned in the Discussion section, with no mention of it in the discussion of experimental settings and results.

questions:
1. What is the rationale behind the specific definition of the optimization objective of the trainer agent, $v(D_{\phi}(s, a)) a^T$ (Equation 8)? It is mentioned that "By incorporating $a^T$ into the reward function, the trainer learns to adjust its policy based on the effectiveness of its previous actions", but this explanation is somewhat unclear. Could you clarify why $a^T$ is specifically multiplied with $v(D_{\phi}(s, a))$?

2. In Figure 4, the performance of RILe remains relatively good when the trainer's replay buffer is 100\% expert data, provided that the proportion of expert data in the student's replay buffer is not too large. This outcome is somewhat puzzling, as I would expect that with 100\% expert data in the trainer's replay buffer, neither the trainer nor the discriminator would be trained on any trajectories generated by the student policy, based on the framework illustrated in Figure 2. Am I misunderstanding something here?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730774009640

thread_type:
official_review

<thread_object>
creator: reviewer_rwcm

<content>
summary:
This paper introduces ‘Reinforced Imitation Learning’, a new trainer-student framework. In this framework, the trainer receives feedback from a discriminator and continuously adjusts the output reward in response to the student's behavior. The student is then trained using reinforcement learning based on the reward signal from the trainer. This approach may enable increased exploration behavior in students and more detailed feedback from the trainer. The proposed approach is evaluated in a maze setting, LocoMujoco, and humanoid MuJoCo. Experimental results indicate that the proposed approach outperforms competitive baselines significantly.

strengths:
**Originality & Significance** 

1. The reviewer found the concepts of a custom reward function and the decoupling of reward function learning from the student and discriminator to be quite innovative. The experimental results suggest that this approach effectively addresses the issue of insufficient guidance, a common challenge in early-stage training for adversarial imitation and reinforcement learning methods. This improvement in guidance appears to be particularly beneficial during the initial phases of the learning process.

2. The background section is well-organized and effectively introduces the key concepts. The authors provide a concise overview of reinforcement learning (RL), inverse reinforcement learning (IRL), adversarial imitation learning (AIL), and adversarial reinforcement learning (AIRL). These explanations establish a solid foundation for understanding the proposed approach and its position within the broader literature. Additionally, the authors clearly articulate the distinctions between existing methods and their proposed approach, further highlighting the motivation and novelty of their work. 


**Quality** 

The claims are supported by experimental results. Particularly, table 1 shows that the proposed approach is able to handle complex tasks that baselines didn’t perform well. 
The reviewer appreciated the presentation of figure 3. It clearly illustrates the dynamics of the reward function of the proposed approach and its difference to existing methods. 

**Clarity** 

While some clarification of  the technical contents is needed,  the paper is overall well organized and easy to follow. Please see the weakness section for details.

weaknesses:
1. The reviewer found the reward function for the trainer agent needs further clarification. Specifically, L 241 reads ‘the trainer’s RL-based approach optimizes for long-term performance rather than immediate feedback’, but it is unclear to the reader how the formulation of the reward function (Eq. 8) could achieve it. 

2. It is unclear to the reviewer the role of $a^T$ in the reward function of the trainer (Eq. 8). With $a^T$ in the objective, would it encourage the trainer to generate $a^T=1$ without further constraint? Also, L 314 reads ‘by incorporating $a^T$ into the reward function, the trainer learns to adjust its policy based on the effectiveness of its previous actions’. It is unclear to the reviewer how this is achieved. Please elaborate.

questions:
1. The reviewer's primary concern is how the trainer's reward function contributes to optimizing long-term feedback. Additionally, the reviewer questions the rationale for incorporating the trainer's action into the reward function. Please refer to the "Weaknesses" section for a detailed explanation.  
2. L313, how is the constant determined in $v(x) = 2x - 1$?  
3. How many training environment steps are used for each task?   
4. Why are confidence intervals not included in Table 1?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel method for adversarial imitation learning with three components: a discriminator that provides feedback to a trainer (an RL agent) which is optimized to reward the student (another RL agent). The paper argues that the additional trainer component can adapt the reward function to accelerate the student agent's policy learning process, perhaps increasing exploration. Experiments demonstrate strong results in locomotion tasks.

Reviewers appreciated the novelty of the proposed approach and strong empirical performance (esp. during the early stages of learning).
Authors appreciated the clear discussion of the relationship with prior work and overall clear writing and motivation.

Several reviewers recommended additional baselines, ablation experiments, and evaluation tasks (e.g., manipulation). They also provided feedback on paper organization and suggested that the paper might be strengthened by including additional theoretical results. Reviewers also had questions about why the reward function (Eq. 8) resulted in maximizing for long term performance and details of the experimental setup. Reviewers also suggested that some of the claims should be clarified (e.g., is it the _adaptability_ that is driving improved performance?) and had concerns about the complexity of the proposed approach, as it seems to stack on RL agent on top of another. (this is briefly discussed in the paper's Discussion).

Overall, despite the novelty of the proposed approach, concerns about the experiments (i.e., whether they support the specific claims made in the paper) compel me to recommend that this paper not be accepted.