Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel algorithm, named adversarial imitation learning with preferences (AILP), for learning from demonstrations and preferences. AILP builds upon the well-known adversarial imitation learning (AIL) framework which uses discriminator to construct reward function. To train the discriminator, AILP uses a combination of a BCE classification loss and a hinge loss per preference pair. Then, AILP uses SAC to update policy where the reward term is composed of the trained discriminator. Finally, the authors provide empirical comparison of AILP and baselines on the 6 different manipulation tasks from the meta-world benchmark.
strength_and_weaknesses: **Strength**

AILP successfully handles feedback from demonstrations and preferences effectively.

**Weakness**

- AILP uses two losses, $L_\text{dem}(\phi_i,\mathcal{D},\mathcal{M}_i)$ and $L_\text{pref}(R_i(\phi_i \mathcal{P})$ to update LDRE $\phi_i$. However, I am wondering if a stable update of $\phi_i$ is possible using both losses. 
- In Eq. (10), The authors remove the KL term, but the entropy term is added due to the usage of SAC. Consequently, $\sum_{t=1}^T\mathbb{E}_{p^\pi(s_t)}[-\log\pi_k(\cdot|s_t)]$ is removed. Does it make sense to remove this term without a theoretical analysis?

**Questions**

- I'm just wondering what's the problem with using a loss like $L_\text{dem}(\phi_i,\mathcal{D},\mathcal{M}_i)+ \alpha L_\text{pref}(R_i(\phi_i), \mathcal{P})$? Here, $\alpha\in[0,1]$ is a hyperparameter.
- Why is there no AIL algorithm as a baseline for experimental evaluation? In addition, do baselines also use the pre-trained policy $\tilde\pi$ to maximize state entropy?

clarity,_quality,_novelty_and_reproducibility: AILP is a novel and interesting algorithm. In addition, The pseudocode provided in the appendix provides a clear understanding.
summary_of_the_review: This paper proposes an interesting policy learning algorithm which uses both demonstrations and preferences. However, I have some questions about this algorithm.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the setting of the combination of learning from demonstration (LfD) and learning from preference (LfP) is studied. Under the adversarial imitation learning (IL) framework, a new algorithm is proposed, whose main idea is to include a new ranking loss term for LfP to combine with the adversarial LfD loss. The proposed method is tested under the meta-world benchmark tasks, showing performance improvement over existing baselines.


strength_and_weaknesses: Strengths:

1. The experiments show significant performance improvement over existing baselines under the LfD + LfP setting.

2. In my view, the paper is technically sound. The proposed approach is reasonable to work in practice.

Weaknesses:

1. I think the technical contribution is somehow limited. Comparing to existing adversarial IL approaches, the essential part of the proposed method is in Equation (9), which is a ranking loss to make use of the preference information. However, the hinge loss is one of the common choices for ranking tasks. 

2. From the paper, I am not quite clear why the proposed method can achieve such good performance over the baselines, since the algorithm itself does not show significant novel improvements. I also suggest including more discussions in the paper.

3. I think it would be useful if the code could released when the paper gets accepted.
clarity,_quality,_novelty_and_reproducibility: I think the paper is overall well-written, while the problem setup is not clearly discussed in Section 3. Even though the introduction of adversarial IL is thorough, it is not clearly described whether all trajectories generated by the learner can be evaluated by the expert to generate preferences, or the preferences are obtained via some query strategy. I suggest describing the problem setup, in special how the preferences are obtained, clearer in the paper.

summary_of_the_review: In summary, I think this is a solid paper proposing an empirically well-performed algorithm. However, I think the technical contribution is relatively weak, which is the major concern to me.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes policy training that combines two commonly used types of human feedback — demonstrations and preferences, each of which has its own strengths and weaknesses. Specifically, the authors propose a method for incorporating preferences into adversarial imitation learning as a unified framework, by reformulating adversarial imitation learning with density ratio estimation, then using the same learned density ratio estimator to learn from preferences. The authors compare their method against other preference based methods and adversarial IL methods, as well as naive combinations of imitation learning and preference based learning.
strength_and_weaknesses: Strengths
- The motivation is compelling and the paper is well written, easy to follow. In particular, the use of a density ratio estimator for both AIL and for judging preferences seems reasonable.
- The empirical results seem compelling, and compared against relevant baselines (of AIL methods, preference-based methods, and naive combinations of the two). The experimental results also present with ablations on their reward formulation choice and compare the effect of providing varying numbers of demonstrations and preferences. In particular, the combination of preferences and imperfect demonstrations is an interesting setting — although in this work it does not seem to show a big improvement over just using the preferences.

Weaknesses
- The main paper of comparison (PEBBLE) shows experiments across a wider set of environments (including locomotion tasks like Quadruped, Walker, etc.) and using real human preferences. It would be an interesting and stronger comparison if those same experiments were replicated here as well, with AILP.
- It also seems like in these domains, a single demonstration is sufficient to saturate performance. It would be interesting to find a domain where more demonstrations are actually helpful / necessary.
- Formatting could be improved. Plots are currently a little hard to read, should increase legend size or move them off the figures.

Questions
- Is relabelling the data in the buffer with the updated reward function time consuming and/or difficult to do as the buffer grows? How frequently is this relabelling necessary?  
- This method assumes preferences are aligned with expert demonstrations. I would be curious to see what happens if preferences weren’t exactly aligned -- e.g. if the preferences were trying to guide for more stylistic behaviours.
clarity,_quality,_novelty_and_reproducibility: The paper presentation is clear and well written. While parts of the method is based on prior work, the proposed method and means of combining preferences and demonstrations seem novel. Code is not provided so reproducibility is unclear.
summary_of_the_review: I recommend a 6. The overall methodology seems sound and experimental results empirically support the author’s claim that AILP manages to combine learning from both demonstrations and preferences. The paper could be improved with more direct comparisons to PEBBLE (more diverse set of tasks, seeing if the method still works when learning from human preferences for stylistic behaviours, etc.).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method that can utilize both preference and demonstration from an expert. While, as one reviewer concerned about, the technical novelty is not high (simply combining two existing methods), the problem formulation is novel and it is well motivated in the paper. I am convinced that this is a needed setup. Also, the paper provided a good set experiments, adding extra experiments reviewers requested, with all positive results. It is a good paper to share with the community.
justification_for_why_not_higher_score: Problem is well-needed, but their proposed approach has limited technical contribution.
justification_for_why_not_lower_score: * Clear problem formulation / motivation / novelty
* Thorough experimental results
Decision: Accept: poster