Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning

Authors: Hanlin Yang, Jian Yao, Weiming Liu, Qing Wang, Hanmin Qin, Kong hansheng, Kirk Tang, Jiechao Xiong, Chao Yu, Kai Li, Junliang Xing, Hongwu Chen, Juchao Zhuo, QIANG FU, Yang Wei, Haobo Fu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: imitation learning, policy diversity, offline learning

Abstract:
Recovering a spectrum of diverse policies from a set of expert trajectories is an important research topic in imitation learning. After determining a latent style for a trajectory, previous diverse polices recovering methods usually employ a vanilla behavioral cloning learning objective conditioned on the latent style, treating each state-action pair in the trajectory with equal importance. Based on an observation that in many scenarios, behavioral styles are often highly relevant with only a subset of state-action pairs, this paper presents a new principled method in diverse polices recovering. In particular, after inferring or assigning a latent style for a trajectory, we enhance the vanilla behavioral cloning by incorporating a weighting mechanism based on pointwise mutual information.
This additional weighting reflects the significance of each state-action pair's contribution to learning the style, thus allowing our method to focus on state-action pairs most representative of that style.
We provide theoretical justifications for our new objective, and extensive empirical evaluations confirm the effectiveness of our method in recovering diverse polices from expert data.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730522719393

thread_type:
official_review

<thread_object>
creator: reviewer_buhv

<content>
summary:
This paper investigates how to recover diverse policies from expert trajectories, proposing a new method that leverages the relevance of state-action pairs to trajectory styles. By introducing Pointwise Mutual Information to model this relationship, the method approaches the problem of policy diversity from a different perspective, ultimately achieving results that surpass previous state-of-the-art methods.

strengths:
1. The motivation is simple but effective, and it will be highly beneficial for the efficient use of imitation learning data in subsequent robot tasks, contributing significantly to the development of the community.
2. The paper is well-written and easy to follow.
3. The experimental results are extensive and convincing.

weaknesses:
1. I would like to know if there are other reweighting methods that can be used for comparison. If so, could some comparative experiments be included?
2. Additionally, the current method evaluates the diversity of state-action pairs. As far as I understand, there are also studies specifically focused on state diversity and action diversity. I wonder if these could serve as a standard for comparison?

[1] Suneel Belkhale, Yuchen Cui, Dorsa Sadigh. Data quality in imitation learning. NeurIPS 2023

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730580787832

thread_type:
official_review

<thread_object>
creator: reviewer_fkfr

<content>
summary:
The paper introduces Behavioral Cloning with Pointwise Mutual Information Weighting (BC-PMI), a new approach for recovering diverse policies in imitation learning. Here, the expert data consists of trajectories coming from different experts, each reflecting a unique approach or style within the task. The authors’ key idea is that, even within a single style, not all state-action pairs equally represent that style’s characteristics. To address this, BC-PMI weights state-action pairs based on their relevance to the style, calculated using pointwise mutual information between $(s,a)$ pairs and the style code $z$. These weighted pairs are then used to learn a style-conditioned policy $\pi(a|s,z)$ using behavioral cloning. This selective weighting allows the model to prioritize pairs that are most representative of the target behavior, thereby enhancing its ability to learn style-conditioned policies that more accurately capture the style within expert trajectories. 

The paper also provides some theoretical insights showing that BC-PMI smoothly interpolates between traditional behavioral cloning and clustering-based behavior cloning, depending on the mutual information between style and state-action pairs.

The authors evaluate BC-PMI in three settings: Circle 2D, a simple 2D setup; Atari Games (Alien, MsPacman, and Space Invaders); and the Professional Basketball Dataset, featuring NBA player movement data. Results show that BC-PMI achieves high style calibration across various metrics, outperforming baselines. However, it is worth noting that the baselines operate in an unsupervised manner and do not have access to the style labels available to BC-PMI.

strengths:
1. The paper is well written and clearly communicates its key ideas.
2. The idea of selectively weighting state-action pairs based on their relevance to a given style, rather than treating entire trajectories uniformly, is both intuitive and impactful. Existing methods typically operate at the trajectory level, but this paper demonstrates that not all state-action pairs within a trajectory equally represent the intended style.

weaknesses:
1. A key disadvantage of the method is that it requires style markers for each trajectory in the training dataset. These can be expensive to obtain. A brief note is made about using labeling functions to obtain these markers, but no experiments are included. This also makes the comparison with existing works unfair since they are unsupervised methods. To make the comparisons fair, the authors should include the supervision in other methods, e.g., they can train the posterior network in InfoGAIL using the style labels.

2. The experimental section is relatively brief, lacking analysis on the method’s performance as the number of styles increases or when labels are noisy. The latter case is quite important given that labels generated by programmable labeling functions (as mentioned in L296) may be imperfect. For a comprehensive understanding of these factors, it would be beneficial to include experiments that measure calibration as the number of styles increases, as well as tests assessing robustness to label noise introduced by such labeling functions.

questions:
1. Why is MINE used to estimate the pointwise mutual information? This can be estimated by simply learning a classifier across different styles, since you are not interested in the $I(S,A;Z)$ rather $p(z|s,a)$. It would be valuable to compare the calibration performance of BC-PMI using a classifier-based approach against the current method with MINE’s statistic network, to assess if simpler alternatives might achieve similar results.
2. What are the calibration metrics used in Tables 3/4? Is it DTW, or ED, or KL?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682601974

thread_type:
official_review

<thread_object>
creator: reviewer_ihl7

<content>
summary:
In this work, the authors focus on developing a method for imitation learning better diversity of policy. It works under assumption that the expert trajectories in training are collected by stylized experts. The key is to using a pointwise mutual information-based weighting strategy to determine the policy importance. The state-action pair with higher posterior probability is given higher importance. The proposed method achieves good experiment results and also is demonstrated to be covered in the extreme cases, such as zero mutual information or no overlap among different policy styles in a single state-action pair. For the extreme cases, the theoretical evidence is provided to show that the proposed strategy degrades to usual strategy without failing.

strengths:
1. The proposed method is motivated by an intuitive insight and the failure of existing alternatives. The use of PMI and MINE are well connected for the purpose.
2. The proposed method achieves good experimental results on the datasets of CIRCLE2D , Atari and Basketball Player dataset.

weaknesses:
1. Given that stylized experts make unbalanced trajectory generation in training, it might be helpful to discuss the disentanglement between the stylized code and the state-action pair. Some previous works have built related benchmark for the purpose such as [1] and [2]. Though the benchmarks may not be originally designed for imitation learning, the corresponding metrics might be worth reference. Such evaluation can be adapted by measuring the mutual information or other metrics between the stylized code and the pair.

2. It remains not unclear, at least not generally convincing across datasets, that PMI shows a significant advance than the usual MI for BC. More evidence or discussion about this can be helpful to enhance the significance of the proposed method as PMI is claimed as a main contribution in this paper. If replacing MI with PMI can result in generalizable performance boosting across datasets, the proposed method can be supported with more experimental significance. 

Reference:

[1] “**An Empirical Study on Disentanglement of Negative-free Contrastive Learning**”, NeurIPS 2022.

[2] “Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations.”, ICML 2019

questions:
Please see my comments in previous sectors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697466696

thread_type:
official_review

<thread_object>
creator: reviewer_6kbz

<content>
summary:
The paper explores a methodology for deriving diverse policies from expert trajectory data. Rising from the traditional conditional behavior cloning (BC) algorithm, the authors  introduce an additional importance weight based on Pointwise Mutual Information (PMI), to assess the correspondence between state-action pairs and trajectory styles. The experimental results prove the proposed method can improve policy diversity with PMI weighting.

strengths:
1. The topic studied in this paper is quite interesting, with an objective to improve policy diversity.
2. This article clearly defines the problem and provides a detailed exposition of the method.
3. This article undertakes extensive comparative experiments across Circle 2D, Atari games and professional basketball player dataset, which valiate the effectiveness of the proposed algorithm compared with the baseline methods.

weaknesses:
1. This paper aims to enhance policy diversity by incorporating a weighting method based on pointwise mutual information to the Conditional Behavioral Cloning framework. The key proposal of this paper is merely the introduction of a data weighting algorithm, which, in my view, does not represent adequate technical contribution.
2. The manuscript does not include sufficient ablation experiments to validate the efficacy of the proposed data weighting method that utilizes pointwise mutual information. It remains unclear how BC-PMI compares to Conditional Behavioral Cloning that is conditioned on style, where each sample point is assigned equal weight. Additionally, its performance against other traditional data weighting methods has not been adequately compared.

questions:
1. Which behavior cloning (BC) algorithm is selected as the baseline method? Does the proposed PMI general well across different BC algorithms?
2. From Table 1, the improvement of the proposed BC-PMI seems marginal compared with CBC.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732206616624

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to express our sincere gratitude for the thorough review and valuable feedback on our paper. The reviewers' insights and suggestions are quite helpful to improve the quality and clarity of our work. 

We are encouraged the topic is interesting (Reviewer 6kbz) and the method is well-motivated (Reviewer ihL7, Reviewer BUhV). The paper is well-written and clearly communicates the key ideas (Reviewer 6kbz, fKFr, BUhV). The method is validated via extensive experiments (Reviewer 6kbz, BUhV)

We have uploaded a revised version of the paper, with changes highlighted in blue within the main text. Additional analyses and experiments have been included in the Appendix. The main revisions covered in the Appendix are as follows: 
- Added more detailed comparison and explanation of the weights of BC-PMI and CBC (Appendix C)
- Added more detailed explanation and experiments of the label function (Appendix D)
- Added more comparison of the calibration with more number of styles (Appendix E.1)
- Added more ablation studies, comparing BC-PMI to BC-classifier, a weighted BC competitor and PMI+CGAIL.(Appendix E.2)

Please see the revision paper for more details.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: This paper proposes Behavioral Cloning with Pointwise Mutual Information Weighting (BC-PMI), a method that enhances policy diversity by weighting state-action pairs based on their relevance to trajectory styles using Pointwise Mutual Information (PMI). Experimental results across three settings, Circle 2D, Atari Games, and the professional basketball datasets, show that BC-PMI outperforms baselines in terms of style calibration, while also providing insights into the smooth interpolation between traditional behavioral cloning and clustering-based behavior cloning.


Strengths and Weaknesses: 

Generally, the reviewers find the topic compelling, the problem clearly defined, and the method well-explained, with an intuitive motivation that addresses the shortcomings of existing alternatives and strong results across multiple datasets, making the paper a significant contribution to improving the efficiency of imitation learning.

While the paper presents a promising approach, several weaknesses were identified by the reviewers. The original manuscript lacks several key ablation experiments and comparisons with other reweighting methods or style-conditioned BC, as well as evaluations on state and action diversity, which are necessary to fully establish the superiority of the proposed method. There were also concerns regarding the fairness of the comparisons due to differing types of supervision. Additionally, the experimental section is brief and does not analyze the method's performance as the number of styles increases or in the presence of noisy labels, which is particularly important given the reliance on potentially imperfect labels. 

During the discussion phase, the authors' responses, along with the additional experimental results, effectively address the reviewers' concerns.

All reviewers recommend acceptance, and ACs support their recommendation. The authors should address the main points raised in the reviews, particularly the additional analyses and experiments, such as validating the effectiveness and applicability of PMI on a broader range of advanced BC policies and conducting more comprehensive comparisons with other data weighting methods, as promised in the rebuttal, when preparing the camera-ready version.