Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper aims to solve a branch of imitation learning scenario that the expert’s and the learner’s feature (observation) spaces are different. This scenario is quite difficult compared with existing imitation learning work under the same observation space. Moreover, the related works relied on the expert’s and learner’s observations can be obtained during the whole learning process, which can be costly in many real-world applications. So this paper tried to solve this problem with limited observations coexistence.

At first, this paper formulated the learning problem as heterogeneously observable imitation learning (HOIL), where some expert demonstrations and an auxiliary policy were given under the expert’s observation space, and the target was to learn a policy under the learner’s observation space. Since these two spaces can be arbitrarily different, they needed the auxiliary policy to sample some data containing both two observations as the bridge between the expert demonstrations and the learned policy. Later on, they limited the number of observation coexistence to lower the cost.

By analyzing this problem, they found the underlying challenges, i.e., the dynamics mismatch and the support mismatch. These two mismatches appeared due to the imperfection of the auxiliary policy. To solve this problem, they proposed an algorithm of importance weighting and rejection (IWRE), where the IW solved the dynamics mismatch and the RE solved the support mismatch.

To evaluate the performance of IWRE, they did some experiments on MuJoCo and Atari benchmarks. Compared with the ablation, domain-adaptation imitation learning methods, and other heterogeneous observation methods, IWRE obtained the best performance among these environments.
strength_and_weaknesses: Strengths：
* This paper studied a challenging but more general learning scenario, with weaker assumptions than related works. Also, the HOIL problem is well-defined.

* For the algorithm part, the analysis of the dynamics mismatch and the support mismatch is quite convincing, with clear mathematical formulations and visualization empirical studies. Meanwhile, the motivated IWRE makes sense.

Weaknesses：

The notations of this work are somehow heavy in this paper. It would be better to simplify some notations.
clarity,_quality,_novelty_and_reproducibility: The writing of this paper is clear and easy to follow. The related works are well discussed, and the experiment is sufficient. The combination of importance weighting and rejection learning is quite novel in the reinforcement learning and imitation learning communities. The paper contained enough details for reproducibility.
summary_of_the_review: This paper studied a challenging but more realistic imitation learning scenario, in which the expert’s and the learner’s observations are heterogeneous. Also, the authors proposed a novel and reasonable method IWRE to solve this problem. The contributions of this work are sufficient. So I recommend this paper be accepted.

After rebuttal:

I have gone through the responses and discussions between the co-reviewers and the authors. I think the authors have clearly answered the questions of other reviewers, and thus believe this paper is a good submission. So I increased my score and stand for accepting it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The bulk of existing work on imitation learning focuses on settings in which the expert demonstrator and learner operate under the **same observation space**; however, in many real-world settings, this isn’t realistic. The work formalizes the problem of heterogeneously observable imitation learning (HOIL) and present an algorithm — Importance Weighting with Rejection (IWRE) to address this.

Concretely, given a set of expert demonstrations in a given observation space, the HOIL setting this work looks at decomposes learning into collecting a set of (suboptimal) “matching correspondence” data online, running a policy that allows one to correlate expert observations with learner observations (even when there’s no explicit overlap in observables — e.g., Atari ALE RAM state vs. visual features). The proposed IWRE framework then tries to address two key problems in this learning setup via an online IRL-like procedure (similar in nature to GAIL); first, the problem of optimality/dynamics mismatch — the “expert demonstrations” are assumed to be optimal, while the correspondence data is not; as a result, directly imitating from the union of both datasets would result in a bad policy. To get around this, similar to prior work in offline RL, this work proposes using importance sampling (the IW in IWRE) to correct for this mismatch, prioritizing the expert demonstrations.

The second problem IWRE addresses is that of a support mismatch; the “correspondence matching” data is again problematic because its suboptimality might mean that the corresponding data only sees a fraction of the state space. As a result, IWRE uses rejection sampling to drive the learner’s behavior policy towards the support of the actual expert observation distribution (exploiting a learned discriminator that partitions the expert demos and correspondence demos), therefore circumventing this issue.

The IWRE approach is evaluated on a set of (albeit somewhat synthetic) environments from the Atari Arcade Learning Environment (expert observations — frames, learner observations — RAM state), and Mujoco Continuous control (expert/learner see disjoint halves of the Mujoco observation space). Compared to traditional imitation learning baselines and reinforcement learning upper bounds (for ceiling results), the IWRE approach is strong, performant, and seems to generalize across environments.

strength_and_weaknesses: I really like this approach, and the motivation behind the HOIL problem. I think it’s definitely true in many robotics settings that there is a mismatch in observation spaces between the expert and the learner, and addressing this is important.

IWRE as an algorithm is also well-motivated, and follows naturally from a large body of prior work; combining importance weighting and rejection sampling in this setting is natural and novel, and clearly the results speak for themselves.

The one weakness I have of this work is with the evaluation; technically, to prove the viability of the algorithm, the existing “synthetic” observation splits in the Atari and Mujoco environments make sense, but I’d really love to see a more realistic evaluation where heterogeneous observations are actually ecologically viable — perhaps in settings in robotic manipulation for example.

clarity,_quality,_novelty_and_reproducibility: This paper is clearly written, with high-quality explanations and derivations of the approach. The care given to describing the heterogeneous observation setting and how it differs from settings studied in prior work are additionally very helpful. This problem setting and proposed algorithm are both novel.

summary_of_the_review: I believe this to be strong work, with an evaluation that demonstrates the viability of the proposed approach and difficulty of the problem setting. However, I’d love to see more ecologically viable/realistic evaluations, rather than the somewhat synthetic tasks studied in this work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method for imitation learning (IL) when the expert and agent operate from different observation spaces. The paper proposes a new algorithm for dealing with this setting, which relies on two components: an importance weighting correction which modifies the standard GAIL objective, and a rejection mechanism to address potential support mismatch between expert and policy distributions. The algorithm is evaluated on 5 Mujoco and 3 Atari environments.

The algorithm operates in two stages. First, GAIL is run in the expert's MDP to learn a decent, but not necessarily optimal policy. Then, the same policy is run in the expert and learner MDPs (this is assumed to be possible), giving a dataset of aligned trajectories. This is then used to compute the importance weighting correction factor. Finally, a classifier is trained to filter out the non-expert parts of the policy, which is used for the rejection sampling mechanism.
strength_and_weaknesses: # Strengths:
- The goal of learning from demonstrations gathered in a different observation space is definitely interesting
- To my knowledge, this combination of importance weighting and rejection sampling is novel.


# Weaknesses:
- I'm not convinced by the setup they're considering here. They make assumptions which seem unrealistic to me: in particular, they assume it is possible to interact with *both* the expert and agent MDPs to some extent. This is necessary for gathering what they call the evolving dataset, which is a set of trajectories where the observations in both MDPs are aligned. I find this to be a strong assumption, and one which other methods (e.g. [1]) do not make - they assume expert trajectories are given and only interaction with the learner's MDP is allowed.

For this reason, although I think the general goal of IL from heterogeneous observations is very interesting (for example, learning from YouTube videos where the perspective of the expert is different from that of the agent), this paper makes overly strong assumptions that one can gather trajectories from both MDPs. This strong assumption limits the significance and impact of this work. Also, the fact that they are only considering discriminator-based approaches and not others (e.g. non-adversarial ones such as RED [2] or DRIL [3]) limits the applicability.

Also, the algorithm proposed is fairly complicated: there are two different learning phases with GAIL subroutines as well as training a classifier on intermediate data. I believe the community would be more open to adopting a simpler, more end-to-end approach.

[1] https://arxiv.org/pdf/1703.01703.pdf
[2] http://proceedings.mlr.press/v97/wang19d/wang19d.pdf
[3] https://openreview.net/pdf?id=rkgbYyHtwB
clarity,_quality,_novelty_and_reproducibility: Clarity:
- The writing is decent, however I think the paper would be easier to understand if pseudocode for the full algorithm were presented in the main paper rather than the appendix.             

Quality/Clarity: 
- As mentioned above, the algorithm is quite complicated - I believe a simpler algorithm would be preferable and easier to build upon. Overall the experiments seem sound given the assumptions made by this work.             

Novelty:
- To my knowledge, this combination of importance weighting and rejection sampling is novel. 

Reproducibility:
- The authors do not mention any code release in their reproducibility statement. 
summary_of_the_review: Overall, I do not recommend acceptance for this paper due to:
- the overly strong assumptions made by this work, which somewhat undermine the otherwise interesting setting they consider
- the complexity of the algorithm
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for imitation learning with heterogeneous observations. Specifically, the dynamics mismatch and the support mismatch are handled using importance weighting and rejection. Experiments show the effectiveness of the proposed IWRE method on both Atari and MuJoCo tasks.
strength_and_weaknesses: Strength:

The proposed method is explained with details. Both dynamics and support set mismatch are handled with the proposed novel algorithm. Experiments show evidence for the improved covering rate in expert data by the learned policy. Explanations for the experimental results are thorough.  


Weakness:

For the t-SNE plot in Fig. 5, is it true that the support set of $\pi_1$ also has some part outside the expert data? How is this part handled by the method? This does not seem to be captured by Fig. 4. 

Adding ablation studies with only importance weighting or the rejection will help to understand their effects better. However, no such ablation study is conducted in the current paper.

The fonts in the figures are too small to read.

In Fig. 6 (a), the ChopperCommand results are a bit weird. Is it too easy or too hard to learn? It seems the reward becomes large very quickly but decreases along the training. Why is this happening? Please explain more in the paragraph.

clarity,_quality,_novelty_and_reproducibility: The paper has good clarity.

The proposed algorithm is novel.

The method is reproducible.

summary_of_the_review: The paper proposes an effective and novel algorithm for handling both dynamics and support set mismatch in imitation learning. The experiments justify the claims that the proposed method helps with a better coverage of the expert data as well as the effectiveness in facing the heterogeneous observations. Some questions need to be answered well in the paragraph. Adding ablation studies will make it a better work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Strengths:
- Addresses an important open problem in learning from demonstrations of learning when demos don't share the agent's observation space. Having only a small "correspondence" dataset that has access to both observation spaces also makes the algorithm more broadly applicable than approaches such as "learning by cheating".
- Well-motivated algorithm design with a technically interesting combination of importance weighting and rejection sampling. Seems straightforward to implement.
- Strong empirical results on their chosen evaluation settings.

Weakness:
- Table 1 contains a somewhat misleading representation of prior works that dealt with different viewpoints of a scene as having the "same observation space", and the author discussion with reviewer j2UZ appears to focus mainly on the semantics of whether the sensors are the same or different. While the original review from j2UZ contained inaccuracies, I agree with them on the points argued in the discussion phase that this is not really a meaningful distinction from having "different observation spaces" --- top and frontal camera views of a scene are just as much different "observation spaces" as having proprioceptive and exteroceptive views of the same scene. I would urge the authors to improve the writing around these claims. 
     - While prior work such as third-person imitation learning (Stadie et al 2017) has addressed related settings, they rely on the existence of some relationships between the views, which allows for classes of methods (domain adaptation) that do not apply to the more dramatically different "observation spaces" considered in this work, for which some kind of correspondence dataset seems inevitable, and this algorithmic choice seems justified.

- The evaluation focuses on rather simplistic and unrealistic settings, namely RAM state versus visual observations in Atari, and disjoint halves of the state space in MuJoCo. This detracts significantly from our ability to evaluate whether the proposed approach is in fact useful in practical settings. 
justification_for_why_not_higher_score: While interesting in its setting and technical approach, the method's evaluation falls short of demonstrating the usefulness of the approach in practical settings. 
justification_for_why_not_lower_score: The paper addresses an important open problem in learning from demonstrations with an intuitive and technically interesting approach.
Decision: Accept: notable-top-25%