Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an algorithm for Heteogeneously Observable Imitation Learning (HOIL), a variant of imitation learning in which the demonstrator and imitator have different observation spaces, and it is expensive to obtain paired samples that depict corresponding imitator and demonstrator states. Specifically, in the HOIL setting, the data available to the agent includes:

1. A small number of paired training samples (corresponding observations under the two observation spaces, along with actions) obtained from some policy $\pi_1$ (which may not be the expert policy).
2. The ability to roll out policies under the demonstrator's observation space alone.
3. The ability to query which expert observation corresponds to a given demonstrator observation.

The proposed method is a form of GAIL that uses importance sampling to reweight the demonstrator/novice observations from data source (2) so that they approximately follow the marginal distribution of the expert policy in the novice observation space. When a given imitator observation falls outside the support of the previously recorded paired data (i.e. when the importance ratio is infinite), the agent actively queries for the corresponding demonstrator observation and adds it to the available paired training data. Experiments on MuJoCo and Atari against various baseline methods for dealing with different observation spaces suggest that this approach yields higher return with fewer paired (expert/novice) observation samples.
strengths_and_weaknesses: **Originality:** There is a substantial amount of existing work on imitation learning from demonstrations that are provided in a different observation space to the one available at evaluation time for the imitator. As I understand it, the novelty of this paper is in (1) explicitly treating the required number of paired observations (i.e. corresponding imitator/demonstrator observations) as something that should be minimised by the learning algorithm, and (2) proposing a combination of importance sampling and active learning to achieve this goal. The proposed approach to this problem seems quite novel to me. However, I'm not convinced of the desirability of minimising the amount of paired data, or of the assumption that active querying for paired data is cheap (this is mentioned more in "significance", below).

**Clarity:** The paper is a bit notationally heavy (here are many subscripts and accents without obvious mnemonic meanings), so it took me a while to digest the meat of the method. Here are some suggestions for things that could be clarified in the paper:

- L6/L115: the paper refers to observation spaces "coexisting". After reading the paper, I think "the two observation spaces coexist" means "the agent can sample paired observations from both observation spaces"; however, I found this terminology confusing at first, since observation spaces are just sets, and it's not clear what it means for two sets to "coexist".
- L11: "dynamics mismatch" is referred to many times but doesn't actually seem to be defined in the paper. What does this term mean?
- Figure 2: it's not clear at this point in the paper what $\pi_1$, $\pi_2$, etc. are, what the script $T$s are, or why they have bars over them. Explaining this in section 1 or moving Figure 2 to later in the paper would be helpful.
- L62-64: "dealing with the dynamic change of the learner's policy during learning is essential" --- what does this actually mean? In what sense does adversarial domain adaptation only need to consider the difference between "two fixed policies"?
- L113: "evolving data" is an odd name for ${\mathcal T}_{\pi_1}$ given that the dataset in question seems to be fixed at the beginning of training, and doesn't "evolve" in any obvious way.
- Figure 4: the customary symbol for the empty set is $\varnothing$ (`\varnothing`), not the Greek letter $\phi$ (`\phi`). The latter is currently used throughout the paper.
- Figure 4: $H$, $O$ and $N$ should be defined at the beginning of the caption, before the explanation of the figure.
- Equations (14)-(16): is $\langle \mathbb I(D^*_w), g^*\rangle$  actually an inner product between two vectors, or is it just multipyling two scalars together? I thought the latter, but the notation suggests they are higher dimensional. (same issue for all three equations)
- Definition (3): IMO it would be easier to read this as $\operatorname{supp}(\rho_{\pi_E}) \cap \operatorname{supp}(\rho_{\pi_1})$ (or at least have the equivalence made explicit).
- Eqn. (18): where are the $x_i$s sampled from?
- In general: it would be helpful to have a table of notation in the appendix.

**Thoroughness/rigour:**  After getting through the notation, I felt that the proposed method made sense. I appreciate that the method explicitly allows for active querying in situations where the support of the expert observations and novice observations do not overlap; without this, it seems like it would be theoretically impossible to solve the proposed problem with importance weighting. I also appreciated the range of expert baselines that take very different approaches to the problem of observation space mismatch (the proposed method is quite complicated, and so it's good to see that complexity justified by a comparison to a broad range of baselines).

Some concerns about thoroughness and rigour:

- L271: how was the ratio of 1/4 chosen? This choice feels somewhat arbitrary, and I feel like Figure (6) does not do a great job of communicating the tradeoff between taking samples from $\pi_1$ and $\pi_2$. Re-plotting the graphs with different ratios would be useful, or perhaps using a different metric like "number of samples of $\pi_1$ vs. number of samples of $\pi_2$ required to get to performance level $x$".
- Figure 6: why is PPO-RAM regularly performing worse than IWRE in most of these plots? It seems like that method should provide an upper bound on performance, given that it doesn't have to learn a reward function.

**Significance:** The empirical results of the paper are quite strong according to the chosen metric (return as a function of samples, with paired/expert observations being more costly). The overall significance of the paper to the field depends on how meaningful this metric is. I feel that this could be better justified in the beginning of the paper. Specifically:

- L28 claims that the observation space of the expert is "often high cost" for the learner to sample from. For what applications in this true? Are these also applications where active querying of expert observations is easy? Just having a motivating application to think about would be useful. I don't really understand the current self-driving car motivation at the beginning of the paper because it doesn't seem like sticking a camera on the dashboard or recording the instrument panel is especially expensive (it's also very hard to actively query "expert" observations in this setting).
- L124 claims that the cost of querying for expert observations is "much lower" than querying for expert actions in a given state (like DAgger does). It's not clear to me when or why this would be the case (in part because I'm not sure what would be "expensive" about reconstructing the expert observation). Again, a motivating example would be useful as an intution pump.
questions: Please see the questions above. I'll copy the specific questions most important to my score here to help the authors prioritise:

- Thoroughness: on L271, how was the ratio of 1/4 chosen? What happens under different ratios?
- Thoroughness: why is PPO-RAM performing worse than IWRE?
- Significance: when would we expect sampling from the expert observation space to be higher cost than getting equivalent samples from the demonstrator observation space? When would the cost of active expert observation queries be lower than that of active expert action queries?
- Limitations: below I ask for failure cases for IWRE. It would be good to know whether the authors encountered any environments where IWRE didn't work well for reasons that are intrinsic to the approach (as opposed to, e.g., a lack of hyperparameter tuning).
limitations: The paper is explicit in setting out its assumptions, but does not do much to delineate the set of problems where IWRE can be expected to perform well. In particular, it's not clear how the environments in Figure 6 were chosen from their respective benchmark suites (Atari/Gym MuJoCo)---what happened on other environments? Are there environments where IWRE does particularly poorly? I know that it is common to only report successful environments in RL papers, but this does lead to a warped view of how effective algorithms actually are on novel problems.

I do not see any obvious ethical issues or anticipate negative social impact specific to this method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new challenging task for imitation learning: heterogeneously observable imitation learning. It is common that the demonstrator and the learner have totally different observation spaces, but it will make the imitator act sub-optimally because of dynamics mismatch and support mismatch. The authors propose the Importance Weighting with Rejection (IWRE) to solve the dynamics mismatch with importance sampling and support mismatch via rejection learning. They conduct experiments on Mujoco and Atari games and their method gets the best imitation performance.
strengths_and_weaknesses: Strengths:

The proposed heterogeneously observable imitation learning is interesting and challenging. The proposed method is novel and performs well across many different environments.

Weaknesses:

1. The experiment setup for Mujoco is strange. I’m sure that the last half of the original observations in MuJoCo are the features of velocity. I think it is unlikely to predict the actions only from the velocity information. Would you please make more clarification about this?
2. The paper lacks sufficient analytical experiments to demonstrate their motivation and arguments. Why the method works is not clear.

Minor issues:

1. The $w_{2}$ under the $max$ should be $w_{1}$ in Eq(5) and Eq(6).
2. A ")" is missing in Eq(14), Eq(15) and Eq(16).
questions: 1. There is no explicit ablation study on your method. The effects of importance weighting and rejection learning are not clear from your current experiments.
2. The experiment design does not clearly verify the arguments in the paper. It is not clear why the method works only from the imitation performance and a simple t-SNE visualization.
3. Some visualization or analysis results to demonstrate that $I(D(x))$ and $g(x)$ correctly find the $H$, $O$, and $N$.
4. Why is there no standard deviation of IWRE and GAIL in Humanoid result (Figure6 (e))?
5. The Mujoco setup is strange. I think it is better to conduct experiments in naturally designed HOIL environments, such as BEV and first-person-view like LBC paper.

limitations: The limitation is mainly in experiment design and setup. Please see above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors consider the problem of imitation learning, focusing on the challenging scenario where (i) the demonstrator and the learner act under different observation spaces, and (ii) there is only limited access to the demonstrator observations. The focus of the paper is thus on a paradigm the authors call 	Heterogeneously  Observable Imitation Learning: they propose methods for overcoming the challenges of the setting, and experimentally demonstrate their good behaviour on different tasks.
strengths_and_weaknesses: To the best of my knowledge, the work has a good level of originality on the theoretical side. Overall the structure of the manuscript is very good, with a clear storytelling. However, some parts are more clear than others. I find for instance the theoretical formulations a bit hard to follow (although I recognise that the theoretical part of the work requires it) while the experiments may be probably presented in a more interpretable way for the unfamiliar reader.
My impression is that the task considered by the authors present interesting challenges and deserves attention, and the advances with respect to existing IL approaches are convincing in Fig. 2. However, the authors should take care of the section of related works, in which the writing is not that clear.
When needed, I clarify my observations in the Questions.
questions: - “…another is that in HOIL, instead of only considering the difference between two fixed policies, dealing with the dynamic change of the learner’s policy during learning is essential.”. I got the point but the sentence is unclear
- “… While in HOIL, expert’s and learner’s observations are totally different from each other, in the sense that the observations themselves are not the obstacle for acting optimally.” This is unclear to me
- AN state-action pair 
- A Trajectory should be probably defined as a set T = {x_i} I=…
- X hat and h bar might become x_E and x_L?
- Why pi hat instead of pi_L?
- GAIL should be explicitly mentioned in the related works
- Dynamics mismatch and the support mismatch are often mentioned but with no introduction. A definition may be beneficial for the non-expert user
- On the experiments, some guide on how to read and interpret the plots may be useful for the understanding. Also it would be good to have a visual impression on the “category” of the other approaches (e.g. do they consider heterogenous states? Do they have access to the whole experts observations?) for an immediate comparison
- Also 20 trajectories seem to me not enough to reach conclusions on the benefit of the method. Some comments on this point are needed
limitations: There are no comments on the  potential negative societal impact of the work. It is suggested to touch the point, focusing in particular on possible application domains of interest (e.g. autonomous driving as suggested by Fig. 1).
review_title: null

Meta-review: metareview: In this paper, the authors tackle the problem of demonstrators and learners having different observation spaces, by proposing an importance-weighted learning algorithm to bring the support of the imitator state marginal closer to that of the expert demonstrator's state marginal. 

All reviewers have voted to weak accept, but it would seem that this year's NeurIPS mechanism of reviewer assignment has resulted in a much higher acceptance rate than typical, with 80%+ of the AC batch having accept votes. As such, I am tasked with the tough job of rejecting some papers that reviewers were only lukewarmly excited about. It pains me to do this to a paper that reviewers have found methodologically correct. I will be recommending this paper be rejected based on calibration against other papers I'm AC'ing, mostly for the following reason:

The HOIL paper assumes a problem setting where:

1. expert observations with more privilege observations than learner: 
2. access to expert observations being high cost and invasive 
3. importance-weighting the data to close the support between learner and expert state distributions mitigates (1) and (2)

However, it's not demonstrated that (1) and (2) is an actual problem in practical applications. Is sensor mismatch between human and autonomous vehicles the actual problem for learning? Do self-driving cars even utilize a policy formulation explored in the HOIL paper? How big are these state / support mismatches in practice, and couldn't they be mitigated by simpler methods? This paper is the first I've heard suggested that sensor mismatch between humans and autonomous vehicle sensors is the bottleneck for performance. If we remove this motivation from the paper and only consider the importance weighting algorithm used for removing out of distribution examples, then it doesn't feel quite as novel as there are many works that propose some kind of distribution-projection step as a way to mitigate state marginal differences between teachers and learners (e.g. CQL). 

In fact, the setting (1) is not only avoided, but actively exploited by some learning papers (Asymmetric Actor Critic, Guided Policy Search, and other ideas) to *boost* the sample efficiency of learning, with the idea that the asymmetry in state support allows experts to provide useful learning signal in a way that the learner cannot adversarially overfit easily. 

From reviewer pD56, 
```
I didn't find the authors' response to be particularly convincing (the self-driving example doesn't seem like a good fit for what they're doing, and I think they're conflating differences in perceptual accuracy with differences in sensing hardware).
```

confidence: Less certain
Decision: Reject