Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: - goal: RL for partial observability
- idea: leverage good fully observable policies (state experts) to help train partially observable policies
- introduced objective: minimizing divergence (e.g. KL) between expert and partially observable agent
- training in such a way is suboptimal, but applying objective in a soft fashion is beneficial
- car-flag: agent learns to accelerate only once it gains more information
- lander: policy less aggressive than expert due to partial observability
- block-picking: agent doesn’t see color. quickly moves to other block once it sees it cannot pick it. real setup: use sim policies with real depth maps.
- Results:
  - does well across the board compared to dagger and RL methods
  - method performs well in both optimality gap and no gap.
- limitation: requires constant access to expert actions at every state during training. proposes fix by learning from limited demonstrations.

strengths_and_weaknesses: strengths:
- paper is pretty well explained
- policy can handle partial observability.
- it seems that the policies are doing as well as the expert but in a more careful manner until they observe the right information because they aren’t as confident as the state expert, this seems like a good thing.
- good results across all tasks, whether there is optimality gap or not

weaknesses:
- it’s not very clear to me in what real world scenario you have access to a state expert?
- not clear how practical or useful this is in the real world
- real setup is pretty limited

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Well written paper, good results on narrow setups, but rather limited real setup, and not clear how practical or useful this is in the real world.
issues: - maybe the authors can motivate better / clarify real life scenarios where the approach is directly applicable? (where a state expert is available)
- The video and voice over is too fast, it’s hard to follow
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes Cross-Observability Soft Imitation Learning (COSIL), a method for augmenting partially-observable RL training with a fully-observable RL policy to improve performance during training. The method introduces the COSIL objective, which is similar to the SAC critic but swaps out the entropy term for a distance metric measuring the similarity between the partially-observable policy and fully-observable policy. The paper studies experimental settings with varying amounts of optimality gap, which consists of the behavior policy performance difference between expert partially-observable policies and fully-observable policies. Specifically, they study 6 different environments that include some form of partial observability, such as withholding geometric or physical properties of the scene.
strengths_and_weaknesses: Strengths:
- The paper is easy to follow and well written
- The motivation and comparison with SAC is helpful

Weaknesses:
- The significance is limited because the empirical settings seem quite contrived (ie. adding features to the environment that are not standard in robot learning). It feels that each of the experimental settings introduce partial observability in fairly forced ways by adding specific MDP adjustments to normal robot learning tasks. More relevant experiments can be motivated by similar works in multi-modal learning that consider state masking or modality masking.
- Similar to the above point, it’s unclear how widely applicable COSIL is or whether it will only work in specific partially observable MDPs; with a very suboptimal state expert, performance can actually be worse, as shown in Figure 6. This may require very careful selection of when and how to apply COSIL.
- The performance of COSIL in low optimality gap environments is not great
- The real world performance improvement is hard to gauge, perhaps due to the small evaluation sample size. The evaluation also introduced a large sim-to-real gap as mentioned in 5.6.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The motivation with SAC is elegant and helps introduce an original and interesting method. My concern is that the impact is limited by the empirical results, which are mediocre even on fairly contrived settings. This seems to suggest challenges for the applicability and significance of the method, even if it is sound and novel.
issues: - Were the fully observable policies trained in parallel or pretrained? In section 2, it’s claimed that a drawback of ADVISOR and A2D are that the state expert must be pretrained, but it’s not clear which setting the experimental settings use. Are there experiments comparing using a fixed pretrained state expert vs. a state expert that is trained in parallel with the partially observable expert?
- Were the BC baselines in 5.2 fully observable or partially observable experts?
- The baseline for ADV-OFF seems poor in the most complex domain, block picking. The variance is high, and there is a strange dip early in training. Can you run more seeds?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper is interested in reinforcement learning in partially observed tasks. Specifically, the approach aims to make use of an online expert with full observability to improve the training of an SAC agent. The key insight is that the fully observable experts actions may not always be optimal for the partially observable agent.

The methods two main components are:
- Adding a BC term to the SAC reward.
- Dynamically adjusting the weight of the BC term to map to a target BC term. 

Experiments suggest that the method outperforms SAC+BC, DAgger, and other relevant baselines on some simulated and one real robot task.
strengths_and_weaknesses: *Strengths*

- Studying training RL policies under partial observability is an interesting and relevant problem for robotic learning.
- The motivation makes sense in partially observable tasks, a fully observable expert actions may not always be optimal for the partially observed agent.
- The paper has experiments on a physical robot. 

*Weaknesses*

First, I have some questions about the method:
- The proposed method as I understand it simply amounts to adding a BC loss to the reward with an online queryable expert at each state. Section 4 frames this as a KL divergence, but looking in the appendix the online expert is deterministic and this is simply amounts to an MSE loss. This added BC objective is commonplace in many IL+RL algorithms algorithms, the paper could do a much better job at explaining how their instantiation is different from prior work.
- In general, I don't understand what about the method is "Soft" Imitation Learning. The introduction and Section 4 emphasizes "soft" but what does this mean and how does it describe the method? I found the paragraph in lines 145-158 unclear in explaining this.
- I also don't understand fully why the weight on the BC term is adapted to hit some target divergence? I understand that the method is trying to balance maximizing reward vs. matching the potentially suboptimal expert, but I would expect the choice of alpha to do this. It would be good to see an ablation of the COSIL method with the adaptive re-weighting part removed. 

Second, I have some questions about the experiments:
- What is different between the proposed method and the SAC+BC baseline? Is it just that in COSIL the BC loss is part of the reward? I find it surprising that performs so much worse than COSIL considering how similar they seem. 
- I would imagine that if the expert is suboptimal as described, the pure SAC agent would eventually outperform COSIL. It would be interesting to see at what point SAC starts to outperform. 

Third, overall the paper clarity could be improved significantly. The paper could better explain what about the current method is novel and why it is important for partially observed tasks. It should also have more details about the method implementation in the main text, including the algorithm block which is currently in the appendix. 

I also have some larger questions about the practicality of the problem setup. For example, the assumption that there is an online expert who gives action labels not just on current online states, but any state the agent wants to query (e.g. for target states)  is strong. For example a human teleoperator cannot do this in the real world, which seems to limit the approach largely to simulation. And if the policies are learning in simulation, then sample efficiency is less of a concern, and pure SAC which will likely eventually reach higher asymptotic performance would be a better choice. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall the paper has some solid insights around learning in partial observability and using a state based expert. But the paper currently does not clearly explain and motivate the method, and I have some open questions about the method, experiments, and general setup that hopefully can be addressed in the rebuttal. 
issues: See Strengths and Weaknesses.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work studies an imitation learning setting in which the learner is subjected to partial observability while the expert has full observability. The proposed approach introduces a regularization term for the actor to encourage the learner to match the action distribution of the expert, which observes the full state. The strength of this term is controlled by \alpha, which is updated by dual gradient descent to match some target divergence. The resulting method COSIL is compared to an array of algorithms on a series of environments with designed partial observability.
strengths_and_weaknesses: Strengths: The paper is overall well-written, and the proposed method COSIL is compared to a strong array of algorithms. COSIL performs either comparably to or better than alternative methods, including ADVISOR which is designed with the same objective in mind. A policy learned by COSIL in simulation is also shown to transfer well onto a physical robot.

Weaknesses: As pointed out in the limitations section,
- to compute the divergence term during training, we require the expert actions at every state visited by the learner, and
- the state expert, while optimal under full observability, can be suboptimal under partial observability.

The second weakness is somewhat concerning to me. The setup studied in this work is a hybrid of RL and IL in that both environment rewards and expert action labels are provided at each time-step, which is strictly more informative than the RL setting and the IL setting. With additional information, I would expect an algorithm designed for this setup should converge to optimal performance or perform better than algorithms that use only rewards and only labels. However, as depicted in Fig. 4a, BC performs better than COSIL when the expert is suboptimal under partial observability.

To account for the potential suboptimality of the state expert, the value of \bar{D} needs to be carefully chosen so that we allow the learner to find a better solution than a suboptimal expert. It seems that too-small values can lead to worse success rates (Fig. 4c) and I suspect that the best choice of \bar{D} would depend on the environment and expert.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this paper studies a practical setting and shows improvements on existing algorithms for this setup. The proposed method is intuitive and rigorously evaluated across multiple environments.

=============

Post-rebuttal: I appreciate the additional analysis on sensitivity to \bar{D} and adapting \alpha. It's helped me understand how alternative design choices affect COSIL.
issues: Questions:
- Why does the transferred ADV-Off policy perform worse in the real world when the performances are similar in simulation?
- In Fig. 4b, how does COSIL perform with fixed values for \alpha? I’m curious whether we need to adapt the strength of the divergence at all.
- Would training for longer in Minigrid-Memory and Block-Picking lead to better converged performance for COSIL?
review_title: null

Meta-review: metareview: This paper initially received (weak accept, 3X weak reject).
During the discussion, an agreement was not reached, and the current scores are (weak accept, borderline accept, 2X weak reject). I base my decision on the reviews and discussion, and on my own review of the paper.

The main point of conflict between reviewers is the limitations of the asymmetric learning setup - having access to a fully observable policy during training, but learning a partially observable policy for test time.

As several reviewers pointed out, this setup is indeed limiting in many realistic scenarios, and the experiments in the paper therefore seem contrived. 

However, POMDPs are a very difficult problem, and this paper proposes a very simple and novel approach. It may be that future work relaxes some of the assumptions (in particular, using sim2real, or in a meta-RL setting where some tasks are known), and exposing this work to the CoRL community may foster such follow-ups.

Strengths:
- Important, difficult problem
- Good empirical results (relatively extensive in sim, limited on real)

Weaknesses:
- Real world practicality not clear enough
- Some important parameters in the method are environment dependent and may be hard to choose
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)