Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors theoretically prove that the recent outcome-conditioned imitation learning methods do not always improve the policy. They analyze the OCBC methods from the reward maximizing view, which is the fundamental goal of imitation learning. And they find that the OCBC is a type of EM policy search and the average step makes it fail to find the optimal policy. Then they propose Normalized OCBC to "cancel out" the average step by importance weighting. The experimental results show that OCBC indeed fails to converge to the policy in practice and the proposed solution works well.
strengths_and_weaknesses: Strengths:

The paper studies a fundamental problem of outcome-conditioned imitation learning. The theory is sound and the proposed method is simple and effective. The writing is good and I enjoy reading the paper.

Weaknesses:

1. Would you please introduce some failure cases in the real-world application of OCBC? in Line171, the failure condition "the tasks have different optimal actions but do visit similar states" is a little too demanding. I worry that in real-world settings, if you collect a diverse enough dataset, this phenomenon may never happen, which will limit the value of this paper.
2. For comparison results on benchmark tasks, there is only one baseline method. Although I don't think it is a big deal, it would be better to compare with more imitation learning methods.
3. The proposed Normalized OCBC is quite similar to some debiasing methods [1,2] in deep learning literature, i.e., using a biased model to cancel out the biasing effect. It is worthwhile to mention them in related work.

[1] Cadene, Remi, et al. "Rubi: Reducing unimodal biases for visual question answering." Advances in neural information processing systems 32 (2019).

[2] Bahng, Hyojin, et al. "Learning de-biased representations with biased representations." International Conference on Machine Learning. PMLR, 2020.
questions: Please see the Weaknesses part above.
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies an existing family of methods for multi-task Reinforcement learning called outcome conditioned behaviour cloning (OCBC). The paper identifies a short-coming of these methods, that, these methods are not guaranteed to maximize rewards when applied to multi-task RL. The paper proposes a correction to these method and shows that the proposed method is guaranteed to obey the policy improvement theorem. The experiments show that the proposed method performs better than OCBC under various settings.
strengths_and_weaknesses: Strengths
+ Seems to follow and study an existing suite of methods and analyzes their theoretical relevance in depth
+ Follows a good step-wise process towards building its results
+ Presents a theoretical, algorithmic and experimental contribution
Weakness
- The problem setting and formulation was not clear
- The experimental setting was also not clear (related to previous weakness) 
- Difficult to understand the significance of these results
questions: My main confusion lies in the problem setting and problem formulation. 
What is multi-task RL? 
How is it different from multi-objective RL? 
The paper presents preliminaries and many mathematical definitions regarding the Q values and distributions of trajectories. However, the paper fails to mention what the aim of the agent in this setting is? 
Does the agent want to maximize rewards? 
Does the agent want to reproduce a trajectory that leads to a state where it achieves a tasks?
What is the relation between task and reward? 
What if the original algorithm OCBC was designed for the agent to achieve the task and it does not really care about maximizing an abstract notion of reward? 
Why is reward maximization important in this setting
Generally speaking, it is fair to say that goal of almost every MDP agent is reward maximization. Is that not true in this case? 

The most simplest experimental setting the paper presents is of a multi-armed bandit. But here again, the paper presets a multi-task multi-armed bandit? What is this setting. Does the actions not have reward associated with each arm? Are the rewards of these arms dependent on each other since each action accomplishes a task? Again, what is the relation between reward and task? 

limitations: The problem setting and formulation for this paper were quite confusing. It is not possible to gather the significance and relevance of the results without a clear understanding of the goal of the agent. The paper will benefit from clearly describing the problem setting and goal. This should naturally answer the question regarding the relationship between reward and multi-tasking.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper discusses about the outcome-conditioned behavior cloning (OCBC) method in the goal-conditioned reinforcement learning field. It suggests that OCBC is not guaranteed to produce optimal (reward-maximizing) policies because of the averaging step (averaging the policies from different tasks). Furthermore, a normalized OCBC is proposed to fix the problem above. Both theoretical and experimental evidences are provided to support the suboptimality of OCBC.  
strengths_and_weaknesses: The discussion of OCBC's suboptimality is novel and interesting. Authors provide solid theoretical analyses and easy-to-understand illustrations. However, it would be better if the significance and related works are more detailedly clarified.
questions: Is the conclusion and proposed method (normalized OCBC) suitable to single-task behavior cloning, both online and offline settings?
Is OCBC the mainstream method in goal-conditioned RL? Are there any other variants of OCBC?
limitations: The authors adequately address the limitations.
review_title: null

Meta-review: metareview: Paper shows that outcome-conditioned behavior cloning (OCBC) is not guaranteed to maximize long-term reward in multi-task RL. A simple, but effective variation of OCBC is proposed that does guarantee policy improvement. 

While the scope of the paper is rather specific (a certain form of behavior cloning in multi-task RL), this family of methods has gained some momentum recently, while there are still many theoretical questions around it. This paper addresses some of these in a clear way and proposes a specific improvement. A clear accept in my view.
confidence: Certain
Decision: Accept