Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a general framework for learning sequential decision-making policies with noisy feedback. Two categories of noisy feedback are considered, noisy reward in reinforcement learning and noisy demonstrations in behavioral cloning. The concept of “correlated agreement” (CA) is proposed as a regularization mechanism for a policy to learn from the noisy feedback without overfitting to it. An additional policy co-training setting is also considered. Empirical evaluations control games and Atari show that CA is effective in enabling policy learning from noisy feedback.
main_review: Post-rebuttal comments: thanks to the authors for providing a detailed response. All my questions are answered satisfactorily. My score will stay the same.

Originality: The application of CA to policy learning from noisy feedback is novel. The discussion of related works is adequate. 

Quality: This paper is technically sound. I appreciate the theoretical analyses for PeerRL and PeerBC, which show convergence to optimal policy with CA. I have the following questions.

1. In the paragraph on BC with weak demonstration (starting at line 120), from which state distribution are the weak demonstrations sampled? I understand the noisy actions are the result of flipping from the true expert action. Are the states generated by following the true expert actions or the noisy ones?

2. I don’t quite understand the point of the toy example given in section 3.1 (starting at line 157). It is true the evaluation score is lower with CA, but does that mean that there is a better policy than the fully memorized one? 

3. I would like to understand what, if any, obstacle exists to extend the noisy model from discrete to continuous. If the reward/action flipping follows a continuous distribution, is any theoretical result impacted?

Clarity: This paper is well-written and easy to read. Good job!

Significance: One limitation the authors acknowledged is that the empirical evaluations are not real-world. I think the significance would improve quite a bit with the inclusion of an example of a real robotic system, for instance. Nevertheless, I think the novelty of CA, the theoretical contributions and the strong performance on the synthetic environments provide enough significance.

Minor comment: This recent ICLR 2021 paper should be included in the related works discussion.
Behavioral Cloning from Noisy Demonstrations, Fumihiro Sasaki, Ryota Yamashina, ICLR 2021

limitations_and_societal_impact: Adequately addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a general and unified framework to study policy learning without strong or high-quality supervision signals. This unified framework comprises, but not limited to, the main approaches for policy learning, including those using rewards signals in reinforcement learning (RL) and demonstrations through behavioral cloning (BC). The authors also propose PeerPL, a solution for efficient policy learning using only weak supervision signals, which evaluates an agent’s learning policy with noisy signals, treated as policy of an imperfect “peer agent”. The weak supervision signal comes in the form of correlation with the imperfect peer agent’s report, while the solution encourages generalization through penalizing identical reports.
main_review: Originality: The originality comes from that the proposed method differs to other noisy or weak supervision methods in which prior knowledge of noisy rates, i.e. the confusion matrices of noisy signals vs true signals, are not needed. Also, the evaluation function is designed so that identical behavior with the weak agent is auto penalized, by utilizing randomly sampled signal pairs. Related works are adequately cited in both Section 1 and Section 2.2. 
Quality: The intuition of how the penalization in correlation assessment is interpreted as overfitting in supervision learning sounds convincing. The theoretical proofs and experimental results look promising. The extension to co-training in partially observable environment using this framework with an RL and BC hybrid algorithm (PeerCT) is clear and well-organized, matching the overall flow of paper.
Clarity: The paper is clearly written, with toy examples which adequately illustrate the core idea of the proposed evaluation function. The experimental setup is also clear, with detailed descriptions on how “imperfect” expert demonstrations were obtained. The authors also provided some valid reasons on performance improvement from PeerPL such as implicitly encouraging higher exploration (presuming through penalizing evaluation function).
Significance: The proposal seems to provide an elegant formulation to address weak supervision signal systematically. The results of single-view co-training show that the proposed solution is feasible and perform significantly better in partially observable conditions compared to other co-training algorithms. The results could serve as good reference for subsequent works either in the field of co-training or weakly supervised policy learning.

limitations_and_societal_impact: There is no discussion section for potential negative societal impacts present in the main paper. Though a short description of why this is excluded is included after the reference section, with the attached submission checklist.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a framework to improve the efficacy of sensorimotor control learning. The main idea consists of augmenting the training loss with a specifically designed term that avoids the policy to "overfit" to noise present in the reward (for RL) or in the demonstrations (for behavioral cloning).  This extra term aims to minimize the probability that when randomly associating rewards and state-action pairs, the policy performs poorly. This idea, borrowed by the theory of correlated agreement, should improve the policy performance when the supervision (in terms of demonstration or reward) is corrupted by noise. Experiments show that the proposed framework achieves comparable or better results than prior works.
main_review: The proposed framework appears to build upon existing work on correlation agreement ([34] in paper), but generalizing such methods to the problem of decision making. Overall, I think that the strong connection with prior work was not adequately discussed. Indeed, it is unclear what are the fundamental differences with existing methods (if it is just an application is not generally a problem but should be clearly stated).

In my opinion, the work presents an interesting approach to a very important and general problem (learning policies with noisy supervision). However, the method and the experiments do not completely back up all the claims. More specifically:

- The proof that the proposed method (asymptotically) converges to the optimal policy despite noisy rewards seems a bit hand-wavy and not completely rigorous. The remark on Lemma 1 (line 193) shows the noisy supervision only differs up to a constant with respect to the proposed peer supervision method. Clearly, this constant can have a strong effect on the final performance, but how much should the noise be? This question also raises the question of whether under the assumption of a large enough sample size and constant noise even the policy without any peer reward would converge to the optimum. Proof A1 in the appendix seems to support this hypothesis since only an affine transformation of the reward (independent of the noise level) is required to guarantee convergence (line 580).
- The experimental results are overall extensive, but comparison with previous work is only partial and sometimes even omitted. Specifically, it would have been interesting to see how the approach would perform in comparison to [7], which exactly aims to solve the same problem (learning from noisy rewards). While this comparison is present for the simplest of the demonstrators (the pendulum) it was omitted for the more challenging ones, e.g. the atari games. Strong baselines on imitation learning, e.g. dagger or dart, are also omitted from the experiments. The evaluation of atari games is by itself a bit strange since no noise in the reward was added (line 326). The paper claims that there is anyway noise given by the real-world application but, to the best of my knowledge, no noise is present in the simulation and the reward observed exactly match the real rewards (I might be wrong on this last point, and clarification in the text could solve the issue).

Overall, I am convinced that the paper is about an important and timely problem, i.e. learning a sensorimotor policy in the presence of noise in the training signal. This problem is relevant for a very large community, including machine learning, robotics, and computer vision. In addition, I really appreciate that an effort was done to cover under the same umbrella a lot of different learning frameworks (RL, Imitation Learning, Co-training). However, I believe that the framework is still not up to its claims and both clarifications in the theoretical part and more extensive experiments could significantly strengthen the contribution.

On a less important side, I think that the current presentation, despite being enough to enable reproducibility, lacks clarity and is sometimes difficult to follow. The main problem is the fact that many terms are used to indicate similar things (e.g. Figure 1, why is the loss function Y indicated as a peer agent? This could cause confusion with the policy co-training part). Similarly, it would be nice to always explicitly state where in the appendix the proofs of the lemma/theorems can be found (this is currently not the case of theorem 2). Finally, it would be nice to put the comparison with the strong imitation learning baselines (GAIL and SQIL) in the main paper.

----------------

Issue solved after rebuttal.
limitations_and_societal_impact: The limitations and possible societal impacts have been properly discussed in the checklist. It would be nice to move some sentences to the main paper, just to have a complete picture in the discussion.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method for RL and imitation learning in the setting where the reward function or expert actions are corrupted with noise. The main idea is to regularize the learned policy to have *high* loss on unaligned state-action-reward examples. Experiments show that the proposed method outperforms baselines that either learn directly using the noisy rewards/actions or learn an auxiliary function that predicts the mean reward/action.

main_review: **Originality**:
The proposed method is (to the best of my knowledge) novel. The paper does good job describing the differences with the most related prior work [Which learns to predict a non-noisy version of the rewards/actions, and then uses them for learning]. My understanding is that the key benefit of the the proposed approach is that it doesn't require knowledge about the noise distribution and doesn't require a separate estimator of the mean reward/action.

**Quality**:
* **Theory**: The main technical claims about sample complexity seem correct (I quickly read through the proofs in the Appendix), building directly on top of prior work. My main concern about the technical claims is that they don't seem to motivate the proposed method. That is, it's unclear if the Theorems show that the proposed method is more sample efficient than either (1) directly learning using noisy rewards/actions or (2) learning using the empirical mean estimates for the rewards/actions. I would highly encourage the authors to discuss this. Said in other words, while it's clear why the proposed method should have the same solution as learning directly on the noisy examples, it's unclear why injecting more noise (but adding a random term to the loss function) should result in faster learning.
* **Experiments**: The paper did a great job comparing both the RL setting and the imitation learning setting, comparing against the most similar prior work. The experiments in the RL setting are limited to two very simple tasks. I'd be curious to see how the method scales to more complex tasks. One suggestion: I would highly recommend including the baseline results on the Pendulum task; omitting baseline results on the tasks where the baseline performs as well as the proposed method is concerning. The imitation learning results on Atari are also nice. I found the co-training experiments quite hard to follow. The discussion of the benefits of PeerRL (L292) seems intriguing, but I'm not sure if I believe the three offered explanations. For example, the argument about human rewards being imperfect seems to contradict the claim that the proposed method optimizes whatever reward function is given. Similarly, the claim that the proposed method decreases variance seems suspicious because (a) it seems like it actually *increases* the variance of the gradient estimator and (b) this same logic would suggest that the baseline from [7] should outperform the proposed method.

**Clarity**:
The paper is generally well written and reads quite well. The related work section is well organized and discusses relevant prior work. The paper does a great job of introducing and explaining notation before using it. See list of minor writing comments at the end of the review. One high-level suggestion is to spend more time explaining the problem settings. Since the problem setting considered in this paper is different from most other RL papers, it's be good to explain why it is useful and interesting. If possible, I'd highly recommend adding an experiment that motivates this setting.

**Significance**:
I am on the fence about the significance of the paper. On the one hand, the proposed method is quite simple and performs well in a variety of experiments. On the other hand, it's unclear if the analysis actually explains *why* the proposed method is performing better (see previous comment), and it's unclear whether the problem being solved is practically relevant.


**Minor comments**:
* L16 "breakthrough" -> "breakthroughs"
* L16 "these techniques" -- Which techniques?
* L19 "expensive to obtain in practice" -- Add a citation.
* L20 "thus" -- Grammar error.
* L21 "are often imperfect due to limited resources or environment noise" -- I'm unsure if this claim is actually correct. Noise in the environment does not affect the distribution $p(a | s)$, which is all we need for behavior cloning.
* L22 - L29 -- I found this paragraph confusing because it was unclear what "weak supervision" was referring to. I'd highly recommend providing *real-world* examples of this problem setting.
* L26 "imperfect solution" -> "solution"
* L28 "Then as a response, we..." -> "We ..."
* L30 - L31 -- I found this sentence confusing.
* L38 "weakness of the supervision" -- Unclear what this means.
* Generally try to avoid adjectives and adverbs (e.g., "intensive analysis" and "jointly demonstrate" and "mainly three-folds").
* At the end of the introduction, I was still unsure what problem was being solved.
* L57 "in-efficient" -> "inefficient"
* L58 "an RL algorithm" -- Add a citation.
* L62 "reasoning" -> "reasoning about"
* L81 "unify" -- Unify with what?
* L104 "both worlds" -- Unclear what "worlds" refers to here.
* L107 - L111 -- I found these few sentences hard to follow.
* L114 - L127 -- These definitions are great, and very helpful!
* L128 -- From reading this paragraph, it was hard to tell whether $C$ was known or not known.
* L132 "taken policy" -- Unclear what this means.
* L136 - L137 -- I found the definition of Eva() pretty confusing. I might recommend explaining the behavior cloning case first without introducing Eva(), and later describing how these results can be extended to the RL setting.
* L137 $J(\pi)$ -- I found it confusing that the same notation, $J(\pi)$, was defined differently in three different places: L92, L98, L137.
* L146 "correlated agreement" -- No need for the quotes.
* L148 "instances" -- Unclear what an "instance" is.
* L151 - L156 -- This intuition is pretty clever!
* L157 - L164 -- This example is somewhat hard to follow. Consider explaining it with less notation, perhaps with pictures.
* L165 "consolidate our ..." -- Unclear what this means.
* L174 "with agreement check" -- Unclear what this means.
* Lemma 1 -- This result is fairly elementary and could be moved to the Appendix.
* L196 - L199 -- I found these sentences a bit confusing.
* L207 "we found that" -- Unclear if this is a theoretical or empirical result.
* Figures -- Please make the figures colorblind-friendly.
* L276 - L277 -- This sentence is pretty wordy and can be greatly shortened.
* L278 "other baselines" -- I would highly recommend adding a separate section that describes the baselines.
* L279 $e=0.4$ -- It could be worth noting that $e \le 0.5$. I initially thought $e \in [0, 1]$ and assumed that 0.4 was a rather small value.
* L281 "which again ... " -- This can be cut.
* L282 "more challenging" -- Inverted pendulum is not really more challenging than CartPole.
* L286 "experiment" -- Grammar error?
* L286 "uniform noise" -- Why is this a reasonable problem setting? Is this similar to some real-world tasks?
* L288 "biased noise" -- Why is the noise biased?
* L300 - L301 -- I didn't understand this sentence.
* L307 "three seeds" -- Is the same expert policy used for each seed? [That's fine, I just wanted to clarify.]
* L315 "GAIL and SQIL" -- These methods are designed for the setting where the agent can interact with the environment. Is this the setting in which they were used?
* L334 "which again ..." -- This sentence can be cut.


-----------------------------
**Review decision**: Overall, the proposed method seems simple and empirically seems to work well. The paper is generally well written. I have two main concerns with the paper. First, it is unclear *why* the proposed method should outperform the baselines; the analysis doesn't seem to answer this question. Second, it is unclear if the paper is actually solving a useful problem. While this problem has been studied in prior work, the problem setting differs from the standard RL setting. I would encourage the authors to address these two concerns in the rebuttal.

**Update after discussion**: Thank you for the discussion and additional experiments studying the "discreteness". I think I finally understand where the "magic" is coming from. At this point, my concerns about (1) why the method should work and (2) why this is a useful problem setting have been well addressed. I have increased my review to score = 7. In the final version of the paper, I would highly encourage the authors to include many of the points from the discussion
limitations_and_societal_impact: The concluding paragraph does a great job discussing limitations and areas for future work. I did not see any discussion of societal impacts, either positive or negative.
review_title: null

Meta-review: This paper was an exemplary case of the value of discussion between the reviewers and the authors. After multiple clarifications, additional experiments and even a code snippet examples, the reviewers agreed that the paper provides a valuable contribution to a problem of weak supervision in RL. I encourage the authors to address reviewers' extensive comments and update the paper for the final revision.
Decision: Accept (Poster)