Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper formulates a framework for reinforcement learning and behavior cloning from weak supervisions (i.e., noisy rewards or imperfect expert demonstration). Specifically, it proposes PeerPL to perform efficient policy learning from the available weak supervisions, which covers PeerRL (for RL with noisy rewards), PeerBC (for imitation learning from imperfect demonstration) and PeerCT (for hybrid setting). The PeerPL idea is based on a new weak supervision objective that is in the form of difference between norm learning loss and a loss incurred by randomly sampling the supervision signals. Experimental results demonstrate that PeerPL significantly outperforms SOTA solutions when the complexity or the noise of the learning environment grows. The proposed idea is useful in practice as it increases the robustness of the learning process to imperfect supervision signals.

Comments: 
1.	The proposed idea seems to share some similar spirit as the contrastive learning, where it also constructs a contrastive training loss from positive and negative examples. The CA loss (1) is somehow similar to contrastive learning with one negative sample. It would be helpful for the authors to add some further discussion in this regard.

2.	In the illustrative toy example, it would be helpful to explain how the numbers 0.75 and 0.25 come up. Based on my understanding, it seems that the calculation should be 1 – (0.8^2+0.2^2)=0.32?  (The probability of a_j and a_k’ being 1 is 4/5 and the probability of them being 0 is 1/5. So the probability of them being equal is (4/5)^2 + (1/5)^2.) Also, it is necessary to clarify that Eva is chosen to be the indicator loss here. Otherwise, it could be confused with the fact that cross-entropy loss is used in practical implementation.

3.	Based on Lemma 1, in order to guarantee convergence in Theorem 1, shouldn’t there be an additional condition of: e_{+} + e_{-1} < 1 (i.e., not too noisy) in Theorem 1 (to keep the same sign of the reward)? Likewise, I think there should be a similar requirement on the PeerBC setting as well (as reflected in the 1-e_{-}+e_{+} in the denominator of the bound in Theorem 2). If this is true, I think this point should be clarified as we do need some (weak) prerequisite/knowledge about noisy model. If this is not required, then it would be important to show some further experiments on this extremely noisy setting of e_{-}+e_{+} > 1. (Note that currently it only shows up to e=0.4.) In addition, I think it would be better to put all other assumptions (if any) on the noise model in the Theorem.

4.	Regarding the sample complexity result of PeerRL relative to RL with true rewards below Theorem 1, I think it might be useful to put it as a separate theorem after Theorem 1.

5.	Please also report the perfect expert (fully converged PPO) results in Table 2 as a performance upper bound.
review_title: Interesting work on learning from noisy supervisions

final_score: rating: 6: Marginally above acceptance threshold
review_content: SUMMARY OF CONTRIBUTION:

This work presents a pair of algorithms, PeerRL and PeerBC, which address certain forms of noise in reinforcement learning and behavioral cloning respectively.  For PeerRL, an arbitrary RL algorithm trains on an augmented reward signal, which is generated by subtracting a past reward signal (sampled randomly from the replay buffer) from the current reward.  PeerBC uses an augmented maximum-likelihood behavioral cloning loss that penalizes a policy for log-probability of a randomly sampled action for a randomly sampled state.  Empirical results show that Deuling DQN with the PeerRL reward outperforms the base DDQN algorithm in a noisy-reward version of the cart-pole task.  They also show that PeerBC outperforms standard behavioral cloning in learning from synthetic demonstrations on several Atari games, as well as the cart-pole and Acrobot tasks, all with noisy versions of the base reward signals.  The also evaluate PeerBC in in combination with reinforcement learning in a co-training setup, demonstrating a significant advantage over co-training with standard BC.

AREAS OF CONCERN:

The main concern with this work is that the gains in performance observed with the PeerRL and PeerBC algorithms may not actually reflect the ability of these methods to correct for a specific type of noise.  For PeerRL, the modified reward is fact that PeerRL is able to outperform the version of DDQN with access to the true reward (Figure 2) suggests that at least part of advantage of PeerRL is better scaling of the reward signal, rather than noise reduction.  It would be helpful to compare PeerRL against noiseless scales and shifts of the true reward.  It also seems likely that simply adding a constant baseline value to the noisy reward would help reduce the variance of the return (PeerRL subtracts a noisy baseline value).  Comparing against this simpler variance reduction method would help us understand the importance of the specific form of the augmented PeerRL reward in improving performance.

For PeerBC, the concern is that the modified loss simply biases against high-entropy policies.  When the true policy is itself stochastic, the learned policy found by PeerRL may end up being the mode of this policy.  The advantage of PeerBC observed in Figure 3 may result from the fact that less noisy policies perform better in these tasks.  The fact that PeerBC outperforms the expert policy in Enduro (Figure 3c) would seem to support this hypothesis (the expert's true policy is more stochastic than the learned policy).  It might be helpful to compare PeerBC to a simpler approach which learns a stochastic policy (under the standard maximum likelihood objective), but always takes the most probable action under this policy during evaluation.

While the authors claim to address the case where the rewards or demonstrated actions are perturbed by some arbitrary confusion matrix.  The theoretical and empirical analysis of PeerRL however are limited to the case of binary rewards. Similarly, the theoretical analysis of PeerBC appears to limited to the case of binary actions, though empirical results consider larger action spaces.

More generally, it is not clear that the noise models (for RL and BC) capture the challenges typical of RL and imitation learning (discussed in the first paragraph of the introduction).  In RL, the issue is often not that the reward is noisy in a fixed state, but that it is sparse within the state space, such that the return under a random policy is noisy.  For behavioral cloning, errors in human-generated data often reflects the fact that the human demonstrator has actually selected a suboptimal strategy, rather than the noisy execution of an optimal strategy.  Distributional shift in behavioral cloning does not reflect noise in the training data itself, but instead reflects the compounding error that occurs when we execute an imperfect policy for many timesteps.

These issues with the noise model could be addressed by making it clear earlier in the document exactly what noise models are being considered, and providing examples in the introduction of settings in which such noise would be expected in the rewards or demonstration data.

The authors attempt to describe their approaches to handling noise for RL and BC as instances of a more general peer evaluation algorithm.  The relationship between the RL and BC solutions appears to be superficial, however, as PeerRL and PeerBC seem to address noisy supervision in very different ways.  Furthermore, there are no theoretical results presented that apply to the general algorithm.  The derivation of the RL solution from the common framework (Equation 2) is incomplete, as the loss Eva^RL is never made precise.  It would likely be more clear to the reader if PeerRL and PeerBC were presented as separate, but analogous algorithms for their respective learning problems.

There are also some apparent technical issues with the theoretical results:

1) At no point is it specified that the error rates must be less than 50% for binary rewards or actions. Theorems 1 and 2 are clearly wrong if this condition does not hold.  While and attentive reader should be able to infer this constraint on the noise model, it should be explicitly stated.

2) The r_peer term (the true peer reward) in Lemma 1 is never defined.

3) It is unclear what the variable theta in tau_theta refers to in section 4.2.

4) There seems to be an error in the proof of Lemma 1. The decomposition of the noisy reward in lines (7) and (8) seems to assume that, even in the absence of noise, the expected reward signal will be zero, regardless of the policy being followed, or the current state.  This decomposition should incorporate the probability of true reward given the current state (or averaged over states given the current policy).  This may mean that Lemma 1, and potentially Theorem 1, are incorrect (though it is likely that this can be corrected without substantially changing the contribution of the paper).

CONCLUSION:

Even though the work considers a relatively constrained class of noise models, there are still practical cases where such noise in rewards (and particularly in demonstrated actions) is an issue, and effective methods for dealing with such noise are potentially valuable.  The issue with this work is that it is not yet clear that the proposed methods are actually effective in mitigating these types of noise in RL or BC.  It seems likely that at least part of the improvement in performance stems from other factors, either the preference for more deterministic policies in PeerBC, or the rescaling of the reward signal in PeerRL.  Without resolving these issue, it is impossible to know whether the proposed methods will actually be useful in settings where we have noisy rewards or demonstration data.  There are also technical issues which raise doubts about the validity of the theoretical results presented in this work.
review_title: This work presents two new methods, PeerRL and PeerBC, which address certain forms of noise in reinforcement learning and behavioral cloning.  Empirical results suggest that these methods are effective at mitigating the effects of noise in rewards and demonstrated actions, but it is unclear that the observed improvements in performance are actually the result of the new methods being more robust to noise.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Sumary

This paper tackles a very important problem of reinforcement learning/imitation learning in the presence of noisy rewards/labels and uses contemporary literature to motivate a simple solution (in a good sense). The empirical results look encouraging on the benchmark problems but the analysis is a little wanting and doesn't get at the heart of the matter. I think with a little more work this paper can be a very good conference paper but at this stage I can't recommend publication in ICLR. 

**Before Author Response** Of course, if the authors address my concerns then I'll increase my rating. 
**After Author Response** I think the authors made a good effort to address the concerns and I have recommended to accept the paper.

## Contributions 

This paper tackles the problem of learning from imperfect reward/imperfect demonstrations using a new evaluation metric called "Correlated Agreement". The proposed metric "regularizes" learning under weak supervision by penalizing "over-agreement" with the supervision signal. The proposed method is meant to be used in situations where the supervision signal is known to be noisy but it does not estimate parameters of the noisy channel   which is corrupting the supervision signal.

## Strengths

The strongest aspect of the paper in my opinion are the strong empirical results on fairly standard RL benchmarks. The paper also explains the concept of learning from "peer agents". The analysis of the methods presented covers the most obvious questions one might ask at the beginning about the proposed method and the appendices cover those well.

## Weaknesses

There are two problems I see with the analysis in the paper that gives me concern:

1. The main result in the paper is Theorem 1 which relies on Lemma 1. Lemma 1 essentially states that the proposed new metric (Peer RL reward) that subtracts a chance agreement baseline (Second term in equation 1) from the standard agreement objective (First term in equation 1) is an affine function of the true reward. Theorem 1 then crucially relies on this result for the ensuing convergence analysis. The problem I see is that an analogous result to Lemma 1 -- and therefore Theorem 1 -- can also be proved for the noisy reward. Specifically equation (12) in Appendix A.1 shows that even the noisy reward 𝔼[\tilde{r}] is an affine function of the true reward, with the exact same slope and just a different intercept. Given that observation the proof for Theorem 1 can be carried out /mutatis mutandis/ to prove convergence with the naive noisy reward instead of the peer reward. So to me all of the analysis in the current paper doesn't really explain "why" PeerRL is actually performing better than using noisy rewards. To me it seems that more attention must be paid to the intercept terms in equation (12) and (17) to really understand the gap in performance which the authors currently do not consider. This is the first issue I see in the analysis.

2. The second issue I have is with the "Correlated Agreement" Objective itself. The "CA with weak supervision" objective shown in equation (1) seems to give unfair advantage to completely random, but high entropy, weak supervision. For example, in the toy example on Page 4 the authors give a demonstration that the "CA metric" will be 0.375 thereby punishing full agreement with a weak baseline. But if instead the weak baseline was a random and unbiased coin toss then the expected CA objective will be 1 - 0.25 - 0.25 = 0.5 > 0.375. So just because the weak supervision signal had higher entropy and therefore lower chance of random agreement therefore its score was higher. This aspect should be dealt with more thoroughly in the paper. 
review_title: Good empirical results but the theoretical justification doesn't seem right

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tackles the problem of policy learning under weak/noisy supervision. The authors present PeerPL, a unified framework that can train agents using behavior cloning under noisy/suboptimal demonstrations, or using reinforcement learning under noisy rewards. PeerPL uses the idea of “Correlated Agreement” by subtracting the original objective with a second term. The second term evaluates on randomly paired state-action tuples and supervisions, punishing the “blind” agreement between the learning agent and the weak supervision. The authors instantiate this idea on both behavior cloning and RL (PG/DQN), and also evaluate it on the problem of policy co-training. The authors demonstrate that PeerPL outperforms the weak supervision baselines on IL and RL setting, and sometimes it even outperforms agents trained with clean supervision. 


Strengths

+ The paper overall is well-written and easy to follow
+ The proposed idea is simple and general, and can be applied to both RL and IL
+ The proposed method does not require prior knowledge on the noise structure
+ Nice and sound convergence analysis
+ Proposed method outperform baselines in most tasks under both BC/RL settings and policy-cotraining


Weaknesses

- Although the experiments prove otherwise, the second term in the objective is not intuitive to me. The learning agent is not supposed to agree with randomly sampled supervision anyways, unless the noise is constructed in a way that encourages the agent to do so. I cannot convince myself that the second term is helpful other than potentially some normalizing factor that helps reduce the variance in the RL setting, or maybe help remove some misleading prior in the BC setting.

- I am confused by the results in Figure 2 that PeerPL outperforms agents trained with *clean* reward. I am not convinced by the authors’ explanation in Section 5.1, as subtracting the reward with randomly sampled {-1,1} will not help exploration.

- The theoretical results are not particularly useful as they assume binary reward/actions, which is rarely the case in practical settings.


Other Comments

- It would have been nice in section 5.2 how PeerBC compares against IRL, which is supposed to handle imperfect demonstrations to some degree.

- For PeerRL, it would be nice to see results on tasks that are harder than CartPole, for example, Atari or other continuous control tasks.

review_title: Well written and presents simple idea, but with some concerns

Meta-review: The reviewers had some initial concerns about this submission. While the authors' rebuttal does a good job to address these concerns, the reviewers still have some doubts about the contribution of this paper and potential impact. In particular, it is not clear whether the performance improvements observed with the proposed algorithms is due to the ability to correct for noisy rewards or whether there are multiple other explanations for the improvement in performance. This makes it hard to predict whether the proposed algorithms will actually be useful in settings where noisy rewards or demonstration data are present.
Decision: Reject