Policy Learning Using Weak Supervision
Authors: Jingkang Wang
Hongyi Guo
Zhaowei Zhu
Yang Liu
Venue:  (2021)
Keywords: Weak Supervision
Policy Learning
Correlated Agreement

Abstract: Most existing policy learning solutions require the learning agents to receive high-quality supervision signals, e.g., rewards in reinforcement learning (RL) or high-quality expert demonstrations in behavior cloning (BC). These quality supervisions are either infeasible or prohibitively expensive to obtain in practice. We aim for a unified framework that leverages the weak supervisions to perform policy learning efficiently. To handle this problem, we treat the ``weak supervisions'' as imperfect information coming from a \emph{peer agent}, and evaluate the learning agent's policy based on a  ``correlated agreement'' with the peer agent's policy (instead of simple agreements). Our way of leveraging peer agent's information offers us a family of solutions that learn effectively from weak supervisions with theoretical guarantees. Extensive evaluations on tasks including RL with noisy reward, BC with weak demonstrations, and standard policy co-training (RL + BC) show that the proposed approach leads to substantial improvements, especially when the complexity or the noise of the learning environments grows. 

Under review as a conference paper at ICLR 2021
POLICY LEARNING USING WEAK SUPERVISION
Anonymous authors
Paper under double-blind review
ABSTRACT
Most existing policy learning solutions require the learning agents to receive high-quality supervision signals, e.g., rewards in reinforcement learning (RL.) or high-quality expert's demonstrations in behavioral cloning (BC). These quality supervisions are either infeasible or prohibitively expensive to obtain in practice. We aim for a unified framework that leverages the weak supervisions to perform policy learning efficiently. To handle this problem, we treat the "weak supervisions" as imperfect information coming from a peer agent, and evaluate the learning agent's policy based on "correlated agreement" with the peer agent's policy (instead of simple agreements). Our way of leveraging peer agent's 's information offers us a family of solutions that learn effectively from weak supervisions with theoretical guarantees. Extensive evaluations on tasks including RL with noisy reward, BC with weak demonstrations and standard policy co-training (RI. + BC) show that the proposed approach leads to substantial improvements, especially when the complexity or the noise of the learning environments grows.
INTRODUCTION
Recent breakthrough in policy learning (PL) opens up the possibility to apply these techniques in real-
world applications such roboties (Mnih et al. 2013 Akkaya al.] 2019) and self-driving (Bojarski) et al. 2016a; Codevilla et al., 2018). Nonetheless, most existing works require agents to receive
high-quality supervision signals, e.g., reward or expert's s demonstrations, which are either infcasible or prohibitively expensive to obtain in practice. For instance, (1) the reward may be collected through
sensors thus not credible (Everitt et al.
2017 Romoff et al. 2018 Wang et al.) 2020); (2) the
demonstrations by an expert in behavioral cloning (BC) are often imperfect due to limited resources
and environment noise (Laskey et al. 2017 Wu et al. 2019 Reddy et all 2020).
Learning from weak supervision signals such noisy rewards F (noisy versions of r) (Wang et al. 2020) low-quality demonstrations DE (noisy versions of Dg) produced by problematic expert Tg (Wu et al., 2019) is one of the outstanding challenges that prevents a wider application of PL. Although some recent works have explored these topics separately in their specific domains (Guo et al. 2019: Wang et al.] 2020:: Lee et al. 2020), there is no unified solution towards performing robust policy learning pip install arxiv-latex-cleaner under this imperfect supervision. In this work, we first formulate a meta-frament framework to study RL/BC with weak supervision signals and call it weakly supervised policy learning. Then as response, we propose theoretically principled solution concept, PeerPL, to perform efficient policy learning using the available weak supervisions.
Our solution concept is inspired by the literature of peer prediction (Miller et al. 2005 Dasgupta & Ghosh 2013, Shnayder et al., 2016). where the question concerns verifying information without ground truth verification. Instead, group of agents' reports (none of which is assumed to be high- quality nor clean) are used to validate each other's information. We adopt similar idea and treat the "weak supervisions" as information coming from a peer agent, and evaluate the learning agent's policy based "correlated agreement" (CA) with the peer agent's. Compared to standard reward/loss functions that impose simple agreements with the weak supervisions, our approach punishes an over- agre-aggeementt to avoid overfitting to the weak supervisions. Our way of leveraging peer agent's information offers us a family of solutions that 1) does not require prior knowledge of the weakness of the supervisions, and 2) learss effectively with strong theoretical guarantees.
We demonstrate how the proposed PeerPL framework adapts in challenging tasks including RL with noisy rewards and behavioral cloning (BC) from weak demonstrations. Furthermore, we provide

---

Under review as a conference paper at ICLR 2021
intensive analysis of the convergence behavior and the sample complexity for our solutions. These results jointly demonstrate that our approach enables agents to learn the optimal policy efficiently under weak supervisions. Evaluations these tasks show strong evidence that PeerPL brings significant improvements over state-of-the-art solutions, especially when the complexity or the noise of the learning environments grows.
To summarize, the contributions in the paper are mainly three-folds: (1) We provide a unified formulation of weakly supervised policy learning model the weak supervision in RL/BC problems; (2) We propose a novel PeerPL solution framework based on calculating a correlated agreement with weak supervisions, novel way for policy evaluation introduced RL/BC tasks; (3) PeerPL is theoretically guaranteed to recover the optimal policy (as if the supervisions are of high-quality and clean) and competitive empirical performances are observed in several policy learning tasks.
2 RELATED WORK
Learning with Noisy Supervision Learning from noisy labels is widely explored within the supervised learning domain. Beginning from the seminal work NNtarajan al., 2013) that first proposed an unbiased surrogate loss function to recover the true loss given the knowledge of noise rates, follow-up works focus on how to estimate the noise rates based on noisy observations (Scott et al.. 2013 Scott 2015 Sukhbaatar & Fergus 2014; van Rooyen & Williamson 2015 Menon et al. 2015). Recent work (Wang et al. 2020) adapts this idea within RL and proposes a statistics-based estimation algorithm. However, the estimation is not efficient especially when the state-action space huge. Moreover, as a sequential process, the error in estimating the noise rate can accumulate and amplify when deploying RL algorithm. In contrast, our solution in this paper does not require priori specification of the noise rates thus offloading the burden of estimation.
Behavioral Cloning (BC) Standard BC (Pomerleau 1991
Ross & Bagnell 2010 tackles the
sequential decision-making problem by imitating the expert's s actions using supervised learning. Specifically, it aims to minimize the one-step deviation error over the expert's 's trajectory without reasoning the sequential consequences of actions. Therefore, the agent suffers from compounding errors when there is a mismatch between demonstrations and real states encountered (Ross & Bagnell
2010 Ross al. 201T). Recent works introduce data augmentations (Bojarski et al.. (016b) and value- based regularization (Reddy et al. 2019) or inverse dynamics models (Trabi et al.
[2018] Monteiro
et al. 2020) to encourage learning long-horizon behaviors. While simple and straightforward, BC has been widely investigated in a wide range of domains GGiusti et al. 20 Justesen & Risi 2017) and
often yields competitive performance (Farag & Salch 2018 Reddy et al. 2019). Our framework is complementary to current BC literature by introducing learning strategy from weak demonstrations (e.g., noisy or from a poorly-trained agent) and provides theoretical guarantees on how to retrieve clean policy under mild assumptions (Song et al. 2019).
Correlated Agreement Peer prediction aims elicit information from self-interested agents without ground-truth verification (Miller et al., 2005 Dasgupta & Ghosh] 2013 Shnayder et al. 2016). The only source of information to serve verification is from the agents' reports. Particularly,
in Dasgupta & Ghosh 2013); Shnayder et al. 2016). correlated agreement (CA) type of mechanism proposed, which evaluates the correlations between agents' reports. addition encouraging some agreement between agents, CA mechanism also punishes over-agreement when two agents always report identically. This property helps reduce the effect of noisy reports by punishing overfitting to them. Recently, Liu & Guo (2020) adapts similar idea in learning from noisy labels for supervised learning. We consider a more challenging weakly supervised policy learning setting and study the convergence rates in sequential decision decision-making problems.
POLICY LEARNING FROM WEAK SUPERVISION
We begin by introducing a general framework to unify PL with low-quality supervision signals. Then provide instantiations of the proposed weakly supervised formulation with two differ- ent applications: (1) RL with noisy reward and (2) behavioral cloning (BC) using weak expert demonstrations.
2

---

Under review as a conference paper at ICLR 2021
3.1 PRELIMINARY OF POLICY LEARNING
The goal of policy learning (PL) is to learn policy π that the agent could follow to perform series of actions in a stateful environment. For RL, the interactive environment characterized as an MDP M = (S, A, R. P. P))). At each time t. the agent in state s, E S takes an action a, E A by following the policy π : S A + R. and potentially receives reward 0e) ) E R. Then the agent transfers to the next state st-1 according to a transition probability function PP. We denote the generated trajectory
t = {($t: de: re)) where T is a finite or infinite horizon. RL algorithms aim to maximize the
expected reward over the trajectory r induced by the policy: J(T) = E ((s,,0e.r))- Ei= y'ra], where 7 (0, 1] is the discount factor.
Another popular policy learning method is through behavioral cloning (BC). The goal of BC is to mimic the expert policy Fe through demonstrations DE = {s a,)] drawn from distribution DE (generated according to kg), where (s,, q,) is the sampled state-action pair from the expert's s trajectory. Typically, training a policy with standard BC corresponds to maximizing the following log-likelihood: J (Tr = >~DD [log x (a]s)).
In both RI. and BC, the agent receives "supervisions" through either the reward r by interacting with environments or the expert policy ge as observable demonstrations. Consider particular policy class 11, the optimal policy is then defined as x² arg max, J (TT): x² obtains maximum expected reward over the horizon T in RL and π corresponds to the clean expert policy Te in BC. In practice. one can also leverage both RI. and BC approaches to take advantage of both worlds (Brys et al.)
2015, Hester et al.] 2018 Guo et al., 2019 Song et al. 2019). Specifically, a recent hybrid framework called policy co-training (Song ct al. 2019h is considered in this paper.
3.2 META FRAMEWORK FOR POLICY LEARNING WITH WEAK SUPERVISION
To unify, we denote these weak supervision signals using Y that is either the reward F for RL or the action à performed by an expert policy मे for BC. Y denotes a weak version of a high-quality supervision signal Y. As a consequence, in an abstract manner, a weakly supervised PL problem can be formulated as learning the optimal policy IT with only accessing a weak supervision sequence denoting as {(..,,,) Y
To unify the discussion, suppose that we have an evaluation function Eva,((s:: az), Yt) that evaluates a taken policy at state (si, a.) with weak supervision Y, In the RL setting, this Eva,, is the loss for different RL algorithms, which is a function of noisy reward F received at (st,a:). a:). While for the BC setting, this Eva, is the loss used evaluate the action taken by the agent given the action taken by the expert. Furthermore, we let J (x) denote the function that evaluates policy x under a set of state
action pairs with weak supervision signals ((s; oz), Y i.e., ((T) = (( [Eva,, ((s, a). Y)]. Note that above unified notations are only for better delivery of our framework and we still treat
PL as sequential decision problem. We focus on the following two instantiations for our weakly supervised settings:
RL with Noisy Reward Consider a finite MDP M = (S,A,,,,,,P)))) with noisy reward channels WWang et al. 2020), where R: : S x A R, and the noisy reward is generated following a certain function F : R + R. Denote the trajectory a policy म. generates via interacting with M as to. Assume the reward is discrete and has |R] levels, the noise rate can be characterized via matrix CRI.
R Ri where each entry Gj,s indicates the flipping probability for generating a perturbed outcome:
= (F Rx/re- = R2). We call r and the true reward and noisy reward respectively.
2.
BC with Weak Demonstration Instead of observing the expert demonstration generated according to Tg. denote the available weak demonstrations by ((*;, a where ā ~ +g(+1*)) is the noisy action and each state-action pair (st, ā.) is drawn from distribution DE. In particular, we assume the noisy action ă; is independent of the state s given the deterministic expert action Te (s), i.e., F(ā,|TT(ss) = P(ã.|s,, (s:).) Similar to RL, assume the noise regime can be characterized by a confusion matrix CBC x LAI where each entry Cj.k indicates the flipping probability for the expert to take an suboptimal action BBC P( (( (8*) = Ax)Tg(3)) = Ay). In this setting, we'd like to recover x² if we were able access the quality expert demonstration ag instead of fg.
3

---

Under review as a conference paper at ICLR 2021
BC with Weak Expert
with Perturbed Reward
Policy Co-Training
Behavior Cloning
Peer Policy Learning
Reinforcement Learning
Agent: (o..k) (5. Du k.
Weak
(s,0),
Peer Agent: i.. 8. j.. 1/.
Noisy Environment
Demonstrations
(5,4)
CA:
Eva Rg). fn) Eva (154a))
F
Figure I: Illustration of weakly supervised policy learning and our PeerPL solution with correlated agreement (CA). We use Y to denote a weak supervision, be it a noisy reward, or noisy demonstra- tion. Eva stands for an evaluation function. "Peer Agent" corresponds to weak supervisions.
Remark We emphasize that we not have nor need the knowledge of the weakness the signals, i.e. CRL [ nor C LA] As we show later, our approach successfully avoided the need of this knowledge. This is also main challenge for designing solution.
4 PEERPL: WEAKLY SUPERVISED PL VIA CORRELATED AGREEMENT
To deal with weak supervisions in PL, we propose a unified and theoretically principled framework PeerPL. We treat the weak supervisions as information coming from a "peer agent", and then evaluates the policy using a certain type of "correlated agreement" function between the learning policy and the peer agent.
4.1 OVERVIEW OF THE IDEA: CORRELATED AGREEMENT WITH WEAK SUPERVISIONS
We first present the general idea of our PeerPL framework that uses concept named "correlated agreement" (CA). For each weakly supervised state-action pair ((s,, a)), Yi). randomly sample state-action pair (æj, a f i, as well as another supervision signal Ys, + i from different state-action pair. Then we evaluate ((s;, 0g), Y;) according to the following:
CA with Weak Supervision: Eva,, ((s;,a;) a;), Y,) Eva., ((s), Vk.
Intuitively, the first term above encourages an "agreement" with the weak supervision (that a policy agrees with the corresponding supervision), while the second term punishes "blind" and "over" agreement that happens when the agent's policy always matches with the weak supervision even on randomly paired traces (noise). The randomly paired samples j, k helps us achieve this check. Note the implementation of our mechanism does not require the knowledge of CRL |R|x|R| nor CBC LA and offers prior-knowledge free way to learn effectively with weak supervsions. This above process is illustrated in Figure
Illustrative toy example Consider a toy BC setting where we learned a policy that outputs a sequence of actions a1 02 0g 1, 04 0, and they have perfectly matched the weak supervision a' di=00 ab = al3 = 1, d4 = 0 (at the same sequence of states). Supposedly Eva, az), af) evaluates how well policy matches the expert demonstration (Eva,
for agreeing, 0 for not). Using only Evar((si, at), af) will return the highest score 1 for agreeing with this noisy/ippeffec/lov- quality supervision. However the correlated agreement evaluation returns (for example for i 1) ((EEv,,(ss az), a() Eva,, ((5), a,), d/e, k * j)] = 1 - (0.75² + 0.25?) = 0.375, where 0.75² + 0.25² is the probability of randomly paired a and a' match each other. The above example shows that full agreement with the weak supervision will instead be punished!
In what follows, we solidate our implementations within each of the settings considered and provide theoretical guarantees under weak supervision.
4.2 PEERRL: PEER REINFORCEMENT LEARNING
Since the reward signals are no longer credible the weakly supervised RL setting, we propose the following objective function that punishes the over-agreement based on CA mechanism
JRL (mo) = E Eva!!! ((si, o.). -E E(Eva RL (($},
(2)
where Eva RL ((s, ass,a,,)) - [((TO, (s)
(3)
4

---

Under review as a conference paper at ICLR 2021
In P.. the first expectation is taken over (st, d,, f.)) Fa and second one is taken over (sj. aj f3) To, (5k; ag, (k)) ~ Tg. where Tg is the trajectory generated by Te with the noisy reward function r. The choice of the loss function ε depends on the RL algorithms used (c.g.. temporal difference error (Mnih et al., 2013, Wang et al. (016) or the policy gradient loss (Sutton et al. 1999). ). Also, the learning sequence encoded in x, therefore, maximizing the objective J (πr) is equivalent maximizing the accumulated reward. ç 2 0 is hyperparameter to balance the penalty for blind agreements induced by CA.
In what follows, we consider the Q-Learning (Watkins & Dayan 1992) as the underlying learning al- gorithm where (TT,, (s, a, f)) - and demonstrate our CA mechanism provides strong guarantees for Q-Learning with only observing the noisy reward. For clarity, we define peer reward as
Peer RL Reward:
5. (p a) = F(s,a) 5.
where r' is randomly sampled reward over all state-action pair and & 2 0 is parameter balance noisy reward and the punishment for blind agreement (with 7'). We set { = (for binary case) in the following analysis and treat each (s, a) equally when sampling the r'. In experiment, 'pper is not sensitive to the choice of ç, and we have kept & constant for each run of the RL experiments.
We show peer reward peer offers the key to guaranteeing the convergence of our Peer RL. algorithm to converge to æ². For clarity, consider the binary reward setting (r+ and r-) and denote the error in F e.+ P(f r-
affine transformation of the true reward in expectation, which
r+),e_ = P( = T+ -+ r-) (a simplification of CRL |||||/ [R]
in the binary setting).
Lemma 1. Let [0, Rmaa]] be a bounded reward. Assume e- e 0, then we have:
E[Fpeerl = (1 -e_ -e+) [E[ = e__ e4) Brrr + const,
where Tpeer Tpeer (s, a) = r(s,))(=) E. the peer RL reward when observing the true reward r.
Lemma I shows that by subtracting the peer penalty term Fl from noisy reward F, Fper recovers the clean and true reward expectation.
Remark It's notable that the expectation the noisy reward [[F] can also be written as:
Fl = -- e+ [rr) + e- +
cunst
However, we claim that the constant in peer reward has less effect on the true reward r. That's because we have that
E[F] (1 -1-e- -c) E[r] +
64
r- + e_
e_ c+ r++)
e_ e+
E El (1 e_ e+) (E[r)
Pr - Ppeer"+)
where Ppeer E [0, denotes the probability that a sample policy gets a reward r+. Since the magnitude T can potentially become much larger than 1 Ppeer and Ppeer in high-noise regime, T-e_- --e__e, T_ will dilute the informativeness of E[r]. On the
of noise terms T-e_-o+ and
contrary, E[epeer] contains moderate constant noise thus maintaining more useful training signals of the true reward in practice.
Based on Lemma
we further offer the following convergence guarantee:
Theorem 1. (Convergence) Given finite MDP with noisy reward, denoting as M = (S, R, F,P, P, 7), the Q-learning algorithm with peer rewards, given by the update rule,
Q(+1(st, = (1 ae)QQ((s, (((())(s)))) as) + as (ppeec(8s a.) + max Qe(s++1, a a')] d.)]. (T(s) = arg max Qr(8,
a'EA
converges w.p. the optimal policy n" as long as Σ, at = co and E, of < oo.
Theorem states that the agent will converge to the optimal policy w.p.l with peer rewards without requiring any knowledge of the corruption rewards (CRL, | as opposed to previous work (Wang et al. 2020) that requires such knowledge). Moreover, we found that, to guarantee the convergence to the number of samples needed for our approach no more than ((1((1 e_ ee)2) times of the one needed when the RL agent observes true rewards perfectly (see Appendix A).
5

---

Under review as a conference paper at ICLR 2021
Even though only presented analysis for the binary case for Q-Learning our approach is rather generic and is ready to be plugged into modern DRL algorithms. Specifically, we pro- vide the multi-reward extension and implementations with DQN (Mnih et al. 2013) and policy gradient (Sutton et al. 1999) using peer reward in Appendix A.
Remark
4.3 PEERBC: PEER BEHAVIORAL CLONING
Similarly, present our CA solution in the setting of behavioral cloning (PeerBC). In BC, the supervision is given by the weak expert' noisy trajectory. The Eva BC function in BC evaluates the agent policy Te and the weak expert' trajectory {[s,, ) using E(TTA (ss, ā:)) where é is an arbitrary classification loss. Taking for instance the cross-entropy, the objective of PeerBC is:
J (T)) = E Eva ((st.u.).:)) - & E[Eva ( ((s),a((
((5).
(4)
where
Eva
BC a) ( ()) log (ā]s).
(5)
In (4) the first expectation is taken over (si, ūs) ~ DE, a, T T--(a) and the second is taken over (s), (g) ~ DE. Oj ~ TT-18)) (sk, āx) DE, ag (-1(s)). Again, the second Eva term in JBC serves the purpose of punishing over-agreement with the weak demonstration. Similarly, & 2 0 is parameter to balance the penalty for blind agreements. At each iteration, the agent learns under weak supervision å, and the training samples are generated from the distribution DE determined by the weak expert.
We prove that the policy learned by PeerBC converges to the expert policy when observing sufficient amount of weak demonstrations. We focus on the binary action setting for the purpose of theoretical analysis, where the action space is given by A = {A+, A-} and the weakness or noise in the weak expert E is quantified by e. = (s) = A_ (s) = A4) and e = P( = A+ (s))) = A-) (a simplification of C x .Al in the binary setting). Let the The be the optimal policy by |Alxx
maximizing the objective in g with imperfect demonstrations De (a particular set of with N i..d. imperfect demonstrations). Note E(-) should be specified as indicator loss: (s) a) when (ss) # a, otherwise (()(s), = 0. We have the following upper bound on the error rate.
Theorem 2. Denote by B = ((s,a)/Dg ( (T (s) + a) the error rate for PeerBC. With probability
at least - 8, it is upper-boundee as: Rps S I-c 1+$ 2/082/5
Theorem states that as long as weak demonstrations are observed sufficiently. i.c., N is sufficiently large. the policy learned by PeerBC is able converge to the clean expert policy TT(s) with convergence rate of (1//NN).
Peer Policy Co-Training Our discussion of BC allows us to study more challenging co-training task (Song et al., 2019). Given finite MDP M, there two agents that receive partial observa- tions and we let # and =B de- note the policies for agent A and B. Moreover. two agents are trained jointly to learn with rewards and noisy demonstrations from each other (e.g., at preliminary training phase). Symmetrically, we consider on the case where agent A learns with the demonstrations from B on sampled trajectories, and πh effec- tively serves as noisy version of expert policy F = arg max ((ssaa)Da (lg
Algorithm 1 Peer policy co-training (PeerCT)
Require: Views A, B, MDPs MA, MM policies TA, mg. mapping functions fA-B, fD-A that maps states from one view to the other view, CA coefficient ç, step size 33 for policy update. repeat
2: Run Æ^ generate trajectories - ((5,) at rr Hii=!
3:
Run x to generate trajectories - = [(5 2g 15 1M
4: Agents label the trajectories for cach other
{a1 N
'mm"
B t (FA- j=1
5:
Update policies: x ((,,) 4- x (A,8) + 3 (
until convergence
For simplicity of demonstration, we focus on recovering the clean expert policy by only adapt- ing the BC evaluation term (ignoring the effect of RL rewards, see Eqn. (6). Denote by T (((+(()))) rr)) the trajectory that A generated via interacting with the partial world

---

Under review as a conference paper at ICLR 2021
202
200
200
DDQN
0015 per epoode
100
130
00035 por optocde
150
saaz per apsaads
1100
100
per
100
100
apro0ce
epledd
aplo0ce
epteode
(a) e 0.1
(b) 0.2
0.3
(d) 0.4
Figure 2: Learning curves of DDQN CartPole with true reward (r) noisy reward (?)
surrogate reward (Wang et al. 2020 (f) and peer reward (ipeer 5 = 0.2)
M Then T/ substitutes each action ag* with its selection at ~ π 3 ( (l-fA-BBs (s())) as the weak supervisions. Similar to PeerRL/ the objective function of peer co-learning (PeerCT) becomes
JCT (To) = E Eva." ((s," a.") (.)),r))) Eva. BC
(6) a B (-IAA B (s:t)) )), and the second 1B (IfAA-=B( l is the loss function a x ) T ok
((s)" aa"),a²"))
E. Eva.
BC ((0) ($5),af az)) ak .(
where the first expectation is taken over (s#t att r(t) Tp.
is taken over (sg) ag rnn)) TØ. (s
defined in Eqn. 5 to measure the policy difference, and Eva Eva BC are defined in Eqn. and We omit detailed discussions on the convergence of PeerCT due to it can be viewed as a straight-forward extension of Theorem in the context of co-training.
G respectively. The full algorithm PeerCT is provided in Algorithm II
5 EXPERIMENTS
We evaluate solution in three challenging weakly supervised PL problems. Experiments control games and Atari show that, without any prior knowledge of the noise in supervisions, our approach is able leverage weak supervisions more effectively.
5.1 PEERRL WITH NOISY REWARD
We first evaluate method in RL with noisy reward setting. Following Wang et al. (2020), consider the binary reward {-1, } for Cartpole where the symmetric noise is synthesized with different error rates e e_ 64. We choose DQN MMih et al. 2013) and Ducling DQN (DDQN) (Wang et al. 2016) algorithms and train the models for 10,000 steps. We repeat each experiment 10 times with different random seeds and leave the results for DQN in Appendix D.
Figure 2 shows the learning curves for DDQN with different approaches in noisy environments ( = 0.2) Since the number of training steps is fixed, the faster the algorithm converges, the fewer total episodes the agent will involve thus the learning curve is on the left side. As consequence, the proposed peer reward outperforms other basclines significantly even in a high-noise regime (e.g.. = 0.4). Moreover, we highlight that peer reward does not require any knowledge of noise rates or complicated estimation algorithms compared to Wang et al. (202)). Table I results the average reward Rang and total episodes Nepi- We find the agents with peer reward lead to a larger Rang (less generalization error) and a smaller Nepi (faster convergence) consistently, which again verifies the effectiveness of our approach.
provides quantitative
Analysis of the benefits PeerRL When the noise rate is small, we observed that the agents with peer reward even lead to faster convergence than ones observing true reward perfectly. This indicates there might be other factors apart from noise reduction that promote the RL learning with peer reward. We hypothesize this is because (1) peer reward scales the reward signals appropriately, which potentially reduces the variance and makes it casier to learn from; (2) the peer penalty term encourages explorations in RL implicitly; (3) the human-spechien
"true reward" is also imperfect which leads to a weak supervision scenario. More discussions and analysis are deferred to Appendix C..6 There might be multiple explanations for the improvement in performance provided by PeerRL depending on the specific RL tasks. However, we emphasize that the property of recovering from noise is non onglegiiiible especially in high-noise regime (e.g., e 0.4).
We analysed the sensitivity of & and found the algorithm performs reasonable when & (0.1, More insights and experiments with varied is deferred to Appendix D.
7

---

Under review as a conference paper at ICLR 2021
Table 1: Numerical performance of DDQN CartPole with true reward (r), noisy reward (?), surrogate reward f (Wang et al. 2020). and peer reward F (5 = 0.2). Rang denotes average reward per episode after convergence, the higher (1) the better; N Nepi denotes total episodes involved in 10,000 steps, the lower (4) the better.
0.1
e
0.4
Ragg
New
Rang
Neg
Rung 1
Nej 4
Rayy T
Nugi
195.6 + 3.1
101.2+33.
195.6 + 3.1
101.2 3.2
195.6+311
101.2++3.2 3.2
105.2+3..0
101.2 - 3.3
DDON
185.2 L. 15.6
114.6 0.6...00 6.0
168.8
1366
123.9
1 9.6
177.1 11.2
133.2
185.5 1 10.9
163.1 1 11.0
183.9 = 10.4
110.6
163.121822
113.9
9.6
192.2 = 10.9
115.5 4.3
179.2
125.8
9.6
poor
198.5
2.3
86.2
1: 5.0
195.5 1 9.1
85.3
1: 5.4
174.1 32.5
88.8 d: 6.3
191.8 1 8.5
106.9 1 9.2
20
80
00
200
0200000
200
batccee (c103)
batches (4700)
bakdhaa (x100)
batchee (x100)
(a) Pong
(b) Boxing
(c) Enduro
(d) Freeway
Figure 3: Learning curves of BC Atari Standard BC PeerBC (ours)
expert
5.2 PEERBC FROM WEAK DEMONSTRATIONS
In BC setting, we evaluate our approach four vision-based Atari games. For each envi-
Table 2: BC from weak demonstrations. Our approach success- fully recovers better policies than expert.
romment, we train an imperfect RL model with PPO (Schulman et al. 2017) algorithm. Here, "imperfect" means the training is terminated before convergence when the performance is about 70% 90% as good as the fully converged model. We then collect the imperfect demonstrations using the expert model and generate 100 trajectories for each environment. The results are reported under three random seeds.
Eavironment
Pong
Boxing
Enduro
Freeway
Lift (t)
Expert
15.1 1 6.6
67.5 1 8.5
150.1 1. 23.0
21.9 1 1.7
Standard BC
14.7 4.2
56.2 2 7.7
138.9 + 14.1
22.0 + 1.3
-6.6%
{
0.
22. 17..9229. 67
0.6
+11.3%
PecBB
{ 0.5
16.6
230.9 75.6
73.0
22.4 1.3
+19.5%
1.0
1..7
6..7+44.7
61.64
8.9
+2.0%
Fully converged PPO 20.9 0.3
89.3 5.1
389.6
216.9
33.3 = 0.8
+70.6%
Figure 3 presents the comparisons for standard BC and PeerBC, from which we observe that our approach outperforms standard BC and even the expert learns from. Note that during the whole training process, the agent never learns by interacting directly with the environment but only have access to the expert's 's trajectories. Therefore, we owe this performance gain to PeerBC's strong ability for learning from weak supervisions. The peer term we add not only provably eliminates the effects of noise but also extracts useful strategy from the demonstrations. In Table 2 we show the quantitative results. Our approach consistently outperforms the expert and standard
BC. As reference, also
compare two other baselines GAIL (Ho & Ermon, 2016) and SQIL (Reddy et al. 2019) and provide the sensitivity analysis of in Appendix D.
Analysis of benefits in PeerBC Similarly, the performance improvement of PeerBC might be also coupled with multiple possible factors. (1) The imperfect expert model might be a noisy version of the convered agent since there are less visited states on which the selected actions of the model contains noise. (2) The improvements might be brought up by biasing against high-entropy policies thus PeerBC is useful when the true policy itself is deterministic. We provide more discussions about the second factor in Appendix C.5
5.3 PEERCT FOR STANDARD POLICY CO-TRAINING
Finally, we verify the effectiveness of PeerCT algorithm in policy co-training setting (Song et al. 2019). This setting is more challenging since the states are partially observable and each agent needs to imitate another agent's 's behavior that is highly biased and imperfect. Note that we adopt the exact same setting as Song et al. (009)) without any synthetic noise included. This implies the potential of our approach to deal with natural noise in real-world applications. Following (Song al. (2019). we mask the first two dimensions respectively in the state vector to create two views for co-training in classic control games (Acrobot and CartPole). Similarly, the agent either removes all even index
8

---

Under review as a conference paper at ICLR 2021
200
300
-200
Nckodde 150
SpISCO rowand
-00
-200
15
400
120
500
2000
sproddo
05:5000
cstudd
(a) Acrobot
(b) CartPole
(c) Pong
(d) Breakout
Figure 4: Policy co-training on control/Atari. Single view
Song et al. 2019
PeerCT (ours)
coordinates (view-A) in the state vector or removing all odd index ones (view-B) on Atari games. As shown in Table and Figure PeerCT algorithm outperforms training from single view, and CoPiEr algorithm consistently on both control games (L 0.5 in Figure 4a 4b and Atari games (5 = 0.2 in
Figure 4c 4d.. In most cases, our approach leads to faster convergence and lower generalization error compared to CoPiEr, which again verify that our ways of leveraging information from peer agent enables recovery of useful knowledge from highly imperfect supervisions.
6 CONCLUSION
To deal with series of RL/BC problems with low-quality super- vision signals, we formulate meta-framework weakly super- vised policy learning to unify the problem instances with weak su- pervision in sequential decision- making. Inspired by the cor- related agreement (CA) mecha- nism, we propose a theoretical principled framework PeerPL that builds on evaluating a learning policy's 's correlated agreements with the weak supervisions. We demonstrate how our method adapts in RL/BC and the combined co-training tasks and provide intensive analyses of their convergence behaviors and sample complex- ity. Experiments on these tasks show our approach leads to substantial improvements over baseline methods.
Table 3: Comparison with single view training and CoPiEr (Song
al. 2019 on standard policy co-training.
Environment
Acrobot
CartPole
Pong
Breakout
Lift (f)
Single View
136.6 + 15.6
172.8 # 5.5
17.8 +18+000 0.6
148.0
16.5
34.7%
126.4444 1: 8.0
186.7 + 8.1
17.7 1: 0.5
137.8
12.5
35.5%
-13..2
174.1
16.8
1.7.5
52.9%
CoPiEr
-131.5
1713
1..5
82.7
72.0%
3.9
188.8 2.7
2.4
263.6
36.0
PearCT
-87.1 6.3
184.7 + 3.9
20.4
268.6
33.6
REFERENCES
Iige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's 's cube with robot hand. CoRR, abs/1910.07113. 2019.
John Asmuth, Michael L. Littman, and Robert Zinkov. Potential- shaping in model-based reinforcement learning. In AAAI, pp. 604-609, AAAI Press, 2008
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.03316, 2016a.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. CoRR, abs/1604.07316, 2016b.
Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreemeentreggulaiiied behavior cloning. In Eighth International Conference on Learning Representations (ICLR), April 2020.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E. Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In IJCAI, pp. 3352-3358, AAAI Press, 2015.
Felipe Codevilla, Matthias Miileer, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to- end driving via conditional behavior cloning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-9. IEEF, 2018.

---

Under review as a conference paper at ICLR 2021
Anirban Dasgupta and Arpita Ghosh. Crowdsoureed judgement elicitation with endogenous profi- ciency. Proceedings of the 22nd international conference on World Wide Web, pp. 319-330, 2013.
Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with corrupted reward channel. In IJCAI, pp. 4705-4713, 2017.
Wael Farag and Zakaria Saleh. Behavior cloning for autonomous driving using convolutional neural networks. 2018 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies, 3ICT 2018, 2018. doi: 10. 11100//CTT2211.885555555
Alessandro Giusti, Jerome Guzzi, Dan C. Ciresan, Fang Lin He, Juan P. Rodriguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jurgen Schmidhuber, Gianni Di Caro, Davide Scaramuzza, and Luca M. Gambardella. A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots. IEEE Robotics and Aulomation Letlers, 1(2):661-67,, 2016. ISSN 23773766. doi: 10. 20.11099RA.22115255000000
Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, and Murray Campbell. Hybrid reinforcement learning with expert state sequences. In AAAI, pp. 3739-3744 AAAI Press, 2019.
Todd Hester, Matej Vecerík, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI, pp. 3223-3230. AAAI Press, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial behavior cloning. In Advances in Neural Information Processing Systems, pp. 4572-4580, 2016.
Tommi S. Jaakkola, Michael I. Jordan, and Satinder P. Singh. Convergence of stochastic iterative dynamic programming algorithms. In NIPS, pp. 703-710, 1993.
Niels Justesen and Sebastian Risi. Learning macromanagement in starcraft from replays using deep learning. In 2017 IEEE Conference Computationall Inaa and Games (CIG), pp. 162-169. IEEE, 2017.
Sham Machandranath Kakade. On the Sample Complexity of Reinforcement Learning. PhD thesis, University of London, 2003.
Michael J. Kearns and Satinder P. Singh. Finite-sample convergence rates for q-learning and indirect algorithms. In NIPS, pp. 996-1002, 1998.
Michael Kearns and Satinder P. Singh. Bias-variance error bounds for temporal difference updates. In COLT, pp. 142 147, 2000.
Michael Kearns, Yishay Mansour, and Andrew Y. Ng. sparse sampling algorithm for near-optimal planning in large markov decision processes. In IJCAI, pp. 1324-1231. 1999.
Michael Laskey, Jonathan Lee, Roy Fox, Anca D. Dragan, and Ken Goldberg. DART: noise injection for robust behavior cloning, In CoRL, volume 78 of Proceedings of Machine Learning Research, pp. 143-156. PMLR, 2017.
Lisa Lee, Benjamin Eysenbach, Ruslan Salakhutdinov, Shixiang, Gu, and Chelsea Finn, Weakly- supervised reinforcement learning for controllable behavior, 2020.
Timothy Lillierap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1500.02971, 2015.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. ICML, abs/1910.03231. 2020.
Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via plass-pro estimation. In ICML, pp. 125-134, 2015.
10

---

Under review as a conference paper at ICLR 2021
Nolan Miller, Paul Resnick, and Richard Zeckhauser. Eliciting informative feedback: The peer- prediction method. Management Science, (1(99)13359-133 -1373, 2005.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Juarez Monteiro, Nathan Gavenski, Roger Granada, Felipe Meneguzzi, and Rodrigo Coelho Barros. Augmented behavioral cloning from observation. CoRR, abs/2200.13529, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K. Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1200 1204, 2013.
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application reward shaping. ICML, pp. 278-287. Morgan Kaufmann, 1999.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97 97.888897111 1991.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: Behavior Cloning via Reinforcement Learning with Sparse Rewards. 2019. URL http://arxiv org/ abt://bs//iei 1905. 11108
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: behavior cloning via reinforcement learning with sparse rewards. ICLR. OpenReview.net, 2020.
Joshua Romoff, Alexandre Piché, Peter Henderson, Vincent François-Laved and Joelle Pineau. Reward estimation for variance reduction in deep reinforcement learning. In ICLR (Workshop). OpenReview.net, 2018.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning, In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/ abss/100..00l 777.06337, 2017.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS, 2015.
Clayton Scott, Gilles Blanchard, Gregory Handy, Sara Pozzi, and Marek Flaska. Classification with asymmetric label noise: Consistency and maximal denoising. In COLT, pp. 489-511, 2013.
Victor Shnayder, Arpit Agarwal, Rafael M. Frongillo, and David C. Parkes. Informed truthfulness in multi-task peer prediction. In EC, pp. 179-196. ACM, 2016.
Jialin Song, Ravi Lanka, Yisong Yue, and Masahiro Ono. Co-training for policy learning. In UAL, pp. 441. AUAI Press, 2019.
Sainbayar Sukhhaatar and Rob Fergus. Learning from noisy labels with deep neural networks. arXiv preprint arXiv: /406.24400.0000 2080, 2(3):4, 2014.
Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour, Policy gradient methods for reinforcement learning with function approximation. In NIPS, pp. 1057-1063. The MIT Press, 1999.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. ICCAI, pp. 0950-4957, ijcai.org, 2018.
11

---

Under review as a conference paper at ICLR 2021
John N. Tsitsiklis. Asynchronous stochastic approximation and q-learning. Machine Learning, 16(3): 185-202, 1994.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI, pp. 2094-2100, 2016.
Brendan van Rooyen and Robert Williamson. Learning in the presence of corruption. arXiv preprint arXiv: (00xv::95000009),, 2015.
John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior (commemora- tive edition). Princeton university press, 2007.
Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In AAAI, 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In ICML, volume 48, pp. 1995- 2003, 2016.
Christopher J. C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pp. 279-292, 1992.
Yueh-Hua WU, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Behavior cloning from imperfect demonstration. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 6818 8-6827. PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2021
A ANALYSIS OF PEERRL
We start this section by providing the proof of the convergence of Q-Learning under peer reward r peer (Theorem D. Moreover, we give the sample complexity of phased value iteration (Theorem AT. In the rest of this section, show how to extend the proposed method to
setting (Sec-
tion A.3) and modern deep reinforcement learning (DRL) algorithms such as policy gradient (Sutton
al. 1999) and DQN (Mnih et al. 2013 van Hasselt al. 2016) (Section A.4).
A.1 CONVERGENCE
Recall that we consider the binary reward case {r.. 7- where r+ and r- are two reward levels. The flipping errors of the reward are defined c P(A -- = r+) and PP r+1r= r-)- The peer reward is defined as T poer (s, a) = r(s, a) - r', where is randomly sampled reward over all state-action pair (s, a). Note that we treat each (s, a) equally when sampling the r' due lack of the knowledge of true transition probability P. In practice, the agent could only noisy observation of peer reward F peer (s. (s.a) a) F(s.a) F². We provide the Q-learning with peer reward in Algorithm Al
Algorithm A1 Q-Learning with Peer Reward
Require: M= A,
learning rate (0, 1), initial state distribution Su. A + IR arbitrarily
I: Initialize Q:
2: while Q is not converged do
:3 4: while is not terminal do
3: Start in state
Calculate и according to Q and exploration strategy
6: a <- T()); 8' ~ ((.
7:
Observe noisy reward F(s, a) and randomly sample another F' from all statc-action pairs Calculate peer reward Fpeer (s, F(s, a) -
8:
Q(s.a) a) (1 a)) ((s,a) + (Fpeer (8, + max, ,(s',a')
10:
8
11:
end while
12:
end while
13:
return as) and (s)
We then show the proposed peer reward fpeer offers affine transformation of true reward in expectation, which the key to guaranteeing the convergence for RL algorithms.
Lemma 1. Let E [0, Rmax] be bounded reward and assume 1 - e_ e. > O. Then, if we define the peer reward F peer (s, a) -iss, a) - F', in which the penalty term 5¹ is randomly sampled noisy reward over all state-action pair (s, a). we have
Eli peer]
(1 c_ = e1-e -
const,
where peer the clean version of peer reward when observing the true reward.
Proof. With slight notation abuse, we let Ppeers r, F, r' also represent the random variables. Let n denotes the RL agent's 's policy. Consider the two terms the RHS of noisy peer reward separately,
P(r r+ (T) Eyer, P(F r- r--r=- = 14) P(j r+Ir = r4) r4]
(7)
P(r = (T) Erer, -Pf=rr-r=r_..r- P(-rr-rr-rrrrrr.------
(r r- (T) Eyer
(8)
(9)
+ IP(r -_|T)) Eper_ [1 - ---re-r--ee +
(10)
P(r = = +++/)) Byery (1 e -e_) . + +e4-_ +erm
(11)
+ IP(r = r-|m) E Eper_ [[1 - c_ -4) r- +err+ +err-]
(12)
(1 6+ - e_ Errr e-rt + eer--
(13)
Since we are treating the visitation probability of all state- action pair (s, a) equally while sampling the peer penalty r', then the probability of true reward under this sampling policy saample is
13

---

Under review as a conference paper at ICLR 2021
constant, denoting Ppeer: i.c. Ppeer P(r [sample) is constant. Then we have,
E P sample
P( r+1 [sample) r+
(14)
= (e+Ppeer + (1 e-) ) (1 (peer)) r- + ((1 e+ Ppper + c- Ppecr) nr
(15)
(1 - ++1(1 Ppeer) Ppeer r+] + e47- + c--++
(16)
As consequence, we obtain the expectation of peer reward satisfies
peer [[ [F'
(17)
= (1 e+ - e-) (1- - -+)](( - Ppeer) - + Ppeer r+1
(18)
(1 c_ - e+ e+) + const.
(19)
Similarly, it is easy to obtain that perr E(r] - [(1 Ppoer) T- + Pprer r4.. Therefore, we have Bly peer] (1 e e.) porrp = (1 e- e. ][(r] const.
Lemma shows the proposed peer reward peer offers us a "noise-free" positive (1 - 6. > 0) linear transformation of true reward in expectation, which is shown the key to govern the convergence. It is widely known in utility theory and reward shaping literature (Ng et al. 1999 Asmuth al. 2008 Von Neumann Morgenstern, 2007) that any positive linear transformations leave the optimal policy unchanged. As a consequence, we consider "transformed MDP" M with reward f = e e+) )r + const, where the const is the same the constant in Eqn.
19
In what follows, we provide the formulation of the concept of "transformed MDP" with the policy invariance guarantee.
Lemma A1. Given finite MDP M (S,A, P,7)) transformed MDP M = AS R. P with positive linear transformation in reward == a v + b, where u, are constants and a >> 0, is guaranteed consistency in optimal policy.
Proof. The Q function for transformed MDP M (denoting Q) is given follows:
((s
la
+b)
t=0
t=
Nr b4b
1=0
1=0
= =a - a) + B,
where - constant. Therefore, there is only a postive linear shift (a > 0) in ((s, thus resulting in invariance in optimal policy for transformed MDP:
# (8) arg max (s, a) = arg max ja Q(8 aEA
aeA
= arg max (s, (s). aEA
Lemma A1 states that we only need to analysis the convergence of learned policy π (ss) to the optimal policy **)s) for transformed MDP M, which is equivalent to the optimal policy (s)) * for original MDP. This result is relevant to potential-basee reward shaping (gg al. 1999 Asmuth al. 2008)
where specific class of state-dependent transformation is adopted to speed up the convergence speed of Q-Learning meanwhile maintaining the optimal policy invariance. Moreover, degenerate case for single-step decisions is studied in utility theory Von Neumann & Morgenstern 2007) which also implies our result.
Finally, we need an auxiliary result (Lemma A2 from stochastic process approximation to analyse the convergence for Q-Learning.
Lemma A2. The random process (Ai) taking values in R and defined as
A+t1(x)= (1 (x())) (z())
converges zero I under the following assumptions:
14

---

Under review as a conference paper at ICLR 2021
0s E,a(t)) and E, (z) oo;
[B(Fi())) l|w S <tlldell with 7 < 1;
(ar(P((=)| S C(1 + || lliv) for C 00 0.
Here Fi = (A..AAAA depend on the past insofar as the above conditions remain valid. The notation l/w refers to some weighted maximum norm.
Fi-1 az ar..... ... stands for the past at step t, az(x) is allowed to
Proof of Lemma A2 See previous literature Jaakkola et al. 1993 Tsitsikis 1994).
Theorem 1. (Convergence) Given finite MDP with noisy reward, denoting as M (S, A, R, F, r),,)), the Q-learning algorithm with peer rewards, given by the update rule,
Qu+1(Se, ar) = - aaee))@t Qr) + ae ]poer(St, ar) + +mxax (:(8)11,6)
(20)
BEa
(s)
arg max (s)
(21)
aEA
converges w.p.l the optimal policy n" (s) as long En Cr 0o and E, aî < 00.
Proof. Firstly, we construct a surrogate MDP M with the positive-linearly transformed reward f = (1 e_ 6+) const, where const -(1 e e+)((1 p) r + p r+) is constant. From Lemma A1 we know the optimal policy for M is precisely the optimal policy for M: 4" (s) = π (s).
Let Q* denotes the optimal state-action function for this transformed MDP M. For notation brevity, we abbreviate 5c. 8141, (st, St41), Qs. Q++1. and a as 5, s', Q, Q', Fpeer and a, respectively.
Subtracting from both sides the quantity (s, a) in Eqn. 21
('(s,a) Q- (s) (s,, - aa) (Q(s,a)-Q*(6
Fpeor + max Q(s' -
Let (s) = ((s,a) (s) and Fi a) ipeer + maxbed Q(s',b) - as)
A4t1(s' = (1 (FF(ss,)
In consequence,
E a)) = EE
peer + y max Q(s' (s,
E
poer
+ 7 Q(6' (() - max (' b)
bea
bEd
E [Pperr]
E [r] +
max BEA
Q(s' b)
max (5(
bEA
=E
max Q(s max Q- (s',b))
Q(s(,b) b) Q' (s', b)|
S B
max
hEA.'ES
B
||Q Q01100 =11QQ Q0 ?1/oo
15

---

Under review as a conference paper at ICLR 2021
In above derivations, utilize the unbiasedness property for peer reward (Lemma and the inequality maxbEA Q b) maxsE (s', & maxbeA.*Es Q(s', - (s', b)
i poer + y max Q('.b) (x, E |peer + 7 max Q(s'. b) -((.a))
Var [F((s.
R
PaA
DEA
E
peer + max ((s', b) - Ppeer + max beA
max Q(s',b) BEA
Var
Ppeer
+ y max Q b)
BEA
Since Peer is bounded, it can be clearly verified that
Var [P,(s, a) S
())
for some constant C" > 0. Then, A, converges to zero w.p.1 from Lemma A2 i.e., Q a) converges to Q* a). As a consequence, we know the policy (s(s) converges to the optimal policy A (s) n" (s).
A.2 SAMPLE COMPLEXITY
In this section, we establish the sample complexity for Q-Learning with peer reward as discussed in Sec 4.2 Since the transition probability P in MDP remains unknown in practice, firstly introduce a practical sampling model G(M) following previous literature (Kearns & Singh 1998 2000 Kearns al. 1999). in which the transition can be observed by calling the generative model. Then the sample complexity is analogous to the number of calls for G(M) to obtain near optimal policy. Definition A1. A generative model G(M) for an MDP M is sampling model which takes state-action pair (&t, at) as imput, and oulputs the corresponding reward r(8t, ax) and the next state 8141 randomly with the probability of P. (St, 8141), i.e., 81+1 P()ss.a).
is known that exact value iteration not feasible when the agent interacts with generative model G(M) (Wang al. 2020; Kakade 2003). For the convenience of analysing sample complexity, introduce phased value iteration following Wang et al. (222)); Kearns & Singh (1998): Kakade (003))
Algorithm A2 Phased Value Iteration
Require: G(M)): generative model of M
number iterations.
1: Set Vs(s)= =
2: for = - ...... do
3: Calling G(M) m times for each state-action pair. # #[(ss, ar) + 8441
Pa(st, S1+1)=
m
4: Set
V(s)) = max A
Σ
Pa S(41) In + (5
S
x(s) arg max V(st) aEd
5: end for
6: return (ss) and (s))
Note that Pa (0t, 11+1) the estimation of transition probability Pa(se, 8t+1) by calling G(M) m times. For the simplicity of notations, the iteration index decreases from T I to O.
could also adopt peer reward phased value iteration by replacing Line Algorithm A2 by
(s+))
max Σ A
Pa St41) (epeer($t, a) (
St+1ES
16

---

Under review as a conference paper at ICLR 2021
Then the sample complexity of one variant (phased value iteration) of Q Q-Learing is given follows: Theorem A1. (Sample Complexity) Let r [0, Rmax] be bounded reward, for an appropriate choice of m, the phased value iteration algorithm with peer reward i ppeer calls the generative model G(M)
(SIAAV times in T epochs, and returns a policy such that for all state E S, O 211-e--44) log
ISILATT
- -=() S w.p. 21 - 6, 0<000 <
1, where - e-
0 is constant.
Proof. Similar to Theorem we firstly construct a transformed MDP M and the optimal policies for these two MDP are equivalent (Lemma ATI. As result, we could analyse the sample complexity of phased value iteration under M.
It is easy to obtain that peer E [0, Rmax] and (*(s) E 0. Bmax ing's inequality, we have
are also bounded. Using Hoeffd-
1-y
Pr (T) [Vti(oer1]]
-2 -
Σ (41))
26 2eep
R2 R2
Pr
B Bl -
Σ 8
Pal 8e4
26
S
2xx
2mme
Rymax
Then the difference between learned value function V (8)) and optimal value function (*())) under transformed MDP at iteration f. is given:
(s) vi(s) = max .EA
[r + VV (S++1)]
VV
max aEA
Σ IES
Palse, S141) pseer a) + +V241($6+1)
max cEA
EE [re]
Σ
(a 8141) Ffer)
SI+EES
+7 max aEA
E
V Viti (se+1)
Σ
Pa Set (1+114)44)))))
BL11ES
$ max
E [r,] - E F Ppoorl
YC2 +
E Vir (ita(seti) (V
Σ y max
1 (s) - ((+
+ +
#ES
Recursing above equation, get
(*((s) v(0) s (a1 + yE2) + 7( r62) +... --... + 762)
max sES
(61 + (ea) (T)
1 1-7
In other words, for arbitrarily small e, by choosing m appropriately, there always exists and such that the value function error is bounded within 6. As a consequence the phased value iteration algorithm can converge to the near optimal policy within finite steps using peer reward.
Let E1 62
(1-2)c
(1+%) then max,s
Note that there are in total [S| transitions under which these conditions must hold, where | represent the number of elements in a specific set. Using union bound, the probability of failure in any condition is smaller than
e1 -11-) (1 (1-7)2
|| exp (-m-
(1 11 71+
R2
17

---

Under review as a conference paper at ICLR 2021
We set above failure probability less than 6, and m should satisfy that
m o ( log
1
S|AT
5
In consequence, after m|S]|A[[ calls, which is, O (
(ISIAAT
log |S||A
the value function con-
verges to the optimal value function (s) for every s in transformed MDP M, with probability greater than - 6.
we know (s) (1 e e+) V*(s) + C, where C' is constant. Let é = 61-6- and V(s) = (1 - e_ - e+) (s)) (s) + C, we have
From Lemma A1
(s) -
V(s) C
IV*(s) - V()]
(22)
(1-e- -e+)
(1 -1-63 e- -+)
(V((s) - V(s) SE
(23)
e_ c+)
This indicates that when the algorithm converges to the optimal value function for transformed MDP M, also finds a underlying value function (s) (s) that converges the optimal value function (s) for original MDP M.
to achieve an e' error As a consequence, we know needs to call (I---_--_) log 5
SSILAT
|S|LT
in value function for original MDP M, which is no more than O needed when the RL agent observes true rewards perfectly. When the noise in high-regime, the algorithm suffers from large (1-e--ey' thus less efficient. Moreover, the sample complexity of phased value iteration with peer reward is equivalent to the one with surrogate reward in Wang et al. (2200) though sampling peer reward less expensive and does not rely on any knowledge of noise rates.
times the one
(T- (1-c--e4
A.3 MULTI-OUTCOME EXTENSION
In this section, we show our peer reward generalizable to multi-class setting. Recall that in Section suppose the reward is discrete and has [R] levels, and the noise rates are characterized as [ Ri Here we make further assumptions on the confusion matrix: the reward is misreported to each level with specific probability, e.g.,
3.2
LtL
EiS1
Cs.
C22 Eieo
G|
Ci
6|
RI. X|R|
(24)
:
61
22
1
Σ E 64
Following the notations in A.1 we detine the peer reward in multi-outcome settings as r(s, a) F(s,a) a) r', where r' is randomly sampled following specific sample policy Tsample over all state-action pairs. Let Rpeer, R. R. and R' denote the random variables corresponding to F Fpeers F,
18

---

Under review as a conference paper at ICLR 2021
r', Gy represents the entry of C
Then we have
(R
R
R
E
P (R-
R
Cit Rg
1=1
1=1
R
PP (R R
1 - Σ
R; +
Σ eg
i=1
[R]
R
[
Σ
IP (R = Ri(TT)
-
R3 +
R3
1=1
1=1
j=1
R
[
Ej
Em
[R]
+
eg Rij
==1
==1
and
E Eapplle
R'
R3
P
R
Rs saaple
R
R
R2
P (R-
R, ssamlle) Cij
i=1
Σ
R2
Σ
PP (R = Rg saample) eg PR = Rg Tsample)
Σ
E4
ig
[R
[
Σ
R2
Σ
TP (RR RS sammple) Bj + PPR
Ry ssample)
Σ
/=1
1=1
[R]
R Σ j=1
E/
Eitaample
[
ej
ej.y.
Then, the peer reward is formulated
E
R
peer
E
R
R
R
R
[
1
-
ej
FT [R]
E Esaampo
R
j=1
[R|
Ej
E, [R]
+ const.
j=1
19

---

Under review as a conference paper at ICLR 2021
A.4 EXTENSION MODERN DRL ALGORITHMS
In this section, we give the following deep reinforcement learning algorithms combined with our peer reward in Algorithm A3 and A4 In Algorithm A3
give the peer reward aided robust corresponds to the loss function [((s, q). loog (a)ss,) which classification calibrated Liu & Guo 2020). So the expectation of the gradient in 25 is an unbiased esitmation of the policy gradient in corresponding clean MDP. In (A4), we present robust DQN algorithm with peer sampling, which the origin loss is {((s, a). 9). also classification calibrated. Thus the robustness can be proved via Liu & Guo (2020).
policy gradient algorithm, where the gradient in Equation 25
Algorithm A3 Policy Gradient (Sutton et al. [999) with Peer Reward
Require: M (S. A,
R, P.7). 7), learning I: Initialize zg: S x A - IR arbitrarily
(0, 1), initial state distribution So. weight parameter ç.
2: for episode = 1 to M do
Collect trajectory To {(si, ai, F:)) where so Bo, ae mo(-|ss), S1+1 P(-Ise,ae)
4: Compute q; Σ -i " for all E (0.1....,
For each index i (0, 1........ ..). we independently sample another two different indices j. k.
6: and update policy parameter 0 tollowing
0 + aa Vo log (o(a.ls)) - -a Vo lo
(25)
7: end for
8: return
Algorithm A4 Deep Q-Network (Mnih al. (0133) with Peer Reward
Require: .M- (S, R, P, r.). learning rate (0, 1), initial state distribution fo. weight parameter 5.
1: Initialize replay memory D to capacity N
Initialize action-value function Q with random weights
3: for episode = to M do
4: for = 1 to 7' do
5: With probability select random action a., otherwise select a, max. (s,
Execute action de and observe reward Pt and observation Sc+1 Store transition (Se, de, fe: #t+1) 1 8: Sample three random minibatches of transitions (8i, ai Pr, 81t)), (8), (s3, 8411))
(sk, Gle Fh, 8 from D.
for terminals; Sct F, y max. a's+1, for non-terminal 81+1 Fe
9:
for terminals; Fe y max,' Q(s++1,a') for non non-terminal Sj+1
10: Set Speer
11: Perform a gradient descent step on (ji Q(s,,a))) - E (Upeer - Q(),,
12: end for
13: end for
14:
return
20

---

Under review as a conference paper at ICLR 2021
B ANALYSIS OF PEERBC
We prove that the policy learned by PeerBC converges to the expert policy when observing a sufficient amount of weak demonstrations in Theorem A2
Theorem A2. With probabiliy at least 6, the error rate is upper-hounded by
1+
2log
2
RD - e - e.
(26)
N
where is the number of state-action pairs demonstrated by the expert.
Proof. Recall Dr denotes the joint distribution of imperfect expert' state-action pair (s, a). Assume there perfect expert and the corresponding state-action pairs (s,a) DE. The indicator classification loss (=)(s), a) is specified here for clean presentation, where (s) a) = when
(s) o a, otherwise (s)(s), a) 0. Let Dr == {(5:, a;) ) be the set of imperfect demonstrations, and DE == {(s;, a.) a.)] be the set of weak demonstrations. Define:
RD (T) = (s aI), R5 (x) B( [1(T()),
(T) = N
1
Σ 1(T(s)), az), Dz (T) = N [E[N]
Σ (TT(s))..)) RE[N]
Note we focus on the analyses of loss in this proof. The negative of loss can be seen reward. Denote by T DE and DD be the optimal policy obtained with minimizing the indicator loss with dataset g and distribution Dg. We shorten Tig as #", which the best policy we can learn from imperfect demonstration with our algorithm. Let nt be the policy for the perfect expert. We would like to see the performance gap of policy learning between imperfect demonstrations and perfect demonstrations, i.c. 1 - 8, we have
(T*). Using Hoeffding's inequality with probability least
log 2/6
i& (T) -R (T) 5 (1+ + 5)
2N
Note we also have
Rs (5*) R (TT))
< <RB (#*) - R (TT)g) + RØg (#*) AB (f*)
RDE (T - -R (T
<0 + 2 max Rin (T) - Rb, (T)
2 2log 2/8
$(1 + E)
N
Before proceeding, we need define constant to show the affect of label noise. When the dimension
of action space is 2, the problem is essentially a binary classification with noisy labels Liu & Guo 2020), where the noise rate (a.k.a confusion matrix) is defined e4 (*) A_ n (8) A+) and e__ = (( = A_ x (s) = A-). Recall the action space is defined as .A = [A4.A_} }. The noise constant is denoted by e e-1 + e+1- Accordingly, when the dimension of action space is
[R] > 2, we can also get similar results under uniform noise where
: (f = u/ (l)="(s) = u'), u' +
(27)
The noise constant is denoted by > | the properties of peer loss functions Liu & Guo 2020 can be used, i.e.
== eu- The feature- indeeee-nddeendeen assumption holds thus
RØ
(7')
Rog (m*)
1
R5 (7*) )-
-R (TT
1 +
2
log 2/5
N
21

---

Under review as a conference paper at ICLR 2021
From definition and deterministic assumption for n², we have RD, (x*) 0. Thus the error rate in the k-th iteration is
+ 1
log 2/0
RD (+)) S Rog (T
N
(28)
1+5
2 2log 2/5
1-6
N
Note RD, (f*) = RDg by definition.
22

---

Under review as a conference paper at ICLR 2021
C SUPPLEMENTARY EXPERIMENTS
C.I EXPERIMENTAL SETUP
We set up our experiments within the popular OpenAI stable-base sabble-baselinee and keras-ril framework. Specifically, three popular RL algorithms including Deep-Q-Netwon (DQN) Mnih at. 2013 van Hasselt et al. 2016), Dueling-DQN (DDQN) (Wang al. 2016) and Proximal Policy Optimization Algorithms (PPO) are evaluated in a varied of OpenAl Gym environments including classic control games (CartPole, Acrobot) and vision-based Atari-2600 games (Breakout, Boxing, Enduro, Freeway, Pong).
C.2 IMPLEMENTATION DETAILS
RL with noisy reward Following Wang et al. 2020), we consider the binary reward {-1, for Cartpole where the symmetric noise synthesized with different error rates e__ 64. We adopted five-layer fully connected network and the Adam optimizer. The model is trained for 10,000 steps with the learning rate of le and the Boltzmann exploration strategy. The update rate of target model and the memory size are 1e-2 and 50,000. The performance is reported under 10 independent trials with different random seeds.
BC with weak expert We train the imperfect expert the framework stable-basel with default network architecture for Atari and hyper-parameters from r sll-baelinnes-21 The expert model is trained for 1, 400, 000 steps for Pong and 000, 000 steps for Boxing, Enduro and Freeway. For each of those environment, We use the trained model to generate 100 trajectories, and behavior cloning performed on these trajectories. We adopt cross entropy loss for behavior cloning and add a small constant (1 x 10 ()=8) for each logit after the softmax operation for peer term to avoid this term become too large. In BC experiments, the batchsize is 128, learning rate is 10- and the e value for Adam optimizer is 1 x 10-8
Policy co-training For the experiments on Gym (CartPole and Acrobot), we mask the first co- ordinate in the state vector for one view and the second for the other, same as Song et al. (019). Both policies are trained with PPO(Schulman et al.) 2017) + PeerBC. In each iteration, we sample 128 steps from each of the parallel environments. These samples are fed to PPO training with a batchsize 256, a learning rate of 2.5 x 10 and a clip range of 0.1. Both learning rate and clip range decay throughout time. We represent the policy by fully connected network with hidden layers, each has 128 units.
For the experiments Atari (Pong and Breakout), the input is game images. We adopt the preprocess introduced in Mnih et al. (201)) and mask the pixels in odd columns for one view and even columns for the other. The policy
use adopts default CNN in stable wwwaabbbbaeeeeeee baselines. Batchsize, learning rate, clip range and other hyper-parametert are the same as Gym experiments. Note that we only add PeerBC after 1000 episodes.
ppo2.ym1# yml
23

---

Under review as a conference paper at ICLR 2021
C.3
SUPPLEMENTARY RESULTS FOR FIGURE 2 AND TABLE
staps per aproode
700
150
pmr eptocde
150
00000 par spiaods
150
DQN
100
100
120
202
200
200
DDQN
0015 per spoode
100
150
03010 por optrdds
120
100
9:0
oprooce
aplocce
aprocds
apoodd
(a) c
(b) 0.2
(c) e 0.3
(d)
0.4
Figure A1: Learning curves on CartPole game with true reward (r) noisy reward (f) surrogate
reward (Wang et al. 2020) (f) and peer reward (Fpsor. 5 = 0.2) Each experiment is repeated 10 times with different random seeds.
Table A1: Numerical performance DDQN on CartPole with true reward (r), noisy reward (F), surrogate reward è (Wang et al. 2020), and peer reward peeer (5 0.2). Raay denotes average reward per episode after convergence, (last five episodes) the higher (1) the better; Neps denotes total episodes involved in 10,000 steps, the lower (4) the better.
0.1
0.2
0.3
0.4
Rang
Nest
Rac
Nasc
Rong
Rong 7
Nep
183.6
101.3
184.0
184.0
101.5+
DQN
189.3 + 12.7
188.3 8.2
101.1+
6.5
192.7 : 9.2
189.7 +: 7.9
110.5 7.1
97.9 6.4
185.4
183.2:19..8
116.9 11.0
130.5 7.7
184.8 + 16.4
169.7 = 18.6
150.2 11.4
123.1 8.6
Fpecr
91.2
195.6 3.1
101.24332 3.2
106.8 + 3.1
101.2
195.6
101.2 3.2
195.2 3.0
101.24 3.3
185.2 = 15.6
114.6 3:6:::00 6.0
168.8 2.8.8:11 13.6
123.9 9.9
177.1 11.2
100.2 9.1
186.5 10.9
163.1 11.0
DDQN
110.6 6.7
163.1 +
192.2
4.3
170.2 6.6
peee
198.5 1:
3:
195.5 d: 9.1
174.1 32.5
C.4 ADDITIONAL RESULTS FOR PEERRL ON PENDULUM
To further evaluate the effectiveness of PeerRL,, we conduct experiments on more challenging continuous control task Pendulum, where the goal is to keep frictionless pendulum standing up. Since the rewards in pendulum are continuous: r (-16.3, 0.0), we discretized into intervals: (-17,-16j, (-16, -15) ... (-1,0) 0), with its value approximated using its maximum point. We experiment the DDPG Lillicrap et all 2015) and uniform noise this environment. Figure A3 the RL agents with proposed CA objective successfully converge to the optimal policy under different amounts of noise. On the contrary, the agents with noisy rewards suffer from the biased noise especially in high-noise regime.
C.5 SENSITIVITY ANALYSIS OF PEER PENALTY E
In this section, we analyze the sensitivity of & in RL and BC tasks. Note that we did not tune this hyperparameter extensively in all the experiments presented above since we found our method works robustly in wide range of g.
RL with noisy reward We repeat the experiment in Figure A1 for DQN but with a varying from 0.1 to 0.4. As shown in Figure A2 method works reasonably and leads to faster convergence compared to baselines. However, we found that the late stage of training, a small & is necessary since the agent already gains useful knowledge and make reasonable actions, therefore, over-large
24

---

Under review as a conference paper at ICLR 2021
800
0.1
190
stago por 6010030
152
stapa per sploods
100
100
120
200
200
0.2
100
S400 par sbtoodd
120
skopp par eptedds
150
3 =
820
eptsode
cploode
spisade
200
200
200
0.3
8000 948 aplocds
150
stopa poo aplesde
120
900
50
80
25
100
eptsode
oploode
spioce
200
200
0.4
00035 ge colendo
sptoods
100
120
100
100
skape por
50
10
100
25
50
100
0
50
100
100
150
ipiode
opisoda
aptende
(a) 0.1
(b) 0.2
0.3
(d) 0.4
Figure A2: Learning curves of DQN on CartPole game with peer reward (f (eperr under different choices of & (from 0.1 to 0.4).
DDPG roward pot oplocde
Isward por uplocde
ronard
200
400 upéncdd
200
400 spisnds
upisode
ugisssi
(a) e
(b)
(c) e
(d) 0.4
Figure A3: Learning curves of DDPG Lilliccrap et al. 2015 Pendulum with reward
noisy reward (7) and peer reward (Fpser
25

---

Under review as a conference paper at ICLR 2021
penalty might avoid the agent achieving simple agreements with the supervision signals, especially in a low- low-noise regime (see Ç = 0.4, e = 0.1). This observation inspires us that a decay schedule of might be helpful in stabilizing the training of PeerRL algorithms. To verify this hypothesis, we repeat the above experiments but with a linear decay & that decreases from 0.4 to 0.1. In Figure (44 we found the linear decay schedule is able stabilize the convergence of PeerRI. algorithms compared to static { = 0.4. The theoretical principles and insights of dynamic peer penalty merit further study,
200
DQN
skeps pet spicods
150
seips per eptocds
150
skeps por 12tlood
100
100
100
120
80
80
30
75
0
50
75
100
50
120
100
190
<piaac
4g 4psoda
aprende
(a) 0.1
(b) 0.2
(c) 0.3
(d) 0.4
(009 per optcod
opocdd
0009 per opo200
00095 pe opoods
DDQN
190
100
800
100
800
80
100
50
150
(e) c 0.1
(f) 0.2
(g) 0.3
(h) 0.4
Figure A4: Learning curves of DQN on CartPole game with peer rewards (Fpeer) decay is applied during training procedure (initial ę 0.4). Compared to static { 0.4, the linear decay peer penalty stabilizes the convergence of RL algorithms.
Here, a linear
BC from weak demonstrations We conduct experiments Pong with 12 different & values. varying from 0.1 to 1.2. As a reference, two other BC baselines GAIL (Ho & Ermon 2016) and SQIL (Reddy et al. 2019) are considered. We do not include GAIL in the figure, since GAIL fails to produce meaningful ful results on vision-based Atari games as also observed (Reddy et al. 2019
Brantley a 2220). From Figure AS can PeerBC outperforms pure behavior cloning and SQIL(Reddy et al. 2019) when is within [0.1, 0.7]. revealing our proposed PeerBC a superior behavior cloning approach able to better elicit information from imperfect demonstrations.
C.6 VARIANCE REDUCTION IN NOISY REWARD SETTING
In practice, there is trade-off question between bias and variance that influence the RL training. Even though variance reduction approaches theoretically do not resolve the challenge when bias presents in the observed rewards, might be beneficial to the training procedure in practice. To investigate whether the variance reduction techniques are helpful in the setting with biased noise, we repeated the experiments in Table D We adopt the variance reduction technique (VRT) proposed in Romott et al. (2018) and found it brings little benefits to noisy reward and surrogate reward as shown in Figure A6 However, peer + VRT performs worse compared to peer reward only. This demonstrates the benefits of scenarios does not mainly come from the variance reduction in our setting. The full quantitative results are presented in Table A2
C.7 STOCHASTIC POLICY FOR BEHAVIORAL CLONING
In this section, we analyze the stochasticity of the imperfect expert model and fully-converged PPO agent (assumed to be the clean expert), and show that our PeerBC can handle both cases when the clean expert is stochastic and when it's rather deterministic.
26

---

Under review as a conference paper at ICLR 2021
000000
baxhes (x100)
bakkhes (1100)
bbakhes (:100)
bbakhes (1100)
(a) f = 0.1
(b) ç 0.2
(c) g = 0.3
(=) 0.4
096006 ROWOR)
10
Sphodi lowad
10
0W600c RONUR
10
5phook
-10
10
-20
-20
100
200
200
103
200
900
200
buxihras (000
baskhhes
bakkhhe (o100)
baiches (1100)
(e) { = 0.5
(l) E = 0.6
(g) E = 0.7
(h) E 0.8
20
20
110
-1
-20
-20
100
900
baakhes
Nbakihes
(i) 0.9
( = 1.0
(k) = 1.1
(l) E = 1.2
Figure A5: Sensitivity analysis of & for PeerBC on Pong with behavior cloning varies from 0.2 to 0.5 and 1.0), expert E, and SQII. reported by SQII. (Reddy et al. 2019). Each experiment is repeated under 3 different random seeds.
PeerBC (
200
DDQN
cheps per ophoode
100
150
60095 per eptoode
+50
150
100
300
900
opio00e
opticde
eptode
(a) e
(b)
0.2
(c) e 0.3 (d) 0.4
Figure A6: Learning curves of DDQN on CartPole with true reward (r) noisy reward (?) noisy
reward + VRT surrogate reward Wang et al. 2020 reward (Fpeer. E 0.2)
(f)
surrogate reward + VAT and peer
Table A2: Numerical performance of DDQN on CartPole with true reward (r), noisy reward (7), surrogate reward [Wang et al. 2020), and peer reward (p = 0.2). + VRT denotes the variance reduction techniques Romoff et al. (2018) are adopted.
0.1
0.2
0.3
0.1
Rasg
Nepi
Rong
Nops
Rang
Nyji
Rang
Nyp
195.6 =: 3.1
101.2 4 3.2
195.6 4 3.1
3:322
195.0 3.1
101.2
195.2 4 3.0
101.2 3.3
DDQN
185.2
15.6
114.6
6.0
168.8
13.6
123.9
177.1
133.2
185.5 10.9
+ 11.0
+VRT
190.2 1: 14.6
111.8 4.5
183.6 1 15.7
118.0
1.001.000 9.5
179.6 1: 19.9
129.6 13.9
162.6 11 13.4
157.2 19.0
183.9
10:4
110.6
165.1 +
1.3.9.99.
19.
4.3
179.2 6.6
125.8 9.6
+ VRT
179.0 + 13.5
112.1 + 5.4
163.4 + 17.0
112.8 0.1
190.1
10.3
112.6 + 3.9
179.8 + 15.1
119.7 + 9.1
Fpoer
198.5 +
96.2
195.5 +
85.3
174.1 32.5
88.8 6.3
191.8 + 8.5
9.2
+ VRT
184.1 =
96.2
193.9 = 6.0
86.5 +
174.2 21.3
93.3 6.2
181.2 15.3
117.7 9.5
We plot the entropy of the PPO agent during training on four environments from the BC task in
Figure A7 and we give the entropy value of the imperfect expert model and the optimal policy in We observe that except for Freeway, the entropy of expert policies is always larger than 1. We calculate the mean value of the highest action probability over 1000 steps for the full-converged PPO agents in Table A4 which again verifies that the true expert policy aim recover might not
Table A3
27

---

Under review as a conference paper at ICLR 2021
Ricking
16
62
24
80
Timedhin
34
05
M
12
80
02
03
12
89
02
Thredone
08
30
12.
(a) Pong
(b) Boxing
(c) Enduro
(d) Freeway
Figure A7: The policy entropy of the PPO agent during training. The imperfect expert model is trained for 0.2 x 10" timesteps as the red line indicates.
be fully deterministic. These results demonstrate the ficxibility of our proposed approach in dealing with both stochastic and deterministic clean expert policies in practice, although a deterministic clean expert policy is assumed in our theoretical analysis.
Also, from Figure A7 and Table we notice that the entropy of imperfect expert models are higher than the fully converged PPO agents, implying that the expert models might contain an amount of noise. That's because there might be states which the expert has not seen enough and the selected actions contain much noise. This is consistent with our claim, that the benefits of PeerBC might come from two aspects, both noise reduction of the imperfect expert and inducing a more deterministic policy.
Timesteps (x107)
Pong
Boxing
Enduro
Freeway
0.2 (Imperfect Expert)
1,201 1.250
1.949
1.637
0.318
1.0 (Fully converged PPO)
1.168
1.126
0.171
Table A3: The policy entropy of the PPO agent during training.
Trained timesteps (x107) Pong
Boxing
Enduro
Freeway
1.0 (Fully converged PPO)
0.492
0.579
0.664
0.903
Table A4: The mean value of the highest action probability over 1000 steps.
28


Reviews:
[{"review_id": "wSluC9D1CJK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Sumary\n\nThis paper tackles a very important problem of reinforcement learning/imitation learning in the presence of noisy rewards/labels and uses contemporary literature to motivate a simple solution (in a good sense). The empirical results look encouraging on the benchmark problems but the analysis is a little wanting and doesn't get at the heart of the matter. I think with a little more work this paper can be a very good conference paper but at this stage I can't recommend publication in ICLR. \n\n**Before Author Response** Of course, if the authors address my concerns then I'll increase my rating. \n**After Author Response** I think the authors made a good effort to address the concerns and I have recommended to accept the paper.\n\n## Contributions \n\nThis paper tackles the problem of learning from imperfect reward/imperfect demonstrations using a new evaluation metric called \"Correlated Agreement\". The proposed metric \"regularizes\" learning under weak supervision by penalizing \"over-agreement\" with the supervision signal. The proposed method is meant to be used in situations where the supervision signal is known to be noisy but it does not estimate parameters of the noisy channel   which is corrupting the supervision signal.\n\n## Strengths\n\nThe strongest aspect of the paper in my opinion are the strong empirical results on fairly standard RL benchmarks. The paper also explains the concept of learning from \"peer agents\". The analysis of the methods presented covers the most obvious questions one might ask at the beginning about the proposed method and the appendices cover those well.\n\n## Weaknesses\n\nThere are two problems I see with the analysis in the paper that gives me concern:\n\n1. The main result in the paper is Theorem 1 which relies on Lemma 1. Lemma 1 essentially states that the proposed new metric (Peer RL reward) that subtracts a chance agreement baseline (Second term in equation 1) from the standard agreement objective (First term in equation 1) is an affine function of the true reward. Theorem 1 then crucially relies on this result for the ensuing convergence analysis. The problem I see is that an analogous result to Lemma 1 -- and therefore Theorem 1 -- can also be proved for the noisy reward. Specifically equation (12) in Appendix A.1 shows that even the noisy reward 𝔼[\\tilde{r}] is an affine function of the true reward, with the exact same slope and just a different intercept. Given that observation the proof for Theorem 1 can be carried out /mutatis mutandis/ to prove convergence with the naive noisy reward instead of the peer reward. So to me all of the analysis in the current paper doesn't really explain \"why\" PeerRL is actually performing better than using noisy rewards. To me it seems that more attention must be paid to the intercept terms in equation (12) and (17) to really understand the gap in performance which the authors currently do not consider. This is the first issue I see in the analysis.\n\n2. The second issue I have is with the \"Correlated Agreement\" Objective itself. The \"CA with weak supervision\" objective shown in equation (1) seems to give unfair advantage to completely random, but high entropy, weak supervision. For example, in the toy example on Page 4 the authors give a demonstration that the \"CA metric\" will be 0.375 thereby punishing full agreement with a weak baseline. But if instead the weak baseline was a random and unbiased coin toss then the expected CA objective will be 1 - 0.25 - 0.25 = 0.5 > 0.375. So just because the weak supervision signal had higher entropy and therefore lower chance of random agreement therefore its score was higher. This aspect should be dealt with more thoroughly in the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "oZ08I6FNUk-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper formulates a framework for reinforcement learning and behavior cloning from weak supervisions (i.e., noisy rewards or imperfect expert demonstration). Specifically, it proposes PeerPL to perform efficient policy learning from the available weak supervisions, which covers PeerRL (for RL with noisy rewards), PeerBC (for imitation learning from imperfect demonstration) and PeerCT (for hybrid setting). The PeerPL idea is based on a new weak supervision objective that is in the form of difference between norm learning loss and a loss incurred by randomly sampling the supervision signals. Experimental results demonstrate that PeerPL significantly outperforms SOTA solutions when the complexity or the noise of the learning environment grows. The proposed idea is useful in practice as it increases the robustness of the learning process to imperfect supervision signals.\n\nComments: \n1.\tThe proposed idea seems to share some similar spirit as the contrastive learning, where it also constructs a contrastive training loss from positive and negative examples. The CA loss (1) is somehow similar to contrastive learning with one negative sample. It would be helpful for the authors to add some further discussion in this regard.\n\n2.\tIn the illustrative toy example, it would be helpful to explain how the numbers 0.75 and 0.25 come up. Based on my understanding, it seems that the calculation should be 1 – (0.8^2+0.2^2)=0.32?  (The probability of a_j and a_k’ being 1 is 4/5 and the probability of them being 0 is 1/5. So the probability of them being equal is (4/5)^2 + (1/5)^2.) Also, it is necessary to clarify that Eva is chosen to be the indicator loss here. Otherwise, it could be confused with the fact that cross-entropy loss is used in practical implementation.\n\n3.\tBased on Lemma 1, in order to guarantee convergence in Theorem 1, shouldn’t there be an additional condition of: e_{+} + e_{-1} < 1 (i.e., not too noisy) in Theorem 1 (to keep the same sign of the reward)? Likewise, I think there should be a similar requirement on the PeerBC setting as well (as reflected in the 1-e_{-}+e_{+} in the denominator of the bound in Theorem 2). If this is true, I think this point should be clarified as we do need some (weak) prerequisite/knowledge about noisy model. If this is not required, then it would be important to show some further experiments on this extremely noisy setting of e_{-}+e_{+} > 1. (Note that currently it only shows up to e=0.4.) In addition, I think it would be better to put all other assumptions (if any) on the noise model in the Theorem.\n\n4.\tRegarding the sample complexity result of PeerRL relative to RL with true rewards below Theorem 1, I think it might be useful to put it as a separate theorem after Theorem 1.\n\n5.\tPlease also report the perfect expert (fully converged PPO) results in Table 2 as a performance upper bound.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XIjrc9la-RK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of policy learning under weak/noisy supervision. The authors present PeerPL, a unified framework that can train agents using behavior cloning under noisy/suboptimal demonstrations, or using reinforcement learning under noisy rewards. PeerPL uses the idea of “Correlated Agreement” by subtracting the original objective with a second term. The second term evaluates on randomly paired state-action tuples and supervisions, punishing the “blind” agreement between the learning agent and the weak supervision. The authors instantiate this idea on both behavior cloning and RL (PG/DQN), and also evaluate it on the problem of policy co-training. The authors demonstrate that PeerPL outperforms the weak supervision baselines on IL and RL setting, and sometimes it even outperforms agents trained with clean supervision. \n\n\nStrengths\n\n+ The paper overall is well-written and easy to follow\n+ The proposed idea is simple and general, and can be applied to both RL and IL\n+ The proposed method does not require prior knowledge on the noise structure\n+ Nice and sound convergence analysis\n+ Proposed method outperform baselines in most tasks under both BC/RL settings and policy-cotraining\n\n\nWeaknesses\n\n- Although the experiments prove otherwise, the second term in the objective is not intuitive to me. The learning agent is not supposed to agree with randomly sampled supervision anyways, unless the noise is constructed in a way that encourages the agent to do so. I cannot convince myself that the second term is helpful other than potentially some normalizing factor that helps reduce the variance in the RL setting, or maybe help remove some misleading prior in the BC setting.\n\n- I am confused by the results in Figure 2 that PeerPL outperforms agents trained with *clean* reward. I am not convinced by the authors’ explanation in Section 5.1, as subtracting the reward with randomly sampled {-1,1} will not help exploration.\n\n- The theoretical results are not particularly useful as they assume binary reward/actions, which is rarely the case in practical settings.\n\n\nOther Comments\n\n- It would have been nice in section 5.2 how PeerBC compares against IRL, which is supposed to handle imperfect demonstrations to some degree.\n\n- For PeerRL, it would be nice to see results on tasks that are harder than CartPole, for example, Atari or other continuous control tasks.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ChtgnHc4XJ1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY OF CONTRIBUTION:\n\nThis work presents a pair of algorithms, PeerRL and PeerBC, which address certain forms of noise in reinforcement learning and behavioral cloning respectively.  For PeerRL, an arbitrary RL algorithm trains on an augmented reward signal, which is generated by subtracting a past reward signal (sampled randomly from the replay buffer) from the current reward.  PeerBC uses an augmented maximum-likelihood behavioral cloning loss that penalizes a policy for log-probability of a randomly sampled action for a randomly sampled state.  Empirical results show that Deuling DQN with the PeerRL reward outperforms the base DDQN algorithm in a noisy-reward version of the cart-pole task.  They also show that PeerBC outperforms standard behavioral cloning in learning from synthetic demonstrations on several Atari games, as well as the cart-pole and Acrobot tasks, all with noisy versions of the base reward signals.  The also evaluate PeerBC in in combination with reinforcement learning in a co-training setup, demonstrating a significant advantage over co-training with standard BC.\n\nAREAS OF CONCERN:\n\nThe main concern with this work is that the gains in performance observed with the PeerRL and PeerBC algorithms may not actually reflect the ability of these methods to correct for a specific type of noise.  For PeerRL, the modified reward is fact that PeerRL is able to outperform the version of DDQN with access to the true reward (Figure 2) suggests that at least part of advantage of PeerRL is better scaling of the reward signal, rather than noise reduction.  It would be helpful to compare PeerRL against noiseless scales and shifts of the true reward.  It also seems likely that simply adding a constant baseline value to the noisy reward would help reduce the variance of the return (PeerRL subtracts a noisy baseline value).  Comparing against this simpler variance reduction method would help us understand the importance of the specific form of the augmented PeerRL reward in improving performance.\n\nFor PeerBC, the concern is that the modified loss simply biases against high-entropy policies.  When the true policy is itself stochastic, the learned policy found by PeerRL may end up being the mode of this policy.  The advantage of PeerBC observed in Figure 3 may result from the fact that less noisy policies perform better in these tasks.  The fact that PeerBC outperforms the expert policy in Enduro (Figure 3c) would seem to support this hypothesis (the expert's true policy is more stochastic than the learned policy).  It might be helpful to compare PeerBC to a simpler approach which learns a stochastic policy (under the standard maximum likelihood objective), but always takes the most probable action under this policy during evaluation.\n\nWhile the authors claim to address the case where the rewards or demonstrated actions are perturbed by some arbitrary confusion matrix.  The theoretical and empirical analysis of PeerRL however are limited to the case of binary rewards. Similarly, the theoretical analysis of PeerBC appears to limited to the case of binary actions, though empirical results consider larger action spaces.\n\nMore generally, it is not clear that the noise models (for RL and BC) capture the challenges typical of RL and imitation learning (discussed in the first paragraph of the introduction).  In RL, the issue is often not that the reward is noisy in a fixed state, but that it is sparse within the state space, such that the return under a random policy is noisy.  For behavioral cloning, errors in human-generated data often reflects the fact that the human demonstrator has actually selected a suboptimal strategy, rather than the noisy execution of an optimal strategy.  Distributional shift in behavioral cloning does not reflect noise in the training data itself, but instead reflects the compounding error that occurs when we execute an imperfect policy for many timesteps.\n\nThese issues with the noise model could be addressed by making it clear earlier in the document exactly what noise models are being considered, and providing examples in the introduction of settings in which such noise would be expected in the rewards or demonstration data.\n\nThe authors attempt to describe their approaches to handling noise for RL and BC as instances of a more general peer evaluation algorithm.  The relationship between the RL and BC solutions appears to be superficial, however, as PeerRL and PeerBC seem to address noisy supervision in very different ways.  Furthermore, there are no theoretical results presented that apply to the general algorithm.  The derivation of the RL solution from the common framework (Equation 2) is incomplete, as the loss Eva^RL is never made precise.  It would likely be more clear to the reader if PeerRL and PeerBC were presented as separate, but analogous algorithms for their respective learning problems.\n\nThere are also some apparent technical issues with the theoretical results:\n\n1) At no point is it specified that the error rates must be less than 50% for binary rewards or actions. Theorems 1 and 2 are clearly wrong if this condition does not hold.  While and attentive reader should be able to infer this constraint on the noise model, it should be explicitly stated.\n\n2) The r_peer term (the true peer reward) in Lemma 1 is never defined.\n\n3) It is unclear what the variable theta in tau_theta refers to in section 4.2.\n\n4) There seems to be an error in the proof of Lemma 1. The decomposition of the noisy reward in lines (7) and (8) seems to assume that, even in the absence of noise, the expected reward signal will be zero, regardless of the policy being followed, or the current state.  This decomposition should incorporate the probability of true reward given the current state (or averaged over states given the current policy).  This may mean that Lemma 1, and potentially Theorem 1, are incorrect (though it is likely that this can be corrected without substantially changing the contribution of the paper).\n\nCONCLUSION:\n\nEven though the work considers a relatively constrained class of noise models, there are still practical cases where such noise in rewards (and particularly in demonstrated actions) is an issue, and effective methods for dealing with such noise are potentially valuable.  The issue with this work is that it is not yet clear that the proposed methods are actually effective in mitigating these types of noise in RL or BC.  It seems likely that at least part of the improvement in performance stems from other factors, either the preference for more deterministic policies in PeerBC, or the rescaling of the reward signal in PeerRL.  Without resolving these issue, it is impossible to know whether the proposed methods will actually be useful in settings where we have noisy rewards or demonstration data.  There are also technical issues which raise doubts about the validity of the theoretical results presented in this work.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers had some initial concerns about this submission. While the authors' rebuttal does a good job to address these concerns, the reviewers still have some doubts about the contribution of this paper and potential impact. In particular, it is not clear whether the performance improvements observed with the proposed algorithms is due to the ability to correct for noisy rewards or whether there are multiple other explanations for the improvement in performance. This makes it hard to predict whether the proposed algorithms will actually be useful in settings where noisy rewards or demonstration data are present.
Decision: Reject