Policy Learning Using Weak Supervision
Authors: Jingkang Wang
Hongyi Guo
Zhaowei Zhu
Yang Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Policy Learning
Noisy Reward
Imperfect Demonstration
Co-training

Abstract: Most existing policy learning solutions require the learning agents to receive high-quality supervision signals, e.g., rewards in reinforcement learning (RL) or high-quality expert demonstrations in behavioral cloning (BC). These quality supervisions are either infeasible or prohibitively expensive to obtain in practice. We aim for a unified framework that leverages the available cheap weak supervisions to perform policy learning efficiently. To handle this problem, we treat the weak supervision'' as imperfect information coming from a peer agent, and evaluate the learning agent's policy based on a  correlated agreement'' with the peer agent's policy (instead of simple agreements). Our approach explicitly punishes a policy for overfitting to the weak supervision. In addition to theoretical guarantees, extensive evaluations on tasks including RL with noisy reward, BC with weak demonstrations, and standard policy co-training (RL + BC) show that our method leads to substantial performance improvements, especially when the complexity or the noise of the learning environments is high. 

Policy Learning Using Weak Supervision
Jingkang Wang*1.2 Hongyi Guo Zhaowei Zhu" Yang Liu"
University of Toronto', Vector Institute?. Northwester University?, UC Santa Cruz wangjkêcs toronto. edu hongyiguo20250u. northwestern edu zwzhuQucsc.edu yangliubucsc.edd
Abstract
Most existing policy learning solutions require the learning agents to receive high- quality supervision signals such as well-designed rewards in reinforcement learning (RL) high-quality expert demonstrations in behavioral cloning (BC). These quality supervisions are usually infeasible or prohibitively expensive to obtain practice. We aim for unified framework that leverages the available cheap weak supervisions to perform policy learning efficiently. To handle this problem, we treat the "weak supervision" as imperfect information coming from a peer agent, and evaluate the learning agent's policy based on a "correlated agreement" with the peer agent's 's policy (instead of simple agreements). Our approach explicitly punishes policy for overfitting to the weak supervision. In addition to theoretical guarantees, extensive evaluations on tasks including RL with noisy rewards, BC with weak demonstrations, and standard policy co- co-training show that our method leads to substantial performance improvements, especially when the complexity or the noise of the learning environments is high.
Introduction
Recent breakthroughs in policy learning (PL) open up the possibility apply reinforcement learning
(RL) or behavioral cloning (BC) real-world applications such as robotics [1, 2] and self-driving [3, 4]. Most existing works require agents receive high-quality supervision signals, e.g., reward expert demonstrations, which are either infeasible or expensive to obtain in practice [5, 6].
The outputs of reward functions in RL are subject to multiple kinds of randomness. For example, the reward collected from sensors on robot may be biased and have inherent noise due to physical conditions such as temperature and lighting [7, 8, 9]. For the human-defined reward, different human instructors might provide drastically different feedback that leads to biased rewards [10]. Besides, the demonstrations by expert in behavioral cloning (BC) often imperfect due to limited resources and environment noise [11, 12, 13]. Therefore, learning from weak supervision signals such as noisy rewards [7] low- low-quality demonstrations produced by untrustworthy expert [12, 14] is one of the outstanding challenges that prevents a wider application of PL..
Although some works have explored these topics separately in their specific domains [7, 15, 14, 16]. there lacks a unified solution for robust policy learning in imperfect situations. Moreover, the noise model as well as the corruption level in supervision signals is often required. To handle these challenges, we first formulate weakly supervised policy learning. Then we propose theoretically principled solution, PeerPL, to perform efficient policy learning using the available weak supervision without requiring noise rates.
fratffrraework to study RL/BC with weak supervision and call
Our solution is inspired by peer loss [17]. a recently proposed loss function for learning with noisy labels but does not require the specification of noise rates. In peer loss, the noisy labels are treated as peer agent's 'ss supervision. This loss function explicitly punishes the classifier from simply agreeing with the noisy labels, but would instead reward it for a "correlated agreement" (CA). We adopt
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

similar idea and treat the "weak supervision" as the noisy information coming from an imperfect peer agent, and evaluate the learning agent's policy based on a "correlated agreement" (CA) with the weak supervision signals. Compared to standard reward and evaluation functions that encourage simple agreements with the supervision, our approach punishes "over-agreemen"" to avoid overfitting to the weak supervision, which offers family of solutions that do not require prior knowledge of the corruption level in supervision signals.
To summarize, the contributions in the paper are: (1) We provide a unified formulation of the weakly supervised policy learning problems: (2) We propose PeerPL, new way to perform policy evaluation for RL/BC tasks, and demonstrate how it adapts in challenging tasks including RL with noisy rewards and BC from weak demonstrations; (3) PeerPL is theoretically guaranteed to recover the optimal policy, as if the supervision are of high-quality and clean. (4) Experiment results show strong evidence that PeerPL brings significant improvements over state-of-the-an solutions. Code is online available at: https: //gi thub. com/ com/wang ks tttttttuuPr....
1.1 Related Work
Learning with Noisy Supervision Learning from noisy supervision is widely explored topic. The seminal work [18] first proposed an unbiased surrogate loss function to recover the true loss from the noisy label distribution, given the knowledge of the noise rates of labels. Follow-up works offered ways to estimate the noise level from model predictions [19, 20, 21, 22, 23, 24, 25, 26, 27] or label consensuses of nearby representations [28]. Recent works also studied this problem in sequential settings including federated bandit [29] and RI. [7]. The former work assumes the noise can be offset by averaging rewards from multiple agents. [7] designs statistics-based estimation algorithm for noise rates in observed rewards, which can be inefficient especially when the state-action space is huge. Moreover, the error in the estimation can accumulate and amplify in sequential problems. Inspired by recent advances of peer loss [17, 30, 31], our solution is able to recover true supervision signals without requiring a priori specification of the noise rates.
Behavioral Cloning (BC) Standard BC [32. 33] tackles the sequential decision -maning problem by imitating the expert actions using supervised learning. Specifically, it aims to minimize the one-step deviation error over the expert trajectory without reasoning about the sequential consequences of actions. Therefore, the agent suffers from compounding errors when there is mismatch between demonstrations and real states encountered [33, 34, 35]. Recent works introduce data augmenta- tions [36] and value-based regularization [37] or inverse dynamies models [38, 39] encourage learning long-horizon behaviors. While being simple and straightforward, BC has been widely inves- tigated in a range of application domains [40, 41] and often yields competitive performance [42, 37]. Our framework is complementary to the current BC literature by introducing a learning strategy from weak demonstrations (e.g.. noisy or from a poorly- torinee agent) and provides theoretical guarantees on how to retrieve clean policy under mild assumptions [43].
Correlated Agreement In [44, 45], correlated agreement (CA) type of mechanism is proposed to evaluate the correlations between agents' reports. In addition to encouraging a certain agreement between agents' reports, CA also punishes over-agreement when two agents always report identically. Recently, [17, 30, 25] adapt a similar idea to noisy label learning thus offloading the burdens of estimating noise rates. We consider more challenging sequential decision-making problem and study the convergence rates under noisy supervision signals.
Policy Learning from Weak Supervision
We begin by reviewing conventional reinforcement learning and behavioral cloning with clean supervision signals. Then we introduce the weak supervision problem in policy learning and define two concrete instantiations: (1) RL with noisy reward and (2) BC using weak expert demonstrations.
2.1 Overview of Policy Learning
The goal of policy learning (PL) is to learn policy that the agent could follow to perform a series of actions a stateful environment. For reinforcement learning, the interactive environment is characterized as an MDP .M- an action o, E A by following the policy æ S x A - RR, and potentially receives reward r(st, ar) R. Then the agent transfers to the next state St+1 according to transition probability
7). At each time I, the agent in state 8, E S takes

---

with Weak Expert
with Perturbed Reward
Policy Co- Co-Training
Behavior Cloning
Peer Policy Learning
Reinforcement Learning
(s.a). a P
Agent: (5. "e)....
s
Weak
Peer Agent: {.... Fo. K
Noisy Environment
a
Demonstrations
(s,0)
CA:
Eva (50.00) ao); 86) Eva ((.. P.P.)
F as Y
Figure I: Illustration of weakly supervised policy learming and our PeerPL solution with correlated agreement (CA). We use Y to denote weak supervision, be it noisy reward, noisy demonstra- tion. Eva stands for an evaluation function. "Peer Agent" corresponds to weak supervision.
function P. We denote the generated trajectory T = {(st. ae, T== where T is finite or infinite horizon. RL algorithms aim to maximize the expected reward over the trajectory induced by the policy: Jokean (T) E(s,,...rr) 12 7're), where 7 E (0, 1] is the discount factor.
Another popular policy learning method is behavioral cloning. Let (- denotes the distribution over actions formed by r, and π (a|s) be the probability of choosing action a given state s and policy и. The goal of BC is to mimic the expert policy gg through set of demonstrations Dg {s, a.) ()) ) drawn from distribution Dg. where (s,, a,) the sampled state-action pair from the expert trajectory and Og (=(--s;;)))))))))))))))(()) Then training a policy with standard BC corresponds to maximizing the following tog-likelihoond Jelsan nT) De
(log π (a|s)).
In both RL and BC, the learning agent receives supervision through either the (clean) reward r by interacting with environments or the expert policy n observable demonstrations. Consider a particular policy class II, the optimal policy is then defined as x² = arg max Jokean (Tr): " obtains the maximum expected reward over the horizon T in RL and ग corresponds to the clean expert policy Te in BC. In practice,
can also combine both RL and BC approaches take advantage of both learning paradigm [46, 47, 15, 43]. Specifically, a recent hybrid framework called policy co-training [43] will be considered in this paper.
2.2 Weak Supervision in Policy Learning
The weak supervision signal Y could be noisy reward for RL or noisy action from an imperfect expert policy Eg for BC, which are noisy versions of the corresponding high-quality supervision signals. See more details below.
RL with Noisy Reward Consider finite MDP M (S, nels [7], where R S x A + R, and the noisy reward is generated following a certain function F: R + R. Denote the trajectory a policy my generates via interacting with M řa. Assume the reward is discrete and has |R] levels. The noisy reward can be characterized via a unknown matrix [ Ri where each entry G, indicates the flipping probability for generating possibly different RL. 5, = P (F: = Ralri = R2). We call r and i the true reward and noisy reward.
with noisy reward chan-
CRL
outcome:
BC with Weak Demonstration Instead of observing the true expert demonstration generated according to π gg denote the available weak demonstrations by {(ss, ā:) J where ā; is is the noisy expert action drawn according to a random variable ā; = FF(;;) 8;) ~ (=))) each state-action pair
(st, ā.) is sampled from distribution DE. Note there may exist two randomness factors in getting āj: uncertainty in true policy T and noise from imperfect policy Rg. In particular, we do not consider the former randomness in theoretical analyses: given the output distribution Tg( (+]ss,,, only one deterministic action πT (s;) is taken by expert. This because with uncertainty in true expert actions, it is hard to distinguish clean case with true expert actions from the weak supervision case without addition knowledge. Similar assumptions are also adopted in [2). The noisy action modeled by unknown confusion matrix CBC [A]x|A| where each entry Cj.A indicates the flipping probability
for taking a sub-optimal action that differs from (s)) and A, denote the k-th and the j-th action from the action space A. In the above definition, we
BC = { = (s(=g(s) = =Ay). A
G.
assume the noisy action a, is independent of the state given the deterministic expert action (s(s).
3

---

i.e., P():/T=(ss)) P(ā, (s:)). We aim to recover x² as if we were able to access the quality
expert demonstration Fe instead of iT..
Knowledge of C Recall C: | [R| or C [[A]x]A] is unknown in practice. While recent works estimate this matrix [26, 23, 28J in supervised classification problems, it is still challenging to generalize them sequential setting [7]. When C is not perfectly estimated, the estimation error of C may lead to unexpected state-action pairs then the eπror of reward estimates will be accumulated in sequential learning. Besides, estimating C involves extra computation burden. In contrast, our method gets rid of the above issues since it is free of any knowledge of C and leads to more robust policy learning algorithms.
Learning Goal With full supervision, both RL and BC can converge to the optimal policy л² However, when only weak supervision is available, with an over-parameterized model such as a deep neural network, the learning agent will easily memorize the weak supervision and learn a biased policy [48]. In our meta framework, instead of converging any biased policy, we focus learning the optimal policy : with only weak supervision sequence denoted as {(1,, 0z), Vi) (RI.) or {(8,, 4;), y (BC).
3 PeerPL: Weakly Supervised PL via Correlated Agreement
To deal with weak supervision in PL, we propose a unified and theoretically principled framework PeerPL. We treat the weak supervision as information coming from a "peer agent", and then evaluate the policy using a certain type of "correlated agreement" function between learning policy and the peer agent's s information.
3.1 A Unified Evaluation Function
We use an evaluation function Eva-((s;; a:), Y2) to evaluate taken policy л at agent state (s;, a;) using the weak supervision V, For RI., Eva, is the instance-wise measure (negative loss) for different RL algorithms, which is function of the noisy reward i received (si, a;). In the BC setting, Eva, the loss to evaluate the action ag taken by the agent given the expert's ' demonstration ä, Note that the larger the Eva,, is at state (s;, a;), the better it follows the supervision Y,. Specifically, we have
Eva RL ((s,a), ( (s, (s,,,))) F))
(RL) and Eva BC ((s, a) log (āls) (BC),
where the RL loss function é can be temporal difference error [49, 50] the policy gradient loss [51]. Furthermore, we let (sr) denote the function that evaluates policy æ under a set of state action pairs with weak supervision sequence {(s; 02), Y, I i.e..
((m) (,,,)~r [Eva, ((,,((),)),))))
Then the goal of weakly supervised policy learning is recover the optimal policy if we receive clean supervision Y. Note that directly maximizing // (T) might result in sub-optimal performance due the weak supervisions. The above unified notations are only for better delivery framework and we still treat PL as sequential decision problem.
3.2 Overview of the Idea: Correlated Agreement with Weak supervision
We first present the general idea of our PeerPL framework using concept named correlated agreement (CA). For each weakly supervised sample ((s;, a;), Yj), we randomly sample (with replacement) two other peer samples indexed by j and k. 'hen we take the state-action pair (sj,a,) of sample and the supervision signal Y, of sample and evaluate ((s,, a;), Y.) follows:
CA with Weak Supervision: Eva, a,), Y.) Eva, (5s),ay) Yz).
This operation is illustrated in Figure 1. We further show intuitions and a toy example below.
Intuition The first term above encourages "agreement" with the weak supervision (that a policy agrees with the corresponding supervision), while the second term punishes "blind" and "over" agreement that happens when the agent's policy always matches with the weak supervision even

---

on randomly paired traces (noise). The randomly paired instances j. k help us achieve this check.
Note our mechanism does not require the knowledge of C [ nor C ||A and offers prior- knowledge free way to learn effectively with weak supervision.
Toy Example Consider toy BC setting where the policy fully memorizes the weak supervision and outputs the same sequence of actions given the same sequence of states, i.e.,
weak-supervision: ä = ق2 = 03- = 1, = O; Outputs: a1 = 02 = 0g = a, = 0.
Let Eva, (88;, dz), ā.) = 1 the policy output agrees with the weak demonstration (a; = ā.), and 0 otherwise. When the policy fully memorizes weak supervisions, we have:
Without CA:
EE[va, az), a:)] 1,
With CA:
E[Eva. ((s;, (3sss.a))))) uz), ā:) Eva, ((5j, az), āz)] 0.375,
where 0.375 = (0.75² + 0.25²) is obtained by considering the probability of randomly paired aj and 0₂ matching each other. The above example shows that full agreement with the weak supervision will instead be punished.
In what follows, we showcase two concrete implementations: PeerRL (peer reinforcement learning) and PeerBC (peer behavioral cloning). We provide algorithms and theoretical guarantees under weak supervisions.
4 PeerRL: Peer Reinforcement Learning
We propose the following objective function to punish the over-agreement of parametric policy me based on CA:
JRI(mT) = E Eva, RL, RT. ((s, a), -[ (s,
- g. F Eva, El. ((s),
where
Eva
(2)
In (1), the first expectation is taken over (8s, 0r, Fz) ~ 7 and second one is taken over (8j, aj. f2) ~ 7, (Sky Og, Fx) f, where F is the trajectory specified by the noisy reward function F. Recall j, k denote two randomly and independently sampled instances. Loss function & depends on the employed RL algorithms, e.g., temporal difference error [49,50] or the policy gradient loss [51]. The learning sequence is encoded in т. The objective (RL((T) represents the accumulated peer RL reward. Parameter & 2 0 balances the penalty for blind agreements induced by CA.
4.1 Peer Reward
In what follows, we consider the Q-Learning [52] as the underlying learning algorithm where (( ()) -f(ss, a) and demonstrate that the CA mechanism provides strong guarantees for Q-Leamnig with only observing the noisy reward. For clarity, we define peer RL reward:
Peer Reward:
Epeer (8, a) F(s,a) E.
where fr Teample (F(ss E S,a A} is reward sampled over all state-action pairs according to fixed policy Tsample- Note the sampling policy Tsample is independent of π and the choice of Tsample does not affect
theoretical results. We adopt a random sampling strategy in practice. Parameter Ç 2 0 balances the noisy reward and the punishment for blind agreement (with F'). We set ç = 1 (for binary case) in the following analysis and treat each (s, a) equally when sampling f'. experiments. we find Fpee is not sensitive to the choice of & and keep & constant for each run.
Robustness to Noisy Rewards
Now show peer reward Fpssrr offers the true reward in expectation, which guarantees that our PeerRL algorithm converges to T Consider the binary reward setting (r_ and r-) and denote the error in F as e. = IP( r-lr Ir r+), P(β = T+ = r_) (a simplification of CRL |
affine transformation of
[R. in the binary setting).
Lemma 1. Let [0, Rmnax] be bounded reward, & 1. Assume 1 - -_ c+ 0. We have:
Eli Blrpeer (8, a)] (1 1----_p -e4) B'rpeer Spee (8, a)] = e_ -e_- (8 a)) + const,
where ppeer a) = r(s, a) - o' the peer RI. reward when observing the true reward v, and r' the true reward corresponding 7'.
5

---

Lemma 1 shows that by subtracting the peer penalty term F' from noisy reward F(s. a), peerr (s. a) recovers the clean and true reward r(s, a) in expectation. Based on Lemma 1, we prove in Theorem A that the Q-learning agent will converge to the optimal policy w.p. with peer rewards without requiring any knowledge of the corruption in rewards (CRL
[ |R|- as opposed to previous work [7] that requires such knowledge). Moreover, we prove in Theorem A2 that to guarantee the convergence to 'T, the number of samples needed for our approach is no more than (11((1 - - e+)²) times of the one needed when the RL agent observes true rewards perfectly (see Appendix A).
Even though we only present an analysis for the binary case for Q-Learning, our approach rather generic and ready be plugged into modern DRL algorithms. We provide multi-rewand extensions, implementations with DQN [49] and policy gradient [51] in Appendix A.
Extension
4.2 Why does Peer Reward Work?
Compared with noisy reward, proposed peer variant a less biased estimation of true reward (Benefit- 1). On the other hand, PeerRL helps break the unstable "tie" states, which might encourage the agent to explore in the early stage [53] (Benefit-2).
Benefit-1: PeerRL reduces the bias We highlight that the biased noise model considered is rather generic, departing from the previous noise assumption such as aero-mean Gaussian noise [8, 9]. In zero-mean noise models, the major focus is on variance reduction increases the variance thus resulting in worse estimation. However, in the discrete biased noise model [18], bias correction also plays important role especially the noise rate is high [7].
adding the random term ff
Similar peer reward (Lemma 1), the expectation of the noisy reward writes as: [F a)] = (1 e. )B[((, a)] + e_r+ + 644- = (1 peer reward has less effect on the true reward r, especially when the noise rate is high. To see this:
)BrBBB(ss a)] + const. But the constant in
noisy reward:
IFF(s, a)] = (BIr(s, a)] + T-a- T- + T ot T++)
peer reward: per (s, a)] = 17 (E(r(s, a)] (1 Ppeer) Ppeer"+)
where 1 e__ - e 0. Ppeer [0, 1] denotes the probability that sample policy sees reward r+ overall. Since the magnitude of noise terms I-c -ei and mej can potentially become much larger than 1 - Ppeer and Ppeer in a high-noise regime, T-e_ T-e__e
T-e__e+-- will dilute the informativeness of E/r(s, a)]. On the contrary, per (s,a), a)] contains moderate constant noise thus maintaining more useful training signals of the true reward in practice. In summary, although peer reward (similar the surrogate reward in previous literature (7]) increases the variance (no free-lunch), it will lead to a better estimation of the true reward due to lower bias.
Benefit-2: PeerRL helps break ties For RL, "tie" states indicate that the rewards for different states are the same, which are less informative as they neither serve positive nor negative examples. Due to the discrete nature of the noise model, adding randomly sampled penalty term helps break the tie states and treats them as either positive examples or negative examples such that it can encourage exploration in the carly stage, which has similar intuitions to some RI. exploration works [53]. It has also been demonstrated that reducing the uncertainty, a.k.a.
2-state Markov process (no actions)
81
82
clampp// 1), min 0, max
T2 clamp|N((0.4 1), min 0, max
Correct
Tie
Incorrect
bascline
54.6%
5.6%
39.8%
pushing confident predictions, makes the learning robust to weak-supervisions in supervised learning [17, 54]. On the other hand, it is known that positive examples are sparse yet important in RL. To leverage these useful experiences sufficiently, experience replay [55, 56] is invented to store and up-sample the positive examples for faster convergence. Tie breaking potentially provides an alternative way to access more positive examples. To illustrate tie-breaking phenomenon when using peer reward, we consider a two-state Markov process (no actions) with bounded Gaussian noise and see how well we could infer which state was better by correcting the reward signals. We collect two observations for each state and conduct 10d trials to calculate the success rate of inferring which state has larger returns ("correct" in the Table). As we can see, PeerRI. exploits the "discreteness" of the reward thus breaking ties to obtain more examples with good-quality supervision. More examples on varied noise models (bounded continuous noise, discrete noise) are deferred to Appendix B.
PeerRL
58.0%
0.3
41.7%

---

200
0005 oporods
snapa optoods
mooe per spiccdt
rone oplonds
350
800
8
30
100
30
103
150
(a) e 0.2
e 0.4
(a) 0.2
(b) 0.4
Figure 2: Learning curves of DDQN on CartPole- Figure 3: Learning curves of DDPG [57] on Pen-
vÔ with true reward (r) noisy reward (7) dulum with true reward (r) noisy reward (7)
surrogate reward [7] (f) and peer reward ( peer. and peer reward (Fpeer: C = 0.2)
{ 0.2)
PeerBC: Peer Behavioral Cloning
Similarly, we present our CA solution in the setting of behavioral cloning (PeerBC). In BC, the supervision is given by the weak expert's 's noisy trajectory. At each iteration, the agent learns under weak supervision à. and the training samples are generated from the distribution DE determined by the weak expert. The Eva BC function in BC evaluates the agent policy тe. parametrized by 0, and the weak trajectory [s,, a using ((To, (st, ā.)), where is an arbitrary classification loss. Taking the cross-entropy for instance, the objective of PeerBC is:
JBC (me) Eva (oaao((s;
((5g, a;), E Eva BC ((s),
(3)
where
Eva BC ((s, a), = - -f( (8, a)) = log To (ā(ss)
(4)
In (3), the first expectation is taken over (st, ā.) ~ DE, ~ T-1s;) and the second is taken over (sj,ūg) āz) ~ DE, U2 m--1a)), (sk, āx) DE, dg T(+ T--1sk). Again, the second Eva, BC term in JBC serves the purpose of punishing aver-agreement with the weak demonstration. Similarly, & 2 0 is parameter to balance the penalty for blind agreements.
Robustness to Noisy Demonstrations We prove that the policy learned by PeerBC converges the expert policy when observing sufficient amount of weak demonstrations. We focus on the binary action setting for theoretical analyses, where the action space is given by = (A+,A-) and the weakness or noise in the weak expert iig is quantified by e+ (s) = A TT(s) = A+) and
e_ (s) = A+|T=(()) = A-). Let TDg be the optimal policy for maximizing the objective in (3) with imperfect demonstrations DE (a particular set of with N i.i.d. imperfect demonstrations). Note E(-) is specified as the 0-1 loss: (TT(s),
when (s) + a, otherwise ((T(ss, a) 0. We have the following upper bound on the error rate.
Theorem 1. Denote by Rpg = P (sssa))DDD (T (s) + a) the error rate
rate for PeerBC. When et +e- <
1, with probability at least 1 - 8, it is upper-bounded as: RT S T
1+6
2log2/5
Theorem states that as long as weak demonstrations observed sufficiently, i.e., N is sufficiently large, the policy learned by PeerBC is able to converge to the clean expert policy TE(S) with convergence rate of (1/VNN)
Peer Policy Co-Training Our discussion of BC allows us study more challenging co-training task [43]. Given a finite MDP M, there are two agents that receive partial observations and we let RA and Ta denote the policies for agent A and B. Moreover, two agents are trained jointly to learn with rewards and noisy demonstrations from each other (e.g.. at the preliminary training phase). Symmetrically, we consider the case where agent A learns with the demonstrations from B on sampled trajectories, and Ty effectively serves as noisy version of expert policy.
Following [43], we assume a mapping function fA-BB exists that transforms states under view A into B. Denote by 74 {( az { the trajectory that a generates via interacting with the partial world M Then TB replaces each action at with its selection a!¹ (B(ff+ (s{")) the weak supervision. To recover the clean expert policy, adapt the BC peer evaluation term to the
7

---

10
spience
-20
100
26
150
batches (c100)
# bucchea (4100)
buscchss (e100)
batches (x100)
(a) Pong
(b) Boxing
(c) Enduro
(d)
Freeway
Figure 4: Learning curves of BC on Atari. Standard BC
PeerBC (ours)
expert
co-learning objective function:
JC (Te) E Eva!" ((("aa/))) r:"" + Eva (($ ((s5",at at
Eva ((s) ,a,) aaf),, (S)
where the first expectation taken over (s,1 aj r;a) ~ TA_ and a. n (f(fA= A 8 (s s/²) )), and the second taken over (s5). az rgg) TA (sf ox rg) A and a TB (f(A )), is the loss function defined in Eqn. (4) to measure the policy difference, and Eva RL Eva BC are defined in Eqn. (2) and (4) respectively. The full algorithm PeerCT is provided in Algorithm 1. We omit detailed discussions on the convergence of PeerCT it can be viewed straight- extension of Theorem in the context of co-training.
6 Experiments
We evaluate our solution three chal- lenging weakly supervised PL prob- lems. Experiments on control games and Atari show that, without any prior knowledge of the noise, our approach is able leverage weak supervision more effectively.
Algorithm Peer policy co-training (PeerCT)
Require:
Views A, MDPs MA, MM, policies T, Ns, map- ping functions fA- fa- that maps states from one view to the other view, CA coefficient step size & for policy update.
1: repeat
2: Run / to generate trajectories 7 {(s a 1())
3: Run π to generate trajectories T = {s Og 5 15m-1.
B B
Experiment Setup & Baselines We evaluate PeerPL on a wide variety of control and Atari games. For RL with noisy reward, we add synthetic noise to reward signals and compare with previous work [7], where an unbiased estimator of true reward is constructed by approximating the confusion matrix. For BC from weak demonstrations, we adopt not fully converged PPO agents the weak experts and unroll the trajectories. We also consider standard policy co-training setting [43] without any synthetic noise added and compare PeerCT with single- view training paradigm and CoPiEr [43].
4: Agents label the trajectories for each other
7" f- [(5) Ta (fR (
7 + [(5) (AA- (8 5/ 3
5:
Update policies: T(A,B) #(A.B) 3 J. (TAAB)
6:
until convergence
6.1 PeerRL with Noisy Reward
CartPole-vO: We first evaluate our method in RL with noisy reward setting. Following [7]. we consider the binary reward 1- for Cartpole where the symmetric noise is synthesized with different error rates e the models for 10,000 steps. We repeat each experiment 10 times with different random seeds and leave extra results in Appendix D. Figure 2 shows the learning curves for DDQN with different approaches in noisy environments (g = 0.2) !. Since the number training steps is fixed, the faster the algorithm converges, the fewer total episodes the agent will involve thus the learning curve is on the left side. As consequence, the proposed peer reward outperforms other baselines significantly in high-nois regime (c.g., = 0.4). Table 1 provides quantitative results the average reward Raeg and total episodes Nept. We find the agents with peer reward lead to a larger Ravg (less generalization error) and smaller N Nepi (faster convergence) consistently.
= e+- We choose DQN [49] and DDQN [50] algorithms and train
We analysed the sensitivity of & and found the algorithm performs reasonable when & E (0.1,0.4). More insights and experiments with varied is deferred to Appendix D.

---

Table 1: Numerical performance of DDQN CartPole with true reward (r). noisy reward (f), surrogate reward F [7], and peer reward peer = 0.2). Ravy denotes average reward per episode after convergence, the higher (1) the better; Nepi denotes total episodes involved in 10,000 steps, the lower (4) the better. Note 0 < E < 0.5.
0.1
e
0.4
Rasg
New
Rayg
New
Rang
Nejo 4
Ray T
Nug
195.6 + 3.1
101.2++3.
195.6 + 3.1
101.2 + 3.2
195.6 3.1
101.2 3.2
195.2 +.0
101.2
3.3
DDQN
185.2 = 15.6
6.
1
13.
123.9
9.0
177.1
11.2
9.3
185.5
10.9
163.1
11.0
183.9
10.4
110.6
6.7
165.1
+ 18.2
113.9
9.6
192.2 + 10.9
115.5
4-44-4.. 4.3
179.2
1.0.2-664
125.8
9.6
ppeer
198.5
2.3
86.2
1
5.0
195.5
i 9.1
85.3
+ 5.4
174.1
32.5
88.8
6.3
191.8
8.5
106.9 1: 9.2
-902
200
300
-203
1000
-900
-401
-303
cotodd
apporm
656000
(a) Acrobot
(b) CartPole
Pong
(d) Breakout
Figure 5: Policy co-training on control/Atari. Single view
[43]
PeerCT (ours)
Pendulum: We further conduct experiments
continuous control task Pendulum, where the goal is to keep a frictionless pendulum standing up. Since the rewards in pendulum are continuous: r (-16.3, 0.0). we discretized it into 17 intervals: (-17, -16] (-16, -15] (-1, 0j. with its value approximated using its maximum point. We test DDPG [57] with uniform noise in this environment following [7]. In Figure 3, the RL agents with the proposed CA objective successfully converge to the optimal policy under different amounts of noise. On the contrary, the agents with noisy rewards suffer from biased noise, especially in high-noise regime.
Analysis of the benefits in PeerRL More surprisingly, we observed that the agents on CartPole with peer reward even lead to faster convergence than the ones observing true reward perfectly when the noise rate e is small. This indicates the possibility of other benefits to further promote peer reward, other than the noise reduction one we primarily focused on. We hypothesize this is because (1) the peer penalty term breaks the tie states (Benefit-2 in Section 4.1) and encourages explorations in RL; (2) PeerRL scales the reward signals appropriately for easier learning: (3) the human-specific "true reward" might be also imperfect which leads to a weak supervision scenario. We emphasize that the advantage of recovering from noisy reward signal is non-negligible, especially in a high-noise regime (e.g., = 0.4 in Figure 2 and 3).
6.2 Pe from Weak Demonstrations
Atari: In BC setting, we evaluate our approach on four vision-based Atari games. For each environment, we train imperfect RL model with PPO [58] algorithm. Here, "imperfect" means the training is terminated before convergence when the performance is about 70% ~ 90% as good as the fully converged model. We then collect the imperfect demonstrations using the expert model and generate 100 trajectories for each environment. The results are reported under three random seeds.
Figure 4 shows that our approach outperforms standard BC and even the expert it learns from. Note that during the whole training process, the agent never learns by interacting directly with the environment but only have access to the expert trajectories. Therefore, we owe this performance gain to PeerBC's strong ability for learning from weak supervision. The peer term we add not only provably eliminates the effects of noise but also extracts useful strategy from the demonstrations. As shown in Table 2. our approach consistently outperforms the expert and standard BC. We provide the sensitivity analysis of & in Appendix D.
Comparison with imitation learning baselines We further extend the empirical study to imitation learning (IL) algorithms on CartPole-vI. To collect weak demonstrations, we train PPO agent for 50k iterations that are not fully converged. As shown in Figure 6, standard IL algorithms such as BC, AIRL [37]. or GAIL [59] cannot handle noisy demonstrations well and lead to sub-optimal performance. Our PeerBC brings 18% improvement over standard BC by penalizing blind agreements with the weak demonstrations. We remark that performance of PeerBC is worse than DAgger due to notorious distribution shift issue. To further improve performance, we train PeerBC in the DAgger fashion DAgger by querying the imperfect expert to augment the training sets. Not surprisingly.

---

Table 2: BC from weak demonstrations. PeerBC successfully recovers better policies than expert.
Environment
Pong
Boxing
Enduro
Freeway
L7)
Expert
15.1 + 6.6
67.5 + 8.5
150.1 + 23.0
21.9 1.7
Standard BC
14.7 +: 3.2
56.2 + 7.7
138.9 14.1
22.0 1.3
-6.6%
0.2
67.2
177.9
29.3
-1.3%
PeerBC
0.5
16.6 +: 4.0
75.6 + 5.4
230.9 73.0
22.4 1.3
+19.5%
1.0
16.7 + 4.3
69.7 + 4.7
230.4 61.6
8.9
4.9
+2.0%
Fully converged PPO
20.9 +: 0.3
89.3 5.4 389.6 +: 216.9
33.3 + 0.8
Table 3: Comparison with single view training and CoPiEr [43] on standard policy co-training.
Environment
Acrobot
CartPole
Pong
Breakout
A
136.6 + 15.6
172.8 5.5
17.8 0.6
148.0 + 16.5
Single View
B
126.44 + 8.0
186.7 + 8.1
17.7 + 0.5
137.8 + 12.5
A
136.2 1: 5.2
174.1
16.8 1: 0.5
107.5 5.8
CoPiEr
B
131.5 = 4.5
174.3 5.4
16.5 + 0.2
82,7 + 6.9
A
87.0
1: 3.9
188 1. 2.7
20.5 1. 0.4
263.6 1. 36.0
PeerCT
B
87.1 +: 6.3
184.7 + 3,9
20.4 1 0.5
268.6 + 33
Peer-DAgger surpasses DAgger by large margin, which indicates that our framework has wide applicability and successfully recovers the true supervision signals. Adapting PeerPL idea to more IL algorithms such as GAIL [59] and DART [35] together with rigorous analysis left as future works.
Analysis of benefits in PeerBC Similarly, the performance improvement of PeerBC might be also coupled with multiple possible factors. (1)
+68%
The imperfect expert model might be noisy version of the colle-co agent since there
Score
are less visited states on which the selected actions of the model contains noise. (2) The improvements might be brought up by biasing against high-entropy policies thus PeerBC is use- ful when the true policy itself is deterministic. We provide more discussions about the second factor in Appendix D.5.
+18%
200
220
25
BC
PeerfiC
DAgger
Pwee-DAgger
AIRL
CAR..
Figure 6: Comparison of imitation learning ap- proaches on CartPole-vl with imperfect expert.
6.3 PeerCT for Standard Policy Co-training
Continuous Control/Atari: Finally, we verify the effectiveness of the PeerCT algorithm in policy co-training setting [43]. This setting more challenging since the states are partially observable and each agent needs to imitate another agent's behavior that is highly biased and imperfect. Note that we adopt the exact same setting as [43] without any synthetic noise included. This implies the potential of our approach to deal with natural noise in real-world applications. Following [43], we mask the first two dimensions respectively the state vector to create two views for co-training in classic control games (Acrobot and CartPole). Similarly, the agent either removes all even index state vector or removing all odd index ones (view-B) on Atari games. As shown in Table 3 and Figure 5, PeerCT algorithm outperforms training from single view, and CoPiEr algorithm consistently on both control games (E = 0.5 in Figure 5a, 5b) and Atari games (E = 0.2 in Figure 5c, 5d). In most cases, our approach leads to faster convergence and lower generalization error compared to CoPiEr, showing that our ways of leveraging information from peer agent enables recovery of useful knowledge from highly imperfect supervision.
coordinates (view-A)
7 Conclusion
We have proposed PeerPL,, weakly supervised policy learning framework to unify series of RL/BC problems with low-quality supervision signals. PeerPL, instead of blindly memorizing the weak supervision, we evaluate a learning policy's correlated agreements with the weak supervision. We demonstrate how our method adapts RL/BC and the hybrid co-training tasks and provide analysis of the convergence rate and sample complexity. Current theorems focus the specific discrete noise model. Future work may extend to more general noise scenarios and evaluate our method on real RL/BC systems, such robotics and self-driving.
10

---

Acknowledgement
We sincerely thank the anonymous reviewers for their insightful suggestions. Our final version benefited substantially from the discussions with Reviewer 58fG. In particular, the tie-breaking analysis together with the code snippet designed and contributed by Reviewer 58fG. This work is partially supported by the National Science Foundation (NSF) under grant IIS-200795I and the Office of Naval Research under grant N00014 100014200-11 1-2240. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
References
[1] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
[2] Igge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng. Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand. CoRR, abs/1910.07113, 2019.
[3] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07331 2016.
[4] Felipe Codevilla, Matthias Miiller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy, End-to-end driving via conditional behavior cloning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-9, IEEE, 2018.
[5] Vibhu Agarwal, Tanya Podchiyska, Juan M Banda, Veena Goel, Tiffany I Leung, Evan P Minty, Timothy E Sweeney, Elsie Gyang, and Nigam Shah. Learning statistical models of phenotypes using noisy labeled training data. Journal of the American Medical Informatics Association, (3(6))1166-11) 1173, 2016.
[6] Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05313. 2018
[7] Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In AAAI, 2020.
[8] Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with a corrupted reward channel. In IJCAI, pages 4705-4713, 2017.
[9] Joshua Romoff, Alexandre Piché, Peter Henderson, Vincent François-Lavet, and Joelle Pineau. Reward estimation for variance reduction in deep reinforcement learning. ICLR (Workshop). OpenReview.net, 2018.
[10] Robert Loftin, Bei Peng, James MacGlashan, Michael I. Littman, Matthew E Taylor, Jeff Huang, and David L Roberts. Learning something from nothing: Leveraging implicit human feedback strategies. In The 23rd IEEE international symposium on robot and human interactive communication, pages 607-612. IEEE, 2014.
(11) Michael Laskey, Jonathan Lee, Roy Fox, Anca D. Dragan, and Ken Goldberg. DART: noise injection for robust behavior cloning. In CoRL, volume 78 of Proceedings of Machine Learning Research, pages 143-156. PMLR, 2017.
[12] Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning, pages 6818-6827. PMLR, 2019.
[13] Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL:: behavior cloning via reinforcement learning with sparse rewards. In ICLR. OpenReview.net, 2020.

---

[14] Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations. 2020.
[15] Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, and Murray Campbell. Hybrid rein- forcement learning with expert state sequences. In AAAI, pages 3739-3746. AAAI Press, 2019.
[16] Lisa Lee, Benjamin Eysenbach, Ruslan Salakhutdinov, Shixiang, Gu, and Chelsea Finn. Weakly- supervised reinforcement learning for controllable behavior, 2020.
[17] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. JCML, abs/ 1910.03331, 2020.
[18] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pages 1 1196-1200 2013.
[19] Clayton Scott, Gilles Blanchard, Gregory Handy, Sara Pozzi, and Marek Flaska. Classification with asymmetric label noise: Consistency and maximal denoising. In COLT, pages 489-511, 2013.
[20] Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS, 2015.
[21] Sainbayar Sukhbaatar and Rob Fergus. Learning from noisy labels with deep neural networks. arXiv preprint 1rXiv:1406.2000 2(3):4, 2014.
[22] Brendan van Rooyen and Robert C Williamson. Learning in the presence of corruption. arXiv preprint arXiv: wrXXvv:154.000900 00091, 2015.
[23] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting
IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-46) 2015.
[24] Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via poosabprbbability estimation. /CML, pages 125-134, 2015.
[25] Zhaowei Zhu, Tongliang Liu, and Yang Liu. second-order approach to learning with instance- dependent label noise. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10113-10123, 2021.
[26] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411 2021.
[27] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. arXiv preprint XrXi:22022.02400, 2021.
[28] Zhaowei Zhu, Yiwen Song. and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. arXiv preprint arXiv:2102.05520 2021.
[29] Zhaowei Zhu, Jingxuan Zhu, Ji Liu, and Yang Liu. Federated bandit: A gossiping approach. In Abstract Proceedings of the 202/ ACM SICMETTRCSS/ntematiooo Conference on Measurement and Modeling of Computer Systems, pages 3-4, 2021.
[30] Jiaheng Wei and Yang Liu. When optimizing SfS-divergence is robust with label noise. In International Conference on Learning Representations, 2021.
[31] Yang Liu. Understanding instance-leved label noise: Disparate impacts and treatments. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 6725-6733 6735. PMLR, 2021.
[32] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1 ((11))88-9))) 97, 1991.
[33] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668, 2010.
12

---

[34] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning, In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635, 2011.
[35] Michael Laskey, Jonathan Lee, Roy Fox, Anca D. Dragan, and Ken Goldberg. DART: noise injection for robust imitation learning. In CoRL, volume 78 of Proceedings of Machine Learning Research, pages 143-156. PMLR, 2017.
[36] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. CoRR, abs/1604.07316, 2016.
[37] Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: Behavior Cloning via Reinforce- ment Learning with Sparse Rewards. 2019.
[38] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In IJCAI, pages 4950 4957. ijcai.org. 2018.
[39] Juarez Monteiro, Nathan Gavenski, Roger Granada, Felipe Meneguzzi, and Rodrigo Coelho Barros. Augmented behavioral cloning from observation. CoRR, 10022004.13529, 2020.
[40] Alessandro Giusti, Jerome Guzzi, Dan C. Ciresan, Fang Lin He, Juan Rodriguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jurgen Schmidhuber, Gianni Di Caro, Davide Scaramuzza, and Luca M. Gambardella. A Machine Learning Approach Visual Perception of Forest Trails for Mobile Robots. IEEE Robotics and Automation Letters, 1(2):66-6677, 2016.
[41] Niels Justesen and Sebastian Risi. Learning macromanagement in starcraft from replays using deep learning. In 2017 IEEE Conference on Computational Intelligence and Games (CIG), pages 162 169. IEEE, 2017.
[42] Wael Farag and Zakaria Saleh. Behavior cloning for autonomous driving using convolutional neural networks. 2018 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies, 3ICT 2018, 2018.
[43] Jialin Song. Ravi Lanka, Yisong Yue, and Masahiro Ono. Co-training for policy learning. In UAI, page 441. AUAI Press, 2019.
[44] Anirban Dasgupta and Arpita Ghosh. Crowdsoureed judgement elicitation with endogenous proficiency. In Proceedings of the 22nd international conference on World Wide Web, pages 319-330, 2013.
[45] Victor Shnayder, Arpit Agarwal, Rafael M. Frongillo, and David C. Parkes. Informed truthful- ness in multi-task peer prediction. EC, pages 179-196. ACM, 2016.
[46] Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew F. Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In ICCAI, pages 3352-3358. AAAI Press, 2015.
[47] Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Amold, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI, pages 3223-3230. AAAI Press, 2018
[48] Yang Liu. The importance of understanding instance-level noisy labels. arXiv preprint arX1v:2102.05336, 2021.
[49] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
[50] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. ICML,, volume 48, pages 1995-2003, 2016.
13

---

[51] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, pages 1057-1063 The MIT Press, 1999.
[52] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pages 279-292, 1992.
[53] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pages 2778- 2787. PMLR, 2017.
[54] Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. In International Conference on Learning Representations, 2021.
[55] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR (Poster), 2016.
[56] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning, In AAAI, pages 2094-2100, 2016.
[57] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, Abs//100.00971. 2015.
[58] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 02//1707.0634/ 2017.
[59] Jonathan Ho and Stefano Ermon. Generative adversarial behavior cloning. In Advances in Neural Information Processing Systems, pages 4572-4580, 2016.
[60] Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In ICML, pages 278-287. Morgan Kaufmann, 1999.
[61] John Asmuth, Michael L. Littman, and Robert Zinkov. Potential-based shaping in model-based reinforcement learning. In AAAI, pages 604-609, AAAI Press, 2008
[62] John Von Neumann and Oskar Morgenstem. Theory of games and economic behavior (com- memorative edition). Princeton university press, 2007.
[63] Tommi S. Jaakkola, Michael I. Jordan, and Satinder P. Singh. Convergence of stochastic iterative dynamic programming algorithms. In NIPS, pages 703-710, 1993.
[64] John N. Tsitsiklis. Asynchronous stochastic approximation and q-learning. Machine Learning. 16(3):185-200 1994.
[65] Michael J. Kearns and Satinder P. Singh. Finite-sample convergence rates for q-learning and indirect algorithms. In NIPS, pages 996-1002, 1998.
[66] Michael J. Kearns and Satinder P. Singh. Bias- variance error bounds for temporal difference updates. In COLT, pages 142-147, 2000.
[67] Michael J. Kearns, Yishay Mansour, and Andrew Y. Ng. A sparse sampling algorithm for near-optimal planning in large markov decision processes. In IJCAI, pages 1324-1231, 1999.
[68] Sham Machandranath Kakade. On the Sample Complexity of Reinforcement Learning. PhD thesis, University of London, 2003.
14


Reviews:
[{"review_id": "xXNzAP3BeYn", "reviewer": "Reviewer_aqkG", "summary": "The authors propose a general and unified framework to study policy learning without strong or high-quality supervision signals. This unified framework comprises, but not limited to, the main approaches for policy learning, including those using rewards signals in reinforcement learning (RL) and demonstrations through behavioral cloning (BC). The authors also propose PeerPL, a solution for efficient policy learning using only weak supervision signals, which evaluates an agent’s learning policy with noisy signals, treated as policy of an imperfect “peer agent”. The weak supervision signal comes in the form of correlation with the imperfect peer agent’s report, while the solution encourages generalization through penalizing identical reports.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The originality comes from that the proposed method differs to other noisy or weak supervision methods in which prior knowledge of noisy rates, i.e. the confusion matrices of noisy signals vs true signals, are not needed. Also, the evaluation function is designed so that identical behavior with the weak agent is auto penalized, by utilizing randomly sampled signal pairs. Related works are adequately cited in both Section 1 and Section 2.2. \nQuality: The intuition of how the penalization in correlation assessment is interpreted as overfitting in supervision learning sounds convincing. The theoretical proofs and experimental results look promising. The extension to co-training in partially observable environment using this framework with an RL and BC hybrid algorithm (PeerCT) is clear and well-organized, matching the overall flow of paper.\nClarity: The paper is clearly written, with toy examples which adequately illustrate the core idea of the proposed evaluation function. The experimental setup is also clear, with detailed descriptions on how “imperfect” expert demonstrations were obtained. The authors also provided some valid reasons on performance improvement from PeerPL such as implicitly encouraging higher exploration (presuming through penalizing evaluation function).\nSignificance: The proposal seems to provide an elegant formulation to address weak supervision signal systematically. The results of single-view co-training show that the proposed solution is feasible and perform significantly better in partially observable conditions compared to other co-training algorithms. The results could serve as good reference for subsequent works either in the field of co-training or weakly supervised policy learning.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion section for potential negative societal impacts present in the main paper. Though a short description of why this is excluded is included after the reference section, with the attached submission checklist.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "gduG3fAatLF", "reviewer": "Reviewer_Hvaz", "summary": "In this paper, the authors propose a general framework for learning sequential decision-making policies with noisy feedback. Two categories of noisy feedback are considered, noisy reward in reinforcement learning and noisy demonstrations in behavioral cloning. The concept of “correlated agreement” (CA) is proposed as a regularization mechanism for a policy to learn from the noisy feedback without overfitting to it. An additional policy co-training setting is also considered. Empirical evaluations control games and Atari show that CA is effective in enabling policy learning from noisy feedback.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-rebuttal comments: thanks to the authors for providing a detailed response. All my questions are answered satisfactorily. My score will stay the same.\n\nOriginality: The application of CA to policy learning from noisy feedback is novel. The discussion of related works is adequate. \n\nQuality: This paper is technically sound. I appreciate the theoretical analyses for PeerRL and PeerBC, which show convergence to optimal policy with CA. I have the following questions.\n\n1. In the paragraph on BC with weak demonstration (starting at line 120), from which state distribution are the weak demonstrations sampled? I understand the noisy actions are the result of flipping from the true expert action. Are the states generated by following the true expert actions or the noisy ones?\n\n2. I don’t quite understand the point of the toy example given in section 3.1 (starting at line 157). It is true the evaluation score is lower with CA, but does that mean that there is a better policy than the fully memorized one? \n\n3. I would like to understand what, if any, obstacle exists to extend the noisy model from discrete to continuous. If the reward/action flipping follows a continuous distribution, is any theoretical result impacted?\n\nClarity: This paper is well-written and easy to read. Good job!\n\nSignificance: One limitation the authors acknowledged is that the empirical evaluations are not real-world. I think the significance would improve quite a bit with the inclusion of an example of a real robotic system, for instance. Nevertheless, I think the novelty of CA, the theoretical contributions and the strong performance on the synthetic environments provide enough significance.\n\nMinor comment: This recent ICLR 2021 paper should be included in the related works discussion.\nBehavioral Cloning from Noisy Demonstrations, Fumihiro Sasaki, Ryota Yamashina, ICLR 2021\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "TDqaQFVk5ta", "reviewer": "Reviewer_58fG", "summary": "This paper proposes a method for RL and imitation learning in the setting where the reward function or expert actions are corrupted with noise. The main idea is to regularize the learned policy to have *high* loss on unaligned state-action-reward examples. Experiments show that the proposed method outperforms baselines that either learn directly using the noisy rewards/actions or learn an auxiliary function that predicts the mean reward/action.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**:\nThe proposed method is (to the best of my knowledge) novel. The paper does good job describing the differences with the most related prior work [Which learns to predict a non-noisy version of the rewards/actions, and then uses them for learning]. My understanding is that the key benefit of the the proposed approach is that it doesn't require knowledge about the noise distribution and doesn't require a separate estimator of the mean reward/action.\n\n**Quality**:\n* **Theory**: The main technical claims about sample complexity seem correct (I quickly read through the proofs in the Appendix), building directly on top of prior work. My main concern about the technical claims is that they don't seem to motivate the proposed method. That is, it's unclear if the Theorems show that the proposed method is more sample efficient than either (1) directly learning using noisy rewards/actions or (2) learning using the empirical mean estimates for the rewards/actions. I would highly encourage the authors to discuss this. Said in other words, while it's clear why the proposed method should have the same solution as learning directly on the noisy examples, it's unclear why injecting more noise (but adding a random term to the loss function) should result in faster learning.\n* **Experiments**: The paper did a great job comparing both the RL setting and the imitation learning setting, comparing against the most similar prior work. The experiments in the RL setting are limited to two very simple tasks. I'd be curious to see how the method scales to more complex tasks. One suggestion: I would highly recommend including the baseline results on the Pendulum task; omitting baseline results on the tasks where the baseline performs as well as the proposed method is concerning. The imitation learning results on Atari are also nice. I found the co-training experiments quite hard to follow. The discussion of the benefits of PeerRL (L292) seems intriguing, but I'm not sure if I believe the three offered explanations. For example, the argument about human rewards being imperfect seems to contradict the claim that the proposed method optimizes whatever reward function is given. Similarly, the claim that the proposed method decreases variance seems suspicious because (a) it seems like it actually *increases* the variance of the gradient estimator and (b) this same logic would suggest that the baseline from [7] should outperform the proposed method.\n\n**Clarity**:\nThe paper is generally well written and reads quite well. The related work section is well organized and discusses relevant prior work. The paper does a great job of introducing and explaining notation before using it. See list of minor writing comments at the end of the review. One high-level suggestion is to spend more time explaining the problem settings. Since the problem setting considered in this paper is different from most other RL papers, it's be good to explain why it is useful and interesting. If possible, I'd highly recommend adding an experiment that motivates this setting.\n\n**Significance**:\nI am on the fence about the significance of the paper. On the one hand, the proposed method is quite simple and performs well in a variety of experiments. On the other hand, it's unclear if the analysis actually explains *why* the proposed method is performing better (see previous comment), and it's unclear whether the problem being solved is practically relevant.\n\n\n**Minor comments**:\n* L16 \"breakthrough\" -> \"breakthroughs\"\n* L16 \"these techniques\" -- Which techniques?\n* L19 \"expensive to obtain in practice\" -- Add a citation.\n* L20 \"thus\" -- Grammar error.\n* L21 \"are often imperfect due to limited resources or environment noise\" -- I'm unsure if this claim is actually correct. Noise in the environment does not affect the distribution $p(a | s)$, which is all we need for behavior cloning.\n* L22 - L29 -- I found this paragraph confusing because it was unclear what \"weak supervision\" was referring to. I'd highly recommend providing *real-world* examples of this problem setting.\n* L26 \"imperfect solution\" -> \"solution\"\n* L28 \"Then as a response, we...\" -> \"We ...\"\n* L30 - L31 -- I found this sentence confusing.\n* L38 \"weakness of the supervision\" -- Unclear what this means.\n* Generally try to avoid adjectives and adverbs (e.g., \"intensive analysis\" and \"jointly demonstrate\" and \"mainly three-folds\").\n* At the end of the introduction, I was still unsure what problem was being solved.\n* L57 \"in-efficient\" -> \"inefficient\"\n* L58 \"an RL algorithm\" -- Add a citation.\n* L62 \"reasoning\" -> \"reasoning about\"\n* L81 \"unify\" -- Unify with what?\n* L104 \"both worlds\" -- Unclear what \"worlds\" refers to here.\n* L107 - L111 -- I found these few sentences hard to follow.\n* L114 - L127 -- These definitions are great, and very helpful!\n* L128 -- From reading this paragraph, it was hard to tell whether $C$ was known or not known.\n* L132 \"taken policy\" -- Unclear what this means.\n* L136 - L137 -- I found the definition of Eva() pretty confusing. I might recommend explaining the behavior cloning case first without introducing Eva(), and later describing how these results can be extended to the RL setting.\n* L137 $J(\\pi)$ -- I found it confusing that the same notation, $J(\\pi)$, was defined differently in three different places: L92, L98, L137.\n* L146 \"correlated agreement\" -- No need for the quotes.\n* L148 \"instances\" -- Unclear what an \"instance\" is.\n* L151 - L156 -- This intuition is pretty clever!\n* L157 - L164 -- This example is somewhat hard to follow. Consider explaining it with less notation, perhaps with pictures.\n* L165 \"consolidate our ...\" -- Unclear what this means.\n* L174 \"with agreement check\" -- Unclear what this means.\n* Lemma 1 -- This result is fairly elementary and could be moved to the Appendix.\n* L196 - L199 -- I found these sentences a bit confusing.\n* L207 \"we found that\" -- Unclear if this is a theoretical or empirical result.\n* Figures -- Please make the figures colorblind-friendly.\n* L276 - L277 -- This sentence is pretty wordy and can be greatly shortened.\n* L278 \"other baselines\" -- I would highly recommend adding a separate section that describes the baselines.\n* L279 $e=0.4$ -- It could be worth noting that $e \\le 0.5$. I initially thought $e \\in [0, 1]$ and assumed that 0.4 was a rather small value.\n* L281 \"which again ... \" -- This can be cut.\n* L282 \"more challenging\" -- Inverted pendulum is not really more challenging than CartPole.\n* L286 \"experiment\" -- Grammar error?\n* L286 \"uniform noise\" -- Why is this a reasonable problem setting? Is this similar to some real-world tasks?\n* L288 \"biased noise\" -- Why is the noise biased?\n* L300 - L301 -- I didn't understand this sentence.\n* L307 \"three seeds\" -- Is the same expert policy used for each seed? [That's fine, I just wanted to clarify.]\n* L315 \"GAIL and SQIL\" -- These methods are designed for the setting where the agent can interact with the environment. Is this the setting in which they were used?\n* L334 \"which again ...\" -- This sentence can be cut.\n\n\n-----------------------------\n**Review decision**: Overall, the proposed method seems simple and empirically seems to work well. The paper is generally well written. I have two main concerns with the paper. First, it is unclear *why* the proposed method should outperform the baselines; the analysis doesn't seem to answer this question. Second, it is unclear if the paper is actually solving a useful problem. While this problem has been studied in prior work, the problem setting differs from the standard RL setting. I would encourage the authors to address these two concerns in the rebuttal.\n\n**Update after discussion**: Thank you for the discussion and additional experiments studying the \"discreteness\". I think I finally understand where the \"magic\" is coming from. At this point, my concerns about (1) why the method should work and (2) why this is a useful problem setting have been well addressed. I have increased my review to score = 7. In the final version of the paper, I would highly encourage the authors to include many of the points from the discussion", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The concluding paragraph does a great job discussing limitations and areas for future work. I did not see any discussion of societal impacts, either positive or negative.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 - 20 (including discussion)"}, "extra_text": {}}, {"review_id": "S0WWa3q3MiI", "reviewer": "Reviewer_xvWU", "summary": "This paper proposed a framework to improve the efficacy of sensorimotor control learning. The main idea consists of augmenting the training loss with a specifically designed term that avoids the policy to \"overfit\" to noise present in the reward (for RL) or in the demonstrations (for behavioral cloning).  This extra term aims to minimize the probability that when randomly associating rewards and state-action pairs, the policy performs poorly. This idea, borrowed by the theory of correlated agreement, should improve the policy performance when the supervision (in terms of demonstration or reward) is corrupted by noise. Experiments show that the proposed framework achieves comparable or better results than prior works.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed framework appears to build upon existing work on correlation agreement ([34] in paper), but generalizing such methods to the problem of decision making. Overall, I think that the strong connection with prior work was not adequately discussed. Indeed, it is unclear what are the fundamental differences with existing methods (if it is just an application is not generally a problem but should be clearly stated).\n\nIn my opinion, the work presents an interesting approach to a very important and general problem (learning policies with noisy supervision). However, the method and the experiments do not completely back up all the claims. More specifically:\n\n- The proof that the proposed method (asymptotically) converges to the optimal policy despite noisy rewards seems a bit hand-wavy and not completely rigorous. The remark on Lemma 1 (line 193) shows the noisy supervision only differs up to a constant with respect to the proposed peer supervision method. Clearly, this constant can have a strong effect on the final performance, but how much should the noise be? This question also raises the question of whether under the assumption of a large enough sample size and constant noise even the policy without any peer reward would converge to the optimum. Proof A1 in the appendix seems to support this hypothesis since only an affine transformation of the reward (independent of the noise level) is required to guarantee convergence (line 580).\n- The experimental results are overall extensive, but comparison with previous work is only partial and sometimes even omitted. Specifically, it would have been interesting to see how the approach would perform in comparison to [7], which exactly aims to solve the same problem (learning from noisy rewards). While this comparison is present for the simplest of the demonstrators (the pendulum) it was omitted for the more challenging ones, e.g. the atari games. Strong baselines on imitation learning, e.g. dagger or dart, are also omitted from the experiments. The evaluation of atari games is by itself a bit strange since no noise in the reward was added (line 326). The paper claims that there is anyway noise given by the real-world application but, to the best of my knowledge, no noise is present in the simulation and the reward observed exactly match the real rewards (I might be wrong on this last point, and clarification in the text could solve the issue).\n\nOverall, I am convinced that the paper is about an important and timely problem, i.e. learning a sensorimotor policy in the presence of noise in the training signal. This problem is relevant for a very large community, including machine learning, robotics, and computer vision. In addition, I really appreciate that an effort was done to cover under the same umbrella a lot of different learning frameworks (RL, Imitation Learning, Co-training). However, I believe that the framework is still not up to its claims and both clarifications in the theoretical part and more extensive experiments could significantly strengthen the contribution.\n\nOn a less important side, I think that the current presentation, despite being enough to enable reproducibility, lacks clarity and is sometimes difficult to follow. The main problem is the fact that many terms are used to indicate similar things (e.g. Figure 1, why is the loss function Y indicated as a peer agent? This could cause confusion with the policy co-training part). Similarly, it would be nice to always explicitly state where in the appendix the proofs of the lemma/theorems can be found (this is currently not the case of theorem 2). Finally, it would be nice to put the comparison with the strong imitation learning baselines (GAIL and SQIL) in the main paper.\n\n----------------\n\nIssue solved after rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and possible societal impacts have been properly discussed in the checklist. It would be nice to move some sentences to the main paper, just to have a complete picture in the discussion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was an exemplary case of the value of discussion between the reviewers and the authors. After multiple clarifications, additional experiments and even a code snippet examples, the reviewers agreed that the paper provides a valuable contribution to a problem of weak supervision in RL. I encourage the authors to address reviewers' extensive comments and update the paper for the final revision.
Decision: Accept (Poster)