Policy Learning Using Weak Supervision
Year: 2022

Abstract:
Most existing policy learning solutions require the learning agents to receive highquality supervision signals such as well-designed rewards in reinforcement learning (RL) or high-quality expert demonstrations in behavioral cloning (BC). These quality supervisions are usually infeasible or prohibitively expensive to obtain in practice. We aim for a unified framework that leverages the available cheap weak supervisions to perform policy learning efficiently. To handle this problem, we treat the “weak supervision” as imperfect information coming from a peer agent, and evaluate the learning agent’s policy based on a “correlated agreement” with the peer agent’s policy (instead of simple agreements). Our approach explicitly punishes a policy for overfitting to the weak supervision. In addition to theoretical guarantees, extensive evaluations on tasks including RL with noisy rewards, BC with weak demonstrations, and standard policy co-training show that our method leads to substantial performance improvements, especially when the complexity or the noise of the learning environments is high.

heading: 1 Introduction
text: Recent breakthroughs in policy learning (PL) open up the possibility to apply reinforcement learning (RL) or behavioral cloning (BC) in real-world applications such as robotics [1, 2] and self-driving [3, 4]. Most existing works require agents to receive high-quality supervision signals, e.g., reward or expert demonstrations, which are either infeasible or expensive to obtain in practice [5, 6].
The outputs of reward functions in RL are subject to multiple kinds of randomness. For example, the reward collected from sensors on a robot may be biased and have inherent noise due to physical conditions such as temperature and lighting [7, 8, 9]. For the human-defined reward, different human instructors might provide drastically different feedback that leads to biased rewards [10]. Besides, the demonstrations by an expert in behavioral cloning (BC) are often imperfect due to limited resources and environment noise [11, 12, 13]. Therefore, learning from weak supervision signals such as noisy rewards [7] or low-quality demonstrations produced by untrustworthy expert [12, 14] is one of the outstanding challenges that prevents a wider application of PL.
Although some works have explored these topics separately in their specific domains [7, 15, 14, 16], there lacks a unified solution for robust policy learning in imperfect situations. Moreover, the noise model as well as the corruption level in supervision signals is often required. To handle these challenges, we first formulate a meta-framework to study RL/BC with weak supervision and call it weakly supervised policy learning. Then we propose a theoretically principled solution, PeerPL, to perform efficient policy learning using the available weak supervision without requiring noise rates.
Our solution is inspired by peer loss [17], a recently proposed loss function for learning with noisy labels but does not require the specification of noise rates. In peer loss, the noisy labels are treated as a peer agent’s supervision. This loss function explicitly punishes the classifier from simply agreeing with the noisy labels, but would instead reward it for a “correlated agreement" (CA). We adopt a
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
similar idea and treat the “weak supervision” as the noisy information coming from an imperfect peer agent, and evaluate the learning agent’s policy based on a “correlated agreement” (CA) with the weak supervision signals. Compared to standard reward and evaluation functions that encourage simple agreements with the supervision, our approach punishes “over-agreement" to avoid overfitting to the weak supervision, which offers us a family of solutions that do not require prior knowledge of the corruption level in supervision signals.
To summarize, the contributions in the paper are: (1) We provide a unified formulation of the weakly supervised policy learning problems; (2) We propose PeerPL, a new way to perform policy evaluation for RL/BC tasks, and demonstrate how it adapts in challenging tasks including RL with noisy rewards and BC from weak demonstrations; (3) PeerPL is theoretically guaranteed to recover the optimal policy, as if the supervision are of high-quality and clean. (4) Experiment results show strong evidence that PeerPL brings significant improvements over state-of-the-art solutions. Code is online available at: https://github.com/wangjksjtu/PeerPL.

heading: 1.1 Related Work
text: Learning with Noisy Supervision Learning from noisy supervision is a widely explored topic. The seminal work [18] first proposed an unbiased surrogate loss function to recover the true loss from the noisy label distribution, given the knowledge of the noise rates of labels. Follow-up works offered ways to estimate the noise level from model predictions [19, 20, 21, 22, 23, 24, 25, 26, 27] or label consensuses of nearby representations [28]. Recent works also studied this problem in sequential settings including federated bandit [29] and RL [7]. The former work assumes the noise can be offset by averaging rewards from multiple agents. [7] designs a statistics-based estimation algorithm for noise rates in observed rewards, which can be inefficient especially when the state-action space is huge. Moreover, the error in the estimation can accumulate and amplify in sequential problems. Inspired by recent advances of peer loss [17, 30, 31], our solution is able to recover true supervision signals without requiring a priori specification of the noise rates.
Behavioral Cloning (BC) Standard BC [32, 33] tackles the sequential decision-making problem by imitating the expert actions using supervised learning. Specifically, it aims to minimize the one-step deviation error over the expert trajectory without reasoning about the sequential consequences of actions. Therefore, the agent suffers from compounding errors when there is a mismatch between demonstrations and real states encountered [33, 34, 35]. Recent works introduce data augmentations [36] and value-based regularization [37] or inverse dynamics models [38, 39] to encourage learning long-horizon behaviors. While being simple and straightforward, BC has been widely investigated in a range of application domains [40, 41] and often yields competitive performance [42, 37]. Our framework is complementary to the current BC literature by introducing a learning strategy from weak demonstrations (e.g., noisy or from a poorly-trained agent) and provides theoretical guarantees on how to retrieve clean policy under mild assumptions [43].
Correlated Agreement In [44, 45], a correlated agreement (CA) type of mechanism is proposed to evaluate the correlations between agents’ reports. In addition to encouraging a certain agreement between agents’ reports, CA also punishes over-agreement when two agents always report identically. Recently, [17, 30, 25] adapt a similar idea to noisy label learning thus offloading the burdens of estimating noise rates. We consider a more challenging sequential decision-making problem and study the convergence rates under noisy supervision signals.

heading: 2 Policy Learning from Weak Supervision
text: We begin by reviewing conventional reinforcement learning and behavioral cloning with clean supervision signals. Then we introduce the weak supervision problem in policy learning and define two concrete instantiations: (1) RL with noisy reward and (2) BC using weak expert demonstrations.

heading: 2.1 Overview of Policy Learning
text: The goal of policy learning (PL) is to learn a policy ⇡ that the agent could follow to perform a series of actions in a stateful environment. For reinforcement learning, the interactive environment is characterized as an MDP M = hS,A,R,P, i. At each time t, the agent in state st 2 S takes an action at 2 A by following the policy ⇡ : S ⇥ A ! R, and potentially receives a reward r(st, at) 2 R. Then the agent transfers to the next state st+1 according to a transition probability
function P . We denote the generated trajectory ⌧ = {(st, at, rt)}Tt=0, where T is a finite or infinite horizon. RL algorithms aim to maximize the expected reward over the trajectory ⌧ induced by the policy: J clean(⇡) = E(st,at,rt)⇠⌧ [ PT t=0 t rt], where 2 (0, 1] is the discount factor.
Another popular policy learning method is behavioral cloning. Let ⇡(·|s) denotes the distribution over actions formed by ⇡, and ⇡(a|s) be the probability of choosing action a given state s and policy ⇡. The goal of BC is to mimic the expert policy ⇡E through a set of demonstrations DE = {(si, ai)}Ni=1 drawn from a distribution DE , where (si, ai) is the sampled state-action pair from the expert trajectory and ai ⇠ ⇡E(·|si) Then training a policy with standard BC corresponds to maximizing the following log-likelihood: J clean(⇡) = E(s,a)⇠DE [log ⇡(a|s)].
In both RL and BC, the learning agent receives supervision through either the (clean) reward r by interacting with environments or the expert policy ⇡E as observable demonstrations. Consider a particular policy class ⇧, the optimal policy is then defined as ⇡⇤ = argmax⇡2⇧ J clean(⇡): ⇡⇤ obtains the maximum expected reward over the horizon T in RL and ⇡⇤ corresponds to the clean expert policy ⇡E in BC. In practice, one can also combine both RL and BC approaches to take advantage of both learning paradigm [46, 47, 15, 43]. Specifically, a recent hybrid framework called policy co-training [43] will be considered in this paper.

heading: 2.2 Weak Supervision in Policy Learning
text: The weak supervision signal eY could be noisy reward r̃ for RL or noisy action ã from an imperfect expert policy ⇡̃E for BC, which are noisy versions of the corresponding high-quality supervision signals. See more details below.
RL with Noisy Reward Consider a finite MDP fM = hS,A,R, F,P, i with noisy reward channels [7], where R : S ⇥A ! R, and the noisy reward r̃ is generated following a certain function F : R ! eR. Denote the trajectory a policy ⇡✓ generates via interacting with fM as ⌧̃✓. Assume the reward is discrete and has |R| levels. The noisy reward can be characterized via a unknown matrix CRL|R|⇥|R|, where each entry cj,k indicates the flipping probability for generating a possibly different outcome: cRLj,k = P (r̃t = Rk|rt = Rj). We call r and r̃ the true reward and noisy reward.
BC with Weak Demonstration Instead of observing the true expert demonstration generated according to ⇡E , denote the available weak demonstrations by {(si, ãi)}Ni=1, where ãi is is the noisy expert action drawn according to a random variable ãi = ⇡̃E(si) ⇠ ⇡̃E(·|si), each state-action pair (si, ãi) is sampled from distribution eDE . Note there may exist two randomness factors in getting ãi: uncertainty in true policy ⇡E and noise from imperfect policy ⇡̃E . In particular, we do not consider the former randomness in theoretical analyses: given the output distribution ⇡E(·|si), only one deterministic action ⇡E(si) is taken by expert. This is because with uncertainty in true expert actions, it is hard to distinguish a clean case with true expert actions from the weak supervision case without addition knowledge. Similar assumptions are also adopted in [23, 28]. The noisy action is modeled by a unknown confusion matrix CBC|A|⇥|A|, where each entry cj,k indicates the flipping probability for taking a sub-optimal action that differs from ⇡E(s): cBCj,k = P(⇡̃E(s) = Ak|⇡E(s) = Aj), Ak and Aj denote the k-th and the j-th action from the action space A. In the above definition, we assume the noisy action ãi is independent of the state s given the deterministic expert action ⇡E(s),
i.e., P(ãi|⇡E(si)) = P(ãi|si,⇡E(si)). We aim to recover ⇡⇤ as if we were able to access the quality expert demonstration ⇡E instead of ⇡̃E .
Knowledge of C Recall C: CRL|R|⇥|R| or C BC |A|⇥|A| is unknown in practice. While recent works estimate this matrix [26, 23, 28] in supervised classification problems, it is still challenging to generalize them to a sequential setting [7]. When C is not perfectly estimated, the estimation error of C may lead to unexpected state-action pairs then the error of reward estimates will be accumulated in sequential learning. Besides, estimating C involves extra computation burden. In contrast, our method gets rid of the above issues since it is free of any knowledge of C and leads to more robust policy learning algorithms.
Learning Goal With full supervision, both RL and BC can converge to the optimal policy ⇡⇤. However, when only weak supervision is available, with an over-parameterized model such as a deep neural network, the learning agent will easily memorize the weak supervision and learn a biased policy [48]. In our meta framework, instead of converging to any biased policy, we focus on learning the optimal policy ⇡⇤ with only a weak supervision sequence denoted as {(st, at), eYt}Tt=1 (RL) or {(si, ai), eYi}Ni=1 (BC).

heading: 3 PeerPL: Weakly Supervised PL via Correlated Agreement
text: To deal with weak supervision in PL, we propose a unified and theoretically principled framework PeerPL. We treat the weak supervision as information coming from a “peer agent”, and then evaluate the policy using a certain type of “correlated agreement” function between the learning policy and the peer agent’s information.

heading: 3.1 A Unified Evaluation Function
text: We use an evaluation function Eva⇡((si, ai), eYi) to evaluate a taken policy ⇡ at agent state (si, ai) using the weak supervision eYi. For RL, Eva⇡ is the instance-wise measure (negative loss) for different RL algorithms, which is a function of the noisy reward r̃ received at (si, ai). In the BC setting, Eva⇡ is the loss to evaluate the action ai taken by the agent given the expert’s demonstration ãi. Note that the larger the Eva⇡ is at state (si, ai), the better it follows the supervision eYi. Specifically, we have
EvaRL⇡ (s, a), r̃ = ` ⇡, (s, a, r̃) (RL) and EvaBC⇡ (s, a), ã = log ⇡(ã|s) (BC),
where the RL loss function ` can be temporal difference error [49, 50] or the policy gradient loss [51]. Furthermore, we let J (⇡) denote the function that evaluates policy ⇡ under a set of state action pairs with weak supervision sequence {(si, ai), eYi}Ni=1, i.e.,
J(⇡) = E(s,a)⇠⌧ [Eva⇡((s, a), eY )].
Then the goal of weakly supervised policy learning is to recover the optimal policy ⇡⇤ as if we receive clean supervision Y . Note that directly maximizing J(⇡) might result in sub-optimal performance due to the weak supervisions. The above unified notations are only for better delivery of our framework and we still treat PL as a sequential decision problem.

heading: 3.2 Overview of the Idea: Correlated Agreement with Weak supervision
text: We first present the general idea of our PeerPL framework using a concept named correlated agreement (CA). For each weakly supervised sample ((si, ai), eYi), we randomly sample (with replacement) two other peer samples indexed by j and k. Then we take the state-action pair (sj , aj) of sample j and the supervision signal eYk of sample k, and evaluate ((si, ai), eYi) as follows:
CA with Weak Supervision: Eva⇡ (si, ai), eYi
Eva⇡ (sj , aj), eYk .
This operation is illustrated in Figure 1. We further show intuitions and a toy example below.
Intuition The first term above encourages an “agreement” with the weak supervision (that a policy agrees with the corresponding supervision), while the second term punishes a “blind” and “over” agreement that happens when the agent’s policy always matches with the weak supervision even
on randomly paired traces (noise). The randomly paired instances j, k help us achieve this check. Note our mechanism does not require the knowledge of CRL|R|⇥|R| nor C BC |A|⇥|A|, and offers a priorknowledge free way to learn effectively with weak supervision. Toy Example Consider a toy BC setting where the policy fully memorizes the weak supervision and outputs the same sequence of actions given the same sequence of states, i.e.,
Weak-supervision: ã1 = ã2 = ã3 = 1, ã4 = 0; Outputs: a1 = a2 = a3 = 1, a4 = 0.
Let Eva⇡((si, ai), ãi) = 1 if the policy output agrees with the weak demonstration (ai = ãi), and 0 otherwise. When the policy fully memorizes weak supervisions, we have:
Without CA: E[Eva⇡((si, ai), ãi)] = 1, With CA: E[Eva⇡((si, ai), ãi) Eva⇡((sj , aj), ãk)] = 0.375,
where 0.375 = 1 (0.752 + 0.252) is obtained by considering the probability of randomly paired aj and ãk matching each other. The above example shows that a full agreement with the weak supervision will instead be punished.
In what follows, we showcase two concrete implementations: PeerRL (peer reinforcement learning) and PeerBC (peer behavioral cloning). We provide algorithms and theoretical guarantees under weak supervisions.

heading: 4 PeerRL: Peer Reinforcement Learning
text: We propose the following objective function to punish the over-agreement of parametric policy ⇡✓ based on CA:
J RL(⇡✓) = E h EvaRL⇡ (si, ai), r̃i i ⇠ · E h EvaRL⇡ (sj , aj), r̃k i , (1)
where EvaRL⇡ (s, a), r̃ = ` ⇡✓, (s, a, r̃) . (2)
In (1), the first expectation is taken over (si, ai, r̃i) ⇠ ⌧̃ and second one is taken over (sj , aj , r̃j) ⇠ ⌧̃ , (sk, ak, r̃k) ⇠ ⌧̃ , where ⌧̃ is the trajectory specified by the noisy reward function r̃. Recall j, k denote two randomly and independently sampled instances. Loss function ` depends on the employed RL algorithms, e.g., temporal difference error [49, 50] or the policy gradient loss [51]. The learning sequence is encoded in ⇡. The objective JRL(⇡) represents the accumulated peer RL reward. Parameter ⇠ 0 balances the penalty for blind agreements induced by CA.

heading: 4.1 Peer Reward
text: In what follows, we consider the Q-Learning [52] as the underlying learning algorithm where `(⇡✓, (s, a, r̃)) = r̃(s, a) and demonstrate that the CA mechanism provides strong guarantees for Q-Learning with only observing the noisy reward. For clarity, we define peer RL reward:
Peer Reward: r̃peer(s, a) = r̃(s, a) ⇠ · r̃0,
where r̃0 ⇡sample ⇠ {r̃(s, a)|s 2 S, a 2 A} is a reward sampled over all state-action pairs according to a fixed policy ⇡sample. Note the sampling policy ⇡sample is independent of ⇡ and the choice of ⇡sample does not affect our theoretical results. We adopt a random sampling strategy in practice. Parameter ⇠ 0 balances the noisy reward and the punishment for blind agreement (with r̃0). We set ⇠ = 1 (for binary case) in the following analysis and treat each (s, a) equally when sampling r̃0. In experiments, we find r̃peer is not sensitive to the choice of ⇠ and keep ⇠ constant for each run.
Robustness to Noisy Rewards Now we show peer reward r̃peer offers us an affine transformation of the true reward in expectation, which guarantees that our PeerRL algorithm converges to ⇡⇤. Consider the binary reward setting (r+ and r ) and denote the error in r̃ as e+ = P(r̃ = r |r = r+), e = P(r̃ = r+|r = r ) (a simplification of CRL|R|⇥|R| in the binary setting). Lemma 1. Let r 2 [0, Rmax] be a bounded reward, ⇠ = 1. Assume 1 e e+ > 0. We have:
E[r̃peer(s, a)] = (1 e e+) · E[rpeer(s, a)] = (1 e e+) · E[r(s, a)] + const , where rpeer(s, a) = r(s, a) r0 is the peer RL reward when observing the true reward r, and r0 is the true reward corresponding to r̃0.
Lemma 1 shows that by subtracting the peer penalty term r̃0 from noisy reward r̃(s, a), r̃peer(s, a) recovers the clean and true reward r(s, a) in expectation. Based on Lemma 1, we prove in Theorem A1 that the Q-learning agent will converge to the optimal policy w.p.1 with peer rewards without requiring any knowledge of the corruption in rewards (CRL|R|⇥|R|, as opposed to previous work [7] that requires such knowledge). Moreover, we prove in Theorem A2 that to guarantee the convergence to ⇡⇤, the number of samples needed for our approach is no more than O(1/(1 e e+)2) times of the one needed when the RL agent observes true rewards perfectly (see Appendix A).
Extension Even though we only present an analysis for the binary case for Q-Learning, our approach is rather generic and is ready to be plugged into modern DRL algorithms. We provide multi-reward extensions, implementations with DQN [49] and policy gradient [51] in Appendix A.

heading: 4.2 Why does Peer Reward Work?
text: Compared with noisy reward, proposed peer variant is a less biased estimation of true reward (Benefit1). On the other hand, PeerRL helps break the unstable “tie” states, which might encourage the agent to explore in the early stage [53] (Benefit-2).
Benefit-1: PeerRL reduces the bias We highlight that the biased noise model considered is rather generic, departing from the previous noise assumption such as zero-mean Gaussian noise [8, 9]. In zero-mean noise models, the major focus is on variance reduction so adding the random term r̃0 increases the variance thus resulting in worse estimation. However, in the discrete biased noise model [18], bias correction also plays an important role especially the noise rate is high [7].
Similar to peer reward (Lemma 1), the expectation of the noisy reward writes as: E[r̃(s, a)] = (1 e e+)E[r(s, a)] + e r+ + e+r = (1 e e+)E[r(s, a)] + const. But the constant in peer reward has less effect on the true reward r, especially when the noise rate is high. To see this:
noisy reward: E[r̃(s, a)] = ⌘ · ⇣ E[r(s, a)] + e+1 e e+ r + e 1 e e+ r+ ⌘ ,
peer reward: E[r̃peer(s, a)] = ⌘ · (E[r(s, a)] (1 ppeer)r ppeerr+),
where ⌘ = 1 e e+ > 0, ppeer 2 [0, 1] denotes the probability that a sample policy sees a reward r+ overall. Since the magnitude of noise terms
e 1 e e+ and e+ 1 e e+ can potentially become
much larger than 1 ppeer and ppeer in a high-noise regime, e 1 e e+ r+ + e+ 1 e e+ r will dilute the informativeness of E[r(s, a)]. On the contrary, E[r̃peer(s, a)] contains a moderate constant noise thus maintaining more useful training signals of the true reward in practice. In summary, although peer reward (similar to the surrogate reward in previous literature [7]) increases the variance (no free-lunch), it will lead to a better estimation of the true reward due to lower bias.
Correct Tie Incorrect
baseline 54.6% 5.6% 39.8%
PeerRL 58.0% 0.3% 41.7% Benefit-2: PeerRL helps break ties For RL, “tie” states indicate that the rewards for different states are the same, which are less informative as they neither serve as positive nor negative examples. Due to the discrete nature of the noise model, adding a randomly sampled penalty term helps break the tie states and treats them as either positive examples or negative examples such that it can encourage exploration in the early stage, which has similar intuitions to some RL exploration works [53]. It has also been demonstrated that reducing the uncertainty, a.k.a. pushing confident predictions, makes the learning robust to weak-supervisions in supervised learning [17, 54]. On the other hand, it is known that positive examples are sparse yet important in RL. To leverage these useful experiences sufficiently, experience replay [55, 56] is invented to store and up-sample the positive examples for faster convergence. Tie breaking potentially provides an alternative way to access more positive examples. To illustrate tie-breaking phenomenon when using peer reward, we consider a two-state Markov process (no actions) with bounded Gaussian noise and see how well we could infer which state was better by correcting the reward signals. We collect two observations for each state and conduct 104 trials to calculate the success rate of inferring which state has larger returns (“correct” in the Table). As we can see, PeerRL exploits the "discreteness" of the reward thus breaking ties to obtain more examples with good-quality supervision. More examples on varied noise models (bounded continuous noise, discrete noise) are deferred to Appendix B.
v0 with true reward (r) , noisy reward (r̃) , surrogate reward [7] (r̂) , and peer reward (r̃peer, ⇠ = 0.2) .
dulum with true reward (r) , noisy reward (r̃) , and peer reward (r̃peer, ⇠ = 0.2) .

heading: 5 PeerBC: Peer Behavioral Cloning
text: Similarly, we present our CA solution in the setting of behavioral cloning (PeerBC). In BC, the supervision is given by the weak expert’s noisy trajectory. At each iteration, the agent learns under weak supervision ã, and the training samples are generated from the distribution eDE determined by the weak expert. The EvaBC⇡ function in BC evaluates the agent policy ⇡✓, parametrized by ✓, and the weak trajectory {(si, ãi)}Ni=1 using `(⇡✓, (si, ãi)), where ` is an arbitrary classification loss. Taking the cross-entropy for instance, the objective of PeerBC is:
J BC(⇡✓) = E h EvaBC⇡ (si, ai), ãi i ⇠ · E h EvaBC⇡ (sj , aj), ãk i , (3)
where EvaBC⇡ s, a), ã = ` ⇡✓, (s, ã) = log ⇡✓(ã|s). (4)
In (3), the first expectation is taken over (si, ãi) ⇠ eDE , ai ⇠ ⇡(·|si) and the second is taken over (sj , ãj) ⇠ eDE , aj ⇠ ⇡(·|sj), (sk, ãk) ⇠ eDE , ak ⇠ ⇡(·|sk). Again, the second EvaBC⇡ term in JBC serves the purpose of punishing over-agreement with the weak demonstration. Similarly, ⇠ 0 is a parameter to balance the penalty for blind agreements.
Robustness to Noisy Demonstrations We prove that the policy learned by PeerBC converges to the expert policy when observing a sufficient amount of weak demonstrations. We focus on the binary action setting for theoretical analyses, where the action space is given by A = {A+, A } and the weakness or noise in the weak expert ⇡̃E is quantified by e+ = P(⇡̃E(s) = A |⇡E(s) = A+) and e = P(⇡̃E(s) = A+|⇡E(s) = A ). Let ⇡ eDE be the optimal policy for maximizing the objective in (3) with imperfect demonstrations eDE (a particular set of with N i.i.d. imperfect demonstrations). Note `(·) is specified as the 0-1 loss: (⇡(s), a) = 1 when ⇡(s) 6= a, otherwise (⇡(s), a) = 0. We have the following upper bound on the error rate. Theorem 1. Denote by R eDE := P(s,a)⇠DE (⇡ eDE (s) 6= a) the error rate for PeerBC. When e++e <
1, with probability at least 1 , it is upper-bounded as: R eDE  1+⇠ 1 e e+
q 2 log 2/
N .
Theorem 1 states that as long as weak demonstrations are observed sufficiently, i.e., N is sufficiently large, the policy learned by PeerBC is able to converge to the clean expert policy ⇡E(s) with a convergence rate of O 1/ p N .
Peer Policy Co-Training Our discussion of BC allows us to study a more challenging co-training task [43]. Given a finite MDP M, there are two agents that receive partial observations and we let ⇡A and ⇡B denote the policies for agent A and B. Moreover, two agents are trained jointly to learn with rewards and noisy demonstrations from each other (e.g., at the preliminary training phase). Symmetrically, we consider the case where agent A learns with the demonstrations from B on sampled trajectories, and ⇡B effectively serves as a noisy version of expert policy.
Following [43], we assume a mapping function fA!B exists that transforms states under view A into B. Denote by ⌧A = {(sAi , aAi , rAi )}Ni=1 the trajectory that ⇡A generates via interacting with the partial world MA. Then ⇡B replaces each action aAi with its selection ãBi = ⇡B(fA!B(sAi )) as the weak supervision. To recover the clean expert policy, we adapt the BC peer evaluation term to the
co-learning objective function:
JCT(⇡✓) = E h EvaRL⇡ (sAi , a A i ), r A i + EvaBC⇡ (sAi , a A i ), ã B i i ⇠ · E h EvaBC⇡ (sAj , a A j ), ã B k i , (5)
where the first expectation is taken over (sAi , aAi , rAi ) ⇠ ⌧A, and ãBi = ⇡B(fA!B(sAi )), and the second is taken over (sAj , aAj , rAj ) ⇠ ⌧A, (sAk , a A k , r A k ) ⇠ ⌧ A, and ãBk = ⇡B(fA!B(s A k )), ` is the loss function defined in Eqn. (4) to measure the policy difference, and EvaRL⇡ ,Eva BC ⇡ are defined in Eqn. (2) and (4) respectively. The full algorithm PeerCT is provided in Algorithm 1. We omit detailed discussions on the convergence of PeerCT - it can be viewed as a straight-forward extension of Theorem 1 in the context of co-training.
6 Experiments
Algorithm 1 Peer policy co-training (PeerCT) Require: Views A, B, MDPs MA, MB , policies ⇡A,⇡B , map-
ping functions fA!B , fB!A that maps states from one view to the other view, CA coefficient ⇠, step size for policy update.
1: repeat 2: Run ⇡A to generate trajectories ⌧A = {(sAi , aAi , rAi )}Ni=1. 3: Run ⇡B to generate trajectories ⌧B = {(sBj , aBj , rBj )}Mj=1. 4: Agents label the trajectories for each other
⌧̃A (sAi ,⇡B fB A(s A i ) N
i=1 ,
⌧̃B (sBj ,⇡A fA B(s B j ) M
j=1 .
5: Update policies: ⇡{A,B} ⇡{A,B} + ·rJCT(⇡{A,B}) 6: until convergence
We evaluate our solution in three challenging weakly supervised PL problems. Experiments on control games and Atari show that, without any prior knowledge of the noise, our approach is able to leverage weak supervision more effectively.
Experiment Setup & Baselines We evaluate PeerPL on a wide variety of control and Atari games. For RL with noisy reward, we add synthetic noise to reward signals and compare with previous work [7], where an unbiased estimator of true reward is constructed by approximating the confusion matrix. For BC from weak demonstrations, we adopt not fully converged PPO agents as the weak experts and unroll the trajectories. We also consider a standard policy co-training setting [43] without any synthetic noise added and compare PeerCT with singleview training paradigm and CoPiEr [43].

heading: 6.1 PeerRL with Noisy Reward
text: CartPole-v0: We first evaluate our method in RL with noisy reward setting. Following [7], we consider the binary reward { 1, 1} for Cartpole where the symmetric noise is synthesized with different error rates e = e = e+. We choose DQN [49] and DDQN [50] algorithms and train the models for 10,000 steps. We repeat each experiment 10 times with different random seeds and leave extra results in Appendix D. Figure 2 shows the learning curves for DDQN with different approaches in noisy environments (⇠ = 0.2) 1. Since the number of training steps is fixed, the faster the algorithm converges, the fewer total episodes the agent will involve thus the learning curve is on the left side. As a consequence, the proposed peer reward outperforms other baselines significantly even in a high-noise regime (e.g., e = 0.4). Table 1 provides quantitative results on the average reward Ravg and total episodes Nepi. We find the agents with peer reward lead to a larger Ravg (less generalization error) and a smaller Nepi (faster convergence) consistently.
1We analysed the sensitivity of ⇠ and found the algorithm performs reasonable when ⇠ 2 (0.1, 0.4). More insights and experiments with varied ⇠ is deferred to Appendix D.
Table 1: Numerical performance of DDQN on CartPole with true reward (r), noisy reward (r̃), surrogate reward r̂ [7], and peer reward r̃peer(⇠ = 0.2). Ravg denotes average reward per episode after convergence, the higher (") the better; Nepi denotes total episodes involved in 10,000 steps, the lower (#) the better. Note 0  e < 0.5.
Pendulum: We further conduct experiments on a continuous control task Pendulum, where the goal is to keep a frictionless pendulum standing up. Since the rewards in pendulum are continuous: r 2 ( 16.3, 0.0], we discretized it into 17 intervals: ( 17, 16], ( 16, 15], · · · , ( 1, 0], with its value approximated using its maximum point. We test DDPG [57] with uniform noise in this environment following [7]. In Figure 3, the RL agents with the proposed CA objective successfully converge to the optimal policy under different amounts of noise. On the contrary, the agents with noisy rewards suffer from biased noise, especially in a high-noise regime.
Analysis of the benefits in PeerRL More surprisingly, we observed that the agents on CartPole with peer reward even lead to faster convergence than the ones observing true reward perfectly when the noise rate e is small. This indicates the possibility of other benefits to further promote peer reward, other than the noise reduction one we primarily focused on. We hypothesize this is because (1) the peer penalty term breaks the tie states (Benefit-2 in Section 4.1) and encourages explorations in RL; (2) PeerRL scales the reward signals appropriately for easier learning; (3) the human-specific “true reward” might be also imperfect which leads to a weak supervision scenario. We emphasize that the advantage of recovering from noisy reward signal is non-negligible, especially in a high-noise regime (e.g., e = 0.4 in Figure 2 and 3).

heading: 6.2 PeerBC from Weak Demonstrations
text: Atari: In BC setting, we evaluate our approach on four vision-based Atari games. For each environment, we train an imperfect RL model with PPO [58] algorithm. Here, “imperfect” means the training is terminated before convergence when the performance is about 70% ⇠ 90% as good as the fully converged model. We then collect the imperfect demonstrations using the expert model and generate 100 trajectories for each environment. The results are reported under three random seeds.
Figure 4 shows that our approach outperforms standard BC and even the expert it learns from. Note that during the whole training process, the agent never learns by interacting directly with the environment but only have access to the expert trajectories. Therefore, we owe this performance gain to PeerBC’s strong ability for learning from weak supervision. The peer term we add not only provably eliminates the effects of noise but also extracts useful strategy from the demonstrations. As shown in Table 2, our approach consistently outperforms the expert and standard BC. We provide the sensitivity analysis of ⇠ in Appendix D.
Comparison with imitation learning baselines We further extend the empirical study to imitation learning (IL) algorithms on CartPole-v1. To collect weak demonstrations, we train a PPO agent for 50k iterations that are not fully converged. As shown in Figure 6, standard IL algorithms such as BC, AIRL [37], or GAIL [59] cannot handle noisy demonstrations well and lead to sub-optimal performance. Our PeerBC brings 18% improvement over standard BC by penalizing blind agreements with the weak demonstrations. We remark that performance of PeerBC is worse than DAgger due to notorious distribution shift issue. To further improve performance, we train PeerBC in the DAgger fashion (Peer-DAgger) by querying the imperfect expert to augment the training sets. Not surprisingly,
Peer-DAgger surpasses DAgger by a large margin, which indicates that our framework has wide applicability and successfully recovers the true supervision signals. Adapting PeerPL idea to more IL algorithms such as GAIL [59] and DART [35] together with rigorous analysis is left as future works.
Analysis of benefits in PeerBC Similarly, the performance improvement of PeerBC might be also coupled with multiple possible factors. (1) The imperfect expert model might be a noisy version of the fully-converged agent since there are less visited states on which the selected actions of the model contains noise. (2) The improvements might be brought up by biasing against high-entropy policies thus PeerBC is useful when the true policy itself is deterministic. We provide more discussions about the second factor in Appendix D.5.

heading: 6.3 PeerCT for Standard Policy Co-training
text: Continuous Control/Atari: Finally, we verify the effectiveness of the PeerCT algorithm in policy co-training setting [43]. This setting is more challenging since the states are partially observable and each agent needs to imitate another agent’s behavior that is highly biased and imperfect. Note that we adopt the exact same setting as [43] without any synthetic noise included. This implies the potential of our approach to deal with natural noise in real-world applications. Following [43], we mask the first two dimensions respectively in the state vector to create two views for co-training in classic control games (Acrobot and CartPole). Similarly, the agent either removes all even index coordinates (view-A) in the state vector or removing all odd index ones (view-B) on Atari games. As shown in Table 3 and Figure 5, PeerCT algorithm outperforms training from single view, and CoPiEr algorithm consistently on both control games (⇠ = 0.5 in Figure 5a, 5b) and Atari games (⇠ = 0.2 in Figure 5c, 5d). In most cases, our approach leads to a faster convergence and lower generalization error compared to CoPiEr, showing that our ways of leveraging information from peer agent enables recovery of useful knowledge from highly imperfect supervision.

heading: 7 Conclusion
text: We have proposed PeerPL, a weakly supervised policy learning framework to unify a series of RL/BC problems with low-quality supervision signals. In PeerPL, instead of blindly memorizing the weak supervision, we evaluate a learning policy’s correlated agreements with the weak supervision. We demonstrate how our method adapts in RL/BC and the hybrid co-training tasks and provide analysis of the convergence rate and sample complexity. Current theorems focus on the specific discrete noise model. Future work may extend it to more general noise scenarios and evaluate our method on real RL/BC systems, such as robotics and self-driving.

heading: Acknowledgement
text: We sincerely thank the anonymous reviewers for their insightful suggestions. Our final version benefited substantially from the discussions with Reviewer 58fG. In particular, the tie-breaking analysis together with the code snippet is designed and contributed by Reviewer 58fG. This work is partially supported by the National Science Foundation (NSF) under grant IIS-2007951 and the Office of Naval Research under grant N00014-20-1-2240. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.

Review summaries:
This paper tackles the problem of policy learning under weak/noisy supervision. The authors present PeerPL, a unified framework that can train agents using behavior cloning under noisy/suboptimal demonstrations, or using reinforcement learning under noisy rewards. PeerPL uses the idea of “Correlated Agreement” by subtracting the original objective with a second term. The second term evaluates on randomly paired state-action tuples and supervisions, punishing the “blind” agreement between the learning agent and the weak supervision. The authors instantiate this idea on both behavior cloning and RL (PG/DQN), and also evaluate it on the problem of policy co-training. The authors demonstrate that PeerPL outperforms the weak supervision baselines on IL and RL setting, and sometimes it even outperforms agents trained with clean supervision. 

This paper formulates a framework for reinforcement learning and behavior cloning from weak supervisions (i.e., noisy rewards or imperfect expert demonstration). Specifically, it proposes PeerPL to perform efficient policy learning from the available weak supervisions, which covers PeerRL (for RL with noisy rewards), PeerBC (for imitation learning from imperfect demonstration) and PeerCT (for hybrid setting). The PeerPL idea is based on a new weak supervision objective that is in the form of difference between norm learning loss and a loss incurred by randomly sampling the supervision signals. Experimental results demonstrate that PeerPL significantly outperforms SOTA solutions when the complexity or the noise of the learning environment grows. The proposed idea is useful in practice as it increases the robustness of the learning process to imperfect supervision signals.

This paper proposes a method for RL and imitation learning in the setting where the reward function or expert actions are corrupted with noise. The main idea is to regularize the learned policy to have *high* loss on unaligned state-action-reward examples. Experiments show that the proposed method outperforms baselines that either learn directly using the noisy rewards/actions or learn an auxiliary function that predicts the mean reward/action. 

This paper proposed a framework to improve the efficacy of sensorimotor control learning. The main idea consists of augmenting the training loss with a specifically designed term that avoids the policy to "overfit" to noise present in the reward (for RL) or in the demonstrations (for behavioral cloning).  This extra term aims to minimize the probability that when randomly associating rewards and state-action pairs, the policy performs poorly. This idea, borrowed by the theory of correlated agreement, should improve the policy performance when the supervision (in terms of demonstration or reward) is corrupted by noise. Experiments show that the proposed framework achieves comparable or better results than prior works.

In this paper, the authors propose a general framework for learning sequential decision-making policies with noisy feedback. Two categories of noisy feedback are considered, noisy reward in reinforcement learning and noisy demonstrations in behavioral cloning. The concept of “correlated agreement” (CA) is proposed as a regularization mechanism for a policy to learn from the noisy feedback without overfitting to it. An additional policy co-training setting is also considered. Empirical evaluations control games and Atari show that CA is effective in enabling policy learning from noisy feedback.

The authors propose a general and unified framework to study policy learning without strong or high-quality supervision signals. This unified framework comprises, but not limited to, the main approaches for policy learning, including those using rewards signals in reinforcement learning (RL) and demonstrations through behavioral cloning (BC). The authors also propose PeerPL, a solution for efficient policy learning using only weak supervision signals, which evaluates an agent’s learning policy with noisy signals, treated as policy of an imperfect “peer agent”. The weak supervision signal comes in the form of correlation with the imperfect peer agent’s report, while the solution encourages generalization through penalizing identical reports.