Policy Learning Using Weak Supervision

1 Introduction . Recent breakthroughs in policy learning ( PL ) open up the possibility to apply reinforcement learning ( RL ) or behavioral cloning ( BC ) in real-world applications such as robotics [ 1 , 2 ] and self-driving [ 3 , 4 ] . Most existing works require agents to receive high-quality supervision signals , e.g. , reward or expert demonstrations , which are either infeasible or expensive to obtain in practice [ 5 , 6 ] . The outputs of reward functions in RL are subject to multiple kinds of randomness . For example , the reward collected from sensors on a robot may be biased and have inherent noise due to physical conditions such as temperature and lighting [ 7 , 8 , 9 ] . For the human-defined reward , different human instructors might provide drastically different feedback that leads to biased rewards [ 10 ] . Besides , the demonstrations by an expert in behavioral cloning ( BC ) are often imperfect due to limited resources and environment noise [ 11 , 12 , 13 ] . Therefore , learning from weak supervision signals such as noisy rewards [ 7 ] or low-quality demonstrations produced by untrustworthy expert [ 12 , 14 ] is one of the outstanding challenges that prevents a wider application of PL . Although some works have explored these topics separately in their specific domains [ 7 , 15 , 14 , 16 ] , there lacks a unified solution for robust policy learning in imperfect situations . Moreover , the noise model as well as the corruption level in supervision signals is often required . To handle these challenges , we first formulate a meta-framework to study RL/BC with weak supervision and call it weakly supervised policy learning . Then we propose a theoretically principled solution , PeerPL , to perform efficient policy learning using the available weak supervision without requiring noise rates . Our solution is inspired by peer loss [ 17 ] , a recently proposed loss function for learning with noisy labels but does not require the specification of noise rates . In peer loss , the noisy labels are treated as a peer agent ’ s supervision . This loss function explicitly punishes the classifier from simply agreeing with the noisy labels , but would instead reward it for a “ correlated agreement '' ( CA ) . We adopt a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . similar idea and treat the “ weak supervision ” as the noisy information coming from an imperfect peer agent , and evaluate the learning agent ’ s policy based on a “ correlated agreement ” ( CA ) with the weak supervision signals . Compared to standard reward and evaluation functions that encourage simple agreements with the supervision , our approach punishes “ over-agreement '' to avoid overfitting to the weak supervision , which offers us a family of solutions that do not require prior knowledge of the corruption level in supervision signals . To summarize , the contributions in the paper are : ( 1 ) We provide a unified formulation of the weakly supervised policy learning problems ; ( 2 ) We propose PeerPL , a new way to perform policy evaluation for RL/BC tasks , and demonstrate how it adapts in challenging tasks including RL with noisy rewards and BC from weak demonstrations ; ( 3 ) PeerPL is theoretically guaranteed to recover the optimal policy , as if the supervision are of high-quality and clean . ( 4 ) Experiment results show strong evidence that PeerPL brings significant improvements over state-of-the-art solutions . Code is online available at : https : //github.com/wangjksjtu/PeerPL . 1.1 Related Work . Learning with Noisy Supervision Learning from noisy supervision is a widely explored topic . The seminal work [ 18 ] first proposed an unbiased surrogate loss function to recover the true loss from the noisy label distribution , given the knowledge of the noise rates of labels . Follow-up works offered ways to estimate the noise level from model predictions [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] or label consensuses of nearby representations [ 28 ] . Recent works also studied this problem in sequential settings including federated bandit [ 29 ] and RL [ 7 ] . The former work assumes the noise can be offset by averaging rewards from multiple agents . [ 7 ] designs a statistics-based estimation algorithm for noise rates in observed rewards , which can be inefficient especially when the state-action space is huge . Moreover , the error in the estimation can accumulate and amplify in sequential problems . Inspired by recent advances of peer loss [ 17 , 30 , 31 ] , our solution is able to recover true supervision signals without requiring a priori specification of the noise rates . Behavioral Cloning ( BC ) Standard BC [ 32 , 33 ] tackles the sequential decision-making problem by imitating the expert actions using supervised learning . Specifically , it aims to minimize the one-step deviation error over the expert trajectory without reasoning about the sequential consequences of actions . Therefore , the agent suffers from compounding errors when there is a mismatch between demonstrations and real states encountered [ 33 , 34 , 35 ] . Recent works introduce data augmentations [ 36 ] and value-based regularization [ 37 ] or inverse dynamics models [ 38 , 39 ] to encourage learning long-horizon behaviors . While being simple and straightforward , BC has been widely investigated in a range of application domains [ 40 , 41 ] and often yields competitive performance [ 42 , 37 ] . Our framework is complementary to the current BC literature by introducing a learning strategy from weak demonstrations ( e.g. , noisy or from a poorly-trained agent ) and provides theoretical guarantees on how to retrieve clean policy under mild assumptions [ 43 ] . Correlated Agreement In [ 44 , 45 ] , a correlated agreement ( CA ) type of mechanism is proposed to evaluate the correlations between agents ’ reports . In addition to encouraging a certain agreement between agents ’ reports , CA also punishes over-agreement when two agents always report identically . Recently , [ 17 , 30 , 25 ] adapt a similar idea to noisy label learning thus offloading the burdens of estimating noise rates . We consider a more challenging sequential decision-making problem and study the convergence rates under noisy supervision signals . 2 Policy Learning from Weak Supervision . We begin by reviewing conventional reinforcement learning and behavioral cloning with clean supervision signals . Then we introduce the weak supervision problem in policy learning and define two concrete instantiations : ( 1 ) RL with noisy reward and ( 2 ) BC using weak expert demonstrations . 2.1 Overview of Policy Learning . The goal of policy learning ( PL ) is to learn a policy ⇡ that the agent could follow to perform a series of actions in a stateful environment . For reinforcement learning , the interactive environment is characterized as an MDP M = hS , A , R , P , i . At each time t , the agent in state st 2 S takes an action at 2 A by following the policy ⇡ : S ⇥ A ! R , and potentially receives a reward r ( st , at ) 2 R. Then the agent transfers to the next state st+1 according to a transition probability function P . We denote the generated trajectory ⌧ = { ( st , at , rt ) } Tt=0 , where T is a finite or infinite horizon . RL algorithms aim to maximize the expected reward over the trajectory ⌧ induced by the policy : J clean ( ⇡ ) = E ( st , at , rt ) ⇠⌧ [ PT t=0 t rt ] , where 2 ( 0 , 1 ] is the discount factor . Another popular policy learning method is behavioral cloning . Let ⇡ ( ·|s ) denotes the distribution over actions formed by ⇡ , and ⇡ ( a|s ) be the probability of choosing action a given state s and policy ⇡ . The goal of BC is to mimic the expert policy ⇡E through a set of demonstrations DE = { ( si , ai ) } Ni=1 drawn from a distribution DE , where ( si , ai ) is the sampled state-action pair from the expert trajectory and ai ⇠ ⇡E ( ·|si ) Then training a policy with standard BC corresponds to maximizing the following log-likelihood : J clean ( ⇡ ) = E ( s , a ) ⇠DE [ log ⇡ ( a|s ) ] . In both RL and BC , the learning agent receives supervision through either the ( clean ) reward r by interacting with environments or the expert policy ⇡E as observable demonstrations . Consider a particular policy class ⇧ , the optimal policy is then defined as ⇡⇤ = argmax⇡2⇧ J clean ( ⇡ ) : ⇡⇤ obtains the maximum expected reward over the horizon T in RL and ⇡⇤ corresponds to the clean expert policy ⇡E in BC . In practice , one can also combine both RL and BC approaches to take advantage of both learning paradigm [ 46 , 47 , 15 , 43 ] . Specifically , a recent hybrid framework called policy co-training [ 43 ] will be considered in this paper . 2.2 Weak Supervision in Policy Learning . The weak supervision signal eY could be noisy reward r̃ for RL or noisy action ã from an imperfect expert policy ⇡̃E for BC , which are noisy versions of the corresponding high-quality supervision signals . See more details below . RL with Noisy Reward Consider a finite MDP fM = hS , A , R , F , P , i with noisy reward channels [ 7 ] , where R : S ⇥A ! R , and the noisy reward r̃ is generated following a certain function F : R ! eR . Denote the trajectory a policy ⇡✓ generates via interacting with fM as ⌧̃✓ . Assume the reward is discrete and has |R| levels . The noisy reward can be characterized via a unknown matrix CRL|R|⇥|R| , where each entry cj , k indicates the flipping probability for generating a possibly different outcome : cRLj , k = P ( r̃t = Rk|rt = Rj ) . We call r and r̃ the true reward and noisy reward . BC with Weak Demonstration Instead of observing the true expert demonstration generated according to ⇡E , denote the available weak demonstrations by { ( si , ãi ) } Ni=1 , where ãi is is the noisy expert action drawn according to a random variable ãi = ⇡̃E ( si ) ⇠ ⇡̃E ( ·|si ) , each state-action pair ( si , ãi ) is sampled from distribution eDE . Note there may exist two randomness factors in getting ãi : uncertainty in true policy ⇡E and noise from imperfect policy ⇡̃E . In particular , we do not consider the former randomness in theoretical analyses : given the output distribution ⇡E ( ·|si ) , only one deterministic action ⇡E ( si ) is taken by expert . This is because with uncertainty in true expert actions , it is hard to distinguish a clean case with true expert actions from the weak supervision case without addition knowledge . Similar assumptions are also adopted in [ 23 , 28 ] . The noisy action is modeled by a unknown confusion matrix CBC|A|⇥|A| , where each entry cj , k indicates the flipping probability for taking a sub-optimal action that differs from ⇡E ( s ) : cBCj , k = P ( ⇡̃E ( s ) = Ak|⇡E ( s ) = Aj ) , Ak and Aj denote the k-th and the j-th action from the action space A . In the above definition , we assume the noisy action ãi is independent of the state s given the deterministic expert action ⇡E ( s ) , i.e. , P ( ãi|⇡E ( si ) ) = P ( ãi|si , ⇡E ( si ) ) . We aim to recover ⇡⇤ as if we were able to access the quality expert demonstration ⇡E instead of ⇡̃E . Knowledge of C Recall C : CRL|R|⇥|R| or C BC |A|⇥|A| is unknown in practice . While recent works estimate this matrix [ 26 , 23 , 28 ] in supervised classification problems , it is still challenging to generalize them to a sequential setting [ 7 ] . When C is not perfectly estimated , the estimation error of C may lead to unexpected state-action pairs then the error of reward estimates will be accumulated in sequential learning . Besides , estimating C involves extra computation burden . In contrast , our method gets rid of the above issues since it is free of any knowledge of C and leads to more robust policy learning algorithms . Learning Goal With full supervision , both RL and BC can converge to the optimal policy ⇡⇤ . However , when only weak supervision is available , with an over-parameterized model such as a deep neural network , the learning agent will easily memorize the weak supervision and learn a biased policy [ 48 ] . In our meta framework , instead of converging to any biased policy , we focus on learning the optimal policy ⇡⇤ with only a weak supervision sequence denoted as { ( st , at ) , eYt } Tt=1 ( RL ) or { ( si , ai ) , eYi } Ni=1 ( BC ) .

Summary: This paper tackles the problem of policy learning under weak/noisy supervision. The authors present PeerPL, a unified framework that can train agents using behavior cloning under noisy/suboptimal demonstrations, or using reinforcement learning under noisy rewards. PeerPL uses the idea of “Correlated Agreement” by subtracting the original objective with a second term. The second term evaluates on randomly paired state-action tuples and supervisions, punishing the “blind” agreement between the learning agent and the weak supervision. The authors instantiate this idea on both behavior cloning and RL (PG/DQN), and also evaluate it on the problem of policy co-training. The authors demonstrate that PeerPL outperforms the weak supervision baselines on IL and RL setting, and sometimes it even outperforms agents trained with clean supervision.