Policy Learning Using Weak Supervision

1 Introduction . Recent breakthroughs in policy learning ( PL ) open up the possibility to apply reinforcement learning ( RL ) or behavioral cloning ( BC ) in real-world applications such as robotics [ 1 , 2 ] and self-driving [ 3 , 4 ] . Most existing works require agents to receive high-quality supervision signals , e.g. , reward or expert demonstrations , which are either infeasible or expensive to obtain in practice [ 5 , 6 ] . The outputs of reward functions in RL are subject to multiple kinds of randomness . For example , the reward collected from sensors on a robot may be biased and have inherent noise due to physical conditions such as temperature and lighting [ 7 , 8 , 9 ] . For the human-defined reward , different human instructors might provide drastically different feedback that leads to biased rewards [ 10 ] . Besides , the demonstrations by an expert in behavioral cloning ( BC ) are often imperfect due to limited resources and environment noise [ 11 , 12 , 13 ] . Therefore , learning from weak supervision signals such as noisy rewards [ 7 ] or low-quality demonstrations produced by untrustworthy expert [ 12 , 14 ] is one of the outstanding challenges that prevents a wider application of PL . Although some works have explored these topics separately in their specific domains [ 7 , 15 , 14 , 16 ] , there lacks a unified solution for robust policy learning in imperfect situations . Moreover , the noise model as well as the corruption level in supervision signals is often required . To handle these challenges , we first formulate a meta-framework to study RL/BC with weak supervision and call it weakly supervised policy learning . Then we propose a theoretically principled solution , PeerPL , to perform efficient policy learning using the available weak supervision without requiring noise rates . Our solution is inspired by peer loss [ 17 ] , a recently proposed loss function for learning with noisy labels but does not require the specification of noise rates . In peer loss , the noisy labels are treated as a peer agent ’ s supervision . This loss function explicitly punishes the classifier from simply agreeing with the noisy labels , but would instead reward it for a “ correlated agreement '' ( CA ) . We adopt a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . similar idea and treat the “ weak supervision ” as the noisy information coming from an imperfect peer agent , and evaluate the learning agent ’ s policy based on a “ correlated agreement ” ( CA ) with the weak supervision signals . Compared to standard reward and evaluation functions that encourage simple agreements with the supervision , our approach punishes “ over-agreement '' to avoid overfitting to the weak supervision , which offers us a family of solutions that do not require prior knowledge of the corruption level in supervision signals . To summarize , the contributions in the paper are : ( 1 ) We provide a unified formulation of the weakly supervised policy learning problems ; ( 2 ) We propose PeerPL , a new way to perform policy evaluation for RL/BC tasks , and demonstrate how it adapts in challenging tasks including RL with noisy rewards and BC from weak demonstrations ; ( 3 ) PeerPL is theoretically guaranteed to recover the optimal policy , as if the supervision are of high-quality and clean . ( 4 ) Experiment results show strong evidence that PeerPL brings significant improvements over state-of-the-art solutions . Code is online available at : https : //github.com/wangjksjtu/PeerPL . 1.1 Related Work . Learning with Noisy Supervision Learning from noisy supervision is a widely explored topic . The seminal work [ 18 ] first proposed an unbiased surrogate loss function to recover the true loss from the noisy label distribution , given the knowledge of the noise rates of labels . Follow-up works offered ways to estimate the noise level from model predictions [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] or label consensuses of nearby representations [ 28 ] . Recent works also studied this problem in sequential settings including federated bandit [ 29 ] and RL [ 7 ] . The former work assumes the noise can be offset by averaging rewards from multiple agents . [ 7 ] designs a statistics-based estimation algorithm for noise rates in observed rewards , which can be inefficient especially when the state-action space is huge . Moreover , the error in the estimation can accumulate and amplify in sequential problems . Inspired by recent advances of peer loss [ 17 , 30 , 31 ] , our solution is able to recover true supervision signals without requiring a priori specification of the noise rates . Behavioral Cloning ( BC ) Standard BC [ 32 , 33 ] tackles the sequential decision-making problem by imitating the expert actions using supervised learning . Specifically , it aims to minimize the one-step deviation error over the expert trajectory without reasoning about the sequential consequences of actions . Therefore , the agent suffers from compounding errors when there is a mismatch between demonstrations and real states encountered [ 33 , 34 , 35 ] . Recent works introduce data augmentations [ 36 ] and value-based regularization [ 37 ] or inverse dynamics models [ 38 , 39 ] to encourage learning long-horizon behaviors . While being simple and straightforward , BC has been widely investigated in a range of application domains [ 40 , 41 ] and often yields competitive performance [ 42 , 37 ] . Our framework is complementary to the current BC literature by introducing a learning strategy from weak demonstrations ( e.g. , noisy or from a poorly-trained agent ) and provides theoretical guarantees on how to retrieve clean policy under mild assumptions [ 43 ] . Correlated Agreement In [ 44 , 45 ] , a correlated agreement ( CA ) type of mechanism is proposed to evaluate the correlations between agents ’ reports . In addition to encouraging a certain agreement between agents ’ reports , CA also punishes over-agreement when two agents always report identically . Recently , [ 17 , 30 , 25 ] adapt a similar idea to noisy label learning thus offloading the burdens of estimating noise rates . We consider a more challenging sequential decision-making problem and study the convergence rates under noisy supervision signals . 2 Policy Learning from Weak Supervision . We begin by reviewing conventional reinforcement learning and behavioral cloning with clean supervision signals . Then we introduce the weak supervision problem in policy learning and define two concrete instantiations : ( 1 ) RL with noisy reward and ( 2 ) BC using weak expert demonstrations . 2.1 Overview of Policy Learning . The goal of policy learning ( PL ) is to learn a policy ⇡ that the agent could follow to perform a series of actions in a stateful environment . For reinforcement learning , the interactive environment is characterized as an MDP M = hS , A , R , P , i . At each time t , the agent in state st 2 S takes an action at 2 A by following the policy ⇡ : S ⇥ A ! R , and potentially receives a reward r ( st , at ) 2 R. Then the agent transfers to the next state st+1 according to a transition probability function P . We denote the generated trajectory ⌧ = { ( st , at , rt ) } Tt=0 , where T is a finite or infinite horizon . RL algorithms aim to maximize the expected reward over the trajectory ⌧ induced by the policy : J clean ( ⇡ ) = E ( st , at , rt ) ⇠⌧ [ PT t=0 t rt ] , where 2 ( 0 , 1 ] is the discount factor . Another popular policy learning method is behavioral cloning . Let ⇡ ( ·|s ) denotes the distribution over actions formed by ⇡ , and ⇡ ( a|s ) be the probability of choosing action a given state s and policy ⇡ . The goal of BC is to mimic the expert policy ⇡E through a set of demonstrations DE = { ( si , ai ) } Ni=1 drawn from a distribution DE , where ( si , ai ) is the sampled state-action pair from the expert trajectory and ai ⇠ ⇡E ( ·|si ) Then training a policy with standard BC corresponds to maximizing the following log-likelihood : J clean ( ⇡ ) = E ( s , a ) ⇠DE [ log ⇡ ( a|s ) ] . In both RL and BC , the learning agent receives supervision through either the ( clean ) reward r by interacting with environments or the expert policy ⇡E as observable demonstrations . Consider a particular policy class ⇧ , the optimal policy is then defined as ⇡⇤ = argmax⇡2⇧ J clean ( ⇡ ) : ⇡⇤ obtains the maximum expected reward over the horizon T in RL and ⇡⇤ corresponds to the clean expert policy ⇡E in BC . In practice , one can also combine both RL and BC approaches to take advantage of both learning paradigm [ 46 , 47 , 15 , 43 ] . Specifically , a recent hybrid framework called policy co-training [ 43 ] will be considered in this paper . 2.2 Weak Supervision in Policy Learning . The weak supervision signal eY could be noisy reward r̃ for RL or noisy action ã from an imperfect expert policy ⇡̃E for BC , which are noisy versions of the corresponding high-quality supervision signals . See more details below . RL with Noisy Reward Consider a finite MDP fM = hS , A , R , F , P , i with noisy reward channels [ 7 ] , where R : S ⇥A ! R , and the noisy reward r̃ is generated following a certain function F : R ! eR . Denote the trajectory a policy ⇡✓ generates via interacting with fM as ⌧̃✓ . Assume the reward is discrete and has |R| levels . The noisy reward can be characterized via a unknown matrix CRL|R|⇥|R| , where each entry cj , k indicates the flipping probability for generating a possibly different outcome : cRLj , k = P ( r̃t = Rk|rt = Rj ) . We call r and r̃ the true reward and noisy reward . BC with Weak Demonstration Instead of observing the true expert demonstration generated according to ⇡E , denote the available weak demonstrations by { ( si , ãi ) } Ni=1 , where ãi is is the noisy expert action drawn according to a random variable ãi = ⇡̃E ( si ) ⇠ ⇡̃E ( ·|si ) , each state-action pair ( si , ãi ) is sampled from distribution eDE . Note there may exist two randomness factors in getting ãi : uncertainty in true policy ⇡E and noise from imperfect policy ⇡̃E . In particular , we do not consider the former randomness in theoretical analyses : given the output distribution ⇡E ( ·|si ) , only one deterministic action ⇡E ( si ) is taken by expert . This is because with uncertainty in true expert actions , it is hard to distinguish a clean case with true expert actions from the weak supervision case without addition knowledge . Similar assumptions are also adopted in [ 23 , 28 ] . The noisy action is modeled by a unknown confusion matrix CBC|A|⇥|A| , where each entry cj , k indicates the flipping probability for taking a sub-optimal action that differs from ⇡E ( s ) : cBCj , k = P ( ⇡̃E ( s ) = Ak|⇡E ( s ) = Aj ) , Ak and Aj denote the k-th and the j-th action from the action space A . In the above definition , we assume the noisy action ãi is independent of the state s given the deterministic expert action ⇡E ( s ) , i.e. , P ( ãi|⇡E ( si ) ) = P ( ãi|si , ⇡E ( si ) ) . We aim to recover ⇡⇤ as if we were able to access the quality expert demonstration ⇡E instead of ⇡̃E . Knowledge of C Recall C : CRL|R|⇥|R| or C BC |A|⇥|A| is unknown in practice . While recent works estimate this matrix [ 26 , 23 , 28 ] in supervised classification problems , it is still challenging to generalize them to a sequential setting [ 7 ] . When C is not perfectly estimated , the estimation error of C may lead to unexpected state-action pairs then the error of reward estimates will be accumulated in sequential learning . Besides , estimating C involves extra computation burden . In contrast , our method gets rid of the above issues since it is free of any knowledge of C and leads to more robust policy learning algorithms . Learning Goal With full supervision , both RL and BC can converge to the optimal policy ⇡⇤ . However , when only weak supervision is available , with an over-parameterized model such as a deep neural network , the learning agent will easily memorize the weak supervision and learn a biased policy [ 48 ] . In our meta framework , instead of converging to any biased policy , we focus on learning the optimal policy ⇡⇤ with only a weak supervision sequence denoted as { ( st , at ) , eYt } Tt=1 ( RL ) or { ( si , ai ) , eYi } Ni=1 ( BC ) .

Summary: This paper proposes a method for RL and imitation learning in the setting where the reward function or expert actions are corrupted with noise. The main idea is to regularize the learned policy to have *high* loss on unaligned state-action-reward examples. Experiments show that the proposed method outperforms baselines that either learn directly using the noisy rewards/actions or learn an auxiliary function that predicts the mean reward/action.