Reinforcement Learning with Perturbed Rewards

1 INTRODUCTION . Designing a suitable reward function plays a critical role in building reinforcement learning models for real-world applications . Ideally , one would want to customize reward functions to achieve application-specific goals ( Hadfield-Menell et al. , 2017 ) . In practice , however , it is difficult to design a function that produces credible rewards in the presence of noise . This is because the output from any reward function is subject to multiple kinds of randomness : • Inherent Noise . For instance , sensors on a robot will be affected by physical conditions such as temperature and lighting , and therefore will report back noisy observed rewards . • Application-Specific Noise . In machine teaching tasks ( Thomaz et al. , 2006 ; Loftin et al. , 2014 ) , when an RL agent receives feedback/instructions from people , different human instructors might provide drastically different feedback due to their personal styles and capabilities . This way the RL agent ( machine ) will obtain reward with bias . • Adversarial Noise . Adversarial perturbation has been widely explored in different learning tasks and shows strong attack power against different machine learning models . For instance , Huang et al . ( 2017 ) has shown that by adding adversarial perturbation to each frame of the game , they can mislead RL policies arbitrarily . Assuming an arbitrary noise model makes solving this noisy RL problem extremely challenging . Instead , we focus on a specific noisy reward model which we call perturbed rewards , where the observed rewards by RL agents are generated according to a reward confusion matrix . This is not a very restrictive setting to start with , even considering that the noise could be adversarial : Given that arbitrary pixel value manipulation attack in RL is not very practical , adversaries in the real-world have high incentives to inject adversarial perturbation to the reward value by slightly modifying it . For instance , adversaries can manipulate sensors via reversing the reward value . In this paper , we develop an unbiased reward estimator aided robust framework that enables an RL agent to learn in a noisy environment with observing only perturbed rewards . Our solution framework builds on existing reinforcement learning algorithms , including the recently developed DRL ones ( Q-Learning ( Watkins , 1989 ; Watkins & Dayan , 1992 ) , Cross-Entropy Method ( CEM ) ( Szita & Lörincz , 2006 ) , Deep SARSA ( Sutton & Barto , 1998 ) , Deep Q-Network ( DQN ) ( Mnih et al. , 2013 ; 2015 ; van Hasselt et al. , 2016 ) , Dueling DQN ( DDQN ) ( Wang et al. , 2016 ) , Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) , Continuous DQN ( NAF ) ( Gu et al. , 2016 ) and Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) algorithms ) . The main challenge is that the observed rewards are likely to be biased , and in RL or DRL the accumulated errors could amplify the reward estimation error over time . We do not require any assumption on knowing the true distribution of reward or adversarial strategies , other than the fact that the generation of noises follow an unknown reward confusion matrix . Instead , we address the issue of estimating the reward confusion matrices by proposing an efficient and flexible estimation module . Everitt et al . ( 2017 ) provided preliminary studies for the noisy reward problem and gave some general negative results . The authors proved a No Free Lunch theorem , which is , without any assumption about what the reward corruption is , all agents can be misled . Our results do not contradict with the results therein , as we consider a specific noise generation model ( that leads to a set of perturbed rewards ) . We analyze the convergence and sample complexity for the policy trained based on our proposed method using surrogate rewards in RL , using Q-Learning as an example . We conduct extensive experiments on OpenAI Gym ( Brockman et al. , 2016 ) ( AirRaid , Alien , Carnival , MsPacman , Pong , Phoenix , Seaquest ) and show that the proposed reward robust RL method achieves comparable performance with the policy trained using the true rewards . In some cases , our method even achieves higher cumulative reward - this is surprising to us at first , but we conjecture that the inserted noise together with our noisy-removal unbiased estimator adds another layer of exploration , which proves to be beneficial in some settings . This merits a future study . Our contributions are summarized as follows : ( 1 ) We adapt and generalize the idea of defining a simple but effective unbiased estimator for true rewards using observed and perturbed rewards to the reinforcement learning setting . The proposed estimator helps guarantee the convergence to the optimal policy even when the RL agents only have noisy observations of the rewards . ( 2 ) We analyze the convergence to the optimal policy and finite sample complexity of our reward robust RL methods , using Q-Learning as the running example . ( 3 ) Extensive experiments on OpenAI Gym show that our proposed algorithms perform robustly even at high noise rates . 1.1 RELATED WORK . Robust Reinforcement Learning It is known that RL algorithms are vulnerable to noisy environments ( Irpan , 2018 ) . Recent studies ( Huang et al. , 2017 ; Kos & Song , 2017 ; Lin et al. , 2017 ) show that learned RL policies can be easily misled with small perturbations in observations . The presence of noise is very common in real-world environments , especially in robotics-relevant applications . Consequently , robust ( adversarial ) reinforcement learning ( RRL/RARL ) algorithms have been widely studied , aiming to train a robust policy which is capable of withstanding perturbed observations ( Teh et al. , 2017 ; Pinto et al. , 2017 ; Gu et al. , 2018 ) or transferring to unseen environments ( Rajeswaran et al. , 2016 ; Fu et al. , 2017 ) . However , these robust RL algorithms mainly focus on noisy vision observations , instead of the observed rewards . A couple of recent works ( Lim et al. , 2016 ; Roy et al. , 2017 ) have also looked into a rather parallel question of training robust RL algorithms with uncertainty in models . Learning with Noisy Data Learning appropriately with biased data has received quite a bit of attention in recent machine learning studies Natarajan et al . ( 2013 ) ; Scott et al . ( 2013 ) ; Scott ( 2015 ) ; Sukhbaatar & Fergus ( 2014 ) ; van Rooyen & Williamson ( 2015 ) ; Menon et al . ( 2015 ) . The idea of above line of works is to define unbiased surrogate loss function to recover the true loss using the knowledge of the noises . We adapt these approaches to reinforcement learning . Though intuitively the idea should apply in our RL settings , our work is the first one to formally establish this extension both theoretically and empirically . Our quantitative understandings will provide practical insights when implementing reinforcement learning algorithms in noisy environments . 2 PROBLEM FORMULATION AND PRELIMINARIES . In this section , we define our problem of learning from perturbed rewards in reinforcement learning . Throughout this paper , we will use perturbed reward and noisy reward interchangeably , as each time step of our sequential decision making setting is similar to the “ learning with noisy data ” setting in supervised learning ( Natarajan et al. , 2013 ; Scott et al. , 2013 ; Scott , 2015 ; Sukhbaatar & Fergus , 2014 ) . In what follows , we formulate our Markov Decision Process ( MDP ) problem and the reinforcement learning ( RL ) problem with perturbed ( noisy ) rewards . 2.1 REINFORCEMENT LEARNING : THE NOISE-FREE SETTING . Our RL agent interacts with an unknown environment and attempts to maximize the total of his collected reward . The environment is formalized as a Markov Decision Process ( MDP ) , denoting as M = 〈S , A , R , P , γ〉 . At each time t , the agent in state st ∈ S takes an action at ∈ A , which returns a reward r ( st , at , st+1 ) ∈ R ( which we will also shorthand as rt ) , and leads to the next state st+1 ∈ S according to a transition probability kernel P , which encodes the probability Pa ( st , st+1 ) . Commonly P is unknown to the agent . The agent ’ s goal is to learn the optimal policy , a conditional distribution π ( a|s ) that maximizes the state ’ s value function . The value function calculates the cumulative reward the agent is expected to receive given he would follow the current policy π after observing the current state st : V π ( s ) = Eπ [ ∑∞ k=1 γ krt+k | st = s ] , where 0 ≤ γ ≤ 11 is a discount factor . Intuitively , the agent evaluates how preferable each state is given the current policy . From the Bellman Equation , the optimal value function is given by V ∗ ( s ) = maxa∈A ∑ st+1∈S Pa ( st , st+1 ) [ rt + γV ∗ ( st+1 ) ] . It is a standard practice for RL algorithms to learn a state-action value function , also called the Q-function . Q-function denotes the expected cumulative reward if agent chooses a in the current state and follows π thereafter : Qπ ( s , a ) = Eπ [ r ( st , at , st+1 ) + γV π ( st+1 ) | st = s , at = a ] . 2.2 PERTURBED REWARD IN RL . In many practical settings , our RL agent does not observe the reward feedback perfectly . We consider the following MDP with perturbed reward , denoting as M̃ = 〈S , A , R , C , P , γ〉2 : instead of observing rt ∈ R at each time t directly ( following his action ) , our RL agent only observes a perturbed version of rt , denoting as r̃t ∈ R̃ . For most of our presentations , we focus on the cases whereR , R̃ are finite sets ; but our results generalize to the continuous reward settings . The generation of r̃ follows a certain function C : S×R → R̃ . To let our presentation stay focused , we consider the following simple state-independent3 flipping error rates model : if the rewards are binary ( consider r+ and r− ) , r̃ ( st , at , st+1 ) ( r̃t ) can be characterized by the following noise rate parameters e+ , e− : e+ = P ( r̃ ( st , at , st+1 ) = r−|r ( st , at , st+1 ) = r+ ) , e− = P ( r̃ ( st , at , st+1 ) = r+|r ( st , at , st+1 ) = r− ) . When the signal levels are beyond binary , suppose there are M outcomes in total , denoting as [ R0 , R1 , · · · , RM−1 ] . r̃t will be generated according to the following confusion matrix CM×M where each entry cj , k indicates the flipping probability for generating a perturbed outcome : cj , k = P ( r̃t = Rk|rt = Rj ) . Again we ’ d like to note that we focus on settings with finite reward levels for most of our paper , but we provide discussions in Section 3.1 on how to handle continuous rewards with discretizations . In the paper , we do not assume knowing the noise rates ( i.e. , the reward confusion matrices ) , which is different from the assumption of knowing them as adopted in many supervised learning works Natarajan et al . ( 2013 ) . Instead we will estimate the confusion matrices ( Section 3.3 ) . 1γ = 1 indicates an undiscounted MDP setting ( Schwartz , 1993 ; Sobel , 1994 ; Kakade , 2003 ) . 2The MDP with perturbed reward can equivalently be defined as a tuple M̃ = 〈S , A , R , R̃ , P , γ〉 , with the perturbation function C implicitly defined as the difference betweenR and R̃ . 3The case of state-dependent perturbed reward is discussed in Appendix C.3

Summary: The authors present work that shows how to deal with noise in reward signals by creating a surrogate reward signal. The work develops a number of results including: showing how the surrogate reward is equal in expectation to the true reward signal, how this doesn't affect the fixed point of the Bellman equation, how to deal with finite and continuous rewards and how the convergence time is affected for different levels of noise. They demonstrate the value of this approach with a variety of early and state-of-the-art algorithms on a variety of domains,, and the results are consistent with the claims.