Reinforcement Learning with Perturbed Rewards

1 INTRODUCTION . Designing a suitable reward function plays a critical role in building reinforcement learning models for real-world applications . Ideally , one would want to customize reward functions to achieve application-specific goals ( Hadfield-Menell et al. , 2017 ) . In practice , however , it is difficult to design a function that produces credible rewards in the presence of noise . This is because the output from any reward function is subject to multiple kinds of randomness : • Inherent Noise . For instance , sensors on a robot will be affected by physical conditions such as temperature and lighting , and therefore will report back noisy observed rewards . • Application-Specific Noise . In machine teaching tasks ( Thomaz et al. , 2006 ; Loftin et al. , 2014 ) , when an RL agent receives feedback/instructions from people , different human instructors might provide drastically different feedback due to their personal styles and capabilities . This way the RL agent ( machine ) will obtain reward with bias . • Adversarial Noise . Adversarial perturbation has been widely explored in different learning tasks and shows strong attack power against different machine learning models . For instance , Huang et al . ( 2017 ) has shown that by adding adversarial perturbation to each frame of the game , they can mislead RL policies arbitrarily . Assuming an arbitrary noise model makes solving this noisy RL problem extremely challenging . Instead , we focus on a specific noisy reward model which we call perturbed rewards , where the observed rewards by RL agents are generated according to a reward confusion matrix . This is not a very restrictive setting to start with , even considering that the noise could be adversarial : Given that arbitrary pixel value manipulation attack in RL is not very practical , adversaries in the real-world have high incentives to inject adversarial perturbation to the reward value by slightly modifying it . For instance , adversaries can manipulate sensors via reversing the reward value . In this paper , we develop an unbiased reward estimator aided robust framework that enables an RL agent to learn in a noisy environment with observing only perturbed rewards . Our solution framework builds on existing reinforcement learning algorithms , including the recently developed DRL ones ( Q-Learning ( Watkins , 1989 ; Watkins & Dayan , 1992 ) , Cross-Entropy Method ( CEM ) ( Szita & Lörincz , 2006 ) , Deep SARSA ( Sutton & Barto , 1998 ) , Deep Q-Network ( DQN ) ( Mnih et al. , 2013 ; 2015 ; van Hasselt et al. , 2016 ) , Dueling DQN ( DDQN ) ( Wang et al. , 2016 ) , Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) , Continuous DQN ( NAF ) ( Gu et al. , 2016 ) and Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) algorithms ) . The main challenge is that the observed rewards are likely to be biased , and in RL or DRL the accumulated errors could amplify the reward estimation error over time . We do not require any assumption on knowing the true distribution of reward or adversarial strategies , other than the fact that the generation of noises follow an unknown reward confusion matrix . Instead , we address the issue of estimating the reward confusion matrices by proposing an efficient and flexible estimation module . Everitt et al . ( 2017 ) provided preliminary studies for the noisy reward problem and gave some general negative results . The authors proved a No Free Lunch theorem , which is , without any assumption about what the reward corruption is , all agents can be misled . Our results do not contradict with the results therein , as we consider a specific noise generation model ( that leads to a set of perturbed rewards ) . We analyze the convergence and sample complexity for the policy trained based on our proposed method using surrogate rewards in RL , using Q-Learning as an example . We conduct extensive experiments on OpenAI Gym ( Brockman et al. , 2016 ) ( AirRaid , Alien , Carnival , MsPacman , Pong , Phoenix , Seaquest ) and show that the proposed reward robust RL method achieves comparable performance with the policy trained using the true rewards . In some cases , our method even achieves higher cumulative reward - this is surprising to us at first , but we conjecture that the inserted noise together with our noisy-removal unbiased estimator adds another layer of exploration , which proves to be beneficial in some settings . This merits a future study . Our contributions are summarized as follows : ( 1 ) We adapt and generalize the idea of defining a simple but effective unbiased estimator for true rewards using observed and perturbed rewards to the reinforcement learning setting . The proposed estimator helps guarantee the convergence to the optimal policy even when the RL agents only have noisy observations of the rewards . ( 2 ) We analyze the convergence to the optimal policy and finite sample complexity of our reward robust RL methods , using Q-Learning as the running example . ( 3 ) Extensive experiments on OpenAI Gym show that our proposed algorithms perform robustly even at high noise rates . 1.1 RELATED WORK . Robust Reinforcement Learning It is known that RL algorithms are vulnerable to noisy environments ( Irpan , 2018 ) . Recent studies ( Huang et al. , 2017 ; Kos & Song , 2017 ; Lin et al. , 2017 ) show that learned RL policies can be easily misled with small perturbations in observations . The presence of noise is very common in real-world environments , especially in robotics-relevant applications . Consequently , robust ( adversarial ) reinforcement learning ( RRL/RARL ) algorithms have been widely studied , aiming to train a robust policy which is capable of withstanding perturbed observations ( Teh et al. , 2017 ; Pinto et al. , 2017 ; Gu et al. , 2018 ) or transferring to unseen environments ( Rajeswaran et al. , 2016 ; Fu et al. , 2017 ) . However , these robust RL algorithms mainly focus on noisy vision observations , instead of the observed rewards . A couple of recent works ( Lim et al. , 2016 ; Roy et al. , 2017 ) have also looked into a rather parallel question of training robust RL algorithms with uncertainty in models . Learning with Noisy Data Learning appropriately with biased data has received quite a bit of attention in recent machine learning studies Natarajan et al . ( 2013 ) ; Scott et al . ( 2013 ) ; Scott ( 2015 ) ; Sukhbaatar & Fergus ( 2014 ) ; van Rooyen & Williamson ( 2015 ) ; Menon et al . ( 2015 ) . The idea of above line of works is to define unbiased surrogate loss function to recover the true loss using the knowledge of the noises . We adapt these approaches to reinforcement learning . Though intuitively the idea should apply in our RL settings , our work is the first one to formally establish this extension both theoretically and empirically . Our quantitative understandings will provide practical insights when implementing reinforcement learning algorithms in noisy environments . 2 PROBLEM FORMULATION AND PRELIMINARIES . In this section , we define our problem of learning from perturbed rewards in reinforcement learning . Throughout this paper , we will use perturbed reward and noisy reward interchangeably , as each time step of our sequential decision making setting is similar to the “ learning with noisy data ” setting in supervised learning ( Natarajan et al. , 2013 ; Scott et al. , 2013 ; Scott , 2015 ; Sukhbaatar & Fergus , 2014 ) . In what follows , we formulate our Markov Decision Process ( MDP ) problem and the reinforcement learning ( RL ) problem with perturbed ( noisy ) rewards . 2.1 REINFORCEMENT LEARNING : THE NOISE-FREE SETTING . Our RL agent interacts with an unknown environment and attempts to maximize the total of his collected reward . The environment is formalized as a Markov Decision Process ( MDP ) , denoting as M = 〈S , A , R , P , γ〉 . At each time t , the agent in state st ∈ S takes an action at ∈ A , which returns a reward r ( st , at , st+1 ) ∈ R ( which we will also shorthand as rt ) , and leads to the next state st+1 ∈ S according to a transition probability kernel P , which encodes the probability Pa ( st , st+1 ) . Commonly P is unknown to the agent . The agent ’ s goal is to learn the optimal policy , a conditional distribution π ( a|s ) that maximizes the state ’ s value function . The value function calculates the cumulative reward the agent is expected to receive given he would follow the current policy π after observing the current state st : V π ( s ) = Eπ [ ∑∞ k=1 γ krt+k | st = s ] , where 0 ≤ γ ≤ 11 is a discount factor . Intuitively , the agent evaluates how preferable each state is given the current policy . From the Bellman Equation , the optimal value function is given by V ∗ ( s ) = maxa∈A ∑ st+1∈S Pa ( st , st+1 ) [ rt + γV ∗ ( st+1 ) ] . It is a standard practice for RL algorithms to learn a state-action value function , also called the Q-function . Q-function denotes the expected cumulative reward if agent chooses a in the current state and follows π thereafter : Qπ ( s , a ) = Eπ [ r ( st , at , st+1 ) + γV π ( st+1 ) | st = s , at = a ] . 2.2 PERTURBED REWARD IN RL . In many practical settings , our RL agent does not observe the reward feedback perfectly . We consider the following MDP with perturbed reward , denoting as M̃ = 〈S , A , R , C , P , γ〉2 : instead of observing rt ∈ R at each time t directly ( following his action ) , our RL agent only observes a perturbed version of rt , denoting as r̃t ∈ R̃ . For most of our presentations , we focus on the cases whereR , R̃ are finite sets ; but our results generalize to the continuous reward settings . The generation of r̃ follows a certain function C : S×R → R̃ . To let our presentation stay focused , we consider the following simple state-independent3 flipping error rates model : if the rewards are binary ( consider r+ and r− ) , r̃ ( st , at , st+1 ) ( r̃t ) can be characterized by the following noise rate parameters e+ , e− : e+ = P ( r̃ ( st , at , st+1 ) = r−|r ( st , at , st+1 ) = r+ ) , e− = P ( r̃ ( st , at , st+1 ) = r+|r ( st , at , st+1 ) = r− ) . When the signal levels are beyond binary , suppose there are M outcomes in total , denoting as [ R0 , R1 , · · · , RM−1 ] . r̃t will be generated according to the following confusion matrix CM×M where each entry cj , k indicates the flipping probability for generating a perturbed outcome : cj , k = P ( r̃t = Rk|rt = Rj ) . Again we ’ d like to note that we focus on settings with finite reward levels for most of our paper , but we provide discussions in Section 3.1 on how to handle continuous rewards with discretizations . In the paper , we do not assume knowing the noise rates ( i.e. , the reward confusion matrices ) , which is different from the assumption of knowing them as adopted in many supervised learning works Natarajan et al . ( 2013 ) . Instead we will estimate the confusion matrices ( Section 3.3 ) . 1γ = 1 indicates an undiscounted MDP setting ( Schwartz , 1993 ; Sobel , 1994 ; Kakade , 2003 ) . 2The MDP with perturbed reward can equivalently be defined as a tuple M̃ = 〈S , A , R , R̃ , P , γ〉 , with the perturbation function C implicitly defined as the difference betweenR and R̃ . 3The case of state-dependent perturbed reward is discussed in Appendix C.3

Summary: The paper aims at studying the setting of perturbed rewards in a deep RL setting. Studying the effect of noise in the reward function is interesting. The paper is quite well-written. However the paper studies a rather simple setting, the limitations could be discussed more clearly and there are one or two elements unclear (see below).