Reward Shifting for Optimistic Exploration and Conservative Exploitation

1 INTRODUCTION . While reward shaping is a well-established practice in reinforcement learning applications and has a long-standing history ( Randløv & Alstrøm , 1998 ; Laud , 2004 ) , specifying a certain reward to incentivize the learning agent requires domain knowledge and deep understanding of the task ( Vinyals et al. , 2019 ; Akkaya et al. , 2019 ; Berner et al. , 2019 ; Elbarbari et al. , 2021 ) . Even with careful design and tuning , learning with a shaped reward that intends to accelerate learning may on the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent ( Florensa et al. , 2017 ; Plappert et al. , 2018 ) . Although Ng et al . ( 1999 ) theoretically points the optimal policy will keep unchanged under a special form of reward transformation , and in the later work of Wiewiora et al . ( 2003 ) a framework is proposed to guide policies with prior knowledge under tabular setting , the investigation of how it accommodates recent Deep Reinforcement Learning ( DRL ) algorithms remains much less explored . In this work , we focus on the simplest case of reward shaping , the linear transformation , in valuebased DRL ( Sutton & Barto , 1998 ; Lillicrap et al. , 2015 ; Mnih et al. , 2015 ; Fujimoto et al. , 2018b ) . We start with understanding how such a specific kind of reward shaping works in value-based DRL algorithms . We show that reward shifting , as the simplest reward transformation , is equivalent to engineering initialization of the Q-function estimation , extending previous discovery of ( Wiewiora et al. , 2003 ) to the function approximation settings . Based on such an equivalence , we bring the key insight of this work : a positive reward shifting leads to conservative exploitation , while a negative reward shifting leads to curiosity-driven exploration . We demonstrate the application of such an insight to three downstream tasks : ( 1 ) for offline RL , we show that conservative exploitation can lead to improved learning performance based on off-the-shelf algorithms ; ( 2 ) for online RL setting , we show multiple value functions with different reward shifting constants can be used as a trade-off between exploration and exploitation , thus improving learning efficiency ; ( 3 ) finally , we introduce a simple yet crucial improvement over a prevailing curiosity-based exploration method , the Random Network Distillation ( Burda et al. , 2018b ) , making it compatible with value-based DRL algorithms . We evaluate our idea on various tasks , including both continuous and discrete action space control , resulting in substantial improvements over previous baselines . Our contributions can be summarized as follows 1. we introduce the key insight that reward shifting is equivalent to diversified Q-value network initialization , which can be used to boost both curiosity-driven exploration and conservative exploitation ; 2. motivated by our key insight , we present three scenarios where the reward shifting can benefit , namely the offline conservative exploitation , the online sample-efficient RL , and the curiosity-driven exploration ; 3. we demonstrate the effectiveness of the proposed method integrated with off-the-shelf baselines on both continuous and discrete control tasks . 2 PRELIMINARIES . 2.1 ONLINE RL . We follow a standard MDP formulation in the online RL settings , i.e. , M = { S , A , T , R , ρ0 , γ , T } , where S ⊂ Rd denotes the d-dim state space , A is the action space ( note for discrete action space |A| < ∞ and for continuous control |A| = ∞ ) , T : S × A 7→ S is the transition dynamics , R : S × A 7→ R is the reward function . ρ0 denotes the initial state distribution , i.e. , ρ0 = p ( s0 ) . γ is the discount factor and T is the episodic decision . Online RL considers the problem of learning a policy π ∈ Π : S 7→ ∆A ( or π ∈ Π : S 7→ A with a deterministic policy class ) , such that the expected cumulative reward in the Markov decision process is maximized , i.e. , π = arg max π Eat∼π , st+1∼T , s0∼ρ0 T∑ t=0 γtrt ( st , at ) , ( 1 ) In the online RL setting , an agent normally learns through trials and errors ( Sutton & Barto , 1998 ) , either with an on-policy paradigm ( Schulman et al. , 2015 ; 2017 ; Cobbe et al. , 2021 ) or an off-policy manner ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Wang et al. , 2016 ; Haarnoja et al. , 2018 ; Fujimoto et al. , 2018b ) . In this work , we focus on the off-policy value-based methods which are in general more sample efficient . Specifically , our discussions assume the policy learning is based on a learned Q-value function , that approximates the cumulative reward an agent can gain in the following part of an episode . The Q-value function is defined as Q ( st , at ) = Eπ , T ∑T τ=t γ tr ( sτ , aτ ) , and can be approximated through the Bellman Operator BQ ( s , a ) = r ( s , a ) + γEQ ( s′ , a′ ) . For value-based methods , the ( soft- ) optimal policy is then produced by π∗α ( a|s ) = exp 1αQ ∗ ( s , a ) ∑ a′ exp 1 αQ ∗ ( s , a′ ) , ( 2 ) where Q∗ is optimal Q-value function . We can also set the temperature parameter close as 0 to have the deterministic policy class . Simplifying the notion we have π ( s ) = arg maxaQ ∗ ( s , a ) . Algorithms like DPG ( Silver et al. , 2014 ) can be used to address the intractable analytical argmax issue arises in continuous action space . We develop our work on top of prevailing baseline algorithms of DQN ( Mnih et al. , 2015 ) , BCQ ( Fujimoto et al. , 2018a ) , and TD3 ( Fujimoto et al. , 2018b ) , and it will be easy to extend to other baseline algorithms . 2.2 EXPLORATION AND THE CURIOSITY-DRIVEN METHODS . One of the most important issues in online RL is the exploration-exploitation dilemma ( Sutton & Barto , 1998 ) that the agent must learn to exploit its accumulated knowledge on the task while exploring new states and actions . Plenty of previous works address the exploration problem from various perspectives : In the tasks with discrete action space , count-based methods like Bellemare et al . ( 2016 ) ; Ostrovski et al . ( 2017 ) ; Tang et al . ( 2017 ) are proposed to motivate the policy to explore more on under-explored states . Curiosity-driven methods are investigated by Houthooft et al . ( 2016 ) ; Pathak et al . ( 2017 ) ; Burda et al . ( 2018a ; b ) , where the intrinsic reward is designed as a supplementary to the primal task reward for better exploration . Self-imitate approaches like Oh et al . ( 2018 ) ; Ecoffet et al . ( 2019 ) ; Sun et al . ( 2019 ) repeat success trajectories but require extra assumptions on the environment . The work of DIAYN and DADS ( Eysenbach et al. , 2018 ; Sharma et al. , 2019 ) show that various skills can be developed even without the primal extrinsic reward . For continuous control tasks , OAC ( Ciosek et al. , 2019 ) improves the SAC ( Haarnoja et al. , 2018 ) with informative action space noise based on the optimism in face of uncertainty ( OFU ) ( Brafman & Tennenholtz , 2002 ; Jaksch et al. , 2010 ; Azar et al. , 2017 ; Jin et al. , 2018 ) . GAC ( Tessler et al. , 2019 ) addresses the exploration issue with a richer functional class for the policy . In the recent work of Rashid et al . ( 2020 ) , the problematic pessimistic initialization is addressed for better exploration , yet the work focuses on specific settings of tabular and discrete control exploration . In the work of Osband et al . ( 2016 ; 2018 ) , ensemble models with diverse initialization and randomized priors are used to resemble the insight of bootstrap sampling and facilitate better value estimation , yet those methods are only applicable to discrete control tasks . Noted that although the reward shifting can be regarded as a special case of these random priors , it can be distinguished by not changing the optimal Q-value , and flexible to be plugged in to both continuous and discrete control algorithms . The Random Network Distillation ( RND ) ( Burda et al. , 2018b ) propose to use the difference between a fixed neural network φ1 and a trainable network φ2 to represents the intrinsic reward , e.g. , rint ( s , a ) = |φ2 ( s , a ) − φ1 ( s , a ) | , ( 3 ) when outputs of both networks are activated by a sigmoid function , and φ2 is optimized to approximate φ1 for the visited ( s , a ) pairs . Henceforth , the value of rint ( s , a ) will decay to 0 when such state-action pairs are visited frequently but remain high for seldom visited pairs . In this work , we show that exploratory behavior can be achieved simply by shifting the reward function with a constant , thus our method is orthogonal to those previous approaches in the sense that our intrinsic exploration behavior is motivated by function approximation error . We demonstrate such an insight by showing that RND , with its original design , is not suitable for value-based methods in developing exploratory behaviors , but integrating RND with a shifted reward function can remarkably improve the learning performance . 2.3 OFFLINE RL . The offline RL , also known as batch-RL , focuses on the problems where the interaction with the environment is impossible , and the policy can only be optimized based on the logged dataset . In those tasks , a fixed buffer B = { si , ai , ri , s′i } i= [ N ] is provided . As the agent in the offline RL setting can not correct its potentially biased knowledge through interactions , the most important issue is to address the extrapolation error ( Fujimoto et al. , 2018a ) induced by distributional mismatch ( Kumar et al. , 2019 ) . To address such an issue , a series of algorithms optimize the policy learning under the constraint of distributional similarity ( Kumar et al. , 2019 ; Wu et al. , 2019 ; Siegel et al. , 2020 ) . Bharadhwaj et al . ( 2020 ) proposed CQL to solve the offline RL tasks with a conservative value estimation . Specifically , CQL learns the Q-value estimation by jointly maximizing the Q-values of actions sampled from the behavior offline dataset and minimizing the Q-values of actions sampled with pre-defined prior distributions ( e.g. , uniform distribution over the action space ) . As we will show in this work , an alternative approach to have a lower bound for the optimal Q-value function is to use an appropriately shifted reward function . This idea leads to the direct application of our proposed framework in the offline setting . In general , reward shift can be plugged in many distribution-matching offline-RL algorithms ( Fujimoto et al. , 2018a ; Kumar et al. , 2019 ; Wu et al. , 2019 ; Siegel et al. , 2020 ) to further improve the performance with conservative Q-value estimation .

Summary: This paper studies how different reward shifting (to be precise, adding a bias term on the reward) affects value-based RL algorithms. It illustrates the idea that different reward shifting corresponds to different initialization of the Q networks. Though linear reward shifting does not change the optimal solution for Q, the initialization does affect the learning process and might lead to different stationary points, when equipped with function approximation. The paper further studies the idea in two settings: (1). offline RL, which shows a position bias term could assign the $(s,a)\in\mathcal{D}$ a large value which helps conservative exploitation while in (2) online RL, a negative bias term could facilitate exploration. Some empirical results verify the ideas.  Reward shifting techniques have been studied in the bandit literature, this paper studies how it affects the value-based RL algorithms with function approximations. This is an interesting and important topic, and this paper provides some intuition and empirical results to help the community systematically understand the effects of reward shifting on both online and offline settings.