Reward Shifting for Optimistic Exploration and Conservative Exploitation

1 INTRODUCTION . While reward shaping is a well-established practice in reinforcement learning applications and has a long-standing history ( Randløv & Alstrøm , 1998 ; Laud , 2004 ) , specifying a certain reward to incentivize the learning agent requires domain knowledge and deep understanding of the task ( Vinyals et al. , 2019 ; Akkaya et al. , 2019 ; Berner et al. , 2019 ; Elbarbari et al. , 2021 ) . Even with careful design and tuning , learning with a shaped reward that intends to accelerate learning may on the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent ( Florensa et al. , 2017 ; Plappert et al. , 2018 ) . Although Ng et al . ( 1999 ) theoretically points the optimal policy will keep unchanged under a special form of reward transformation , and in the later work of Wiewiora et al . ( 2003 ) a framework is proposed to guide policies with prior knowledge under tabular setting , the investigation of how it accommodates recent Deep Reinforcement Learning ( DRL ) algorithms remains much less explored . In this work , we focus on the simplest case of reward shaping , the linear transformation , in valuebased DRL ( Sutton & Barto , 1998 ; Lillicrap et al. , 2015 ; Mnih et al. , 2015 ; Fujimoto et al. , 2018b ) . We start with understanding how such a specific kind of reward shaping works in value-based DRL algorithms . We show that reward shifting , as the simplest reward transformation , is equivalent to engineering initialization of the Q-function estimation , extending previous discovery of ( Wiewiora et al. , 2003 ) to the function approximation settings . Based on such an equivalence , we bring the key insight of this work : a positive reward shifting leads to conservative exploitation , while a negative reward shifting leads to curiosity-driven exploration . We demonstrate the application of such an insight to three downstream tasks : ( 1 ) for offline RL , we show that conservative exploitation can lead to improved learning performance based on off-the-shelf algorithms ; ( 2 ) for online RL setting , we show multiple value functions with different reward shifting constants can be used as a trade-off between exploration and exploitation , thus improving learning efficiency ; ( 3 ) finally , we introduce a simple yet crucial improvement over a prevailing curiosity-based exploration method , the Random Network Distillation ( Burda et al. , 2018b ) , making it compatible with value-based DRL algorithms . We evaluate our idea on various tasks , including both continuous and discrete action space control , resulting in substantial improvements over previous baselines . Our contributions can be summarized as follows 1. we introduce the key insight that reward shifting is equivalent to diversified Q-value network initialization , which can be used to boost both curiosity-driven exploration and conservative exploitation ; 2. motivated by our key insight , we present three scenarios where the reward shifting can benefit , namely the offline conservative exploitation , the online sample-efficient RL , and the curiosity-driven exploration ; 3. we demonstrate the effectiveness of the proposed method integrated with off-the-shelf baselines on both continuous and discrete control tasks . 2 PRELIMINARIES . 2.1 ONLINE RL . We follow a standard MDP formulation in the online RL settings , i.e. , M = { S , A , T , R , ρ0 , γ , T } , where S ⊂ Rd denotes the d-dim state space , A is the action space ( note for discrete action space |A| < ∞ and for continuous control |A| = ∞ ) , T : S × A 7→ S is the transition dynamics , R : S × A 7→ R is the reward function . ρ0 denotes the initial state distribution , i.e. , ρ0 = p ( s0 ) . γ is the discount factor and T is the episodic decision . Online RL considers the problem of learning a policy π ∈ Π : S 7→ ∆A ( or π ∈ Π : S 7→ A with a deterministic policy class ) , such that the expected cumulative reward in the Markov decision process is maximized , i.e. , π = arg max π Eat∼π , st+1∼T , s0∼ρ0 T∑ t=0 γtrt ( st , at ) , ( 1 ) In the online RL setting , an agent normally learns through trials and errors ( Sutton & Barto , 1998 ) , either with an on-policy paradigm ( Schulman et al. , 2015 ; 2017 ; Cobbe et al. , 2021 ) or an off-policy manner ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Wang et al. , 2016 ; Haarnoja et al. , 2018 ; Fujimoto et al. , 2018b ) . In this work , we focus on the off-policy value-based methods which are in general more sample efficient . Specifically , our discussions assume the policy learning is based on a learned Q-value function , that approximates the cumulative reward an agent can gain in the following part of an episode . The Q-value function is defined as Q ( st , at ) = Eπ , T ∑T τ=t γ tr ( sτ , aτ ) , and can be approximated through the Bellman Operator BQ ( s , a ) = r ( s , a ) + γEQ ( s′ , a′ ) . For value-based methods , the ( soft- ) optimal policy is then produced by π∗α ( a|s ) = exp 1αQ ∗ ( s , a ) ∑ a′ exp 1 αQ ∗ ( s , a′ ) , ( 2 ) where Q∗ is optimal Q-value function . We can also set the temperature parameter close as 0 to have the deterministic policy class . Simplifying the notion we have π ( s ) = arg maxaQ ∗ ( s , a ) . Algorithms like DPG ( Silver et al. , 2014 ) can be used to address the intractable analytical argmax issue arises in continuous action space . We develop our work on top of prevailing baseline algorithms of DQN ( Mnih et al. , 2015 ) , BCQ ( Fujimoto et al. , 2018a ) , and TD3 ( Fujimoto et al. , 2018b ) , and it will be easy to extend to other baseline algorithms . 2.2 EXPLORATION AND THE CURIOSITY-DRIVEN METHODS . One of the most important issues in online RL is the exploration-exploitation dilemma ( Sutton & Barto , 1998 ) that the agent must learn to exploit its accumulated knowledge on the task while exploring new states and actions . Plenty of previous works address the exploration problem from various perspectives : In the tasks with discrete action space , count-based methods like Bellemare et al . ( 2016 ) ; Ostrovski et al . ( 2017 ) ; Tang et al . ( 2017 ) are proposed to motivate the policy to explore more on under-explored states . Curiosity-driven methods are investigated by Houthooft et al . ( 2016 ) ; Pathak et al . ( 2017 ) ; Burda et al . ( 2018a ; b ) , where the intrinsic reward is designed as a supplementary to the primal task reward for better exploration . Self-imitate approaches like Oh et al . ( 2018 ) ; Ecoffet et al . ( 2019 ) ; Sun et al . ( 2019 ) repeat success trajectories but require extra assumptions on the environment . The work of DIAYN and DADS ( Eysenbach et al. , 2018 ; Sharma et al. , 2019 ) show that various skills can be developed even without the primal extrinsic reward . For continuous control tasks , OAC ( Ciosek et al. , 2019 ) improves the SAC ( Haarnoja et al. , 2018 ) with informative action space noise based on the optimism in face of uncertainty ( OFU ) ( Brafman & Tennenholtz , 2002 ; Jaksch et al. , 2010 ; Azar et al. , 2017 ; Jin et al. , 2018 ) . GAC ( Tessler et al. , 2019 ) addresses the exploration issue with a richer functional class for the policy . In the recent work of Rashid et al . ( 2020 ) , the problematic pessimistic initialization is addressed for better exploration , yet the work focuses on specific settings of tabular and discrete control exploration . In the work of Osband et al . ( 2016 ; 2018 ) , ensemble models with diverse initialization and randomized priors are used to resemble the insight of bootstrap sampling and facilitate better value estimation , yet those methods are only applicable to discrete control tasks . Noted that although the reward shifting can be regarded as a special case of these random priors , it can be distinguished by not changing the optimal Q-value , and flexible to be plugged in to both continuous and discrete control algorithms . The Random Network Distillation ( RND ) ( Burda et al. , 2018b ) propose to use the difference between a fixed neural network φ1 and a trainable network φ2 to represents the intrinsic reward , e.g. , rint ( s , a ) = |φ2 ( s , a ) − φ1 ( s , a ) | , ( 3 ) when outputs of both networks are activated by a sigmoid function , and φ2 is optimized to approximate φ1 for the visited ( s , a ) pairs . Henceforth , the value of rint ( s , a ) will decay to 0 when such state-action pairs are visited frequently but remain high for seldom visited pairs . In this work , we show that exploratory behavior can be achieved simply by shifting the reward function with a constant , thus our method is orthogonal to those previous approaches in the sense that our intrinsic exploration behavior is motivated by function approximation error . We demonstrate such an insight by showing that RND , with its original design , is not suitable for value-based methods in developing exploratory behaviors , but integrating RND with a shifted reward function can remarkably improve the learning performance . 2.3 OFFLINE RL . The offline RL , also known as batch-RL , focuses on the problems where the interaction with the environment is impossible , and the policy can only be optimized based on the logged dataset . In those tasks , a fixed buffer B = { si , ai , ri , s′i } i= [ N ] is provided . As the agent in the offline RL setting can not correct its potentially biased knowledge through interactions , the most important issue is to address the extrapolation error ( Fujimoto et al. , 2018a ) induced by distributional mismatch ( Kumar et al. , 2019 ) . To address such an issue , a series of algorithms optimize the policy learning under the constraint of distributional similarity ( Kumar et al. , 2019 ; Wu et al. , 2019 ; Siegel et al. , 2020 ) . Bharadhwaj et al . ( 2020 ) proposed CQL to solve the offline RL tasks with a conservative value estimation . Specifically , CQL learns the Q-value estimation by jointly maximizing the Q-values of actions sampled from the behavior offline dataset and minimizing the Q-values of actions sampled with pre-defined prior distributions ( e.g. , uniform distribution over the action space ) . As we will show in this work , an alternative approach to have a lower bound for the optimal Q-value function is to use an appropriately shifted reward function . This idea leads to the direct application of our proposed framework in the offline setting . In general , reward shift can be plugged in many distribution-matching offline-RL algorithms ( Fujimoto et al. , 2018a ; Kumar et al. , 2019 ; Wu et al. , 2019 ; Siegel et al. , 2020 ) to further improve the performance with conservative Q-value estimation .

Summary: This paper shows a key observation that the linear transformation of reward shaping is equivalent to diversified Q-value network initialization. Based on such observation, the paper presents three scenarios where the reward shifting can benefit: the offline conservative exploitation, the online sample-efficient RL, and the curiosity-driven exploration. The algorithm is evaluated on the continuous control tasks and discrete control tasks.