Exploit Reward Shifting in Value-Based Deep-RL: Optimistic Curiosity-Based Exploration and Conservative Exploitation via Linear Reward Shaping
Authors: Hao Sun
Lei Han
Rui Yang
Xiaoteng Ma
Jian Guo
Bolei Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reward Shifting
Value-Based Deep RL
Reward Shaping
Conservative Exploitation
Optimistic Exploration
Curiosity-Driven Exploration
Online RL
Offline RL

Abstract: In this work, we study the simple yet universally applicable case of reward shaping in value-based Deep Reinforcement Learning (DRL). We show that reward shifting in the form of a linear transformation is equivalent to changing the initialization of the $Q$-function in function approximation. Based on such an equivalence, we bring the key insight that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration. Accordingly, conservative exploitation improves offline RL value estimation, and optimistic value estimation improves exploration for online RL. We validate our insight on a range of RL tasks and show its improvement over baselines: (1) In offline RL, the conservative exploitation leads to improved performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to tackle the exploration-exploitation dilemma for better sample efficiency; (3) In discrete control tasks, a negative reward shifting yields an improvement over the curiosity-based exploration method.

Exploit Reward Shifting in Value-Based Deep-RL: Optimistic Curiosity-Based Exploration and Conservative Exploitation via Linear Reward Shaping
Hao Sun
Lei Han?
Rui Yang
Xiaoteng Ma
Jian Guo
Bolei Zhou
Abstract
In this work, we study the simple yet universally applicable case reward shaping in value-based Deep Reinforcement Learning (DRL). We show that reward shifting in the form of a linear transformation is equivalent to changing the initialization the Q-function in neural approximation. Based such an equivalence, bring the key insight that a positive reward shifting leads to conservative exploitation, while negative reward shifting leads to curiosity-driven exploration. Accordingly, conservative exploitation improves offline RL value estimation, and optimistic value estimation improves exploration for online RL. We validate our insight on a range of RL tasks and show its improvement over baselines: (1) In offline RL, the conservative exploitation leads to improved performance based on off- the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used tackle the exploration-exploittiiii dilemma for better sample efficiency; (3) In discrete control tasks, a negative reward shifting yields improvement over the curiosity-based exploration method. wtwplle..cm/vvv/////e//o wtt.:o//....o////////eaaarraaaaa
Introduction
While reward shaping is well-established practice in reinforcement learning and has a -ong-standing history 11. 21. specifying certain reward to in- centivize the learning agent requires domain knowl- edge and a thorough understanding of the task (3-6]. Even with careful design and tuning, learning with shaped reward that intends to accelerate learn- ing may on the contrary hinder the learning perfor- mance by incurring the sub-optimal behaviors of the agent Z H. Although Ng et al. (9] theoretically points that optimal policy remains unchanged under certain form of reward transformation, and the later work of Wiewiora al. 100 framework is proposed to guide policies with prior knowledge un- der tabular setting, how reward shifting accommo- dates recent Deep Reinforcement Learning (DRL) algorithms remains much less explored.
Exploration
World Gutaee Eourb-lased
1.0
greedy
Count Dased
0.8
Success Rate
Reward
0.6
0.4
0.1k Breaad Shirt
0.2
04
250
500
750
1000
Episodes
Figure 1: Our work inspired by the observa- tion that reward shifting remarkably helps explo- ration and outperforms count-based exploration in Q learning (left). Reward shifting does not change primal optimal Q-value landscape, and is able learn near-optimal Q-value (right). While count- based exploration suffers from the curse of dimen- sionality, reward shifting can be seamlessly applied to high-dim tasks including continuous control.
In this work, we study a special linear transforma- tion, which is the simplest form of reward shaping.
hs 789 @ ac..kc.uk. University of Cambridge; Tencent RoboticsX: "Hong Kong University of Science and Technology; Tsinghua University; IDEA; University California, Los Angeles
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in value-based DRL 1+1. We start with understanding how such specific kind of reward shaping works in basee DRL function approximations and show that reward shifting equivalent to different Q-value initialization, extending previous discovery of [CC] to the function approximation setting. Figure showcases reward shifting benefits exploration count-based and r-greedy exploration and learns near optimal value function.
maze game and outperforms
Based on such an equivalence, we introduce the key insight of this work:
A positive reward shifting leads to conservative exploitation, and a negative reward shifting leads to (a new type off curiosity-driven exploration.
We demonstrate the application of such an insight in three Deep-RL scenarios: (S1) for offline RL, we show that conservative exploitation induced by reward shifting improves learning performance of off-the-shelff algorithms: (S2) for online RL setting, we show multiple value functions with different reward shifting constants can be used to trade-off between exploration and exploitation, thus improve learning efficiency; (S3) for curiosity-driven exploration, we introduce a simple yet crucial adaptation on a prevailing curiosity-based exploration algorithm, the Random Network Distillation [15,, making it compatible with value-based DRL algorithms. Experiments on a diverse set of tasks including continuous and discrete action space control show our method brings substantial improvement over baselines.
Our contributions can be summarized as follows:
1. Analytically, introduce the key insight that reward shifting is equivalent diversified Q-value network initialization in value-hased DRI., which can be applied to curiosity-driven exploration and conservative exploitation;
2. Practically, we instantiate the key insight to three different scenarios, namely the offline con- servative exploitation, sample-efficient continuous control, and curiosity-driven exploration, to contrast the generality of reward shifting;
3. Empirically, we demonstrate the effectiveness of the proposed method integrated with multiple off-the-shelff baselines on both continuous and discrete control tasks.
2 Preliminaries and Related Work
2.1 Online RL
We follow a standard MDP formulation in the online RL setting, i.e., M (S, R, Po, 7, T. where S IRd denotes the d-dim state space, A is the action space (note for discrete action space [A| < oo and for continuous control |A| 0o), we consider deterministic transition dynamics T SxA +> S and deterministic reward function state distribution. yy is the discount factor and T is the episodic decision horizon. Online RL considers the problem of learning policy π IT S + A(A) (or π E IT: S + A with deterministic policy class), such that the expected cumulative reward Ba,~ 1tw.::...1111..._____
S + TR. Po = p(sg) E ASS) denotes the initial
7'r,( a.) in
the Markovian decision process is maximized. In online RL setting, an agent learns through trials and errors [Л]. through either on-policy [[ or off-policy paradigm [12 1244 19 (T]. In this work, we focus on the off-policy value-based methods which are in general more sample efficient. Specifically, our discussions assume the policy learning is based on a learned Q-value function that approximates the cumulative reward an agent can gain in the following part of an episode.
Formally, Q(st; az) Ex,T ΣT= 7'r(s,, ). and can be estimated by propagating the Bellman operator [B( a) r(s.a) a) + FIBQ(s', a²). For basue- methods, the (soft- coftt-)optimal policy is then produced by
exp (s) Ea exp (x)
Th (als)
where Q" is the optimal Q-value function. We can also set the temperature parameter close to 0 to have the deterministic policy class. Simplifying the notion we have π 77(ss = arg max, Q"'ss, a). Algorithms like DPG po can be used to address the intractable analytical argmax issue that arises in continuous action space. We choose to develop our work on top of prevailing baseline algorithms of
"The novelty this type of curiosity is presented in Section 4.3.2

---

Table I: Reward shifting is flexible to be plugged into both online and offline RI. algorithms to guarantee conservative exploitation pursue optimistic exploration. It covers both discrete and continuous control tasks, with only little additional computational expense. Moreover, optimal policy learned with shifted reward not biased.
Covered Topics
Related Work
Plug-in
Online
Offline
Discrete
Continuous
Unbiased
Examples
Curiosity
Burda al.
[6]
Encemble
Oshand et al.
Exploration
Initialization
/
Rushid et
Optimism
Closek et
Exploitation
Conservatism Policy Constraints
3
5
/
Bharadhwaj
Fojimoto et al.
Reward Shifting
/
Ours
DQN H
minimal example to isolate the source of gains. It should be straightforward to extend the method on top other learning algorithms.
BCQ
2
CQL
24
and TD3 [4]
2.2
Exploration and the -uriosity-Driven Methods
One of the most important issues in online RL is the exploration-explootttion dilemma that the agent must simultaneously learn to exploit its accumulated knowledge on the task and explore new states and actions. Previous works address the exploration problem from various perspectives: for discrete action space tasks, count count-based methods like 28-30 are proposed to motivate the revisiting of under-explored state-action pairs. Specifically, Choshen et al. [3] extended the idea into general settings by constructing an additional MDP for Exploration-valle estimation, as a generalized counter for count-based exploration. To boost exploration in continuous state tasks, curiosity-drive methods are investigated by Houthooft et al. 32.. Pathak et al. [33), Burda et al. 34 [5], where variety of
intrinsic rewards are designed as supplementary to the primal task reward for better exploration. Self-
imitate approaches like Oh et al. [35], Ecoffet et al. [36]. Sun et al. 37 repeat success trajectories but require extra assumptions on the environment.
DORA G1 constructed an additional MDP to estimate the Exploration- value as generalised counter for count- based exploration, yet those count- count-based methods are orthogonal reward shifting: in intrinsic reward methods, an agent must first experience a new (s, a) pair before receiving a high intrinsic reward - this is extremely hard with arg-max style policy. On the other hand, with optimistic initialization, the rarely-visited (s, a) pairs will naturally have higher Q-values before experiencing
the frequently-visited pairs have updated their values with negatively shifted reward. From such a perspective, reward shifting not only works by itself motivates exploratory behaviors, but can also be seamlessly plugged into intrinsic reward methods encourage the first visitation of new states.
The works of DIAYN and DADS G8 39 show that various skills can be developed even without
the primal extrinsic reward. For continuous control tasks, OAC (77] improves the SAC [20]
with
informative action space noise based the optimism in face of uncertainty (OFU) 40 4
GAC 44
addresses the exploration issue with a richer functional class for the policy.
In the work of Rashid et al. 26 the problematic pessimistic initialization is addressed for better exploration, yet the work focuses on specific settings of tabular and discrete control exploration. In the work of Osband et al. 45
25], ensemble models with diverse initialization and randomized priors are used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet those methods are only applicable discrete control tasks. Noted that although the reward shifting can be regarded as a special case of these random priors, it can be distinguished by not changing the optimal Q-value, and its flexibility to be plugged into both continuous and discrete control algorithms.
Random Network Distillation (RND) (5) defines the intrinsic reward as the output difference between a fixed neural network Φ and trainable network dh given stateacctions as the inputs. e.g.,
(ine a) = (0)(s, - 01 a))a
where both networks are activated by a sigmoid function. After optimizing the learnable Φ2 to approximate ói with seen (s, a) pairs the value of riut a) will decay 0 for such state-action pairs that are frequently visited but remain high for thoase are seldom visited.
Or computed with only states, i... (i))s) (d2(s) 01 (s)].
3

---

In this work, we show that exploratory behavior can be achieved simply by shifting the reward function with constant. Thus our method is orthogonal to those previous approaches in the sense that our intrinsic exploration behavior is motivated by high function approximation error in the under-explored state-action pairs. We demonstrate such insight by showing that RND its original design is not suitable for value-based methods in developing exploratory behaviors, but integrating RND with a shifted reward function remarkably improves the learning performance.
2.3 Offline RL
The offline RL, also known as batch-RL, focuses on the problems where interaction with the environ- ment impossible. and the policy can only be optimized based on the logged dataset. In those tasks, a fixed buffer B = {8;. 0g, ri, & [=[NN, collected from some unknown behavior policy æs. is provided.
In general, such a dataset can either be generated by rolling out an expert that generates high-quality
solutions to the task 23 46
47 or non-expert that executes sub-optimal behaviors [47 51] or be
a mixture of both 24 As the agent in the offline RL setting can not correct its potentially biased knowledge through interactions, the most important issue is to address the extrapolation error [ induced by distributional mismatch 40 To address such an issue, a series of algorithms optimize the policy learning under the constraint of distributional similarity
52 X
Bharadhwaj al. 24 propose CQL to solve the offline RL tasks with a conservative value estimation. Specifically, CQL learns the Q-value estimation by jointly maximizing the Q-values of actions sampled from the behavior offline dataset and minimizing the Q-values of actions sampled with pre-defined prior distributions (e.g., uniform distribution over the action space). As we will show in this work, an alternative approach to have a lower bound for the optimal Q-value function is to use an appropriately shifted reward function. This idea leads to the direct application of our proposed framework in the offline setting. In general, reward shift can plugged many distribution-matchin
offline-RL algorithms 23 48
49
52
to further improve the performance with conservative Q-value
estimation.
Table | contextualizes reward shifting with respect to related works we discussed above.
3 A Motivating Example
We start with two intuitive remarks and a "counter-intuitive" motivating example in this section before introducing our method.
7] where |A| < oo, scaling the reward function with linear transformation, i.e., RR.b k. R + b, vk > 0,b E R. such that r kr, + E Rk. does not change the optimal policy induced by the corresponding value function Qi, a) = E, 7//:
Remark 1. Given an MDP . (S,
arg max (s, arg max kQ" (s, + I-7 arg max Q" (s,
(3)
Remark 2.
When |A| oo, scaling the reward function with linear transformation does not change the optimal policy induced by deterministic policy gradient 22 given proper learning rate o/ = n/k: k:
VoJ(uo) E.,, IV.Q" (s,, VolLo(se)
R.z, V. Ti (ellmmmotee) Voploss))/4;; (4)
Remark D and Remark 2 declare the fact that constant reward shifting does not affect the optimal
policy induced by the optimal Q-value function calculated by the shifted reward. However, Figure T
presents "counter-intuitive" results in a demonstrative exploration task of Grid World. In this task, an agent located at the upper left corner of a map needs to explore without reward and reach the
goal point located at the lower right corner. A non-trivial reward of +1 will be assigned only when
the goal is reached. We report learning curves with regard to the episodic success rate in reaching the goal point and the learned Q-values. In this toy example, we find a negative reward shifting remarkably boosts the learning efficiency of Q-learning, and surpasses the conventional count- method for exploration. Moreover, Remark is empirically verified with such toy example: reward shifting does not change the optimal Q- Q-value as well as its induced policy. but it does change the learning dynamics and accelerates the discovery of (near-)optimal policy.
In the following of this work, we study the effect of varying constant bias and fix the scaling factor k 1 to avoid trivial discussions the choices of learning rate- there should be no surprise that

---

Q-
F:Q. Q- (s,,, 1pp1
Q
P: 0b+ : - 22-1 ()
Q
bt
0-1 10
0-
b
Not 0%-1 Raat--
Qiac Qute
A
A
b"
Quap Qo
Figure 2: Illustrative figure for conservative exploitation, with positive constant bias added to the reward function. We use Black lines denote the original value function, and use Red lines to denote the shifted value function with the constant reward shift. Orange lines are used to denote the function approximation during different learning stages and Green line shows the equivalent pessimistic initialization. (a) shifting the reward function with positive bias term b* leads to an uniformly increased Q-value function, namely
0 = + I- during learning, neural network estimator Q initialized with Qo & 0 is optimized to approximate the Q-value functions (e.g., through TD or MC estimation). (b) for any value-based RL algorithm, the value optimization step can be regarded as function F that minimizes the difference between the estimated Q-value function Q: and the optimal one Q", given the interaction experience with the environment (e.g.. a replay buffer &6 for off-pot off-policy methods). Note Q: approximating Q' better than Qi-1 holds in expectation as long as the RL algorithm is designed to approximate the optimal value function. (c) similarly, the optimizer given the same interactive experience (c.g., replay buffer B) will learn to minimize the difference between Q-value function Q and the optimal one Qi-, after re-labeling the rewards in the buffer by r' + (d) according to Remark the optimization conducted in (c) is equivalent (b) with the neural network Q-value estimator initialized as QLa,o Q b+
5+
6+.
i-ş- rather than Qo i.e. by shifting the reward with proper positive value b², we are able to initialize the Q-value network that lower-bounds the optimal Q Q-value.
choosing an appropriate learning rate empirically important. We focus on revealing the importance of selecting the universal bias term b in the reward function through the lens of initialization priors in function approximation and show such a bias is generally helpful for both online and offline settings. In the online settings, it improves learning efficiency, and in the offline settings, it promotes conservative exploitation.
Shifted Priors for Q-value Estimation
4.1 Reward Shift Equals Different Initialization
We start by formally introducing notions and the key idea of this work: reward shifting equals different initialization. We use Figure S illustrate how reward shifting affects function approximation, hence changing the learning dynamics for value-based algorithms. The original optimal Q-value function denoted as Q", and plotted the figures as Black curves. We then denote the shifted optimal Q-value function as Q I which is the Q-value function with the shifted reward r' = + bt. We use Red curves figures to denote those shifted Q-value functions. In this section we provide analysis with positive bias b+ > 0 to illustrate how positively shifted value function affects function approximation, and coordinates the normally applied near-zero initialized function approximators (e.g., neural networks. Orange curves in the figure) to inspire conservative behaviors. The discussion of negative biases is elaborated the next section.
To summarize, shifting the reward function with positive constant is equivalent to initializing the value network with smaller value as the Q-value of unseen state-action pairs during training are much lower than their shifted optimal values, those actions will not be selected in argmax-style policy updates - leading to conservative learning behaviors that benefit policy learning in offline settings.
4.2 (S1) Offline RL: Conservative Exploitation
According the key insight presented in Section conservative Q-value estimation can emerge with a positively shifted reward. And such a value estimation empirically bowed-buunss the optimal Q-value function. As shown in Figure [a), positive constant b+ added to the reward function will lead to a universal positively shifted optimal Q-value function, and the gap between the primal 5+ Q-value function and the new one is -- Optimizing the Q-value approximator with logged data will minimize the difference between the predicted value and the optimal value with observed data. For
5

---

Das Q
Oua: 2
q'
1-7
Ounz 2 2
(a)
(h)
(6)
Figure 3: Illustrative figure for curiosity-driven exploration with negative shifted reward. (a) While adding negative constant value b- on the reward function leads negatively shifted optimal Q-value function Q5- (b) Minimizing the difference between Q-value approximator and the optimal Q-value will enable calculating an upper-bound estimation for Q" which can be used for optimistic exploration. (c) A positive constant shift added to the reward function can be used for conservative policy update.
the unobserved data, the pessimistically initialized Q-value approximator guarantees the prediction is lower than the positively shifted optimal ones, thus conservative exploitation can be achieved with argmax-style value propagation and action execution such value function.
4.3 (Curiosity-Driven Optimistic Exploration
On the other hand, if shift the reward function to the negative side, is equivalent to optimistic initialization. Figure (a-b) illustrate how adding negative bias leads to curiosity-driver exploration. With sufficiently small b (so that |6 larger than the maximal value of any (s, -)-air),, such an upper bound of Q" can be used to conduct curiosity-driven exploration. Intuitively. initializing value network that always predicts value larger than the true optimal value will lead to curiosity- driven exploration behavior, any visited state will be assigned a relatively smaller value and the argmax-style policy will tend to choose under-explored actions.
Based on the analysis above that (1) positive constant shift added to the reward function can be used for conservative policy updates and (2) negative constant shift added to the reward function can be used for ouriosity-driver exploration, we are ready to access both the upper bound, i.c., the optimistic estimation with b=, and the lower bound, i.e., the conservative estimation with b+ of the optimal value function. Formally, we use
b
QLB,((S,)) = (st
(5)
1-?
denote the empirical lower bound of value estimation (cf. Figure d), Orange line), and
b- QUB.i(S a) = Q0-,((s,0) 1-7
(6)
denote the empirical upper bound of value estimation (cf. Figure B(),, Orange line). In both notions, denotes the optimization step. When QLB.O(S, a) = 6+ is able to lower bound the unknown optimal Q-value given sufficiently large bt. (ef. Figure d), Green line). Similarly, Q-value is upper-bounded by QUB,O(s, a) with sufficiently small
with near-zero initialization Qo
Following those notions, we introduce our sample-effiee efficiieff algorithms for both continuous control and discrete action space respectively. We propose practical algorithm for sample- -fffcietf continuous control in Section 4.3.I and focus on special class of -urion exploration method, the RND [5] in Section 4.2
4.3.1
(S2) Online RL: Trading-off Exploration and Exploitation with Reward Shift
According to the principle of optimism in the face of uncertainty (OFU), an exploration bonus that manifests the uncertainty of the Q-value function can be introduced into the value estimation:

---

integrating optimistic exploration with conservative exploitation,
Q a) (s, + QUR a) (s.
= (1 - (0-))QQ - 1-7) 11100-1(..00 - 1 1
b
= (1- (1-=)Q.++))) (s, + ((,,) (1 - 9-b+ + +,3
where the second term with coefficient / denotes exploration bonus that is composed of uncertainty.
For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for neural network, both Qo+ (8, a) and Qb- (8,, a) will give near-zero predictions as consequence of initialization (detailed implications are provided in Appendix C Hence, the explorative bonus
becomes (1-8)b 4.56 which is equivalent to applying another constant reward shift with value of
C = -16++ + +..bbfoomallll formally, we have
Proposition 4.1. Assuming we have access an unbiased estimator for the optimal value function Q", e.g., with Monte-Carlo estimation Q' = E E. 7"r, and the optimization is based on minimizing the MSE between the unbiased estimator and the function approximator i.e., Es = (Qr "2 is equivalent to a linear
20=00 Q 2ny (Q:-1 ),), then the linear combination in Equation combination of the constants with value of G. - B)b -.36.
The proof can be found in the Appendix B According to Proposition 4.1 a grid search for trading-off
between the three hyper-parameters, i.e., the exploration bias b=, the exploitation bias b+ and the coefficient 3 in Equation 7 is trivial as they only lead to a linear combination as C. = (1 3)b+
3b indicating that
Corollary 4.2. Changing the reward shifting constant b is sufficient to trade off between exploration and exploitation.
The corollary says, in principle, a meta-learner can be trained to monitor the learning process and select a proper reward shifting constant automatically [54] [55] to balance exploration and exploitation. For the ease of exposition, in this work we choose focus on the simplest yet effective uniform sampling strategy from multiple shift constants, which has been shown as strong baseline of those meta-learner approaches [56 (57), and leave more complicated meta-leamer--asel shifting constant adjustment for future investigation.
Specifically, we multiple Q-networks to learn with transition tuples as, sampled from the identical buffer that collects the policy's ' historcal interactions with the environment. In propagating Q-values through the temporal difference loss, the primal recorded reward value r is replaced with shifted rewards with different constant biases to update their individual Q-networks. We then uniformly sample one of those learned Q-networks for the optimization of policy networks (e.g., with DPG )22)). It is worth noting that our approach only requires post-hoc revision of the primal reward function, rather than interacting with the environment multiple times collect samples for each value network. We dub the proposed method Random Reward Shift (RRS), and provide the pseudo-code in Algorithm of Appendix D.2
4.3.2 (S3) Compatible Curiosity: Tailored Curiosity-Drivee Exploration for Value-Based RL
In previous works, the curiosity-driven exploration methods are always work with policy-based methods. In this section, cast the key insight introduced above to RND [5] one of the leading algorithms for exploration - to answer why is its vanilla design not suitable for value-based algorithms like Q-learning, and propose variant of RND that is tailored for DQN.
The vanilla RND use two randomly initialized networks Φ1 and d2 to generate the intrinsic reward Tint |0 (s, a) Φ2(5, a)l 0 for exploration. During learning, Φy is a fixed network and the parameters of Φ1 is optimized to approximate the output of d2 for frequently-visited states. We follow Burda al. [15] to bound the intrinsic reward below 1 and use riat.t denote the intrinsic reward after t step of optimization. Specifically, rint, (s,,,o(s,0 a) = 1, V(s,a) - an universally positive bonus is added to the primal reward function beginning.
According to our analysis previous sections, such a positive reward shift is equivalent to pessimistic initialization and will lead to conservative behaviors in Q-value estimation. Therefore, the exploration
7

---

Happol wnllar 100
Pepper Expert 000
Hogper Coper OOL
Pesper Medium
Mestum
Epecot 3mmmm
CENCON Telure
400
Oplimicatinn.m Shap
Optenicaionn Stes
Step
Optmmiationn Stoy
Figure 4: Results on offline RL settings. We verify our key insight that a positive reward shift equals to conservative exploitation thus helps offline value estimation, while a negative reward shift leads to worse performance. Results are from 10 runs with shaded areas indicating the 25%-75% quantiles.
behaviors at the beginning of learning will be hindered, rather than boosted. To overcome such conservative behavior induced by the pessimistic initialization, we proposed use Vimt (s, a) |o (8,a) - (2(5, a)22 T, Vs, a, where 7 is positive constant that assures 'int S 0 is negatively initialized for optimistic exploratory behaviors.
A New Type of Curiosity To further understand the difference between the curiosity-driven method in ways of intrinsic reward and our reward shifting- thiseing-based optimistic initialization, let us consider when do those curiosities work in each case: for intrinsic reward methods, an agent must first experience a new (s, a) pair before receiving a high intrinsic reward - this is extremely hard with an arg-max style policy. On the other hand, with optimistic initialization, the rarely-visited visited (s, a) pairs will naturally have higher Q-values before experiencing it itt-an as the frequently-visitee pairs have updated their values with a negatively shifted reward. From such perspective, reward shifting not only works by itself motivates exploratory behaviors, but can also be seamlessly plugged into intrinsic reward methods to encourage the first visitation of new states.
Experiments
5.1 (S1): Offline RL with Conservative Q-value Estimation
Experiment Setup We start our experiments by demonstrating the effectiveness of reward shifting in offline RL benchmarks. As discussed in Section 4 positive reward shift is equivalent to pessimistic initialization that benefits conservative exploitation. In general, our proposed method can be plugged in any off-the-shelf offline RL algorithm, we choose to verify the effectiveness and generality of such a conservative Q-value estimation based on BCQ (23] and CQL 244, i.e., both Aatributiio-matchingg approach and conservative value estimation approaches in offline RI. To verify our insight, we experiment with both positive reward shift (Pos.) and negative reward shift (Neg.), added on either BCQ or CQL.
Results Figure shows our experiment results. We experiment with both the dataset generated in (23 (Hopper Imitate) and the dataset used in 47 (others), and find in our experiments that learning with the CQL dataset is much more stable. The first two panels show results with BCQ as the backbone algorithm. We observe a clear performance drop during the training of BCQ as the policy overfits the batched dataset. Differently, positive reward shift can alleviate such a problem and outperform BCQ in terms of both best-achieved performance and performance after convergence. The following three panels in Figure 4 use CQL as the backbone. Implementation details and more results can be found in Appendix D.I
Take-Away Message In all experiments, shifting the reward with positive constant improves learning performance, while negative reward shift impedes to efficient learning - as expected.
5.2 (S2): Online RL with Randomized Priors
Experiment Setup We then conduct experiments in the MuJoCo locomotion benchmarks to demon- strate reward shifting improves learning efficiency in the online RL settings. As our implementation is based on TD3, we use TDs-based variants as our baselines: The TD3 is trained with default settings according to Fujimoto et al. [14. We also include Ensemble TD3 and Bootstrapped TD3 as basclines due to they are similar to our work in using multiple Q-nttworks in value estimation.
8

---

Htspperu2
23/2
HanCheetnn
Numanabbd2
Ephood Reture
Spacisl 301-T
TAN-ON BALLE
Figure 5: Results on continuous control tasks, Random Reward Shift (RRS) outperforms its value-based baselines in most environments. Results are from 10 runs with shaded areas indicating the 25%-75%% quantiles.
We follow Osband et al. 45 but extend to the continuous control settings. In continuous control settings, the argmax operator is approximated by the policy network, and multiple policy networks are needed to cooperate with the multiple bootstrapped Q-valuee networks. Otherwise, multiple Q-value networks are not independent of each other thus breaking the condition of bootstrapped value estimation. The Ensemble TD3 presents the baseline performance when multiple Q-networks are used for value estimation in TD3, and works as an ablation of our method where all shift priors are set to be O.
As has been illustrated in Sec. 4.3.1 learning with different reward shifting values is equivalent to learning with optimistic or conservative initialization. In our instantiating of RRS, we use 3 Q-networks with different priors. We empirically find +0.5, 0 work universally good for all environments. Though, future investigation on hyper-parameter may help to further improve the performance.
Results Results are shown in Figure 5 RRS outperforms the vanilla TD3 in all five environments and outperforms all baseline methods in most tasks. In the experiment of Bootstrapped TD3 and Ensemble TD3, we also use 3 Q-networks for fair comparison. Note that there is a trade-off between computational complexity and sample efficiency, i.e., using more Q-networks may further improve the performance at the cost of more computational expenses, as reported in implementation details, pseudo-code of RRS, and ablation studies can be found in Appendix [.2
45 More
Take-Away Message Shifting the reward function can trade-off between exploration and exploita- tion. The ensemble performance of multiple value networks with random reward shifting drastically improve learning efficiency in continuous control.
5.3
(S3): Optimistic Random Network Distillation
Vecamiincargg
Hinklnd Mi Ritooom 8P St
100-13
Eploodo
Figure 6: Value-based RND with shifted prior: Plugging the vanilla RND into DQN is not well-motivated according to our analysis in Section 4.3.2 The insight of equivalence between negative reward shifting and curiosity-driven exploration motivates us to negatively shift the intrinsic reward of RND, which drastically improves DQN-based RND. Results are from 10 runs with shaded areas indicating the 25%-75% quantiles.
Experiment Setting To demonstrate the key insight and effectiveness of reward shifting in opti- mistic exploration, we benchmark with five discrete exploration tasks, including the classic Moun- tainCar control and four MiniGrid navigation tasks [58] environment details can be found at Ap- pendix D33
Results Figure 6 presents the results of 5 different methods for comparison: besides the as-is DDQN and RND baselines, we use DQQ -0.5 to indicate DQN with -0.5 reward shift, and RND 1.0 5 RND -1.5 to indicate RND with -1.0, -1.5 reward shift, respectively. In the following, we use between methods to indicate the former outperforms the latter. Comparing the results:

---

1. reward shifting is equivalent to optimistic initialization, thus boosting exploration
Q:: negative reward shift is equivalent to optimistic initialization and helps exploration.
2. RND is effective for value-based exploration - as long as a negative intrinsic reward is used
① 2: vanilla RND with positive intrinsic reward is always worse than DQN: adding positive
intrinsic reward like RND is harmful for exploration as it is equivalent to a pessimistic initialization.
4 - D: > 3: RND is effective for exploration (i.e., improve over DQN) as long as the intrinsic reward bonus is always negative.
3. reward shifting for optimistic initialization is orthogonal to other exploration methods
2 > 4 increasing the magnitude of the negatively shifted reward can further improve the exploration performance reward shifting can either work in isolation get combined with other exploration algorithms as they are working orthogonally.
Tway-Away Message Reward shifting is equivalent to optimistic initialization, it can help with ex- ploration in value-based methods. Importantly, such an intrinsic motivation is orthogonal to previous count-based methods, such that it can either work in isolation be combined with conventional curiosity-driven exploration methods.
Conclusion
In this work, we studied how reward shifting affects policy learning value-based deep reinforcement learning algorithms. Although constant reward shift should not change the optimal policy induced by the optimal value function, practice such a constant shift does affect the function approximation. and leads to different learning behaviors. Our detailed analysis manifests the fact that a constant reward shift is equivalent to using different initialization in the value function approximation. The proposed idea is then verified through variety of application settings. Specifically, we show that a negative reward shift leads to curiosity- conservative exploitation. Importantly, our analysis reveals that changing reward shifting constant itself is sufficient trading-off between exploration and exploitation. We empirically verify the effectiveness of the proposed method in variety of experiments, including better exploitation in offline RL, cample-efficient learning in continuous control benchmarks, and enhanced curiosity-drive drivenn exploration in value-based discrete control.
exploration, while a positive reward shift helps
While our experiments demonstrate the performance gain is quite robust to the shifting constant, we would like to point out that the theoretical guidance for such a shifting constant is missing. Potential solutions may lie in analysis from the perspective of optimization for black-box models, yet out of the scope of the current empirical study and left for future research.
7 Acknowledgement
We thank all anonymous reviewers, ACs, PCs for their efforts and time in the reviewing process and in improving our paper. This work is done with the warm supports from the MMLab members. We acknowledge the insightful discussions with Ziping Xu, the Hot Spring Harbor group and the van der Schaar Lab members in improving the presentations of this paper. We thank Takuya Kanazawa in pointing out the concurrent work of Dubey et al. [59] that also discusses the effects of shifting terms in reward function. We thank all insightful comments and discussions during the conference poster sessions that helps us in improving the presentation of the paper.
'Code is available at GitHub
10

---

References
[1] Jette Randløv and Preben Alstrøm. Learning to drive a bicycle using reinforcement learning and shaping. In ICML, volume 98, pages 463-471, Citeseer, 1998.
[2] Adam Daniel Laud. Theory and application of reward shaping in reinforcement learning. University of Illinois at Urbana- Chanaa-iaaaa,nn,, 2004.
[3] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster in starcraft using multi-agent reinforcement learning. Nature, 575(7782) -345000--,,--0 2019.
[4] Ilge Akkaya, Marcin Andrychowicz, Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's s cube with a robot hand. arviv preprint arXiv:1910.071 13, 2019.
[5] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 19/2.0660, 2019.
[6] Mahmoud Elbarbari, Kyriakos Efthymiadis, Bram Vanderborght, and Ann Nowé. Ltlf-based reward shaping for reinforcement learning. In Adaptive and Learning Agents Workshop 2021, 2021.
[7] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pages 482-495. PMLR, 2017.
[8] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 2018.
[9] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In Ieml, volume 99, pages 278-287, 1999.
[10] Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforcement learning agents. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 792-799, 2003.
[11] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 1998.
[12] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wttp:::09900799 2015.
[13] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):599533, 2015.
[14] Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.09777 2018.
(15) Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:18100 /2894, 2018.
[16] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pages 1889-1897, 2015.
[17] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 06347, 2017.
[18] Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference Machine Learning, pages 2020-2027. PMLR, 2021.

---

[19] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv: 1,01.12 2016.
[20] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
[21] Hao Sun, Ziping Xu, Yuhang Song, Meng Fang, Jiechao Xiong, Bo Dai, and Bolei Zhou. Zeroth-order supervised policy improvement. arXiv preprint arXiv:2006.0660 2020.
[22] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. PMLR, 2014.
[23] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900 2018.
[24] Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinchart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety crities for exploration. arXiv preprint arXiv:2010. 1449970000..4497, 2020.
[25] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep rein- forcement learning. In Advances in Neural Information Processing Systems. pages 8617-8629, 2018.
[26] Tabish Rashid, Bei Peng. Wendelin Boehmer, and Shimon Whiteson, Optimistic exploration even with pessimistie initialisation. arXiv preprint arXiv:2002. 12174, 2020.
[27] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In Advances in Neural Information Processing Systems, pages 1785- 1796, 2019.
[28] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29:1471-1479, 2016.
[29] Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. International conference on machine learning, pages 2721-2730 PMLR, 2017.
[30] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems (NIPS), volume 30, pages 1-18, 2017.
[31] Leshem Choshen, Lior Fox, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. arXiv preprint arXiv:1804.04012, 2018
[32] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Variational information maximizing exploration. 2016.
[33] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017.
[34] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint 180i 04355, 2018.
[35] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint arXiv: 1806:8800000 1806.06635, 2018
[36] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-cxploration problems. arXiv preprint 10I1:1901.11095, 2019.
12

---

[37] Hao Sun, Zhizhong Li, Xiaotong Liu, Bolei Zhou, and Dahua Lin. Policy continuation with hindsight inverse dynamics. In Advances Neural Information Processing Systems, pages 10265-10275, 2019.
[38] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprini arXiv: #rxXiv/802.06070 2018.
[39] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint arXiv:190..01657, 2019.
[40] Ronen Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(0ct):213-311, 2002.
[41] Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 1 1563 2010.
[42] Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 263-272. JMLR. org, 2017.
[43] Chi Jin, Zeyuan Allen-Zhre Sebastien Bubeck, and Michael Jordan. Is q-learning provably efficient? In Advances in Neural Information Processing Systems, pages 4863-4873, 2018.
[44] Chen Tessler, Guy Tennenholtz, and Shie Mannor. Distributional policy optimization: alternative approach for continuous control. arXiv preprint arXiv:1905.09885, 2019.
[45] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pages 4026-4034, 2016.
[46] Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor Prasanna. Brac+: Going deeper with behavior regularized offline reinforcement learning. 2020.
[47] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[48] Yifan Wu. George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:9911.11611, 2019.
[49] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00940 2019.
[50] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[51] Daniel Jarrett, Alihan Hüyük, and Mihaela Van Der Schaar. Inverse decision modeling: Learning interpretable representations of behavior. In International Conference on Machine Learning. pages 4755 4771. PMLR, 2021.
[52] Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08336. 2020.
[53] Rui Yang, Chenjia Bai, Xiaoteng Ma. Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. arXiv preprint arXiv:2206.0288 2022.
[54] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 161 1.02779, 2016.
13

---

[55] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. Conference on Robot Learning, pages 535-855 PMLR, 2020.
[56] Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Au- tomated curriculum learning for neural networks. In international conference on machine learning, pages 131 1320. PMLR, 2017.
[57] Tambet Matüsen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems, 310))3372233740, 2019.
[58] Maxime Chevalier-Boisver, Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for openai gym. https://gtt httpg/////gtttbb...o///cciiciccb htt.hh/....oo/mnacceeeeeeeaaaaaa wwwp:eeebbbmmmmmiiiiiiii. 2018.
[59] Rachit Dubey, Thomas L Griffiths, and Peter Dayan. The pursuit of happiness: A reinforcement learning perspective habituation and comparisons. 2022.
[60] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforce- ment learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[61] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[62] Clare Lyle, Marc G Bellemare, and Pablo Samuel Castro. A comparative analysis of expected and distributional reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages
1,11,,0199
[63] Gabriel
Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional determin- istic policy gradients. arXiv preprint (00:::/800.00111 2018.
[64] Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, Clara Kwon Starkweather, Demis Hassabis, Rémi Munos, and Matthew Botvinick. A distributional code for value dopamine- reinforcement learning. Nature, 577(7792):67) 1-777, 2020.
[65] Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprint arXiv:2102.05371, 2021.
[66] Grégoire Delétang, Jordi Grau-Moya, Markus Kunesch, Tim Genewein, Rob Brekelmans, Shane Legg, and Pedro A Ortega. Model-free risk-sensitive reinforcement learning. arXiv preprint arXiv:2111.02907, 2021.
[67] Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via g-ensembles. arXiv preprint 10X.::7706005022 2017.
[68] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pages 6131-6141. PMLR, 2021.
[69] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-basee offline reinforeement learning with diversified qensemble. Advances in neural information processing systems, 34:743674447, 2021.
[70] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q- learning: Learning fast without model. arXiv preprint arXiv:2101.05992 2021.
[71] Balaji Lakshminarayananl Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[72] Hao Sun, Boris van Breugel, Jonathan Crabbe, Nabeel Seedat, and Mihaela van der Schaar. Daux: a density-based approach for uncertainty explanations. arXiv preprint arXiv:2207.05100 2022.
14

---

[73] William R Clements, Bastien Van Delft, Menoit-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning, arXiv preprint arX5v 1e0.. 2019.
[74] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[75] Rui Yang. Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint 2r02022200.04478, 2022.
[76] Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob MeGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes) We discussed in the Conclusion
that this work is mainly limited in empirical studies.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our research studies the general problem of reward shifting in reinforcement learning, aiming improve learning efficiency.
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Please refer to Appendix B
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL)? [Yes] Implementation details can be found at Appendix D Code is open-sourced at http:/ghwwwbb.ooo/ooooosssun/n/e
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Implementation details can be found at Appendix D.I D.2 and
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] In our experiments, the results are averaged over 10 runs, and shaded areas indicate the 25% 75% quantile values.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] MIT license.
(c) Did you include any new assets either the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] We follow BCQ and CQL to use the dataset they used in the papers.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
15

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] Not applicable.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Not applicable.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Not applicable.
16


Reviews:
[{"review_id": "XAzD-gDqJ92", "reviewer": "Reviewer_wKjq", "summary": "The paper first shows that reward shifting is equivalent to diversified Q-value network initialization in deep-RL. This kind of phenomenon can influence learning efficiency. Then, using the key insight, the paper presents three scenarios to show the benefit of reward shifting. Finally, the paper conducts experiments to prove the key insight in three scenarios.", "questions": "1. In lines 40-41, is there any theoretical proof of this conclusion? Can you make sure in most of the conditions, the key insight is true?\n2. Why the baselines in 5.2 and 5.3 are different?\n3. How about the total amount of compute between the baselines and the algorithms in this paper?\n4. What are the advantages and disadvantages of the proposed method compared to similar exploration enhancement algorithms? Lack of description or theoretical comparisons.\n5. Why does EnsemleTD3 perform worse than TD3 in Figure 5?\n6. How can fairness be guaranteed by using only 3 Q networks in Figure 5?\n7. There is no doubt that negative reward shifting will lead to more exploratory behavior, but is the Reward Shift approach better than Count-Based or other approaches? In Figure 1, Count-Based and epsilon-greedy can also be adjusted for exploration ability by hyperparameters, are these adjusted to be optimal in the experiment?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper shows that reward shifting is equivalent to diversified Q-value network initialization in deep-RL. This is the key point throughout the paper.\n2. In 4.3.1, the paper proposes that Q-value can be written in the form of a linear combination.\n3. The experiments compare many conditions in one environment. \n\nWeakness:\n1. The experiment results are not reliable enough. In Figure 4, the paper shows four kinds of Hopper environment and one kind of Walker environment. And the result in the Walker environment is not so obvious. More environments and more experiments in each environment are needed to compare. The experiment shown in Figure 6 has the same problem. In Figure 5, only 1e6 steps of environmental interactions are used, resulting in policies that have not yet converged in most tasks.\n2. The theoretical proof is not good insufficient. We should be more concerned about how to find the balance of the conservative and optimistic exploration, rather than attempting different reward shifting settings for each environment.\n3. The innovation of this paper requires further discussion. This paper is based on an intuitive idea to optimize multiple classes of methods such as offline and online. This paper treats each class of methods differently, and there is only one baseline algorithm for each class of methods, which makes me worry about the universality of this optimization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P8N4RWJ07X", "reviewer": "Reviewer_ZKxh", "summary": "This paper proposes a reward shaping mechanism that trades of exploitation/exploration, by considering adding a negative vs. positive intrinsic constant leads to curiosity driven and conservative exploitation respectively, also they  study the effect of positive constant reward multiplication acts as a different learning rate. Based on this simple remark they study how offline and online RL can benefit from this shifting approach.\n\n\n", "questions": "Questions:\n(See above)\n. can the Q_UB-Q_LB provide a good uncertainty estimate locally? \nMinor:\n. missing meaning of each line if both fig.3 and fig. 2 \n. yellow in line166 (no yellow lines exist)\n", "limitations": "- The paper studies how constant bias shift affect the value based learning agents as the constant is the same for all action/state space. In intrinsically motivated exploration this is not the case and it is a different one per state/action. Can the authors elaborate on insights of what this means and whether the current approach can make certain cases blind to exploration since the gap now is too big or small.\n- how other OFU methods apply in this setting, such as Rmax.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper is very well written and clear it studies a simple yet widely applicable problem in RL and curiosity driven approaches for exploration. It provides a derivation to how reward shifts affect the optimal value function, and their equivalence to different initialisations.\nIt tackles an important problem in exploration and provides interesting insights.\n\nWeaknesses:\nThe paper does not show case the ranges under which this constant shifts work well, for too large gaps the the agent will never exploit or explore depending on sign? \nThere is also a trade-off between the multiplicative weight k and the bias? Can the authors clarify how these interact with each other?\nHow to best estimate a good b that provides a LB or UB? Can these parameters be learned? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "N1wk8skA2CU", "reviewer": "Reviewer_FqMf", "summary": "This paper studies how reward shifting can be used to encourage or discourage exploration without modifying optimal behavior in deep Q-learning algorithms. The authors demonstrate how reward shifting can be used in different contexts where exploration is desirable or undesirable and show that it can improve performance on a range of tasks.", "questions": "Some other suggestions (less critical stuff):\n\n-Consider using a non-gridworld example for figure 1. This paper is mostly concerned with deep RL and continuous MDPs, so leading with gridworld results seems odd.\n\n-Figures 2 and 3 I found kind of confusing. I see and like the intent, but it's hard to piece together what's happening based on the caption and equations in the panels. Consider cleaning up the captions and labeling the lines with descriptive labels rather than equations/symbols that haven't been fully introduced/explained at that point in the paper.\n\n-I don't like table 1. It's a matter of preference, but I don't find it informative. Consider focusing on the domains/use-cases for reward shifting instead of the comparison to previous methods.\n\n-I didn't understand the point about RND-1.0 being compared to vanilla DQN. RND has a variable reward bonus, so I'm not sure why it should neatly cancel out when reward shifted.\n\n", "limitations": "This paper doesn't discuss the limitations of reward shifting, which I think is a significant issue. The experiments presented show reward shifting to work well on a variety of tasks, but when does it perform poorly? \n\nI think it might have issues with stochastic rewards, for example (offhand it seems like it might work but with worse bounds?)\n\nHow does reward shifting perform on harder exploration tasks, or larger/more complex off-policy datasets? Even if it under-performs compared to dedicated methods there is value in a simpler and more general but less specialized algorithm, so I'd love to see the results regardless of outcome.\n\nRegarding societal impact, I don't think this work has any direct negative impact risks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The core idea proposed by this paper, reward shifting, is shown to be both simple and effective. A simple method (both conceptually and to implement) is more likely to see wider adoption and further work, so that is a significant strength of this paper.\n\nI do feel there are a number of significant issues to be addressed to make this paper great, however. To be clear, I like this work, but I find it hard to recommend it be accepted without substantial modifications (particularly the points below). If these issues can be addressed I think this will be a good paper and a valuable contribution to the field.\n\n-The clarify of the writing is poor. I found the paper generally understandable, so this isn't a fatal flaw, but it made understanding more difficult than it needed to be. Apologies if I'm mistaken, but this paper comes across as written by scientists not fluent in English, and regardless of acceptance status I strongly encourage the authors to take a careful editorial pass (preferably with an English-fluent colleague's help) to improve the language. This will greatly improve the presentation and lead to more people reading and citing this paper in the future.\n\n-The paper feels somewhat padded. Given the simplicity of the core idea it doesn't take much space to explain how reward shifting works, and the paper as is (IMO) spends more space than needed on it. This space could be better spent addressing some of the points below, or increasing the size of the results figures (which are a little too small to read without zooming in).\n\n-The idea of reward shifting is not novel, though I do not know of any work that explores it in the way this paper does. I think the novel contribution is significant and worthy of publication here, but a longer discussion of related work seems important for a paper like this one that applies an old idea in a new way to good effect.\n --There is related work using value function initialization, which is equivalent to reward shifting, to perform exploration. For example DORA, which is cited in this paper but not discussed, uses biased initialization for exploration, which should perform similarly to reward shifting, and seems like a natural point of comparison.\n--Similarly, biased value ensembles have been proposed before, but such methods don't seem to be discussed (Some related works: \"A distributional code for value in dopamine-based reinforcement learning\" by Dabney et al, \"UCB Exploration via Q-Ensembles\" by Chen et al)\n\n-The experiments cover several cases and domains, which is good, but the comparison to RND seems odd as it focuses on simple minigrid environments and doesn't include any hard exploration continuous MDP tasks where baseline naive exploration methods perform poorly, which were the intended use case for RND. For example, the Sparse HalfCheetah environment proposed by VIME or other continuous control exploration tasks similar to the MuJoCo gym tasks already tested would be easy to compare on without additional modification to the algorithm.\n\n-I'd like to see some exploration of the limitations of reward shifting (see limitations section).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept