Reward Shifting for Optimistic Exploration and Conservative Exploitation
Authors: Hao Sun
Lei Han
Jian Guo
Bolei Zhou
Venue: ICLR 2022 Submitted (2022)
Keywords: Reward Shift
Reinforcement Learning
Batch RL
Offline RL
Online RL
Curiosity-Driven Method

Abstract: In this work, we study the simple yet universally applicable case of reward shaping, the linear transformation, in value-based Deep Reinforcement Learning. We show that reward shifting, as the simplest linear reward transformation, is equivalent to changing initialization of the $Q$-function in function approximation. Based on such an equivalence, we bring the key insight that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration. In this case, a conservative exploitation improves offline RL value estimation, and the optimistic value estimation benefits the exploration of online RL. We verify our insight on a range of tasks: (1) In offline RL, the conservative exploitation leads to improved learning performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to trade-off between exploration and exploitation thus improving learning efficiency; (3) In online RL with discrete action space, a negative reward shifting brings an improvement over the previous curiosity-based exploration method.

Under review as a conference paper at ICLR 2022
REWARD SHIFTING FOR OPTIMISTIC EXPLORATION AND CONSERVATIVE EXPLOITATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we study the simple yet universally applicable case of reward shap- ing, the linear transformation, in value-based Deep Reinforcement Learning. We show that reward shifting, as the simplest linear reward transformation, equiva- lent to changing initialization of the Q-function in function approximation. Based on such an equivalence, we bring the key insight that positive reward shift- ing leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration. this case, a conservative exploitation improves offline RL value estimation, and the optimistic value estimation benefits the ex- ploration of online RL. We verify our insight on range of tasks: (1) In offline RI., the conservative exploitation leads to improved learning performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value func- tions with different shifting constants can be used to trade-off between exploration and exploitation thus improving learning efficiency; (3) In online RL with discrete action space, negative reward shifting brings an improvement over the previous curiosity-based exploration method.
INTRODUCTION
While reward shaping is well-established practice in reinforcement learning applications and has a long-standing history (Randløv & Alstrom 1998 Laud 2004). specifying certain reward to in- centivize the learning agent requires domain knowledge and deep understanding of the task (Vinyals) itt 2019 Akkaya et al. 2011 Berner et al. [019 Elbarbari et al. (021). Even with careful design and tuning. learning with a shaped reward that intends to accelerate learning may on the contrary hinder the learning performance by incurring the sub-optimal behaviors of the agent (Florensa et al. 2017 [Plappert al., 2018). Although Ng et al.] (1999) theoretically points the optimal policy will keep unchanged under a special form of reward transformation, and in the later work of Wiewioral jet al. (2003) a framework proposed to guide policies with prior knowledge under tabular setting. the investigation of how it accommodates recent Deep Reinforcement Learning (DRL) algorithms remains much less explored.
In this work, we focus the simplest case of reward shaping, the linear transformation, in value- based DRL (Sutton & Barto 1998 Lillicrap et al. 2015) Mnih et al.) 2015) Fujimoto et al. 2018b). We start with understanding how such specific kind of reward shaping works in value- based DRL algorithms. We show that reward shifting, as the simplest reward transformation, is equivalent to engineering initialization of the Q-function estimation, extending previous discovery of (Wiewioral et al. 2003) to the function approximation settings. Based on such an equivalence, we bring the key insight of this work: a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to suriosity-driver exploration. We demonstrate the application of such an insight to three downstream tasks: (1) for offline RI, we show that conservative exploitation can lead to improved learning performance based on off-the-shed algorithms; (2) for online RL setting, we show multiple value functions with different reward shifting constants can be used as a trade-off between exploration and exploitation, thus improving learning efficiency; (3) finally, we introduce simple yet crucial improvement over a prevailing curiosity-basee Network Distillation (Burda et al. 2018b), making it compatible with value-basee DRL algorithms. We evaluate our idea on various tasks, including both continuous and discrete action space control, resulting in substantial improvements over previous baselines.
exploration method, the Random

---

Under review as a conference paper at ICLR 2022
Our contributions can be summarized as follows
1. we introduce the key insight that reward shifting is equivalent to diversified Q-value net- work initialization, which can be used boost both curiosity-driven exploration and con- servative exploitation;
2. motivated by our key insight, we present three scenarios where the reward shifting can benefit, namely the offiin conservative exploitation, the online sample-efficient RI., and the curiosity-driven exploration;
3. we demonstrate the effectiveness of the proposed method integrated with aff-the-shell base- lines on both continuous and discrete control tasks.
PRELIMINARIES
2.1 ONLINE RL
We follow standard MDP formulation in the online RL settings, i.e., M - 7; T) where S C Id denotes the d-dim state space, A is the action space (note for discrete action space the transition dynamics, R S x A + IR is the reward function. Po denotes the initial state distribution, i.e., Pu p(su). the discount factor and T the episodic decision. Online RL considers the problem of learning policy E IT S + AA (or π E IT: S + A with a deterministic policy class), such that the expected cumulative reward the Markov decision process maximized, i.e.,
|Al Do and for continuous control |A| oo), T S A
arg max Ea 380000000000000
Entra (r),
2m
In the online RL setting, an agent normally learns through trials and errors (Sutton & Barto 1998), either with on-policy paradigm (Schulman al. 2015 2017 Cobbe al. (0221)
off of tof-oolicy
manner (Mnih et al., 2015) Lillicrap et al.) 2015] Wang et al. [016)] al. 2018b). In this work, we focus on the off-policy value-based methods which more sample efficient. Specifically, our discussions assume the policy learning is based on learned Q-value function, that approximates the cumulative reward an agent can gain in the following part of an episode. The Q-value function is defined as Q(s;, a,) = Ex.T ΣT rrss ar), and can be approximated through the Bellman Operator (Q((s, a) = r(s,a) 7BQ((s d'). For value-based methods, the (soft-)optimal policy is then produced by
20266]||aamaa Haarnoja et al. 2018 Fujimoto
general
exp (s)
TH (als) E. exp (s,
where Q" is optimal Q-value function. We can also set the temperature parameter close 0 have the deterministic policy class. Simplifying the notion we have ((s) arg max, Q" (s a). Algo- rithms like DPG (Silver al.] (014)) can used to address the intractable analytical argmax issue arises in continuous action space. We develop our work on top of prevailing baseline algorithms of DQN (Mihh al., 2015), BCQ (Fujimoto al. (2088),, and TD3 (Fujimoto et al.) 2018b), and will be easy to extend other baseline algorithms.
2.2 EXPL.ORATION AND THE CURIOSITY-DRIVER METHODS
One of the most important issues in online RL is the -etrtitton-exlloiatiiinn dilemma (Sutton & Barto 1998) that the agent must learn exploit its accumulated knowledge on the task while ex- ploring new states and actions. Plenty of previous works address the exploration problem from various perspectives: In the tasks with discrete action space, count-based methods like Bellemare 2017); Tang et al. (2017) are proposed to motivate the policy to ex- states. Curiosity- driven methods are investigated by [outhooff et al. (2016); Pathak et al. 2);17); Burda et al. (2018a b). where the intrinsic reward is designed
et al. (2016); Ostrovski et al.
plore more on under-explored
supplementary to the primal task reward for better exploration. Self-imitate approaches like Oh (2019): Sun et al. (2019) repeat success trajectories but require extra al. (2018) Fcoofet al.
2

---

Under review as a conference paper at ICLR 2022
assumptions on the environment. The work of DIAYN and DADS (Eysenbach et al. 2018 Sharma jet al. 2019) show that various skills can be developed even without the primal extrinsic reward.
For continuous control tasks, OAC (Ciosek et al., 2019) improves the SAC (Haarnoja et al. 2018)
with informative action space noise based on the optimism in face of uncertainty (OFU) (Brafman
& Tennenholtz] 2002 Jaksch al. 2010: Azar al. 2017 Jin al., 2018). GAC (Tessler al.)
2019) addresses the exploration issue with richer functional class for the policy.
In the recent work of Rashid al. (2020), the problematic pessimistic initialization is addressed for better exploration, yet the work focuses on specific settings of tabular and discrete control explo- ration. In the work of [Osband et al. (2016 2018). ensemble models with diverse initialization and randomized priors are used to resemble the insight of bootstrap sampling and facilitate better value estimation, yet those methods are only applicable to discrete control tasks. Noted that although the reward shifting can be regarded as a special case of these random priors, it can be distinguished by not changing the optimal Q-value, and flexible to be plugged in to both continuous and discrete control algorithms.
The Random Network Distillation (RND) (Burda et al. 201 propose to use the difference be- tween fixed neural network á1 and a trainable network d2 to represents the intrinsic reward, e.g.
Time (s, a) (s2 a) +1 (s, as)
when outputs of both networks are activated by a sigmoid function, and d2 is optimized to approx- imate ④i for the visited (s, a) pairs. Henceforth, the value of Tine Tine(s, a) will decay to 0 when such state-action pairs are visited frequently but remain high for seldom visited pairs.
In this work, we show that exploratory behavior can be achieved simply by shifting the reward func- tion with a constant, thus our method orthogonal to those previous approaches the sense that intrinsic exploration behavior is motivated by function approximation error. We demonstrate such an insight by showing that RND, with its original design, is not suitable for value- based meth- ods in developing exploratory behaviors, but integrating RND with shifted reward function can remarkably improve the learning performance.
2.3 OFFLINE RL.
The offline RL, also known as batch-RL, focuses on the problems where the interaction with the environment is impossible, and the policy can only be optimized based on the logged dataset. In those tasks, a fixed buffer B = {si, ds ri "/ is provided. As the agent in the offline RL setting can not correct its potentially biased knowledge through interactions, the most important issue is to address the extrapolation eиror (Fujimoto et al.) 2018a) induced by distributional mismatch (Kumar al. 2019). To address such an issue, series of algorithms optimize the policy learning under the constraint of distributional similarity (Kumar et al. 2019 Wu et al. 2019 Siegel et al. 2020).
Bharadhwaj et al. (2020) proposed CQL to solve the offline RL tasks with a conservative value es- timation. Specifically, CQL learns the Q-value estimation by jointly maximizing the Q-values of actions sampled from the behavior offline dataset and minimizing the Q-values actions sampled with pre-defined prior distributions (e.g., uniform distribution over the action space). As we will show in this work, an alternative approach to have a lower bound for the optimal Q-value func- tion is to use an appropriately shifted reward function. This idea leads to the direct application of our proposed framework in the offline setting. In general, reward shift can be plugged in many distribution-matching offline-RL algorithms (Fujimoto et al., 2018a| Kumar et al. [019 Wu al.)
2019 Siegel et al. 2020) to further improve the performance with conservative Q-value estimation.
A MOTIVATING EXAMPLE
We start with motivating example that may look counterintuitive: in the prevailing contin- uous control environment of Humanoid, a three-dimensional bipedal robot is simulated based on the MuJoCo engine. The learning objective is control the robot to walk forward as fast as possible, without falling over. There are two types of positive reward signals: (I) to en- courage forward-moving, i.e., the reward proportional to the horizontal displacement, and
3

---

Under review as a conference paper at ICLR 2022
(2) to encourage the robot avoid falling over, i.e., binary alive bonus whether the mass center of the robot is higher than some certain threshold. Although the design of (2) is intended to provide a curriculum for the agent, we show in Figure that such a currieulum, the contrary, hinders the learning performance of such bipedal robot to run fast.
Humanoid
Episodic Return
7000
5000
2000
2000
TD3
The orange curve, TD3 w/o Alive Bonus, shows the smoothed learning curve of a TD3 agent trained with the alive bonus set be O; the blue curve, TD3 w/ Alive Bonus, shows the av- eraged learning curve of vanilla TD3 agent trained with the alive bonus set to be as default. Noted that for both ap- proaches, we use identically the same primal task in policy evaluation to make sure the curves are comparable. In gen- eral curriculum learning improves learning efficiency (Graves) ctt al. 2017, Matüsen et al., 2019; 9 Portelas et al. 2020), how- ever, in the environment of Humanoid, such curriculum design, which aims at helping the agent learning to run after knowing how to avoid falling over, hinders the learning efficiency and asymp- totic performance.
TD3 Shifted
Timesteps
Figure 1: Humanoid agents trained with a linear reward shift drastically gain asymptotic performance.
Remark 1. Given MDP M (S,A,,, Pa, 7, T) where IA < xo, scaling the reward function with linear transformation, i.e., Ra,s = R- Ak > 0,b E R, do not change the optimal policy induced by
b
n.' (s) arg max Li,6(s a) arg max kQ' (s) aeA
1-7 arg max ("ss, a), aEA
(4)
aEA
Remark 2. Given an MDP M = [S,A,T, R, Po. Y.T. where |A| = oc. scaling the reward function with linear transformation, i.e., Rx,5 k- b. wk > 0, R. do not change the optimal policy induced by deterministic policy gradient (Silver et al. 2014), given proper learning rate:
VoJ(Hu)) = E., Q= (:)a))) (e)ss=()())) Velo(s:)] E., V a (i, (i)la- [oploss)]/ks (5)
In the following, we will focus on the scenarios when k empirically important) discussions on different learning rates. We reveal the importance of selecting the universal bias term, i.e., the reward function, through the lens of initialization priors in function approximation. We further show such a bias term can be utilized not only in the online RL. settings to improve learning efficiency but also the offline RL settings to conduct conservative exploitation with batched data.
avoid the trivial (though maybe
SHIFTED PRIORS FOR VALUE ESTIMATION
4.1 REWARD SHIFT EQUALS TO DIFFERENT INITIALIZATION
We start by introducing the central idea of this work: reward shift equals different initialization. Specifically, we illustrate the basic idea with Figure 2 The black curves denote the primal optimal Q-value functions (e.g., Q*), and the red curves denote the optimal Q-value functions with shifted is positive constant bias, and we will show different
reward r' = + b+. where b+ choice of such a bias leads to a different motivation in Q-value estimation. The yellow lines denote Q-value estimators, e.g., neural network predictions of the corresponding Q-values. (a) shift the reward function with a positive bias term 1+ will lead to an uniformly increased Q-value function, b!_ namely Qf+ Q' T-y. during learning, neural network estimator Q initialized with Qo R 0 is optimized to approximate the Q-value functions (c.g.. through Temporal Difference or Monte Carlo estimation).
(b) for any value-based RL algorithm, the value optimization step be regarded function F that minimizes the difference between the estimated Q-value function Qu and the optimal one Q' given the interaction experience with the environment (c.g., a replay buffer for off -ooicooicy methods).
4

---

Under review as a conference paper at ICLR 2022
Q Q. (s,
Q I-y
R0. - Q'1 <0|=
021
Qr
1
Qo
(a)
(b)
F: Po - 1..1 (s.
Ro- Cg+1 10ote-1
1
bt Qia 0
1-y Quao 0o I-y
(c)
(d)
Figure 2: Illustrative figure for conservative exploitation, with a positive constant bias added to the reward function.
(c) similarly, the optimizer given the same interactive experience (e.g., replay buffer B) will learn to minimize the difference between Q-value function Qs+ and the optimal one Q after re-labeling the rewards in the buffer by r' = b
(d) according Remark the optimization conducted in (c) equivalent (b) with the neural network Q-value estimator initialized at Qo ~ 0 1-7" rather than Qo R O. i.e., by shifting the reward with proper positive value b*, we are able to initialize the Q-value network that lower- bounded the optimal Q-value.
To summarize, shifting the reward function with positive constant is equivalent initializing the value function network with smaller value, hence during training, the Q-value of unseen state- action pair is far lower than the optimal value and hence will not be selected in policy update, leading to conservative learning behavior.
4.2 CONSERVATIVE EXPLOITATION
We begin with natural application the offline setting, where policy doesn't interact with the en- vironment and only learns through a logged data set collected from an unknown behavior policy Tg, which can either an expert that generates high-quality solutions to the task (Fujimoto et al. 2018a] Zhang et al. 2020 Fu et al. 2020) non-expert that provides actions that are sub-optimal (Wu
ct al. 2019 Kumar et al. 2019 Agarwal et al. 2020 Fu et al. 2020 Jarrett et al. 2021 or a mixture
of both [Bharadhwaj et al. 2020).
we introduce a simple yet effective approach for conservative Q-value estimation that learns heuristically put lower bound on the optimal Q-value function in batch settings. Hark back to Figure 2 a), a positive constant b+ added to the reward function will lead to uniformly (positively) shifted optimal Q-value function, and the gap a+ between the primal Q-value function and the new one is I- Optimizing the Q-value function with logged data (e.g., a fixed replay buffer) will minimize the difference between the predicted value and the optimal value with observed data. For the unobserved data point (in the state-action space A), the near-zero initialization guarantees the prediction is lower than the optimal Q-value, thus conservative exploitation be conducted with such value function.
Based on the basic idea we presented in Section 4.1
5

---

Under review as a conference paper at ICLR 2022
Que:
Qute 1-y
Oca 2 a
b
1-y
4o =
0% Q' b-y
Q Oety
Diar S
(a)
(b)
(c)
Figure 3: Illustrative figure for curiosity-driven exploration with a negative shifted reward
4.3 (CURIOSITY-RRIVES OPTIMISTIC EXPLORATION
On the other hand, if we shift the reward function to the negative side, it is equivalent optimistic initialization. Figure 3 (a-b) illustrate how adding a negative bias leads to curiosity-driven explo- ration: while adding negative constant value 6 on the reward function lead negatively shifted optimal Q-value function Q7- (Figure 3 (a)), minimizing the difference between Q-value approx- imator and the optimal Q-value will enable calculating an upper- bound estimation for Q", as shown in Figure b). With sufficiently large b (so that b larger than the maximal value of any 8, a a-pair), such upper bound of Q* can be used conduct curiosity-drive driven exploration. Intuitively. initializ- ing a value network that predicts value larger equal than the true value will lead to curiosity exploration, as any visited state will be assigned low value and the policy that learns to perform the action with higher value will tend to choose novel actions.
Based on the discoveries above that (I) a positive constant shift added to the reward function can be used for conservative policy update, as shown in Figure () and (2) negative constant shift added to the reward function can be used for curiosity-driven exploration, as shown in Figure 3Jb). We are ready to access both the upper bound, i.e., the optimistic estimation with b and the lower bound, ie., the conservative estimation with 1i of the optimal value function. Henceforth, we are ready to introduce sample-efficient algorithms for both continuous control and discrete action space respectively. We propose a practical a algorithm for general continuous control in Sec. 4.3.1 while focusing on a special class of curiosity- driven exploration method, the RND, in Sec. 4.3.2
4.3. EMPPLEEE CONTINUOUS CONTROL WITH REWARD SHIFT
Based on the principle of optimism the face of uncertainty (OFU), we introduce a exploration bonus that manifests the uncertainty of the Q-value function. Specifically, the basic idea starts from integrating optimistic exploration with conservative exploitation, i.e.,
(aa.:(s,)++@Quuuee ((+, a) = (LB, a) + ((QUB,((s.a) a) (s,,
= (1 - 3) (s(,a) - 1-7 3(Qb ,(s,a) b
b
(6)
= (1 - B) ((s, a) + (Qs-,((s,)) (1 --6+ + ,6
where the second term with coefficient 3 denotes exploration bonus that is composed of uncertainty.
For those under-explored state-action pairs, i.e., extremely out-of-distribution samples for ral network, both Qb+ (s, and Q (s, a) will give near-zero predictions consequence of initialization (detailed implications are provided in Appendix (B). henceforth, the explorative bonus becomes (1-9)** +00 Note this is equivalent to applying another constant reward shift with 1- B)b+ _0=-
value e, = (1
For the explored state-action pairs close to the samples from the replay buffer, have Proposition 1. Assuming we have access to an unbiased estimator for the optimal value function Q', e.g., with Monte- Carlo estimation Q- = E E, yrr, and the optimization is based on minimizing the MSE between the unbiased estimator and the function approximator, i.e., ? =Q²- Q2)

---

Under review as a conference paper at ICLR 2022
i -1--1-22(Q--1-- Q*), then combining the linear combination in Equation 6 is equivalent to using a linear combination of the constants with value of c. = (1-3)6 + 36
According to Proposition I a grid search for trading- off between the three pyper-parameters: the exploration bias b_ the exploitation bias by and the coefficient in Equation (6) is trivial as they only lead to a linear combination as 6, = (1 - =B)b+ + _bb_ . In principle, meta-learner can be trained to monitor the learning process and select a proper constant automatically (Graves et al.) 2017] Matüsenn et al., 2019; Portelas et al. (020). In this work we focus on simple yet effective uniform sampling strategy from multiple shift constants, which is shown as a strong baseline in Graves et al. (017 ; Matiien et al. (2019), and leave more complicated meta-learner-bat based approaches in future investigation.
Specifically, we use multiple Q-networks to learn with transition tuples (s, s') sampled from the identical buffer that collects the policy's interaction history with the environment. For each Q-network, the primal r is replaced with a new reward with shifted constant bias for temporal difference updates. Those learned Q-networks are sampled uniformly during the training of policy network. It is worth noting that our approach only requires post-hoc revision of the primal reward function, rather than interacting with the environment multiple times to collect samples for each value network.
4.3.2 IMPROVING VALUE-BASEE CURIOSITY-DRIVED EXPLORATION
The discussion above casts the curiosity-driven exploration method as a special case in the reward shift equals to initialization perspective: exploration with RND (Burda al. 2018h) is equivalent to selecting b+ = 0, i.e., using the primal task reward for exploitation, and using b as a fixed random function over S x A, i.e., b" (s,a) |0 101.0($. (5, s, a) (2 a)l² Vs, O, where Φ1,0 and Φ2 are two neural networks with different random initializations. While Φ1.0 is learnable, Φ2 is set to be fixed during learning as a random curiosity prior (Osband et al. [ 2018). Without loss of generality, we can assume &2 = 1 as constant initialization, and 61.0 O. Then the exploration behavior is fully controlled by the scale of external reward, which is set to be 1.0 with external reward clipped to [-1, in (Burda) et al. 2018b). The final equivalent objective in this case is to c, (1.1 1
2, ft= a)
0, 1,2,.....
Specially, 01,0 = and c, = 1 at beginning.
According to our analysis in the previous section, such positive shift directly leads to conservative behaviors in Q- estimation, therefore may hinder the exploration behaviors at the beginning, i.e., when |01,1 1| > O. The exploration bonus in RND only becomes effective when visiting seldom visited states after the predictions of Φ1, of frequently-visiter states are close to 1, and hence encouraging stepping into those novel states and discover new knowledge. To overcome the conservative tendency induced by Q-value estimation in RND, we propose to use b b=(s a) = (01,0(s.a) a)) Φ2(s,a) 1² I, Vss a, where (2(5s, 2) that assures b- (s,a) is negative-initien regative-initializedd for optimistic exploratory behaviors.
positive constant (e.g.. max, a (01.0 a)
is worth noting that the curiosity introduced by exploratory bonuses like RND focuses on increas- ing the temporary Q-value of rarely visited states, which still relies the visitation behavior itself in random exploration. i.e., without stepping into a non-frequent visited state, the agent will never receive the novelty bonus based the state. On the other hand, the mechanism of shifting the reward by a constant is an universal optimistic exploration bonus as the curiosity is fused into the initialization across the entire state-action support.
EXPERIMENTS
5.1 OFFLINE REINFOR LEARNING WITH CONSERVATIVE Q- Q-VALUE ESTIMATION
We first demonstrate the proposed method in the offline RL setting. As has been discussed in Sec- tion 4.2, shifting the primal reward function with a positive constant provides a natural way of conservative exploitation. By adding positive reward shift, any visited (s, a) saved in the given replay buffer B (logged dataset) will be assigned a large reward, while under-explored (s, a) pairs remain have the low values due to initialization, as illustrated in Figure our proposed method can be plugged in to any off-the-shelf offline RL algorithm. in this work we demonstrate the effectiveness of such conservative Q-value estimation based BCQ (Fujimoto)
2 (d). Although in general
7

---

Under review as a conference paper at ICLR 2022
Hopper Imitate BCQ
Hopper Expert
Hopper Expert COL
3030
3000
Egisodic Return
Episodic Return
Eolsodic Return
5300
20:0
0.40
10:00
2000
3300
1500
530
140
5200
2300e isease 260300 25068 200303
203006
600309
#o00e
300e93
1000093
5950es 1009900 199300 250099 25009922688999
Optimization Step
Optimization Step
# Optimization Step
Hopper Medium COL
Walker Medium CQL
Vanilla
Episodic Return
2030
Episodic Return
Vanila-Pos (Durs)
2093
Vanila-Neg. (Conbrat
2530
3090
Loas
910
100383 (E 2at.
Optimization Step
Optimization Step
Figure 4: Results offline RL settings. We verify our key insight that positive reward shift equals to conservative exploitation thus helps offline value estimation, while a negative reward shift leads to worse performance.
et al. 201 and CQL (Bharadhwaj et al. 2020). i.e., both distributtin-matching approach and conservative value estimation approaches in offline RL.
To verify our insight, we experiment with both positive reward shift (Pos.) and negative reward shift (Neg.), added on either BCQ or CQL. Figure shows our experiment results. We experiment with both the dataset generated in BCQ (Hopper Imitate BCQ) and the dataset used in CQL (Fu al., 2020) (others), and find in our experiments that learning with the CQL dataset is much more stable. the first two figures show results with BCQ as the backbone algorithm, where the former shows results on the vanilla BCQ dataset, while the latter shows results on the CQL dataset. The following three experiments in Figure use CQL as the backbone. In all experiments, using a positive reward shift leads to improved learning performance, while negative reward shift leads to performance decay, as expected. Implementation details and ablation study can be found in Appendix C.I
5.2 ONLINE REINFORCEMENT LEARNING WITH RANDOMIZED PRIORS
We then conduct experiments the online RL settings. We demonstrate proposed method in the MuJoCo locomotion benchmarks. As our implementation is based on TD3, we use TD3-based variants as our baselines: The TD3 is trained with default settings according to Fujimoto et al. (2018b). We also include Ensemble TD3 and Bootstrapped TD3 as basclines due to they are similar to work in using multiple Q-networks in value estimation. We follow [Ohand et al. (016)) but extend it to the continuous control settings. Noting that in the continuous control setting, the argmax operator is approximated by the policy network, multiple policy networks are needed to cooperate with the multiple bootstrapped Q-value networks. Otherwise, multiple Q-value networks are not independent of each other thus breaking the condition of bootstrapped value estimation. The Ensemble TD3 presents the baseline performance when multiple Q-networks are used for value estimation in TD3, which also works as an ablation of our method when all reward shift priors are set to be O. As has been illustrated in Sec. 1.3.1 learning with different reward shifting values equivalent to learning with optimistic or conservative initialization. In our method of Random Reward Shift (RRS), we use 3 Q-networks with different priors. In our experiments, we found +0.5, 0 works universally as a default setting. Though, further investigation on hyper-parameter may help to further improve the performance.
Results are shown in Figure 5 RRS outperforms the vanilla TD3 in all five environments and outperforms all baseline methods in most tasks. In all experiments, use 3 Q-networks for fair comparison. Note that there is a trade-off between computational complexity and sample efficiency, i.e. using more Q-networks may further improve the performance at the cost of more computational
8

---

Under review as a conference paper at ICLR 2022
Hopper-v2
Walker2d-v2
Ant-v22
10.00
50.00
6606
Episodic Return
30.0
Episodic Return
Eo/sodic Return
02:00
-0.00
2006
2000
0090
1000
500
1006
10936
A60200
19386
RX662
103603
20006
660383
63066
1606693
22044 403031
6X661
(0000) JE0083)
Interactions
Interactions
# Interactions
HallCheetah-vv
Humanoid-v2
5000
Booistrapped TD3
Ep/sodic Return
30008
Bolsodic Return
RRS (Duns)
6100
Ersembl TD3
.000
TD3
2008
200300
JC0930
500093
8X008
100K0C
200003
400333 C00820
003802
2200006
Interactions
Interactions
Figure 5: Results on continuous control tasks, the method of Random Reward Shift (RRS) outper- forms its value-based baselines in most environments.
expensive, as reported in (Osband et al. and ablation studies can be bound in Appendix C.2
2016).
More implementation details, pseudo code of RRS,
5.3 OPTIMISTIC RANDOM NETWORK DISTILLATION
Finally, we experiment with five discrete exploration tasks, namely the MountainCar-vO, and four navigation tasks of MiniGrid suite (Chevalier-Boisse) Boisvert et al. 2018). namely the task of Empty- Random, MultiRoom, and FourRooms, to verify our insight on improving RND for value-based duriosity-driven exploration. More environment details are provided in the Appendix
C.3
We compare the vanilla DQN, vanilla RND, as well improved DQN and RND according to our proposed insight. DQN -0.5 indicates the results with a reward shifting of -0.5. Comparing DQN 0.5 with the vanilla DQN, our insight of negative reward shift leads to curiosity-drivee again verified. RND -1.0 indicates the results when a reward shifting of 1.0 is added to RND based on DQN. RND -1.0 improves the performance of RND and DQN in most environments, showing the effectiveness of the RND-based exploration bonus. Note that for fair comparison, RND 1.0 should be compared with the vanilla DQN as the -1.0 reward shift just cancels the positive exploration bonus introduced by RND.
exploration
Moreover, we can further improve RND by equipping it with servrd-shiftinggbaae based drivenn exploration. RND -1.5 (i.c., RND with -1.5 reward shift) can be compared with the DQN -0.5, as both receive an -0.5 exploration bonus for unseen states. We find all experiments that our negative reward shift remarkably improve exploration, not only working with RND to improve its performance but also work effectively in isolation.
6 CONCLUSION
In this work, we study how reward shifting affects policy learning in value-based deep reinforce- ment learning algorithms. Although constant reward shifting should not change the optimal policy induced by the optimal value function, in practice such a constant shift does affect the function ap- proximation. Our detailed analysis manifests the fact that constant reward shift equivalent us- ing different initialization in the value function approximation. Specifically, we show that a negative reward shift leads to curiosity-driven exploration, while a positive reward shift helps conservative exploitation. The proposed idea is then verified through a variety of application scenarios, includ- ing offline RL, sample-efficient continuous control, and curiosity-driven exploration in value-based methods. Thus reward shifting is a simple yet effective technique that deserves further investigation.

---

Under review as a conference paper at ICLR 2022
MountainCar-v0
MiniGrid Baaeed--mmttyypaayaee Random 6x6-v0
MiniGrid-MMuttiRoom--M
-230
Episodic Return
1.00
Episodic Return
Episodic Return
-230
-140
9.5
-120
04
Mmw
-330
2000
2000
3006
4300
2043
Episodes
# Episodes
Episodes
FiniGrid-FourRoenn 7x7-V0
90
www
DON 0.5
Episadic Return
3.0
Episodic Return
RND -10
RND -15
3.4
Episodes
Episodes
Figure 6: Value-based RND with shifted prior: Plugging the vanilla RND into DQN is not well- motivated according to our analysis in Section 4.3.2 The insight of equivalence between negative reward shifting and curiosity- driven exploration motivates shift the vanilla RND with a con- stant, which drastically improves the performance of RND when working with DQN.
ETHICS STATEMENT
In this work, we study how the linear reward shaping in reinforcement learning benefits offline con- servative estimation, online continuous control, and curiosity-drit exploration in discrete control tasks. Although in our work we experimented on variety of benchmark environments, there are plenty of real-world applications: Improving the learning stability as well the asymptotic per- formance of offline RL empowers the application of RL in scenarios where interaction with the environment extremely expensive or unethical, c.g., healtheare, robotics, finance, etc. Moreover, discussions continuous and discrete control tasks open up promising direction in pursuance of sample-efficient learning without introducing heavy extra computational burdens. Large-scale applications of our method can help improving efficiency in exploration, which is always conducted through computing additional curiosity networks explicitly in previous works.
REPRODUCIBILITY STATEMENT
We include our code in the supplementary materials. More details for our experiments on offline RL, continuous control and discrete control can be found in Appendix C.I Appendix C22 and Appendix C.3) separately.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Igge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXXv:19100071113 2019.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 263-272. IMLR. org, 2017.
10

---

Under review as a conference paper at ICLR 2022
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count- bount-based exploration and intrinsic motivation. Advances in neural information processing systems, 29:1471-1479 2016.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung. Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1912.0666.,,, 2019.
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Ani- mesh Garg. Conservative safety crities for exploration. arXiv preprint arXiv:2010.1 14497. 2020.
Ronen Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near- optimal reinforcement learning. Journal of Machine Learning Research, 2(0ct):213-231,22 2002.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study curiosity-driven learning. arXiv preprint arXiv: 808.0435, 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810. 12894, 2018b.
Maxime Chevalier- Boisvent, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. hitps: / cwm www.mmcw..iiawtttaattmmc..cccctt http://awwwc.cca//aaaaiaiaraa/aa sym-minigrid, 2018.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In Advances in Neural Information Processing Systems, pp. 1785-1796, 2019.
Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference on Machine Learning, pp. 2020-2027. PMLR, 2021.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint 1rXiv11901.10995 2019.
Mahmoud Elbarhari, Kyriakos Efthymiadis, Bram Vanderborght, and Ann Nowé. Ltlf-based reward shaping for reinforcement learning. In Adaptive and Learning Agents Workshop 2021, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint Xiv: 1802.06600 2018.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on robot learning, pp. 482-495. PMLR, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: 18Xiv:1812.02900, 2018a.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint Xiv: htfi::88020947.,, 2018b.
Alex Graves, Mare Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pp- 1311-1320. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Variational information maximizing exploration. 2016.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(3A1):1533-1600 2010.

---

Under review as a conference paper at ICLR 2022
Daniel Jarrett, Alihan Hüyük, and Mihaela Van Der Schaar. Inverse decision modeling: Learning interpretable representations of behavior. International Conference on Machine Learning, pp- 4755 4771- PMLR, 2021.
Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael Jordan. Is q-learning provably effi- cient? In Advances in Neural Information Processing Systems, pp. 4863 0113-487,,4 2018.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint MrXiv:1906.00990 2019.
Adam Daniel I Laud. Theory and application of reward shaping in reinforcement learning. University of Illinois Urbana- Chbaaa-Champaign, 2004.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint M..Kiv:159.009999 2015.
Tambet Matüsenn Avital Oliver, Taco Cohen, and John Schulman. Teacher Teacher-student student curriculum learn- ing. IEEE transactions neural networks and learning systems, 31(9):3732-3743 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 518(7540):529-533 2015.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In leml, volume 99, pp. 278-287, 1999.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint arXiv:1806.05635, 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8617-8629, 2018.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.0944 800.09944, 2018.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pp. 835-853. PMLR, 2020.
Jette Randløv and Preben Alstrøm. Learning to drive a bicycle using reinforcement learning and shaping. In ICML, volume 98, pp. 463-471. Citeseer, 1998.
Tabish Rashid, Bei Peng, Wendelin Bochmer, and Shimon Whiteson. Optimistic exploration even with a pessimistic initialisation. arXiv preprint arXiv:2002.12114, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
12

---

Under review as a conference paper at ICLR 2022
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awarr unsupervised discovery of skills. arXiv preprint TXXix:1977.015777 2019.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degrís, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp- 387-395. PMLR, 2014.
Hao Sun, Zhizhong Li, Xiaotong Liu, Bolei Zhou, and Dahua Lin. Policy continuation with hind- sight inverse dynamics. In Advances in Neural Information Processing Systems, pp. 10265- 10275, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 1998.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In 3 3st Conference on Neural Information Processing Systems (NIPS), volume 30, pp. 1-18, 2017.
Chen Tessler, Guy Tennenholtz, and Shie Mannor. Distributional policy optimization: An alternative approach for continuous control. arXiv preprint arXiv:1905.09855 2019.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782) 2019.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01222 2016.
Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforce- ment learning agents. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 792-799, 2003.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1911:1911..1361, 2019.
Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor Prasanna. Brac+: Going deeper with behavior regularized offline reinforcement learning. 2020.
A PROOF OF PROPOSITION
Proof. the estimated Q-value Q(s, a) is composed by the two estimators with function approxima- tion error, defined as Eb1 (s,a) =
bt I- Q= a), and E, (s,a) = Q _, a)
d= - a))
I-
(1 1--))A,, + BEB,
=2n(11 - Q) + (1 (1- (1- (11))(1- 2 + 3(1 2n)
=217(" + (1- 27)((1 - -3) + +8Q
4(1 ((1-3)r Q- + 3(1 - -21)) Q +
4,1,²
- -210 + +11 27) - B)(1 2n) 2A.0
Q-1
1 (1- -21)
1- (1 21)2))
41²
2nn22 + +11 +(1-27)1C 27) 271 - 3) 3QB,o)
1- (1 2m)
EC.,
(7)
13

---

Under review as a conference paper at ICLR 2022
where C = B1 3B and the last line requires QA,0- QB.0- Qc, identical initializa-
tion.
With this notion, Equation (6 can be re- re-written
( a) = (* + (1- B)eb (8, + Beb- (8,a)
= (s, a) + + -,6
(8)
= (s, + (s,
where the second line relies the linear assumption of the approximation error (1 3)6, (s. + Beb- (s, a). We further have (1 B)Qu+ + 3Q,- Q and Q(s,a) Qr (s, telling us that trading-off between the constant b- used for exploration and the constant bt used for exploitation with the coefficient 33 is equivalent to use another constant with value of = (1 (1-3)b"* 3)b+ + 3b.
B IMPLICATIONS OF ASSUMPTION IN SEC. 4.3.
In our main text, the estimated values for extremely o.o.d. samples are assumed to be near zeros. We provide detailed implications and explanations this section.
On the one hand, it's clear that such an assumption holds for the tabular settings, that un-visited state-action pairs have the value in tabular initialization.
On the other hand, we acknowledge it as a mild assumption that there always exists o.o.d. samples that have the Q-values near zero for function approximation settings. Interpolation between those o.o.d. samples and other state-action pairs will clearly lead to an "in-between" value estimation, which in practice can be achieved with properly regularized neural networks.
The key insight we want to emphasize in Sec. 4.3.1 is that for frequently visited state-action pairs, the value discrepancy with different initialization are small, while for seldomly-visite visitedd state-action pairs, the discrepancy are relatively large, enabling the usage of such discrepancy as exploration bonus.
C IMPLEMENTATION DETAILS AND ABLATION STUDIES
Hardware and Training Time We experiment on server with 8 TITAN X GPUs and 32 Intel(R) ES-2640 CPUs. In general, shifting the reward does not introduce further computation burden except in the continuous control tasks, our method of Random Reward Shift (RRS) requires two additional Q-value networks. In our PyTorch- byTeec implementation, those additional networks can be easily implemented and optimized in a parallel manner, and the extra computational burden is equivalent to using a v3 times wider neural network during optimization. It is worth noting that RRS is computationally much cheaper than the Bootstrapped TD3, where additional policy networks are also needed.
Network Structure Our implementation of TD3, BCQ and CQL are based on code released by the authors, without changing hyper-parameters. We implement DQN based on a 3-layer fully connected neural network with 64 hidden units for the Q-value function, using ReLU and linear activation respectively. We use the Adam optimizer with learning rate 0.001, and use an epsilon- two 4-yer fully connected
greedy approach naive exploration strategy. In RND, neural networks with 512 units and ReLU activation in each hidden layer, and a softmax activation for the output layer. Adam optimizer is used for the optimization of the RND networks with learning rate 0.0001.
Our code is provided in the supplementary materials, and will be made public available.
14

---

Under review as a conference paper at ICLR 2022
C.I OFFLINE RL
In our experiments, we use fixed dataset with 10k offline trainsition tuples for offline RL learning. Our implementation BCQ and CQL are both based on the code provided by the authors. The only change we made to verify our insight is to shift the reward by a constant. In most environments, we find r' = r 8 provides good enough performance. While in Hopper Medium CQL we find using a smaller positive reward shift r' = r + works better than r' = r r 8, and for Walker Medium CQL, using a larger reward shift of = r 50 further improves the result with r' = +
Figure Shows different performance under different choices of the reward shift constant. We denote a positive reward shift r' = + 8 as Pos.1, denote = r + 20 as Pos.2 and denote = r 50 as Pos.3 for all experiments excetp in the Hopper Medium CQL we use Pos.1 to denote r' = - 1.
In the experiments based on BCQ (first two figures). We can observe uniformly performance im- provement with all choices of reward shift constants. As the algorithm of CQL has already taken the conservative value estimation into consideration, in the experiments based on CQL. the performance is more closely related to the constant we use. Specifically, in Hopper Expert, while using any of the positive reward shift constants improve the learning stability, r' = r + 8 performs better on preserving the learning efficiency during early learning stage. For Hopper Medium, we find using larger positive constants hinder the performance. For Walker Medium, using a larger constant in reward shift performs much better than using a smaller one.
Hopper Imitate BCQ
Hopper Expert BCQ
Hopper Expert COL
3,00
Episodic Return
Episodic Return
Eolsodic Return
22:00
0000
20:00
3.:03
2000
BCQ
BCQ
COL
Pos
10:00
BCQ Pox
Pos.1
500
Pos 2
1:00
BCQ Pos.3
BCQ Pos. 3
COL Pos
20006
<60383
saace
220483 200003//0c000
Optimization Step
Optimization Step
Optimization Step
Hopper Medium COL
Walker Medium COL
2500
Episodic Return
Episodic Return
2333
3083
Pos.1
COL Pos.2
OOL Pos. 2
COL Pos.3
CGL Pos. 3
Optimization Step
Optimization Step
Figure 7: Performance with different reward shift constants.
C.2 CONTINUOUS CONTROL
Pseudo-Code for Random Reward Shift
The pseudo-code of RRS is provided in Algorithm I
Although we find in the motivating example that a -5 reward shift is able to remarkably improve the asymptotic performance of TD3, in this work we aim at proposing an uni- formly suitable method based on the insight behind the motivating example. Therefore to use =0.5, 0 as the reward shifting constants. We find in experiment that the sampling frequency does not affect the performance. And in the experiments use a fixed value network throughout a whole trajectory. i.e., one of the K Q-networks is sampled uniformly after each episode with length of 1000 timesteps. Intuitively, searching for more suitable reward randomization designs may further improve the performance, yet that is beyond the coverage of this work.
Details of RRS
propose
follow BDQN Osband et al. (2016) to
Ablation Studies We experiment with different number of Q-value networks as well as dif- ferent choices of the random reward shifting ranges. Results are presented in Figure g We denote RRS with 7 reward shifting constants ( and therefore also 7 Q-networks) RRS-
15

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Sample-Efficient Continuous Control with Random Reward Shift
Require
the size of mini-batch N, smoothing factor 7 > 0, K reward shift values r/ + bc, 1, K. Random initialized policy network To, target policy network Ter, 8' + 6.
K random initialized Q networks, and corresponding target networks, parameterized by uk, u/k w/ 4 for k = I,...,K. K. (e.g., ModuleList in PyTorch).
for iteration = 1, 2... do
Uniformly sample one of the K Q-functions, Qua for policy update for 1,2..... do
# Interaction
Run policy na, and collect transition tuples (st, de, $6. vt).
Sample mini-batch of transition tuples {(s, 8', (0,,),,,)))))) r))
# Update Qu (in parallel)
Calculate the k-th target Q value yk,i r. + be + Qug, (66, To (s!))
Update wt with loss Σ Li=1 ( Quu as),
# Update Te Update policy No with Qwa end for
# Update target networks
0' < 70 + (1 -)0'
w/ t Twk (1 - T)uk, K I....,K. K.
end for
7, and denote RRS with 3 reward shifting constants and therefore also 3 Q-networks) as RRS-3. The constants following RRS-3/RRS-7 are the ranges of those random con- stants. Specifically, we use [-0.5, 0.5] for the RRS-3 0.5 settings, [-1.0, 1.0] for the RRS-3 1.0 settings, [-0.5, 0.33. 0.117 0.17,,0.....00 0.33, 0.5] for the RRS-7 0.5 settings and [-1.0, -0.67, 0.33, 0, 0.33, 0.67, 1.0] for the RRS-7 1.0 settings. According to the experimental results, RRS is not sensitive to ppper-paramtters, showing the robustness of the proposed method. We believe further search for those hyper-parameters can further improve the learning efficiency, yet this is off the main scope of this work and therefore left for the future research.
Hopper-v2
Walker2d-v2
Ant-v2
20:00
30000
0000
Episodic Return
3000
Episedic Return
c Retur
5000
25:00
20:00
1200
BRS-7
RS5-7
35
18:00
BRS-7 3.0
RS5-7
2008
100
230
20300
Sreass
103064
2000e
1000001
2500E
(29131
exzes
(CE9)) 100003)
Interactions
Interactions
# Interactions
HalfCheetah-v2
Humanoid-v2
Egisodic Return
J000e
Eoisodic Return
12.0
3001
4830
8008
4008
RAS-7 05
BR5-7 05
2000
R15-005
R15-7
885-3 0.5
B65-) 3.0
200000
4CC930
500433
#30009
L0000e
6
200000
430008
500600
300003
2300008
Interactions
# Interactions
Figure Performance with different reward shift constants and different number of nentworks.
C.3 RANDOM NETWORK DISTILLATION
Environments In this work, experiment with five discrete (sparse reward) exploration tasks namely the MountainCar-v0, and four navigation tasks of MiniGrid suite (Chevalier- Boisvert et al, 2018), namely the task of Hmppy-Randdon MultiRoom, and FourRooms, to verify our insight on
16

---

Under review as a conference paper at ICLR 2022
improving RND for value-based curiosity-drive exploration. Figure shows example of different tasks.
Figure 9: Examples of environments used Section 5.3 The first figure shows the MountainCar-v0 environment where needs to accumulate potential energy reach the flag, receive positive reward. The second figure shows the maze of the Empty- Random task with size of 6, the third one shows the MultiRoom of level S2-N4, where there are 2 rooms with size 4, the last figure shows example of FourRoom task with size 17. In our experiments, as we use the vanilla DQN as the baseline, which not suitable for partial observable tasks, we use smaller maze of size and to avoid further dependency memories. In all tasks of the MiniGrid domain, the triangular red agent need to navigate to the green goal square, and the observable region only a 7x7 square the agent is facing to (i.e., the regions with shallower color in the last three figures).
Ablation Studies We experiment with different reward shifting constants in the dis- crete control settings. We use a relatively large range in choosing constants, i.c., -.0.05.-0.155 1.0, 1.5, 2.0, 2.5, -50, 10.0). Results are presented in Figure I0 In all experiments, using moderate reward shifting constant like {-1.0,- 1155, -2.0, 22.5) remarkably improves the learning efficiency. On the other hand, too aggressive reward shifting will lead too much curiosity exploration and hinder the learning efficiency in the limited number of interactions.
17

---

Under review as a conference paper at ICLR 2022
MiniGrid-Emppe Emply-Randomi -
MiniGrid-F Settrid-FoorRRooe FourRooms- 7x7-v0
MiniGrid- Mitt:idd-Muutitoooo--------
Episodic Return
Episodic Return
Episodic Return
RND
MAD
RND -0:5
RM0m,0 0.05
RND -0.05
RND 15
2.1m_0.15
AND 15
RND -1.0
KNOm 1.0
580 -1.0
MND -15
MMDm.15
140 3.5
HND -2.0
RMDm 2.0
124 2.0
022
RND
RND
RNDm 25
RNDm_5.0
RNO -5.0
520 -25
RND -12.
RNDm
1NO -10.0
2000
8383
Episodes
# Episodes
Episodes
MindGrid-FourRooms--y9sk
MountainCar-v0
-130
RND
RND -0.05
AND
BND
AKD1,0.05
RND -1.0
mxmm
RND -1.5
MND.11.
Egisodic
RMDm11.
-.5
-120
AND -10.0
MND
-190
5000 28003 15800 23000 2500 50000 30000
Episodes
Episodes
Figure 10: Performance with different reward shift constants in RND.
18


Reviews:
[{"review_id": "urQRzGjOUaS", "reviewer": "Reviewer_4ENc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- I do like the topic this paper studies, as I believe it is also what the deep RL community observes, that reward shifting does have an impact on the final learned policy. \n- The paper discusses the reward shifting from both online and offline settings, which is interesting to see how positive/negative bias affects exploration and exploitation. \n- Empirical studies in both online and offline setting verify the effectiveness of the reward shifting.\n\nWeakness:\n- The main argument in this paper is that reward shifting corresponds to different initialization, and pessimistic or optimistic initialization does affect the final learned policy, which is known to the community for a while. However, I think it lacks some intermediate important steps in the paper, such as how the initialization affects the learning dynamics? It seems all the arguments in the paper intuitively make sense, but lack strong theoretical support.\n- The experiments though illustrates how the positive/negative bias helps in online/offline setting, which is well-known. A more important question is how to determine an appropriate bias in different tasks and environments. As shown in Appendix B.1, the magnitude of the bias does affect the learning behavior and outcome. It would be great to discuss this in a more principle way, instead of saying something like, $r+8$ seems work. \n- Also, the author aims to study the linear transformation in the reward shift, but just ignores the scaling factor, which also seems have some significant effect in the learning process. It would be great to investigate more on this.", "summary_of_the_paper": "This paper studies how different reward shifting (to be precise, adding a bias term on the reward) affects value-based RL algorithms. It illustrates the idea that different reward shifting corresponds to different initialization of the Q networks. Though linear reward shifting does not change the optimal solution for Q, the initialization does affect the learning process and might lead to different stationary points, when equipped with function approximation. The paper further studies the idea in two settings: (1). offline RL, which shows a position bias term could assign the $(s,a)\\in\\mathcal{D}$ a large value which helps conservative exploitation while in (2) online RL, a negative bias term could facilitate exploration. Some empirical results verify the ideas.\n\nReward shifting techniques have been studied in the bandit literature, this paper studies how it affects the value-based RL algorithms with function approximations. This is an interesting and important topic, and this paper provides some intuition and empirical results to help the community systematically understand the effects of reward shifting on both online and offline settings.", "summary_of_the_review": "I like the topic this paper studies. However, the main argument in the paper is intuitive and seems well-known to the community. Beyond some empirical validation, I do not think it gives enough insight on how different initialization affects the learning dynamics and final learned policy. Also, there are important questions left, such as: (1). How to select the optimal bias $b$ to facilitate the learning process and (2). Does the scaling factor matter, empirically? Though the topic is interesting, given the current status of the paper, I would recommend a reject and I believe it would be a valuable paper if the authors study more deeply on the underlying causes of reward shifting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iE4x8Ox3M82", "reviewer": "Reviewer_Mgm9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n+ Overall, the paper is technically sound and the major claims are well supported with experimental results. \n+ Various settings and scenarios are considered in the experimentation, including offline RL, online RL with continuous and discrete action spaces. \n+ Related work in different domains, offline and online RL, is adequately cited.    \n\nWeaknesses/Concerns: \n1.\t[Clarity] Some parts of the paper are not clear enough in my opinion. Please find the list of items below for details. \n-\t1.a. It is not super clear to me how the Humanoid example in Section 3 connects to the major points made in this paper? Turning off the alive bonus is only applicable and effective for selective states, while the reward shifting is universal across all state-action pairs. To me, it only seems that the reward function for Humanoid is kind of problematic and is not designed well. Thus, it seems a bit off as a motivating example for reward shifting. Additional elaboration could be helpful here.  \n-\t1.b. It is important to clarify the sign of parameter $b^-$ in the revision. Section 4.3 states that $b^-$ is a positive constant, while Figure 3 and Equation (6) should interpret $b^-$ as a negative constant I believe. Please make sure the definition of $b^-$ is consistent throughout the paper including Appendices. \n-\t1.c. It would be helpful to clearly state, for example at the end of the first paragraph under Section 4.3, that 4.3.1 is for continuous control and 4.3.2 is for improving curiosity-driven exploration with RND, so that the readers may be clearer that these two subsections are talking about different scenarios with their corresponding designs.    \n-\t1.d. I had an impression that all the Q-functions should be seen as function approximators with neural networks. If that is the case, the statement from Section 4.3.1., “For under-explored state-action pairs, both $Q_A$ and $Q_B$ will give near-zero predictions as a consequence of initialization”, is not true to me. Even for unseen state-action pairs, the Q-network would extrapolate based on observed data, and the Q-values would not stay near-zeros for all under-explored state-action pairs. To be more rigorous, I encourage the authors to state this as an assumption in the paper or specify that this part of analysis only applies to tabular Q-learning settings.   \n-\t1.e. The statement made in Proposition 1 is not clear enough. Appendix A definitely helps but I would still suggest elaborate a bit more in the main paper, especially for this part “combining the linear combination in Equation (6) is equivalent to using a linear combination of the constants”. Accordingly, it would be very helpful if the authors can describe more about the Random Reward Shift (RRS) in Section 4.3.1 or in Section 5.2. Again, I understand RRS much better by looking at Algorithm 1 in the Appendix, and I suggest make clearer connections between the algorithm RRS and Section 4.3.1, e.g., Equation (6) and how 4.3.1 leads to the design of RRS?  \n2.\t[Experiments/Baselines]  \n-\t2.a. How many seeds were used in the experiments Figure 4 - 6? I don’t think it is stated in the main paper? \n-\t2.b. To my understanding, RRS is training K = 3 different Q networks, each trained with reward shifting $b_k$, $k = 1, 2, 3$. Are these three shifting constants $b_1 = 0, b_2 = 0.5, b_3 = -0.5$ or they are sampled from $[0, 0.5, -0.5]$ randomly? In addition, it seems to me that RRS is similar to the ensemble of Q-networks with randomized additive priors [1]. Therefore, it is interesting to see how RRS would compare with [1] (or Ensemble TD3 + Randomized Prior Functions) when $K = 3$. It would be an interesting baseline to include in the paper. \n\nMinor comments:\n-\tRemove the line break right below Figure 2. \n-\tMake sure the sign of $b^-$ is consistent in the paper. \n-\tSection 4.3.2, the penultimate paragraph, I believe it should be $I = max_{s, a} |A – B|^2$, the square is missing.    \n\n[1] Osband, Ian, John Aslanides, and Albin Cassirer. \"Randomized prior functions for deep reinforcement learning.\" arXiv preprint arXiv:1806.03335 (2018).\n\n", "summary_of_the_paper": "This paper studies the effectiveness of reward shifting in value-based deep reinforcement learning. Particularly, it points out that (1) a positive reward shifting is equivalent to pessimistic initialization of Q values, thus, leading to conservative exploitation; and (2) a negative reward shifting equals to optimistic initialization and hence leads to curiosity-driven exploration. By leveraging these insights, the paper proposes to modify existing RL algorithms by simply adding reward shifting to improve their performance. Empirically, experimental results showed that reward shifting helps improve the baselines (or the vanilla algorithms) under various scenarios, including offline RL, online continuous control and online value-based curiosity-driven exploration. ", "summary_of_the_review": "The paper provides helpful insights to better understand the effectiveness of reward shifting in deep reinforcement learning. Major claims are well supported by empirical experiments in various domains, including offline and online RL. My main concerns are about the clarity of the paper, which I believe needs to be further improved, and additional baselines (please see the above Concerns for details). Therefore, I vote for a weak reject for the current submission. I am willing to adjust my scores should my concerns be addressed in the rebuttal period.  \n\n\n=== Updated ===\n\nMy major concerns were addressed during the rebuttal period, so increased my score accordingly.      \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a3Z2vKT0_jH", "reviewer": "Reviewer_fFaW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \nThe strengths are summarized below:\n\n1. This paper tries to unify optimistic exploration and conservative Q learning through reward shaping, the idea by itself is very interesting.\n2. The paper provides several nice plots to help understand the algorithm.\n3. The experiments are pretty solid including various settings (e.g., offline/online, discrete/continuous) and environments.\n\nWeakness:\nI would like to raise several questions to the paper: \n\n1. In Sec.3 the motivating example, the design of the alive bonus is not just to provide the curriculum. Such design may also have the practical effect of preventing the robotic to break itself. For example, falling down can cause the robotic to break some of its components. In addition, since the reward is shifted, I am not sure how to read the curve in the figure. Is the orange curve representing the shifted reward and the blue one is the unshifted reward? If that is the case, they are not directly comparable. Thus, I think this moving example is problematic to some extent and needs more explanation. \n\n2. In Figure. 2, it seems not correct that the initialization $\\tilde{Q}_0$ is always smaller than $Q^*$. This would be interpreted as always having a pessimistic initialization of the $Q$ function. \n\n3. In the same figure, the labeling: $|Q_t - Q^*| < |Q_{t-1} - Q^*|$ seems to be very strict and cannot be achieved in practice. Even we are doing optimization over the $Q$ function, I don't think it is a practical condition.\n\n4. Same problem in Sec.4.2, the author states \"the near-zero initialization guarantees the prediction is lower than the optimal Q-value, thus\nconservative exploitation can be conducted with such a value function\". Which I think is not valid, a simple negative reward case would cause this 0 initialization to be optimistic rather than conservative.\n\n5. In the optimistic/pessimistic in face of uncertainty criterion, the algorithm design is always state-action dependent. Thus, I am having trouble understanding just adding a uniform bias can achieve such criterion, since it is not a function of state-action at all. \n\n6. In the experiments, I would particularly like to see how the authors choose the value of bias. My feeling is that it will be very sensitive to the magnitude, but such magnitude is environment-dependent. I think adding an ablation on a larger range of values will help clarify this.\n\n7. In addition, some of the other optimistic initialization/exploration designs are not sufficiently discussed. Such as https://arxiv.org/pdf/2002.12174.pdf.\n\nClarity: The paper is well-written and clear in the flow. In addition, the figure is pretty straightforward to illustrate the point.\n\nFeedbacks & Questions: Please see details in the weakness.", "summary_of_the_paper": "This paper shows a key observation that the linear transformation of reward shaping is equivalent to diversified Q-value network initialization. Based on such observation, the paper presents three scenarios where the reward shifting can benefit: the offline conservative exploitation, the online sample-efficient RL, and the curiosity-driven exploration. The algorithm is evaluated on the continuous control tasks and discrete control tasks.", "summary_of_the_review": "Overall, I am still having some concerns about the claim of the paper. These concerns are pretty important and thus I encourage the author to engage in the discussion period and clarify these if there is any misunderstanding. I am happy to re-evaluate if the author convinces me. Based on my current evaluation, I don't recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8FPiSChes7m", "reviewer": "Reviewer_dgHr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantage:\nWith the idea of reward shifting, this paper considered different reward shifting schemes for a lot of different scenarios such as offline RL, online RL, and curious-driven RL. The intuition of using pessimistic Q-value for offline RL and optimistic Q-value to encourage exploration is easy to understand.\n\nSome questions:\n1. Is constant reward shifting can be regarded as solving a different MDP or is actually a way to design reward but not a more efficient way to accelerate the Q-value convergence? \n2. It seems to minus/plus a constant reward to the Q-estimates can be seen as using pessimistic penalty in offline RL or optimistic bonus for exploration in online RL. The previous works designed different values of the bonus $b(s, a)$ corresponding to different $(s,a)$ pairs, which may be more efficient than adding a constant reward for all the Q-values? However such kinds of baselines seem not to be covered or even discussed? I'm not sure whether they are related such as [1],[2].\n3. It is claimed in Section 2.1 that this paper considers off-policy cases? So will the data be generated under a fixed behavior policy or generated conditioned on some Q-values?\n4. It is a little bit counterintuitive why changing the initialization of Q-value will bring benefits for the algorithms. Maybe the biggest change brought by reward shifting is to change the landscape of the Q-value (or say the optimal Q-values after changing the reward) but not the change of the initialization? If that landscape and all the local and global minima of a loss function does not change, changing the initialization helps just means the algorithm is not robust which depends a lot on the initialization?\n5. Will the initialization of Q-values be too conservative or optimistic which may hinder the performance of the algorithm? And how to choose the reward shifting magnitude?\n6. It seems a lot more baselines involving other kinds of pessimistic or optimistic additional terms instead of constant reward shifting will be more convincing?\n\n[1] Jin, Chi, et al. \"Is Q-learning provably efficient?.\" arXiv preprint arXiv:1807.03765 (2018).\n\n[2] Rashidinejad, Paria, et al. \"Bridging offline reinforcement learning and imitation learning: A tale of pessimism.\" arXiv preprint arXiv:2103.12021 (2021).", "summary_of_the_paper": "This paper proposes a linear combination of reward shifting to improve the performance of value-based reinforcement learning. Using the equivalence between reward shifting and a new initialization for the Q-value function approximation, this work shows that the optimal policy will not change and the method can achieve better results compared to no reward shifting.", "summary_of_the_review": "In my opinion, this work is slightly below the acceptance threshold. But I'm happy to change my score if the problems are addressed well.\n\nAdvantage:\nIt formulates the addition of a pessimistic penalty or an optimistic bonus when learning Q-value estimations as a reward shifting, which is may of independent interest.\nTargeting different settings in RL, it designs different algorithms using reward shifting.\n\nDisadvantage:\nI'm not sure whether the intuition that changing the initialization of the Q-value network is the main reason why the algorithm with reward shifting outperforms other baselines.\nIf we see reward shifting as an extra term added for Q-values, there are some other ways instead of adding constant to do that, while there are no baselines about that is involved.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and active participation in the discussions. The majority of reviewers have concers with this paper, in particular, regarding the motivation of the method [dgHr], clarity [Mgm9], and theorethical support [4ENc]. I side with reviewers 4ENc, dgHr and fFaW, and recommend rejection of this paper. I want to encourage the authors to use the feedback by the reviewers to improve their paper.
Decision: Reject