Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning
Authors: Haozhe Ma
Zhengding Luo
Thanh Vinh Vo
Kuankuan Sima
Tze-Yun Leong
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Reward Shaping
Thompson Sampling
Self-Adaptive Exploration-Exploitation Balance
Primary area: reinforcement learning
TL;DR: We propose a self-adaptive and highly efficient reward shaping mechanism by incorporating historical success rates into shaped rewards to improve sample efficiency and convergence stability in sparse-reward reinforcement learning environments.

Abstract: Reward shaping is a reinforcement learning technique that addresses the sparse-reward problem by providing frequent, informative feedback. We propose an efficient self-adaptive reward-shaping mechanism that uses success rates derived from historical experiences as shaped rewards. The success rates are sampled from Beta distributions, which evolve from uncertainty to reliability as data accumulates. Initially, shaped rewards are stochastic to encourage exploration, gradually becoming more certain to promote exploitation and maintain a natural balance between exploration and exploitation. We apply Kernel Density Estimation (KDE) with Random Fourier Features (RFF) to derive Beta distributions, providing a computationally efficient solution for continuous and high-dimensional state spaces. Our method, validated on tasks with extremely sparse rewards, improves sample efficiency and convergence stability over relevant baselines.

Published as a conference paper at ICLR 2025
HIGHLY EFFICIENT SELF-ADAPTIVE REWARD SHAP- ING FOR REINFORCEMENT LEARNING
Haozhe Ma
Zhengding Luo"
School Computing
School of Electrical and Electronic Engineering Nanyang Technological University luoz002 2.1 (e ntt..ud..ss.
National University of Singapore haozhe. ma nus. edu
Thanh Vinh Vo School of Computing National University of Singapore votvênus. edu.sg
Kuankuan Sima
Department of Electrical and Computer Engineering National University of Singapore kua kuan_s ma @u nus edu
Tze- Yue-Yun/oe Leong
School of Computing
National University of Singapore leongtyênus. edu. sg
ABSTRACT
Reward shaping is reinforcement learning technique that addresses the sparse- reward problem by providing frequent, informative feedback. We propose an ef- ficient self-adaptive reward-shaping mechanism that uses success rates derived from historical experiences shaped rewards. The success rates are sampled from Beta distributions, which evolve from uncertainty to reliability data accu- mulates. Initially, shaped rewards are stochastic to encourage exploration, grad- ually becoming more certain to promote exploitation and maintain natural bal- ance between exploration and exploitation. We apply Kernel Density Estimation (KDE) with Random Fourier Features (RFF) to derive Beta distributions, provid- ing computationally efficient solution for continuous and high-dimensional state spaces. Our method, validated on tasks with extremely sparse rewards, improves sample efficiency and convergence stability relevant baselines.
INTRODUCTION
Environments with extremely sparse rewards present notable challenges for reinforcement learning (RL). In such contexts, the reward model lacks immediate signals, agents receive feedback only after long horizons, making the ability to quickly discover beneficial samples crucial for successful learning (Ladosz al. 2022). To address this, straightforward solution is reconstruct the ward models by introducing auxiliary signals that assess the agent's 's hehavior, which has led to the popular technique of Reward Shaping (RS) (Strehl & Littman 2008 Gupta et al.) 2022). Inverse reinforcement learning, which extracts reward functions from human knowledge or expert demon- strations, represents intuitive approach within this framework Arora & Doshi (202T). However, it heavily relies on extensive human input, which can be difficult to obtain, especially in complex environments. Alternatively, fully autonomous approaches have emerged an attractive direction.
Automatically maintained reward shaping can be broadly categorized into two branches: intrinsic motivation-based rewards, which are task-agnostic, and inherent value-based rewards, which are typically task-specific. The former mainly introduces exploration bonuses to encourage agents to explore wider range of states, commonly by rewarding novel or infrequently visited states (Burda et al. 2018; Ostrovski et al. 2017; Tang al. 2017 Bellemare et al.) 2016). While these methods effectively enhance exploration, they tend to overlook the internal values the states. This can
"The authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
lead to the "noisy TV" problem, where agents fixate highly novel but meaningless regions, thus trapping them in suboptimal behaviors MMavor-Parker et al., 2022). In contrast, the latter leverages high-level heuristics to guide agents in extracting meaningful values from learning experiences, which helps stabilize convergence. However, these methods often struggle in early exploration due to non-directional guidance (Ma et al. 2024a Memarian al. 2021 Trott al. 2019).
To overcome the limitations of existing RS methods and combine the advantages of exploration- encouraged and inherent value-based rewards, this paper introduces a novel Self-Adaptive Success Rate-based reward shaping mechanism (SASR) The success rate, defined as the ratio of a state's presence in successful trajectories its total occurrences, serves as an auxiliary reward distilled from historical experience. This success rate assesses a state's 's contribution toward successful task completion, which closely aligns with the agent's ' original objectives, offering informative guidance for learning. Furthermore, to mitigate overconfidence caused by deterministic success rates, we adopt Beta distributions to model success rates from a probabilistic perspective. Beta distributions enable self-adaptive evolution of confidence in approximating state's success rate, ensuring the system gradually converges to reliable rewards as more data is collected, while avoiding premature certainty. To derive Beta distributions, we use kernel density estimation (KDE) with random Fourier features (RFF) to efficiently estimate success and failure counts. The main contributions of this paper are summarized as follows:
We propose SASR, an autonomous reward-shapp shaping mechanism for sparse-rewart environments. By deriving success rates from historical experiences aligned with the agent's 's optimization objec- tives, SASR effectively augments the environmental rewards.
We introduce a novel self-adaptive mechanism. Initially, low-confidence Beta distributions pro- vide uncertain rewards, encouraging exploration by perturbing the reward function and assigning higher rewards to unvisited states. As more experience accumulates, high-contidence Beta distri- butions deliver more reliable and precise rewards enhance exploitation.
To derive Beta distributions in continuous state spaces, we use KDE with RFF, creating an effi- cient approach that eliminates the need for additional neural networks or models for learning the auxiliary shaped rewards, thereby achieving remarkably low computational complexity.
SASR is evaluated on various extremely sparse-reward tasks, significantly outperforming several baselines in sample efficiency, learning speed, and convergence stability.
RELATED WORK
Reward shaping (RS) methods can generally be categorized based on the source of learning: ei- ther from human knowledge or the agent's own experiences. Techniques that derive reward models from human knowledge, such as Inverse Reinforcement Learning (IRL) (Arora & Doshi 2021 Ra- machandran & Amir 2007: /Ziearr et al. [0088] HadfilldMMenell et al. 2016) and Inverse Optimal Control (IOC) (Schultheis et al.] 2021 Zhang et al. a (2025). ), aim to extract reward or objective func- tions from expert demonstrations. Subsequently, transferring the learned reward models to new tasks has received considerable efforts (Bryık et al. 2022 Wu et al. 2021 Ellis et al. 2021 Cheng et al.) 2021 Adamczyk al. 2023 Luo al. 2024 Lyu et al. 2024). However, these methods rely heav- ily on genean-eenerated data and often struggle adapting out-of-distributin
scenarios. Thus, our focus shifts toward autonomous self-learning approaches, which can be further divided into intrin- sic motivation- and inherent value-based rewards depending on the nature of the rewards.
Intrinsic notivation-based RS explores general heuristics task-agnostic metrics encourage exploration. Potential- algorithms define the shaped reward as 7 - -e(s), where 4(-) potential function. This ensures the reward cancels out in the Bellman equation, preserving the optimal policy (Devlin & Kudenko 2012; Asmuth et al. 2008; Wiewwora 2003). However, designing the potential function highly depends on the environmental dynamics, making more ap- plicable to model-based RL. More commonly, methods incorporate exploration bonuses to reward
novel states (Mahankali et al. 2024: [iu et al.) [2025] Devidze et al., (2022) [Badia et al. 2020 Hong 2019). Count-based strategies, for instance, track visitation counts and
et al. 2018 [Eyenbach et al.
assign higher rewards to less frequently visited states (Lobel et al. 2023) Machado et al. 2020 Fox
et al. 2018; Fu et al. 2017 7 Martin et al. (017). In continuous spaces, state counting challeng-
ing, so Tang et al. (2017) introduced a hash function to discretize the state space, Bellemare et al.
'The source cade is accessible at: httpss///gitthb...c///aaaaaaa/t
2

---

Published as a conference paper at ICLR 2025
2016) proposed pseudo-counts based recording probabilities, and Ostrovski al. (2017 used
PixelCNN Van den Oord et al. (2016) to simulate density. Additionally, random network distillation
based methods measure state novelty by neural networks (Yang al. 2024b Liu et al. 2024 Burda
et al. 2018), while curiosity driven approaches reward agents for encountering surprising or unpre-
dictable states (Yang al.) 2024a [Sun al. 2022; Burda et al.] 2019 Pathak et al.) 2017 Zhang
iet al. 2023). Although intrinsic motivation has proven effective in enhancing exploration, only
considering novelty while ignoring the inherent values of states can lead suboptimal policies.
Inherent value based RS, on the other hand, focuses on task-related signals that highlight how states contribute to achieving higher rewards and their underlying significance. For instance, Trott itt al.] (2019) introduced additional rewards based on the distance between a state and the target; Stadie al. (2200) derived informative reward structures using a Self-Tuning Network to optimize guidance; Memarian et al. (021) captured the preferences among different trajectories by ranking them via a trained classifier; Zheng al. (2018) minimized the Kl--divergence between learned and original rewards to align their distributions. Mguni et al. (2023) used an auxiliary agent competing against the original agent in a Markov game; Ma et al. (2024a) introduced ReLara, collabora- tive framework where reward agent automatically generates rewards to guide the policy agent. Moreover, incorporating multiple agents or hierarchical structures to share and transfer knowledge through synchronized reward functions is another promising research direction (Park et al.
2023
Ma et al. 2024b Gupta et al. 2023
Hu et al. 2020 Raileanu & Rocktäschel 2020 Yi et al.
2022
PRELIMINARIES
Reinforcement Learning (RL) aims to train an agent to interact with an environment, which is commonly modeled as Markov Decision Process (MDP). An MDP represented as (S, A, T. R, 7), involves four main components: S is the state space, A is the action space, T A x S + [0. is the probability of transitioning from one state to another given a specific action, and R S the reward model. The objective in RI. is to learn a policy π [a|s) that maximizes the expected cumulative rewards G = E 7 RR(s))), where T7 (a)s) () indicates the likelihood of selecting action
a in state s, and 7 is the discount factor (Sutton & Barto 2018).
Beta Distribution is defined on the interval [0, 1], making it ideal for modeling proportions or probabilities. is parameterized by a and 3, which represent prior counts of successes and failures of binary outcome. The probability density function of Best-distr variable X is:
[ ar 3) B(a, 3)
where Bia, 3) the beta function. The key attribute Beta distribution is its adaptability: as more data accumulates, the values of and /3 increase, narrowing the distribution's 's shape and increas- ing confidence in the estimated probabilities. This feature is particularly useful in adaptive online learning, aligning with our objective of balancing exploration and exploitation.
Kernel Density Estimation (KDE) is a non-parametric method for approximating the probabil- ity density function of random variable from data samples. Given n data points {r:] KDE smooths these points to approximate the density function as follows:
d(x)
1 nh
Σ
n
where h is the bandwidth, and (.) is kernel function such Gaussian kernel, Laplacian kernel, or Cauchy kernel. KDE particularly useful in scenarios where the actual distribution complex or poorly defined, such as in continuous state spaces in RL environments.
METHODOLOGY
We propose a Self-Adaptive Success Rate based reward shaping mechanism (SASR) to accelerate RL algorithms in extremely- spansel---sss--eeaardd environments. Figure illustrates the principles of the SASR mechanism: The diagram consists of two parts representing the early and late learning stages. As experiences accumulate with learning progresses, the Beta distributions modeling the success
3

---

Published as a conference paper at ICLR 2025
distritution
with lowee and
4,66)
with ligher S R,
X.M
Rebba
KDB & RFF
K
29 45
nepping
sumpliy
Spaco
State Space
duaped sentadd
Chaped rosued
Early learning stage
Late learning stage
charpleshil
Figure 1: A schematic diagram of the self-adaptive success rate based reward shaping mechanism. KDE: Kernel Density Estimation; RFF: Random Fourier Features.
rates evolve from being stochastic to deterministic. This autonomous adaption closely aligns with the agent's exploration-exploiaaiinn balance. Section 4.1 introduces how Beta distributions evolve and how shaped rewards are generated from them. Additionally, to achieve highly efficient com- putation, we use KDE and RFF to estimate success and failure counts, which are used to derive the corresponding Beta distributions, integration SASR into the RL agent and the overall algorithmic flow.
detailed in Section 4.2 Lastly, Section presents the
4.1 SELF-ADAPTIVE SUCCESS RATE SAMPLING
We formulate the augmented reward function in SASR as adding an auxiliary shaped reward RtS (ts(s) to the environmental reward R (s), weighting by a factor A:
RSASR(() = (s) + (s)
We assign the shaped reward R5 of a given state based on its success rate defined as the ra- tio of the state's presence in successful trajectories to its total occurrences. This metric provides a meaningful reward from statistical perspective: higher success rate, reflected in a higher shaped reward, indicates a greater likelihood that the state will guide the agent toward successful task com- pletion. Formally, the success rate based shaped reward R (sx) is given by:
(s:))) f (s:): (())==(N((()))++ NF((s)
Ns((s))
(4)
where (s;)s) and (s:) denote the counts of state s; appearing in successful and failed historical trajectories, respectively. To enhance scalability and adaptability. f()) is linear scaling function that maps the original success rate from [0, to desired scale RRnin Rmar], making the magnitude of the shaped rewards more flexible, i.c., f(z) Rmin + (Rmax Rmin).
Given (s(s;) and (8;);) directly using deterministic success rate may lead to overconfidence in the estimation of the true value. To address this, inspired by the principles of Thompson sam- pling (Thompson] [933] Agrawal & Goyal, 2012), we adopt a probabilistic perspective for success rate estimation. Specifically, the success rate of each state is approximated distribution, with shape parameters set as a = (s(s;) + 1 and 3 -Nr(s:) + 1:
variable in Beta
r: Beta(r; a, a,3) B(Ns(#;) NF
No(a.) (1- Ne(e))
(1-r
(5)
where the beta function B(--)) the normalization factor. By sampling from this distribution, we obtain probabilistic estimate of the true success rate. This sampled value, r-S. is then processed through the scaling function f(-) to produce the shaped reward: R (s.)) = f(r?))
As (s:):) and NF(s:) progressively increase throughout the learning process, they influence the shape and sampling variability of the Beta distribution. Generating the shaped reward from these evolving Beta distributions offers several advantages:
Encourage Exploration. In the early phases, lower counts of Ns(s;) and Nr(s;) result in higher- variance Beta distributions, making the sampled rewards more stochastic. This acts as a noisy perturbation of the reward function. Given that the environmental rewards mostly zero, this perturbation optimizes the agent in diverse directions through small adjustments, shifting the an- chors from which the stochastic policy samples actions. Meanwhile, varttevisited states are likely
4

---

Published as a conference paper at ICLR 2025
fail, leading to decrease in their success rates, while unvisited states retain the initial Beta distribution Beta(1, (1, 1), receiving relatively higher rewards. This mechanism drives the agent to explore novel regions, aligning with the principles of intrinsic motivation.
Enhance Exploitation. In the later phases, as the counts Ns(+;) and Nx(s;) increase, the Beta distribution gradually sharpens, concentrating generated rewards around the true success rate. The more certain reward signals with higher confidence highly support the agent's exploitation, facilitating faster convergence towards optimal policies.
Consistent Optimization. The peak of the Beta distribution, given by (N(s))/(Ng(ss) + (s)).) equals the success rate. Meanwhile, the expectation, (1) + (s));) 2), closely approximates the success rate. This ensures that, despite stochasticity, the overall re- ward remains consistent with policy optimization.
+
4.2 HIGHLYY EFFICIENT BETA DISTRIBUTION DERIVATION
In this section, present how the success and failure counts, Ns((s) and Nr(s;), derived for the Beta distributions. To efficiently estimate these counts in high-dimensional, continuous, infinite state spaces, successes and failures from accumulated experience. Specifically, we maintain two buffers, Ds and Dr. to store the states in successful and failed trajectories, respectively. By treating these states as scattered data instances distributed across the state space, KDE estimates the density as:
Kernel Density Estimation (KDE) to approximate the densities of
Dxl
dx(s:)
K(8; - 88); Xe F),
(6)
Dx
where (--) is the kernel function and |Dxx is the buffer size. We select Gaussian kernel in our implementation. The estimated density dx (s:) approximates the likelihood of encountering state & in success or failure scenarios, providing a statistically sound basis for estimating Nx (s:). By multiplying dx (s:) by the total number of observed states N, the count Nx(s:) is estimated as:
Px
N
||st
(x N dx))))=
exp
(e(
[Dx
2h2
where hyperparameter h is the bandwidth of the Gaussian kernel.
We further integrate Random Fourier Features (RFF) (Rahimi & Recht 2007 reduce compu- tational complexity, as calculating the Gaussian kernel can be expensive, especially in scenarios involving high-dimensional state spaces and large buffers. RFF approximates the kernel function of the original k-dimensional states through an inner product of M-dimensional randomized features:
2 ((s,,)) = 2(s):) 2(3)). 2(s) V M
cos T
s b),
(8)
where x(-) is the RFF mapping function with W (1)
(10) E R and b
[b()) ..... B(M)T" RM is randomly sampled from the following distributions:
w (m) o ((m) Uniform(O, 2m), m m= I,..,,, M,
where Ig is the ls identity matrix. Equation 9 kernels and the detailed derivations of the RFF method are provided in Appendix A.]
is applied for the Gaussian kernel, while different
4.2. IMPLEMENTATION DETAILS
Retention Rate. We introduce hyperparameter, the retention rate Φ E (0, 1]. to regulate the volume and diversity of states stored in the buffers. Rather than storing encountered states, we uniformly retain specific portion of σ. The motivations behind this are: (1) adjacent states in one trajectory tend to be highly similar, especially those near the initial state are repetitive and uninformative, retaining portion of states skip redundant states and increase sample diversity; (2) using lower retention rate in the early stage keeps Ng and F lower, resulting in broader Beta distributions and preventing premature overconfidence.
5

---

Published as a conference paper at ICLR 2025
Algorithm Self-Adaptive Success Rate based Reward Shaping
Require: Environment & and agent AA.
Require: Experience replay buffer
Require: State buffers for success D3 and failure DE-
Require: RFF mapping function 2 : RR.
Sample W and based on Equation
/ do 2: for each environmental step do
1: for each trajectory
3:
(8t, 1t,,0,,4420 8141, rr) CollectTransition((,.
interact with the environment store the transition in the replay buffer record the state in the trajectory
4:
D+ at. s2,1, T
5:
6: end for
7: if trajectory is successful: Ds t Dg Ut
store the trajectory the success buffer >> otherwise, store the trajectory in the failure buffer
8: else: Dp t De U
9: end for
10: for each update step do
11: ((s,, ae. re s(+))s) D
B sample a batch of transitions from the replay buffer D estimate the success counts D estimate the failure counts b sample the success rate from the Beta distribution compute the SASR reward
12:
Ns = 2(5)) ((ss)
13:
Nr = ==(8))"=((pp
14:
ri ~ Beta(r; Ns + 1. +
rt SASR rf + Af(r))
16: Update agent A with {(5t, or st41k}
SASR
end for
Defining Success and Failure. tasks where sparse rewards are only given the end of an episode to indicate task completion, the entire trajectory can be classified either success or failure based on the episodic reward. For tasks with sparse rewards that not explicitly indicate task completion, we segment the trajectories by positive reward occurrences. Specifically, if reward is obtained within pre-defined maximum steps, the corresponding sub-sequence is classified as a success; otherwise, is considered failure.
4.2.2 TIME AND SPACE COMPLEXITY OF SASR
Suppose the buffer size of Dx is D and the batch size is B per iteration, the computational complex- ity to compute the count Nx is D(MB), indicating linear complexity (detailed in Appendix (A.2)) RFF converts nonlinear kernel computations into linear vector operations, significantly speeding up computation by leveraging the vectorization capabilities of GPUs (Dongarra et al.) 2014]. This highlights its superior efficiency compared to methods that rely on network updates and inferences involving extensive nonlinear computations.
Given the retention rate d. the space complexity of maintaining two buffers is O(o]|s)), where T the total iterations and |s] is the size of a single state. Moreover, storage space is significantly reduced by leveraging the existing replay buffer in off-policy RL algorithms, like SAC (Haarnoja et al.] 20188a and TD3 (Fujimoto et al. 2018). Specifically, we augment the replay buffer with a flag that marks each state either success (flag 1) failure (flag 0). This allows for the efficient management of space requirements with minimal overhead from indexing.
For supporting experimental results in time and space complexity, please refer to Appendix A.3
4.3 THE SASR MECHANISM FOR RL AGENTS
Building upon the SASR reward, for demonstration, we employ the soft actor-critic (SAC) algorithm by HHarrojj al. (2018a) as the foundational agent. Let Q the parameterized Q-network and ny be the parameterized policy network. The Q-network is optimized by the following loss function:
((v) (st, (r XR (st) + + (1141, (++)) 2
(10)
where Q obtained from secondary frozen target network to maintain fixed objective (Mnih) al. 2015). Notably, the environmental reward r. is retrieved from the replay buffer, conversely,

---

Published as a conference paper at ICLR 2025
the shaped reward (s;) is computed in real-time using the most recently updated Ns(si) and (s),,, ensuring it reflects the latest learning progress.
We optimize the policy network by maximizing the expected Q-value and the policy entropy (TT)--)s))), following Haarnoja et al. (2018b:
(()) = Bu, en) ( 0x) + log mo (ax (ax]s))]-
(11)
The flow of the SAC-embedded SASR algorithm is summarized in Algorithm
5 EXPERIMENTS
We evaluate SASR in high-dimensional environments, including four MuJoCo tasks (Todorov et al., 2012), four robotic tasks (de Lazcano et al. 2023), five Atari games, including the well-known Mon- lezuma's Revenge Bellemare et al. (2013), and a physical simulation task (Towers et al. 2023), as shown 2 All tasks provide extremely sparse rewards, with reward of granted only upon reaching the final objective within the maximum permitted steps. To ensure robust validation, 10 instances per setting with different random seeds and report the average results. We also maintain consistent hyperparameters and network architectures across all tasks, detailed in Appendix A.6
AutStand
AutiFar
RobotReach
RobotSlide
Pithil
Frogget
Dlmazkeopp
Prewway
Figure 2: MuJoCo, robotic, Atari games and physical simulation tasks in our experiments. Detailed descriptions and the environmental reward models of each task are provided in Appendix A.7
5.1 COMPARISON AND DISCUSSION
Baselines. We compare SASR with ten baselines to benchmark its performance: (a) the online Dis-
tributional Random Network Distillation (DRND) (Yang et al. 2024b), (b) RL with an Assistant
Reward Agent (Relara) (Ma et al. 2024a). (c) General Function Approximation Reward-Free Ex-
ploration (GFA-RFE) (Zhang et al. 2024). (d) RL Optimizing Shaping Algorithm (ROSA) Mgguni
cet al. 2023), (e) Exploration-Gurded RS (ExploRS) (Devidze et al. 2022), (f) Count-based static
hashing exploration (#Explo) (Tang et al. 2017). (g) Random Network Distillation (RND) Burda
al. 2018), (h) Soft Actor-Critic (SAC) (Haarnoja et al.) 2018a). (i) Twin Delayed DDPG
(TD3) (Fujimoto et al. 2018), and (j) Proximal Policy Optimization (PPO) (Schulman et al. 2017).
Algorithms (a) to (g) are all reward shaping methods, incorporating either exploration bonuses
auxiliary agents to shape rewards, while algorithms (h) to (j) are advanced RL algorithms.
Figure 3 shows the learning performance of SASR compared with the baselines, while Table reports the average episodic returns with standard errors achieved by the final models over 100 episodes. Our findings indicate that SASR surpasses the baselines terms of sample efficiency, learning stability, and convergence speed. The primary challenge in these environments is the ex- tremely sparse reward given after a long horizon, making exploration crucial for obtaining successful trajectories in timely manner. Although exploration strategies of algorithms such as ExploRS, #Ex- plo, and RND are designed to reward novel states, effectively expanding the early exploration space with direct additional targets, they continue prioritizing novelty, overlooking the implicit values of these states. As a result, they fail to return to the final objectives.
SASR outperforms the baselines primarily due to its self-adaptive reward evolution mechanism. In the early phases, SASR encourages exploration by injecting substantial random rewards, optimizing the agent in multiple directions, and increasing the likelihood of collecting positive samples. More- over, since most states are initially classified as failures, their success rates decrease. As a result, unvisited states receive relatively higher rewards, encouraging further exploration. This mechanism

---

Published as a conference paper at ICLR 2025
AurStadd
Autfar
RoborBecch
Robodtikle
Pitrell
Pragerr
MantorrreR eveage
HatrerStand
Humaskers
RobesPash
RobaPhiAPlee
Soluris
Freewey
YougtaiiCa
Epindd Gpiedees
(N157)
AMM
Pock-it
Figure The learning performance SASR compared with the baselines.
Table 1: The average episodic returns and standard errors of all models tested over 100 episodes.
Tisks
SASR
DRND-online
ReLara
GFA-RFE
ROSA
ExploRS
#Explo
RND
SAC
TD3
PPO
AntStand
94.9--0.0
67.3+20.0
905+-17
54.2+000
3.8-00.4
5.14:0.4
17.9000
4.0+00222
31.6-0.0
0:0:000
4.9-20.1
AmFFrr
130.8:0000
93.2000
1157400
$64200
10:00
12.02.2
75.10:0
16
25.3=0.0
1.0000
7.80000
HumanStund
7988222
50.6:0000
76.240077
58.2000
8882000
9.300.0
72.7000
9.3-0.1
99:00
53=00
9.0=0.11
HumanKeep
154.5400.0
169.7=000
1950+00
23+00.
RohotReach
170.2::0.00
09
187.9:0.00
42. 42.0000
0.12000
0.720.00
4.60000
60:1:00
156.520.
0.0000
79.520.0
RoboeSlide
132.3-1:.3
127.2400.0
1116622
115.8#20
11.2
4.3+00.1
3.54:0
48+02
0.70022
0.5044
0.2=0.2
RobotPush
49:10:00
00:00.0
37400
00:00
0.0:0.0
RobwePPakPPace
1.0-0.0
1.0+0.0
0.5:000
60:00
00=00.0
00:00
00:00
000:000
0:0000
0.0:000
Piraali
93.0::0.0
52.00:0.0
40:3-::000
394400
0.0:00000
57.6::00
00:00
00:000
4.6:20.0
0.5:2000
0.0:2000
Frogzer
14.2::0.0
11.72:00
116500
794000
988000
8.5
11:9::00
105500
0.200.0
0.7000
0.0:0.0
Montezuma
6757.9:40.0
6828.5=0.0
242199300
4755.3:20.0
4294
397155000
1400.1000
5494.320.0
0.0:0000
0.0=00
0.0:0.0
42.1--0.0
21.3-2077
2033-00
26:3-:000
0.1000
170+00
12408
98400
15=000
Froeway
22
19:8:00
21.5
101:000
18:0:0.0
175200
691:0
130:000
0.1
0.2000
0.020.0
MountainCaar
1,000.0
L0::00
1.000.0
10:40.0
.09000
-100200
10:00
10:200
00.000
0::000
0.9:0.0
resembles intrinsic motivation, assigning higher rewards to novel states, and effectively guiding the agent expand the exploration space. As more data is collected, the success rate estimation be- comes more accurate, the shaped reward provides more reliable guidance, enhancing exploitation and stabilizing convergence. Together, these strategies improve SASR's sample efficiency and con- vergence stability in challenging tasks.
While ReLara used a similar exploration mechanism by perturbing reward functions, relies on an independent black-box agent, requiring more iterations to converge. In contrast. SASR's 's success rate sampling more direct, reducing delays in acquiring valuable information. ReLara's 's advantage lies in incorporating the policy agent's actions into reward construction, seen in the RobotReach task, where the target point randomly selected in each episode. In this case, ReLara outperforms SASR due to access to action information. However, SASR can achieve the same by incorporating actions as additional features in the state vector.
5.2 EFFECT OF SELF-ADAPTIVE SUCCESS RATE SAMPLING
SASR introduces a novel self-adaptive mechanism that balances exploration and exploitation by modulating the randomness of the shaped rewards. To further investigate the effect of this mecha- nism, we use the AntStand task as a case study, analyzing the shaped rewards learned at different training stages. Figure | (bottom) shows the learning curve, while Figure (top) illustrates the dis- tributions of generated rewards over the "height of the ant" feature, dimension in the state vector.
R-1
R-1
R3
R-
R-5
slupod rowanl
R-5
100 Episodic Resurn
R-4
R-1
200
400
Steps (x 10
600
800
1000
Figure 4: Distributions of the shaped rewards over the height of the ant robot in the AntStand task at different training stages. Red diamonds represent the estimated success rate, while the blue polylines show the actual shaped rewards sampled from the Beta distribution.
As learning progresses, the shaped rewards exhibit two key attributes: values transition from ran- dom meaningful, and variance decreases from uncertain to deterministic. Although the sampled
8

---

Published as a conference paper at ICLR 2025
KASR
pehabetioss.o aciiss and
ReLara
PEgpe sovading saucky
toweddng sourity
xcliem
NodryNel pettatbaliziio
parsnnttess
Carx Earx-pattion
Carxpostion
Careeaation
Figure 5: The density of visited states in the MountainCar task for four training periods.
rewards fluctuate, their means gradually show a positive linear correlation with the robot's height. In the early phases, the shaped rewards contain significant randomness due to high uncertainty. While these random signals offer limited information, they drive the agent take small optimiza- tion steps in diverse directions, effectively shifting the policy anchors, expanding the action space sampled from SAC's stochastic policy, promoting exploration, and increasing sample diversity. In later phases, rewards stabilizes, closely aligning with the robot's 's height meaningful and intuitive metric enhancing the agent's exploitation.
To further investigate SASR's 's exploration behavior, we compare the visited state density throughout training the MountainCar task with five representative exploration strategies: (1) Rel.ara, which perturbs both rewards and actions; (2) #Explo and (3) RND, which rewards novel states; (4) SAC, which uses entropy-rege regulay-reuularized exploration; and (5) NoisyNet (Fortunato et al. 2018), which per- turbs network weights. The state density for every 25k steps is shown in Figure 5 We observe that SASR progressively covers a wider range of the state space. From 50k to 100k steps, SASR reaches positions near the goal, driven by its success rate mechanism. In contrast, ReLara and RND cover similar ranges to SASR, but are less smooth and take longer to reach the right side. #Explo shows clear rightward shift, it prioritizes novelty and ignores the inherent value of states. SAC's ploration is relatively narrow, making it prone to getting trapped in local optima. NoisyNet's range narrows over time as perturbations diminish through optimization. Overall, SASR demonstrates more effective exploration and collects valuable samples sooner, leading to faster convergence.
5.3 ABLATION STUDY
We conduct ablation studies investigate key components of SASR. We select six representative tasks and report the experimental results, with quantitative data provided in Appendix A.5
Sampling from Beta distributions. (Figure We examine variant of SASR that omits Beta dis- tribution sampling, instead directly using the success rate (N.ss))(Nssss) + (s:).. In the early stages, limited experience makes this success rate an unreliable estimate, and using fixed, overly confident value can mislead the agent. Furthermore, skipping Beta distribution sampling eliminates exploration driven by random rewards, leading to narrower exploration. The results highlight the critical role of Beta distribution sampling in effective learning.
Reward function over state-action pair. (Figure (b) We extend SASR with a reward function over state-action pairs, r(s, a). The comparison results show that both settings perform similarly. However, encoding actions into the reward function increases dimensionality, complicating density estimation and correlation assessment. Furthermore, the state and action vectors may have different distributions, potentially reducing KDE estimation accuracy.
Retention rate é. (Figure 6c The retention rate directly influences the confidence level of Beta distributions. A high retention rate (o 1) preserves all samples, resulting in a densely populated, redundant state pool, which makes the Beta distribution prematurely overconfident and degrades performance. Conversely, a low retention rate (ò = 0.01) slows convergence as more iterations are required to gather sufficient samples. The results suggest that appropriate retention rate is crucial.

---

Published as a conference paper at ICLR 2025
Aut Stand
AniFar
SunannSunn
HumenKeep
RobatReach
RubotPush
(a) The impact of omitting Bcta distribution sampling on the performance of SASR.
AniSlzed
AntFer
HumenStand
HumseKepp
RoheRRech
RabotPudh
Sxp
send
(b) Learning performance of SASR with shaped reward function Rd (s, a) and (s).
AmiSand
AntFar
HumanSund
RobotRench
RobotPush
Mxpu 201)
(c) Learning performance SASR with different retention rates d.
HumanKesp
10%
(d) Learning performance of SASR with diffcrent RFF feature dimensions M.
AniStand
AnsFar
HummnSond
HummKer
RobufRench
RobotPush
(e) Learning performance SASR with different shaped reward weight factors A.
Figure 6: Ablation study: the impact of key components on the performance of SASR.
RFF feature dimensions M. (Figure 6d SASR shows relatively low sensitivity to M, provided is sufficiently large to capture the complexity of the states. Results show that values like M = 500, 1000, 2000 all yield similar performance, while significantly lower dimensions, such as M 50, degrade performance. These findings align with the original RFF study (Rahimi & Recht 2007).
Shaped reward weight factor A. (Figure beb SASR performs better with intermediate values like 0.4, 0.6, 0.8. At = 0.2, the minimal shaped reward scale reduces state differentiation, leading suboptimal performance. At À aligning the shaped reward scale with the environmental reward introduces excessive uncertainty and randomness, potentially overwhelming feedback and hindering learning. The findings emphasize that maintaining balanced reward scale is important for optimal learning outcomes.
CONCLUSION AND DISCUSSION
In this paper, we propose SASR, a self-adaptive reward shaping algorithm hased on success rates to address the sparse-reward challenge. SASR achieves a balance between exploration and exploitation by generating shaped rewards from evolving Beta distributions. Experiments demonstrate that this adaptability significantly improves the agent's convergence speed. Additionally, the of KDE and RFF provides computationally efficient approach to deriving Beta distributions. This mecha- nism also offers sound alternative traditional count-based RS strategies, adapting effectively to continuous environments. Our evaluations confirm the superior performance of SASR in terms of sample efficiency and learning stability.
While SASR is designed for sewrse-reward environments, in dense-reward settings, the additional shaped rewards may be unnecessary. Extending SASR to such scenarios presents a promising direc- tion for further research. Moreover, the derivation of Beta distributions relies on the samples stored in the success and failure buffers. Currently, our method does not consider the relationships or vary- ing importance of different states within the trajectory, making it sensitive to the retention rate. Therefore, developing an adaptive retention rate or improved buffer management mechanisms crucial for future improvement.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research is supported by the National Research Foundation Singapore and DSO National Labo- ratories under the Al Singapore Programme (Award Number: AISG2-RP-2220-016) This research is also partially supported by an Academic Research Grant No. MOE-T2EP20011-00015 from the Ministry of Education in Singapore.
REFERENCES
Jacob Adamczyk, Argenis Arriojas, Stas Tiomkin, and Rahul Kulkarni. Utilizing prior solutions for reward shaping and composition in entropy-rreuularized reinforcement learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pp. 6658-6665, 2023.
Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit prob- lem. Conference on Learning Theory, pp. 39-1. JMLR Workshop and Conference Proceedings, 2012.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, meth- ods and progress. Artificial Intelligence, 297:103500, 2021.
John Asmuth, Michael L Littman, and Robert Zinkov. Potential-based shaping in model-based reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 604 609, 2008.
Adrià Puigdomënech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Rep- resentations, 2020.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Pro- cessing Systems, 29, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Erdem BIyik, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources human feedback: Optimally inte- grating demonstrations and preferences. The International Journal of Robotics Research, 41(1): 45-67, 2022.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-drivar learning. In International Conference Learning Repre- sentations, 2019.
Ching-An An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learn- ing. Advances in Neural Information Processing Systems, 34:13550-13563, 2021.
Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjac Ryan Lee, and Jordan
Terry. Gymnasium robotics, 2023. URL
Gymnasiumm inmmaaiin-RRooiiccs
Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guided reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 35:5829-5842. 2022.
Sam Michael Devlin and Daniel Kudenko. Dynamic potential based reward shaping. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, pp. 433 440. IFAAMAS, 2012.
11

---

Published as a conference paper at ICLR 2025
Jack Dongarra, Mark Gates, Azzam Haidar, Jakub Kurzak, Piotr Luszczek, Stanimire Tomov, and Ichitaro Yamazaki. Accelerating numerical dense linear algebra calculations with gpus. Numeri- cal computations with GPUs, pp. 3-28, 2014.
Christian Ellis, Maggie Wigness, John Rogers, Craig Lennon, and Lance Fiondella. Risk averse bayesian reward learning for autonomous navigation from human demonstration. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 8928-8935. IEEE, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Represenia- tions, 2019.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference Learning Representations, 2018.
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. In International Conference on Learning Reprexentations, 2018.
Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep
reinforcement learning. Advances in neural information processing systems, 30, 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking re- ward shaping: Understanding the benefits of reward engineering on sample complexity, Advances in Neural Information Processing Systems, 35:15281-15222 2022.
Dhawal Gupta, Yash Chandak, Scott Jordan, Philip Thomas, and Bruno C da Silva. Behavior alignment via reward function optimization. Advances in Neural Information Processing Systems, 36, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint
2018b.
Dylan Hadeicld-Menell, Stuart Russell, Picter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in Neural Information Processing Systems, 29, 2016.
Zhang- Wei Hong, Tzu- Yun Shann, Shih-Yang Su, Yi-Hsiang Chang. Tsu-Jui Fu, and Chun- Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. Advances in neural infor- mation processing systems, 31, 2018
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning utilize shaping rewards: A approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931-15944 2020.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guided policy regularization for offline reinforcement learning. In Forty-first International Conference on Machine Learning, volume 235, pp. 31406 31424. PMLR, 2024.
Tenglong Liu, Jianxiong Li, Yinan Zheng, Haoyi Niu, Yixing Lan, Xin Xu, and Xianyuan Zhan. Skill expansion and composition in parameter space. In International Conference on Learning Representations, 2025.
12

---

Published as a conference paper at ICLR 2025
Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learning, pp- 22594-22613. PMLR. 2023.
Zhengding Luo, Haozhe Ma, Dongyuan Shi, and Woon-Seng Gan. Gfanc-rl: Reinforcement learning-based generative fixed-filter active noise control. Neural Newworks, 180:106687, 2024.
Jiafei Lyu, Xiaoteng Ma, Le Wan, Runze Liu, Xiu Li, and Zongging Lu. Seabo: A simple search- based method for offline imitation learning. In International Conference on Learning Represen- tations, 2024.
Haozhe Ma, Kuankuan Sima, Thanh Vinh Vo, Di Fu, and Tze- Leong, Reward shaping for reinforcement learning with an assistant reward agent. In Forty-first International Conference on Machine Learning, volume 235, pp. 33925-33939, PMLR, 2024a.
Haozhe Ma, Thanh Vinh Vo, and Tze- Yun Leong. Mixed-initiative bayesian sub-goal optimization in hierarchical reinforcement learning. In Proceedings of the 23rd International Conference Autonomous Agents and Multiagent Systems, pp. 1328-1336, 2024b.
Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp- 5125-5133, 2020.
Srinath Mahankali, Zhang-Wee Hong, Ayush Sekhari, Alexander Rakhlin, and Pulkit Agrawal. Ran- dom latent exploration for deep reinforcement learning. International Conference on Machine Learming. 2024.
Jarryd Martin, Suraj Narayanan, Tom Everitt, and Mareus Hutter. Count-based exploration in fea- ture space for reinforcement learning. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 2471-2478, 2017.
Augustine Mavor-Parker, Kimberly Young, Caswell Barry, and Lewis Griffin. How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In International Conference Machine Learning, pp. 15220-15240. PMLR, 2022.
Farzan Memarian, Wonjoon Goo, Rudolf Lioutikov, Scott Niekum, and Ufuk Topcu. Self-supervised online reward shaping in sparse-reward environments. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2369-2375. IEEE, 2021.
David Mguni, Taher Jafferjee, Jianhong Wang, Nicolas Perez-Nieves, Wenbin Song, Feifei Tong, Matthew Taylor, Tianpei Yang. Zipeng Dai, Hui Chen, et Learning to shape rewards using game of two partners. In AAAI Conference Artificial Intelligence, pp. 11604-11612, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 5 518(7540):529-33, 2015.
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning, pp. 2721-2730. PMLR, 2017.
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability unsupervised skill discovery. In International Conference on Machine Learning, pp. 27225-27245. PMLR, 2023.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-2787. PMLR, 2017.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in Neural Information Processing Systems, 20, 2007.
13

---

Published as a conference paper at ICLR 2025
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- environments. In International Conference on Learning Representations, 2020.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In International Join! Conference on Artificial Intelligence, volume 7, pp. 2586-2591, 2007.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Matthias Schultheis, Dominik Straub, and Constantin A Rothkopf. Inverse optimal control adapted to the noise characteristics the human sensorimotor system. Advances in Neural Information Processing Systems, 34:9429-9442, 2021.
Bradly Stadie, Lunjun Zhang, and Jimmy Ba. Learning intrinsic rewards as a bi-level optimization problem. In Conference on Uncertainty in Artificial Intelligence, pp. 111-120. PMLR, 2020.
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-133) 2008.
Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploit reward shifting in value-based deep-RL: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems, 35:37719-37753 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Chen, Yan Duan, John Schul- man, Filip DeTurek, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. Advances Neural Information Processing Systems, 30, 2017.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of samples. Biometrika, 25(3-4):285-294, 1933.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012.
Mark Towers, Jordan K. Terry Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente Vicente, An- drea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymna- sium, March 2023. URL
Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self- hifanialnning shaped rewards, Advances Neural Information Processing Systems, 32, 2019.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in Neural Information Processing Systems, 29, 2016.
Eric Wiewiora.
Potential- based shaping and q value initialization are equivalent. Journal of Artificial Intelligence Research, 19:205-201 208, 2003.
Rand R Wilcox. Introduction to robust estimation and hypothesis testing. Academic press, 2011. 1.
Yuchen Wu, Melissa Mozifian, and Florian Shkurti. Shaping rewards for reinforcement learning with imperfect demonstrations using generative models. In IEEE International Conference on Robotics and Automation, pp. 6628-6634. IEEE, 2021.
Kai Yang, Zhirui Fang, Xiu Li, and Jian Tao. Cmbe: Curiosity-driven model-based exploration for multi-agent reinforcement learning in sparse reward settings. 2024 International Joint Conference Newral Networks (IJCN), pp. 1-8. IEFE, 2024a.
14

---

Published as a conference paper at ICLR 2025
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti-exploration with distributional ran- dom network distillation. In Forty-first International Conference on Machine Learning. PMLR, 2024b.
Yuxuan Yi, Ge Li, Yaowei Wang, and Zongqing Lu. Learning share in networked multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 55:15119-15133 2022.
Hai Zhang, Hang Yu, Junqiao Zhao, Di Zhang, Chang Huang, Hongtu Zhou, Xiao Zhang, and Chen Ye. How to fine-tune the model: Unified model shift and model bias policy optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Hai Zhang, Boyuan Zheng, Tianying Ji, Jinhang Liu, Anqi Guo, Junqiao Zhao, and Lanqing Li. Scrutinize what ignore: Reining in task representation shift of context-based offline meta reinforcement learning. In International Conference on Learning Representations, 2025.
Junkai Zhang, Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Uncertainty-awan reward-free free exploration with general function approximation. International Conference on Machine Learning, 2024.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI Conference Artificial Intelligence, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DERIVATION OF RANDOM FOURIER FEATURES
We incorporate Random Fourier Features (RFF) (Rahimi & Recht 2007) to approximate the kernel functions in the KDE process for the SASR algorithm. Let the original state be k-dimensional, denoted as IR and the kernel function be k(s;, sy). RFF approximates the kernel function by projecting the input k-dimensional space into a M-dimensiional feature space using mapping function z I Ad The RFF--based kernel function is then defined as follows:
k(s;,, 85) R =(8))" (*,))
(12)
We provide the derivation of the RFF approximation in this section.
First, we clarify that RFF primarily targets shift-invariant kernels, that satisfy k(s;, 8,) = ((5:-8)) Common shift-invariant kernels include Gaussian kernels, Laplacian kernels, and Cauchy kernels. Given a shift-invariant kernel function k(A), we perform the inverse Fourier transform:
k(s 8;) p(w)e :w (a.-8;) dw
(13)
= =E [eiun (s=-a)))
(14)
where we can consider w ~ p(w) based on the Bochner's theorem, and p(w) is called the spectral distribution of kernel function. For the three types of shift-invariant kernels, the corresponding spectral distributions are listed in T
Table 2: Some thift-invariant kernels and their associated spectral distributions.
Kernel
Kernel function, k(s; -8))
Spectral density, p(w)
|; -8112
Vi. exp
h
Gaussian
exp
2
2/TT
4
M
Laplacian
exp - ||s; - -syll1)
II
T
w
m=1
2
Cauchy
exp ( ||2||1
(s, - 82))
Next, we perform the Fuler's formula transformation, which retains only the cosine term since we are dealing with realevalued functions, the kernel function can be further derived as:
k(ss, 8j) = Ew i
(a. (a.-
(15)
Ew cos( wn (s--8)))
(16)
E, cos( -0 (s: (s:-8))) Em [Ea[cos( (s: + 8;) + 26]]]
(17)
= Em [cos( (n" (8: 8,)) + cos(w" (s: + 88)) ) + 26]]]
(18)
Ew [., 2 cos( "T s; + b) c w0 8; + b]]]
(19)
where b Uniform 2T).. Equation 17 holds since Unifonnninnn (7, cos(t + 26)] for any
Equation 19 is obtained from cos( B) + cos( A + B) = cos( A) (B)( where = wt 8; b.
B w 8, b.
We define the mapping (s) /2 cos 8 b), then the kernel function can be approxi- mated by the inner product of two vectors and the expectation can be approximated by Monte Carlo sampling:
k(s;, s,) = Eu Es (el (w
(20)
M
1
2
Zwabaa (8:) 2w Rw (83)
(21)
M
m==1
=(())) 2(3)).
(22)
16

---

Published as a conference paper at ICLR 2025
Therefore, we have derived the mapping function 2s)) 2/M cos(WTs 's b), where W RMxx and RM,, The RFF-based kernel function can be approximated by the inner product of the mapped features the M-dimensional space.
A.2 COMPUTATIONAL. COMPLEXITY
In this section, we derive the computational complexity to retrieve the success or failure counts Ns and Nr for each iteration. Suppose the buffer size of Dxx is D, the batch size of B is B, the corresponding counts are retrieved by calculating:
Nx = N x xBB)"=((x))
(23)
where the mapping function defined as:
2 V M cos (W
2ss)
W WERAXM BERM.
(24)
For each state, the mapping function calculation involves:
1. Matrix multiplication WT k.M Addition W s + b: M.
3. Cosine calculation cos WTs b): M.
Therefore, the computational complexity for calculating zs) for one state is O(kM).
For each pair of states (8;, 8j), calculating the kernel involves multiplications and M addi- tions, thus, the complexity is O(M).
For each iteration, we calculate the RFF mapping for all states the buffer and the batch and then compute the kernel between them. The complexities involve three parts: RFF mapping for the buffer, RFF mapping for the batch and kernel calculation:
O(DkM) + ((BM)) + O(DBB).
(25)
Since the first two terms O(Dk.) and O(BkM) are dominated by the last term OMMDB) when the buffer size and the batch size are large, the overall computational complexity to retrieve the corresponding counts can be approximated (MDB).
A.3 EXPERIMENTS TIME AND SPACE COMPLEXITY
A.3.1 TIME AND SPACE COMPLEXITY COMPARISON
In this section, analyze the time and space overhead introduced by SASR and other representative reward-shaping methods. Below, we summarize the computational and memory costs of the RS baselines, introduced by the shaped reward generation.
SASR (ours) calculates shaped rewards using RFF, which essentially is matrix operations, without additional networks/models learning processes. Regarding the memory costs, the buffers Ds and Dr are much smaller than the replay buffer used in the backbone SAC algorithm, due to the retention rate Φ. While considering the scalability for larger problems, we have implemented an alternative approach by augmenting the original replay buffer in the backbone SAC algorithm with a success or failure flag. This approach avoids the need for additional buffers.
ReLara (Ma et al. 2024a) requires an additional RL agent (of the same scale as the original RL agent) and an additional replay buffer.
ROSA (Mguni al. 2023) involves a competition agent (the same sacle as the original RL agent) and switching model (a neural network).
EXploRS (Devidze et al. 2022) requires learning two parameterized networks: one for a self- supervised reward model and another for the exploration bonus.
#Explo (Tang et al. 2017) requires a hash function to discretize the state space and a hash table store the state-visitation counts.
RND (Burda et al. 2018) uses a random network distillation module to compute the intrinsic rewards.
17

---

Published as a conference paper at ICLR 2025
Furthermore, report the computational and memory costs of SASR and the RS baselines in two tasks: AntStand and Frogger, the results are shown in Table 3 and Table respectively. To provide more intuitive comparison, report the relative value normalized our SASR, in this case, if the value > 1, it indicates that the baseline is more computationally or memory expensive than SASR, and vice versa.
Table Average maximum memory consumption during the training process, normalized SASR.
Tasks
SASR
ReLara
ROSA
ExploRS
#Explo
RND
AntStand Frogger
3.67
4.12
2.05
0.89
0.12
1
5.21
4.33
2.64
0.92
0.09
Table 4: Average training time, normalized to SASR.
Tasks
SASR
ReLara
ROSA
ExploRS
#Explo
RND
AntStand
!
1.87
2.12
1.67
1.08
1.11
Frogger
1
1.98
3.17
1.72
1.24
1.06
A.3.2 COMPARISON OF SASR WITH AND WITHOUT RFF
To evaluate the effect of introducing RFF, we compare the training time of SASR with and with- out RFF, also with the backbone SAC algorithm, the results are shown in Table [ The tests are conducted on the NVIDIA RTX A6000 GPUT. The results show that excluding SAC's inherent optimization time, RFF significantly saves time in the SASR algorithm, while with varying effects across tasks.
Table 5: Comparison of training time (in hours) for SASR with and without RFF.
Algorithms
AntStand
AntFar
HumanStand
HumanKeep RobotReach RobotPush
SAC (backbone)
5.87
5.08
4.87
5.67
5.42
6.3
SASR KDE+RFF
7.15
7.52
6.92
6.20
7.07
8.13
SASR w/o RFF
8.12
8.72
8.37
6.53
11.12
9.21
A.4 AUTO HYPERPARAMETER SELECTION
To improve the robustness and generalization of the SASR algorithm, we propose some potential autonomous hyperparameter selection strategies, mainly designed for the bandwidth of the kernel function and the RFF feature dimension M.
For bandwidth h, can the empirical formula Silverman's 'ss Rule of Thumb [Wiloxx 2011
h 1.06 N
(26)
or cross-validation to determine the optimal bandwidth.
For the RFF dimension M, directly related the bandwidth h. After determining we can use the formula mentioned in the RFF theory to determine M:
N
M 0 log
(27)
where and å are the error and confidence parameters. Another method is compare the Frobe- nius norm error between the RFF approximated kernel matrix K RPF and the true kernel matrix K Gauesion to select M: RFF K Gauesian
18

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study #1: The average episodic returns and standard errors of SASR and the variant without sampling from Beta distributions.
Tasks
SASR (with sampling)
SASR (without sampling)
AntStand
94.92.10.00
54.48 1.29
AntFar
139.84-1..00
92.77 + 1.53
HununStand
79.83.1.2.03
9.77 + 0.02
HumanKeep
195.77=0.00
185.00 0.00
RobotReach
170.18++0.00
110.29 2.93 86.82 = 0.00
RobotPush
167.14=:000
Table 7: Ablation study #2:
The average episodic returns and standard errors of SASR with reward function on state-action pair or state only.
Tasks
SASR (with R (s))
SASR (with R (s. a))
AntStand
94.92110.00
85.61+1.30
AntFar
139.84=0.00
132.49=2.92
HumanStand
79.83=+2.03
78.93=00.65
HunanKeep
195.77=
192.54--0.16
RobotReach
170.18::0000
151.955+5.74
RobotPush
167.14+0.00
179.76-
Table 8: Ablation study #3: The average episodic returns and standard errors of SASR with different retention rates.
Tasks
0.1 (default)
0.01 á
AntStand
45.71
94.92:+0.00
62.85++3.49
AntFar
70.07+3.30
139.84-10.00
103.65=2..77
SundnStand
9.88-10.01
79.83.1.2.03
66.46.12.99
HumanKeep
195.00:10.00
195.77-10.00
194.7711.100
RobotReach
154.32-10.89
170.18:10.00
112.46:10.90
RobotPush
2.96+1197
167.14=0.00
1.75
Table 9: Ablation study #4: The average episodic returns and standard errors of SASR with different RFF feature dimensions M.
Tasks
M = 50
. = 500
M 1000 (defauut)
A = 2000
AniStand
5.2110.45
50.68:16.40
94.92110.00
96,80118.42
AntFar
98.88112.64
72.17=5.007
139.84:10.00
129.87110.3
HunanStand
9.8710001
78.82-10.52
79.83:12.03
1..73
HumanKeep
193.84.10.61
194.7111..15
195.77-00000
195.86.10.00
RobotReach 119.094426.92
122.89-4.51
170.18=0.00
94.87+15 5,56
RobotPush
71.67+27.53
161.70++1..4
167.14=0.00
150.20++8.70
Table 10: Ablation study #5: The average episodic returns and standard errors of SASR with differ- ent shaped reward weight factors.
Tasks
= 0.6 (default)
= 0.8
A 1.0
AntStand
35.71=0.92
59.82-12.71
94.92:10.00
75.61+1.41
3.16::0.3
AntFar
99.83+3.25
119.82=1.18
139.84110.00
119.35+1.80
80.71:1..74
HunanStand
9.81++0.02
75.35++1.06
79.83+-2.03
70.96+0.51
28.94440.46
Human
194068-0.08
194.21-0.38
195.77+-0.00
193.85-t0.22
194.89+0.10
RobotReach
131.6144.51
154.16-5.20
170.18-+0.00
169.00+2.56
74.23+4.23
RobotPush
13.0741.65
193.56=3.85
167.1411.000
178.90-10.00
192.07
19

---

Published as a conference paper at ICLR 2025
A.5 SUPPLEMENTARY EXPERIMENTAL RESULTS FOR ABLATION STUDY
In this section, we provide the detailed quantitative results of the ablation study.
Bandwidth h of Gaussian kernel. The bandwidth h controls the smoothness of the kernel func- tions. Beyond fixed bandwidths, we also test a linearly decreasing configuration (h- : 0.5 + 0.1), which reflects increasing confidence in KDE. Results indicate that a small bandwidth (h 0.01) increases the distance between samples, causing many to have zero estimated density, while large bandwidth (h 1) makes samples indistinguishable due an overly flat kernel function. Both cases result in suboptimal performance. The decreasing bandwidth setting offers no significant im- provement and tends to reduce stability due to inconsistent density estimations.
Ablation study #6: The average episodic returns and standard errors of SASR with differ- ent bandwidths h of Gaussian kernel.
Table 11 1:
Tasks
h= 0.01
h 0.1
h = 0.2 (default)
h h=
h = 0.5 + 0.1
AntStand
10.71+2.52
57.22=#3.87
94.92++0.00
17.74=2.53
68,40=1.554
AntFar
17.58=2.84
99.80=4.41
139.84=00.00
25.83=8.62
136.49:4..15
HumanStand
9.89=0.01
64.47=1.87
79.83+22.03
9,90=0.02
58.7942.76
HumanKeep
194.92-0.02
194.00=10.57
195.77-10.00
193.06::046
194.59-1..18
RobotReach
128.57
97.35
170.18:10.00
134.02+42.02
59.39
RobotPush
2.29+1162
122.45+33..58
167.14::0.00
0.00-+0.00
0.01
A.6 NETWORK STRUCTURES AND HYPERPARAMETERS
A.6.1 NETWORK STRUCTURES
Figure 7 illlsttates the structures of the policy network and Q-network employed in our experiments. The agent utilizes simple multilayer perceptron (MLP) models for these networks. Given the use of stochastic policies, the policy network features separate heads to generate the means and standard deviations of the inferred normal distributions, which are then used to sample actions accordingly.
observation
FC layer (observation 256)
(observation, action)
ReLU
FC layer (obsernation + action shope, 256)
FC layer (256, 256)
ReLU
ReLU
ReLU
FC layer
FC layer (256, action shape)
FC layer (256, 256)
(256, action awt.aooocooapppp shape)
tanh
ReLU
means
standard deviations
FC layer (256,1)
Normal distributions
Q-value
sample
actions
(a) policy network structure
(b) Q-network structure
Figure 7: The structures of policy network and Q-network in our implementation.
A.6.2 HYPERPARAMETERS
We have observed that SASR demonstrated high robustness and was not sensitive to hyperparameter choices. Table 12 shows the set of hyperparameters that we used in all of our experiments.
20

---

Published as a conference paper at ICLR 2025
Table 12: The hyperparameters used in the SASR algorithm.
Hyperparameters
Values
reward weight À (default)
0.6
kernel function bandwidth
0.2
random Fourier features dimension M retention rate (default)
1000
0.1
discounted factor Y
0.99
replay buffer size [D] batch size
1 x 10
256
actor module learning rate critic module learning rate SAC entropy term factor a learning rate policy networks update frequency (steps) target networks update frequency (steps)
x 10-4
1 x 10
x 10
2
1
target networks soft update weight burn-in steps
10-3
5000
A.6.3 COMPUTE RESOURCES
The experiments in this paper conducted computing cluster, with the detailed hardware configurations listed Table 13 The computing time for the SASR algorithm in each task (running 1,000,000 steps) was approximately 6 + hours.
Table 13: The compute resources used in the experiments
Component
Specification
Operating System (OS)
Ubuntu 20.04
Central Processing Unit (CPU)
2x Intel Xeon Gold 6326 256GB Ix NVIDIA A 100 20GB Supermicro 2022
Random Access Memory (RAM)
Graphics Processing Unit (GPU)
Brand
A.7 CONFIGURATIONS OF TASKS
In this section, we provide the detailed configurations of the tasks in the experiments.
AntStand:
The ant robot is trained to stand over a target position. The reward given if the ant robot reaches the target height. Maximum episode length is 1000 steps.
AntFar: The ant robot is trained to reach a target position far from the starting point. The reward is given if the ant robot reaches the target position. Maximum episode length is 1000 steps.
HumanStand: The human robot trained to stand over a target position. The robot initialized by lying the ground, and the reward is given if the robot reaches the target height. Maximum episode length is 1000 steps.
HumanKeep: The human robot is trained to keep target height. The robot is initialized by standing, and the reward is given if the robot maintains the target height. Maximum episode length is 1000 steps.
RobotReach: The robot arm is trained to reach target position. The target position is randomly generated in the workspace, and the reward is given if the robot reaches the target position. Max- imum episode length is 500 steps.
RobotPush: The robot arm is trained to push an object to a target position. The target position is randomly generated the table, and the reward is given if the object reaches the target position. Maximum episode length is 500 steps.
RobotSlide: The robot arm is trained to slide object to target position. The target position is randomly generated on the table, and the reward given the object reaches the target position. Maximum episode length is 500 steps.
21

---

Published as a conference paper at ICLR 2025
RobotPickPlace: The robot arm is trained to pick and place object to target position. The target position is randomly generated in the space, and the reward is given if the object reaches the target position. Maximum episode length is 500 steps.
Pitfall: The agent is tasked with collecting all the treasures in a jungle while avoiding the pitfalls. The reward is given if the agent collects treasure, while if the agent falls into a pitfall, the episode ends. Maximum episode length is 2000 steps.
Frogger: The agent is trained cross frogs on river. The reward given when each frog is crossed, and the episode ends if all frogs are crossed or fall into the river. Maximum episode length is 2000 steps.
Montezumalevenge: The agent is trained to navigate through series of rooms collect keys and reach the final room. The reward is given if the agent successfully reaches one new room. Maximum episode length is 5000 steps.
Solaris: The agent controls a spaceship to blast enemies and explore new galaxics. The reward is given if the agent destroys one enemy spaceship and enters new galaxy. Maximum episode length is 2000 steps.
Freeway: The agent is trained to guide the chicken across multiple lanes of heavy traffic. The reward is given if one chicken crosses one lane, while the episode ends if all chickens are crossed hit by car. Maximum episode length 2000 steps.
MountainCar: The car is trained to reach the top of the right hill. The reward is given if the car reaches the top. Maximum episode length is 1000 steps.
Furthermore, we provide the detailed dimensions of the states in our evaluated tasks Table 14
Table 14: The dimensions of the states in the evaluated tasks.
Domain (Tasks)
Dimension
Ant robot (AntStand, AntFar)
105
Humanoid robot (HumanStand, HunanKeep)
348
RobotReach
20
RobotPush, RobotSlide and RobotPickPlace
35
Atari games (MontezumaRevengee Pitfall, Frouger, Solaris, Freeway) MountainCar
84 x 84 = 7056
2
22


Reviews:
[{"review_id": "fr4L1k27st", "reviewer": "Reviewer_XbGP", "summary": "The paper proposes SASR, a novel method for adaptive reward shaping in RL with sparse binary rewards, based on the idea that states that belong to trajectories ending in success are more valuable than states that belong in a trajectories that end in failure, so visiting such states should be encouraged by increasing the immediate stage reward leading to them. This mechanism is a form of temporal credit assignment, a central question in RL with sparse rewards, and appears to be very effective in speeding up learning in multiple test problems.", "questions": "The use of a state's success rate reminds me of the use of eligibility traces, whose purpose is also to assign credit to states and actions taken in them. Eligibility traces typically depend on how far in the future success happened, whereas in this paper, the success of a state does not depend on the time horizon of a positive reward. Have the authors considered time-dependent or other variable weights depending on distance to the goal state, similar to eligibility traces? \n\nCould the authors clarify what they mean by \"parameter-free\" and \"learning-free\" in the context of KDE estimation? Could the authors discuss potential automated methods for selecting parameters like bandwidth h, number of RFF M, and reward weight λ?\n\nRegarding getting to successful outcomes earlier rather than later, does SASR provide any benefits in the very early stages of learning before successful trajectories are found?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The main strength of the paper is perhaps the principled approach in augmenting the reward function of RL with sparse rewards with a component that encourages visiting states that have proven to be useful in completing a sequential decision making task in past trials. The proposed method of using beta distributions to model the probability that a state is useful avoids overconfidence in this probability from very few samples, and is helpful in encouraging exploration in the early stages of learning. The proposed use of kernel density estimation combined with random Fourier features solves the problem of keeping tally of which states are successful in continuous state spaces. The experimental verification is rigorous and of high quality and leaves no doubt in the effectiveness of the method on representative benchmark problems of significant complexity. The paper is well written and easy to follow, and the related work section is thorough and informative.", "weaknesses": "The method still needs at least some trajectories with successful outcomes, so it cannot speed up the initial discovery of such trajectories, which is perhaps the most difficult problem in RL with sparse rewards. \n\nA claim has been made that the method for estimating the beta distributions of expected success of individual states is parameter-free and learning-free, and I am not sure I agree with that. At the very minimum, the KDE estimator depends on the bandwidth parameter h, and the ablation study in Section 5.3 clearly shows that its value does matter a lot as regards the accuracy of estimation and the overall success of the algorithm. Usually, the bandwidth of KDE schemes is determined by a search routing of some kind, either grid search or cross-validation, which is a form of learning. Other parameters that matter are M, the number of RFF, and lambda, the weight of the additional reward term, and they must be determined somehow, preferably automated by some form of learning.\n\nA minor typo:\nP. 8. L. 427: \"broadens\" -> \"broadening\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aIFnKksVhH", "reviewer": "Reviewer_JGvx", "summary": "The paper presents a method called Self-Adaptive Success Rate based Reward Shaping (SASR) aimed at addressing the sparse-reward problem in reinforcement learning. SASR uses success rates from past interactions to construct a shaped reward functions. Success rates are sampled from an evolving Beta distribution, estimated via Kernel Density Estimation with Random Fourier Features to improve efficiency. The authors experimentally validate SASR and ablate their design choices.", "questions": "See **Weaknesses**. Additionally:\n1. Does the method require to know $R_\\min$ and $R_\\max$? If not, are they estimated? How?\n2. In line 305, it is stated that \"if a reward is obtained within a pre-defined maximum number of steps, the corresponding sub-sequence is classified as a success\". How is the threshold for the reward and number of steps decided? Is the algorithm sensitive to how these are chosen?\n3. In the following grid-world environment, where B is a bad state with very low reward, O is an empty state, G is the goal state, W is a wall and A is the agent, would SASR encourage the agent A to encourage the agent to take the longer path that goes down, right and then up because the state between B and G would be counted in many successful and failure trajectories, so it would be deemed a bad state? How could you make sure that in this case, the agent still takes the shortest path?\n```\n| B | O | G | O |\n| W | A | W | O |\n| W | O | W | O |\n| W | O | O | O |\n```", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The use of Beta distributions to manage exploration-exploitation trade-offs in sparse-reward environments seems to be simple yet effective.\n- The use of KDE with RFF makes the estimation of Beta distributions more efficient. Moreover, the success rates based method is more computationally efficient than similar previous work (ReLara), which uses a neural network to learn shaping rewards.\n- SASR shows improved convergence stability in sparse-reward settings compared to several baselines.", "weaknesses": "1. **Inconsistent Notation.** The notation is inconsistent in some parts of the paper.\n\t1. Equation 6 uses boldface notation for state variables, while the rest of the paper does not.\n\t2. Similarly, in Algorithm 1, in lines 12 and 13 the states and states buffers are bolded, when they are not everywhere else.\n\t3. In Section 4.3, the environmental reward uses a different notation from Equation 3. The authors should choose to either use $R^E(s_{t})$ or $r_t^E$.\n2. The retention rate $\\phi$ in line 265 has not been introduced yet. It can be helpful to introduce it before using it in presenting the space complexity of SASR.\n3. Section 4.3 seems to be partially copied from the ReLara paper.\n\t1. The $\\mathcal A_P$ in the Q-function is redundant.\n\t2. The notation for the policy should be made consistent. At the moment, the policy appears with three different notations.\n4. This method was only tested for state-only reward functions. In contrast, ReLara can also work for state-action reward functions. Why is the reward function considered in SASR state-only? Can the authors also show the efficacy of SASR for state-action reward functions or provide arguments as to why this is not necessary?\n5. Montezuma Revenge is know to be a hard exploration problem. It would be insightful to have a comparison of SASR with other exploration algorithms, such as RND.\n6. SASR appears to be quite similar to ReLara, except it is restricted to using a Beta distribution for success rates, rather than a more expressive neural network to propose shaping rewards. The paper would benefit from a more thorough discussion on why ReLara seems to perform worse, despite using an overparametrized neural network, which should in theory be able to learn a Beta distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lpb2z0nsUe", "reviewer": "Reviewer_NDw1", "summary": "This paper introduced Self-Adaptive Success Rate based reward shaping (SASR) for reward-sparse RL problems. SASR incorporates success rates, defined as the ratio of a state's presence in successful trajectories to its total occurrences, as auxiliary rewards. Further, SASR adopt Kernel Density Estimation (KDE) and Random Fourier Features (RFF) to derive Beta distributions and sample success rates from that distribution.", "questions": "- How would SASR perform on some famous hard exploration tasks, i.e., Montezuma's Revenge and DM-Hard-8?\n- Can SASR solve difficult maze tasks with sparse reward, i.e., `AntMaze_HardestMaze-v5` from https://robotics.farama.org/envs/maze/ant_maze/?\n- For a hard exploration task where we only receive nonzero reward when the task is solved. If it's very difficult to collect the first successful trajectories. How will SASR perform when the success rates for all states are zero? Will the beta distribution be accurate enough to solve such problems?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of using success rate as auxiliary reward is interesting.\n- The writing is clear and the paper is easy to follow.\n- SASR shows promising results in the experiments.", "weaknesses": "- Some of the selected environments are simple toy tasks. For example, the authors select the simpler Reach and Push tasks instead of the more challenging PickAndPlace and Slide tasks. It's still unclear if SASR is effective for hard-exploration tasks from the current experiment results.\n- Many baselines are old. The paper should compare to more recent baselines in exploration. For example,\n    - DRND (https://github.com/yk7333/DRND)\n    - RLE (https://github.com/Improbable-AI/random-latent-exploration)\n    - GFA-RFE (https://github.com/uclaml/GFA-RFE)\n- There are no pixel-based experiments, i.e., Atari games. It's unclear how would SASR perform with pixel-based inputs?\n- Typos: (Line 296) \"The motivations behind this are\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KYao8w7GZS", "reviewer": "Reviewer_52SB", "summary": "The paper proposes a reward shaping method, SASR, for deep reinforcement learning by incorporating success rates derived from historical experiences. These success rates are sampled from Beta distributions with decreasing uncertainty. Kernel Density Estimation (KDE) and Random Fourier Features (RFF) are used to derive the Beta distributions. The method is evaluated on sparse reward tasks and compared with eight baseline methods. The results demonstrate that the proposed method outperforms the baseline methods on the chosen tasks.", "questions": "- What is the formula for the scaling function $f$ in Equation 4?\n\n- In sparse reward tasks, the most challenging part is at the beginning of the learning process when all the trajectories fail or have zero rewards. How does the proposed method handle this situation? What advantage does it offer over other methods in this situation?\n\n- It seems Table 1 computes the average reward over the entire learning process. Why not use the final reward as the metric?\n\n- RND achieves the best performance on MountainCar. Why is it not included in Figure 5?\n\n- What are the dimensions of the chosen tasks? It appears that the state space is a vector of tens of dimensions. How does the method perform on tasks with high-dimensional state space or images-based tasks like Atari games or dmcontrol tasks?\n\n- Since all tasks provide extremely sparse rewards, is the proposed method only suitable for sparse reward tasks? If not, how does it perform on tasks with dense rewards?\n\n- Given that 7 out of 8 baseline methods were not tested on these chosen tasks in the original papers, how did you finetune the hyperparameters of these baseline methods? Did you compare their performance on the tasks reported in the original papers?\n\n- See Weaknesses.\n\nI am willing to increase my score if all my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-organized and easy to follow. \n- The idea makes sense. The method is original and the authors provide a clear explanation of the method.\n- The authors compare their method with 8 baseline methods, which is a good practice.", "weaknesses": "- The scope of the proposed method is unclear. Although the authors mention \"sparse reward tasks\" several times, it remains uncertain whether the method is exclusively suitable for sparse reward tasks. If so, this should be clearly stated [1].\n\n- The authors motivate the method by mentioning that \"most real-world environments involve high-dimensional state spaces\". However, the tasks chosen for the experiments appear to involve vectors with only tens of dimensions.\n\n- The chosen tasks are not standard in the RL community. The authors compare their method with 8 baseline methods, but 7 of these were not tested on the chosen tasks in the original papers. Additionally, there are no implementation details or results of hyperparameter searches for these baseline methods, making it unconvincing that the proposed method outperforms them. I recommend testing the proposed method on standard tasks used in the chosen baselines, such as sparse reward Atari games like Montezuma's revenge and Gravitar chosen by ROSA, #Explo, RND and PPO, or mujoco tasks (though they may not be sparse reward tasks) chosen by #Explo, SAC, TD3, and PPO. In addition, if the proposed method is intended for sparse-reward, high-dimensional, continuous control tasks, dmcontrol conld be a good choice [2,3].\n\n- The method introduces many parameters to tune, such as the weighting factor $\\lambda$, the bandwidth of the Gaussian kernel $h$, the retention rate $\\phi$, and the Fourier features dimension $M$. These parameters are sensitive to performance, as shown in the ablation study (Figure 6). Additionally, some functions need to be predefined, such as the scaling function $f$ and the kernel function $K$. The interaction between these parameters and functions is complex, making it difficult to understand how they influence each other and how can the method be tuned for other tasks.\n\n- The method introduces additional computational costs (reward computation) and memory costs (three buffers). It is unfair to compare it with other methods without considering these costs. A comprehensive comparison with other methods in terms of computational and memory costs, in addition to performance, would be more appropriate.\n\n\n[1] Devidze, Rati, Parameswaran Kamalaruban, and Adish Singla. \"Exploration-guided reward shaping for reinforcement learning under sparse rewards.\" Advances in Neural Information Processing Systems 35 (2022): 5829-5842.\n\n\n[2] Tassa, Yuval, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden et al. \"Deepmind control suite.\" arXiv preprint arXiv:1801.00690 (2018).\n\n\n[3] Yarats, Denis, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. \"Mastering visual continuous control: Improved data-augmented reinforcement learning.\" arXiv preprint arXiv:2107.09645 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)