Residual Kernel Policy Network: Enhancing Stability and Robustness in RKHS-Based Reinforcement Learning
Authors: Yixian Zhang
Huaze Tang
Huijing Lin
Wenbo Ding
Venue: ICLR 2025 Poster (2025)
Keywords: policy learning
reproducing kernel Hilbert space
representation learning
variance reduction
Primary area: reinforcement learning
TL;DR: We introduce the Residual Kernel Policy Network (ResKPN), which integrates representation learning with a residual layer to mitigate gradient variance and enhance the robustness of the RKHS-based policy.

Abstract: Achieving optimal performance in reinforcement learning requires robust policies supported by training processes that ensure both sample efficiency and stability. Modeling the policy in reproducing kernel Hilbert space (RKHS) enables efficient exploration of local optimal solutions. However, the stability of existing RKHS-based methods is hindered by significant variance in gradients, while the robustness of the learned policies is often compromised due to the sensitivity of hyperparameters. In this work, we conduct a comprehensive analysis of the significant instability in RKHS policies and reveal that the variance of the policy gradient increases substantially when a wide-bandwidth kernel is employed. To address these challenges, we propose a novel RKHS policy learning method integrated with representation learning to dynamically process observations in complex environments, enhancing the robustness of RKHS policies. Furthermore, inspired by the advantage functions, we introduce a residual layer that further stabilizes the training process by significantly reducing gradient variance in RKHS. Our novel algorithm, the Residual Kernel Policy Network (ResKPN), demonstrates state-of-the-art performance, achieving a 30% improvement in episodic rewards across complex environments.

Published as a conference paper at ICLR 2025
RESIDUAL KERNEL POLICY NETWORK: ENHANCING STABILITY AND ROBUSTNESS IN RKHS-BASED REIN-
FORCEMENT LEARNING
Yixian Zhang, Huaze Tang '& Huijing Lin
Tsinghua Shenzhen International Graduate School, Tsinghua University (yixian-2244 ranghz24, ,, 1inhj24 2 @mails. tsinghua. edu en
Wenbo Ding
Tsinghua Shenzhen International Graduate School, Tsinghua University Shanghai Artificial Intelligence Laboratory ding. , wenbols sz. tsinghua. edu. cn
ABSTRACT
Achieving optimal performance in reinforcement learning requires robust policies supported by training processes that ensure both sample efficiency and stability, Modeling the policy in reproducing kernel Hilbert space (RKHS) enables efficient exploration of local optimal solutions. However, the stability of existing RKHS- based methods is hindered by significant variance in gradients, while the robust- ness of the learned policies is often compromised due to the sensitivity of hyper- parameters. In this work, we conduct a comprehensive analysis of the significant instability in RKHS policies and reveal that the variance of the policy gradient increases substantially when a wide-bandwidth kernel is employed. To address these challenges, we propose novel RKHS policy learning method integrated with representation learning to dynamically process observations in complex en- vironments, enhancing the robustness of RKHS policies. Furthermore, inspired by the advantage functions, we introduce residual layer that further stabilizes the training process by significantly reducing gradient variance in RKHS. Our novel algorithm, the Residual Kernel Policy Network (ResKPN), demonstrates state-of- the-art art performance, achieving a 30% improvement in episodic rewards across complex environments.
INTRODUCTION
Reproducing Kernel Hilbert Space (RKHS) methods have emerged powerful tools in reinforce- ment learning (RL) due to their ability to model policies nonparametrically, allowing for flexible function approximation and efficient exploration of the solution space (Lever & Stafford, 2015 Pa- ternain et al. 2020). By leveraging kernels, RKHS-based policies can capture complex relationships in high-dimensional observation spaces, leading expressive models that adapt well to diverse vironments (Paternain et al.) 2022). However, despite these advantages, RKHS- policies face significant challenges that limit their practical applicability. A critical issue the high variance in policy gradients inherent to RKHS methods (Smith & Egeland 2024). This excessive variance arises because the RKHS gradient updates do not fully exploit all previously sampled episodes, leading to instability during training and difficulties in converging to optimal policies (Dastider et al.) (022). Additionally, the robustness of RKHS models often compromised due to hyperparameter sensitiv- ity. Different environments may require distinct hyperparameter settings, making it challenging for RKHS policies to maintain consistent performance across varied and complex environments (Liu & 2024). In environments with large action spaces, such the Humanoid environment (Todorov 2012) RKHS methods struggle to scale effectively.
Lian
et al.
"Contribute equally to this work. 'Corrsponding author

---

Published as a conference paper at ICLR 2025
From review of existing approaches, grid search methods are commonly employed for hyper- parameter tuning to align the data distribution with the RKHS kernel (Montesinos López et al,) 2022 Hsu & Lin 2002 Wilson et al. 2016). The requirement for meticulous hyperparameter tun- ing introduces additional computational complexity and limits the adaptability of RKHS policies to new tasks. Regarding variance reduction, various techniques such learning rate search (Le) iet al. 2019). symmetric estimation (Paternain et al., 2020). and policy search (Chen et al.) (016)) proposed stabilize the training process, while the effectiveness of these variance reduction methods is often constrained to specific environments. Alternatively, variance can be mitigated by introducing bias into the estimation process, such as leveraging predefined kernel orthogonal ba- sis (Mazoure et al. 2020) or employing an online clustering approach to aggregate similar kernel orthogonal bases into central representations (Wang & Principe 2021). Despite their promise, the implementation of these methods in high-dimensional spaces poses significant challenges due to computational inefficiencies.
To fully leverage the advantages of RKHS methods and develop stable, robust policies across di- verse environments, we introduce the Residual Kernel Policy Network (ResKPN). Our approach integrates representation learning with RKHS policy models dynamically process observations, enhancing the adaptability and robustness RKHS policies in complex environments. By incor- porating neural network for feature extraction, we adjust the distribution of inputs better align with the chosen kernel, mitigating hyperparameter sensitivity. Furthermore, inspired by the variance reduction capabilities of advantage functions, we introduce residual layer (He al. (016b) that significantly reduces gradient variance within RKHS. This addition stabilizes the training process and enables the discovery of tigh-performing policies with improved robustness. Specifically, the following key contributions are made:
We propose novel RKHS policy learning algorithm that employs a neural network to dynamically represent observations, enhancing the adaptability and robustness RKHS policies across diverse environments by aligning observation distributions with the chosen kernel.
We conduct in-depth analysis of the high variance issue in RKHS policy gradients. Our findings reveal that learning with traditional RKHS policies, particularly in wandibanndiddh kernels, leads significant instability and high variance during training.
We introduce variance reduction technique by designing residual layer for the RKHS policy. Our analysis demonstrates that this approach effectively reduces gradient variance and stabilizes the training process. Combined with representation learning, our ultimate algorithm, ResKPN, achieves superior performance across various challenging environ- ments, including a 30% episodic rewards improvement in the Humanoid environment.
BACKGROUND
In this section, review previous studies on reinforcement learning within RKHS. Additionally, we provide a brief overview of variance reduction techniques in reinforcement learning to facilitate the subsequent introduction of the variance reduction methods designed for RKHS policies. To describe the methodology of the ResKPN algorithm, we also examine prior research on the integration of kernel methods and deep learning techniques.
REINFORCEMENT LEARNING WITHIN RKHS
Reinforcement learning algorithms attempt to learn the optimal Q-function for the cumulative rewards or the optimal policy (Sutton & Barto 2018). This paper concentrates on the latter one, which leads the policy gradient algorithm. The cumulative rewards (Bedi al. 2024) is define as
U UTTw) = = BTap(T:Fw)
Σ
- (st, ai)
where p (T: p(s1) I p(s Tw represents the distribution of the trajectory 7 ((s))ar))) (82, 02). following the policy म., and r(s,, a,) denotes the instant reward of state- action pair (st, a:). 7 is the discounted factor of reward. The policy gradient is to find the gradient
2

---

Published as a conference paper at ICLR 2025
direction to maximize U (TTw) that
Ve U (Tw)
Ba (alm)(als) (s) 1Q log (a)s)]
(I)
D r(a,, st)lai 81 s) is the Q-function and pa (s) represents the marginal density of the state under policy Tw-
where QTH (a, s) Brap(r
Reproducing Kernel Hilbert Space (RKHS) is the vector valued Hilbert Space HK where an
elements K (x,,) )) E H. satisfies the reproducing property (K(z, Ky..)) = (x, y). Despite the policy is modeled by the parameter policy is directly modeled as a function h in RKHS HK. where the updating gradient for is also a function. In detail, the action is chosen from multivariate normal distribution (h)))),
with particular parameterized functions, the stochastic
Th. (n,s(a 7
(h())-o) ---(0)))) (b(())aa)
where the mean value is dependent the function h Hg. The gradient for the RKHS policy is then derived (Paternain et al. 2020 Lever Stafford 2015) as
1 Eu, (() h(s))]
V (Th)
where the derivative with respect to function h uses the Fréchet derivative (Megillivray & Oldenburg, 1990j. To compute the stochastic gradient V₂U (TT). a common approach is use Monte Carlo approximation (Lever & Stafford, 22155) which incurs high computational costs, particularly in complex environments. In (Pontil et al. 2005 Paternain et al. 2020), a pair (s, a) sampled from trajectories to obtain an unbiased estimate of (,UU(,)) The accuracy of this estimate depends on the precision of the Q-function approximation (²*(2), for which actor-critic methods (Dastider & Lin 2022) employed to ensure more stable estimations. In this paper, the estimation of 0 Th is obtained from the one pair estimation in (Cervino et al., 2021)
VuÚ (mA) K )
-) '(x ((ss)) (ak,
where 17 represents the learning rate.
2.2 VARIANCE REDUCTION AND DEEP KERNEL LEARNING
Variance in policy gradient is seen as the main factor influencing the performance of policy gra- dient algorithms (Hafner & Riedmiller 201 Aiming to reduce the variance, a series classic reinforcement learning algorithms are proposed. Designed with the famous actor-critic framework Gronndmn et al. 2012), baseline control variate is proposed for leveraging the vibration in Q- function (Greensmith al. 2004). It introduces baseline function, which is mainly chosen as the state value function ((s) E, (Q(, s)]. to eliminate unnecessary variance introduced by state values, which formulates
V.ū (Tw) A Tw (ax, Sk. V. log (ax(ss))))
where ATT (ak, st) TT (ax, sk) VT= (sk) is known the advantage function. The update size of the policy gradient is further limited in the generalized advantage estimation method (Schulman) iteration optimization algorithm for trust region policy optimiza- et al.) 2016) by introducing Combining the well-performance variance techniques, the proximal tion (Schulman et al.] 2015). policy optimization (PPO) algorithm is proposed (Wu et al. 202T), achieving the overall best perfor- mance, which is viewed as the baseline algorithm in the successive research. The ultimate gradient V U (mw)
in PPO is formulated
(w Tw. (ax(s))))
Tan (au (sn) TT. (ak(sx)
e, (e V log t., (anlsx),
TT TT (ax, sx) min
clip
(4)
where (als) represents the policy in the last iteration, and e the ratio of the clip, limiting the update size of the new policy. Based on this, the discovered policy optimisation (DPO) algorithm (Lu ct al. 2022) is designed to further smooth the training process while encouraging the exploration, ahieving the overall best performance.
3

---

Published as a conference paper at ICLR 2025
Deep Kernel Learning is widely studied with the development of kernel methods and neural net- works. Kemel methods can learn a wide range of conditional distributions and predictive functions conditioned on context sets of arbitrary sizes (Kim et al., 2019). However, their applicability is of- ten constrained by the necessity of designing task-specific kernels. To address this limitation, deep learning techniques are introduced parameterize observations through learnable network layers, giving rise to deep kerell learning (Wilson et al.) (016)), which enhances the scalability and per- formance of the model on complex tasks. For example, (Papamarkou al., 2024) employs deep kernel processes to improve digit classification accuracy, while (Kristiadi et al. (2200) incorporates kernels into ReLU networks for more efficient predictions. Moreover, wide residual networks can be interpreted as performing kernel regression within the associated RKHS (Lai et al. 2023), offering a theoretical explanation for the smoother functions they learn. This smoothness contributes to the networks' superior generalization capabilities (Tirer et al. 2022).
Despite the strong performance of deep kernel learning in typical deep learning tasks, few stud- ies integrate deep learning methods with kernels in RKHS for policy gradient. To the best of our knowledge, this the first paper to incorporate representation learning through neural networks into RKHS policy gradient methods for the rapid identification of locally optimal policies. We present a novel formulation of the RKHS policy gradient and provide a detailed analysis of the algorithm's stability.
PROPOSED METHOD: THE RESIDUAL KERNEL POLICY NETWORK
In this section, we introduce the proposed RKHS policy gradient method, ensuring stable training process across multiple environments. We begin by analyzing the limitations in the current RKHS policy gradient: the insufficient representational capacity and the excessive variance. In order to enhance representational capacity for learning complex environments, we describe representation learning in Section 3.2 This representation learning is integrated with neural network updated using the Proximal Policy Optimization (PPO) algorithm, enabling the adaptive adjustment of obser- vation distributions across different environments. In Section [33 we introduce advantage functions to reduce the variance in the RKHS gradient. Additionally, based on the representation learning, we design residual layer as the baseline function for the RKHS policy to further minimize variance.
3.1 THE LIMITATIONS IN RKHS POLICY GRADIENT
In this section, detailed analysis is conducted to illustrate two main drawbacks in the RKHS policy gradient: the insufficient representational capacity (Wilson et al. 2016) and excessive variance (Le
al. 2019).
Insufficient representational capacity is the primary issue limiting the learning adaptability of kernel methods (Wu & Wang 2009). The hyperparameters of a kernel are highly sensitive to the distribution of imput data, leading to under-fittin fitting in uneven data distributions (Wang et al.) 2022)) straightforward method to select hyperparameters computational complexity limits its use in high dimensional spaces. To adaptively learn the hyper- parameters, they are modeled with linear functions or separation index. In (Wilson et al. 2016). the input of kernel is directly learned by neural networks, which achieves excellent performance in orientation extraction and magnitude recovery tasks. To illustrate the insufficient representational capacity of RKHS policies, we investigate their learning performance under different hyperparam- || v
grid search (Hsu & Lin| 2002), while the
eter settings. Specifically, we employ the Gaussian kernel defined as K K(x, y) exp
to learn from observations, applying the gradient as described in Equation (3). As demonstrated in Figure Ia the RKHS policy fails to update when 2² 0.01 and 5.0. In contrast, episodic returns improve during training for o² = 0.05, 0.1, 0.5, and 1.0. To the best of our knowledge, there is currently general method for performing RKHS policy learning across various environments that offers strong robustness and insensitivity to hyperparameters.
Excessive variance is prevalent issue in policy gradient methods (Greensmith et al. 2004). When applying the RKHS policy gradient, the problem becomes more prominent. We now compare the variance of gradient between the RKHS policy and the linear policy Ta(a)s) defined in Appendix.
4

---

Published as a conference paper at ICLR 2025
Invereed fendalum
Inverted Randulum
e1-3.01
gouzion komel
signel some! Inear po
01-0.05
teerr keme!
Eolooole erd
pFw3.0
Eploace
Epaode
(a)
(b)
Figure I: The insufficient representational capacity and excessive variance in RKHS policy gradient within Inverted Pendulum environment. (a) The learning performance choosing different o². (b)
The variance in training for different kernels compared with linear policy.
A.1 where the corresponding estimated gradient denoted as Voú (TO). We can prove that the variance ratio R = Vara, (V (mxs)) is highly dependent on the choice of kernel K K-,-)).
Vatug (Vo)(no)
Lemma 3.1 A.7 Assuming that QT (a, s) and QTTM (a. s) conditioned on sy are linear functions, Le., QTe (a, 8k) = of a di, TTh (a, 8x) = ei a d2, where Ci, di, C22 d2 are learnable parameters. Then, when the dimension of the action space is 1, the variance ratio R satisfies:
Varaz (v,0 (TT))
(* 8k))
207
Esg.
E.g
Varaz VV (To))
2c E (c20" Sk + d2)
We prove Lemma 3.1 in Appendix. A. Following this, it observed that the variance of the RKHS policy is highly dependent on the chosen kernel, which leads to excessive variance when wide- bandwidth kernel selected. We test three different kernels in the Inverted Pendulum environment, comparing them with linear policy in Figure 1b The results illustrate that the variance of all three kernels significantly higher than that of linear policy, which coincides with the findings Lemma 3.1
3.2
LEARNING REPRESENTATIONS FOR RKHS KERNEL
The neural networks enable the learning model extract multiple features from data using the back- propagation algorithm (LeCun et al. 2015). In order to adjust the data distribution for compatibility with the RKHS kernel, observations are initially input into neural network for feature representa- tion, and the subsequently distribution- distribution-adjused representations are utilized for gradient iteration in the RKHS policy. Without loss of generality, we use (-)()) represent the neural networks with parameters 1 for representation learning, where the RKHS policy in Equation (2) is adapted as
(a = 7. - -
M (+)))()))-a)
(5)
For the sake of clarity, the parameters of the policy defined using the actor-critic scheme similar to the PPO algorithm, the critic V is also modeled as neural networks with parameters where the gradient derived from TD error is formulated as
(0, h, E). We update vha(-)
N
(v (st) R.) (s)))Ra)))
V (T
V
N
where R2 is the target value minimum calculation min
We use the operator T (ak, sK) to represent the T= (acleee)) (e. [xa] - e, + <)) in Equation (4) Based on the chain rule in derivative, the gradient of the neural networks (-) is also derived
Wu et al. leg)
2021
(ot sn] clip
-a== V.ú (mT) A= (ak: 8h T (ax, 8k) 1 V
5

---

Published as a conference paper at ICLR 2025
This formulation resembles the loss function of the PPO algorithm, differing primarily in the inclu- sion of the RKHS function h())()))), where is updated using the RKHS gradient with the only modification being that the state 8n represented by the neural network (sx):
(m²) n (vo(ss)) (a ((
8%).
(6)
Following the gradient estimation, the Kernel Policy Network (KPN) algorithm is shown in Algo- rithm [ It should be noticed that the function only updates for I. times to avoid the explosion update parameters of neural networks for
of the gradient. Meanwhile, the update epoch is times in each training, ensuring sufficient learning of critic and representation networks. With the integration of representation neural networks (aa(), the RKHS policy can update dynamically during each training epoch. Experiment results show that the KPN algorithm achieves further better performance than pure RKHS policy even with tuned hyperparameters. Nevertheless, the excessive variance problem still exists, leading to the extreme reward cliff (Sullivan et al. 2022) in the training process.
Algorithm KPN algorithm
Hyperparameters: Total number of training steps I, mini-batch size N, step times of agent each training T and the update epoch J critic and neural networks eha(-) each train- ing.
1: Initialize critic (s), RKHS function h 0, actor To and replay buffer B.
2: Start with the initial state
3: for l= L..... do
4: Using policy Tw. collect and store transitions (st, Ct, Pt, 86+1) in replay buffer B. do 6: Sample mini-batch {(st, ag; T2, Ta, 81-1) =........ N) from B.
5:
7: Estimate the critic gradient VgTD (ww) and update the parameters 8.
8: Estimate the neural networks vi gradient Voú (m=) and update the parameters 0.
9:
end for
10: Sample a transition (s; az, Tr, 8141) from B.
11: Estimate the RKHS gradient V (Tw) and update the function / = h + VnÚ (TT)
end for
3.3
THE RESIDUAL LAYER FOR VARIANCE REDUCTION
In this section, we introduce two proposed variance reduction methods designed for the KPN algo- rithm, attaining a stable training process.
The advantage function is widely used surrogate for the Q-function in policy gradient methods. is proven in (Weaver & Tao 2001)) that using the advantage function does not introduce bias into the estimation of the policy gradient V. (ww) In this paper, introduce the advantage function decrease the variance for RKHS policy gradient V (T=) in the KPN algorithm. We adopt the gradient estimation in Equation 6 as:
(7) where TT (ak, 91) = QT= (ak, ) - V (sx). We denote the introduced algorithm with advan- tage functions as AdvKPN. Moreover, based on the representation networks in KPN algorithm, the residual layer is specifically designed to diminish the variance in RKHS gradient.
V (ma)) (K()oo EE- 'ax
sk),
The residual layer is initially introduced in resnet structure address the vanishing gradient prob- lem (He et al. 20166). The key idea of the residual layer is to learn the additive residual function F(x) + (x). where fx) is usually chosen as the identity mapping f(x) = x. assumed that when the is the optimal value, F will converge to the zero mapping skip the network layer (He et al., 2016a). In this paper, inspired by the convergence property of the residual layer, we also design fully connected layer for the AdvKPN algorithm to form the final ResKPN algorithm. The complete algorithmic scheme is illustrated Figure 2 with step-by hopp-b--ttep details provided the subsequent text. The policy integrated with the residual layer is expressed follows:
1 (a
π (als)) = z

---

Published as a conference paper at ICLR 2025
Critie networks
((s+)
(aa 522
Reward
5p
Gradient in RKHS (TU
Observat ions
Vo(ss
h(Wo(s))
F
Represen- tations
RKHS Kemel
Gradients calculation
Action
Representation networks
Residual Layer
Pi(ag(s))
V (TTS))
(T
Gradient for representation networks parameters
Figure 2: The scheme of the ResKPN algorithm.
where pt(+)) (-) is the fully connected layer with parameters t. Therefore, it is easily derived that our ultimate RKHS gradient:
VaOA== (m=,)) = n (et)))))(())))),,,))))))) )E h(vb (1b (sx))) (ak. sk). (8)
should be noticed that the gradient of the representation network is also changed to Voû (Tww.)) due to the integration of residual layer:
A (ay, sk) Tx)T== $k 1a h(vo(ss)) (v
(1ls (sx))). (9)
Following this, the ResKPN algorithm obtained by substituting the gradient RKHS and the representation network in Algorithm [ with Equation (9) and Equation (8). The advantage value A (0x, 8k) is computed by combining the state value (sx),, represented by the critic networks, with the reward r(ax, ((0k,,8). 8k). The detailed methodology, along with supplementary explanations for Figure 2 is provided in Appendix B. Meanwhile, we can prove that the variance the RKHS
gradient above follows the following order:
Theorem 3.2
(A.2) Assuming that QT (ax, sk) 2 (s)()) 2 sr, (sb (8k)) when they following the same policy Taass the variance of the RKHS policy gradient gradually decreuses as:
L. [QT (ak, sk)] V (sk) and ak
Varay (Vnū (T=)) 2 Varay (VRŪATo (m=)) 2 Vara, (VRŪATH (TT,))
A.2 The introduction of the residual layer is motivated by the variance reduction properties observed in the advantage function. Specifically, by incorporat- ing a baseline function V* (8t) = E,, xQ== (ax, sk)]. which exhibits greater stability compared to QT=
We prove Theorem 3.2 in Appendix.
the variance during training can be decreased. Similarly, for RKHS policies, stable baseline function can be leveraged to reduce the variance of the RKHS gradient, which is shown in the proof of Theorem 3.2 An ideal candidate for this baseline the residual network (.))). Fur- thermore, the integration of fully connected layer enhances representation learning, and together, these components jointly contribute ResKPN's superior overall performance. To directly il- lustrate the variance reduction effect of this baseline function, an intuitive comparison is provided in Appendix[[ C
Based on the aforementioned adaptation, the RKHS policy retains naramaraet representations suitable for high-dimensional environments and enhances its representational capacity. Meanwhile, the excessive variance in the RKHS policy gradient is mitigated, resulting in more stable learning process. The incorporation of a residual layer introduces an additional learner, leveraging the advan- tages of ensemble learning as well. In order to achieve the overall best performance, the techniques PPO and DPO algorithms are also integrated into all algorithms proposed above to stabilize the training process, where the technique details described in Appendix B.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTAL RESULTS
In this section, various experiments are employed to verify the effectiveness of the representation learning and the variance reduction residual layer. We divide this section into three parts: the environment setting of our experiment, (2) the comparison results including ablation study across multiple environments for the aforementioned algorithms, and (3) the analysis of the effectiveness of the proposed methods.
4.1 EXPERIMENT SETTING
Esioode
Expacc IMAUTT
Mean
Ealacde
Vean AdMEPN (oung
Pan
PRC Sod PAQ
Hean DPO Sod DFO
Essodd Snd Origin Origin Heme!
Sto XPM
St:: AdViin
Std ReckPN
Figure 3: The episodic reward in multiple MuJoCo environments for proposed algorithms.
We evaluate our proposed algorithm on six continuous control tasks from the MuJoCo environments (Toorovv et al., 2012), including the Inverted Pendulum, Hopper, Half Cheetah, Walker2D, Hu- manoid, and Humanoid Standup. These tasks are ordered by increasing complexity, ranging from low to high. The primary objective of these environments is to optimize the performance of simu- lated robotic agents to enhance the forward progress and maintain stability with minimized energy consumption (Batra et al. 2024). The detailed setup of our experiment is described in Appendix
4.2 COMPARISONS
We compare our results with the original RKHS policy (Origin-Kernel), DPO algorithm and the widely used baseline PPO algorithm in the comparison. The episodic reward of the training agents serves the primary metric for assessing the effectiveness of the algorithms. We show the training performance of different methods in Figure
As illustrated in the figure, our ultimate model, ResKPN, achieves the overall best performance. It converges faster than other baseline algorithms in the Inverted Pendulum, Hopper, Humanoid, and Humanoid Standup environments, achieving up to 30% higher episodic rewards compared to the DPO algorithm. In the Walker2D and Half Cheetah environments, ResKPN also attains comparable results, while both the PPO and DPO algorithms struggle to perform well in these settings. The RKHS policy with representation networks takes more time to explore initially, leading to lower episodic returns at the start of training, shown in the Inverted Pendulum, Hopper, and Humanoid environments. However, after sufficient exploration the environment space, the RKHS policy achieves higher episodic rewards due its sampling efficiency.
Considering the ablation results, is significant that the AdvKPN and KPN fail to learn effective policies in complex environments such Humanoid and Humanoid Standup. This reflects how the ResKPN algorithm leverages the advantages of ensemble learning with integrated representation networks. In contrast, the original RKHS method can only achieve similar performance other
8

---

Published as a conference paper at ICLR 2025
Chedth
Hepper
Humanekd
Splaze
Faleze
Eplise
Eplace
Epkize
Fpkidd
Std KPN (ou's)
Stu AdvEP lours
Sid RcsOMI lourt
Std PP
560 DPO
Figure 4: The standard deviation in MuJoCo environments for proposed algorithms.
algorithms in the simpler Inverted Pendulum environment, while fails to learn effectively more complex scenarios.
We also present the training variance in Figure We observe that when algorithms succeed in learning the local optimal policy in the Inverted Pendulum environment, the ResKPN algorithm exhibits the most stable training process overall. After integrating the advantage functions, the variance in AdvKPN decreases significantly compared to the KPN algorithm, which exhibits the highest variance similar to that the PPO algorithm. For the insufficient learning observed in KPN and AdvKPN algorithms within complex environments, their variances in these situations are abnormally lower than those of algorithms that successfully learn a good policy, indicated by dashed lines in the figure. When considering the learned policies, it is observed that the ResKPN algorithm achieves relatively low variance, except in the Humanoid environment. More experiment comparisons shown Appendix D
Hopper
0'-04
(Pw07
o1-1.
10-00k
(a)
(b)
Figure 5: The comparison of distributions and the learning performance after adjusting hyperpa- rameters (a) The boxplot demonstrates the distribution differences of the observations (obs) and representations (rep) different dimensions. (b) Comparison of episodic rewards for the RKHS policy with optimized hyperparameters versus alternative settings. The performance of the KPN algorithm is also illustrated.
4.3 ANALYSE FOR EFFECTIVENESS
In this section, we conduct a comprehensive evaluation addressing two key questions: (1) How effectively does representation learning adjust the observations of the original environment to align with the RKHS kernel? (2) Does the integration of advantage functions and the residual layer lead reduction in the variance of the RKHS gradient?

---

To answer (2), utilize classic variance estimation formula, with detailed methodology provided The variance of the RKHS gradient across various environments is illustrated in Published as a conference paper at ICLR 2025
To answer (1), we consider the Gaussian kernel ((x,y) y) exp
at |-=yyl o²
where the hyperpa-
rameter can be viewed scaling of the variance of the observation distribution. The setting of the hyperparameter can imitate the scaling of observation embedding learned by the representation learning appaorch. In detail, let mean (varobs) meannvarross) denote the mean variance of the original observa- tions, and mean vaIrep) represent the mean variance of the representations. Then the controller
(varr(varre))
is set as o² meann(vas (vaFrep) The comparison of the distributions between observations and repre-
(vaooa)a
sentations illustrated in Figure 5a is shown that the representation networks tend to reduce the variance in the distribution of observations while maintaining relatively unchanged mean. After tuning the hyperparameter, we observe significant improvement in the learning performance the RKHS policy, shown in Figure 5b) The results indicate that the original RKHS policy achieves the best overall performance compared to other hyperparameter settings, although it still underperforms relative to the KPN, which dynamically processes observations through neural networks.
Eo sodde
Eplisde
Colodd
Ephodd
Figure 6: The estimated variance in MuJoCo environments for proposed algorithms.
in Appendix B.2
Figure 6
The results indicate that the introduction of advantage functions significantly reduces the variance in KPN. Nevertheless, ephemeral fluctuations still observed, particularly in the Inverted Pendulum, Hopper, and Humanoid Standup environments. The ResKPN further mitigates these fluctuations during training, demonstrating rapid decrease in gradient variance the beginning the training process, notably in the Humanoid and Walker2D environments.
CONCLUSION
In this paper, we examine the limitations RKHS policies, focusing on how inadequate representa- tion learning and excessive variance cause unstable training and low robustness across environments. To address these issues, we introduce the ResKPN model, which capitalizes on the high sample efficiency of RKHS policies. Our theoretical analysis of ResKPN is validated, confirming the effec- tiveness of the proposed variance reduction methods. Comprehensive experiments demonstrate that algorithm outperforms baseline methods.
However, several challenges remain. Unlike gradient updates neural networks, which use mini- batches for averaging gradients, RKHS gradient updates cannot currently leverage minibatches due to computational complexity. Additionally, the kernel's separate learning high- nigh-dimensional spaces may hinder performance in multi-agent settings. Future research will focus kernel embed- ding techniques to enable batch RKHS gradient updates and on approximating kernel computations to improve sample efficiency and reduce variance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to acknowledge the support from the National Key R&D Program of China under Grant No. 2022ZD0160504 and the Shenzhen Key Laboratory of Ubiquitous Data Enabling under Grant No. ZDSYS9022062777440015
REFERENCES
Sumeet Batra, Bryon Tjanaka, Matthew Christopher Fontaine, Aleksei Petrenko, Stefanos Niko- laidis, and Gaurav S. Sukhatme. Proximal policy gradient arborescence for quality diversity reinforcement learning. In International Conference on Learning Representations, 2024.
Amrit Singh Bedi, Anjaly Parayil, Junyu Zhang, Mengdi Wang, and Alec Koppel. On the sample complexity and metastability of heavy-tailed policy search in continuous control. Journal of Machine Learning Research, 25(39):1-58, 2024.
Juan Cervino, Juan Andrés Bazerque, Miguel Calvo-Fullad Fullana, and Alejandro Ribeiro. Multi-task reinforcement learning in reproducing kernel hilbert spaces via cross-learning. IEEE Transactions Signal Processing, 69:5947-5962, 2021.
Zhitang Chen, Pascal Poupart, and Yanhui Geng. Online relative entropy policy search using repro- ducing kernel Hilbert space embeddings. In Artificial Intelligence and Statistics, pp. 573-581, 2016.
Apan Dastider and Mingjie Lin. Non-parametric stochastic policy gradient with strategic retreat for non non-stationary environment. In 2022 IEEE 18th International Conference on Automation Science and Engineering, pp. 1377-1384, 2022.
Apan Dastider, Sayyed Jaffar Ali Raza, and Mingjie Lin. Learning adaptive control in dynamic environments using reproducing kernel priors with bayesian policy gradients. In Proceedings of the 37th ACM/SIGAPP Symposium Applied Computing, pp. 748-757, 2022.
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. The Journal of Machine Learning Reseurch, 5:1471
2004.
Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, part C, 42(6): 1216))29111309 291-1307, 2012.
Roland Hafner and Martin Riedmiller. Reinforcement learning in feedback control: Challenges and benchmarks from technical process control. Machine learning. 84:137-169, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision 2016: 14th European Conference 2016, Proceedings, Part IV 14, pp. 630-645, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016b.
Chi Hsu and Chih-Jen Lin. A comparison of methods for multiclass support vector machines. IEEE transactions on Neural Networks, 13(2):415-425, 2002.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv: arXiv:/901.055 /90100561, 2019.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just bit, fixes overconfidence relu networks. In International conference on machine learning, pp. 5436- 5446, 2020.
11

---

Published as a conference paper at ICLR 2025
Jianfa Lai, Zixiong Yu, Songtao Tian, and Qian Lin. Generalization ability of wide residual net- works. arXiv preprint arXiv: 18506, 2023.
Tuyen Pham Le, Vien Anh Ngo, P Marlith Jaramillo, and TaeChoong Chung. Importance sampling policy gradient algorithms in reproducing kernel hilbert space. Artificial Intelligence Review, 52: 2039-2059, 2019. 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 4444 2015.
Guy Lever and Ronnie Stafford. Modelling policies MDPS in Reproducing Kernel Hilbert Space. In Artificial intelligence and statistics, pp. 590-598, 2015.
Jiamin Liu and Heng Lian. Kemel-based decentralized policy evaluation for reinforcement learning.
IEEE Transactions on Neural Networks and Learning Systems, pp. 1-10, 2024.
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35:16455- 16468, 2022.
Bogdan Mazoure, Thang Doan, Tianyu Li, Vladimir Makarenkov, Joelle Pineau, Doina Precup, and Guillaume Rabusseau. Representation of reinforcement learning policies in reproducing kernel hilbert spaces. arXiv preprini arXiv:2002.02888 2020.
Peter R Mcgillivray and DW Oldenburg. Methods for calculating fréchee derivatives and sensitivities for the non- non-linear inverse problem: A comparative study 1. Geophysical prospecting, 38(5):499- 524, 1990.
Osval Antonio Montesinos López, Abelardo Montesinos López, and Jose Crossa. Reproducing kemel Hilbert spaces regression and classification methods. In Multivariate Statistical Machine Learning Methods for Genomic Prediction, pp. 251-336. 2022.
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, and Laurence et Aitchison. Posi- tion: Bayesian deep learning is needed in the age of large-scale AI. In Proceedings of the 41st International Conference on Machine Learning, pp. 39556-39586. 2024.
Santiago Paternain, Juan Andrés Bazerque, Austin Small, and Alejandro Ribeiro. Stochastic policy gradient ascent in Reproducing Kernel Hilbert Spaces. IEEE Transactions Automatic Control, 66(8):3429-3444 2020.
Santiago Paternain, Juan Andrés Bazerque, and Alejandro Ribeiro. Policy gradient for continuing tasks in discounted markov decision processes. IEEE Transactions on Automatic Control, 67(9): 4467-4482, 2022.
Massimiliano Pontil, Yiming Ying, and Ding-Xuan Zhou. Error analysis for online gradient descent algorithms in reproducing kernel hilbert spaces. Proceedings of Technical Report, pp. 1-20, 2005.
Xin Qian and Diego Klabjan. The impact of the mini-batch size on the variance gradients in stochastic gradient descent. arXiv preprint arXiv:2004.13146, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Con- ference on Learning Representations, 2016.
Torbjørn Smith and Olav Egeland. Learning of Hamiltonian dynamics with Reproducing Kernel Hilbert Spaces. In 2024 European Control Conference, pp. 3876-3883, 2024.
Ryan Sullivan. Jordan K Terry, Benjamin Black, and John P Dickerson. Cliff diving: Exploring reward surfaces in reinforcement learning environments. In International Conference on Machine 2022.
Learning, pp.
12

---

Published as a conference paper at ICLR 2025
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, 2018.
Tom Tirer, Joan Bruna, and Raja Giryes. Kemel-based smoothness analysis of residual networks. In Mathematical and Scientific Machine Learning, pp. 921-954. PMLR, 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Hailun Wang, Daxing Xu, and Alexander Martinez. Parameter selection method for support vector machine based on adaptive fusion of multiple kernel functions and its application in fault diagno- sis. Neural Computing and Applications, 32(1):188 19118933,, 2020.
Yiwen Wang and Jose C Principe. Reinforcement learning in reproducing kernel hilbert spaces. IEEE Signal Processing Magazine, 38(4):34-45, 2021.
Lex Weaver and Nigel Tao. The optimal reward baseline for gradient- reinforcement learn- ing. In Proceedings of the Seventeenth Conference Uncertainty in Artificial Intelligence, pp-
538-545, 2001.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378, 2016.
Kuo-Ping Wu and Sheng-De Wang. Choosing the kernel parameters for support vector machines by the inter-cluster distance in the feature space. Pattern Recognition, 42(5):710-717, 2009.
Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, haiyin piao, and Hankz Hankui Zhuo Coordinated proximal policy optimization. In Advances in Neural Information Processing Systems, pp. 26437- 26448, 2021.
Tingting Zhao, Hirotaka Hachiya, Gang Niu, and Masashi Sugiyama. Analysis and improvement of policy gradient estimation. Advances in Neural Information Processing Systems, 24:262-270, 2011.
13

---

Published as a conference paper at ICLR 2025
A VARIANCE ANALYSE
A.1 THE VARIANCE RATIO
In this section, provide proof to Lemma
3.1
which indicates that the variance ratio R satisfies:
Varaz (Vrú (mT))
K² (s sk)
2c
Ees Vara, (VoU (To))
2 E.g
20 + E-1(c20 8k + +2)2
First, we define the linear policy.
Definition A.1 (Linear policy) The linear policy ne(als) is modeled as 1 - -2 =-==)) x -0T s-a)
To Ta(a)s)
Z
where 0 is the trainable parameter for policy. Linear policy embeds state in a linear manner as 0¹..
Comparing with RKHS policy in Equation P the only difference of lienar policy is the modeling of the mean vector 0 and h(s). Therefore, the estimated gradient of linear policy V (Ta) is derived (Zhao et al. 2011) as
VoÙ (TO) - 11 8 - (ax 0 88 @TH (ax, 8k).
Then, apply this definition in the variance ratio R. Considering the expectation of variance ratio conditioned on 0, combine with Eq, 3 we can have the variance ratio R. as
R Vara, (V,ū (TT)) n2² Vara. (K(sk -))-))--- (ax h h(s))) (ak: sk))
Varaz (VoÚ (TT))
17² Varaz (skE (ax 0T (ax, 8k))
(K(sk, ) K K(sk, Varaz (Ks,, -)- (ax h(sx))) (ak,,) Sk), of Varai (2' (ax: 0T sR) QTTM (kk:
K(sk, sk) Vara. [K(s, ) (ax - ((*x))(=TH (ak. sk))
t Vara, (E-1 (ak- - 0 sx) QT (ax: 8k)) where the last step uses the reproducing kernel property that Vx. U, (K(r, ") K(v.)) -) = K(r,y).
Considering the Assumption A.1 that Q (ax, s&) and Q (ok, sk) are linear functions conditioned on 8k, i.e., QT* (a, sk) cf d1, QTT (a, su) 0, a + d2, we can have that
R K(sk, sk) Varaz (E- (ax - h(ss))(crax + d1)) $t Vara, (Σ -1 (a+ 0 54) (cea+ +d2)
We first consider the variance in the numerator part. Denote y Σ 3 (ox-h(ss)),, then have that y (O, 1), then we can write Var (ax h(sx))(e1aa + dz)) Var E (Σ- by((cax d:))- Further, we have
Var
Σ plcian + di))
3 yyccaa di)) (15 b dii))
E8
(u(c y h(sx)) + di)) E i h(sw) + di)))
Σ
'E
Σー'
21 2g (3 + d Bby
+ (aM))
N (c)
(cth(sw) + +
1+ (ch((s)) +
3e + (a + +d)
= 20 + (c +d))²
14

---

Published as a conference paper at ICLR 2025
Using the similar method, the variance Varai (Voù (me)) is derived as 2c + E- -02(000" Sy + dz)?. Therefore, the variance ratio is
R K (*K, 8k) In this case,
2ef + E- (c1)(8k)) + +1)²
K2 (SK, 8k)
2e
2c + Σ- (c23k + +d2)2
2
26 + Σー (c20
8k + d2)
Varaz (vnū (TTA))
K2 (8k, 8k)
207
Eys
2 Esg
Varai Vol' (ST))
20 + Σ E (cg8)
8k + d2)
A.2 THE VARIANCE IN DIFFERENT RKHS GRADIENT ALGORITHMS
In this section, we prove the Theorem 3.2
Ad Advantage Function Effect First, we prove that Varag (Vrū (m=)) 2 Vares (VrŪATo (m=)).
For simplification of notations, we denote Vara, (vr (m=)) = Ban [(T> Eag, [T)]) with
A = VRÜATO (m V Vlog क (ak|sk)
and Varaz (V (mw)) Eag Q E.g [T) with
Te - V (TT( = V log (ag (ax+ 8k).
Similarly, we denote that Iv Ea, [Tv) with V log T (aulss)) (sk). Since AT QTn (ak, 8k) - VR (8*), we can derive that:
A Eog [TA]
=V log To (an|5k) A = - E [V log To (ax sk ) A
=V log Tw (ax|sk) (Q (ak, sk) - V (sk)) - Ea V log To (aklsk) (Q (ax, sk) - V (sx)))
= -(V log (cx 10= (ak: 8t) - Ea, Vlog T- (ax 8/: Q (ax, st)]
(V T (axl 8k) V* (sk) - -aee . log no (au|sa) V= (s:))
= (TQ Eu, [T) (Tv Eug [Tv]).
Therefore, we have that
Vara, UT (Tm))
-Baa (T' - Eus [2
-Ba: ((To= Eac [Tq] - (I'v -Ea [Kvl))²
-Ba, (To - Rn, [T2])²² 2 (TQ Emn (T2]) (Tv - Es, [Tv)) + (Tv - -Bo, [v]])²]
Vara, (V,ū (T) + Faz [2 Fag [T)]) (Tv) + (Tv))
Vara, Ü (T))) )) + Eag (Iv)²- -21''I')
(VV log The (ax|s*), V Vlo To (arls;)) 2V Em, V log - (aw s ) (ak, 8k), V log n- (aw[s*))] ))},
Varaa Ü ú (Ta) )) + (V=T)²
(3 )
where we use the Exs V log To (s ) V== [sx]] = 0 property for V log Tww (a* WWaver & Tao 2001). Therefore, is easily derived that
r8 - r2- = (V*=) )
Te (ax lsx). V log T= (axl [x)]+
2 Es. (V log T= (at|s) (at, 8t), V log कं (ax18x)))
20,
where the last inequality uses the assumption ("=(au+ 84) 2 F (Qx= sn)l = _ Therefore, it is shown that Var., (V,ū (m=)) > Vara, (VrU Arm (m=))
15

---

Published as a conference paper at ICLR 2025
Residual Effect Now consider the variance of VnUAAO (Twze)- Similarly, we write Eas [TTr. Ea, [T wheere Th VnÜna (Tm,)
Vara, [V U Arw (Tm,))
((sk,) E (as h (Va (sw)
He, (v. (5x)))A4= ( sk). Additionally, we denote that Varas (VAUAr (Tw)) = Eax [[T Bos [Ta]2]] with Ta = K(sk,)) (tho (8:))) = (ag, 8:) and Fµ = K(sk. -) (M, (eho (sx))) (ak, sx).
Like the derivations
in the advantage function effect, it is easily observed that
J. Eag [[ (I's Eos (T)) (T -Ba [Tl))
Based on the definition, we have that
Varai (VrūATo (+)
-Bog [[TA,. Eag [T ['n,,)))²
-Box ( -Bax [T)]) - (Tp= Eug (T))) )
Vara (VnUSTo (m=)) + Bas - - Eug [T (T₂) + (Tw)²
-1 + Bos [[-2K (sk, :) --- (ax h (bb (sx))) A T (ak, sk), K (skr M uo (vo (Sk)) T (ax, sk))
+(K (sk; E E-7mm (vo (sk)) A (ax, sk): K (sks :) Σ- M4 (ve (sk)) AT sk)]]
where we use the property Ea, [ (sk. :) E 1ー11, Mr (tha (8k)) A ( (kk, = 0 due to Exg = sk)] = O, then we can derive that
Varag (VAUA-= (m)) Vara. (VRUA-= (T=,))
-En (2K (sk, -) - (ax (v's (sk))) A =- (ak, sk), K (5k: Σ-1 2-11. pt (wo (sk)) A (T= sk))
(K (8k: .) -ー uu, (who (8x)) A (aR, 5k K (sk, .) --1 μr (va (8x)) A (ak, 8R))
> Eay [KK (sk, -) Σ Pr. (vh (sw)) ) A - (ak: 8 K (sK: .) Σ p4. (v/ (sk)) A (ak: 81))
(K (Sk, -) E 1 p. (wbo (sx) A (az, sx), K (Sk: E 3, (woo (sx)) A (ax: sk]]]
20
Vara, (V.ū AN= (TT)) Vara, U..A (m=,)), where the last inequality uses the assumption as - ((va((K)) N
Therefore
we can attain that Varag (V,Ū (Tm))
I
24 (1o (sk)).
B DETAILS IN EXPERIMENT
The experimental details are shown in this section, including a scheme figure of the ResKPN al- gorithm, the techniques in PPO and DPO integrated with the proposed algorithms, the function to estimate the variance of the RKHS gradient, and the hyperparameters in the experiments.
Without loss of generality, we adopt the Gaussian kernel for the experiments due its well- established properties in capturing smooth and continuous relationships in high-dimensional spaces. To leverage GPU acceleration for training and simulation, we utilize the Brax simulator, which is developed using Jax (Freeman et al. (021). The experiments are conducted on a cluster server, with each experiment utilizing an NVIDIA RTX A6000 GPU and 32 cores of an Intel(R) Xeon(R) Gold 5218 CPU running at 2.30 GHz.
B. THE IMPLEMENTATION DETAILS IN RESKPN
The methodology scheme of ResKPN is shown in Figure where V,û (Tw.) represents the gradient in the residual layer n4 (wo (-)), and the notation FC means the Fully Connected layer, which composes the representation networks. The observations are initially input into the repre- sentation networks for distribution adjustment. Both RKHS function h())(s*)) and residual layer
16

---

Published as a conference paper at ICLR 2025
Critie networks
((s+)
(aa 522
Reward
5p
Gradient RKHS (T
Observat ions
Vo(ss
h(Wo(s))
Represen- tations
RKHS Kemel
Gradients calculation
Action
Representation networks
Residual Layer
Pi(agts))
V (TTO))
(T
Gradient for representation networks parameters
Figure 7: The scheme of the ResKPN algorithm.
Me (vo (s+))) uses the representations for action choosing. Additionally, the covariance matrix --1 is parameterized as the network parameters, which are updated through the backpropagation. The main difference between ResKPN and AdvKPN algorithm the usage of the residual layer, stabi- lizing the training process.
To leverage the classic variance reduction techniques, We employ the techniques of Proximal Policy Optimization (PPO) algorithms in the KPN, AdvKPN, and ResKPN algorithms to reduce variance. Specifically, the optimization objective PPO utilizes Trust Region Policy Optimization (TRPO) (Schulman et al., (215)), which we adapt to optimize our gradient. The optimization objective in TRPO given by
(ax 88)) As
maximize
To (as 88))
subject to
E, [KL [Toael sk), To [x)]]] s 6,
where fol represents the parameters before the update. Inspired by this formulation, since
VTTLUR 8k) Th (ak 8k) That (ak Ske) Thau (ak $k:)
V log (ax sk);
(a (sx)
adapt the RKHS policy gradient by scaling the original gradient with the policy ratio Additionally, we incorporate the clipping technique used PPO, where the policy ratio is clipped follows:
Rh Fha )
(a 8b) Thaa
sk) (ax sk) Tha (as (sh)
Th (ax
e,, e, e,,+)))
min
clip
The threshold value is set to 0.2 in the PPO algorithm. In contrast, we set to 1 in the RKHS policy to encourage more exploration, leveraging the high sample efficiency of the RKHS gradient. To balance the exploration-exxloittiio trade-off, we introduce the drift function D from Discov- ered Policy Optimization (DPO). The main difference between DPO and the PPO algorithm the inclusion of drift function that depends the advantage function values, allowing the adjustment of hyperparameters to control the degree of exploration and exploitation in the policy. The drift function is calculated
17

---

Published as a conference paper at ICLR 2025
(aka [sk)
==
Th [akak 18k)
Th
(ak
[8k)
max
0,
- 1 T arp tanh
20,
Tha (ak 8x)
Or
D
Tn(ax
8k)
A
0, Th (ak
sk) A= 3D tanh
Thau (ak 5k)
max
otherwise,
Thau (ak. 9k)
3D
where the hyperparameters and ßD balance the exploration and exploitation of the policy, The drift function is integrated into the RKHS gradient as:
VnOAA= (Tai)) = (K((0(5)))))) (ax- h(vo(ss)) (.(o((s))) (AT (ax, sk) D).
Table I: Comparison of Hyperparameters among ResKPN, PPO, and DPO.
Parameter
ResKPN
PPO
DPO
Optimization
Learning Rate
3
10"4
3 10"4
3 10
Learning Rate for RKHS
1 x 10-1
N/A
N/A
Adam for RKHS
0.9
N/A
N/A
Max Gradient Norm
0.5
0.5
0.5
Environment
Number of Environments in Parallel
2048
2048
2048
Training Schedule
Number of Steps per Update
10
10
10
Total Timesteps
5 107
5 107
x 107
Number of Minibatches
32
32
32
Algorithm Specific
GAE Lambda
0.95
0.95
0.95
Clip Epsilon
0.2
0.2
0.2
Clip Epsilon for RKHS
N/A
N/A
DPO Alpha
2.0
N/A
2.0
DPO Beta
0.6
N/A
0.6
DPO Alpha RKHS
1
N/A
N/A
DPO Beta RKHS
0.6
N/A
N/A
Model Architecture
Feature Dimension
256
N/A
N/A
Activation Function
tanh
tanh
tanh
B.2 THE ESTIMATION OF VARIANCE IN THE RKHS GRADIENT
To estimate the variance of RKHS gradient in the experiment, we conduct 9 parallel experiments. It is observed that the RKHS gradient can be denoted a (st,)), where a represents the coeffi- cient including the learning rate, the derivative of log policy and the Q-function value or advantage function, and the kerne is determined by the observations 8t. Therefore, we derive the following
18

---

Published as a conference paper at ICLR 2025
variance estimation equation:
1 Var (VoU (Ti))
Σ
(or())),))
1
Σ
0 K (
1
Σ
a (si; ))
g
g
g i=1
t=1
i=1
1
8t)
201
s(s
04 K
(8),
g
g²
(10)
where we use the reproducing property of the RKHS kernel.
B.3 HYPERPARAMETERS IN THE EXPERIMENT
The hyperparameters for PPO, DPO and ResKPN are shown in Table 1 The Max Gradient Norm only limits the gradient update of the representation networks in ResKPN. We run environments in parallel based on the JAX, which supports the acceleration of simulation in GPUs. All experiments run for 5 x 10" steps to compare the learning speed and convergence performance.
C MORE EXPLANATIONS FOR THE VARIANCE REDUCTION EFFECT IN THE RESIDUAL LAYER
One of the main reasons that the advantage function (a* 8k) Q== (ak, 88) - V (sk) can reduce the variance of the gradient is the natural properties that V (sk) Eag 1QT (ax sk)], which indicates that the state value function V* (sx) is potentially stable than the Q-function Q (ag,&s, (Greensmith et al. 2004). We compare these three values in the Hopper environment, which is shown in Figure 8 It can be seen that after subtracting the state value function V (sk), the advantage function shows more stability and low-value curve, which avoids the overestimation (Greensmith al. 2004).
Hopper
Q"u(ax. Sk)
20
(na(s))
ATr(ar.
Value
1000
1350
2005
Episode
Figure 8: The variance reduction by introducing the advantage function in Hopper environment.
Inspired by the baseline function in the advantage framework, we introduce the residual layer (+(()) as a "baseline function" within the RKHS policy. This addition significantly enhances training stability compared to the RKHS function h(-). To demonstrate the variance reduction achieved by the residual layer, compare the values of a; /t( h(()(s))) nu (voo a₃ h((o(ss)), and p. (x)) in Figures and 10 across the Inverted Pendulum and Hopper environments. These comparisons reveal that introducing the residual layer a "baseline function" stabilizes the product term (az - (()))()))) ps. (wh (sx))) relative to the original RKHS term (az- h()o(s=))). This stabilization directly contributes to the variance reduction in the RKHS gradient:
V AT ( = 7 (wo(su),
- -
((bb(s))) - 4.(2 (su))) (aR, 81).
This also provides additional evidence supporting the correctness of Theorem 3.2
19

---

Published as a conference paper at ICLR 2025
Inveried Pendulum
Inverted Pendulum
Maye Alwiiall
1000e
Vilue
Eplooda
Episoda
(a)
(b)
Figure 9: Variance reduction achieved by introducing the residual layer (.(-) in the Inverted Pen- dulum environment. (a) Comparison of ag (x))) and po (vo(s))) (b) Comparison of
ag (s)) and ax h(tbo(s)) (.((bo(s)))
Maula Alwisall
1080e
Episade
Uplonde
(a)
(b)
Figure 10: Variance reduction achieved by introducing the residual layer (-)-) in the Hopper envi- ronment. (a) Comparison of a hh1o and (*())(s))). (b) Comparison of ag h(vb(ss))
and aІ h())(sk)) (s)))
D MORE EXPERIMENT COMPARISONS
D.1 MORE VARIANCE REDUCTION METHODS COMPARISON
In this section, we delve into variance reduction techniques from related literature and present an experiment the minibatch method to highlight the distinctions and advantages of our approach compared to prior research.
Table 2: Variance Reduction Methods for RKHS Policies.
Variance Method
Reduction
Paper
Tested Environment
State
Action
Learning rate search / Ker- nel matching pursuit
Le et al.
2019
Quadrotor Navigation
13
3
Paternain
Self-Charging Surveillance Robot
Symmetric estimation
3
2
et al. (020)
Mazoure et al.
Predefined kernel orthogo- nal basis
Pendulum
3
1
2020)
Nonhuman
primate per-
& Principe 2021)
Wang
Kernel orthogonal bases clustering
forming avoidance task
obstacle-
1
Table summarizes various variance reduction methods for RKHS-based reinforcement learning policies, highlighting their tested environments, techniques, and applicable state-action dimensions. These approaches. while effective in specific scenarios, face significant challenges when applied to high-dimensional environments like MuloCo. For instance, the model-based based method in (Paternain al. 2020) reduces gradient variance by averaging gradients of symmetric transitions, strategy
20

---

Published as a conference paper at ICLR 2025
that is effective in simple tasks such self-charging surveillance robots but struggles in complex settings where symmetric transitions are difficult to identify. Similarly, (Le et al. 2019) introduces kereel matching pursuit method that reduces variance through gradient regression, yet it suffers from computational instability in tasks with large state-action spaces, such as those found in MuloCo environments.
The method in (Mazoure et al. 2020) proposes truncating RKHS embeddings to represent policies using kernel orthogonal bases, theoretically reducing Q-function variance. However, its reliance on partitioning the state space into fixed bins renders it impractical for high-dimensional environ- ments. For example, the Hopper environment with 17-dimensional state space, even simple binary division across dimensions leads to an exponential number of kernel bases, making the ap- proach computationally infeasible. Similarly, (Wang Principe, 2021 employs an online clustering method to aggregate kernel orthogonal bases, reducing computational costs and variance in neural signal processing tasks. While this technique offers valuable insights, its focus on Q-function learn- ing rather than policy gradient optimization, coupled with the bias introduced by clustering, limits its applicability to RKHS policy frameworks in high-dimensional settings. These methods collec- tively underscore the challenges of applying variance reduction techniques effectively in complex environments like MuJoCo, where computational efficiency and scalability remain critical obstacles.
Although variance reduction techniques have been specifically designed for RKHS gradients, mini- batch gradient computation (Qian & Klabjan, 2020) is widely adopted in deep learning and many machine learning tasks due to its effectiveness in stabilizing gradient updates. However, applying minibatch gradients directly to RKHS methods introduces significant computational challenges. The mean of RKHS gradients must be explicitly expressed as Σ; Oi (Ks,)) 1), where represents the minibatch size. This results in quadratic increase in computational complexity as n grows, making minibatch gradients computationally prohibitive for RKHS based methods.
To evaluate the impact of minibatch size on computational cost, we conducted experiments with varying minibatch sizes in two environments, and the results are summarized in Table 3
Table 3: Training Time for Different Minibatch Sizes Across Environments (in minutes).
Training Time (min.)/ Minibatch Size
Environment
n=
n
= 2
n =
n.
= 4
n = 5
Half Cheetah
13.19
61.25
140.74
199.49 169.03
205.07 202.61
Hopper
10.83
58.91
123.12
The results demonstrate that training time increases quadratically with minibatch size in both the Hopper and Half Cheetah environments. For example, using a minibatch size of n. 5 requires over three additional hours of computation compared to n = 1. These findings underscore the computa- tional infeasibility of employing minibatch gradients in RKHS bbssed methods for high-dimensional environments. Despite the significant computational cost, using minibatches remains important due their ability to reduce training variance. We conducted experiments with a minibatch size of (as larger sizes are prohibitively slow) to evaluate training performance and variance, as shown in Figures| [T and [2 These figures indicate that the training performance with n 2 minibatches shows minimal differences compared to the original approach. However, incorporating an addi- tional transition per training period effectively reduces the variance of both AdvKPN and ResKPN, highlighting the potential of the minibatch approach.
If the computational cost associated with minibatches can be addressed, we believe that this design has the potential to further reduce variance in RKHS policy methods, offering promising direction for future research.
D.2 EXPERIMENTS ON OTHER ENVIRONMENTS
In this section, extend the experimental evaluation of our proposed algorithms to two additional environments: Pusher and Reacher. The episodic rewards obtained during training are presented in Figure 13
21

---

Published as a conference paper at ICLR 2025
Hepwer
Epid
Mean ResKPN (ours)
Mean AdvKPN AdvICN
Mean PPO Miniberch Std FPO Minibatch
Kean ResKPN Minibechh2 ReskPN Min betbetcc2
ResKPN
Figure 11: The episodic reward Hopper environment for minibatch size n
Hopper
332
Std
372
232
Episade
(ours)
AdVKPN-miinbbtchh2
AdvKPN
ResKPN Minibatch2
Figure 12: The standard deviation in Hopper environment for minibatch size 2.
Iourt
Oplece
Ieen AdvERN (wund
Nean ReccpN (uung)
Mean Sid
Mean
Origin
Figure 13: The episodic reward Pusher and Reacher environments for proposed algorithms.
22

---

Published as a conference paper at ICLR 2025
It is illustrated in the Figure 13 that the ResKPN consistently achieves the best performance in the Pusher environment. In the Reacher environment, all algorithms except KPN and Origin- Kernel converge to the optimal reward. This demonstrates the robustness of ResKPN, particularly in com- plex environments requiring precise control. To further analyze stability during training, the standard deviations of the episodic rewards for the proposed algorithms shown in Figure 14
From Fig-
Now
Fjjasd
Std KFW (SUS
Sic Adv rours)
Std RastPP rours]
Std PPO
Sid DPO
Figure 14: The standard deviation in Pusher and Reacher environments for proposed algorithms.
ure 14 is evident that ResKPN, DPO, and AdvKPN achieve the smallest overall variance in the Pusher environment, indicating stable learning dynamics. In the Reacher environment, while all algorithms (except KPN) converge, the PPO algorithm exhibits significant oscillations the final phase of training. In contrast, algorithms such as ResKPN maintain high stability throughout train- ing. These supplementary results further demonstrate the efficacy and robustness of the proposed ResKPN algorithm.
D.3 OTHER ABLATION EXPERIMENTS
In this section, we investigate whether the integration of residual layer can enhance the perfor- mance or reduce the training variance PPO and DPO algorithms. The episodic rewards during training across multiple MuloCo environments are illustrated in Figure
Talicte
Calonde
Aeturr
Sploode
Eplosse
Eclodd
9rd Ad4i9%
Sed ReelOh
Srd OPO
Szs 990
690-me
Figure 15: The episodic reward in multiple MuloCo environments for additional baseline algo- rithms.
The results show that the PPO-res algorithm exhibits improvements in Walker2D and Hopper, main- tains similar performance Half Cheetah and Humanoid Standup. but experiences decline in performance in Inverted Pendulum and Humanoid. For DPO-res, its performance remains largely
23

---

Published as a conference paper at ICLR 2025
unchanged, with slight decrease observed in Half Cheetah. A plausible explanation for this trend that the existing neural network structures PPO and DPO may already provide sufficiently well-learned representations, rendering the addition of residual layer less impactful. The residual layer might only offer benefits in specific cases, such as addressing issues of overfitting or gradient dispersion. However, it does function does in ResKPN, to further enhance performance.
additional representation learning component, it
The variance reduction effect of integrating the residual layer with AdvKPN is another significant consideration. The standard deviations during training for the additional baseline algorithms are presented in Figure 16
Epease Percsken
Frelaee Waterrd
Hareroid
ADVKK
ResKIN
PRO
PRO
DFOves
Figure 16: The standard deviation in multiple MuloCo environments for additional baseline algo- rithms.
As shown in Figure T6 the addition of the residual layer has minimal impact on training variance across all tested environments. Both PPO-res and DPO-res maintain similar variance levels their respective baselines, PPO and DPO. This can be attributed to the similar gradient calculation mech- anisms used in PPO, DPO, and their residual-layer variants. Adding a residual layer directly into the neural network architecture may not effectively reduce variance in these cases.
In contrast, for policies with inherently high variance, such as KPN and AdvKPN, integrating stable learner, like the residual network, significantly stabilizes training, as demonstrated in our theoretical analysis and the visualizations provided in Appendix[C This highlights the effectiveness of the residual layer in scenarios where the original policy struggles with variance-related instability.
D.4 ALTERNATIVE KERNEL CHOICES
In this section, evaluate alternative kernel choices within the ResKPN algorithm by testing Lin- ear, Laplacian, and Sigmoid kernels, and comparing their performance against the Gaussian kernel and the PPO algorithm. The episodic rewards achieved by ResKPN with different kernels across multiple MuJoCo environments are presented in Figure 17 As shown in the figure, the Gaus- sian kernel achieves the best overall performance, with the Laplacian kernel yielding comparable episodic rewards. The Sigmoid kernel performs slightly worse but still outperforms PPO in most environments, except for Half Cheetah. The Linear kernel, however, produces the lowest perfor- mance due to its inability to capture non-linear relationships, which are critical for complex tasks requiring intricate patterns and interactions. We also examine the variance in episodic rewards, as shown in Figure 18
Observing the figure, it is evident that ResKPN with Gaussian, Sigmoid, and Laplacian kernels exhibits similar stability, with relatively low variance across episodes in most environments. In contrast, the Linear kernel demonstrates significantly higher variance, especially in complex envi- ronments such Half Cheetah and Hopper. This elevated variance underscores the limitations of the Linear kernel, as its inability to model non-linear relationships results in less stable and reliable performance.
24

---

Published as a conference paper at ICLR 2025
Filede
Folodd
Selodd
Epkaae Beak?W) 9nd ineer RecciN
Evsode
ReaKFN
Vean Sigmold ReaktW
Hean Lapliccion ReakeN
Maan Srd ARO
Ses Gerrsion Beadliy
Sez Sagnsic Rec.Ph
Sidt Lepiacion Ree.PPh
Figure 17: The episodic reward in multiple MuJoCo environments for ResKPN within different kernels.
Splece Namerad Standor
Ephidd WaterrD
Irietted Phrccker
Figure 18: The standard deviation in multiple MuJoCo environments for ResKPN within different kernels.
25

---

Published as a conference paper at ICLR 2025
E THE COMPUTATIONAL COST
In this section, we show the computational cost in Table 4 All algorithms are accelerated using JAX, significantly reducing computation time (Freeman al., 2021). Among the tested methods, PPO and DPO exhibit the lowest overall computational costs. Integrating the RKHS policy with representation learning increases computation time due the additional computational requirements of the RKHS function, which vary depending on the chosen kernel. Specifically, Linear ResKPN and Sigmoid ResKPN achieve moderate computation times. whereas Gaussian ResKPN and Laplacian ResKPN exhibit the highest computational costs, attributed to the intensive calculations required for these kernels.
A comparison of episodic reward performance and computational costs reveals a trade-off between performance and efficiency. For applications prioritizing policy performance, the Gaussian or Lapla- cian kernel may be preferred due to their superior episodic rewards. Conversely, for scenarios em- phasizing computational efficiency while maintaining reasonable policy performance, the Sigmoid kernel offers balanced alternative. The choice of kernel should be guided by the specific require- ments and constraints the application environment.
Table 4: Runtime Comparison Across Different Algorithms and Environments (in minutes).
Gaussian ResKPN
Laplacian ResKPN
Linear ResKPN
Sigmoid ResKPN
Environment
KPN
PPO
DPO
Half Cheetah
13.09
13.18
3.95
4.68
14.21
6.08
6.24
Humanoid Standup
13.70
13.77
2.08
2.14
12.56
7.37
6.61
Inverted Pendulum
10.29
10.31
3.50
3.86
11.22
2.60
2.63
Walker2d
11.00
11.06
3.69
4.12
10.59
3.55
3.69
Hopper
10.69
10.80
1.54
1.55
13.56
3.30
3.34
Humanoid
13.07
13.23
2.13
2.28
15.43
5.96
6.12
26


Reviews:
[{"review_id": "79zkvSWI4h", "reviewer": "Reviewer_eyTM", "summary": "This paper applies Reproducing Kernel Hilbert Space (RKHS) methods to policy gradient to enhance sample efficiency and stability in training. Additionally, it introduces a variance reduction technique inspired by residual networks, further improving the stability and effectiveness of the policy training process.", "questions": "1. Based on the numerical results, it appears that the main improvement stems from the residual design. However, the comparison models are baseline models without any variance reduction techniques, raising questions about the fairness of the comparison. Additionally, variance reduction methods introduced in previous works should be considered.\n2. There is existing literature on combining RKHS with residual networks, and a discussion of these studies would add valuable context.\n3. The numerical section would benefit from testing in more complex environments to strengthen the evaluation.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduce a new RKHS policy learning algorithm.\n2. This paper introduces a variance reduction technique by designing a residual layer for the RKHS policy\n3. The numerical results demonstrate the validity of the proposed method", "weaknesses": "1. While applying RKHS to reinforcement learning (RL) is not novel, this paper lacks a discussion of existing methods. Relevant references include: \n[1] Mazoure, Bogdan, et al. \"Representation of reinforcement learning policies in reproducing kernel Hilbert spaces.\" arXiv preprint arXiv:2002.02863 (2020). \n[2] Wang, Yiwen, and Jose C. Principe. \"Reinforcement learning in reproducing kernel Hilbert spaces.\" IEEE Signal Processing Magazine 38.4 (2021): 34-45. \nAdditionally, some kernel-based methods, although not specifically RKHS-based, are also relevant to consider. \n2. Existing work, such as reference [2], introduces variance reduction techniques. A comparison or discussion of these approaches with the methods in this paper would provide valuable insights. Although RKHS is rarely applied to RL, there is extensive work on integrating RKHS with general machine learning problems. \n3. The idea of applying RKHS to RL appears straightforward, and the key distinctions from previous approaches remain unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "acU8zWFa2x", "reviewer": "Reviewer_sr1j", "summary": "The paper, titled \"Residual Kernel Policy Network: Enhancing Stability and Robustness in RKHS-Based Reinforcement Learning,\" addresses the instability and sensitivity in RKHS-based reinforcement learning policies. The authors show significant gradient variance and hyperparameter sensitivity and propose the Residual Kernel Policy Network (ResKPN). This network incorporates representation learning to adaptively align observations with the kernel's structure. The Authors also employ a residual architecture to further stabilize training. Experiments on MuJoCo tasks demonstrate ResKPN's performance, reportedly surpassing baseline algorithms like PPO and DPO by up to 30% in episodic rewards.", "questions": "Could the authors expand on how ResKPN might handle multi-agent or cooperative environments? Given the scalability challenges, it would be valuable to understand the model's limitations in such settings. How would the approach adapt to environments where action spaces vary significantly in scale or complexity? Neural networks often succeed in settings with large amount of training data, would such a setting be appropriate for a non-parametric method such like RKHS? 106: h is a functional (function -> values), but notation h(s) is used, s is a state, not a function so why do we call h a functional?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The technical claims are well-founded, and the experimental results are robustly supported by rigorous methodology. The integration of residual layers with RKHS gradients appears to reduce gradient variance, as confirmed by extensive empirical evidence on MuJoCo environments. The variance analysis is theoretically grounded, and experimental setups align well with the claims, ensuring soundness across technical aspects.\n\nThe presentation is clear overall, though there are instances where dense technical language or unclear phrasing makes comprehension difficult, especially in theoretical sections. Improved structuring or additional context around complex derivations could enhance readability.\n\nThis work contributes meaningfully to reinforcement learning research by empirically identifying a weakness in a common reinforcement learning approach. It attempts to solve this by introducing a model with enhanced stability and robustness through representation learning and a residual layer. The originality lies in effectively merging RKHS gradient variance reduction with neural network-based feature extraction, a strategy not previously well-addressed. The approach is promising for applications requiring adaptive, high-dimensional policy learning. However, just adding a residual neural network to an existing method has limited originality.\n\n- Significance: Tackling gradient variance in RKHS-based reinforcement learning is critical for real-world applications, and the results demonstrate potential for improved robustness.\n- Experimental Rigor: Extensive tests across six MuJoCo tasks validate ResKPN’s efficacy and its edge over comparable baselines in terms of episodic rewards and convergence rates.\n- Practical Impact: The adaptability of ResKPN to complex, high-dimensional environments shows promise for real-world reinforcement learning scenarios.", "weaknesses": "Complexity of Variance Analysis: While theoretically thorough, the variance analysis may benefit from simplification or additional visual explanations. This complexity could present a barrier for researchers less familiar with RKHS.\n\nComputational Cost: Given the use of RKHS, the method may face scalability limitations in more extensive settings or when applied to multi-agent environments.\n\nLimited Discussion on Alternative Kernels: While Gaussian kernels are utilized effectively, the paper could explore the feasibility of other kernels or adaptive kernel selection strategies to further broaden the model's applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M3cxhdbXo9", "reviewer": "Reviewer_CJFW", "summary": "This paper sets out to make a new SOTA in RL policy gradient algorithms, by modifying a method from reproducing kernel Hilbert space (RKHS) reinforcement learning, where policies are represented as Gaussians in a RKHS.  This allows policies to be learned in a space that captures relationships and correlations in large-dimensional action spaces.\n\nThe paper argues that previous RKHS RL approaches have suffered for two reasons.  First, the selection of the kernel is important and difficult, and an improper kernel choice leads to underperformance.  Second, RKHS RL is particularly vulnerable to high variance in the gradients, leading to unstable learning.\n\nThe paper addresses these issues by introducing ResKPN. This policy algorithm addresses the representation problem by applying the kernel to a learned representation of the state rather than to the state itself, and the high variance problem by introducing a residual layer in the representation to empirically decrease this variance.\n\nThe paper shows that policies learned via ResKPN outperform or compete closely with benchmark algorithms like PPO on a variety of standard RL problems.", "questions": "- How slow is this?  Please provide some training time comparisons with PPO.\n- Do you have any further explanation or intuition for the variance-kills-RKHS-methods argument?  Would minibatches mitigate this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is clearly explained.  It is clear what problems with RKHS RL the authors are setting out to fix, and how those problems motivate the produced algorithm.\n- The mathematics and notation are professionally done, and are easy enough to follow (though I didn't go through the derivations in the appendices).\n- The writing is clear.\n- The experiments in the experimental section are comprehensive and convincing.\n- A more effective RL baseline is significant... if it's usable (see weaknesses/questions).", "weaknesses": "- An important argument of the paper is that representation and variance problems cause RKHS RL to fail.  Accordingly, something like the illustrations of the representation and variance problems (Figure 1) are probably necessary, but I do not find these particular illustrations very effective.  They show that on one problem, some Gaussian kernels are ineffective, and that on another (single) problem, high variance can be seen.  I don't think they reinforce the strong causal relationship that the authors intend to convey, particularly when the high variance is itself dependent on the kernel selection.  Representation problems are certainly easy enough to believe, but the fact that the fully connected layer is effective *because it diminishes variance*, rather than (for example), just because it augments the representation, is not so clearly argued.\n- How slow is this? Seems like it might be very slow... Is it slow enough to be near-unusable?  I think this should be addressed with a table of training times in the appendix.\n\n**Minor things**\n- The system being trained in this algorithm is complex, with lots of different sets of parameters ($\\theta, \\iota, \\delta...$).  I think Figure 6 is important enough that it should probably be promoted to the regular paper, as the explanation is not clear enough to stand on its own without it.  The critic network should also be integrated into this figure.\n- On line 96, $U(w)$ should be $U(\\pi_w)$.\n- On line 191, should be $\\sigma^2=0.3, 0.5, 0.7,$ and 0.9.\n- Wording on lines 262-263 is not correct.\n- On line 299, \"The key idea of residual layer\" should be \"They key idea of the residual layer\" (or \"motivating the residual\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7lzmOfZ6AQ", "reviewer": "Reviewer_KaJy", "summary": "This paper addresses the challenges of achieving optimal performance in RL using policies modeled in reproducing RKHS. While RKHS-based methods offer efficient exploration of local optima, they suffer from significant instability due to high variance in policy gradients and sensitivity to hyperparameters. The authors analyze the causes of instability, particularly highlighting the increased gradient variance with wide-bandwidth kernels. To resolve these issues, they propose the ResKPN, a novel approach that integrates representation learning to process complex observations and introduces a residual layer inspired by advantage functions. This residual layer reduces gradient variance, thereby improving training stability and policy robustness. The ResKPN algorithm achieves state-of-the-art performance, with a 30% increase in episodic rewards across multiple complex environments.", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper presents a clear and well-defined contribution by addressing the instability and sensitivity issues in RKHS-based reinforcement learning methods. The introduction of the ResKPN and the integration of representation learning and a residual layer provide a novel solution to these challenges. The contribution is clearly articulated, with a strong emphasis on how the proposed method improves stability and performance in complex environments. The significant 30% improvement in episodic rewards further highlights the effectiveness of the approach.", "weaknesses": "A notable weakness of the paper is the absence of available code for the proposed ResKPN. The lack of code limits reproducibility and hinders other researchers from validating the results or building upon the work. Providing access to the implementation would significantly enhance the paper's impact and facilitate further exploration of the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)