Policy Newton Algorithm in Reproducing Kernel Hilbert Space
Authors: Yixian Zhang
Huaze Tang
Changxu Wei
Chao Wang
Wenbo Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
RKHS
Newton method
Primary area: reinforcement learning
TL;DR: We propose the first second-order optimization method for RL policies in RKHS.  Proves quadratic convergence rate and achieves superior performance in experiments.

Abstract: Reinforcement learning (RL) policies represented in Reproducing Kernel Hilbert Spaces (RKHS) offer powerful representational capabilities. While second-order optimization methods like Newton's method demonstrate faster convergence than first-order approaches, current RKHS-based policy optimization remains constrained to first-order techniques. This limitation stems primarily from the intractability of explicitly computing and inverting the infinite-dimensional Hessian operator in RKHS. We introduce Policy Newton in RKHS, the first second-order optimization framework specifically designed for RL policies represented in RKHS. Our approach circumvents direct computation of the inverse Hessian operator by optimizing a cubic regularized auxiliary objective function. Crucially, we leverage the Representer Theorem to transform this infinite-dimensional optimization into an equivalent, computationally tractable finite-dimensional problem whose dimensionality scales with the trajectory data volume. We establish theoretical guarantees proving convergence to a local optimum with a local quadratic convergence rate. Empirical evaluations on a toy financial asset allocation problem validate these theoretical properties, while experiments on standard RL benchmarks demonstrate that Policy Newton in RKHS achieves superior convergence speed and higher episodic rewards compared to established first-order RKHS approaches and parametric second-order methods. Our work bridges a critical gap between non-parametric policy representations and second-order optimization methods in reinforcement learning.

Published as a conference paper at ICLR 2026
POLICY NEWTON ALGORITHM IN REPRODUCING KER- NEL HILBERT SPACE
Yixian Zhang', Huaze Tang', Changxu Wei', Chao Wang', Wenbo Ding"
Tsinghua University
*Corresponding Author
{ diig. wenbowsz ttabboss..siiggua.eeee en
ABSTRACT
Reinforcement learning (RL) policies represented Reproducing Kernel Hilbert Spaces (RKHS) offer powerful representational capabilities. While second-order optimization methods like Newton's 's method demonstrate faster convergence than first-order approaches, current RKHS-based policy optimization remains con- strained to first-order techniques. This limitation stems primarily from the in- tractability of explicitly computing and inverting the infiniteddimensional Hessian operator in RKHS. We introduce Policy Newton in RKHS, the first second-orde optimization framework specifically designed for RL policies represented in RKHS. Our approach circumvents direct computation of the inverse Hessian operator by optimizing a cubic regularized auxiliary objective function. Crucially, we lever- age the Representer Theorem to transform this infinite- diinnte-dimenssinalll optimization into an equivalent, computationally tractable finite-dimensional problem whose dimensionality scales with the trajectory data volume. We establish theoretical guarantees proving convergence to a local optimum with a local quadratic con- vergence rate. Empirical evaluations validate these theoretical properties, while experiments on standard RI. benchmarks demonstrate that Policy Newton in RKHS achieves superior convergence speed and higher episodic rewards compared to established first-order RKHS approaches and parametric second-order methods. Our work bridges critical gap between non-parann non-parametric policy representations and second-order optimization methods in reinforcement learning.
a toy financial asset allocation problem
INTRODUCTION
Representing policies within Reproducing Kernel Hilbert Spaces (RKHS) offers powerful non- parametric alternative, leveraging its definition in an infinite-dimensional functional space to provide strong representational capability and universal approximation (Barreto et al. 2016 Lee et al. 2023). Crucially, the RKHS framework facilitates dynamic complexity adaptation: the policy updates are efficiently restricted the dimen-dimeenional span of observed data points, allowing the model size to adapt precisely to the task complexity. These properties are particularly advantageous in data- constrained environments, where sample efficiency is paramount, and in safety-critical applications, where the norm-induced smoothness provides policies with superior robustness and stability against noise and uncertainty (Paternain et al. 2020, Morimura et al. 2010). This approach has demonstrated success in various RL domains, including RL (Lee et al. 2023) and distributional RL (Morimural al. 2010). Despite these representational advantages, optimization methods for RKHS policies have remained primarily limited to first-order approaches. The RKHS Policy Gradient (Paternain et al. 2020), which achieves policy updates by adding gradient-derived functions RKHS, represents the current standard. However, this approach inherits the fundamental convergence limitations common to all first-order methods namely slow convergence in complex optimization landscapes characterized by high curvature narrow valleys.
In parametric policy representations, second-order optimization methods have emerged as effective solutions to these convergence challenges. While first-order methods like Policy Gradient (Sutton jet al.) 1999) are widely implemented due to their simplicity, they often exhibit slow convergence and sensitivity to the optimization landscape's curvature, particularly when dealing with ill-conditioned

---

Published as a conference paper at ICLR 2026
problems (Furmstoo et al. 2016). Second-order methods, exemplified by the Policy Newton algorithm (Li et al.) 2023; Jha et al. 2020), address these limitations by incorporating Hessian curvature information, enabling potentially faster convergence rates and more appropriately scaled updates. These advantages make second-order methods particularly compelling candidates for accelerating learning in RKHS policy optimization.
The natural progression towards faster optimization ppimiizaionn-deeelopiing a Policy Newton method di- rectly within the RKHS-poses significant theoretical and practical challenges. Unlike the finite- dimensional case, the Hessian analogue in RKHS corresponds to the second-order Fréchet derivative of the expected cumulative reward function. This derivative an operator acting on the function space, and computing its inverse explicitly, as required by standard Newton methods, is generally intractable in this infniitediimensional setting. Existing research on second-order methods RKHS has primarily focused on regret bounds in online learning settings with specific distributed data (Ca landriello et al. 2017a 7a
et al. 2016 Le et al.) 2013), leaving critical gap for policy optimization in RL where the data distribution shifts with the policy.
To bridge this gap, this paper introduces the Policy Newton RKHS algorithm, the first second-order optimization framework specifically tailored for policies represented within RKHS in the RL context. Our approach circumvents the explicit computation of the
Hessian operator in
policy optimization by reformulating the problem through cubic regularized auxiliary objective function within the RKHS (Maniyar et al. 2024
Doikov et al. 2024). Crucially, we leverage the Representer Theorem (Schölkopf al., 2001 to demonstrate that this infiite-dimensional optimization problem is equivalent to solving a finite-dimensional optimization problem in Euclidean space, whose dimension scales with the amount of trajectory data used in the estimate. This makes the approach computationally feasible.
Our main contributions are summarized as follows:
We propose the first second-order optimization algorithm for policy in RKHS, comprised of two key components: (1) We derive the second-order Fréchet derivative as the Hessian oper- ator and introduce cubic regularized auxiliary function to find the update step, avoiding the need to compute the intractable inverse operator; (2) We reformulate the infinitediimnsiional optimization problem into equivalent finite dimeessiiaiii problem in Euclidean space using the Representer Theorem, making the approach computationally tractable.
We establish theoretical guarantees for the proposed algorithm, proving convergence local optimum, and demonstrating a quadratic convergence rate. Our empirical evaluations toy problem verify these theoretical properties and show that Policy Newton in RKHS achieves superior performance in terms of episodic reward compared to baseline methods, with an enhanced ability escape local optima.
PRELIMINARIES
2.1 POLICY NEWTON IN REINFORCEMENT LEARNING
In reinforcement learning, Markov decision process is defined by the tuple (S, A, P. where denotes the state space; denotes the action space; P(s 8t, at) represents the transition probability function; r(st, (4)) is the reward function; 7 E [0, 1) is the discount factor; and p(0o) is the initial state distribution. Actions are selected according to a policy x(a 8t), which defines probability distribution over actions conditional on the current state. A trajectory is denoted by w (so, ao...... -T 8T)), where so ~ p(50) and T is the episode length. We denote the probability of trajectory w following a policy as p(w; T). Standard reinforcement learning aims to maximize the expected cumulative reward. However, to align with the standard conventions of reformulate this minimization
gradient descent and Newton-type optimization frameworks, problem. Throughout this paper, we define the instantaneous cost as the negative reward, effectively setting r(8e 0t) < -r(st, (t), and minimize the expected discounted cumulative term given by ((T) Eu-p(ar)) E --1, (st, at)]. where y is the discount factor. Typically, the policy is parameterized by a vector 0 Rd and the notation me used shorthand for the distribution
2

---

Published as a conference paper at ICLR 2026
(as 0). The target of RL is to find parameter Sutton & Barto 2018
O*
argmin (TO).
dE
The Policy Gradient method (Maniyar et al. 2024 Williams 1992) is utilized to find the optimal parameter using the gradient (Tg) of the expected reward J (TO):
VOJ(TO) = Buwptwaa)
Σ Vi(w))Volgg (at 1155)
where (w) Eー (8i; a,) denotes the cumulative reward starting from (st, 4z) in trajec- tory w. The Policy Gradient has enjoyed success in many fields, but it is not scale invariant and the search direction is often poorly-scaled (Furmston et al. 2016). To accelerate the optimization, the second-order information is integrated in the Policy Newton method by using the Hessian ( (Shen et al. 2019:
(TO( = -E (ump()
V. (a, sc;0)
V log π (av (s;())
#=
T-1
Σ V log ग (at 8 0)
=E
[H (w; TO)
where Ho (w; TT) represents the Hessian matrix. During the training of the RL, the direct calculation of the gradient is infeasible. Therefore, in the k-th iteration of the training, the gradient VoJ (0x) is estimated among the sampled trajectory set 7 (Sutton et al. 1999):
Vo.(Ou) N Σ
Σ
(w(w)) log (at s1; 0x)
where N denote the size of the trajectory set TN. Then the parameter 0 is updated through 0.+1
8 + v ( where is the learning rate. For the Policy Newton method, the Hessian is similarly estimated as (1a)(0)) v EwErn Ho(w; ng), and the policy is updated through 0x+1 where V is the Newton step. The calculation for
8g ( the inverse of the Hessian is computationally unstable and costly. A direct way to alleviate this drawback is introduce the regularization term and optimize auxiliary function to obtain the Doikov et al. 2024):
Newton step (Mnniyrr et al.
2024
8x+1 = argmin 6E
V J (0x) 0s) + (Z) (0x) 0x) 0x 0x)+ 110 1a11 (3)
where B is the hyperparameter of the regularization term
2.2 POLICY GRADIENT IN RKHS
Reproducing Kernel Hilbert Space (RKHS) is the Hilbert Space H. where the element K K(x, -) E Hx and f E HK satisfy the reproducing property (Kx f(x). Despite the policy is modeled by the parameter 0 with particular parameterized functions, the stochastic policy π is directly modeled with a function h in RKHS HK. where the updating gradient for it is also function (Lever & Stafford 2015). Particularly, we denote the policy as Ihy, (at 8t) 3 Th(ssaa) for discrete action space, where Z = -E' aTh((s,,a)) is the normalization constant and T is the temperature. Through the definition of the Fréchet derivative (Megilliraay Oldenburg 1990), the Policy Gradient in
RKHS is derived as (Mercier et al. 2025 Lever & Stafford 2015 Patemain et al. 2020):
VnJ((T) -Eunpp(urr.) (w: m.)) Eump(umn))
W. (e Vn log n, (at st)
(4)
Σ (.(w) (K ((11, (z). -E, (an
[K ((st, d'),,))))
where K ((s,, a.),)) )) is the kernel section induced by the state action pair (sz, at). Without loss of generality, the action space is set discretely in the rest of the paper. The estimation of the gradient is
3

---

Published as a conference paper at ICLR 2026
similar to Equation where we denote it as (TA)) Nv Ewery Zn(w;)), Then the policy is updated iteratively by hR+11 = hi + (TT)) For simplicity, we denote J (TT) as ((h in the rest of this paper.
The use of RKHS policy improves the sample efficiency greatly during training (Paternain et al. 2022), and the overall performance is better than traditional gradient methods (Zhang et al. (025). However, there is still a lack of research on the Policy Newton algorithm within RKHS. A potential obstacle for its derivation is that the Hessian of (TA, is infinite, where its inverse is infeasible to represent explicitly. In previous research, Newton optimization is mainly studied in the online learning problem. In (Calandriello et al. 2017a), the Newton optimization scheme derived where the inverse of the Hessian in RKHS is approximated iteratively. The authors in (Calandriello et al. 2017b) integrate the adaptive embedding with the inverse approximation, which alleviates the computational burden. Despite this success in RKHS Newton optimization, the learning scheme is only suitable for data sampled from the same distribution during training (Lu et al. 2016:] Le et al., (001)), while the distribution of transitions in RL is related to the updating policy. As far we investigated, this is the first paper to derive the Policy Newton in RKHS, and the convergence of our algorithm is also guaranteed.
THE POLICY NEWTON IN RKHS
In this section, we present the derivation of the Policy Newton in RKHS. In the conventional Policy Newton method, it is simple to obtain the Hessian by deriving the second-order derivative of the expected discounted cumulative reward J (TTT)) with respect to the parameter 0. However, in RKHS space, the second-ords ooderr Fréchet derivative may not be implicitly represented. Before the derivative of the Hessian within RKHS, we first introduce the following definition:
Definition 3.1 Defining the outer product Hr @ HK as new RKHS with operator [K((s, at) (6). a() K ((8+ o4)..) & K ((o),a)): E HK @0 HK (Kubrusly & Vieira 2008
Szabó & Sriperumbudur] 2018 Kumari el al.) (017), it satisfies that:
[K((E, ue); ; (o/, a))) (166, a.),, ) = K ((ss, at), K ((d d)), a"") < <K((s;; (t); (s/s d))), K((s", (6"" aa"")) (6"",""") > = K ((s,, at), (6% a()) K ((6 d)), (ol", a²"))
While the first-order Fréchet derivative (h) is an element in Hk. the second-order Fréchet derivative is an operator on this space, which we denote by the symbol Vi This operator can be identified with an element in the tensor product space Hk & HK. as shown in the following lemma.
(w); mn),, where Lemma 3.1 The second-order Fréchet derivative VIJ((T) Bu-p(werr)) (w; HA (w) mx) is
T-1 VT( Cov [K ((se, a(); )}
Σ V. (e log
& Σ V log logmh
Σ T=
Here Vn log T/ = Vn log TA (ar(s)) and Cov [K ((s,, a(), - -)] denotes the covariance opera- tor for kernel section K ((5,, a)), .), which is detailed as:
Egr~a((-1e)) [K ((s,, )'), ) @ K ((s,, -a K ((s,, 3)0 K ((8g, d)),
The detailed derivation is shown in Appendix A Here, we only introduce simple example, (h) Th(e)..))) K ((st, at), ), which is a component in VAJ(m)) to present the core concept for intro- ducing the outer product in RKHS when implementing the Fréchet derivative.
The second-order Fréchet derivative Let h. E Hx and (h) = Th(ss K ((s), . x K ((st, at), ). Then according the definition of the Fréchet derivative (Megillivray & Oldenburg 1990), we testify that D(h)= = (h):
11(h + g) Uh) ((())ggl |le Th(x) K(x,) :) (TTe)) - [1))]]
||9||
|||||
Th(r) |K (z. I| MT² 19(x))1
MT²
e Th(x)) (K (2, x)) x)) |lll| 9-0 0,
|||||
2
4

---

Published as a conference paper at ICLR 2026
where the last inequality is due to SchwayySchwarz. Through Lemma 3.1 could find that the second-order Fréchet derivative (TT) is infeasible to present explicitly. Calculating its inverse is further infeasible for the Policy Newton methods in RKHS. Fundamentally, this is because the Hessian operator is trace-class in the RKHS, and therefore compact. In an infinite-dimensional space, compact operator does not have a bounded inverse (Kreyszig 199T), rendering the standard Newton step ill-posed. However, we can still obtain the RKHS Newton step Ah through optimizing the regularized auxiliary function similar Equation 3
Ah = argmin (VIJJ (hx) .h + 2 (i (hx) o REHx
3 6
(5)
where V (1x) + HA (w: TT) is the estimated RKHS Hessian. Although this estimation
is not computationally feasible, the corresponding second-order component (V (hx) o h h) is easy to calculate according to the Definition 3.1 The optimization in HK is still hard to proceed, but through the Representer Theorem in RKHS, we could easily transform the parameter space in this optimization problem from Hx into
Lemma 3.2 Representer Theorem (Schölkopf et al. 2001). Suppose we are given a nonempty set X, positive definite real-valued kernel -(.,-) X x X, a training sample (æ1. v1) :..., (IM, yM)) X x R. a strictly monotonically increasing real-valued functiio G. an arbitrary cost function c. Then any h E HK minimizing the regularized functional
e c( V1, (x1)) ..... MM UM. h (ra))) + G(((A])
admits a representation of the form h(() = Em a;K (zw.), where ag is the weight for kernel sections.
Applying the Representer Theorem (Lemma 3.2 Optimization Problem 5 is equivalent to finding ax via:
a argmin aERNT
(VnJ (hk)), Ra)
V (ha)) has ha T
(6)
where ha >i ΣT a ((s), af)..). Here, (s/, at) denotes the acteon pair for the i-th trajectory at time step t, and a {af) {=i is the set of kernel weights. These weights can be vectorized as ā RENT, where the I-th element corresponds af with k (i 1)T + Applying the Representer Theorem transforms the search for the optimal function perturbation into a search for finite coefficients a. By substituting the expansion ha back into the operator-based objective
Equation aan utilizing the reproducing property {K (x, )) K K(y, .)) = k(x, y). we can explicitly derive the algebraic form of the quadratic and cubic terms with the following theorem.
The optimization of the Policy Newton step in RKHS is equal to the optimization of the following quadratic optimization with cubic regularization:
Theorem 3.3
a² argmin GERNT
(u. (Hā, a) +
|all
(7)
Here, u E NT is the first-order coefficient vector where
T
2: Σ
Vi(w) (K ((si, (St, a.)) Ea [K a(s1,
Let H E NTXNT be the second-order coefficient matrix given by:
H T N
bc
T N
Σ (I(w))()
NT
(8)
Here, E NT and c NT are vectors, and 5) information. The components by, Co and the related covariance terms (!!) are defined as:
represents component-related covariance
NT
NT
b. Ve(w) (Ku Ea (Ka),
Σ
q=
Σ
(Ka E. (K())
E
Cov
[K,, Kdd]
5

---

Published as a conference paper at ICLR 2026
In these expressions, the kernel terms are Ka K ai))
at), (s1,
and K2 K ((ss, az), (s),
The detailed derivation for Theorem 3.3 is presented in Appendix It observed from this theorem that the Policy Newton in RKHS is similar to the traditional Policy Newton method, but the complexity dependent on the volume of data, i.e., x T. which possesses
of the Optimization Problem 7 the same property of other RKHS methods like support vector machine (Burges 1998) and radial basis function networks (Park& & Sandberg 1991). We show in the next section the suboptimality and convergence rate for the proposed Policy Newton in RKHS.
Intuitive Interpretation of the Reduction. Before proceeding to the convergence analysis, we briefly clarify the physical meaning of the terms in the Anite-dimensionnl optimization problem 7 The vector o represents the projection of the functional gradient V,J onto the data-dependent subspace spanned by { [K ((s,, a.), The matrix H encapsulates the curvature information: the
T term N de corresponds to the outer product of first-order gradients (the first term in Equation 8).
while the term involving (0 captures the covariance structure of the policy's s action distribution (the second term).
4 THE SUBOPTIMALITY AND CONVERGENCH RATE OF POLICY NEWTON IN RKHS
this section, we first detail the Policy Newton RKHS algorithm. We then analyze convergence properties, demonstrating that despite optimizing via a surrogate function, the resulting policy converges to local optimum. Furthermore, we show that our proposed Policy Newton RKHS exhibits second-order convergence rate, in contrast to the first-order rate achieved by Policy Gradient in RKHS.
4.1
THE POLICY NEWTON IN RKHS METHOD
The Optimization Problem Gaadmits two primary solution approaches:
(1) Directly computing the derivative of objective function, setting it to zero, and solving for critical points.
(2) Optimizing it using various classic optimization methods, including gradient descent, the Newton method, and the conjugate gradient method (Lasdon al. 2003).
While the analytic method (1) is conceptually simple and direct, can introduce significant instability into the training process. In complex environments, this instability can lead to exponential error growth. Consequently. method (2) represents a more practical optimization approach. We select the conjugate gradient method as optimization method. More settings are detailed in Appendix H
Algorithm Policy Newton RKHS Method
Input: Number of iterations M, trajectory batch size N, learning rate
1: Initialize RKHS function h +- 0, actor policy Th, based on h,, trajectory set
2: for m 1,.... do
3: Sample N trajectories using the current policy The.. store in
4: Estimate the first-order coefficient vector v and sccond-order coefficient matrix H using
(according to Theorem B.3).
5: Solve the Optimization Problem using conjugate gradient descent method, output the optimization result a.
6: Construct the RKHS update step Ah using & via Lemma 3.2
7: Update the RKHS function: hm+1 hm 172h and the actor policy Thmma based on hm+1-
8: end for
9: return final policy Thorts"

---

Published as a conference paper at ICLR 2026
4.2 SUBOPTIMALITY ANALYSIS THE PROPOSED ALGORITHM
Theoretically, we assume that the optimal solution to Problem 7 is consistently achieved by Algo- rithm To establish the convergence properties of our algorithm, introduce the following lemmas and assumptions.
Lemma 4.1 (Monte Carlo convergence rate (C.1) Assuming Ea--ptum w 2 (w; (www))))l xh)l S o and (u-ptusmy) l (wz; Th I I of. the Monte Carlo estimation of first and second-order Frécher derivative, namely V (hx) and V7 (hx), achieve the convergence rate of O (x))))
||V (he) - VaJ (hx) 112
Eump((as)) N
[1V (h+) VJJJ (hx) 11 2 N
The proof is straightforward by using the property of expectation, which we show in the Appendix C.I Establishing convergence also requires Lipschitz continuity assumption for the Hessian, similar to other gradient-based methods (Xiao 2022 Zhang et al. 2020).
Assumption 4.1 (Lipschitz continuous) The Hessian operator (h) is Lipschitz continuous with constant 0 < S B, i.e., for all h1, h2 Hy: (we discuss the validation of this assumption in Appendix
[V))) -Vh S Llih h2ll-
Through this assumption, could establish the upper bound for J(hh) with respect the Hessian and step norm:
Lemma 4.2 (Taylor upper bound Under Assumption 4.1 for any h1, h2 E Hx:
L
(hha) s J(hi) + (VnJ(h))) h1) (h11) (h2 h1), - h1) 11112 hn
The fundamental approach to proving convergence centers on establishing relationship between the expected gradient norm, E| (hx))] and function denoted by L(3, o. of.N). .V. Following standard techniques in convergence analysis, the norm of the update step, / h& ll. is employed as an intermediate quantity to construct this relationship. this end, an upper bound for this step norm is derived in Lemma 4.3
Lemma 4.3 (Step norm upper bound
Denoting the updating times for Policy Newton in RKHS as M, and the number of trajectories sampled in each updating N, the updating step can be upper bounded as:
E [|bR+1 hRll 36(J(h))- - JJ)
48v3
do
3/2
864 of
3M
B3/2
33/4
B3
N3/2
where a random variable uniformly distributed on (1...., M). such that P(R=K) k) 1/M.
This lemma provides an upper bound on the expected cubed norm of the update step involving randomly selected iteration R, which is key quantity used subsequently to establish convergence bounds in expectation for the Policy Newton in RKHS. Next, we establish the lower bound of the norm for the update step, which relates to the RKHS gradient.
Lemma 4.4 (Step norm lower bound
C.4
The updating step can be lower bounded
1
of
B[ 1
E
L
+ B
VN
2N(L+
Having established both lower and upper bounds for the step norm, we can now use these results to construct the main convergence theorem.
Theorem 4.5 (Convergence property (C.5) Given Lemmas 4.3 and let R be random vari-
able uniformly distributed on (l,..., MJ. The sequence {hx} generated by iterative optimization in Theorem 3.3 satisfies
lim [hI[V](h++)]]]====
M.N
This theorem indicates that the expected gradient norm at randomly chosen iteration converges to zero, implying convergence towards stationary point.
7

---

Published as a conference paper at ICLR 2026
4.3 THE SECOND-ORDER CONVERGENCE RATE
Establishing the convergence rate for stochastic Policy Newton methods typically requires specific assumptions regarding problem structure. While the convergence properties of Newton's method in fiite-dimeeniionll Euclidean spaces are well-established (Furmston et al. 2016). extending these guarantees to the
with the cubic regularization auxil- lary function- requires rigorous verification. We demonstrate that the siiteeddmessionall reduction derived Theorem 3.3 preserves the desirable quadratic convergence properties of the original pperator-theoretie problem. To establish baseline performance characteristics, this section analyzes the convergence rate under idealized conditions, while a comprehensive analysis under more realistic stochastic assumptions remains for future work. Specifically, we assume access to the true gradient [hb)( and Hessian Vi at each iteration k. effectively setting [ha) V and ((2g)) = Jh, Under this deterministic scenario, we prove that the method achieves local quadratic convergence rate.
RKHS
Our analysis relies on a key assumption: that the inverse of the regularized Hessian is uniformly bounded. This is standard condition in the analysis of Newton-type methods, required to ensure the update step is well-defined by excluding potential singularities (Nesterov & Polyak 2006a Nocedal & Wright 2006a).
Theorem 4.6 (Local Quadratic Convergence (D) Consider the deterministic Policy Newton RKHS method (Algorithm with V = VnJ and Vi,((h)) = (ha()))) Assuming the
norm of the inverse operator (VV J(hx) + 21 11 II) I| is bounded by constant B, and assume that the update step is sufficiently small that || | S Lllex]l-
If the initial iterate bo is sufficiently close to h", the sequence {hu} converges quadratically h". That is, there exists a constant Cg > 0 such that
| 11ll S Cg1lhx- - h²1?
for all k sufficiently large. The validation of the assumptions is discussed in Appendix E
5 NUMERICAL EXPERIMENT
This section presents an empirical evaluation of our proposed Policy Newton method in Reproducing Kernel Hilbert Space (RKHS) across two distinct experimental settings: (a) a simplified Asset Alloca- tion environment designed specifically to demonstrate the quadratic convergence properties of Policy Newton RKHS (Yoo et al., 2023), and (b) complex control tasks from the Gymnasium framework (Towers al.) 2024), including CartPole and Lunar Lander. Throughout all experiments, utilize standard Gaussian kernel for the RKHS representations. The Asset Allocation environment serves to empirically validate the theoretical convergence guarantees established in Section 4 Additionally, benchmark Policy Newton in RKHS against several baseline methods in complex environments to demonstrate its superior performance characteristics.
5.1 QUADRATIC CONVERGENCE TESTED IN THE TOY EXPERIMENT
We empirically validate the quadratic convergence properties of Policy Newton in RKHS using simplified Asset Allocation environment (Lee et al. 2004 Yoo al. 2023). While the complete asset allocation problem presents substantial analytical challenges, we utilize reduced- reduued-complexitt variant (detailed in Appendix F where the global optimal policy can be explicitly represented, enabling precise quantification of convergence properties for both the policy and cumulative reward J (Tr)
For comparative analysis, we implemented four distinct methodologies. The conventional Policy Gra- dient and Policy Newton methods (Maniyar et al. 2024) utilize discrete policies with parameterized action probabilities. The Policy Gradient in RKHS implementation follows the approach described in (Lever & Stafford 2015), while our Policy Newton in RKHS method is implemented according to Algorithm] All policies were initialized with uniform distributions.
The experimental results presented in Figure reveal several important findings. Figure La demon- strates that both Policy Gradient in RKHS and Policy Newton RKHS converge rapidly toward the maximum expected episodic reward. Notably, Policy Newton in RKHS exhibits clear quadratic
8

---

Published as a conference paper at ICLR 2026
Furctton Convergetce with Condidence Intervah
Kilyy IUS300E
Poliy
11d
Polly Recssaic
Polcy Gradieed salisee
Polo henon ninors
144
10
Policy Grodeet AXHS
Policy Nowton
Policy Grodlett
dimendoon
Polcy eutc
Optine palicy
peiky
Cum FOweddR
Policy Newton In RKHS
100000
200000
100000
500000
Pytcy Convergerce with confidence intervals
1-0
10
Palkyy hawiton
Patryy Gradeet
10
Palay henton in I1Q5
Pulay Gradlent RC15
200
400
330
1000
19 65
1..0
03
3.0 15
20
(a)
(b)
Figure I: Experimental results demonstrating the quadratic convergence of Policy Newton in RKHS within the simplified Asset Allocation environment. Three benchmark methods are compared: conventional Policy Gradient, Policy Newton, and Policy Gradient in RKHS. Ta illustrates the convergence metrics during training. [b] visualizes the optimization trajectories of all methods in PCA-projected policy space, where the background surface is interpolated return landscape. This plot highlights how different algorithms move through the policy space toward the optimal policy.
convergence behavior as it approaches the optimal policy, In contrast, the conventional methods show different sharacteristics-wiile the standard Policy Newton method converges more rapidly than conventional Policy Gradient (as shown Figures[Taland instability and ultimately converges to suboptimal local maximum. It is important note that in this toy environment all four compared methods share the same representational capacity: the state and action spaces are finite, and each algorithm directly optimizes the probability value assigned to every (state, action) pair. Thus, all methods are able to exactly represent the optimal policy. The differences observed in Figure [b] [b] therefore arise purely from the optimization geometry rather than the expressiveness of the policy class. In particular, RKHS-based updates span the data-dependent kerne basis, which yields richer of descent directions and enables the optimizer to out of suboptimal attraction regions that trap parametric methods.
training trajectory exhibits greater
5.2 TRAINING PERFORMANCE IN RL TESTING ENVIRONMENT
To evaluate the efficacy and universality of the proposed Policy Newton in RKHS algorithm, we conduct experiments utilizing standard RI environments from the Gymnasium suite (Towers et al. 2024), covering both discrete and continuous action spaces.
Discrete Control Tasks. We first evaluate the method CartPole and Lunar Lander. These environments feature discrete action spaces, directly utilizing the theoretical framework established in Section 3. The baseline policies (Policy Gradient and Policy Newton) are parameterized using a linear model augmented with a polynomial transformation (Maniyar et al. 2024) to ensure fair comparison representational power.
To demonstrate the method's 's scalability to digeeniimnsional continu- ous control, we further evaluate on Inverted Pendulum and Hopper. For these tasks, we implement the Policy Newton in RKHS using the continuous Gaussian policy formulation. The rigorous theoret- ical derivation for the second-order RKHS step in the Gaussian policy provided Appendix The baselines utilize the same polynomial feature expansion as in the discrete case but map to continuous action outputs.
Continuous Control Tasks.
Performance Analysis. The experimental results summarized in Figure In the discrete tasks (Figs. [a and (b)), Policy Newton in RKHS exhibits rapid convergence and superior sample efficiency, significantly outperforming the first-order RKHS baseline and the parametric Policy Newton method.

---

Published as a conference paper at ICLR 2026
In the continuous tasks (Figs. 2c and 2d the advantage of our method is equally pronounced. In Inverted Pendulum, our method stabilizes quickly. the challenging Hopper environment, Policy Newton in RKHS demonstrates superior sample efficiency, achieving high rewards with fewer iterations than the baselines. We attribute this performance to the effective utilization of curvature information via the Hessian operator, which aids in navigating complex optimization landscapes, and the flexible representational capacity of the RKHS.
Moteton ROMS Palicy Gradd ant B005
Pekic Nowior Pelicy Crackent
Pulky Nowton HONS Fallcy Gr Gradiet RGIS
Policy Nowton Palccy Crudiect
Eptoodi reward
Taining steps
Training steps
(a) CartPole (Discrete)
(b) Lunar Lander (Discrete)
Pallyy Newton
Newton
1100)
000
1990
Average Return
600
Average
-00
400
332
2
Training steps
Training steps
(c) Inverted Pendulum (Continuous)
(d) Hopper (Continuous)
Figure 2: Comparative analysis of Policy Newton in RKHS against established baseline methods across discrete and continuous Gymnasium environments. The plots display mean episodic rewards across 5 independent runs, and the shaded regions denote the 95% confidence intervals. (a-b) Performance on discrete tasks, utilizing the formulation in Section 3. (c-d) Performance on continuous control tasks, utilizing the extension derived in Appendix || Our method demonstrates consistent superior sample efficiency and rapid convergence speed in both settings.
6 CONCLUSION AND FUTURE WORK
This paper successfully introduced Policy Newton in RKHS, the first practical second-order opti- mization method for reinforcement learning policies represented within Reproducing Kernel Hilbert Spaces. We established its theoretical foundations, proving convergence to a local optimum and demonstrating a local quadratic convergence rate. These theoretical properties were empirically validated on both toy problem and standard RL benchmarks, where Policy Newton in RKHS achieved significantly faster convergence superior episodic rewards compared first-order and parametric Newton baselines.
While the current results are promising, extending the application of Policy Newton in RKHS to highly complex RL problems, such as the Humanoid environment (Todorov et al., 2012), may reveal challenges related to robustness and stability. A promising avenue for future research the integration of neural networks with the Policy Newton in RKHS framework, potentially drawing inspiration from architectures similar to those proposed in (Zhang et al., 2025). to enhance performance such demanding scenarios. The primary focus of this paper has been the rigorous theoretical establishment of Policy Newton RKHS, laying the groundwork for these and other exciting explorations in future work.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This study uses only a synthetic toy Asset Allocation environment (Appendix F) and public Gymna- sium benchmarks (CartPole, Lunar Lander) (Towers al. 2024): no human subjects, personal data, or sensitive attributes are involved, and no proprietary datasets are introduced. We caution against de- ploying the method in high-stakes settings (e.g.. financial decision systems) without Nomain-spectie governance, distribution- -hiff monitoring, and independent validation. The compute footprint is modest. The authors disclose no conflicts of interest and no external sponsorship that could bias the work.
REPRODUCIBILITY STATEMENT
We provide anonymous supplementary archive containing the full implementation, dependency specifications, and runnable scripts to reproduce all figures. The algorithmic procedure is given in Algorithm [ the finite-dimensional reduction and coefficient construction appear in Theorem 3.3 with derivations in Appendix B and RKHS second-order calculus in Appendix A Experimental settings, hyperparameters, and random seeds are documented in Appendix the toy environment is fully specified Appendix These materials enable end-to-end reproduction of the reported results.
ACKNOWLEDGMENTS
This work was supported by Shenzhen Science and 11ZD202409033000005008 and Ant Group.
Technology
Program (No.
REFERENCES
André MS Barreto, Doina Precup, and Joelle Pineau. Practical kernel-based reinforcement learning. Journal of Machine Learning Research, 17(67):1-70, 2016.
Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 60(2):223-3) 2018.
Corentin Boyer and Antoine Godichon- Bagghonn.. the asymptotic rate of convergence of stochastic newton algorithms. Computational Optimization and Applications, 85(2):589-633 2023
2023.
Christopher JC Burges. A tutorial support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2):12 1998.
Daniele Calandriello, Alessandro Lazaric, and Michal Valko. Second-order kernel online convex optimization with adaptive sketching. In International Conference on Machine Learning, pp.
645-653, 2017a.
Daniele Calandriello, Alessandro Lazaric, and Michal Valko. Efficient second-order online kernel learning with adaptive embedding. Advances in Neural Information Processing Systems, 30, 2017b.
Nikita Doikov, Konstantin Mishchenko, and Yurii Nesterov. Super-universal regularized newton method. SIAM Journal on Optimization, 34(1):27-56, 2024.
Thomas Furmston, Guy Lever, David Barber. Approximate newton methods for policy search in markov decision processes. Journal of Machine Learning Research, 17(226):1-51, 2016.
Devesh K Jha, Arvind U Raghunathan, and Diego Romeres. Quasi-newton trust region policy optimization. In Conference on Robot Learning, pp. 945-954, 2020.
Erwin Kreyszig, Introductory functioonl analysis with applications. 1991.
Carlos S Kubrusly and Paulo CM Vieira. Convergence and decomposition for tensor products of hilbert space operators. Operators and Matrices, 203):407-416, 2008.
Rani Kumari, Jaydeb Sarkar, Srijan Sarkar, and Dan Timotin. Factorizations of kernels and reproduc- ing kernel hilbert spaces. Integral Equations and Operator Theory, 87(2):225-25 2017.

---

Published as a conference paper at ICLR 2026
L Lasdon, S Mitter, and A Waren. The conjugate gradient method for optimal control problems. IEEE Transactions on Automatic Control, 12(2):132-133 138, 2003.
Quoe Le, Tamás Sarlós, Alex Smola, et al. -aproianda kernel expansions in loglinear time. International Conference on Machine Learning, pp. 244-252, 2013.
Jae Won Lee, Jongwoo Lee, Byoung- Tak Zhang, et al. Dynamic asset allocation exploiting predictors in reinforcement learning framework. In European Conference on Machine Learning, pp. 298-309, 2004.
Suyoung Lee, Myungsik Cho, and Youngchul Sung. Parameterizing non- an-paaametric meta- reinforcement learning tasks via subtask decomposition. In Advances in Neural Information Processing Systems, volume 36, 2023.
Guy Lever and Ronnie Stafford. Modelling policies in mdps in reproducing kernel hilbert space. In Artificial intelligence and statistics, pp. 590-598, 2015.
Haoya Li, Samarth Gupta, Hsiangfu Yu, Lexing Ying, and Inderjit Dhillon. Approximate newton
policy gradient algorithms. SIAM Journal on Scientific Computing, 45(5)::2585-A260) 2023.
Jing Lu, Steven CH Hoi, Jialei Wang. Peilin Zhao, and Zhi- Yong Liu. Large scale online kernel learning. Journal of Machine Learning Research, 17(47):1-43, 2016.
Mizhaan P Maniyar, LA Prashanth, Akash Mondal, and Shalabh Bhatnagar. A cubic-regularized policy newton algorithm for reinforcement learning. In International Conference on Artificial Intelligence and Statisties, pp. 4708 4716. PMLR, 2024.
Peter R Mcgillivray and DW Oldenburg. Methods for calculating fréchet derivatives and sensitivities for the non- non-linear inverse problem: A comparative study 1. Geophysical Prospecting, 38(5): 499-524, 1990.
Jean-Marc Mercier, Aguirre Max, et al. A scalable kernel approach to reinforcement learning. SSRN, 2025.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In International Conference on Machine Learning, pp. 799-806, 2010.
Yurii Nesterov and Boris T. Polyak. Cubic regularization of newton method and its global performance. Mathematical Programming. 108(1):177-255, 2006a.
Yurii Nesterov and Boris T. Polyak. Cubic regularization of newton method and its global performance. Mathematical Programming, 108(1):177-205, 2006b.
Jorge Nocedal and Stephen J. Wright. Numerical Optimization. 2006a.
Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer, 2006b.
Jooyoung Park and Irwin W Sandberg. Universal approximation using Neural Computation, 3(2):246-25 257, 1991.
networks.
Santiago Paternain, Juan Andrés Bazerque, Austin Small, and Alejandro Ribeiro. Stochastic policy gradient ascent in Reproducing Kernel Hilbert Spaces. IEEE Transactions on Automatic Control, 66(8):3429-344 -444, 2020.
Santiago Paternain, Juan Andrés Bazerque, and Alejandro Ribeiro. Policy gradient for continuing tasks in discounted markov decision processes. IEEE Transactions Automatic Control, 67(9): 4467-4482, 2022.
Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient in lipschitz markov decision processes. Machine Learning, 100(2-3):4494477 479, 2015.
Bernhard Schölkopf, Ralf Herbrich, and Alex Smola. A generalized representer theorem. In
International Conference Computational Learning Theory, pp. 416-426, 2001.
12

---

Published as a conference paper at ICLR 2026
Zebang Shen, Alejandro Ribeiro, Hamed Hassani, Hui Qian, and Chao Mi. Hessian aided policy gradient. In International conference on machine learning, pp. 5729-5738. PMLR, 2019.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: introduction. 2018.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
Zoltán Szabó and Bharath K Sriperumbudur. Characteristic and universal tensor product kernels. Journal of Machine Learning Research, 18(233):1 29, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:240.11032, 2024.
Ronald J Williams. Simple statistical gradient-followingg algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-2) 2222,556..1 1992.
Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1-36, 2022.
Scong Joon Yoo, Yeong Hyeon Gu, al. Safety aarl: Weight adjustment for reinforcement-leamiing based safety dynamic asset allocation strategies. Expert Systems with Applications, 227:12029) 2023.
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal Control and Optimization, 58(6): 3586-3612, 2020.
Yixian Zhang, Huaze Tang, Huijing Lin, and Wenbo Ding. Residual kernel policy network: Enhancing stability and robustness in rkhs-based reinforcement learning. International Conference on Learning Representations, 2025.
A THE DERIVATION OF THE SECOND-ORDER FRÉCHET DERIVATIVE IN RKHS
This section details the derivation of second-order Fréchet derivatives of the log-policy, log m. (a, st). with respect to the function E Hx. The first-order Fréchet derivative introduced in (Mercier 2025). for convenience's 's sake, we also detail it in this section. The policy defined as Th (a, sx) 1 ah(s where
E''Es Th(ss,a") the normalization constant. For brevity in this section, will denote (st, a,) (s, a) when the context is clear for single state-action pair for which the log-policy is being differentiated. The kernel section K ((ss, a), -) is an element in HK-
We first derive the first-order Fréchet derivative VA log (a 8). The log-policy log F (a T Th(s,, a) - log Z.
The Fréchet derivative of the first term is:
=T(Th(s.a)=TK((()))))
For the second term, - llog Z:
1 V
Vi( Z) =
We compute V Z:
Z Σ Th(s,aa)
a'EAA
13

---

Published as a conference paper at ICLR 2026
V 2 [a((eThial)) = Σ
=T E Th(s,d') ( (((s,a'),)
Substituting this back:
2
- - Σ Th(s,a)) (((s,)))(((d))))
Recognizing that Thiraaa)
Z Tdl 18), we simplify:
Vn(- log Z) - malal ())(((s)'))))) -- (119) [K((s,a'),
Combining the derivatives of both terms, we obtain the first-order Fréchet derivative of the log-policy:
V (a s) = (K((s, a), - TEa
Factorizing T:
V log T ( (-1s) [K((sa'') a'),
(9)
This expression forms the core of the Policy Gradient in RKHS shown in Equation 4 of the main paper when appropriately weighted and summed.
Next, we derive the second-order Fréchet derivative (Hessian operator) of the log-policy with respect to h, denoted as V log (a s). This is obtained by differentiaiing Equation
V lag 5) Vn T [(((),)))))
Ema(a' [mmadds))((s())))))
a'EA
Since T is constant and K((s,a),)) is fixed element in Hx (not depending h for this differentiation), its derivative is zero:
V log (a = TVA
Σ mn(d' (){((s,))))))
a'EA
Applying the product rule for Fréchet derivatives (treating K((s, a)), as a constant vector in Hr for each d'):
V log (a = -T Σ (VaTa s)) & K((s,a'),)
d'EA
Here, @ denotes the outer product as defined in Definition 3.1
Using the quotient rule (x( D Let No. Th(o,,,), V, Th(aa,)) (((,aa)) Let D = Z Ear Th(na""), so = Dn Th(na") ((s,aa""
Now, we compute Va s). Recall (a s) Thiaaaa)
Fnta(d|(s) (TeTH(s)))(sssdd)) Z - Th(nao) Th(sa"""))
72
T e Th(s,a')
Z K((s,a').-) Thla,a)) Enp Th((sa")) (((,aa")"(
Z
=aalad1 s) TTala' |s) E Ta(a" | (s){(s)
a²
Va = TTu (8) (K((s, Ea [K((s, (sssa")))))) )))
Substituting this expression for V Va s) back into the equation for V log (a s).
To align with the result in Lemma 3.1 of the main paper, which states V log mn (a 8t)
14

---

Published as a conference paper at ICLR 2026
-T Cov (t) (a)) a'), the substitution effectively uses Vana(aa s)
Vi log (a 6) Σ
An(d 8) (K(((,)))) - Earrwaatte) ( 8 (((s,a)),)
d'EA
--T
Σ
(a ())(]]))))))))))K((((()))))
a'EAA
Σ an(a' (Ba" ( [K((s " @k((s.d))))
a'EAA
(Be'' [K((s,a));) (((ss,)),))
(Ben [K((s, 60
Σ WEA
Ta(a' In(alIs)<((ss0)))) [s)
-T (Bar (Ear (11s) [K(s,, d'), @ K ((s,a'),,)
-Bd' (11s) [Ks d'), )) & Ea" [K((s,"))
(10)
This can be compactly written using the covariance operator as defined in Lemma 3.1 (using St, at for generality):
V log (a (t) T Cova'ma. [K(ss, .)).
This expression for V log TA (a: 8() is the component used in constructing the Hessian operator in Lemma 3.1 and the estimated Hessian Vi (h&) in the paper. By substituting the first and second derivative in with v log (a; st) and V log (a; st), the Lemma 3.1 proved.
B THE DERIVATION OF THEOREM 3.3
Theorem 3.3 transforms the RKHS optimization problem for the Newton step Ah (Equation 5 into an equivalent minee-dimensional optimization problem (Equation 7 The RKHS update step is AA = (-(-) = E KgKK(rg ((x, -), where = (sk, ag) are state-action pairs from the N x T trajectory data points (k is a flattened index from to M = NT), and ā E RNT is the coefficient vector.
The objective function in Equation is:
L(x)
V R (hx) ha) + V7.j (hx) ha, ha)
| 6
We derive the forms for the first two terms. The third term, |a|l directly uses the Euclidean norm of a as stated in Equation
Let M NT. The set of basis functions is {K( }} The perturbation is ha(:) EMT a; ().. We use index i (or j) for the coefficients as and the basis functions K K(ma, ) We use index i (or I') for data points from the batch of M samples when defining the gradient and Hessian operators.
B. FIRST-ORDER TERM:
(ha,))))
The estimated first-order Fréchet derivative (hxz) (denoted gop for operator form) is given by adapting Equation for the empirical average over N trajectories, M NT total samples:
T M N
(.) vh.((h)))) = Σ
vi(w) (K(Z),)) Easoa((s)) ('s,ss))))
15

---

Published as a conference paper at ICLR 2026
where xı = (st, at) is the 1-th data point in the batch, and (w)) is its associated cumulative reward. The inner product is:
- (). Σ [a,((an))
V ha)
M
Σ as (VnJ(hk), ((x)))((ss)))))
Kxxs:)) (by linearity of inner product)
(V, K K(x..)) )) Using the reproducing property and the expression for Vn
Let v;
T
M
Σ
Vi(w) (K(m..) (e [K((s), ((()
=
N
1=1
T
vi(w) (K(I) (ed))() =z))
N
Thus, (Vnj(ha), ha) En a.u, T. ā. This definition of v matches Theorem 3.3
B.2 SECOND-ORDER TERM:
[V..J (hx) has ha)
Let Hest (h.))) Based the updated Lemma 3.1 and Equation the estimated Hessian
operator (ha) includes two components:
1
Σ Vr(w) log (1i)
&
vn log TA (are))
Σ Cov a' [K(a,, (a)]
HD be the operator for the first part (outer product) and H'P for the second part (covariance sum), such that (hb) 2 (H!)) H!!)).
Let
1. Contribution from H Let V log (x))()) = (K(x,) - Barwa(( [K((s, a(),),,)),))))), ))) Let
(1(ha)= (Vn log TA (37), ha).
M
Xi(ha) = T Σ ai [K( -E'NT(()e)) [K(s)) a'), x.:])
M The quadratic form from H'' is (HF()) ha)) (EM VI(() EM= Xx(hha) Using the definitions b, and a from Theorem 3.3 (with Ka K (x mi) and K/4 K( a'), and summation index for data points):
M
6
vi(w) (K(z), Ea (x;, (st, d'))l)
M
C:
Σ
(K(;; - Ba(NN((He)) [K(r,,
Then, MM, V (w)
T> 1 as;; = =(āTb). And, EM (ha())) = T -
15c =
T(a T c). So, the contribution to (hbj( ha ha) from this first part is:
H has ha) 1 (Ta (TTTC) =a (FrbOT N
Contribution from H !! ou ΣM (w)T Cov sx) (x) (K((()))))) -) u. The
quadratic form is ( ha, ha) IN Vi(W)T (Cova' ( (K(ss) a'), ha, ha).
16

---

Published as a conference paper at ICLR 2026
The inner term is Val (K(,, a') d'),)) b, ha)l (o (K(()) x4), K ((s), d'), 2z
ā T where
Σ
Using
K
K (2,, (s), this is So, the contribution to
Cove' Ka K3g matching
Theorem
3.3
(hx)) har ha, ha) from this second part is:
1
v (i ā I
T
N (H ha: ha) N á
Vi(W)) ā = ā
Combining terms for the matrix H: The full quadratic form for the second-oud -oder term in the objective function ja H where the matrix H given by:
T N
T M N
H
be T
This matches Equation 8 in Theorem
3.3
C THE PROOF OF THE CONVERGENCE
C.I PROOF MONTE CARLO CONVERGENCE
Let the Monte Carlo estimates be defined as the average of N independent and identically distributed (i.i.d.) samples, denoted by 9 (6) and ( corresponding trajectories w; p(w; nt)
1
Vn. (1u)
9h
N
N
V2. (ha) = 1
H
N ==1
The true Fréchet derivatives are the expectations of these samples (we use E[-] as shorthand for Eunnp(wss) [:)):
VnJ (hx) Elgn]
J. (ha) E[H.]
Consider the expected squared norm for the first-order derivative estimate. Since the Monte Carlo estimator is unbiased (hv)) = (hw)), the expected squared norm equals the variance:
V (hw) V (h*) ||] Var Var(V
E
The variance of the mean of N random variables is 1/N times the variance of a single variable:
N
Var V(ha)) = Var
N
1
gh
(4)
=
N2
Var(gr (i)
N
Var(gn)
1=1
==1
The variance of yh defined as:
Var (gh) = Ellggh - E[gh]11
Using the property Var(X) E [|| ||[[X]] (which holds according the definition of RKHS) and the fact that |
2 0:
Var(gn) = E[[ gm [[]] B[lle,11
Applying the assumption |[[lgn || 5 o/:
Var(gh)
o
17

---

Published as a conference paper at ICLR 2026
Substituting this back, we find the standard convergence rate for the expected squared norm:
E
(h+) - V V
(bw))l
- o
N
Similarly, for the second-order derivative estimate:
Vi. (ha) V (hx)) 1 = Var( ((hxa)
E
Var Vi Var
1 N
Σ H
N Var(HH)
Var(HA) = 18I1 - E| 11 s B1|
Applying the assumption []][ S of:
Var Hh) S of
Substituting back, the standard convergence rate for the expected squared norm is:
(ma) (ha
of
E
N
This completes the proof.
C.2 PROOF FOR TAYLOR UPPER BOUND
Let Ah h2- - h1 Define the auxiliary function Φ: [0, 1] + R by 6(1) J(hi + -1hh). By the chain rule for Fréchet derivatives:
o'(t) +h Ah)
<'(t) (V, +ah) Ah, Ah)
Using Taylor's theorem with integral remainder for <(t):
(11) = ((0) + (0 + / (1 (1)d
Substituting the expressions for Φ, 6', and d":
((b2) = (h1) + (h,, / (1 () + +ta) o Ah, Ah)dt
We introduce the second-order term h1. Note that So t)dt 1/2. Thus, Ar, Adding and subtracting this term within the integral expression for (hh):
(hi Ad, An) = L (1 (1- (1-1)(V?,(hi) Sh, Ah)dt
J(h2) = J(h)) + (ViJ(hi), (hi)( Ah, Ah)
(1 () (1)) Ah. Ah)dt
(1 ((V Ah. Ah)di
= J(()) + (V.J(M)) (V), o Ah, AM)
+ +1 1) 1-DX()) +( vi.(hh)) Ah, Ah)dt
18

---

Published as a conference paper at ICLR 2026
Let R2 be the remainder term:
R2 L (1 -1-1) (1 + (AA) - vi,((h)) a Al, Ah)dt
Through Cauchy-Schwarr Schwarz inequality, the quadratic form (Au, 2 ||
(1) + (Δ)) - ((,)( Ah, AM) S [h, + (Ah) (
Using the Lipschitz continuity of the Hessian in Assumption 4.1
Jh, + (ah) (bz))l - (h + (ah) - h1ll - L|| = LeIAA||
(since & 2 0)
Substituting this bound into the integral for Rz:
R₂ - 1 (1 +[TН|АМIIIАИ]
Ll|AI| f(01 (1111)ia
|| [ (t - (t-+))dt
L
Substituting this upper bound for R2 back into the expression for (h2) and replacing Alt with
lo2 - h1:
J(h2) S J(h1) (VnJ(hi), Mn) + ! (hi))) (h2 h1), h2 h1d +
6 E11b2 h1 1|
L
This completes the proof.
C.3 PROOF FOR THE UPPER BOUND OF THE STEP NORM
To prove this upper bound, we first need to introduce lemma to show the optimality conditions of the iteration step:
Lemma C.1 (Optimality conditions) Let
{ (h+) (ha), hi) +
1 + [VR (h+) ( o h h, + 3 |b]l
h argmiin
then it satisfies that:
B | Ah. 0 (necessary condition),
VnJ (hx) + (7.j(ha) (hx) Ah +
2
3 (V + | u. 2 vu E Hg (sufficient condition),
where I the identity operator on HK HK + Hk.
Proof: To simplify the notation in the proof, we denote g = vJ(hu) E HK. H v²J(hu) and the objective function M HR + l::
M(h) (g. hi + 5 ((H2 o h)) + 8 |7/
6
The first Fréchet derivative of M given by:
VM = + Hy oh 11 h
Since Ah is minimizer, must satisfy the necessary condition VM (Ah) 0:
+ H o Ah / 11ANIAA= 0.
19

---

Published as a conference paper at ICLR 2026
We now prove the standard second-order necessary condition for Ah.
Let Ah be minimizer of (h) and let - | |1lA Then the operator H2 + 3 |Ahl must be positive semi-definite, i.c..
((llx u, u) 1 ( 2 Hx +
8 IIAN
We proceed by contradiction. Assume that H& + | is not positive semi-definite (H2 S|AAi|T ž 0). This implies that there exists a direction e Hx with associated quadratic form is negative:
||u|| 1 such that its
P = ((H& + [IANIT] o u, u) <0
Consider the Taylor expansion M around the minimizer Ah along the direction u for a small step R. Using Taylor's 's theorem in Hilbert spaces:
( + eu) M (A)) + e <(V u) + _ ((V u, u) O(e²)
Since V.M (A)) = from the first-order condition, this simplifies to:
M(Ah + eu) ((A) + ((V2 u, u) + O(E")
2
The second Fréchet derivative (Hessian operator) of M h is calculated
8 h ② h
v²h H +
|k|lz
1
Evaluating at Ah (assuming Ah + O, which implies - |/ > O; the case Ah 0 requires separate,
simpler verification) and substituting 8
2 || = - || yields:
2 ((A)) H + 3 Ah @ Ah
2 A (1
3
- IAADD)
B
3 Ah & Ah
+
2 |AI||
2 |
Now, substitute this Hessian back into the Taylor expansion. The quadratic term is:
B
((V² M(Ah) u, u) = ((Hx +
[ANIIT) o u. u) + 2|| (1Ah, u),
Using the definition of μ and properties of the inner product, this becomes:
B
(V)M((A)) u, u) 2||A
(Ah, u)²
The Taylor expansion for the difference is thus:
2²
8
M(Ah + Eu) M(AA)
1 +
2 || (Ah,
+ 0(6))
2
B Let K /4 + ||| (Ah 2 By assumption, l < 0. The second term ||| (Ah, u)² is non- negative. Since the assumption H&+ $ must be false. Therefore, must conclude that H4 + A 2 0, which complete the proof.
2 | Y leads to a contradiction in all cases, the assumption
Now we continue the proof for the upper bound of the step norm. Through the Taylor upper bound in lemma 4.2 we know that:
J (h2) - (fi) + (VnJ) h1) 1 (k)) ( (h2 h1), h2 h1) +
I 1 hu
6
For Ah fu h1 that satisfies the optimality conditions, use the necessary condition to establish that:
(122) (h)) + (V)J(hn)) (ho), Ah) + + (hi) vRJ((n)) o (AA), Ah)+
- ||NN| 5 (v.J(..)) (AN), Ah) --1 Ah|3
(11)
20

---

Published as a conference paper at ICLR 2026
For L I B and the sufficient condition in Lemma C.1 could find that
L | - 8 vhi(())) (Ah), Dh) - - |
6
8
Substituting back into the inequality we can find thar:
8 / - J(m) -hh2)
12
(12)
1
+ (VnJ(hi) Vn. (Iea) Ah) + 2
(Ah), Ah)
For the gradient enror term, applying Cauchy-Sehhee Schwarz and Young's inequality (ab I Cia + cb²
with a = || [h+)) Vn,J(h))) b= || | 1. 3/2, g = 3, and 33/6)):
(ha)) [(ax), Aha)
S (has())) - Vn
4v3
3
3/8 [ha(())) V, ((ww))
36
For the Hessian error term, applying generalized Cauchy-Schwarz and Young's inequality (ed <
O2c + edf with [(b,) V (hw)). || p 3, q 3/2, and 3/36);
2 (V) (A/x) Abp)
2
24
(i,((h)) - vi.i(h1)13
||
82
36
Substituting these bounds back into 12
A 3 [h(k) (1b+11
4/3
[V - Jhec(1)) +
3 3
36
24
8
Ahv 113
3a
36
Rearranging terms yields:
8
3
8
||hn|l S J(hi) -[hh++1)
12
36
4/3
24 [[7. vihi(1))1
[nJ((h)) J1a(2)))
3/B
5a
which simplifies to:
8
4/3 V (v. 1 < J(ha) J(b+1)) 3/B
24 vi.((h))
vh+((1))
36
(13)
Now, we take the total expectation E||] over all randomness. Using Lemma C.1 and properties of expectation (Jensen's 's inequality), we bound the expected error terms:
E V J(hn) Jh s E [(hn)
J(h)))))))
3/4
o
3/4
0
3/2
N3/1
o
(See note below)
VN
N3/2
21

---

Published as a conference paper at ICLR 2026
Substituting these bounds into the expectation of
8 E [1
<B(J(ha)]
[[J 13]
36
4v3
24
o
3vB
N3/4
N3/2
Summing this inequality over the total iterations lt ,,...,... M:
8
1
1x119
[B[J(h+)] ((J((++))
36
k=1
M
4v3 00
of
+
3/B N3/4
8a N3/2
The first the right-hand side telescopes to [ (h,1 ) E B[J(mM++a)) }] Assuming h, is determin- istic and (h) 2 J" for some minimum value J", this sum is bounded by ((1) - J The second sum consists of terms independent of the summation index k:
Σ
M
= M (...)=
3/2 4v3
0o
of
3/3 N3/4 B N3/2
1=1
Combining these results:
B
Σ E [1a+11 - hell S J(ha) M
4v3 To
3/2
24 of
36
3/3 N3/4
Ba
Na/2
Let R be a random variable uniformly distributed on (1...., M). such that P(R = k) 1/M.
Then [[]]/R+1
h
IE [|]a++ - //.113.. Dividing the inequality by M: Ein
3 E
hR 13 J(hi) - J
24 of 3/B N3/4
3/2 4433 0
[1
+
36
Ba N3/2
Finally, multiplying by 36/B isolates the expected cubic step norm for a randomly chosen iteration
R:
36(J(h))-JF
36
4v3 do
3/2
24
E [[ hRll3]
+
BM
B
3 3/3 N3/4
B N3/2
36(J(h1) -J")
48 3 do
3/2
864 of
+
(14)
BM
B3/2
33/4
(3 N3/2
We box the final result for emphasis:
36(J(hi) J*)
3/2 48.3
864 of N734
E [1] hall
do
3M
B3/2
B3 N3/22
This completes the derivation of the upper bound on the expected cubic step norm.
C.4 PROOF FOR THE LOWER BOUND OF THE STEP NORM
From the proof of the Taylor upper bound in Appendix C.2 could similarly derive the first-order Taylor upper bound as
(. & VJ(h1) + (Vr)( o (h2 h1), h2 h1) + 2 11/122 - h2113
(15)
22

---

Published as a conference paper at ICLR 2026
Through this, could prove the lower bound by first constructing this auxiliary equation through the optimality conditions Lemma C.I
(li++1)) = V ((k++1) (VnJ(ha) + (hh())) o Ah + B
| (A
2
[vJ(h VJ(ha)) J²)(),) o Ahg]
(Term 1)
+ VJ(hk)) vnh( (Term 2)
+ JVa)( - vi,5(h))) Ahk (Term
|A (Term 4)
Taking norms and applying the triangle inequality:
[VJ(h++])]] V vJ(h))) (2.(( o Ahgll
+ [/v))))) vnJ(ha)]
+ J(hb)((h)) - vi,((ha)) Ahall
B
+1
2
We bound the terms using Assumption 4.1 for Term norm properties for Term and direct calculation for Term 4:
L.
J S ||
VJ(he) Jhuill
_ - -V (L (L+ B)]lAn
11
Applying Young's inequality ab - error:
a² 2c
Cb² +
with L + to the term involving the Hessian
2
J(be) Vh,
2(I.+ 19)
2
Substituting this back and collecting terms with ||
L +
L+ 3
3
J 1
||
2
V22(hk) - Vi.((ba)1
+ [|v](ha) - VaJ(Nll1
2(T. (3)
= 3) |
+l)V [haill [V2((a)) Vi.j(he)11 2(L $)
Now, take the total expectation [[-] Using Lemma C.I and Jensen's inequality:
[hw) [hw]]] S V E J[v vnn(1)))
<
Oa
VN
[hi) (ini(1 - of
N
Applying expectation to the inequality for || V (hb+1)]
[|( < (T. + (3) [E[I [1
E [ - vi.((nn)1
E [ J(u(ll (I+ 11
2(L + 3)
of
VN
2N 3)
23

---

Published as a conference paper at ICLR 2026
Rearranging to isolate the expected squared step norm:
of
(L + 3) |||| R I2 2 |l| V (h
d0
VN
2 + 3)
of
B1 1x L+
E| [
00
VN
2N + +
This completes the proof for the lower bound based on the gradient norm, using the corrected regularization term.
C.5
PROOF FOR CONVERGENCE THROREM
From Lemma
4.3 the upper bound on the expected cubic step norm for R Uniform [1, ,.M is:
E [[|4++ hall"] 36(J(h)) J*)
48v3 do +
3/2
864 of
3M
68/2 N3/4
53
N3/22
As the number of iterations M + oo and the batch size N + cc, the right-hand side approaches zero. Thus,
lim E [[]hR+1 - hRll"] = 0. Mo
(16)
Using Lyapunov's 's inequality, h hR|l? & (Bil h [all91)2/33 Taking the limit as M, N and using 16
lim E [1]b++1 - hnll 0. M.N
(17)
From Lemma
4.4 rearrange the inequality which holds for any iteration k:
of
V J(x+1)ll & (L 3) - hall
ao VN
2N
Now, take the expectation the random index R ~ Uniform 11, ................................ M). Since R selects one of the iterations E {1,...,M} uniformly, taking the expectation of the inequality with respect to R effectively averages it:
ER [ < ER (L + h) hRll
2N + B)
Here, ER denotes the expectation the choice of R. Let TE[-] denote the total expectation (over the process history and R). The inequality becomes:
E J((n++)))l s (L + B)][|lbr+r hnll31 do of
VN
+ 2N
Taking the limit as M + oo and + oc:
of
(L B)BB111++1+11111111 hhal
lim [l]]v](h++)]ll N
lim M.N
+
VN
2N (L + 3)
of
(I. +1++3) 3)
lim
B[ - hnll
lim
ou
+ lim
M.N -c
N
VN
No00
2N + B
(T. + 3) x 0 0+0+0 (Using 17
0.
Since E[] 2 0, we conclude that:
lim E 0. M.N
As E [(())) differs from E[ V by terms that vanish M oo (typically 1
M E
we can equivalently state:
lim M
E [ 0.
This proves that the expected gradient norm randomly chosen iteration converges
24

---

Published as a conference paper at ICLR 2026
D PROOF FOR QUADRATIC CON CONVERGENCE
Let the error at iteration i be eg = hig - h² The update gives hg+1 h² h2 h- A so.
ex+1 + Ahg. The update step satisfies the optimal condition C.1
which is given by:
VJ(hx) + V2 (ha) Sh + 2 ||
Ahg =
Rearranging yields:
(V² + 2 Ah A/e = V((())
where I is the identity operator.
We expand (hx) around h* using Taylor's theorem (similar to the derivation in Appendix C.2 (ha)) VJ(h')+ ++'(")+ (hy- h²) + R2 (hy; h²)
where VJ(h") = 0 and the remainder term satisfies [| h h+]l S t l1 10
flexx 12
for
some constant y when hy is near ht" Thus,
J(hh)) = J((")) o + (1)x,111
Substitute this into the rearranged update equation:
(ha) + 2 || IZ) o A = -V v2 ((h"))))) ex - Ri(hx,
Let H4 V2 ((kx) and 11 (h"). Let A₁ H2 +
AhalII. The equation is A Ahe
-H" o cg - (h,,, h*). Assuming the norm of the inverse operator ||4 'lop will be bounded by
some constant B, and we assume that the update step is sufficiently small that |Ah/l S Lllexll-
Now, substitute Ahs = ex+1 - Ck into the equation A a Ahe
H o ock R₂ (hah,"))
As (Ex+1 ex) - H ek
R1 (hg,, h²)
A₁ o Ck+1 = AA o Ex H' o - Rr(hx,h")
3
A o ek+1 = =HH + 2
A1x II -H*) oex - Ri(h,,
3 |halllee R2 h²)
A& o ck+1 (Hę H o ex +
Applying the inverse Ar
3 1 - RRines h))
Ch- = Ao1o (H - H*) o +
2
Taking norms and using the triangle inequality:
3 ? Ahullee (iabsshh))
|ex+1|| $ 1
(HI 11*) ex
3 + Alllllllessl + (| (1z, ht]ll -
< 2 Plop
||Hss Hollllle
2
Substitute the bounds: || 11 llop [Rr(h, h²'ll < |llex 1|2.
S B, |
H* - T.llexll. ||Ablll s T.llexll. and
+ _ [I]lex1)\leall + -lealle))
|ex+1
S B
(L]lex
< B
L.[ex 1|2 +
3L
|exl1
L
[|eal1?
2
3L + ()
< B
L+ +
||
Setting C, = B(L + 8L ) which is a positive constant independent of k. we have shown that
|| h1 C2 1=112
This demonstrates local quadratic convergence for the deterministic version of the algorithm, provided hy is sufficiently close h²*
25

---

Published as a conference paper at ICLR 2026
E VALIDITY OF THE ASSUMPTIONS
E.1 VALIDITY OF ASSUMPTION 4.1 (LIPSCHITZ CONTINUITY OF THE HESSIAN)
Claim. Under mild regularity conditions standard in second-order analysis Nesterov & Polyak
2006b; Nocedal & Wright 2006b) and Lipschitz MDPs (Pirotta et al. 2015) y the RKHS Hessian operator Vi is Lipschitz on neighborhood N C Hx; L.e., there exists > 0 such that
J(.)) - (h2a)lo S L
- hell V2 E..
Sufficient conditions. It suffices that the following hold:
1. Bounded horizon and rewards. Finite horizon T < oo (as our experiments) or discounted infinite horizon with 7 < 1, and [r(s, a)l & Rmaxx
2. Lipschitz MDP. The transition and reward are Lipschitz in metric d (cf. (Pirotta al. 2015)):
a), (() L, d((s, a), (s',
< (s)((s) and (r(s,a) r(s''a')) s
3. Kernel boundedness and smoothness. The kernel sections satisfy [[K((s, a), S x and the map (s, a) - ((ss,)),.) is Lipschitz in RKHS norm with constant T.k.
4. Softmax RKHS policy with Lipschitz log-policy. For Ty (a|s) o exp(Th(s, a)) with finite T and h restricted to ||l]| < B, the log-policy and its first two Fréchet derivatives are Lipschitz in h.
5. Uniform integrability. The trajectory weights Vr (w) admit finite moments ensuring inter- change of expectation and differentiation (dominated convergence).
Sketch. By Lemma 3.2,
(h)(h)= = [2 Me Vn logTh & Er V log log -B Σ ue Cov Cova'Nta(la)) ((5,, a'),
For the outer-product term, VA lo (als) T(K((s,a)::)- Eag [K(ss, a'), ))). Because (s, a) + ((ss,a).)) is Lipschitz (Assumption 3) and + Th is smooth and Lipschitz on |||||| < B. the map h VA log TT is Lipsehitz; bilinearity of the outer product then yields Lipschitz bound on the first expectation. For the covariance term, the maps h + Ea'Nay [K((s, a²), and h h+ Ba'sor, [K((s, d'), -)) K((s,a'), a)) are Lipschitz by the same reasoning, hence h (s)
Cov [K((s, a') ), Lipschitz. Multiplying by bounded | and taking expectations over huuned-hhoriznng (or discounted) trajectories preserves Lipschitzness. Therefore there exists
ir |. This mirrors the sypschitz-gradicee results for value functions in Lipschitz MDPs (Pirotta et al. (2055) and the smoothness assumptions used in
such that [hr,) Vi, J(h2))) llop L |hi
cubic-regularizeedNNewoo methods (Nesterov & Polyak 2006b Nocedal & Wright 2006b).
E.2 VALIDITY OF THE ASSUMPTIONS IN THEOREM 4.6 (LoCaL QUADRATIC CONVERGENCE)
The theorem rests on three local assumptions. Below we give standard sufficient conditions and why they hold in our setting.
B.1 Bounded inverse of the regularized Hessian. We require
[(b) + ||)
If ht" is strict local minimizer, then H [h, is positive definite. By continuity (Assump- tion 4.1), there exists a neighborhood V that Amin (h) + [| I) 2 μ. so the inverse is uniformly bounded by B < 1/u. The cubic regularizer only enlarges the spectrum, standard safeguard in cubic-regularized Newton analyses (Nesterov & Polyak] 2006b Nocedal & Wrright 2006b).
26

---

Published as a conference paper at ICLR 2026
B.2 Step error proportionality || I K |exll- Let h--h". The step solves (hs))
1 I) Ahk (hg)) By Taylor's theorem with Lipschitz Hessian (Assumption 4.1), (ha) ((h")) + 7% with ||rll S cllex|1". Multiplying by the bounded inverse from B.1
gives
| $ || (I) || + ||r=ll) s B(]) + ellex [exl S K|
for |ex/l small enough, establishing the proportionality in the local region.
B.3 Initialization inside the local basin. Quadratic convergence for Newton-type methods is inherently local: starting sufficiently close to h" ensures the Newton map is a contraction and that iterates remain in N. This ansin-of-attraction requirement is standard (Nocedal & Wright (006b) and is precisely the regime where cubic regularization attains its classical rates (Nesterov & Polyak 2006b).
B.4 Deterministic scope of Section 4.3. Rates for stochastic Newton-type methods depend on curvature, noise, and step-sizz size policies and typically require separate concentration and bias variance controls; see, e.g.. (Boyer & Modichon-Baggione 2023 Bottou et al. 2018). Our Section 4.3 therefore establishes the baseline local quadratic rate in the deterministic setting-the hallmark behavior of Newton's 's method methood-rovvidnng principled rationale for second-order approach to RKHS policies and foundation for future stochastic analysis.
F ASSET ALLOCATION EXPERIMENT
our investment planning MDP, we formulate a state-action framework that models investment decisions under varying market conditions and resource constraints. This model captures the funda- mental trade-offs between risk and return across different market states while accounting for resource dynamics.
In
The state E S is characterized by a tuple (r, m) where:
r E {0, 1,..., Rmaz - 1} represents the discrete resource level, with Rmar being the maximum possible resource level
E {0, 1, corresponds to market conditions (recession, stability, and prosperity, respec- tively)
The cardinality of the state space is |S| Rynax 3.
The action space A comprises three distinct investment strategies:
a Conservative investment (low risk/low return)
Balanced investment (moderate risk/moderate return)
2: Aggressive investment (high risk/high return)
The state transition function P(se+1 | St, 0z) models the stochastic evolution of both resource levels and market conditions:
Resource Dynamics: The probability of resource level transitions depends on the chosen action:
Conservative strategy:
P(reatre: 0) = [0.1, 0.1, 0.0] for Ar
(-1,0,+1,++
Balanced strategy: +,, +3}
P(retara, = 1) = (02,00..004.022.000 for Ar E
Aggressive strategy: P(realr,, = 2) [0.4.0.1,011,0.22002) for Ar (11,0,+1,+2, +3}
2. Market Dynamics: Market state transitions follow a Markov chain with the following probabilities:
Recession: P(m Im, 0) 0.3,, for m4+1 (0,1,2)
27

---

Published as a conference paper at ICLR 2026
Stability: P(me+1|m 1) [0.3, for m411 E {0.1,2)
Prosperity: Pmm 2) [0.1, 0.6] for m,+1 (0,
The joint transition probability is computed as:
([(r+t,me++)le mz), 4+) = P(retalre, 0z) P(m++1lme)
The reward function r(st, oz) captures the expected immediate return for taking action dj in state 8, (ra, m+):
Ti ++1
(e)r,))).))) B(m,, az) Rmaz
where B(m,, a+) is the base reward that depends on the market state and chosen action:
Conservative strategy (a, 0):
Base reward = 1.0 for all market states, except
Base reward = 0.5 in prosperity (m, = 2) to represent opportunity cost
Balanced strategy (a, 1):
Base reward- 0.5 in recession (m, = 0)
Base reward 2.0 in stability (m, 1)
Base reward 1.5 in prosperity (m, = 2)
Aggressive strategy (a. 2):
Base reward 11.0 in recession (m: 0)
Base reward 1.0 in stability (m, 1)
Base reward 3.0 in prosperity (m+ = 2)
The resource scaling factor r++1 ensures that higher resource levels amplify rewards.
Rmux
The initial state distribution p(so) typically set to start with a medium resource level and randomly selected market state:
p((ro [Rmea/2], mg m)) = for m E (0, 1,2
In our experiment, Rmar is set as 5, balancing sufficient environmental complexity with simplicity for visualization and optimal policy calculation.
G VISUALIZATION OF POLICY LANDSCAPES
To visualize the optimization behavior of different algorithms in Figure 1(b), we project the high- dimensional policy space onto two-dimensional plane and overlay an approximate value landscape and the optimization trajectories.
PCA projection of policy space. Let e(m)) IR denote the flattened policy parameter vector of method m at iteration t, and let 0 be the analytically known optimal policy in the toy asset allocation setting. We collect all visited policies across all methods,
D D= (0 Vm, Vt) {0*).
and form the data matrix X E N with = (D]. We apply PCA to X, obtain the first two principal components WT, W2 D and project any policy 0 to
2 w (0 0), w (0 0) E E
where 0 is the empirical mean of D. This 2D plane captures the dominant variation of all policies and makes their relative positions (initial policies, intermediate iterates, and the optimum) directly comparable.
28

---

Published as a conference paper at ICLR 2026
Value surface reconstruction. Let Z; z(0,) be the 2D projection of policy 0; D and 0/ J(0;) its corresponding expected return. We construct dense grid over the convex hull of {zi) and approximate the value landscape by interpolating the pairs {(z, 2 using standard scattered-data interpolation (cubic interpolation when possible, and linear interpolation in sparse regions). This yields smooth surrogate J(z) whose level sets define the background contour in Figure ((b).
Trajectory smoothing. For each method m, the projected iterates (z(0(=))) )T= are connected into smooth curve using B-spline interpolation with small smoothness penalty. This reduces visual clutter due to small step-to-step fluctuations while preserving the global optimization trend towards z(0"), which is marked as the optimal policy in Figure ((b).
Additional high-contrast visualization. The main-panel Figure 1(b) was primarily designed to highlight the geometric differences between optimization trajectories and their distances to the optimal policy in the PCA-projected space. As noted by the reviewer, reward differences in the high-return region can appear visually subtle due the relatively flat landscape around the optimum. To make these differences more apparent, we provide in Figure an additional visualization with an adjusted color scale (and identical trajectories), which enhances the contrast of the reward values while keeping the underlying optimization paths unchanged.
Policy Gradient
Pollcy Gradient
Policy Newton
Policy Newton
Pollcy Gradient in RKHS
Policy Gradient in RKH5 end
Policy space PCA dimension 2
1.0
Policy Newton in RKHS
A
The start policy for methods
Newton
Cumulated reward ((m)
Optimal palicy
0.5
-1.0
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Policy space PCA dimension
Figure 3: Alternttvve visualization of the policy landscape with enhanced reward contrast. The optimization trajectories are identical to Figure 1(b); only the color scale of the value surface is adjusted.
H OPTIMIZATION DETAILS
To solve the optimization problem formulated in Equation 7 we implemented a conjugate gradient optimization framework based on the Newton-CG method. This approach combines the second-ord -oddr convergence properties of Newton's method with the computational efficiency of the conjugate gradient algorithm, making particularly suitable for our problem where the dimensionality of the Hessian matrix H scales with the volume of trajectory data N T.
29

---

Published as a conference paper at ICLR 2026
Specifically, we address the following optimization problem:
argmin
(u, a + (Hā a) + |al
GERNT
This optimization problem incorporates a linear term (u, @.) a quadratic term 1 (Hā, ã), and a cubic 2. The objective function and its gradient computed
regularization term f/2
f(ā) = (v, α)) + (Hā, ā 6 I
Vf(a) = u + Ha + a
In each Newton iteration, determine the search direction by solving the linear system (H ||a| (()).. The conjugate gradient method employed to efficiently solve this linear system, avoiding the high computational cost of directly computing (Hf + - = This method constructs set of conjugate directions {P:} and progressively approximates the optimal solution through orthogonal projections. The algorithm proceeds as follows:
1. Initialize residual ro (()( (v + Hōo |0 and initial search direction
= rp
2. For each iteration k:
I Tr
Compute optimal step size 0%k =
P/ [M+ [x))
Update solution A Aăæ + aiPb
Update residual Tk+1 = rи og H + 2 c T)ps:
Calculate conjugate direction update coefficient Bg 2e 7241
Update search direction Pk+1 T + 8ыPи
In our implementation, we utilized the minimi ze function from the SciPy optimization library, configured with the 'Newton-CG' method. To balance optimization accuracy and computational efficiency, we set the convergence tolerance to 10-3 and the maximum number of iterations to 500.
H.1 COMPUTATIONAL COST ANALYSIS
To address the practical concerns regarding the computational overhead of second-order methods, we present a comparison of the training time (wall-clock time) for the proposed method and baselines.
All experiments reported in this subsection were conducted the CartPole environment. The total training duration corresponds to 1.2 x 107 training steps for each method. All runs were executed on the same hardware infrastructure (Intel Xeon Gold 5218 CPU) to ensure a fair comparison.
Table I
summarizes the total wall-clock time required. We compare our Policy Newton in RKHS against first-order RKHS methods, as well as parametric methods with varying model complexities (Polynomial features of degree and 3).
Analysis. The results indicate that the Policy Newton RKHS requires approximately 1.97 the computation time of its first-order counterpart (Policy Gradient in RKHS). This overhead is primarily attributed to the construction of the Hessian operator and the Conjugate Gradient (CG) iterations required to solve the Newton step.
However, it is crucial to interpret this cost in the context of sample efficiency:
Relative Overhead: The 2x cost factor is consistent with the overhead observed in parametric second-order methods (e.g., Policy Newton Poly-3 vs. Gradient Poly-3), suggesting that the RKHS formulation does not introduce disproportionate computational burdens.
Thus, while computationally more intensive per update, Policy Newton in RKHS remains practically feasible for standard RI. benchmarks.
30

---

Published as a conference paper at ICLR 2026
Table Comparison of total training time (in minutes) for 1.2 x 107 training steps on the CartPole task. The RKHS methods utilize non-parametric representation scaling with sample size. Poly-1 and Poly-3 denote parametric policies using polynomial feature representations of degree and degree 3, respectively. We note that the degree 3 is used in the experiment of the main paper.
Method
Optimization Type
Time (min)
Policy Newton in RKHS (Ours)
Second-Order (RKHS)
43.5
Policy Gradient in RKHS
First-Order (RKHS)
22.1
Policy Newton (Poly 3)
Second-Order (Parametric)
18.8
Policy Gradient (Poly 3)
First-Order (Parametric)
9.7
Policy Newton (Poly 1)
Second-Order (Parametric)
16.3
Policy Gradient (Poly 1)
First-Order (Parametric)
8.5
I EXTENSION CONTINUOUS ACTION SPACES
In the main text, we focused discrete action spaces establish the theoretical foundation of Policy Newton in RKHS. In this appendix, we demonstrate that our framework naturally extends to continuous action spaces. Specifically, derive the second-order optimization steps for Gaussian policy parameterized by a RKHS function.
1.1 GAUSSIAN POLICY IN RKHS
We consider a continuous action space A IRd The policy is modeled multivariate Gaussian distribution with fixed covariance matrix E E Rdddd and deteedeepndnnt mean ju(s) represented by a function h in a Reproducing Kernel Hilbert Space (RKHS) H The policy is defined as:
(a 8) (2m)d
1
exp (- h(s)) ta (a-(ss))
(ahss)))
(18)
Here, h = [hi...., ha] where each component h2 E HK is a function in scalar RKHS associated with kernel K(,,), )). The reproducing property implies h(s) (h, K 2),, (component-wise).
The log-policy is given by:
log (a 8) - (h-))))) h(ss) (-)sss)) +C,
-
(19)
where a constant independent of
I.2 FRÉCHET DERIVATIVES IN RKHS
We derive the Fréchet derivatives of the expected return J(m) with respect the function h.
First-Order Derivative. The gradient of the log-policy with respect to h is simply the kernel function scaled by the score vector (Zhang al. 2025
VA log (a 8)) = K(s,) [ '( M(s)] H
(20)
Note that this standard element in the RKHS. Consequently, the Policy Gradient in RKHS is:
(AT( = (EuNp(WSsTa)
Σ
(((W) [E - -(ss)))]]
(21)
Second-Order Derivative (Hessian Operator). Differentiating the log-policy again yields the Hessian operator. The Gauss-Newton approximation (outer product of gradients) involves the tensor product of the gradient element with itself. The curvature term arises from the second derivative of the log- likeliiod.ood. Specifically, the Hessian of the log pog-poliy is the operator:
V log (a s)) (K(s,))E-)) )) -)
K (s..).
(22)
31

---

Published as a conference paper at ICLR 2026
This notation denotes operator T Hi + H% such that for any E Hk. Tu (s,-) (K(,) u) ((s u()]]
The full Hessian of the objective JTT is thus:
(Tn) = (uwp((;TA)
Vn log X
Σ 1'=0
VA log mh
1=0
(23)
Ve(w) (-(K -
K(ss.)))
1.3 FINITE-DIMENSIONAL REDUCTION
By applying the Representer Theorem, seek the update step in the form:
Ah() Σ K(m,))an
(24)
where xy = (8), 0z) corresponds to the 1-th sample in the dataset, and wi E Rd are coefficient vectors. We define the full coefficient vector RRd by stacking w1..... WNT-
Theorem L.1 For a continuous Gaussian policy, the optimization of the Policy Newton step RKHS is equivalent to minimizing:
L(o)= = (v. α) + (Hā, ā) 8 |
(25)
The vector NTd is composed of blocks Rd (i 1. N...
U
(((w) 8:)= h(si))
(26)
The matrix E NTáxNNT is given by:
NT H Σ Vi(w) GG @ (8,0) - )-
(27)
where 8, E- ((ss)) Rdd and G kik, E RNNTNNT is the outer product of the kernel K (s), sNT)] The Kronecker product A B here assumes the standard block layout where A (the kernel matrix part) dictates the block structure and B (the action dimension part) dictates the content of each block.
column vector Kj
LLM USAGE DISCLOSURE
We used large language model solely for writing polish. Its assistance was limited grammar and style edits, wording suggestions for ttules/bstract/caapions consistency of terminology, and minor LaTeX phrasing (e.g., figure/table captions and cross-reference text).
32


Reviews:
[{"review_id": "AYH7sf7l2j", "reviewer": "Reviewer_Z1vx", "summary": "This paper presents a policy iteration algorithm for reinforcement learning problems where the policies are formulated directly as elements of a reproducing kernel Hilbert space (RKHS). The method extends second-order optimisation algorithms to the RKHS setting by deriving computationally tractable approximations to the Hessian and the resulting optimal step direction. Theoretical guarantees are provided regarding the approximation error and convergence to an optimal solution, and experiments complement the theoretical results with demonstrations in practical settings, where the algorithm achieves superior performance in contrast to first-order methods and parametric policy iteration approaches.", "questions": "* Was the objective $J(\\pi)$ supposed to be written as the negative cumulative reward? How do you ensure the Newton step is leading in a direction that maximises the expected cumulative reward?\n* Is there an alternative reference for the outer product kernel in Definition 3.1? Kubrusly and Vieira (2008) only introduce tensor products between general Hilbert spaces, not particularly RKHSs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Paper is well written and follows a clear structure.\n* Rigorous theoretical analysis with resulting guarantees.\n* Experimental evaluations show significant performance improvements.", "weaknesses": "* A minimisation problem over $J(\\pi_\\theta)$ is introduced in Sec. 2.1. Yet, $J$ is formulated as the expected cumulative reward, which an agent should be seeking to maximise, instead of minimise. The result of the regularised Newton step in Eq. 5 also seems to be leading in a descent, instead of ascent, direction.\n* Experimental evaluation is limited to a toy experiment and relatively simple classic RL problems (e.g., CartPole).\n* Notation for temperature and trajectories set use the same symbol $\\mathcal{T}$.\n* Non-standard notation for gradient term in first expectation Eq. 4.\n* The kernel for the numerical experiments is not specified. Was it a standard Gaussian or Matern kernel? The specific details should be stated in the paper, or at least in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NNmg6RMjkS", "reviewer": "Reviewer_6Lar", "summary": "This paper studies second-order policy optimization (Newton) when the policy is modelled by a function belonging to a RKHS. It introduces a tractable method for computing (matrix-vector products with) the inverse Hessian operator, which is infinite-dimensional due to the RKHS assumption. It proves (quadratic) convergence rates for the proposed algorithm, and provides a few empirical evaluations.", "questions": "- Why regularisation is cubic instead of quadratic in equation (3)?\n\n- How does RKHS Policy Newton do in wall clock time? (Figure 1 and 2)\n\n- How do RKHS policy methods do when state and/or actions are high-dimensional?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is written clearly, and seems to be correct. \nResults are novel and interesting, the proposed method may have a strong impact.", "weaknesses": "I do not see any major weaknesses, however I can highlight a couple of minor issues:\n\n- Section 4.3 seems unnecessary, it’s just a re-statement of known results about convergence rate of Newton’s method on strongly convex losses. Perhaps this space could be used instead to extend Section 3, which represents the main contribution and it’s not very easy to grasp. \n\n- Line 94: “The objective of RL is to minimize…” It should be “maximize”. Similarly, the following equation should be “argmax”, not “argmin”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8zKcf5ApMv", "reviewer": "Reviewer_ZjiM", "summary": "The paper proposes a method for utilizing second-order optimization for policies represented in Reproducing Kernel Hilbert Spaces (RKHS). Direct second-order optimization is not feasible due to the infinite-dimensional Hessian operator in RKHS. Hence, the authors introduce a finite-dimensional optimization problem, whose solution is equivalent to the Newton step. The authors compare their method to the vanilla Policy Gradient and second-order Policy Newton method, as well as the Policy Gradient in RKHS, demonstrating faster convergence (in terms of training iterations).", "questions": "1. The introduction is relatively vague about the advantages of RKHS policy representations, simply stating that RKHS \"offer a powerful non-parametric alternative, [...], valued for its representational flexibility, potential for improved sample efficiency, and capacity for dynamic adjustment during learning\". Perhaps the introduction could be more explicit about what makes this representation more suitable, and in which kinds of tasks might benefit the most from these representations.\n\n2. The description of plot 1b is too short. What exactly does the plot visualize? How is the PCA reduction done? There is no interpretation of the results. Also, the difference in reward between the optimal policy and suboptimal points is hard to assess, as large parts of the plot seem to have more or less the same color.\n\n3. Lines 435-436 state that the \"policy optimization in RKHS effectively leverages infinite-dimensional feature representations, enabling the optimization process to escape local optima\". How does the feature representation help with escaping local optima?\n\n4. What are the shaded areas in Figure 2?\n\n5. Line 477 states that \"Policy Newton in RKHS achieved significantly faster convergence to superior episodic rewards compared to first-order and parameteric Newton baselines\", but some of the baselines in Figures 2(a) and (b) did not converge yet, so from the plot, it is not clear whether Policy Newton actually converges to superior episodic rewards.\n\nComments:\n\n1. Specifying the training progress in \"training iterations\" in Figures 1 and 2 makes it hard to compare the convergence speed of the methods to other RL algorithms, consider changing the x-axis labels to environment steps.\n\n2. Line 101 cites Maniyar et al. for the policy gradient method. The method, however, goes back to [1], which is not cited here.\n\n[1] Ronald J. Williams \"Simple statistical gradient-following algorithms for connectionist reinforcement learning.\" Machine learning 8.3 (1992).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors provide extensive theory for their method, proving a quadratic convergence rate. The empirical evaluation results reflect the superior convergence rate.", "weaknesses": "1. The authors evaluate their method only on three tasks, which are all very low-dimensional, discrete, and relatively simple. I encourage the authors to add more tasks to the evaluation. Is the method also applicable to continuous control tasks?\n\n2. While the proposed method achieves the highest reward out of the methods compared, it still does not seem to solve the LunarLander task consistently (the Gymnasium documentation specifies a reward threshold of 200 for an episode to be considered solved). Furthermore, all of the progress of the RKHS methods in the LunarLander task seems to happen in the first couple of iterations, which are not shown in the plot. For the rest of the training, the performance stagnates. What might be preventing the method from learning to solve the task?\n\n3. The main drawbacks of second-order methods are the increased computation time and the limited scalability to larger models. The paper is lacking an evaluation of the computation time compared to first-order optimization. Furthermore, a comparison of the computation required for different policy sizes would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)