Integral Performance Approximation for Continuous-Time Reinforcement Learning Control
Authors: Brent A. Wallace
Jennie Si
Venue: ICLR 2025 Poster (2025)
Keywords: Continuous-Time Reinforcement Learning (CT-RL)
Optimal Control
Integral Performance Approximation (IPA)
Adaptive/Approximate Dynamic Programming (ADP)
Flight Control
Hypersonic Vehicles (HSVs)
Primary area: reinforcement learning

Abstract: We introduce integral performance approximation (IPA), a new continuous-time reinforcement learning (CT-RL) control method. It leverages an affine nonlinear dynamic model, which partially captures the dynamics of the physical environment, alongside state-action trajectory data to enable optimal control with great data efficiency and robust control performance. Utilizing Kleinman algorithm structures allows IPA to provide theoretical guarantees of learning convergence, solution optimality, and closed-loop stability. Furthermore, we demonstrate the effectiveness of IPA on three CT-RL environments including hypersonic vehicle (HSV) control, which has additional challenges caused by unstable and nonminimum phase dynamics. As a result, we demonstrate that the IPA method leads to new, SOTA control design and performance in CT-RL.

Published as a conference paper at ICLR 2025
INTEGRAL PERFORMANCE APPROXIMATION CONTINUOUS-TIN REINFORCEMENT LEARNING
FOR
CONTROL
Brent A. Wallace & Jennie Si
Department of Electrical Engineering Arizona State University
Tempe, AZ, USA [bawal 1a2,, s4 Basu)aasueee
ABSTRACT
We introduce integral performance approximation (IPA), new continuous-tima reinforcement learning (CT-RL) control method. It leverages an affine nonlinear dynamic model, which partially captures the dynamics of the physical environ- ment, alongside state-action trajectory data to enable optimal control with great data efficiency and robust control performance. Utilizing Kleinman algorithm structures allows IPA to provide theoretical guarantees of learning convergence, solution optimality, and closed-loop stability. Furthermore, we demonstrate the effectiveness of IPA on three CT-RL environments including hypersonic vehicle (HSV) control, which has additional challenges caused by unstable and nonmini- mum phase dynamics. As result, we demonstrate that the IPA method leads to new, SOTA control design and performance in CT-RL.
INTRODUCTION
Many important applications such as flight control (Stengel 2022). robotics (Craig 2005 process control (Morari & Zafiriou| [199), and waste water treatment (Yang et al., 2022) are inherently continuous-time in their dynamics and require real-time high performance controls. However, an accurate dynamic model required in the traditional classical control methods. This poses great challenges, for example, in the HSV context, for which aeroproppolsive/arrelssiii effects are ex- ceptionally difficult to model (Bolender & Doman 2006b Dickeson et al., 2009b). This modeling challenge extends to almost all control applications, feasible. Reinforcement learning control designs have provided a new way of circumventing this stringent requirement, as RL can learn from data to accommodate unmodeled dynamics.
modeling complex systems perfectly is not
Currently, there are two major classes algorithms for continuous time RL (CT-RL). The SOTA deep RL (DRL. Fitted Value Iteration (FVT) methods (Lutter al. 2021; 2023b), which have achieved some of the greatest empirical successes in CT-RI to-date, require data from over 5,000,000 simulations to solve the simple inverted pendulum control task (cf. Section 5 For mission-critical applications like HSVs, achieving millions of test flights is impossible typically only few test flights can be conducted. Thus, the large data requirements of deep CT-RL methods presents great challenges to many real-world applications where simulation data from hardware is expensive collect and limited in quantity. The adaptive dynamic programming (ADP) methods represented by seminal works of (Vrabie & Lewis, 2009) Vanvoudakis & Lewis, 2010 Jiang & Jiang 2014 Yang et al. 2024) (ef. Appendix [M have achieved promising theoretical results and tracted substantial research attention and body of work since about fifteen years ago. Yet, few ADP works have resulted in meaningful controllers, as evaluations of those methods have been limited to simple toy problems with closed-form solutions known a priori (Wallace & Si 2024).
Even though discrete- -imee (DT) RL algorithms (Sutton & Barto
2018 Lewis et al. 2012b Si et al.
2004 Bertsekas 2017) have demonstrated extensive theoretical guarantees and demonstrations in applications, CT-RL algorithms have only developed handful of results. One approach to CT-RL discretizes the CT environment and/or value integral in order approximate CT problem by DT

---

Published as a conference paper at ICLR 2025
one (Kim et al. 2021 Yildiz et al. 2021). Discretization of CT environments, however, may cause acute numerical issues for weal- systems, especially ones with complex dynamics like the HSV (Chen & Francis, 1995). Even if discretization can be formed, this casts the problem into com- pletely different class of DT control problem. The CT HJB equation (a first-order nonlinear PDF) not equivalent to the DT Bellman equation (a difference equation). For additional discussion, (Wallace & Si [2044] Cao & Pan 2024). Another approach resorts to LQR/linearization to eliminate the nonlinearity challenge, as in seminal works (Bradtke et al., 1994) and subsequent ADP (Jiang & Jiang, 2012 Tha et al. 2019) and Q Q-learning- based methods (Possieri & Sassano 2022). A third approach addresses CT-RL problems under general nonlinear (non-afine) dynamics (Yildiz et al.) Problerss
2 (Sandoval et al., 2023). ). However, fully-nonlinear algorithms are at an early stage. The exist- ing evaluations study simple cart/pendulum variant systems and second-order academic examples. Comprehensive theoretical results and meaningful designs without stringent assumptions are yet to be developed. Perhaps the most studied fourth approach deals with affine nonlinear systems, includ- ing the aforementioned ADP works and deep CT-RL continuous fitted value iteration (cFVI) (Lutter) et al. ) and robust FVI (rFVT) (Lutter et al.) 2023b). While both ADP works and FVIs require an affine-nonlinear model, the FVIs have demonstrated learning with low variance and meaningful control performance, thus standing as SOTA in CT-RL.
From an applications standpoint of CT-RL, take HSVs as an example. CT-RL methods have been developed for HSVs, yet these studies are of limited scope as they do not address the model un- certainty of HSVs and consequently their controls. For instance, Zhao et al. (2023) develops a composite RL/observer-base simplified approximation of the standard Wang and Stengel model (Wang & Stengel, 2000 Marrison & Stengel 1998 Shaughnessy et al. 1990) in which Mach dependencies are neglected, significant limitation in the high-Mach hypersonic regime. The neural control methods in (Xu al.] 2013 2015) use this same simplified model. Along similar vein, the developed stability results of many RL HSV works (Zhao et al. 2023 Xu et al., 2013 2015 Bu et al.) 2019; Bu & Qi) [022]] Qiao et al.] 2019) which are essential in flight control applications require that multiple complex stability inequalities hold simultancously along trajectories, and no constructive method is provided for ensuring that the inequalities met. Furthermore, these RL-based HSV control works do not provide substantive generalization studies to unmodeled dynamics.
attitude control method for HSVs. However, the HSV model used
A realistic HSV control problem is challenging, as these nigh-performance aircraft fly 20 times faster than commercial jets. Their long shape and rearward-set propulsion systems give HSVs a nose-up pitch instability (placing hard lower bounds on how "slow" the HSV can be controlled). Meanwhile, the parasitic coupling from upward deflections of the tail causing a temporary dip in altitude results in nonminimum phase behavior (placing hard upper bounds on how "fast" the HSV can be controlled) (Bolender & Doman (2066a). The combination of these two constraints makes HSV control exceptionally challenging (Rodriguez et al. 2008).
Contributions. We propose new based CT-RL learning method that leads to the following three contributions: 1) Our novel IPA CT-RL design approach takes advantage of an affine nonlinear dynamic model of the environment and quadratic cost performance structure, thus to exploit the Kleinman's solution framework. By using state-action data-driven learning, we address unmodeled dynamics with great data efficiency and robust control performance. 2) We provide theoretical guarantees of IPA CT-RL learning convergence, solution optimality, and system stability alongside comprehensive evaluations and comparisons to demonstrate IPA-enabled SOTA CT-RL results in optimal control problems. 3) We demonstrate, perhaps for the first time, that a CT-RL method (IPA) has successfully addressed the challenging optimal control of HSVs.
2
METHOD
The IPA method takes advantage of an affine nonlinear model of the environment, which is usually not an accurate representation of the physical environment, and utilizes state-action data with learn ing to accommodate uncertainties. It is therefore an organic integration of classical control design principles and RL for adaptive optimal control. We assume an affine nonlinear environment of form:
t = f(x) + g(z)u,
(1)
where x E R" is the state, u E R' is the control, f : R" > R", and g R" + paxm As standard, we assume f and g are Lipschitz compact set Z C RD" containing the origin in
2

---

Published as a conference paper at ICLR 2025
its interior, and that f(0) 0. We consider the infinite-horizon undiscounted cost (aro = -o (a:" Qu + u²" Ru) dr,
(2)
where Q TRax Q Q² and R E Rmxm R R² 0 are the state and control penalties.
2.1 THE IPA ALGORITHM
Bellman Optimality for Continuous- Time Systems: The HJB Equation. Consider the control affine nonlinear dynamics of (T). and the horizon undiscounted performance index of ( CT-RI and CT optimal control both aim to solve the common HJB equation for optimal value (a continuous-time analogy of the Bellman equation in discrete time), from which an optimal control is computed. We begin with the system Hamiltonian function (Lewis et al. 2012a).
av (Ir, u. ön = (f(z) + g(x))) av
On + +xx + u Ru.
(3)
The goal of CT-RL is to find the optimal control policy μ² which minimizes the value in (2), yielding the optimal value V Such a solution can be obtained from the necessary condition based on Pontryagin's minimum principle: Tu Hamiltonian H (3) in the case of affine nonlinear dynamics reduces to:
Afl = O. Plugging in Pontryagin's minimum principle to the
0 =
AH Dw (x) g" (x)
OV Oz (x) + 2Rp"(z)
p² - (
oV dx (z).
(4)
Note that we able reach this nice, closed-form control solution for the optimal nonlinear policy yu" given the optimal value V* because of the quadratic cost structure in the performance index ? in combination with the control affine nonlinearity in the form of (T). Plugging now the closed-form
optimal policy μ² (4) into the Hamiltonian H (3) and applying Pontryagin's ' minimum principle we arrive the HJB equation for affine nonlinear systems (Figure Block #1):
8H
8h.
0 (x de f (x) ov -V
dh (())
(x) ov xT Qz.
(5)
Our focus now is to solve for V from the HJB (5), in turn yielding the optimal policy μ²" from (4).
Setting up Critic Value and Respective Policy Based Generalized HJB Equation. We now inspect the value V (critic value) from perspective of the generalized HJB (GHJB) equation (Beard & McLain 1998) below. Here, the goal of the critic value V is to approximate the cost index J in achieved when the control (x(x)) is applied to the nonlinear system. function V approximates the cost J of current policy μ4 if and only if V satisfies the GHJB:
(f+ gHu)
OV Oz
+x=Qx + pu R14 0.
This is a differential equivalent to the value integral in (2) indeed, given any function V and policy H4 with cost J 2 V = J if and only if V satisfies the GHJB (6) (Beard & McLain 1998).
CT Temporal Difference Equation. Let jr be the control policy at the i-th iteration with respective value V. Given the performance index ( re-written below in the current context for convenience,
((((t) S (x"Qx= Qr + +x((=)))))))))))))))))( (x) dr,
(7)
we can obtain the following CT temporal difference equation (Figure Block #4), analogous the discrete-ting time Bellman equation as follows,
V(z(to)) v(x(tt)) J +u
(8)
where in the above, the left-hand side is the CT temporal value difference, and the right-hand side is called the integral reinforcement signal note that requires requires only state-action data. Just as in discrete time where the temporal difference points to the stage cost, in CT the temporal difference becomes an integral reinforcement signal. Similarly to DT, Bellman's 's optimality principle from the HJB (5) dictates that the optimal cost (xx(4)) time to is the same as minimizing the cost accrued over the interval [to. t1] plus the optimal cost V V(x(t1) )) at time t1. Analogous to discrete time TD, we use CT TD to learn the value function in (7). Given the time indices to t1, we examine the CT temporal value difference V(x(to)) V(z(h)) (Figure [] Block #2). the integral reinforcement signal fd x e'Qr + of Rx))))( dr, and the quadratic cost (2) (Figure Block #3) to develop CT learning rule for solving for V from the HJB equation (Figure 1 Block#1)
3

---

Published as a conference paper at ICLR 2025
IPA Learning Structure
ft gv" dx
de 9 9
av"
+ x Qu =0
HJB Equation 5
((a(to))- -
J(wo) = 10= x. Qu u² Ru dr
CT Temporal Value Difference
Performance Index
(Quadratic Q-R Cost)
V(z(to)) V(a(ti)) l. x''Q= Qr 1. (x) (p
Temporal Difference Equation
((x) x" (x, = x= P.
Sh x'Qu Qu + pe RuGd 2 St x² eQ Au7 Sus dr oz
Critic w/ Quadratic Bases
Integral Performance Approximation (IPA)
x (to)Pax(to) - x² (tt) = li x² =0(QQ +
0u. dz CT Temporal Difference by IPA
R R x
+ cs E, Critic Weight Learning Update
C (PP
(e,
Actual Environment
8
Trajectory Data
w/ Uncertainty
p641
ov 2
Closed-Form
Policy Update
10
IPA Learning Loop
Figure 1: IPA algorithm block diagram. IPA RL begins with the HJB equation (Block #1), which when combined with the CT temporal value difference (Block #2) and IPA's quadratic Q-R cost index (Block #3) yields the CT temporal difference (TD) equation (Block #4). When IPA's critic with quadratic bases (Block #5) is combined with the novel integral performance approximation scheme (Block #6), this yields the IPA continuous-time TD (Block #7), which in turn forms the basis of the IPA critic weight learning update (Block #8). IPA learning occurs in closed loop with the actual physical process with model uncertainty (Block #9), which supplies state-action data (x, u) for the IPA critic weight update (Block #8). IPA updates its policy via the closed- form HJB- based update (Block #10), from which new weight update is constructed, and so on.
The Critic and its Bases. We construct the critic network as in 9 below,
( x" (x.
Here, C1 E RR,, n A m(n+1) are the critic weights at the i-th iteration. We have chosen quadratic
2
bases ( R2 approximation features of the value V leverage the quadratic Q-R cost
structure in (2) and we define these bases as
( v) 1 [2r V1, F132 + #2th1,.... Ti2n + Tn11, 2022/, 2r,,,n]
(10)
To further see that our choice of bases is quadratic, the critic V 9 can be represented equivalently by the following quadratic form ++ P,, P. = P." Raxx (Figure I Block #5);
(x) = कπ (2,
कh (p.
=x'
(11)
where for symmetric matrix P = PT E R define its "vectorization" ol vPP) E R as
(P) [pi, 2p12. 2p p22, 2p23...., 2pn-1.n: Pun
(12)
Indeed, quick linear algebra shows that for each critic weight vector c; E I2, there exists a unique symmetric matrix P; pf such that e; v(P.). The last equality in (11) is also quick algebra.
4

---

Published as a conference paper at ICLR 2025
Equation 11 renders quadratic function approximant of the integral cost which is of the Q-R form. It is well-known in control theory and engineering that this Q-R cost structure is an appropriate form to be used feedback control design objective, it leads to physical insights of system performance (Lewis et al. 2012a [Barr & McL.ii 1998).
Integral Performance Approximation (IPA). Now, define the following matrices to be used for integral performance approximation only (these are not the IPA policies):
K; _ du (1)())))==0
(13)
Then (x) = -K,r, to first order, more precisely, p= = - K₁ + o(|x=]), where the residual ())(()))Ka)) Karl
O. We may then approximate the integral reinforcement signal () with what we term integral performance approximation (IPA) (Figure I Block #6):
<[[=)]] satisfies |il- [
x Qr + mh (c) dr Rs {-¹ x² =² KV RK:) dr,
(to o([==11))).
(14)
CT Temporal Difference by IPA. Now. substituting the quadratic bases (II) as the TD on the left-hand side of the CT TD equation (8. and substituting the IPA scheme (14) as the integral reinforcement signal in the right-hand side of (8), we get the CT TD by IPA (Figure T Block #7):
('o))) - x² (4) R I.h xT(Q+ RK.)
(to o((l=11²)) (15)
With the above equation, and since P; related the critic weights e₁ as discussed in the above, we have now reached the basis of the IPA learning algorithm. As will be seen, its quadratic structure enables our suite of theoretical guarantees (Section 3) as well as high data efficiency (Section 5).
IPA for Critic Learning. We now use IPA-facilitated CT temporal difference in (5) (which com- prises a single trajectory sample) to construct value function weight update. To solve for the weights ct we use / N trajectory samples. The update rule proceeds by designer first selecting a sequence of sample instants {4x (/ and reference command input r(t) for exploration (see Ap- pendix FF. We apply r(t) to the actual environment (Figure I Block #9) under initial stabilizing
policy Po. to collect the resulting state-action trajectory data (x((t, u(t)) } xe(to.t]]" Applying [15]
at the sample instants ( we arrive at the learning update (Figure Block #8):
θ Ej.
(16)
The exact form of the matrices θ E Rlxa, E, E R' and their algebraic derivation are provided in Appendix C In essence, the matrices capture the IPA CT temporal difference features in (5)) via:
af (1o1))) xF (11))a))))
1h x =T((0 K.T RK,) dr
E
(17)
x7
SLL TT(Q + K? R
Policy Update using Closed-Form Formula. As derived in (4). given the affine nonlinear dynamics (f.g), the optimal policy update can be solved directly from:
Mi+1 (x) = -
av dz (z).
(18)
Having solved for the critic weights 16). we update the nonlinear control policy μ4+1 (Figure Block #10) via 18), and so on, iteratively learning the optimal policy.
THEORETICAL RESULTS
Our choice of critic network structure V nonlinear policy structure μi (18). and use of IPA [4) allows us to take advantage of classical control results in Kleinman's well-tested algorithm Kleinman 1968) to be combined with state-action data (ar, u) from the actual physical environment in learning. This enables us to develop our key theoretical guarantees. To state the properties of IPA, let (A, B) denote the linearization of the affine nonlinearity (f.g) ( Then for all i 2 0
a (f(x) + ((x))1(2))] = A BK,
(19)
8z
5

---

Published as a conference paper at ICLR 2025
Similarly, f(x) + (())) = (A BK, ar ol [z]]) Let P E R p = 0 be the Riccati equation solution for (A, B) Rodriguez 2004), and K E Raxm the optimal LQR control.
Theorem 3.1 (Local Convergence, Optimality, Closed-Loop Stability, and Robustness of IPA) Suppose that the initial policy μo stabilizes the system and that the sample instants {4x ) are chosen such that the IPA matrix I E R given by
I= fro Φ(x, x)) dr
S 2(1,z) dr ]"
(20)
has full rank n (n being the number of bases in critic approximation). Then identifying the quadratic
form of the bases G vP..) from (II). IPA produces identical sequences of matrices {Pi}
{K;) as Kleinman's 's algorithm (Kleinman 1968) to first order if the Kleinman control sequence
produced based on a linearized actual non process that is unknown IPA. Thus:
Each of the policies {u: _ in (18) stabilizes the nonlinear system (f.g) (1) under feedback.
pa P:+1 S P. for all 2 and lim P. P Thus, in the limit the value function V 11
<-oc
optimal to second-order (stronger than first-order); i.e., V(x) = x. P3 - (*) + o[|[=]|] 2)
Ox 14, = K* Thus, in the limit the nonlinear policies hi 188) are optimal to first- order; i.e., (x) + μ²(xx) o[[[=]]).
lim 0
As direct result, IPA inherits the guaranteed stability robustness margins of Kleinman's algorithm:
||Sul| < 0 dB, I I dB, where ||- denotes the 7 norm, and where Su, Tu denote the sensitivity and complementary sensitivity closed-loop maps, respectively, at the control loop breaking point u (Rodrigue7. 2004). For definitions and further discussion, see Appendix F
Proof: An induction argument presented in Appendix
EXPERIMENT SETUP FOR EVALUATIONS
In this section, we show how IPA learning and control performance compares to the two most promising classes of CT-RL methods: 1) the foundational ADP CT-RL methods as reviewed in (Wallace & Si. 2024), and 2) the SOTA FVIs (Lutter et al., 2021 2023b).
We perform in-depth evaluations on three CT-RL environments: 1) second order system (SOS) Vamvoudakis & Lewis (010) that is what ADP methods were demonstrated on as being solvable, however, only in the case of using exact bases induced from priori knowledge of the solution (but not solvable if the bases slightly differ from the true structure of the value function, refer to Appendix M. 2) a pendulum that extensively evaluated in (Lutter et al. 2021 2023b), the results of which stand as SOTA in current CT-RL, and 3) a hypersonic vehicle (HSV) (Wang & Stengel 2000 Shaughnesss al. 1990) that, for the first time, is being extensively evaluated in this study, which is considered a SOTA environment that has ever been studied in CT-RL.
Due to ADP's inability to solve even SOS under realistic conditions (see evaluations in Appendix Mj. this study focuses mainly on IPA and the FVIs for extensive and systematic evaluations. All implementation details and hyperparameter selections for these studies be found in Appendix
The dynamics, physical insights, and realistic modeling discrepancy issues of the environments are provided in Appendices K for the SOS, pendulum, and HSV. respectively.
Baseline Methods. These include ADP and FVIs with a strong focus on FVIs, as they have successfully synthesized meaningful controllers for the environments being evaluated:
"Continuous FVI (cFVI)): SOTA deep CT-RL method (Lutter et al. 2021) "Robust FVI (rFVI)": Robust variant of SOTA FVI (Lutter et al. 2023b).
Questions Addressed. Our evaluations aim to quantitatively address the following: Q1: How does IPA CT-RL learning performance (average cost, learning speed and variance, and success rate) compare to baseline FVIs?
Q2: How time/data efficient is IPA compared to FVIs?
Q3: How does IPA learning performance generalize to environment uncertainty compared to FVIs?
Q4: How does IPA cost performance generalize environment uncertainty compared to FVIs?
Q5: How does IPA critic estimation generalize to environment uncertainty compared to FVIs?
Q6: How do IPA time responses generalize environment uncertainty compared FVIs?

---

Published as a conference paper at ICLR 2025
Remark on Simulating Unmodeled Dynamics. For Q3 13-Q above, in order simulate uncertainty present in the actual physical process, we study the performance of the algorithms under systematic variations of model uncertainty parameter (nominally 1) inserted in the dynamies. For de- tailed description of this parameter relation to each of the three environments studied, please see For details on performance measures, refer to Appendix
Appendices
D
Remark on Environments. For details on SOTA environments evaluated in leading CT-RL algo- rithms, please refer Appendix A where shown that the pendulum and the HSV environments are SOTA. We use the SOS environment because it is the only one that can be successfully solved by all leading methods: ADP-based (if the bases are selected using priori knowledge of the value function structure), FVIs, and the current IPA-basee CT-RL.
5 COMPARISONS BETWEEN IPA AND DRL FVIS
This evaluation focuses on comparing IPA to SOTA deep RL cFVI and rFVI Lutter et al. 2021 2023b). Training and evaluation procedures described in detail in Section 4 and Appendix H
100
45
90
Average Clst..
3.5
Average Cost
A Average Cost
35
70
30
25
50
20
15
0.5
Iteration
Iteration
Iteration
Figure 2: Learning curves of IPA and FVIs obtained over 20 seeds for the nominal models v = 0% of the SOS (left), pendulum (middle) and HSV (right). The shaded area displays the min/max range among seeds, as in the original works (Lutter et al. 2021
2023b).
:PV1
(ft/s)
100
1
((t) (deg)
120
80
( (deg)
0.8
100
80
60
0.6
40
0.4
20
0.2
20
IPVI
IPVI
0.5
1
2.5
Time (s)
Time (s)
Time (s)
Figure 3: Closed-loop time-domain responses nominal models 0%. Left: Swing-up response of pendulum model f(t) from hanging pendulum position 6n = 180 Middle: HSV response to 100 ft/s step velocity command. Right: HSV response 1 step FPA command. Overall, IPA exhibits fast, well-behaved responses. FVIs on the HSV have large oscillatory transients.
Q1: IPA exhibits the fastest learning convergence. Its cost performance and evaluation success rate meet or exceed FVIs. The learning curves of the tested methods trained to 20 seeds for the three respective nominal models (model uncertainty level v 0%) are plotted in Figure 2 where IPA exhibits the fastest learning convergence of the three methods on all three environments. Except for cFVI on the pendulum benchmark, IPA also delivers the lowest cost and variance. The average cost (closer to is better) of the tested methods evaluated on the three CT-RL environments in Table 2 each at three levels of modeling error 0%. 10%. 25%. Success rates summarized in Table for evaluations of the three environments and different tasks under three levels of modeling enror. See Table Appendix G for definitions of task criteria.
7

---

Published as a conference paper at ICLR 2025
Examining the performance in Figure and Table 3 overall the three methods (IPA, cFVI, rFVI) perform quite comparably on the baselmne SOS environment. However, examining the closed-loop performance in the pendulum environment swinging up from full free-hanging pendulum displace- ment (0n = 180%) ) in Figure g one can see that the IPA response is the fastest of the three methods and is well-behaved with overshoot. This hence points to trade-off between lower cost (achieved by cFVI) and faster swing-up up performance (achieved by IPA). Finally, IPA exhibits a clear learning performance edge over FVIs on the HSV environment, with the fastest convergence, lowest cost, and 100% evaluation success rate on all tasks (Figure Table 3).
Q2: IPA orders of magnitude more data efficient and 3 orders of magnitude more time efficient than FVIs. Due to IPA's low-dimension quadratic bases enabled by our choice of quadratic Q-R cost, it achieves substantial learning efficiency compared to the FVIs training deep networks. Iists key time/data complexity parameters for IPA and the FVIs. In many cases, IPA exhibits orders of magniude less time/data complexity than the SOTA FVI works. To illustrate, we examine the ratio of IPA/FVI on the HSV model. For number of data samples required: 1/5,750,000, training episodes required: IPA/FVI 1/5,250,000, training time: 1/1,500, algorithm iterations: 1/5.
Table 1: Comparison of time/data complexity of IPA and FVIs on 3 environments
SOS
Pendulum
HSV
Parameter
IPA
cFVI/IFVI
IPA
cFVI/rFVI
IPA
cFVI/rFVI
Traj. data samples
10
2.88e+7
15
8.63e+7/2.88er
30
1.73e+8
# Data episodes
1
6.40e+5
1
2.63e+6/6.4006
1
5.25e+6
Avg train time (s)
0.20
1.4243/1.41143
0.22
1.760+3/1.466
2.75
160c+3/4.79643
# Alg. iterations i
5
25
5
25
10
50
IPA learning generalizes the best on all three environments as modeling errors are introduced. In Table [ we see that regardless of the environment tested, IPA achieves aver- age cost that meets or exceeds the FVIs when modeling error increases. Exam- ining Table 2 on the HSV at // = 25% modeling error, IPA cost (32.45, a 31% degradation from the nominal model) far surpasses that of cFVI (63.80, 114% degradation) and rFVI (74.15, a 122% degradation). Turning to the evaluation success rates for the HSV in Table 3. we see that IPA achieves a 100% success rate all tasks regard- less of modeling error severity. FVIs exhibit a high success rate on the lax regulation Tasks 1 and 2; however, they struggle with the stringent Tasks and 4 for reasons discussed in Q6 below.
Q3:
Table 2: Average cost varying modeling error levels
Cost
SOS
Pendulum
HSV
IPA
0.50++0.5
12.12+17.96
24.69
0%
cFVI
0.50+0.66
8.99+10.04
29.79+:31.51
rFVI 0.50++0.67
16.26=17.61
33.37+33..47
IPA 0.53+10.69
8.48=11.85
27.19+40.63
10% cFVI
0.53+0.70
7.7348.70
36.42+36.49
rFVI
0.53
16.75
41.54
IPA
0.58
5.77#6.31
32,45155038
25% cFVI
0.58+0.78
7.01
43.70
rFVI
0.56+00.76
23.15
74.15
Q4: FVIs deliver good cost performance results on the nominal model, but IPA generalizability is superior. Tigure[4shows the relative cost performance J.FvI JTPA between cFVI and IPA (left two plots) and J,FvI JIPA between rFVI and IPA (right two plots) on the nominal HSV model
0% and at 25% modeling error. Full plots can be found in Figures 9 10 and II of Appendix I on the SOS, pendulum, and HSV, respectively. This data is tabulated in Tables 12 and [3] Appendix
! On the nominal HSV model (first and third plots of Figure 4 FVIs IPA perform comparably near the origin, but IPA begins to outperform FVIs by ~ 10- 20 in the majority of the operating region of the states. There are two fringes of the domain (corresponding to large initial flightpath angle (FPA) displacements 7 +1 deg) in which FVIs slightly outperform IPA; however, in the mean IPA outperforms cFVI by 13.14 and rFVI by 16.45 (cf. Table [3). Cost performance of both FVI algorithms degrades significantly when modeling error is introduced, in particular at the severe 25% modeling error (second and fourth plots of Figure 4). Here, IPA outperforms FVIs pointwise, in the mean outperforming cFVT by 34. 10 and rFVI by 35.90.
8

---

Published as a conference paper at ICLR 2025
Table Success rates for different environments and tasks varying modeling error 0%, 10%,
25% (cf. Table 5
for task criteria)
Success
SOS
Pendulum
HSV
HSV
HSV
HSV
HSV
HSV
HSV
[%]
Task
Task 1
Task
Task 2
Task
Task 4
Task 5
Task 6
Task 7
IPA
100
100
100
100
100
100
100
100
100
0%
cFVI
100
100
100
100
31.95
15.30
100
3.90
3.90
IFVI
100
100
100
100
19.20
8.75
1.25
1.25
IPA
100
100
100
100
100
100
100)
100
100
10%
cFVI
100
100
100
99.50
32.70
9.95
99.50
2.35
2.35
FVI
100
100
100
100
15,50
7.70
100
1.45
1.45
IPA
T00
100
100
100
100
100
100
100
100
25%
cFVI
100
100
100
99.20
48,00
8.30
99.20
4.20
4.20
#VVI
100
100
100
100
16.45
5.70
100
1.95
1.95
0.8
0.4
[deg)
022
0.2
0.2
0.2
28
0.4
4.4
4.4
48.4
18
0.6
0.5
300
-50
50
100
1300
-50
50
100
100
-50
0
50
100
-100
-50
100
(8./8)
V (ft/ss
V (ft/s)
(ft/s)
Figure 4: Relative cost performance J.FVI- JIPA (* cr > means IPA better) on the HSV environment. Left to right: cFVI 0% uncertainty, cFVI 25% uncertainty, rFVI 0% uncertainty, rFVI 25% uncertainty. FVIs exhibit slightly higher cost on average than IPA on the nominal model, and the performance gap increases significantly when environment uncertainty is introduced.
Q5: IPA exhibits more consistent critic approximation behavior environment uncertainty is introduced. Figure shows the critic network error J- V for IPA (left column), cFVI (middle col- umn), and rFVI (right column) on the nominal HSV model v = 0% (top row) and at 25% modeling error (bottom row). Full plots can be found in Figures 12 13 and 14 Appendix [ the SOS, pen- dulum, and HSV. respectively. On the nominal HSV model v = 0% (left column of Figure I4]. one can see that both VI and rFVI struggle with overestimation
the band y [-0.5 0.5 in par- ticular for large velocity displacements V = 100 ft/s. IPA approximation error remains lower and more consistent in this region. Furthermore, rFVI begins to underestimate policy performance for 11, where cFVI performs comparably better. Overall, approxima-
large FPA displacements tion performance significantly degrades for FVIs. Here, both FVI algorithms begin to overestimate policy performance nominal model near 7 = +10, it performs comparably better than cFVI at 25% modeling error. This suggests that rFVI's adversarial input successfully improves generalization properties of this robust variant somewhat. By comparison, IPA's approximation error remains at comparable levels and in a similar radial pattern its performance on the nominal model.
large degree. Where rFVI begins underestimates policy performance on the
Q6: IPA exhibits the fastest closed-loop performance with least overshoot on all environments, performance advantage retained with modeling error. Figure 3 plots the closed-loop time re- sponses of the tested methods on the nominal model for 1) the pendulum swing-up task (left), 2) HSV 100 ft/s step velocity command issued from trim (middle)). and 3) HSV 1 step FPA command issued from trim (right). Full plots for the SOS, pendulum, and HSV can be found in Figures 15] I6 and 17 of Appendix ! respectively. The trends are similar as modeling error is introduced. pendulum, we can see that the IPA swing-up is more responsive than either of the FVI methods. For the HSV velocity response (middle plot of Figure we can see that both FVI algorithms have large overshoot compared to IPA. Furthermore, the FVI responses exhibit pronounced and slow-decaying transients about the 100 ft/s reference command setpoint. Similar transients are seen in the HSV FPA response for the two FVIs (right plot of Figure B). Such oscillations in FPA (i.e., in deflec- tion the vehicle velocity vector) are highly undesirable in flight control settings, as they cause wear on the fuselage and may excite the HSV flexible modes (Bolender & Doman 2006a). This time-
the

---

Published as a conference paper at ICLR 2025
0.6
0.6
0.6
(deg)
-00
40
-0.6
40.6
-0.6
-80
-80
-90
-100
-50
50
100
-100
-50
50
100
-100
-40
50
100
(ft/s)
(ft/ /8)
(ft/s)
40
40
20
0.4
0.4
0.4
100g
0.2
-20
:20
-20
-02
-02
-0.2
004
.60
.60
008
-0.8
-0.8
-1
100
-100
100
(ft//)
V (ft//s
(ft/s)
(xx of nominal HSV model (top row) and at 25% modeling error (bottom row) for IPA (left), cFVI (middle), and rFVI (right). IPA exhibits consistent approximation performance, and the FVIs struggle to generalize.
Figure 5: Critic NN approximation error ((x - (x)
domain behavior is also clear visual indicator of why FVIs struggle with evaluation success rate the more stringent HSV regulation Tasks 3 and 4 (cf. Table 3). By comparison, IPA exhibits less overshoot, faster settling time, and well-behaved system response without oscillatory transients.
CONCLUSION AND DISCUSSION
We introduce IPA, new, model-based CT-RL control method which innovatively combines partially-known affine nonlinear dynamic model of the actual nonlinear environment, state-action trajectory data, Kleinman control structures, and reference control input for learning exploration. These give our IPA method significant data efficiency, as we have summarized in comparison to the FVIs in Table [ When compared to the FVIs, IPA offers substantial thcoretical guarantees, and its learning performance at least matches, and often outperforms, the SOTA FVIs in terms of average return, evaluation success rate, critic network approximation accuracy, closed-loop time-domain per- formance, and generalization to unmodeled dynamics. The IPA performance advantage observed on the HSV control task demonstrates that IPA achieves SOTA results in both the domains of learning and dynamical control. Finally, would like to point the need of spending extra time tune the FVIs for convergence on the HSV environment (for detailed discussions and implementation, see Remark H.l of Appendix H.
Limitations of this Study. The theoretical guarantees and design of IPA are based on an affine nonlinear model, not the general nonlinear systems in some recent works (Yildiz al. 2021 Sandoval et al. 2023). However, as discussed in Section I fully-nonlinear algorithms are at an early stage. Comprehensive theoretical results and meaningful designs without stringent assumptions are still under development. As result, few methods have synthesized meaningful controllers. For example, the ADP CT-VI for general nonlinear systems (Appendix MM fails to synthesize for simple second-order systems with known closed-form solutions (Wallace & Si 2024).
Reproducibility Statement. All IPA code and all datasets for this study are available in Supple- mental and at WWallace & Si 2025). All FVI results (Lutter et al. 202 2023b) are generated by the open-source code developed by the authors available at Lutter et al. (202aa). For an in-depth discussion of our setup and a complete list of numerical hyperparameter selections, see Section 4 and Appendix H. All theoretical assumptions can be found our method setup of Section 2 and all proofs of theoretical results can be found in Appendix D
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the NSF under Grants 1808752 and 2211740. The work of Brent A. Wallace was supported in part by the NSF under Graduate Research Fellowship Grant 026257-001.
REFERENCES
R. W. Beard and T. T. McLain. Successive galerkin approximation algortihms for nonlinear optimal and robust control. Int. J. Contr., 71:717-743, 1998.
D. Bertsekas. Value and policy iterations in optimal control and adaptive dynamic programming. IEEE TNNLS, 28(3)5000-509 March 2017.
T. Bian and Z.-P. Jiang. Reinforcement learning and adaptive optimal control for continuous-tima nonlinear systems: A value iteration approach. IEEE TNNLS, (117)))281-2790 July 2022. doi: St0.///INN.S..2..........
M. A. Bolender and D. B. Doman. A non-linear model for the longitudinal dynamics of hypersonic air-breathing vehicle. In AIAA Guid., Nav., Contr. Conf. Exhib., pp. 1-22, August 2005. doi: 022514/6.2005-6255 AIAA 2005-6255.
M. A. Bolender and B. Doman. Flight path angle dynamics of air-breathing hypersonic vehicles. In AIAA Guid., Nav. Contr. Conf. Exhib., August 2006a. AIAA 2006-6692.
M. A. Bolender and D. B. Doman. Modeling unsteady heating effects on the structural dynamics of hypersonic vehicle. In AIAA Atmos. Flight Mech. Conf. Exhib., August 2006b. AIAA 2006-6646.
S. J. Bradtke, B. E. Ydstie, and A. G. Barto. Adaptive linear quadratic control using policy iteration. In 1994 IEEE ACC, pp. 3475-3479, June 1994.
X. Bu and Q. Qi. Fuzzy optimal tracking control of hypersonic flight vehicles via single-network adaptive critic design. IEEE Trans. Fuzzy Syst., 30(1):270-277 January 2022. doi: 10.1 10.1/ TFU722.2020.30367066
X. Bu, Xiao, and Lei. adaptive critic design-based fuzzy neural controller for hypersonic vehicles: Predefined behavioral nonaffine control. JEEE/ASME Trans. Mechatron.. 24(4):1871- 1881, August 2019. doi: 10. 10.1109TMC...211.2222222
W. Cao and W. Pan. Impact of computation integral reinforcement learning for continuous-time control. Int. Conf. Learn. Rep. (ICLR), May 2024.
T. Chen and B. Francis. Optimal Sampled-Datt Control Systems. Springer Verlag, Berlin, Germany, 1995.
J. J. Craig. Introduction to Robotics: Mechanics and Control. Pearson Education, Upper Saddle River, NJ, USA, 3 edition, 2005.
R. Dhaouadi and A. Abu Hatab. Dynamic modelling of diffcrential-drive mobile robots using la- grange and newton-euler methodologies: A unified framework. Adv. Robo. Auto., 2(2):1-7, Jan- uary 2013.
J. Dickeson, A. A. Rodriguez, S. Sridharan, J. Benevides, and D. Soloway. Decentralized control of an airbreathing scramjet-powered hypersonic vehicle. In AIAA Guid,, Nav, Contr: Conf. Exhib., pp. 1-25, August 2009a. doi: 10.2514/6.2009-2222 AIAA 2009-6281.
J. Dickeson, A. A. Rodriguez, S. Sridharan, D. Soloway, A. Korad, J. Khatri, J. Benavides, A. Kelkar, and J. Vogel. Control-relevant modeling, analysis, and design scramjet-poweret hypersonic vehicles. In A/AA/DLR/DGLR Ins. Space Planes Hyper. Syst. Tech. Conf., pp. 1-45, October 2009b. doi: 10.2514/6.2009- 7287. AIAA 2009-7287.
R. Enns and J. Si. Apache helicopter stabilization using neural dynamic programming, AIAA J. Guid., Contr, & Dyn., 2591):19-25 January 2002.

---

Published as a conference paper at ICLR 2025
R. Enns and J. Si. Helicopter flight-control reconfiguration for main rotor actuator failures. AIAA J. Guid., Contr. & Dyn., 26(4):572-588 July 2003a.
R. Enns and J. Si. Helicopter trimming and tracking control using direct neural dynamic program- ming. IEEE TNN, 14(4):929-939, July 2003b.
K. Jha, S. B. Roy, and S. Bhasin. Initial excitation-basel iterative algorithm for approximate optimal control of completely unknown Iti systems. IEEE TAC, 64(12):523-55237, December 2019.
Y. Jiang and P. Jiangg Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica, 28440) October 2012.
Y. Jiang and Z.-P. Jiang. Robust adaptive dynamic programming and feedback stabilization of nonlinear systems. IEEE TNNLS, 25(5):882-893, January 2014. doi: 10.1109/TNNLS.2133. 2294968.
Kim, J. Shin, and I. Yang. Hamilton-Jacoe deep Q-learning for deterministic continuous-ting systems with Lipschitz continuous controls. J. Mach. Learn. Res., 22(1):9363-9396 September 2021.
D. Kleinman. On an iterative technique for Riccati equation computations. JEEE TAC, 13(1):114- 115. February 1968. doi: 10. 19.1.99/ 1098829.
L. Lewis, D. Vrabie, and L. Syrmos. Optimal Control. John Wiley & Sons, edition, 2012a.
L. Lewis, D. Vrabie, and K. G. Vamvoudakis. Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers. 1EEE Contr. Syst. Mag., 105,6-105, 2012b.
M. Lutter, S. Mannor, J. Peters, D. Fox, and A. Garg. Value iteration in continuous actions, states and time. In Proc. 38th Int. Conf. Mach. Learn. (ICML), volume 139, pp. 7224-7234, July 2021. URL httttpps://proceeddnnge
mlr. wrwssrrsssseeeeeep httpss///vv11//tttttte1 html
M. Lutter al. Continuous & robust fitted value iteration. 2023a. Accessed: 2023-01-12.
M. Lutter et al. Continuous-time fitted value iteration for robust policies. IEEE Trans. Pall. Anal.
Mach. Intel., (5(5):553455548 May 2023b. doi: (009/TPAM..0222221555555
C. I. Marrison and R. Stengel. Design of robust control systems for a hypersonic aireraft. AIAA
J. Guid., Contr, & Dyn., 21(1):58-63, January 1998. doi: 10.2514/2.4199
M. Morari and E. Zafiriou. Robust Process Control. Prentice Hall, Englewood Cliffs, NJ, USA, 1989.
C. Possieri and M. Sassano. Q-learning for continuous-time linear systems: A data-driven im- plementation of the Kleinman algorithm. IEEE Trans. Syst., Man, Cybern. A: Syst., 52(10): 6487-6497, October 2022.
H.-Y. Qiao, H. Meng, M.-J. Wang, W. Ke, and J.-G. Sun. Adaptive control for hypersonic vehicle with input saturation and state constraints. Aero. Sci. Tech., 84:107-119, January 2019. doi: 10.1016/j.ast.0118 20............1........ 10.018.
Quanser. Quanser courseware and resources. httpp://www.qqaaser..cc////oo//s
controllsystenm 2018.
A. A. Rodriguez. Analysis and Design of Multivariable Feedback Control Systems. CONTROL3D, Tempe, AZ, USA, 2004.
A. A. Rodriguez, J. Dickeson, O. Cifdaloz, A. Kelkar, J. Vogel, D. Soloway, R. McCullen, J. Bena- vides. and S. Sridharan. Modeling and control of powerde hypersonic vehicles: Chal- lenges, trends, and tradeoffs. In A/AA Guid,, Nav. Contr. Conf. Exhib., pp. 1-40, August 2008. AIAA 2008-6793.
12

---

Published as a conference paper at ICLR 2025
1. L. Sandoval, P. Petsagkourakis, and E Antonio del Rio-Chanona. Neural odes as feedback policies for nonlinear optimal control. Pape--apersnnLLne, 56(2):4816 January 2023.
J. D. Shaughnessy. S. Z. Pinckney, J. D. McMinn, C. I. Cruz, and M.-L. Kelley. Hypersonic vehicle simulation model: Winged-cone configuration. NASA TM-1026 November 1990.
Si, A. G. Barto, W. B. Powell, and D. C. Wunsch. Handbook of Learning and Approximate Dynamic Programming. Wiley, Piscataway, NJ, USA, 2004.
R. F. Stengel. Flight Dynamics. Princeton University Press, Princeton, NJ, USA, 2 edition, 2022.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge,
MA, USA, 2 edition, 2018.
K. G. Vamvoudakis and F. L. Lewis. Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica, 46(5):878 888, 2010. doi: 10.1016/j. automatica.2010.02.00
D. Vrabie and F. L. Lewis. Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neur. Net., 22(3):337-266, 2009. doi: 10.1016/ j j.nemnet2009.00 03.008.
B. A. Wallace and J. Si. Continuous-time reinforcement learning control: A review of theoretical results, insights on performance, and needs for new designs. IEEE TNNLS, 3518)11019,--0211,, August 2024.
1109/TNN1SSS 2023. 3245980
B. Wallace and Si. IPA (ICLR 2025). htttps:://iithub.com/// 2ttpb/aa22aa01100022222220222 2025.
P. Wang, Z. Man, Z. Cao, J. Zheng, and Y. Zhao. Dynamics modelling and linear control of quad- copter. In IEEE Int. Conf. Adv. Mech. Syst. (ICAMechS), pp. 498-503, November 2016.
Q Wang and R. F. Stengel. Robust nonlinear control of a hypersonic aircraft. AIAA J. Guid., Contr, & Dyn., (374)5577- July 2000. doi: 10.2514/2.4580.
B. Xu, D. Wang, F. Sun, and Z. Shi. Direct neural control of hypersonic flight vehicles with predic- tion model in discrete time. Neurocomp., 11:339 48, September 2013. doi: 10.1016/j neccom. 2012. 12.028.
B. Xu, C. Yang, and Y. Pan. Global neural dynamic surface tracking control of strict-feedback systems with application to hypersonic flight vehicle. IEEE TNNLS, 26(10):2563-25 2575, August 2015. doi: 1099TNN/
2015. 20.19//NNSS.2....45555999999
H. Xu, M. D. Mirmirani, and P. A. Ioannou. Robust neural adaptive control of a hypersonic aircraft. In AIAA Guid., Nav. Contr. Conf. Exhib., pp. 1-8, August 2003, doi: 10.2514/6.2003-555 AIAA 2003-5641.
H. Xu, M. D. Mirmirani, and P. A. Toannou. Adaptive sliding mode control design for hypersonic flight vehicle. AIAA J. Guid., Contr, & Dyn., 27(5):829-833 September 2004. doi: 10.2514/1. 12596.
Q Yang, W. Cao, W. Meng, and J. Si. Deinnooreeeentlearmiig lewrning-aaannnaeeee based tracking control of waste water treatment process under realistic system conditions and control performance requirements. IEEE Trans. Syst. Man, Cybern. A: Syst., (2(8):52445529) August 2022.
Y. Yang, Y. Pan, C.-Z. Xu, and C. Wunsch. Hamiltonian adaptive dynamic programming with efficient experience replay. IEEE TNNLS, 35(3):3278-3290 March 2024.
C. Yildiz, M. Heinonen, and H. Lähdesmäki. Continuous-time model-based reinforcement learning. In Proc. 38th Int. Conf. Mach. Leurn. (ICML), pp. 12009-12018, July 2021.
S. Zhao, J. Wang, H. Xu, and B. Wang. Composite observer-based optimal sutitude-tracking con- trol with reinforcement learning for hypersonic vehicles. IEEE Trans. Cybern., 53(2):913 926, February 2023. doi: 10. (99/TCCBB.022231228771
13

---

Published as a conference paper at ICLR 2025
APPENDIX A ENVIRONMENTS STUDIED AND KEY THEORETICAL ASSUMPTIONS REQUIRED BY LEADING AND SOTA CT-RL WORKS
We provide an overview of the environments studied in the evaluations of the leading and SOTA CT-RL works in Table below as discussed in Section
Table 4: Environments in SOTA CT-RL evaluation
Alg
System
Order
# inputs
Source of model parameters
SOS
Identical SPI as benchmark
Pendulum
Identical to FVIs as benchmark
IPA
HSV
NASA Langley aeropropulsive data
(Shaughnessy et al. 1990)
Unstable, nommınmmu phase, complex
Pendulum
Quanser STEM curriculum
FVIs
Cart Pendulum
4
resources Quanser 2018)
Furatura Pendulum
SOS
2
I
Non-physical, optimal known priori Non-physical, optimal known a priori Non-physical LQQ example
IRL
SOS
1
Simple Linear
SPI
SOS
2
Non-physical, optimal known a priori Non-physical for illustration
Simple Engine
RADP
Simple Power Bus
2
Non-physical for illustration
SOS
1
Non-physical, optimal known a priori
CT-VI
Simple Robot Arm
4
2
Non-physical for illustration
Theoretical Assumptions Required by Leading CT-RL Works. shown below, IPA among the
least restrictive in CT-RL in its thcoretical assumptions. As note, all methods require that be Lipschitz near origin to assure well-posedness solutions to the system differential equations. Also note that none of the ADP designs resulted in meaningful controllers, please refer to (Wallace & Si 2024).
IPA (present work):
Linearization of nonlinear system (A, B) stabilizable and (Q1/2, detectable (for
well dell-posedness and definiteness of regulation problem)
Full column rank & x(n+1) of the IPA matrix I R easy to satisfy, as the algorithm can continue to collect > n samples until is met. is also virtually instantaneous verify (matrix rank calculation). We observe issue satisfying this assumption on the complex HSV in practice
20 This assumption is
Initial stabilizing nonlinear policy μo
FVIs (Lutter et
2021 2023b):
and are smooth in their partial derivatives in the state v and model uncertainty parameters and these partials all known a priori
Undiscounted problem = can be approximated by discounted problem < < < Discrete-time rumning cost r(æ, u) can be approximated by continuous counter- part: r(z, u) At (a u) with sample time At
Strict convexity of action penalty 9-
Availability of convex conjugate function to action penalty 9c
Higher-order terms in Taylor series expansion of optimal value V are negligible
Existence of priori state grid x E D contain trajectories for fitting procedure Trajectories leaving the grid x E D can be instantancously re- re-initialized to the previ- position inside the grid
14

---

Published as a conference paper at ICLR 2025
IRL (Vrabie & Lewis 2009):
There exists sequence of sampling instants to < <1 <... t such that the IRL regression matrix has full rank. This assumption is qualitatively similar to IPA, but the method does not lead to meaningful controllers in practice, there is constructive method to ensure the full rank condition (Wallace & Si, 2024)
Chosen basis functions approximate optimal value and its gradient uniformly com- pact sets
Basis functions for critic network linearly-independent
Initial stabilizing policy
SPI
Vamvoudakis & Lewis 2010
Existence and uniqueness of least-squares solution to approximate HJB equation
PE assumption on various learning signals
Chosen basis functions approximate optimal value and its gradient uniformly on com- pact sets
Chosen basis functions approximate optimal policy uniformly on compact sets
Basis functions for critic network linearly-independent
Basis functions for actor network are linearly-independent
Initial stabilizing policy
RADP (Jing & Jiang 2014
Optimal value can be bounded from above and below by priori known class Koo functions
Existence of priori known compact set So for which the closed-loop system under the initial policy is invariant with respect to the probing noise d
PE assumption on various learning signals
Chosen basis functions approximate optimal value and its gradient uniformly on com- pact sets
Chosen basis functions approximate optimal policy uniformly compact sets
Basis functions for critic network are linearly-ineependent
Basis functions for actor network are linearly-independent
Initial stabilizing policy
CT-VI (Bian Jiang 2022):
Existence and uniqueness of solutions an uncountable family of finite-horizon HJB equations
Properness of each solution to the finite-horizon HJB equation
Convergence of family of solutions of finite-horizon HJB equation to the infinite- horizon HJB solution
Invariance of closed-loop state/action trajectory to compact set with respect to the probing noise d
Initial globally asymptotically stabilizing policy
PE assumption on various learning signals
Chosen basis functions approximate optimal value and its gradient uniformly com- pact sets
Chosen basis functions approximate optimal policy uniformly compact sets
Chosen basis functions approximate optimal Hamiltonian uniformly on compact sets
Basis functions for critic network are innearlyinneependent
Basis functions for actor network are linearl--independent
Basis functions for Hamiltonian network are linearly-independentt
15

---

Published as a conference paper at ICLR 2025
APPENDIX B KLEINMAN'S ALGORITHM FOR LINEAR SYSTEMS KLEINMAN I968)
We adapt some successive approximation concepts from Kleinman's 's algorithm to the proposed non- linear IPA algorithm for data efficiency. Classical Kleinman's 's algorithm considers the linear time- invariant system ă Ar + Bu. We assume that (A. B) is stabilizable and (Q'/², .A) is detectable, for well- pole-dossdeess [Rodriguez] 2004). Kleinman's 's algorithm iteratively solves for the optimal LQR control K* equation (Rodrigue7, 2004), as follows. For iteration = 0, l,.... on the current policy K, let P. E RR P. P > 0 be the solution of the algebraic Lyapunov equation (ALE)
B" p where p TR=x" P p-7 > is the solution of the Riceati
(A- (.-B),PPP.. P.(A- BK,) K? RK, + Q 0.
(21)
Then, P. solved from
21 leads to the new policy Kj+1 E R as
K B
(22)
The following theorem is needed to prove the theoretical results of Section 3
Theorem B.1 (Convergence, Optimality, and Closed-Loop Stability of Kleinman's Algorithm (Kleinman 1968)) Suppose the initial policy Ko is such that A BKo is Hurwitz Then we have the following:
ii A - BK; is Hurwitz for all 2 0.
i P S P.4 P, for all i 0, and lim P. = Pa, lim K; K*
APPENDIX C ADDITIONAL DERIVATIONS OF IPA METHOD (SECTION 2
The following algebraic manipulation of the IPA CT temporal difference equation T5) is necessary in order to 1) cast it into the form for a learning update (16). which requires that all terms pertaining to the value function weights c; appear as regression parameters, and 2) accommodate when the con- trol input u(() includes excitation signals. The integral reinforcement (8) and surrounding equations only hold if the control u(t) = N4 (z))) is applied without excitation signals, yet these excitation signals are necessary for good data/learning quality and thus need to be accommodated by the IPA framework. First, we must formally establish some of the properties of the operators e (12 and +
10
Proposition C.I The operators
[2) and + (TO) satisfy the following:
The restriction of the symmetric matrices is linear isomorphism; thus, for each R
there exists a unique P E Raxn, P == pT such that
c= v(P).
(23)
(ii) Whenever P E Raxx, P PT the following identity holds
Φ T (x))) Py. Va,yER"
(24)
We are ready to proceed with the derivation. Beginning with the i-th iteration policy μ4, first note that as x - 0, we have
f(x) + ((x)ps())
- E(x) A Ax + (ry(
(25)
at a rate o[[==1), where (A, B) denotes the linearization of (f. g) (1). Combining the quadratic value
function approximator structure V(x) = T P.r 11] with the GHJB equation (6) implies the first order that
25 (x) IT TQQ Ru: 0.
(26)
16

---

Published as a conference paper at ICLR 2025
Next, we differentiate the value function integral ((x(t)) 7 along the trajectories of the nonlinear system, yielding
((a(4))) - (2(())) ((((t)))(((((())) = -." dr (V())) dr dr= S. (J(x) + g(>)))²
OV (x) (x) dr.
(27)
dr
The crux of the CT temporal difference formulation in (27) is that it accommodates arbitrary control signals u. In terms of the quadratic bases chosen (11). we can express (27) as
x (ta))a)))))) - x x² (to) = J. (f(z) g(x)u)" o(z))) P.c.
(28)
Next, we need to re-introduce the integral reinforcement signal J.) =² 1², R Rp (2))d as
found in the original CT TD equation (8) back into the CT TD formulation (28) so that IPA (14)
may be applied to 28 To do so, we add an integral reinforcement constant based on 25 to both sides of (28):
If ((1) =f -f (16)P:a(())) -2 2 ["'
l (f(x) g())) ((x))
(1) =2
P.
(29)
Plugging the approximate GHJB equation
26 into 29 and rearranging, we have
-2 %/'² (f(x) + g(x)u - E(x:)) do + [a (()))())) - (1o)
pf (p) dr. (30)
Finally, by using the identification c, v(P)) and algebraic identities from Proposition 2.1, in particular with c = v(P)) and the bilinear algebra of $:
IJ (tt()t)))))))))))))))))) - (to) (to)) = [0 (x((1), x(t))) x" x(to)l
= = (x((t) + x(to), x(t)) =(to))
(31)
then (30 becomes
1 - to
(((x)) E(x),, x) dr + (x((1) + x(1o), xx((o)) 4
x -xx + o7 (x) dr ~ x- -f/" xTQx+ +xT KT RK5 dr.
(32)
Here, the last approximation in (2) is precisely IPA (14). Equation 32 is of the final form required
for learning update, all terms pertaining to the value function via the weights e; now appear as regression to update the critic network weights, and the control u(t) has been kept arbitrary so as accommodate excitation signals r(t) for training. The learning update matrices in (16) now follow directly from applying (32) at the sample instants (t& K
2 1/ क (f(x) + g(x)u E(x)) 2)) dr +Φ7 (x(t)) x(t)), x(ti) =(to))
fht T (x,x) dr
I A
2 Sh of(()) E(x), x) dr + (z(ti) + (ttt a(t)) a(tl-1))
fti Φ7 (T(xxx))
(33)
E, --=II(Q+K??)
(34)
Remark C.1 (IPA Increases Data Efficiency) IPA via 33 allows the current policy informa- tion to be pulled out of the integral reinforcement signal, enabling re-use of a single trajectory {a(t))re(taa)) from the real environment, which may include nonlincarity, uncertainty, and unmod- eled dynamics. This single trajectory can now generate all of the critic weights {c and policy iterates } from a single integration matrix I E Rix", without having to re-integrate the inte- gral reinforeement signal at each iteration i. For this reason, we call I (33) the IPA matrix. The IPA matrix has further structural significance to the IPA algorithm; indeed, its full rank forms the basis of our theoretical guarantees (ef. Section
17

---

Published as a conference paper at ICLR 2025
APPENDIX D PROOF OF THEOREM 3.1
Suppose that the sample count I E N and sample times {t=))== are such that I 20 has full rank n. We now proceed by induction i.
Suppose it has been proven for iteration 2 O that BK; is Hurwitz and that Kg. ....... K2 agree as generated from Kleinman's algorithm and IPA via 13
We claim that Ko.-.., K4t1 agree.
(6) has full column rank n. For suppose PPP) R is such that (şv(P) (P) = O. Examining 32 and proceeding through the derivation in
We first establish that the least-squares matrix θ E R x m
Section 2 and Appendix [ after applying the identity (24) we note for any symmetric matrix that
2,v = [v))), where N | N =N is given by
N (A BK,) P P(A BK.).
(35)
However, 35) is an ALE. Since N = -N and since it's been established that A BKG is Hurwitz, 335 has the unique solution P Jo ((A-BKK)* (-NNee NN-ee--AABBBB dt (Rodriguez 2004 Now, full rank o and that [vV = 0 imply e(N) = O, whence N = O. That N = 0 now implies vPP) 0 by the above. We have shown that θ; has trivial right null space, hence full column rank n.
Having established that θ; has full rank, now claim that P. R P. P.T > 0 uniquely solves the ALE (21) if and only c;
v(P.) satisfies the regression [6]. The forward direction was
already proved in the derivation ( (32 of Section [ and Appendix C Conversely now, suppose that u (P) E R2 minimizes the regression
16 Since it has been established that θ has full column p. >> 0 be the unique solution of the ALE 21,. the forward direction establishes that v(P)) E IR satisfies (16) at equality. Thus, v(P) = (P.), and as
rank, vPP) I2 unique. Next, letting
a result P P. (Proposition C.I The result is proved.
Having established the preceding, the proof now follows by induction on the algorithm iteration i 0,
APPENDIX E DECENTRALIZABLE ENVIRONMENT FOR FURTHER DATA EFFICIENCY
Consider a decentralized environment (J.g) (1) with N distinct control loops. To illustrate, we
present N loops; however, results readily generalize N loops:
- [ [
i i
(1)
9n(x)) g12(x) (21(z) 922(2)
][
+
(36)
U/2
No assumptions are made dynamic coupling between the loops; i.e., the loops may be coupled. (j = 1 1,....
Here, we define 2y E R Uj E R.m
.......) .N) with Σラ=1 ny = n and Σ my = m.
Such partitions appear in a variety of real-world applications, in particular, for the aircraft control problem studied (Stengel (022). In fact. decentralization is standard practice in control of HSVs (Dickeson al. 2009a b). so we refer the reader to these references for further details. Furthermore, such partitions can be found in robotics (Craig] 2005| Dhaouadi & Abu Hatab, 2013). helicopters (Enns & Si 2002 2003b a). and UAVs (Wang et 2016)) In this case, the IPA learning (T6) occurs in decentralized fashion each of the loops, thus reducing problem dimensionality. Note that the IPA method holds for general affine nonlinear systems (T) without decentralization. If the physics of the environment permit decentralization, it only helps further improve solution efficiency.
APPENDIX F REFERENCE COMMAND INPUT AND IPA ROBUSTNESS
F.1 REFERENCE COMMAND INPUT
Many SOTA ADP CT-RL algorithms require the persistence of excitation (PE) condition in proofs of algorithm properties (Wallace & Si 2024). PE is analytical condition, but it is not construc- tive, nor is there a verification procedure for PE. In essence, PE is in alignment with the notion of exploration enable learning and convergence. To achieve PE, it is standard practice apply
18

---

Published as a conference paper at ICLR 2025
probing noise d to the system ① in a feedback control of the form == ju(x))+ = sical feedback control principles, a good feedback control attenuates plant input disturbances; thus, plant-input probing noise excitation is inherently problematic practice (Rodriguez) 2004). Its consequence and insights on why such excitation signals may cause learning issues are thoroughly analyzed in (Wallace & Si 2024). We instead propose reference command input solution excite the closed-loop system at the favorable reference command r. Critically, reference command input compatible with current RL formulations and is of the required form u. = μu(x) d. Thus, refer- ence command input can improve learning of existing CT-RL algorithms. We have included block diagram illustrating reference command input a standard negative feedback structure (Rodriguez) 2004] in Figure 6 Standard in ADP CT-RL is to apply the control
According clas-
μ(xz) + as shown in Figure (i.e., with no reference command r(() = 0 and output = x). Instead, we form an error signal from user-designed reference command input in order to improve excitation properties.
probiing noise
impur
r
Up
f(x + g
Jelerence
crrur
control
plant
emvronment output
command
ingut
ingut
Control Policy
Nonlinear Environment
Figure 6: Reference command input in standard negative feedback structure.
To illustrate the issue with inserting probing noise alone, we first define few closed-loop maps. Definition F.1 (Closed-Loop Maps (Rodriguez 2004)) Examining Figure 6 we define:
Sy å Ta-uu, (sensitivity at the controls u): The closed-loop map from plant input distur- bance d to control
Tu A Ta-ou (complementary sensitivity at the controls u): The closed-loop map from plant input disturbance d to plant input Up-
Se A Te (sensitivity at the error e): The closed-loop map from reference command r to error e.
T. è Tr-y (complementary sensitivity at the error e): The closed-loop map from reference command to plant output
PSa. A Ta- (P-sensitivity): The closed-loop map from plant input disturbance d to plant
output y.
To illustrate typical input/output behavior, Figure 7 shows these two closed-loop frequency re- sponses for the HSV. Let us examine the frequency response in loop with flightpath angle
of the HSV (associated 7, yellow dashed curve). Since probing noise is inserted at the plant in- put, the effective closed-loop map from probing noise d output is the P-sensitivity Ta-ty- Thus, one glance at the yellow dashed SISO Ta-ty response in Figure Talreveals issues: Any probing noise will be attenuated by least -25 dB (a factor of about 20) at best-case. Furthermore, any probing noise frequency content below 10-2 rad/s and above 2.5 rad/s will be attenuated by more than -40 dB, or factor of 100. In light of this simple analysis, it is clear that achieving sufficient system excitation via probing noise injection alone proves to be significant issue for learning the nonlinear flightpath angle dynamics.
19

---

Published as a conference paper at ICLR 2025
P-Sensitivity Tag
PS.
Complementary Sensitivity at Error Try T.
-10
10
MOMO SISO
P MIMO
8150
-20
Pa
Pb SI50
Magnitude (dB)
Magnitude (dB)
-40
-10
-50
-15
10aa
10a
10
10
10*
10a
10
10
10
10
10*
10a
Frequency (rad/s)
Frequency (rad/s)
(a)
(b)
Figure 7: Closed-loop frequency responses: The probing noise injection issue visualized. (a): P. sensitivity Ta-ry- (b): Complementary sensitivity Ty-
F.2 IPA ROBUSTNESS
Theorem 3.1 shows that IPA inherits the guaranteed stability robustness margins of Kleinman's algorithm. These are, repeated here:
|Sull I dB,
"Tullz 2 dB,
where || denotes the 00 norm, and where Su, Ta denote the sensitivity and complementary sensitivity closed-loop maps the controls u, respectively (see Definition F.I.. These stability robustness margins are substantial, as they offer theoretical guarantees of stability to disturbances inserted in the feedback loop in Figure
For example, let denote the plant map Up - y in Figure Po be the nominal model of the actual plant P. and consider model uncertainty A the nominal model Po satisfying || S . for some M > O, but which is otherwise arbitrary. Then given each of the following uncertainty models, no such uncertainties A can destabilize the closed-loop system under the IPA policy provided the following stability robustness test holds:
Uncertainty Type
Uncertainty Model P Po A
Stability Robustness Test
Multiplicative
A
Divisive
P = P A1-1
S. #
M
owing to the small gain theorem (Rodriguez 2004).
APPENDIX G PERFORMANCE MEASURES
In these studies, we provide comprehensive evaluations of standard performance measures in: aver- age cost, learning success rate, and generalization with respect system ICs and modeling error. In addition, we analyze performance with respect to the following learning control measures:
cost (æo) obtained via the integral (2) delivered by the policy in the nonlinear optimal control task. As note, in controls conventions the cost (zz) > 0
Cost Performance: The
a positive number to be minimized (lower is better).
Relative Cost Performance: The difference J-Fvr(a)) JrPA(2: between the cost performance
of the respective FVI algorithm (i.c., * = c or r) and IPA (Note: means IPA is better; i.c., lower
cost).
Estimation Error: The difference ((x) - (xr) between the cost J(x) ( and the value function approximation (x)
11) & E IR..
20

---

Published as a conference paper at ICLR 2025
Learning Trial Success. In this work, we study 9 different closed-loop evaluation tasks on the three environments tested. A policy achieves a "success" in each of these tasks its closed-loop time response satisfies the respective criteria defined in Table 5
Table 5: Definitions of closed-loop evaluation tasks
Task
Description
Definition
SOS Task
State 21 regulation
(t))) within + 0.1 f 5
Pendulum Task
Pendulum angle / regulation
((t) within + + deg at f = 5s
HSV Task
Lax velocity V regulation
(t) within + 10 ft/s at = 50 s
HSV Task 2
Lax FPA y regulation
(t)) within + 0.1 deg f 50s
HSV Task 3
Strict velocity V regulation
(t)) within + ft/s at t = 25
HSV Task 4
Striet FPA 7 regulation
(() within + 0.01 deg at 25
HSV Task 5
Lax V and 7 regulation
HSV Tasks and 2 satisfied
HSV Task 6
Striet and y regulation
HSV Tasks and 4 satisfied
HSV Task 7
Lax and strict V and y regulation
HSV Tasks 1, 2, 3, and satisfied
APPENDIX H ADDITIONAL IMPLEMENTATION DETAILS
Hardware. We use PyTorch 1.13.1 for FVI implementations, and MATLAB R2022b for IPA im- plementations. All results are obtained on an NVIDIA RTX 2060, Intel i7 (9th Gen) processor.
Software: All Code/Data Available. All IPA and FVI code and datasets developed for this work is available in Supplemental and at (Wallace & Si 2025). FVI results (Lutter et al.) [021 20236)
were generated from the open-source repository developed by the authors (Lutter et al. 2023a)
implemented on the three environments studied.
H.1 ADDITIONAL. TRAAINING/EVALLAATTIOOO PROCEDURE DETAILS
This section provides additional implementation details for the training/evaluation procedures dis- cussed in Section 4
Training Procedure. An episode is initialized by resetting the environment and terminated time T' of the training horizon for collecting the state-action trajectory data (z,, u). A trial is a complete training process that contains series of consecutive episodes.
IPA learning requires state-action trajectory data from a single episode which usually has on the order of 1 100 total samples for the three evaluated environments. This low data complexity allows IPA to learn online from the actual physical process. Deep RL FVIs require training data from over million episodes (cf. Table [. for details see (Lutter et al. [02T (02TT As result, the only practical means of training FVI is in simulation. Since the modeling error v for a given system not known a priori, this means that FVI must train on the nominal model (modulo adversary perturbations in the rFVI case (Lutter al. 2023b)).
Random Seeds. Training and evaluation for each of the methods are based on 20 seeds for random number generation (RNG): 0-19 for training, and 100-119 for evaluation. In the case of the FVIs, the seeds are used in the environment, Numpy, and PyTorch for number generation. In the case of IPA, we have set MATLAB's master RNG seed for number generation.
Network Weight Initialization. For FVIs' deep networks, we use the identical network initializa- tion procedure in the original works (Lutter et al. 2021 2023b); namely, Xavier normal distribu- tion. The initialization gains of the layers used on each system can be found in Table of Appendix the same in previous studies (Lutter et al.) 2021 2023b).
For IPA's simpler quadratic network structure, we need only initialize the critic weights oo in We initialize each element of the critic weight vector co E R.. in the following uniform distributions
21

---

Published as a conference paper at ICLR 2025
for the SOS, pendulum, and HSV, respectively:
co ~ U(-15, 15),
(37)
co U(-25, 25),
(38)
(0((-3,,3))
for weights in translational loop j = for weights in rotational loop j = 2
(39)
Initializing the Environments. System initial conditions (ICs) for training and evaluation are gen- erated using uniform distributions U, where the ranges for the SOS, pendulum, and HSV cover the dynamics broadly, well beyond their linear regimes.
System Initial Condition Generation Training. System ICs for training and evaluation are generated using uniform distributions U. where the ranges for the SOS, pendulum, and HSV cover the dynamics broadly, well beyond their linear regimes. We use the following uniform distributions for the SOS, pendulum, and HSV, respectively:
Jo ~ (11,, 5o ~ (= rad, +M rad/s), To 150 ft/s, 1.5 deg, =5 deg, +5 deg/s, #0.01 ft).
(40)
(41)
(42)
where these distributions are centered about the respective system equilibrium point ze (cf. Appen- dices [or discussion of equilibria of each system).
System Initial Condition Generation - Evaluation. For the learning curves plotted Figure 2] at each algorithm iteration the return of the trained policies is evaluated over 100 episodes of the environment. For evaluation, system ICs for training are generated via the following uniform distributions U for the SOS, pendulum, and HSV, respectively:
Tg ~ =11, Zu U(= rad, = rad/s),
(43)
(44)
Zg ~ 100 ft's, 11 deg, #0.01 deg, #0.01 deg/s, #0.01 ft).
(45)
For display purposes of generating the surface plots in Figures 4 and 5 we evaluate the final polices of single trial for each method over the following evaluation grids x E G2 for the SOS, pendulum, and HSV, respectively:
G₂ 1inspace(-1,1, 150) x 1inspace(-1,1,150
(46)
G linspace(-xx m, 150) rad x Linspace(=7/4.7/ 150) rad/s,
(47)
G₂ = 1inspace(-1100) 100, 150) ft/s x 1inspace(-1,11 150) deg/s,
(48)
where all other ICs the higher-order HSV environment
their equilibrium values Ze-
Note that we choose the bounds of these grids the same as the bounds of the respective evaluation IC distributions // in (43) (45). This is done for illustration purposes display the policy performance over the evaluation regime. It is these grids which are used to generate the surface plots of Figures and [5 of the manuscript.
Modeling Error Generation. In the modeling error generalization studies of Section 5 modeling error tested at three severity levels: 0% (corresponding to the nominal model), 10% and / = 25% modeling error. Please see Appendices [ for an in-depth discussion of the modeling error parameters studied for each system. These are summarized in Table
Table 6: Modeling error parameters studied
Environment
Parameter
Equation Reference
SOS
212 unstable nonlinearity vxx)
(57)
Pendulum
Pendulum length L
59
HSV
Lift coefficient C2
22

---

Published as a conference paper at ICLR 2025
H.2
HYPERPARAMETER SELECTIONS
H.2.1
SHARED HYPERPARAMETERS
For the SOS, we use identical penalty selections those in the original ADP study (Vamvoudakis & Lewis 2010); namely,
I2:
R1 = 1.
(49)
For the pendulum, we use identical penalty selections those in the original FVI studies LLutter et al. 2021 2023b); namely,
di (1, (1,0.),
R1 0.5.
(50)
For the HSV, consider the decentralized design framework described in Appendix the following cost structure
L.2 choose
Q di (2, 2),
Ri 2.5,
Q2 = di aia5(2.5, 5, 0.05, 0),
R2 = 1.
(51)
H.2.2 IPA
For sake of illustration, on IPA we use simple LQ initial policies μo. We make the note that, of course, IPA accommodates nonlinear initial policies Po. However, to illustrate the practicality of our implementations for real-world design problems, we choose LQ initial policies as natural designer first-choice. For the SOS, we use the initial stabilizing policy
(o Koæ,
Kg 3.1952 6.7099
(52)
which we obtained from cost structure selections pendulum, use the initial stabilizing policy
Qı di dia a0..5 0.25), and Rú = 0.01. For the
Koo
K₂ 13.5108
5.8316 ],
(53)
which we obtained from cost structure selections Q a0.5 0.25), and R = 0.01. For the HSV in loop (j 1,,), use the initial stabilizing policies
no,1 (11) 11,
K 0.8944
2.7117 ],
(54)
(0.2(12) 2,,
Ko.2 1.5811
6.7007
0.8837 0.7135 ].
(55)
which we obtained from a decentralized design with cost structure selections 51 The remainder of the IPA hyperparameter selections can be found in Table 7
Table 7: IPA hyperparameter selections
SOS
Pendulum
HSV
Hyperparameter
Loop j =
Loop j =
Loop j =1
Loop j =
Sample Period T,, (s)
0.1
Number of Samples ly
10
15
15
15
Final Iteration iz
5
10
Ref Cmd ry
sin(5t)
10 10 10
50
100 t)
0.25
2
100
([-) deg ft/s, deg)
+5 sin 23
+5s
()
+0.5 sin
+5 cos
2
t)
+0.03 cos
Initial Policy Mo.j
52
53
54
55
H.2.3 CFVI, RFVI
Hyperparameter selections for cFVI and rFVI can be found in Table These parameter selections are overall quite standard and have indeed demonstrated great learning performance successes on second-order, unstable systems in previous studies (Lutter et al., 2021 2023b).
23

---

Published as a conference paper at ICLR 2025
Remark H.1 (FVI Implementation Challenges HSV System) The FVIs hyperparameter lections for the HSV Table 8 are a result of a search over the hyperparameter space in order to achieve the best policy performance possible. This process took approximately two weeks of trial and error. We find that FVIs struggle to converge on the HSV environment, likely due to its higher order, complex nonlinear structure, and its unstable, nonminimum-phar dynamics. In order to achieve FVI convergence, we found it necessary to initialize the weights to a policy trained on 40 iterations with the same hyperparameters given in Table 8 except a lower learning rate of le-7. In order to make IPA's learning procedure more analogous to FVT's on the HSV, we thus center its uniform initial weight distribution (39) about the weights of a nominal LQR policy performed on the nominal linearized dynamics (A,
As with our selections of the pendulum model structure and parameters (cf. Appendix K). for our pendulum studies have selected hyperparameters identical to those of the original CVT evaluations (Lutter et al. 2021 2023b). with two exceptions. In (Lutter et al. 2021 2023b). the au- thors use logcos control penalty function scaled so that its curvature the origin so that its curvature agrees with that of quadratic penalty u" Ru. In order to make comparisons consistent across the methods studied, and in order to produce more widely-applicable perfor- mance benchmark for real-world designers, we have decided to apply the standard quadratic control penalty u² Ru function of the pendulum angle state to be periodic in [0, 2Tr., a practice which we have dropped for consistency of comparison and generalizability of benchmarking. Finally, due to these changes we observed that more iterations were necessary for rFVI to converge in training the pendulum system
0 2R: i.c.
methods. Likewise, the authors in (Lutter et al. 2021 2023b) wrap the penalty
(cf. Figure 2). so we increased its iteration count from 100 previously (Lutter et al. 2021 2023h) to 150 here (cf. Table 8)
Table 8: cFVI, rFVI hyperparameter selections
SOS
Pendulum
HSV
Hyperparameter
cFVI
rFVI
cFVI
rFVI
cFVI
rFVI
Time Step (s)
0.008
0.008
0.008
0.008
0.04
0.04
Time Horizon (s)
20
20
Discounting 7
0.99
0.99
0.99
0.99
0.99
0.99
Network Dimension
96]
[3 x 96]
[3 96]
[3 x 96)
[3 96]
[3 x 96)
# Ensemble
4
4
4
4
4
Activation
Tanh
Tanh
Tanh
Tanh
Tanh
Tanh
Learning Rate
le-6
Ie-6
lc-5
le-5
lc-4
le-4
Weight Decay
1e-6
le-6
1e-6
le-6
1e-6
le-6
Hidden Layer Gain
1.41
1.41
1.41
1.41
1.41
1.41
Output Layer Gain
1.00
1.00
1.00
1.00
1.00
1.00
Output Layer Bias
-0.1
-0.1
-0.1
-0.1
-0.1
-0.1
Diagonal Softplus Gain B
1.0
1.0
1.0
1.0
7.5
7.5
Batch Size
128
128
256
128
256
256
# Batches
200
200
200
200
200
200
Eligibility Trace
0.85
0.85
0.85
0.85
0.95
0.95
n-step Trace Weight
1e-4
le-4
le-4
le-4
1e-3
1e-3
# Iterations
25
25
25
25
50
50
Epochs//teration
20
20
20
20
20
20
State Adversary
[|E
0.0
0.025
0.0
0.025
0.0
0.001
max
Action Adversary
Eu
0.0
0.1
0.0
0.1
0.0
0.05
Model Adversary
Eo
0.0
0.15
0.0
0.15
0.0
0.125
maax
Obs Adversary
0.0
0.025
0.0
0.025
0.0
0.01
24

---

Published as a conference paper at ICLR 2025
APPENDIX EVALUATIONS: QUANTITATIVE COMPARISONS BETWEEN IPA & FVIS
In this section, we examine the effects of modeling error / on 1) policy cost performance J, 2) critic network approximation error J- V, and 3) glosed-loop performance. All hyperparameter selections
can be found in Appendix H
I.1 AVERAGE COST AND EVALUATION SUCCESS RATE GENERALIZATION
IPA, cFVI, and rFVI are trained with trajectory data generated by the same training uniform IC distribution for each of the respective environments, and the average return is then evaluated over the respective evaluation uniform IC distribution. These distributions are given in Appendix [H] The learning curves are plotted Figure 2 20 training seeds. The corresponding average return and variance is tabulated in Table 9. and the corresponding evaluation success rate in Table 10
Table 9: Average cost at varying modeling error v/ = 0%, 10%, 25%
Cost
SOS
Pendulum
HSV
[I + 2x]
IPA
0.50=0.65
12.12=17.96
24.69: 14
0%
cFVI
0.50=0.66
1.99=10.04
29.79:t31.51
rFVI 0.60-0.67
16.26+
33.37++36.47
IPA
0.53-0.69
8.48111.85
27.191140.3
10%
cFVI
0.53=0.70
7.73+8..0
36.42436..49
FFVI
0.53=0.70
16.85+17.79
41.54+142.29
IPA
0.58=0.75
5.77+6.31
32.45
25%
cFVI
0.58=0.78
7.01
63.80:+44.73
rFVI
0.56
23.15
74.15
Table 10: Success rates for different environments and tasks at varying modeling error 0%,
10%, 25% (cf. Table for task criteria)
Success
SOS
Pendulum
HSV
HSV
HSV
HSV
HSV
HSV
HSV
V
[%]
Task
Task 1
Task
Task 2
Task
Task 4
Task 5
Task 6
Task 7
IPA
100
100
100
100
100
100
100
0%
cFVI
100
100
100
100
31.95
15.30
100
3.90
3.90
#FVI
100
100
100
100
19.20
8.75
100
1.25
1.25
IPA
100
100
100
100
100
100
100
100
100
10%
cFVI
100
99.50
32.70
9.95
99.50
2.35
2.35
FFVI
100
100
100
100
15.50
7.70
100
1.45
1.45
IPA
100
100
100
100
100
100
100
100
100
25%
cFVI
100
100
100
99.20
48,00
8.30
99.20
4.20
4,20
#FVI
100
100
100
5.70
1.95
1.95
Average Cost Performance. The learning curves of the tested methods trained to 20 seeds are plotted F The average cost of the tested methods is evaluated on three CT-RL enviromeets in Table 9 at three levels of modeling error = 0%. 10%, 25%. We include the evaluation success rates with respect to the closed-loop control tasks for the three environments and three levels of modeling emorr in Table T0 (cf. Table for for definitions of task criteria). Examining Figure 8 we see that IPA exhibits the fastest learning convergence of the methods tested, converging or less for all three environments. By comparison, FVIs converge at similar rate on the simpler SOS benchmark but require x 10 iterations to converge on the pendulum and 30 iterations on the HSV. With the exception of cFVI the pendulum, IPA also exhibits the lowest cost and variance between seeds.
iterations
Average Cast Performance/Success Rate SOS. Turning to Table 9 first that all three methods perform comparably on the SOS benchmark, exhibiting similar average and standard de- viation evaluated cost all three modeling error levels Similarly, all methods achieve 100%
25

---

Published as a conference paper at ICLR 2025
evaluation success the SOS regardless of modeling enror severity (cf. Table I0.. Thus, all meth- ods successfully generalize with respect to modeling error for this system.
Average Cost Performance/Succes Rate - Pendulum. Examining Figure 8 we that cFVI out- performs IPA in terms of average cost and variance on the nominal model V/ 0%. However, we would like highlight this result in the context of the closed-loop performance observed of these two methods. We examine the swing-up performance of the pendulum beginning from full pendu- lum displacement 0g = 180 in Figure As can be seen, the IPA response is the fastest of the three methods and is well-behaved with no overshoot. This hence points to trade-off between lower cost (achieved by cFVI) and faster swing-up up performance (achieved by IPA). Furthermore, IPA exhibits the best modeling error generalization on the pendulum (ef. Table 2 eventually outperforming cFVI in terms of average cost and variance when modeling error is increased. By comparison, rFVI exhibits the highest cost on the nominal pendulum model and struggles to generalize (ef. Table (9) However, all methods achieve 100% success rate the pendulum angle regulation task regardless of modeling error severity (cf. Table IOP.
Average Cost Performance/Success Rate- HSV. On the flagship HSV environment, IPA exhibits clear cost and evaluation success advantage over both FVI methods as well as the best modeling error generalization. Examining Table 9 the nominal model, IPA has the lowest average policy cost (24.69), followed by cFVI (29.79) and rFVI (33.37). The performance discrepancy increases with modeling error. At V = 25%, IPA cost (32.45, a +-11% degradation from nominal) far surpasses that of cFVI (63.80, a + 114% degradation) and rFVI (74.15, +122% degradation).
Tuming to the evaluation success rates for the HSV in Table 10 we see that IPA achieves a 100% success rate on all tasks regardless of modeling error severity. FVIs exhibit 100% success rate on the lax velocity regulation Task 1, and (with the exception of > 99% of cFVI at v 10% and 25% modeling error) 100% success on the lax FPA regulation Task 2. However, FVIs struggle with the stringent velocity regulation Task 3 and FPA regulation Task 4. As will be examined shortly (cf. Figure [7 and surrounding discussion), FVIs struggle with evaluation success rate due slow- decaying transients in their closed-loop time-domain responses, which prevent them from falling within the regulation thresholds.
100
45
90
Average Cost
3.5
:PV1
Average Cost
Average Cost
FV
35
2.5
30
25
50
20
15
0.5
10
15
Iteration
Iteration
Iteration
Figure 8: Learning curves of IPA and FVIs obtained over 20 seeds for the nominal models v = 0% of the SOS (left), pendulum (middle) and HSV (right). The shaded area displays the min/max range among seeds, as in the original works (Lutter et al. 2021: 2023b).
I.2 MODELING ERROR GENERALIZATION. COST PERFORMANCE
Figure 9 shows the cost difference J.PVı - JIPA between cFVI and IPA (first row), and the differ- ence J,rVI JIPA between rFVI and IPA (second row) for the nominal SOS model = 0% (left column), 10% modeling eтror 10% (middle column), and 25% modeling error 25% (right column). Note that wherever this difference is positive, IPA delivers better performance than the respective FVI algorithm. Table 11 presents the corresponding min, max, average, and standard deviation data. Figure 10 and Table 12 are laid out analogously for the pendulum, and Figure II and Table 13 are laid out analogously for the HSV.
Cust Performance SOS. Overall, the three methods tested perform comparably with respect to cost on the SOS benchmark. However, IPA exhibits slight performance edge, achieving the lowest
26

---

Published as a conference paper at ICLR 2025
cost pointwise regardless of modeling error level. The most pronounced performance difference occurs with cFVI at / = 25% modeling error (cf. top right plot of Figure 9 Here, one can see two distinct regions corresponding higher cost than IPA. As a note, this result is intuitive given that the modeling error parameter v (57) selected primarily affects the unstable 22 dynamics (cf. Appendix for discussion). Meanwhile, we can see that rFVI manages to successfully generalize to the modeling error, exhibiting similar performance IPA even the severe 25% modeling error.
large initial displacements in the state 22 for which cFVI has
Cost Performance Pendulum. Examining the performance on the nominal model in Figure 10 we see that cFVI performs quite well on the pendulum benchmark. IPA outperforms cFVI large region around the origin; however, cFVI outperforms IPA on the fringes of the state domain near full pendulum displacement advantage of cFVI erodes; in the mean, cFVI and IPA perform almost comparably at 10% modeling error, while IPA outperforms cFVI at 25% modeling error (ef. Table [2]. Thus, we conclude that IPA generalizes better with respect to modeling error than cFVI on this benchmark.
180 Once modeling error is introduced, the performance
Meanwhile, rFVI performs worse than cFVI on the pendulum overall and struggles to generalize. On the nominal model (bottom left plot of Figure TO). IPA outperforms rFVI in the same regions where it outperforms cFVI to a larger degree, and the regions in which rFVI outperforms IPA are limited to the very corners of the state domain. IPA outperforms rFVI in the mean for all modeling error levels v, and the performance discrepancy increases by a large degree from 4.99 nominally 25% modeling error (ef. Table 12). This degradation is corroborated visually in Figure I0 (cf. bottom row, moving left to right).
0% 19.02
Cost Performance HSV. On the flagship HSV model, IPA exhibits the lowest mean cost regard- Tess of the modeling error, and a distinct generalization advantage as the modeling error severity increases. On the nominal model (left column of Figure TT). FVIs and IPA perform comparably near the origin, but IPA begins to outperform FVIs by z 10 20-20 in the majority of the state domain. There are two fringes of the domain (corresponding to large initial FPA displacements r =1 deg) in which FVIs slightly outperform IPA; however, in the mean IPA outperforms cFVI by 13.14 and rFVI by 16.45 on the nominal model 0%.
Cost performance of both FVI algorithms degrades significantly when modeling error is introduced, in particular at the severe 25% modeling error (right column of Figure [T). Here, IPA outperforms FVIs pointwise, in the mean outperforming cFVI by 34. 10 and rFVI by 35.90.
27

---

Published as a conference paper at ICLR 2025
1
0.8
0.77
0.07
0.07
0.4
0.66
0.06
0.4
0.06
0.2
0.2
0.2
0
0.049
0.04
0.04
0.2
0.03
40.2
0.43
40.2
0.03
-06
-08
0.01
0.01
0.01
-1
-1
-0.5
0.5
-0.5
0.5
-1
4.5
0.5
21
0.08
0.08
0.8
0.8
0.6
0.07
0.07
0.6
0.07
0.06
0.06
0.05
0.05
0.04
0,04
0.04
4.2
-02
-0.2
0.03
004
4.4
0.4
4.6
0.02
0.02
0.01
0.01
0.01
-6.
0.5
-0.5
0.5
-4.5
0.5
Figure 9: Cost performance results of SOS model. First row: cFVI cost difference JeFVI - JIPA
2). Second row: rFVI cost difference J,FVI JIpa ② Left: Nominal model 0%. Middle:
10% modeling error v = 10%. Right: 25% modeling error v = 25%.
Table 11: SOS training cost/approximation data
Function
Data
V
0%
10%
25%
min 1.34--06
1.80c-06
3,24c-06
JervI JIPA
max
0.03
0.04
0.09
avg
0.01
0.01
0.02
std
0.01
0.01
0.02
min 2.29e-06
1.71e-06
1.99e-06
max 0.05
0.04
0.04
J.PVI JIPA
avg 0.01
0.01
0.01
std 0.01
0.01
0.01
min -0.01
-0.01
-0.01
J J-P IPA
max 9.166-08
0.03
0.10
avg -6.28e-04
0.01
0.03
std -6.83e-04
0.01
0.02
min 2.51e-06
6.53e-06
1.28e-05
J-V
cFVI
0.06
0.14
0.30
avg 0.02
0.05
0.10
std 0.01
0.03
0.08
min -0.11
-0.06
8,88c-03
J
J-V
rFVI
max -3.24e-06
-3.67e-07
0.12
avg -0.03
-0.01
0.03
std
0.03
0.01
0.03
28

---

Published as a conference paper at ICLR 2025
45
40
30
40
30
40
30
30
[deg, 8
15
15
15
10
10
3
0
30
-10
-30
-10
-30
-10
-45
-45
-180-135
(deg)
(deg)
0 (deg)
|deg/ 8
15
15
20
20
10
10
10
3
-15
-30
45
-45
-5
-180-135
-45
90 135
180
-180-135 -50
-45
45 (deg)
135
180
-180-135 80
0
90 135 180
(deg)
0 (deg)
Figure 10: Cost performance results of pendulum model. First row: cFVI difference J.FVT JIPA ② Second row: rFVI cost difference JrEVı JIPA ( Left: Nominal model = 0%. Middle: 10% modeling error 10%. Right: 25% modeling error 25%.
Table 12: Pendulum training cost/approximation data
Function
Data
0%
10%
25%
min -18.53
-10.21
-3.33
max
10.17
9.05
8.91
JeFVI - JIPA
avg -3.04
-0.52
1.48
std
6.43
3.91
2.87
min 14.25
-4.94
-2.68
max 27.36
31.35
48.74
JTFVI JIPA
avg 4.99
9.39
19.02
std
8.54
8.87
12.40
min 40.61
-35.84
-6.200
J-V IPA
max 4.78e-05
0.04
1.44
avg -9.88
-8.79
-5.59
11.13
9.91
7.63
min -8.90
-9.17
-9.15
J-
cFVI
max 10.71
6.85
5.20
avg
-0.02
-1.31
-2.06
std
4.02
3.40
3.17
min
-27.49
-26.56
-23.22
J-V
J
rFVI
max 13.71
15.33
43.98
avg -7.07
-6.48
0.40
std
9.11
9.19
13.87
29

---

Published as a conference paper at ICLR 2025
0.8
0.8
0.6
40
0.6
40
0.6
40
0.4
30
0.4
0,4
30
(deg)
0.2
0.2
20
20
10
10
0.4
-0.
0.6
-0.6
-0.6
0.8
-0.8
-0.8
-10
-50
V (ft/s)
(ft.
V (ft/s)
0.6
40
40
0.4
30
0.4
0.4
30
(deg)
0.2
0.2
0.2
10
10
10
0.6
-0.6
.05
0.8
-0.8
-0.8
-1
-100
-50
50
100
100
-100
-50
50
100
V (ft/s)
(ft/s)
V (ft/s)
Figure 11: Cost performance results of HSV model. First row: cFVI cost difference J.FVT JIPA 2). Second row: rFVI cost difference J,FVI JIPA ② Left: Nominal model v/ = 0%. Middle: 10% modeling error y 10%. Right: 25% modeling error 25%.
Table 13: HSV training cost/approximmation data
Function
Data
0%
10%
25%
min
-11.16
-8.51
19.18
max
13.14
21.77
45.83
J&FVI - JIPA
avg
5.63
9.76
34.10
std
5.23
5.60
5.56
min
-6.92
-2.90
23.61
J.PVI JIPA
max
16.45
21.75
49.97
avg 7.24
11.37
35.90
std
4.13
4.60
4.54
min 46.62
-50.55
-56.11
J-V
IPA
max 1.83e-03
0.02
0.20
avg -13.04
-14.76
-17.66
std
9.61
10.87
13.10
min 49.48
-37.75
-0.63
J-
J
cFVI
max 28.33
31.21
56.68
avg -7.77
-0.96
29.12
std
17.15
14.22
10.53
min -92.74
-79.03
41.03
J-V
#FVI
max
27.66
30.60
57.32
avg -25.27
-18.46
11.81
std
29.52
26.41
21.36
30

---

Published as a conference paper at ICLR 2025
1.3 MODELING ERROR GENERALIZATION: APPROXIMATION PERFORMANCE
13 and 14 show the critic network error J - V for IPA (first row), cFVI (second row), and rFVT (third row) the SOS, pendulum, and HSV systems, respectively. The corresponding data is tabulated in Tables 11 12
Figures 12
and [3 respectively. In general, it is desirable for the critic approximation error J V to be as small in magnitude as possible (so the critic is accurate).
1
0.8
0.8
0.8
0.6
0.2
0.2
0.6
0.2
0.15
0.15
0.15
0.1
0.1
0.1
-0.
0.05
40.2
0.05
002
0.05
-04
-0.4
004
0.6
-0.6
006
0.00
-0.0
0.00
-08
-0.8
-0.5
0.5
0.5
31
21
0.8
0.25
0.8
0.25
0.8
0.25
0.6
0.2
0.6
0.2
0.6
0.2
0.4
0.4
0.2
0.1
0.1
0.1
-02
0.05
40.2
-0.4
40.4
0
-0.4
-0.6
0.00
-0.6
-0.0
00.6
0.01
-08
00.8
.0.8
-0.1
001
27
371
0.8
0.25
0.25
0.8
0.25
0.6
0.6
0.6
0.2
0.4
0.4
0.4
0.15
0.15
0.15
0.2
0.2
0.1
0.1
0.1
00.4
0
0.4
-066
0.00
40.6
.0.6
0.00
0.8
00.8
008
0.1
-1
-0.1
Ø1
Figure 12: Critie NN approximation error J V. of SOS model. Left: Nominal model 0%.
Middle: 10% modeling error v = 10%. Right: 25% modeling error v 25%. First row: IPA.
Second row: cFVI (Lutter et al. 2021). Third row: rFVI (Lutter al. 2023b
Approximation Performance SOS. Examining Figure 12 IPA exhibits the lowest critic network approximation eтror regardless of modeling error severity and the best generalization overall. As the case with cost performance, cFVI struggles the most to generalize the SOS. Meanwhile, rFVI tends to underestimate policy performance on the nominal model and exhibits slightly higher approximation enror than IPA when modeling error increases.
31

---

Published as a conference paper at ICLR 2025
45
45
46
40
40
(deg/ (S)
20
15
20
10
10
0
-15
-10
-15
-10
-15
-10
-20
-20
-30
-55
400
-45
-40
-55
40
-180-135
-00
-45
0 (deg)
45
90
135
180
-180-135
400
-55
45 (deg)
90
135
180
-180-135
-00
-45
45 0 (deg)
0
90
135
160
45
45
45
40
40
30
30
30
20
20
20
3 (deg/ S
-10
-15
-10
-15
-10
-30
-30
400
-45
-40
-40
11801135
900 45
45
90 135
180
-180-135
90
445
45
90
135
180
-180-135 900
-45
45 (deg)
0
90 135 180
0 (deg)
0 (deg)
0
45
45
45
N
30
30
30
30
30
20
20
20
(deg 8
0
-15
-15
-10
-15
-30
-30
-30
-30
-30
330
-180-155
-45
90
180
.180-135
45
45 (deg)
135
-180-135
80
135
0 (deg)
0 (deg)
Figure 13: Critic NN approximation error J- V of pendulum model. Left: Nominal model v = 0%. Middle: 10% modeling error
10%. Right: 25% modeling error y/ 25%. First row: IPA.
Second row: cFVI (Lutter et al. (021). Third row: rFVI (Lutter et al. 20236).
Approximation Performance Pendulum. Examining Figure 13 IPA approximation error in- creases in the fringes near full pendulum displacement 1180 where IPA cost performance also struggles (cf. Figure TO). However, IPA exhibits low approximation error in wide band around the origin encompassing the majority the state domain, meeting or outperforming cFVI in this region. Furthermore, IPA estimation error improves as modeling error is introduced, demon- strating good generalization. Nevertheless, cFVI performs the best overall in terms of estimation error the pendulum benchmark, exhibiting both excellent performance and generalization to modeling error. Mcanwhile, rFVI performs worse on the nominal model and struggles to generalize in comparison IPA cFVI.
the nominal model
32

---

Published as a conference paper at ICLR 2025
Approximation Performance HSV. As is the case with cost performance, IPA exhibits pro- nounced approximation performance advantage on the nominal HSV model and the best generaliza- tion modeling error. On the nominal HSV model 0% (left column of Figure [4p, one
that both cFVI and rFVI struggle with overestimation in the band 7 E [-0.5", 0.5" % in particular
for large velocity displacements V +100 ft/s. IPA approximation error remains lower in this region. Furthermore, rFVI begins to underestimate policy performance for large FPA displacements y =1, where cFVI performs comparably better.
A Approximation performance significantly degrades for FVIs, in particular at the 25% modeling error (right column of Figure [4). Here, both FVI algorithms begin to overestimate policy performance to a large degree. Where rFVI begins underestimates policy performance on the nominal model near y 1 it performs comparably better than cFVI 25% modeling error. This suggests that rFVI adversarial input successful improves generalization properties of this robust variant somewhat. By comparison, IPA's approximation eroor remains at comparable levels and in similar radial pattern to its performance on the nominal model.
0.8
40
0.8
0.6
0.6
20
20
20
[deg]
-20
-20
0.2
0.2
-0.2
0.4
0.4
-0.4
0.6
-60
4.6
-60
-0.6
-80
-80
800
-100
-50
50
100
-100
-50
100
-100
-50
50
100
V (ft/s)
V
(ft/)
V (ft/s)
0.8
0.8
0.8
0.6
0.6
20
20
0.4
0.4
(deg)
0
-20
-20
0.2
40.2
0.2
0.4
-0.4
-0.
0.6
0.6
-06
4.8
-80
0.8
-80
-0.8
-100
-100
-100
V (ft/1
(ft/s)
V (ft/s)
0.8
40
40
0.6
0.8
0.6
0.4
0.4
0.4
(deg)
0.2
0.2
-20
-20
20
-40
-40
40
0.6
-0.6
-06
0.8
0..8
-0.
-100
-50
50
100
-100
V (ft/
(ft /s)
V (ft/s)
Figure 14: Critic NN approximation enror J - V of HSV model. Left: Nominal model 0%.
Middle: 10% modeling error = 10%. Right: 25% modeling error и 25%. First row: IPA.
Second row: cFVI LLttter et al. 2021). Third row: rFVI (Lutter al. (023b).
33

---

Published as a conference paper at ICLR 2025
1.4 MODELING ERROR GENERALIZATION: CLOSED-LOGD LOOP PERFORMANCE
We examine the closed-loop time-domain responses of all three environments at 0%, 10%, and 25% modeling error v. The associated plots for the SOS, pendulum, and HSV can be found Figures
15
16
and 17 respectively.
Closed-Loop Performance SOS. Figure 15 we plot the closed-loop regulation responses of the tested methods to the initial condition 2g II 1J" As can seen, regardless of the modeling error severity IPA and the FVIs perform very similarly. This corroborates the cost and approximation results in Figures 9 12 and Table 11 which establish that these methods successfully learn the same
optimal policy.
1
IPA
0.9
IPA
0.9
IPA
dPVI
0.8
<PVI
0.7
0.7
0.7
0.5
0.5
0.5
0.3
0.3
0.3
0.1
0.1
0.1
25
15
0.5
3.5 4.5
Time (s)
Time (s)
Time & (s)
Figure 15: Closed-loop response (t) SOS model to initial condition xy = [1, 11 Left: Nominal model 0%. Middle: 10% modeling error 10%. Right: 25% modeling error V 25%.
Closed-Loop Performance Pendulum. We examine the swing-up performance of the pendulum beginning from full pendulum displacement 6g = 180 Figure 16 As can be seen, the IPA response the fastest of all three methods regardless of the modeling error tested. We would like to highlight this result in the context of the cost and approximation performance of IPA illustrated in Figures and 13 respectively. In these two figures, it can be that IPA struggles for large initial pendulum displacements 0 = 180 however, Figure 16 shows that IPA's swing-up performance full pendulum displacement 0g 180 is the most responsive of the three methods. This hence points to trade-off between lower cost/approximation error (achieved by FVIs) and faster swing- up performance (achieved by IPA). This trade-off corroborated between the FVIs themselves. Comparing FVIs in T6] we see that rFVI exhibits faster response than cFVI; meanwhile, examining the state [0, 0j= [180", 0j in Figures 10 and 13 we see that cFVI achieves lower cost and approximation error than rFVT this same initial condition.
160
.PV1
140
FVVI
140
8()) (deg)
120
120
120
100
100
100
80
(1
20
0
0.5
Time (s)
Time
Time (8)
Figure 16: Swing-up closed-loop response of pendulum model 0(t) from hanging pendulum position
0o = T80% Left: Nominal model y 0%. Middle: 10% modeling error v 10%. Right: 25%
modeling emor 25%.
34

---

Published as a conference paper at ICLR 2025
Closed-Loop Performance- HSV. We examine the closed-loop performance the HSV in Figure 17 issuing a I00 ft/s step velocity command (top row) and a 1 step FPA command (bottom row). Beginning on the velocity response (top row of Figure 17),
that both FVI algorithms have large overshoot compared to IPA. Furthermore, the FVI responses exhibit pronounced and slow- decaying transients about the 100 ft/s reference command setpoint. This time-domain behavior is clear visual indicator of why FVIs struggle with evaluation success rate in velocity regulation (cf. Table 3 HSV Tasks I and 3). By comparison, IPA exhibits less overshoot, faster settling time, and a well-behaved system response without oscillatory transients.
Examining the FPA response (bottom row of Figure 17). we see that IPA and FVIs yield similar FPA overshoot. However, FVIs exhibit similar oscillatory transients as in the velocity response, giving IPA faster settling time. because of these oscillations that FVIs struggle with evaluation success rate in FPA regulation (ef. Table 3 HSV Tasks 2 and 4). Furthermore, such oscillations in FPA (i.e., in deflection the vehicle velocity vector) are highly undesirable in flight control settings, as they cause wear on the fuselage and may excite the HSV flexible modes (Bolender & Doman 2006a).
(ft/s)
40
40
TPA
IPA
IP4
cFV]
0
0
0
Time
Time (s)
Time
7()) (deg)
((E) (deg)
(degg)
IPA cFV]
IPA cFV]
IPA CPVI
50
Time (s)
Time (s)
Time (8)
Figure 17: Closed-loop responses of HSV model. Top row: 100 ft/s step velocity V command.
Bottom row: 1 step FPA 7 command. Left: Nominal model v 0% Middle: 10% modeling error V 10% Right: 25% modeling error V = 25%.
Time/Data Efficiency. Table lists key algorithm complexity parameters for IPA and FVI. In many cases, IPA exhibits orders of magnitude less time/data complexity than the SOTA FVI works. To illustrate, we examine the ratio of IPA/FVI on the HSV model. For number of data samples required: 1/5,750,000, training episodes required: IPA/FVI 1/5,250,000, training time: 1/1,500, algorithm iterations: 1/5.
35

---

Published as a conference paper at ICLR 2025
Table 14: Comparison of time/data complexity of IPA and FVIs 3 environments
SOS
Pendulum
HSV
Parameter
IPA
cFVI\FVI
IPA
cFVI/rFVI
IPA
cFVI/IFII
10
2.88c+07
15
8.63e+07
30
1.73e+08
# Trajectory Data Samples
12.88e+07
1
6.40c+05
1
2.63e+06
5.25e+06
# Episodes (Data Collection)
16.40c+05
0.20
1.42c+03
0.22
1.76c+03
2.75
4.60e+03
Avg Training Time (s)
/1.41c+03
/1.46c+03
4.79/c+03
5
25
5
25
10
50
# Algorithm Iterations i
SECOND ORDER ACADEMIC EXAMPLE (VAMVOUDAKIS & APPENDIX
LEWIS 2010)
J.1 SECOND ORDER SYSTEM (SOS) MODEL.
We consider the identical second order system (SOS) model used in the seminal ADP work Vamvoudakis & Lewis 2010), which has the following dynamics
11
21 + x2
0
2
0.5zz 0z5x(())
cos(2x1) + u,
(56)
(xx) A v vo(o),
vo(x) - x2 x2 (cos(2:2) 2x + 2)7).
(57)
The function db 57 contributes to the nonlinearity in the it drift dynamics f(x) ( for this system. also determines the stability properties of the origin equilibrium x. = 0. This system is open-loop unstable, and as the modeling error parameter v (57) increases above its nominal value v = 1, the system nonlinearity grows stronger and the SOS becomes more unstable. We have included the cigenvalues of the open-loop linearization of the SOS in Table 15 Thus, in this study we examine the more challenging perturbation direction >
Table 15: SOS instability versus modeling error parameter (7
// 57
Unstable Pole Location
1 nom, 0%)
3.8979
1.1 (= 10%)
4.3058
1.25 (= 25%)
4.9155
APPENDIX K PENDULUM MODEL & DESIGN FRAMEWORK
K.1 PENDULUM MODEL.
We consider the identical pendulum model used in the cFVI evaluations for this work, which has the following dynamics
(Lutter et al,
2021
2023b
i W,
mg L
sin 0
+ 7
(58)
21
where 0 is the pendulum angle (measured zero pointing upward, positive counterelockwise). w is the pendulum angular velocity, and the torque applied the pendulum base. The numerical values of all model constants are chosen identical to the cFVI evaluations (Lutter et al.] 2021 2023b) and are available in Table 16 We examine the upright pendulum equilibrium [o rad, rad/s]"". This equilibrium is naturally unstable (upright pendulum instability).
[te, wel
36

---

Published as a conference paper at ICLR 2025
Table 16: Pendulum model parameters
Definitin
Symbol
Value
Pendulum length
Lo = I m (nominal)
Pendulum mass
1 kg
Gravitational field constant
g
9.81 m/s²
Pendulum moment of inertia
3m1?
The pendulum length is a central physical parameter in the dynamics (58), as it determines the severity of the upright pendulum instability. In the studies conducted in this work, will focus how modeling errors the pendulum length affect the pendulum dynamics and learning perfor- mance. Specifically, we study modeling errors of the form
L = v To:
(59)
where Lg IR a nominal value of the pendulum length, and R is the modeling error parameter (nominally 1). As < 1 decreases, L < Lo decreases. Table 17 shows the inverted pendulum instability as function of the modeling error v (59). As can be seen, the pendulum becomes more unstable with decreasing pendulum length I.. Thus, the modeling error perturbation direction < (59 studied in this work.
Table 17: Pendulum instability and control effectiveness versus modeling error parameter v (59
1 (59)
Unstable Pole Location
(nom, 0%)
3.8360
1.1 (= 10%)
3.6575
1.25 (= 25%)
3.4310
APPENDIX L HSV MODEL AND DECENTRALIZED DESIGN FRAMEWORK
L.1 HSV MODEL
The HSV model used this study is the standard Wang and Stengel model developed (Wang & Stengel 2000); Marrison & Stengel, 1998) based on NASA Langley's 's winged-cone tabular aero- propulsive data (Shaughnessy et al. 1990.. This model has served as a standard testbed for HSV control development and has since been used in seminal classically-based works such as (Xu et al.) 2003 2004), and simplified variants of it have been used in state-of-the-a RL-based control works
such (Zhao et al. 2023 Xu et al. 2013 2015):
T Coos
D
p.sing
V
m
x2
L + T ssn
(μ - V²r) cos
3
mV
2
Ô q.
q = M
Iyv
h sin 7
(60)
The HSV (60) is fifth-order, with states x = [V, 7, 0, M²" Here, V is the vehicle airspeed, the flightpath angle (FPA), a the angle of attack (AOA), 0 a + the pitch attitude, y the pitch rate, and h the vehicle altitude. In addition, r(h)
RØ the total distance from the earth's 's center to the HSV, Rg = 20, 903, 500 is earth's 's radius, and μ 2 Gmmg = 1.39 10 ft3/s², where G is Newton's 's gravitational constant and mg is the earth's 's mass. As note, the notation in (60) is standard in flight control literature and is not to be confused with the learning community's use μa denote a control policy. L, D, T, M the lift, drag, thrust, and pitching moment, respectively, and are given by
L - -=
1
D D= DV-SSn
T=
M - -V
1
(61)
37

---

Published as a conference paper at ICLR 2025
where p is the local air density, S 3603 ft is the wing planform area, and 80 ft is the mean acrodynamic chord of the wing. Air density p and speed of sound a are modeled as functions of altitude h by
0. 0.00238e
24,000
(62)
a 8.99 x 10 %/2 9.16 x 10 h/ 996,
(63)
and Mach number M V The aerodynamic coefficients CL. CD. CM, and Or are heavily nonlinear functions of the flight condition as follows
Cr. Cr,a + CL,Eg
(64)
1.91
CL.a CL,a0,
CL - (0.493 +
(65)
M
CL, (-0.2356a² - 0.004518a 0.02913) dg,
(66)
CD 0.0082 (171a² + 1.15a + +1) (0.0012M 0.054M 1),
(67)
CM CM, CM.. + CM,6n
(68)
Cи. 10 (0.06 - -e- (-65650² 6875a + 1),
(69)
CM.4
qC
(-0.025M 1.37) (-6.83a² + 0.303a 0.23).
(70)
2V
CM,6
0.0292(6 a),
(71)
[0.0105 (1 + M) (1+0.15) 8 <1 87 21,
CT
(72)
0.0105 (1+ M (1+00156)
17
where óg the elevator setting, 8r is the throttle, and v E R is the modeling error parameter
(nominally 1) in the basic lift increment coefficient CL.a (65) The controls are given by "
18r, óg and we examine the outputs y [V, ")" As in (Wang & Stengel 2000 Marrison & Stengel, 1998). we examine level flight condition = 0, = 0% at M. = 15, he = =110, 000 ft, which corresponds to an airspeed V. = 15, 060 ft/s. At this flight condition, the vehicle is equilibrated at ae = 1.7704 by the controls ~τ,- 0.1756 (T. = 4.4966 x 10a lb), òg.e -0.3947%.
HSV Dynamical Challenges. The HSV is significant control challenge due to its dynamical features. First, the HSV open-loop unstable. Linearization of the model about the equilibrium (Ie, ue) has cigenvalues at s -0.8291, 0. 7..7666 shhort-priood modes), -0.00001 + 0.0276j (phugoid modes), and = 0.0005 (altitude mode). The unstable short-period mode at s = 0.7165 associated with the vehicle pitch-up instability (long vehicle forebody, aftward-set center of mass due to propulsion system). As is common with tail-controlled aircraft, the elevator-FPA map is nonminimum phase (Bolender & Doman 2005). The linearization has transmission zeros at 8 8.3938, -8.4620, the right half plane zero at = 8.3938 attributed to the elevator-FPA map (parasitic negative lift increment due pitch-up elevator deflections).
Reducing the lift coefficient 1 < (65) represents degraded lift efficiency and a more difficult vehicle to control dynamically. We have calculated the unstable pole location and right-half-ples plan- zero location as a function of the modeling eтror parameter // (65) in Table [8] As can be seen, the system instability decreases slightly with increasing modeling error, but the nonminimum phase zero gets closer to the origin. Thus, the pole/zero ratio drops from 11.72 nominally (u = 0%). to 11.66 (v 10%), to 10.88 (v 25%), presenting a greater control design challenge. Thus, we study the modeling error perturbation direction < 1 in this study.
Table 18: HSV instability and nonminimum phase zero versus modeling error parameter v (65
(65
Unstable Pole Location Nonminimum Phase Zero Location
(nom, 0%)
0.7165
8.3938
0.9 (= 10%)
0.7011
7.9619
0.75 (= 25%)
0.6681
7.2664
38

---

Published as a conference paper at ICLR 2025
L.2 HSV DECENTRALIZED CONTROL FRAMEWORK
This work implements a decentralized design methodology inspired by (Dickeson et al. 2009a) de- veloped for HSVs and extensively tested on HSVs. Here, policies are designed separately for the velocity loop (associated with the airspeed V and throttle control ōγ) and rotational loop (associated with the FPA y. attitude 8, g, and elevator control ,g). As in (Dickeson et al. 2009a), for controlla- bility reasons we do not feed back altitude h in the learning control, though altitude is still included in the nonlinear simulation. To reduce overshoot due to step reference commands r and initial con- dition Zo transients, we include in the design a reference command pre-filter outside the feedback loop (a standard control design practice for HSVs, for further discussion see, e.g., (Dickeson et al.) 20.9a)).
In order to achieve zero steady-state error to step reference commands, we augment the plant at the output with the integrator bank f y dr (av, =n"" [JV V dt, I the state/control vectors are thus partitioned as 21 = [zv, VJ=, u1 = dr (11 = 2, mn = 1) and
dr] T For decentralization.
I2 = 12y, 7. 0, q]" U2 = óg (n2 = 4, m2 = 1).
APPENDIX M ADP-BASED LEADING CT-RL DESIGN INSIGHTS AND PERFORMANCE LIMITATIONS
ADP approaches have been developed largely within the scope of seminal works such as inte- gral reinforcement learning (IRL) (Vrabie & Lewis, 2009), synchronous policy iteration (SPI) Vamvoudakis & Lewis, 2010), robust ADP (RADP) (Jiang & Jiang] 2014). and continuous-time value iteration (CT-VD) (Bian & Jiang 2022). These methods achieve substantial theoretical results. As result of ADP's theoretical frameworks in adaptive and optimal control, Lyapunov arguments are available to prove qualitative properties including weight convergence and closed-loop stabil- ity results. However, the results require restrictive theoretical assumptions. For complete list of assumptions required by 1) IPA, 2) leading ADP CT-RL, works (see above), and 3) the SOTA FVI works (Lutter et al. [021] 2023b), please see Appendix A In the case of ADPs, these are difficult to satisfy for even simple academic examples, and as a result these methods exhibit empirical issues (Wallace & Sil 2024). These issues impede algorithm performance long before the methods may be substantively evaluated for generalization to varying system ICs and modeling error, as we do for IPA and FVIs Section [ The in-depth numerical studies (Wallace & Si 2024) find that these methods fail to synthesize controls for simple second-order systems with known closed-form solu- tions. A small change in the basis of such small problem led to learning failure. We will illustrate this same phenomenon in the coming evaluation.
M.1 SYSTEM AND ABLATION OF CRITIC BASES
Second Order System (SOS) Studied /Vamvoudakis & Lewis 2010). In order to illustrate these empirical issues, we will study the performance of the four seminal ADP works the academic second order system (SOS) studied in this work (cf. Appendix D). The SOS is a suitable benchmark environment for these ADP methods; indeed, the model was originaly developed for evaluation in the SPI work (Vamvoudakis & Lewis 2010). The dynamics of the SOS are given in Equation (56). The SOS is academic example constructed such that, for the nominal model
and with the choice of state and control penalties Q = /2. R = 1 given in 49). the optimal value V* and optimal
policy fe" are known priori in analytic form as
V*(x) = 2 + 22
(73)
2
4 (x) = (cos((2r) + 2)0².
(74)
As is the case with all classes of learning algorithms, central hyperparameter of ADPs is the basis functions chosen for the critic neural network. The seminal ADPs all study a linear approximation structure of the following form
V(x)= = (x))))
(75)
where ((x) Φ1 (x)
(2)
(x) 1 E R is the critic bases consisting of M E N
basis functions, and E TR- is the critic weight vector.
39

---

Published as a conference paper at ICLR 2025
Critic Bases. In this study, we will demonstrate ablation sensitivity of the ADPs with respect to the following two natural choice critic bases:
Ablation:
<(x) = x <(x) = o
01
(76)
2122
of 2/02
xa2z 21 1",
(77)
The first choice of bases Φ (76) is identical to the choice of bases in the original SPI study for this system (Vamvoudakis & Lewis 2010). The second choice of bases ø (77) contains the first as subset, and is chosen for illustration because it is identical to the choice of bases in the original IRL study (Vrabie & Lewis 2009) on similar academic example. Thus, these bases are well-motivated choices for study and have been demonstrated previously by the leading ADP works.
Crucially, both choices of bases (6).. (77 can achieve exact approximation of the optimal value V* and inspection of (73) shows that the optimal critic weights c* for the bases (76). (77) are given respectively as
1 1 T
(78)
0 001000001
(79)
M.2 SETUP AND IMPLEMENTATION
M.2.1
NETWORKS AND INITIALIZATION
Initial Stabilizing Policy. For all methods, we initialize the critic weights to = [1, 0, 4|" for (76) and cp = [1, 0, 4, O, O, O, 0, OJ for (76). These critic weights were chosen to implement the same initial stabilizing policy pi for all works, given by
po(x) - 4(cos (2x1) 2),2
2 R g (x) dz (o)(x)c)
(80)
This policy Ao chosen so that it can be implemented in the single-network control policy struc- tures used by IRL. (Vrabie & Lewis 2009) and SPI (Vamvoudakis & Lewis 2010).
Actor Neural Network: RADP and CT-VL Of the four seminal ADPs, the RADP (Jiang & Jiang, 2014) and CT-VI (Bian & Jiang 2022) methods make use of self-standing actor neural network, whose basis functions we will denote (ar). For these networks, we choose the minimal bases required to approximate the optimal policy n² (74) and in order to implement the common initial stabilizing policy pn (80
(xx
It2
cos(2x1) )az I". cos(2x1) )x
(81)
For reference, the optimal actor network weights w E R24 for this problem are w" = [0, -2, 0, -1'
Hamiltonian Neural Network: CT-VI. The CT-VI method makes use of a novel network to ap- proximate the optimal Hamiltonian function, for details see the original work (Bian Jiang/ 2022) and subsequent discussions in (Wallace & Si, (024). We shall denote the basis functions of the Hamiltonian network by (zr), and likewise we use minimal bases to approximate the Hamiltonian:
(x)
x2 cos? (2x1)
x2 (20()2))
x x2 l".
(82)
For reference, the optimal Hamiltonian network weights o² E IR.t for this problem o² [1, 4, -1, 3
M.2.2 HYPERPARAMETER SELECTIONS
All ADPs collect state-action data under feedback control of the form u(x) + d, where d is a probing noise excitation. The policy μ may be kept constant at the initialization 14g, or it may be updated online (for specific details, the respective reference of each method).
40

---

Published as a conference paper at ICLR 2025
Algorithm Hy Hyperparameters. We choose hyperparameters for the ADPs with the following siderations: 1) The selections reflect in-depth quantitative hyperparameter ablation studies con- ducted on these algorithms previously (Wallace & Si. 2024). 2) The selections reflect those of the original ADP studies (Vrabie & Lewis, 2009; Vamvoudakis & Lewis 2010 [Tang & Jiang 2014] kept constant across methods wherever possible for sake of consistent benchmarking, and 4) The selections represent natural designer first-choices to illustrate key design insights in the algorithm empirical behavior. For descriptions of the algorithm hyperparameters and learning procedure, see the respective development work
Bian & Jiang. 2022p. 3) The selections
All methods except IRL accommodate probing noise. For SPI, RADP, and CT-VI, we use the prob- ing noise d(t) = 5 cos(t). For these three methods, we initialize the state to the origin zo = 0. Since IRI. does not accommodate probing noise, its only means of excitation is through the initial condition, which we set to to [10. 10 For IRI., we run i* 5, collecting / = 15 data samples sample period T. 0.1 s. For SPI, which tunes its weights dynamically online, we tune the weights for f 500 s. with tuning gains a a2 10, Fi O, and F2 = 5/M. For RADP, we run i = 10 iterations, collecting = 15 data samples at a sample period T, = s. For CT-VI, which tunes its weights dynamically at an independent time the simulation time t, we collect data for t = 50 s and tune for s = 50 s.
M.3 ABLATION STUDY
We first run each algorithm using the tuning hyperparameters discussed in Appendix M.2 on the minimal basis (76). The results of this training can be found in the top row of Figure 18 which plots the critic weight responses for the four seminal ADP algorithms. As can seen, all methods successfully converge to the optimal weights c* (78) for the small basis & (76). For SPI in par- ticular, this is an independent validation of the convergence results established in the original study
Vamvoudakis & Lewis 2010). Thus, we have established a performance baseline: All methods suc- cessfully converge for these hyperparameters and choice of critic bases. The critic weight responses
well-behaved and exhibit fast convergence.
We then run all four algorithms with the same hyperparameters on the slightly expanded basis ø (7) and plot the resulting critic weight responses in the bottom row of Figure 18] All methods fail to converge to the optimal weights c* (79) for the expanded basis & (77). even though this basis can achieve exact approximation of the known optimal value V (73).
We now turn to the bottom row of Figure 18 for an analysis the failure modes of each algorithm. iterations. For SPI (Vamvoudakis & Lewis 2010), one can see that the weights do not diverge. However, the weight values c(t) after f 6.75 s for SPI fail to stabilize the closed-loop system. To see this, we have plotted the corresponding state response ztt) of SPI's training in Figure 19 As can be seen, the state diverges due the destabilizing SPI policy. For RADP (Jiang Jiang 2014). the weights successfully converge; however, RADP fails to converge to the optimal weights e* (79); indeed, elements oj
For IRL Vrabie & Lewis, 2009), the weights diverge after
r 0, yet examination of the RADP response Figure T8 reveals that RADP zeros none of these weights. Finally, for CT-VI (Bian & Jiang 2022). the weights diverge. These weight divergence issues have been exhibited by CT-VI in previous evaluations (Wallace & 2024).
The elegant results of ADP-based leading CT-RL methods have made significant con- tributions to CT-RL. However, further algorithm development work is required to enable these al- gorithms to synthesize for meaningful applications, beyond second order academic examples with priori known optimal solutions.
Conclusion.
41

---

Published as a conference paper at ICLR 2025
2.5
Iteration
Time (8)
Iteration
Training Time (500)
2.5
1110
-10
1113
0.11
Iteration
Time (8)
Itcration
Training Time (sec)
Figure 18: Critic weight responses of ADP methods for minimal basis Φ 76 (top row) and expanded
basis @ (77) (bottom row). First column: IRL, second columm: SPI, third column: RADP, fourth
column: CT-VI All methods successfully converge to the optimal weights e* 78 for the small basis
766). but all fail to converge to the optimal weights e* (79) for the expanded basis ø (77).
Time
Figure 19: State response :rt) of SPI when training on the expanded basis 77
The state diverges
after & 2J 6.75 as the SPI weights fail to stabilize the closed-loop system.
42


Reviews:
[{"review_id": "RABjaK48IU", "reviewer": "Reviewer_ZBUg", "summary": "This paper introduces integral performance approximation (IPA) for continuous-time model-based reinforcement learning control of control-affine nonlinear systems.", "questions": "1. Does the state-dependent part of the cost need to be quadratic? Specifically, can the $x^T Q x$ term be generalized to some positive semi-definite function Q(x). I can understand the challenges generalizing so for the $u^T R u$, but $x^T Q x$  must be easy if linearizations are being used in the integral approximation anyway.\n\n2. Does the linearization in (13) need to be necessarily around x=0? What would be difficult about constructing a function $K_i(x)=-\\frac{\\partial}{\\partial x}\\mu_i (x)$? across a range of values of x, similar to gain scheduling methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, the contributions are clear and of significant importance for solving continuous-time optimal control problems. The simulation and ablation studies are extensive and make a compelling case for the developed controller.", "weaknesses": "My main concerns are regarding the general mathematical rigor in the theoretical results. Specifically, there are a lot of linearized approximations throughout the paper, which by itself is not necessarily problematic. However, then one would expect the unknown approximation error to be accounted for in the analysis with some appropriate bounds. Instead, there are a lot of $\\approx$ relations in the paper (e.g., (14), (15), (30) etc.) without accounting for the errors.  Specifically, in Eq. (13), K_i is defined as $$-\\frac{\\partial}{\\partial x}\\mu_i (x)|_{x=0},$$ and then it is said \n$$\\mu_i \\approx -K_i x.$$\nThis linearization is foundational to the entire paper. At that point one can also linearize the dynamic model itself, i.e., \n$$f(x)+g(x)u \\approx Ax+Bu$$ and then apply the entire development accordingly. This makes me wonder if the development is appropriate for the nonlinear system in Eq. (1) like it is claimed.\nInstead of writing $\\mu_i \\approx -K_i x,$ it would be more appropriate to write $\\mu_i = -K_i x + O(||x||^2)$. The linearization error would then grow quadratically with the states, but for the region $||x||\\leq r$, there exists some constant $c_1$ such that can be bounded by $ O(||x||^2) \\leq c_1 r^2$. Similar analysis needs to be performed for all of the other approximations in the paper to achieve a local stability result with robustness to small perturbations. The way the paper stands now, the analysis is valid only in an infinitesimal neighborhood of the origin.\n\nBesides, the literature review surrounding CT-RL and ADP methods is sparse. In Appendix M, the following vague comment is made about ADP methods:\n\n \"As a result of ADP’s theoretical frameworks in adaptive and optimal control, Lyapunov arguments are available to prove qualitative properties including weight convergence and closed-loop stability results. However, the results require restrictive theoretical assumptions which are difficult to satisfy for even simple academic examples, and as a result these methods exhibit empirical issues.\" \n\nHowever, it is not specified what theoretical assumptions are restrictive and difficult to satisfy. Besides the literature and ablation study on ADP is mostly focused on the old result by Vamvoudakis and Lewis (2010). Besides, a lot of further development has happened after this classical work. For example, see the following works and references citing them/therein:\n\nKamalapurkar, R., Walters, P. and Dixon, W.E., 2016. Model-based reinforcement learning for approximate optimal regulation. Automatica, 64, pp.94-104.\n\nVamvoudakis, K.G., 2017. Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach. Systems & Control Letters, 100, pp.14-20.\n  \nIn Appendix F, the authors mention \"Many SOTA ADP CT-RL algorithms require the persistence of excitation (PE) condition in proofs\nof algorithm properties\". There are many newer results which relax the PE condition with initial/interval/finite-time excitation conditions. See the following references for example:\n\nJha, S.K., Roy, S.B. and Bhasin, S., 2019. Initial excitation-based iterative algorithm for approximate optimal control of completely unknown LTI systems. IEEE Transactions on Automatic Control, 64(12), pp.5230-5237.\n\nYang, Y., Pan, Y., Xu, C.Z. and Wunsch, D.C., 2022. Hamiltonian-driven adaptive dynamic programming with efficient experience replay. IEEE Transactions on Neural Networks and Learning Systems.\n\nThe authors can discuss these newer results that relax the PE condition and analyze how they relate to or could potentially improve the proposed IPA method.\n\nEDIT: Increased the score to 8 after the revisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kuEGxf39IK", "reviewer": "Reviewer_5RXP", "summary": "This paper introduces Integral Performance Approximation (IPA), a new method for continuous-time reinforcement learning (CT-RL) control. IPA utilizes an affine nonlinear dynamic model that partially captures the environment's dynamics, alongside state-action trajectory data, to enable highly data-efficient and robust control. The approach incorporates structures from the Kleinman algorithm to ensure theoretical guarantees for learning convergence, solution optimality, and closed-loop stability. The effectiveness of IPA is demonstrated across three CT-RL environments, including hypersonic vehicle (HSV) control, which presents additional challenges due to unstable and non-minimum phase dynamics.", "questions": "All the questions are listed in the weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a CT-RL method, IPA CT-RL, which leverages an affine nonlinear dynamic model and quadratic cost structure for data-efficient, robust control. It provides theoretical guarantees for convergence, optimality, and stability, validated through extensive evaluations. Finally, it demonstrates IPA CT-RL's successful application to HSV control.", "weaknesses": "However, there are still some aspects of this paper that require clarification:\n\n1. The paper mentions several SOTA methods that are not restricted to control-affine systems, such as Yildiz et al. (2021). However, the authors did not include these methods in their simulations for comparison. Could the authors explain the reasoning behind this choice?\n\n2. The authors briefly mention the discretization of continuous-time environments in Yildiz et al. (2021), suggesting that this process could lead to significant numerical issues for real-world systems. However, their proposed method also relies on discrete data points when performing integration, which could introduce discretization errors. A previous study [1] has analyzed this issue in detail. I believe the authors should at least discuss the impact of discretization error in their approach.\n\n3. The authors suggest that HSV represents a SOTA environment, but it still appears to be relatively low-dimensional. It’s unclear why their approach cannot scale to higher dimensions, especially since the method in Yildiz et al. (2021) seems capable of handling more complex systems. The authors should discuss the limitations that might prevent their approach from scaling up.\n\n4. I am unclear about the theoretical advantage of the IPA method. Is this benefit primarily due to a linearization structure or another feature? I recommend that the authors provide a clear explanation of this through a example. Additionally, it’s unclear if IPA applies effectively to all control-affine systems or only to those with high linearity. The systems simulated by the authors do not exhibit a high degree of nonlinearity, so clarification here would be helpful.\n\nIf the authors can address these four points clearly in the updated version of the paper, I would be inclined to raise my score.\n\n[1] Cao W, Pan W. Impact of Computation in Integral Reinforcement Learning for Continuous-Time Control[C]. The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "71scAZajMW", "reviewer": "Reviewer_K99L", "summary": "This paper proposes a continuous-time reinforcement learning control method. The critic network design is novel. However, the theoretical analysis is questionable, which is difficult to follow. Simulation results on three optimal control tasks show that the proposed method outperforms SOTA methods.", "questions": "(1) It is suggested to introduce the value function $V$ (formula (7)) before Section 2.1. In addition, please use different notations to represent the value function (e.g. $V$) and the critic network (e.g. $\\hat V$). In this paper, the authors sometimes replace the value function with the critic network in their theoretical analysis.\n\n(2) In (5), the notation $V$ in the function $H$ should be $V^*$.\n\n(3) In Fig. 1, the HJB equation is unrelated to the CT temporal difference equation. \n\n(4) In (13), why do you linearize the controller $\\mu (x)$ at $x=0$? For states which are far away from the origin, the linearized function could not approximate $\\mu (x)$ accurately.\n\n(5) The reviewers are confused with the content in Appendix C, as formulas (16) and (17) can be obtained directly based on the Bellman equation (8), the critic network (9), and linearized controller (13).\n\n(6) The proof of Theorem 3.1 is hard to follow. It seems the authors overlook that the system considered in this paper is affine nonlinear.\n\n(7) In appendix F, the authors should explain clearly that how the reference command input and the error influence the control policy, which could be an important trick in improving the learning performance of the proposed method.\n\n(8) Since the system is known, the authors are encouraged to further compare their method with some classic control methods which do not consider optimality. In addition, could the authors explain why the SOTA baselines are significantly sample-inefficient?\n\n(9) Is it possible to discuss the learning robustness of the proposed method from a theoretical perspective?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) Extensive simulations are conducted.\n\n(2) The proposed method is robust to model uncertainty empirically.", "weaknesses": "(1) The proposed method is model-based, which only considers affine nonlinear systems.\n\n(2) There are some theoretical issues in the paper.\n\n(3) The advantage of the proposed method in improving learning robustness is not analyzed theoretically.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)