Online Control-Informed Learning
Authors: Zihao Liang
Tianyu Zhou
Zehui Lu
Shaoshuai Mou
Venue: Accepted by TMLR ()

Abstract: This paper proposes an Online Control-Informed Learning (OCIL) framework, which employs the well-established optimal control and state estimation techniques in the field of control to solve a broad class of learning tasks in an online fashion. This novel integration effectively handles practical issues in machine learning such as noisy measurement data, online learning, and data efficiency. By considering any robot as a tunable optimal control system, we propose an online parameter estimator based on extended Kalman filter (EKF) to incrementally tune the system in an online fashion, enabling it to complete designated learning or control tasks. The proposed method also improves the robustness in learning by effectively managing noise in the data. Theoretical analysis is provided to demonstrate the convergence of OCIL. Three learning modes of OCIL, i.e. Online Imitation Learning, Online System Identification, and Policy Tuning On-the-fly, are investigated via experiments, which validate their effectiveness.

Published in Transactions on Machine Learning Research (03/2025)
Online Control-Informed Learning
Zihao Liang
1ang3331 @purdue. edu
School of Aeronoutics and Astronautics
Purdue University
Tianyu Zhou
2hou10438yurdue.cc edu
School Acronoutics and Astronautics
Purdue University
Zehui Lu
hu8463purdue.. edu
School of Aeronautics and Astronautics Purdue Universily
Shaoshuai Mou
mous@purdue... edu
School of Acronautics and Astronautics Purdue University
Reviewed on OpenReview: https: // ottp:///eeervvee....m net/ hta/fforus forum? id= LDavZEV115
Abstract
This paper proposes an Online Control-Informn Learning (OCIL) framework, which employs the well-established optimal control and state estimation techniques the field of control to solve a broad class of learning lasks in an online fashion. This novel integration effectively handles practical issues in machine learning such as noisy measurement data, online learning, and data efficiency. By considering any robot an online parameter estimator based on extended Kalman filter (EKF) to incrementally tune the system in an online fashion, enabling it to complete designated learning or control lasks. The proposed method also improves the robustness learning by effectively managing noise the data. Thcoretical analysis provided to demonstrate the convergence OCIL. Three learning modes of OCIL,, iee. Online Imitation Learning, Online System Identification, and Policy Tuning On-the-fly, are investigated via experiments, which validate their effectiveness.
tunable optimal control system, we propose
1 Introduction
Informed Machine Learning (IML) (Von Rueden al. 2021 represents emerging approach that integrates prior knowledge into the machine learning (ML) process. While classic classification tasks in unsupervised, semi-supervised, or supervised MT primarily focus on extracting patterns from labeled or unlabeled data (LeCun et al. 2015), IML leverages prior knowledge such as physical laws, expert knowledge, or existing models to uncover underlying connections within data (Karniadakis et al.) (021). This integration enables models to produce more reliable and interpretable predictions, especially when dealing with noisy data. This approach espocially advantagoous the domains where thcorctical understanding well-ostablished and thus can guide ML. One notable example of IML is physics-informed machine learning (Wu et. al.) 2018 Karniadakis et al.) 2021 Kashinath al. 2021) which is particularly valuable for solving partial differential equations for computational fluid dynamics.
Control-informed learning (CIL) is subset of IML tailored for system control, autonomy, and robotics. This approach merges standard control theory with ML techniques to enhance the capabilities of autonomous systems. The integration leverages the complementary strengths of control and lcarning. Control theory provides model structures and optimization guidance that enable efficient and reliable algorithms for handling complex tasks. Meanwhile, ML improves these models by learning from data, a capacity that conventional

---

Published in Transactions on Machine Learning Research (03/2025)
control methods lack (Jin et al., 2020 2021b). This paper aims to tackle learning tasks in autonomous systems that are governed by optimal control (OC) systems. An optimal control system usually consists of dynamics, a control policy, and an objective function. From a unified perspective, learning these components can be understood as addressing a common problem with unknown parameters in different parts of the system and using different loss functions. For example, in learning dynamics, the task involves parameteriving a differential equation, with the loss function measuring the prediction error between the model's output and the target data. In learning policies, the unknown parameters are within the feedback policy and the control objective itself serves as loss function. When learning control objective functions, the objective is parameterized, and the loss measures the discrepancy between the reproduced trajectory and observed demonstrations.
To tackle these problems, many works in the field of so-called Learning for Dynamics and Control aim to leverage the integration of learning and control but often treat them as separate or sequential tasks. For example, control thcorics are used for algorithm development and convergence analysis of online unconstrainod constrrained optimizations (Casti et. al.
2023
Bastianello et. al.] 2024 Lu et al.
2024g; model-based
reinforcement learning (Heess et al.) 2015 Gu et al.
2016), improves sample efficiency by using dynamics
models; Koopman-operator control Proctor et al.
2018 Abraham & Murphey 2019 Hao et al.) 2024),
employs learning to transform nonlinear systems into linear observable space, simplifying control design. In contrast, CIT. integrates these processes, allowing learning algorithms to incorporate control insights directly. The integration enables ML and control techniques to perform simultaneously, reducing computational complexity, and improving practical applicability. CIL differentiates itself by utilizing Pontryagin's maximum principle, foundational concept in OC theory. This principle defines the optimality conditions for the state and input trajectorics of an OC system. CIL. employs these conditions to provide gradicnts for machine learning (Jin et al. 2020 2021b Böttcher et al. 2022). CIL integrates these gradients directly into its learning process, ensuring that machine learning outcomes are efficient while remaining consistent with established control theories and physical models. This approach enhances both the reliability and accuracy of the results.
1.1 Related Work
This section presents existing research on learning various components of an autonomous system and explores related learning frameworks that address these problems from a unified perspective.
Learning dynamics. To learn a nonlinear system with possibly noisy measurement, Markov decision-
procoss-based methods widely used, such as lincar regression (Haruno et al. 2001), observation-traanition
modeling (Finn et al. 2016). : latent space modeling (Watter et al. 2015), (deep) neural networks (NN) (Li
et al. 2018 Li & Hao 2018 Han et al.) 2019 Zhang et al. 2019 Bemning et al., 2019 Liu & Markowich
2020 Beintema et al. 2023 Pillonetto et al. 2025), Gaussian processes (Deisenroth & Rasmussen 2011), and
transition graphs (Zhang et al. 2018). Despite their widespread use, these methods often must balance data efficiency with prediction accuracy. To improve both meirics, physics-informed learning approaches Lutter
ett al. 2019 ; Xu et al. (2020); (Saemundsson et al.] [2020): ; SShamma et al. (2023) incorporate physical laws into
learning models. Koopman operator theory offers method for lifting states to infinite-dimensional lincar
observable space (Mauroy et al. 2020 Liang et al. 2023 Hao et al. 2023 Liu et al. 2024
Learning objective functions. Objective learning is typically referred to as inverse reinforcement learning (IRL) in the ML community and inverse optimal control (IOC) in the system control community. These methods aim to deduce a control objective function with observed optimal demonstrations. (Brown et al, 2019) The objective function is generally represented as weighted sum of features (Abbeel & Ng 20044
Ratliff et al. 2006: Ziebart al., 2008 Arora & Doshi 2021). p. Approaches to find these unknown weights include feature matching (Abbeel & Ng 2004), maximum entropy (Ziebart et al. 2008), maximum margin
Ratliff et al. 2006), and approximate variational reward imitation learning (Chan & van der Schaar 2021
As for learning nonlinear parameter mapping of objective functions, prior and system-dependent knowledge is required to further exlend the methods above. On the other hand, with system dynamics, IOC aims for efficient learning approaches (Mombaur et al., 2010). For example, some methods (Keshavarz et al. 2011 Jin et al. [2019] 2021a] Jiin & Moul 2021: Liang et al.) 2022 2023) directly calculate unknown weights by minimizing the violation of optimality conditions by the observed demonstration data, which avoids repeatedly solving OC problems.
2

---

Published in Transactions on Machine Learning Research (03/2025)
Learning control policies. Learning policies are generally termed reinforcement learning (RL) and OC in the MI. and control communities, respectively. In the RL, community, there are mainly two streams of research, namely model-free and model-based RL. Model-free RL learns policies by directly interacting with the environment, without using a model of it (Mnih et al. 2013:
2015 Oh et al. 2016). To improve data complexity, model-based RL learns a dynamics model before policy learning (Schneider 1997 Abbeel el al.)
2006;
Deiscnroth & Rasmussen 2011 Levine & Abbeel 2014; Gu et al., 2016). For OC, the first strategy
is based on dynamical programming, such as the linear quadratic regulator (LQR) (Scokaert & Rawlings 1998), which solves the OC problem with linear dynamics and quadratic cost, the linear quadratic Gaussian AAthans 1971), which combines LQR with Kalman filter to solve OC problem with linear system affected by Gaussian noise, the iterative linear quadratic regulator (iLQR) (Li & Todorov 2004), which linearizes the dynamics and quadratizes the value function, and differential dynamical programming, which quadratizes the dynamics and value function. Another strategy relies on Pontryagin's maximum/ /minimal principle (PMP)
PPnntrygginn 2018); such as shooting methods (Bock & Plitt 1984) and collocation methods (Patterson & Rao 2014 4). These open-loop methods are further improved by closed-loop methods such as model predictive control (MPC) (Schwenzer et al.] 2021). which repeatedly solves an OC problem over finite horizon to generate control inputs. Recently, Jin et al. (2020) proposed a framework for learning optimal policy based on differentiating Pontryagin's 's Maximum Principle.
Many research studies also focus on incremental policy tuning. One of the most popular tracks is transfer learning, which exploits the generalization of existing knowledge so that it can be transferred across different domains [Taylor & Stone, 2009). Recently, transfer learning has been implemented to speed up the learning
process in RL (Taylor & Stone 2009). Another popular method is behavior cloning (Torabi et al.] 2018
Czarnecki et al., 2019 (Sasaii & Yamashina, 2021 In the control community, tuning OC systems initially
refers to neighboring extremal optimal control (NEOC) [Bryson] 1975; Ghaemi et al. 2009). There are other
popular methods including adaptive control (Ioannou & Sun 2012 Bertsekas 2022 Luo et al., 2023 Guo & Pan 2023), which adjusts its parameters in real-time to maintain optimal performance, even in the presence of uncertainties or changes in system dynamics, and Bayesian optimization for controller tuning, (Khosravi
et al.
2021 Sorourifar et. al. 2021 Berkenkamp et al. 2023
To sum up: there are numcrous existing methods focused on individual tasks. These approaches are effective when only one component of the system is unknown. However, in many real-world scenarios, multiple components may be unavailable or uncertain simultaneously. For instance, in autonomous driving, the dynamics of the vehicle may be unknown due to changes in road conditions or vehicle wear and tear. Simultancously, the control policy may also be unavailable due to a lack of predefined rules or data. In such cases, existing methods often fall short, as they are not designed to handle the joint learning of multiple interdependent components, limiting their applicability in more complex or incomplete systems.
Unified learning frameworks. Several studies have explored unified learning frameworks to tackle learning challenges in autonomous systems. These approaches integrate an implicit planner directly within the policy Okada et al. 2017 Percira et al.) 2018 Amos et al. 2018 Srinivas ct al.) 2018). The main challenge in these methods is learning the OC system, which is very similar to the goal of this work. (Okada et al. 2017; Pereira et al.) 2018) learn a path-integral OC system (Kappen] 2005), which is a special class of OC systems. [Srinivas et al. 2018 learns an OC system in a latent space. These methods rely on an "unrolling" strategy to make differentiation casier. Essentially, they treat solving an OC problem as an "unrolled" computational graph created by applying gradient deeccntt repeatedly. This allows automatic different iation tools (Abadi (i) it requires storing all intermediate steps, making it memory-intensive and (ii) the accuracy of the gradients depends on how many steps are included in the graph, leading to a trade-off between computational cost and accuracy. To tackle these issues, Amos et al. (018)) proposed a differentiable MPC framework. In the forward pass, it uses an LQR approximation of the OC system, and in the backward pass, gradients are computed by differentiating this LQR approximation. This framework has a major challenge: differentiating LQR requires solving a large linear equation, involving the inversion of a matrix with size proportional to the time horizon, making it very costly for long-horizon systems. To address the challenges of the framework mentioned above, Jiin et al. (2020) proposed Pontryagin's 's differential programming (PDP). PDP avoids unrolled computational graphs by only storing the resulting trajectory without concern about how it is solved. Instead of relying on intermediate
et al. 2016 to be used directly. This approach faces few challenges:
3

---

Published in Transactions on Machine Learning Research (03/2025)
LQR approximations, it directly differentiates through Pontryagin's Maximum Principle (PMP) to obtain exact gradients. Furthermore, its backward pass uses an auxiliary control system to obtain the gradient, reducing memory and computational complexity, First, it lacks the ability for online learning, as it relies on gradient descent to update unknown parameters in the OC system, roquiring significant computation time to reach convergence. This drawback particularly problematic in applications like aulonomous driving, where quick adaptation to new sccnarios is essential for safcty and performance. Second, PDP does not account for noisy measurement data, limiting its effectiveness in real-world situations where sensor data is often unreliable or noisy.
1.2 Contributions
This paper introduces online learning framework called Online Nontrol-Informee Learning (OCIL). This framework is designed to be data efficient for various learning and control tasks while providing robustness against noisy data. In this paper, we consider an autonomous system as an OC system, which is parameterized by tunable parameters within different components of the system, including dynamics, policy, and objective function. By tuning the OC system in an online fashion, the proposed OCIL tackles three learning tasks in robotics, namely Online Imitation Learning, Online System Identification, and Policy Tuning On-the-fly. The proposed OCIL consists of two main components, both of which are inspired by control theory. Specifically, the framework first proposes online parameter estimator based on the classic online state estimation techniques in control theory. The estimator continually updates the parameter estimates in an online fashion as new data becomes available, aiming to minimize a cumulative loss defined for a specific task. To do so, the gradient information for the loss with respect to the tunable parameter is required. Therefore, OCIL employs a gradient generator (GG) based on Pontryagin Differential Programming in OC theory to calculate the exact gradient.
Notations. || denotes the Euclidean norm. Given a matrix A E Raxm let A' denotes its transpose. For positive integers and m, let I, be the n x n identity matrix; O,, RK" denotes a vector with all value 0; Onxm denotes m matrix with all value O. Let col(v),.... } denote a column stack of elements U1...., Va, which may be scalars, vectors or matrices, ie. col(v,..... va a [ug....' vn].
2 Problem Formulation
Consider the following class of OC systems Σ(O'), where 0r E Rp denotes the unknown and constant. parameter. The behavior of E(0*) is determined by minimizing a control objective function:
(xiT(0), 10-T-1 = arg min
[ uo:T-1, 0²) ΣT=O T- o(2, u, 0*) + h(xr,
(Ia)
Fi.T uo.
s.t.
L x441= = f(,,, u, 0*). with Do given.
(1b)
where t 1, , is the time index with T being the final time; ot E Ra and u, E T denote the system state and control input, respectively; (o.T(0") & col (0oo0"),.... #T(0")} and (o-T-1(0*) A col (u"))"..... (r-1( denote the states and inputs trajectory given parameter 0-, respectively; 2 (0*) and u, (0r) denote the state and input given 0* at time respectively; f IR x Rmm x RR + RD denotes a twiee-ddiferenniabll time-invariant system dynamics; C: R" x Rmm x R + R and h: x R. + R denote running cost the final cost, respectively, both of which are assumed to be twice-diflerentiable.
Remark 1. Including the parameter 8" in the system dynamics allows for the representation of both partially known and completely unknown dynamics. For parlially known dynamics, il is paromelerized via a known physsccal dynamic model with unknoum physical parameters. For erample, this could be a quadrotor dynamics with known structure and unknown inertia and mass [Wang et al. [2014] Jin et al.) [2020] Revach et al. 2022). In the case of completely unknown dynamics, parameterization is done by neural networks. In this case, the neural network captures the evolution of the state, where the parameter 0" represents the weights and biases of the newral network (Kumpati et al. 1990: Lewis et al. 1998: Nelles & Nelles 2020).
For notation simplicity, we define the unknown trajectory of the optimal control system E(00) as
E(0²)= col (2l.r()(00)) (00T-1(0")) (T+)))+TT
(2)
4

---

Published in Transactions on Machine Learning Research (03/2025)
The goal of this paper is to estimate 0r. Define O Rp arbitrary estimation of 0". Then for estimation
purposes, copy; I(0), of the autonomous system E(0") can be proposed by replacing 0 with 0 in ① i.e.
{orr (0), uo.r uo:r-(00)) arg min TiT; 40:T-1
J(Z:T 1) Et=o c(æ,, u, 0) h(xr, 0)
(3a)
s.t.
a++1 f uu 0), with zu given.
(3b)
At each time t, a noisy measurement O2 E R" is observed, where
O = h()((")))) + vt.
(4)
Here, h 1u + R" denotes a twice-differentiable measurement function; ((())) = (0)) u (00)):
Us (O,, R,) denotes the measurement noise which is multivariate Gaussian, with R. E gg being the covariance matrices of the measurement noise.
With the measurement equation
4 defined, this paper considers signed residual function:
((E;()), O, (() E RR.
(5)
Then, the performance of the entire trajectory can be evaluated by a cumulative loss which is assumed to be twice-differentiable:
L(E(0)) = Σ | (0),
(6)
The problem of interest is to develop online method to update the estimation 0, E Rp of 0 at every time t, such that its trajectory {(O.) from minimizes task-specific cumulative loss (({
To achieve a specific learning or control task, one needs to select the most suitable measurement O,. Below, we will present three modes of the proposed OCIL framework. is worth noting that different applications, adjustments the configuration of system Σ()) are required according to the task.
Online SysID: For a SysID problem, the goal is to identify the dynamics model of physical system from the state-input trajectory 8" = ( Mu:r-1), where the superscript denotes the observed trajectory. The tra trajectory is often generated by persistent excitation of the system without considering any control objectives
Keesman 2011). Therefore, we can set (ID.T U0,T-1,,) 0) = O:
dynamics:
L++1 = f(z,, u. θ). with Zo given,
(
objective:
J u0 0) =.
To identify the model dynamics, namely finding the 0* in the dynamics fær ur 0*), one could design the signed residual function to represent the discrepancy between the observed trajectory and the trajectory produced by 0, i.e. (0), E (0), where &i is a slice of & at time t. In the SysID mode, the mcasurement O, received at time is a slice of the trajectory of a physical system E/-
Online Imitation Learning: The objective function and the model dynamics are parameterized by an unknown 0" The OC system follows 3 Suppose one can observe the measurement of the expert demonstration yī at each time l. Then, the signed residual function can be designed as ((E,()), yē) = y: g( (())) In this case, the measurement O, received at time t is the expert demonstration y!: The optimal demonstration can vary between being continuous or sparse, depending on practical application scenarios.
Tuning Policy On-tho-fly: For an autonomous system, one would like to obtain a control policy such that the trajectory minimizes certain task loss. This mode considers a feedback controller which is parameterized by 0, i.. u, p(xx 0). Then the OC system is written follows:
dynamics:
F._1 1 = f(x,, u1 0)); with 3o given,
I(θ):
(8)
objective:
J u0. 0) = ΣT= c(x,, ulz, 0)) + h(xr).
Then we can design the signed residual function such that represents trajectory tracking. For instance, the signed residual function could be (() (0),)) E! Ed (θ), where & is a slice of desired trajectory to track
time t.
5

---

Published in Transactions on Machine Learning Research (03/2025)
3 Main Results
The proposed OCIL consists of two main components, both of which are inspired by control theory. Specifically, OCIL first proposes an online parameter estimator based on the extended Kalman filter (EKF). Going forward, we will show the challenge of obtaining the Kalman gain. To tackle this challenge, the gradient information for the loss with respect the tunable parameter is required. Therefore, OCTL employs a gradient generator (GG) based on Pontryagin Differential Programming calculate the exact gradient. Then the proposed OCIL framework will be introduced and supported with theoretical analysis.
3.1 Online Parameter Estimator
To minimize the cumulative task loss (E)))) with measurement O₁, which is unavailable until time t, the optimization problem that needs to be solved in online fashion is:
min 0
El (0), O,)1 subject to { is the trajectory of (3)
(9)
The optimization problem (9) is essentially a least squares problem, although under constraints. One of the most famous methods to solve the least squares problems incrementally is the EKF (Bertsekas) 1996. Ribeiro 2004). The EKF was proposed to incrementally estimate the state of system using measured output available at each time step. In our problem setting, instead of estimating the state estimate the parameter 0* by utilizing the measurement O, that is available at each time Therefore, by considering the parameter l the state to be estimated, one introduce new dynamical system:
system, our goal is to
dynamics: 0tt1 0,, with 0g = 0" mcasurement: O hc(E((,)) +
(10)
The online estimation of 0 via EKF can done as follows
[Ribeiro 2004):
0, = 0,-1, P = P
(11a)
K, = P. LL, + R P. = (I, K.L.).. O, : O, Ki(0; -((E((,))))
(11b)
L (E;)) O.)
d0: 0. 6- E TR'XP
(12)
where (1a) predicts the dynamics; 11b updates the parameter estimate. Here, the superscript means the term is not yet updated by measurement residual; P. E RP*p is positive-definite matrix that denotes the covariance of the estimate; K, E Rexr denotes the Kalman gain. Throughout the estimation process, all of the terms are known except L, It is challenging to obtain this term as the signed residual function (0), Oz) is not an explicit function of 0. Tn the next subsection, we will present a gradient generator which computes the exact value for Lt.
3.2 Gradient Generator
In this section, for notation simplicity, the parameter estimate 0, is simplified to 0; (I...
d0;
0. 07
is
written (0)) To obtain the gradient (0)) one can employ the chain rule by definition,
d0
(O) al(E: (+)
(13)
d0
(
00
where (01) is known since the signed residual function pre-designed. The challenge that remains is to 0Q.(0)
find the partial derivative (g) 30 ie. an analytical relation between trajectory & and the tunable parameter
0. To tackle this challenge, the gradient generator in Jin al. (2000) is used to obtain the exact value of (g) 09
Given the OC system
3 one can obtain the Hamiltonian equation
H. o(wr, ur 0) f u4 0) A
(14)

---

Published in Transactions on Machine Learning Research (03/2025)
for all 0.... . where X I" denotes the Lagrangian multiplier associated with the equality
constraint of model dynamics. With the definition of E(O), one has #E(0) defining
coll dan r()) Buo (0)
}. By
00
( X2
( U
E RExp,
d0 E Rmxp,
(15)
00
one can utilize the following lemma from Jin et al. (2020) to obtain the partial derivatives (c,(0)
00
Lemma 1.
Jin et al.
(2000)
Define the Jocobian and Hessian matrices related to E(0) as:
0f
0f
0f
8 He H.
8 He dz
F dx,
G.
E,
H.
(H!
du
00
dz H.d
(16)
8 H, tv,Uu.
H
H
h
0hh
H
H
H
H
0x,00'
du:00'
dxrZxt
~xr
- - If H." is incertible for all t = 0, ,
1, the following recursions from t = T to = 0 hold:
V, C, + A/ + V,) V1+
(17)
W. A,(I V4+ (,) (W I+ (W) V1+1MM No:
with VT = H and WT = H Here, A, = F. HH B. G. M, = E,
G,( C, = H.Ch HH. (H:")
- H N, = H." are all known given 16
0E(O)
Then, the partial derivative
can be oblained by recursively solving the following equalions from to
00
T 1 with (o(0)
U. = - (H! H.Th X, + H.C + G Br. + V4 + W+11),, X = F,X, + GUU, + E.,
(18)
The terms in 16 are based on the trajectory ( and the associated Lagrangian multiplier Aurr-1 According to the discrete-ting time Pontryagin Maximum Principle (Jin et al. 2020 the trajectory the Lagrangian multiplier can be obtained by
Oh dxr
OH, dc 0h dx,
Ay
X
A4t1. for t -----11
(19)
da.
da,
Remark 2. Lemma proposes recursive way to obtain the exact gradient of the trajectory {(0) with respect to the parameter 0, i.c. 0)
3.3 OCIL Framework
With the online parameter estimator and the gradient generator, we propose the Online Control-Informed Learning framework in Fig. The framework is summarized in Algorithm 2
10,
System trajectory
Bi(e, ( O.,)
Online Parameter
{(O,)
(,,
((E,(
Chain rule
Estimator
d (0, ).O,) d0,
81(E, (0, ),O,) 8g, (0,)
(θ,
(,,
00,
Gradient Generator
80,
L
Figure 1: Framework of Online Control-Informer Learning.

---

Published in Transactions on Machine Learning Research (03/2025)
As shown in Fig. 1 at each time step, the predefined OC system Σ(θ.) generates system trajectory <(O,) by performing optimal control with given 2o and Oz. The trajectory E(O.) is then fed into the signed residual function ((.), O+) and the gradient generator. Along with the information O, obtained al time 4, the signed residual function generales ( while _c,(0,) is generated by the gradient generator in 86,
(6,
Algorithm I The chain rule is then performed to obtain the Jacobian matrix Lr, which is then passed into the online parameter estimator for the estimation of 0-.
Algorithm 1: Gradient Generator (GG)
Algorithm 2: Ouline Control-Informed Learning System and Residual: 20) and ((E:(0), O:) Initialize: 0,, Po for ta, ta,... do Obtain new information O2:
Input: Trajectory {(0,) from Σ(0,)
Compute the coefficient matrices in 16
Set VI = Hf and WT Hf;;
for t+- T to O by At do
2
Solve E(O,) from current OC system [0):);
Update Vi and W. using 17
Obtain (6,(0,) with GG in Algorithm 1
Set Xu(0,)= =
00,
for + 0 to T by Al do
Obtain (. from L(E, (6,)
Update X.(0,) and U.(O,) using 18)
(g,(,,)
6 Obtain L, via the chain rule 13
Output: ag(o_) 80,
[X1r (0,) (1,))
7 Update 0, using the estimator
3.4 Convergence Analysis
This subsection presents the convergence analysis of the online parameter estimator. The analysis employs a candidate Lyapunov function and introduces how the measurement covariance matrix R2 affects the convergence of the cumulative loss L(E(0)). In this section, for brevity, the signed residual function (O), O²) is written as (0). Suppose for a specific task, the optimal cumulative loss I.(E(O)) 0.
Then, we define the estimation error as 0, = 0 - 0 Furthermore, we define
Measurement error: e, (({(0)) - ((((),))
(20)
Prediction error: 0, = 0 0
To perform the convergence analysis, a candidate Lyapunov function is employed:
V = :.:.P.-0.
(21)
The goal here is to determine conditions for which the candidate Lyapunov function (V1h=1,2... is a decreasing sequence, iee. Vi41 - V S O, Vt. For rigorous analysis of the candidate Lyapunov function, as proposed in Boutayeb et al. (1997). unknown diagonal matrices F. E RR" and G, E RPxx are introduced to model the
measurement and prediction error defined in (20):
L,
= -.0.--
(22)
To ensure convergence of the proposed estimator, the following assumptions need to be made.
Assumption 1. The derivative L, (0,)) is of full rank for every 0,
do,
Remark 3. The discrete- time dynamical system 10 satisfies the observability rank condition, i.e., for every 0, rank(col( 1) dic.,(d, 1) d0, do, Ip. ..., Assumption satisfied for every 0,, the system assures that Pt is bounded matriz from above and below [Song E Grizzle 1992. Boutayeb & Aubry. 1999
dlig.
Ig 11) = (Song & Grizzle 1992). That means if
db
(10) is observable for every O,
The observability condition
As common in the EKF analysis, we adopt the following assumption: Assumption 2. L, is a uniformly bounded matriz.
We have the following lemma to show how the measurement covariance matrix R, affects the convergence of the tunable parameter. The proof can be found in Appendix A
8

---

Published in Transactions on Machine Learning Research (03/2025)
Lemma 2. Let Assumptions 1 and 2 hold. If the following inequalities satisfied:
(F, 1-11)) PL L + +R)
(23)
S. P.
(24)
Then the proposed estimator couvergence, i.e. lim 0. 0.
11 when used as an observer for the system
10 ensures local asymptotic
Remark 4. Lemma provides sufficient conditions for the convergence of O..
As the diagonal matrices Ft For ezample, one can set the is satisfied, which means the
and 9, are unknown, one can design the matria Rr to satisfy inequalities 23
matriz R, to be sufficiently large, f.c. much larger than L,P, L, so that 23
potumeler estimalor can tolerale arbilrary large imilial prediclion ervor. It is worth to nole that long (3)) and (24) are satisfied, 02 converges to 0 and consequently F; and 9 become identity matrix. In the case when there is no measurement noise, i.e. R = 0xxa, Ft and G, can only be identity matrices to satisfy the inequalities (25) and 24), indicating the convergence of 0. to r
and 24 indicate one of the limitaations of the estimalor, which is the selection of initial guess. If the instial guess of 0" results Fo and 96 that do not satisfy 23 and 24), the value of the Lyapunov function 21 becomes larger, which leads to even larger F, and Gr, cousing the estimation to diverge.
Remark 5. Equalion
We have the following main theorem shows how the measurement covariance matrix R, alfects the convergence of cumulative loss L(E(0)) by utilizing the inequalities introduced in Lemma The proof can be found in Appendix B
Theorem 1. Let Assumptions and 2 hold. If the incqualities in Lemma 2 are met, then estimating 0
with the proposed eslimator [11] employing the gradient generator in [17 18 ensures local asymplolic
convergence of the cumulative loss L in to 0, iec lim L(g(0))= = 0.
4 Applications to Different Online Learning Modes and Experiments
This section demonstrates the capability of the proposed OCIL framework with its three modes by three applications, Online Imitation Learning, Online System Identification, and Learning Policy on-the-fly. This section includes a performance comparison with some saateoff-theart frameworks for three environments that are summarized in Table Let O h(E)(0")) denotes the measurement without noise. The mcasurement noise is subject to a multivariate Gaussian distribution ((O;,o²I,)
To highlight the flexibility of OCIL, each experiment includes two phases: 1) online phase, where OCIL keeps lcarning the unknown parameter while new data comes in before the final time T; 2) offline phase, where OCIL keeps learning the parameter given the learned parameter at time T and the entire trajectory obtained from time = 0 to time T. For each environment and task, terminal time T E Z defined to represent a desired time duration where the system shall finish the task.
To unify the data visualization of both online and offline phases, the horizontal axis represents the number of data points, where vertical red line corresponds to the final time T, i.e. the end of the online phase. The number of data points reflects the number of iterations multiplied by the total number of time steps for each iteration. The solid blue curves indicate the online portion of OCIL, whereas the dashed blue curves indicate the oflline portion. For every environment and every method, 5 trials are performed given random initial conditions due the high computational cost for other methods. The computational performance and analysis for OCIL are shown in Section 5 of the Appendix.
Online Imitation Learning. The control objective is parameterized as weighted distance to the goal. Set the signed residual function of imitation learning ((), v,)) y: ( u. (.)). The optimal cumulative loss is zero, i.e. L(E(O*) = 0, with full knowledge of the parameter. Four existing methods are used for comparisons: (i) inverse KKT (Englert et al. 2017) (ii) neural policy cloning (Bojarski et al., 2016) and (iii) PDP (Jin al. 2020). These methods don't handle measurement noise well because of their limitations, so we performed the experiments without including measurement noise for these methods. For OCIL, o 0.1 for all of the systems.
9

---

Published in Transactions on Machine Learning Research (03/2025)
Table Experiment Environments
Systems
Dynamics parameter Odgn cart mass, pole mass and length mass, wing length, inertia matrix mass, rocket length, incrtia matrix
Objective parameter Pobj
Cartpole
c(æ, u) Ooty | x + |
6-DoF Quadrotor
h(x;) = Dbbj || Izg
6-DoF Rocket
Fig. 2a 2c summarize the comparison result, where OCIL converges faster and obtains lower loss than the other offline methods, in both online and offline phases. The imitial loss for each method is different because the learning representation (parameterization) is different. Thus, it hard to guarantee that an initial neural network has the same loss as another initial parameter vector. Nevertheless, the initial representation of each method adjusted such that OCIL does not take advantage of good initialization. Fig. [a validate the effectiveness of OCIL's 's both online and offline performance, even with measurement noise.
10a
Imitation Loss
Imitation Loss
Imitation Loss
10a
10a
10"
10-2
10-2
10-2
10"4
10
10"4
C3. Oheenied Whe Oalrcttie
0C3 Progendi
OC. Progonni verien PIS Decendd
10-6
aooreii Other
10
ralon OEte
10-4
PDP palis 20
Heanippll moneii Office
Sourd solee Offer
Ncard palcesio rabe Offece
10'
10-4
10"
10'
Number of Data Points
Number of Data Points
Number of Data Points
(a) Cartpole
(b) Quadrotor
(c) Rocket
Figure 2: Imitation loss v.s. number of data points
Figure 3: Trajectories of the cartpole in online imitation learning. Blue solid lines: learned trajectory. Green solid lines: observed noisy trajectory. Red dashed lines: ground truth.
Online System Identification The signed residual function is set to be ((E, <?) E
(.) The
optimal cumulative loss zero, i.e. T({((0))= = 0, with full knowledge of the parameter. Three other methods are used for comparison: (i) Pytorch Adam solver (Pillonetto et al. 2025), (ii) DMDc (Proctor et al. 2016). and (iii) PDP (Jin et al. 2020). No measurement noise are injected into observed data for existing methods due to their inherent limitations. For OCIL, 0.05 for all of the systems.
Fig. ba 5c summarize the result, where OCIL outperforms PDP for faster convergence and lower loss, in both online and oflline phases. Different Ihan Online Imitation Learning, OCIL does not decrease its SysID loss significantly at first because the number of data points is not sufficient for online learning. Once the number of data points becomes sufficient, the SysID loss starts decreasing significantly. This phenomenon can also be observed the other methods, but their critical number of data points significantly larger than OCIL's. OCIL and other methods applied to learn the neural dynamics using the same observed
In Fig. 5d 5f
10

---

Published in Transactions on Machine Learning Research (03/2025)
4.-
16
18
Figure 4: Trajectories of the quadrotor in online imitation learning. Blue solid lines: learned trajectory. Green solid lines: observed noisy trajectory. Red dashed lines: ground truth.
trajectory. It can be seen that OCIL purperfs other methods for lower loss. Fig. 6 demonstrates the capability of OCIL to deal with neural dynamics that have different sizes of NN.
10a
OOL seroddact SACT
10
10a
InDOSNNT SusK UhiMtrr offine
SysiD Loss
10
SysiD Loss
102
SysiD Loss
102
100
10
300
10
10
10
10
10
100
10
10
10a
10a
10
0
10a
10
10a
10a
10
10-
10a
10
10'
10"
10*
Number of Data Points
Number of Data Points
Number of Data Points
(a) Cartpole
(b) Quadrotor
(c) Rocket
10a
10a
102
CCL do Nirdi
10a
COL, Thopsoe wiress D460: a/t 10le
OCL ofterree IN rob.
240 a 1096
104
Pytorth deaa edles
10a
104
SysiD Loss
SysiD Loss
10a
SysiD Loss
10a
10a
10a
101
10
101
10a
109
10-1
10-1
109
10
0.0
0.2
0.4
0.6
0.8
1.0
10-2
0.0
0.2
0.4
0.6
0.8
1.0
10
0.0
0.2
0.4
0.6
0.8
1.0
Number Data Points 1e4
Number of Data Points 1ea
Number of Data Points 164
(d) Cartpole, offlin,, NN dynamics (e) Quadrotor, offline, NN dynamics
(f) Rocket, offline, NN dynamics
Figure 5: SysID loss v.s. number of data points
Policy Tuning On-tho-fly. The parameterized OC system in g is used here, where the policy is in a stade-feedback form and parameterized by the tunable parameter Ô. The signed residual function is set to be ((E:(0)) { E (0), where & is the trajectory that needs to be tracked. The optimal cumulative loss
11

---

Published in Transactions on Machine Learning Research (03/2025)
10a
10
10c
MK sow: (nemiiinemjmm
NW cow tn-mipon-miom
MV sow: Inemistn-mjjm
10a
NR size 21n+mik20r+mi0 mlom
MN sice 3 miom
10a
NN sic.
MN sie: 21n-moxion+mimm
31o-mo3mn-mio
10"
MV S2e.
MN sze: din=mX2inmmumm
30 midin
NB sue 4In middon mimm
SyslD Loss
SystD Loss
SysiD Loss
10a
10
10
10
10"1
10a
10"
10 0.0
2.0
2.5
3.0
10
0.0
0.5
1.0
1.5
10
1.5
2.0
Number Data Points
1e3
Number of Data Points 1e3
Number Data Points 1e3
(a) Cartpole w/o noise
(b) Quadrotor w/o noise
(c) Rocket w/o noise
Figure 6: SysID Loss v.s. number of data points, given different sizes of neural dynamics
is zero, i.c. L(E(O)))= with full knowledge the paramcter. Other methods are used for comparison (i) iLQR (Li & Todorov 2004) (ii) GPS (Levine & Abbeel 2014), and (iii) PDP (Jin al.) 2020). No measurement noise is included for existing methods due to their limitations. For OCIL, 0.1 for cartpole and quadrotor; = 0.25 for rocket.
Fig. 9c summarize the result, where the loss and its variation of OCIL converge very quickly. The buffers
in Fig. 3a indicate 3 times standard deviation. Fig. [ presents the online phase of OCTL given 1000 random trials, which further validates the effectiveness and robustness of OCIL given measurement noise.
10-
-4-
Figure 7: Trajectories of the cartpole in policy tuning on-the-fly. Blue solid lines: learned trajectory. Green solid lines: observed noisy trajectory. Red dashed lines: ground truth.
In general, OCIL from all figures does not have smooth loss trajectory the other offline methods. This is because at the online phase, optimal gain matrix K, from (lla) is computed to update O,, whereas the other methods either use constant or iteration step size. The optimal gain is conceptually similar to searching an optimal step size in the line-search optimization algorithms. Thus, it is observed that the loss variation, as represented by blue buffers, is relatively high initially but starts decreasing siguificantly as new data comes because K, is continually updated. In contrast, the loss variation barely changes for the other offline methods after some data points.
5 Online Computational Performance
The experiments with OCIL were performed on a desktop with one Intel Core 17-8700k CPU with 8GB RAM. No GPU was used. The experiments with other methods were performed on a desktop with one AMD Ryzen 5900X CPU, one Nvidia Geforee RTX 4070ti, and 32 GB RAM. more powerful PC was selected for the other methods because of their high computational cost. As noted at the beginning of Section only 5 trials were conducted due to the computational expense.
12

---

Published in Transactions on Machine Learning Research (03/2025)
4.-
16
Figure 8: Trajectories of the quadrotor in policy tuning on-the-fly. Blue solid lines: learned trajectory. Green
solid lines: observed noisy trajectory, Red dashed lines: ground truth.
10
10a
10
Policy Tuning Loss
Policy Tuning Loss
Policy Tuning Loss
(rogoonds
10a
Nolly
zut
10"
101
30*-
10-1
10"
10
1000
2000
2000
4000
5000
10-1
1000
2000
3000
4000
5000
10a2
1000
2000
3000
400p
5000
Number of Data Points
Number of Data Points
Number of Data Points
(a) Cartpole
(b) Quadrotor
(c) Rocket
1000
Policy Tuning Loss
Policy Tuning Loss
10"
Policy Tuning Loss
2000
400
200
10"
10"
15 Number of Data Points
20
25
30
5 30 15
20
25
35
Number Data Points
301
60 50
60
Number of Data Points
(d) Cartpole, online phase, with (c) Quadrotor, ouline phase, with noise
noise
Rocket, online phase, with noise
Figure 9: Policy Tuning Loss v.s. number of data points. Buffers represent loss variation under 30 with random initial conditions.
To demonstrate that the computational performance OCIL is enough to be used in an online fashion, we recorded the computational time for OCIL in different modes for 100 trials. The code implemented Python, utilizing the CasADi library with the IPOPT solver solve the OC problem. Table summarizes the OCIL's computational performance for the system identification task for three environments, where OCIL
13

---

Published in Transactions on Machine Learning Research (03/2025)
Time indicates the computational time of running OCIL at each time t, i.. the iteration within the for-loop of Algorithm 2 GG Time indicates the computational time of running gradient generator (GG) at each time i.c. Algorithm Estimator Time indicates the computational time of updating θ, i.c. Line 7 of Algorithm 2 A indicates the time slep of each environment, i.e, the Lime duration belween two conseculive data measurements or the maximum allowed time duration of online algorithms to perform computation: Percentage indicates the percentage of the average OCIL time with respect to A. The header of Table 4 and Table 2 are the same. Roughly speaking, OCIL time GG time + Estimator time + Optimal Control computation time.
Table illustrates that OCIL can estimate the dynamical system with neural network representation in an online fashion, within the system frequency of getting new data. Table illustrates that OCIL can tune the neural policy online. As indicated in Line 2 of Algorithm 2 the most computationally heavy part is solving optimal control trajectory in an online fashion, instead of GG and the parameter estimator. As demonstrated at the beginning of this section, OCIL does not require huge computational resources, such as GPU. Therefore, OCIL has the capability to run in an online fashion.
4
Table 2: Computational Performance for Online Imitation Learning
Env.
OCIL Time [ms]
GG Time [ms]
Estimator Time [ms]
A [ms]
Percentage
Cartpole
62.10 + 6.63
7.47 + 0.25
0.031 + 0.0023
100
62.10 %
Quadrotor
81.70 + 2.51
21.72 + 0.84
0.058 + 0.039
100
81.70 %
Rocket
72.25 + 13.91
19.77 + 6.26
0.060 + 0.012
100
72.25%
Table 3: Computational Performance for SysID with Neural System
Env.
OCIL Time [ms]
GG Time [ms]
Estimator Time [ms]
A [ms]
Percentage
Cartpole
17.18 5.15
6.05 2.59
1.93 + 0.85
50
34.36 %
Quadrotor
35.53 + 8.98
12.18 + 4.77
16.18 + 6.11
100
35.53 %
Rocket
29.54 8.29
11.25 + 4.67
12.89 + 5.29
200
14.77 %
Table 4: Computational Performance for Policy Tuning with Noural Policy
Env.
OCIL Time [ms]
GG Time [ms]
Estimator Time [ms]
A [ms]
Percentage
Cartpole
16.41 + 5.35
7.72 + 3.07
3.96 + 1.91
50
32.82 %
Quadrotor
62.67 + 9.51
30.86 + 2.25
22.47 + 8.33
100
62.67 %
Rocket
59.02 + 7.25
33.99 + 2.98
12.94 + 5.62
100
59.02 %
6 Limitations.
This section discusses the major limitations of the proposed framework from three perspectives.
Local convergence: Since OCIL is based first-order gradients, it can only achieve local minima for general non-convex optimal control problems in (3). Furthermore, the general problem proposed in this paper belongs to a bi-level optimization framework. Under certain assunptions such as convexity and smest on models (e.g., dynamics model, policy, loss function, and control objective function), global convergence of the bi-level optimization can be establishod. However, such conditions are too restrictive in the context of dynamic control systems. Therefore, the local convergence analysis based on general nonlinear optimization is enough.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Parameterization matters for global convergence: When performing experiments, we find that how models are parameterized matters for good convergence performance. For example, in online SysID mode, we observe that using a neural network dynamics (in Fig. 5d 5f is more likely to get trapped in local minima than using the true dynamics with unknown paramcters (in Fig. 5a 5et). In general, more complex parameterivation will bring extreme non-convexity lo the optimization problem, making the algorithm more easily trapped local minima. Determining the paramcterization of an object to be learned requires prior or expert knowledge, which common in ML.
Initialization matters: As OCIL borrows how optimal gain updates from EKE, they share the same drawback that convergence depends on the selection of initialization. As shown in Remark 5 bad initial guess might cause the estimator to diverge according to Lemmn 2 Therefore, if a relatively good initial guess is hard to retrieve, one might need to use other methods to cold start OCIL.
7 Conclusions
This paper proposes Online learning tasks. By considering an optimal control system with tunable parameter, OCIL is a unified learning framework that effectively addresses tasks such online imitation learning, online system identification, and luning policy on-the-fly. By designing signed residual function specific to each task and treating the parameter as a state of a new system, we employ the online parameter estimator to estimate the parameter online and minimize the signed residual at each time step. Thcoretical analysis establishes the convergence conditions for OCIL, while experiments on various environments, tasks, and existing methods are done to validate data efficiency, versatility, and robustness against measurement noise.
Learning (OCIL), an online learning method tailored for diverse
Acknowledgments
This material is based upon work supported by the Office of Naval Research (ONR) and Saab, Inc. under the Threat and Situational Understanding of Networked Online Machine Intelligence (TSUNOMI) program (grant no. N00014-23-C- C-1016). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the ONR, the U.S. Government, or Saab, Inc.
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint 103.00003 2016.
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, pp. 1 8, 2004.
Pieter Abbeel, Morgan Quigley, and Andrew Y Ng. Using inaccurate models in reinforcement learning. Tn International Conference on Machine Learning, pp. 1 1-8, 2006.
Ian Abraham and Todd D Murphey. Active learning of dynamics for data-driven control using koopman operators. JEEE Transactions on Roboties, 3575)11071,, 2019.
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and Zico Kolter. Differentiable mpe for end-to-end planning and control. In Advances in Neural Information Processing Systems, pp. 8289-838 8300,,80,0000 2018.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Michael Athans. The role and use of the stochastic linear-quadratic gaussian problem in control system design. IEF.. Transactions on Automatic Control, 16(6):529 25252,9-5,,... 1971.
Nicola Bastianello, Ruggero Carli, and Sandro Zampieri. Internal model-based online optimization. 1EEE Transactions Automatic. Control, 69(1 (669 696, 2024.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Gerben I Beintema, Maarten Schoukens, and Roland Tóth. Deep subspace encoders for nonlinear system identification. Aulomalicn, 156:111210) 2023.
Martin Bemning, Elena Celledoni, Matthias J Ehrhardt, Brynjulf Owren, and Carola-Bibiane Schönlieb. Deep learning as optimal control problems: models and numerical methods, ar.iv preprint arXio:1904.05637, 2019.
Felix Berkenkamp, Andreas Krause, and Angela P Schoellig, Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning, 112(10):3713-374) 2023.
Dimitri Bertsekas. Lessons from AlphaZero for optimal, model predictive, and adaptive control. Athena Scientific, 2022.
Dimitri P. Bertsekas. Incremental least squares methods and the extended kalman filter. SIAM Journal on Oplimization, 6(3):807 822, 1996.
Hans Georg Bock and Karl-Josef Plitt. A multiple shooting algorithm for direct solution of optimal control problems. IFAC Proceedings Volumes, 17(2):1603-1600) 1984.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint orXiv: 10046000440066 2016.
Lucas Böttcher, Nino Antulov-Pantulin, and Thomas Asikis. Ai pontryagin or how artificial neural networks learn to control dynamical systems. Nature Communications, 13(1):333, 2022.
M. Boutayeb and D. Aubry. A strong tracking extended kalman observer for nonlinear discrete-time systems. IEEE Transactions on Automatic Control, 44(8):1550-155 1999.
M. Boutayeb, H. Rafaralahy, and M. Darouach. Convergence analysis of the extended kalman filter used observer for nonlinear deterministic discrete-time systems. IEEE Transactions Automatic Control, 42(4):581 586, 1997.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International Conference Machine Learning, pp.
PMLR, 2019.
Arthur Earl Bryson. Applied optimal control: optimization, estimation and control. CRC Press, 1975.
Umberto Casti, Nicola Bastiancllo, Ruggero Carli, and Sandro Zampicri. A control thcoretical approach to online constrained optimization. arXiu preprinl arXiv:2309. 15499, 2023.
Alex J Chan and Mihaela van der Schaar. Scalable bayesian inverse reinforcement learning, arXiv preprint arXiv:2102.06483 2021.
Wojeiech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1331-1340. PMLR, 2019.
Mare Deisenroth and Carl E. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 465-472, 2011.
Peter Englert, Ngo Anh Vien, and Marc Toussaint. Inverse kkt: Learning cost functions of manipulation tasks from demonstrations. The International Journal of Robotics Research, 36(13-14):1474 1488, 2017.
Chelsea Finn, lan Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in Neural Information Processing Systems, pp. 64-72, 2016.
Reza Ghaemi, Jing Sun, and Tlya V Kolmanovsky. Neighboring extremal solution for nonlinear discrete-time optimal control problems with state inequality constraints. IEEE Transactions on Automatic Control, 54 (11):2674-2677) 2009.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with
model-based acceleration. In International Conference on Machine Learning, pp. 2829-282220 2299-288,,- 2016.
Kai Guo and Yongping Pan. Composite adaptation and learning for robot control: A survey. Annual Reviews
in Control, 55:279-290 2900 2023.
Jiequn Han, Qianxiao Li, et al. mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 6(1):1-41, 2019.
Wenjian Hao, Paulo C Heredia, Bowen Huang, Zehui Lu, Zihao Liang, and Shaoshuai Mou. Policy learning based on deep koopman representation. arXiv preprint arXiv:2305. 15188, 2023.
Wenjian Hao, Zehui Lu, Devesh Upadhyay, and Shaoshuai Mou. A distributed deep koopman learning algorithm for control. arXiv preprint arXiv:241..77211 2024.
Masahiko Haruno, Daniel M Wolpert, and Mitsuo Kawato. Mosaic model for sensorimotor learning and control. Newral Computation, (3(10))2201-29)) 2001.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Aduances in Neural Information Processing Systems, pp.
2015.
Petros A Ioannou and Jing Sun. Robust adaptive control. Courier Corporation, 2012.
Wanxin Jin and Shaoshuai Mou. Distributed inverse optimal control. Aulomatica, 129:109658, 2021. ISSN 0005-1098.
Wanxin Jin, Dana Kulić, Jonathan Feng-Shun Lin, Shaoshuai Mou, and Sandra Hirche. Inverse optimal control for multiphase cost functions. IEFE Transactions Robotics, 35(6):1387-1338 1398, 2019.
Wanxin Jin, Zhaoran Wang, Zhuoran Yang, and Shaoshuai Mou. Pontryagin differentiable programming: An end-to-end learning and control framework. Advances in Neural Information Processing Systems, 33: 7979-7992, 2020.
Wanxin Jin, Dana Kulić, Shaoshuai Mou, and Sandra Hirche. Inverse optimal control from incomplete trajectory observations. The International Journal of Robotics Research, 40(6-7):848-885))) 2021a.
Wanxin Jin, Shaoshuai Mou, and George J Pappas. Safe pontryagin differentiable programming, Advances in Neural Information Processing Systems, 34:16034-16050, 2021b.
Hilbert J Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Mechanics: Theory and Experiment, 2005.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics- 4444222440022 informed machine learning. Nature Reviews Physics, 3(6):422-44) 440, 2021.
Karthik Kashinath, M Mustafa, Adrian Albert, JL Wu, Jiang, Soheil Esmaeikaddeh, Kamyar Aviwadenesheli, R Wang, Ashesh Chattopadhyay, A Singh, et al. Physics-informed machine learning: case studies for weather and climate modelling, Philosophical Transactions of the Royal Society A, (799(194):00200099 2021.
Karel Keesman. System identification: an introduction. Springer Science Business Media, 2011.
Arczou Keshavarz, Yang Wang, and Stephen Boyd. Imputing a convex objective function. In IEEE International Symposium on Intellignn Control, pp. 613 619, 2011.
Mohammad Khosravi, Varsha N Behrunani, Piotr Myszkorowski, Roy S Smith, Alisa Rupenyan, and John Lygeros. Performance-driven cascade controller tuning with bayesian optimization. IEEE Transactions Industrial Electromics, 69(1):1032-100 2021.
Jack B Kuipers. Quaternions and rolation sequences, volume 66. Princeton University Press, 1999.
17

---

Published in Transactions on Machine Learning Research (03/2025)
S Narendra Kumpati, Parthasarathy Kannan, et al. Identification and control of dynamical systems using neural networks. IEEE Transactions on Neural Networks, 1(1):4-27, 1990.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Sergey Levine and Picter Abbeel. Learning neural network policics with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071-1079, 2014.
FW Lewis, Suresh Jagannathan, and Aydin Yesildirak. Neural network control of robot manipulators and non-linear systems. CRC press, 1998.
Qianxiao Li and Shuji Hao. An optimal control approach to deep learning and applications to discrete-weight neural networks. Xie prepriul 18X3v:1803.012 2018.
Qianxiao Li, Long Chen, Cheng Tai, and E Weinan. Maximum principle based algorithms for deep learning. Journal of Machine Learning Research, 18(165):1-29, 2018.
Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In International Conference on Informalics in Control, Aulomation and Robolics, pp. 222-229, 2004.
Zihao Liang Wanxin Jin, and Shaoshuai Mou. An iterative method for inverse optimal control. In 2022 13th Asian Control Conference (ASCC), pp. 959-964, 2022.
Zihao Liang, Wenjian Hao, and Shaoshuai Mou. A data-driven approach for inverse optimal control. In 2023 62nd IEEE Conference on Decision and Control (CDC), pp. 3632 3637, 2023.
Hailiang Liu and Peter Markowich. Selection dynamics for deep neural networks. Journal of Differential Equations, 289(12): 2020.
Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. Advances in Newral Information Processing Systems, 36, 2024.
Zehui Lu, Tianpeng Zhang, and Yebin Wang. Torque constraint modeling and reference shaping for servo systems. IEEE Control Systems Letters, 8:2637-2642, 2024.
Rui Luo, Zhinan Peng, Jiangping Hu, and Bijoy Kumar Ghosh. Adaptive optimal control of affine nonlinear systems via identifier critic neural network approximation with relaxed pe conditions. Neural Networks, 167:588-600. 600, 2023.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. arXiv preprint ar Xiv: 04490, 2019.
Alexandre Mauroy, Y Susuki, and Igor Mezić. Koopman operator in systems and control. Springer, 2020.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1812, 5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Jocl Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540))522 :529 533, 2015.
Katja Mombaur, Anh Truong, and Jean-Paul Laumond. From human to humanoid locomotion an inverse optimal control approach. Autonomous Rohots, 28(3):369 (8333336-388, 2010.
Oliver Nelles and Oliver Nelles. Nonlinear dynamic system identification. Springer, 2020.
Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. arXiv preprint arXiv:1605. 09128,, 2016.
Masashi Okada, Luca Rigazio, and Takenobu Aoshima. Path integral networks: End-to-end differentiable optimal control. Xiv preprint or Xiv: 096909 2017.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Michael A Patterson and Anil V Rao. Gpops-ii: A matlab software for solving multiple-phase optimal control
problems using hi-adaptive gaussian quadrature collocation methods and sparse nonlinear programming. ACM Transactions on Mathematical Software, 41(1):1, 2014.
Marcus Pereira, David D Fan, Gabriel Nakajima An, and Evangelos Theodorou. Mpe-inspired neural network policies for sequential decision making. arXiv preprint orXiu:1802.05880 2018.
Gianluigi Pillonctto, Alcksandr Aravkin, Daniel Godon, Lennart Ljung, Antônio Ríbeiro, and Thomas B Schön. Deep networks for system identification: a survey. Aulomatica, 171:111907, 2025.
Lev Semenovich Pontryagin. Mathematical theory of optimal processes. Routledge, 2018
Joshua L Proctor, Steven L Brunton, and J Nathan Kutz. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamicall Systems, 15(1):142-185 1611111,1111------ 2016.
Joshua L Proctor, Steven L Brunton, and J Nathan Kutz. Generalizing koopman theory to allow for inputs and control. SIAM Journal on Applied Dynamical Systems, 17(1):909-99) 00099999-30,,100 2018.
Nathan Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In International Conference Machine Learning, pp. 729-736, 2006.
Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adrià López Escoriza, Ruud J. G. van Sloun, and Yonina C. Fldar. Kalmannet: Neural network aided kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70:1532 1547, 2022.
Maria Isabel Ribeiro. Kalman and extended kalman filters: Concept, derivation and properties. Instilute for Systems and Robotics, (7(44):3736-377 2004.
Steindor Saemundsson, Alexander Terenin, Katja Hofmann, and Mare Deisenroth. Variational integrator networks for physically structured embeddings. In International Conference Artificial Intelligence and Statistics, pp. 3078 3087, 2020.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. International Conference on Learning Representations, 2021.
Jelf G Schneider. Exploiting model uncertainty estimates for safe dynamic control learning. In Advonces in Neural Information Processing Systems, pp. 1047-1053, 1997.
Max Schwenzer, Muzaffer Ay, Thomas Bergs, and Dirk Abel. Review on model predictive control: An engineering perspective. The International Journal of Advanced Manufacturing Technology, 117(5):1327- 1349, 2021.
Pierre OM Scokaert and James Rawlings. Constrained linear quadratic regulation. IEEE Transactions on Automatic Control, 166331166,,, 1998.
Pushan Sharma, Wai Tong Chung, Bassem Akoush, and Matthias Ihme. A review of physics-informed machine learning in fluid mechanics. Energies, 16(5):2343, 2023.
Yongkyu Song and Jessy W. Grizzle. The extended kalman filter as local asymptotic observer for nonlinear discrete-time systems. In 1992 American Control Conference, pp. 3365 3365-3369, 1992.
Farshud Sorourifar, Georgios Makrygirgos, Ali Mesbah, and Joel Paulson. A data-driven automatic tuning method for mpe under uncertainty using constrained bayesian optimization. IFAC-PapersOnLiin, 54(3): 243-250, 2021.
Aravind Srinivas, Allan Jabri, Picter Abbeel, Sergey Levine, and Chelsca Finn. Universal planning networks. arXiv preprint arxiv: 1eXic:1804.0060 00645, 2018.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
19

---

Published in Transactions on Machine Learning Research (03/2025)
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXio:1805.0195. 2018.
Laura Von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit. Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, et al. Informed machine learning- a taxonomy and survey of integrating prior knowledge into learning systems. IEEE Transactions on Knowledge and Dala Engineering, 35(1):614 633, 2021.
Hai Wang, Zhihong Man, Weixiang Shen, Zhenwei Cao, Jinchuan Zheng, Jiong Jin, al. Robust control for steer-by-wire systems with partially known dynamics. IEEE Transactions Industrial Informalics, 10(4): 2003-2015, 2014.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally lincar latent dynamics model for control from raw images. In Advonces in Neural Information Processing Systems, pp. 2746 2754, 2015.
Jin-Long Wu, Heng Xiao, and Eric Paterson. Physics-informed machine learning approach for augmenting turbulence models: A comprehensive framework. Physical Review Fluids, 3(7):074602, 2018.
Peng Xu, Fred Roosta, and Michael W Mahoney. Second-order optimization for non-convex machine learning: An empirical study, In SIAM International Conference on Data Mining, pp. 199-207, 2020.
Amy Zhang, Sainbayar Sukhbaatar, Adam Lerer, Arthur Szlam, and Rob Fergus. Composable planning with attributes. In International Conference Machine Learning, pp. 5842-5851 2018.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Painless adversarial training using maximal principle. ar preprint MXXw:1955.06% 2019.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 1433 1438, 2008
20

---

Published in Transactions on Machine Learning Research (03/2025)
Appendix
A Proof of Lemma 2
Since the matrices P. and L, are bounded according to Assumplion and
2
from
11b
one will have:
K, = P.
(25)
PT + Ra)
(26)
Then, by laking the inverse of
25 and 26 one will gel:
P. = (
L R Lt.
(27)
Substituting 25
into 11b
and subtracting both sides from O,, one will have:
0, == 0, P.L,R, et.
(28)
Then, plug 28
into the Lyapunov function 21
Ve = 0
(29)
(0, P. P.L.R,eesPP. P. (0, P,L,R, et)
(30)
=(0, (0,) J.. 0 -0,) e, e L,0, + e,R, L,P,L,, L,R.LL,,,,RRRR e
(31)
Next, we plug 27 into 31
V (0, (CPP() L L. )
LL, c, L,0, + RR + L.P.IR, = Vi + (0,) 1, L.O, ) [ R. e, e R, L,O, + eR L P.L.RR te,
(32)
(33)
where
VT = (0))(P:))-1;;
(34)
(35)
Using
22
33
becomes:
V1 VT + (0,) L,0, (0, LL, e, - L + C.R.
L.P.L,,,,
(36)
= V, - eFF,R F F,e:- F, Ri e. e,
-R-iF,e. Fie: + R2 L, L.P.IRR,
(37)
= =V - ef - - R RF + R. L.L.R. )e..
(38)
To ensure that the Lyapunov function (V decreasing sequence, V, V. 0.
V V,-1
(39)
= eF - - R FF R2 L.P.IR S.. O.
(40)
(41)
Therefore, to ensure the Lyapunov funetion is decreasing sequence,
F R R L.L.R, 50,
(42)
and
G - P
(43)
With some manipulations:
(F. - R, (Fr - 1,) R
- 50,
(44)
FF-1.), (F, - I.) R L.
(45)
21

---

Published in Transactions on Machine Learning Research (03/2025)
By letting I, (L,P. L R.), R have
(Ft - (LiP. L R.) 0..
(46)
Since Ft and Rd are diagonal matrices, we will have
(L,P, + Ra) 00
(47)
which at the end yields:
( RL, L Rd)
(48)
therefore the proof is completed.
B Proof of Theorem
This proof is straightforward once Lemma 2 is provided. Consider the assumptions to Lemma with the exact gradient generated by the gradient generator in (17 18 lim O. = 0. As the
1 andd 2 are met, according
estimated 0e converges to the true 0", where the true parameter gives zero cumulative loss, the cumulative loss ((E())) goes O.
C Experiment Details
C.1 System/Environmeen Setups
Cartpole. We consider the following continuous dynamics of the cartpole
P
(F + m. 162 sin(0)
m1 cos(
(49)
(1) oass (n) myd sin(e)
u1
"ip (o0
where p E is the horizontal displacement of the cart; 0 is the pole angle; F E R denotes the horizontal force applied the cart which is between -1 and +1: E IR the length of the pole; my my are the masses of the pole and total cartpole, respectively. By defining the states and control inputs of the cartpole
P
0 8
and
u F
(50)
respectively.
Quadrotor UAV. We consider a quadrotor UAV with the following dynamics
P1 V1,
mē, = mgr + Fr:
(51)
4n/T = i(wB ((Ww))a//t 9a/1.
JghB = MB - w x JBWB.
Here, the subscription and denote quantity expressed in the body frame and inertial (world) frame, respectively; m and Jg E 3x3x3 are the mass and moment of inertia with respect to the body frame of the UAV, respectively. is the gravitational constant (g = 10 m/s²), = g, E R33 and w E R are the position and velocity vector of the UAV; wB R the angular velocity vector the UAV; qB/1 E R24 is the unit quaternion Kuipers ((wB) is defined
1999) that describes the attitude of the UAV with respect to the inertial frame;
0 W/
Wl
W2
N(WB) =
0
W
wy
(52)
Wy
Wg
0
W
W=
Wy
W
22

---

Published in Transactions on Machine Learning Research (03/2025)
Mg R is the torque applied to the UAV; F, R is the force vector applied to the UAV center of mass. The total force magnitude / = ||Fy|l E IR (along z-axis of the body frame) and torque M [Ma. My Ma]' are generated by thrust from four rotating propellers [Ti. T2, T3, Ti]', their relationship can be expressed as:
f
1
1
T
M
T2
(53)
Mg
1w/2
1w/2
0
T3
M
T4
where tu is the wing length of the UAV and is fixed constant. The state and input vectors of the UAV are defined as:
p V
d'
w'l'
E RR Ta) ER.
(54)
T
T2 T3
Rocket. The rocket is treated as rigid body subject constant gravitational acceleration, 9r E IR3, and neglects aerodynamic forces. The vehicle assumed to actuate single gimbaled rocket engine to generate a thrust vector within feasible range of magnitudes and gimbal angles. We assume that at the landing phase, the depletion of fuel is insignificant. Therefore, omit the dynamics of rocket mass. The rocket has the following dynamics:
Pr = VI,
1
0z CI/B TT gz: m
(55)
45/T = (((w))
JgwB = Ms [JB
Here, the subscription B and z denote quantity expressed in the body frame and inertial (world) frame, respectively; and Js E R3x3 are the mass and moment of inertia with respect to body frame of the rocket, respectively. IR3 and v E R3 are the position and velocity vector of the rocket; Wy E R is the angular velocity vector of the rocket; 4B/z = [9o, Q 42, Q3] is the unit quaternion that describes the attitude of rocket with respect to the inertial frame; Ts E R2 the commanded thrust vector; Mg E R is the torque applied the rocket; CB/I is the direction cosine matrix that encodes the attitude transformation from body frame inertia frame and related qByz by the following relationship:
1 - 2(42 + 93)
2(q192 + qugs)
2(9193 9092)
CB/I
2(1112 (043)
- 2( + o3)
2 + 2091
2(q1qs + 9042)
2 9091)
1 - 2( + +
The inverse transformation is denoted Cz/B
Czyzi
The skew-symmetric matrices [W and ((WB) are defined as follow:
W/
Wy
W/
Wy
[Sx A
Wz
0
Wr
N(wB)
Wy
Wz
0 Wz
W2
0
W
-Wy
Wx
0
Wy
W
The state and input vectors of the rocket are defined as:
w/s 1' E 13
Ppz ozz
18/I
(56)
u = Ts Tn Ty T2) E
Discretization. Discretization done by the following discrete-time form
Eo+1 mt + A g( uc) A $ f (x)),
(57)
23

---

Published in Transactions on Machine Learning Research (03/2025)
where A is the discretization interval.
C.2 Online Imitation Learning
Data acquisition. The dataset of expert demonstrations is generated by solving optimal control system with the true dynamics and control objective parameter 0 = [Odnn dotj given. We generate five trajectories with different initial conditions Zg and time horizons T.
PDP. We employed the PDP in Jin et al. (2200) to solve this problem. The learning rate is fj = 10-4, Five trials were run given random initial 0g.
Inverse KKT method. We choose the inverse KKT method Englert et al. (017 for comparison bocause it is suitable for learning objective functions for high- dig-dimmesiionl continuous santinuuus-spaee systems. We adopt the inverse KKT method and define the KKT loss the norm-2 violation of the KKT condition by the demonstration data:
(1 |Oo Oxoor
( "o: -
OUo.T ( Vo: )
aL
)
min 0, 0,A
+
Neural policy cloning. For the neural policy cloning, we directly learn neural network policy u = p(xx 0) from the dataset using supervised learning, that is
min
(58)
C.3 Online System Identification
Data acquisition. Tn the system identification experiment, we collect a total number of five trajectories from systems with dynamics known, wherein different trajectories 5 = ( uoT--1 have different initial conditions 2o and horizons T (T ranges from 10 to 20 depending on different environment and task), with random inputs uo:7-1 drawn from uniform distribution.
PDP.
We employed the PDP in Jin et al. (020)) to solve this problem. The learning rate is 11 10-4, Five trials were run given random initial 0g. For the neural dynamics case, the learning rate is 77 10-5.
Pytorch Adam to learn neural dynamics. We consider the dynamics of each system (cartpole, quadrotor, and rocket.) are represented by a fully-comeectdd feed-forward neural network f(æ,, Ut, 0). The neural network has a layer structure of (n + m)-2(n + m)-n with tanh activation functions, i.e., there is an input layer with (n + m) neurons equal to the dimension of state, one hiddon layer with 2(n + m) neurons and one output layer with n neurons. The _ (w uo:T-1 } obtained previously are used in stage loss. We conducted five trials for each method with different initial 0. We use Pytorch Adam train the neural network by minimizing the following residual
min Σ |+ fa u. 0)1
(59)
t=
DMDc. The DMDc method Proctor et al. (2016) is a method that is based on Koopman theory to represent nonlinear dynamics with linear dynamics of observables. Observables vb(x;) are some basis functions of states. The observable space has a much higher dimension compared to state space. The estimation the dynamics is achieved by the following optimization:
min Σ A.B 1=0
(bb())))))) Axb(x?) Bu,l
(60)
C.4 Policy Tuning On-the-fly
Neural State Feedback Policy. In this application, we learn the parameters of a neural state feedback policy by minimizing given control objective functions. Specifically, fully connected
24

---

Published in Transactions on Machine Learning Research (03/2025)
neural network that has layer structure of 3n-3n-m with tanh activation functions, i..,, there is an input layer with 3n neurons equal to the dimension of state, one hidden layer with 3n neurons and one output layer with m neurons. The policy parameter 0 is the neural network parameter. For comparison, we apply the guided policy search (GPS) method Levine & Abboel (2014) and iLQR Li & Todorov (2004) to solve the same problem.
PDP. We employed the PDP in Jin al. (2020) to solve this problem. The learning rate is set to be 7) 10 10-4 = 10-6. Five trials were run given random initial 0g. For the neural objective function case, the learning rate is 10-5
25


Reviews:
[{"review_id": "rMzj0C2DE5", "reviewer": "Reviewer_7UGk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nMeaningful Contribution to the Field: The manuscript presents a novel and impactful Online Control-Informed Learning (OCIL) framework, bridging control theory and machine learning to enable real-time adaptation in robotics. This integration addresses critical challenges such as noise robustness and data efficiency, making the work highly relevant to modern robotics applications.\n\nRobust Methodology: By leveraging an Extended Kalman Filter and gradient-based optimization, the framework demonstrates a strong methodological foundation that aligns with the objectives of real-time control and learning systems.\n\nWeaknesses:\n\nClarity in Problem Formulation: The transition from general cost functions to specific quadratic forms (e.g., Equation (2) to Equation (4)) is unclear. Providing upfront definitions and assumptions, such as convexity or differentiability, would help readers better understand the scope and applicability of the proposed method.\n\nIncomplete Comparisons: The simulation results do not include comparisons with classical methods like LQG or state-space models. Such comparisons would highlight the practical advantages of OCIL in terms of metrics like convergence speed and noise robustness, significantly strengthening the evaluation of the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript makes a meaningful contribution to the field of online learning and control. By proposing the Online Control-Informed Learning (OCIL) framework, effectively bridges control theory and machine learning to enable real-time adaptation in robotics. Leveraging an Extended Kalman Filter and gradient-based optimization, the method addresses critical issues such as noise robustness and data efficiency. However, certain aspects of the paper could be strengthened, as outlined below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Introduction:\nThe introduction and literature review provides an overview of the field but could benefit from improvements in clarity and focus. Repetition and verbosity dilute the impact of the narrative, and a more precise problem statement would enhance the justification for the proposed approach.  \n\n- **Sentence structure:** Streamlining verbose sentences would improve readability, e.g., reducing redundancy in phrases like \"integration enables models to produce more reliable predictions with enhanced interpretability.\"\n- **Technical jargon:** Terms such as \"optimal control principles\" should be briefly explained for accessibility, especially for a broader audience with a machine learning background.\n- **Verb consistency:** Ensure consistent present tense throughout the introduction for a cohesive tone.\n- **Examples or citations:** Claims like \"CIL is beneficial for robotic applications\" require supporting examples or references for credibility.\n- **Literature review gaps:** Key connections to related fields, such as model predictive control, iLQR, finite-time LQR, and LQG problems, are underexplored. Highlighting their similarities and differences with OCIL would significantly strengthen the context.\n\n### Technical Comments:\n1. **Problem formulation:** The definition of the class of optimal control systems in the problem formulation section could be clearer. Referring to the dynamical system alongside its cost to be optimized might reduce confusion.\n2. **Cost function:** The transition from a general cost in equation (2) to a quadratic finite-time cost in equation (4) is unclear. Define the cost function upfront, including assumptions such as convexity or differentiability. I understand it is mentioned as a limitation at the end of the paper but it would be better for the readers to see it sooner. \n3. The claim on page 4 that \"we can introduce any loss function to be minimized\" is too broad. Specify the required characteristics of the loss function, such as convexity. Same as my previous comment. \n4. **Lemma 1 format:** The presentation of Lemma 1 is unclear. Separate the statement from the proof and include remarks on its innovation relative to the cited work.\n5. **Regret analysis:** The regret analysis is incomplete and lacks essential details, such as the rate of convergence. The current content in Appendix C is insufficient and should either be expanded or omitted.\n6. **Simulation results and comparisons:** The system identification convergence analysis is overly dependent on \\(R_t\\) and \\(Q_t\\), which are handled more robustly in classical approaches like LQG. To that end, including comparisons with LQG or state-space models in the simulations would highlight the advantages of the proposed method.\n7. **Broader comparisons:** The simulations should demonstrate how OCIL outperforms classical methods (SYSID in the state-space) in key metrics such as convergence speed and robustness under noise.\n8. I find the connection between the state-space model and \\(\\theta\\) unclear. Clarifying the relationship and specifying the dimension of \\(\\theta\\) would be helpful. Including an example in the paper to illustrate the purpose and role of this new parameter \\(\\theta\\) would greatly enhance understanding. In particular, how does one select the dimension of this new parameter?", "broader_impact_concerns": "Future iterations of the framework should address potential safety failures and maybe this can be extended to safe learning.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h9L5kMKIep", "reviewer": "Reviewer_dQYf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Using the gradient generator from Jin et al. (2020) aligns very well with the extended Kalman filter framework. The proposed solution is quite flexible, as demonstrated by its application to three different problems.\n\nThe paper has three main weaknesses. The method is proposed to be an \"online\" method, i.e., executed during the system's runtime. However, all the experiments seem to be executed in a collected dataset and not in a running system. It's understandable not presenting results on the three real systems due to the amount of work. However, at least a running simulation (Gazebo, Mujoco, Isaac Sim, analytical model) of one of the systems should be given (which would also result in an interesting video). If the experiments were actually performed in a running simulation, this information should be more clear.\n\nThe second one is the statement that the method handles noisy measurements, but not a single experiment with noisy measurements is performed. The authors argue that no noisy data was used because the compared baselines couldn't handle this case. Nevertheless, at least an additional trial with noisy data should be performed with the OCIL to contribute to the statement. Another baseline that handles noise could also be included, e.g., low pass filter + iLQR, Robust iLQR.\n\nThe statement \"Therefore, this paper believes that OCIL has the capability to run in an online fashion.\" is very problematic. Using the verb \"believe\" can be very troublesome in a scientific paper; it would be ok if it was related to a potential contribution. But it's connected to one of the main features of the method: online learning. In general, sentences with \"believe\" should be reformulated to eliminate vague possibilities or clarify them as a potential secondary contribution. About the mentioned statement: the computational time is evaluated, but the values are pretty high to be used on a real system with the specified control frequency, especially regarding \"online imitation learning\" and policy tuning. Nyquist frequency can be used as a rule of thumb: 2x the control frequency. The authors could still state that the method works in an online fashion for OC problems with a medium number of variables, but it suffers from the curse of dimensionality from OC.\n\nAn additional issue: more details about the experiments need to be included. How were the systems simulated? The explanation about how epsilon* is computed for the \"policy tuning\" is unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an interesting Online Control-Informed Learning framework to combine control theory and machine learning. The main contribution is integrating a gradient generator with an Extended Kalman Filter, which is applied to online system identification, online imitation learning, and online policy/control tuning. The authors present a theoretical converge analysis.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Address the comments about the three main weaknesses.\n\n- The method seems flexible for imitation learning, but the \"online imitation learning\" described in Section 2 is not different than reference trajectory tracking. Why not consider also the expert action in the loss? In imitation learning, you will usually have pars of states and actions. However, in the presented problem, only the error between the observations is minimized. I suggest changing to a more meaningful name, e.g., reference trajectory tracking or explaining why only the expert's observations are tracked.\n \n- The authors could add additional plots of the system's trajectory (for at least one of them). Especially for the \"imitation learning\" and policy tuning, a baseline using the ground truth parameter values could be included to demonstrate how close to the optimal tracking the OCIL gets.\n\n- Real time is about having a deterministic and not a fast system. To state a system is a real time system, some analysis or arguments about the code complexity and the worst execution time should be given. In the text, many times, \"runtime\" would be more appropriate than \"real-time.\"\n \n- In Appendix D.1: \"g is the gravitational constant (g = 10kg/m2 )\", why use 10 instead of 9.81? And the unity is entirely absurd.\n\n- Check the entire paper for typos, e.g., the last sentence of Online System Identification: \"against m measurement.\"\n\n- Notation consistency, the optimal state and control action at instant 't' should be denoted as x_{t:T+t} and u_{t:T+t}\n\n- The related work could be improved by adding adaptive control techniques", "broader_impact_concerns": "It's ok.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fuJ6PgoBtV", "reviewer": "Reviewer_sujy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n(S1) Ambition to propose a unifying framework capturing different learning-based control problems could have some benefits (albeit this goal and the potential benefits aren't clearly articulated).\n\n(S2) It is interesting that the experiments combine dynamics and cost function parameters in one framework.\n\n\n\n## Weaknesses\n\n(W1) The manuscript is hard to follow.  I had trouble identifying key ideas, as well as following the line of argumentation in multiple parts.\n\n\n\n(W2) No or only weak technical contributions. \n\n* Unfortunately and based on my reading of the current manuscript, I fail to see any substantial technical contribution in this work. The authors say that their framework has two components: the EKF and the \"Gradient Generator\" (GG).  The EKF (Sec. 3.1) is well-known and has been applied to a huge range of problems. The GG (Sec. 3.2) builds basically directly on prior work (Jin et al., 2020). Hence, there seems to be no major novelty. \n* A *potentially* interesting aspect might be in the way that the observation equations in Sec. 3.1 is formulated as an implicit equation via the loss.  But what is missing is proper motivation for why this is beneficial, as well as comprehensive discussion of whether or not this has been considered in the context of the EKF.  Even *if* there is some novelty, I don't think it is major, simply because EKF and gradient-methods are such established techniques, and I could not see any major difficulty here.\n\n\n\n(W3) The problem formulation is both unclear and lacks specificity.\n\n* I failed to see how any of the mentioned problems at the end of Sec. 2 (online SysID, online imitation  learning, and policy tuning) are new.  This should be clearly stated and also reflected in the related work (see prior (W4) below). \n* It seems that the problem that the authors are after is a (somehow) unified description of different types of learning problems in the context of control.  This does, however, not become clear from Section 2.  Further, I fail to see why this is desirable if methods for the individual problems exist.  In summary, I do not find the problem formulation, nor its motivation clear and convincing. What is the specific (technical) problem that shall be solved?\n* I find the stated \"problem of interest\" very unclear. Is this an estimation problem (theta shall be updated), or a control problem (some cumulative loss shall be minimized)?  Further, what is the link between J and L, which both shall be optimized somehow.\n* There are further imprecise or misleading formulations:\n  * \"(1) can be determined by solving the optimal control problem\" -- This is a very general nonlinear control problem. Under what conditions can it be solved? Or do the authors simply assume that it can be solved? (which might be considered a strong assumption depending on the problem)\n  * Notation is hard to follow and seems partly unnecessary: Introducing k seems unnecessary. How is O_t related to the state? Why is this not given in the form of an observation model?\n  * Later in the manuscript, it is used that O_t is not related to theta.  This assumption should be stated in the problem formulation (Sec. 2).  Furthermore, is this really the case for the stated example of O_t representing control input (u_t should depend on theta in general).\n  * The statement \"subject to \\xi(\\theta) is in 2\" is unclear, and IMO not a valid constraint. Do the authors mean that \\xi shall be a trajectory of (2)?\n\n\n\n(W4) Discussion of related work is insufficient and not helpful to understand the potential contributions.\n\n* Many references in the related work are rather old/dated. Only few are from 2023.  Learning-based control is a very active field (see, e.g., L4DC conference and recent sessions at CDC), so I would expect a more up-to-date review here.\n* Topics and scope of the related work need to be better chosen and explained. Important areas are missing. For example:\n  * I don't understand the sentence \"The proposed OCIL framework includes three modes for three sets of classic problems in both ML and system control communities\" and thus the way that the categories for the related work discussion have been chosen.\n  * What is the relation of OCIL to adaptive control?  For example, the sentence \"online learning algorithms adjust their parameters or models in response to new information without training from scratch\" also applies to adaptive control.\n  * \"Tuning policy on-the-fly\" is missing a large body of work on Bayesian optimization for controller tuning (and adaptive/time-varying Bayesian optimization for online approaches) \n  * The main technical components are the EKG and GG. Thus, related work needs to discuss relevant work on EKF and gradient methods (or state that there is no novelty). As for gradient-based approaches, there are different approaches both in control (e.g., extremum seeking control, other local controller tuning methods) and machine learning.\n* The methods in \"Online system identification\" seem to give a broad overview over all sorts of learning methods used for identifying dynamical system. It is missing focus on *online* methods / implementation, as per the claimed contributions. Accordingly, the novelty of the claimed contributions is not supported by the discussion.\n* Likewise, the discussions on \"Online imitation learning\" and \"Tuning policy on-the-fly\" lack focus and are not helpful to underline the contribution either. Overall, the discussion remains very high-level and generic; it lacks focus with regards to the claimed contributions.\n\n\n\n(W5) Doubts about the meaningfulness and strength of the empirical results\n\n* Choice of comparison methods for \"*Online Imitation Learning*\": The comparison algorithms are rather old. Do these represent the state of the art in Imitation Learning? Furthermore, have results from these papers been reproduced to have some certainty that these methods are properly tuned and setup?\n* *Online System Identification*: References are missing for the chosen methods, as well as reasoning as to why *these* methods are chosen. Some more standard comparisons for SysID should be included (like linear methods, e.g., subspace identification, and nonlinear ones, see e.g. Schoukens/Ljung, \"Nonlinear System Identification: A User-Oriented Road Map\")\n* *Policy Tuning On-the-fly*: \n  * Again references are missing.\n  * Apparently a state-feedback policy is used; so, I assume the control remains in a linear regime, which might not be the most interesting setting for learning-based control.\n  * Further, if linear state feedback is used, parameter estimation + optimal control design should give very good results and probably be included for comparison.  Recent work on event-triggered learning (e.g., for linear quadratic control) also seems relevant in this context, as the parameters only change occasionally and not continuously.\n  * Finally, it should be stated what is the theoretical optimum in each case (e.g., with full parameter knowledge) for comparison.  Is this just 0?\n* The empirical results do not include measurement noise (R_t = 0, or very small value). I regard this as a severe limitation as in realistic learning scenarios, one almost always has to deal with uncertain data. And it is not clear if the proposed method can cope this this. \n* Code is not shared to reproduce results.\n\n\n\n### Minor comments and suggestions\n\n* Abstract, \"which synthesizes the well-established control theories to solve a broad class of learning and control tasks in real time.\" -- The wording is strange (what does synthesize control theories mean?)\n* Is \"Control-informed learning\" a new term?  If not, please provide proper references. (More than just one reference.)  If yes, I wonder why such a new term is needed.  The combination of control and ML is often referred to as \"Learning-based control\" or \"Learning for dynamics and control\", see, e.g., https://sites.google.com/umich.edu/l4dc2025/ or sessions at CDC.  If there is special meaning to the term beyond this intersection, the authors should elaborate.\n* Section 4: I don't understand the offline phase.  What is the additional information that is given?\n* The measurement equation (8) is given in non-standard form.  This should be elaborated.  Why is this?  And how do the equations (9b) then follow? For the loss that is assumed for the online SysID, online imitation learning, and policy tuning, can't the measurement equation be written in standard form?  \n* What is the loss actually in the policy tuning case - this isn't stated (Sec. 2).\n* References for the EKF and equations (9) should be stated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript aims to propose a general learning control framework, named \"Online Control-Informed Learning\" (OCIL). At its core, the framework combines an Extended Kalman Filter (EKF) with an unusual observation equation capturing the \"loss\" of a learning problem, with a gradient estimation technique from prior work, needed in the EKF update. The framework is then applied to three learning control use cases in numerical examples.  While such a unified framework might carry some benefits, this reviewer unfortunately failed to see a clear motivation, as well as any technical contribution beyond combining existing techniques and results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Unfortunately, the technical contributions are unclear to me.\n\nIn addition, I have concerns about the presentation, empirical results, discussion of related work, as pointed out above (weaknesses).\n\nIn my evaluation, this manuscript needs a substantial revision. Even if the presentation is improved, I have doubts about any clear contribution to the machine learning and control community.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel combination of existing methods and demonstrates its applicability to different optimal control problems. Although it arguably lacks significant novelty, the approach is sound and at least somewhat interesting and therefore I argue that it meets the acceptance criteria of TMLR.
Decision: Accept (poster)