Online Reference Tracking For Linear Systems with Unknown Dynamics and Unknown Disturbances
Authors: Nariman Niknejad
Farnaz Adib Yaghmaie
Hamidreza Modares
Venue: Accepted by TMLR ()

Abstract: This paper presents an online learning mechanism to address the challenge of state tracking for unknown linear systems under general adversarial disturbances. The reference trajectory is assumed to be generated by unknown exosystem dynamics, which relaxes the common assumption of known dynamics for exosystems. Learning a tracking control policy for unknown systems with unknown exosystem dynamics under general disturbances is challenging and surprisingly unsettled. To face this challenge, the presented online learning algorithm has two stages: In the first stage, an algorithm identifies the dynamics of the uncertain system, and in the second stage, an online parametrized memory-augmented controller accounts for the identification error, unknown exosystem dynamics as well as disturbances. The controller's parameters are learned to optimize a general convex cost function, and learning the control parameters is formulated as an online convex optimization problem. This approach uses the memory of previous disturbances and reference values to capture their effects on performance over time. Besides, it implicitly learns the dynamics of the exosystems. The algorithm enables online tuning of controller parameters to achieve state tracking and disturbance rejection while minimizing general convex costs. It is shown that the algorithm achieves a policy regret of $\mathcal{O}({T}^{\frac{2}{3}})$. In the simulation results, the performance of the presented tracking algorithm was compared with the certainty equivalent $H_{\infty}$-control and linear quadratic regulator.

Published in Transactions on Machine Learning Research (01/2024)
Online Reference Tracking For Linear Systems with Un- known Dynamics and Unknown Disturbances
Nariman Niknejad
nisnejad@msu.edu
Deportment of Mechanical Engineering
Michigaan State University
Michigan, USA
Farnaz Adib Yaghmaic
farnaz www..atdbb.h.ooooo www.hbihggggmmmmiiaiaa
Paculty of Electrical Engineering Linköping University Linköping, Sweden
Hamidreza Modares
modaresh @msu edu
Faculty of of Mechanical Engineering Michigan State University Michigan, USA
Reviewed on OpenReview: https: // openreview. net/ forum? id= p VayaUNe
net/forum?
Abstract
This paper presents an online learning mechanism to address the challenge of state tracking for unknown linear systems under general adversarial disturbances. The reference trajectory assumed be generated by unknown exosystem dynamics, which relaxes the common assumption of known dynamics for exosystems. Learning tracking control policy for un- known systems with unknown exosystem dynamics under general disturbances is challenging and surprisingly unsettled. To face this challenge, the presented online learning algorithm has two stages: In the first stage, an algorithm identifies the dynamics of the uncertain system, and in the second stage, an online parametrized memory-auummented controller ac- counts for the identification error, unknown exosystem dynamics as well as disturbances. The controller's parameters are learned to optimize a convex cost function, which not necossarily quadratic, and lcarning the control parameters formulated as an online con- vex optimization problem. This approach uses the memory of previous disturbances and reference values to capture their effects performance over time. Besides, it implicitly learns the dynamics of the exosystems. The algorithm enables online tuning of controller parameters to achieve state tracking and disturbance rejection. It is shown that the algo- rithm achieves a policy regret of (T2/3). In the simulation results, the performance of the presented tracking algorithm is compared with the certainty equivalent Hoc-contol and linear quadratic regulator.
1 Introduction
Reference tracking is fundamental problem in control theory (sidori 1985 Huang 2004 Dixon et al, 2004 Vamvoudakis et al., (017)) and it has many applications (Zare et al. 2022 where the goal is to design a control policy that steers the closed-loop system towards a reference tra trajectory or set-point. Significant progress has been made toward developing reference tracking controllers for both linear and nonlinear sys- tems. Most of the existing results, however, have made all or some of the following assumptions: 1) the system dynamics are known; the exosystem dynamics generating the reference trajectories are known; the system is deterministic and might be under bounded energy disturbances the system is stochastic and

---

Published in Transactions on Machine Learning Research (01/2024)
under independent and identically distributed (i..d.)) noise, mostly Gaussian noise; and 4) only asymptotic tracking is of concern, and no optimality of performance is concerned.
Several learning-basse or adaptive controllers have been presented to deal with epistemic uncertainties (i.e., uncertainties that can be reduced by collecting data) in system dynamics and exosystem dynamics (i.e. to relax numbers and 2 of the above-mentioned assumptions). Traditional adaptive controllers do not provide performance guarantees, as they only optimize an instantaneous cost function (Lewis) 1986 Sutton
& Barto 2018
Bertsekas 2019: Zhang & Lewis, 2012). In contrast, reinforcement learning (RL) Yao &
Yao 2022 Wang et al.) 2022b Chen et al. 2019 Gao & Jiang 2022) Deng et al.] 2021 Mei et al. 2022
Modares et al. 2016; Rabice & Safari, 2023) lcads to learning adaptive optimal control policies by optimizing
a long-horizon cost function using collected data. Despite this advantage, existing RL-based control solutions
for continuous state-actions are limited to deterministic systems with no disturbances (Li & Wu 2020 Hao
ctt al.] 2021] or bounded disturbances (Li et al., 2020 Mohammadi et al. 2021) and stochastic systems
with Gaussian noise (Cheng et al.) 2019). For the case of bounded disturbances, RL algorithms typically reformulate the H2-optimal control design into robust min-max or Hx-optimal control problem (Khalil
2002 Modares et al. 2015), which can be overly conservative. To this end, some works proposed combining controllers such as resonant control with Ho for a better performance (Dadkhah & Mohcimani 2023 For stochastic systems, existing RL algorithms are either based on policy gradient under which the controller is parametrized and its parameters are learned through the gradient descent method or based on policy iteration method under which the policies are iteratively evaluated until convergence. The former requires approximating the expected gradient of the cost function, and the latter requires approximating the expected cost itself. These results are typically limited to systems with Gaussian noise. However, in practical control systems, adversarics aiming to degrade the control performance can act adversarial disturbancos that are unpredictable and do not follow any distribution.
In this paper, the main focus is on constructing tracking controllers for linear systems when the underlying system dynamics and exosystem dynamics are unknown, and disturbances are adversarial. Adversarial dist urbances can take any arbitrary form and are not restricted to those that are bounded with a known bound or follow a probability distribution. A control policy in the form of (disturbance, reference)-actionn is developed that leverages fixed-size history of disturbances and reference values in its actions. The
presented online learning algorithm extends the results of Yaghmaie & Modares (2023): Hazan et al. (2020)
to systems with unknown dynamics and has two stages: In the first stage, system identifier algorithm estimates the unknown dynamics and uncertainty, and in the second stage, an online parameterized memory- augmented controller is learned to optimize a convex cost function while accounting for the identification
error, unknown exosystem dynamics, and adversarial disturbances. Yaghimaie & Modares (2023) provides a concise parameterization of the control policy resulting in O(VT) regret bound benchmarking against the best linear control policy when the system dynamics known. In this extension to their work, our algorithm achieves regret bound of O(32/3) for linear systems with unknown dynamics. Besides the theoretical guarantees, in the simulation results, we compare the performance of the presented algorithm with couple of relevant solutions including the Ho and Linear Quadratic Regulator (LQR) control to highlight its superior performance.
2 Related works
In this section, the related works to the problem of optimal tracking are summarized. These works are focused on sccnarios with an inducod disturbance on the states and also where the dynamics of the system are unknown.
System identification: Linear dynamic system identification is the process of determining the mathemat- ical model that describes the input-output behavior of a lincar dynamic system and it has been studied in Ljung (1998). The least-squares method and its variants, such as total least. squares and recursive least squares, are widely used to identify the system's parameters (Tatari et al. 2021 Faradonbeh et al.) 2017
Sarkar et al.] 2019). System identification using machine learning techniques, such as artificial neural net- works and support vector machines, has also gained popularity in recent years (Nagumo & Nodal 1967]
Weber et al. 2019: Chiuso & Pillonetto 2019 Mehrzad et al. 2023). The choice of method depends on
2

---

Published in Transactions on Machine Learning Research (01/2024)
the system's characteristics, the available data, and the required accuracy of the identified model. A prop- erly identified model can facilitate the design of robust controllers and the prediction of system behavior under different operating conditions. In situations where there is an adversarial disturbance, the use of the least-squares method may produce unreliable estimates. Thus, this paper exploits the method introducod by (Theorem 19
(Hazan et al. 2020
Output regulation theory:
The output regulation theory, as introduced by Isidori and Huang in their works (Isidori 1985 Huang 2004), has been widely utilized in the design of model-free reinforcement learning
(RL) algorithms for solving optimal tracking problems, well as in attenuating the effects of disturbances
(Gao et al. 2017: Chen et al. 2022 Jiang et al.) 2020b Chen et al., 2019 Jiang et al., 2020a; Gao & Jiang, 2016 2015p. However, a limitation of RL and adaptive dynamic programming (ADP) approaches based on the output regulation theory is that they assume the disturbance generated by a dynamical system, which is not always the case many real-world applications. This constraint restricts the applicability of the output regulation theory in practical scenarios. Additionally, ADP methods typically optimize risk-ncutral (expected) risk-aware measures of the cost. function under the assumption of i.i.d and Gaussian noise. This assumption is made because either the value function is learned directly based on collected data to estimate expected or risk-aware accumulated rewards in policy interaction or value iteration methods, or the expected or risk-aware cost function or its derivative with respect to control parameters is learned from data in policy gradient methods. For general disturbances, usually a robust control approach is utilized which is discussed below.
Robust control design: To handle general disturbances with limited energy, the - theory is often employed to guarantee an Lz-gain performance bound (Doyle, 1995; Khalil 2002 Modares et al. 2015). That said, the Hx-approach is known to be overly conservative, the resulting robust controller is designed to hedge against the worst-case disturbance sequence, which is rarely encountered in reality, To that end, some works proposed online compensation for unknown stochastic disturbances for motion planning and control (Faust et al. 2015).
Gaussian disturbance: The Linear Quadratic Regulator (I.QR) can be used design an optimal controller for linear systems subject Gaussian process disturbance (noise) (Bertsekas) 2012). It is also the optimal controller for noisc-free lincar systems. However, in many practical control systems, the disturbance does not follow Gaussian distribution or the cost function is not quadratic. The provided guarantees so far in the literature are for the Gaussian case.
3 Optimal Reference Tracking Problem
Notations and preliminaries: Let I represent an identity matrix with the appropriate size. Let 1 and 0 denote matrices with appropriate sizes consisting of all ones and all zeros, respectively. The gradient of a function fz) with respect to x is denoted by V.f. The Lg-morm of x is denoted by ||ж||e, (E ||x) where ||ar*|| || is the instantaneous Fuclidean norm of the vector Ig- For matrix A, the spectral norm is denoted by ||A||, and the Frobenius norm is denoted by |A Let lg be an indicator function set E. For a time-dependent variable Fn, the notation Kzjj, j 2 i is defined as Zai = I,,,,,, The notation O() is leveraged throughout the paper to express the regret upper bound as a function of T.
3.1 Tracking Problem
Consider the following linear dynamical system
Tk+1 Az + Buy + Wk:
(1)
where the variables T and uy E R. represent the state and control input of the system, respectively.
In equation [ Wh E "R" denotes the adversarial (unknown and arbitrary) disturbance. Only a bound, which also does not need to be known a priori, is assumed on the disturbance for theoretical reasons. It can be assumed that zo = 0 without loss of generality and incorporate the initial condition into wo.
The objective of this paper is to choose the input variable uy in such a way that the state of the system 24 follows an arbitrary reference signal rk that is not known beforehand. This reference signal is only revealed
3

---

Published in Transactions on Machine Learning Research (01/2024)
sequentially after the control imput has been applied.
2k+1 S2k:
(2)
7k = F2x.
where Ea E RP, Th E IR" represent the state and the output of the reference generator.
In the sequel, list of few definitions and results are brought that are related to equation D eqqution 2 the tracking problem.
and
Definition 1
Agarral el al. 2019 Consider
28+1 = Ar + Buk:
and 7 E 10,1), > 1. A linear controller K is (K, (8,,))ssbble if ||||| S and || S (1 where Ax = A BK. Equivalently, a linear controller K is also (k. 7) stable if there erist decomposition of Åx = QIQ such that ||A||| S (1 7), and |||], [II, IIQII. QIQ | & K.
Definition 2 (Strong Controllability) (introllabbltts))eeiiiitttto (Definition 7 in (Hazan et al. 20201) A linear dynamical system (A, B) is said to have comtrollabillly index X if the matrin Gx is full-row runki and
Gx AB, 12.. AB....A-11).
where Gx is defined as the matrix associated with (A, B) for X 221 1. In addition, such system is also defined (A,k) strongly controllable if [[(GAG))= k.
In controllable system, the controllability index has upper bound of the number of states in the system. It is worth noting that, due to the Cayley-Hamiltor theorem, the controllability index of controllable system is never greater than the dimension of the state space. Assuming that the system (A BK. B) is (A,x) strongly controllable, similar to the concept. of stability, a measurable counterpart of controllability is initially presented by (Cohen et al. 2018
Lemma 1 (Maintaining Stability) (Lemma 15 in Hazan et al. 2020 ) Consider an identified dynamical system with (A, B). Assume that the original system is (K, -lssronlyy stable. It can be shown that control gain K is (K++EAB. 7 2x CA,B- strongly stable for (Å, B), long as || A|i, || B|| S EA.B. For the system with estimated matrices, one has || | | CA.. Assuming [IQII. I|Q I|, | || ||], ||| S K, we define Ā + BK Qi.Q-1 where one can show || 1 - 2K3CA,B, where == L + ((Á A) (B B)K). Moreover, the Q matrix coincide in both the actual system (A, B) and the one with estimated system matrices B).
Theorem is typically leveraged to present a fundamental discovery outlining the necessary and sufficient condition for the existence a linear feedback strategy that can solve the state tracking problem. Specifically, the problem concerns ensuring that zx + r in the absence of disturbances. In this context, the term 'lincar feedback policy' denotes a particular approach uuilized to address the problem.
Theorem [sidori 1985) Consider the dynamical system in equation and the reference signal in equa- tion Assume that ws = O, (A, B) is stabilizable. Assume that the learner has previous knowledge on K such that A + BK is strongly stable. Then, the controller
u (k₁) Ko0 + K1jzk.
(3)
solves the classical state tracking problem Iy + Tk. if and only if there exist matrices 1I xxx and I' e gmxp such that
ITS Al + BT. - =
(4)
and K,f - KT

---

Published in Transactions on Machine Learning Research (01/2024)
3.2 Performance index
The main goal of this paper is to design control policy (Zink) wi:k-1, Ti:k) + ug that optimizes an average cost function, reflecting the designer's intentions. The total cost linked to given policy æ is determined as follows
T JI( = Σ (Tx, up);
(5)
where is the rolling cost. Also, the average cost of a policy x is defined as the below equation
JI(T)= T
1
Σ ca
(6)
3.3
The presented policy
The conventional linear controller in the form of equation aims to mitigate the impact of adversarial arbitrary disturbances the cost function by determining gains Kys and K,f using Hcc-ootrol design. However, this approach is overly conservative and encounters difficulties in online control design due to the non-convexity of the cost function (I ux) with respect to Ky and Kys. To tackle this issue, we work with the class memory augmented policies which is capable of handling adversarial disturbances and unknown dynamics Yaghmaie & Modares 2023 Agarwal et al. 2019).
Definition 3 A Memorr--uummentel Control Policy is denoted by n (K, M, P)
1
u (K, M,
P) P) Krr +
M (r
Wk
pl rk-8
(7)
t=1
==0
where K E K. is a fixed matriz and Y [M, PI [M to be learned. The domains K., y are defined
M _m 11, [p|0] ,..., pmm.- 1]] y are parameters
K = [K A BK (k, 7) - stable),
(8)
y = (Y [M ..... -l 1), plo) ..... plm, 11 [|] ||P((]] S k0xx 7)
Since the policy parameters are learned, which are changing over time, My = [M =11 and P
[P[0] Ph. ppm.- are representing the policy parameters at step k.
Observer that the class of memory-angmenterd policies is more general than the class of linear controller policies. Indeed, a linear control policy is a special case of the memory-angmented policy.
Let o be the state attained upon execution of the policy x (K, Mox-1, Po--1) that generates the control input equation at time k. One can show that the state attained upon execution of memory-augmented control policy is lincar in M. This is established in the next lemma. Consequently, this implics linearity of the memory-augme mugneee- control policy in M. Since the cost function (I u&) is convex in Fk. ug and zi, ug are linear in M, one can conclude that e is convex in M. Let Ax = A + BK and define
h-1
V K.h (M lsk-1) = AV S + E.y
Σ 1=0
BM
k-j-1 11 SSm
(9)
K.h Vk (Pa-h-1t-1) = Σ
A
BP 1-1---1-11 11
(10)
5

---

Published in Transactions on Machine Learning Research (01/2024)
Lemma 2 (Yagghmaie & Modares, 2023) Let 2i be the state attained upon execution of the policy a (, Mo Po:k-1) that generales the comtrol inpul. in equation 7 at time k. Then
may +h-1
5x = 2x (Mox-1, Posk-1) -A
v k.h (M R.y
1:k- 1)Wkk (MM
(11)
m. m,th-1
Vk,z (P8-
K.h
equivalently
k-1
x x = 2t (Mosk-1, Pack-1)
Σ y=0
K.k kyy (Mork
1)kkk
v- +
k,z
K.k
(okk-))kaz-
(12)
2=0
3.4 Assumptions and the optimal tracking problem
In this subsection, list of the assumptions is brought to be used throughout the paper and define the optimal tracking problem in the presence of adversarial disturbances.
Assumption 1 (dynamical system) The pair (A, B3 is unknown but stabilizable. The actual system matrices (A, B) and the identified system dynamics matrices B) are bounded, i.c., |||||, BII s x and [| S BA,,, where EAB denoles the distance belueen the actual and identified system matrices.
Assumption 2 (disturbance) The disturbance sequence we is bounded, i.e., ||wxll S Kw for some Kew > O. Moreover, the disturbance wi does not depend on the control input ug.
Assumption (reference signal) The dynamics of the reference signal generator are unknown but de- tectable. The state of the reference signal 2g is not measurable but the output Tk measurable and Tke 24 are bounded, i.e., ||rx| fir and || Kzz-
Assumption (Known Linear Controller) A control gaān in equalion 7 that makes the unknown system B), (K, (6,7)-stabl stable is available to the learner. In other words, the set equation is known.
Remark 1 Partial knowledge the system model can be typically extracted for many practical systems using the physical information available, which can be leveraged to design robust controllers salisfying Assumplion 4 That is, even though no knowledge of the system models is used during learning, pertial knowledge of the system models (e.g., the matrices A and B belonging to sets of possible system models) is required to find the starting control policy. This a slandard assumption in most control systems since, without a stabilizing control policy start with, the system' state can quickly become large and useless to learn from, and the system can also fail before any learning occurs.
Assumption 3 stipulattss that the reference signal must be bounded, lead to an unbounded average cost. This assumption is commonly employed in analyzing average cost, observed in works like Abbasi- Yadkori et al. (2014) and Adib Yaghmaie et al. (2019). Nevertheless, the issuc of tracking unbounded reference signals can be tackled by exploring discounted cost settings, where an appropriate discounting factor can ensure the boundedness the discounted cost. This approach has been demonstrated in other studies, including Kiumaasi et al. (0144)
unbounded reference signal may
Assumption (cost function) The cost es (xL, ug) conner Tk, uy- Moreover, when || ||u||| S D, it holds that (cx 8D2 and [Z, [V,,, u)ll S G.D for some and G. 0.
Assumption broadens the scope of applicable cost functions beyond quadratic forms, thereby enhancing the inclusiveness of the assumption.

---

Published in Transactions on Machine Learning Research (01/2024)
Problem 1 (Optimal Tracking Against Adversarial Disturbances with Unknown Dynamics)
Consider the dynamicol system in equation 7 the reference generulor in equatãon equation 2 function in equation 5 Let Assumptions
and the cost.
hold. Design a policy in the form of equation 7
B&w
mr
u (K, P) = Kot +
M M Wk
Σ
pll rk-s
from the class of ceamory-augmenter policies in Definition optimize the total cost in equation
Jr(T) Σ ck (2x, ug)
4 Memory-augmented online state-tracking algorithm
We propose Algorithm 1 to solve Problem which will be defined in the sequel.
The algorithm uses the concept of truncated state and cost
4.1 Truncated state, input and cost
Similar to (agghmaie & Modares 2023), we limit everything to a fixed memory length of H. Let my, ug.; fu represent the truncated state, input, and cost if the system had started at 8%- = O. The expressions for ak. Ug are
HM Pk- Pk-H-1:k --k-
(13)
ma + H-1
m, + H-1
K.H. (MA-H-IA- +
K.H.
ky
k,z
y=0
z=0
üg (M H- 1ske Pk 1:k)
(14)
ma
m,
K K (M (M-HH-ik-D Pk- 1
M M 1
pl. plo) Tk-s:
Wk-t +
and the truncated cost fx reads
fa Mr 1. -1. Pb-1) Pa-1, Pa-H-1161) - Tk,
(15)
= (87 (M (M
In Appendix B some theoretical results are provided regarding the memory-augmennted controllers and the associated states and costs which are essential in obtaining the main result in Theorem
4.2 The overall online learning algorithm (Algorithm 1)
Algorithm [ jnvolves two stages. In the first stage, the system dynamics are identified, then, in the second stage, the online controller is learned. This approach is commonly known the explore-then-comee pipeline and will be explained in the sequel.
The algorithm starts in Line 1 with the selection of a stabilizing controller gain K, as well as other necessary parameters.
System identification (Algorithm SysId): Following the initiation of the algorithm, the identification stage begins by executing Algorithm Sysld. As detailed in Algorithm Sysld, the controller in equation 18 is used to collect T3 samples. In this algorithm, it is assumed that the learner has access to stabilizing

---

Published in Transactions on Machine Learning Research (01/2024)
control gain K. System identification using binary inputs with values of and 1 (also called bipolar imputs) is method to analyze and model the behavior of system. By systematically applying binary input sequences consisting of -1 and 1, the system's 's response is measured and recorded in Line 3.In Lines 4,5,6, dummy variables G and Q are computed. Then, in Line 7, a innerministic-equivaalee
matrix pair (Á, B) is identified through an iteralive procedure that determines matrices of the form (A + BK)*B, followed by solving a linear system of equations to recover the original matrix A. Note that during this stage, the trajectory to be followed is disregarded. The estimated dynamics (A, B) is fed to the presented robust tracking algorithm. This identification procedure is inspired by (Hazan et al 2020
Remark 2
Generating stochastic binary signal often involves the introduction of white Gaussian noise, which is subsequently filtered using a carefully selected linear filter. The resulting signal's sign is then extracted, conforming it to a desired binary level. A signal is deemed favorable for the identification of linear systems when it erhibits a small crest factor. The crest factor, with a minimum value of 1, is achieved when employing a binary waveform. However, binary signals prove less useful for nonlinear system identification due their limåted information content and lack of varied excitation levels (Novak et al., 2009
[Ljung 1995)
Robust tracking (Algorithm RobTrack): Upon identificatinn of the system, the domain set is ini- tialized and loop starts. In Line 7, the reference signal rk is recorded. ug in equation 7 is calculated and applied to the system. Next, in Line 8, the next state Fx+1 is observed, and the disturbance the is estimated by equation I6 Selection of WR according to equation 16 ensures that the state, action, and cost coincide with those of the actual system. In Line 9, the algorithm suffers the cost (e(e,, u&). Then Line 10, the truncated state and inputs are computed from equation 13 equatton 14 using the latest values of M, P, and the truncatod cost. fa(M))), Mbm=-)) plol ...,, p(m, I)) is calculated from equation 15 In Line 11, the weights M, P are adjusted with projected gradient descent on the truncated cost fa (M ..... [m pl) p(m. based equation 17
produced by Algorithm| 1
It should be noted that during cach itcration k of the algorithm outlined in Algorithm 1 the values of 01++1:11 and +ib+1sk are already known and accessible. Additionally, for all k < To, 0y and T are defined to be equal to the zero vector. Thus, it is possible to compute the expressions in equation 13 eequaaion [5 for any given iteration of the algorithm. The projection operator 1l and 1l are matrix projection operators with C2 norm of Kg (1 y)) 7).
The properties related to Algorithms SysTd and 1 are given in Appendices
4.3 Regret Analysis
The standard measure for online control based on the gradient descent is the policy regret (Agarwal et al. 2019), which definod here as the difference between cumulative cost of the designed paramctorized control policy æ learned by Algorithm and that of the optimal linear control policy in the form of equation 3
Definition 4 Consider the system in equation 7 Let the control policy be designed to generate the control action ug in equation 7 at time k. Let Algorithm 7 be used to update the parometers of uk. Then, its regret is defoned as
T cs (Z ux) min Jr (K,),
Regret=
Kyek
where ()) is the total cost in equation 5 of the linear feedback controller in equation 3+
The regret compares the performance of Algorithm generatiin controllers from the class of feasible memory- augmented control policies with the best. linear control policy in hindsight.
8

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm Online state tracking algorithm
1: Initialize: Set a stabilizing controller gain K, perturbation horizon Mar: reference horizon Mars rounds of system identification To, number of iterations after system identification T, and horizon of identification
2: Stage 1: System Identification
(A SysId(To. A).
Stage 2: Robust Tracking (Algorithm RobTrack)
4: Initialize y = {Y [M[0] ..... M(m=-11, plol .... Plm. ||]| ||Pid|] S 15 - 7
5: Set d0s = 0 for all k S To and ih = TTa-
6: for k To + 1, 1,.., To + T do
Record Fæ and execute
mr.
" (K, P) Ko +
M ukgt
plel rk-s
Observe th+1 and record an estimate
Φ2 Fk+1 Ave Bug-
(16)
Suffer Ce(,,, ug). 10: Compute S& fx(M(0) Mimu-1) plo) ... plm, in equation 13 eequation 15 for A, B, and w.
11: Update M, P.
M 11 1/ M (M(M M 1m plo plmm-11)
(17)
P - P (M .... mm -1, plll p plm plm.-1l))
-
Algorithm Sysld System identification by inducing random inputs
1: Inputs: To, X.
2: for h = 1,... To do
3: Induce the control
Kæx + Me, Th widdd {
(18)
4: Observe and record the resulting state
Calculate Q, To-X Eteo To-A-1 044s+1 Vj E {A}.
Form Go = (Qo.... Qx-1), G. (Q),.... Q.).
Outputs: Å and B
β = Qo. A = (G A A' BKB..
In the sequel, we give the regret analysis of Algorithm The main technical difficulty in the regret analysis lies in combining the system identification and the online control approach where both the estimated and true dynamics are present. Another aspect to find the right balance between the identification and control horizons to guarantee a sublinear regret bound.

---

Published in Transactions on Machine Learning Research (01/2024)
Theorem Suppose Algorithm 1 is exccuted under Assumptions 1 learning rate nf and the memory size H satisfy rj 0 Genu vT
Let H m mr. Select the
H (log " and To 72/3 Then,
Regret (TT2/5)
Proof: Let K = arg minkek Ek=1 Ck cx uz). We decompose the regret as
T
Regret
(Cx, uk)
Ck
uk))
minx EK
To
T (Cx( un)
(I2, ux)
(Ik, ux))
+
(ohs, ux))
minx EK
mink EX
k=1
k=
To
(CN (IRs u&)
Ck
(IK, U&)) + Σ
(ck (Zx,, ux)
Ck
(IKRs ux))
minx
mink
To
(Ck (TKK ux)
(x),
uk)) + [[A A, B, (r) - J(K* B, (r))
minx EX
k=
+ J(K*),, (10, (1) J(K)A,, fw), (r)).
R2
The term Jo contains the regret for the system identification stage in Algorithm Sysld. The regret analysis is given in Lemma g where show that R₁ OTT)). The term R1 compares the total cost by algorithm using the estimated dynamics with that of the best linear controller using the estimated dynamics.
In Theorem 4 we prove that RG O(VT). The term R2 compares the total cost by the best linear
controller using the estimated dynamics with that of using the original dynamics. In Lemma I we show that R2 = 1/3 Selecting To 22/3, the regret is concluded.
5 Simulation results
In this section, the simulation results are given.
5.1
The dynamical system, reference, and cost function
Consider the following tracking problem where the dynamics of the system is considered as
1
1 0
E++1
0
1
0 1
+
(19)
and the reference signal is generated by
0
1
1.5
0
Zk: 20
[1, 2,, 0.5]T
0
1
(20)
F'k
Zk-
where
Fik
Wik
F1k
E1k
Tik rik
wk
rk
[2k
W2k
r2k
E22
T2k
12k
10

---

Published in Transactions on Machine Learning Research (01/2024)
A quadratic cost with Q 20I2, R 12 is considered; that is
Ck = =e + ug Rug.
Note that the presented algorithm is designed to handle any convex cost function, but quadratic cost is chosen for comparison with classical control approaches such as Lincar Quadratic Regulator (LQR) and -on-oontsolsers
5.2 Disturbances
We consider 7 dilferent cases of disturbance. In each case, the disturbance is introduced at the start of the simulation, and as result, the sequence of disturbances is consistent across all algorithms. The first three cases involve randomly generated disturbances, while the remaining cases involve contimuous disturbances, which allows us to study how the algorithms perform when the disturbances are not stochastic. Also, a worst-case disturbance case considered assess the performance of the tracking algorithm against. an adversary.
Uniformly sampled disturbance It is considered the disturbance to be uniformly sampled from the interval [0,
Constant disturbance The constant disturbance is considered w w2x 1.
Amplitude modulation disturbance The disturbance is considered as Wik W2k sin sin(6xk/500) sin(8xk/500).
Sinusoidal disturbance A sinusoidal disturbance is considered as Wik W2k sin (8t(k//1100)
Gaussian disturbance Gaussian disturbances is utilized where wIk NOO, 0.01) and wax N(0,0.01) in this study. If the system's dynamic is known, the optimal controller for an LQR cost is a linear one (Bertsekas) 2012). The support for the Gaussian noise is not finite for the LQR method. That said, the theoretical results require the disturbance to be bounded. However, the actual bound does not necessarily need to be known and could be large, in contrast to robust control met hods such as Hx. In this paper's simulation, the Gaussian noise generator (uumpyrandomm normal) is utilized which is provided by Numpy Python that generates boundod samples.
Random walk disturbance It is assumed that the disturbance follows a random walk and is generated by wi = 0.999 + 7 where Th- ~ (/,0000000000 0.01). The internal dynamics of the random walk is chosen to be 0.999 instead
in order to ensure the boundedness the disturbance. When the noise follows a random walk, the optimal TQQR controller linear. To illustrate this, the random walk disturbance in equation [
replaced with
Tk+1
Ars + Bu + 0.999wk-1 + 7/k-1-
Here, in each time step k, the state Ty is measured, and according to Assumption 1 A new state variable 5k [x=] wf is introduced, then one obtain
Wk-1 is known.
LA 0.999/
Tk+1
0.9997 Fx +
0 u +
T/K-1-
(21)
Thus, equation with random walk disturbance can be viewed as extended system described by equation 21 where the noise Mh-1 is Gaussian. Consequently, the optimal controller for this extended system is the LQR.
Worst-casc disturbance (Adversary) Tracking an unknown reference signal with an adversary agent can be formulated as two-player zero-sum game in which the control policy seeks to minimize the value function, while the disturbance policy wa desires to maximize it. The goal is to find the
11

---

Published in Transactions on Machine Learning Research (01/2024)
feedback saddle point (ug, wt) such that if we take the rolling cost c; 2x,Qz + u/! Ru; u2, Wt: one has
(zx) min max Up
le Qe: uz Ru; u2u wa].
(22)
The worst disturbance based on the formulation of the zero-sum game in (Kiumarsi et al. be computed at each state as wi - ww (2 Tk) where Kw can be computed as
2017 can
Kw =I= D"
B² PB PB C B'B
where P satisfies the game algebraic Riccati equation (GARE)
R+ BTBPP DB
BTPD
B D
P=A'PA+Q-IA'D ATP
DJ -21
5.3 The compared control approaches
The effectiveness of the presented online tracking algorithm is shown by comparing it to other linear control methods, including the LQR and Ho approaches. These approaches optimize quadratic performance index and are considered optimal for Gaussian and worst-case disturbances, making them the best performers in scenarios where these types of disturbances are present.
In the LQR approach, we consider two cases where the actual model of the system is known, as well as when an estimated model is used for the design. We set To 464, X when we identify the dynamics of the system in approach. The details are given in the description of cach algorithm. In the case of Adversarial disturbance the identified model from the Gaussian noise is used since the adversarial disturbance makes the open-loop system unstable. Learning a system model despite adversarial disturbance is daunting challenge and an open problem.
Online state tracking in Algorithm 1 During the exccution of the algorithm, the value of K is maintained unchanged, which can be obtained based priori knowledge of the systems' dynamics. Note that this K can be any stabilizing controller that the algorithm is assumed to have access to. The other paramcters are chosen as H 5, and P are initialized as zero matrices. In this algorithm, no information about the dynamics of the reference signal is needed. This algorithm only relies on the measured outputs of the reference signal Tk. Similarly, no information about the disturbance is Incorporated into this algorithm.
5, m = 5, and 0.0001, and M
The feedback controller gain Kys is chosen as - -R++ TT+BBP,, -B where P, is calculated using ARE(A, (A, B, Q, Rt,, assuming knowledge of the dynamics of the reference signal. Subsequently, Kss is calculated using the approach mentioned in Theorem I The control law 2 = K Ko + K2 requires knowledge of the state of the reference signal, denoted as 2k: which is constructed from Thp using the dynamics of the reference signal as described in Lemma 1 of (Yaghmaic Modares
LQR:
2023
Certainty equivalent (C.E.) LQR and LQR for random walk: The feedback controller gain Kyb is chosen as -(R + BT P, - whore P, is calculated using ARE(Â, B, Q,R,R, R), with (Á, being the identified system. Subsequently, Kzz is computed with the method mentioned in Theorem [ Also, the control law = K KSDTR + Kyyzı requires knowledge of the state of the reference signal, denoted as 2k, which is constructed from F& using the dynamics of the reference signal as described in Lemma. of (Yaghmaie & Modares 2023)
It is observed in Subsection 5.2 that when the disturbance is a random walk, the system dynamics can be extended according to equation [1] The extended dynamics involve a Gaussian disturbance, and consequently, LQR for the extended dynamics is used as the optimal controller. In this case, the algorithm is referred to as "LQR for randoma walk."
12

---

Published in Transactions on Machine Learning Research (01/2024)
The final maximum and normal difference between the identified dynamics of the system (A, and the actual one (A, B). "Maximum difference" refers to the maximum difference in the identification of entries matrices A and
Table 1:
Disturbance
Max Difference for Norm of Difference for Max Difference B Norm of Difference for
Constant
0.14
0.27
0.03
0.04
Amplitude mod.
0.11
0.17
0.06
0.08
Sinusoidal
0.16
0.27
0.03
0.01
Gaussian
0.07
0.07
0.03
0.03
Random walk
0.22
0.28
0.07
0.10
Uniformly sam.
0.06
0.08
0.02
0.03
Certainty equivalent (C.E.) -c-control: The Certainty equivalent (C.E.) Hoo-control ap- proach aims to design a controller, denoted as K fs,, for the system described by equation D such that the Cz-norm of the system's output, scaled by vQ, divided by the C2-norm of the worst-case disturbance input, denoted as w, less than or equal to a threshold. For the sake of comparison, is assumed that the dynamics of the reference input, described by equation 20 are known. This knowledge is utilized to construct z from F& described in Lemma 1 of (Vaghmaie & Modares, 2023). Then, the control input ug is calculated as ug K x + K5 where K,f is the feedforward controller computed with the method mentioned in Theorem [ Certainly equivalent -contol approach is conservative, as it ensures a finite Cy-gain for the worst-case disturbance.
5.4 Evaluation of the identification algorithm
In this subsection, the performance of the identification algorithm is discussed for the cases of the distur- bance in Subsection 5.2 In Table[] the maximum difference between the actual system and the identified one is summarizod, as well as the norm of their difference. When the noise is non-Gaussian, system iden- tification using random binary inputs and the least squares method have different implications. Random binary inputs can provide diverse frequency content for analysis, but they may be more sensitive to outliers and non-linear distortions, resulting in potentially less accurate parameter estimates. Meanwhile, the least squares method assumes Gaussian noise, and when this assumption is violated, the parameter estimates ob- tained may be biased or less accurate due to the influence of non-Gaussian noise. Thus, the choice wetn these methods should consider the specific non-Gaussian characteristics the noise to ensure reliable system identification results. In table [ the results gained from the presented system identification show that even though the disturbances are not Gaussian except for one case, the identification errors are very close it, with maximum L2 norm error of 0.28 for random walk noisc and minimum La norm error of 0.03 for Gaussian noise.
5.5 Evaluation of the Tracking Algorithm
In Fig. 1 algorithrs in Subsection 5.3 are run for T = 10000 steps and the final average costs are brought in Table 2 The performance of the algorithms over t = [9900, 10000] is depicted in Fig. g
the reference signal is plotted over [9900, 10000] for the representation purpose. The
When the disturbance follows a non non-Gaussian or non-random walk distribution, there is no analytical ap- proach to determine the optimal linear feedback policy. In such cases, the contootr is commonly employed design a lincar foedback policy that ensures a finite Lz-gain for the worst-casc disturbance, al- beit with a conservative approach. If the actual disturbance is not the worst-case scenario, the coontolll may not yield the best performance. According to Fig. 2 the presented algorithm has an even better performance in constant, amplitude modulation, sinusoidal, random walk, uniformly distributed, and adver- sarial disturbances. Additionally, the performance of the presented algorithm is comparable to the actual LQR and certainty-equivalent LQR when Gaussian noise is present..
In the case where the disturbance is Gaussian and the dynamics of the reference signal and the actual system are known, the optimal linear foedback policy can be determined by selecting Kn = -(R + Td B) P,A, where P, ARF(A, B, Q, R), and subsequently calculating Kzz- For the sake of the
13

---

Published in Transactions on Machine Learning Research (01/2024)
The final average cost, as introduced equation incurred by the different algorithms over a duration of T 10000 steps presented. Notably, the most competitive average cost values, indicated bold, are reported for each respective disturbance case. It noteworthy that the evaluation of LQR for random walk is solely applicable to scenarios involving random walk disturbances, and thus its performance is only assessed in such instances. C.E. refers to certainty equivalent and R. W. refers to random walk.
Table 2:
Disturbance
Algorithm 1
C.E. LQR
C.E. Hoc
C.E./Actua LQR R.W.
LQR
Constant
8.04
40.07
29.18
N.A.
57.76
Amplitude mod.
7.83
16.74
12.72
N.A.
17.58
Sinusoidal
15.21
27.71
20.08
N.A.
30.21
Gaussian
5.62
5.32
5.29
N.A.
5.25
Random walk
17.75
236.11
163.02
19.22 / 15.48
236.68
Uniformly sam.
10.09
19.51
16.09
N.A.
21.99
Adversarial
13.76
17.21
14.75
N.A.
17.17
experiment, one take Kys computed this way the initial stabilizing control gain for the presented algorithm well. The results be seen in Table 2 The average cost of the presented tracking approach is considerably close to the optimal control when the noise is Gaussian (LQR). This can be seen in Figure 9 A similar discourse is applicable to the scenario of a random-wall disturbance, as elucidated in Subsection
5.2 where is demonstrated that the optimal controller for the system in the presence a random-walk disturbance can be obtained by solving an LQR problem for the extended system. That said, the uncer- lainty that is present in the model identification the system resulted poorer performance of the certainly equivalents as compared the presented algorithm.
6 Conclusion
In this paper, the challenge of state tracking in the presence of general disturbances is addressed, even when the dynamics of the actual system unknown. An algorithm that combines an identification period with memory-augmented robust tracking algorithm is introduced. This presented algorithm enables online tuning of the controller parameters to achieve state tracking and disturbance rejection while minimizing convex costs. is shown that the presented online algorithm achieves a policy regret of OTT233). In our future research, we plan to extend our approach to partially observable dynamical systems and eliminate the bounded assumption on the reference signal.
Helevrcee
Heterenco
-8.0
9623
8540
3550
8880
1300
8850
9840
3600
3950
Time
Time
Figure 1: The reference signals used for the evaluation and comparison of various control algorithms under different disturbances.
14

---

Published in Transactions on Machine Learning Research (01/2024)
e1k
-1
3
comality equaderree
Contaity EquvolerLLR
Centalndy Equivalent
Time
Time
Figure 2: Tracking error for constant disturbance for the presented Algorithm 1, versus certainty equivalent =oonoolr,,, certainty equivalent LQR control, and LQR control knowing the dynamics of the system using the reference signals Fig.
Algortin
Cenarryy Equiveeent..
IQR Contrnty Fopainient QR
Algondimn
comsiny
1ipe
Centarty Equivalenn
Time
Time
Figure 3: Tracking error for amplitude modulation disturbance for the presented Algorithm 1, versus cer- lainty equivalent -ontrol,, certainly equivalent LQR control, and LQR control knowing the dynamics of the system using the reference signals in Fig.
Acknowledgment
Nariman Niknejad and Hamidreza Modares are supported by the Department of Navy award N00014-22-1- 2159 issued by the Office of Naval Research, USA. Farnaz Adib Yaghmaie is supported by the Excellence Center at Linköping Lund in Information Technology (ELLITT), ZENITH, and partially by Sensor infor- matics and Decision-making for the Digital Transformation (SEDDIT).
References
Yasin Abbasi- Abbasi-Yadkori, Peter Bartlett, and Varun Kanade. Tracking adversarial targets. In International Conference Machine Learning, pp. 377. PMLR, 2014.
Farnaz Adib Yaghmaie, Svante Gunnarsson, and Frank L. Lewis. Output regulation of unknown linear systems using average cost reinforcement learning. Automatioa, 110:108549, 2019. ISSN 00051098, doi: http://jjaatommttia2111....00..0
URL sttps:///olt org/10. 101///.101//] automatica... 2wt.amaaaaaa..........m.. 2019. 108549
Naman Agarwal, Brian Bullins, Elad Hazan, Sham M. Kakade, and Karan Singh. Online control with adversarial disturbances. 36th International Conference Machine Learning, ICML 2019, 2019-June:
15

---

Published in Transactions on Machine Learning Research (01/2024)
Algonth:
Algonthm
IQR
122
Deptuint Enpawvicna
Optairtt Equindient
Time
Time
Figure 4: Tracking error for simusoidal disturbance for the presented Algorithm 1, versus certainty equivalent =oonoolr,,, certainty equivalent TQQ control, and LQR control knowing the dynamics of the system using the reference signals Fig.
Alpithi
Alpoatin
Octaorty EqUKaictH,
Contanty
LQR
EQuialerLLOR
Ceralitty
WW
-1
Time
Time
Figure 5: Tracking error for Gaussian disturbance for the presented Algorithm 1, versus the certainty equivalent H certainty equivalent LQR control, and LQR control knowing the dynamics of the system using the reference signals in Fig.
154-165, 2019.
Dimitri Bertsekas. Dynamic programming and optimal control: Volume I. Athena scientific, 2012.
Dimitri Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2019.
Ci Chen, Hamidreza Modares, Kan Xie, Frank L. Lewis, Yan Wan, and Shengli Xie. Reinforcement Learning- Based Adaptive Optimal Exponential Tracking Control of Linear Systems with Unknown Dynamics. IEEE Transactions on Automatic Control, 64(11):4423 4438, 2019. ISSN 15582523. doi: 1..1099/AC.20199 2905215.
Chen, Lihua Xie, Yi Jiang, Kan Xie, and Shengli Xie. Robust output regulation and reinforcement Dearning-based based output tracking design for unknown linear discrete-time systems. IEEE Transactions on Automatic Control, pp. 1-1, 2022.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-criticn continuous control tasks. In Proceedings of the AAAI conference artificial ûntelligence, volume 33, pp. 3387-353 -397-3395, 2019.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Aignttor
Algarthin
tarrty Laphodee
Cartardy Eqquiadent
Cestuurty Equavalont
Contant Equacient LOR
Cit Lage for zandoon wellt rancom
Ci LUH random wetl
forrandon
Time
Time
Figure 6: Tracking error for the random walk disturbance for the presented Algorithm 1, versus certainty equivalent -conoolr certainty equivalent LQR control, certainty equivalent LQR for random walk, LQR control, and LQR control for random walk disturbance knowing the dynamics of the system using the reference signals in Fig.
gunitinn 1
Algonthmm
LQR Conainy equiealerngee
13R
Centäiny equhaten
Time
Time
Figure 7: Tracking error for the uniformly sampled disturbance for the presented Algorithm 1, versus cer- tainty equivalent Ha- certainty equivalent LQR control, and the LQR control knowing the dynamics of the system using the reference signals in Fig. 1
Chiuso and G. Pillonctto. System identification: A machine learning perspective. An- vual Review of Control, Robolics, and Aulonomous Systems, 2(1):281 304, 2019. doi: 10.1146/ anmurev-control maatrol-053018-02777- URL https:///oi. hecl/aii.rgg/// 10.
Alon Cohen, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar. Online lincar quadratic control. In International Conference on Machine Learning, pp. 1029-1038. PMLR, 2018.
Diyako Dadkhah and SO Reza Moheimani. Combining Hos and resonant control to enable measurements with a MEMS force sensor. Mechatronics, 96: 96:103086, 2023.
Chao Deng, Xiao-Zheng Jin, Wei- Wei Che, and Hai Wang. Learning- based distributed resilient fault-tolerant control method for heterogeneous mass under unknown leader dynamic. IEEE Transactions on Neural Networks and Learning Systems, 33(10):5504 5515--511,, 2021.
Warren Dixon, Marcio de Queiroz, Darren M Dawson, and Terrance J Flynn. Adaptive tracking and regulation of a wheeled mobile robot with controller /update law modularity. IEEE Transactions on control systems lechnology, 12(1):138 147, 2004.
17

---

Published in Transactions on Machine Learning Research (01/2024)
Algonthm
Centrrty Dqundtent
e1k
Aggaibenn
coomliity equaaerree
Contaity
Time
Time
Figure 8: Tracking error for the adversary disturbance for the presented Algorithm 1, versus the Hon-col certainty equivalent contool certainty equivalent EQR control, and LQR control knowing the dynamics of the system using the reference signals in Fig. !
John Doyle, Robust and optimal control. Proceedings of 35th IEEE Conference on Decision and Control, 2: 1595 1598 vol.2, 1995.
Quan-Yoma Fan, Yucong Sun, and Bin Xu. Improved data-driven control design based on LMI and its applications in lithium-ion battcrics. IEEE Transactions on Circuits and Systems II: Express Briefs, 2023.
Mohamad Kazem Shirani Faradonbeh, Ambuj Tewari, and George Michailidis. Finite time identification in unstable linear systems, 2017. URT
01852
Aleksandra Faust, Nick Malone, and Lydia Tapia. Preference- motion planning under stochastic disturbances. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 3555 3562, IEEE, 2015.
Weinan Gao and Zhong Ping Jiang. Global Optimal Output Regulation of Partially Linear Systemss via Robust Adaptive Dynamic Programming. JFAC-PapersOnnLine, 48(11 11): 747, 2015. ISSN 24058963. doi: 10. 10.1016/j.ifacoo..200 2011..00...00......7 URL http:///......../...........//// ifacol.....f.oooooooooooooooolll ifacol. www.co.o.......00 2015. .78
Weinan Gao and Zhong Ping Jiang. Adaptive Dynamic Programming and Adaptive Optimal Output Reg- ulation of Lincar Systems. JEEE Transactions on Automatic Control, 61(22):4164-4166, 2016. ISSN 00189286. doi:
Weinan Gao and Zhong-Ping Jiang. Learning- based adaptive optimal output regulation of linear and non- linear systems: overview. Control Theory and Technology, 20(1):1-19, 2022.
Weinan Gao, Zhong-Ping Jiang, Frank L. Lewis, and Yebin Wang. Cooperative optimal output regulation of multi-agent systems using adaptive dynamic programming. In 2017 American Control Conference (ACC), pp. 2674-2679, 2017.
Gaofeng Hao, Zhuang Fu, Xin Feng, Zening Gong, Peng Chen, Dan Wang, Weibin Wang, and Yang Si. A deep deterministic policy gradient approach for vehicle speed tracking control with robotic driver. IEEE Transactions on Aulomation Science and Engineering, 19(3):2514 45255, 2021.
Elad Hazan, Sham M. Kakade, and Karan Singh. The Nonstochastie Control Problem. In Algorithmic Learning Theory, pp. 408 -421, 2020. URL attp:://arxi..rrgaabss.. 12178 12178
Jie Huang. Nonlinear output regulation: theory and applications. STAM, 2004.
Alberto Isidori. Nonlinear control systems: introduction. Springer, 1985.
18

---

Published in Transactions on Machine Learning Research (01/2024)
Yi Jiang, Bahare Kiumarsi, Jialu Fan, Tianyou Chai, Jinna Li, and Frank L. Lewis. Optimal Output Regulation of Linear MiscreteTTime Systems with Unknown Dynamics Using Reinforcement Learning. IEEE Transactions on Cybernetics, 50(7):3147 3156, 2020a. ISSN 21682275. doi: 10. 10.1109/TCYB.201S. 2890046.
Yi Jiang, Bahare Kiumarsi, Jialu Fan, Tianyou Chai, Jinna Li, and Frank L. Lewis. Optimal output regulation of linear discrete-time systems with unknown dynamics using reinforcement learning. 1EEE Transactions on Cybernetics, 50(7):3147-31 3156, 2020b.
Hassan K. Khalil. Nonlinear Systems. Prentice Hall, second edition, 2002.
Bahare Kiumarsi, Frank L. Lewis, Hamidreza Modares, Ali Karimpour, and Mohammad Bagher Naghibi- Sistani. Reinforcement Q-learning for optimal tracking control linear discrete-time systems with un- known dynamics. Automatica, 50(4):1167-1175, 2014. ISSN 00051098. doi: 10.1016/j.automatica.
02.015. URL http://dx doi. wtw.://oooooo.....ooo//.. 1016/j automatica. 2014.02....
Bahare Kiumarsi, Frank L Lewis, and Zhong-Ping Jiang. Ho0 control of linear discrete-time systems: Off- policy reinforcement learning. Automatica, 78:144-152, 2017.
Frank. L. Lewis. Optimal control((book)). New York, Interscnntescceeeee, 1986, 371, 1986.
Bohao and Yunjie Wu. Path planning for uav ground target tracking via deep reinforcement learning.
IEEE occess, 8:29064 29074, 2020.
Hongyi Li, Ying Wu, and Mou Chen. Adaptive fault-tolerant tracking control for discrete- time multiagent systems via reinforcement learning algorithm. TEEE Transactions on Cybernetics, 51 51(3):1163-111 -1741 2020.
Lennart Ljung. System identification toolbox: User's guide. Citeseer, 1995.
Lennart Ljung. System identification. Springer, 1998.
Alireza Mehrzad, Milad Darmiani, Yashar Mousavi, Miadreza Shafie-Khah, and Mohammadreza Aghamo- hammadi. A review on bata-driven security assessment of power systems: Trends and applications of artificial intelligence. IEFE Access, 2023.
Di Mei, Jian Sun, Lihua Dou, and Yong Xu. Learning-based distributed adaptive control of heterogeneous multi-agent systems with unknown leader dynamics. IET 2022.
Systems: Theory & Applications,
Hamidreza Modares, Frank L. Lewis, and Zhong Ping Jiang. H Tracking Control of Completely Unknown Continuous-Time Systems via Off-Policy Reinforcement Learning. IEFE Transactions Neural Networks and Learning Systems, 255000)2550-2560 2015. ISSN 21622388. doi:
Hamidreza Modares, Subramanya P Nageshrao, Gabriel A Delgado Lopes, Robert Babuška, and Frank L Lewis. Optimal modcl-free output synchronization heterogencous systems using off-policy reinforcement learning. Aulomaticu, 71:334 341, 2016.
Mchdi Mohammadi, Mohammad Mchdi Arcfi, Peyman Sctoodoh, and Okyay Kaynak. Optimal tracking control based on reinforcement learning value iteration algorithm for time-delayed nonlinear systems with external disturbances and input constraints. Information Sciences, 554:84-99 19884499---1 2021.
J. Nagumo and A. Noda. A learning method for system identification. IEEE Transactions on Automatic Control, 12(3):282-22)) 1967. doi: 10.109//ACC.997..08885
Antonin Novak, Laurent Simon, František Kadlec, and Pierrick Lotton. Nonlinear system identification using exponential swept-sine signal. IEEE Transactions on Instrumentation and Measurement, 59(8):2220 2229, 2009.
Pedram Rabice and Amirsacid Safari. Safe exploration in reinforcement learning: Training backup control barrier functions with zero training time safety violations, 2023.
19

---

Published in Transactions on Machine Learning Research (01/2024)
Tuhin Sarkar, Alexander Rakhlin, and Munther A. Dahleh. Nonparametric finite time Iti system identifica- tion, 2019. URL https://arxti: http:////arxiv..o/abbba/s/// 19tp//992..oo 01848
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Farzaneh Tatari, Majid Mazouchi, and Hamidreza Modares. Fixed-time system identification using con- current. learning. JEFE Transactions on Neural Networks and Learning Systems, pp- 1-11, 2021. doi: 10.1100/INNSS2222..111113
Kyriakos G Vamvoudakis, Arman Mojoodi, and Henrique Ferraz. Event-triggered optimal tracking control of nonlinear systems. International Journal of Robust and Nonlinear Control, 27(4):598 619, 2017.
Licheng Wang, Engang Tian, Changsong Wang, and Shuai Liu. Secure estimation against malicious attacks for lithium-ion batteries under cloud environments. IEEE Transactions Circuits and Systems I: Regulor Pupers, (9(10):433744214 2022a.
Ning Wang, Ying Gao, Chen Yang, and Xueleng Zhang. Reinforcement learning-based finile-time tracking control of an unknown unmanned surface vehicle with input constraints. Neurocompating, 484:26 37, 2022b.
Thomas Weber, Johannes Sossenheimer, Steffen Schäfer, Moritz Ott, Jessica Walther, and Eberhard Abele. Machine learning based system identification tool for data-based energy and resource modeling and simula- tion. Procedia CIRP, 80:683 688, 2019. ISSN 2212-8271. doi: /000:////0 wttp://jpooirr..0001 201.2.22.11.22 26th CIRP Con-
www.sennwweneeettcccoooooo URL httpps//ww..cddssddee........o wttpe//eerreet...oo/ccciieeeeeee httpe//ccceaaartce/////////////1 ference Life Cycle Engineering (LCE) Purdue University, West Lafayette, IN, USA May 7-9, 2019.
Farnaz Adib Yaghmaie and Hamidreza Modares. Online optimal tracking linear systems with ad- versarial disturbances. Transactions on Machine Learning Reseorch, 2023. ISSN 2835-8856. URL
https://openrreveew.. http://ee//fforreeeee
Zhikai Yao and Jianyong Yao. Toward reliable designs data-driven reinforcement learning tracking control for euler-lagrange systems. Neural Networks, 153:564-575, 2022.
Soroush Zare, Mohammad Reza Hairi Yazdi, Mehdi Tale Masouleh, Dan Zhang, Sahand Ajami, and Amirhos- sein Afkhami Ardekani. Experimental study on the control of a suspended cable-driven parallel robot for object tracking purpose. Robotica, 40(11))3863-3877) 2022.
Hongwei Zhang and Frank L Lewis. Adaptive cooperative tracking control of higher-order nonlinear systems with unknown dynamics. Aulomatica, 48(7):1432 1439, 2012.
20

---

Published in Transactions on Machine Learning Research (01/2024)
A Real-world Application Example
In the main body of the paper, we compared the performance of the presented algorithm with LQR, Hoo and their certainty equivalent versions. In this subsection, the implementation of the algorithm is brought on a real-world application. The operational efficincy lithium-ion batteries inherently inftuenced by various factors real-world situations, such as ambient temperature, battery aging, and operational status. These factors pose signifieant risks the secure functioning of lithium batteries and may potentially lead to undesirable damage to the connected equipment. Consider the following tracking constant signal problem where the dynamics of the system of Lithium-ion battery from (aa et al. 2023
Wang et al. 2022a
shown in Figure 10 is
1
0
0
R1C
1k+1
0
TaC2
0
Jk
Uk
WKI
(33
where nominal values are (C1, C22 Cb) reference signal is generated by
(39.91 F,3.50 F, 40 F) and (R1, R2) = 7,399) and the
1
0
0
1k++1
1
0
Zh: = -..200
-5.20, 0.89)"
1
(24)
1
0
0
Tk
0
Zk.-
1
where
21
wik
Tik
E1k
Zik T1k
Wrk
2/2
12k
C2k
32k 22k
233
r3k
C3k
A quadratic cost with Q 13, R = 1 is considered; that is
Ck = u Rug.
A.1 Disturbances
Same as the previous simulation, in this section we brought the comparison of the different control algorithms implemented on the system under different disturbances.
Uniformly sampled disturbance Tt is considered the disturbance to be uniformly sampled from the interval [0, 0.5).
Constant disturbance The constant disturbance is considered wik w2x Wak 0.05.
Amplitude modulation disturbance The disturbance is considered as wik Wok 0.05 x sin(6mk/500) sin (88kk)0000)
Wagk
Sinusoidal disturbance A sinusoidal disturbance is considered as wik Wek Wss 0.05 sin(8nk/100).
Gaussian disturbance Gaussian disturbances is utilized where wik (/ 0.05) and Wox N.000.05) in this case.
Worst-case disturbance (Adversary) Same as explained Section the worst-case disturbance generated using the zero-sum game formulation.
21

---

Published in Transactions on Machine Learning Research (01/2024)
Table 3: The final maximum and normal difference between the identified dynamics of the system (A, and the actual one (A, B). "Maximum difference" refers to the maximum difference in the identification of entries matrices A and
Disturbance
Max Difference
Norm of Difference for Max Difference B Norm of Difference for
Constant
0.89
0.89
0.00
0.00
Amplitude mod.
0.09
0.11
0.00
0.00
Sinusoidal
0.19
0.25
0.00
0.00
Gaussian
0.32
0.45
0.00
0.00
Uniformly sam.
0.65
0.68
0.27
0.27
A.2
The compared control approaches
the previous simulation example, the below control algorithms are employed to assess their perfor- mance against the previously mentioned disturbances.
Online state tracking in Algorithm During the exccution of the algorithm, the value K is maintained unchanged, which can be obtained based on a priori knowledge of the systems" dynamics. Note that this K can be any stabilizing controller that the algorithm is assumed have access to. The other parameters chosen as H 5, my my 5, and rj 0.0001, and M and P are initialized zero matrices.
LQR with the actual system matrices
Certainty equivalent (C.E.) LQR
Certainty equivalent (C.E.) -ontrol
The settings of the last three algorithms are the same as defined before.
A.3 Evaluation of the identification algorithm
In this subsection, the performance of the identification algorithm is discussed for the 5 cases of the dis- turbance mentioned in the previous section. The identification period for this system is 293. In Table 3 the maximum difference between the actual system and the identified one is summarized, as well as the L2 norm of their difference. As can be seen in Table for the 'Constant disturbance' case with C2(A A) = 0.89 which similar to the results of the previous example. The best identification accuracy happened for the "Amplitude modulation' case.
3 the performance of the system identification is the worst
A.4 Evaluation of the Tracking Algorithm
for this example, the algorithms are run for T = 5000 steps and the final average costs are brought in Table Tracking the constant signals for the three states of the system = [4900, 5000] depicted in Fig. 11
As can be seen, due to the inherent limitations computing the feedforward term during tracking, particularly when matrix lacks square dimensions, the accuracy of the feedforward term compromised. Consoquently, across all algorithms reliant on this computation such C.E. LQR and H discernible decline in tracking performance becomes evident, emphasizing the superiority of the presented algorithm. It can be seen in Table 4 that the presented algorithm outperforms all other methods in the tracking of the defined reference signal.
22

---

Published in Transactions on Machine Learning Research (01/2024)
The final average cost, as introduced equation incurred by the different algorithms over a duration of T 5000 steps presented. Notably, the most competitive average cost values, indicated in bold, are reported for each respective disturbance case.
Table 4:
Disturbance
Algorithm
C.E. LQR
C.E. Ho0
LQR
Constant
2.35
28.28
51.15
30.77
Amplitude mod.
1.34
23.32
2.02
22.80
Sinusoidal
1.17
27.36
20.94
22.63
Gaussian
1.19
30.51
70.02
22.63
Uniformly sam.
1.94
31.45
26.01
23.61
Adversarial
1.77
12.66
2.60
7.58
Appendix B: Theoretical Results
Results related to the memory augmented controller
In this section, list of results related to the memory-augmented controllers brought.
Lemma
Pk,z provides bounds for K.h and K.h
Lemma gives bounds on the states and inputs.
Lemma
provides a bound on the tracking error.
Lemma defines the Lipschitz condition on the truncated cost.
Lemma gives bound on the gradient of the truncated cost for the tracking algorithm.
Lemmas
8
given in (Yaghmaie & Modares 2023 and Lemmas
5 inspired by Yaghmmae &
Modares
2023 and are essential in proving the main results in Theorem
Lemma 3 (Lemma 4 in (Yaghmaic & Modares, 2023 Let Assumptions hold. Suppose that K is (k, -tstoongly slable. Then,
I Kih k.y S (1- " Sh + m 2. kg[11 - -7)011
(25)
K.A 1,,
<m,,' 2n 7)=-1
Lemma 4 Let Assumptions 7
hold. Define
Yo.k == [Mo.ks Pock]
YH.: = [M H:k, Pk -H.k].
D := k + (n + mw (1 - 7)) 1n (Bw + Korite )
x²21 7)
Suppose that K and Kjs are ( strronnlyy stable. Define ziinn( (K Kir) as the system state corresponding to optimal linear feedback controller. Then, one has
(nax(1<x (Yo. 1[ 111. (K7 KT1)1) $ D. max (Yo (Y DD
(26)
(Voo(o)))1) - (V,,,,-))) S 5x²(1 - H D,
u (Yo.x) - (YH.)) (1) "D
23

---

Published in Transactions on Machine Learning Research (01/2024)
Proof: Using equation one has
m,+H-1
- <| k -||=T==11 m1l + Kiw E
K.H (MA-H-Lik-1)ll + Rr
K.H (PA-H-1:8-1))
1l
< - 7) " | + kwy (1²2 + moo 12 -
(6-1--)-11 1) + Kr7 (m, (i]{( - 7)-
The above recursion satisfies
| S 5 kin + (R + kw (1 -7) -)==)) km
x²(1 7)
Similarly, from equation 13 one has
mir H-1
Mn+
[Y
1
K,H
+ Σ
1 (1N---118--)
K.H
Kiy
==0
S S k 7 Mw + K,m,,, )x Ri: - -7)-1 S.D D.
where the last inequality is obtained because 1=1- -1)" $. Moreover,
1-1
1-1
1 Nin (K jb: Kit)D1 )1 =
Aij Wh --1 +
Σ
A'yy, BKj z
y=
1=0
Sg n22 (Kw + kKbbiz)) S D.
Besides, one has
m,-1
Jug (Yo:))l
[| (Yo:k-1) +
M1 Wk Σ
plolyk-ol
Man
mn-1
x xx (Yox-11)l + Now
Kitt" 3 (1 - (t-1) + Krr
Σ Kot (1 - 7)
t=1
==0
Sy K + (K + 10 (l - -)) - (Niw + Kep )oo
S D.
x= 7)
?
Similarly,
my
m,-1
| (YH,))] |( (YH, +
Σ
M 1t-
Wk-t Σ plaly--al
min
mw-1 1
S x| (YH,x11))l Kew
Kibk 31 (1 -7) 1) + Krr
Σ
Kixx 7)a
8=0
(Niw + Ken )obks
S - kwn + 7 (Kro + Ky )x 2np(1 - 7) +
S D.
To bound the difference between the actual and truncated state, from equation 13 and equation 11 one has
| (Yo:k-1) 2Y = 1 (1-
which gives
(Yous) (YH,,))
H (Yo (aax---1)) 551(1 - D. |
This completes the proof.
24

---

hold. The following gra- Published in Transactions on Machine Learning Research (01/2024)
Lemma 5 Let Assumptions
hold. Suppose that K is (K, stsronnlly stable. Define the tracking ervor e/ = 2x (Yo.k-1) zR.
Then,
1-1
lexll S Now Y (1² + mwrk n2 - 7)) + Kr, , (1) "m, Kg + Krr
Σ Kg.
Proof:
Without loss of generality and for simplicity, assume that |F] = =|F-11 S n. The tracking error
reads
ek
v K.k k.
(Po:k-1)Tk-> F2k. k,z
K.k
Using the bounds in equation
25
k-1
||ea||
Σ
(n2(1 7)" IISkI + mw x3](1 - -7)0-1) + KNg +
Σ
m
2 kig (1 - -7)²-1
Kir
8=0
==0
- (n² + m x1 1)
) RK; KrY 2 m 2
which is based on the fact that En (1 - 7)²² S 1 the last inequality.
Lemma 6 (Lemma 7 in YYaghmaic & Modares 2023 Let Assumptions 1 hold. Define YH.k
[Y,,..., Yi,..., Y2al = [M Pk-H-k] and YH.k = VV.,.., Ye..., Y2hl where YH has all its elements the
same as YH,K, except one element. Then, the truncated cost function in equation 15 satisfies the following Lipschitz condition
[fa(Y) Yo.,. Y2n) fa (Y......... Yo..., Y2H)1 s LsllY, Y.ll
where
L = 3G D Kon + k..)
Lemma 7 (Lemma in (Yaghmaie & Modares, 2023 Let Assumptions 1 dient bound satisfied
(rs, O7 mnb = G5
Nynnaa(Yaalll S GH&G,
where d max (n, m).
C Results related to Algorithm Sysld
In this subsection, list of the properties related to Algorithm Sysld is brought. More specifically:
Theorem gives the bounds on the estimated dynamics.
Lemma gives bounds on the state and the input while Algorithm SysId is running.
To this end, Some additional notations will be required along with the proofs. Let
J(AJA, B, (w), (r}) be representing the total cost associated with exccuting the algorithm A over the T time steps. With some abuse of notations, one can say J(K] [A, B, (w), (r}) shows the total cost associated with executing the linear controller K,
25

---

Published in Transactions on Machine Learning Research (01/2024)
B, (r}) be the state visited at time k, and
ux[A]A, B, (w), (r)) be the control input at time k.
Also, if instead of B) in the above notations, B) are used, it means that they are associated with the identified system instead of the actual one.
Theorem (Theorem 19 in (Hazan et al., 2020) If the system identification algorithm run for To sleps, the oulpul pair (A, B) salisfies, with probubility
6, that l| Al.oo B Bl S c, where
To 100 Amn? 10
log
kmn
E
8
Lemma 8 Assume that Algoritho SysId is run for To steps. Select the imput as up Krg ak, where Tle Mel. .. Memm ] Ukj {=1}, ,,,,, m. Define
33 Dis = - (Now + 1o kmVm) + vm.
Then,
[x S
Did, ||u/||| Did:
(27)
To
Jo =
Σ
[[c ux)
Ck (ox, un)ll < 4TGG.DD mink EK
(28)
k=1
Proof. From the strong stability of K, for 2a, one has
(A + BK) + Bn))ll
i=0
x2 (1 (om
Based on Ilk {=1)", one has || S mm. As a result, using the fact that E/ 7)2 S
x² S
||
- (Nim + Kb mm).
(29)
7
For ||u]|, one derive the following
x² || KK mell (Niw koVm) Vm Dia-
Next, an upper bound for (Tx, up) min C (Dk, uz)ll is computed. Based on Assumption 5 one has
[] ux) min cx ux)l
KeK
< G.Diallar - 2z Opt JA, B, (w), (r)) -A ODt A, B, (w), {r))]]
< 2G. + -G. < 4G
The result in equation
28 is concluded by summing the above inequality To steps.
26

---

Published in Transactions on Machine Learning Research (01/2024)
D Result related to Algorithm |
The following lemma provides an estimation of the upper bounds for the state, control input, and perturbation during the tracking stage.
Lemma 9 In the tracking step of Algorithm for ko To +1 subsequently,
x² (Kw + k6Vm)) + 3 Kir) =: Dzi
| I ?
(Rw + Ni (Kim + Eu +
(30)
x3
k²
||u S xD2 + Kbb - (Kio
(15r + Em +
-
(Kim + Kb kbVm)) + Kib li--n Kr = Du.
(31)
and
1 wxll S (D,n(D2= + (nDr + ni Rb (Nim + EuT +
x²
(Niw + Ki k6VmL) + xis Rib krr)) Ewe
(32)
?
?
x3 2
K3
CA + 7 (Niw + KKb KDVM)) + lib Kir) =: Fow,
where Dz, Du and Ew are the upper bounds to the state, control imput and perturbation estimation error, respeclively- Tw is the upper bound on the perturbation error al =T
Proof. We prove the result by induction. First note that for equation if the input is chosen as
u
Ko
MD-1u Ur
p rk
(33)
8=0
the state 2 reads
mu
m.
Tk+1 Σ
(A BK) *
B
MDS 10 10
B
>
pl pk rk-s)-
As result,
2k+1
Σ
(A + BK) (0K))~((ww) + B M wx + B Σ pla) p
(34)
i=0
==0
k
k (1 - - + 1ob 1201 ||
kab (1 - 7)) + Kb |T
Σ Kbn (1 - ))
1=0
At time step & = To, one defines (r. FTo whose upper bound computed in equation 29 Thus, for = To:
one has
2 (Kw + Kib mm)
|0r S
Based on the fact that Σk (1 - 7)² < 1 for ts = To. one gets
| S 1.2
(No + 1sb 33 x2 (Ro + Kb (bym) + nib n.2 kr) =: Da.To11-
27

---

Published in Transactions on Machine Learning Research (01/2024)
Then,
| + 1 |K ||
Σ
|M 11
:11 + Σ
pls] rk-s
2=1
==0
S &KDr,T611 + +10x||
Kgr (1 - --)
1- + ||
Σ
Kgr (1 -7)
j=1
==
K3
x
x3
S KDDo.T++1 + Kb
New + Kb k6vm)) + kbb Kir = D Do+To
Assuming that a| A|I.|BB - B|| CABB at k To
|| - w & II((A A) + (B B)un)l S cABBlEE + tallluul
Thus,
2²
R3 x2
(No + Kbb kb/m) + xig A: (k=Kk))) -),))
100 wall S EA.B( C (Kiw + No
22
x33 (Nem + kbb kbVm)) + Kib -Kr) krr) ) = Ew,
(n,n(KD22TT) + kb
and
[ S max(Ew,, + New (10TT
Then, if the upper bound for || is replaced with Tu, + Kiw + 11 ll in equation 34 the bounds in equation 30 equation 31 are concluded.
Helper results for the proof of the regret bound E
To prove the regret bound, few results are needed.
Lemma 10 is technical result to be used in Lemma
Lemma gives an upper bound for the difference in the costs for the real and estimated systems using linear controller.
Lemma 12 provides an upper bound for the difference between the cost of using the tracking algo- rithm and the minimum cost. that be achieved by the same class of controller.
Lemma 10 For any matria pair P, P,P,, such that || P + API $ - 7 it holds
|AP|
I P'1 S
22
Proof. This proof is based on an inductive argument. First,we we prove that the inequality [I -Pils IPll ||А[||[[ - 7) holds true. Then, the validity of this claim can be easily verified for = 0 and Next, it is shown that this claim is valid for the case i + Observe that
[I(P + AP pi+! ||
< [(P AP) + (PP) - PP)
[P((P+ APY P*) +
28

---

Published in Transactions on Machine Learning Research (01/2024)
It is known from the claim that ((P+AP))-P1 s d||AP][( -7)=1 and it is assumed that ||+AAP| $ 11-7 thus,
[P((P + ||PPP++ PP
$ <|1-0/\APP(0->>>>~~~|||PPPP
(i+ 1))
Then,
30
20
II(P + AP) P'I Σ Σ
IIP+ AP)(tt) p(+ti S
8 (1 (0+1)(1-))1l 1001->))~110
Hazan ct al. 2020) in Lemma 17 showed that Σ i(1 - 7))=1 S 2/2 Thus, the proof is concluded.
1BB BII EA,B, where EA,B S 0.25k and that K is ()- strongly stable with respect to the pair B). Then, from Lemma g for any perturbation sequence satisfying ||w wx & Fre, and it is assumed that || S Wo, the following statement holds
Lemma 11 Assuming that ||A All S EA,B,
B, ((K,), (. (r}1 S poly(K, 7. m, n, Kww G.T
y for (A, and | 1 - + 2x A.B from Lemina It can be said that -= L+ AL and |||||| < 22 Using Lemma 10 for L and it can be stated that if one take EA, = -x will have [T]||] and || z. The linear controller K is equation 3 and is (1,, n-sttongyy stable for (A, B). It holds
Proof. One has ||]|| S 1
7
x* KKA, B, (w), (r)) -( (A+ (B wkk)) wk-i)-
Thus, with the assumption that Nob S k without any loss of generality,
| (K B, B, [w), {r})ll s k'ng
+ 2=pm0
Similarly it can be said that by Lemma since EA = 4nz K > 1, and 0 I
< 1 one has
|| <
||||| EA and || < [B]]] EAB hence
| S K +
7
S K +
3 < 2
4x3
B
S k +
7
S k
+
S 2 k.
3x
4
Thus, K is (2k, 3) -tsronyyyyy stable for (Å, B) and
[K]A, B, (0), (r))1 - (2x)'
(2x)² Kzz
32x Axzz
8x2 W
Wo
7/2
7/2
7
29

---

Published in Transactions on Machine Learning Research (01/2024)
Subsequently:
| B, (w), - ( B, (w)), {r}]]
k (I + BBBB)KKK.BB.a.. (A+ Bky ux= + (A+
+44
(A+ Bx) BK
i=0
(A+ (I (A BK (b) x0==00
II(A BK (b) 0xx= (A + Bkyo) mi
+ |I(A + BK (A+ ) Bk
Starting from the first term, one has
122 2 +
II(A + BK wk-i (A Bkib)' 10 S
2=0
For the second term, one has || + yk S QQ~ and ||A+ Bkyall S SQ-ILQ. Since |4],
|1 1 3.
then [| Bkll ||A + Bkll s (1-(1-3))))001111 Q. Additionally, ||Q]], ||Q-1|| -1 S k. Using Lemma 10 and knowing
|| || $ 22
(II +
(A + Bkbu) (xxxil)
< K Wo Σ (01))-((1)
k=0
I 8K B. Wo-
72
For the third term, following the same steps from the last part,
k
I((A +
(Å (KK
An e
72
Thus,
(K|A, B, [W], [r]) 2x+1 (K|, B, (10), fr\iil
6²2 En + x2 ()(11)))Wo= +
8W 8WoN" CA.B
8k CA
72
From assumption 5 it can be shown that || 11. || < D and ( w)]l I ll (ucu(z, w)ll & G.D. With
abuse of the use of the notation, let D denote the bound related to the identified system. It holds:
u (K|A, B, (1), {r}) (K) (w)) + K5fz441.
30

---

Published in Transactions on Machine Learning Research (01/2024)
Thus,
Wo
32x Riz
11 (KJA, (w),, fr))il
+
+ Kfiz =: D.
7
As result, we have
B, (-0),tr) - ( (r3) &T..DI (K|A, B, (w), [w)](r) (r))
- ( B, (,), B, (r)il
8k W
+ kK²)( 32mºk; +
8K 8k
WOEA,B
8x BA'EA,BRZ + x²2 - Wo n²² x²=u
< TG,
+
7
7/2
7
From equation 32 one can derive E S poly(ki fG, kb. m, Kws y Ry CAAB and Wo I E Also, from Theorem known that To 2 A, Niw; m, n). Thus, one can write the above inequality as
S poly(k, 1 X, m. n, Rugsy 16 GT. ?
-x)
Lemma 12 Let Assumplions
5 hold. Let xx denote the state using the optimal memorr-uuumeeteed ,P',,,.,,
controller uh Set H my my- Let YH == , ,.... ,....P" Po] denote the optimal weights learned
by Algorithm 7 each one of the weights repeated for H times, and 55 states and control using these optimal weights according to equation 13 equation 14 Then
(Y H.k): 0% (Y B.k) denote the truncated
- rk. cx cx Tk, 22G 7)
Proof of Lemma 12: Stacking optimal learned weights for times makes You == [M"...., M .",... P*], and then stacking them for times defines YH = [M* ...., M P ..., P*). Based on Lemma one has
le (e Tk, (YY))) c (xx rh ux ){ < G. (1 ( - rxl]ll + G.D ul (Yo üß (YH,,)) x2 71
This completes the proof.
Theorem Suppose A Algorithmm RobTrack is executed under Assumptions 7 Let H = mw my
Select the learning rate y and the memory size to satisfy n = O G.Nu / H = (log
x²T ?
and To = T2/3
The regret of Algorithm Rob Track on the identified system (Ā, B) and the perturbation {w) is
J(A AAA
where J(A4Pt]AA B, (w), (r)) denotes the total cost associated with the optimal memory angmene-dgmented policy ut"
31

---

Published in Transactions on Machine Learning Research (01/2024)
Proof: To begin, one has
J(A B, (w). (r]) J[A ([A (w), [w){(r))
Σ
Ck (ek, uk) Σ
ch (x - rk, u
h=1
T
Ce(EN (York un (oot-ox-1)))
Σ k=1
fu(HH))
Or
H.H.) Σ Sa(
Fr
+
fx Σ
ck (2)
Aem
CT
p(0]_ where Y = [M/[- M|
1|-
p( 11]] E (Rmxn)2H denote the optimal weights learned by Algo-
rithm D satisfying the conditions in 3
The regret analysis is split into three parts: ar denotes the difference between the cost of Algorithm D and the truncated cost. 87 denotes the difference between the truncated and optimal truncated costs. Çr denotes the difference between the optimal truncated cost and the optimal memory-augmentee control policy.
The bound of the first term or is given by
(ek(ek, ux) fv((H s G. D (1 (Yo.-1)) rw) - - llu (Yo.-1) ag (YH,))
2G, (1-
where one can use Lemma to get the above result. Therefore,
||T||
Σ ca (ex, uk) Σ (Vm.))) ST (T (1-))) O((T)
(35)
where the last equality is obtained based on = T).
The term ST can be bounded by Theorem 4.6 of Agarwal et al. 2019 and the results of Lemmas and as
Σ (<HYn))) Ea(rs
< _ -M + TGPn + L,HtHGGTT
1=1
Ollog (T)); then Br where Mo = 2 2vdnon3 d m). By selecting 1) O (.Ru H O(VT).
The last term is the difference between the truncated cost of the algorithm and the cost by the optimal memory-augmented controller. For the third term, using Lemma 12
fa(Y (0T Tk, " ux') s 2T (2
where the last equality is obtained based on = O(log (T)). Observe that
If ef S 2 Lemmna guarantees that & is (2x, -tsstryy stable on (A,
If I poly(K, 2), the iterates obtained by running Algorithm A (trajectory tracking algorithm)
satisfy ||xx|l | I poly(Ns 1, n) + kw)) guaranteed by Lemma
32

---

Published in Transactions on Machine Learning Research (01/2024)
Given the aforementioned observations and the proof before that, it is ensured that
J(AJA, J, (0). J[A'H) IA, (.). -(A9))( S (OVT)).
Also, for the sake of completeness and self-contaimment of this article, it can be mentioned that from Yagh- maie & Modares 2023), it is known that
and following the same steps as before one can conclude
J(A) A, B, - (C)) OVVT)
33

---

Published in Transactions on Machine Learning Research (01/2024)
Apalts
Cannry
Average Cor
Average Coss
223
000
143
000
89
20.3
Time
Time
(a) Ganssian
(b) Constant
Appalins
Apprtii
Octariy Squnderr
Cate by Esavalent I.
200
Deltaly crpioentt..
Average Con
A0
Average
000
2g
%
Time
Time
(c) Random walk
(d) Amplitude modulation
Appetrs
Appittie
Average Cont
Awerage Cost
600
(e) Uniformly sampled
(f) Sinusoidal
Apiitrnd
Auenage Coot
80
600
Time
(g) Adversary
Figure 9: The changes in the average cost Jr, as given by equation over time T is compared between Algorithm I and other control methods, namely certainty equivalent H -ontoolo certainly equivalent LQR control, and LQR control with knowledge of system dynamics, for different types of disturbances such as Gaussian, random walk, uniformly sampled, constant, amplitude modulation, simusoidal, and adversary disturbances.
34

---

Published in Transactions on Machine Learning Research (01/2024)
Ra
V
V/2
Vic
C1
C2
R₂
R₂
Figure 10: Circuit a Lithium-ion battery.
Coruuity Sunvilet
Corariyy Coundeed LOR
Coroity Dundell LOR
Figure 11: Tracking performance for constant disturbance for the presented Algorithm 1, versus certainty equivalent Ho-conntol, certainty equivalent LQR control, and TQQ control knowing the dynamics of the system.
35

---

Published in Transactions on Machine Learning Research (01/2024)
Coruity Convien
Ceranry Connlee LOR
Ceroety Convles LOR
Figure 12: Tracking performance for amplitude modulation disturbance for the presented Algorithm 1, versus certainty equivalent H -contol,,,, certainly equivalent LQR control, and LQR control knowing the dynamics of the system.
Coantty Donvibl
Countyy bondded LOF
Countyy bundel LOF
Figure 13: Tracking performance for sinusoidal disturbance for the presented Algorithm 1, versus certainty equivalent Hon-control, certainty equivalent LQR control, and LQR control knowing the dynamics of the system.
36

---

Published in Transactions on Machine Learning Research (01/2024)
Mann
Coroity Convien
Ceroery Conaee LOR
Figure 14: Tracking performance for Gaussian disturbance for the presented Algorithm 1, versus the certainty equivalent H =contolll,, certainty equivalent LQR control, and LQR control knowing the dynamics of the system.
Coannty Donvabol 120
Coultiy boundee LOF
Counrty Donndel LOn
Figure 15: Tracking performance for the uniformly sampled disturbance for the presented Algorithm 1, versus certainty equivalent Hoc-oontoo,, certainty equivalent LQR control, and the LQR control knowing the dynamics the system.
37

---

Published in Transactions on Machine Learning Research (01/2024)
Dekenn
Cenare Conssles LOR
Certiny Coholes
Figure 16: Tracking performance for the adversary disturbance for the presented Algorithm 1, versus the =ontorl,,,, certainty equivalent Hxcccontoo, certainly equivalent LQR control, and LQR control knowing the dynamics of the system.
38

---

Published in Transactions on Machine Learning Research (01/2024)
800
800
1g3
Equiokrrugg uge
Average Cost
Average Cost
200
250
350
3000
2:30
3000
10:00
3000
.000
3000
/0:0
Time
Time
(a) Gaussian
(b) Constant
800
800
densiee cauldler
Gersiter caulderr
100
1.0
Cerairy Eaiiskrsuqs uge
Average Cost
Average Cost
200
200
150
3000
-5:00
3000
40:0
1000
2000
3000
4000
Time
Time
(c) Amplitude modulation
(d) Uniformly sampled
200
300
Gensiter sauiderr M.
Certaray Sainskri ug3
Average Cost
200
Average Cost
200
200
250
3000
2520
3000
4000
1000
200
3000
4000
Time
Time
(e) Sinusoidal
(f) Adversary
Figure 17: The changes in the average cost Jr, given by equation over varying values of T, is pared between Algorithm 1 and other control methods, namely certainty equivalent -conoollo,, certainty equivalent LQR control, and LQR control with knowledge of system dynamics, for different types of dis- turbances such as Gaussian, uniformly sampled, constant, amplitude modulation, sinusoidal, and adversary disnurbances.
39


Reviews:
[{"review_id": "aVMs8sVCLa", "reviewer": "Reviewer_X7qZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed approach is novel and of value to the control community.\n\n2. The proposed method is theoretically sound. The authors presented thorough analysis of the approach and showed that the method can achieve slightly worse regret bound but with unknown dynamics, which is a great result.\n\n3. The authors performed simulated experiments that show the method can outperform prior methods such as $H_{\\infty}$ and LQR control methods in linear control.\n\nWeaknesses:\n\n1. The experiments done in the paper are fairly simple and done in toy environments. It is unclear if the proposed method can work well in more complex and realistic environment. The method is motivated by working well in uncertainties and disturbance. Therefore, it is even more important to show some experiments in realistic tasks that can demonstrate the effectiveness and practicality of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel approach for designing reference tracking controllers for linear systems under uncertainties and adversarial disturbances.  The paper tackles the challenges not addressed by prior works, including unknown system and exosystem dynamics, and adversarial disturbances that are beyond bounds or probability distributions. The paper also introduces a (disturbance, reference)-action control policy that leverages past disturbances and reference values, enabling adaptation to adversarial settings.The proposed algorithm first estimates unknown dynamics and uncertainties, then learns a parameterized memory-augmented controller, optimizing a convex cost function while accounting for identification errors, unknown exosystem dynamics, and adversarial disturbances. The paper proves a regret bound of $O(T^\\frac{2}{3}))$ for the proposed algorithm, demonstrating its effectiveness compared to the $O(\\sqrt{T})$ bound achieved by previous approaches with known system dynamics. The authors show simulation results that compare the proposed controller to $H_{\\infty}$ and LQR control, demonstrating its superior performance in dealing with uncertainties and adversarial conditions. This work offers contributions to the field of control theory by enabling robust tracking control under challenging real-world uncertainties and adversarial influences.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please provide empirical evidence in more complex and realistic environment and tasks.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jMt1hCIiIf", "reviewer": "Reviewer_FBmt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. This paper analyzes the regret bound thoroughly for the presented algorithm\n2. This paper compares the presented algorithm against the typical control algorithms.\n\nWeakness:\n1. It seems that the work is a trivial extension from [Yaghmaie & Modares (2023)].\nIn Remark 1 in the paper [Yaghmaie & Modares (2023)]; it is mentioned that it is a trivial extension to the case where the system is unknown.  And indeed this seems the trivial combination of the algorithm 2 of [Hazen et al (2020)] and [Yaghmaie & Modares (2023)].\n2. The main technical novelty for proving the bound is not really summarized in the main body of the paper; making it hard to see the novelty beyond the trivial combination of the above.\n3. Figures and the legends are strange in experimental section.\n4. It is not really well explained how the initial stabilizing controller gain K is obtained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work relaxes the assumption that the dynamical system is known which was made in the work [Yaghmaie & Modares (2023)] by using the \"exploit-then-commit\" type algorithm combining the algorithms of [Hazen et al (2020)] and [Yaghmaie & Modares (2023)].", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I will list some major/minor concerns here.\n\nMajor:\n1. Definition 1 is a bit vague; what do you mean by “This can also be represented as…”.  Is it a part of the definition or some derived properties?\n2. In Remark 1 in the paper [Yaghmaie & Modares (2023)]; it is mentioned that it is a trivial extension to the case where the system is unknown.  And indeed this seems the trivial combinations of the algorithm 2 of [Hazen et al (2020)] and the mentioned work.\n3. The stabilizing controller gain K is known?  And K in Algorithm 2 seems strangely typed.\n4. Figures and the legends are strange.  Please refine the experimental sections carefully.\n5. Please describe briefly what the technical novelty for proving the bound in the main body.\n\nMinor:\n1. page 1 after “an instantaneous cost function” maybe better to wrap references; and others\n2. page 1 Despite this advantage, existing RL-based control…  is it for continuous state/control case?\n3. page 2 “the presented online learning…” the must be The.\n4. page 2 “Estimating this expected cost, however, is only computationally tractable if the noise is Gaussian” isn’t it too strong statement?\n5. page 2 “sqrt(O(T)) → O(sqrt(T))”?\n6. Related work: may need “:” after paragraph\n7. LQR controller may provide some guarantee but it hasn’t yet provided it for non-Gaussian cases.\n8. Definition 2: G_lambda is full-row rank, and … → G_lambda defined by … where … is full-row rank?  … is also → is said to be… ?  Perhaps move the sentence “controllability index…” out of the definition.\n9. What is “identified dynamical system”?  Better to rephrase it or define it properly.\n10. Define Lhat.\n11. Memory augmented policy has been used in the literature; please cite some of them before presenting the definition.\n12. what do you mean by “general convex function” in page 6\n13. page 6 what is “:” between truncated control and cost f?\n14. L2 norm it is mathecal in some part and not in some part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zqqbs9AS6E", "reviewer": "Reviewer_Mjcu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ For the most part, the main problem formulation and algorithmic contribution are clearly motivated and stated.\n  + I did get somewhat lost trying to understand Section 3.3, which seems to heavily rely on Yaghmaie & Modares (2023).\n  + Problem 1 could be easier to understand with one more level of notation/equations. The last sentence says \"design a memory-augmented control policy in the form of eq 9 to optimize eq 7\" --- it could be nice to also write out the min/argmin equation to make it extremely concrete and easy to see the optimization problem of interest.\n+ The experimental evaluation is clearly described and the baselines of C.E. LQR, and C.E. $H_\\infty$ control are reasonable. The proposed methodology and this experiment seem to back up the claims and evidence, which is the acceptance criteria for TMLR.\n\nWeaknesses\n+ As stated in the introduction, the paper reads as a relatively straightforward combination of the optimal tracking of [Yaghmaie and Modares](https://openreview.net/forum?id=5nVJlKgmxp) with the non-stochastic control problem in [Hazan et al.](https://arxiv.org/abs/1911.12178).  Algorithm 1 of the submitted paper closely follows the structure of [Algorithm 1 in Hazan et al.](https://arxiv.org/abs/1911.12178) of performing system identification with random inputs and then solving the online control problem. The difference is that Hazan et. al don't perform reference tracking and consider a more general control cost while this paper uses the reference tracking using [Algorithm 1 in Yaghmaie and Modares](https://openreview.net/forum?id=5nVJlKgmxp). Experimentally, I do not completely understand the difficulty/novelty in combining these approaches, but the paper seems like it would be a good reference for anybody wanting to do this.\n+ Even though the system identification portion using random perturbations was taken from Hazan et al., it seems important to justify or briefly say when that works. Especially if the approach is generalized to non-linear systems, there are many systems where random perturbations using control values of -1 and 1 would not be able to adequately recover the dynamics. Maybe this is ok for linear systems and there are results explaining why perturbations using control values of just -1 and 1 work and how fast that can approach the true system.\n+ The formulation of the paper throughout the introduction and in problem 1 state that the method is robust to adversarial perturbations. However, experimentally, adversarial disturbances are not investigated --- Section 5.2 only lists uniformly sampled, constant, amplitude modulation, sinusoidal, Gaussian, and random walk disturbances. While these go beyond the usual Gaussian perturbance assumptions, they still don't involve an adversary. I have the same comment about the settings from [Yaghmaie and Modares](https://openreview.net/forum?id=5nVJlKgmxp) which also do not experimentally consider adversarial disturbances.\n+ The experimental evaluation in setting 5.1 is for a very specific 2D linear dynamical system with very specific values set for the reference system. While it is a reasonable demonstration of the setting, I would be much more convinced by a system and reference tracking problem directly taken from a real-world dynamical system and application. The would be the most interesting for real non-linear systems (e.g., a vehicular, aerospace, or other robotic/mechanical system), so I acknowledge that it's out-of-scope for the submitted paper's contribution on linear systems.\n\nMinor\n+ In `Algorithm SysId`, $\\mathbb{K}$ is used as the linear controller, but it should just be $K$ from definition 1? The other parts of the SysId algorithm block are also not very well-described in comparison to Algorithm 1 which briefly describes the inputs/initialization with text", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies a method for online reference tracking for a system with unknown dynamics and disturbances. The first few sections of the paper provide preliminary and background information on tracking problems, building up to the main problem formulation of the paper in Problem 1. The proposed method in Algorithm 1 has two steps: 1) system identification by sending binary inputs into the system, followed by 2) tracking the error approximations and solving the state-tracking problem online.  Theorem 2 analyzes the regret of this method and Section 5 experimentally evaluates the method on the system in eq (22) under various disturbances.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not have any", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree, and I concur, that the paper contributes an important and thoroughly analyzed and evaluated method for trajectory tracking of a linear system under disturbances. The contributed algorithm is a collection of the dispersed previous ideas -- nevertheless the paper offer a solid contribution of a not well-explored problem. The authors successfully addressed all the reviewers questions.

Optionally, I would recommend that the authors look into [1] as an example for a work that addresses online compensation for unknown stochastic disturbances on control affine systems with the unknown dynamics, for a goal-conditioned navigation. 

[1] A. Faust, N. Malone and L. Tapia, "Preference-balancing motion planning under stochastic disturbances," 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA, 2015, pp. 3555-3562, doi: 10.1109/ICRA.2015.7139692.
Decision: Accept (poster)