Year: 2021

Abstract:
We develop a learning-based algorithm for the control of autonomous systems governed by unknown, nonlinear dynamics to satisfy user-specified tasks expressed via time-varying reference trajectories. Most existing algorithms either assume certain parametric forms for the unknown dynamic terms or resort to unnecessarily large control inputs in order to provide theoretical guarantees. The proposed algorithm addresses these drawbacks by integrating neural-network-based learning with adaptive control. More specifically, the algorithm learns a controller, represented as a neural network, using training data that correspond to a collection of system parameters and tasks. These parameters and tasks are derived by varying the nominal parameters and the reference trajectories, respectively. It then incorporates this neural network into an online closed-form adaptive control policy in such a way that the resulting behavior satisfies the user-defined task. The proposed algorithm does not use any a priori information on the unknown dynamic terms or any approximation schemes. We provide formal theoretical guarantees on the satisfaction of the task. Numerical experiments on a robotic manipulator and a unicycle robot demonstrate that the proposed algorithm guarantees the satisfaction of 50 user-defined tasks, and outperforms control policies that do not employ online adaptation or the neuralnetwork controller. Finally, we show that the proposed algorithm achieves greater performance than standard reinforcement-learning algorithms in the pendulum benchmarking environment.

heading: 1 INTRODUCTION
text: Learning and control of autonomous systems with uncertain dynamics is a critical and challenging topic that has been widely studied during the last decades. One can identify plenty of motivating reasons, ranging from uncertain geometrical or dynamical parameters and unknown exogenous disturbances to abrupt faults that significantly modify the dynamics. There has been, therefore, an increasing need for developing control algorithms that do not rely on the underlying system dynamics. At the same time, such algorithms can be easily implemented on different, heterogeneous systems, since one does not need to be occupied with the tedious computation of the dynamic terms.
A promising step towards the control of systems with uncertain dynamics is the use of data obtained a priori from system runs. However, engineering systems often undergo purposeful modifications (e.g., substitution of a motor or link in a robotic arm or exposure to new working environments) or suffer gradual faults (e.g., mechanical degradation), which might change the systems’ dynamics or operating conditions. Therefore, one cannot rely on the aforementioned data to provably guarantee the successful control of the system. On the other hand, the exact incorporation of these changes in the dynamic model, and consequently, the design of new model-based algorithms, can be a challenging and often impossible procedure. Hence, the goal in such cases is to exploit the data obtained a priori and construct intelligent online policies that achieve a user-defined task while adapting to the aforementioned changes.
There has been a large variety of works that tackle the problem of control of autonomous systems with uncertain dynamics, exhibiting, however, certain limitations. The existing algorithms are based on adaptive and learning-based approaches or the so-called funnel control (Krstic et al. (1995); Kanellakopoulos et al. (1991); Xu & Ioannou (2003); Ge & Wang (2004); Wen et al. (2011); Chen et al. (2011); Slotine & Li (1987); Slotine & Coetsee (1986); Chen et al. (2008); Xian et al. (2004); Vamvoudakis & Lewis (2010); Berger et al. (2018); Bechlioulis & Rovithakis (2014); Joshi et al. (2020); Capotondi et al. (2020); Bertsekas & Tsitsiklis (1996); Sutton & Barto (2018)). Nevertheless,
adaptive control methodologies are restricted to system dynamics that can be linearly parameterized with respect to certain unknown parameters (e.g., masses, moments of inertia), assuming the system structure perfectly known; funnel controllers employ reciprocal terms that drive the control input to infinity when the tracking error approaches a pre-specified funnel function, creating thus unnecessarily large control inputs that might damage the system actuators. Data-based learning approaches either consider some system characteristic known (e.g., a nominal model, Lipschitz constants, or global bounds), or use neural networks to learn a tracking controller or the system dynamics; the correctness of such methods, however, relies on strong assumptions on the parametric approximation by the neural network and knowledge of the underlying radial basis functions. Finally, standard reinforcementlearning techniques (Bertsekas & Tsitsiklis (1996); Sutton & Barto (2018)) usually assume certain state and/or time discretizations of the system and rely on exhaustive search of the state space, which might lead to undesirable transient properties (e.g., collision with obstacles while learning).

heading: 1.1 CONTRIBUTIONS AND SIGNIFICANCE
text: This paper addresses the control of systems with continuous, unknown nonlinear dynamics subject to tasks expressed via time-varying reference trajectories. Our main contribution lies in the development of a learning-based control algorithm that guarantees the accomplishment of a given task using only mild assumptions on the system dynamics. The algorithm draws a novel connection between adaptive control and learning with neural network representations, and consists of the following steps. Firstly, it trains a neural network that aims to learn a controller that accomplishes a given task from data obtained off-line. Secondly, we develop an online adaptive feedback control policy that uses the trained network to guarantee convergence to the given reference trajectory and hence satisfaction of the task. Essentially, our approach builds on a combination of off-line trained controllers and on-line adaptations, which was recently shown to significantly enhance performance with respect to single use of the off-line part (Bertsekas (2021)).
The major significance of our contribution is twofold. Firstly, we guarantee the theoretical correctness of the proposed algorithm by considering only mild conditions on the neural network, removing the long-standing assumptions on parametric approximations and boundedness of the estimation error. Secondly, we demonstrate via the experimental results the generality of the algorithm with respect to different tasks and system parameters. That is, the training data that we generate for the training of the neural network in the first step correspond to tasks that are different from the given one to be executed1. Additionally, we employ systems with different dynamic parameters to generate these data. We evaluate the proposed algorithm in numerous scenarios comprising different variations of the given task and different system dynamic parameters, which do not necessarily match the training data. We show that the algorithm, owing to its adaptation properties, is able to guarantee the satisfaction of the respective tasks in all the aforementioned scenarios by using the same neural network.

heading: 1.2 RELATED WORK
text: A large variety of previous works considers neuro-adaptive control with stability guarantees, focusing on the optimal control problem (Vamvoudakis & Lewis (2010); Yang et al. (2020); Cheng et al. (2007); Fan et al. (2018); Kiumarsi et al. (2017); Zhao & Gan (2020); Vrabie & Lewis (2009); Sun & Vamvoudakis (2020); Kamalapurkar et al. (2015); Huang et al. (2018b); Mo et al. (2019); Joshi et al. (2020)). Nevertheless, the related works draw motivation from the neural network density property (see, e.g., (Cybenko (1989)))2 and assume sufficiently small approximation errors and linear parameterizations of the unknown terms of the form W (x)θ (dynamics, optimal controllers, or value functions), with W and θ known and unknown, respectively. Similarly, more traditional adaptive control methodologies that handle uncertain nonlinear systems assume either linear parameterizations of the unknown dynamic terms (Krstic et al. (1995); Hong et al. (2009); Chen (2019); Huang et al. (2018a); Kanellakopoulos et al. (1991); Xu & Ioannou (2003); Ge & Wang (2004); Wen et al. (2011); Chen et al. (2011); Slotine & Li (1987); Slotine & Coetsee (1986); Chen et al. (2008)), use known upper-bound functions (Slotine & Coetsee (1986)), or provide local stability results dictated by the dynamic bounds (Xian et al. (2004); Chen et al. (2008)). This paper relaxes the aforementioned assumptions and proposes a non-parametric neuro-adaptive controller, whose stability guarantees
1The task difference is illustrated in Section 4. 2A sufficiently large neural network can approximate a continuous function arbitrarily well in a compact set.
rely on a mild boundedness condition of the closed-loop system state that is driven by the learned controller. The proposed approach exhibits similarities with (Liu et al. (1994)), which employs off-line-trained neural networks with online feedback control, but fails to provide convergence guarantees. Similarly, Zeng et al. (2020) develops an algorithm that combines a learning module and a physics-based controller for robot throwing of arbitrary objects, assuming however access to a physics simulator and not providing theoretical guarantees.
Other learning-based related works include modeling with Gaussian processes (Capotondi et al. (2020); Leahy et al. (2019); Jain et al. (2018); Berkenkamp & Schoellig (2015)), or use neural networks (Ma et al. (2020); Shah et al. (2018); Yan & Julius (2021); Liu et al. (2021); Hahn et al. (2020); Cai et al. (2021); Wang et al. (2020); Camacho & McIlraith (2019); Hahn et al. (2020); Riegel et al. (2020); Hu et al. (2020); Ivanov et al. (2019)) to accomplish reachability, verification or temporal logic specifications. Nevertheless, the aforementioned works either use partial information on the underlying system dynamics, or do not consider them at all. In addition, works based on Gaussian processes usually propagate the dynamic uncertainties, possibly leading to conservative results. Similarly, data-driven model-predictive control techniques (Nubert et al. (2020); Maddalena et al. (2020)) use data to over-approximate additive disturbances or are restricted to linear systems.
Control of unknown nonlinear continuous-time systems has been also tackled in the literature by using funnel control, without necessarily using off-line data or dynamic approximations (Berger et al. (2018); Bechlioulis & Rovithakis (2014); Lindemann et al. (2017); Verginis & Dimarogonas (2018); Verginis et al. (2021)). Nevertheless, funnel controllers usually depend on reciprocal time-varying barrier functions that drive the control input to infinity when the error approaches a pre-specified funnel, creating thus unnecessarily large control inputs that might damage the system actuators.

heading: 2 PROBLEM FORMULATION
text: Consider a continuous-time dynamical system governed by the 2nd-order continuous-time dynamics
ẍ = f(x̄, t) + g(x̄, t)u(x̄, t), (1)
where x̄ := [x>, ẋ>]> ∈ R2n, n ∈ N, is the system state, assumed available for measurement,and u : R2n × [0,∞) → Rn is the time-varying feedback-control input. The terms f(·) and g(·) are nonlinear vector fields that are locally Lipschitz in x̄ over R2n for each fixed t ≥ 0, and uniformly bounded in t over [0,∞) for each fixed x̄ ∈ R2n. The dynamics (1) comprise a large class of nonlinear dynamical systems (Zhong & Leonard (2020); Yu et al. (1996); Doya (1997)) that capture contemporary engineering problems in mechanical, electromechanical and power electronics applications, such as rigid/flexible robots, induction motors and DC-to-DC converters, to name a few. The continuity in time and state provides a direct link to the actual underlying system, and we further do not require any time or state discretizations.
We consider that f(·) and g(·) are completely unknown; we do not assume any knowledge of the structure, Lipschitz constants, or bounds, and we do not use any scheme to approximate them. Note also that we do not assume global Lipschitz continuity or global boundedness of f(·, t) and g(·, t) or the solution x̄(t) of (1). Nevertheless, we do assume that g(x̄, t) is positive definite:
Assumption 1. The matrix g(x̄, t) is positive definite, for all (x̄, t) ∈ R2n × [0,∞).
Such assumption is a sufficiently controllability condition for (1); intuitively, it states that the multiplier of u (the input matrix) does not change the direction imposed to the system by the underlying control algorithm. Systems not covered by (1) or Assumption 1 consist of underactuated or non-holonomic systems, such as unicycle robots or underactuated aerial vehicles. Nevertheless, we provide an extension of our results for a non-holonomic unicycle vehicle in Appendix B. Moreover, the 2nd-order model (1) can be easily extended to account for higher-order integrator systems (Slotine et al. (1991)).
Consider now a time-constrained task expressed as a time-varying reference trajectory pd : R≥0 → Rn. The objective of this paper is to construct a time-varying feedback-control algorithm u(x̄, t) such that the state of the closed-loop system (1) asymptotically tracks pd, i.e., limt→∞(x(t)− pd(t)) = 0.

heading: 3 MAIN RESULTS
text: This section describes the proposed algorithm, which consists of two steps. Firstly, it learns a controller, represented as a neural network, using training data that correspond to a collection of different tasks and system parameters. Secondly, we design an adaptive, time-varying feedback controller that uses the neural-network approximation and guarantees tracking of the reference trajectory, consequently achieving satisfaction of the task.

heading: 3.1 NEURAL-NETWORK LEARNING
text: As discussed in Section 1, we are inspired by cases where systems undergo changes that modify their dynamics and hence the underlying controllers no longer guarantee the satisfaction of a specific task. In such cases, instead of carrying out the challenging and tedious procedure of identification of the new dynamic models and design of new model-based controllers, we aim to exploit data from off-line system trajectories and develop an online policy that is able to adapt to the aforementioned changes and achieve the task expressed via pd. Consequently, we assume the existence of offline data from a finite set of T system trajectories that satisfy a collection of tasks, corresponding to bounded reference trajectories, including pd, and possibly produced by systems with different dynamic parameters. The data from each trajectory i ∈ {1, . . . , T} comprise a finite set of triplets {x̄s(t), t, us(t)}t∈Ti , where Ti is a finite set of time instants, x̄s(t) ∈ R2n are system states, and us(t) ∈ Rn are the respective control inputs, compliant with the dynamics (1). We use the data to train a neural network in order to approximate the respective controller u(x̄, t). More specifically, we use the pairs (x̄s(t), t)t∈Ti as input to a neural network, and us(t)t∈Ti as the respective output targets, for all trajectories i ∈ {1, . . . , T}. For given x̄ ∈ R2n, t ∈ R≥0, we denote by unn(x̄, t) the output of the neural network. Note that the controller u(x̄, t), which the neural network aims to approximate, is not associated to the specific task expressed via pd and mentioned in Section 2, but a collection of several tasks. Therefore, we do not expect the neural network to learn how to track pd, but rather to be able to adapt to the entire collection of tasks. This is an important attribute of the proposed scheme, since it can generalize over tasks. The motivation for training the neural network with different tasks and dynamic parameters is the following. Since the tasks correspond to bounded trajectories, the respective stabilizing controllers compensate successfully the dynamics in (1). Therefore, as will be clarified in the next section, the neural network aims to approximate an “average” controller the retains this property, i.e., the boundedness of the dynamics of (1). By using such approximation, the online feedback-control policy - illustrated in the next section - is able to guarantee tracking of pd without using any explicit information on the dynamics.

heading: 3.2 FEEDBACK CONTROL DESIGN
text: As mentioned in Section 3.1, we do not expect the neural-network controller to accomplish tracking of pd, since the system (1) is trained on potentially different tasks and different system parameters, and (2) the neural network provides only an approximation of a stabilizing controller; potential deviations in certain regions of the state space might lead to instability. Moreover, the neural-network controller has no error feedback with respect to the open-loop trajectory pd; such feedback is substantial in the stability of control systems with dynamic uncertainties. Therefore, this section is devoted to the design of a feedback-control policy to track the trajectory pd(t) by using the output of the trained neural network (see Fig. 1). The goal is to drive the error e := x − pd to zero. As mentioned in Section 3.1, the motivation for training the neural network with several different tasks and dynamic parameters is the learning of a controller that is able to retain the property of the training controllers to compensate the system dynamics (1). This is officially stated in the following assumption regarding the closed-loop system trajectory that is driven by the neural network’s output. Assumption 2. The output unn(x̄, t) of the trained neural network satisfies
‖f(x̄, t) + g(x̄, t)unn(x̄, t)‖ ≤ d‖x̄‖+B (2)
for positive constants d, B, for all x̄ ∈ R2n, t ≥ 0.
Intuitively, Assumption 2 states that the neural-network controller unn(x̄, t) is able to maintain the boundedness of the system state by the constants d, B, which are considered to be unknown. The assumption is motivated by the property of neural networks to approximate a continuous function
arbitrarily well in a compact domain for a large enough number of neurons and layers (Cybenko (1989))3. As mentioned before, since the collection of tasks, which the neural network is trained with, correspond to bounded trajectories, the system states are expected to remain bounded. Since f(x̄, t), and g(x̄, t) are continuous in x̄ and bounded in t, they are also expected to be bounded as per (2). Contrary to the related works (e.g., (Vamvoudakis & Lewis (2010); Yang et al. (2020); Cheng et al. (2007); Fan et al. (2018))), however, we do not adopt approximation schemes for the system dynamics and we do not impose restrictions on the size of d, B. Moreover, Assumption 2 does not imply that the neural network controller unn(x̄, t) guarantees tracking of the open-loop trajectory pd. Instead, it is merely a growth condition. Additionally, note that inequality 2 does not depend specifically on any of the tasks that the neural network is trained with. We exploit this property in the control design and achieve task generalization; that is, the open-loop trajectory pd to be tracked (corresponding to the task ϕ) can be any of the tasks that the neural network is trained with.
We now define the feedback-control policy. Consider the adaptation variables ˆ̀1, ˆ̀2, corresponding to upper bounds of d, B in (2), with ˆ̀1(0) > 0, ˆ̀2(0) > 0. We design first a reference signal for ẋ as
vd := ṗd − k1e, (3)
that would stabilize the subsystem ‖e‖2, where k1 is a positive control gain constant. Following the back-stepping methodology (Krstic et al. (1995)), we define next the respective error ev := ẋ− vd and design the neural-network-based adaptive control law as
u(x̄, ˆ̀1, ˆ̀2, t) = unn(x̄, t)− k2ev − ˆ̀1ev − ˆ̀2êv (4a) ˙̂ `1 = k`1‖ev‖2, ˙̂ `2 = k`2‖ev‖ (4b)
where k2, k`1 , k`2 are positive constants, and êv = ev ‖ev‖ if ev 6= 0, and êv = 0 if ev = 0. The control design is inspired by adaptive control methodologies (Krstic et al. (1995)), where the time-varying gains ˆ̀1(t), ˆ̀2(t), adapt to the unknown dynamics and counteract the effect of d and B in (2) in order to ensure closed-loop stability. Note that the policy (3), (4) does not use any information on the system dynamics f(·), g(·) or the constants B, d. The tracking of pd is guaranteed by the following theorem. Theorem 1. Let a system evolve according to (1) and let an open-loop trajectory pd : R≥0 → R6 encoding a user-defined task. Under Assumption 2, the control algorithm (4) guarantees limt→∞(e(t), ev(t)) = 0, as well as the boundedness of all closed-loop signals.
We provide a sketch of the proof here and give the details in Appendix A.
Proof (Sketch). In view of (2), one can find positive constants d1, d2, D such that 1
g ‖ge+ f(x̄, t) + g(x̄, t)unn(x̄, t)− v̇d‖ ≤ d1‖e‖+ d2‖ev‖+D,
3For simplicity, we consider that (2) holds globally, but it can be extended to hold in a compact set.
for all x̄ ∈ R2n, t ≥ 0, where g is the minimum eigenvalue of g(x̄, t), which is positive owing to the positive definitiveness of g(·). The constants d1, d2, and D define upper bounds `1 and `2, which we aim to approximate by the adaptation variables ˆ̀1 and ˆ̀2, respectively. We next define the continuously differentiable function
V (x̃) := 1 2 ‖e‖2 + 1 2g ‖ev‖2 + ∑ i∈{1,2} 1 2k`i (ˆ̀i − `i)2
which represents an energy-like function of the system, and which we wish to drive to zero. One can prove, by analyzing the derivative V̇ and using the control policy (4), that V is non-decreasing along the solutions of the closed-loop system and that limt→∞ V̇ (t) = 0, which implies that limt→∞ e(t) = limt→∞ ev(t) = 0.
Note that, contrary to works in the related literature (e.g., (Verginis & Dimarogonas (2020); Bechlioulis & Rovithakis (2014))), we do not impose reciprocal terms in the control input that grow unbounded in order to guarantee closed-loop stability. The resulting controller is essentially a simple linear feedback on (e(t), ev(t)) with time-varying adaptive control gains, accompanied by the neural network output that ensures the growth condition (2). The positive gains k1, k2, k`1 , k`2 do not affect the stability results of Theorem 1, but might affect the evolution of the closed-loop system; e.g., larger gains lead to faster convergence but possibly larger control inputs.
The proposed control algorithm does not require any of the long-standing assumptions on the system dynamics (1), such as linear parameterization, growth conditions, or boundedness by known functions (e.g., Krstic et al. (1995); Hong et al. (2009); Chen (2019); Huang et al. (2018a); Kanellakopoulos et al. (1991); Xu & Ioannou (2003); Ge & Wang (2004); Wen et al. (2011); Chen et al. (2011); Slotine & Li (1987); Slotine & Coetsee (1986); Chen et al. (2008); Slotine & Coetsee (1986); Chen et al. (2008); Xian et al. (2004)). Additionally, we do not assume the boundedness of the solution of (1) or of the dynamic terms f(·, t), g(·, t); instead, the control algorithm guarantees via Theorem 1 the boundedness of the system state as well as the asymptotic tracking of pd(t). The only boundedness condition that we require is (2) in Assumption 2, which can be accomplished by neural-network component in view of the universal approximation property Cybenko (1989).

heading: 4 NUMERICAL EXPERIMENTS
text: This section is devoted to a series of numerical experiments. More details can be found in Appendix B. We first test the proposed algorithm on a 6-dof UR5 robotic manipulator with dynamics
ẍ = B(x)−1 (u− C(x̄)ẋ− g(x) + d(x̄, t)) (5)
where x, ẋ ∈ R6 are the vectors of robot joint angles and angular velocities, respectively; B(x) ∈ R6×6 is the positive definite inertia matrix, C(x̄) ∈ R6×6 is the Coriolis matrix, g(x) ∈ R6 is the gravity vector, and d(x̄, t) ∈ R6 is a vector of friction terms and exogenous time-varying disturbances.
The workspace consists of four points of interest T1, T2, T3, T4 (end-effector position and Euler-angle orientation), as depicted in Fig. 2, which correspond to the joint-angle vectors c1, c2, c3, c4. More information is provided in Appendix A. We consider a nominal task expressed via the spatio-temporal constraint ∧ i∈{1,...,4}G[0,∞)FIi(‖x1 − ci‖ ≤ 0.1), where G and F are the always and eventually operators respectively. The task consists of visits of x1 to ci ∈ R6 (within the radius 0.1) infinitely often within the time intervals dictated by Ii, for i ∈ {1, . . . , 4}. We create 150 problem instances by varying the positions of ci, the time intervals Ii, the dynamic parameters of the robot (masses and moments of inertia of the robot’s links and actuators), the friction and disturbance term d(·), the initial position and velocity of the robot, and the sequence of visits to the points ci, as dictated by φ, i.e., one instance
might correspond to the visit sequence ((x(0), 0) → (c1, t11) → (c2, t12) → (c3, t13) → (c4, t14), and another to ((x(0), 0) → (c3, t13) → (c1, t11) → (c4, t14) → (c2, t12). We separate the aforementioned 150 problem instances into 100 training instances and 50 test instances. We generate trajectories using the 100 training instances from system runs that satisfy different variations of one cycle of the task (i.e., one visit to each point). Each trajectory consists of 500 points and is generated using a nominal model-based controller. We use these trajectories to train a neural network and we test the control policy (4) in the 50 test instances. We also compare our algorithm with the non-adaptive controller uc(x̄, t) = unn(x, t)− k1e− k2ė, as well as with a modified version ud(x̄, t) of (4) that does not employ the neural network (i.e., the term unn(x̄, t)). The comparison results are depicted in Fig. 3, which depicts the mean and standard deviation of the signal ‖e(t)‖+ ‖ė(t)‖ for the 50 instances and 20 seconds. It is clear from the figure that the proposed algorithm performs better than the non-adaptive and no-neural-network policies both in terms of convergence speed and steady-state error. It is worth noting that the non-adaptive policy results on average in unstable closed-loop system.
We next test the proposed algorithm, following a similar procedure, on a unicycle robot (the respective dynamics and control algorithm can be found in Appendix B). Fig. 5 depicts the mean and standard deviation of the errors ed(t), eβ(t) for 50 test instances. We note that the performance of the noneural-network control policy is much more similar to the proposed one than in the UR5 case. This can be attributed to (1) the lack of gravitational terms in the unicycle dynamics, which often lead to instability, and (2) the chosen control gains of the no-neural-network policy, which are sufficiently large to counteract the effect of the dynamic uncertainties.
Finally, we compare the performance of the proposed control policy with the reported data of (Wang et al. (2019)) on the benchmarking enivronment of the pendulum, where a single-link mechanical structure aims to reach the upright position. Following (Wang et al. (2019)), the reward and costs are rpend(t) = − cos q(t) − 0.1 sin q(t) − 0.1q̇(t) − 0.001u(t)2 and Jpend = ∑H t=1 γ trpend(t), respectively.
Similar to the previous cases and in contrast to Wang et al. (2019), we generate 150 instances by varying the system parameters (pendulum length and mass), the external disturbances, and the initial conditions. We generate 100 trajectories, consisting of 500 points each, for the 100 training instances, by employing a nominal controller, and we use them train a neural network. Moreover, in order to guarantee the feasibility of the proposed algorithm (4) we consider larger control-action bounds than in (Wang et al. (2019)). The original bounds render these systems under-actuated, which is not included in the considered class of systems (1) and consist part of our future work. It should be noted that such larger bounds affect negatively the acquired rewards. We test the control policy (4) on the 50 test instances, for 5000 steps each, corresponding to 10 seconds. We set H = 200 and γ = 1 (Wang et al. (2019)). Fig. 4 depicts the
mean and standard deviation of the time-varying reward and cost functions, illustrating successful regulation to the upright position. After 5000 steps, we obtain a mean reward and cost of 0.99 and 200, respectively, showing better performance than the reported cost of 180 in (Wang et al. (2019)). Moreover, the proposed control algorithm achieves this performance without resorting to exhaustive exploration of the state-space, which is the case in the reinforcement-learning algorithms used in (Wang et al. (2019)). This is a very important property in practical engineering systems, where safety is of paramount significance and certain areas of the state space must be avoided.

heading: 5 DISCUSSION AND LIMITATIONS
text: As shown in the experimental results, the control algorithm is able to asymptotically track reference trajectories which were not considered when generating the training data. Similarly, the trajectories used in the training data were generated using systems with different dynamical parameters, and not specifically the ones used in the tests. The aforementioned attributes signify the ability of the proposed algorithm to generalize to different tasks and systems with different parameters. Nevertheless, the proposed control policy is currently limited by systems satisfying Assumption 1 and is not able to take into account under-actuated systems (e.g., the acrobot or cart-pole system); such systems consist part of our future work. Moreover, the proposed algorithm is dependent on the performance of the learning component to satisfy Assumption 2. Lack of satisfactory training might cause the neural-network output to add extra terms in the dynamics that render the closed-loop system unstable. A less conservative and more robust approach consists of using the off-line data to train multiple neural networks, each for a certain region of the state space; such an approach constitutes part of our future work. Finally, the discontinuities of (4), (12) might be problematic and create chattering when implemented in real actuators. A continuous approximation that has shown to yield satisfying performance is the boundary-layer technique (Slotine et al. (1991)).

heading: 6 CONCLUSION AND FUTURE WORK
text: We develop a novel control algorithm for the control of robotic systems with unknown nonlinear dynamics subject to tasks expressed as reference trajectories. The algorithm integrates neural networkbased learning and adaptive control. We provide formal guarantees and perform extensive numerical
experiments. Future directions will focus on relaxing the considered assumptions and extending the proposed methodology to underactuated systems.

heading: 7 REPRODUCIBILITY STATEMENT
text: We provide the proof of Theorem 1 in Appendix A. Additionally, we elaborate on the required assumptions (Assumptions 1, 2, 3) throughout the text - see pages 3, 5, and 7. Finally, we provide implementation details, instructions, and the respective code, required to reproduce the results, in Appendix C and the supplementary material.

heading: A APPENDIX
text: We provide here the proof of Theorem 1. We first give some preliminary notation and background on systems with discontinuous dynamics.
NOTATION
Given a function f : Rn → Rk, its Filippov regularization is defined as (Paden & Sastry (1987)) K[f ](x) := ⋂ δ>0 ⋂ µ(N̄)=0 co(f(B(x, δ)\N̄), t), (6)
where ⋂ µ(N̄)=0 is the intersection over all sets N̄ of Lebesgue measure zero, co(E) is the closure of the convex hull co(E) of the set E, and B(x, δ) is the ball with radius δ centered at x.
NONSMOOTH ANALYSIS
Consider the following differential equation with a discontinuous right-hand side:
ẋ = f(x, t), (7)
where f : D × [t0,∞)→ Rn, D ⊂ Rn, is Lebesgue measurable and locally essentially bounded. Definition 1 (Def. 1 of (Fischer et al. (2013))). A function x : [t0, t1)→ Rn, with t1 > t0, is called a Filippov solution of (7) on [t0, t1) if x(t) is absolutely continuous and if, for almost all t ∈ [t0, t1), it satisfies ẋ ∈ K[f ](x, t), where K[f ](x, t) is the Filippov regularization of f(x, t). Lemma 1 (Lemma 1 of (Fischer et al. (2013))). Let x(t) be a Filippov solution of (7) and V : D × [t0, t1)→ R be a locally Lipschitz, regular function4. Then V (x(t), t) is absolutely continuous, V̇ (x(t), t) = ∂∂tV (x(t), t) exists almost everywhere (a.e.), i.e., for almost all t ∈ [t0, t1), and V̇ (x(t), t) a.e ∈ ˙̃V (x(t), t), where
˙̃ V := ⋂ ξ∈∂V (x,t) ξ> [ K[f ](x, t) 1 ] ,
and ∂V (x, t) is Clarke’s generalized gradient at (x, t) (Fischer et al. (2013)).
Corollary 1 (Corollary 2 of (Fischer et al. (2013))). For the system given in (7), let D ⊂ Rn be an open and connected set containing x = 0 and suppose that f is Lebesgue measurable and x 7→ f(x, t) is essentially locally bounded, uniformly in t. Let V : D × [t0, t1) → R be locally Lipschitz and regular such that W1(x) ≤ V (x, t) ≤W2(x), ∀t ∈ [t0, t1), x ∈ D, and
z ≤ −W (x(t)), ∀z ∈ ˙̃V (x(t), t), t ∈ [t0, t1), x ∈ D,
where W1 and W2 are continuous positive definite functions and W is a continuous positive semidefinite on D. Choose r > 0 and c > 0 such that B̄(0, r) ⊂ D and c < min‖x‖=rW1(x). Then for all Filippov solutions x : [t0, t1) → Rn of (7), with x(t0) ∈ D := {x ∈ B̄(0, r) : W2(x) ≤ c}, it holds that t1 =∞, x(t) ∈ D, ∀t ∈ [t0,∞), and limt→∞W (x(t)) = 0.
Proof of Theorem 1. We re-write first the condition of Assumption 2. Note that pd(t) and its derivatives are bounded functions of time. Moreover, since e = x − pd, ė = ẋ − ṗd, it holds that ‖x̄‖ ≤ ‖e‖ + ‖ė‖ + ‖pd‖ + ‖ṗd‖ and v̇d = p̈d − k1ė, as well as ev = ẋ − vd = ė + k1e implying ė = ev − k1e. Therefore, in view of (2), one can find positive constants d1, d2, D such that
1 g ‖ge+ f(x̄, t) + g(x̄, t)unn(x̄, t)− v̇d‖ ≤ d1‖e‖+ d2‖ev‖+D, (8)
for all x̄ ∈ R2n, t ≥ 0, where g is the minimum eigenvalue of g(x̄, t), which is positive owing to the positive definitiveness of g(·). Inequality (8) will be used later in the proof.
4See (Fischer et al. (2013)) for a definition of regular functions.
Let now a constant α such that d1α2 < k1. As will be clarified later, the adaptation variables ˆ̀ 1, ˆ̀ 2 aim to approximate the constants d12α + d2 and D, respectively. Therefore, let `1 := d1 2α + d2,
`2 := D, and the respective error terms ˜̀1 := ˆ̀1 − `1, ˜̀2 := ˆ̀2 − `2, as well as the overall state x̃ := [e>, e>v , ˜̀ 1, ˜̀2]> ∈ R2n+2. Since the control policy is discontinuous, we use the notion of Filippov solutions. The Filippov regularization of u is K[u] = unn(x, t)−k2ev− ˆ̀1ev− ˆ̀2Êv , where Êv :=
ev ‖ev‖ if ev 6= 0 and Êv ∈ (−1, 1) otherwise. Note that, in any case, it holds that e > v Êv = ‖ev‖.
Let now the continuously differentiable function
V (x̃) := 1 2 ‖e‖2 + 1 2g ‖ev‖2 + ∑ i∈{1,2} 1 2k`i ˜̀ i
which satisfies W1(x̃) ≤ V (x̃) ≤ W2(x̃) for W1(x̃) := min{ 12 , 1 2g , 1 2k`1 , 12k`2 }‖x̃‖2, W2(x̃) :=
max{ 12 , 1 2g , 1 2k`1 , 12k`2 }‖x̃‖2. According to Lemma 1, V̇ (x̃)
a.e. ∈ ˙̃V (x̃), with ˙̃V := ⋂ ξ∈∂V (x̃) K[
˙̃x]. Since V (x̃) is continuously differentiable, its generalized gradient reduces to the standard gradient and thus it holds that ˙̃V (x̃) = ∇V >K[ ˙̃x], where∇V = [e>, 1g e > v , 1 k`1 ˜̀ 1, 1 k`2 ˜̀ 2] >. By differentiating V and using (3), one obtains
V̇ ⊂W̃s := e>(ẋ− ṗd) + 1 g e>v ( f(x̄, t) + g(x̄, t)u− v̇d ) + 1 k`1 ˜̀ 1 ˙̂ `1 + 1 k`2 ˜̀ 2 ˙̂ `2
and by using ẋ = ev + vd and substituting the control policy (3), (4), and inequality (8),
W̃s :=− k1‖e‖2 + 1 g e>v ( ge+ f(x̄, t) + g(x̄, t)unn(x̄, t)− v̇d ) − 1 g e>v g(x̄, t) ( k2ev + ˆ̀1ev + ˆ̀2Êv ) + ˜̀1‖ev‖2 + ˜̀2‖ev‖
≤ − k1‖e‖2 + d1‖ev‖‖e‖+ d2‖ev‖2 +D‖ev‖ − 1
g e>v g(x̄, t)
( k2ev + ˆ̀1ev + ˆ̀2Êv ) + ˜̀1‖ev‖2 + ˜̀2‖ev‖
Note from (4) and the fact that ˆ̀1(0) > 0, ˆ̀2(0) > 0 that ˆ̀1(t) > 0 and ˆ̀2(t)) > 0 for all t ≥ 0. Moreover, recall that g(x̄, t) is positive definite for all x̄ ∈ R2n, t ≥ 0, with g being its minimum (and positive) eigenvalue. Therefore, we conclude that W̃s becomes
W̃s ≤− k1‖e‖2 + d1‖ev‖‖e‖+ d2‖ev‖2 +D‖ev‖ − (k2 + ˆ̀1)‖ev‖2 − ˆ̀2‖ev‖+ ˜̀1‖ev‖2 + ˜̀2‖ev‖ By incorporating the term α in the term d1‖ev‖‖e‖, we obtain d1‖ev‖‖e‖ = d1α ‖ev‖α ‖e‖ and by using the property ab ≤ 12a 2 + 12b 2 for any constants a,b, we obtain d1α ‖ev| α ‖e‖ ≤ d1α 2 ‖e‖
2 + d1 2α‖ev‖ 2. Therefore, W̃s becomes
W̃s ≤− ( k1 − d1α
2
) ‖e‖2 + ( d1 2α + d2 ) ‖ev‖2 +D‖ev‖ − (k2 + ˆ̀1)‖ev‖2 − ˆ̀2‖ev‖+ ˜̀1‖ev‖2 + ˜̀2‖ev‖
=− ( k1 − d1α
2
) ‖e‖2 + `1‖ev‖2 + `2‖ev‖ − (k2 + ˆ̀1)‖ev‖2 − ˆ̀2‖ev‖+ ˜̀1‖ev‖2 + ˜̀2‖ev‖
=− ( k1 − d1α
2
) ‖e‖2 − k2‖ev‖2 =: −Q(x̃)
Therefore, it holds that ζ ≤ −Q(x̃), for all ζ ∈ ˙̃V , with Q being a continuous and positive semi-definite function in R2n+2, since k1 − d1α2 > 0. Choose now any finite r > 0 and let c < min‖x̃‖=rW1(x̃). Hence, all the conditions of Corollary 1 are satisfied and hence, all Filippov solutions starting from x̃(0) ∈ Ωf := {x̃ ∈ B(0, r) : W̃2(x̃) ≤ c} are bounded and remain in Ωf , satisfying limt→∞Q(x̃(t)) = 0. Note that r, and hence c, can be arbitrarily large allowing and finite initial condition x̃(0). Moreover, the boundedness of x̃ implies the boundedness of e, ė, ev, ˜̀1, and˜̀ 2, and hence of ˆ̀1(t) and ˆ̀2(t), for all t ∈ R≥0. In view of (5), we finally conclude the boundendess of u(·), ˙̂`1, and ˙̂`2, for all t ∈ R≥0, leading to the conclusion of the proof.

heading: B APPENDIX
text: Extension to non-holonomic unicycle dynamics
As mentioned in Section 1, the dynamics 1 do not represent all kinds of systems, with one particular example being when non-holonomic constraints are present. In such cases, the control law design (4) and Theorem 1 no longer hold. In this section, we extend the control policy to account for unicycle vehicles subject to first-order non-holonomic constraints. More specifically, we consider the dynamics
ṗ1 = v cosφ, ṗ2 = v sinφ, φ̇ = ω (9a)
Mθ̈ = u+ fθ(x̄, t) (9b)
where x = [p1, p2, φ]> ∈ R3 are the unicycle’s position and orientation, (v, ω) are its linear and angular velocity (see Fig. 6), θ := [θR, θL]> ∈ R2 are its wheel’s angular positions, and u = [uR, uL]
> ∈ R2 are the wheel’s torques, representing the control input. The unicycle vehicle is subject to the non-holonomic constraint ṗ1 sinφ − ṗ2 cosφ = 0, which implies that the vehicle cannot move laterally. Additionally, M ∈ R2×2 is the vehicle’s inertia matrix, which is symmetric and positive definite, and fθ(·) is a function representing friction and external disturbances. The velocities satisfy the relations v = r2 (θ̇R + θ̇L), ω = r 2R (θ̇R − θ̇L)), where r and R are the wheels’ radius and axle length, respectively. The terms r, R, M , and fθ(·) are considered to be completely unknown. As before, the goal is for the vehicle’s position p := [p1, p2]> to track a desired trajectory pd = [pd,1, pd,2]> ∈ R2. Towards that end, we define the error variables e1 := p1 − pd,1, e2 := p2 − pd,2, ed := ‖p− pd‖, as well as the angle β measured from the the longitudinal axis of the vehicle, i.e., the unicycle’s direction vector [cosφ, sinφ], to the error vector −[e1, e2] (see Fig. 6). The angle β can be derived by using the cross product between the aforementioned vectors, i.e., ed sin(β) = [cosφ, sinφ] × [−e1,−e2] = e1 sinφ − e2 cosφ. The purpose of the control design, illustrated next, is to drive ed and β to zero. By differentiating the latter and using (9) as well as the relations e1 = −ed cos(φ+ β), e2 = −ed sin(φ+ β) (see Fig. 6), we derive
ėd = −v cosβ + ṗd,1 cos(φ+ β) + ṗd,2 sin(φ+ β) (10a)
β̇ = −ω + sinβ ed v − ṗd,1 ed sin(φ+ β) + ṗd,2 ed cos(φ+ β) (10b)
In view of (10), we set reference signals for the vehicle’s velocity as
vd := 1
cos(β) (ṗd,1 cos(β + φ) + ṗd,2 sin(β + φ) + kded) (11a)
ωd :=− sin(φ)ṗd,1 cos(β)ed + cos(φ)ṗd,2 cos(β)ed + kd tanβ + kββ (11b)
where kd, kβ are positive gains, aiming to create exponentially stable subsystems via the terms kded and kββ. We define next the respective velocity errors ev := v − vd, eω := ω − ωd and design the
adaptive and neural-network-based control input as u(x̄, d̂, t) := [uS+uD2 , uS−uD 2 ] >+unn(x̄, t), with
uS := ˆ̀v v̇d − (kv + ˆ̀1)ev − ˆ̀2êv + ed cosβ − β sinβ
ed (12a)
uD := ˆ̀ωω̇d − (kω + ˆ̀1)eω − ˆ̀2êω + β (12b)
˙̂ `v := −kvev v̇d, ˙̂`ω := −kωeωω̇d (12c) ˙̂ `1 := k1(e 2 v + e 2 ω) ˙̂ `2 := k2(|ev|+ |eω|) (12d)
where ˆ̀v, ˆ̀ω, ˆ̀i are adaptation variables (similar to (4)), with ˆ̀v(0) > 0, ˆ̀ω(0) > 0 and kv, kω, ki, are positive gains, i ∈ {1, 2}; êa, with a ∈ {v, ω}, is defined as êa = ea|ea| if ea 6= 0 and êa = 0 otherwise. We now re-state Assumption 2 to apply for the unicycle analysis as follows.
Assumption 3. The output unn(x̄, t) of the trained neural network satisfies ‖unn(x̄, t) + fθ(x̄, t)‖ ≤ d‖x̄‖+B, for positive, unknown constants d, B.
Similar to assumption 2, assumption 3 is merely a growth-boundedness condition by the unknown constants d and B. The stability of the proposed scheme is provided in the next corollary.
Corollary 2. Let the unicycle system (9) and let an open-loop trajectory pd : R≥0 → R2. Assume that β(t) ∈ (−β̄, β̄), |ṗd,1 sinφ− ṗd,2 cosφ| < edα1, sinβ < edα2 for positive constants β̄ < π2 , α1, α2 and all t ≥ 0. Under Assumption 3, the control policy (12) guarantees limt→∞(ed(t), β(t), ev(t), eω(t)) = 0, and the boundedness of all closed-loop signals.
Proof of Corollary 2. The derivatives of ed, β in (10) can be written as[ ėd β̇ ] = G [ v ω ] + [ ṗd,1 cos(φ+ β) + ṗd,2 sin(φ+ β) ṗd,2 ed cos(φ+ β)− ṗd,1ed sin(φ+ β) ] where
G :=
[ − cosβ 0
sin β ed
−1 ] It can be verified that the reference velocity signals, designed in (11), can be written as[
vd ωd
] = G−1 [−ṗd,1 cos(φ+ β)− ṗd,2 sin(φ+ β)− kded − ṗd,2ed cos(φ+ β) + ṗd,1 ed sin(φ+ β)− kββ ] and therefore, by using the relations v = ev + vd and ω = eω + ωd, ėd and β̇ can be written as[
ėd β̇
] = [ −kded −kββ ] +G [ ev eω ] = [ −kded −kββ ] + [ −ev cosβ ev
sin β ed − eω
] (13)
The control design follows the back-stepping methodology (Krstic et al. (1995)). Let, therefore, the function V1 := 12 (e 2 d + β 2), which, after time differentiation and use of (13), yields
V̇1 =− kde2d − kββ2 − edev cosβ + βev sinβ
ed − βeω (14)
We will cancel the two last terms of V̇1 using the control input (12).
First, we note that the inertia matrix of the system M , appearing in the unicycle dynamics (9), has the form (Ivanjko et al. (2010)) M = [ M1 M2 M2 M1 ] with M1 := mr 2 4 + (IC+md 2)r2 4R2 + I0, M2 := mr2 4 − (IC+md 2)r2
4R2 , and where IC is the moment of inertia of the vehicle with respect to point C (see Fig. 6), I0 is the moment of inertia of the the wheels, and d is the distance of the between
point C and the vehicle’s center of mass (p1, p2). Therefore, the second part of the unicycle dynamics (9) can be written as
M1θ̈R +M2θ̈L = uR + fθ,R(x̄, t)
M2θ̈R +M1θ̈L = uL + fθ,L(x̄, t)
with fθ,R and fθ,L denoting the elements of fθ. By summing and subtracting the aforementioned equations, we obtain
(M1 +M2)(θ̈R + θ̈L) = uR + uL + fθ,R(x̄, t) + fθ,L(x̄, t) (15a)
(M1 −M2)(θ̈R − θ̈L) = uR − uL + fθ,R(x̄, t)− fθ,L(x̄, t) (15b)
From the definition of ev and eω, it holds that ėv = v̇ − v̇d, ėω = ω̇ − ω̇d, which, by using the relations v = r2 (θ̇R + θ̇L), ω = r 2R (θ̇R − θ̇L) and (15), becomes
ėv = r
2(M1 +M2)
( uR + uL + fθ,R(x̄, t) + fθ,L(x̄, t) ) − v̇d (16a)
ėω = r
2R(M1 −M2)
( uR − uL + fθ,R(x̄, t)− fθ,L(x̄, t) ) − ω̇d (16b)
Define now `v := 2M1+M2r , `ω := 2R M1−M2 r . The adaptation terms ˆ̀ v, and ˆ̀ω, used in the control mechanism (12), aim to approximate `v and `ω, respectively. Note that ˆ̀v and ˆ̀ω are positive, and we define the errors ˜̀v := ˆ̀v − `v , ˜̀ω := ˆ̀ω − `ω . We re-write next the condition of Assumption 3. It holds that ‖p‖ = √ p21 + p 2 2 ≤ 2ed+ |pd,1|+ |pd,2|. Moreover, in view of (11) as well as the fact that β ∈ (−β̄, β̄) ⊂ (−π2 , π 2 ), where β̄ is the bound of β, stated in Corollary 2, it holds that |v| ≤ |ev|+ |vd| ≤ |ev|+ 1cos(β̄) (|ṗd,1|+ |ṗd,2|+ kded), |ω| ≤ |eω|+ |ωd| ≤ |eω|+ 1cos β̄ (α1 +kd)+kβ |β|, where we also use the fact that |ṗd,1 sinφ−ṗd,2 cosφ| ed
≤ α1. By also recalling that φ moves on the unit circle, we conclude that Assumption 3 becomes
‖fθ(x̄, t) + unn(x̄, t)‖ ≤ dded + dβ |β|+ d|ev|+ d|eω|+D, (17)
with dd := d(2 + kdcos β̄ ), dβ := dkβ , and D := d(2p̄d + 2π + 1 cos β̄ (α1 + 2v̄d + 1)), where p̄d and v̄d are the upper bounds of pd,i, vd,i, respectively, i ∈ {1, 2}. Let now positive constants γd and γβ such that kd > 2ddγd and kβ > 2dβγβ , and define
`1 := dd γd + dβ γβ + 4d (18)
which is the constant to be approximated by ˆ̀1; its derivation will be clarified later. Let also `2 := 2D, which we aim to approximate with the adaptation variable ˆ̀2. We define hence the respective errors˜̀ 1 := ˆ̀1−`1 and ˜̀2 := ˆ̀2−`2 and define the overall vector x̃ := [ed, β, ev, eω, ˜̀v, ˜̀ω, ˜̀1, ˜̀2]> ∈ R8. Since the control mechanism is discontinuous, we use again the notion of Filippov solutions. The Filippov regularization K[u] of u differs from u only in the terms êv and êω, which are replaced by Êv, defined as Êv := ev|ev| if ev 6= 0, and Êv ∈ (−1, 1) otherwise, and Êω, defined as Êω := eω |eω| if eω 6= 0, and Êω ∈ (−1, 1), respectively. Consider now the continuously differentiable and positive definite function
V (x̄) := V1 + `v 2 e2v + `ω 2 e2ω + 1 2kv ˜̀2 v + 1 2kω ˜̀2 ωi + ∑ i∈{1,2} 1 2ki ˜̀2 i
which satisfies W1(x̃) ≤ V (x̃) ≤ W2(x̃) for W1(x̃) := min{ 12 , `v 2 , `ω 2 , 1 2kv , 12kω , 1 2k1 , 12k2 }‖x̃‖ 2,
W2(x̃) := max{ 12 , `v 2 , `ω 2 , 1 2kv , 12kω , 1 2k1 , 12k2 }‖x̃‖
2. According to Lemma 1, V̇ (x̃) a.e. ∈ ˙̃V (x̃),
with ˙̃V := ⋂ ξ∈∂V (x̃) K[ ˙̃x]. Since V (x̃) is continuously differentiable, its generalized gradient reduces to the standard gradient and thus it holds that ˙̃V (x̃) = ∇V >K[x̃], where ∇V = [ed, β, `vev, `ωeω, 1 kv ˜̀ v, 1 kω ˜̀ ω, 1 k1 ˜̀ 1, 1 k2 ˜̀ 2] >. Therefore, by differentiating V and employing (14) and (15), we obtain
V̇ ⊂ W̃ :=− kde2d − kββ2 − edev cosβ + βev sinβ
ed − βeω + ev
( uR + uL + fθ,R + fθ,L − `v v̇d ) + eω ( uR − uL + fθ,R − fθ,L − `ωω̇d ) + 1
kv ˜̀ v ˙̂ `v + 1 kω ˜̀ ω ˙̂ `ω + ∑ i∈{1,2} 1 ki ˜̀ i ˙̂ `i
By substituting the control and adaptation policy (12), W̃ becomes
W̃ =− kde2d − kββ2 − kve2v − kβe2ω + ev v̇d ˜̀v + eωω̇d ˜̀ω − ˆ̀1(e2v + e2ω)− ˆ̀2(|ev|+ |eω|) + ev(unn,R + unn,L + fθ,R + fθ,L) + eω(unn,R − unn,L + fθ,R − fθ,L)
− ˜̀vev v̇d − ˜̀ωeωω̇d + ˜̀1(e2v + e2ω) + ˜̀2(|ev|+ |eω|) where uR,nn and uL,nn are the elements of unn, i.e., unn = [uR,nn, uL,nn]>. It is easy to conclude that |unn,R +unn,L + fθ,R + fθ,L| ≤ |unn,R + fθ,R|+ |unn,L + fθ,L| ≤ 2‖unn + fθ‖ and |unn,R−unn,L + fθ,R − fθ,L| ≤ |unn,R + fθ,R|+ |unn,L + fθ,L| ≤ 2‖unn + fθ‖ and hence, in view of (17),
|unn,R + unn,L + fθ,R + fθ,L| |unn,R − unn,L + fθ,R − fθ,L|
} ≤ D̄ := 2dded + 2dβ |β|+ 2d|ev|+ 2d|eω|+ 2D
Therefore, W̃ becomes
W̃ =− kde2d − kββ2 − kve2v − kβe2ω − ˆ̀1(e2v + e2ω)− ˆ̀2(|ev|+ |eω|) + D̄(|ev|+ |eω|)
+ ˜̀1(e2v + e2ω) + ˜̀2(|ev|+ |eω|) The term D̄(|ev|+ |eω|) yields
D̄(|ev|+ |eω|) =2dded|ev|+ 2dβ |β||ev|+ 2de2v + 4d|ev||eω|+ 2D|ev|+ 2dded|eω| + 2dβ |β||eω|+ 2de2ω + 2D|eω|
By setting 2dded|ev| = 2ddγded |ev|γd , 2dded|eω| = 2ddγded |eω| γd , and 2dβ |β||ev| = 2dβγβ |β| |ev|γβ , 2dβ |β||eω| = 2dβγβ |β| |eω|γβ and completing the squares, one obtains
D̄(|ev|+ |eω|) ≤2ddγde2d + 2dβγββ2 + ( dd γd + dβ γβ + 4d ) (e2v + e 2 ω) + 2D(|ev|+ |eω|)
=2ddγde 2 d + 2dβγββ 2 + `1(e 2 v + e 2 ω) + `2(|ev|+ |eω|)
Hence, W̃ becomes
W̃ ≤− (kd − 2ddγd)e2d − (kβ − 2dβγβ)β2 − kve2v − kβe2ω − ˆ̀1(e2v + e2ω)− ˆ̀2(|ev|+ |eω|)
+ `1(e 2 v + e 2 ω) + `2(|ev|+ |eω|) + ˜̀1(e2v + e2ω) + ˜̀2(|ev|+ |eω|)
=− (kd − 2ddγd)e2d − (kβ − 2dβγβ)β2 − kve2v − kβe2ω =: Q(x̃)
Therefore, it holds that ζ ≤ −Q(x̃), for all ζ ∈ ˙̃V , with Q being a continuous and positive semidefinite function in R8, since kd− 2ddγd > 0, kβ − 2dβγβ > 0. Choose now any finite r > 0 and let c < min‖x̃‖=rW1(x̃). Hence, all the conditions of Corollary 1 are satisfied and hence, all Filippov solutions starting from x̃(0) ∈ Ωf := {x̃ ∈ B(0, r) : W̃2(x̃) ≤ c} are bounded and remain in Ωf , satisfying limt→∞Q(x̃(t)) = 0.
Note that r, and hence c, can be arbitrarily large allowing and finite initial condition x̃(0). Moreover, the boundedness of x̃ implies the boundedness of e, ė, ev, ˜̀1, and ˜̀2, and hence of ˆ̀1(t) and ˆ̀2(t), for all t ∈ R≥0. Finally, in view of the conditions | sinφṗd,1−cosφṗd,2|ed < α1 and sin β ed
< α2, one concludes the boundedness of all closed-loop signals, for all t ≥ 0
The assumptions |ṗd,1 sinφ− ṗd,2 cosφ| < edα1, sinβ < edα2 are imposed to avoid the singularity of ed = 0; note that β and ωd are not defined in that case. Intuitively, they imply that ed will not be driven to zero faster than β or ṗd,1 sinφ− ṗd,2 cosφ; the latter becomes zero when the vehicle’s velocity vector v aligns with the desired one ṗd. In the experiments, we tune the control gains according to kβ ∼ 10kd in order to satisfy these assumptions.

heading: C APPENDIX
text: We provide here more information on the experimental results of Section 4. All the results were obtained on MATLAB environment using the ODE simulator. We performed the training of the neural networks in with the pytorch library in Python environment. The neural networks consist of 4 fully connected layers of 512 neurons; each layer is followed by a batch normalization module and a ReLU activation function. For the training we use the adam optimizer and the mean-square-error loss function. In all cases we choose a batch size of 256, and we train until a desirable average (per batch) loss of the order of 10−4 is achieved. All the values of the data used for the training were normalized in [0, 1].
Regarding the robotic-arm task, the points of interest are set as T1 = [−0.15,−0.475, 0.675, π2 , 0, 0] >, T2 = [−0.6, 0, 2.5, 0,−π2 ,− π 2 ] >, T3 = [−0.025, 0.595, 0.6,−π2 , 0, π] >, and T4 = [−0.525,−0.55, 0.28, π, 0,−π2 ]
>, and the corresponding joint-angle vectors as c1 = [−0.07,−1.05, 0.45, 2.3, 1.37,−1.33]>, c2 = [1.28, 0.35, 1.75, 0.03, 0.1,−1.22]>, c3 = [−0.08, 0.85,−0.23, 2.58, 2.09,−2.36]>, c4 = [−0.7,−0.76,−1.05,−0.05,−3.08, 2.37]> (radians). We set a nominal value for the time intervals as Ii = [0, 20] (seconds), and we create 150 problem instances by varying the following attributes: firstly, we add uniformly random offsets in [−0.3, 0.3] (radians) to the elements of all ci, and in [−2, 2] (seconds) to the right end-points of the intervals Ii; secondly, we add random offsets to the dynamic parameters of the robot (masses and moments of inertia of the robot’s links and actuators) and we set a different friction and disturbance term d(·), leading to a different dynamic model in (5); thirdly, we set different sequences of visits to the points ci, i ∈ {1, . . . , 4}, as dictated by φ, i.e., one trajectory might correspond to the visit sequence ((x(0), 0)→ (c1, t11)→ (c2, t12)→ (c3, t13)→ (c4, t14), and another to ((x(0), 0)→ (c3, t13)→ (c1, t11) → (c4, t14) → (c2, t12). Finally, we add uniformly random offsets in [−0.5, 0.5] to the initial position of the robot (from the first point of the sequence), and we set its initial velocity randomly in the interval [0, 1]6.
Regarding the dynamics (5), we use the methodology described in (Siciliano et al. (2009)) to derive the B, C, and g terms. We set nominal link masses and moments of inertia as m = [1, 2.5, 5.7, 3.9, 2.5, 2.5, 0.7] (kg) and I = [0.02, 0.04, 0.06, 0.05, 0.04, 0.04, 0.01] (kgm2), respectively, and we add random offsets in (−m2 , m 2 ), (− I 2 , I 2 ) in the created instances. Regarding the function d() used in (5); we set d(x̄, t) = dt(t) + df (x̄), where
dt = At sin(η1t+ ϕ1)... sin(η6t+ ϕ6)  , df = Btẋ⊗ ẋ At = diag{ Ati}i∈{1,...,6} ∈ R6×6, Ati is a random term in (0, 2mi), ηi is a random term in (0, 1), ϕi is a random term in (0, 2), Bt ∈ R6×36 is a random matrix taking values in (0, 2mi), and ⊗ denotes the Kronecker product. We chose the control gains of the control policy (4) as k1 = 1, k2 = 10, and k`1 = k`2 = 10.
Further experimental results are depicted in Figs. 7-11; Fig. 7a depicts the mean and standard deviation of ‖ev(t)‖ of the proposed control policy for the 50 test instances, whereas Figs. 10a depicts the control input that results from the control policy (4) as well as the the neural-network output. Note that the control input converges to the neural-network output, i.e., limt→∞(u(t)− unn(t)) = 0, which can be also verified by (4) and the fact that limt→∞ ev(t) = 0. Fig. 11a depicts the mean and standard deviation of the adaptation signals ˆ̀1(t), ˆ̀2(t) for the 50 test instances. Finally, Fig. 9 shows timestamps of one of the test trajectories, illustrating the visit of the robot end-effector to the points of interest at the pre-specified time stamps.
Regarding the unicycle experiments, the dynamic terms in (9) have the form
M = [ M1 M2 M2 M1 ] fθ(x̄, t) = d(x̄, t)
with M1 := mr 2 4 + (IC+md 2)r2 4R2 + I0, M2 := mr2 4 − (IC+md 2)r2
4R2 , and where IC is the moment of inertia of the vehicle with respect to point C (see Fig. 6), I0 is the moment of inertia of the the wheels, and d is the distance of the between point C and the vehicle’s center of mass (p1, p2). The term d(x̄, t) is chosen as in the robotic-manipulator case. The points to visit are chosen here as c1 = [0, 0]
>, c2 = [0, 2]>, c3 = [2, 0]>, c4 = [2, 2]> and Ii is chosen as Ii = [0, 20], i ∈ {1, . . . , 4}. We derive 150 problem instances by varying the following attributes: we vary ci with random offsets in [−0.3, 0.3] and the right end-points of Ii with random offsets in [−2, 2]; we add random offsets to the dynamic parameters (elaborated subsequently) and the function d(x̄, t), and we set different sequences of visits to the points ci; we further vary the unicycle’s initial position from the first ci with random offsets in [−0.3, 0.3], and its initial orientation with random offsets in [−0.25, 0.25] (rad) from θ(0) = arctan(e2(0)/e1(0)); we further set random values in [−0.25, 0.25] (rad/s) to the initial wheel velocities.
We set nominal values for the dynamic and geometric parameters, appearing in the inertia matrix, as m = 28 (kg), IC = 0.1 (kgm2), I0 = 0.01 (kgm2), r = 0.01 (m), d = 0.01 (m), and we added random offsets in (−m2 , m 2 ), (− IA 2 , IA 2 ), (− I0 2 , I0 2 ), (− r 2 , r 2 ), (− d 2 , d 2 ), respectively, for the problem instances. We chose the control gains of (12) as kd = 0.25, kβ = 1, kv = kω = k`1 = k`2 = 10, kv = kω = 1. The non-adaptive control policy we compared our algorithm with was selected as
uS = MS v̇d − kvev + ed cosβ − β sinβ ed uD = MDω̇d − kωeω + β
where vd, ωd are chosen as (11) and MS , MD are static estimates of 2M1+M2r , 2R M1−M2
r , respectively. More specifically, we set a deviation of 25% in the parameters appearing in these terms to form MS and MD. Further experimental results are depicted in Figs. 7-11; Fig. 7b depicts the mean and standard deviation of the velocity errors ev(t), eβ(t) for the 50 test instances, showing convergence to
zero, whereas Figs. 10b depicts the control input that results from the control policy (4) as well as the the neural-network output. Similarly to the robotic-manipulator case, the control input converges to the neural-network output. Finally, Fig. 11b depicts the mean and standard deviation of the adaptation signals ˆ̀v(t), ˆ̀β(t), ˆ̀1(t), ˆ̀2(t) for the 50 test instances.
Finally, regarding the pendulum environment, we consider the dynamics
q̈ = g
L sin q + u+ d(t)
where L is the pendulum’s link length, g is the gravitational constant, and d(t) is a term of exogenous disturbances. We created 150 instances by varying the mass and link length of the pendulum, the external disturbances, and setting its initial position and velocity randomly in [−1, 1] (rad) and [−1, 1] (rad/s), respectively. We selected unitary value for its nominal link length, and we added random offsets in (−0.5, 0.5) in each instance. We set the external disturbances as d = A sin(ηt+ φ), with A, η, and φ taking random values in [0, 0.2], [0, 1], [0, 2], respectively. In contrast to the robotic manipulator case, we assume feedback of sin q, cos q, q̇, and set the error e in (3) as 1− cos(q − π). Finally, we chose the control gains as k1 = k2 = 1, and k`1 = k`2 = 10.

Review summaries:
In this paper, the authors propose an adaptive controller that can be applied to a certain class of dynamical systems in order to fulfill Signal Interval Temporal Logic (SITL) tasks. The proposed controller leverages a nominal feedback law learned from past trajectory data collected on possibly many different SITL tasks and system parameters. The authors test their method in simulation on a manipulator arm, unicycle robot, and a pendulum with varying physical parameters and external forces.

The paper describes an adaptive controller which includes a learned component. The paper proves that this controller is asymptotically stable. Importantly the paper develops theory which proves that it is not necessary for the learned component to achieve bounded error on the task of predicting the control input given the state, but instead only that the output of the combined system be bounded by a simple function of the current state. The method incorporates a proposal for generating training data for the learned component by sampling trajectories (open-loop) of the system which conform to an SITL specification. The method is evaluated on several common simulation environments which comprise "reach-like" motion planning tasks. The results demonstrate that the proposes adaptive controller converges to the desired reference trajectory, sometimes faster than adaptive methods which do not incorporate the learned component. 

This paper presents a learning + adaptive control approach to synthesize control policies that fulfills STL specifications. The open-loop trajectory is generated using (Lindemann & Dimarogonas (2020)), the neural network is trained to replicate the expert demonstration consisting of state-action pairs with trajectories collected following multiple tasks (including the target task). This input is used as the open-loop input and a closed-loop trajectory tracking controller is added to correct the trajectory deviation using adaptive control techniques.