Non-Parametric Neuro-Adaptive Control Subject to Task Specifications

1 INTRODUCTION . Learning and control of autonomous systems with uncertain dynamics is a critical and challenging topic that has been widely studied during the last decades . One can identify plenty of motivating reasons , ranging from uncertain geometrical or dynamical parameters and unknown exogenous disturbances to abrupt faults that significantly modify the dynamics . There has been , therefore , an increasing need for developing control algorithms that do not rely on the underlying system dynamics . At the same time , such algorithms can be easily implemented on different , heterogeneous systems , since one does not need to be occupied with the tedious computation of the dynamic terms . A promising step towards the control of systems with uncertain dynamics is the use of data obtained a priori from system runs . However , engineering systems often undergo purposeful modifications ( e.g. , substitution of a motor or link in a robotic arm or exposure to new working environments ) or suffer gradual faults ( e.g. , mechanical degradation ) , which might change the systems ’ dynamics or operating conditions . Therefore , one can not rely on the aforementioned data to provably guarantee the successful control of the system . On the other hand , the exact incorporation of these changes in the dynamic model , and consequently , the design of new model-based algorithms , can be a challenging and often impossible procedure . Hence , the goal in such cases is to exploit the data obtained a priori and construct intelligent online policies that achieve a user-defined task while adapting to the aforementioned changes . There has been a large variety of works that tackle the problem of control of autonomous systems with uncertain dynamics , exhibiting , however , certain limitations . The existing algorithms are based on adaptive and learning-based approaches or the so-called funnel control ( Krstic et al . ( 1995 ) ; Kanellakopoulos et al . ( 1991 ) ; Xu & Ioannou ( 2003 ) ; Ge & Wang ( 2004 ) ; Wen et al . ( 2011 ) ; Chen et al . ( 2011 ) ; Slotine & Li ( 1987 ) ; Slotine & Coetsee ( 1986 ) ; Chen et al . ( 2008 ) ; Xian et al . ( 2004 ) ; Vamvoudakis & Lewis ( 2010 ) ; Berger et al . ( 2018 ) ; Bechlioulis & Rovithakis ( 2014 ) ; Joshi et al . ( 2020 ) ; Capotondi et al . ( 2020 ) ; Bertsekas & Tsitsiklis ( 1996 ) ; Sutton & Barto ( 2018 ) ) . Nevertheless , adaptive control methodologies are restricted to system dynamics that can be linearly parameterized with respect to certain unknown parameters ( e.g. , masses , moments of inertia ) , assuming the system structure perfectly known ; funnel controllers employ reciprocal terms that drive the control input to infinity when the tracking error approaches a pre-specified funnel function , creating thus unnecessarily large control inputs that might damage the system actuators . Data-based learning approaches either consider some system characteristic known ( e.g. , a nominal model , Lipschitz constants , or global bounds ) , or use neural networks to learn a tracking controller or the system dynamics ; the correctness of such methods , however , relies on strong assumptions on the parametric approximation by the neural network and knowledge of the underlying radial basis functions . Finally , standard reinforcementlearning techniques ( Bertsekas & Tsitsiklis ( 1996 ) ; Sutton & Barto ( 2018 ) ) usually assume certain state and/or time discretizations of the system and rely on exhaustive search of the state space , which might lead to undesirable transient properties ( e.g. , collision with obstacles while learning ) . 1.1 CONTRIBUTIONS AND SIGNIFICANCE . This paper addresses the control of systems with continuous , unknown nonlinear dynamics subject to tasks expressed via time-varying reference trajectories . Our main contribution lies in the development of a learning-based control algorithm that guarantees the accomplishment of a given task using only mild assumptions on the system dynamics . The algorithm draws a novel connection between adaptive control and learning with neural network representations , and consists of the following steps . Firstly , it trains a neural network that aims to learn a controller that accomplishes a given task from data obtained off-line . Secondly , we develop an online adaptive feedback control policy that uses the trained network to guarantee convergence to the given reference trajectory and hence satisfaction of the task . Essentially , our approach builds on a combination of off-line trained controllers and on-line adaptations , which was recently shown to significantly enhance performance with respect to single use of the off-line part ( Bertsekas ( 2021 ) ) . The major significance of our contribution is twofold . Firstly , we guarantee the theoretical correctness of the proposed algorithm by considering only mild conditions on the neural network , removing the long-standing assumptions on parametric approximations and boundedness of the estimation error . Secondly , we demonstrate via the experimental results the generality of the algorithm with respect to different tasks and system parameters . That is , the training data that we generate for the training of the neural network in the first step correspond to tasks that are different from the given one to be executed1 . Additionally , we employ systems with different dynamic parameters to generate these data . We evaluate the proposed algorithm in numerous scenarios comprising different variations of the given task and different system dynamic parameters , which do not necessarily match the training data . We show that the algorithm , owing to its adaptation properties , is able to guarantee the satisfaction of the respective tasks in all the aforementioned scenarios by using the same neural network . 1.2 RELATED WORK . A large variety of previous works considers neuro-adaptive control with stability guarantees , focusing on the optimal control problem ( Vamvoudakis & Lewis ( 2010 ) ; Yang et al . ( 2020 ) ; Cheng et al . ( 2007 ) ; Fan et al . ( 2018 ) ; Kiumarsi et al . ( 2017 ) ; Zhao & Gan ( 2020 ) ; Vrabie & Lewis ( 2009 ) ; Sun & Vamvoudakis ( 2020 ) ; Kamalapurkar et al . ( 2015 ) ; Huang et al . ( 2018b ) ; Mo et al . ( 2019 ) ; Joshi et al . ( 2020 ) ) . Nevertheless , the related works draw motivation from the neural network density property ( see , e.g. , ( Cybenko ( 1989 ) ) ) 2 and assume sufficiently small approximation errors and linear parameterizations of the unknown terms of the form W ( x ) θ ( dynamics , optimal controllers , or value functions ) , with W and θ known and unknown , respectively . Similarly , more traditional adaptive control methodologies that handle uncertain nonlinear systems assume either linear parameterizations of the unknown dynamic terms ( Krstic et al . ( 1995 ) ; Hong et al . ( 2009 ) ; Chen ( 2019 ) ; Huang et al . ( 2018a ) ; Kanellakopoulos et al . ( 1991 ) ; Xu & Ioannou ( 2003 ) ; Ge & Wang ( 2004 ) ; Wen et al . ( 2011 ) ; Chen et al . ( 2011 ) ; Slotine & Li ( 1987 ) ; Slotine & Coetsee ( 1986 ) ; Chen et al . ( 2008 ) ) , use known upper-bound functions ( Slotine & Coetsee ( 1986 ) ) , or provide local stability results dictated by the dynamic bounds ( Xian et al . ( 2004 ) ; Chen et al . ( 2008 ) ) . This paper relaxes the aforementioned assumptions and proposes a non-parametric neuro-adaptive controller , whose stability guarantees 1The task difference is illustrated in Section 4 . 2A sufficiently large neural network can approximate a continuous function arbitrarily well in a compact set . rely on a mild boundedness condition of the closed-loop system state that is driven by the learned controller . The proposed approach exhibits similarities with ( Liu et al . ( 1994 ) ) , which employs off-line-trained neural networks with online feedback control , but fails to provide convergence guarantees . Similarly , Zeng et al . ( 2020 ) develops an algorithm that combines a learning module and a physics-based controller for robot throwing of arbitrary objects , assuming however access to a physics simulator and not providing theoretical guarantees . Other learning-based related works include modeling with Gaussian processes ( Capotondi et al . ( 2020 ) ; Leahy et al . ( 2019 ) ; Jain et al . ( 2018 ) ; Berkenkamp & Schoellig ( 2015 ) ) , or use neural networks ( Ma et al . ( 2020 ) ; Shah et al . ( 2018 ) ; Yan & Julius ( 2021 ) ; Liu et al . ( 2021 ) ; Hahn et al . ( 2020 ) ; Cai et al . ( 2021 ) ; Wang et al . ( 2020 ) ; Camacho & McIlraith ( 2019 ) ; Hahn et al . ( 2020 ) ; Riegel et al . ( 2020 ) ; Hu et al . ( 2020 ) ; Ivanov et al . ( 2019 ) ) to accomplish reachability , verification or temporal logic specifications . Nevertheless , the aforementioned works either use partial information on the underlying system dynamics , or do not consider them at all . In addition , works based on Gaussian processes usually propagate the dynamic uncertainties , possibly leading to conservative results . Similarly , data-driven model-predictive control techniques ( Nubert et al . ( 2020 ) ; Maddalena et al . ( 2020 ) ) use data to over-approximate additive disturbances or are restricted to linear systems . Control of unknown nonlinear continuous-time systems has been also tackled in the literature by using funnel control , without necessarily using off-line data or dynamic approximations ( Berger et al . ( 2018 ) ; Bechlioulis & Rovithakis ( 2014 ) ; Lindemann et al . ( 2017 ) ; Verginis & Dimarogonas ( 2018 ) ; Verginis et al . ( 2021 ) ) . Nevertheless , funnel controllers usually depend on reciprocal time-varying barrier functions that drive the control input to infinity when the error approaches a pre-specified funnel , creating thus unnecessarily large control inputs that might damage the system actuators . 2 PROBLEM FORMULATION . Consider a continuous-time dynamical system governed by the 2nd-order continuous-time dynamics ẍ = f ( x̄ , t ) + g ( x̄ , t ) u ( x̄ , t ) , ( 1 ) where x̄ : = [ x > , ẋ > ] > ∈ R2n , n ∈ N , is the system state , assumed available for measurement , and u : R2n × [ 0 , ∞ ) → Rn is the time-varying feedback-control input . The terms f ( · ) and g ( · ) are nonlinear vector fields that are locally Lipschitz in x̄ over R2n for each fixed t ≥ 0 , and uniformly bounded in t over [ 0 , ∞ ) for each fixed x̄ ∈ R2n . The dynamics ( 1 ) comprise a large class of nonlinear dynamical systems ( Zhong & Leonard ( 2020 ) ; Yu et al . ( 1996 ) ; Doya ( 1997 ) ) that capture contemporary engineering problems in mechanical , electromechanical and power electronics applications , such as rigid/flexible robots , induction motors and DC-to-DC converters , to name a few . The continuity in time and state provides a direct link to the actual underlying system , and we further do not require any time or state discretizations . We consider that f ( · ) and g ( · ) are completely unknown ; we do not assume any knowledge of the structure , Lipschitz constants , or bounds , and we do not use any scheme to approximate them . Note also that we do not assume global Lipschitz continuity or global boundedness of f ( · , t ) and g ( · , t ) or the solution x̄ ( t ) of ( 1 ) . Nevertheless , we do assume that g ( x̄ , t ) is positive definite : Assumption 1 . The matrix g ( x̄ , t ) is positive definite , for all ( x̄ , t ) ∈ R2n × [ 0 , ∞ ) . Such assumption is a sufficiently controllability condition for ( 1 ) ; intuitively , it states that the multiplier of u ( the input matrix ) does not change the direction imposed to the system by the underlying control algorithm . Systems not covered by ( 1 ) or Assumption 1 consist of underactuated or non-holonomic systems , such as unicycle robots or underactuated aerial vehicles . Nevertheless , we provide an extension of our results for a non-holonomic unicycle vehicle in Appendix B . Moreover , the 2nd-order model ( 1 ) can be easily extended to account for higher-order integrator systems ( Slotine et al . ( 1991 ) ) . Consider now a time-constrained task expressed as a time-varying reference trajectory pd : R≥0 → Rn . The objective of this paper is to construct a time-varying feedback-control algorithm u ( x̄ , t ) such that the state of the closed-loop system ( 1 ) asymptotically tracks pd , i.e. , limt→∞ ( x ( t ) − pd ( t ) ) = 0 .

Summary: This paper presents a learning + adaptive control approach to synthesize control policies that fulfills STL specifications. The open-loop trajectory is generated using (Lindemann & Dimarogonas (2020)), the neural network is trained to replicate the expert demonstration consisting of state-action pairs with trajectories collected following multiple tasks (including the target task). This input is used as the open-loop input and a closed-loop trajectory tracking controller is added to correct the trajectory deviation using adaptive control techniques.