Non-Parametric Neuro-Adaptive Control Subject to Task Specifications

1 INTRODUCTION . Learning and control of autonomous systems with uncertain dynamics is a critical and challenging topic that has been widely studied during the last decades . One can identify plenty of motivating reasons , ranging from uncertain geometrical or dynamical parameters and unknown exogenous disturbances to abrupt faults that significantly modify the dynamics . There has been , therefore , an increasing need for developing control algorithms that do not rely on the underlying system dynamics . At the same time , such algorithms can be easily implemented on different , heterogeneous systems , since one does not need to be occupied with the tedious computation of the dynamic terms . A promising step towards the control of systems with uncertain dynamics is the use of data obtained a priori from system runs . However , engineering systems often undergo purposeful modifications ( e.g. , substitution of a motor or link in a robotic arm or exposure to new working environments ) or suffer gradual faults ( e.g. , mechanical degradation ) , which might change the systems ’ dynamics or operating conditions . Therefore , one can not rely on the aforementioned data to provably guarantee the successful control of the system . On the other hand , the exact incorporation of these changes in the dynamic model , and consequently , the design of new model-based algorithms , can be a challenging and often impossible procedure . Hence , the goal in such cases is to exploit the data obtained a priori and construct intelligent online policies that achieve a user-defined task while adapting to the aforementioned changes . There has been a large variety of works that tackle the problem of control of autonomous systems with uncertain dynamics , exhibiting , however , certain limitations . The existing algorithms are based on adaptive and learning-based approaches or the so-called funnel control ( Krstic et al . ( 1995 ) ; Kanellakopoulos et al . ( 1991 ) ; Xu & Ioannou ( 2003 ) ; Ge & Wang ( 2004 ) ; Wen et al . ( 2011 ) ; Chen et al . ( 2011 ) ; Slotine & Li ( 1987 ) ; Slotine & Coetsee ( 1986 ) ; Chen et al . ( 2008 ) ; Xian et al . ( 2004 ) ; Vamvoudakis & Lewis ( 2010 ) ; Berger et al . ( 2018 ) ; Bechlioulis & Rovithakis ( 2014 ) ; Joshi et al . ( 2020 ) ; Capotondi et al . ( 2020 ) ; Bertsekas & Tsitsiklis ( 1996 ) ; Sutton & Barto ( 2018 ) ) . Nevertheless , adaptive control methodologies are restricted to system dynamics that can be linearly parameterized with respect to certain unknown parameters ( e.g. , masses , moments of inertia ) , assuming the system structure perfectly known ; funnel controllers employ reciprocal terms that drive the control input to infinity when the tracking error approaches a pre-specified funnel function , creating thus unnecessarily large control inputs that might damage the system actuators . Data-based learning approaches either consider some system characteristic known ( e.g. , a nominal model , Lipschitz constants , or global bounds ) , or use neural networks to learn a tracking controller or the system dynamics ; the correctness of such methods , however , relies on strong assumptions on the parametric approximation by the neural network and knowledge of the underlying radial basis functions . Finally , standard reinforcementlearning techniques ( Bertsekas & Tsitsiklis ( 1996 ) ; Sutton & Barto ( 2018 ) ) usually assume certain state and/or time discretizations of the system and rely on exhaustive search of the state space , which might lead to undesirable transient properties ( e.g. , collision with obstacles while learning ) . 1.1 CONTRIBUTIONS AND SIGNIFICANCE . This paper addresses the control of systems with continuous , unknown nonlinear dynamics subject to tasks expressed via time-varying reference trajectories . Our main contribution lies in the development of a learning-based control algorithm that guarantees the accomplishment of a given task using only mild assumptions on the system dynamics . The algorithm draws a novel connection between adaptive control and learning with neural network representations , and consists of the following steps . Firstly , it trains a neural network that aims to learn a controller that accomplishes a given task from data obtained off-line . Secondly , we develop an online adaptive feedback control policy that uses the trained network to guarantee convergence to the given reference trajectory and hence satisfaction of the task . Essentially , our approach builds on a combination of off-line trained controllers and on-line adaptations , which was recently shown to significantly enhance performance with respect to single use of the off-line part ( Bertsekas ( 2021 ) ) . The major significance of our contribution is twofold . Firstly , we guarantee the theoretical correctness of the proposed algorithm by considering only mild conditions on the neural network , removing the long-standing assumptions on parametric approximations and boundedness of the estimation error . Secondly , we demonstrate via the experimental results the generality of the algorithm with respect to different tasks and system parameters . That is , the training data that we generate for the training of the neural network in the first step correspond to tasks that are different from the given one to be executed1 . Additionally , we employ systems with different dynamic parameters to generate these data . We evaluate the proposed algorithm in numerous scenarios comprising different variations of the given task and different system dynamic parameters , which do not necessarily match the training data . We show that the algorithm , owing to its adaptation properties , is able to guarantee the satisfaction of the respective tasks in all the aforementioned scenarios by using the same neural network . 1.2 RELATED WORK . A large variety of previous works considers neuro-adaptive control with stability guarantees , focusing on the optimal control problem ( Vamvoudakis & Lewis ( 2010 ) ; Yang et al . ( 2020 ) ; Cheng et al . ( 2007 ) ; Fan et al . ( 2018 ) ; Kiumarsi et al . ( 2017 ) ; Zhao & Gan ( 2020 ) ; Vrabie & Lewis ( 2009 ) ; Sun & Vamvoudakis ( 2020 ) ; Kamalapurkar et al . ( 2015 ) ; Huang et al . ( 2018b ) ; Mo et al . ( 2019 ) ; Joshi et al . ( 2020 ) ) . Nevertheless , the related works draw motivation from the neural network density property ( see , e.g. , ( Cybenko ( 1989 ) ) ) 2 and assume sufficiently small approximation errors and linear parameterizations of the unknown terms of the form W ( x ) θ ( dynamics , optimal controllers , or value functions ) , with W and θ known and unknown , respectively . Similarly , more traditional adaptive control methodologies that handle uncertain nonlinear systems assume either linear parameterizations of the unknown dynamic terms ( Krstic et al . ( 1995 ) ; Hong et al . ( 2009 ) ; Chen ( 2019 ) ; Huang et al . ( 2018a ) ; Kanellakopoulos et al . ( 1991 ) ; Xu & Ioannou ( 2003 ) ; Ge & Wang ( 2004 ) ; Wen et al . ( 2011 ) ; Chen et al . ( 2011 ) ; Slotine & Li ( 1987 ) ; Slotine & Coetsee ( 1986 ) ; Chen et al . ( 2008 ) ) , use known upper-bound functions ( Slotine & Coetsee ( 1986 ) ) , or provide local stability results dictated by the dynamic bounds ( Xian et al . ( 2004 ) ; Chen et al . ( 2008 ) ) . This paper relaxes the aforementioned assumptions and proposes a non-parametric neuro-adaptive controller , whose stability guarantees 1The task difference is illustrated in Section 4 . 2A sufficiently large neural network can approximate a continuous function arbitrarily well in a compact set . rely on a mild boundedness condition of the closed-loop system state that is driven by the learned controller . The proposed approach exhibits similarities with ( Liu et al . ( 1994 ) ) , which employs off-line-trained neural networks with online feedback control , but fails to provide convergence guarantees . Similarly , Zeng et al . ( 2020 ) develops an algorithm that combines a learning module and a physics-based controller for robot throwing of arbitrary objects , assuming however access to a physics simulator and not providing theoretical guarantees . Other learning-based related works include modeling with Gaussian processes ( Capotondi et al . ( 2020 ) ; Leahy et al . ( 2019 ) ; Jain et al . ( 2018 ) ; Berkenkamp & Schoellig ( 2015 ) ) , or use neural networks ( Ma et al . ( 2020 ) ; Shah et al . ( 2018 ) ; Yan & Julius ( 2021 ) ; Liu et al . ( 2021 ) ; Hahn et al . ( 2020 ) ; Cai et al . ( 2021 ) ; Wang et al . ( 2020 ) ; Camacho & McIlraith ( 2019 ) ; Hahn et al . ( 2020 ) ; Riegel et al . ( 2020 ) ; Hu et al . ( 2020 ) ; Ivanov et al . ( 2019 ) ) to accomplish reachability , verification or temporal logic specifications . Nevertheless , the aforementioned works either use partial information on the underlying system dynamics , or do not consider them at all . In addition , works based on Gaussian processes usually propagate the dynamic uncertainties , possibly leading to conservative results . Similarly , data-driven model-predictive control techniques ( Nubert et al . ( 2020 ) ; Maddalena et al . ( 2020 ) ) use data to over-approximate additive disturbances or are restricted to linear systems . Control of unknown nonlinear continuous-time systems has been also tackled in the literature by using funnel control , without necessarily using off-line data or dynamic approximations ( Berger et al . ( 2018 ) ; Bechlioulis & Rovithakis ( 2014 ) ; Lindemann et al . ( 2017 ) ; Verginis & Dimarogonas ( 2018 ) ; Verginis et al . ( 2021 ) ) . Nevertheless , funnel controllers usually depend on reciprocal time-varying barrier functions that drive the control input to infinity when the error approaches a pre-specified funnel , creating thus unnecessarily large control inputs that might damage the system actuators . 2 PROBLEM FORMULATION . Consider a continuous-time dynamical system governed by the 2nd-order continuous-time dynamics ẍ = f ( x̄ , t ) + g ( x̄ , t ) u ( x̄ , t ) , ( 1 ) where x̄ : = [ x > , ẋ > ] > ∈ R2n , n ∈ N , is the system state , assumed available for measurement , and u : R2n × [ 0 , ∞ ) → Rn is the time-varying feedback-control input . The terms f ( · ) and g ( · ) are nonlinear vector fields that are locally Lipschitz in x̄ over R2n for each fixed t ≥ 0 , and uniformly bounded in t over [ 0 , ∞ ) for each fixed x̄ ∈ R2n . The dynamics ( 1 ) comprise a large class of nonlinear dynamical systems ( Zhong & Leonard ( 2020 ) ; Yu et al . ( 1996 ) ; Doya ( 1997 ) ) that capture contemporary engineering problems in mechanical , electromechanical and power electronics applications , such as rigid/flexible robots , induction motors and DC-to-DC converters , to name a few . The continuity in time and state provides a direct link to the actual underlying system , and we further do not require any time or state discretizations . We consider that f ( · ) and g ( · ) are completely unknown ; we do not assume any knowledge of the structure , Lipschitz constants , or bounds , and we do not use any scheme to approximate them . Note also that we do not assume global Lipschitz continuity or global boundedness of f ( · , t ) and g ( · , t ) or the solution x̄ ( t ) of ( 1 ) . Nevertheless , we do assume that g ( x̄ , t ) is positive definite : Assumption 1 . The matrix g ( x̄ , t ) is positive definite , for all ( x̄ , t ) ∈ R2n × [ 0 , ∞ ) . Such assumption is a sufficiently controllability condition for ( 1 ) ; intuitively , it states that the multiplier of u ( the input matrix ) does not change the direction imposed to the system by the underlying control algorithm . Systems not covered by ( 1 ) or Assumption 1 consist of underactuated or non-holonomic systems , such as unicycle robots or underactuated aerial vehicles . Nevertheless , we provide an extension of our results for a non-holonomic unicycle vehicle in Appendix B . Moreover , the 2nd-order model ( 1 ) can be easily extended to account for higher-order integrator systems ( Slotine et al . ( 1991 ) ) . Consider now a time-constrained task expressed as a time-varying reference trajectory pd : R≥0 → Rn . The objective of this paper is to construct a time-varying feedback-control algorithm u ( x̄ , t ) such that the state of the closed-loop system ( 1 ) asymptotically tracks pd , i.e. , limt→∞ ( x ( t ) − pd ( t ) ) = 0 .

Summary: The paper describes an adaptive controller which includes a learned component. The paper proves that this controller is asymptotically stable. Importantly the paper develops theory which proves that it is not necessary for the learned component to achieve bounded error on the task of predicting the control input given the state, but instead only that the output of the combined system be bounded by a simple function of the current state. The method incorporates a proposal for generating training data for the learned component by sampling trajectories (open-loop) of the system which conform to an SITL specification. The method is evaluated on several common simulation environments which comprise "reach-like" motion planning tasks. The results demonstrate that the proposes adaptive controller converges to the desired reference trajectory, sometimes faster than adaptive methods which do not incorporate the learned component.