Non-Parametric Neuro-Adaptive Control Subject to Task Specifications
Authors: Christos Verginis
Zhe Xu
ufuk topcu
Venue: ICLR 2022 Submitted (2022)
Keywords: Neural-network-control
nonlinear systems
continuous control
adaptive control
task specification
signal temporal logic

Abstract: We develop a learning-based algorithm for the control of autonomous systems governed by unknown, nonlinear dynamics to satisfy user-specified spatio-temporal tasks expressed as signal temporal logic specifications. Most existing algorithms either assume certain parametric forms for the unknown dynamic terms or resort to unnecessarily large control inputs in order to provide theoretical guarantees. 
The proposed algorithm addresses these drawbacks by integrating neural-network-based learning with adaptive control. More specifically, the algorithm learns a controller, represented as a neural network, using training data that correspond to a collection of system parameters and tasks. These parameters and tasks are derived by varying the nominal parameters and the spatio-temporal constraints of the user-specified task, respectively.  It then incorporates this neural network into an online closed-form adaptive control policy in such a way that the resulting behavior satisfies the user-defined task. The proposed algorithm does not use any a priori information on the unknown dynamic terms or any approximation schemes. We provide formal theoretical guarantees on the satisfaction of the task. Numerical experiments on a robotic manipulator and a unicycle robot demonstrate that the proposed algorithm guarantees the satisfaction of 50 user-defined tasks, and outperforms control policies that do not employ online adaptation or the neural-network controller. Finally, we show that the proposed algorithm achieves greater performance than standard reinforcement-learning algorithms in the pendulum benchmarking environment.

Under review as a conference paper at ICLR 2022
NON-PARAMETE NEURO- CONTROL
Anonymous authors Paper under double-blind review
ABSTRACT
We develop maaming--bssed algorithm for the control of autonomous systems governed by unknown, nonlinear dynamics to satisfy user-specified tasks expressed via vare-ia reference trajectories. Most existing algorithms either assume certain parametric forms for the unknown dynamic terms or resort to unnecessarily large control inputs in order to provide theoretical guarantees. The proposed algo- rithm addresses these drawbacks by integrating neural-network-basee learning with adaptive control. More specifically, the algorithm learns a controller, represented as neural network, using training data that correspond to a collection of system pa- rameters and tasks. These parameters and tasks are derived by varying the nominal parameters and the reference trajectories, respectively. It then incorporates this neu- ral network into an online closed-fore adaptive control policy in such a way that the resulting behavior satisfies the user-defined task. The proposed algorithm does not use any a priori information on the unknown dynamic terms or any approximation schemes. We provide formal theoretical guarantees on the satisfaction of the task. Numerical experiments on a robotic manipulator and a unicycle robot demonstrate that the proposed algorithm guarantees the satisfaction of 50 user-defined tasks, and outperforms control policies that do not employ online adaptation or the neural- network controller. Finally, we show that the proposed algorithm achieves greater performance than standard reinforcement-leaming algorithms in the pendulum benchmarking environment.
INTRODUCTION
Learning and control of autonomous systems with uncertain dynamics is critical and challenging topic that has been widely studied during the last decades. One can identify plenty of motivating reasons, ranging from uncertain geometrical or dynamical parameters and unknown exogenous disturbances to abrupt faults that significantly modify the dynamics. There has been, therefore, an increasing need for developing control algorithms that do not rely on the underlying system dynamics. At the same time, such algorithms can be easily implemented on different, heterogeneous systems, since one does not need to be occupied with the tedious computation of the dynamic terms.
promising step towards the control of systems with uncertain dynamics is the use of data obtained priori from system runs. However, engineering systems often undergo purposeful modifications (e.g., substitution of a motor or link in a robotic arm or exposure to new working environments) or suffer gradual faults (e.g., mechanical degradation), which might change the systems' dynamics or operating conditions. Therefore, one cannot rely on the aforementioned data to provably guarantee the successful control of the system. On the other hand, the exact incorporation of these changes in the dynamic model, and consequently, the design of new model-based algorithms, can be challenging and often impossible procedure. Hence, the goal such cases is exploit the data obtained priori and construct intelligent online policies that achieve a user-defined task while adapting to the aforementioned changes.
There has been large variety of works that tackle the problem of control of autonomous systems with uncertain dynamics, exhibiting, however, certain limitations. The existing algorithms are based on adaptive and learning-based
approaches
the so-called funnel control (Krstic et al. (1955):
Kanellakopoulos et al. [199]);]2 Xu & loannou (2003); Ge & Wang (2004); Wen et al. (2011 Chen Slotine& & Coetsee 1986); Chen et al. (2008) Xian et al. (2004);
et al. (2011); 1>; Slotine & Li (187 Sl
Vamvoudakis & Lewis (2010): Berger et al.] (2018); Bechlioulis & Rovithakis 2014); Joshi et al.
(2020); Cappooodi al. 2020) Bertsekas & Tsitsiklis (199)); Suttoon & Barto (001)). Nevertheless,

---

Under review as a conference paper at ICLR 2022
adaptive control methodologies restricted to system dynamics that can be linearly parameterized with respect to certain unknown parameters (c.g., masses, moments of inertia), assuming the system structure perfectly known; funnel controllers employ reciprocal terms that drive the control input to infinity when the tracking error approaches pre-specified funnel function, creating thus unnecessarily large control inputs that might damage the system actuators. Data-based learning approaches either consider some system characteristic known (c.g., nominal model, Lipschitz constants, or global bounds), or use neural networks to learn a tracking controller or the system dynamics; the correctness of such methods, however, relies on strong assumptions on the parametric approximation by the neural network and knowledge of the underlying radial basis functions. Finally, standard reinforcement- learning techniques (Bertsekas & Tsitsiklis (1996); Sutton & Barto (2018) ) usually assume certain state and/or time discretizations of the system and rely on exhaustive search of the state space, which might lead to undesirable transient properties (e.g., collision with obstacles while learning).
1.1 CONTRIBUTIONS AND SIGNIFICANCE
This paper addresses the control of systems with continuous, unknown nonlinear dynamics subject to tasks expressed via time-van varying reference trajectories. Our main contribution lies in the development of a learning-basee based control algorithm that guarantees the accomplishment of a given task using only mild assumptions on the system dynamics. The algorithm draws a novel connection between adaptive control and learning with neural network representations, and consists of the following steps. Firstly, it trains neural network that aims to learn controller that accomplishes given task from data obtained off-line. Secondly, we develop an online adaptive feedback control policy that uses the trained network to guarantee convergence to the given reference trajectory and hence satisfaction of the task. Essentially, our approach builds on a combination of off-line trained controllers and on-line adaptations. which was recently shown to significantly enhance performance with respect single use of the off-line part (Bertsekas (202T)).
The major significance of our contribution is twofold. Firstly, we guarantee the theoretical correctness of the proposed algorithm by considering only mild conditions on the neural network, removing the long-standing assumptions on parametric approximations and boundedness of the estimation error. Secondly, we demonstrate via the experimental results the generality of the algorithm with respect to different tasks and system parameters. That is, the training data that we generate for the training of the neural network in the first step correspond to tasks that are different from the given one to be executed) [ Additionally, we employ systems with different dynamic parameters to generate these data. We evaluate the proposed algorithm in numerous scenarios comprising different variations of the given task and different system dynamic parameters, which do not necessarily match the training data. We show that the algorithm, owing its adaptation properties, is able to guarantee the satisfaction of the respective tasks in all the aforementioned scenarios by using the same neural network.
1.2 RELATED WORK
A large variety of previous works considers neuro-adaptive control with stability guarantees, focusing on the optimal control problem (Vamvoudakis & Lewis] (2010) Yang et al. (2020); Cheng et al.
(2007); Fan al. (2018); Kiumarsi et al. (017)) Zhao & Gan (2020): V Vrabie & Lewis (2009); Sun
& Vamvoudakis (2020): Kamalapurkar et al. (2015); Huang et al. (2018)); Mo et al. Joshi
al. 22020). Nevertheless, the related works draw motivation from the neural network density
property (see, e.g., (Cybenko 1989)) ? and assume sufficiently small approximation errors and linear parameterizations of the unknown terms of the form ((x)d (dynamies, optimal controllers, or value functions), with W and 9 known and unknown, respectively. Similarly, more traditional adaptive control methodologies that handle uncertain nonlinear systems assume either linear parameterizations of the unknown dynamic terms (Krstic et al. y HHng et al.] (2009); Chen (2019): [Huang et al. (2018a); Kanellakopoulos et al.) (1991) Xu & Ioannou| (2003): Ge & Wang (2004 y; Wen et al. (2011): Slotine & Coetsee (1966); Chen et al. (2008)), use known upper-bound functions (Slotine & Coetsee (1986). or provide local stability results dictated by the dynamic bounds (Xian et al. al.] (2004): Chen et al. (2008)). This paper relaxes the aforementioned
Chen et al. (201 Slotine & Li
assumptions and proposes a non non-parametric neuro-adaptive controller, whose stability guarantees
'The task difference is illustrated in Section
A sufficiently large neural network can approximate continuous function arbitrarily well in a compact set.
2

---

Under review as a conference paper at ICLR 2022
rely on mild boundedness condition of the closed-loop system state that is driven by the learned controller. The proposed approach exhibits similarities with (Liu et al. (1994)), which employs off off-line-rrained neural networks with online feedback control, but fails to provide convergence
guarantees. Similarly, ZZeng et al. 2020) develops an algorithm that combines a learning module and physics-based controller for robot throwing of arbitrary objects, assuming however access to physics simulator and not providing theoretical guarantees.
Other learning-based related works include modeling with Gaussian processes (Capotondi al. 2200)) Leahy et al.] (2019): Jain et al. (0188):
Berkenkamp & Schoellig (2015), or use neural
networks (Ma et al. (2020); Shah et 2018); Yan & Julius (2021); Liu et (2021 II Hahn et al.)
(2020); Cai et al. (2021) Wang et al. (2020); Camacho & McIlraith (2019); Hahn et al. (2020);
Riegel et al. 2020); Hu et al. (2020): Ivanov et al. (2019)) to accomplish reachability, verification or temporal logic specifications. Nevertheless, the aforementioned works either use partial information on the underlying system dynamics, or do not consider them all. In addition, works based on Gaussian processes usually propagate the dynamic uncertainties, possibly leading to conservative results. Similarly, data-driven model-predictive control techniques (Nubert et al. (2020); Maddalena) restricted to linear systems.
et al. 2020g) use data over-approxinate additive disturbances
Control of unknown nonlinear continuous-ting systems has been also tackled in the literature by using funnel control, without necessarily using off-line data dynamic approximations (Berger et al. (2018); Bechlioulis & Rovithakis (2014); Lindemann et al. (201)); Verginis & Dimarogonas (2018):
Verginis et al. (2021)). Nevertheless, funnel controllers usually depend on reciprocal dime-varying barrier functions that drive the control input to infinity when the error approaches a pre-specified funnel, creating thus unnecessarily large control inputs that might damage the system actuators.
PROBLEM FORMULATION
Consider a continuous-ting time dynamical system governed by the 2nd-order continuous time dynamics
ã = f(z 8) + g(2, t),
(1)
where 2 [x [ E RR"", n N, is the system state, assumed available for measurement,and
222n x 10, 00) + R" is the vame-narying feedback-control input. The terms f(-) and g(-) are nonlinear vector fields that are locally Lipschitz in F over 2R20 for each fixed 2 0, and uniformly bounded in É over [0, oo) for each fixed & E R220_ The dynamics
comprise large
class of nonlinear dynamical systems (Zhong & Leonard (2020): Yu et al. (1996); Doya (1997)) that capture contemporary engineering problems in mechanical, electromechanicl and power electronics applications, such as rigid/flexible robots, induction motors and DC-to-DC converters, to name a few. The continuity in time and state provides direct link to the actual underlying system, and we further do not require any time or state discretizations.
We consider that f(-) and g(-) are completely unknown; do assume any knowledge of the structure, Lipschitz constants, or bounds, and we do not use any scheme to approximate them. Note also that we do not assume global Lipschitz continuity or global boundedness of f(-,t) and g( or the solution a(() of [ Nevertheless, we do assume that g(ā, is positive definite:
Assumption 1. The matrix g(.. 1) is positive definite, for all (H, t) E 2R x [0,
Such assumption sufficiently controllability condition for I intuitively, states that the multiplier of (the input matrix) does not change the direction imposed to the system by the underlying control algorithm. Systems not covered by (T) or [ non-holooomic systems, such as unicycle robots underactuated acrial vehicles. Nevertheless, provide an extension of our results for a non-holonomic unicycle vehicle in Appendix B Moreover, the 2nd-order model (1 can be easily extended to account for higher-order integrator systems (Slotine et al. (1991)).
I consist of underactuated
Consider now a constraind task expressed as vimevnarying reference trajectory Pa R20
R". The objective of this paper is to construct a time-varying feedback- control algorithm u(ē t) such that the state of the closed-loop system (1) asymptotically tracks pa, i.e., lim -a(()) O.
3

---

Under review as a conference paper at ICLR 2022
3
MAIN RESULTS
This section describes the proposed algorithm, which consists of two steps. Firstly, it learns a controller, represented as neural network, using training data that correspond to a collection of different tasks and system parameters. Secondly, we design an adaptive, time-varying feedback controller that uses the neural-network approximation and guarantees tracking of the reference trajectory, consequently achieving satisfaction of the task.
3.1 NEURAL-NETWORK LEARNING
As discussed in Section ! we are inspired by cases where systems undergo changes that modify their dynamics and hence the underlying controllers longer guarantee the satisfaction of a specific task. In such cases, instead of carrying out the challenging and tedious procedure of identification of the new dynamic models and design of new model- based controllers, we aim to exploit data from off off-line system trajectories and develop an online policy that is able to adapt to the aforementioned changes and achieve the task expressed via Pa- Consequently, we assume the existence of offline data from a finite set of T system trajectories that satisfy collection of tasks, corresponding to bounded reference trajectories, including Pa. and possibly produced by systems with different dynamic parameters. The data from each trajectory i E (1,...,T) T comprise finite set of triplets {T, (t), i, us (1) where Ti is a finite set of time instants, H. (1) E IR are system states, and u, (t) E R" are the respective control inputs. compliant with the dynamics (T). We use the data to train neural network in order to approximate the respective controller u(z,t), t). More specifically, we use the pairs (t) t)ee, as input neural network, and u. (4)eTT the respective output targets, for all trajectories i E (1..... T). For given T E R2n 1 E Ry0, we denote by ( t) the output of the neural network. Note that the controller u( t), which the neural network aims to approximate, is not associated to the specific task expressed via and mentioned in Section 2 but collection of several tasks. Therefore, we do not expect the neural network learn how to track Pa, but rather to be able to adapt to the entire collection of tasks. This is an important attribute of the proposed scheme, since it can generalize over tasks. The motivation for training the neural network with different tasks and dynamic parameters is the following. Since the tasks correspond to bounded trajectories, the respective stabilizing controllers compensate successfully the dynamics in ① Therefore, as will be clarified in the next section, the neural network aims to approximate an "average" controller the retains this property, i.c., the boundedness of the dynamics of [ By using such approximation, the online feedback-contre policy- illlsstraee in the next section is able to guarantee tracking of Pa without using any explicit information on the dynamics.
3.2 FEEDBACK CONTROL DESIGN
we do not expect the neural-network controller to accomplish tracking of Pa. since the system (1) is trained on potentially different tasks and different system parameters, and
As mentioned in Section 3.1
(2) the neural network provides only an approximation of a stabilizing controller, potential deviations
in certain regions of the state space might lead to instability. Moreover, the neural-network controller has no error feedback with respect to the open-loop trajectory Pa: such feedback is substantial in the stability of control systems with dynamic uncertainties. Therefore, this section is devoted to the design of a feedback-contro policy to track the trajectory pa(1) by using the output of the trained neural network (see Fig- D. The goal is to drive the error e x Pa zero. As mentioned in Section 3.T the motivation for training the neural network with several different tasks and dynamic parameters is the learning of controller that able retain the property of the training controllers to compensate the system dynamics ( This is officially stated in the following assumption regarding the closed-loop system trajectory that is driven by the neural network's 's output.
Assumption 2. The output Unn (ā, t) of the trained neural network satisfies
|| t) + g t) t)u (I, (I,t))l < d||>|| + +B
for positive constants d, for all ă E 2" 2 0.
Intuitively, Assumption 2 states that the neural-network controller (m, t) is able to maintain the boundedness of the system state by the constants d, B, which are considered to be unknown. The assumption is motivated by the property of neural networks approximate continuous function
4

---

Under review as a conference paper at ICLR 2022
(ps, Pa)
Online Control
Policy
Unn
Neural Network
(x,
System Dynamics
Figure Block diagram of the proposed algorithm.
arbitrarily well in a compact domain for a large enough number of neurons and layers CCyhenko
1989 As mentioned before, since the collection of tasks, which the neural network is trained with, correspond to bounded trajectories, the system states are expected to remain bounded. Since f(z, t), and ( t) are continuous in z and bounded in f. they are also expected be bounded as per (2. Contrary to the related works (e.g.. (Vamvoudakis & Lewis (2010); Yang et al. (2220); [Cheng al.] (007): Fan et al. (2018)), however, we do not adopt approximation schemes for the system dynamics and we do not impose restrictions on the size of d, B. Moreover, Assumption 2 does not
imply that the neural network controller Unn ( ()) guarantees tracking the open-loop trajectory Pa- Instead, it is merely growth condition. Additionally, note that inequality 2 does not depend specifically on any of the tasks that the neural network is trained with. We exploit this property in the control design and achieve task generalization; that is, the open-loop trajectory Pa to be tracked (corresponding to the task ()) can be any of the tasks that the neural network is trained with.
We now define the feedback-control policy. Consider the adaptation variables é,, f2, corresponding to upper bounds of d, B in with (0) > 0, (0) > O. We design first a reference signal for å as
Fd
Au kjc,
(3)
that would stabilize the subsystem ||]]?. where L is positive control gain constant. Following the back-stepping methodology (Krstic al. (1995)). we define next the respective error e, 1a and design the ceural-network-band adaptive control law as
u(t é1, 8, 1) = Unn(T () - kize -lies - EEgs
(4a)
? ke, ||c,112. ig = ke ||clll
(4b)
e. if ce F 0, and è, 0 if c. 0. The control where k2, ke, ke, are positive constants, and é. |e-ll
design is inspired by adaptive control methodologies (Krstic et al. (1995)). where the time-varying gains (1). & (t), adapt to the unknown dynamics and counteract the effect of and in (2) in order to ensure closed-loop stability. Note that the policy (3) (4) does not use any information on the system dynamics f(-). g()) or the constants B, d. The tracking of pa is guaranteed by the following theorem.
Theorem 1. Let a system evolve according to 7 and let an open-loop trajectory Pa -R20 R encoding a user-defined lask. Under Assumption 2 the control algorithm guarantees lim; (e(t), (t) = 0, as well as the boundedness of all closed-loop signals.
We provide sketch of the proof here and give the details in Appendix
Proof (Sketch), view of 2 one can find positive constants di, d2, D such that
Ige + f(z, 1) + g(, () ( (z, -all S dillel| + da d.lle..l + D,
For simplicity, we consider that holds glohally, but it he extended to hold in compact set.
5

---

Under review as a conference paper at ICLR 2022
for all E RR²", 2 0, where g is the minimum eigenvalue of g which is positive owing to the positive definitiveness of g(-). The constants d1, d2, and D define upper bounds & and lz. which we aim approximate by the adaptation variables & and ég. respectively. We next define the continuously differentiable function
(
|ell
11e,11+
Σ xE(1.2)
(2e 2,)
2g
2ke.
which represents an energy-like function of the system, and which we wish to drive to zero. One can prove, by analyzing the derivative V and using the control policy 4 that V non-decreasing
along the solutions of the closed-loop system and that lime-so V(I) 0, which implies that lim e(t) = limy-soo (t) = 0.
Note that, contrary to works in the related literature (e.g., (Verginis & Dimarogonas 2020) Bech lioulis & Rovithakis (2014))) we do not impose reciprocal terms in the control imput that grow unbounded in order to guarantee closed-loop stability. The resulting controller is essentially simple linear feedback on (e(t), (t)) with time-inrying adaptive control gains, accompanied by the neural network output that ensures the growth condition (2). The positive gains k1. K2, kēz ke, do not affect the stability results of Theorem but might affect the evolution the closed-loop system; e.g., larger gains lead faster convergence but possibly larger control inputs.
The proposed control algorithm does not require any of the long- long-standing assumptions on the system dynamies (I). such linear parameterization. growth conditions, boundedness by known functions
(e.g.. Krstic et al. (1955); Hong et al. (2009); Chen (2019) Huang et al. (2018a) Kanellakopoulle
al. (1991); Xu & loannou (2003) Ge & Wang (2004); Wen al. (2011) Chen et al. (201 Slotinn
Li (987)) ; Slotine & Coetsee 1986): Chen et al. ( Slotine & Coetsee ( (1986); Chen et al.
(2008)) Xian et al. (2004)). Additionally, we do not assume the boundedness of the solution of
or of the dynamic terms [', f), g(t, t); instead, the control algorithm guarantees via Theorem ] the boundedness of the system state as well as the asymptotic tracking of Pu(t). The only boundedness condition that we require is ( in Assumption 2 which can be accomplished by neural-
component in view of the universal approximation property Cybenko (1889)
4 NUMERICAL EXPERIMENTS
This section is devoted series of numerical experiments. More details can be found in Appendix We first test the proposed algorithm on 6-dof UR5 robotic manipulator with dynamics
म = B(x) (u - C(z)+ g(x) + d(z, t))
(5)
where x, i E E6 are the vectors of robot joint angles and angular velocities, respectively; B(x) e Rex6 the positive definite inertia matrix, C(z) E R is the Coriolis matrix, g(x) R is the gravity vector, and d(ã, t) E IR is a vector of friction terms and exogenous time-varying disturbances. The workspace consists of four points of interest Ti, T2.
Ta. T4 (end-effector position and Euler-angle orientation). as depicted in Fig. 2 which correspond to the joint-angle
vectors C1. C2. C3. Cq. More information is provided in Appendix[A] We consider a nominal task expressed via the spatio-temporal constraint NE G(o.oo)FF (1|æ11 cal S 0.1), where G and are the always and eventually operators respectively. The task consists of visits of æ to e E 6 (within the radius 1) infinitely often within the time intervals dictated by 4, for i {I....,4).
T
We create 150 problem instances by varying the positions of the time intervals 4, the dynamic parameters of the robot (masses and moments inertia of the robot's 's links and actuators), the friction and disturbance term d(-), the initial position and velocity of the robot, and the sequence of visits the points G, dictated by qi i.e., instance
Figure 2: A URS robot in workspace with four points of interest T, E {1....,4},

---

Under review as a conference paper at ICLR 2022
x x10 12
+ |(ē(t)
2000
8
Proposed
Non-adaptive
No-NN
1500
6
mean meean(]ee e|e((t)]l
1000
4
500
2
0
0
0
10
20
10
20
0
10
20
t sec
x
x10
12
0.6
2000
10
(td( + ((((t))))
Proposed
Non-adaptive
No-NN
8
1500
0.4
6
1000
4
0.2
500
2
0
0
10
20
10
20
10
20
t
[sec]
Figure 3: Mean (top) and standard deviation (bottom) [|e(t]] + [|ě(()]] for the proposed (left), non-adaptive (center), and no-neural-netwon (right) control policies.
might correspond the visit sequence ((x(0).)) (c1, + (cz, 112) - (cg,tla) - (c4, tr.). and another to ((z(00,, 0) (c3, (1,) + (1, + (c4,414) (c2,41z). We separate the aforementioned 150 problem instances into 100 training instances and 50 test instances. We generate trajectories using the 100 training instances from system runs that satisfy different variations of one cycle the task (i.c., one visit to each point). Each trajectory consists of 500 points and is generated using nominal model-based controller. We these trajectories train neural network and we test the control policy (4) in the 50 test instances. We also compare our algorithm with the non-adaptive controller uz( t) Unn (an, (xr,t kie kzě, as well with a modified version uz(æ of (4) that does not employ the neural network (i.e., the term Wan (2, t)). The comparison results are depicted in Fig. 3 which depicts the mean and standard deviation of the signal ||e(t]] [|é(t)]) for the 50 instances and 20 seconds. It is clear from the figure that the proposed algorithm performs better than the non-au adaptive and no-neural-netwon and steady-state error. It is worth noting that the non-adaptive policy results on average in unstable closed-loop system.
policies both in terms of convergence speed
We next test the proposed algorithm, following similar procedure, on unicycle robot (the respective dynamics and control algorithm can be found in Appendix [B. Fig. depicts the mean and standard deviation of the errors ea(t), et(() for 50 test instances. We note that the performance of the no- netwook control policy is much more similar to the proposed one than the URS case. This be attributed to (1) the lack of gravitational terms in the unicycle dynamics, which often lead to instability, and (2) the chosen control gains of the no-neural-netwe policy, which are sufficiently large to counteract the effect of the dynamic uncertainties.

---

Under review as a conference paper at ICLR 2022
Finally, compare the performance of the proposed control policy with the re- ported data of (Wang et al. (2019)) on the benchmarking enivronment of the pen- dulum, where single-link mechanical structure aims to reach the upright position. Following (Wang et al. (2019)), the reward and costs are [pena(t) = -cos 0.14(t)
0.1 sin (t))
0.001u(t)2 and Jpend Similar to the previous cases and in contrast to
respectively.
Wang et al. (2019).
generate 150 instances by varying the system parameters (pendulum length and mass). the external disturbances, and the initial conditions. We generate 100 trajec- tories, consisting of 500 points each, for the 100 training instances, by employing a nomi- nal controller, and we use them train a neural network. Moreover, in order to guarantee the feasibility of the proposed algorithm 4 we con- sider larger control-action bounds than in (Wang ct al. (2019)). The original bounds render these systems under-actuated, which is not included in the considered class of systems T and con- sist part of our future work. It should be noted that such larger bounds affect negatively the ac- quired rewards. We test the control policy (4) on the 50 test instances, for 5000 steps each, corre- sponding 10 seconds. We H 200 and
cueum(pewardde (Ce
stull (reward)))))) (()
100
(ean(coutt())
std(ccut(())
-200
[sec]
Figure 4: Mean and standard deviation of the re- ward and cost for the pendulum environment.
(Wang et al.) 2019)). Fig. 4 depicts the
mean and standard deviation of the varying reward and cost functions, illustrating successful regulation to the upright position. After 5000 steps, we obtain a mean reward and cost of 0.99 and 200, respectively, showing better performance than the reported cost of 180 in (Wang et al. (2019)) Moreover, the proposed control algorithm achieves this performance without resorting to exhaustive exploration of the state-space, which is the case in the reinforcement-leate learning algorithms used in (Wang et al. (2 This is very important property in practical engineering systems, where safety is of paramount significance and certain areas of the state space must be avoided.
DISCUSSION AND LIMITATIONS
As shown in the experimental results, the control algorithm is able asymptotically track reference trajectories which were not considered when generating the training data. Similarly, the trajectories used in the training data were generated using systems with different dynamical parameters, and specifically the ones used in the tests. The aforementioned attributes signify the ability of the proposed algorithm to generalize to different tasks and systems with different parameters. Nevertheless, the proposed control policy is currently limited by systems satisfying Assumption [ and is not able to take into account under-actuated systems (e.g., the acrobot or cart-pole system); such systems consist part of our future work. Moreover, the proposed algorithm is dependent on the performance of the learning component to satisfy Assumption [ Lack of satisfactory training might cause the neural-network output to add extra terms in the dynamics that render the closed-loop system unstable. less conservative and more robust approach consists of using the off-line data to train multiple neural networks, each for certain region of the state space; such approach constitutes part of our future work. Finally, the discontinuities of 44. (12) might be problematic and create chattering when implemented in real actuators. A continuous approximation that has shown to yield satisfying performance is the boundary-layer technique (Slotine et al. (1991).
CONCLUSION AND FUTURE WORK
We develop novel control algorithm for the control of robotic systems with unknown nonlinear dynamics subject to tasks expressed as reference trajectories. The algorithm integrates neural network- based learning and adaptive control. We provide formal guarantees and perform extensive numerical
8

---

Under review as a conference paper at ICLR 2022
Proposed
mean (e. (t)))) [m]
Non-adaptive
No-nn
Proposed
Non- con-adaptive
No-nn
stdl
t [sec]
0.1
mean mean( (3()) [rad]
(td( Irad]
0.05
0.4
0.3
Proposed
Proposed
-0.05
0.2
Non-adaptive
Non-adaptive
-0.1
No-mn
0.1
No-nn
-0.15
0
10
15
20
0
10
15
20
t
[sec]
Figure 5: Mean (left) and standard deviation (right) of ea(t), (tt) for the proposed, non-adaptive, and no-neural-netwer control policies.
experiments. Future directions will focus on relaxing the considered assumptions and extending the proposed methodology to underactuated systems.
REPRODUCIBILITY STATEMENT
We provide the proof of Theorem in Appendix A Additionally, we elaborate on the required assumptions (Assumptions 22 throughout the text see pages 3, 5, and 7. Finally, we provide implementation details, instructions, and the respective code, required reproduce the results, in Appendix and the supplementary material.
REFERENCES
Charalampos P Bechlioulis and George A Rovithakis. A low-complexity global approximation-fii -feee control scheme with prescribed performance for unknown pure feedback systems. Automatica, 50 (4):1217-1226, 2014.
Thomas Berger, Huy Hoång Lê, and Timo Reis. Funnel control for nonlinear systems with known strict relative degree. Automatica, 87:345-357, 2018.
Felix Berkenkamp and Angela P Schoellig. Safe and robust learning control with gaussian processes. European Control Conference (ECC), pp. 2496-2501, 2015.
Dimitri Bertsekas. Lessons from alphazero for optimal, model predictive, and adaptive control. arXiv preprint arXiv:2108. 10315, 2021.
Dimitri Bertsekas and John Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, MIT, 1996.
Mingyu Cai, Mohammadhosein Hasanbeig, Shaoping Xiao, Alessandro Abate, and Zhen Kan. Modular deep reinforcement learning for continuous motion planning with temporal logic. arXiv preprint arXiv:2 102. 2021.
Alberto Camacho and Sheila Mcllraith. Towards neural-guided program synthesis for linear temporal logic specifications. arXiv preprint arXiv: 19/2. 2019.

---

Under review as a conference paper at ICLR 2022
M Capotondi, G Turrisi, C Gaz, Valerio Modugno, Giuseppe Oriolo, and A De Luca. An online learning procedure for feedback linearization control without torque measurements. Conference on Robot Learning, pp. 1359-1368, 2020.
Jian Chen, Aman Behal, and Darren M Dawson. Robust feedback control for class of uncertain mimo nonlinear systems. JEEE Transactions on Automatic Control, 53(2):591-596 2008.
Mou Chen, Shuzhi Sam Ge, and Beibei Ren. Adaptive tracking control of uncertain mimo nonlinear systems with input constraints. Automatica, 47(3):452-465, 2011.
Zhiyong Chen. Nussbaum functions in adaptive control with time-varying unknown control coeffi- cients. Automatica, 102:72-79, 2019.
Tao Cheng, Frank Lewis, and Murad Abu-Khalaf. A neural network solution for fxed-final time optimal control of nonlinear systems. Automatica, 43(3):482-490, 2007.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 303):43,, 1989.
Kenji Doya. Efficient nonlinear control with actor-tutor architecture mozer, jordan, petsche (eds.) advances in neural information processing systems 9, 1997.
Bo Fan, Qinmin Yang. Xiaoyu Tang, and Youxian Sun. Robust adp design for continuous -imee nonlinear systems with output constraints. IEEE transactions on neural networks and learning systems, 2966222127- 2018.
Nicholas Fischer, Rushikesh Kamalapurkar, and Warren E Dixon. Lasalle-yoshizawa corollaries for nonsmooth systems. IEEE Transactions Automatic Control, (8(9))2333-23333 2013.
Shuzhi Sam Ge and Cong Wang. Adaptive neural control uncertain mimo nonlinear systems. IEEE Transactions on Neural Networks, 15(3):774-692, 2004.
Christopher Hahn, Frederik Schmitt, Jens U Kreber, Markus N Rabe, and Bernd Finkbeiner. Teaching temporal logics to neural networks. arXiv preprint arXiv:2003. 04218, 2020.
Hong, SS Ge, Ren, and TH Lee. Robust adaptive control for a class of uncertain strict-feedback nonlinear systems. International Journal of Robust and Nonlinear Control: IFAC-Affiliate Journal, 19(7):746-767, 2009.
Haimin Hu, Mahyar Fazlyab, Manfred Morari, and George J Pappas. Reach-sdp: Reachability
analysis of closed-loop systems with neural network controllers semidefinite programming. IEEE Conference Decision and Control (CDC), pp. 5929-5934, 2020.
Jiangshuai Huang, Wei Wang, Changyun Wen, and Jing Zhou. Adaptive control of a class of strict- feedback time-varying nonlinear systems with unknown control coefficients. Automatica, 93: 98-105, 2018a.
Xiucai Huang, Yongduan Song, and Junfeng Lai. Neuro-adaptive control with given performance specifications for strict feedback systems under full-state constraints. IEEE transactions on neural networks and learning systems, 30(1):25-33) 2018b.
Edouard Ivanjko, Toni Petrinic, and Ivan Petrovic. Modelling mobile robot dynamics. 7th EUROSIM Congress Modelling and Simulation, 2, 2010.
Radoslav Ivanov, James Weimer, Rajeev Alur, George Pappas, and Insup Lee. Verisig: verifying safety properties of hybrid systems with neural network controllers. Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, pp. 169-178, 2019.
Achin Jain, Truong Nghiem, Manfred Morari, and Rahul Mangharam. Learning and control using gaussian processes. ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS), pp. 140-149, 2018
Girish Joshi, Jasvir Virdi, and Girish Chowdhary. Asynchronous deep model reference adaptive control. Conference Robot Learning, 2020.
10

---

Under review as a conference paper at ICLR 2022
Rushikesh Kamalapurkar, Huyen Dinh, Shubhendu Bhasin, and Warren E Dixon. Approximate optimal trajectory tracking for continuous- time nonlinear systems. Automatica, 51:40 48, 2015.
Ioannis Kanellakopoulos, Petar V Kokotovic, and A Stephen Morse. Systematic design of adaptive controllers for feedback linearizable systems. American control conference, pp. 649-654, 1991.
Bahare Kiumarsi, Kyriakos G Vamvoudakis, Hamidreza Modares, and Frank L. Lewis. Optimal and autonomous control using reinforcement learning: A survey. IEEE transactions on neural networks and learning systems, 29(6):20422262,, 2017.
M. Krstic, I. Kanellakopoulos, and P. Kokotovic. Nonlinear and Adaptive Control Design. Publisher: Wiley New York, 1995.
Kevin Leahy, Eric Cristofalo, Cristian-Ioan Vasile, Austin Jones, Eduardo Montijano, Mac Schwager. and Calin Belta. Control in belief space with temporal logic specifications using assion-base localization. The International Journal of Robotics Research, (8(6):702-722) 2019.
Lars Lindemann, Christos K Verginis, and Dimos Dimarogonas. Prescribed performance control for signal temporal logic specifications. IEEE Conference Decision and Control (CDC), pp. 2997-3002, 2017.
Ke Liu, Robert Tokar, and Brain Mc An integrated architecture of adaptive neural network control for dynamic systems. Advances in neural information processing systems, 7:1031-1038, 1994.
Wenliang Liu, Noushin Mehdipour, and Calin Belta. Recurrent neural network controllers for signal temporal logic specifications subject to safety constraints. IEEE Control Systems Letters, 2021.
Meiyi Ma, Ji Gao, Lu Feng, and John Stankovic. Stinet: Signal temporal logic enforced multivariate recurrent neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Emilio Tanowe Maddalena, CG da Moraes, Gierri Waltrich, and Colin N Jones. A neural network architecture to learn explicit mpe controllers from data. IFAC-PapersOnLLine, 13(2):11362-1111111 1362-11367, 2020.
Lipo Mo, Xiaolin Yuan, and Yongguang Yu. Neuro-adaptive leaderless consensus fractional-order multi-agent systems. Neurocomputingg 339:17-25, 2019.
Julian Nubert, Johannes Köhler, Vincent Berenz, Frank Allgöwer, and Sebastian Trimpe. Safe and fast tracking on a robot manipulator: Robust mpe and neural network control. IEEE Robotics and Automation Letters, 5(2):3050-3057. 2020.
Brad Paden and Shankar Sastry. calculus for computing filippov's differential inclusion with application to the variable structure control of robot manipulators. IEEE transactions circuits and systems, 34(1):73-82, 1987.
Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, al. Logical neural networks. arXiv preprint arXiv:2006. 33155, 2020.
Ankit Jayesh Shah, Pritish Kamath, Shen Li, and Julie A Shah. Bayesian inference of temporal task specifications from demonstrations. 2018
Bruno Siciliano, Lorenzo Sciavicco, Luigi Villani, and Giuseppe Oriolo. Modelling, planning and control. Advanced Textbooks in Control and Signal Processing. Springer, 2009.
J-JE Slotine and JA Coetsee. Adaptive sliding controller synthesis for non-linear systems. Interna- tional Journal of Control, 1636):663111651 1986.
Jean-Jacques E Slotine and Weiping Li. On the adaptive control of robot manipulators. The international journal of robotics research, 6(3):49-59, 1987.
Jean-Jacques Slotine, Weiping Li, et al. Applied nonlinear control, volume 199. Prentice hall Englewood Cliff's, NJ, 1991.
11

---

Under review as a conference paper at ICLR 2022
Chuangchuang Sun and Kyriakos G Vamvoudakis. Continuous-timm safe learning with temporal logic constraints in adversarial environments. American Control Conference (ACC), pp. 4786 4791, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Kyriakos G Vamvoudakis and Frank L. Lewis. Online actor-critic algorithm solve the continuous- time infinite horizon optimal control problem. Automatica, 46(5):878 888, 2010.
Christos Verginis and Dimos V Dimarogonas. Asymptotic tracking of second-order nonsmooth feedback stabilizable unknown systems with prescribed transient response. IEEE Transactions on Automatic Control, 2020.
Christos K Verginis and Dimos V Dimarogonas. Timed abstractions for distributed cooperative manipulation. Autonomous Robots, 42(4):781-779, 2018.
Christos K. Verginis, Dimos V. Dimarogonas, and Lydia E. Kavraki. Kdf: Kinodynamic motion
planning via geometric sampling-based algorithms and funnel control. 2021.
Draguna Vrabie and Frank Lewis. Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Networks, 22(3):237-265 2009.
Chuanzheng Wang. Yinan Li, Stephen L Smith, and Jun Liu. Continuous motion planning with temporal logic specifications using deep neural networks. arXiv preprint (rxxiv2004.026/10 2020.
Tingwu Wang. Xuchan Bao. Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang. Pieter Abbeel, and Jimmy Ba. Benchmarking model- based reinforcement learning. arXiv preprint arXiv: ww0.: 02057, 2019.
Changyun Wen, Jing Zhou, Zhitao Liu, and Hongye Su. Robust adaptive control of uncertain nonlinear systems in the presence of input saturation and external disturbance. IEEE Transactions Automatic Control, 56(7):1672-1673 2011.
Bin Xian, Darren M Dawson, Marcio S de Queiroz, and Jian Chen. A continuous asymptotic tracking control strategy for uncertain nonlinear systems. IEEE Transactions on Automatic Control, 49(7): 1206-1211, 2004.
Haojian Xu and Petros A loannou. Robust adaptive control for a class of mimo nonlinear systems with guaranteed enror bounds. IEEE Transactions on Automatic Control, 48(5):72--742, 2003.
2003.
Ruixuan Yan and Agung Julius. Neural network for weighted signal temporal logic. arXiv preprint arXiv:2104.05435, 2021.
Yongliang Yang, Kyriakos G Vamvoudakis, and Hamidreza Modares. Safe reinforcement learning for dynamical games. International Journal of Robust and Nonlinear Control, 30(9):3706-3726, 2020.
Ssu-Hsin Yu, Anuradha M Annaswamy, DS Touretzky, MC Mozer, and ME Hasselmo. Neural control for nonlinear dynamic systems. Advances in Neural Information Processing Systems, pp. 1010-1016, 1996.
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4): 1307-1319, 2020.
Jingang Zhao and Minggang Gan. Finite-horizon optimal control for continuous-tim uncertain nonlinear systems using reinforcement learning. International Journal of Systems Science, 51(13): 2429-2440, 2020.
Yaofeng Desmond Zhong and Naomi Leonard. Unsupervised learning of lagrangian dynamics from images for prediction and control. Advances in Neural Information Processing Systems, 33, 2020.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
We provide here the proof Theorem systems with discontinuous dynamics.
1
We first give some preliminary notation and background on
NOTATION
Given function f : + RR its Filippov regularization is defined as (Paden & Sastry (1987)
K[A(x)) = n
())
(6)
620 p(N)=D
where (=)= is the intersection over all sets N of Lebesgue measure zero, co E)) is the closure of the convex hull EE) of the set E,, and ((x, 5) is the ball with radius å centered at
NONSMOOTH ANALYSIS
Consider the following differential equation with a discontinuous right-hand side:
i == f(r
(7)
where D x 1/. x6) - R", C RR, is Lebesgue measurable and locally essentially bounded.
Definition (Def. of [Fscherr et al. (2.13)). A function x (1) - RR" with t >> lo, is called Filippov solution of 7 on [to. 1 if xtt) is absolutely continuous and if. for almost all t [ttort), satisfies v E (if)(z,
(z, t). where [zfl t) is the Filippov regularization of fx,tt.
Lemma (Lemma of (Fischer al. (013))).. Let x(1) be Filippov solution of 7 and V D [1o,41 (1) > IR be locally Lipschitz, regular function Then V (x(((),)) 1) is absolutely continuous, 8e Vzx(t), t) exists almost everywhere (a.e.), ie., for almost all E [to,t]), and ((x(t),t) ae ()) where
(=((t,,) a
ET
(EOV(a.t)
and ÖV(x, (x, f) Clarke's generalized gradient (ar, t) Fischer et 2013)).
Corollary (Corollary 2 of (Fischer al.] (2013))) For the system given in 77). let 'D C R. be an open and connected set containing x O and suppose that f is Lebesgue measurable and + f(z, is essentially locally bounded, uniformly in t. Let V D [to,ti) + IR be locally Lipschitz and regular such that (1(x) S V(x, t) I W2(x), VE [tu, t2), D, and
5 >w(((),,, VZ ()((),), LE (1,,41, ED.
where Wi and W2 continuous positive definite functions and W is continuous positive semi- definite on D. Choose r > and e such that B(0, r) C D and e < minj Wi(x) (x). Then for all Filippov solutions x [t. tn) + R of 7 with x x(to) E D = {r E B(0,r) W2(x) S c),
holds that 1 co, x(t) Dj Vt E [fo, 20), and lim W ((x(t)) 0.
Proof of Theorem 7 We we-weite first the condition of Assumption 2 Note that pa(() and its deriva- tives are bounded functions of time. Moreover, since = x Pa. è = Pa. it holds that | < ||e|| + |é|| ||aal |aall and Un Pa kjě, well e, = I E. k,c. Therefore, in view (). one can find positive constants d1, dz, D such that
+ kie implying
Ige + f(z, + t) (z, t) rall s di||el| + dzlle, + D. g
(8)
for all E E 2", 2 0. where g is the minimum eigenvalue of ( t), which is positive owing to the positive definitiveness of g()). Inequality (8) will be used later in the proof.
See Fischer al. 2013 for definition of regular functions.
13

---

Under review as a conference paper at ICLR 2022
Let now constant such that dia k1- As will be clarified later, the adaptation variables f1 di + d2 and D. respectively. Therefore, let 6 = 2n
2
&g aim to approximate the constants
d. - d2.
2n
l2 == D, and the respective error terms & == l l1 f2 = l f2, as well as the overall state == [eT ,, f Ed]T R2*+2 Since the control policy is discontinuous, we use the notion of Filippov solutions. The Filippov regularization of u is K[u) = u t) kge, - le. - EgE.. where E.- ||_T| if e, o 0 and E. E (-1,1) otherwise. Note that, in any case, it holds that e, E. - |
Let now the continuously differentiable function
V +
|ell
1
Σ
4
2g
2h
(E(1,2)
which satisfies W
( for W (3) min
| (2
2g Ekr
2kry
max ( 1
2g Zkz, 3kr, H|p According to Lemma
(ā) E
V(),, with V = Neearaa) K[
Since V (ā) is continuously differentiable, its generalized gradient reduces to the standard gradient
and thus it holds that V()) = VV VVTK(a], where VV VV= =V=(e" [cT LeT
11 Keg
[T] By differentiating
Keg
V and using (3), one obtains
1 c. (f(x,t) + (), da) V. CW, e (+ pa)
1 17 E +
g
kit
kitz
and by using c2 + va and substituting the control policy 3p. ④4. and inequality 18)
W, = - 1 --1llellf
+ e (ge + f(z,t) t) + g(ā, (z, iu) cI. (koee + + + EgE.)
_
+ +Elleo + E2lle11
1 1 g6t (( + +2E.)
S < + di dille_llle|| + delle,11 DIlcoll
g
+ + 12 |eelll
Note from (4) and the fact that (0) > 0, (2( > 0 that (t) > 0 and (t) > 0 for all t 2 0. Moreover, recall that g(.,t) is positive definite for all æ E R²n. t 2 O, with g being its minimum
(and positive) eigenvalue. Therefore, we conclude that W, becomes
W. S kn /|el| + di [|ee|llle|| + dalle=11? DI|e= (K2 + &.)le..11 &|le_ll & ||e 2 Ea Eelle.l
By incorporating the term a in the term di ||e> ||]le]] we obtain di |||e|| = dia ||=-lll| ||e|| and by using the property ab < fa²² + jb² for any constants a.b, we obtain dia [e_- di ||e..112. Therefore, W, becomes
|el| s 41 1|e
a
d1a)
|ell + (
di
W. <
ki
(k2 + 1e) Ealeelll + Ealle.1l + Ealleull + dz /leulf + Dile. - (k2
2
2a
dia
kı
|ell + filleul1* + Balleall (k2 + E.)l &lle. Elle. 112 &
2
dia 2 ||e|| ka]lev -(())
Therefore, holds that Ç < -(Ø)) for all ( V. with Q being continuous and positive semi-definite function in 2n since kn dia > O. Choose now any finite >> 0 and let 2
e < minjaii= (T) Hence, all the conditions of Corollary I are satisfied and hence, all Filippov solutions starting from (())) E 22 == (ā E B(0,r) : W2( S c} are bounded and remain in Uy. satisfying lim Q()(t)) O. Note that r, and hence c, can be arbitrarily large allowing and finite initial condition (()). Moreover, the boundedness of implies the boundedness of é, ev. Z, and 22 and hence of (t) and (z))), for all t E Rzo- In view of (5), we finally conclude the boundendess of u(-), é, and é, for all L E Rzo. leading to the conclusion of the proof.
14

---

Under review as a conference paper at ICLR 2022
Cd
R
Figure 6: unicycle vehicle.
B APPENDIX
Extension to non-holonomic unicycle dynamics
As mentioned in Section I the dynamics do not represent all kinds of systems, with one particular example being when non-holonomic constraints are present. In such cases, the control law design (4) and Theorem I no longer hold. In this section, we extend the control policy to account for unicycle vehicles subject to first-order non-holonomic constraints. More specifically, we consider the dynamics
PI = cos o p22 = sin ø e x=
(9a)
M/ + fo(
(9b)
where [21..2.0]] Tj E R3 are the unicycle's position and orientation, (u,w) are its linear and angular velocity (see Fig. 6). = [0 01] E R² are its wheel's angular positions, and [un, uz]" E R² the wheel's torques, representing the control input. The unicycle vehicle is subject to the non-holonomic constraint é sin á P2 cos e = O, which implies that the vehicle cannot move laterally. Additionally, M 2x the vehiele's inertia matrix, which is symmetric and positive definite, and (o()) is a function representing friction and external disturbances. The velocities satisfy the relations
_(nn + 0z), w = 2 (ŌR 0z)), where and R are the wheels' radius and axle length, respectively. The terms r. R. M, and fo(-) are considered to be completely unknown. As before, the goal is for the vehicle's 's position == [pI,p2] to track desired trajectory pa = Ip4, Pa.2] E IR?. Towards that end, we define the error variables ei := P1 Pa Pall. well as the angle 3 measured from the the longitudinal axis of the vehicle, i... the unicycle's 's direction vector [cos Φ sin Φ), to the eтror vector lle,, e2] (see Fig. 6). The angle 3 can be derived by using the cross product between the aforementioned vectors, i.e., sin = [cos ci sin Φ) x [-e1, -c2] = en sin Φ e2 cos Φ. The purpose of the control design, illustrated next, is drive ed and 3 to zero. By differentiating the latter and using (9) well the
62 = P2 Pa.2, Cd
relations - Ed cos(Φ + 3). e2 = - sin @ + 3) (see Fig. 6 we derive
éæ U cos + Pa.1 cos( + 3) pa.2 sin( + 3)
(10a)
sin 3 pa.1 sin( (Ø 3) Pa.2
-
cos( + B)
(10b)
ed ea
ea
In view of 100 set reference signals for the vehicle's velocity
0y 1
(B)(B) (0a,1 cos(3 + Φ) + pa.2 sin (3 + Φ) + kses)
(11a)
sin ④ (0) cos( cos())ia.2
Wa (3)ed (3)(
ha tan B kaß
(11b)
where kg, kg are positive gains, aiming to create exponentially stable subsystems via the terms k and kgB We define next the respective velocity enrors e, va, eu wa and design the
15

---

Under review as a conference paper at ICLR 2022
adaptive and neural -et-nnk--wwseeee control input u(x, d,t) = [ #a=
Unn (3. t), with
sin B
ug == Es (Kr + fieer - lze, + cd cos B - B
(12a)
"D = Eubis (kw + (ww - lgew +
(12b)
è. kkeepda
è. -kwewad
(12c)
è = (e + ew)
i = k2 + (ew))
(12d)
where lor, lui & are adaptation variables (similar to 4 with (0) > Ea (0) > 0 and k. kws kis are positive gains, i E (1, eas with E {v, w). is defined as &a f if ca o and éa 0 otherwise. We re-state Assumption to apply for the unicycle analysis as follows.
Assumption 3. The oupput (nn 1) of the trained neural network satisfies [1, fo t)ll S d||=|| + B, for positive, unknown constants d, B.
Similar to assumption 2 assumption 3 is merely a growth-boundednes. condition by the unknown constants d and B. The stability of the proposed scheme is provided in the next corollary.
Corollary 2. Let the unicycle system (9) and let open-loop trajectory Pa R20 R..
Assume that B(t) (-B,8), Iøa,1 sin Ø Pa.2 cos Φl Cg011 sin 3 < ed02 for positive
stants B < x, 01,, 02 and all 0. Under Assumption 3 the control policy (I2) guarantees
som ((t), ew (t)) = O, and the boundedness of all closed-loop signals.
Proof of Corollary
2 The derivatives of ed. 8 in 110) can be written as
Ted [
v
Pd.1 cos + B) + pa.2 sin( + 8)
=
G
+
Pa. cos( + B) fi1 sin(d + 8)
where
cos
G =
It can be verified that the reference velocity signals, designed in 11 can be written
Pa,1 cos(g) (b + B3 pa.2sin(e + B) kaea]
Cd
G-1
wid
Pu. cos( B} + P..1 ea
sin( B) kg
ea
and therefore, by using the relations
= bn + va and = ew + was éa and can be written as
Ex |
kaea kag
[ep
- kiged kgB
E, cos
+
G
=
+
sin
(13)
ew
Cv
Cw
The control design follows the back-stepping methodology (Krstic et al.) (1955) ). Let, therefore, the function 1/1 = 2 (e3 + 32), which, after time differentiation and use yields
sin
V - kuce - ko.82 egei cos 3 + Be,
Bew
(14)
ed
We will cancel the two last terms of V using the control input 12
First, note that the inertia matrix of the system M, appearing in the unicycle dynamics (
M
M2
has the form (Ivanjko et al. (2110) M = M2
with Mi = mp?
+ Io.
Mi
1R
M2 (o+mdd)) and where Ig is the moment of inertia of the vehicle with respect to point AR2
C (see Fig. 6.. To the moment of inertia of the the wheels, and d is the distance of the between
16

---

Under review as a conference paper at ICLR 2022
point C and the vehicle's center of mass (p1. (P1.P2). P2). Therefore, the second part of the unicycle dynamics can be written as
MiRr + M20r, un + (0,n(3,() () MaOR + M101 un + (z,t)
with fo.n and fo.1 denoting the elements of fo- By summing and subtracting the aforementioned equations, we obtain
(M1 + M2) + 0L. =u ur + fo,n (ū,)
(15a)
(M1 (2R 01) u uy + R0, (o,
(15b)
From the definition of Ey and e it holds that é, v òg ew is Was which, by using the relations [On + f 2R (0n θ1) and 15 becomes
E. 2(M + M2) r
uR + un + (0, (z,) àia
(16a)
& 2R(M1 M2)
WR ur. + So, 1) so,((; Wa
(16b)
Define now = 2 MM lw = 2R M1 The adaptation terms L,, and lw, used in the control mechanism (1)). aim to approximate bu and lur respectively. Note that l, and ly are positive, and we define the errors E. = lu Ev la = lu Ey.
We we-write next the condition of Assumption 3 It holds that |||| V v S 2ea + |pa.|| |pa.2
Moreover, in view of II as well as the fact that 8 E (- -, C 3. $) where B is the bound of 3, stated in Corollary
holds that |u| I |e,.1 + |tal S |e, cas(B) (lpa,1l |Pa.2l + kaea). |w S
|ewl + |wal S lewl + cos (on + ka) + lig (3). where we also use the fact that | a Au. cos o1-
By also recalling that Φ moves on the unit circle, we conclude that Assumption becomes
[T + Umn (7, 1) & daea + d=| + d/eal + d/ew + D,
(17)
with da = d(2 + -ي dg = dks, and D = d(2pg + 2T + 3 (al + 20a + 1)), where Pa and Ty the upper bounds of pa,s Va,i, respectively, E [1,2].
Let now positive constants "aa and 71 such that kis > 2da and k2 > 2dg7/8 and define
da +
dg +
4
4d
(18)
7
7p
which is the constant to be approximated by f its derivation will be clarified later. Let also f2 = 2D, which we aim to approximate with the adaptation variable & We define hence the respective errors l1 2== 2 & and & & --f, l2 and define the overall vector == [ea. 3, eu, Cw. Pw, l, E2] R Since the control mechanism is discontinuous, we use again the notion of Filippov solutions. The Filippov regularization K[u] of differs from u only in the terms é, and éw. which are replaced by E,, defined as É, = Co. if e, * 0, and É, E (-1,1) otherwise, and É.,, defined as É [ea]
eu
Tew
ew * O, and É.,, (-1, 1), respectively.
Consider now the continuously differentiable and positive definite function
V(z) = V1 1 +
Lg
l
2
1
72
2
2
2k.
2h
2kg
iE (1.)
which satisfies W) S V
S (2
for W (8)
= min
2h 11
21
2
26,
28
2k
W2(T) max 2 3 24,
4. fy
1712 According to Lemma
v
2
24
26
17

---

Under review as a conference paper at ICLR 2022
with V = keeov K[a]- Since (T) is continuously differentiable, its generalized gradi-
ent reduces to the standard gradient and thus it holds that v - VV K where VV
lea, B, ev Ewr Ko kw
Lee Lew,
L ET.. Therefore, by differentiating V and employing 14
kg
and (15 we obtain
sin 3
V c W. = kady - kz3 - Edeo cos B + Seu
Bee + co co(uRR + un + fo + fo.l - tvds)
ed
1 Eglo + 1 Indl + Σ
1
+ e (uR uu + So. - fo (wwia)
+
ko
kiw
hig
sE(1,2)
By substituting the control and adaptation policy 12
W becomes
W- = -kaez kg.²² - ke kkge + e, cgly + ewiil li(e, : + + -22-le ul + lewl)
+ ev (u Unn.L + S0. + f0, + ew ("mm Una + so (o,
- Eswwiis + Ee(e" + + + E2(leol + l(wl)
where uR and an nn are the elements of unn. i.c., Una Ru U. .n It easy to conclude that Mon, + Lnn. + f0.R + fa. I |nnn. + fo.Rl + [nn,L + fe.L. < 2 Un + fell and |unn Unn,
fo. fo.L| S nt + f0 + [u,,LL L + fo,6| 2m + foll and hence, in view of 17
Man, + Unn, + f0.R= + fo.z|
< = 2dgee + 2dg|81 + 2d + 2d + 2D
I I.n. + So.R - -sozl
Therefore, W becomes
W = vV= kae3 kz/32 - k kzd - (e + +x)) - -22(1eo + (c
D(leol + lea))
+ (c + +x²)= +22lcoo + lewl)
The term D((eal + le yields
D(leol + ((ml) 22daealer + 2dg1B/leed + 2de + 4djemllewl +DDeeul + 2dgealewl
+ +2dp|3]lewl + 2de + 2
By setting 2dgaalevl 2daraea Isal /dwe 2daraed |لع|, and 2dg|Bllenl 2d (1 7a
lew.
2dg[ = 2d 81 Neع1 and completing the squares, one obtains
D(levl + lewl) 22d + 2 + (d + ds + 4d| (et + E + +2D(le + lewl)
da
na
7a
=2d + 2 + lile + + + +zlleell + +lew)
Hence, W becomes
W < - (ks- 2ddra)e - -kg-2kkpp 2d ko kze - (e + + - -Ee(levl + lewl)
+ (e + ex) -ba(llell lew)) + Ei(eet + 12(e,l + (ewl)
(ks 1la-2ada 2d (kg 2d )2 ko k3c =: Q(T)
Therefore, it holds that ( < -Q()), for all ( E V. with Q being a continuous and positive semi- definite function in R2$ since kis 2d > 0, ki 2d > 0. Choose now any finite r > 0 and let
c < min W1(3) (T). Hence, all the conditions of Corollary I are satisfied and hence, all Filippov
solutions starting from (()) E 22 == (T E r) W2(T) < c} are bounded and remain in Py. satisfying lim ((*(1) = 0.
Note that r, and hence c, can be arbitrarily large allowing and finite initial condition 2(0). Moreover, the boundedness of implies the boundedness of e, é, e,, Eg and Eg, and hence of (1) and (t), for all t E R220 Finally, in view of the conditions |sin drfu, cas drpa. < ai and sin / < a2, one
concludes the boundedness of all closed-loop signals, for all 12 0
18

---

Under review as a conference paper at ICLR 2022
The assumptions [ua.1 sin Pa.2 cos Φ Cd01, sin B < cd02 imposed avoid the singularity of ea = 0: note that B and W/ are not defined in that case. Intuitively, they imply that ed will not be driven to zero faster than 3 or Pa. sin Ø Pa. cos cos o; the latter becomes zero when the vehicle's velocity vector v aligns with the desired one pa- In the experiments, we tune the control gains according to kg 10kg in order satisfy these assumptions.
C APPENDIX
We provide here more information on the experimental results Section 4 All the results were obtained on MATLAB environment using the ODE simulator. We performed the training of the neural networks in with the pytorch library in Python environment. The neural networks consist of 4 fully connected layers of 512 neurons; each layer is followed by batch normalization module and ReLU activation function. For the training we use the adam optimizer and the mean-square-erre loss function. In all cases choose batch size of 256, and we train until desirable average (per batch) loss of the order of 10-4 4 is achieved. All the values of the data used for the training were normalized in [0, 1).
Regarding the robotic-arm task, the [-0.15, 0.475, 0.675, g 0, T2
T [-0.6, 0, 2.5, 0, - - q. C1 = 1-0.07.-1000 0.45, 2.3, 2.3,, 1.37. -133
points of interest are set
as
T3
and T4 = corresponding joint-angle vectors as
[-0.025, 0.595, 0.6, -2,6--2,00 0, -,
[-0.525, 0.55, 0. 0, 0,-51 and
the
22 [1.28, 0.35, 1.75, 0.03, 0.1, 11.22
C3
[-0.08, 0.85, 0.223, 2.58, 2.09, -236
44 = 1-07 -0776, - -1.05, -0055, - 3.08, 2.37] (radians).
We set nominal value for the time intervals as = [0, 20] (seconds), and create 150 problem instances by varying the following attributes: firstly, we add uniformly random offsets in [-0.3, 0.3) (radians) to the elements of all ç, and in [-2, 2] (seconds) the right end-points of the intervals I; secondly, we add random offsets to the dynamic parameters of the robot (masses and moments of inertia of the robot's links and actuators) and we set a different friction and disturbance term d(-), leading to different dynamic model in (5): thirdly, we set different sequences of visits to the points C, (1......4] 4). as dictated by o, i.e. one trajectory might correspond the visit sequence (x)) (ci, (1,) - (62, (12) + (c3, 41,) - (ca, (1,), and another to ((x(0), 0) - (Ca, (1g) - (cI,4,) (c4, - (c2, l1,). Finally, we add uniformly random offsets 0.5, 0.5] to the initial position of the robot (from the first point of the sequence), and we set its initial velocity randomly in the interval [0, 1/6.
Regarding the dynamics (). we use the methodology described in (Siciliano et al. (2009) to derive the B, C. and terms. We set nominal link masses and moments of inertia as m = [1, 2.5.5.7, 3.9, 2.5, 2.5, 0.7] (kg) and I [0.02, 0.04, 0.06, 0.05, 0.04, 0.04, 0.01] (kgm²), respec- tively, and we add random offsets in (- 2: m (- 2 in the created instances. Regarding the function d() used in 55: we set d(ī, t) = (t) + d (f). where
de = A
sin(nal + p6)
A, diagf AG, (1......))) R6x6 Av. is random term in (0, 2m,). 77s is random term in (0, 1). ii is random term in (0, 2). B, E R6x36 is random matrix taking values in (0, 2m,). and @ denotes the Kronecker product. We chose the control gains of the control policy 4 as
k 1, A2 10, and ke, = k2_ 10.
Further experimental results are depicted in Figs. 7 Fig. 7 depicts the mean and standard deviation of [()]] of the proposed control policy for the 50 test instances, whereas Figs. 10p depicts the control input that results from the control policy 4 as well as the the neural-network output. Note that the control input converges the
output, i.c. lim, (o (t)) which can be also verified by (4) and the fact that lim (t) = O. Fig. 11p depicts the mean and standard deviation of the adaptation signals (t), (t) for the 50 test instances. Finally, Fig, 9 shows timestamps of one of the test trajectories, illustrating the visit of the robot end-effector to the points of interest at the pre-specified time stamps.
19

---

Under review as a conference paper at ICLR 2022
1.5
0.7
0.6
mean (I el (t)l)
0.5
std [1 (t) ())
0.4
0.2
0.1
10
15
20
10
15
20
t
[sec]
t
[sec]
Figure 7: Mean (left) and standard deviation (right) of (t) for the proposed control policy.
2.5
Proposed
meanfe. ( mm [e]
mean(e (e., (Q)) [pad/a
Non-adaptivn
No-un
1.5
Progosed
Non-ndaptive
0.5
No-an
-1
-1.
0.
Proposed
Proposed
ste(eu(t) [m/4 m//
Non-sdaptive
(e, (+)) [pad/s]
Non-adaptive
Nowin
No-ni
[sec]
[sec]
Figure 8: Mean (top) and standard deviation (bottom) of (t) and (t) for the proposed, non- adaptive, and no-neural-network control policies.
20

---

Under review as a conference paper at ICLR 2022
1=5 sec
[=10.ser sec
T
T
(b)
1-15 sec
sec sec
3
T
T <
(d)
Figure 9: Illustration of the execution of one of the test trajectories of the URS robotic arm, visiting the points of interest at the pre-specified time stamps.
Regarding the unicycle experiments, the dynamic terms in (9)
have the form
M1
M2
M
M2
M1
fo (F, t) d(ō,t)
with M1 = my?
1 + (fe+md"))" ARR + Io. M2 = mr? (Ie tmd+ and where Ic is the moment of 1
1R2
inertia of the vehicle with respect to point C (see Fig. 66, To the moment of inertia of the the wheels, and d is the distance of the between point C and the vehicle's center of mass P2). The term d(ī, t) is chosen as the robotic- manoilllam case. The points to visit are chosen here as C1 [0, 0] C2 [0,2] C3 [2, 0jT C1 22.2 and 1, is chosen as 1, [0, 20), i {l,.... 4). We derive 150 problem instances by varying the following attributes: we vary C; with random offsets in 0.3, 0.3] and the right end-points of 7, with random offsets in 2, 2]; add random offsets to the dynamic parameters (elaborated subsequently) and the function d(ē. f), and we set different sequences of visits the points e; further vary the unicycle's 's initial position from the first e with random offsets in [-0.3, 0.3], and its initial orientation with random offsets in [-0.25, 0.25] (rad) from (0)) = arctan(e²(0)) (0)); we further set random values in [-225 0.25] (rad/s) the initial wheel velocities.
We nominal values for the dynamic and geometric parameters, appearing in the inertia matrix, as m = 28 (kg), Ic 0.1 (kgm²), ), To = 0.01 (kgm²), ), 0.01 (m), d = 0.01 (m), and we added random offsets in (- 2 ).
(- 2. 2 d respectively, for the problem instances. We chose the control gains of (12) as ki 0.25, kig 1, k. k. kūg keg 10,
m
La LA ).
Io la
2 2
2
k, = ken = 1. The non-adaptive control policy we compared our algorithm with was selected as
ug Msda k.e, + cos 3 3 sin 3
ed
uD = Mpus - kw +
where Va, wis are chosen as and Ms. MD are static estimates of 2 M 2R M1 respec- tively. More specifically, set deviation of 25% in the parameters appearing in these terms to form Ms and Mp. Further experimental results are depicted in Figs. 7 IT Fig. 77p depicts the mean and standard deviation of the velocity emror e, (1), e for the 50 test instances, showing convergence
21

---

Under review as a conference paper at ICLR 2022
mean (u(t)) [kgm 3
std (w(()) [kgmm
200
mean (M) [kggee)]
[kgm?
8g
20
190
std (()())
10
20
mean mean (Monn (f)) [kgm=)]
std (t)) [kgm²)
80
(0) [kgmm
Stdd Muna (t) [kgme
Vinui Vend
Ree.
Men
20
mean (Mem (1)
40
8
20
.20
10
f [sec]
[sec]]
[ssec
t [sec]
(a)
(b)
Figure 10: (a): Mean (left) and standard deviation (right) of u(t) (top) and tmm (t) (bottom) for the proposed control policy and the robotic-manipulatoo environment. (b): Mean and standard deviation (right) of ut) (top) and unn(1) (bottom) for the proposed control policy and the unicycle environment.
cueau (t.(0)
mesal (200)
mean (f)n(f.(t)
meanf ((()
cuean (2)(0)
mecon (2(0))
0
stdl (td((((()) s
std (t)) (t.((_ std(f, (ti) ste(fo(()
0
10
15
20
[sec]
[snc]]
(a)
(b)
Figure 11: (a): Mean (top) and standard deviation (bottom) of & (t), (t() for the proposed control policy and the robotic-manipulator environment. (b): Mean (top) and standard deviation (bottom) of (t). (t). (t), (t) for the proposed control policy and the unicycle environment.
zero, whereas Figs. 10p depicts the control input that results from the control policy 4 well the the neural-network output. Similarly to the nobotic-manipulator case, the control input converges to the neural-network output. Finally, Fig. [P depicts the mean and standard deviation of the adaptation signals (t), (t), (t), (t) for the 50 test instances.
Finally, regarding the pendulum environment, we consider the dynamics
q = sin q + + d(t)
where I. is the pendulum's link length, g the gravitational constant, and d(t) is term of exogenous disturbances. We created 150 instances by varying the mass and link length of the pendulum, the external disturbances, and setting its initial position and velocity randomly in [-1, 1] (rad) and [-1, 1] (rad/s), respectively. We selected unitary value for its nominal link length, and we added random offsets in (-0.5, 0.5) in each instance. We set the external disturbances as d = A sin( + Φ), with A, n7, and é taking random values in [0, 0.2), [0, 1], [0, 2], respectively. In contrast to the robotic manipulator case, we assume feedback of sin q. cos q, q. and set the error in (3) as 1 - cos(q - T). Finally, we chose the control gains k k and biz kit 10.
22


Reviews:
[{"review_id": "nMNaKAcbo-P", "reviewer": "Reviewer_YVCm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The approach presented in this paper is novel and differs significantly from past proposals.\n2. The approach is demonstrated on robotic applications.\n3. The concept of using a learned component within a control task is of interest to this community. \n\nWeaknesses\n1. The paper claims that the proposed method can be used to achieve behaviors of the controlled system which satisfy SITL specifications. This appears to be subtely incorrect. Theorem 1 describes asymptotic performance guarantees, not performance of the system during a finite horizon. Nothing can be said about the satisfaction of the \"prefix\" portion of the SITL specification. \n2. The paper correctly claims that the theorem makes no special requirements about the error of the learned component, and moreover that the selection of training data does not affect the claim. This is not the same thing as saying that these elements \"don't matter\", what is promised is that things won't go terribly wrong, not that the outcomes will be good. The experimental section could be improved by ablations of the off-policy data both in terms of variations of the system dynamics, the quantity of the data, and the diversity of tasks.\n3. The learning problem in which the learned component predicts the next control input seems like it could have issues (which do not affect the theoretical guarantees) in the case that there are two tasks with the same \"prefix\" trajectory but a different end trajectory. It seems that the network would then be forced to predict the average of the two outcomes. Does the methodology seek to avoid sampling such training data. What are the implications?\n4. In general the paper oversells some of the claims. See above. it could be improved by taking a less adversarial approach in the writing and acknowledging the trade off in relaxing the approach to ask that the learned component doesn't add arbitrarily large errors. In addition more discussion could be added about the controllability requirement and how limiting this is; I acknowledge that the paper has made some efforts to be precise here. \n5. A sketch of the proof in the main body could be useful. It is extremely technical. Moreover, it could be useful for the author's to discuss whether the learned component must be a neural network and specific properties of neural networks were necessary for the proof. It seems that the setup could be more general. \n6. The use of SITL seems needlessly complex and not the main point of the paper. Beyond its use as a specification language for generating training examples it seems ancillary. See point 1. The paper could be improved by incorporating specifications that use more properties of the SITL formalism for task specification (e.g. beyond reach like tasks). \n7. In the related work some discussion of work like: https://arxiv.org/pdf/1903.11239.pdf could be included. In this reference the authors also learn a component to augment control inputs. They do not have any similar guarantees but it is a better motivated setting. ", "summary_of_the_paper": "The paper describes an adaptive controller which includes a learned component. The paper proves that this controller is asymptotically stable. Importantly the paper develops theory which proves that it is not necessary for the learned component to achieve bounded error on the task of predicting the control input given the state, but instead only that the output of the combined system be bounded by a simple function of the current state. The method incorporates a proposal for generating training data for the learned component by sampling trajectories (open-loop) of the system which conform to an SITL specification. The method is evaluated on several common simulation environments which comprise \"reach-like\" motion planning tasks. The results demonstrate that the proposes adaptive controller converges to the desired reference trajectory, sometimes faster than adaptive methods which do not incorporate the learned component. ", "summary_of_the_review": "This paper describes an approach to adaptive control incorporating a learned component. The novelty relies in the development of a proof technique which doesn't require the learned component to have Lipschitz like properties or exhibit universally bounded error on a space of inputs. The paper is interesting to the community and combines theory with experiment in a coherent manner. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jH8Jn8yjRbn", "reviewer": "Reviewer_HYqY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, here are the strenght and weakness of the paper.\n\nStrength: \n1) by combining control techniques and supervised learning, the learned controller can potentially give a good prior to the online adaptive tracking\n2) the presentation of the proposed method is clear.\n\nWeakness:\n1) I'm not convinced that the supervised learning is necessary as given the open loop trajectory, there are plenty of existing methods to track it.\n2) Theorem 1 does not mean much as the transient tracking can be really bad, thus violating the STL formula.\n3) Infinite control input should not be allowed.\n4) It seems that if you drop the learning controller, the open-loop trajectory and online adaptive control can still satisfy the system task, making the learning obsolete.\n\nMinor comments:\n1) Assumption 1, do you mean the matrix is PSD, or the entries are positive definite? g may not be square\n2) Assumption 2 seems trivial as any bounded controller with bounded f and g satisfies that\n3) When mixing trajectories under different tasks, there could be states with totally different inputs (for different tasks), does that make the learned controller inferior to even some heuristic controller? Why not include the task information in the learning process?", "summary_of_the_paper": "This paper presents a learning + adaptive control approach to synthesize control policies that fulfills STL specifications. The open-loop trajectory is generated using (Lindemann & Dimarogonas (2020)), the neural network is trained to replicate the expert demonstration consisting of state-action pairs with trajectories collected following multiple tasks (including the target task). This input is used as the open-loop input and a closed-loop trajectory tracking controller is added to correct the trajectory deviation using adaptive control techniques.\n\n", "summary_of_the_review": "It seems to me that the open-loop trajectory and the online adaptive tracking controller are doing the heavy-lifting here and the learning is not that important. The authors should demonstrate the benefit of learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R2DlvuXHLSt", "reviewer": "Reviewer_GDik", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*** STRENGTHS ***\n\n(1) The paper is generally well-written, and is timely given a recent surge in works at the intersection of parametric/non-parametric learning and adaptive control theory. \n\n(2) The experiments in the paper are extensive; it is good to see multiple different robotic systems included. The experimental results seem detailed and reproducible.\n\n*** WEAKNESSES ***\n\nThe core algorithmic content of the paper is spread across: (1) learning a nominal feedback law as a neural network to bound the system dynamics, (2) computing open-loop trajectories satisfying SITL formulae, and (3) adaptive controller design.\n\nRegarding (1): It seems naive to train a single feedback law on data from many different tasks and models (e.g., different controllers are often used for different system dynamics). Moreover, the authors provide no convincing evidence (rigorous, intuitive, or otherwise) that their learned neural network feedback law should satisfy their Assumption 2. Even if the neural network is trained on bounded trajectories, this does not guarantee that the closed-loop dynamics satisfy the particular affine bound in Assumption 2. Overall, the neural network learning content of the paper is not substantive and not convincingly conducive to the adaptive control theory presented later on.\n\nRegarding (2): the paper's narrative is muddied by its focus on SITL. In section 3.2, the authors describe that, using _existing_ work, a smooth open-loop trajectory in R^n satisfying all of the requirements of a SITL task can be computed. Given that the authors do not make any contributions to SITL work and just reduce any SITL task to a trajectory tracking problem, sections 2.1 and 3.2 should be removed entirely. If desired, the authors could simply include a comment on applications of adaptive trajectory tracking control to SITL tasks in a motivation or discussion section.\n\nSo far, the possible algorithmic contribution of the paper has been narrowed down to the design of the adaptive controller on pages 5-6 (assuming any nominal controller that bounds the dynamics according to Assumption 2). However, the adaptive controller design contribution is not adequately contextualized against the adaptive control literature. Other than citing an adaptive control textbook from Krstic, et al, the authors need to cite related works and establish the novelty of their design. For example, Equations (3)-(4) seem reminiscent of well-known sliding-mode adaptive controllers, e.g., from:\n\n    J.-J. E. Slotine and J. A. Coetsee. \"Adaptive sliding controller synthesis for non-linear systems\". International Journal of Control, 1986.\n\n    J.-J. E. Slotine and W. Li. \"On the adaptive control of robot manipulators\". International Journal of Robotics Research, 1987.\n\nIndeed, the authors' \"e_v\" is exactly Slotine, et al's sliding variable \"s\", and the authors' \"v_d\" is Slotine, et al's reference velocity \"q_r\". The authors need to discuss how their design differs from such well-known existing work.\n\nFinally, the content on the adaptive control design for a unicycle system, which spans over a page, seems ill-fitted to the main content of an ICLR paper submission.", "summary_of_the_paper": "In this paper, the authors propose an adaptive controller that can be applied to a certain class of dynamical systems in order to fulfill Signal Interval Temporal Logic (SITL) tasks. The proposed controller leverages a nominal feedback law learned from past trajectory data collected on possibly many different SITL tasks and system parameters. The authors test their method in simulation on a manipulator arm, unicycle robot, and a pendulum with varying physical parameters and external forces.", "summary_of_the_review": "Of the three sections in the \"Main Results\" of the paper, only the adaptive control design section is substantive; the proposed neural network training is just naive regression on state-input tuples, and the inclusion of SITL needlessly complicates the paper's exposition before being reduced to a trajectory tracking task. Adaptive control theory alone may be an unsuitable fit for ICLR. Regardless, the novelty of the adaptive control design is questionable, as it appears similar to well-known previous work and the authors' have not contextualized their design amongst the adaptive control literature.\n\n%%\nNote: I have raised my score after the authors' rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledge that the paper is well written and contains interesting ideas to combine adaptive control and learning. However, they identified issues regarding the claims about transient tracking and STL formula. Moreover, the significance of the presented learning rule was unclear regarding one reviewer. While the authors could respond well to the identified transient tracking issue, they also needed to weaken their claims, limiting the contribution of the paper. The reviewers therefore stayed with a a reject rating.
Decision: Reject