Meta-Adaptive Nonlinear Control: Theory and Algorithms
Authors: Guanya Shi
Kamyar Azizzadenesheli
Michael O'Connell
Soon-Jo Chung
Yisong Yue
Venue: NeurIPS 2021 Poster (2021)
Keywords: Online Learning
Representation Learning
Adaptive Control
Nonlinear Control
Meta-learning

Abstract: We present an online multi-task learning approach for adaptive nonlinear control, which we call Online Meta-Adaptive Control (OMAC). The goal is to control a nonlinear system subject to adversarial disturbance and unknown \emph{environment-dependent} nonlinear dynamics, under the assumption that the environment-dependent dynamics can be well captured with some shared representation. Our approach is motivated by robot control, where a robotic system encounters a sequence of new environmental conditions that it must quickly adapt to. A key emphasis is to integrate online representation learning with established methods from control theory, in order to arrive at a unified framework that yields both control-theoretic and learning-theoretic guarantees. We provide instantiations of our approach under varying conditions, leading to the first non-asymptotic end-to-end convergence guarantee for multi-task nonlinear control. OMAC can also be integrated with deep representation learning. Experiments show that OMAC significantly outperforms conventional adaptive control approaches which do not learn the shared representation, in inverted pendulum and 6-DoF drone control tasks under varying wind conditions.

Meta-Adaptive Nonlinear Control: Theory and Algorithms
Guanya Shit, Kamyar Azizzadenesheli, Michael O'Connell', Soon-Jo Chung', Yisong Yue Michael
'Caltech Purruee University
{gshi moc, s jjchung.yyue)occalteed edu. kamyar edu
Abstract
We present an online multi-task learning approach for adaptive nonlinear control, which we call Online Meta-Adaptive Control (OMAC). The goal is to control a nonlinear system subject to adversarial disturbance and unknown environment- dependent nonlinear dynamics, under the assumption that the environment- dependent dynamics can be well captured with some shared representation. Our approach is motivated robot control, where robotic system encounters quence of new environmental conditions that it must quickly adapt to. A key emphasis is integrate online representation learning with established methods from control theory, in order to arrive at unified framework that yields both control-theoretic and learning-theowe thhooreicc guarantees. We provide instantiations of our approach under varying conditions, leading to the first non-asymptotic end- to-end convergence guarantee for multi-task nonlinear control. OMAC can also be integrated with deep representation learning. Experiments show that OMAC significantly outperforms conventional adaptive control approaches which do not learn the shared representation, in inverted pendulum and 6-DoF drone control tasks under varying wind conditions
Introduction
One important goal in autonomy and artificial intelligence is to enable autonomous robots to learn from prior experience to quickly adapt new tasks and environments. Examples abound in robotics, such as drone flying in different wind conditions [I] a manipulator throwing varying objects (2), quadruped walking empirical evidence, when designing such adaptive systems, two important theoretical challenges arise, discussed below.
changing terrains
6 Though those examples provide encouraging
First, from a learning perspective, the system should be able to learn an "efficient" representation from prior tasks, thereby permitting faster future adaptation, which falls into the categories of representation learning or meta-learning. Recently, line of work has shown theoretically that learning representations (in the standard supervised setting) significantly reduce sample complexity Empirically, deep representation learning or meta-learning has achieved success in many applications 77), including control, in the context of mett-rriinforeement learning [8]] However, theoretical benefits (in the end-to-end sense) of representation learning or meta-learning for adaptive control remain unclear.
new tasks
Second, from a control perspective, the agent should be able to handle parametric model uncertainties with control-theoretic guarantees such stability and tracking error convergence, which is a common adaptive control problem [ For classic adaptive control algorithms, theoretical analysis often Moreover, many recent
involves the use of Lyapunov stability and asymptotic convergence [11]
Code and video: http:://gthhhcooooo/aa///l/aaaar
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

studies aim to integrate ideas from learning, optimization, and control theory to design and analyze adaptive controllers using learning-theor theoretic metrics. Typical results guarantee non-asymptotic convergence in finite time horizons, such as regret 133177 and dynamic regret [18-20]. However, these results focus on a single environment or task, multi-task extension, especially whether and how prior experience could benefit the adaptation in new tasks, remains an open problem.
Main contributions. In this paper, we address both learning and control challenges in unified framework and provide end-to-end guarantees, We derive a new method of Online Meta-Adaptive Control (OMAC) that controls uncertain nonlinear systems under a sequence of new environmental conditions. The underlying assumption is that the devironment-depende well be captured by a shared representation, which OMAC learns using meta-adapter. OMAC then performs eavironment-specifee updates using an inner--ddaper.
unknown dynamics can
We provide different instantiations of OMAC under varying assumptions and conditions. In the jointly and element-wise convex cases, we show sublinear cumulative control error bounds, which to our knowledge is the first non-asymptotic convergence result for multi-task nonlinear control. Compared to standard adaptive control approaches that do not have a meta-adapter, we show that OMAC possesses both stronger guarantees and empirical performance. We finally show how to integrate OMAC with deep representation learning, which further improves empirical performance.
2 Problem statement
We consider the setting where a controller encounters sequence of N environments, with each environment lasting 7 time steps. We use outer iteration to refer to the iterating over the N environments, and inner iteration refer to the T time steps within an environment.
Notations: We use superscripts (e.g., (i) in x; ((()) to denote the index of the outer iteration where
1 <is i N, and subscripts (c.g.. t in x; to denote the time index of the inner iteration where 1 TT T. We use step(i, t) to refer to the inner time step at the 2th outer iteration. || || denotes the 2-norm of a vector the spectral norm of matrix. || denotes the Frobenius norm of a matrix and Amin(-) (-) denotes the minimum eigenvalue of real symmetric matrix. vec(.) E I denotes the vectorization of a m x n matrix, and @ denotes the Kronecker product. Finally, we use w1zt. to denote sequence {ur; u2, ",,,"",,, J
We consider a discrete-time nonlinear control-affine system 21 22 with envirommeent-dependent uncertainty f(z c). The dynamics model at the 7th outer iteration is:
(8) xt-1 = fo(xx") B(2
(0) uu
( f(( (()) + w 1stsT,
(1)
where the state x1") R", the control u." E FT fo R + R" is known nominal dynamics model, B R= + =x is known state-dependent actuation matrix, c(i) C R² is the unknown parameter that indicates an environmental condition, ʃ R" x R + R is the unknown c(O)_ (6) is disturbance, potentially adversarial. For simplicity we define dependent dynamies model, and w
B(") =aa and f( = f(x (2) c.)).
Interaction protocol. We study the following adaptive nonlinear control problem under N unknown environments. At the beginning of outer iteration i, the environment first selects (+)) (adaptively and adversarially), which is unknown the controller, and then the controller makes decision u under unknown dynamics (x)
(i)
(8) clO) and potentially adversarial disturbances (8) To summarize:
Outer iteration i. policy encounters environment (i E (1...., .,, N)), associated with unob- served variable (99 (e.g., the wind condition for flying drone). Run inner loop (Step 2).
(8) 2. Inner loop. Policy interacts with environment for T time steps, observing xt and taking action (.) with state/action dynamics following |
3. Policy optionally observes ct at the end of the inner loop (used for some variants of the analysis).
Increment i = i and repeat from Step I.
We average control error (ACE) our performance metric:

---

Definition (Average control error). The average control error (ACE) of N outer iterations (i.e., N environments) with each lasting T time steps, is defined ACE TN Ei Σ ||=]""]
ACE can be viewed as a non-asymptotic generalization of steady-state error in control the following assumptions on the actuation matrix B, the nominal dynamics fo. and disturbance (
23 We make
Assumption (Full actuation, bounded disturbance, and e-ISS assumptions). We consider fully- actuated systems, i.e., for all I, vank(B((z) = || & W,vt. Moreover, the nominal dynamics fo is exponentially input-to- st-ttte stable (e-ISS): let constants B. y and S p For sequence V1-t-1 E R", consider the dynamics 2k+1 fo(Øx) + 1 s & S - 1. 24 satisfies:
| || S Bp |0
k=1
With the e-ISS property in Assumption average squared loss between B Ut
we have the following bound that connects ACE with the (8)
+ we and ft
Lemma 1. Assume 11 O, Vi. The average control error (ACE) is bounded as:
|00
4
11 +
r-ll TN
(3)
1 1-
TN
The proof can be found in the Appendix A.1 We assume 21 for simplicity: the influence of (8) non-zero and bounded zi is constant term in each outer iteration, from the e-ISS property (
(7)
Remark the e-ISS assumption and the ACE metric. Note that an exponentially stable linear system fo(2;) = Ax, (i.e., the spectral radius of A is < 1) satisfies the exponential ISS (c-ISS) assumption. However, in nonlinear systems e-ISS a stronger assumption than exponential stability. For both linear and nonlinear systems, the c-ISS property of fo is usually achieved by applying some
stable feedback controller to the system i.e., fo is the closed-loop dynamics I 24 IC
e-ISS
assumption is standard in both online adaptive linear control 124 16 [4] and nonlinear control [13]
and practical robotic control such as drones (25] ACE we consider regulation task, but can also capture trajectory tracking task with vime-variant nominal dynamics fo under incremental We only consider the regulation task this paper for simplicity.
stability assumptions
[
Generality. We would like to emphasize the generality of our dynamics model I The nominal control-affine part can model general fully-actuated robotic systems via Euler- Langrange equations and the unknown part f(r, c) is nonlinear in and We only need to assume the disturbance and nonlinear cases. For example, wi can model extra (æ, c)-dependent uncertainties or adversarial disturbances. Moreover, the environment sequence c(1:N) could also be adversarial. In term of the extension to under-actuated systems, all the results in this paper hold for the matched uncertainty setting [3] i.c.
2112
22
w
bounded, which is more general than stochastic settings in linear
in the form #++ = fo(a;") + Bxx (( - f( (6) c()))
)) + (+) where B(x is not necessarily Generalizing other
full rank (e.g.. drone and inverted pendulum experiments in Section 5
under-actuated systems is interesting future work.
3 Online meta-adaptive control (OMAC) algorithm
The design of our online meta-adaptie adaptive control (OMAC) approach comprises four pieces: the policy class, the function class, the inner loop (within environment) adaptation algorithm A2, and the outer loop (between environment) online learning algorithm A1.
Policy class. We focus on the class certainty-equivaleence controllers 13 general class of model-based controllers that also includes linear feedback controllers commonly
26 14
which is a
For example, consider z++1 Ja + 2sin v ue- With feedback controller ū, Mr xt sin
the closed-loop dynamics is 2s+1 fe Ue. where fo(x) 1 e-ISS.
3

---

studied in online control 26 14|
After model is learned from past data, controller is designed
by treating the learned model as the truth [2]] Formally, at step(i. t), the controller first estimates
(( (an estimation of f()) based on past data, and then executes " B ft where (-)1 is the pseudo inverse. Note that from Lemma || the average control error of the omniscient controller using P( = f(4) (i.e., the controller perfectly knows f(r c)) is upper bounded ACE & yW/(1 - p).
ACE (Cmonisciee) can be viewed as a fundamental limit of the certainty seuivalt-e policy class.
Function class F' for representation learning. In OMAC, we need to define a function class F(o(2; θ), é) to compute f(()) (i.e., f( F()(z)) θ). where (parameterized by θ) is representation shared by all environments, and is an environment-specffe latent state. From a theoretical perspective, the main consideration of the choice of F()(x), is how it effects the resulting learning objective. For instance, Φ represented by a Deep Neural Network (DNN) would lead to a highly non-convex learning objective. In this paper, we focus on the setting θ E RP, C E IR", and p >> h. i.e., it is much more expensive to learn the shared representation & (e.g., DNN) than "fine-ttning" via è in specific environment, which is consistent with meta-learning 9 and practices.
representation learning
7
Inner loop adaptive control. We take a modular approach in our algorithm design, in order to cleanly leverage state-of-the-art methods from online learning, representation learning, and adaptive control. When interacting with a single environment (for T time steps), we keep the learned representation ø fixed, and use that representation for adaptive control by treating è as unknown low-dimensional parameter. We can utilize any adaptive control method such as online gradient descent, velocity gradient, or composite adaptation 27
Outer loop online learning. We treat the outer loop (which iterates between environments) as an online learning problem, where the goal is to Iearn the shared representation & that optimizes the inner loop adaptation performance. Theoretically, we can reason about the analysis by setting up a hierarchical or nested online learning procedure (adaptive control nested within online learning).
Design goal. Our goal is design meta-adaptive controller that has low ACE, ideally converging to ACE(omniscien)) as T, N - cc. In other words, OMAC should converge to performing as good as the omniscient controller with perfect knowledge f(x, c).
Algorithm D describes the OMAC algorithm. Since & is environment-invarian and p >> h, we only adapt θ at the end of each outer iteration. On the other hand, because c( varies in different environments, we adapt each inner step. As shown in Algorithm at step(i, t), after applying
(8) the controller observes the next state o and computes: yi (4) [)( + B,ah un
(8)
((()) wi
(8) which is a disturbed measurement of the ground truth ft We then define e, ( 2) &
[[F(((z))) (1), 6), c) 2 || as the observed loss step(i, t), which is squared loss between the disturbed measurement of (((i) and the model prediction F(o(2{)) Ô). ( E).
AI, A2, ObserveEnv), we consider three settings: Instantiations. Depending on (F(d(1; 0), (F
Convex case (Section 4.1 The observed loss (()) is convex with respect to ê and è
Element- wise convex case (Section 4.2): Fixing Ô è, e(9) is convex with respect the other.
Deep learning case (Section 5p: In this case, we use a DNN with weight é to represent ø.
4
Main theoretical results
4.1
Convex case
In this subsection, we focus on a setting where the observed loss 4 (Ô, è) = [[F((((;;; 0). - 112
convex with respect to Ô and é We provide the following example to illustrate this case and highlight its difference between conventional adaptive control (e.g.. [3 [2]).
"Tii upper bound is tight. Consider scalar system 2141 = ax; + u f(xx) + w with [a| < 1 and w constant. In this case p Jal, 1, and the omniscient controller u, f(x)) yields ACE 7/wl/(1 p).

---

Algorithm 1: Online Meta-Adaptive Control (OMAC) algorithm
Input: Meta-adapter A inner-adapter A2; model (x((x;; O), e); Boolean ObserveEnv
for 1, .NV do
The environment selects e(3) for 1 = 1, 1, ,T do
Compute Jt F(o(a ( (.)))
Execute up
(i) B, (i)t p(()
//certannt--quiivlleeet
policy
Observe T 144 Vi ( = f(xt (0) dd0) - w and el ( (Ô. = (P) 1
1/y6)) is noisy measurement of and g(i)
observed loss
(2) (2) by A2 1/9t (9 is function of
Construct inner cost function Di
Inner-adaptation: 241 - Az(d, .9:1)
(i)
ObserveEnv then Observe c(0) //only used some instantiations
Construct an outer cost function ()) by A l/G is a function of Q Meta-adaptation: (1+1) A1 G(I))
Example 1. We consider model ((b(2;; θ), -1))))))()0000
()(x) + Y2(2) to estimate f: A = Yi (4) 1 + Y2(2,
(4)
where Yi RD - Rnxp, Y2 R - R are two known bases. Note that conventional adaptive control approaches typically concatenate and è and adapt on both at each time step, regardless of the environment changes (e.g., [3]. Since h, such concatenation is computationally much more expensive than OMAC, which only adapts θ outer iterations.
Because e.th (Ô, é) is jointly convex with respective to and é, the OMAC algorithm in this case falls into the category of Nested Online Convex Optimization (Nested OCO) (28) The choice of
Pi G6,, (i, A2 and ObserveEnv are depicted in Table Note that the convex case OMAC does not need to know c(0) in the whole process (ObserveEnv = False).
F(((z;0),)) d (c)
Any F model such that (G, Vell (6() e(f)) è
é) is convex (e.g.. Example
(()
E V g(0) (6))) cl )
A
With convex set 1G, Ai initializes ( Kr, and returns ( Kc, Vi. Ai sublinear regret, i.c., the total regret of A, is o(N) (c.g., online gradient descent)
Aa
With convex set K22, Vi, A2 initializes a( Kig and returns e K2,, Vt. A2 has sublinear regret, i.e., the total regret of Aa is N o(T) (e.g,, online gradient descent)
ObserveEnv
False
Table I: OMAC with convex loss
As shown in Table 1 at the end of step(i, t) fix ê 6 and update (6)
Kr using (t) On the other hand, at the end of outer Ae( -1:)) which is an OCO problem with linear costs 91:t-
(6)
iteration i, we update ((+1) E Ki using (,()()) G(a))), which is another OCO problem with
linear costs ((a)), From 288
we have the following regret relationship:
Lemma 2 (Nested OCO regret bound, (28
OMAC (Algorithm 7 specified by Table 1 has regret:
N
T
N
i
100 (0,
min BE
min EKa
t=
t=1
(5)
Σ
(6(0)))
min
())
g (c
Σ
min
Σ
91
(c(0))
ek22
1=1
the total regret of As (T-o(N)
the total regret No.)-o(T)
5

---

Note that the total regret of A1 is - o(N) because G is scaled up by factor of T, With Lemmas and we have the following guarantee for the average control error.
Theorem 3 (OMAC ACE bound with convex loss). Assume the unknown dynamics model is [xx F( θ), c). Assume the true parameters θ K1 and clt) K22 Vi. Then OMAC (Algorithm /
specified by Table 7 ensures the following ACE guarantee:
( (N)
ACE
W2
To further understand Theorem 3 and compare OMAC with conventional adaptive control approaches, provide the following corollary using the model in Example
Corollary 4. Suppose the unknown dynamics model is fxx c) Yı (z)) where Yı R + Raxp, Y2 R - m are known bases. We assume ||||| S Ke and || ||e( S Ke, Vi. Moreover, we assume ||(()) < K1 and |||(())]| < K2, Vz. In Table ! we use f( Y2(=("))10) Y2 (()) and Online Gradient Descent (OGD) 277 for both A1 and A2, with learning rates 17() (4) respectively. We set K₁ =O: ||ô|| S Ke} and K2 {ē: ||e|| I Ke). we schedule the learning rates as:
and 1ts
2K
2K.
( KKi + 4K1KgK. + 2K W)TVi "
(4K2 + K2 2KK22K Ke + 2K2W)
G.
then the average control performance is bounded as:
1 VN
ACE(OMAC) & 1-p W2 + 3
KaCi
+
KcC2
VT
Moreover, the buseline adaptive control which uses OGD to aulppt Ô and at each time step safisfies:
? W2 3 K2 + K3 C C2 ACE(baseline adaptive control) - 1- pV
1 VT
Note that ACE(baseline adaptive control) does not improve as N increases (i.e., encountering more environments has no benefit). p >> h, potentially have K1 >> K₂ and Ke >> Ke, C₁ >> C2- Therefore, the ACE upper bound of OMAC better than the baseline adaptation if N T, which is consistent with recent representation learning results 55. It is also worth noting that the baseline adaptation is much more computationally expensive, because it needs to adapt both Ô and è each time step. Intuitively, OMAC improves convergence because the meta-adapter A1 approximates the environment-invariinn part Y (z)),, which makes the inner-adapter Aa much more efficient.
4.2 Element-wise convex case
In this subsection, we consider the setting where the loss function ( is element-wise convex
with respect to θ and 2, i.e., when one of Ô or è fixed, ((ه) is convex with respect to the other one.
For instance, F could function as depicted in Example bilinear models are also studied in representation learning
Outside the context of control, such
5 and factorization bandits 29
Example We consider a model F(ò(:;; 6), = Y (x) θe estimate f:
f = Y y (0))))
(6)
where Y: n - R is a known basis, 60) E Roxh and 2(00 E R.A Note that the dimensionality
of Ô is p = ph. Conventional adaptive control typically views Ôč as vector in Rd and adapts it at each time step regardless of the environment changes
Compared Section the element-wise convex case poses new challenges: the coupling between é and brings inherent non-identifiability issues; and ii) statistical learning guarantees

---

These challenges are further amplified by the underlying unknown nonlinear system ① Therefore in this section we set ObserveEnv = True, i.c., the controller has access to the true environmental condition e( at the end of the 7th outer iteration, which is practical in many systems when c(*) has concrete physical meaning, e.g., drones with wind disturbances 3
typical need i.i.d. assumptions on ((+) and J
(4)
4 5
9 (c) F(o(z;)) C).
Any F model such that (,, is clement-wise convex (c.g., Example (6(+), 2)
(6)
Σe (6,
Ai & A ObserveEnv
Same Table
True
Table 2: OMAC with element-wise convex loss
The inputs to OMAC for the element-wise convex case are specified in Table Compared to the convex case in Table difference includes i) the cost functions 9t and (l are convex, not necessarily linear; and ii) since ObserveEnv = Truc, in G++) we use the true environmental condition cO) With inputs specified in Table 2 Algorithm has ACE guarantees the following theorem.
Theorem (OMAC ACE bound with element wise convex loss). Assume the unknown dynamics model is f(x,c) c) F( +), c). Assume the true parameter θ E Kr and c E K22, Vi. Then
OMAC (Algorithm)
7 specified by Table 2 eensures the following ACE guarantee:
o(T) o(N) ACE 1 1 p W2
N
4.2.1 Faster convergence with sub Gaussian and environment diversity assumptions
Since the cost functions 9r (2) and G in Table are not necessarily strongly convex, the ACE upper bound in Theorem is typically - W2 O(L/VT) O(L/VN) (e.g., using OGD for both AA, and A2). However, for the bilinear model in Example 2 is possible to achieve faster convergence with extra sub Gaussian and environment diversity assumptions
F(o(2; 2). of (C)
bilinear model in Example
( (6()), ||| + Σ Ee= (6, c()) with some > 0
())
A
( = arg ming (1) (Ô) (i.e., Ridge regression)
As ObserveEnv
Same as Table
True
Table 3: OMAC with bilinear model
With a bilinear model, the inputs to the OMAC algorithm are shown in Table With extra assumptions on w (i) and the environment, have the following ACE guarantees.
Theorem 6 (OMAC ACE bound with bilinear model). Consider an unknown dynamics model f( (=) where Y IR" R is known basis and E Roxh, Assume each component of the disturbance w. is Gausb-aa the true parameters ||||| S Ke. ||c(O]| S Ke, Vi, and
[x)(=)) < Ky,Vx. Vx. Define z(,1) c(J)T (x( R and assume environment diversity: Amin Ej= Σr= z(5)T z(5)) 2ii). Then OMAC (Algorithm specified by Table has the following ACE guarantee (with probability at least - 8);
7 ACE < 1-p W2 +
o(T) T
+ o Jog(NN// log(N)
N
If we relax the environment diversity condition to 2(Vi), the last term becomes Of (TR(NT/5)
VN
The Lub-Gaussian assumption is widely used in statistical learning theory to obtain concentration bounds 32 4. The environment diversity assumption states that (() provides "new information"
7

---

in every outer iteration such that the minimum eigenvalue of Σ i=1 E1=12011 2,"¹ grows linearly increases. Note that do not need Amin (E Ei= Z 2 to increase T goes up. Moreover, if we relax the condition to (2(v),, the ACE bound becomes worse than the general element-wise convex case (the last term is O()/VN)), which implies the importance of "linear" environment diversity (i))). Task diversity has been shown to be important for representation learning We provide proof sketch here and the full proof can be found in the Appendix A.6
Proof sketch. In the outer loop we use the martingale concentration bound diversity assumption bound [|x((+1) l O ( ), Vi with probability at least
2 and the environment
Then, use Lemma to show how the outer loop concentration bound and the inner loop regret bound of Ag translate to ACE.
Deep OMAC and experiments
We now introduce deep OMAC, deep representation learning based OMAC algorithm. Table shows the instantiation. As shown in Table Deep OMAC employs a DNN to represent Φ, and uses gradient descent for optimization. With the mode p θ) é, the meta-adapter A1 updates the representation & via deep learning, and the inner- A2 updates a linear layer č.
F(o(z;0) 2)
cir; ②) where 6(2;; 0) R"
R is neural network with weight O
(()(6) (C)
(+())))
A
Gradient descent: ( -- ( Vo E (6, 2
A ObserveEnv
Same Table
False
Table 4: OMAC with deep learning
To demonstrate the performance of OMAC, we consider two sets of experiments:
Inverted pendulum with external wind, gravity mismatch, and unknown damping. The continuous-time model is ml2@ miğ sin é/0 is the angular velocity/acceleration,, m the pendulum mass and the length, is the gravity estimation, cis the unknown parameter that indicates the external wind condition, and f(0, 0, c) is the unknown dynamics which depends on but also includes c-invariant terms such as damping and gravity mismatch. This model generalizes 35 by considering damping and gravity mismatch.
[00, 0 c), where is the pendulum angle,
6-DoF quadrotor with 3-D wind disturbances. We consider a 6-DoF quadrotor model with unknown wind-dependent aerodynamic force (x, E R3, where x E IR13 is the drone state (including position, velocity, attitude, and angular velocity) and the unknown parameter indicating the wind condition. We incorporate realistic high-fidelity aerodynamic model from [36].
We consider different controllers in the experiment (see more details about the dynamics model and controllers in the Appendix A.7
No-adapt is simply using ft O, and omniscient is using ((() f((0)
OMAC (convex) is based on Example where p((0) = (x )(0)) + Y2(x, (8) (( We use
random Fourier features 37 K3 to generate both Y, and Y2. We use OGD for both Ai and As in
Table I
OMAC (bi-convex) based on Example 2 where f(()) = Y( ((0( Similarly, we use random Fourier features to generate Y. Although the theoretical result in Section requires in Table instead of >T ((3) (Ô, c(.). We also deploy OGD for A₁ and A2. Baseline has the same procedure except with ( (1),, i.c.. calls the inner-adapter Ag every step and does not update é, which standard in adaptive control
ObserveEnv Truc, we relax this our experiments and use ()) Σ
'The intuition behind this structure is that, any analytic function f(r, e) can be approximated by ((x)c(c) with universal approximator ó (3 We provide detailed and theoretical justification in the Appendix A.7

---

Baseline
(convex)
convex) ACE 0.074m
(deap)
0.199m
0.125m
04
Postonn (m)
0.2
0.0
-02
-04
-5-
(a) xz-position trajectories (top) and force predictions (bottom) in the quadrotor experiment from one random sced. The wind condition is switched randomly every 2s (indicated by the dotted vertical lines).
Average control error for each 25 wind condition
Average force prediction error (Newtons) for each 25 wind condition
0.5
5
DAWC (bi-conves)
CMAC (convext
DMAC (deep)
3
2
10
11
12
13
15
14
Wind conditonn Index
Wind condition Indox
(b) The evolution of control eтrоr (left) and prediction error (right) in each wind condition. The solid lines average 10 random seeds and the shade show standard deviations. The performance OMAC improves the number of wind conditions increases (especially the bi-convex and deep variants) while the baseline
Figure 1: Drone experiment results.
OMAC (deep learning) is based on Table We use a DNN with spectral normalization 38 251
39 40
represent Φ, and use the Adam optimizer 41 for A1. Same other methods, A is also an OGD algorithm.
no-adapt
baseline 0.311 + 0.112
OMAC (convex)
0.663 + 0.142
0.147 + 0.047 0.047
external wind
OMAC (bi-convex)
OMAC (deep)
omniscient
0.129 + 0.044
0.093 + 0.027
0.034 0.017 0.017
UI
no-adapt
baseline
OMAC (convex)
0.374 + 0.044
0.283 + 0.043
0.251 + 0.043
OMAC (bi-convex)
OMAC (deep)
omniscient
050 + 0.019
0.141 + 0.024
0.100 + 0.018
Table 5: ACE results in pendulum (top) and drone (bottom) experiments from 10 random seeds.
For all methods, we randomly switch the environment (wind) c every 2s. To make fair comparison, except no-adapt or omniscient, all methods have the same learning rate for the inner-adapter Ag and the dimensions of è are also same (dim(é) 20 for the pendulum and dim(E) = 30 for the drone). ACE results from 10 random seeds are depicted in Table Figure I visualizes the drone experiment results. We observe that i) OMAC significantly outperforms the baseline; ii) OMAC adapts faster and faster as it encounters more environments but baseline cannot benefit from prior experience, especially for the bi-convex and deep versions (see Figure D.. and iii) Deep OMAC achieves the best ACE due to the representation power of DNN.

---

We note that in the drone experiments the performance of OMAC (convex) is only marginally better than the baseline. This is because the aerodynamic disturbance foree in the quadrotor simulation is a multiplicative combination of the relative wind speed, the drone attitude, and the motor speeds; thus, the superposition structure f( Y1 (1 ( ) Y2 (1) ) cannot easily model the unknown force, while the bi-convex and deep learning variants both learn good controllers, In particular, OMAC (bi-convex) achieves similar performance as the deep learning case with much fewer parameters. On the other hand, in the pendulum experiments, OMAC (convex) is relatively better because a large component of the c-invariant part in the unknown dynamics superposition with the c-dependent part. For more details and the pendulum experiment visualization we refer to the Appendix A.7
Related work
Meta-learning and representation learning. Empirically, representation learning and meta-learning have shown great success in various domains 77] In terms of control, reinforeiitrtt learning is able to solve challenging mult-task RL problems [8 [0] We remark that learning representation for control also refers to learn state representation from rich observations
42 but we consider dynamics representation in this paper. On the theoretic side, 4 5 33 have shown representation learning reduces sample complexity on new tasks, and "task diversity" is critical. Consistently, we show OMAC enjoys better convergence theoretically (Corollary D and empirically, and Theorem 6 also implies the importance of environment diversity. Another relevant line of theoretical work 45 47 uses tools from online convex optimization to show guarantees for gradient-based meta-lcarning, by assuming closeness of all tasks to a single fixed point in parameter space. We eliminate this assumption by considering hierarchical or nested online optimization procedure.
Adaptive control and online control. There is a rich body of literature studying Lyapunov stability and asymptotic convergence in adaptive control theory 12) Recently, there has been increased interest in studying online adaptive control with non-asymptotic metrics (e.g., regret) from learning theory. largely for settings with linear systems such as online LQR or LQG with unknown dynamics 14
The most relevant work [K] gives the first regret bound of adaptive nonlinear control with unknown nonlinear dynamics and stochastic noise. Another relevant work studies online robust control of nonlinear systems with a mistake guarantee on the number of robustness failures 49. However, all these results focus on the single-task case. To our knowledge, we show the first non-asymptotic convergence result for multi-task adaptive control. On the empirical side, [I BT combine adaptive nonlinear control with meta-learning, yielding encouraging experimental results.
16 24
Online matrix factorization. Our work bears affinity online matrix factorization, particularly the bandit collaborative filtering setting 6C 50 29 In this setting, one typically posits a linear low-rank projection as the target representation (e.g.. a low-rank factorization of the user-item matrix), which similar to our bilinear case. Setting aside the significant complexity introduced by nonlinear control, a key similarity comes from viewing different users as "tasks" " and recommended items as "actions". Prior work in this area has by and large not been able to establish strong regret bounds, in part due to the non-identifiabiiity issue inherent in matrix factorization. In contrast, in our setting, one set of latent variables (e.g., the wind condition) has concrete physical meaning that we are allowed to measure (ObserveEnv in Algorithm Db. thus side-stepping this non- non-identifiability issue.
7 Concluding remarks
We have presented OMAC, a meta-algorithm for adaptive nonlinear control in a sequence of unknown environments. We provide different instantiations of OMAC under varying assumptions and condi- tions, lcading to the first non-asymptotic convergence guarantee for multi-task adaptive nonlinear control, and integration with deep learning. We also validate OMAC empirically. We use the average control error (ACE) metric and focus on fully-actuated systems in this paper. Future work will seek to consider general cost functions and systems. It is also interesting study end-to-end convergence guarantees of deep OMAC, with ideas from deep representation learning theories. Another interesting direction is to study how to incorporate other areas of control theory, such as roboust control.
Broader impacts. This work primarily focused on establishing theoretical understanding of meta-adaptive control. Such fundamental work will not directly have broader societal impacts.
10

---

Acknowledgments and Disclosure of Funding
This project was supported in part by funding from Raytheon and DARPA PAI, with additional support for Guanya Shi provided by the Simoudis Discovery Prize. There is no conflict of interest.
References
[1] Michael O'Connell, Guanya Shi, Xichen Shi, and Soon-Jo Chung. Meta-lcarning-batn based robust adaptive flight control under uncertain wind conditions. arXiv preprint arXiv:2103.01952 2021.
[2] Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36 (4):1307-1319, 2020.
[3] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), 2020.
[4] Simon S Du, Wei Hu, Sham Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.0944 2020.
[5] Nilesh Tripuraneni, Chi Jin, and Michael I Jordan. Provable meta-learning of linear representa- tions. arXiv preprint arXiv:2002.11684 2020.
[6] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1-32, 2016.
[7] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):
1798 1828, 2013.
[8] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Picter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. arXiv preprint arXiv: 1802.07255 2018.
[9] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostie meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1133 2017.
[10] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. arXiv preprint 1rXiv:1803.111
2018.
[11] Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control, volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
[12] Karl Áström and Björ Wittenmark. Adaptive control. Courier Corporation, 2013.
[13] Nicholas M Boffi, Stephen Tu, and Jean-Jaqques E Slotine. Regret bounds for adaptive nonlinear control. In Learning for Dynamics and Control, pages 471-483. PMLR, 2021.
[14] Max Simchowitz and Dylan Foster. Naive exploration optimal for online lqr. International Conference Muchine Learning, pages 8937-8948. PMLR, 2020.
[15] Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, pages 1-47, 2019.
[16] Sahin Lale, Kamyar Azizzadenesheli, Babak Hassibi, and Anima Anandkumar. Logarithmic regret bound in partially observable linear dynamical systems. arXiv preprint arXiv:2003. 11277, 2020.
[17] Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Infor- mation theoretic regret bounds for online nonlinear control. arXiv preprint arXiv:2006. /2466, 2020.
11

---

[18] Yingying Li, Xin Chen, and Na Li. Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In NeurIPS, pages 14858-14870, 2019.
[19] Chenkai Yu, Guanya Shi, Soon-Jo Chung. Yisong Yue, and Adam Wierman. The power of predictions in online control. Advances in Neural Information Processing Systems, 33, 2020.
[20] Yiheng Lin, Yang Hu, Haoyuan Sun, Guanya Shi, Guannan Qu, and Adam Wierman. Perturbation- based regret analysis of predictive control in linear time varying systems. arXiv preprint arXiv:2106. 2021.
[21] Richard M Murray, Zexiang Li, and S Shankar Sastry. A mathematical introduction to robotic manipulation. CRC press, 2017.
[22] Steven M LaValee. Planning algorithms. Cambridge university press, 2006.
[23] Karl Johan Åström and Richard M Murray. Feedback systems. Princeton university press, 2010.
[24] Alon Cohen, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar. Online linear quadratic control. In International Conference on Machine Learning. pages 1029-1022 1038. PMLR, 2018.
[25] Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. In 2019 International Conference on Robotics and Automation (ICRA), pages 9784-9790. IEEE, 2019.
[26] Horia Mania, Stephen Tu, and Benjamin Recht, Certainty equivalence is efficient for linear quadratic control. arXiv preprint 1rXXv:1902.07826. 2019.
[27] Elad Hazan. Introduction online convex optimization. arXiv preprint 100.v::999.05207 2019.
[28] Naman Agarwal, Elad Hazan, Anirudha Majumdar, and Karan Singh. A regret minimization approach to iterative learning control. arXiv preprint arXiv:2102.134770 2021.
[29] Huazheng Wang, Qingyun Wu, and Hongning Wang. Factorization bandits for interactive recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
[30] Jaya Kawale, Hung H Bui, Branislav Kveton, Long Tran-Thanke and Sanjay Chawla. Efficient thompson sampling for online matrix-factorizatto recommendation. In Advances in neural information processing systems, pages 1297-1305, 2015.
[31] Spencer M Richards, Navid Azizan, Jean-Jacques E Slotine, and Marco Pavone. Adaptive- control-oriented meta-learning for nonlinear systems. arXiv preprint arXiv:2103.0444 2021.
[32] Yasin -abasi-Yadkorii Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In NIPS, volume 11, pages 2312-2320, 2011.
[33] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. arXiv preprint arXiv:2006. 16500, 2020.
[34] Lloyd Trefethen. Multivariate polynomial approximation in the hypereube. Proceedings of the American Mathematical Society, (4:(11)
2017.
[35] Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue. Robust regression for safe exploration in control. In Learning for Dynamics and Control, pages 608-60 619. PMLR, 2020.
[36] Xichen Shi, Patrick Spieler, Ellande Tang, Flena-Sorina Lupu, Phillip Tokumaru, and Soon-Jo Chung. Adaptive nonlinear control of fixed-wing vtol with airflow vector sensing. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5321-5327. IEEE, 2020.
[37] Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In NIPS, volume 3, page 5. Citeseer, 2007.
12

---

[38] Takeru Miyato, Toshiki Kataoka, Masanori Koyama. and Yuichi Yoshida. Spectral normalization for generative adversarial networks, arXiv preprint arXiv:1802.05957, 2018.
[39] Guanya Shi, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung. Neural-swarm: Decentralized close-proximity multirotor control using learned interactions. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 3241-3247. IEEE, 2020.
[40] Guanya Shi, Wolfgang Hönig, Xichen Shi, Yisong Yue, and Soon-Jo Chung. Neural-swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. IEEE Transactions on Robotics, 2021.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[42] Timothée Lesort, Natalia Díaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108:379-392, 2018.
[43] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pages 2170-2179 PMLR, 2019.
[44] Amy Zhang, Zachary C Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandkumar, Laurent Itti, Joelle Pineau, and Tommaso Furlanello. Learning causal state representations of partially observable environments. arXiv preprint arXiv: 1eeii.966..143 2019.
[45] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pages 1920-1930. PMLR, 2019.
[46] Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-leat stochastic gradient descent with biased regularization. In International Conference on Machine Learning, pages 1566-1575. PMLR, 2019.
[47] Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient- based meta- learning methods. arXiv preprint vrXiv:1906.02711 2019.
[48] Yasin Abbasi- Yadkori and Csaba Szepesvári. Regret bounds for the adaptive control of linear quadratic systems. In Proceedings of the 24th Annual Conference on Learning Theory, pages 1-26. JMLR Workshop and Conference Proceedings, 2011.
[49] Dimitar Ho, Hoang Le, John Doyle, and Yisong Yue. Online robust control of nonlinear systems with large uncertainty. In International Conference on Artificial Intelligence and Statistics, 2021.
[50] Guy Bresler, Devavrat Shah, and Luis Filipe Voloch. Collaborative filtering with low regret. In Proceedings of the 2016 ACM SIGMETRICS International Conference Measurement and Modeling of Computer Science, 2016.
[51] Dario Brescianini, Markus Hehn, and Raffaello Andrea. Nonlinear Quadrocopter Attitude Control: Technical Report. Report, ETH Zurich, 2013.
13


Reviews:
[{"review_id": "vBmeTOst0fi", "reviewer": "Reviewer_BB7T", "summary": "This paper introduces an integrated representation learning and adaptive control approach named Online Meta-Adaptive Control (OMAC). The approach uses a meta-adaptor to learn a shared representation that can be used across different environments, and uses an inner adaptor to perform environment-specific updates. Non-asymptotic convergence guarantee is provided for the approach. The approach and several baseline methods are evaluated to address a nonlinear pendulum control problem in the experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The research problem and its challenges are well defined and justified. The proposed approach is novel and has a theoretically sound basis. It can also be applied to various instantiations such as convex, element-wise convex and deep learning cases. The paper is also well organized and well written.\n\nThe following comments, mainly on the experiments, may help improve the quality of the paper:\n\n* The research problem in the paper is motivated using several robotic adaptation scenarios; however, the experiments only address a relatively simple pendulum control scenario. Evaluation using a robotics application, e.g., drone control, can make the experiments more relevant and convincing.\n\n* As a follow-up comment, the paper argues the generality of the approach, e.g., it can be used to model fully actuated robotic system. But such statements are not reflected in or supported by the experiments.\n\n* The experiments only provide results obtained by the proposed approach and its simplified versions as baselines (e.g., in Table 5). No comparison with other state-of-the-art adaptive control methods is provided in the experiments. It is difficult to evaluate whether the proposed approach outperforms the current state-of-the-art. \n\n* Providing video demonstrations of pendulum controls by different methods can be helpful for the audience to intuitively evaluate and understand the experimental results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Several future work topics are briefly mentioned in Section 7. No negative societal impact is identified.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "jPKBvq9qi72", "reviewer": "Reviewer_H6i6", "summary": "The paper proposes a novel algorithm for adaptive nonlinear control that can learn to stabilize a sequence of dynamical systems that share a common part of their dynamics, but differ in another part that can be estimated from data. It is assumed that the unknown part can be learned in terms of a set of shared features, which also need to be learned from data. The algorithm is decomposed into an outer and inner loop, where features are learned in the outer loop, and the specific component of the dynamics for a given environment is learned in an inner loop. The authors consider several approximation schemes for the unknown dynamical component (convex, element-wise convex, and a deep neural net), and prove asymptotic convergence for each of them. Empirical verification on the problem of stabilizing a pendulum with unknown wind acting on it demonstrates that the richest function approximation, the deep neural net, achieves the lowest control error, which is only greater than that of an omniscient controller by a factor of about 3.   ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is probably the convergence guarantee of the proposed learning algorithm. This appears to be the first in the field. The approximation schemes for the unknown parts of the system dynamics are well known from previous research. Related work is adequately cited, and connections are made to both work in the field of adaptive control, as well as representation learning in the fields of machine and reinforcement learning. In fact, another merit of this paper is that it draws on research from both fields to propose a unified view on the problem of meta-learning of system dynamics. \n\nThe submission appears to be technically sound, and the empirical verification is informative. One additional issue that the authors might address is how good these approximation schemes are with respect to the functions that they are trying to approximate. My understanding is that the results in Table 5 are averaged over all trials (as per Definition 1), which includes the learning period. How about learning all functions  $\\hat{f}_t^{(i)}$ from a sufficiently large amount of data in a supervised learning manner, and then using them for control, similar to the omniscient controller? To use Example 1, for the convex case this would mean finding the parameters in Eq. 4 using supervised learning (regression, in this case). If the environment-specific functions are estimated in this way, the bias in comparison due to the varying quality of the approximation bases could be removed.\n\nThe paper is written very clearly, and is very well organized. A minor typo: line 301, \"roboust\" -> \"robust\". I believe the results are very significant, because, first, the paper provides convergence proofs, and second, the method does look practical for solving problems involving control of a family of related systems that differ by an unknown part of their dynamics. I think it is a solid advance of the state of the art in adaptive control and meta learning.   \n\nUpdate: I am satisfied with the effort the authors made to investigate the effect of learning the features in a supervised learning manner, and I think they have addressed my review adequately. Seeing that my rating was a bit of an outlier in comparison with the others, I have revised it to be more in line with them. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have made a good effort to assess the limitations and potential negative societal impact of their work.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "I do not see any particular ethical concerns with this paper, beyond general concerns about potential dual use of control technology and robotics systems. This is mainly theoretical work that is far removed from potential dangerous applications."}}, {"review_id": "exGx_te_nyz", "reviewer": "Reviewer_UjXF", "summary": "This paper proposes and analyzes a multi-task learning approach to adaptive nonlinear control under changing environments.  The paper assumes that the dynamics can be decomposed as the sum of a known (exponentially input-to-state-stable) component and an unknown and changing environment dependent component.  The adaptive control problem is conducted over N episodes (the outer loop) of T time steps (the inner loop).  By assuming that the unknown environmental dependent component can be defined in terms of a set of common features across environments $\\phi$ and an environment dependent term $c^{i}$ for the i-th environment, a modular interaction protocol, called Online Meta-Adaptive Control (OMAC) is suggested wherein the outer loop is treated as an online learning problem with the goal of learning the shared representation $\\phi$, and the inner loop is viewed as an adaptive control problem wherein the environment dependent component $c^{i}$ is the unknown term to be learned.  It is then shown under suitable convexity assumptions that the Average Control Error (the average of the state norms over both inner and outer iterations) achieved by OMAC is sublinear in both the number episodes $N$ and the task horizon length $T$.  A deep learning based approach for learning the common features $\\phi$ is also proposed.  The paper concludes with an illustrative numerical example (an uncertain inverted pendulum) comparing various flavors of OMAC to non-adaptive and traditional adaptive control methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI thoroughly enjoyed reading this paper.  I found it to be clearly written, and the problem being posed and addressed to be of significant interest to the learning and control community.  The results are rigorously stated and technically correct.  In my view, the main novelty/contributions of the paper are (i) defining a formalism for reasoning about meta-learning in the context of nonlinear adaptive controls and (b) recognizing that once suitably posed, the resulting problem can be naturally addressed using (nested) online convex optimization techniques, leading to clean and intuitive theorem statements and proofs.\n\nThat being said, I do have some questions/comments that I would like to have clarified (and I believe would also help improve the quality of the paper if addressed):\n\n- How would the results change for a system that isn’t fully actuated?  This is a very strong assumption that not even the simple inverted pendulum example used in the numerical experiments section satisfied.\n\n- The assumption of exponential input-to-state-stability (E-ISS) is not standard in nonlinear adaptive control: it is in fact a very strong assumption made in [12] (Boffi et al.) that may also be difficult satisfy.  It is also not appropriate to cite [12] as a “standard nonlinear adaptive control” reference, as it is very recent and makes many non-standard assumptions, including E-ISS.  I want to emphasize that I have no problem with this assumption being made, especially when attempting to derive a first result for a novel problem formulation, but take issue with its framing as being standard.\n\n- A comment is made early on that initial conditions can be assumed to be 0 wlog, as by E-ISS, they only result in a constant additive term.  This is obviously true if all initial conditions are drawn from a compact set, e.g., that $||x^{i}_0||\\leq B_0$ for all episodes $i$ and some $B_0>0$.  However, the “environment switching” process suggested in the experimental section is that episodes do not require hard resets (specifically, a 60s task is divided into 30 x 2s episodes) — it is therefore not clear that the assumption of $||x^{i}_0||\\leq B_0$ holds anymore.  In particular, if $f_0 + f - \\hat{f}$ is unstable, then the state may be blowing up on you, in which case $||x^{i}_0||\\leq B_0$ is no longer obviously true. \n\n- Related to the above question, if hard resets indeed end up being required to ensure the desired error bounds, then is it realistic to ask for $N>T$?  For a 10s task with sampling time of 0.1s, this would mean a minimum of 100 novel environments.  Similarly, what was the sampling time $\\delta$ used to discretize the continuous time dynamics for the inverted pendulum used in Section 5: was it such that $30>2/\\delta$?\n\n- It would be good to see at least an informal quantification of the computational complexity tradeoffs associated with choosing larger or smaller $N$ vs. $T$.  In particular, if I consider the example in Section 5 conducted over an overall horizon of $H=60/\\delta$, why shouldn’t I set $N=60$ and $T=1/\\delta$ instead of $N=30$ and $T=2/\\delta$?  Given that the model for environmental dependent dynamics assumes that the environment specific term $c$ is constant over an episode, this suggests smaller $T$ and larger $N$ will lead to both better bounds relative to standard adaptive control, and a more dynamic controller.  However, unless I’m misunderstanding, we recover standard adaptive control in the setting where $N=H$ and $T=1$.  I believe what’s missing from the theory is capturing the computational complexity of the approach (which increases with $N$ as this means more meta-updates over an overall horizon $H$), but I’m not sure.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above regarding limitations with respect to translating theory into practice.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "FFk6whAlYq-", "reviewer": "Reviewer_ZMnJ", "summary": "This paper presents an approach to adaptive nonlinear control in a changing environment: an agent must regulate a nonlinear dynamical system which changes from episode to episode. The work focuses on fully actuated, control affine systems, and takes an approach which aims to identify the true dynamics of the system and choose controls to cancel out the effect of these dynamics to regulate the system. The approach involves multi-task online learning, whereby non-varying elements of the dynamics are learned between episodes, while during an episode, the model adapts to estimate the part of the dynamics that vary.\n\nThe authors show that depending on the structure of the unknown dynamics, tools from online convex optimization can yield algorithms with sublinear cumulative control error bounds. The same structure can be used with a neural network model, albeit without guarantees on performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach takes a clean formulation to the problem of adaptive control in environments that change from episode to episode. The choice of control scheme (certainty equivalent) and focus on fully actuated control affine systems allow for a straightforward application of results from the online convex optimization literature to this domain. I thought the organization of the paper was good, starting from the general formulation, and clearly stating the assumptions for each instantiation before detailing the theoretical results. While making strong assumptions (that the true system lies in the model class without any model mismatch), there are some interesting theoretical insights, such as the connection of environment diversity to cumulative regret, and the results guiding the selection of learning rates for Corollary 4.\n\nHowever, the paper does have some weaknesses, primarily related to the connection of the theoretical results to practice:\n- First, the experimental results do not reflect the theoretical procedure except in the joint convex case. The assumption that for an element-wise convex system that the true value of c would be presented after the trial is strong, and theoretical results on handling the error from using $\\hat{c}^{(T)}$ would strengthen the paper.\n- Corollary 4 considers an adaptive control baseline which optimizes both Theta and c. However, in the experiments the Baseline only adapts c, and not both. This severely limits the representational capacity of the Baseline, and not a fair comparison. Indeed, the impressive performance of the deep learning solution suggest that representational capacity is key to performing well on the chosen experiment.\n- The experimental domain is very simple with only 2 states, and a nominal model that is very accurate, apart from minor mismatch in wind and gravity. Furthermore, it does not match the theory, as it is under actuated (B is not full rank). It would have been helpful to see how the algorithm scales to larger state and action spaces.\n- Comparison to other meta-adaptive control literature is missing. Even though many of these works do not perform online learning, it would be helpful to see the difference in ACE in running the nominal controller for K episodes to gather data, meta-training, and subsequently deploying the meta-trained adaptive controller.\n\nOverall, this paper presents what an application of results from the OCO literature to meta-adaptive nonlinear control. While this connection is novel to me, the paper is hindered by limited experiments which do not showcase the applicability of the theoretical results to realistic systems, and do not compare to strong enough baselines.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion of limitations could be expanded to highlight the strong assumption made in the theoretical results that the true system is in the modeling class. Given the theoretical nature of this work, the lack of a section on negative societal impact is alright.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to tackle the problem of adaptive nonlinear control over a set of related environments, by learning a shared representation. The approach is sufficiently evaluated and the paper well structured.
This is an interesting paper that would be a positive contribution to the conference.
The authors are advised to make the promised changes.
Decision: Accept (Poster)