Meta-Adaptive Nonlinear Control: Theory and Algorithms
Authors: Guanya Shi
Kamyar Azizzadenesheli
Michael O'Connell
Soon-Jo Chung
Yisong Yue
Venue: NeurIPS 2021 Poster (2021)
Keywords: Online Learning
Representation Learning
Adaptive Control
Nonlinear Control
Meta-learning

Abstract: We present an online multi-task learning approach for adaptive nonlinear control, which we call Online Meta-Adaptive Control (OMAC). The goal is to control a nonlinear system subject to adversarial disturbance and unknown \emph{environment-dependent} nonlinear dynamics, under the assumption that the environment-dependent dynamics can be well captured with some shared representation. Our approach is motivated by robot control, where a robotic system encounters a sequence of new environmental conditions that it must quickly adapt to. A key emphasis is to integrate online representation learning with established methods from control theory, in order to arrive at a unified framework that yields both control-theoretic and learning-theoretic guarantees. We provide instantiations of our approach under varying conditions, leading to the first non-asymptotic end-to-end convergence guarantee for multi-task nonlinear control. OMAC can also be integrated with deep representation learning. Experiments show that OMAC significantly outperforms conventional adaptive control approaches which do not learn the shared representation, in inverted pendulum and 6-DoF drone control tasks under varying wind conditions.

Meta-Adaptive Nonlinear Control: Theory and Algorithms
Guanya Shit, Kamyar Azizzadenesheli, Michael O'Connell', Soon-Jo Chung', Yisong Yue Michael
'Caltech Purruee University
{gshi moc, s jjchung.yyue)occalteed edu. kamyar edu
Abstract
We present an online multi-task learning approach for adaptive nonlinear control, which we call Online Meta-Adaptive Control (OMAC). The goal is to control a nonlinear system subject to adversarial disturbance and unknown environment- dependent nonlinear dynamics, under the assumption that the environment- dependent dynamics can be well captured with some shared representation. Our approach is motivated robot control, where robotic system encounters quence of new environmental conditions that it must quickly adapt to. A key emphasis is integrate online representation learning with established methods from control theory, in order to arrive at unified framework that yields both control-theoretic and learning-theowe thhooreicc guarantees. We provide instantiations of our approach under varying conditions, leading to the first non-asymptotic end- to-end convergence guarantee for multi-task nonlinear control. OMAC can also be integrated with deep representation learning. Experiments show that OMAC significantly outperforms conventional adaptive control approaches which do not learn the shared representation, in inverted pendulum and 6-DoF drone control tasks under varying wind conditions
Introduction
One important goal in autonomy and artificial intelligence is to enable autonomous robots to learn from prior experience to quickly adapt new tasks and environments. Examples abound in robotics, such as drone flying in different wind conditions [I] a manipulator throwing varying objects (2), quadruped walking empirical evidence, when designing such adaptive systems, two important theoretical challenges arise, discussed below.
changing terrains
6 Though those examples provide encouraging
First, from a learning perspective, the system should be able to learn an "efficient" representation from prior tasks, thereby permitting faster future adaptation, which falls into the categories of representation learning or meta-learning. Recently, line of work has shown theoretically that learning representations (in the standard supervised setting) significantly reduce sample complexity Empirically, deep representation learning or meta-learning has achieved success in many applications 77), including control, in the context of mett-rriinforeement learning [8]] However, theoretical benefits (in the end-to-end sense) of representation learning or meta-learning for adaptive control remain unclear.
new tasks
Second, from a control perspective, the agent should be able to handle parametric model uncertainties with control-theoretic guarantees such stability and tracking error convergence, which is a common adaptive control problem [ For classic adaptive control algorithms, theoretical analysis often Moreover, many recent
involves the use of Lyapunov stability and asymptotic convergence [11]
Code and video: http:://gthhhcooooo/aa///l/aaaar
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

studies aim to integrate ideas from learning, optimization, and control theory to design and analyze adaptive controllers using learning-theor theoretic metrics. Typical results guarantee non-asymptotic convergence in finite time horizons, such as regret 133177 and dynamic regret [18-20]. However, these results focus on a single environment or task, multi-task extension, especially whether and how prior experience could benefit the adaptation in new tasks, remains an open problem.
Main contributions. In this paper, we address both learning and control challenges in unified framework and provide end-to-end guarantees, We derive a new method of Online Meta-Adaptive Control (OMAC) that controls uncertain nonlinear systems under a sequence of new environmental conditions. The underlying assumption is that the devironment-depende well be captured by a shared representation, which OMAC learns using meta-adapter. OMAC then performs eavironment-specifee updates using an inner--ddaper.
unknown dynamics can
We provide different instantiations of OMAC under varying assumptions and conditions. In the jointly and element-wise convex cases, we show sublinear cumulative control error bounds, which to our knowledge is the first non-asymptotic convergence result for multi-task nonlinear control. Compared to standard adaptive control approaches that do not have a meta-adapter, we show that OMAC possesses both stronger guarantees and empirical performance. We finally show how to integrate OMAC with deep representation learning, which further improves empirical performance.
2 Problem statement
We consider the setting where a controller encounters sequence of N environments, with each environment lasting 7 time steps. We use outer iteration to refer to the iterating over the N environments, and inner iteration refer to the T time steps within an environment.
Notations: We use superscripts (e.g., (i) in x; ((()) to denote the index of the outer iteration where
1 <is i N, and subscripts (c.g.. t in x; to denote the time index of the inner iteration where 1 TT T. We use step(i, t) to refer to the inner time step at the 2th outer iteration. || || denotes the 2-norm of a vector the spectral norm of matrix. || denotes the Frobenius norm of a matrix and Amin(-) (-) denotes the minimum eigenvalue of real symmetric matrix. vec(.) E I denotes the vectorization of a m x n matrix, and @ denotes the Kronecker product. Finally, we use w1zt. to denote sequence {ur; u2, ",,,"",,, J
We consider a discrete-time nonlinear control-affine system 21 22 with envirommeent-dependent uncertainty f(z c). The dynamics model at the 7th outer iteration is:
(8) xt-1 = fo(xx") B(2
(0) uu f(( (()) + w 1stsT,
(
(1)
where the state x1") R", the control u." E FT fo R + R" is known nominal dynamics model, B I= + =x is known state-dependent actuation matrix, c(i) C R² is the unknown parameter that indicates an environmental condition, ʃ R" x R + R is the unknown c(O)_ (6) is disturbance, potentially adversarial. For simplicity we define dependent dynamies model, and w
B(") =aa and f( = f(x (2) c.)).
Interaction protocol. We study the following adaptive nonlinear control problem under N unknown environments. At the beginning of outer iteration i, the environment first selects (+)) (adaptively and adversarially), which is unknown the controller, and then the controller makes decision u under unknown dynamics (x)
(i)
(8) clO) and potentially adversarial disturbances (8) To summarize:
Outer iteration i. policy encounters environment (i E (1...., .,, N)), associated with unob- served variable (99 (e.g., the wind condition for flying drone). Run inner loop (Step 2).
(8) 2. Inner loop. Policy interacts with environment for T time steps, observing xt and taking action (.)
with state/action dynamics following |
3. Policy optionally observes ct at the end of the inner loop (used for some variants of the analysis).
4. Increment i = i and repeat from Step I.
We average control error (ACE) our performance metric:

---

Definition (Average control error). The average control error (ACE) of N outer iterations (i.e., N environments) with each lasting T time steps, is defined ACE TN Ei Σ ||=]""]
ACE can be viewed as a non-asymptotic generalization of steady-state error in control the following assumptions on the actuation matrix B, the nominal dynamics fo. and disturbance (
23 We make
Assumption (Full actuation, bounded disturbance, and e-ISS assumptions). We consider fully- actuated systems, i.e., for all I, vank(B((z) = ||/|"|| & W,vt. Moreover, the nominal dynamics fo is exponentially input-to- st-ttte stable (e-ISS): let constants B. y and S p For sequence V1-t-1 E R", consider the dynamics 2k+1 fo(Øx) + 1 s & S - 1. 24 satisfies:
| || S Bp |0
k=1
With the e-ISS property in Assumption average squared loss between B Ut
we have the following bound that connects ACE with the (8)
+ we and ft
Lemma 1. Assume 11 O, Vi. The average control error (ACE) is bounded as:
|00
4
11 +
r-ll TN
(3)
1 1-
TN
The proof can be found in the Appendix A.1 We assume 21 for simplicity: the influence of (8) non-zero and bounded zi is constant term in each outer iteration, from the e-ISS property (
(7)
Remark the e-ISS assumption and the ACE metric. Note that an exponentially stable linear system fo(2;) = Ax, (i.e., the spectral radius of A is < 1) satisfies the exponential ISS (c-ISS) assumption. However, in nonlinear systems e-ISS a stronger assumption than exponential stability. For both linear and nonlinear systems, the c-ISS property of fo is usually achieved by applying some
stable feedback controller to the system i.e., fo is the closed-loop dynamics I 24 IC
e-ISS
assumption is standard in both online adaptive linear control 124 16 [4] and nonlinear control [13]
and practical robotic control such as drones (25] ACE we consider regulation task, but can also capture trajectory tracking task with vime-variant nominal dynamics fo under incremental We only consider the regulation task this paper for simplicity.
stability assumptions
[
Generality. We would like to emphasize the generality of our dynamics model I The nominal control-affine part can model general fully-actuated robotic systems via Euler- Langrange equations and the unknown part f(r, c) is nonlinear in and We only need to assume the disturbance and nonlinear cases. For example, wi can model extra (æ, c)-dependent uncertainties or adversarial disturbances. Moreover, the environment sequence c(1:N) could also be adversarial. In term of the extension to under-actuated systems, all the results in this paper hold for the matched uncertainty setting [3] i.c.
2112
22
w
bounded, which is more general than stochastic settings in linear
in the form #++ = fo(a;") + Bxx (( - f( (6) c()))
)) + (+) where B(x is not necessarily Generalizing other
full rank (e.g.. drone and inverted pendulum experiments in Section 5
under-actuated systems is interesting future work.
3 Online meta-adaptive control (OMAC) algorithm
The design of our online meta-adaptie adaptive control (OMAC) approach comprises four pieces: the policy class, the function class, the inner loop (within environment) adaptation algorithm A2, and the outer loop (between environment) online learning algorithm A1.
Policy class. We focus on the class certainty-equivaleence controllers 13 general class of model-based controllers that also includes linear feedback controllers commonly
26 14
which is a
For example, consider z++1 Ja + 2sin v ue- With feedback controller ū, Mr xt sin
the closed-loop dynamics is 2s+1 fe Ue. where fo(x) 1 e-ISS.
3

---

studied in online control 26 14|
After model is learned from past data, controller is designed
by treating the learned model as the truth [2]] Formally, at step(i. t), the controller first estimates
(( (an estimation of f()) based on past data, and then executes " B ft where (-)1 is the pseudo inverse. Note that from Lemma || the average control error of the omniscient controller using P( = f(4) (i.e., the controller perfectly knows f(r c)) is upper bounded ACE & yW/(1 - p).
ACE (Cmonisciee) can be viewed as a fundamental limit of the certainty seuivalt-e policy class.
Function class F' for representation learning. In OMAC, we need to define a function class F(o(2; θ), é) to compute f(()) (i.e., f( F()(z)) θ). where (parameterized by θ) is representation shared by all environments, and is an environment-specffe latent state. From a theoretical perspective, the main consideration of the choice of F()(x), is how it effects the resulting learning objective. For instance, Φ represented by a Deep Neural Network (DNN) would lead to a highly non-convex learning objective. In this paper, we focus on the setting θ E RP, C E IR", and p >> h. i.e., it is much more expensive to learn the shared representation & (e.g., DNN) than "fine-ttning" via è in specific environment, which is consistent with meta-learning 9 and practices.
representation learning
7
Inner loop adaptive control. We take a modular approach in our algorithm design, in order to cleanly leverage state-of-the-art methods from online learning, representation learning, and adaptive control. When interacting with a single environment (for T time steps), we keep the learned representation ø fixed, and use that representation for adaptive control by treating è as unknown low-dimensional parameter. We can utilize any adaptive control method such as online gradient descent, velocity gradient, or composite adaptation 27
Outer loop online learning. We treat the outer loop (which iterates between environments) as an online learning problem, where the goal is to Iearn the shared representation & that optimizes the inner loop adaptation performance. Theoretically, we can reason about the analysis by setting up a hierarchical or nested online learning procedure (adaptive control nested within online learning).
Design goal. Our goal is design meta-adaptive controller that has low ACE, ideally converging to ACE(omniscien)) as T, N - cc. In other words, OMAC should converge to performing as good as the omniscient controller with perfect knowledge f(x, c).
Algorithm D describes the OMAC algorithm. Since & is environment-invarian and p >> h, we only adapt θ at the end of each outer iteration. On the other hand, because c( varies in different environments, we adapt each inner step. As shown in Algorithm at step(i, t), after applying
(8) the controller observes the next state o and computes: yi (4) [)( + B,ah un
(8)
((()) wi
(8) which is a disturbed measurement of the ground truth ft We then define e, ( 2) &
[[F(((z))) (1), 6), c) 2 || as the observed loss step(i, t), which is squared loss between the disturbed measurement of (((i) and the model prediction F(o(2{)) Ô). ( E).
AI, A2, ObserveEnv), we consider three settings: Instantiations. Depending on (F(d(1; 0), (F
Convex case (Section 4.1 The observed loss (()) is convex with respect to ê and è
Element- wise convex case (Section 4.2): Fixing Ô è, e(9) is convex with respect the other.
Deep learning case (Section 5p: In this case, we use a DNN with weight é to represent ø.
4
Main theoretical results
4.1
Convex case
In this subsection, we focus on a setting where the observed loss 4 (Ô, è) = [[F((((;;; 0). - 112
convex with respect to Ô and é We provide the following example to illustrate this case and highlight its difference between conventional adaptive control (e.g.. [3 [2]).
"Tii upper bound is tight. Consider scalar system 2141 = ax; + u f(xx) + w with [a| < 1 and w constant. In this case p Jal, 1, and the omniscient controller u, f(x)) yields ACE 7/wl/(1 p).

---

Algorithm 1: Online Meta-Adaptive Control (OMAC) algorithm
Input: Meta-adapter A inner-adapter A2; model (x((x;; O), e); Boolean ObserveEnv
for 1, .NV do
The environment selects e(3) for 1 = 1, 1, ,T do
Compute Jt F(o(a ( (.)))
Execute up
(i) B, (i)t p(()
//certannt--quiivlleeet
policy
Observe T 144 Vi ( = f(xt (0) dd0) - w and el ( (Ô. = (P) 1
1/y6)) is noisy measurement of and g(i)
observed loss
(2) (2) by A2 1/9t (9 is function of
Construct inner cost function Di
Inner-adaptation: 241 - Az(d, .9:1)
(i)
ObserveEnv then Observe c(0) //only used some instantiations
Construct an outer cost function ()) by A l/G is a function of Q Meta-adaptation: (1+1) A1 G(I))
Example 1. We consider model ((b(2;; θ), -1))))))()0000
()(x) + Y2(2) to estimate f: A = Yi (4) 1 + Y2(2,
(4)
where Yi RD - Rnxp, Y2 R - R are two known bases. Note that conventional adaptive control approaches typically concatenate and è and adapt on both at each time step, regardless of the environment changes (e.g., [3]. Since h, such concatenation is computationally much more expensive than OMAC, which only adapts θ outer iterations.
Because e.th (Ô, é) is jointly convex with respective to and é, the OMAC algorithm in this case falls into the category of Nested Online Convex Optimization (Nested OCO) (28) The choice of
Pi G6,, (i, A2 and ObserveEnv are depicted in Table Note that the convex case OMAC does not need to know c(0) in the whole process (ObserveEnv = False).
F(((z;0),)) d (c)
Any F model such that (G, Vell (6() e(f)) è
é) is convex (e.g.. Example
(()
E V g(0) (6))) cl )
A
With convex set 1G, Ai initializes ( Kr, and returns ( Kc, Vi. Ai sublinear regret, i.c., the total regret of A, is o(N) (c.g., online gradient descent)
Aa
With convex set K22, Vi, A2 initializes a( Kig and returns e K2,, Vt. A2 has sublinear regret, i.e., the total regret of Aa is N o(T) (e.g,, online gradient descent)
ObserveEnv
False
Table I: OMAC with convex loss
As shown in Table 1 at the end of step(i, t) fix ê 6 and update (6)
Kr using (t) On the other hand, at the end of outer Ae( -1:)) which is an OCO problem with linear costs 91:t-
(6)
iteration i, we update ((+1) E Ki using (,()()) G(a))), which is another OCO problem with
linear costs ((a)), From 228
we have the following regret relationship:
Lemma 2 (Nested OCO regret bound, (28
OMAC (Algorithm 7 specified by Table 1 has regret:
N
T
N
i
(0,
min BE
min EKa
t=
t=1
(5)
Σ
(6(0)))
min
())
g (c
Σ
min
Σ
91
(c(0))
ek22
1=1
the total regret of As (T-o(N)
the total regret No.)-o(T)
5

---

Note that the total regret of A1 is - o(N) because G is scaled up by factor of T, With Lemmas and we have the following guarantee for the average control error.
Theorem 3 (OMAC ACE bound with convex loss). Assume the unknown dynamics model is [xx F( θ), c). Assume the true parameters θ K1 and clt) K22 Vi. Then OMAC (Algorithm /
specified by Table 7 ensures the following ACE guarantee:
( (N)
ACE
W2
To further understand Theorem 3 and compare OMAC with conventional adaptive control approaches, provide the following corollary using the model in Example
Corollary 4. Suppose the unknown dynamics model is fxx c) (z)) where Yı R + Raxp, Y2 R - m are known bases. We assume ||||| S Ke and || ||e( S Ke, Vi. Moreover, we assume ||(()) < K1 and |||(())]| < K2, Vz. In Table ! we use f( Y2(=("))10) Y2 (()) and Online Gradient Descent (OGD) 277 for both A1 and A2, with learning rates 17() (4) respectively. We set K₁ =O: ||ô|| S Ke} and K2 {ē: ||e|| I Ke). we schedule the learning rates as:
and 1ts
2K
2K.
( KKi + 4K1KgK. + 2K W)TVi "
(4K2 + K2 2KK22K Ke + 2K2W)
G.
then the average control performance is bounded as:
1 VN
ACE(OMAC) & 1-p W2 + 3
KaCi
+
KcC2
VT
Moreover, the buseline adaptive control which uses OGD to aulppt Ô and at each time step safisfies:
? W2 3 K2 + K3 C C2 ACE(baseline adaptive control) - 1- pV
1 VT
Note that ACE(baseline adaptive control) does not improve as N increases (i.e., encountering more environments has no benefit). p >> h, potentially have K1 >> K₂ and Ke >> Ke, C₁ >> C2- Therefore, the ACE upper bound of OMAC better than the baseline adaptation if N T, which is consistent with recent representation learning results 55. It is also worth noting that the baseline adaptation is much more computationally expensive, because it needs to adapt both Ô and è each time step. Intuitively, OMAC improves convergence because the meta-adapter A1 approximates the environment-invariinn part Y (z)),, which makes the inner-adapter Aa much more efficient.
4.2 Element-wise convex case
In this subsection, we consider the setting where the loss function ( is element-wise convex
with respect to θ and 2, i.e., when one of Ô or è fixed, ((ه) is convex with respect to the other one.
For instance, F could function as depicted in Example bilinear models are also studied in representation learning
Outside the context of control, such
5 and factorization bandits 29
Example We consider a model F(ò(:;; 6), = Y (x) θe estimate f:
f = Y y (0))))
(6)
where Y: n - R is a known basis, 60) E Roxh and 2(00 E R.A Note that the dimensionality
of Ô is p = ph. Conventional adaptive control typically views Ôč as vector in Rd and adapts it at each time step regardless of the environment changes
Compared Section the element-wise convex case poses new challenges: the coupling between é and brings inherent non-identifiability issues; and ii) statistical learning guarantees

---

These challenges are further amplified by the underlying unknown nonlinear system ① Therefore in this section we set ObserveEnv = True, i.c., the controller has access to the true environmental condition e( at the end of the 7th outer iteration, which is practical in many systems when c(*) has concrete physical meaning, e.g., drones with wind disturbances 3
typical need i.i.d. assumptions on ((+) and J
(4)
4 5
9 (c) F(o(z;)) C).
Any F model such that (,, is clement-wise convex (c.g., Example (6(+), 2)
(6)
Σe (6,
Ai & A ObserveEnv
Same Table
True
Table 2: OMAC with element-wise convex loss
The inputs to OMAC for the element-wise convex case are specified in Table Compared to the convex case in Table difference includes i) the cost functions 9t and (l are convex, not necessarily linear; and ii) since ObserveEnv = Truc, in G we use the true environmental condition cO) With inputs specified in Table 2 Algorithm has ACE guarantees the following theorem.
Theorem (OMAC ACE bound with element wise convex loss). Assume the unknown dynamics model is f(x,c) c) F( +), c). Assume the true parameter θ E Kr and c E K22, Vi. Then
OMAC (Algorithm)
7 specified by Table 2 eensures the following ACE guarantee:
o(T) o(N) ACE 1 1 p W2
N
4.2.1 Faster convergence with sub Gaussian and environment diversity assumptions
Since the cost functions 9r (2) and G in Table are not necessarily strongly convex, the ACE upper bound in Theorem is typically - W2 O(L/VT) O(L/VN) (e.g., using OGD for both AA, and A2). However, for the bilinear model in Example 2 is possible to achieve faster convergence with extra sub Gaussian and environment diversity assumptions
F(o(2; 2). of (C)
bilinear model in Example ( (6()), ||| + Σj= Ee= (6, c()) with some > 0
())
A
( = arg ming (1) (Ô) (i.e., Ridge regression)
As ObserveEnv
Same as Table
True
Table 3: OMAC with bilinear model
With a bilinear model, the inputs to the OMAC algorithm are shown in Table With extra assumptions on w (i) and the environment, have the following ACE guarantees.
Theorem 6 (OMAC ACE bound with bilinear model). Consider an unknown dynamics model f( (=) where Y IR" R is known basis and E Roxh, Assume each component of the disturbance w. is Gausb-aa the true parameters ||||| S Ke. ||c(O]| S Ke, Vi, and
[x)(=)) < Ky,Vx. Vx. Define z(,1) c(J)T (x( R and assume environment diversity: Amin Ej= Σr= z(5)T z(5)) 2ii). Then OMAC (Algorithm specified by Table has the following ACE guarantee (with probability at least - 8);
7 ACE < 1-p W2 +
o(T) T
+ o Jog(NN// log(N)
N
If we relax the environment diversity condition to 2(Vi), the last term becomes Of (TR(NT/5)
VN
The Lub-Gaussian assumption is widely used in statistical learning theory to obtain concentration bounds 32 4. The environment diversity assumption states that (1) provides "new information"
7

---

in every outer iteration such that the minimum eigenvalue of Σ i=1 E1=12011 2,"¹ grows linearly increases. Note that do not need Amin (E Ei= Z 2 to increase T goes up. Moreover, if we relax the condition to (2(v),, the ACE bound becomes worse than the general element-wise convex case (the last term is O()/VN)), which implies the importance of "linear" environment diversity (i))). Task diversity has been shown to be important for representation learning We provide proof sketch here and the full proof can be found in the Appendix A.6
Proof sketch. In the outer loop we use the martingale concentration bound diversity assumption bound [|x((+1) l O ( ), Vi with probability at least
2 and the environment
Then, use Lemma to show how the outer loop concentration bound and the inner loop regret bound of Ag translate to ACE.
Deep OMAC and experiments
We now introduce deep OMAC, deep representation learning based OMAC algorithm. Table shows the instantiation. As shown in Table Deep OMAC employs a DNN to represent Φ, and uses gradient descent for optimization. With the mode p θ) é, the meta-adapter A1 updates the representation & via deep learning, and the inner- A2 updates a linear layer č.
F(o(z;0) 2)
cir; ②) where 6(2;; 0) R"
R is neural network with weight O
(()(6) (C)
(+())))
A
Gradient descent: ( -- ( Vo E (6, 2
A ObserveEnv
Same Table
False
Table 4: OMAC with deep learning
To demonstrate the performance of OMAC, we consider two sets of experiments:
Inverted pendulum with external wind, gravity mismatch, and unknown damping. The continuous-time model is ml2@ miğ sin é/0 is the angular velocity/acceleration,, m the pendulum mass and the length, is the gravity estimation, cis the unknown parameter that indicates the external wind condition, and f(0, 0, c) is the unknown dynamics which depends on but also includes c-invariant terms such as damping and gravity mismatch. This model generalizes 35 by considering damping and gravity mismatch.
[00, 0 c), where is the pendulum angle,
6-DoF quadrotor with 3-D wind disturbances. We consider a 6-DoF quadrotor model with unknown wind-dependent aerodynamic force (x, E R3, where x E IR13 is the drone state (including position, velocity, attitude, and angular velocity) and the unknown parameter indicating the wind condition. We incorporate realistic high-fidelity aerodynamic model from [36].
We consider different controllers in the experiment (see more details about the dynamics model and controllers in the Appendix A.7
No-adapt is simply using ft O, and omniscient is using ((() f((0)
OMAC (convex) is based on Example where p((0) = (x )(0)) + Y2(x, (8) (( We use
random Fourier features 37 K3 to generate both Y, and Y2. We use OGD for both Ai and As in
Table I
OMAC (bi-convex) based on Example 2 where f(()) = Y( ((0( Similarly, we use random Fourier features to generate Y. Although the theoretical result in Section requires in Table instead of >T ((3) (Ô, c(.). We also deploy OGD for A₁ and A2. Baseline has the same procedure except with ( (1),, i.c.. calls the inner-adapter Ag every step and does not update é, which standard in adaptive control
ObserveEnv Truc, we relax this our experiments and use ()) Σ
'The intuition behind this structure is that, any analytic function f(r, e) can be approximated by ((x)c(c) with universal approximator ó (3 We provide detailed and theoretical justification in the Appendix A.7

---

Baseline
(convex)
convex) ACE 0.074m
(deap)
0.199m
0.125m
04
Postonn (m)
0.2
0.0
-02
-04
-5-
(a) xz-position trajectories (top) and force predictions (bottom) in the quadrotor experiment from one random sced. The wind condition is switched randomly every 2s (indicated by the dotted vertical lines).
Average control error for each 25 wind condition
Average force prediction error (Newtons) for each 25 wind condition
0.5
5
DAWC (bi-conves)
CMAC (convext
DMAC (deep)
3
2
10
11
12
13
15
14
Wind conditonn Index
Wind condition Indox
(b) The evolution of control eтrоr (left) and prediction error (right) in each wind condition. The solid lines average 10 random seeds and the shade show standard deviations. The performance OMAC improves the number of wind conditions increases (especially the bi-convex and deep variants) while the baseline
Figure 1: Drone experiment results.
OMAC (deep learning) is based on Table We use a DNN with spectral normalization 38 251
39 40
represent Φ, and use the Adam optimizer 41 for A1. Same other methods, A is also an OGD algorithm.
no-adapt
baseline 0.311 + 0.112
OMAC (convex)
0.663 + 0.142
0.147 + 0.047 0.047
external wind
OMAC (bi-convex)
OMAC (deep)
omniscient
0.129 + 0.044
0.093 + 0.027
0.034 0.017 0.017
UI
no-adapt
baseline
OMAC (convex)
0.374 + 0.044
0.283 + 0.043
0.251 + 0.043
OMAC (bi-convex)
OMAC (deep)
omniscient
050 + 0.019
0.141 + 0.024
0.100 + 0.018
Table 5: ACE results in pendulum (top) and drone (bottom) experiments from 10 random seeds.
For all methods, we randomly switch the environment (wind) c every 2s. To make fair comparison, except no-adapt or omniscient, all methods have the same learning rate for the inner-adapter Ag and the dimensions of è are also same (dim(é) 20 for the pendulum and dim(E) = 30 for the drone). ACE results from 10 random seeds are depicted in Table Figure I visualizes the drone experiment results. We observe that i) OMAC significantly outperforms the baseline; ii) OMAC adapts faster and faster as it encounters more environments but baseline cannot benefit from prior experience, especially for the bi-convex and deep versions (see Figure D.. and iii) Deep OMAC achieves the best ACE due to the representation power of DNN.

---

We note that in the drone experiments the performance of OMAC (convex) is only marginally better than the baseline. This is because the aerodynamic disturbance foree in the quadrotor simulation is a multiplicative combination of the relative wind speed, the drone attitude, and the motor speeds; thus, the superposition structure f( Y1 (1 ( ) Y2 (1) ) cannot easily model the unknown force, while the bi-convex and deep learning variants both learn good controllers, In particular, OMAC (bi-convex) achieves similar performance as the deep learning case with much fewer parameters. On the other hand, in the pendulum experiments, OMAC (convex) is relatively better because a large component of the c-invariant part in the unknown dynamics superposition with the c-dependent part. For more details and the pendulum experiment visualization we refer to the Appendix A.7
Related work
Meta-learning and representation learning. Empirically, representation learning and meta-learning have shown great success in various domains 77] In terms of control, reinforeiitrtt learning is able to solve challenging mult-task RL problems [8 [0] We remark that learning representation for control also refers to learn state representation from rich observations
42 but we consider dynamics representation in this paper. On the theoretic side, 4 5 33 have shown representation learning reduces sample complexity on new tasks, and "task diversity" is critical. Consistently, we show OMAC enjoys better convergence theoretically (Corollary D and empirically, and Theorem 6 also implies the importance of environment diversity. Another relevant line of theoretical work 45 47 uses tools from online convex optimization to show guarantees for gradient-based meta-lcarning, by assuming closeness of all tasks to a single fixed point in parameter space. We eliminate this assumption by considering hierarchical or nested online optimization procedure.
Adaptive control and online control. There is a rich body of literature studying Lyapunov stability and asymptotic convergence in adaptive control theory 12) Recently, there has been increased interest in studying online adaptive control with non-asymptotic metrics (e.g., regret) from learning theory. largely for settings with linear systems such as online LQR or LQG with unknown dynamics 14
The most relevant work [K] gives the first regret bound of adaptive nonlinear control with unknown nonlinear dynamics and stochastic noise. Another relevant work studies online robust control of nonlinear systems with a mistake guarantee on the number of robustness failures 49. However, all these results focus on the single-task case. To our knowledge, we show the first non-asymptotic convergence result for multi-task adaptive control. On the empirical side, [I BT combine adaptive nonlinear control with meta-learning, yielding encouraging experimental results.
16 24
Online matrix factorization. Our work bears affinity online matrix factorization, particularly the bandit collaborative filtering setting 6C 50 29 In this setting, one typically posits a linear low-rank projection as the target representation (e.g.. a low-rank factorization of the user-item matrix), which similar to our bilinear case. Setting aside the significant complexity introduced by nonlinear control, a key similarity comes from viewing different users as "tasks" " and recommended items as "actions". Prior work in this area has by and large not been able to establish strong regret bounds, in part due to the non-identifiabiiity issue inherent in matrix factorization. In contrast, in our setting, one set of latent variables (e.g., the wind condition) has concrete physical meaning that we are allowed to measure (ObserveEnv in Algorithm Db. thus side-stepping this non- non-identifiability issue.
7 Concluding remarks
We have presented OMAC, a meta-algorithm for adaptive nonlinear control in a sequence of unknown environments. We provide different instantiations of OMAC under varying assumptions and condi- tions, lcading to the first non-asymptotic convergence guarantee for multi-task adaptive nonlinear control, and integration with deep learning. We also validate OMAC empirically. We use the average control error (ACE) metric and focus on fully-actuated systems in this paper. Future work will seek to consider general cost functions and systems. It is also interesting study end-to-end convergence guarantees of deep OMAC, with ideas from deep representation learning theories. Another interesting direction is to study how to incorporate other areas of control theory, such as roboust control.
Broader impacts. This work primarily focused on establishing theoretical understanding of meta-adaptive control. Such fundamental work will not directly have broader societal impacts.
10

---

Acknowledgments and Disclosure of Funding
This project was supported in part by funding from Raytheon and DARPA PAI, with additional support for Guanya Shi provided by the Simoudis Discovery Prize. There is no conflict of interest.
References
[1] Michael O'Connell, Guanya Shi, Xichen Shi, and Soon-Jo Chung. Meta-lcarning-batn based robust adaptive flight control under uncertain wind conditions. arXiv preprint arXiv:2103.01952 2021.
[2] Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36 (4):1307-1319, 2020.
[3] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), 2020.
[4] Simon S Du, Wei Hu, Sham Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.0944 2020.
[5] Nilesh Tripuraneni, Chi Jin, and Michael I Jordan. Provable meta-learning of linear representa- tions. arXiv preprint arXiv:2002.11684 2020.
[6] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1-32, 2016.
[7] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):
1798 1828, 2013.
[8] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Picter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. arXiv preprint arXiv: 1802.07255 2018.
[9] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostie meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1133 2017.
[10] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. arXiv preprint 1rXiv:1803.111
2018.
[11] Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control, volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
[12] Karl Áström and Björ Wittenmark. Adaptive control. Courier Corporation, 2013.
[13] Nicholas M Boffi, Stephen Tu, and Jean-Jaqques E Slotine. Regret bounds for adaptive nonlinear control. In Learning for Dynamics and Control, pages 471-483. PMLR, 2021.
[14] Max Simchowitz and Dylan Foster. Naive exploration optimal for online lqr. International Conference Muchine Learning, pages 8937-8948. PMLR, 2020.
[15] Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, pages 1-47, 2019.
[16] Sahin Lale, Kamyar Azizzadenesheli, Babak Hassibi, and Anima Anandkumar. Logarithmic regret bound in partially observable linear dynamical systems. arXiv preprint arXiv:2003. 11277, 2020.
[17] Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Infor- mation theoretic regret bounds for online nonlinear control. arXiv preprint arXiv:2006. /2466, 2020.
11

---

[18] Yingying Li, Xin Chen, and Na Li. Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In NeurIPS, pages 14858-14870, 2019.
[19] Chenkai Yu, Guanya Shi, Soon-Jo Chung. Yisong Yue, and Adam Wierman. The power of predictions in online control. Advances in Neural Information Processing Systems, 33, 2020.
[20] Yiheng Lin, Yang Hu, Haoyuan Sun, Guanya Shi, Guannan Qu, and Adam Wierman. Perturbation- based regret analysis of predictive control in linear time varying systems. arXiv preprint arXiv:2106. 2021.
[21] Richard M Murray, Zexiang Li, and S Shankar Sastry. A mathematical introduction to robotic manipulation. CRC press, 2017.
[22] Steven M LaValee. Planning algorithms. Cambridge university press, 2006.
[23] Karl Johan Åström and Richard M Murray. Feedback systems. Princeton university press, 2010.
[24] Alon Cohen, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar. Online linear quadratic control. In International Conference on Machine Learning. pages 1029-1022 1038. PMLR, 2018.
[25] Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. In 2019 International Conference on Robotics and Automation (ICRA), pages 9784-9790. IEEE, 2019.
[26] Horia Mania, Stephen Tu, and Benjamin Recht, Certainty equivalence is efficient for linear quadratic control. arXiv preprint 1rXXv:1902.07826. 2019.
[27] Elad Hazan. Introduction online convex optimization. arXiv preprint 100.v::999.05207 2019.
[28] Naman Agarwal, Elad Hazan, Anirudha Majumdar, and Karan Singh. A regret minimization approach to iterative learning control. arXiv preprint arXiv:2102.134770 2021.
[29] Huazheng Wang, Qingyun Wu, and Hongning Wang. Factorization bandits for interactive recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
[30] Jaya Kawale, Hung H Bui, Branislav Kveton, Long Tran-Thanke and Sanjay Chawla. Efficient thompson sampling for online matrix-factorizatto recommendation. In Advances in neural information processing systems, pages 1297-1305, 2015.
[31] Spencer M Richards, Navid Azizan, Jean-Jacques E Slotine, and Marco Pavone. Adaptive- control-oriented meta-learning for nonlinear systems. arXiv preprint arXiv:2103.0444 2021.
[32] Yasin -abasi-Yadkorii Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In NIPS, volume 11, pages 2312-2320, 2011.
[33] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. arXiv preprint arXiv:2006. 16500, 2020.
[34] Lloyd Trefethen. Multivariate polynomial approximation in the hypereube. Proceedings of the American Mathematical Society, (4:(11)
2017.
[35] Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue. Robust regression for safe exploration in control. In Learning for Dynamics and Control, pages 608-60 619. PMLR, 2020.
[36] Xichen Shi, Patrick Spieler, Ellande Tang, Flena-Sorina Lupu, Phillip Tokumaru, and Soon-Jo Chung. Adaptive nonlinear control of fixed-wing vtol with airflow vector sensing. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5321-5327. IEEE, 2020.
[37] Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In NIPS, volume 3, page 5. Citeseer, 2007.
12

---

[38] Takeru Miyato, Toshiki Kataoka, Masanori Koyama. and Yuichi Yoshida. Spectral normalization for generative adversarial networks, arXiv preprint arXiv:1802.05957, 2018.
[39] Guanya Shi, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung. Neural-swarm: Decentralized close-proximity multirotor control using learned interactions. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 3241-3247. IEEE, 2020.
[40] Guanya Shi, Wolfgang Hönig, Xichen Shi, Yisong Yue, and Soon-Jo Chung. Neural-swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. IEEE Transactions on Robotics, 2021.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[42] Timothée Lesort, Natalia Díaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108:379-392, 2018.
[43] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pages 2170-2179 PMLR, 2019.
[44] Amy Zhang, Zachary C Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandkumar, Laurent Itti, Joelle Pineau, and Tommaso Furlanello. Learning causal state representations of partially observable environments. arXiv preprint arXiv: 1eeii.966..143 2019.
[45] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pages 1920-1930. PMLR, 2019.
[46] Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-leat stochastic gradient descent with biased regularization. In International Conference on Machine Learning, pages 1566-1575. PMLR, 2019.
[47] Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient- based meta- learning methods. arXiv preprint vrXiv:1906.02711 2019.
[48] Yasin Abbasi- Yadkori and Csaba Szepesvári. Regret bounds for the adaptive control of linear quadratic systems. In Proceedings of the 24th Annual Conference on Learning Theory, pages 1-26. JMLR Workshop and Conference Proceedings, 2011.
[49] Dimitar Ho, Hoang Le, John Doyle, and Yisong Yue. Online robust control of nonlinear systems with large uncertainty. In International Conference on Artificial Intelligence and Statistics, 2021.
[50] Guy Bresler, Devavrat Shah, and Luis Filipe Voloch. Collaborative filtering with low regret. In Proceedings of the 2016 ACM SIGMETRICS International Conference Measurement and Modeling of Computer Science, 2016.
[51] Dario Brescianini, Markus Hehn, and Raffaello Andrea. Nonlinear Quadrocopter Attitude Control: Technical Report. Report, ETH Zurich, 2013.
13


Reviews:
[{"review_id": "xO0TfQsmgEB", "reviewer": "Reviewer_7eB6", "summary": "This manuscript investigates the online multi-task adaptive control of nonlinear systems subject to unknown uncertainty and disturbances. Several instantiations of the proposed approach are investigated with theoretical guarantees on the resulting average control error. A numerical example is used to demonstrate the effectiveness of the proposed instantiations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This manuscript proposes a novel multi-task meta-learning approach for the adaptive control of nonlinear systems subject to unknown uncertainties and disturbances. The manuscript is well-organized and easy to follow, and the theorems and proof look rigorous. The reviewer admires the authors' effort in presenting various instantiations with a thorough investigation of their theoretical properties. Nonetheless, the experimental validation is certainly a weak point of the manuscript. And the reviewer has some concerns over the generality of the proposed approach. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Major comments:\n\n1) The nonlinear systems investigated in this manuscript are assumed to be fully actuated, which excludes a lot of underactuated systems such as autonomous vehicles/boats, legged robots, and even the cartpole system. Nevertheless, the manuscript still claims that generality is a merit of the proposed approach. \n\n2) The experimental part only provides a 2D numerical example to benchmark the performance of OMAC, without comparison with other existing meta-learning control approaches. In the reviewer's opinion, the provided example can't prove the general applicability of the proposed approach nor benchmark its performance against existing meta-learning control approaches. \n\nMinor comments:\n\n1) Some notations are used before proper introduction, e.g. line 162\n\n2) In section 5, which approach does baseline refer to?\n\n3) The emphasis on adversarial disturbances seems unnecessary, as the adversarial disturbances are not particularly considered either in the theorems or the experiments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "w53kulKT3Z3", "reviewer": "Reviewer_ic62", "summary": "This work proposes online adaptive nonlinear control under environment-dependend dynamics by assuming shared representations across environments. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of applying representation learning/meta-learning through shared representation to nonlinear control problems is important; also the reviewer appreciates the inclusion of the codes at this point too.\nDecoupling shared representations and environment parameters seems to be the main novelty.\nHere are the reviewer's concerns.\n\nMajor concerns:\n1) This work considers control affine system as a nominal model; augmenting the state by control input and regarding the time difference of control input as the new \"control\", general nonlinear system can be translated into control affine form.  However, this work further assumes that the system is fully actuated.  In such a case, the overall assumptions on the system seem to be too restrictive.  If any strong claim holds for this system, it would have some insights; however, the theoretical analysis seems to be a rather simple application of the existing results. \n(basically, based on the assumptions (stability, rank of B, minimum eigenvalue...), exploration is not required)\n\n2) DNN is usually used for scaling up problems; however, the experiments considered in this paper seems to be rather too simple.  If there is no theoretical claims under the use of DNN or the applications to more complex dynamics, it is hard to see the main messages delivered through the experiments.  It is mentioned that decoupling shared representations and environment has several (computational and other) benefits, and this point should be important for this work; however it is not really discussed in example cases etc. \n\nMinor concerns:\n1) When the author(s) say \"multi-task\", what does it mean?  For model-based scenarios, even if the environment (or the system) does not change, it would be able to deal with different tasks using the learned dynamical system model.\n\n2) there are some minor typos  (e.g. in Alg 1  The environment select -> selects), please recheck them.\n\nIf the significance of decoupling Theta and c becomes clearer in terms of applications, the reviewer is happy to raise the score.\n\n--------------------\nAfter initial response\n\nThe reviewer still has some concerns about the significance and applicability of the proposed framework; however, the reviewer agrees with the author(s)' arguments \"We showed the first end-to-end theoretical results for multi-environment control in nonlinear systems....\" and now believes it can be beneficial to place this work as one of the initial attempts of meta learning via shared representations of dynamical systems under variety of environments.  \nSo, the reviewer raises the score to 6 from 5.\nThe reviewer hopes the author(s) continue to build on this work to offer more significant insights in theory and applications too.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations may be stated in the concluding remarks in a indirect way; however, it is hard to see how restrictive those limitations are.\nAlso, please describe limitations of the assumptions on the class F, and limitations when DNN is used (e.g. how restrictive they are).  These will help readers think about how the work can be scaled up or generalized.\n\nAlthough there is no direct social impact, it is recommended to discuss any possible misuse etc. of the proposed work.", "review_text": "", "extra_scores": {"ethical_concerns": "The reviewer believes there is no significant ethical concerns in this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "aoYa9EEA1hM", "reviewer": "Reviewer_xzK3", "summary": "This paper proposes and studies a meta-learning algorithm inspired from nonlinear adaptive control. The algorithm posits that the uncertainty in the model can be decomposed into a shared representation, and an environment specific representation. In each inner loop, the algorithm adapts to the environment specific parameters, whereas in the outer loop the algorithm adapts to the shared representation parameters. A regret-like bound is shown for the “average control error”, which decays not to zero, but to a ball scaling as the maximum size of the disturbances. A simple experiment is run which shows that meta-learning outperforms a baseline adaptive controller which does not attempt to reuse features across tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and addresses a relevant question in the learning-to-control/RL literature. However, I am recommending a reject because the theoretical results are not very strong; the results are a simple extension combining the recent work of Boffi et al. [12] and Agarwal et al. [24]. Specifically, Boffi et al. [12] show how the e-ISS assumption on the nominal dynamics yields regret bounds for a single task, and the nested OCO framework of Agarwal et al. [24] gives a very clear path to extending the single task result to the multi task setting. Given knowledge of both these works, piecing them together (as is done in the paper) is a relatively straightforward task. I’m open to changing my mind here if the authors could point out what the challenges were in deriving the results. \n\nSome comments / questions / nits: \n\nIn the definition of ACE, the OMAC algorithm is not actually competing with the omniscient controller, but rather an *upper bound* for the ACE of the omniscient controller. Thus, when the omniscient controller upper bound is not saturated, this means that the results do not imply OMAC approaches the omniscient controller performance even in the limit of infinite data. This is dealt with in Boffi et al. [12] by restricting the size of the noise. At the very least, I would recommend the authors make a short remark addressing this issue.\n\nIn Equation 1, are both f and c^{(i)} unknown, or is just c^{(i)} unknown (but the functional form of f known)? I assume that the functional form of f is known (otherwise how does one even take its gradient). Please clarify the exposition around lines 64-65, which can be interpreted as f itself is unknown.\n\nSmall nit: consider calling the “element-wise convex” model “bi-convex” instead. \n\nOn Line 247, it is noted that Y_1 \\in R^{30} and Y_2 \\in R^{20}. This is a typo, as Y_1 and Y_2 should be matrices. \n\nMy main takeaway from Figure 1 was not that meta-learning was a significant win, but rather I was surprised at how well the baseline adaptive algorithm performed. An error of 0.2 radians seems fairly negligible. Perhaps the authors can try to make the learning task harder by increasing the dimensionality of the shared parameters? Or better yet, include a plot that captures the performance degradation of all algorithms as the number of shared parameters increases.\n\nIf possible, please include a version of Figure 1 where “no adaptation” is plotted, so we can see how difficult the task is without any adaptation. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "8mpg24mc_J8", "reviewer": "Reviewer_vw5a", "summary": "In this paper the authors show the advantages of keeping track of (and learning) a shared representation across different 'environments' that are switched, while learning to control a partially unknown system.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written well and the math looks solid, but it has some weaknesses that prevent me from accepting it:\n\n- Equation (1) is not general enough unlike the claims of Generality (line 99). For a control-affine system, the most general case is B(x,c) having environment dependence. To model it as part of the disturbances w, you'd need to know the bound W, which in this case is not so mild.\n\n- Having the context c revealed by the environment (in the bi-convex case) is not a reasonable or mild assumption in my opinion.\n\n- The method takes the pseudo-inverse of B (as a 'certainty-equivalent' control approach) for computing the controls, hence decoupling the time steps, whereas the whole idea of control theory is to optimize (given known or partly known models) with the future in mind. Hence I fear that the paper does not make a serious contribution to the adaptive control / RL literature. In fact I think the paper does not do justice to the field of adaptive control, reducing it to a case of bandit / regret minimization. Consequences of bad control (safety, stability) are not considered.\n\n- As for meta-learning literature, I also think that the improvement (keeping track of a shared representation and showing that it improves over methods that disregard it) is not a big contribution, although I am not an expert in this field. \n\n- The authors only show one experiment (partly known pendulum with switching-environment) and even here there is only one baseline. I wonder if the meta learning / adaptive control / RL literature has other methods that can be applied whenever the dynamics switches regularly?\n\n===== UPDATE =====\n\nAfter reading the rebuttal and re-reading the paper I decided to keep my rating the same. Basically my main concern about not advancing the state of the art of adaptive control (due to only using a certainty-equivalent greedy pseudo-inverse based approach) remains. The contribution to meta-learning I maintain is also not a big one (as also described more convincingly by Reviewer xzK3).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall there is not enough support here from reviewers for me to recommend acceptance, even though one reviewer raised their score from 5 to 6.

Reviewers agreed on some real strengths to the paper, including that it is well-written and addresses an important topic:
* Reviewer 72B6: "The manuscript is well-organized and easy to follow"
* Reviewer xzK3: "The paper is well written and addresses a relevant question in the learning-to-control/RL literature"
* Reviewer vw5a: "The paper is written well and the math looks solid"

The main weakness is described by Reviewer xzK3: "the theoretical results are not very strong; the results are a simple extension combining the recent work of Boffi et al. [12] and Agarwal et al. [24]" and "Given knowledge of both these works, piecing them together (as is done in the paper) is a relatively straightforward task." (discussed in detail in an exchange during the discussion period, with more details). Reviewer vw5a concurred after the discussion: "The contribution to meta-learning I maintain is also not a big one (as also described more convincingly by Reviewer xzK3)." A second concern is around performance shown in experiments, though these were improved to some degree during the discussion period.

Given these issues and the overall scores, I suggest rejection. However, I encourage the authors to improve the paper given the feedback from the reviewers, especially xzK3.
Decision: Accept (Poster)