Meta-Adaptive Nonlinear Control: Theory and Algorithms

Abstract: We present an online multi-task learning approach for adaptive nonlinear control, which we call Online Meta-Adaptive Control (OMAC). The goal is to control a nonlinear system subject to adversarial disturbance and unknown \emph{environment-dependent} nonlinear dynamics, under the assumption that the environment-dependent dynamics can be well captured with some shared representation. Our approach is motivated by robot control, where a robotic system encounters a sequence of new environmental conditions that it must quickly adapt to. A key emphasis is to integrate online representation learning with established methods from control theory, in order to arrive at a unified framework that yields both control-theoretic and learning-theoretic guarantees. We provide instantiations of our approach under varying conditions, leading to the first non-asymptotic end-to-end convergence guarantee for multi-task nonlinear control. OMAC can also be integrated with deep representation learning. Experiments show that OMAC significantly outperforms conventional adaptive control approaches which do not learn the shared representation, in inverted pendulum and 6-DoF drone control tasks under varying wind conditions.


Reviews and author responses:

 My original response was written before the updated simulation results. Thanks for bringing this to my attention.

While the new results do strengthen the experimental section, they do not obviate my remaining concerns outlined in (1) and (2) which have the highest influence on my rating. So unfortunately my rating remains the same. 

 Thanks for your input so far, Reviewer xzK3.

On point #3 in your latest comment, the authors provided an update to their "General Response to all Reviewers" top-level comment which includes a results table with "simulation result for a drone with a quadratic air drag model". Did you already take that into account? If not, does it change your thinking at all?

 Thanks for the authors' responses. 
The author's justification about the generality does not seem promising to me, and the implicit properties of the proposed system in the response are unclear. Thus I can't be convinced by the generality of the "systems with matched uncertainty". Nonetheless, it is good to see the proposed method outperforms the baseline in the new experiment. The reviewer would like to encourage the authors to further extend the experiment part to fully show the potential of the proposed approach. At this stage, I can't further raise my scores given the current contents of the paper. 

In this paper the authors show the advantages of keeping track of (and learning) a shared representation across different 'environments' that are switched, while learning to control a partially unknown system.  The paper is written well and the math looks solid, but it has some weaknesses that prevent me from accepting it:

- Equation (1) is not general enough unlike the claims of Generality (line 99). For a control-affine system, the most general case is B(x,c) having environment dependence. To model it as part of the disturbances w, you'd need to know the bound W, which in this case is not so mild.

- Having the context c revealed by the environment (in the bi-convex case) is not a reasonable or mild assumption in my opinion.

- The method takes the pseudo-inverse of B (as a 'certainty-equivalent' control approach) for computing the controls, hence decoupling the time steps, whereas the whole idea of control theory is to optimize (given known or partly known models) with the future in mind. Hence I fear that the paper does not make a serious contribution to the adaptive control / RL literature. In fact I think the paper does not do justice to the field of adaptive control, reducing it to a case of bandit / regret minimization. Consequences of bad control (safety, stability) are not considered.

- As for meta-learning literature, I also think that the improvement (keeping track of a shared representation and showing that it improves over methods that disregard it) is not a big contribution, although I am not an expert in this field. 

- The authors only show one experiment (partly known pendulum with switching-environment) and even here there is only one baseline. I wonder if the meta learning / adaptive control / RL literature has other methods that can be applied whenever the dynamics switches regularly?

===== UPDATE =====

After reading the rebuttal and re-reading the paper I decided to keep my rating the same. Basically my main concern about not advancing the state of the art of adaptive control (due to only using a certainty-equivalent greedy pseudo-inverse based approach) remains. The contribution to meta-learning I maintain is also not a big one (as also described more convincingly by Reviewer xzK3).
 N/A

 Thank you for the response. We have updated the drone simulation result in the general response, where our method significantly improves on the baseline.

 Thank you for the review and comments. Since some comments overlap, we will first have a general response to all reviewers, and then respond individually. Please find the general response in the following.

**(A)** Regarding the fully-actuated assumption, beyond Eq. (1), OMAC paradigm supports general robotics systems with matched uncertainty, where $B$ is not necessarily full rank, in the form $$x_{t+1}=f_0(x_t)+B_t(u_t-f(x_t,c))+w_t.$$ We will provide formal justifications for a variety of under-actuated systems in the paper, e.g., pendulums and drones.  

To summarize, our model supports not only general fully-actuated systems in Eq. (1), but the above dynamics with matched uncertainty. Meta-adaptive control for other systems requires something beyond direct disturbance compensation (i.e., it needs some multi-step planning), and is definitely an interesting future direction. We will add a remark about it in the paper.

We believe our setting is a promising start towards understanding the theoretical benefits of representation learning in control. Compared to literature, our setting indeed generalized previous theoretical results: (i) Most non-asymptotic convergence guarantees for adaptive control consider linear systems (e.g., paper [13-16]) in a single environment. The only nonlinear system result also requires matched uncertainty (i.e., [12]) in a single environment. (ii) Previous representation learning or meta-learning results did not consider dynamics (e.g., paper [4-5]). We showed the first end-to-end theoretical results for multi-environment control in nonlinear systems. We also presented a novel unified framework of online representation learning and adaptive control.

**(B)** Regarding the experiments and baselines, as a theoretical paper, our experiments are meant to demonstrate the high-level ideas of the algorithm, and we select the current baseline because it plays the role of ablation analysis. The baseline can be viewed as a special case of our method by setting the outer loop learning rate to 0, and all methods (including the deep-learning one) have the same learning rate and dimensionality in the inner loop. We believe such a design is a reasonable way to demonstrate why meta-adaptation matters. Moreover, please note that most other adaptive control methods, e.g., velocity gradient and composite adaptation, do not directly apply to the multi-environment setting and require extra assumptions, such as a known Lyapunov function.

To better showing the applicability, we further incorporated a new set of experiments with drones. This set of experiments indicates how desirably well the proposed algorithm scales. We have also conducted further experiments on (offline) meta-learning methods for performance comparison, e.g., first collecting $K$ outer iterations’ data and using a MAML-style meta-regression algorithm to meta-train the representation. We observed that OMAC provides superior performance.

**(C)** Regarding the theoretical novelty, our work derives new methods and finds new insights. For the convex case (Theorem 3), we use the recent Nested OCO result from [24] to derive the ACE bound. However, it is the only ⅓ of our results. For the bi-convex and bi-linear case (Theorem 5 and 6), [24] is no longer applicable (because the cost function is not convex anymore) and we have to derive new methods to give ACE bounds, and we also found some non-trivial insights, e.g., the importance of the “environment diversity” in multi-environment control, which is not covered in previous literature.

--------------
**Update**: The following table shows the simulation result for a drone with a quadratic air drag model, where the state dimension is 6 and the unknown dynamics $f(x,c)$ is a 3-d vector.

| | no-adapt  |  omniscient | baseline | OMAC (convex) | OMAC (bi-convex) | OMAC (deep learning) |
|---|---|---|---|---|---|---|
| ACE | 1.117 | 0.125 | 0.622 | 0.397 | 0.336 | 0.218 |

 Thank you for the response; please see the updated review.

-----
The reviewer still has some concerns about the significance and applicability of the proposed framework; however, the reviewer agrees with the author(s)' arguments "We showed the first end-to-end theoretical results for multi-environment control in nonlinear systems...." and now believes it can be beneficial to place this work as one of the initial attempts of meta learning via shared representations of dynamical systems under variety of environments.  
So, the reviewer raises the score to 6 from 5.
The reviewer hopes the author(s) continue to build on this work to offer more significant insights in theory and applications too.

This work proposes online adaptive nonlinear control under environment-dependend dynamics by assuming shared representations across environments.   The idea of applying representation learning/meta-learning through shared representation to nonlinear control problems is important; also the reviewer appreciates the inclusion of the codes at this point too.
Decoupling shared representations and environment parameters seems to be the main novelty.
Here are the reviewer's concerns.

Major concerns:
1) This work considers control affine system as a nominal model; augmenting the state by control input and regarding the time difference of control input as the new "control", general nonlinear system can be translated into control affine form.  However, this work further assumes that the system is fully actuated.  In such a case, the overall assumptions on the system seem to be too restrictive.  If any strong claim holds for this system, it would have some insights; however, the theoretical analysis seems to be a rather simple application of the existing results. 
(basically, based on the assumptions (stability, rank of B, minimum eigenvalue...), exploration is not required)

2) DNN is usually used for scaling up problems; however, the experiments considered in this paper seems to be rather too simple.  If there is no theoretical claims under the use of DNN or the applications to more complex dynamics, it is hard to see the main messages delivered through the experiments.  It is mentioned that decoupling shared representations and environment has several (computational and other) benefits, and this point should be important for this work; however it is not really discussed in example cases etc. 

Minor concerns:
1) When the author(s) say "multi-task", what does it mean?  For model-based scenarios, even if the environment (or the system) does not change, it would be able to deal with different tasks using the learned dynamical system model.

2) there are some minor typos  (e.g. in Alg 1  The environment select -> selects), please recheck them.

If the significance of decoupling Theta and c becomes clearer in terms of applications, the reviewer is happy to raise the score.

--------------------
After initial response

The reviewer still has some concerns about the significance and applicability of the proposed framework; however, the reviewer agrees with the author(s)' arguments "We showed the first end-to-end theoretical results for multi-environment control in nonlinear systems...." and now believes it can be beneficial to place this work as one of the initial attempts of meta learning via shared representations of dynamical systems under variety of environments.  
So, the reviewer raises the score to 6 from 5.
The reviewer hopes the author(s) continue to build on this work to offer more significant insights in theory and applications too. Limitations may be stated in the concluding remarks in a indirect way; however, it is hard to see how restrictive those limitations are.
Also, please describe limitations of the assumptions on the class F, and limitations when DNN is used (e.g. how restrictive they are).  These will help readers think about how the work can be scaled up or generalized.

Although there is no direct social impact, it is recommended to discuss any possible misuse etc. of the proposed work.

 Thanks to the authors for their response. 

Unfortunately, I am not able to change my rating based on their response.

Here are my responses to the authors' responses:

1. I do not believe general response (C) to be compelling. I agree that the bi-convex and bi-linear models are more challenging, but by assuming that $c^{(i)}$ is observed (`ObserveEnv` is true), this simplifies away much, if not all, of the complexity in the analysis. For example, the proof of Theorem 5 is a simple modification of the proof of Theorem 3.  As for Theorem 6, the part of the proof related to control is not any different than Theorem 5, the only additional complexity comes in needing to invoke a (by now very well-known) martingale concentration result to bound the quality of solutions to regularized least-squares problems. Regarding "environment diversity", the assumption given in Theorem 6 is the straightforward one needed to ensure parameter recovery of $\Theta$ at a $1/k$ rate, and is nearly identical to the definition of diversity given in reference [5].

2. Regarding my concern about the ACE of OMAC to the ACE upper bound of the omniscient controller, I appreciate the fact that the noise could be worst case. However, the bounds given in this paper are non-adaptive in the following sense. When the noise sequence actually is stochastic, the difference between the ACE of OMAC bounds given and the ACE of the comparator will most likely converge to a constant (even in expectation) even in the limit of infinite as $T$ and $N$ both tend to infinity, which is not desirable. 

3. Finally, the authors do not address my concerns regarding the performance of the baseline adaptive algorithm. As I mentioned before, the wins in Figure 1 for OMAC seem to be not very substantial. While the authors indicate they do have some new experiments (with drones), since these results do not appear in the paper and the authors have not provided any links to new figures, I do not believe we can incorporate these new experiments into our reviews.

 Thank you for the insightful review and comments. We will answer the reviewer’s questions as follows.

* Regarding the theoretical novelty, please see the general response **(C)**. Moreover, beyond [12] or [24], we show that our meta-algorithm is compatible with modern deep learning practices.

* Regarding the question about the definition of ACE, indeed OMAC is comparing with an upper bound of the ACE of the omniscient policy. But we would like to emphasize that this ACE upper bound of the omniscient policy is tight because the disturbance $w_t$ is adversarial (see footnote 2). [12] could get rid of $W$ in their bounds because their $w_t$ is assumed to be zero-mean i.i.d., and the regret is in expectation, not the worst-case regret. Our paper considers a worst-case-based metric.

* Regarding the question about the functional form of $f$, indeed we need to assume the form is known in theoretical claims. We explained this in Section 3 (line 117-125). We will add the clarification in Section 2 as well.

* Regarding the experimental section, we will certainly fix the typos and add a figure for the “no adaptation” method.

 Thank you for the insightful review and comments. We answer the reviewer’s questions as follows.

* Regarding major comment 1, please see the general response **(A)**.

* Regarding major comment 2, please see the general response **(B)**.

* Regarding the minor comments, we define the baseline in lines 253-254. We believe the emphasis on adversarial and $(x,u,c)$-dependent disturbance is important because in the convex and bi-convex cases (Theorem 3 and 5) the disturbance $w$ is adversarial, but in the bi-linear case (Theorem 6), $w$ is assumed to be stochastic to show a tighter ACE bound. Different assumptions on $w$ yield different theoretical results. Meanwhile, the adversarial setting is more general than in previous literature.


 Thank you for the insightful review. We will answer the reviewer’s questions as follows.

* Regarding major concern 1, please see the general response **(A)** and **(C)**. The role of exploration (in terms of regret minimization) is by and large not well understood in nonlinear systems, let alone the multi-environment setting. Considering exploration strategy in under-actuated systems is definitely an interesting future direction. We will add a remark.

* Regarding the major concern 2, please see the general response **(B)**. The use of DNN in the pendulum experiment is not for scaling up the state space, but for raising the representational capacity in the latent space.

* Regarding the significance of decoupling $\Theta$ and $c$, we believe it is proved by Corollary 4, where the ACE bound of the non-decoupling method gets much worse, even though the non-decoupling method needs more computation. Moreover, we conducted a new experiment comparing OMAC with the non-decoupling method in the convex setting (i.e., the baseline adaptive control in Corollary 4). We observed that the OMAC framework provides superior performance which is consistent with the claim in Corollary 4.

* Regarding minor concern 1, by “multi-task”, we mean multi-environment, not multiple cost functions or multiple control objectives. We will clarify it.

 Thank you for the insightful review. We will answer the reviewer’s questions individually as follows. 

* Regarding the question about generality (bullet points 1 and 3 in the review), please see the general response **(A)**. OMAC paradigm supports $B(x,c)$ if it is known. Moreover, we would like to point out that this paper focuses on a nonlinear multi-environment setting with a non-asymptotic metric, which is not covered in previous adaptive control/RL or bandit/regret minimization settings.

* Regarding the question about the contribution in meta-learning (bullet point 4 in the review), as we mentioned in the general response **(A)**, we showed the first end-to-end theoretical results for meta-learning or representation learning in control.

* Regarding the question about observing $c^{(i)}$ (bullet point 2 in the review), as the reviewer mentioned, we also believe the bi-convex theoretical results without knowing $c^{(i)}$ at the end of each outer iteration is interesting. However, as we mentioned in lines 198-203, the theoretical analysis for this case is very challenging because of the inherent non-identifiability issues (i.e., the “optimal” $\hat{c}^{(i)}$ is not unique). Actually even in the statistical representation learning literature, non-asymptotic results with bi-convex loss came only recently (paper [4-5]), with strong i.i.d. sampling assumption and the bi-linear structure. Moreover, as discussed in lines 285-292, in robotics the environment condition $c^{(i)}$ often has a concrete physical meaning (e.g., the wind condition in [1]), and we only need to measure $c^{(i)}$ after the outer iteration $(i)$ is finished, which has a lot of applications (e.g., [1,2] and [27]). Therefore, while the assumption of observing $c^{(i)}$ is strong, it is actually satisfiable in many applications. 

* Regarding the question about experiments (bullet point 5 in the review), please see the general response **(B)**.


This manuscript investigates the online multi-task adaptive control of nonlinear systems subject to unknown uncertainty and disturbances. Several instantiations of the proposed approach are investigated with theoretical guarantees on the resulting average control error. A numerical example is used to demonstrate the effectiveness of the proposed instantiations.  This manuscript proposes a novel multi-task meta-learning approach for the adaptive control of nonlinear systems subject to unknown uncertainties and disturbances. The manuscript is well-organized and easy to follow, and the theorems and proof look rigorous. The reviewer admires the authors' effort in presenting various instantiations with a thorough investigation of their theoretical properties. Nonetheless, the experimental validation is certainly a weak point of the manuscript. And the reviewer has some concerns over the generality of the proposed approach.  Major comments:

1) The nonlinear systems investigated in this manuscript are assumed to be fully actuated, which excludes a lot of underactuated systems such as autonomous vehicles/boats, legged robots, and even the cartpole system. Nevertheless, the manuscript still claims that generality is a merit of the proposed approach. 

2) The experimental part only provides a 2D numerical example to benchmark the performance of OMAC, without comparison with other existing meta-learning control approaches. In the reviewer's opinion, the provided example can't prove the general applicability of the proposed approach nor benchmark its performance against existing meta-learning control approaches. 

Minor comments:

1) Some notations are used before proper introduction, e.g. line 162

2) In section 5, which approach does baseline refer to?

3) The emphasis on adversarial disturbances seems unnecessary, as the adversarial disturbances are not particularly considered either in the theorems or the experiments.

This paper proposes and studies a meta-learning algorithm inspired from nonlinear adaptive control. The algorithm posits that the uncertainty in the model can be decomposed into a shared representation, and an environment specific representation. In each inner loop, the algorithm adapts to the environment specific parameters, whereas in the outer loop the algorithm adapts to the shared representation parameters. A regret-like bound is shown for the “average control error”, which decays not to zero, but to a ball scaling as the maximum size of the disturbances. A simple experiment is run which shows that meta-learning outperforms a baseline adaptive controller which does not attempt to reuse features across tasks.  The paper is well written and addresses a relevant question in the learning-to-control/RL literature. However, I am recommending a reject because the theoretical results are not very strong; the results are a simple extension combining the recent work of Boffi et al. [12] and Agarwal et al. [24]. Specifically, Boffi et al. [12] show how the e-ISS assumption on the nominal dynamics yields regret bounds for a single task, and the nested OCO framework of Agarwal et al. [24] gives a very clear path to extending the single task result to the multi task setting. Given knowledge of both these works, piecing them together (as is done in the paper) is a relatively straightforward task. I’m open to changing my mind here if the authors could point out what the challenges were in deriving the results. 

Some comments / questions / nits: 

In the definition of ACE, the OMAC algorithm is not actually competing with the omniscient controller, but rather an *upper bound* for the ACE of the omniscient controller. Thus, when the omniscient controller upper bound is not saturated, this means that the results do not imply OMAC approaches the omniscient controller performance even in the limit of infinite data. This is dealt with in Boffi et al. [12] by restricting the size of the noise. At the very least, I would recommend the authors make a short remark addressing this issue.

In Equation 1, are both f and c^{(i)} unknown, or is just c^{(i)} unknown (but the functional form of f known)? I assume that the functional form of f is known (otherwise how does one even take its gradient). Please clarify the exposition around lines 64-65, which can be interpreted as f itself is unknown.

Small nit: consider calling the “element-wise convex” model “bi-convex” instead. 

On Line 247, it is noted that Y_1 \in R^{30} and Y_2 \in R^{20}. This is a typo, as Y_1 and Y_2 should be matrices. 

My main takeaway from Figure 1 was not that meta-learning was a significant win, but rather I was surprised at how well the baseline adaptive algorithm performed. An error of 0.2 radians seems fairly negligible. Perhaps the authors can try to make the learning task harder by increasing the dimensionality of the shared parameters? Or better yet, include a plot that captures the performance degradation of all algorithms as the number of shared parameters increases.

If possible, please include a version of Figure 1 where “no adaptation” is plotted, so we can see how difficult the task is without any adaptation.  Yes

Meta-review: Overall there is not enough support here from reviewers for me to recommend acceptance, even though one reviewer raised their score from 5 to 6.

Reviewers agreed on some real strengths to the paper, including that it is well-written and addresses an important topic:
* Reviewer 72B6: "The manuscript is well-organized and easy to follow"
* Reviewer xzK3: "The paper is well written and addresses a relevant question in the learning-to-control/RL literature"
* Reviewer vw5a: "The paper is written well and the math looks solid"

The main weakness is described by Reviewer xzK3: "the theoretical results are not very strong; the results are a simple extension combining the recent work of Boffi et al. [12] and Agarwal et al. [24]" and "Given knowledge of both these works, piecing them together (as is done in the paper) is a relatively straightforward task." (discussed in detail in an exchange during the discussion period, with more details). Reviewer vw5a concurred after the discussion: "The contribution to meta-learning I maintain is also not a big one (as also described more convincingly by Reviewer xzK3)." A second concern is around performance shown in experiments, though these were improved to some degree during the discussion period.

Given these issues and the overall scores, I suggest rejection. However, I encourage the authors to improve the paper given the feedback from the reviewers, especially xzK3.

Decision: accept