Simultaneous Online System Identification and Control using Composite Adaptive Lyapunov-Based Deep Neural Networks

Authors: Omkar Sudhir Patil, Emily J. Griffis, Wanjiku A Makumi, Warren Dixon
Primary area: applications to robotics, autonomy, planning
Keywords: Adaptive control, Online Learning, Control Theory, Robotics

Abstract:
Although deep neural network (DNN)-based controllers are popularly used to control uncertain nonlinear dynamic systems, most results use DNNs that are pretrained offline and the corresponding controller is implemented post-training. Recent advancements in adaptive control have developed controllers with Lyapunov-based update laws (i.e., control and update laws derived from a Lyapunov-based stability analysis) for updating the DNN weights online to ensure the system states track a desired trajectory. However, the update laws are based on the tracking error, and offer guarantees on only the tracking error convergence, without providing any guarantees on system identification. This paper provides the first result on simultaneous online system identification and trajectory tracking control of nonlinear systems using adaptive updates for all layers of the DNN. A combined Lyapunov-based stability analysis is provided, which guarantees that the tracking error, state-derivative estimation error, and DNN weight estimation errors are uniformly ultimately bounded. Under the persistence of excitation (PE) condition, the tracking and weight estimation errors are shown to exponentially converge to a neighborhood of the origin, where the rate of convergence and the size of this neighborhood depends on the gains and a factor quantifying PE, thus achieving system identification and enhanced trajectory tracking performance. As an outcome of the system identification, the DNN model can be propagated forward to predict and compensate for the uncertainty in dynamics under intermittent loss of state feedback. Comparative simulation results are provided on a two-link manipulator system and an unmanned underwater vehicle system with intermittent loss of state feedback, where the developed method yields significant performance improvement compared to baseline methods.

Decision: reject
Reviewer scores: 8, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1729647861375

thread_type:
official_review

<thread_object>
creator: reviewer_n4x4

<content>
summary:
This paper presents a kind of Lpapunov-based adaptive framework, that can update all layers of DNN. The proposed method can handle nonlinear-in-parameters uncertainties. Moreover, a dynamic state-derivative estimator is utilized to obtain the state-derivative information. Overall, some novel theoretical results are developed in this paper with rigid proofs. The presentation is also clear. However, some drawbacks exist and many improvements can be further considered. There are some inappropriate statements and comparisons. The simulation tests are not enough to show its efficiency. Please refer to below for more details.

strengths:
1. The main contribution is that a Lpapunov-based adaptive framework is proposed to update **all layers** of DNN.
2. Rigid convergence analysis.
3. Two applied examples, despite only numerical simulations.

weaknesses:
1. The Abstract is too long, preventing the reader from capturing key points quickly. It is recommended to only highlight the main contributions in the Abstract and technique details can be removed.

2. It is claimed that the tracking, state-derivative, and weight estimation errors can be guaranteed to converge to bounded sets. **The factors that determine the upper bounds** of convergence sets should be provided in the Abstract.

3. The previous work (OConnell, 2022) is compared in the Intro. It is claimed the limitation of the composite adaptive approach used by OConnell, 2022, is the inner-layer weight cannot be online updated. However, the considered case of OConnell, 2022 is different from the one of this paper. OConnell mainly focuses on a composited disturbance, which comes from external disturbance and internal state-related uncertainties. The last layer of DNN is updated online to handle external disturbances, while the inner layers correspond to internal state-related uncertainties, which would not change in application. However, the internal state-related uncertainty is mainly considered in this paper, i.e., $f({x, \dot{x}})$. **The direct comparison with (OConnell, 2022) is inappropriate**.

4. One important problem is only simulation examples are demonstrated in this paper, and no noises exist in the measured states, despite the theorems that seem to be relatively complete. A small upper bound of the convergence set depends on a large gain. However, the gain may enlarge the noises in a real system. Thus, **the effect of the real application is questionable**.

5. In the simulation of two link manipulators, it is recommended to cover the ESO comparison and the composite adaptive method developed in (Slotine and Li, 1989). The gain selection strategy for all comparison methods should be provided to ensure fairness.

questions:
1. If the proposed framework could be combined with offline learning?  It seems the proposed update strategy only relies on current measurements and has no historical data-mining procedure. It is a kind of traditional adaptive control, instead of modern data-based learning. Maybe a control journal is more applicable to this paper.

2. If the considered uncertainty $f({x, \dot{x}})$ can be extended to the composited disturbance, like $f({x, \dot{x}, d})$, where $d$ denotes the external disturbance. It will be valuable in real applications.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729934128482

thread_type:
official_review

<thread_object>
creator: reviewer_bqxc

<content>
summary:
This paper provides the first result on simultaneous online system identification and trajectory tracking control of nonlinear systems using adaptive updates for all layers of the DNN. The Lyapunov-based stability analysis is provided, which guarantees that the tracking error, state-derivative estimation error, and DNN weight estimation errors are uniformly ultimately bounded.

strengths:
This is first application of the Jacobian of the DNN to develop simultaneous online system identification and control. The theoretical content of the paper is good. The literature research is sufficient.

weaknesses:
The practical application of this method requires high computing resources and is not suitable for personal computers. The presence of measurement noise does not seem to be considered in the two simulation tests, which is unreasonable. Control inputs of two simulations should also be presented. Moreover, the nonlinear dynamics of the selected simulation system is weak.

questions:
1)	The existence of measurement noise should be considered, which is common in practical engineering; Please provide the control inputs of two simulation tests;
2)	I wonder if this method is effective for highly dynamic systems like quadcopters?
3)	Provide more experimental details, such as control inputs, weights update, and the selected control parameters.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612464661

thread_type:
official_review

<thread_object>
creator: reviewer_xnmo

<content>
summary:
This paper addresses a methodology for simultaneously performing online system identification for the plant system and adaptation in the feedback control logic. Under some technical assumptions, stability conditions are presented, more specifically, the asymptotic stability of the equilibrium point of the entire feedback control system is ensured.

strengths:
The simultaneous approach to online system identification and adaptation in the control logic, addressed in this paper, is well-motivated and justified with attractive numerical experiments.  In addition, as claimed by the authors, the convergence analysis for the identification error and control error is novel.

weaknesses:
Assumptions 1 and 2 are mathematically severe.  While this paper claims its contribution lies in simultaneous system identification and control, Assumption 1 implies that performing system identification can achieve arbitrary control performance (arbitrary system dynamics is realized by u= g^+(r - f(x,\dot{x})). In this sense, the problem addressed in this paper is not essentially simultaneous.  

Furthermore, there are too many technical assumptions on the modeling accuracy, meaning the existence of \bar{vareplison}, \bar{\theta}, etc.

questions:
Could you relax Assumptions 1 and 2?  In particular, as commented in Weakness, Assumption 1 is mathematically (and practically) severe.  The authors comment that the developed methods can be extended to underactuated systems. However the details of the extension are scarcely explained, and no theoretical analysis is provided.  The reviewer believes that the extension to underactuated cases and its convergence analysis should be the main contribution of this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626532848

thread_type:
official_review

<thread_object>
creator: reviewer_vqh4

<content>
summary:
This paper is introducing an adaptive DNN-based controller. Standard adaptive DNN-based controllers are based on Lyapunov-based analysis and allow updates on the last layers of the NNs only. They only use the tracking error as a metric to indicate when adaptation is needed, and only provide guarantees on the tracking error convergence. This paper introduces a dual (composite) method, for continuous system identification combined with trajectory tracking, and guarantees that the tracking error, state-derivative estimation error, and DNN weight estimation errors are uniformly ultimately bounded. The last two reflect identifying the dynamics of the system.
The system identification is performed via a dynamic state-derivative estimator and under the assumption of persitence of excitatiton. The controller is evaluated in simulation on a two-link manipulator system and an unmanned underwater vehicle system with intermittent loss of state feedback, and shows improvement compared to baseline methods.

strengths:
The paper is on a very timely topic, DNN-based control. With the emergence of AI-based approaches, a principled treatment of a learning-based controller is an impactful contribution. Introducing a controller which allows for updates in all layers, in contrast to the state of the art where only thelast layers can be updated, and where system dynamics is not considered explicitely, is a big contribution. 
Given the complexity of the paper, it is very clearly presented. The simulation examples are relevant and not just datasets, but dynamical systems.

weaknesses:
1. In the simulations, there is only comparison to DNN-based controllers. 
It would highly interesting to see how is the peroformance, compared to more standard controllers used in robotics (for example, MPC-based, or RL-based controllers). I am willing to raise my rating if this is added.

questions:
1. Shouldn’t there be an assumption for the activation functions to be convex? How do you deal with the nonconvex dependence of the underlying loss function on weights of hidden layers? Is it playing a role?

2. How can input and state constraints be integrated in the proposed approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732432083765

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Changes

comment:
We are grateful to all the reviewers for their positive and constructive feedback. We appreciate that all reviewers acknowledge the contribution of our work: **"the paper is on a very timely topic", "a principled treatment of a learning-based controller is an impactful contribution", "introducing a controller which allows for updates in all layers, in contrast to the state of the art where only the last layers can be updated, and where system dynamics is not considered explicitly, is a big contribution"** (reviewer vqH4), **"... is well-motivated and justified with attractive numerical experiments"** (reviewer XnMo), **"This is first application of the Jacobian of the DNN to develop simultaneous online system identification and control. The theoretical content of the paper is good. The literature research is sufficient."** (reviewer bqXC) , **" some novel theoretical results are developed in this paper with rigid proofs. The presentation is also clear"** (reviewer N4X4).

All changes in the paper have been marked in blue. For the convenience of the reviewers, we are providing a summary of revisions in the manuscript below.

**1. More baselines:** The robot manipulator and UUV simulations (Sections 4.1 and 4.2, respectively) are now expanded with nonlinear MPC, nonlinear PD, and observer-based disturbance rejection controllers as additional baselines for simulations.

**2. Measurement Noise:** All of the simulations were performed again incorporating state measurement noise to make the simulations more realistic. Sections 4.1 and 4.2 have been updated accordingly.

**3. More Details on Simulations:** Based on the reviewers' requests, additional simulation details on control inputs, weights update, and the selected control parameters along with their selection strategy are provided in Appendix D.

**4. Expanded Justification of Assumptions:** More discussion is added in Section 2 to justify Assumption 1 and 2. 

**5. Further Discussion on Related Works:** A more detailed discussion regarding the related work (O'Connell, 2022) is added in Appendix B.2. 

**6. Modified Abstract:** The abstract is made more concise and the factors that determine the upper bounds of convergence sets are provided.

**7. Future Work:** More discussion is added in Section 6 on possible future extensions of this work to incorporate state and input constraints.

**8. Extension to underactuated case of nonholonomic mobile robot:** An extension of the development to nonholonomic mobile robots is added in Appendix F.

---
**Summary of Discussion Period**

1. **Reviewer vqH4** suggested incorporating comparisons with more standard baselines, such as Model Predictive Control (MPC), and asked questions about convexity. In response, we **added comparisons with nonlinear MPC and nonlinear PD as baselines** and provided detailed explanations of how conditions analogous to convexity emerge in the form of Persistent Excitation (PE) and gain conditions in the Lyapunov analysis. The reviewer was satisfied with these additions and **raised the score to 8**.

2. **Reviewer XnMo** critiqued Assumptions 1 and 2, requesting more justification and **details on extending the method to underactuated systems** along with a theoretical analysis. We addressed these concerns by providing stronger justifications for the assumptions and clarifying that **theoretical extensions of any controller to underactuated systems are not universal but inherently case-specific**. To illustrate the process of such extension, we **included an extension to a nonholonomic mobile robot** in Appendix F. While the reviewer was satisfied with our justification for Assumption 2, they have not yet commented on the extension for mobile robots. Furthermore, the reviewer stated Assumption 1 eliminates the challenges, which we disagree with. **The challenges we addressed are in the DNN all-layer update part, our key contribution**, which the reviewer did not comment on yet.

3. **Reviewer bqXC** recommended incorporating measurement noise in the simulations and adding more experimental details. **We revised the manuscript accordingly.** Additionally, the reviewer inquired whether the method could handle strongly nonlinear systems like quadcopters. We affirmed that it could, noting that the Unmanned Underwater Vehicle example in our work includes nonlinear dynamics comparable to those of a quadcopter. The reviewer acknowledged our rebuttal, remarked that 6 is an appropriate rating for our current manuscript, and gave constructive feedback on improving the quality of our work in future.

4. **Reviewer N4X4** suggested improvements to the related work section, simulations, and abstract. We implemented most of their recommendations in our revisions. The reviewer raised follow-up questions about the **connections between deep learning and adaptive control** in the context of our contribution. Our detailed responses convinced the reviewer, leading to an **increased score of 8**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper demonstrates a result for simultaneous identification and stabilization of a control system with a deep neural network. The reviewers were generally positive, but with one reviewer leaning towards rejection. The positive reviewers were rather lacking in detail, whereas the negative reviewer demonstrated greater expertise. In addition, reviewers did not actively participate in discussion. 

Therefore, I decided to intervene as meta-reviewer to assess the paper on its own merits. 
(a) this paper establishes a simultaneous tracking and stabilization for nonlinear systems. They claim this is novel, because classical results only apply to linear parameter uncertainty. 
(b) This paper mostly makes good on its claim
(c) I believe the presentation of the paper is below the standards of ICLR, and will be somewhat impenetrable for the community. For one, results rely on a number of terms which are defined haphazardly throughout. Second, certain constants (e.g. the Taylor expansion should depend on the smoothness of G) are omitted. Moreover, NNs are typically highly non-smooth, so framing arguments as Taylor expansions seem misleading. Third, the authors claimed limitation of Slotine and Li, '89 is that they require a low pass filter, which necessitates LPV systems. But I do not seem to see any form of filtering in the paper, and indeed the authors seem to act as if they have access to time derivatives. Moving beyond this point is quite poorly explained. 
(d) Ultimately, it seems that the theoretical of this content has little to do with neural networks at all, and should be about tracking/estimation with non-LPV uncertainty. The presentation has great room for improvement, and the reviewer who appeared most knowledgable also seems to side with rejection. 

Constructive suggestions to the authors: (a) rewrite proofs, exposition, clean up theorem statements and remove excess and unnecessary terms. (b) I would suggest submitting to a revised manuscript to a controls venue, where the work can be appropriately evaluated by a community with a more extensive knowledge of nonlinear adaptive control.