A priori guarantees of finite-time convergence for Deep Neural Networks
Authors: Anushree Rankawat
Mansi Rankawat
Harshal B. Oza
Venue:  (2021)

Abstract: In this paper, we perform Lyapunov based analysis of the loss function to derive an a priori upper bound on the settling time of deep neural networks. While previous studies have attempted to understand deep learning using control theory framework, there is limited work on a priori finite time convergence analysis. Drawing from the advances in analysis of finite-time control of non-linear systems, we provide a priori guarantees of finite-time convergence in a deterministic control theoretic setting. We formulate the supervised learning framework as a control problem where weights of the network are control inputs and learning translates into a tracking problem. An analytical formula for finite-time upper bound on settling time is provided a priori under the assumptions of boundedness of input. Finally, we prove that our loss function is robust against input perturbations. 

Under review as a conference paper at ICLR 2021
A PRIORI GUARANTEES OF FINITE-TIME CONVER- GENCE FOR DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we perform Lyapunov based analysis of the loss function to derive an a priori upper bound on the settling time of deep neural networks. While previous studies have attempted to understand deep learning using control theory frame- work, there limited work on a priori finite time convergence analysis. Drawing from the advances in analysis of finite-time control of non-lineat systems, we provide priori guarantees of finite-time convergence in a deterministic control theoretic setting. We formulate the supervised learning framework problem where weights of the network are control inputs and learning translates into a tracking problem. An analytical formula for finite-time upper bound on settling time is provided a priori under the assumptions of boundedness of input. Finally, prove that our loss function is robust against input perturbations.
control
INTRODUCTION
Over the past decade. Deep neural networks have achieved human-like performance in various ma- chine learning tasks, such as classification, natural language processing and speech recognition. Despite the popularity of deep learning, the underlying theoretical understanding remains relatively less explored. While attempts have been made to develop deep learning theory by drawing inspi- ration from other related fields such as statistical learning and information theory, comprehensive theoretical framework is still in an early developmental stage. It is difficult to perform mathematical analysis on Deep neural networks due to the large number of parameters involved. Other problems in deep neural networks revolve around the stability and desired convergence rate of the training. Since the performance of the network depends highly on the training data and the choice of the opti- mization algorithm, there is no guarantee that the training will converge. Our work attempts to give finite-time convergence guarantees for training of a deep neural network by utilizing an established stabilization framework from control theory.
Existing works in deep learning theory have attempted to bridge the gap in understanding deep learning dynamics by focusing simple models of neural networks [Saxe et al. (2013). Li & Yuan (2017), [Arora et al. (2018), Jacot et al. (2018)]. This could be attributed to the fact that current deep learning models highly complex structures analyze. Jacot al. (2088)
proved that a multilayer fully-connected network with infinite width converges to a deterministic limit at initialization and the rate of change of weights goes to zero. [axx et al. (2013) analyzed deep linear networks and proved that these networks, surprisingly, have a rich non-linear structure. The study shows that given the right initial conditions, deep linear networks are a finite amount slower than shallow networks. Following this work, Arora et al. (2018) proved the convergence of gradient descent to global minima for networks with dimensions of every layer being full rank in di- mensions. While these studies give important insights into the design of neural network architecture and the behavior of training, their results may need to modified order to provide convergence guarantees for the conventional deep neural networks. Du et al. (2018) extended the work of Jacot cct al. (2018) further by proving convergence for gradient descent to achieve zero training loss in deep neural networks with residual connections.
When it comes to convergence of certain state variables of dynamical system, control theory pro- vides a rich mathematical framework which can be utilized for analyzing the non-linear dynamics of deep learning (Liu & Theodorou (. One of the early works relating deep learning to con- trol theory of LeCun et al. (1988), which used the concept of optimal control and formulated

---

Under review as a conference paper at ICLR 2021
back pock-prppggation..m optimization problem with non-linear constraints. Non-linear control has gained increasing attention over the past few years in the context of neural networks, especially for recurrent neural networks [Allen- -hu et al. (2119), Xiao [2017] and reinforcement learning (Xu
et al. (2013), Gupta et al. 2019), Wang et al. 2019), Kaledin et al. (2020]] A new class of recurrent neural networks, called Zhang Neural Networks (ZNN), was developed that expressed dynamics of
the network as a set of ordinary differential equations and used non-linear control to prove global
or exponential stability for time-varying Sylvester equation [Zhang al. (2002), Guo et al. (201T).
Li et al. 2 introduced the sign bi-power activation function for Zhang Neural Networks (ZNN)
which helps prove the existence of finite-time convergence property. |Haber & Ruthotto (2017)
presents deep learning as a parameter estimation problem of non-linear dynamical systems to tackle
the exploding and vanishing gradients.
The focus of this paper is on deriving priori guarantee of attaining finite-time convergence of train- ing in supervised learning framework under some assumptions on inputs. The novelty lies in the fact that the weight update is cast a finite-time control synthesis such that the loss function is proven to be a valid Lyapunov function. The resulting training update is derived as a function of time such that it ensures the convergence of Lyapunov function in finite time. The only assumption used is that the magnitude of atleast one input is greater than zero. Thus, the learning problem is converted into finite time stabilization problem studied rigorously in Bhat & Bernstein (2000). The contributions of the proposed study are twofold. First, we propose a Lyapunov candidate func- tion to be used as loss function. Second, we modify the weight update of the neural network in such a way that the supervised training is converted into a dynamical control system. This allows us use results from Bhat & Bernstein (2000) priori guarantee finite time convergence on the training. To the best of our knowledge, guarantee of finite-time convergence is being studied for the first time in context of training a general multi-layer neural network. The proposed results will enable time bound training that will be useful in real-time applications.
The paper is organized as follows. Section 2 starts with introducing the Lyapunov function from the control theory perspective. Section 2 derives the weight update and Цyарunov loss function for a single neuron case and proves that it satisfies the conditions required for finite-time stability theorems developed in Bhat & Bernstein (2000) to be applicable. Section 2.2 then proves that a similar result extends to a multi-layer perceptron network under reasonable assumptions on the input. In Section 2.3 we state the equations to compute upper bounds on the convergence time for training neural networks. Section p.opiivides an extension to the case when bounded perturbations are admitted at the input and convergence guarantees are shown to hold true. In Section 3 some numerical simulations are presented for both single neuron and multi-layer perceptron cases for regression, Section S colleetts conclusions and discusses future scope.
2 PROPOSED METHOD TO CONVERT SUPERVISED LEARNING INTO DYNAMICAL CONTROL SYSTEM
This section motivates the development of a priori bounds on settling time with certain assumptions on the input. The weight update problem for supervised learning in neural networks is similar to the tracking problem of non-linear control systems. The idea is to synthesize a feedback control law based on certain Lyapunov function Vxx). A Lyapunov function is a positive definite function, i.e. V(x)) > 0, and its time derivative is negative definite along the given system dynamics, i.e. V << 0. We cast the supervised learning problem as a dynamical system which admits a valid Lyapunov function as the loss function with the weight update is designed as a function of time.
2.1 SINGLE NEURON CASE
We start with simplistic single neuron case. Let IR be the input to the network where xx]] |2,|| < c = 1,2,............., ........ n holds true for some a priori but arbitrary scalar e E (0, oo). Let y" be the target output and = Σx= Wya; + be the linear combination of weights w, with inputs z; and bias 6. The definition of signum function used here is sign(æ:)
x = [i] x2
V2 > O, sign(2) = -1, Vz < 0, sign(x) E [-1, 1], x = O. For our analysis, we choose sigmoid function our activation function, i.e. d(2) = 14e-z. The output of the neural network is given
by y (=). Let the error in output be defined as é = y D² The first objective is convert
2

---

Under review as a conference paper at ICLR 2021
loss function into candidate Lyapunov function in order apply the control theoretic principles. Consider a continuous function ((a) to be a candidate Lyapunov function as follows:
|e (a+1)
E
(1)
(a + 1)
where a E (0, 1) is a user-defined parameter. The second objective is to define the temporal rate of weight the control input to enforce the stability of the origin é 0 - oo. The Lyapunov function in ① is used to show that is indeed plausible to achieve this asymptotic stability goal. Taking the temporal derivative of the candidate Lyapunov function T produces (by chain rule of differentiation)
dE dE dē dy dz dw dt dē dy dz dw dt where de dy ds dw the weight update for standard gradient descent algorithm and du is the addi- tional term we introduce to the weight update equation. Using (T). we get:
dE dē
de
dE
(1T
|el sign(E)
(1 (1+)-1)33
(0)10) rgtii in)
dl
Define, 11 - Wi, U2 A u2 .. Un - Wn. and
u; - ki sign( (ign (1+ i= 1.,,,.....
(3)
where k; > 0, for all i, are tuning parameters to be chosen by the user. It can be noted that all control inputs U1, U2.. -.. Un remain bounded due to boundedness assumption of all the inputs z; and that into produces
of ef Substituting
dE dt = -1
72
Σ 1m1
k kg
(4)
Assumption 1. least one input of all x4, = 1, ... n is non-zero such that |ı, where y is priori known scalar for some integers j [1.
>
can be noted that Assumption iis reasonable for many practical applications in that some inputs will always be nonzero with known lower bound its magnitude. First main result of the paper is in order.
Theorem 1. Assuming Assumption ] holds true, let the output of the neural network be given by
y (2)) Let all the inputs I, 2, ... be bounded by some priori known scalar a (0, oo) such that |a,) < a holds true for all i. Then, weight update 3 causes the error = y
converge zero in finite time.
Proof. The proof of the theorem is furnished using standard I. Lyapunov analysis arguments. Consider E defined by (T) as candidate Lyapunov function. Observing 44) can be concluded that the right hand side of the temporal derivative of remains negative definite since it involves power terms and norm inputs. Furthermore, (4) can be rewritten under the assumption [ as follows:
dE - kiinn F.B
(5)
dt
where Kmin 1.2, a and |æ| 11 E has been utilized. Noting that F. is a positive definite function and scalars kmin and 7 always positive, the proof is complete by applying (Bhat Bernstein 2000 Theorem 4.2).
2.2 MULTI NEURON CASE
Consider lylti-layer perceptron with N layers where the layers are connected in feed-forward
manner (Bishop) 1995 Chapter 4). Let E 12" be the input to the network where =
[2]
En and [2,]| c, - 1, holds true for some priori but arbitrary scalar
(0, oo). Let E Rm define the multi-neuron output to the network where = [//1 y/ ... ym]-
Let y E R define the target output values of the network where yy [v] 3/
vtn] The
3

---

Under review as a conference paper at ICLR 2021
error in the output layer can be expressed as é the scalar candidate Lyapumov function can be written as follows:
||
vil
1//2
vēl
Vill. Hence,
E (a+1)
Em (a+1)
== E1
+
Em
(6)
(a
(a +1+1)
As is usually done in the case of feed-forver med-forward networks, consider unit j of layer / that computes its output
dj Σ 3
using its inputs 2 from layer I where bias parameter has been embedded inside the linear combi- nation and of (d, where a is non-linear activation function. The aim of this section is to extend the single neuron case to multi-neuron one. The simplest way do achieve this to find sensitivity of E to the weight wts, of given layer I, which is given by
dEm awi, jz
θEm dd
oag ow
Using standard notation 0j 2 aEm with results in: 8a
DEm
83
(9)
Ow'j
is straightforward to compute 6 that belongs the output layer shown below:
F.m o' da ( ÖEm
(10)
da
Oym
where 2m is replaced by ym as it is the output layer. Finally, computation of 8) for all hidden units given by
FEm aa
d da 1+1 Oa!
5)
(11)
Da [+1
where units with label & includes either hidden layer units output unit in layer I + 1. Com- bining 7 24 (aa) and 03. & 0Em produces dat
8) o=((aa)) Σ
1+1 kj
g++
(12)
A slightly modified version of assumption is required before the next result of the paper is pre- sented.
Assumption 2. At least one input of all zı, 1,2, ... ,,, is non-zero such that |zn| > 7 > where is a priori known scalar for some integers n E [1, L] where is the number of units in layer I.
Theorem 2. Let the weight update for connecting unit layer I to unit j of layer + 1 of multi-layer neural network be given by
id kJs sign(5)
(13)
with some scalar 3 E (0, 1) such that a + < and kjs > a tuning parameter. Then the output vector y converges to 15" in finite time.
Proof. Consider the candidate Lyapunov function E given by (6) The temporal derivative of the Lyapunov function is given by
d
Σ
Em = Σ
(14)
Dwiz
m
m
4

---

Under review as a conference paper at ICLR 2021
which can be simplified using 133 and
9
E = E
(15)
Using Assumption is easy to conclude that for some kmin min Ky; > 0, the following inequal- ity holds true:
R.3.
< kimin ? a++
E
(16)
Noting that E is a positive definite function and scalars kiiin 7 are always positive, the proof is complete by applying (Bhat & Bernstein 2000 Theorem 4.2).
Remark 1. It can be seen that weight update (13) (or respectively (3)) is in a feedback control form where state 2, and 8, (respectively 2; and é) are being used for inftuencing the learning process.
2.3 SETTLING TIME FOR NEURAL NETWORK TRAINING
Theorem and 2 prove that using candidate Lyapunov function as the loss function and the temporal rate of change the loss function the weight update equation, can convert supervised learning framework into a control problem. (Bhat & Bernstein, 2000) Theorem 4.2) states that if there is continuous function that is positive definite and it's rate of change with respect time negative definite, then it's finite time stable equilibrium is at origin. The theorem also gives the settling time when the above conditions are satisfied by the control system. For our case, this means that the Lyapunov based loss function will converge in finite time and time taken to converge is as given below:
The settling time for Single Neuron case:
1
L (11-B)
T S mmnn 71 B)
(17)
The settling time for Multi Neuron case:
T
E (1-8)
(18)
Kmin a a 3)
ii
where y is the lower bound on the inputs, kmin is the minimum value of tuning parameter k. a is the scalar used the Lyapunov loss function and Enni is the initial value loss, i.c. loss at t=
2.4 SENSITIVITY PERTURBATIONS
This section considers the robustness of training neural network based on our proposed algorithm. control theoretic terms, perturbation can understood either as a disturbance to the process or as modelling uncertainty. When supervised learning is viewed as control process, perturbation in each neuron of the input layer can be seen as external noise. This perturbation could cause the control system to diverge. In this section, develop theoretical claims that with perturbed inputs, training a neural network with the proposed algorithm will converge. The following assumption of the upper bound the perturbation of inputs is invoked.
Assumption 3. There exists a priori known constant M O such that all imputs F,, 1,2,-.. N admit additive perturbations Az; such that
Anal < M
(19)
for all i where N is the number of inputs.
The following result is in order.
Theorem 3. Let assumptions and 3 true. Let the weight update for connecting unit of layer I to unit j of layer + of multi-layer neural network be given by (13). Then the output vector
converges to y" in finite time in the presence of additive perturbations Ara (1, T.) kmin M.
5

---

Under review as a conference paper at ICLR 2021
Proof. It be seen that the perturbation is considered only in inputs. Hence all hidden layer weights are updated as done in the proof of Theorem 2 the following revised temporal derivative of Lyapunov function:
Hence, the dynamics of learning results in
== -= E Σ
2
at1
Σ p=0
k1plopEp1 sign(6,2))6
(20)
where K1p is the gain parameter for training of all the input layer weights. The expression in (0 can be simplified using Assumption 3 as follows:
E S - Σ kys 0 a++1
+
E kip
xpp jat1 M,
pm 0
(21)
- E
(kye 21
where inputs 2; now collect inputs as well. Similar to the proof of Theorem 2 re-written by applying Assumption 2 as follows:
21 can be
E. -kmin M)
(22)
Since Kminn M, the proof is complete by applying (Bhat & Bernstein 2000, Theorem 4.2).
EXPERIMENTS
This section presents empirical evidence that the theoretical results derived in the above sections hold for real-life applications.
Convergence Rate for training. To analyze the training convergence rate of the proposed method, we train a multi-layer perceptron (covered in Section 2.2p to perform regression on the Boston Housing dataset (Harrison Jr & Rubinfeld (1978)). Experiment single neuron case is covered in Appendix A.2.2
Training details. We compare our proposed method (Lyapunov based loss and modified weight update equation) with traditional L,, L2 loss functions and standard SGD weight update equa- tion. The modifications to the weight update equation for Lyapunov loss are as stated in (15) for multi-neuron case. We plot the training and test loss with respect to time in Figure The details about the training examples, epochs, learning rate etc. are specified under the description of the figures. We observe that the Lyapunov Loss function converges faster than L1 and La loss functions, demonstrating the results proven extensively in Section This attributed mainly to the non-Lipschitz weight updates given by and (13).
0.6
0.4
(papanoo Loos
0.6g
The settling time as well as metrics of the trained networks time taken by the proposed algorithm to con- verge falls within the priori upper bound. We also observe that the metrics (Accuracy for sin- gle neuron case and rmse (Root mean square er- ror) for multi-neuron case) achieved by the pro- posed method better than the baselines. Ad- mittedly, the theoretical upper bound on settling time for single neuron and multi-layer percep- tron produced by (5) and (16) are very conser- vative, yet it is an a priori deterministic upper bound and the aggressive learning proves to be faster than traditional loss functions in the sce- narios considered here.
Time In socondal
reported in Table [ The
0.6
0.4
Lpppunion Loss
0.6g
Time In sacurdda
Comparison of convergence with re- spect to time for the multi-layer perceptron trained on the Boston Housing dataset. All networks are trained for 4000 epochs with learning rate = 0.02 and a 0.68.
Figure 1:

---

Under review as a conference paper at ICLR 2021
Experiment
Theoretical Upper Bound
Exp. Convergence Time
Metric
(in seconds)
(in seconds)
L1
L2
Lyap.
L1
Lg
Lyap.
SingleNeuron (acc)
3.4e3
0.065
0.064
0.025
0.95
0.95
1.0
MLP (rmse)
~ 1.4e10
4.3e3
3.4e3
3.1e3
0.091
0.179
0.085
Table 1: Settling time in seconds for each experiment conducted. We compare the time taken for convergence by three different loss functions, L1, T22 and Lyapunov Loss function. The training conditions were similar for individual cases in the experiment. The single neuron case trained on Iris dataset and the metric used is Accuracy (higher is better). For Multi-layer perceptron, we use Boston Housing dataset and the metric used for this is rmse (lower is better).
This can be attributed to the fact that we operate in the realm of non-smooth functions which tend to be more aggressive than the smooth L2 function we usually encounter in most learning problems. We can clearly observe that the loss value does not converge to zero. According to (Bhat & Bern- stein 2000 Theorem 5.2), control system will converge to non-zero value at steady state, if there is constant perturbation in the input. Usually in real life datasets, there is an inherent bias or constant perturbation, which prevents the loss to converge to zero. Of course, by setting a O, all persisting disturbances (Orlov) (2005) and force the loss to converge to zero, but this may result in discontinuity in back propagation (discussed in Appendix A.T and result into large numerical errors.
reject
Hyperparameter analysis and its effect on training. In this section, we attempt give an em- pirical analysis as to how changes in the hyperparameters like k and a affect the training of the neural network. In order to study the effect of tuning k; we work with the single neuron case for simplicity and ease of understanding. We use Iris dataset for this experiment Dua & Graff (2017). Existing literature in control theory suggests that a monotonic increase the tuning parameter, should result in a monotonic decrease in the corresponding loss. In neural network terminology, k can be considered as the 'learning rate'. We can clearly see in Figure 2 that behaves like learning rate parameter where increasing k makes the learning more aggressive.
1.0
1.0
1.0
L1 loss
L1 icss
loss for k-0.001
0.8
loss Lyepunov loss
Training Loss for k=00.005
0.8
Lyspunov loss
loss
Training Loas for K= 07
0.8
Lyspunov loss
loss
0.6
0.6
0.6
Treiing loss
0.4
0.4
0.4
08.0
0.01
0.03
0.04
0.05
08.00
0.03 seconds)
08.00
0.03
seconds)
1.0
(b)
loss
12 loss
loss
L2 loss
1-00007
0.8
Lyepunov loss
Test Logs Las for c-0.005
C.8
Lyepunov loss
Test Lass for 1-0.01
Lyapunov loss
0.6
0.6
0.6
Logs
0.4
0.4
0.4
0.7
0.7
0.2
0800
0.01
0.02
0.03
0.04
0.05
00.00
0.01
0.02
0.03
0.04
08.00
0.01
0.02
0.03
0.04
0.05
Time (In seconds) (d)
Time (In seconds)
Time (in secordss)
Figure2 2: Comparison of training and test convergence with respect to time for different values of tuning parameter, l single neuron trained on the Iris dataset. We convert the problem into a binary classification problem by considering only two output classes. All networks are trained for 600 epochs with 80 training examples and 20 test examples. The learning rate for all L1, L2 and Lyapunov is set to the tuning parameter value l and a = 0.8
From Table 2 we observe that the settling time for the training loss of our Lyapunov loss function is always less than Ln and Lzz We also observe that the Accuracy of method is either better
7

---

Under review as a conference paper at ICLR 2021
or similar to the baselines, which shows that the optimized solution achieved by our method is either better or on par with the baselines. For the multi nulti-neuron case, we have defined kiy for every weight. Similar to adaptive learning rate, we can either tune Kl for every weight (which can be quite cumbersome for larger neural networks) or we can set it arbitrarily to the same value for all weights. It must be noted that kig can be set to any value greater than an a priori known positive constant M.
Experiment for single
Theoretical Upper Bound (in seconds)
Exp. Convergence Time (in seconds)
Accuracy
on Test Set
neuron case on Iris dataset
L1
L2
Lyap.
L1
L2
Lyap.
0.001
37213.848
0.312
0,310
0.0918
0.6
0.95
0.95
A 0.005
7442.769
0.0943
0.0942
0.0344
0.95
1.0
1.0
k = 0.01
~ 3721.385
0.0566
0.0573
0.0221
0.95
1.0
1.0
Table 2: Settling time in seconds for different values of tuning parameter, k for the single neuron case on Iris dataset. We compare the time taken for convergence by three different loss functions,
L1, Le and Lyapunov Loss function. The training conditions were similar for individual cases in the experiment.
We will discuss the impact of different values of a on the training process our neural network. For the given experiments, we tested the network for various values of ranging between (0, 1). We observed that the training loss follows equations proved in above sections as long as a E (0.5, 0.9). If a happens be too close zero, the resulting control law becomes discontinuous thereby resulting in numerical instabilities owing to the fact that the current ODE solvers are unable to handle functions that are discontinuous. More details about the case a 0 is given in Appendix A.1
Bhat & Bernstein 2000 Theorem 5.2).
Stability of learning in the presence of amplitude-bounded perturbations.
1.0
Triningg for ponurtation 0.2
Training Loss tor geturbatio 0.3
Trainin for peturtation
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
loss
loss
loss
0.22
lass
0.2
lass
P0.2
12 loss
Lyapunov
Lyspunov loss
Lyapunov loss
0.0g
50
75
300
125
0.0
25
75
100
125
0.0g
25
100
125
1.0
(8)
2.0
Validation Las for perturaation 0.3
Validation perturtation
Volidation Lass perturbatton
0.8
0.8
0.8
0.6
0.5
0.6
0.4
10.4
loss
lass
L1 loss
0.2
0.2
0.2
Lyapunow loss
Lyspunov loss
(yapunov loss
0.0g
50
75
100
125
0.0
25
50
75
100 125
25
50
75
100
125
Time lin seconds)
Time (In seconds)
secones)
(d)
Figure 3: Comparison of training loss convergence with respect to time for different values of input perturbations, Ar for a multi-layer perceptron trained on the IMDB Wiki Faces dataset. The a priori upper bound on input perturbations Learning rate/k = 0.0009, a 0.8, epochs=100
follows: (a) Ar 0.1, (b) Ar 0.2, (c)Ar 0.3
In this section, we present results that demonstrate the robustness of proposed algorithm to bounded perturbations in the input while maintaining convergence of training dynamics. For this case, we train a multi-layer perceptron on a regression task of predicting the age given the image of a face. We use IMDB Wiki Faces Dataset Rothe et al. (2015) which has over 0.5 million face images of celebrities with their age and gender labels. Only part of the dataset is used for this experiment, ie. 20,000 images for training and 4, 5,000 images each for validation and test.
8

---

Under review as a conference paper at ICLR 2021
For adding imput perturbations, we asssume that a priori upper bound, M is known the am- plitude possible imput perturbations following Assumption We add input perturbations to each pixel in the image using randomized uniform distribution ranging from (-Az, Ar). Hence, additive noise remains in the abovementioned a priori bounds. We vary the values of M from (0.1, 0.3) with 0.1 increment, giving three training cases. Figure shows that the proposed training loss still manages to achieve steady state error and does not diverge due to perturbations introduced in the input dataset. Since we are dealing with noisy data, the loss values converge to non-zero value in the steady-state
shows
Experiments on larger datasets. In this section, we present results to demonstrate the performance of our proposed algorithm on larger dataset. We use the entire IMDB Wiki Faces Dataset Rothe al. (015) with 0.5 million images for this experiment. Training dataset consists of ~0.2 million images and the test and validation set consists of ~0.1 million images each. As described in the previous section, multi-layer perceptron is trained to predict the age from the image of a face. The MLP is trained for 100 epochs with learning rate 0.0 0.0005 for all three loss functions and with = 0.7 for Lyapunov loss function. From Figure 4 and Table 3 we can see that our proposed algorithm achieves similar generalization / testing mmse (Root mean square error) L1 and L2 baselines while providing finite time convergence guarantees even on large datasets. The convergence time is also within the theoretically derived upper bound.
1.0
10.6
Lyspurar
Lyspurow
0.6
2000
4503
0000
8000
30000
2000
4000
5000
5000
10000
(50
Figure 4: Comparison of training convergence with respect to time for 0.5 million IMDB Wiki dataset
Experiment
Theoretical Upper Bound (in seconds)
Exp. Convergence Time
Metric rmse
(in seconds)
L
T,2
Lyap.
L
L.2
Lyap- 0.416
IMDB Wiki
8.3e6
980.40
712.99
467.96 .6.
0.414
0.415
Table 3: rmse on IMDB Wiki test dataset for L,, L2 and Lyapunov Loss function.
4 CONCLUSION AND FUTURE WORK
This paper studies the training of a deep neural network from control theory perspective. We pose the supervised learning problem control problem by jointly designing loss function Lyapunov function and weight update as temporal derivative of the Lyapunov function. Control theory princi- ples are then applied to provide guarantees on finite time convergence and settling time of the neural network. Through experiments on benchmark datasets, our proposed method converges within the priori bounds derived from theory. It also observed that in some cases our method enforces faster convergence as compared to standard L, and L2 loss functions. We also prove that our method is robust to any perturbations in the input and convergence guarantees still hold true. The given a pri- ori guarantees for the convergence time is desirable result for training networks that are extremely difficult to converge, specifically in Reinforcement Learning. A future scope of this work may be to convert the continuous time analysis framework to discrete time. This study introduces novel per- spective of viewing neural networks as control systems and opens up the field of machine learning research to a plethora of new results that be derived from control theory.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. On the convergence rate of training recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 6673-6685, 2019.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. convergence analysis of gradient descent for deep linear neural networks. arXiv preprint arXiv:1810.02281. 2018.
S.P. Bhat and D.S. Bernstein. Fimete-Time Stability of Continuous Autonomous Systems. SIAM Journal of Control and Optimization, 38(3)751-766, 2000.
Christopher M. Bishop. Neural Networks for Pattern Recognition. OXFORD University Press, 1995.
Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint 1r1iv: 00804,, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL BtLhhttp//archhii ics wuc..edu/m...
Dongsheng Guo, Chenfu Yi, and Yunong Zhang. Zhang neural network versus gradient-based neural network for time-varying linear matrix equation solving. Neurocomputing, 74((7):3708-371) 2011.
Harsh Gupta, R Srikant, and Lei Ying. Finite-time performance bounds and adaptive learning rate selection for two time-scale reinforcement learning. In Advances in Neural Information Process- ing Systems, pp. 4706-4715, 2019.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.
David Harrison Jr and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. 1978.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Advances in neural information processing systems, pp. 8571- 8580, 2018
Maxim Kaledin, Eric Moulines, Alexey Naumov, Vladislav Tadic, and Hoi-To Wai. Finite time analysis of linear two-timescale stochastic approximation with markovian noise. arXiv preprint arXiv:2002.01268 2020.
Yann LeCun, D Touresky, G Hinton, and T Sejnowski. A theoretical framework for back- propagation. In Proceedings of the 1988 connectionist models summer school, volume 1, pp- 21-28. CMU, Pittsburgh, Pa: Morgan Kaufmann, 1988.
Shuai Li, Sanfeng Chen, and Bo Liu. Accelerating a recurrent neural network to finite-time conver- gence for solving time-varying sylvester equation by using sign-bi-power activation function. Neural processing letters, 37(2):189-20S. 2013.
Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with relu activation. In Advances neural information processing systems, pp. 597-607, 2017.
Guan Moann-Hongg& Liu and Evangelos Theodorou. Deep learning theory review: An optimal control and dynamical systems perspective. arXiv preprint arXiv: 10920, 2019.
Yury Orlov. Finite-Time Stability and Robust Control Synthesis of Uncertain Switched Systems. 2005.
SIAM Journal of Control and Optimization,
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from single image. In IEEE International Conference on Computer Vision Workshops (ICCVW), December 2015.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv: /312::/311.6//00 2013.
10

---

Under review as a conference paper at ICLR 2021
Gang Wang, Bingcong Li, and Georgios B Giannakis. A multistep Lyapunov approach for finite-tine analysis of biased stochastic approximation. arXiv preprint arXiv:/909.04299, 2019.
Lin Xiao. Accelerating a recurrent neural network to finite-time convergence using a new design formula and its application to time- timevaryying matrix square root. Journal of the Franklin Institute, 354(13):5667-567) 2017.
Bin Xu, Chenguang Yang, and Zhongke Shi. Reinforcement learning output feedback nn control using deterministic learning technique. IEEE Transactions Neural Networks and Learning Systems, 25(3):635 (21)),66.66..222 2013.
Yunong Zhang, Danchi Jiang, and Jun Wang. A recurrent neural network for solving sylvester equation with time-varying coefficients. IEEE Transactions Neural Networks, 13(5):1053- 1063, 2002. 2002.
A APPENDIX
A.1 IMPACT OF a ON TRAINING
The case when a = 0, as depicted in Figure raises continuity issues.
Lyspunov L.2 Loss
0.07
0.06
0.03
0.02
0.01
2000
4000
6000
8000
10000
12000
epochs
Figure 5: Depiction of the numerical instability observed when we take = 0. Effectively, at this point, the loss function contains a discontinuous signum function.
Consider the left and right limits of the function f(g) = (ig where 0 = 6,24 as appearing
in (15.. It can be seen that lim f(e) g (()) = 0 for a (0, 1). However, for 0.
limg-+0- f(() = - and g f(e) = 1. It should be noted that function f(e) is non-Lipschitz 0 reduces the loss function to LI loss, but the control update becomes purely discontinuous due to the presence of f(e) in (15)
since /0g tends to infinity the limit e + We agree that
We do deal with this case for the reasons of continuity as mentioned in the main paper.
A.2 EXPERIMENTS
A.2.1 EXPERIMENTAL SETUP
For our experiments, core CPU with 32GB of RAM for training. The training code is in Python. All networks are trained on data with 80% 20% train data-test data split.
the NVIDIA GEFORCE GTX1080TI GPU card with 12GB RAM, 4-

---

Under review as a conference paper at ICLR 2021
A.2.2 CONVERGENCE EXPERIMENTS ON SINGLE NEURON
In this experiment, we perform binary classification on the Iris dataset (Dua & Graff (2017) representative example of single neuron case presented in Section 2.1
10
loce
Iaee
werght3
lose
Ioea
L2 werght3
Lyspurce Uno
Lyupa noy lown
Lyppurio welgle
Telnin Ua
3.8
Teal Loze
0.6
WWO VAL
0.6
c2
0.2
0.2
SICOOD
Bibcoo 0.0523
90:00 6.0025
TIme in sevendo)
Time iir caronns)
Time On seronin)
Figure 6: Comparison of convergence with respect to time for single neuron trained the Iris dataset. We convert the problem into a binary classification problem by considering only two of the three output classes. All networks trained for 2100 epochs with 80 training examples and 20 test examples, a = 0.8, learning rate / k 0.01
From the Figure 6 we can clearly see that Lyapunov loss function with control weight update con- verges much faster towards zero compared to In and L2 loss with standard gradient descent weight update. This shows that explicitly adding control weight update (I5) that drives the loss towards zero is helpful in reaching convergence faster.
A.2.3 BOSTON HOUSING DATASET EXPERIMENT WITH PERTURBATIONS
The Boston Housing Dataset predicts the price of houses in various areas of Boston Mass, given 14 different relevant attributes, like per capita crime rate and pupil-teacher ratio by town. The 506 examples in the data are divided into a 80-20 training-tes split to give us 505 training examples and 101 test examples respectively. We consider three cases here, M 0.1, M 0.2 ,nd M 0.3. Table presents the theoretical upper bounds for the proposed Lyapunov function's 's settling time and experimental results for the training conducted for different upper bounds assumed for the input perturbation. We observe that the settling time for the proposed Lyapunov function is similar to the one observed for Lg loss function whereas it performs way better than the L1 loss function.
Experiment for MLP case Boston dataset
Theoretical Upper Bound (in seconds)
Exp. Convergence Time
Metric rmsc L2
to within 10c-9 (in seconds)
L
L2 Lyap.
L1
Lyap.
M 0.1
I.97e
1277.66
906.61
755.59
0.095
0.145
0.092
M 0.2
5.67c
1297.11
1097.62
914.78
0.096
0.146
0.093
M = 0.3
20
1290.48
1197.53
998.05
0.093
0.147
0.101
Table Settling time in seconds for different values of the upper bound on additive input perturba- tions, M for the multi layer perceptron case on Boston Housing dataset. We compare the time taken for convergence by three different loss functions, L,, L2 and Lyapunov Loss function. The training conditions were similar for individual cases in the experiment.
12

---

Under review as a conference paper at ICLR 2021
A.2.4
IMDB WIKI FACES EXPERIMENT ON PERTURBATIONS
33.0
31.0
33.0
31.0
33.0
33.0
33.0
33.0
33.0
31.0
33.0
31.0
33.0
33.0
33.0
33.0
Figure 7: Pictorial representation of the effect of random additive noise with an upper bound of 0.2 on IMDB Wiki Faces Dataset. this experiment, we work with monochrome images for the sake of simplicity.
Figure 7 shows how the noise affects the training images. We take the specific case where M 0.2. We can clearly observe that the images obtained after adding input perturbations happens to be quite noisy. Figure 8 and Figure Figure/Ishow show that even when inputs are perturbed, our proposed method converges in finite time.
1.0
Training Loss
Training Loss
Training Loss
0.6
0.4
L1 loss
L1 loss
L1 loss
0.2
0.2
L2 loss
L2 loss
L2 loss
Lyapunov loss
Lyapunov loss
Lyapunow loss
0.9
50
100
0.0g
50
100
0.0g
100
Time (in seconds)
Time (in seconds)
Time seconds]
(a)
(b)
(c)
1.0
1.0
0.9
0.8
Training Loss
Training Loss
Training Loss
0.6
0.6
0.6
0.4
0.4
0.4
L1 loss
L1
0.2
0.2
0.2
L2 loss
L2 loss
L2 loss
Lyapunov loss
Lyapunov loss
Lyapunov loss
0.0g
50
100
0.0g
50
100
150
0.0g
50
100
150
Time (in seconds)
Time (in seconds)
Time (in seconds)
(d)
(e)
(t)
Figure 8: Comparison of training loss convergence with respect to time for different values of input perturbations, Ax for multi-layer perceptron trained the IMDB Wiki Faces dataset. The priori upper bound on input perturbations are as follows: (a) Ar 0.1, (b) Az = 0.2, (c)Ax 0.3, (d) Ar 0.4, (e) Ar = 0.5, and (f) Ar 1.2.
13

---

Under review as a conference paper at ICLR 2021
1.0
1.0
1.0
0.8
0.8
0.8
Testing Loss
Testing Loss
0.6
0.6
0.6
Testing
0.4
0.4
0.4
loss
0.2
0.2
0.2
Lyapunov loss
Lyapunov loss
Lyapunov
0.0
50
100
0.0
50
100
0.0
50
100
Time (In seconds)
Time (in seconds)
Time (in seconds)
1.0
1.0
0.8
0.8
0.8
Testing Less
Testing Loss
Testing Loss
0.6
0.6
0.6
0.4
0.4
0.4
L1 loss
L1 loss
loss
0.2
0.2
loss
L2 loss
loss
Lyapunov
Lyapunov
Lyapunov
0.00
100
0.0g
150
0.0
50
150
Time (in seconds)
(in seconds)
seconds)
Figure 9: Comparison of test loss convergence with respect to time for different values of input perturbations, Ar for multi-layer perceptron trained on the IMDB Wiki Faces dataset. The priori upper bound on input perturbations are as follows: (a) Ar 0.1, (b) Az = 0.2, (c)Az 0.3, (d) Ar 0.4, (e) Ar 0.5, and (f) Ar 1.2.
14


Reviews:
[{"review_id": "tYlEQok6A6P", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors in this paper make an attempt in providing finite time convergence guarantees of the training process of neural networks, using ideas from control theory.  The loss function in the training process is framed as a Lyapunov function.  The training process at each time step is seen as  assigning dynamics to the Lyapunov function over time. The convergence of which can then be analyzed using standard control theoretic techniques. \n\nFor some fixed input - output target, the idea is to come up with a weight update rule which guarantees the convergence rate with some assumptions on the inputs. This  is the novelty in the paper. The extension to multi-layer case is  an extension of the back propagation algorithm. \n\nThough the above is an interesting contribution in itself,  I am not convinced that the results for a fixed input case would generalize well to the batched input case. Which in my opinion is more general,  and has enabled the training of large scale neural networks. The authors have analyzed the robustness to perturbations  in Section 2.4. Specifically Eqn 22. Where the authors have bounds on the perturbation limits, under which it can still guarantee convergence rates. For the batched case it might need some restrictions on the choice of samples in a batch. \n\nThe next concern I have is the experiments definitely looks very insufficient. The current experiments include much smaller datasets. I would be interested to see how this technique performs on some of the larger  neural network architectures. Since convergence guarantees become more important, only when the time it takes to train a network is much longer.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "eYO6UnyXyLC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies the finite-time convergence for neural networks. In particular, it tries to recast the problem of training neural networks as a control problem. Supervised learning is then reformulated as a non-linear control problem with a Lyapunov based loss. The weight update is then transformed to be the control inputs. Finally, convergence results are obtained with standard theory from non-linear systems.\n\nOverall, connecting neural networks with classical control theory is an interesting direction. However, results presented in this paper seems limited, and it is not clear what contributions the current work really bring to the community.\n(1) there does not seem to be enough innovation in this paper. To me, the result simply follows from the classical control theory. The authors simply try to mimic the theory by having a candidate Lyapunov loss and continuous weight update equations. It is not clear why these are used for neural networks at the first place; rather, it seems that these are only applied for the sake of proving some technical results. For example, does the candidate Lyapunov loss actually generalize better (theoretically or empirically)? what's the property of it? Howe does it compare to traditional loss function? It is not convincing for me why someone should use it for training neural networks. It seems to be an artifact used solely for the theorem.\n(2) relatedly, the experiments focus on plotting the training loss of Lyapunov loss and l1/l2 loss. From my perspective, this is not informative. The loss function is likely not on the same scale; it is probably better to plot a \"normalized\" version so that the comparison indeed makes sense. Again, such a comparison does not reveal any interesting property about the new loss, e.g.., generalization/testing error? \n(3) Please clarity to what extent, the results hold with respect to different activation function. In Section 2.1, it is explicitly mentioned that sigmoid activation is used. In Section 2.2, the authors use the same notation \\sigma. Does the result hold for other common activation functions? If not, any comments on the difficulties?\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YzWdj58SZDr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to make strides towards a theoretical understanding of Deep neural networks, which remains elusive to date. This paper uses a control theoretic formulation to analyze the convergence rate of deep neural networks. More specifically, a Lyapunov based analysis of the loss function is used to derive a priori upper bound on the settling time of a restricted set of fully connected neural network architectures with some assumptions on the input space. \n\nI'm interested to know, for what kind of real-world tasks or datasets is their assumption on the boundedness of the input valid?\nAlthough the proposed Lyapunov loss provides the possibility of analyzing convergence guarantees a-priori, how does this affect the performance of the underlying model on test data? \n\nThe paper provides experiments supporting their theoretical claims for MLPs on a regression task and a single neuron on a classification task. They show that their proposed Lyapunov loss converges faster than the L1 and L2 losses, and faster than the a-priori upper bound. Can a similar loss function for MLPs on classification tasks be easily derived? In other words, do these results easily extend to classification tasks?\n\nAnd what effect does the new loss have on overfitting?\n\nI'm a bit confused by the theoretical upper bound. The derived upper bounds in Table 1 are orders of magnitude higher than the actual time taken, even with the L1 and L2 losses. What does this mean? What's the use of the upper bound in this case?", "extra_scores": {}, "extra_text": {}}, {"review_id": "OaGQC2Ge0yb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a Lyapunov based analysis of the loss function in neural network training and derives a priori upper bounds on the settling time of the training, which somewhat complements existing studies. The supervised neural network learning problem is formulated as a control problem with the weight parameters being the control input, and the learning problem as a tracking problem. Analytic formula for computing the finite-time upper bound on the settling time is provided under suitable assumptions on the input. Furthermore, the loss function is also shown robust against input perturbations.\n\nThis paper contains some interesting ideas in revealing relationships between control and neural network learning, which is a plus. Hopefully, this can further motivate exploration and application of more control-theoretic tools to understanding of neural network training. Although this paper is fairly readable, the presentation and organization can be improved. Several detailed comments are provided below. \n\ni) In control, particularly tracking problem, it is known that there is a given reference signal y(t) one wants the control plant to track. Nonetheless, here in the discussion the y^\\ast I guess is determined by the loss function, training method, data, as well as the neural network architecture altogether, right? What exactly is this y^\\ast? How is it related to e.g., the equilibrium point of the weight parameters and stationary points in the optimization context? \n\nii) The current analysis in Section 2 pertains to a single data point? How would having more data points affect the analysis and the results? In that case, what would be the y^\\ast? Or will be functions of the input? \n\niii) In the experiments, since the Lyapunov loss function and other loss functions are plotted, how does the loss function convergence correspond to the learned weights parameters? In the context learning, one is more interested in the neural network parameters that not only capture the training data but also predict well the unseen ones? So it would also be interesting to present the corresponding testing results?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to study the convergence of deep neural networks training via a control theoretic analysis. This is a very interesting approach to establish theoretical understanding of deep learning. However, there are several concerns raised by the reviewers:

1.	The contribution of this paper is limited. The results simply follow from standard optimal control. It is not clear what new insight the paper provides.
2.	There are already quite a few works on control theoretic analysis of deep learning. This paper did not do a good job on presenting its novelty and difference with existing works.
3.	The experimental part is weak. It only involves small data set and very simple networks.

Based on these, I am not able to recommend acceptance for the current manuscript. But the authors are encouraged to continue this research.

Decision: Reject