Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper aims to make strides towards a theoretical understanding of Deep neural networks, which remains elusive to date. This paper uses a control theoretic formulation to analyze the convergence rate of deep neural networks. More specifically, a Lyapunov based analysis of the loss function is used to derive a priori upper bound on the settling time of a restricted set of fully connected neural network architectures with some assumptions on the input space. 

I'm interested to know, for what kind of real-world tasks or datasets is their assumption on the boundedness of the input valid?
Although the proposed Lyapunov loss provides the possibility of analyzing convergence guarantees a-priori, how does this affect the performance of the underlying model on test data? 

The paper provides experiments supporting their theoretical claims for MLPs on a regression task and a single neuron on a classification task. They show that their proposed Lyapunov loss converges faster than the L1 and L2 losses, and faster than the a-priori upper bound. Can a similar loss function for MLPs on classification tasks be easily derived? In other words, do these results easily extend to classification tasks?

And what effect does the new loss have on overfitting?

I'm a bit confused by the theoretical upper bound. The derived upper bounds in Table 1 are orders of magnitude higher than the actual time taken, even with the L1 and L2 losses. What does this mean? What's the use of the upper bound in this case?
review_title: A priori guarantees of finite-time convergence for Deep Neural Networks

final_score: rating: 7: Good paper, accept
review_content: This paper presents a Lyapunov based analysis of the loss function in neural network training and derives a priori upper bounds on the settling time of the training, which somewhat complements existing studies. The supervised neural network learning problem is formulated as a control problem with the weight parameters being the control input, and the learning problem as a tracking problem. Analytic formula for computing the finite-time upper bound on the settling time is provided under suitable assumptions on the input. Furthermore, the loss function is also shown robust against input perturbations.

This paper contains some interesting ideas in revealing relationships between control and neural network learning, which is a plus. Hopefully, this can further motivate exploration and application of more control-theoretic tools to understanding of neural network training. Although this paper is fairly readable, the presentation and organization can be improved. Several detailed comments are provided below. 

i) In control, particularly tracking problem, it is known that there is a given reference signal y(t) one wants the control plant to track. Nonetheless, here in the discussion the y^\ast I guess is determined by the loss function, training method, data, as well as the neural network architecture altogether, right? What exactly is this y^\ast? How is it related to e.g., the equilibrium point of the weight parameters and stationary points in the optimization context? 

ii) The current analysis in Section 2 pertains to a single data point? How would having more data points affect the analysis and the results? In that case, what would be the y^\ast? Or will be functions of the input? 

iii) In the experiments, since the Lyapunov loss function and other loss functions are plotted, how does the loss function convergence correspond to the learned weights parameters? In the context learning, one is more interested in the neural network parameters that not only capture the training data but also predict well the unseen ones? So it would also be interesting to present the corresponding testing results?
review_title: Interesting connections between neural network training and control, as well as control-theoretic analysis of the neural network loss function 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work studies the finite-time convergence for neural networks. In particular, it tries to recast the problem of training neural networks as a control problem. Supervised learning is then reformulated as a non-linear control problem with a Lyapunov based loss. The weight update is then transformed to be the control inputs. Finally, convergence results are obtained with standard theory from non-linear systems.

Overall, connecting neural networks with classical control theory is an interesting direction. However, results presented in this paper seems limited, and it is not clear what contributions the current work really bring to the community.
(1) there does not seem to be enough innovation in this paper. To me, the result simply follows from the classical control theory. The authors simply try to mimic the theory by having a candidate Lyapunov loss and continuous weight update equations. It is not clear why these are used for neural networks at the first place; rather, it seems that these are only applied for the sake of proving some technical results. For example, does the candidate Lyapunov loss actually generalize better (theoretically or empirically)? what's the property of it? Howe does it compare to traditional loss function? It is not convincing for me why someone should use it for training neural networks. It seems to be an artifact used solely for the theorem.
(2) relatedly, the experiments focus on plotting the training loss of Lyapunov loss and l1/l2 loss. From my perspective, this is not informative. The loss function is likely not on the same scale; it is probably better to plot a "normalized" version so that the comparison indeed makes sense. Again, such a comparison does not reveal any interesting property about the new loss, e.g.., generalization/testing error? 
(3) Please clarity to what extent, the results hold with respect to different activation function. In Section 2.1, it is explicitly mentioned that sigmoid activation is used. In Section 2.2, the authors use the same notation \sigma. Does the result hold for other common activation functions? If not, any comments on the difficulties?
 
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors in this paper make an attempt in providing finite time convergence guarantees of the training process of neural networks, using ideas from control theory.  The loss function in the training process is framed as a Lyapunov function.  The training process at each time step is seen as  assigning dynamics to the Lyapunov function over time. The convergence of which can then be analyzed using standard control theoretic techniques. 

For some fixed input - output target, the idea is to come up with a weight update rule which guarantees the convergence rate with some assumptions on the inputs. This  is the novelty in the paper. The extension to multi-layer case is  an extension of the back propagation algorithm. 

Though the above is an interesting contribution in itself,  I am not convinced that the results for a fixed input case would generalize well to the batched input case. Which in my opinion is more general,  and has enabled the training of large scale neural networks. The authors have analyzed the robustness to perturbations  in Section 2.4. Specifically Eqn 22. Where the authors have bounds on the perturbation limits, under which it can still guarantee convergence rates. For the batched case it might need some restrictions on the choice of samples in a batch. 

The next concern I have is the experiments definitely looks very insufficient. The current experiments include much smaller datasets. I would be interested to see how this technique performs on some of the larger  neural network architectures. Since convergence guarantees become more important, only when the time it takes to train a network is much longer.  
review_title: Using Lyapunov function to model training is interesting, but scalability of the results is an issue.

Meta-review: This paper aims to study the convergence of deep neural networks training via a control theoretic analysis. This is a very interesting approach to establish theoretical understanding of deep learning. However, there are several concerns raised by the reviewers:

1.	The contribution of this paper is limited. The results simply follow from standard optimal control. It is not clear what new insight the paper provides.
2.	There are already quite a few works on control theoretic analysis of deep learning. This paper did not do a good job on presenting its novelty and difference with existing works.
3.	The experimental part is weak. It only involves small data set and very simple networks.

Based on these, I am not able to recommend acceptance for the current manuscript. But the authors are encouraged to continue this research.

Decision: Reject