Nonparametric Learning of Two-Layer ReLU Residual Units

Abstract: We describe an algorithm that learns two-layer residual units using rectified linear unit (ReLU) activation: suppose the input $\mathbf{x}$ is from a distribution with support space $\mathbb{R}^d$ and the ground-truth generative model is a residual unit of this type, given by $\mathbf{y} = \boldsymbol{B}^\ast\left[\left(\boldsymbol{A}^\ast\mathbf{x}\right)^+ + \mathbf{x}\right]$, where ground-truth network parameters $\boldsymbol{A}^\ast \in \mathbb{R}^{d\times d}$ represent a nonnegative full-rank matrix and $\boldsymbol{B}^\ast \in \mathbb{R}^{m\times d}$ is full-rank with $m \geq d$ and for $\boldsymbol{c} \in \mathbb{R}^d$, $[\boldsymbol{c}^{+}]_i = \max\{0, c_i\}$. We design layer-wise objectives as functionals whose analytic minimizers express the exact ground-truth network in terms of its parameters and nonlinearities. Following this objective landscape, learning residual units from finite samples can be formulated using convex optimization of a nonparametric function: for each layer, we first formulate the corresponding empirical risk minimization (ERM) as a positive semi-definite quadratic program (QP), then we show the solution space of the QP can be equivalently determined by a set of linear inequalities, which can then be efficiently solved by linear programming (LP). We further prove the statistical strong consistency of our algorithm, and demonstrate its robustness and sample efficiency  through experimental results.

Reviews and author responses:

 Thanks for the follow-up experiments confirming your claim. I could not find enough reasons to increase my score substantially. It was a pleasure to review your article.

 I would like to thank the authors for their response. However, regretfully, my concerns have not been fully addressed.

1) My first point was about the assumption that $A^*$ has positive entries. My concern is that the whole analysis builds on this assumption (so I'd say it is a bit more than "not all of the results are generalizable to non-positive A", as stated in the reply of the authors), starting from the warm-up on vanilla linear regression. I understand that this model has been used before (as the references pointed out in the response testify), but it is nevertheless rather restrictive.

2) My second point was about sample complexity. While I do not see a clear way to fix 1), this point could have actually been discussed better, in my opinion. The authors just prove that the sample complexity of linear regression is exponential, while the sample complexity of their algorithm is sub-exponential. I believe it should be possible to find more accurate bounds (which of course would depend on the data distribution). 

For these reasons, I plan to keep my score for this.

 Following the comments from the reviewers, we added an appendix to the draft (Appendix A) that includes an empirical running time analysis of our algorithm compared to SGD, and an analysis of the behavior of our algorithm and SGD as the condition number of the ground-truth parameters changes. Corresponding code script changes are also updated in the supplementary.

This paper gives an algorithm for learning two-layer neural networks with ReLU activation (realizable case). The network also consists of skip connections to avoid the problems of identifiability. As a warm up, the authors first give a simple algorithms based on linear regression, whenever enough samples could be observed with all-positive and all-negative entries. 

They start with an algorithm for recovering the layer two parameters. For this step, they use the $\ell_2$ loss and show that its minimizer is unique and matches with the ground truth parameters under certain condition. Subsequently, the authors replace the expectation in the objective with an average over the samples (based on the empirical risk minimization principle), which they could write as a convex quadratic program without making any parametric assumption. Finally, they show that the quadratic program can be equivalently written as a linear program, which lets them recover the parameters of the second layer (in fact the inverse matrix of this weight matrix).

Once layer 2 is learnt, the original problem reduces to that of learning the single layer non-linearity. They write the objective as a sum of weight estimator and a function estimator. They show that the minimizer of this loss is achieved when the estimated weight matrix is a diagonal matrix times the ground truth and the function measures the difference between the true nonlinear function and the scaled wright matrix. Again, they learn the function non-parametrically using a quadratic program, which they equivalently write as a linear program to get the scaled weight matrix. The scaling factors in the diagonal matrix can then be learnt using a linear regression.

Finally, the authors conduct extensive experimentation to validate that their algorithms performs superior to the vanilla SGD-based algorithm. They also show that their algorithm, especially the QP algorithm is robust to noise. Strengths: The proposed techniques are simple and lead to easy-to-implement algorithms. Also, it is interesting to know that an algorithm apart from the SGD works well for learning the parameters. The theoretical results are well-supported by experiments.

Weakness: One concern I had is that the authors solve the problem of learning the layer 2 parameters by learning its inverse matrix. Computing matrix inverse may bring in numerical issues and ill-conditioning problem. I would have liked if the authors had done some experiments regarding this. Are there cases where learning the weight matrix itself instead of its inverse may lead to better performance? Another limitation of the paper is that there are no finite-sample guarantees, as opposed to asymptotic guarantees. Understanding theoretical properties of neural networks is a major unsolved problem in learning theory. Considering that, this paper makes an important progress using simple algorithms and techniques. Although setting is somewhat simple, I felt the results should be encouraged. It would be interesting to extend this result to more complicated classes of neural networks.

 Thank you for your comments.

Regarding point 1 in weaknesses:

While not all of the results are generalizable to non-positive A, we clarify that such an assumption is reasonable and presents an expressive class of neural networks. Nonnegativity constraints on neural networks appeared in different contexts of their learning.

We give some references below where this is discussed in different ways. A really well-known example is that of binary neural networks (by Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David), where the weights are even further constrained to be 0 or 1. Finally, we want to note that we do not limit the input support, so the first layer can have a diverse set of activation values, including negative values.


[1] Hosseini-Asl, Ehsan, Jacek M. Zurada, and Olfa Nasraoui. "Deep learning of part-based representation of data using sparse autoencoders with nonnegativity constraints." IEEE transactions on neural networks and learning systems 27.12 (2015): 2486-2498.

[2] ​​Chorowski, Jan, and Jacek M. Zurada. "Learning understandable neural networks with nonnegative weight constraints." IEEE transactions on neural networks and learning systems 26.1 (2014): 62-69.

[3] Gamarnik, David, Eren C. Kızıldağ, and Ilias Zadik. "Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networks." arXiv preprint arXiv:2103.01887 (2021).

[4] Ge, Rong, et al. "Learning One-hidden-layer Neural Networks with Landscape Design." International Conference on Learning Representations. 2018.

[5] Tian, Yuandong. "An analytical formula of population gradient for two-layered relu network and its applications in convergence and critical point analysis." International Conference on Machine Learning. PMLR, 2017.

[6] Arora, Sanjeev, et al. "Provable bounds for learning some deep representations." International conference on machine learning. PMLR, 2014.

Regarding point 2 in weaknesses:

(2) Regarding sample complexity:

We believe that Appendix D quite clearly addresses the issue of the sample complexity, demonstrating that the rate of convergence of our algorithm is sub-exponential, while the complexity of the vanilla linear regression algorithm is clearly exponential. More specifically, our justification for a non-exponential sample complexity for our algorithm is based on properties of the input distribution and the ground-truth parameters (the input distribution term is based on a “splitting number” that breaks the input distribution sample space into two parts, and calculates the probability of a sample falling into each part).
Our results are complemented by strong consistency results, which we would not consider to be a low bar by any means -- these results are in “almost surely” convergence which is stronger than “with probability” convergence of sample complexity. We note that well-received papers on related topics (theoretical analysis of similar networks) such as Pilanci & Ergen (2020) do not include such sample complexity analysis.

Regarding point (a) in the main review:

Thank you for this comment. We agree, and we will follow up on this.

Regarding point (b) in the main review:

No, the results do not change. Since we do not restrict the input distribution except for its support space and second order moment, what we aim here is to let the input distribution be as general as possible in the numerical experiments to eliminate the possibility of unintentionally using any properties of specific distributions, e.g. pure Gaussian [1], symmetric distributions [2].

Regarding point (c) in the main review:

Fixed in the revision. Thanks for spotting it.

[1] Shanshan Wu, Alexandros G. Dimakis, Sujay Sanghavi. “Learning Distributions Generated by One-Layer ReLU Networks”, NeurIPS 2019.

[2] Ge, Rong, et al. "Learning Two-layer Neural Networks with Symmetric Inputs." International Conference on Learning Representations. 2018.


 Thank you for your comment regarding the condition number and for your patience as we study this. We now have updated the draft with an analysis of the condition number in Appendix A.2. We demonstrate that compared to SGD, our algorithm is quite robust to the condition number, following on the justification we provided to you earlier.


 Thank you for your comments.

Regarding the ground-truth parameter assumption: In our analysis, we make the common “realizability” assumption that the underlying "ground-truth" model conforms to the hypothesis class we aim to learn, because we focus on the learnability of the model. For fixed hyperparameters of a model, only allowing scaling/permutation invariance does not indicate anything about its expressiveness. Having no scaling/permutation invariance also does not indicate whether the model is simple or not.

Regarding the advantages/disadvantages of our algorithm compared to previous work and a comparison to Pilanci and Ergen:

Our work stands in stark contrast to Pilanci and Ergen, for example, as the underlying network model is different, and their work does not handle the non-linear activations nonparametrically. We also provide convergence results, while they focus on computational complexity results. We agree that we should further expand on the differences and similarities between our work and previous work, and we will follow on that.

Regarding the comparison to SGD:

SGD and its derivatives are not comparable here with respect to their computational complexity because:
1. SGD is run on standard L^2 output loss with bad local minima. Also, SGD gives worse accuracy of learnt results than our method. Comparing these two algorithms when the outputs are different is somewhat flawed.

2. Experimentally, the scoping of SGD epoch/iteration is different from interior point methods used for convex QP/LP optimization.

3. The original purpose of conducting SGD in numerical experiments is to verify that the conventional L^2 loss has bad local minima, and to numerically justify the difficult optimization landscape when following conventional learning patterns.

Pilanci & Ergen (2020) do not seem to develop polynomial sample complexity but only polynomial running time complexity. Our algorithm als runs in polynomial time (see Appendix B). Also, the convex training approach of Pilanci & Ergen (2020) is not comparable in terms of computational complexity, because the network architectures we are learning are different (ours: two-layer ResNets; theirs: two-layer affine/convolutional neural nets).

Regarding the nonparametric solution:

We believe there is some confusion here with respect to the way our algorithm works. It is only the estimation of nonlinear activations, e.g. hidden layer activations, that makes use of nonparametric variables, to tackle the nonlinearity in the model. The training algorithm does estimate matrix A and B that can be used as usual with the actual network, i.e. our algorithm is an estimation algorithm that returns the same type of result that backpropagation would return. During inference time, predictions are directly computed by the model estimation in A and B. No additional programs are needed to be solved.

Regarding real-world datasets:

We agree that practical use is important in the use of neural networks. In addition, we note that most of the recent research on neural network learnability decomposes neural networks to much simpler blocks, e.g. two-layer (Ge et al., 2018, 2017), ReLU-summation (Tian, 2017). Such decomposed blocks are rather limited for fitting real data.  Like other work, the purpose of this work is to focus on the learning properties of two-layer ReLU ResNets. The experiments are designed to confirm the correctness of our method in recovering the ground-truth and the issue with local optima with backprop.


 Thank you for your comments.

Regarding inversion during learning:

Thank you for this suggestion. It does make sense, and we note that based on the convergence properties discussed in Appendix D, our method hardly suffers from numerical errors when performing inverse estimation for B*.

We demonstrate in Appendix D there is less of an issue with taking the inverse due to the discrete nature of our algorithm, in which the ground-truth solution has a certain probability to fall on divisive planes as determined by the samples.

In addition, the experiments in this paper are all based on estimating the inverse of B*, which, from the numerical results reported, already showed very accurate recovered results with ~0 relative error for B* estimation.

We do value this suggestion though, and we will include further experiments that demonstrate this point.


Regarding finite-sample guarantees, we do provide an analysis of sample efficiency in Appendix D. This analysis, in conjunction with the consistency results, gives a rather complete picture of the rate of convergence of the algorithm.


 Thank you for your comments.

We agree that practical use is important in the use of neural networks. More specifically, we note that most of the recent research on neural network learnability decomposes neural networks to much simpler blocks, e.g. two-layer (Ge et al., 2018, 2017), ReLU-summation (Tian, 2017).

Such decomposed blocks are rather limited for fitting real data.

The mentioned research also runs numerical experiments using synthetic data. Like other work, the purpose of this work is to focus on the learning properties of two-layer ReLU ResNets. The experiments are designed to confirm the correctness of our method in recovering the ground-truth and the issue with local optima with backprop.


This paper considers the problem of learning the parameters of a two-layer ReLU network with a residual unit: given samples of the form $(x_i, y_i)$, with $y_i = B^*[(A^* x_i)^+ + x_i]$, the authors provide an algorithm that provably learns $A^*$ and $B^*$, as the sample size grows. It is crucially assumed that the entries of $A^*$ are non-negative and, by exploiting this fact, the learning problem is formulated via quadratic programming (QP): the authors show that there exist quadratic functionals whose minimisers can be used to deduce the ground-truth parameters. Furthermore, the QPs are re-written as linear programs (LPs), which are simpler to optimise and have the same solution space. Numerical results demonstrate the superiority of the proposed approach with respect to gradient descent methods.   STRENGHTS:

(1) The problem is timely: residual networks have enjoyed great popularity in practical applications, but from a theoretical perspective, many basic questions about their optimization are still open. 

(2) The algorithm is novel and its computational complexity is polynomial in the number of samples, due to the QP (and, subsequently, LP) simplification. As for the dependence on the number of samples, the algorithm is consistent (the error vanishes, as the sample size grows), it significantly improves upon the vanilla linear regression and the numerical simulations also show a significant improvement over gradient descent.

----------------------------------------------------------------

WEAKNESSES:

(1) The whole paper (starting from the vanilla linear regression to the non-parametric learning of the two layers) is based on the fact that $A^*$ has positive entries. This assumption is rather restrictive, not particularly practical, and it simplifies the problem significantly. As the paper appears to rely strongly on this assumption, it is not clear at all to me whether the methods discussed here can still work for a more general choice of $A^*$. 

(2) The sample complexity is only briefly discussed in Appendix D: here, the authors provide some rather heuristic arguments to justify that the sample complexity is polynomial (as opposed to vanilla linear regression whose sample complexity is exponential). I understand that, ultimately, in order to get a convergence speed result, one has to make assumptions on the data distribution. However, I guess that also the vanilla linear regression is a consistent algorithm, so proving just consistency appears to be a rather low bar. In order to fully understand the power of the QP/LP simplification presented here, the authors should make assumptions on the data distribution and show sample complexity results (instead of only giving experimental evidence).  While I generally appreciate the problem tackled in this paper (provable optimization of two-layer ReLU residual networks) and the idea of re-writing the learning problem as a QP (or even LP), unfortunately the paper in its current status appears to be (slightly) below the acceptance bar, because of the weaknesses (1)-(2) mentioned above.

On the one hand, weakness (1) seems to be inherent to the approach pursued in the manuscript (but I would still appreciate if the authors could comment on it and suggest possible ways to overcome it). On the other hand, weakness (2) does not seem fundamental, but it would require additional work. I encourage the authors to pursue the direction of giving concrete sample complexity bounds since, in my opinion, it would add value to their paper. This would mean not only to show that the sample complexity is polynomial, but also to understand the degree of such a polynomial. It would also add value to provide simulation results that indicate a certain polynomial dependence of the algorithm on the number of samples (e.g., linear, quadratic, etc) -- even better if the simulation results confirmed the theoretical bounds.

A couple of other comments are below:

(a) The discussion on the computational complexity is deferred to Appendix B. I think that this part is actually important and I would suggest to discuss it directly in the main paper. This is especially important if the authors are able to obtain sample complexity bounds, since the computational complexity bounds would be complementary to them.

(b) The input distribution chosen for the numerical experiments (mixture of N(-0.1, 1) and U(-0.9, 1.1)) seems a bit ad-hoc. Do the results change for different input distributions? (say only Gaussian, only uniform, and as a function of the mean of the Gaussian and of the uniform distributions)

(c) Page 7. nonparamatric --> nonparametric 

In this paper, the author describes an algorithm that learns a two-layer residual unit/network with ReLU activation in a teacher-student setup, under the assumption that the first-layer weights are nonnegative (without specific assumption on the data distribution), and shows that the student network is able to recover the teacher parameters in the infinite data limit (Theorem 6.2 and 6.3).  The paper is in general in good shape, I've not checked the details of the proof, but the main lines of arguments and numerical results in Section 7 look compelling. Related previous efforts are appropriately discussed. 

From a technical perspective, this paper contains some interesting ideas that are, I would say, inspired by previous efforts but adapted to the problem under study.

Since one of the major contributions of this work is that the results do not rely on strong assumptions on the data distribution, it would be of interest to provide numerical experiments on more "realistic" data and tasks (instead of the toy model presented in Section 7).
  The major novelty and contribution of this work lies in the study of the residual unit model, without relying on the assumption on the (specific) data distribution. The paper contains some interesting ideas that would be of interest in the study of more involved neural network models.


This paper considers the special case of two-layer ReLU residual units and introduces a nonparametric convex approach to optimize the training objective. The experiments also highlight the effectiveness of the proposed approach with respect to SGD applied to the original non-convex training objective. 

 The paper is mostly well written, however, there are issues requiring further discussions and explanations. See below for my detailed comments:

* Can the authors comment on the ground truth parameter assumption? This does seem quite restrictive to me since it precludes any scaling/permutation invariance which is one of the main challenges in neural network training problems. Is this really necessary for the analysis if so where exactly?

* The authors cited a series of papers training shallow neural networks via convex optimization, e.g., (Pilanci & Ergen, 2020), what are the advantages/disadvantages of the proposed method here compared to the ones in these papers? I think there should be a detailed discussion regarding this comparison?

* In Appendix B, the authors provided a complexity analysis of the algorithm, however, I think they should also include a comparison with respect to the standard SGD applied the non-convex training problem and some baselines e.g., (Pilanci & Ergen, 2020) also proposed a convex approach with a polynomial complexity in the problem parameters.

* After solving the nonparametric problem is there a way to recover a proper two-layer Residual network parameters? Does that require solving additional problems? If so can you provide some details, e.g., computational complexity?

* Since the proposed method is nonparametric, how do you obtain the estimates during the inference time? Do you need to solve additional optimization problems?

* I appreciate the experiments provided to verify the theory, however, there should have been some experiments on real benchmark datasets. Are there any specific reasons preventing such experiments, e.g., the scalability of the algorithm to larger datasets?  

-Pilanci M, Ergen T. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. ICML 2020 I think that the main contribution of this paper is incremental due to the following reasons. First of all, the model seems to be so simple that it even doesn't allow permutation and/or scaling invariance. Moreover, the paper lacks satisfactory comparative experimental analysis. 

Meta-review: In this paper, the authors propose a non-parametric approach for learning a two-layer neural net. I agree with the authors and reviewers that this is a timely problem. However, the solution in this paper come short of achieving this goal. In particular, the assumtions are very strong and cannot be generalized (e.g., non-negativity). The authors also need to better spell out the sample complexity.

Decision: Reject