Title: Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Convex optimization

deep learning theory

path norm

group sparsity

polynomial-time training

ReLU networks

parallel architectures

global optimality

computational complexity
Abstract: Understanding the fundamental principles behind the success of deep neural networks is one of the most important open questions in the current literature. To this end, we study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape. We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases. We then show that pathwise regularized training problems can be represented as an exact convex optimization problem. We further prove that the equivalent convex problem is regularized via a group sparsity inducing norm. Thus, a path regularized parallel ReLU network can be viewed as a parsimonious convex model in high dimensions. More importantly, since the original training problem may not be trainable in polynomial-time, we propose an approximate algorithm with a fully polynomial-time complexity in all data dimensions. Then, we prove strong global optimality guarantees for this algorithm. We also provide experiments corroborating our theory.

Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks

Abstract

Understanding the fundamental principles behind the success of deep neural networks is one of the most important open questions in the current literature. To this end, we study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape. We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases. We then show that pathwise regularized training problems can be represented as an exact convex optimization problem. We further prove that the equivalent convex problem is regularized via a group sparsity inducing norm. Thus, a path regularized parallel ReLU network can be viewed as a parsimonious convex model in high dimensions. More importantly, since the original training problem may not be trainable in polynomial-time, we propose an approximate algorithm with a fully polynomial-time complexity in all data dimensions. Then, we prove strong global optimality guarantees for this algorithm. We also provide experiments corroborating our theory.

(a) 2-layer NN with WD

(b) 3-layer NN with WD

(c) 3-layer NN with PR (

Ours

)

Optimal2-layerwithWD

Optimal3-layerwithWD

Optimal3-layerwithPR

0.5

0.5

0.5

0.0

口

0.0

0.0

口

-0.5

-0.5

-0.5

0

1

Q

1

1

1 Introduction

Deep Neural Networks (DNNs) have achieved substantial improvements in several fields of machine learning. However, since DNNs have a highly nonlinear and non-convex structure, the fundamental principles behind their remarkable performance is still an open problem. Therefore, advances in this field largely depend on heuristic approaches. One of the most prominent techniques to boost the generalization performance of DNNs is regularizing layer weights so that the network can fit a function that performs well on unseen test data. Even though weight decay, i.e., penalizing the glyph[lscript] 2 2 -norm of the layer weights, is commonly employed as a regularization technique in practice, recently,

(

∏

)

it has been shown that glyph[lscript] 2 -path regularizer [1], i.e., the sum over all paths in the network of the squared product over all weights in the path, achieves further empirical gains [2]. Additionally, recent studies have elucidated that convex optimization reveals surprising generalization and interpretability properties for Deep Neural Networks (DNNs). [3-6] Therefore, in this paper, we investigate the underlying mechanisms behind path regularized DNNs through the lens of convex optimization.

j

1

.

.

.

j

2

.

.

.

j

3

x

w

1

j

1

w

2

j

1

j

2

w

3

j

2

j

3

Input Data

X

Sub-network 1

Sub-network 2

. . .

Sub-network

K

Output

Parallel ReLU Network

f

θ

(

X

) =

∑

K

k

=1

(

(

XW

1

k

)

+

W

2

k

)

+

w

3

k

Path Regularization

R

(

θ

) =

∑

j

1

,j

2

,j

3

(

‖

w

1

j

1

‖

2

2

w

2

2

j

1

j

2

w

2

3

j

2

j

3

)

+

2 Parallel Neural Networks

Although DNNs are highly complex architectures due to the composition of multiple nonlinear functions, their parameters are often trained via simple first order gradient based algorithms, e.g., Gradient Descent (GD) and variants. However, since such algorithms only rely on local gradient of the objective function, they may fail to globally optimize the objective in certain cases [7, 8]. Similarly, [9, 10] showed that these pathological cases also apply to stochastic algorithms such as Stochastic GD (SGD). They further show that some of these issues can be avoided by increasing the number of trainable parameters, i.e., operating in an overparameterized regime. However, [11] reported the existence of more complicated cases, where SGD/GD usually fails. Therefore, training DNNs to global optimality remains a challenging optimization problem [12-14].

To circumvent difficulties in training, recent studies focused on models that benefit from overparameterization [15-18]. As an example, [19-23] considered a new architecture by combining multiple standard NNs, termed as sub-networks, in parallel. Evidences in [20-25] showed that this way of combining NNs yields an optimization landscape that has fewer local minima and/or saddle points so that SGD/GD generally converges to a global minimum. Therefore, many recently proposed NN-based architectures that achieve state-of-the-art performance in practice, e.g., SqueezeNet [26], Inception [27], Xception [28], and ResNext [29], are in this form.

Notation and preliminaries: Throughout the paper, we denote matrices and vectors as uppercase and lowercase bold letters, respectively. For vectors and matrices, we use subscripts to denote a certain column/element. As an example, w lkj l -1 j l denotes the j l -1 j th l entry of the matrix W lk . We use I k and 0 (or 1 ) to denote the identity matrix of size k × k and a vector/matrix of zeros (or ones) with appropriate sizes. We use [ n ] for the set of integers from 1 to n . We use ‖ · ‖ 2 and ‖ · ‖ F to represent the Euclidean and Frobenius norms, respectively. Additionally, we denote the unit glyph[lscript] p ball as B p := { u ∈ R d : ‖ u ‖ p ≤ 1 } . We also use ✶ [ x ≥ 0] and ( x ) + = max { x, 0 } to denote the 0-1 valued indicator and ReLU, respectively.

In this paper, we particularly consider a parallel ReLU network with K sub-networks and each sub-network is an L -layer ReLU network (see Figure 2) with layer weights W lk ∈ R m l -1 × m l , ∀ l ∈ [ L -1] and w Lk ∈ R m L -1 , where m 0 = d , m L = 1 1 , and m l denotes the number of neurons in the l th hidden layer. Then, given a data matrix X ∈ R n × d , the output of the network is as follows

Remark 1. Most commonly used neural networks in practice can be classified as special cases of parallel networks, e.g., standard NNs and ResNets [33] see Appendix A.2 for details.

where we compactly denote the parameters as θ := ⋃ k { W lk } L l =1 with the parameter space as Θ and each sub-network represents a standard deep ReLU network.

2.1 Our Contributions

We prove that training the path regularized parallel ReLU networks (1) is equivalent to a convex optimization problem that can be approximately solved in polynomial-time by standard convex solvers (see Table 1). Therefore, we generalize the two-layer results in [6] to multiple nonlinear layer without any strong assumptions in contrast to [21] and a much broader class of NN architectures including ResNets.

As already observed by [6, 21], regularized deep ReLU network training problems require exponential-time complexity when the data matrix is full rank, which is unavoidable. However, in this paper, we develop an approximate training algorithm which has fully polynomial-time complexity in all data dimensions and prove global optimality guarantees in Theorem 2. To the best of our knowledge, this is the first convex optimization based and polynomial-time complexity (in data dimensions) training algorithm for ReLU networks with global approximation guarantees.

We show that the equivalent convex problem is regularized by a group norm regularization where grouping effect is among the sub-networks. Therefore the equivalent convex formulation reveals an implicit regularization that promotes group sparsity among sub-networks and generalizes prior works on linear networks such as [34] to ReLU networks.

We derive a closed-form mapping between the parameters of the non-convex parallel ReLU networks and its convex equivalent in Proposition 1. Therefore, instead of solving the challenging non-convex problem, one can globally solve the equivalent convex problem and then construct an optimal solution to the original non-convex network architecture via our closed-form mapping.

2.2 Overview of Our Results

Given data X ∈ R n × d and labels y ∈ R n , we consider the following regularized training problem

where Θ := { θ ∈ Θ : W lk ∈ R m l -1 × m l , ∀ l ∈ [ L ] , ∀ k ∈ [ K ] } is the parameter space, L ( · , · ) is an arbitrary convex loss function, R ( · ) represents the regularization on the network weights, and β > 0 is the regularization coefficient.

For the rest of the paper, we focus on a scalar output regression/classification framework with arbitrary loss functions, e.g., squared loss, cross entropy or hinge loss. We also note that our derivations

can be straightforwardly extended to vector outputs networks as proven in Appendix A.11. More importantly, we use glyph[lscript] 2 -path regularizer studied in [1, 2], which is defined as

where w lkj l -1 j l is the j l -1 j th l entry of W lk . The above regularizer sums the square of all the parameters along each possible path from input to output of each sub-network k (see Figure 2) and then take the squared root of the summation. Therefore, we penalize each path in each sub-network and then group them based on the sub-network index k .

We now propose a scaling to show that (2) is equivalent to a group glyph[lscript] 1 regularized problem.

Lemma 1. The following problems are equivalent 2 :

where w Lk ∈ R m L -1 are the last layer weights of each sub-network k , and Θ s := { θ ∈ Θ : ∑ j 1 ,j 2 ,...,j L -2 ( ‖ w 1 kj 1 ‖ 2 2 ∏ L -1 l =2 w 2 lkj l -1 j l ) ≤ 1 , ∀ j L -1 , ∀ k ∈ [ K ] } denotes the parameter space after rescaling.

The advantage of the form in Lemma 1 is that we can derive a dual problem with respect to the output layer weights w Lk and then characterize the optimal layer weights via optimality conditions and the prior works on glyph[lscript] 1 regularization in infinite dimensional spaces [35]. Thus, we first apply the rescaling in Lemma 1 and then take the dual with respect to the output weights w Lk . To characterize the hidden layer weights, we then change the order of minimization for the hidden layer weights and the maximization for the dual parameter to get the following dual problem 3

where L ∗ is the Fenchel conjugate function of L , which is defined as [36]

The dual problem in (3) is critical for our derivations since it provides us with an analytic perspective to characterize a set of optimal hidden layer weights for the non-convex neural network in (1). To do so, we first show that strong duality holds for the non-convex training problem in Lemma 1, i.e., p ∗ L = d ∗ L . Then, based on the exact dual problem in (3), we propose an equivalent analytic description for the optimal hidden layer weights via the KKT conditions.

We note that strong duality for two-layer ReLU networks has already been proved by previous studies [6, 19, 21, 22, 37, 38], however, this is the first work providing an exact characterization for path regularized deep ReLU networks via convex duality.

3 Parallel networks with three layers

Here, we consider a three-layer parallel network with K sub-networks, which is a special case of (1) when L = 3 . Thus, we have the following training problem

where Θ= { ( W 1 k , W 2 k , w 3 k ) : W 1 k ∈ R d × m 1 , W 2 k ∈ R m 1 × m 2 , w 3 k ∈ R m 2 } . By Lemma 1,

linear classi

!

er

(2,2)

(3,3)

(1,0)

x

y

linear classi

!

er

(2,2)

(3,3)

(1,0)

x

y

linear classi

!

er

(2,2)

(3,3)

(1,0)

x

y

linear classi

!

er

(2,2)

(3,3)

(1,0)

x

y

𝐃

!

=

1

0

0

0

1

0

0

0

1

𝐃

"

=

1

0

0

0

1

0

0

0

0

𝐃

#

=

0

0

0

0

0

0

0

0

0

𝐃

$

=

0

0

0

0

0

0

0

0

1

Then, taking the dual of (5) with respect to the output layer weights w 3 k ∈ R m 2 and then changing the order of the minimization for { W 1 k , W 2 k } and the maximization for the dual variable v yields

∥ ∥ Here, we remark that (4) is non-convex with respect to the layer weights, we may have a duality gap, i.e., p ∗ 3 ≥ d ∗ 3 . Therefore, we first show that strong duality holds in this case, i.e., p ∗ 3 = d ∗ 3 as detailed in Appendix A.4. We then introduce an equivalent representation for the ReLU activation as follows.

Since ReLU masks the negative entries of inputs, we have the following equivalent representation where ¯ w j 1 = | w 2 j 1 | w 1 j 1 , I j 1 = sign( w 2 j 1 ) ∈ {-1 , +1 } , and we use the following alternative representation for ReLU (see Figure 3 for a two dimensional visualization)

where D ∈ R n × n is a diagonal matrix of zeros/ones, i.e., D ii ∈ { 0 , 1 } . Therefore, we first enumerate all possible signs and diagonal matrices for the ReLU layers and denote them as I j 1 , D 1 ij 1 and D 2 l respectively, where j 1 ∈ [ m 1 ] , i ∈ [ P 1 ] , l ∈ [ P 2 ] . Here, D 1 ij 1 and D 2 l denotes the masking/diagonal matrices for the first and second ReLU layers, respectively and P 1 and P 2 are the number diagonal matrices in each layer as detailed in Section A.10. Then, we convert the non-convex constraints in (6) to convex constraints using D 1 ij 1 , D 2 l and I j 1 .

Using the representation in (7), we then take the dual of (6) to obtain the convex bidual of the primal problem (4) as detailed in the theorem below.

Theorem 1. The non-convex training problem in (4) can be cast as the following convex program

where ‖· ‖ F, 1 denotes a d × m 1 dimensional group Frobenius norm operator such that given a vector u ∈ R dm 1 P , ‖ u ‖ F, 1 := ∑ P i =1 ‖ U i ‖ F , where U i ∈ R d × m 1 are reshaped partitions of u . Moreover, the convex set C is defined as

where I s ij 1 l ∈ { +1 , -1 } , M = 2 m 1 , and z , z ′ ∈ R dm 1 MP 1 P 2 are constructed by stacking z s ij 1 l , z s ′ ij 1 l ∈ R d , ∀ i ∈ [ P 1 ] , l ∈ [ P 2 ] , j 1 ∈ [ m 1 ] , s ∈ [ M ] , respectively. Also, the effective data matrix ˜ X ∈ R n × dm 1 MP 1 P 2 is defined as ˜ X := I M ⊗ ˜ X s and

We next derive a mapping between the convex program (8) and the non-convex architecture (4).

Proposition 1. An optimal solution to the non-convex parallel network training problem in (4) , denoted as { W ∗ 1 k , w ∗ 2 k , w ∗ 3 k } K k =1 , can be recovered from an optimal solution to the convex program in (8) , i.e., { z ∗ , z ′ ∗ } via a closed-form mapping. Therefore, we prove a mapping between the parameters of the parallel network in Figure 2 and its convex equivalent.

Next, we prove that the convex program in (8) can be globally optimized with a polynomial-time complexity given X has fixed rank, i.e., rank( X ) = r < min { n, d } .

Proposition 2. Given a data matrix such that rank( X ) = r < min { n, d } , the convex program in (8) can be globally optimized via standard convex solvers with O ( d 3 m 3 1 m 3 2 2 3( m 1 +1) m 2 n 3( m 1 +1) r ) complexity, which is a polynomial-time complexity in terms of n, d . Note that here globally optimizing the training objective means to achieve the exact global minimum up to any arbitrary machine precision or solver tolerance.

Below, we show that the complexity analysis in Proposition 2 extends to arbitrarily deep networks. Corollary 1. The same analysis can be readily applied to arbitrarily deep networks. Therefore, given rank( X ) = r < min { n, d } , we prove that L -layer architectures can be globally optimized with O ( d 3 ( ∏ L -2 j =1 m 3 j ) 2 3 ∑ L -1 j =1 m j n 3 r ( 1+ ∑ L -2 l =1 ∏ l j =1 m j ) ) , which is polynomial in n, d .

)

4

5

6

7

8

9

# of features (d)

10 -1

10 0

10 1

10 2

10 3

Objective value

(a) Objective value (

↓

)

4

5

6

7

8

9

# of features (d)

0

1000

2000

3000

4000

5000

#of hyperplane arrangements (P)

Exact

Proposed

(b)

#

of hyperplane arrangements (

P

3.1 Polynomial-time training for arbitrary data

Based on Corollary 1, exponential complexity is unavoidable for deep networks when the data matrix is full rank, i.e., rank ( X ) = min { n, d } . Thus, we propose a low rank approximation to the model in (4). We first denote the rankr approximation of X as ˆ X r such that ‖ X -ˆ X r ‖ 2 ≤ σ r +1 , where σ r +1 represents the ( r +1) th largest singular value of X . Then, we have the following result.

Theorem 2. Given an R -Lipschitz convex loss function L ( · , y ) , the regularized training problem

can be solved using the data matrix ˆ X r to achieve the following optimality guarantee

where p r denotes the objective value achieved by the parameters trained using ˆ X r .

Remark 2. Theorem 1 and 2 imply that for a given arbitrary rank data matrix X , the regularized training problem in (4) can be approximately solved by convex solvers to achieve a worst-case approximation p ∗ 3 ( 1 + √ m 1 m 2 Rσ r +1 β ) 2 with complexity O ( d 3 m 3 1 2 3( m 1 +1) n 3( m 1 +1) r ) , where r glyph[lessmuch] min { n, d } . Therefore, even for full rank data matrices where the complexity is exponential in n or d , one can approximately solve the convex program in (8) in polynomial-time. Moreover, we remark that the approximation error proved in Theorem 2 can be arbitrarily small for practically relevant problems. As an example, consider a parallel network training problem with glyph[lscript] 2 loss function, then the upperbound becomes (1 + √ m 1 m 2 σ r +1 β ) 2 , which is typically close to one due to fast decaying singular values in practice (see Figure 4).

3.2 Representational power: Two versus three layers

Here, we provide a complete explanation for the representational power of three-layer networks by comparing with the two-layer results in [6]. We first note that three-layer networks have substantially higher expressive power due to the non-convex interactions between hidden layers as detailed in [39, 40]. Furthermore, [41] show that layerwise training of three-layer networks can achieve comparable performance to deeper models, e.g., VGG-11, on Imagenet. There exist several studies analyzing two-layer networks, however, despite their empirical success, a full theoretical understanding and interpretation of three-layer networks is still lacking in the literature. In this work, we provide a complete characterization for three-layer networks through the lens of convex optimization theory. To understand their expressive power, we compare our convex program for three-layer networks in (8) with its two-layer counterpart in [6].

[6] analyzes two-layer networks with one ReLU layer, so that the data matrix X is multiplied with a single diagonal matrix (or hyperplane arrangement) D i . Thus, the effective data matrix is in the form of ˜ X s = [ D 1 X . . . D P X ] . However, since our convex program in (8) has two nonlinear ReLU layers, the composition of these two-layer can generate substantially more complex features via locally linear variables { w s ij 1 l } multiplying the d -dimensional blocks of the columns of the effective data matrix ˜ X s in Theorem 1. Although this may seem similar to the features in [21], here, we have 2 m 1 variables for each linear region unlike [21] which employ 2 variables per linear region. Moreover, [21] only considers the case where the second hidden layer has only one neuron, i.e., m 2 = 1 , which is not a standard deep networks. Hence, we exactly describe the impact of having one more ReLU layer and its contribution to the representational power of the network.

4 Experiments

In this section 4 , we present numerical experiments corroborating our theory.

Low rank model in Theorem 2: To validate our claims, we generate a synthetic dataset as follows. We first randomly generate a set of layer weights for a parallel ReLU network with K = 5 subnetworks by sampling from a standard normal distribution. We then obtain the labels as y = ∑ k ( ( XW 1 k ) + W 2 k ) + w 3 k +0 . 1 glyph[epsilon1] , where glyph[epsilon1] ∼ N ( 0 , I n ) . To promote a low rank structure in the data, we first sample a matrix from the standard normal distribution and then set σ r +1 = . . . = σ d = 1 . We consider a regression framework with glyph[lscript] 2 loss and ( n, r, β, m 1 , m 2 ) = (15 , d/ 2 , 0 . 1 , 1 , 1) and present the numerical results in Figure 4. Here, we observe that the low rank approximation of the objective p r is closer to p ∗ 3 than the worst-case upper-bound predicted by Theorem 2. However, in Figure 4b, the low rank approximation provides a significant reduction in the number of hyperplane arrangements, and therefore in the complexity of solving the convex program.

Toy dataset: We use a toy dataset with 5 samples and 2 features, i.e., ( n, d ) = (5 , 2) . To generate the dataset, we forward propagate i.i.d. samples from a standard normal distribution, i.e., x i ∼ N ( 0 , I d ) , through a parallel network with 3 layers, 5 sub-networks, and 3 neurons, i.e., ( L, K, m 1 , m 2 ) = (3 , 5 , 3 , 1) . We then train the parallel network in (4) on this toy dataset using both our convex program (8) and non-convex SGD. We provide the training objective and wall-clock time in Table 2, where we particularly include 5 initialization trials for SGD. This experiment shows that when the number of sub-networks K is small, SGD trials fail to converge to the global minimum achieved by our convex program. However, as we increase K , the number of trials converging to global minimum gradually increases. Therefore, we show the benign overparameterization impact.

Image classification: We conduct experiments on benchmark image datasets, namely CIFAR-10 [42] and Fashion-MNIST [43]. We particularly consider a ten class classification task and use a parallel network with 40 sub-networks and 100 hidden neurons, i.e., ( K,m 1 , m 2 ) = (40 , 100 , 1) . In Figure 5, we plot the test accuracies against wall-clock time, where we include several different optimizers as well as SGD. Moreover, we include a parallel network trained with SGD/Adam and Weight Decay (WD) regularization to show the effectiveness of path regularization in (4). We first note that our convex approach achieves both faster convergence and higher final test accuracies for both dataset. However, the performance gain for Fashion-MNIST seems to be significantly less compared to the CIFAR-10 experiment. This is due to the nature of these datasets. More specifically, since CIFAR-10 is a much more challenging dataset, the baseline accuracies are quite low (around ∼ 50% ) unlike Fashion-MNIST with the baseline accuracies around ∼ 90% . Therefore, the accuracy improvement achieved by the convex program seems low in Figure 5b. We also observe that weight decay achieves faster convergence rates however path regularization yields higher final test accuracies. It is normal to have faster convergence with weight decay since it can be incorporated into gradient-based updates without any computational overhead.

5 Related Work

Parallel neural networks were previously investigated by [22, 23]. Although these studies provided insights into the solutions, they require impractical assumptions, e.g., sparsity among sub-networks in Theorem 1 of [23]) and linear activations and hinge loss assumptions in [22].

Recently, [6] studied weight decay regularized two-layer ReLU network training problems and introduced polynomial-time trainable convex formulations. However, their analysis is restricted to standard two-layer ReLU networks, i.e., in the form of f θ ( X ) = ( XW 1 ) + w 2 . The reasons for this restriction is that handling more than one ReLU layer is a substantially more challenging optimization problem. As an example, a direct extension of [6] to three-layer NNs will yield doubly exponential complexity, i.e., O ( n rn r ) for a rankr data matrix, due to the combinatorial behavior of multiple ReLU layers. Thus, they only examined the case with a single ReLU layer (see Table 1 for details and references in [6]). Additionally, since they only considered standard two-layer ReLU networks, their analysis is not valid for a broader range of NN-based architectures as detailed in Remark 1. Later on, [21] extended this approach to three-layer ReLU networks. However, since they analyzed glyph[lscript] 2 -norm regularized training problem, they had to put unit Frobenius norm constraints on

(a) CIFAR-10

(b) Fashion-MNIST

0.55

0.90

Test Accuracy

0.50

Test Accuracy

0.88

0.45

0.86

0.40

Convex(Ours)

0.84

Convex(Ours)

SGD

SGD

Adam

Adam

Adagrad

Adagrad

0.35

Adadelta

0.82

Adadelta

SGD-WD

SGD-WD

Adam-WD

Adam-WD

0.30

0.80

0

200

400

600

800

1000

0

200

400

009

800

1000

Time(s)

Time(s)

the first layer weights, which does not reflect the settings in practice. In addition, their analysis is restricted to the networks with a single neuron in the second layer (i.e. m 2 = 1 ) that is in the form of f θ ( X ) = ∑ k ( ( XW 1 k ) + w 2 k ) + w 3 k . Since this architecture only allows a single neuron in the second layer, each sub-network k has an expressive power that is equivalent to a standard two-layer network rather than three-layer. This can also be realized from the definition of the constraint set C in Theorem 1. Specifically, the convex set C in [21] has decoupled constraints across the hidden layer index j 1 whereas our formulation sums the responses over hidden neurons before feeding through the next layer as standard deep networks do. Therefore, this analysis does not reflect the true power of deep networks with L > 2 . Moreover, the approach in [21] has exponential complexity when the data matrix has full rank, which is unavoidable.

However, we analyze deep neural networks in (1) without any assumption on the weights. Furthermore, we develop an algorithm which has fully polynomial-time complexity in data dimensions and prove strong global optimality guarantees for this algorithm in Theorem 2.

6 Concluding Remarks

We studied the training problem of path regularized deep parallel ReLU networks, which includes ResNets and standard deep ReLU networks as its special cases. We first showed that the non-convex training problem can be equivalently cast as a single convex optimization problem. Therefore, we achieved the following advantages over the training on the original non-convex formulation: 1) Since our model is convex, it can be globally optimized via standard convex solvers whereas the non-convex formulation trained with optimizers such as SGD might be stuck at a local minimum, 2) Thanks to convexity, our model does not require any sort of heuristics and additional tricks such as learning rate schedule and initialization scheme selection or dropout. More importantly, we proposed an approximation to the convex program to enable fully polynomial-time training in terms of the number of data samples n and feature dimension d . Thus, we proved the polynomial-time trainability of deep ReLU networks without requiring any impractical assumptions unlike [6, 21]. Notice that we derive an exact convex program only for three-layer networks, however, recently [19] proved that strong duality holds for arbitrarily deep parallel networks. Therefore, a similar analysis can be extended to deeper networks to achieve an equivalent convex program, which is quite promising for future work. Additionally, although we analyzed fully connected networks in this paper, our approach can be directly extended to various NN architectures, e.g., threshold/binary networks [44], convolution networks [45], generative adversarial networks [46], NNs with batch normalization [47], autoregressive models [48], and Transformers [49, 50].

References

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pages 1376-1401, Paris, France, 03-06 Jul 2015. PMLR.

Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Tolga Ergen and Mert Pilanci. Convex geometry of two-layer relu networks: Implicit autoencoding and interpretable models. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 4024-4033, Online, 26-28 Aug 2020. PMLR.

Tolga Ergen and Mert Pilanci. Convex geometry and duality of over-parameterized neural networks. Journal of Machine Learning Research , 22(212):1-63, 2021.

T. Ergen and M. Pilanci. Convex optimization for shallow neural networks. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 79-83, 2019.

Mert Pilanci and Tolga Ergen. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 7695-7705. PMLR, 13-18 Jul 2020.

Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah. Failures of gradient-based deep learning. arXiv preprint arXiv:1703.07950 , 2017.

Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning , volume 1. MIT press Cambridge, 2016.

Rong Ge, Jason D. Lee, and Tengyu Ma. Learning one-hidden-layer neural networks with landscape design, 2017.

Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International Conference on Machine Learning , pages 4433-4441. PMLR, 2018.

Animashree Anandkumar and Rong Ge. Efficient approaches for escaping higher order saddle points in non-convex optimization. In Conference on learning theory , pages 81-102, 2016.

Bhaskar DasGupta, Hava T Siegelmann, and Eduardo Sontag. On the complexity of training neural networks with continuous activation functions. IEEE Transactions on Neural Networks , 6(6):1490-1504, 1995.

Avrim Blum and Ronald L Rivest. Training a 3-node neural network is np-complete. In Advances in neural information processing systems , pages 494-501, 1989.

Peter Bartlett and Shai Ben-David. Hardness results for neural network approximation problems. In European Conference on Computational Learning Theory , pages 50-62. Springer, 1999.

Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. SGD learns over-parameterized networks that provably generalize on linearly separable data. CoRR , abs/1710.10174, 2017.

Simon S Du and Jason D Lee. On the power of over-parametrization in neural networks with quadratic activation. arXiv preprint arXiv:1803.01206 , 2018.

Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In 35th International Conference on Machine Learning, ICML 2018 , pages 372-389. International Machine Learning Society (IMLS), 2018.

Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization of neural networks. arXiv preprint arXiv:1805.12076 , 2018.

Yifei Wang, Tolga Ergen, and Mert Pilanci. Parallel deep neural networks have zero duality gap. In The Eleventh International Conference on Learning Representations , 2023.

Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 3004-3014. PMLR, 18-24 Jul 2021.

Tolga Ergen and Mert Pilanci. Global optimality beyond two layers: Training deep relu networks via convex programs. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2993-3003. PMLR, 18-24 Jul 2021.

Hongyang Zhang, Junru Shao, and Ruslan Salakhutdinov. Deep neural networks with multibranch architectures are intrinsically less non-convex. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1099-1109, 2019.

Benjamin D Haeffele and René Vidal. Global optimality in neural network training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7331-7339, 2017.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In Advances in neural information processing systems , pages 550-558, 2016.

Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360 , 2016.

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence , 2017.

François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1251-1258, 2017.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492-1500, 2017.

Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. In 6th International Conference on Learning Representations, ICLR 2018 , 2018.

Surbhi Goel, Adam Klivans, Pasin Manurangsi, and Daniel Reichman. Tight Hardness Results for Training Depth-2 ReLU Networks. In James R. Lee, editor, 12th Innovations in Theoretical Computer Science Conference (ITCS 2021) , volume 185 of Leibniz International Proceedings in Informatics (LIPIcs) , pages 22:1-22:14, Dagstuhl, Germany, 2021. Schloss Dagstuhl-LeibnizZentrum für Informatik.

Vincent Froese, Christoph Hertrich, and Rolf Niedermeier. The computational complexity of relu network training parameterized by data dimensionality, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Zhen Dai, Mina Karzand, and Nathan Srebro. Representation costs of linear neural networks: Analysis and design. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 26884-26896. Curran Associates, Inc., 2021.

Saharon Rosset, Grzegorz Swirszcz, Nathan Srebro, and Ji Zhu. L1 regularization in infinite dimensional feature spaces. In International Conference on Computational Learning Theory , pages 544-558. Springer, 2007.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Tolga Ergen and Mert Pilanci. Convex geometry and duality of over-parameterized neural networks. Journal of Machine Learning Research , 22(212):1-63, 2021.

Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research , 18(1):629-681, 2017.

Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Huy Tuan Pham and Phan-Minh Nguyen. Global convergence of three-layer neural networks in the mean field regime. In International Conference on Learning Representations , 2021.

Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to imagenet. In International conference on machine learning , pages 583-593. PMLR, 2019.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. http://www.cs. toronto.edu/kriz/cifar.html , 2014.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.

Tolga Ergen, Halil I. Gulluk, Jonathan Lacotte, and Mert Pilanci. Globally optimal training of neural networks with threshold activation functions. In The Eleventh International Conference on Learning Representations , 2023.

Tolga Ergen and Mert Pilanci. Implicit convex regularizers of cnn architectures: Convex optimization of two- and three-layer networks in polynomial time. In International Conference on Learning Representations , 2021.

Arda Sahiner, Tolga Ergen, Batu Ozturkler, Burak Bartan, John M. Pauly, Morteza Mardani, and Mert Pilanci. Hidden convexity of wasserstein GANs: Interpretable generative models with closed-form solutions. In International Conference on Learning Representations , 2022.

Tolga Ergen, Arda Sahiner, Batu Ozturkler, John M. Pauly, Morteza Mardani, and Mert Pilanci. Demystifying batch normalization in reLU networks: Equivalent convex optimization models and implicit regularization. In International Conference on Learning Representations , 2022.

Vikul Gupta, Burak Bartan, Tolga Ergen, and Mert Pilanci. Convex neural autoregressive models: Towards tractable, expressive, and theoretically-backed models for sequential forecasting and generation. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3890-3894, 2021.

Tolga Ergen, Behnam Neyshabur, and Harsh Mehta. Convexifying transformers: Improving optimization and understanding of transformer networks. arXiv, 2022.

Arda Sahiner, Tolga Ergen, Batu Ozturkler, John Pauly, Morteza Mardani, and Mert Pilanci. Unraveling attention via convex duality: Analysis and interpretations of vision transformers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 19050-19088. PMLR, 17-23 Jul 2022.

Daniel Smilkov Carter and Shan. A neural network playground. https://playground.tensorflow.org/.

Michael Grant and Stephen Boyd. CVX: Matlab software for disciplined convex programming, version 2.1. http://cvxr.com/cvx , March 2014.

Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision , 5(1):42-60, 2018.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim. Do we need hundreds of classifiers to solve real world classification problems? Journal of Machine Learning Research , 15(90):3133-3181, 2014.

Maurice Sion. On general minimax theorems. Pacific J. Math. , 8(1):171-176, 1958.

Piyush C Ojha. Enumeration of linear threshold functions from the lattice of hyperplane intersections. IEEE Transactions on Neural Networks , 11(4):839-850, 2000.

Richard P Stanley et al. An introduction to hyperplane arrangements. Geometric combinatorics , 13:389-496, 2004.

RO Winder. Partitions of n-space by hyperplanes. SIAM Journal on Applied Mathematics , 14(4):811-818, 1966.

Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers , (3):326-334, 1965.

Arda Sahiner, Tolga Ergen, John M. Pauly, and Mert Pilanci. Vector-output relu neural network problems are copositive programs: Convex analysis of two layer networks and polynomial-time algorithms. In International Conference on Learning Representations , 2021.

Supplementary Material

Table of Contents

A Appendix

A.1 Additional numerical results and details

In this section, we provide new numerical results and detailed information about our experiments in the main paper.

Decision boundary plots in Figure 1: In order to visualize the capabilities of our convex training approach, we perform an experiment on the spiral dataset which is known to be challenging for 2-layer networks while 3-layer networks can readily interpolate the training data (i.e. exactly fit the training labels) [51]. As the baselines of our analysis, we include the two-layer convex training approach in [6] and recently introduced three-layer convex training approach (with weight decay regularization and several architectural and parametric assumptions) in [21]. As our experimental setup, we consider a binary classification task with y ∈ { +1 , -1 } n and squared loss. We choose ( n, m 1 , m 2 , P 1 , P 2 , β ) = (30 , 5 , 1 , 11 , 11 , 1 e -4) and for [6] we use P = 50 neurons/hyperplane arrangements. We also use CVPXY with MOSEK solver [52-54] to globally solve the convex programs. As demonstrated in Figure 1, baselines methods, especially [21], fit a function that is significantly different than the underlying spiral data distribution. This clearly shows that since [6] is restricted two-layer networks and [21] have multiple assumptions, i.e., unit Frobenius norm constraints on the layer weights ( ‖ W lk ‖ F ≤ 1 , ∀ l ∈ [ L -2] ) and last hidden layer weights cannot be matrices ( w ( L -1) k ∈ R m L -2 ), both baseline approaches fail to reflect true expressive power of deep networks ( L > 2 ). On the other hand, our convex training approach for path regularized networks fits a model that successfully describes the underlying data distribution for this challenging task.

Additional experiments: We also conduct experiments on several datasets available in UCI Machine Learning Repository [55], where we particularly selected the datasets from [56] such that n ≤ 500 . For these datasets, we consider a conventional binary classification framework with ( m 1 , m 2 , K, β ) = (100 , 1 , 40 , 0 . 5) and compare the test accuracies of non-convex architectures trained with SGD and Adam with their convex counter parts in (8). For these experiments, we use the 80% -20% splitting ratio for the training and test sets. Furthermore, we train each algorithm long enough to reach training accuracy one. As shown in Table 3, our convex approach achieves higher or the same test accuracy compared to the standard non-convex training approach for most of the datasets (precisely 20 and 19 out of 21 datasets for SGD and Adam, respectively). We also note that for this experiment, we used the unconstrained form in (11) with the approximate version in Remark 3.3 of [6].

Details for the experiments in Table 2 and Figure 5: We first note that for the experiments in Table 2, we use CVX/CVPXY [52-54] to globally solve the proposed convex program in (8). For these

Highest test accuracy

20/21

19/21

experiments, we use a laptop with i7 processor and 16GB of RAM. In order to tune the learning rate of SGD/GD, we first perform training with a bunch of learning different learning rates and select the one with the best performance on the validation datasets, which is 0 . 005 in these experiments.

For comparatively larger scale image classification experiments in Figure 5, we utilize a cluster GPU with 50GB of memory. However since the equivalent convex program in (8) has constraint which are challenging to handle for these datasets, we propose the following unconstrained convex problem which has the same global minima with the constrained version as discussed in [48]

where λ > 0 coefficient to penalize the violated constraints and h C ( z ) is a function to sum the absolute value of all constraint violations defined as

Thus, we obtain an unconstrained version of convex optimization problem (11), where one can use commonly employed first-order gradient based optimizers, e.g., SGD and Adam, available in deep learning libraries such PyTorch and Tensorflow. For both CIFAR-10 and Fashion-MNIST, we use the same training and test splits in the original datasets. We again perform a grid search to tune the learning rate, where the best performance is achieved by the following choices

as the learning rates for CIFAR-10 and Fashion-MNIST, respectively. We choose the momentum coefficient of the SGD optimizer as 0 . 9 . In addition to this, we set P 1 P 2 = K and λ = 1 e -5 .

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Stage

55

60

65

70

75

80

85

90

CIFAR-10 Test Accuracy

c

onvex

layerwise training

More importantly, we remark that these experiments are performed by using a small sampled subset of hyperplane arrangements rather than sampling all possible arrangements as detailed in Remark 3.3 of [6]. In particular, we first generate random weight matrices from a multivariate standard normal distribution and then solve the convex program using only the arrangements of the sampled weight matrices.

Details for the experiments in Figure 6: Layerwise training with shallow neural networks was proven to work remarkably well. Particularly, [41] shows that one can train three-layer neural networks sequentially to build deep networks that outperform end-to-end training with SOTA architectures. In Figure 6, we apply this layerwise training procedure with our convex training approach. In particular, each stage in this figure is our three-layer convex formulation in Theorem 1. Here, we use the same experimental setting in the previous section. We observe that making the network deeper by stacking convex layers resulted in significant performance improvements. Specificaly, at the fifth stage, we achieved almost 85% accuracy for CIFAR-10 unlike below 60% accuracies in Figure 5.

A.2 Parallel ReLU networks

The parallel networks f θ ( X ) models a wide range of NNs in practice. As an example, standard NNs and ResNets [33] are special cases of this network architecture. To illustrate this, let us consider a parallel ReLU with two sub-networks and four layers, i.e., K = 2 and L = 4 . If we set W lk = W l ∀ k ∈ [2] , l ∈ [4] then our architecture reduces to a standard four-layer network where W 1 ∈ R d × m 1 , W 2 ∈ R m 1 × m 2 , W 3 ∈ R m 2 × m 3 , w 4 ∈ R m 3 . For ResNets, we first remark that since residual blocks are usually used after a ReLU activation function, which is positively homogeneous of degree one, in practice, each residual block takes only nonnegative entries as its inputs. Thus, we can assume X ∈ R n × d + without loss of generality. We also assume that weights obey the following form: W 11 = W 1 , W 21 = W 2 , W 12 = W 22 = I d , W 31 = W 32 = W 3 , and w 41 = w 42 = w 4 then

which is a shallow ResNet as demonstrated in Figure 1 of [25].

A.3 Proof of Lemma 1

r kj L -1 = 0 , this means that k th sub-network does not contribute the output of the parallel network in (1). Therefore, we can remove the paths with r kj L -1 = 0 without loss of generality. Now, we use the following change of variable

We now note that

Then, (2) can be restated as follows

We also note that one can relax the equality constraint as an inequality constraint without loss of generality. This is basically due to the fact that if a constraint is not tight, i.e., strictly less than one, at the optimum then we can remove that constraint and make the corresponding output layer weight arbitrarily small via a simple scaling to make the objective value smaller. However, this would lead to a contradiction since this scaling further reduces the objective, which means that the initial set of layer weights (that yields a strict inequality in the constraints) are not optimal.

A.4 Proof of Theorem 1

To obtain the bidual problem of (4), we first utilize semi-infinite duality theory as follows. We first compute the dual of (6) with respect to the dual parameter v to get

where ‖ µ ‖ TV denotes the total variation norm of the signed measure µ . Notice that (12) is an infinitedimensional neural network training problem similar to the one studied in [38]. More importantly, this problem is convex since the model is linear with respect to the measure µ and the loss and regularization functions are convex [38]. Thus, we have no duality gap, i.e., d ∗ 3 = p ∗ ∞ . In addition to this, even though (12) is an infinite-dimensional convex optimization problem, it reduces to a problem

with at most n +1 neurons at the optimum due to Caratheodory's theorem [35]. Therefore, (12) can be equivalently stated as the following finite-size convex optimization problem where K ∗ ≤ n +1 . We further remark that given K ≥ K ∗ , (13) and (5) are the same problems, which also proves strong duality as p ∗ 3 = p ∗ ∞ = d ∗ 3 . In the remainder of the proof, we show that using an alternative representation for the ReLU activation, we can achieve a finite-dimensional convex bidual formulation.

Now we restate the dual problem (6) as

We first note that using the representation in (7), the dual constraint in (14) can be written as

where Θ s = { ( W 1 , W 2 ) : ∑ m 1 j 1 =1 ‖ w 1 j 1 ‖ 2 2 w 2 2 j 1 j 2 ≤ 1 , ∀ j 2 ∈ [ m 2 ] } , we apply a variable change as w j 1 = | w 2 j 1 | w 1 j 1 and define the set C il as

We also note that P 1 and P 2 denote the number of possible hyperplane arrangement for the first and second ReLU layer (see Appendix A.10 for details).

Then we have

where we use M := |{± 1 } m 1 | = 2 m 1 to enumerate all possible sign patterns {I j 1 } m 1 j 1 =1 of size m 1 . Using the equivalent representation above, we rewrite the dual (14) as

Since the problem above is convex and satisfies the Slater's condition when all the parameters are set to zero, we have strong duality [36], and thus we can state (16) as

Due to Sion's minimax theorem [57], we can change the order the inner minimization and maximization to obtain closed-form solutions for the maximization over the variable v . This yields the following problem

Finally, we introduce a set of variable changes changes as z s ij 1 l = √ m 2 γ s il I s ij 1 l w s ij 1 l and z s ′ ij 1 l = √ m 2 γ s ′ il I s ′ ij 1 l w s ′ ij 1 l such that (20) can be cast as the following convex problem

where the constraint set C s il are defined as

Notice that (21) is a constrained convex optimization problem with 2 dm 1 MP 1 P 2 variables and 2 n ( m 1 +1) MP 1 P 2 constraints in the set C s il .

A.5 Proof of Proposition 1

In this section, we prove that once the convex program in (21) is globally optimized to obtain a set of optimal solutions { z s ∗ ij 1 l , z s ′∗ ij 1 l } i,j 1 ,l,s , one can recover an optimal solution to the non-convex training

problem (4) via a simple closed-form mapping as detailed below where

with k ′ = k -MP 1 P 2 . Hence, we achieve an optimal solution to the original non-convex training problem (4) as { W ∗ 1 k , W ∗ 2 k , w ∗ 3 k } 2 MP 1 P 2 k =1 , where W ∗ 1 k ∈ R d × m 1 , W ∗ 2 k ∈ R m 1 × m 2 , and w ∗ 3 k ∈ R m 2 respectively. Next, we confirm that the proposed set of layer weights are indeed optimal by plugging them back to both the convex and non-convex objectives.

We first verify that both the optimal convex and non-convex layer weights give the same network output as follows (8), i.e.,

Now, we show that the proposed set of weight matrices for the non-convex problem achieves the same regularization cost with (21), i.e.,

Since { W ∗ 1 k , W ∗ 2 k , w ∗ 3 k } 2 MP 1 P 2 k =1 yields the same network output and regularization cost with the optimal parameters of the convex program in (21), we conclude that the proposed set parameters for the non-convex problem also achieves the optimal objective value p ∗ 3 , i.e.,

A.6 Proof of Theorem 2

We start with defining the optimal parameters for the original and rankk approximation of the rescaled problem in (5) as

and the objective value achieved by the parameters trained using ˆ X r as

Then, we have

where ( i ) and ( iii ) follow from the optimality definitions of the original and approximated problems in (22). In addition, ( ii ) and ( iv ) follow from the relations below

where we use the convexity and R -Lipschtiz property of the loss function, convexity of glyph[lscript] 2 -norm, 1 -Lipschitz property of the ReLU activation, ‖ x ‖ 1 ≤ √ m ‖ x ‖ 2 for x ∈ R m , and max k ∑ m 1 j 1 =1 ‖ ˆ w 1 kj 1 ‖ 2 2 ˆ w 2 2 kj 1 j 2 ≤ 1 from the rescaling in Lemma 1 for (1) , (2) , (3) , (4) , (5) , and (6) respectively.

A.7 Proof for the dual problem in (3)

In order to prove the dual problem, we directly utilize Fenchel duality [36]. Let us first rewrite the primal regularized training problem after the application of the rescaling in Lemma 1 as follows

The corresponding Lagrangian can be computed by incorporating the constraints into objective via a dual veriable as follows

We then define the following dual function

Hence, we write the dual problem of (23) as

However, since the hidden layer weights are the variables of the outer minimization, we cannot directly characterize the optimal hidden layer weight in the form above. Thus, as the last step of the derivation, we change the order of the minimization over θ and the maximization over v to obtain the following lower bound

A.8 Hyperplane arrangements

Here, we review the notion of hyperplane arrangements detailed in [6].

We first define the set of all hyperplane arrangements for the data matrix X as

where |H| ≤ 2 n . The set H all possible { +1 , -1 } labelings of the data samples { x i } n i =1 via a linear classifier w ∈ R d . We now define a new set to denote the indices with positive signs for each element in the set H as S := { {∪ h i =1 { i }} : h ∈ H } . With this definition, we note that given an element S ∈ S , one can introduce a diagonal matrix D ( S ) ∈ R n × n defined as

D ( S ) can be also viewed as a diagonal matrix of indicators, where each diagonal entry is one if the the corresponding sample labeled as +1 by the linear classifier w , zero otherwise. Therefore, the output of ReLU activation can be equivalently written as ( Xw ) + = D ( S ) Xw provided that D ( S ) Xw ≥ 0 and ( I n -D ( S )) Xw ≤ 0 are satisfied. One can define more compactly these two constraints as (2 D ( S ) -I n ) Xw ≥ 0 . We now denote the cardinality of S as P , and obtain the following upperbound

where r := rank ( X ) ≤ min( n, d ) [58-61].

A.9 Low rank model in Theorem 2

In Section 3.1, we propose an glyph[epsilon1] -approximate training approach that has polynomial-time complexity even when the data matrix is full rank. Here, you can select the rank r by plugging in the desired approximation error and network structure in equation 10. We show that the approximation error proved in Theorem 2 can be arbitrarily small for practically relevant problems. As an example, consider a parallel architecture training problem with glyph[lscript] 2 loss function, then the upperbound becomes (1 + √ m 1 m 2 σ r +1 β ) 2 , which can be arbitrarily close to one due to presence of noise component (with small σ r +1 ) in most datasets in practice (see Figure 4 for an empirical verification). This observation is also valid for several benchmark datasets, including MNIST, CIFAR-10, and CIFAR-100, which exhibit exponentially decaying singular values (see Figure 7) and therefore effectively has a low rank structure. In addition, singular values can be computed to set the target rank and the value of the regularization coefficient to obtain any desired approximation ratio using Theorem 2.

A.10 Proof of Proposition 2 and Corollary 1

We first review the multi-layer hyperplane arrangements concept introduced in Section 2.1 of [21]. Based on this concept, we then calculate the training complexity to globally solve the convex program in Theorem 1.

If we denote the number of hyperplane arrangements for the first ReLU layer as P 1 , then from Appendix A.8 we know that

In order to obtain a bound for the number of hyperplane arrangements in the second ReLU layer we first note that preactivations of the second ReLU layer, i.e., ( XW 1 ) + w 2 can be equivalently represented as a matrix-vector product form by using the effective data matrix ¯ X := [ I 1 D 11 X I 2 D 12 X . . . I m 1 D 1 m 1 X ] due to the equivalent representation in (7). Therefore, given rank( ¯ X ) = r 2 ≤ m 1 r

However, notice that ¯ X is not a fixed data matrix since we can choose each diagonal D 1 j 1 among a set { D i } P 1 i =1 of size P 1 due to (24) and the sign pattern I j 1 among the set { +1 , -1 } of size 2 . Thus, in the worst-case, we have the following upper-bound

0

100

200

300

400

500

600

700

800

SV index

0.00

0.02

0.04

0.06

0.08

Normalized SVs

(a) MNIST-Linear scale

0

100

200

300

400

500

600

700

800

SV index

10 -5

10 -4

10 -3

10 -2

10 -1

Normalized SVs

(b) MNIST-Log scale

0

500

1000

1500

2000

2500

3000

SV index

0.00

0.02

0.04

0.06

0.08

0.10

Normalized SVs

(c) CIFAR-10-Linear scale

0

500

1000

1500

2000

2500

3000

SV index

10 -5

10 -4

10 -3

10 -2

10 -1

Normalized SVs

(d) CIFAR-10-Log scale

Notice that given fixed scalars m 1 and r , both P 1 and P 2 are polynomial terms with respect to the number of data samples n and the feature dimension d .

Remark 3. Notice that Convolutional Neural Networks (CNNs) operate on the patch matrices { X b } B b =1 instead of the full data matrix X , where X b ∈ R n × h and h denotes the filter size. Hence, even when the data matrix is full rank, i.e., r = min { n, d } , the number of hyperplane arrangements P 1 is upperbounded as P 1 ≤ O ( n r c ) , where r c := max b rank ( X b ) ≤ h glyph[lessmuch] min { n, d } (see [45] for details). For instance, let us consider a CNN with m 1 = 512 filters of size 3 × 3 , then r c ≤ 9 independent of data dimension n, d . As a consequence, weight sharing structure in CNNs dramatically limits the number of possible hyperplane arrangements. This also explains efficiency and remarkable generalization performance of CNNs in practice.

Training complexity analysis: Here, we calculate the computational complexity to globally solve the convex program in (8). Note that (8) is a convex optimization problem with 2 dm 1 MP 1 P 2 variables and 2 n ( m 1 +1) MP 1 P 2 constraints. Therefore, due to the upperbounds in (24) and (25) of Appendix A.8, the convex program (8) can be globally optimized by a standard interior-point solver with the computational complexity O ( d 3 m 3 1 2 3( m 1 +1) n 3( m 1 +1) r ) , which is a polynomial-time complexity in terms of n, d .

The analysis in this section can be recursively extended to arbitrarily deep parallel networks. First notice that if we apply the same approach to obtain an upperbound on P 3 , then due to the multiplicative pattern in (25), we obtain P 3 ≤ P ′ 3 (2 P 2 ) m 2 ≤ O ( n m 2 m 1 r ) . In a similar manner, the number of hyperplane arrangements in the l th layer is upperbounded as P l ≤ O ( n r ∏ l -1 j =1 m j ) , which is polynomial in both n and d for fixed data rank r and fixed layer widths { m j } l -1 j =1 .

A.11 Extension to vector outputs

In this section, we extend the analysis to parallel networks with multiple outputs where the label matrix is defined as Y ∈ R n × C provided that there exist C classes/outputs. Then the primal non-convex training problem is as follows

Applying Lemma 1 and each step in the proof of Theorem 1 yield d ∗ v := max V min θ ∈ Θ s -L ∗ ( V ) s.t. ∥ ∥ ∥ V T ( ( XW 1 k ) + . . . W ( L -1) k ) + ∥ ∥ ∥ F ≤ β, ∀ k ∈ [ K ] , where the corresponding Fenchel conjugate function is

Notice that above we have a dual matrix V instead of the dual vector in the scalar output case. More importantly, here, we have glyph[lscript] 2 norm in the dual constraint unlike the scalar output case with absolute value. Therefore, the vector-output case is slightly more challenging than the scalar output case and yields a different regularization function in the equivalent convex program.

The rest of the derivations directly follows the steps in Section A.4 and [62].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a theoretical analysis of optimization for path-regularized parallel ReLU networks.\nThey demonstrate how to represent this non-convex problem as a regularized convex problem.\nWhile convex problems in general admit polynomial time solutions, the size of this convex\nproblem is exponential in the rank $r$ of the data matrix. However, its size is polynomial\nin the data dimension $d$ and number of examples $n$ if those are treated as being independent\nof the rank $r$. The authors then establish approximation guarantees when operating on a\nrank $r$ approximation of a (potentially) full rank data matrix. For matrices with quickly\ndecaying singular values, this allows the convex problem to be effeciently solved at a decent accuracy.\nThey then run experiments on a toy task and image classification datasets to demonstrate that\ntheir method can outperform gradient descent and other optimizers in both final performance\nand time.\n\nSTRENGTHS:\n- Theoretical understanding of neural networks is an important and impactful topic.\n- Paper is well written and introduces novel analysis.\n- Emperical results support theory.\n\nWEAKNESSES:\n- Although the authors claim that their solution is polynomial time in both data dimension $d$ and number of examples $n$, it is exponential in the rank $r$ of the data matrix. While they prove that using a low rank approxiation will approximate the\noptimal solution well if the low rank approximation is good, their approximation bound will be poor if the data matrix is inherently high dimensional. So their solution still has exponential complexity in the \"implicit\" dimensionality of the inputs.\n\nTypos:\n- line 166: I see $RR$ in one of the expressions. Should this be $\\sqrt{m_1m_2}R$ instead?\n\nQUESTIONS:\n- How does the algorithm scale to larger problem sizes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper studies the problem of training parallel networks with three-layer subnetworks under path regularization. It is shown that the non-convex optimization problem of minimizing the regularized loss can be cast as a convex optimization problem, which can be solved efficiently.

STRENGTHS:
1. The study of parallel networks is interesting.
2. Finding efficient algorithms for training neural networks is important and the proposed solution seems novel and performs well.

WEAKNESSES:
1. In all the experiments the networks have $m_2=1$? what happened if $m_2$ is large, does it affect the training time?

QUESTIONS:
See "Weaknesses"

LIMITATIONS:
See "Weaknesses"
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows how parallel ReLU networks can be trained to approximate optimality in polynomial time by convex programming when employing a path-regularization (rather than weight decay or other regularization of the network weight parameters) and a low-rank approximation of the data matrix. Some numerical experiments demonstrate practical benefits of the approach.\n\n-- update: I have read and acknowledged all other reviews and the authors rebuttals, see discussion. --\n\nSTRENGTHS:\nPositive and practical results on optimal training of neural networks are quite rare, so I see this work as an interesting contribution. The numerical experiments demonstrate that the proposed approach can work in practice as well (however, it should be mentioned in the main paper, not only as an aside in the supplementary document (Sect. A.1, ll. 491ff), that the image classification experiments did not use all hyperplane arrangements for the convex program, but only a small sampled subset, thus deviating from the theoretical model to achieve practical speed-up).\n\nWEAKNESSES:\nI have read this paper (or a precursor preprint) a year or two ago, so the results aren't exactly new anymore, which may be a weakness. The techniques (to obtain a convex program solvable in poly-time) are very similar to earlier works on two-layer ReLU networks, which may make this work seem somewhat derivative; I don't think this is necessarily a weakness, but it may open a way to significantly shorten the very technical parts and referring to the earlier work (by Tolga&Pilanci) instead. In the (main) paper, Tables and Figures are not placed near where they are mentioned/discussed in the text body, which should be fixed as it breaks the flow or reading in a distracting way. Some statements are repetitive and should be consolidated accordingly (e.g., \"Remark 1\" essentially repeats what was mentioned at the end of the first paragraph of Sect. 2, and the sentence in lines 86-87 repeats Footnote 1). Throughout, there are various small typos and missing words (short ones like \"of\")\n\nFinally, it appears to have become common practice to submit papers to NeurIPS (and ICML) whose actual, main content is put in a separate \"Supplementary Material\" document whose length exceeds that of the supposed main paper. This paper is only a partial exception -- the main paper provides enough information and details to follow the ideas, but all (admittedly very technical) proofs can then be found in the very long Appendix (supplementary document) along with some further additional information. Thus, it may be considered a weakness of the paper to have such a long Appendix, because this format bears the danger of the formally most important parts of the work (proofs; algorithm details and specific setups) not being reviewed thoroughly due to the short review period and high review load of reviewers at these conferences. I cannot exclude myself from this -- I simply did not have the time to rigorously check all the details in the long supplementary document, and therefore cannot give a definitive answer regarding the proofs' correctness beyond \"believing\" everything appears to be well in order, especially since it is so reminiscent of the earlier work on two-layer ReLU networks, which I had read carefully at some point. In this regard, I cannot help but wonder if a full journal paper would not be the better way to publish results that simply do not fit into the 9-page limit.\n\nQUESTIONS:\nIs it correct that the entries in the last column of Table 2 are all identical? \nAlso, in Remark 2, l. 166, is it meant to say \"RR\" in the nominator?\n\nLIMITATIONS:\nLimitations have been adequately addresses, as far as I can tell.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study considers training the parallelized multi-layer neural networks with path-wise norm regularization. Through the duality argument, the authors reduce the regularized empirical minimization problem, which is highly non-convex, to the convex programming problem. They show that when the data matrix has a small rank, the solving time will only be polynomial time. In general, in cases where the data matrix does not shrinkage, an approximation method that also runs in polynomial time is proposed through the matrix approximation technique. Numerical experiments are conducted to show the effectiveness of the proposed regularization and the solving algorithm.\n\nSTRENGTHS:\n- The theoretical results seem to be solid and clearly written.\n- The reduction of the neural network training problem beyond two-layer to the convex programming is a novel, and the proposed algorithm seems to work effectively in the experiment section.\n\nWEAKNESSES:\nThe exhibited results are interesting from the theoretical view, but I think there are restrictive from the practical view.\n- Whereas the reduction to the convex programming is interesting, the obtained time complexity $O(d^3m_1^3m_2^32^{3(m_1+1)m_2}n^{3(m_1+1)r})$ will be extremely large in the practical models even in it is polynomial. Indeed, the numerical experiment seems to end with narrow networks.\n- Apart from the above concern, although the approximate algorithm is provided, obtaining the low-rank approximation of the data matrix requires much time when using a large dataset (especially to which deep learning is applied).\n\nQUESTIONS:\n- Do you consider that the time complexity obtained in Proposition 2 is just a theoretical one, and it will be faster in practice?\n- I have several concerns about the parameter dependence of the obtained time complexity in Proposition 2. First, does it not depend on the $K$, the number of parallelized networks? Second, the time complexity exhibited in Corollary 1 seems to be inconsistent with Proposition 2. When we apply Corollary 1 to the 3-layer networks, the term two will be powered by $3(m_1+m_2)$, but it becomes $3(m_1+1)m_2$ in Proposition 2. Which is correct?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This work studies the training of 3-layer parallel ReLU networks with path regularization. Specifically, the authors show that minimizing the non-convex learning objective with an \ell_2 pathwise weight decay is equivalent to a convex program that can be solved optimally in polynomial-time complexity (Proposition 1), except when the data matrix is full-rank. The authors also propose a more efficient polynomial-time algorithm (Theorem 2) for arbitrary data by using a local approximation of the data matrix on the training objective. The paper also provides numerous empirical results on toy and real data to support the theoretical results.

STRENGTHS:
+ This work provides a convex duality for parallel ReLU networks with path regularization.

+ An optimal, polynomial-time algorithm for arbitrary data, including full-rank data.

WEAKNESSES:
- This work is basically an extension of techniques and results in [17] to a slightly different regularization, so there is a concern about novelty.

- The performance of the proposed algorithm does not improve others on Fashion-MNIST, and why is the gain much higher in CIFAR-10, which is a more challenging dataset?

QUESTIONS:
1. The authors claim that “we proved the polynomial-time trainability of deep ReLU networks without requiring any impractical assumptions unlike [17, 29].” Can the authors compare and contrast the assumptions made in this and those works?

2. How practical is the algorithm in Theorem 2? I can’t tell from Proposition 1 what closed-form mapping is and how one constructs optimal network weights from it. Can the authors clarify?

3. In Table 1, the bounds are exactly the same between this work and [29] for 2-layer networks. Why?

4. How does low-rank approximation error populate to the optimality?

LIMITATIONS:
Above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-22