SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tolga Ergen
LG AI Research
tergen@lgresearch.ai
Mert Pilanci
pilanci@stanford.edu
Understanding the fundamental principles behind the success of deep neural networks is one of the most important open questions in the current literature. To this end, we study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape. We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases. We then show that pathwise regularized training problems can be represented as an exact convex optimization problem. We further prove that the equivalent convex problem is regularized via a group sparsity inducing norm. Thus, a path regularized parallel ReLU network can be viewed as a parsimonious convex model in high dimensions. More importantly, since the original training problem may not be trainable in polynomial-time, we propose an approximate algorithm with a fully polynomial-time complexity in all data dimensions. Then, we prove strong global optimality guarantees for this algorithm. We also provide experiments corroborating our theory.
Optimal 2-layer with WD
(a) 2-layer NN with WD
Figure 1: Decision boundaries of 2-layer and 3-layer ReLU networks that are globally optimized with weight decay (WD) and path regularization (PR). Here, our convex training approach in (c) successfully learns the underlying spiral pattern for each class while the previously studied convex models in (a) and (b) fail (see Appendix A.1 for details).
Optimal 3-layer with WD
(b) 3-layer NN with WD
Optimal 3-layer with PR
(c) 3-layer NN with PR (Ours)
Deep Neural Networks (DNNs) have achieved substantial improvements in several fields of machine learning. However, since DNNs have a highly nonlinear and non-convex structure, the fundamental principles behind their remarkable performance is still an open problem. Therefore, advances in this field largely depend on heuristic approaches. One of the most prominent techniques to boost the generalization performance of DNNs is regularizing layer weights so that the network can fit a function that performs well on unseen test data. Even though weight decay, i.e., penalizing the  $\ell_2^2$ -norm of the layer weights, is commonly employed as a regularization technique in practice, recently,
Table 1: Complexity comparison with prior works (n: # of data samples, d: feature dimension,  $m_{l}$ : # of hidden neurons in layer l,  $\epsilon$ : approximation accuracy, r: rank of the data,  $\kappa \ll r$ : chosen according to (10) such that  $\epsilon$  accuracy is achieved)
Loss function	2-layer complexity	L-layer complexity	Globally optimal
[30]	Convex loss	O(2m1ndm1poly(n,d,m1))	-	✓(Brute-force)
[31]	l2-loss	2O(m5/ε2)poly(n,d)	-	X(NP-hard)
[32]	lploss	O(2m1ndm1poly(n,d,m1))	-	✓(Brute-force)
[6]	Convex loss	O(nrpoaly(d,r))	-	✓(Convex, exact)
Ours	Convex loss	O(nrpoaly(d,r))	O(nrΠj=1L-2mjpoly(d,r,Πj=1L-2ml)	✓(Convex, exact)
Ours	Convex loss	O(nkpolly(d,k))	O(nkΠj=1L-2mjpoly(d,k,Πj=1L-2ml)	✓(Convex, ε-opt)
it has been shown that  $\ell_2$ -path regularizer [1], i.e., the sum over all paths in the network of the squared product over all weights in the path, achieves further empirical gains [2]. Additionally, recent studies have elucidated that convex optimization reveals surprising generalization and interpretability properties for Deep Neural Networks (DNNs). [3-6] Therefore, in this paper, we investigate the underlying mechanisms behind path regularized DNNs through the lens of convex optimization.
Figure 2: (Left): Parallel ReLU network in (1) with  $K$  sub-networks and three layers ( $L = 3$ ) (Right): Path regularization for a three-layer network.
2 Parallel Neural Networks
Although DNNs are highly complex architectures due to the composition of multiple nonlinear functions, their parameters are often trained via simple first order gradient based algorithms, e.g., Gradient Descent (GD) and variants. However, since such algorithms only rely on local gradient of the objective function, they may fail to globally optimize the objective in certain cases [7, 8]. Similarly, [9, 10] showed that these pathological cases also apply to stochastic algorithms such as Stochastic GD (SGD). They further show that some of these issues can be avoided by increasing the number of trainable parameters, i.e., operating in an overparameterized regime. However, [11] reported the existence of more complicated cases, where SGD/GD usually fails. Therefore, training DNNs to global optimality remains a challenging optimization problem [12-14].
To circumvent difficulties in training, recent studies focused on models that benefit from overparameterization [15-18]. As an example, [19-23] considered a new architecture by combining multiple standard NNs, termed as sub-networks, in parallel. Evidences in [20-25] showed that this way of combining NNs yields an optimization landscape that has fewer local minima and/or saddle points so that SGD/GD generally converges to a global minimum. Therefore, many recently proposed NN-based architectures that achieve state-of-the-art performance in practice, e.g., SqueezeNet [26], Inception [27], Xception [28], and ResNext [29], are in this form.
Notation and preliminaries: Throughout the paper, we denote matrices and vectors as uppercase and lowercase bold letters, respectively. For vectors and matrices, we use subscripts to denote a certain column/element. As an example,  $w_{lkj_{l-1}jl}$  denotes the  $j_{l-1}j_l^{th}$  entry of the matrix  $\mathbf{W}_{lk}$ . We use  $\mathbf{I}_k$  and  $\mathbf{0}$  (or  $\mathbf{1}$ ) to denote the identity matrix of size  $k \times k$  and a vector/matrix of zeros (or ones) with appropriate sizes. We use  $[n]$  for the set of integers from 1 to  $n$ . We use  $\| \cdot \|_2$  and  $\| \cdot \|_F$  to represent the Euclidean and Frobenius norms, respectively. Additionally, we denote the unit  $\ell_p$  ball as  $\mathcal{B}_p := \{\mathbf{u} \in \mathbb{R}^d : \| \mathbf{u} \|_p \leq 1\}$ . We also use  $\mathbb{1}[x \geq 0]$  and  $(x)_+ = \max\{x, 0\}$  to denote the 0-1 valued indicator and ReLU, respectively.
In this paper, we particularly consider a parallel ReLU network with  $K$  sub-networks and each sub-network is an  $L$ -layer ReLU network (see Figure 2) with layer weights  $\mathbf{W}_{lk} \in \mathbb{R}^{m_{l-1} \times m_l}$ ,  $\forall l \in [L-1]$  and  $\mathbf{w}_{Lk} \in \mathbb{R}^{m_{L-1}}$ , where  $m_0 = d$ ,  $m_L = 1^1$ , and  $m_l$  denotes the number of neurons in the  $l^{th}$  hidden layer. Then, given a data matrix  $\mathbf{X} \in \mathbb{R}^{n \times d}$ , the output of the network is as follows
$$
f _ {\theta} (\mathbf {X}) := \sum_ {k = 1} ^ {K} \left(\left(\mathbf {X W} _ {1 k}\right) _ {+} \dots \left. \mathbf {W} _ {(L - 1) k}\right) _ {+} \mathbf {w} _ {L k}, \right. \tag {1}
$$
where we compactly denote the parameters as  $\theta := \bigcup_{k} \{\mathbf{W}_{lk}\}_{l=1}^{L}$  with the parameter space as  $\Theta$  and each sub-network represents a standard deep ReLU network.
Remark 1. Most commonly used neural networks in practice can be classified as special cases of parallel networks, e.g., standard NNs and ResNets [33] see Appendix A.2 for details.
2.1 Our Contributions
- We prove that training the path regularized parallel ReLU networks (1) is equivalent to a convex optimization problem that can be approximately solved in polynomial-time by standard convex solvers (see Table 1). Therefore, we generalize the two-layer results in [6] to multiple nonlinear layer without any strong assumptions in contrast to [21] and a much broader class of NN architectures including ResNets.
- As already observed by [6, 21], regularized deep ReLU network training problems require exponential-time complexity when the data matrix is full rank, which is unavoidable. However, in this paper, we develop an approximate training algorithm which has fully polynomial-time complexity in all data dimensions and prove global optimality guarantees in Theorem 2. To the best of our knowledge, this is the first convex optimization based and polynomial-time complexity (in data dimensions) training algorithm for ReLU networks with global approximation guarantees.
- We show that the equivalent convex problem is regularized by a group norm regularization where grouping effect is among the sub-networks. Therefore the equivalent convex formulation reveals an implicit regularization that promotes group sparsity among sub-networks and generalizes prior works on linear networks such as [34] to ReLU networks.
- We derive a closed-form mapping between the parameters of the non-convex parallel ReLU networks and its convex equivalent in Proposition 1. Therefore, instead of solving the challenging non-convex problem, one can globally solve the equivalent convex problem and then construct an optimal solution to the original non-convex network architecture via our closed-form mapping.
2.2 Overview of Our Results
Given data  $\mathbf{X} \in \mathbb{R}^{n \times d}$  and labels  $\mathbf{y} \in \mathbb{R}^n$ , we consider the following regularized training problem
$$
p _ {L} ^ {*} := \min  _ {\theta \in \Theta} \mathcal {L} \left(f _ {\theta} (\mathbf {X}), \mathbf {y}\right) + \beta \mathcal {R} (\theta), \tag {2}
$$
where  $\Theta := \{\theta \in \Theta : \mathbf{W}_{lk} \in \mathbb{R}^{m_{l-1} \times m_l}, \forall l \in [L], \forall k \in [K]\}$  is the parameter space,  $\mathcal{L}(\cdot, \cdot)$  is an arbitrary convex loss function,  $\mathcal{R}(\cdot)$  represents the regularization on the network weights, and  $\beta > 0$  is the regularization coefficient.
For the rest of the paper, we focus on a scalar output regression/classification framework with arbitrary loss functions, e.g., squared loss, cross entropy or hinge loss. We also note that our derivations
<sup>1</sup>We analyze scalar outputs, however, our derivations extend to vector outputs as shown in Appendix A.11.
can be straightforwardly extended to vector outputs networks as proven in Appendix A.11. More importantly, we use  $\ell_2$ -path regularizer studied in [1, 2], which is defined as
$$
\mathcal {R} (\theta) := \sum_ {k = 1} ^ {K} \sqrt {\sum_ {j _ {1} , j _ {2} , \dots , j _ {L}} \left(\| \mathbf {w} _ {1 k j _ {1}} \| _ {2} ^ {2} \prod_ {l = 2} ^ {L} w _ {l k j _ {l - 1} j _ {l}} ^ {2}\right)},
$$
where  $w_{lkj_{l-1}j_l}$  is the  $j_{l-1}j_l^{th}$  entry of  $\mathbf{W}_{lk}$ . The above regularizer sums the square of all the parameters along each possible path from input to output of each sub-network  $k$  (see Figure 2) and then take the squared root of the summation. Therefore, we penalize each path in each sub-network and then group them based on the sub-network index  $k$ .
We now propose a scaling to show that (2) is equivalent to a group  $\ell_1$  regularized problem.
Lemma 1. The following problems are equivalent:
$$
\min  _ {\theta \in \Theta} \mathcal {L} (f _ {\theta} (\mathbf {X}), \mathbf {y}) + \beta \mathcal {R} (\theta) = \min  _ {\theta \in \Theta_ {s}} \mathcal {L} (f _ {\theta} (\mathbf {X}), \mathbf {y}) + \beta \sum_ {k = 1} ^ {K} \| \mathbf {w} _ {L k} \| _ {2},
$$
where  $\mathbf{w}_{Lk} \in \mathbb{R}^{m_{L-1}}$  are the last layer weights of each sub-network  $k$ , and  $\Theta_s := \{\theta \in \Theta : \sum_{j_1, j_2, \ldots, j_{L-2}} \left( \| \mathbf{w}_{1kj_1} \|_2^2 \prod_{l=2}^{L-1} w_{lkj_{l-1}j_l}^2 \right) \leq 1, \forall j_{L-1}, \forall k \in [K]\}$  denotes the parameter space after rescaling.
The advantage of the form in Lemma 1 is that we can derive a dual problem with respect to the output layer weights  $\mathbf{w}_{Lk}$  and then characterize the optimal layer weights via optimality conditions and the prior works on  $\ell_1$  regularization in infinite dimensional spaces [35]. Thus, we first apply the rescaling in Lemma 1 and then take the dual with respect to the output weights  $\mathbf{w}_{Lk}$ . To characterize the hidden layer weights, we then change the order of minimization for the hidden layer weights and the maximization for the dual parameter to get the following dual problem<sup>3</sup>
$$
p _ {L} ^ {*} \geq d _ {L} ^ {*} := \max  _ {\mathbf {v}} - \mathcal {L} ^ {*} (\mathbf {v}) \quad \text {s . t .} \max  _ {\theta \in \Theta_ {s}} \left\| \mathbf {v} ^ {T} \left(\left(\mathbf {X} \mathbf {W} _ {1}\right) _ {+} \dots \mathbf {W} _ {(L - 1)}\right) _ {+} \right\| _ {2} \leq \beta , \tag {3}
$$
where  $\mathcal{L}^*$  is the Fenchel conjugate function of  $\mathcal{L}$ , which is defined as [36]
$$
\mathcal {L} ^ {*} (\mathbf {v}) := \max  _ {\mathbf {z}} \mathbf {z} ^ {T} \mathbf {v} - \mathcal {L} \left(\mathbf {z}, \mathbf {y}\right).
$$
The dual problem in (3) is critical for our derivations since it provides us with an analytic perspective to characterize a set of optimal hidden layer weights for the non-convex neural network in (1). To do so, we first show that strong duality holds for the non-convex training problem in Lemma 1, i.e.,  $p_{L}^{*} = d_{L}^{*}$ . Then, based on the exact dual problem in (3), we propose an equivalent analytic description for the optimal hidden layer weights via the KKT conditions.
We note that strong duality for two-layer ReLU networks has already been proved by previous studies [6, 19, 21, 22, 37, 38], however, this is the first work providing an exact characterization for path regularized deep ReLU networks via convex duality.
3 Parallel networks with three layers
Here, we consider a three-layer parallel network with  $K$  sub-networks, which is a special case of (1) when  $L = 3$ . Thus, we have the following training problem
$$
p _ {3} ^ {*} = \min  _ {\theta \in \Theta} \mathcal {L} \left(f _ {\theta} (\mathbf {X}), \mathbf {y}\right) + \beta \sum_ {k = 1} ^ {K} \sqrt {\sum_ {j _ {1} , j _ {2}} \left\| \mathbf {w} _ {1 k j _ {1}} \right\| _ {2} ^ {2} w _ {2 k j _ {1} j _ {2}} ^ {2} w _ {3 k j _ {2}} ^ {2}}. \tag {4}
$$
where  $\Theta = \left\{\left(\mathbf{W}_{1k},\mathbf{W}_{2k},\mathbf{w}_{3k}\right):\mathbf{W}_{1k}\in \mathbb{R}^{d\times m_1},\mathbf{W}_{2k}\in \mathbb{R}^{m_1\times m_2},\mathbf{w}_{3k}\in \mathbb{R}^{m_2}\right\}$ . By Lemma 1,
$$
p _ {3} ^ {*} = \min  _ {\theta \in \Theta_ {s}} \mathcal {L} \left(f _ {\theta} (\mathbf {X}), \mathbf {y}\right) + \beta \sum_ {k = 1} ^ {K} \| \mathbf {w} _ {3 k} \| _ {2}. \tag {5}
$$
2All the proofs are presented in the supplementary file.
<sup>3</sup>We present the details in Appendix A.7.
$$
\mathbf {D} _ {1} = \left[ \begin{array}{c c c} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right]
$$
$$
\mathbf {D} _ {2} = \left[ \begin{array}{c c c} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{array} \right]
$$
$$
\mathbf {D} _ {3} = \left[ \begin{array}{c c c} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{array} \right]
$$
$$
\mathbf {D} _ {4} = \left[ \begin{array}{c c c} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{array} \right]
$$
Figure 3: Illustration of possible hyperplane arrangements that determine the diagonal matrices  $\mathbf{D}_i$ . Here, we have three samples in two dimensions and we want to separate these samples with a linear classifier.  $\mathbf{D}_i$  basically encodes information regarding which side of the linear classifier samples lie.
Then, taking the dual of (5) with respect to the output layer weights  $\mathbf{w}_{3k} \in \mathbb{R}^{m_2}$  and then changing the order of the minimization for  $\{\mathbf{W}_{1k}, \mathbf{W}_{2k}\}$  and the maximization for the dual variable  $\mathbf{v}$  yields
$$
p _ {3} ^ {*} \geq d _ {3} ^ {*} := \max  _ {\mathbf {v}} - \mathcal {L} ^ {*} (\mathbf {v}) \quad \text {s . t .} \max  _ {\theta \in \Theta_ {s}} \left\| \mathbf {v} ^ {T} \left(\left(\mathbf {X} \mathbf {W} _ {1}\right) _ {+} \mathbf {W} _ {2}\right) _ {+} \right\| _ {2} \leq \beta . \tag {6}
$$
Here, we remark that (4) is non-convex with respect to the layer weights, we may have a duality gap, i.e.,  $p_3^* \geq d_3^*$ . Therefore, we first show that strong duality holds in this case, i.e.,  $p_3^* = d_3^*$  as detailed in Appendix A.4. We then introduce an equivalent representation for the ReLU activation as follows.
Since ReLU masks the negative entries of inputs, we have the following equivalent representation
$$
\begin{array}{l} \left(\left(\mathbf {X} \mathbf {W} _ {1}\right) _ {+} \mathbf {w} _ {2}\right) _ {+} = \left(\sum_ {j _ {1} = 1} ^ {m _ {1}} \left(\mathbf {X} \mathbf {w} _ {1 j _ {1}}\right) _ {+} w _ {2 j _ {1}}\right) _ {+} = \left(\sum_ {j _ {1} = 1} ^ {m _ {1}} \left(\mathbf {X} \bar {\mathbf {w}} _ {j _ {1}}\right) _ {+} \mathcal {I} _ {j _ {1}}\right) _ {+} \\ = \mathbf {D} _ {2} \sum_ {j _ {1} = 1} ^ {m _ {1}} \mathcal {I} _ {j _ {1}} \mathbf {D} _ {1 j _ {1}} \mathbf {X} \bar {\mathbf {w}} _ {j _ {1}}, \tag {7} \\ \end{array}
$$
where  $\bar{\mathbf{w}}_{j_1} = |w_{2j_1}|\mathbf{w}_{1j_1},\mathcal{I}_{j_1} = \mathrm{sign}(w_{2j_1})\in \{-1, + 1\}$ , and we use the following alternative representation for ReLU (see Figure 3 for a two dimensional visualization)
$$
(\mathbf {X} \mathbf {w}) _ {+} = \mathbf {D X w} \Longleftrightarrow \begin{array}{l} \mathbf {D X w} \geq 0 \\ (\mathbf {I} _ {n} - \mathbf {D}) \mathbf {X w} \leq 0 \end{array} \Longleftrightarrow (2 \mathbf {D} - \mathbf {I} _ {n}) \mathbf {X w} \geq 0,
$$
where  $\mathbf{D} \in \mathbb{R}^{n \times n}$  is a diagonal matrix of zeros/ones, i.e.,  $\mathbf{D}_{ii} \in \{0,1\}$ . Therefore, we first enumerate all possible signs and diagonal matrices for the ReLU layers and denote them as  $\mathcal{I}_{j_1}$ ,  $\mathbf{D}_{1ij_1}$  and  $\mathbf{D}_{2l}$  respectively, where  $j_1 \in [m_1], i \in [P_1], l \in [P_2]$ . Here,  $\mathbf{D}_{1ij_1}$  and  $\mathbf{D}_{2l}$  denotes the masking/diagonal matrices for the first and second ReLU layers, respectively and  $P_1$  and  $P_2$  are the number diagonal matrices in each layer as detailed in Section A.10. Then, we convert the non-convex constraints in (6) to convex constraints using  $\mathbf{D}_{1ij_1}$ ,  $\mathbf{D}_{2l}$  and  $\mathcal{I}_{j_1}$ .
Using the representation in (7), we then take the dual of (6) to obtain the convex bidual of the primal problem (4) as detailed in the theorem below.
Theorem 1. The non-convex training problem in (4) can be cast as the following convex program
$$
\min  _ {\mathbf {z}, \mathbf {z} ^ {\prime} \in \mathcal {C}} \mathcal {L} (\tilde {\mathbf {X}} (\mathbf {z} - \mathbf {z} ^ {\prime}), \mathbf {y}) + \frac {\beta}{\sqrt {m _ {2}}} \left(\| \mathbf {z} \| _ {F, 1} + \| \mathbf {z} ^ {\prime} \| _ {F, 1}\right), \tag {8}
$$
where  $\| \cdot \|_{F,1}$  denotes a  $d\times m_1$  dimensional group Frobenius norm operator such that given a vector  $\mathbf{u}\in \mathbb{R}^{dm_1P}$ ,  $\| \mathbf{u}\|_{F,1}\coloneqq \sum_{i = 1}^{P}\| \mathbf{U}_i\| _F$ , where  $\mathbf{U}_i\in \mathbb{R}^{d\times m_1}$  are reshaped partitions of  $\mathbf{u}$ . Moreover, the convex set  $\mathcal{C}$  is defined as
$$
\mathcal {C} := \left\{\mathbf {z}: \mathbf {z} _ {i j l} ^ {s} \in \mathcal {C} _ {i l} ^ {s}, \forall i \in [ P _ {1} ], l \in [ P _ {2} ], s \in [ M ] \right\}
$$
$$
\mathcal {C} _ {i l} ^ {s} := \left\{\left\{\mathbf {w} _ {j _ {1}} \right\} _ {j _ {1}}: (2 \mathbf {D} _ {2 l} - \mathbf {I} _ {n}) \sum_ {j _ {1} = 1} ^ {m _ {1}} \mathcal {I} _ {i j _ {1} l} ^ {s} \mathbf {D} _ {1 i j _ {1}} \mathbf {X} \mathbf {w} _ {j _ {1}} \geq 0, (2 \mathbf {D} _ {1 i j _ {1}} - \mathbf {I} _ {n}) \mathbf {X} \mathbf {w} _ {j _ {1}} \geq 0, \forall j _ {1} \in [ m _ {1} ] \right\}
$$
where  $\mathcal{I}_{ij_1l}^s\in \{+1, - 1\}$ $M = 2^{m_1}$  , and  $\mathbf{z},\mathbf{z}'\in \mathbb{R}^{dm_1MP_1P_2}$  are constructed by stacking  $\mathbf{z}_{ij_1l}^s,\mathbf{z}_{ij_1l}^{s'}\in \mathbb{R}^d$ $\forall i\in [P_1],l\in [P_2],j_1\in [m_1],s\in [M]$  , respectively. Also, the effective data matrix  $\tilde{\mathbf{X}}\in \mathbb{R}^{n\times dm_1MP_1P_2}$  is defined as  $\tilde{\mathbf{X}}\coloneqq \mathbf{I}_M\otimes \tilde{\mathbf{X}}_s$  and
$$
\tilde {\mathbf {X}} _ {s} := \left[ \mathbf {D} _ {2 1} \mathbf {D} _ {1 1 1} \mathbf {X} \dots \mathbf {D} _ {2 l} \mathbf {D} _ {1 i j _ {1}} \mathbf {X} \dots \mathbf {D} _ {2 P _ {2}} \mathbf {D} _ {1 P _ {1} m _ {1}} \mathbf {X} \right].
$$
We next derive a mapping between the convex program (8) and the non-convex architecture (4).
Proposition 1. An optimal solution to the non-convex parallel network training problem in (4), denoted as  $\{\mathbf{W}_{1k}^{*},\mathbf{w}_{2k}^{*},w_{3k}^{*}\}_{k = 1}^{K}$ , can be recovered from an optimal solution to the convex program in (8), i.e.,  $\{\mathbf{z}^*,\mathbf{z}'^*\}$  via a closed-form mapping. Therefore, we prove a mapping between the parameters of the parallel network in Figure 2 and its convex equivalent.
Next, we prove that the convex program in (8) can be globally optimized with a polynomial-time complexity given  $\mathbf{X}$  has fixed rank, i.e.,  $\mathrm{rank}(\mathbf{X}) = r < \min\{n, d\}$ .
Proposition 2. Given a data matrix such that  $\mathrm{rank}(\mathbf{X}) = r < \min\{n, d\}$ , the convex program in (8) can be globally optimized via standard convex solvers with  $\mathcal{O}(d^3 m_1^3 m_2^3 2^{3(m_1 + 1)m_2} n^{3(m_1 + 1)r})$  complexity, which is a polynomial-time complexity in terms of  $n, d$ . Note that here globally optimizing the training objective means to achieve the exact global minimum up to any arbitrary machine precision or solver tolerance.
Below, we show that the complexity analysis in Proposition 2 extends to arbitrarily deep networks.
Corollary 1. The same analysis can be readily applied to arbitrarily deep networks. Therefore, given  $\mathrm{rank}(\mathbf{X}) = r < \min\{n, d\}$ , we prove that  $L$ -layer architectures can be globally optimized with  $\mathcal{O}\left(d^{3}\left(\prod_{j=1}^{L-2} m_{j}^{3}\right) 2^{3 \sum_{j=1}^{L-1} m_{j}} n^{3r\left(1 + \sum_{l=1}^{L-2} \prod_{j=1}^{l} m_{j}\right)}\right)$ , which is polynomial in  $n, d$ .
(a) Objective value  $(\downarrow)$
(b) # of hyperplane arrangements  $(P)$
Figure 4: Verification of Theorem 2 and Remark 2. We train a parallel network using the convex program in Theorem 1 with  $\ell_2$  loss on a toy dataset with  $n = 15$ ,  $\beta = 0.1$ ,  $m_2 = 1$ , and the low-rank approximation  $r = \lfloor \frac{d}{2} \rfloor$ . To obtain a low-rank model, we first sample a data matrix from a standard normal Gaussian distribution and then set  $\sigma_{r+1} = \ldots = \sigma_d = 1$ . Please see the subsection for low rank models in Section 4 for more details.
3.1 Polynomial-time training for arbitrary data
Based on Corollary 1, exponential complexity is unavoidable for deep networks when the data matrix is full rank, i.e.,  $\mathrm{rank}(\mathbf{X}) = \min \{n,d\}$ . Thus, we propose a low rank approximation to the model in (4). We first denote the rank- $r$  approximation of  $\mathbf{X}$  as  $\hat{\mathbf{X}}_r$  such that  $\| \mathbf{X} - \hat{\mathbf{X}}_r\| _2\leq \sigma_{r + 1}$ , where  $\sigma_{r + 1}$  represents the  $(r + 1)^{th}$  largest singular value of  $\mathbf{X}$ . Then, we have the following result.
Theorem 2. Given an  $R$ -Lipschitz convex loss function  $\mathcal{L}(\cdot, \mathbf{y})$ , the regularized training problem
$$
p _ {3} ^ {*} = \min  _ {\theta \in \Theta} \mathcal {L} \left(f _ {\theta} (\mathbf {X}), \mathbf {y}\right) + \beta \sum_ {k = 1} ^ {K} \sqrt {\sum_ {j _ {1} , j _ {2}} \left\| \mathbf {w} _ {1 k j} \right\| _ {2} ^ {2} w _ {2 k j _ {1} j _ {2}} ^ {2} w _ {3 k j _ {2}} ^ {2}}, \tag {9}
$$
Table 2: Training objective of a three-layer parallel network trained with non-convex SGD (5 independent initialization trials) on a toy dataset with  $(n,d,m_1,m_2,\beta, \text{batch size}) = (5,2,3,1,0.002,5)$ , where the convex program in (8) are solved via the interior point solvers in CVX/CVXPY.
Method	Non-convex SGD	Convex(Ours)
Run #1	Run #2	Run #3	Run #4	Run #5
K = 5 Training objective Time(s)	0.0221	0.0239	0.0031	0.0027	0.0027	0.0007
11.62	11.62	11.62	11.62	11.62	4.947
K = 20 Training objective Time(s)	0.0010	0.0027	0.0009	0.0010	0.0027	0.0007
44.37	44.37	44.37	11.55	44.37	4.947
K = 40 Training objective Time(s)	0.0008	0.0009	0.0009	0.0009	0.0008	0.0007
91.87	91.87	91.87	91.87	91.87	4.947
can be solved using the data matrix  $\hat{\mathbf{X}}_r$  to achieve the following optimality guarantee