SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \stackrel {(6)} {=} \sqrt {m _ {1}} R \sigma_ {r + 1} \sum_ {k = 1} ^ {K} \| \hat {\mathbf {w}} _ {3 k} \| _ {1} + \mathcal {L} \left(\sum_ {k = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {m _ {2}} \left(\left(\hat {\mathbf {X}} _ {r} \hat {\mathbf {W}} _ {1 k}\right) _ {+} \hat {\mathbf {w}} _ {2 k j _ {2}}\right) _ {+} \hat {w} _ {3 k j _ {2}}, \mathbf {y}\right) \\ \leq \sqrt {m _ {1} m _ {2}} R \sigma_ {r + 1} \sum_ {k = 1} ^ {K} \| \hat {\mathbf {w}} _ {3 k} \| _ {2} + \mathcal {L} \left(\sum_ {k = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {m _ {2}} \left(\left(\hat {\mathbf {X}} _ {r} \hat {\mathbf {W}} _ {1 k}\right) _ {+} \hat {\mathbf {w}} _ {2 k j _ {2}}\right) _ {+} \hat {w} _ {3 k j _ {2}}, \mathbf {y}\right), \\ \end{array}
$$
where we use the convexity and  $R$ -Lipschitz property of the loss function, convexity of  $\ell_{2}$ -norm, 1-Lipschitz property of the ReLU activation,  $\| \mathbf{x} \|_1 \leq \sqrt{m} \| \mathbf{x} \|_2$  for  $\mathbf{x} \in \mathbb{R}^m$ , and  $\max_k \sum_{j_1 = 1}^{m_1} \| \hat{\mathbf{w}}_{1kj_1} \|_2^2 \hat{w}_{2kj_1j_2}^2 \leq 1$  from the rescaling in Lemma 1 for (1), (2), (3), (4), (5), and (6) respectively.
A.7 Proof for the dual problem in (3)
In order to prove the dual problem, we directly utilize Fenchel duality [36]. Let us first rewrite the primal regularized training problem after the application of the rescaling in Lemma 1 as follows
$$
p _ {L} ^ {*} = \min  _ {\hat {\mathbf {y}} \in \mathbb {R} ^ {n}, \theta \in \Theta_ {s}} \mathcal {L} (\hat {\mathbf {y}}, \mathbf {y}) + \beta \sum_ {k = 1} ^ {K} \| \mathbf {w} _ {L k} \| _ {2} \text {s . t .} \hat {\mathbf {y}} = \sum_ {k = 1} ^ {K} \left(\left(\mathbf {X} \mathbf {W} _ {1 k}\right) _ {+} \dots \left. \mathbf {W} _ {(L - 1) k}\right) _ {+} \mathbf {w} _ {L k}\right). \tag {23}
$$
The corresponding Lagrangian can be computed by incorporating the constraints into objective via a dual variable as follows
$$
L (\mathbf {v}, \hat {\mathbf {y}}, \mathbf {w} _ {L k}) = \mathcal {L} (\hat {\mathbf {y}}, \mathbf {y}) - \mathbf {v} ^ {T} \hat {\mathbf {y}} + \mathbf {v} ^ {T} \sum_ {k = 1} ^ {K} \left((\mathbf {X} \mathbf {W} _ {1 k}) _ {+} \dots \mathbf {W} _ {(L - 1) k}\right) _ {+} \mathbf {w} _ {L k} + \beta \sum_ {k = 1} ^ {K} \| \mathbf {w} _ {L k} \| _ {2}.
$$
We then define the following dual function
$$
\begin{array}{l} g (\mathbf {v}) = \min  _ {\hat {\mathbf {y}}, \mathbf {w} _ {L k}} L (\mathbf {v}, \hat {\mathbf {y}}, \mathbf {w} _ {L k}) \\ = \min  _ {\hat {\mathbf {y}}, \mathbf {w} _ {L k}} \mathcal {L} (\hat {\mathbf {y}}, \mathbf {y}) - \mathbf {v} ^ {T} \hat {\mathbf {y}} + \mathbf {v} ^ {T} \sum_ {k = 1} ^ {K} \left(\left(\mathbf {X} \mathbf {W} _ {1 k}\right) _ {+} \dots \left. \mathbf {W} _ {(L - 1) k}\right) _ {+} \mathbf {w} _ {L k} + \beta \sum_ {k = 1} ^ {K} \| \mathbf {w} _ {L k} \| _ {2} \right. \\ = - \mathcal {L} ^ {*} (\mathbf {v}) \text {s . t .} \left\| \mathbf {v} ^ {T} \left(\left(\mathbf {X} \mathbf {W} _ {1 k}\right) _ {+} \dots \mathbf {W} _ {(L - 1) k}\right) _ {+} \right\| _ {2} \leq \beta , \forall k \in [ K ], \\ \end{array}
$$
where  $\mathcal{L}^*$  is the Fenchel conjugate function defined as [36]
$$
\mathcal {L} ^ {*} (\mathbf {v}) := \max _ {\mathbf {z}} \mathbf {z} ^ {T} \mathbf {v} - \mathcal {L} (\mathbf {z}, \mathbf {y}).
$$
Hence, we write the dual problem of (23) as
$$
p_{L}^{*} = \min_{\theta \in \Theta_{s}}\max_{\mathbf{v}}g(\mathbf{v}) = \min_{\theta \in \Theta_{s}}\max_{\mathbf{v}} - \mathcal{L}^{*}(\mathbf{v})  \text{s.t.}\left\| \mathbf{v}^{T}\left((\mathbf{X}\mathbf{W}_{1k})_{+}\dots \mathbf{W}_{(L - 1)k}\right)_{+}\right\|_{2}\leq \beta ,\forall k\in [K].
$$
However, since the hidden layer weights are the variables of the outer minimization, we cannot directly characterize the optimal hidden layer weight in the form above. Thus, as the last step of the derivation, we change the order of the minimization over  $\theta$  and the maximization over  $\mathbf{v}$  to obtain the following lower bound
$$
\begin{array}{l} p _ {L} ^ {*} \geq d _ {L} ^ {*} = \max  _ {\mathbf {v}} \min  _ {\theta \in \Theta_ {s}} - \mathcal {L} ^ {*} (\mathbf {v}) \text {s . t .} \left\| \mathbf {v} ^ {T} \left(\left(\mathbf {X} \mathbf {W} _ {1 k}\right) _ {+} \dots \mathbf {W} _ {(L - 1) k}\right) _ {+} \right\| _ {2} \leq \beta , \forall k \in [ K ] \\ = \max  _ {\mathbf {v}} - \mathcal {L} ^ {*} (\mathbf {v}) \text {s . t .} \max  _ {\theta \in \Theta_ {s}} \left\| \mathbf {v} ^ {T} \left(\left(\mathbf {X} \mathbf {W} _ {1}\right) _ {+} \dots \mathbf {W} _ {(L - 1)}\right) _ {+} \right\| _ {2} \leq \beta . \\ \end{array}
$$
A.8 Hyperplane arrangements
Here, we review the notion of hyperplane arrangements detailed in [6].
We first define the set of all hyperplane arrangements for the data matrix  $\mathbf{X}$  as
$$
\mathcal {H} := \bigcup \left\{\left\{\operatorname {s i g n} (\mathbf {X w}) \right\}: \mathbf {w} \in \mathbb {R} ^ {d} \right\},
$$
where  $|\mathcal{H}| \leq 2^n$ . The set  $\mathcal{H}$  all possible  $\{+1, -1\}$  labelings of the data samples  $\{\mathbf{x}_i\}_{i=1}^n$  via a linear classifier  $\mathbf{w} \in \mathbb{R}^d$ . We now define a new set to denote the indices with positive signs for each element in the set  $\mathcal{H}$  as  $S := \left\{ \left\{ \cup_{h_i = 1} \{i\} : \mathbf{h} \in \mathcal{H} \right\} \right\}$ . With this definition, we note that given an element  $S \in S$ , one can introduce a diagonal matrix  $\mathbf{D}(S) \in \mathbb{R}^{n \times n}$  defined as
$$
\mathbf {D} (S) _ {i i} := \left\{ \begin{array}{l l} 1 & \text {i f} i \in S \\ 0 & \text {o t h e r w i s e} \end{array} \right..
$$
$\mathbf{D}(S)$  can be also viewed as a diagonal matrix of indicators, where each diagonal entry is one if the corresponding sample labeled as  $+1$  by the linear classifier  $\mathbf{w}$ , zero otherwise. Therefore, the output of ReLU activation can be equivalently written as  $(\mathbf{Xw})_{+} = \mathbf{D}(S)\mathbf{Xw}$  provided that  $\mathbf{D}(S)\mathbf{Xw} \geq 0$  and  $(\mathbf{I}_n - \mathbf{D}(S))\mathbf{Xw} \leq 0$  are satisfied. One can define more compactly these two constraints as  $(2\mathbf{D}(S) - \mathbf{I}_n)\mathbf{Xw} \geq 0$ . We now denote the cardinality of  $S$  as  $P$ , and obtain the following upperbound
$$
P \leq 2 \sum_ {k = 0} ^ {r - 1} \binom {n - 1} {k} \leq 2 r \left(\frac {e (n - 1)}{r}\right) ^ {r}
$$
where  $r\coloneqq \mathrm{rank}(\mathbf{X})\leq \min (n,d)$  [58-61].
A.9 Low rank model in Theorem 2
In Section 3.1, we propose an  $\epsilon$ -approximate training approach that has polynomial-time complexity even when the data matrix is full rank. Here, you can select the rank  $r$  by plugging in the desired approximation error and network structure in equation 10. We show that the approximation error proved in Theorem 2 can be arbitrarily small for practically relevant problems. As an example, consider a parallel architecture training problem with  $\ell_2$  loss function, then the upperbound becomes  $(1 + \frac{\sqrt{m_1m_2}\sigma_{r+1}}{\beta})^2$ , which can be arbitrarily close to one due to presence of noise component (with small  $\sigma_{r+1}$ ) in most datasets in practice (see Figure 4 for an empirical verification). This observation is also valid for several benchmark datasets, including MNIST, CIFAR-10, and CIFAR-100, which exhibit exponentially decaying singular values (see Figure 7) and therefore effectively has a low rank structure. In addition, singular values can be computed to set the target rank and the value of the regularization coefficient to obtain any desired approximation ratio using Theorem 2.
A.10 Proof of Proposition 2 and Corollary 1
We first review the multi-layer hyperplane arrangements concept introduced in Section 2.1 of [21]. Based on this concept, we then calculate the training complexity to globally solve the convex program in Theorem 1.
If we denote the number of hyperplane arrangements for the first ReLU layer as  $P_{1}$ , then from Appendix A.8 we know that
$$
P _ {1} \leq 2 r \left(\frac {e (n - 1)}{r}\right) ^ {r} \approx \mathcal {O} \left(n ^ {r}\right). \tag {24}
$$
In order to obtain a bound for the number of hyperplane arrangements in the second ReLU layer we first note that preactivations of the second ReLU layer, i.e.,  $(\mathbf{X}\mathbf{W}_1)_+\mathbf{w}_2$  can be equivalently represented as a matrix-vector product form by using the effective data matrix  $\bar{\mathbf{X}} := [\mathcal{I}_1\mathbf{D}_{11}\mathbf{X} \mathcal{I}_2\mathbf{D}_{12}\mathbf{X} \dots \mathcal{I}_{m_1}\mathbf{D}_{1m_1}\mathbf{X}]$  due to the equivalent representation in (7). Therefore, given  $\mathrm{rank}(\bar{\mathbf{X}}) = r_2 \leq m_1r$
$$
\bar {P} _ {2} \leq 2 \sum_ {k = 0} ^ {r _ {2} - 1} {\binom {n - 1} {k}} \leq 2 r _ {2} \left(\frac {e (n - 1)}{r _ {2}}\right) ^ {r _ {2}} \leq 2 m _ {1} r \left(\frac {e (n - 1)}{m _ {1} r}\right) ^ {m _ {1} r}.
$$
However, notice that  $\bar{\mathbf{X}}$  is not a fixed data matrix since we can choose each diagonal  $\mathbf{D}_{1j_1}$  among a set  $\{\mathbf{D}_i\}_{i=1}^{P_1}$  of size  $P_1$  due to (24) and the sign pattern  $\mathcal{I}_{j_1}$  among the set  $\{+1,-1\}$  of size 2. Thus, in the worst-case, we have the following upper-bound
$$
P _ {2} \leq \bar {P} _ {2} \left(2 P _ {1}\right) ^ {m _ {1}} \leq \frac {2 ^ {2 m _ {1} + 1} (e (n - 1)) ^ {2 m _ {1} r}}{m _ {1} ^ {m _ {1} r - 1} r ^ {2 m _ {1} r - m _ {1} - 1}} \approx \mathcal {O} \left(n ^ {m _ {1} r}\right). \tag {25}
$$
(a) MNIST-Linear scale
(c) CIFAR-10-Linear scale
Figure 7: The values of normalized singular values of the data matrix for the MNIST and CIFAR-10 datasets. As illustrated in both figures, the singular values follow an exponentially decaying trends indicating an effective low rank structure.
(b) MNIST-Log scale
(d) CIFAR-10-Log scale
Notice that given fixed scalars  $m_1$  and  $r$ , both  $P_1$  and  $P_2$  are polynomial terms with respect to the number of data samples  $n$  and the feature dimension  $d$ .
Remark 3. Notice that Convolutional Neural Networks (CNNs) operate on the patch matrices  $\{\mathbf{X}_b\}_{b=1}^B$  instead of the full data matrix  $\mathbf{X}$ , where  $\mathbf{X}_b \in \mathbb{R}^{n \times h}$  and  $h$  denotes the filter size. Hence, even when the data matrix is full rank, i.e.,  $r = \min\{n, d\}$ , the number of hyperplane arrangements  $P_1$  is upperbounded as  $P_1 \leq \mathcal{O}(n^{r_c})$ , where  $r_c := \max_b \text{rank}(\mathbf{X}_b) \leq h \ll \min\{n, d\}$  (see [45] for details). For instance, let us consider a CNN with  $m_1 = 512$  filters of size  $3 \times 3$ , then  $r_c \leq 9$  independent of data dimension  $n, d$ . As a consequence, weight sharing structure in CNNs dramatically limits the number of possible hyperplane arrangements. This also explains efficiency and remarkable generalization performance of CNNs in practice.
Training complexity analysis: Here, we calculate the computational complexity to globally solve the convex program in (8). Note that (8) is a convex optimization problem with  $2dm_{1}MP_{1}P_{2}$  variables and  $2n(m_{1} + 1)MP_{1}P_{2}$  constraints. Therefore, due to the upper bounds in (24) and (25) of Appendix A.8, the convex program (8) can be globally optimized by a standard interior-point solver with the computational complexity  $\mathcal{O}(d^{3}m_{1}^{3}2^{3(m_{1} + 1)}n^{3(m_{1} + 1)r})$ , which is a polynomial-time complexity in terms of  $n, d$ .
The analysis in this section can be recursively extended to arbitrarily deep parallel networks. First notice that if we apply the same approach to obtain an upperbound on  $P_{3}$ , then due to the multiplicative pattern in (25), we obtain  $P_{3} \leq P_{3}^{\prime}(2P_{2})^{m_{2}} \leq \mathcal{O}(n^{m_{2}m_{1}r})$ . In a similar manner, the number of hyperplane arrangements in the  $l^{th}$  layer is upperbounded as  $P_{l} \leq \mathcal{O}(n^{r\prod_{j=1}^{l-1}m_{j}})$ , which is polynomial in both  $n$  and  $d$  for fixed data rank  $r$  and fixed layer widths  $\{m_{j}\}_{j=1}^{l-1}$ .
A.11 Extension to vector outputs
In this section, we extend the analysis to parallel networks with multiple outputs where the label matrix is defined as  $\mathbf{Y} \in \mathbb{R}^{n \times C}$  provided that there exist  $C$  classes/outputs. Then the primal non-convex training problem is as follows
$$
p _ {v} ^ {*} := \min  _ {\theta \in \Theta} \mathcal {L} \left(\sum_ {k = 1} ^ {K} f _ {\theta , k} (\mathbf {X}), \mathbf {Y}\right) + \beta \sum_ {k = 1} ^ {K} \sqrt {\sum_ {j _ {1} , j _ {2} , \ldots , j _ {L}} \left(\| \mathbf {w} _ {1 k j _ {1}} \| _ {2} ^ {2} \prod_ {l = 2} ^ {L - 1} w _ {l k j _ {l - 1} j _ {l}} ^ {2} \| \mathbf {w} _ {L k j _ {L - 1} j _ {L}} ^ {2} \| ^ {2}\right)}.
$$
Applying Lemma 1 and each step in the proof of Theorem 1 yield
$$
d _ {v} ^ {*} := \max  _ {\mathbf {V}} \min  _ {\theta \in \Theta_ {s}} - \mathcal {L} ^ {*} (\mathbf {V}) \mathrm {s . t .} \left\| \mathbf {V} ^ {T} \left((\mathbf {X} \mathbf {W} _ {1 k}) _ {+} \dots \mathbf {W} _ {(L - 1) k}\right) _ {+} \right\| _ {F} \leq \beta , \forall k \in [ K ],
$$
where the corresponding Fenchel conjugate function is
$$
\mathcal {L} ^ {*} (\mathbf {V}) := \max _ {\mathbf {Z}} \operatorname {t r a c e} \left(\mathbf {Z} ^ {T} \mathbf {V}\right) - \mathcal {L} \left(\mathbf {Z}, \mathbf {Y}\right).
$$
Notice that above we have a dual matrix  $\mathbf{V}$  instead of the dual vector in the scalar output case. More importantly, here, we have  $\ell_2$  norm in the dual constraint unlike the scalar output case with absolute value. Therefore, the vector-output case is slightly more challenging than the scalar output case and yields a different regularization function in the equivalent convex program.
The rest of the derivations directly follows the steps in Section A.4 and [62].