SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here, we briefly review the cone decomposition concept introduced by [8]. We first note that (3) is the exact convex reformulation of the standard non-convex regularized training problem in (2). However, as mentioned in the main paper, the number of constraints in (3) can be prohibitively large, precisely can be  $\mathcal{O}(n^d)$ , and consequently prevents solving the exact convex optimization problem especially for large scale datasets. To remedy this issue, [8] proposed the unconstrained formulation in (8), but, this relaxation corresponds to another neural network architecture called gated ReLU network in (5). Although this relaxation breaks dependence between the arrangements and preactivations of a ReLU activation, [8] proved that once the gated ReLU network training problem in (8) training problem is solved, one can construct a solution for the ReLU network model in (1). This procedure is named as cone decomposition and we provide details regarding the implementation steps below.
Let us assume that we solve the gated ReLU network training problem in (8) and denote the optimal solution as  $\{\mathbf{w}_i^*\}_{i=1}^p$ . Then, since each  $\mathbf{D}_i \mathbf{X} \mathbf{w}_i^*$  can have both negative and positive entries (due to the absence of nonnegativity constraint), we need to decompose  $\mathbf{D}_i \mathbf{X} \mathbf{w}_i^*$  into two separate cones to maintain nonnegativity property of a ReLU activation. We formulate this decomposition step as the following convex optimization problem
$$
\min  _ {\mathbf {v} _ {i}, \mathbf {u} _ {i}} \| \mathbf {v} _ {\mathbf {i}} \| _ {2} + \| \mathbf {u} _ {i} \| _ {2} \quad \text {s . t .} \quad \mathbf {D} _ {i} \mathbf {X} \mathbf {w} _ {i} ^ {*} = \mathbf {D} _ {i} \mathbf {X} (\mathbf {v} _ {i} - \mathbf {u} _ {i}), \begin{array}{l} (2 \mathbf {D} _ {i} - \mathbf {I} _ {n}) \mathbf {X} \mathbf {v} _ {i} \geq \mathbf {0} \\ (2 \mathbf {D} _ {i} - \mathbf {I} _ {n}) \mathbf {X} \mathbf {u} _ {i} \geq \mathbf {0}. \end{array} \tag {28}
$$
After solving the convex optimization problem in (28) for all  $i \in [p]$ . We declare the optimal solutions,  $\{\mathbf{v}_i^*, \mathbf{u}_i^*\}_{i=1}^p$  as a solution to the exact ReLU model in (1). Thus, instead of directly solving the convex reformulation with extremely large number of constraints in (3), we first solve the relaxed gated ReLU problem in (8) and then map the solutions to the proper cones to enforce the nonnegativity property of the ReLU activation.
E.1 Computational Complexity
Although [8] reduce the computational complexity by eliminating the constraints in the convex optimization problem in (3), the number of variables, denotes as  $p$ , can still be exponentially large in feature dimension  $d$ . However, there are multiple ways to remedy this issue.
- We can change the architecture. Particularly, we can replace fully connected networks with convolutional networks. Then, since CNNs operate on the patch matrices  $\{\mathbf{X}_b\}_{b=1}^B$  instead of the full data matrix  $\mathbf{X}$ , where  $\mathbf{X}_b \in \mathbb{R}^{n \times h}$  and  $h$  denotes the filter size, even when the data matrix is full rank, i.e.,  $r = \min(n, d)$ , the number of hyperplane arrangements  $p$  is upperbounded as  $p \leq \mathcal{O}(n^{r_c})$ , where  $r_c := \max_b \operatorname{rank}(\mathbf{X}_b) \leq h \ll \min(n, d)$ . For instance, let us consider a CNN with  $3 \times 3$  filters, then  $r_c \leq 9$  independent of  $n, d$ . As a consequence, weight sharing structure in CNNs dramatically limits the number of possible hyperplane arrangements and avoids exponential complexity. This also explains efficiency and remarkable generalization performance of CNNs in practice.
- We can also use a sampling based approach where one can randomly sample a tiny subset of all possible hyperplane arrangements and then solve the convex program with this subset. Thus, although the resulting approach is not exact, the training complexity will not be exponential in  $d$  anymore. The experimental results in Appendix C show that this approximation in fact works extremely well, specifically resulting in models that outperform the NTK in 26/33 UCI datasets as detailed in Table 1.