SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The conditional expectation is the probability of the event that  $\mathbf{x}^T\mathbf{g}\geq 0$  given that  $\mathbf{x}^{\prime T}\mathbf{g}\geq 0$ . This can be geometrically seen as the ratio of  $\pi$  minus the angle between  $\mathbf{x}$  and  $\mathbf{x}'$  to  $\pi$ . So, we get
$$
\mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {g} \geq 0 \right\} \right] = \frac {1}{2} \cdot \frac {1}{\pi} \left(\pi - \operatorname {a r c c o s} \left(\frac {\mathbf {x} ^ {T} \mathbf {x} ^ {\prime}}{\| \mathbf {x} \| _ {2} \| \mathbf {x} ^ {\prime} \| _ {2}}\right)\right).
$$
Combining this with (18) yields the desired result.
A.2 Proof of Lemma 2.3
The gradient feature map of (11) is
$$
\nabla_ {\boldsymbol {\theta}} f _ {r} \left(\mathbf {x}, \boldsymbol {\theta}\right) = \frac {1}{\sqrt {2 m}} \left[ \begin{array}{c} \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} _ {1} ^ {(+)} \geq 0 \right\} \mathbf {x} \\ \vdots \\ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} _ {m} ^ {(+)} \geq 0 \right\} \mathbf {x} \\ - \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} _ {1} ^ {(-)} \geq 0 \right\} \mathbf {x} \\ \vdots \\ - \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} _ {m} ^ {(-)} \geq 0 \right\} \mathbf {x} \end{array} \right].
$$
The finite width random NTK given by  $H_{m,\pmb{\theta}}^{r}(\mathbf{x},\mathbf{x}^{\prime}):= \nabla_{\pmb{\theta}}f_{r}(\mathbf{x},\pmb{\theta})^{T}\nabla_{\pmb{\theta}}f_{r}(\mathbf{x}^{\prime},\pmb{\theta})$  is
$$
\begin{array}{l} H _ {m, \boldsymbol {\theta}} ^ {r} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \frac {1}{2 m} \sum_ {j = 1} ^ {m} \left(\mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} _ {j} ^ {(+)} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {w} _ {j} ^ {(+)} \geq 0 \right\} \mathbf {x} ^ {T} \mathbf {x} ^ {\prime} + \right. \\ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} _ {j} ^ {(-)} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {w} _ {j} ^ {(-)} \geq 0 \right\} \mathbf {x} ^ {T} \mathbf {x} ^ {\prime}). \\ \end{array}
$$
By the Law of Large numbers since  $\mathbf{w}_j^{(+)},\mathbf{w}_j^{(-)}$  are iid,  $H_{m,\pmb{\theta}}^r (\mathbf{x},\mathbf{x}^\prime)$  converges in probability to the expectation of the quantity in the sum as  $m\to \infty$ . We denote this infinite limit by  $H^{r}(\mathbf{x},\mathbf{x}^{\prime})$ . Therefore,
$$
\begin{array}{l} H ^ {r} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \frac {1}{2} \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} ^ {(+)} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {w} ^ {(+)} \geq 0 \right\} + \right. \\ \left. \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} ^ {(-)} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {w} ^ {(-)} \geq 0 \right\} \right] \mathbf {x} ^ {T} \mathbf {x} ^ {\prime}, \\ \end{array}
$$
where  $\mathbf{w}^{(+)}\sim \mathcal{N}(\mathbf{0},\mathbf{I_d})$  and  $\mathbf{w}^{(-)}\sim \mathcal{N}(\mathbf{0},\mathbf{I_d})$ . Since  $\mathbf{w}^{(+)}$  and  $\mathbf{w}^{(-)}$  are i.i.d., we can combine the expectations to get
$$
H ^ {r} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {w} ^ {(+)} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {w} ^ {(+)} \geq 0 \right\} \right] \mathbf {x} ^ {T} \mathbf {x} ^ {\prime}.
$$
Comparing to (18), we get the desired result.
A.3 Proof of Lemma 4.2
Since  $\mathcal{G}$  is complete,  $\forall i\in [p],\exists \mathbf{g}_i\in \mathcal{G}$  such that  $\mathrm{diag}(\mathbb{1}\{\mathbf{X}\mathbf{g}_i\geq 0\}) = \mathbf{D}_i$ . Additionally, these gates  $\mathbf{g}_i$  are unique since  $\mathrm{diag}(\mathbb{1}\{\mathbf{X}\mathbf{g}_i\geq 0\})\neq \mathrm{diag}(\mathbb{1}\{\mathbf{X}\mathbf{g}_j\geq 0\})\Rightarrow \mathbf{g}_i\neq \mathbf{g}_j$ . Therefore, there is a subset of gates  $\mathcal{G}$  of size  $p$  that are uniquely associated to each mask  $\mathbf{D}_i$ . Since  $\mathcal{G}$  is also minimally complete, this subset must be  $\mathcal{G}$  and this proves the lemma.
By Lemma 3.1, the MKL problem given by
is equivalent to the group lasso problem (14). Additionally, (14) is also equivalent to the standard group lasso (15) since they have the same regularization paths. Since  $\mathcal{G}$  is minimally complete, (7) (which is equivalent to (6) in the minimally complete gate set setting) is equivalent to the group lasso problem (15) and this proves the theorem.
A.5 Proof of Theorem 5.1
The weighted masking kernel is given by
$$
\mathbf {K} _ {\mathcal {G}} (\tilde {\boldsymbol {\eta}}) = \sum_ {i = 1} ^ {p} \tilde {\eta} _ {i} \mathbf {D} _ {i} \mathbf {X} \mathbf {X} ^ {T} \mathbf {D} _ {i}. \tag {19}
$$
Now, the NTK matrix  $\mathbf{H}$  can be written as (using (18)),
$$
\mathbf {H} = \mathbb {E} _ {\mathbf {g} \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {\mathrm {d}})} \left[ \operatorname {d i a g} \left(\mathbb {1} \left\{\mathbf {X} \mathbf {g} \geq \mathbf {0} \right\}\right) \mathbf {X} \mathbf {X} ^ {T} \operatorname {d i a g} \left(\mathbb {1} \left\{\mathbf {X} \mathbf {g} \geq \mathbf {0} \right\}\right) \right].
$$
We can condition on the events that diag  $(\mathbb{1}\{\mathbf{Xg}\geq \mathbf{0}\}) = \mathbf{D}_i\forall i\in [p]$  . Since  $\mathcal{D}_{\mathbf{X}}$  covers all possible data maskings  $\mathbf{D}_i$  , we know that  $\sum_{i = 1}^{p}\mathbb{P}[\mathrm{diag}(\mathbb{1}\{\mathbf{Xg}\geq \mathbf{0}\}) = \mathbf{D}_i] = 1$  . Therefore,
$$
\begin{array}{l} \mathbf {H} = \sum_ {i = 1} ^ {p} \mathbb {E} _ {\mathbf {g} \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {\mathbf {d}})} \left[ \mathbf {D} _ {i} \mathbf {X} \mathbf {X} ^ {T} \mathbf {D} _ {i} \mid \operatorname {d i a g} \left(\mathbb {1} \left\{\mathbf {X} \mathbf {g} \geq \mathbf {0} \right\}\right) = \mathbf {D} _ {i} \right] \mathbb {P} [ \operatorname {d i a g} \left(\mathbb {1} \left\{\mathbf {X} \mathbf {g} \geq \mathbf {0} \right\}\right) = \mathbf {D} _ {i} ] \\ = \sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \mathbf {X} ^ {T} \mathbf {D} _ {i} \mathbb {P} [ \operatorname {d i a g} (\mathbb {1} \left\{\mathbf {X} \mathbf {g} \geq \mathbf {0} \right\}) = \mathbf {D} _ {i} ]. \\ \end{array}
$$
Plugging in  $\tilde{\eta}_i = \mathbb{P}[\mathrm{diag}(\mathbb{1}\{\mathbf{Xg}\geq \mathbf{0}\}) = \mathbf{D}_i]$  and comparing to (19), we get that  $\mathbf{K}_{\mathcal{G}}(\tilde{\boldsymbol{\eta}}) = \mathbf{H}$  and this proves Theorem 5.1.
A.6 Proof of Corollary 5.2
First, we define a single combined masked data matrix  $\tilde{\mathbf{X}}\in \mathbb{R}^{n\times pd}$  as follows
$$
\tilde {\mathbf {X}} := \left[ \begin{array}{c c} \mathbf {D} _ {1} \mathbf {X} & \mathbf {D} _ {2} \mathbf {X} \dots \mathbf {D} _ {p} \mathbf {X} \end{array} \right].
$$
We also define a diagonal regularization weighting matrix  $\mathbf{R} \in \mathbb{R}^{pd \times pd}$  as follows
$$
\mathbf {R} := \operatorname {d i a g} \left(\left[ \begin{array}{c c c c} \tilde {\eta} _ {1} \mathbf {1} _ {\mathbf {d}} & \tilde {\eta} _ {2} \mathbf {1} _ {\mathbf {d}} & \dots & \tilde {\eta} _ {p} \mathbf {1} _ {\mathbf {d}} \end{array} \right]\right),
$$
where  $\mathbf{1_d}$  is a  $d$ -dimensional all-1s row vector. Now, we can rewrite (16) with  $\Phi_i = \mathbf{D}_i\mathbf{X}$  and regularization weights  $\tilde{\eta}$  as
$$
\min  _ {\mathbf {w} \in \mathbb {R} ^ {p d}} \left\| \tilde {\mathbf {X}} \mathbf {w} - \mathbf {y} \right\| _ {2} ^ {2} + \lambda \left\| \mathbf {R} ^ {- \frac {1}{2}} \mathbf {w} \right\| _ {2} ^ {2}. \tag {20}
$$
Setting the gradient of the convex objective in (20) to 0, we get the following equation for the solution  $\tilde{\mathbf{w}}$ ,
$$
2 \tilde {\mathbf {X}} ^ {T} \left(\tilde {\mathbf {X}} \tilde {\mathbf {w}} - \mathbf {y}\right) + 2 \lambda \mathbf {R} ^ {- 1} \tilde {\mathbf {w}} = 0.
$$
Solving for  $\tilde{\mathbf{w}}$ , we get
$$
\tilde {\mathbf {w}} = \left(\tilde {\mathbf {X}} ^ {T} \tilde {\mathbf {X}} + \lambda \mathbf {R} ^ {- 1}\right) ^ {- 1} \tilde {\mathbf {X}} ^ {T} \mathbf {y}.
$$
By applying the matrix inversion lemma [53], we can rewrite this as
$$
\tilde {\mathbf {w}} = \mathbf {R} \tilde {\mathbf {X}} ^ {T} \left(\tilde {\mathbf {X}} \mathbf {R} \tilde {\mathbf {X}} ^ {T} + \lambda \mathbf {I} _ {\mathbf {n}}\right) ^ {- 1} \mathbf {y}.
$$
Now, we notice that
$$
\sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \tilde {\mathbf {w}} _ {i} = \tilde {\mathbf {X}} \tilde {\mathbf {w}} = \tilde {\mathbf {X}} \mathbf {R} \tilde {\mathbf {X}} ^ {T} \left(\tilde {\mathbf {X}} \mathbf {R} \tilde {\mathbf {X}} ^ {T} + \lambda \mathbf {I} _ {\mathbf {n}}\right) ^ {- 1} \mathbf {y}. \tag {21}
$$
Additionally, observe that  $\tilde{\mathbf{X}}\mathbf{R}\tilde{\mathbf{X}}^T = \sum_{i=1}^{p} \tilde{\eta}_i \mathbf{D}_i \mathbf{X}\mathbf{X}\mathbf{D}_i = \mathbf{H}$  where the last equality is by Theorem 5.1. Plugging this into (21), we get
$$
\sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \tilde {\mathbf {w}} _ {i} = \mathbf {H} \left(\mathbf {H} + \lambda \mathbf {I} _ {\mathbf {n}}\right) ^ {- 1} \mathbf {y}
$$
and this proves Corollary 5.2.
A.7 Proof of Theorem 6.1
We begin with the optimality of  $\hat{\mathbf{w}}$
$$
\frac {1}{n} \left\| \sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \hat {\mathbf {w}} _ {i} - \mathbf {y} \right\| _ {2} ^ {2} + \lambda \sum_ {i = 1} ^ {p} \| \hat {\mathbf {w}} _ {i} \| _ {2} \leq \frac {1}{n} \left\| \sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \mathbf {w} _ {i} ^ {*} - \mathbf {y} \right\| _ {2} ^ {2} + \lambda \sum_ {i = 1} ^ {p} \| \mathbf {w} _ {i} ^ {*} \| _ {2}.
$$
By plugging in the model  $\mathbf{y} = \sum_{i=1}^{p} \mathbf{D}_i \mathbf{X} \mathbf{w}_i^* + \epsilon$ , we simplify this to
$$
\frac {1}{n} \left\| \sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \left(\hat {\mathbf {w}} _ {i} - \mathbf {w} _ {i} ^ {*}\right) - \epsilon \right\| _ {2} ^ {2} + \lambda \sum_ {i = 1} ^ {p} \| \hat {\mathbf {w}} _ {i} \| _ {2} \leq \frac {1}{n} \| \epsilon \| _ {2} ^ {2} + \lambda \sum_ {i = 1} ^ {p} \| \mathbf {w} _ {i} ^ {*} \| _ {2}.
$$
We expand the square on the LHS and obtain the following bound on the prediction error
$$
\begin{array}{l} \frac {1}{n} \left\| \sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \left(\hat {\mathbf {w}} _ {i} - \mathbf {w} _ {i} ^ {*}\right) \right\| _ {2} ^ {2} \leq \frac {2}{n} \sum_ {i = 1} ^ {p} \epsilon^ {T} \mathbf {D} _ {i} \mathbf {X} \left(\hat {\mathbf {w}} _ {i} - \mathbf {w} _ {i} ^ {*}\right) + \lambda \sum_ {i = 1} ^ {p} \left(\left\| \mathbf {w} _ {i} ^ {*} \right\| _ {2} - \left\| \hat {\mathbf {w}} _ {i} \right\| _ {2}\right) \\ \leq \frac {2}{n} \sum_ {i = 1} ^ {p} \left\| \boldsymbol {\epsilon} ^ {T} \mathbf {D} _ {i} \mathbf {X} \right\| _ {2} \left\| \hat {\mathbf {w}} _ {i} - \mathbf {w} _ {i} ^ {*} \right\| _ {2} + \lambda \sum_ {i = 1} ^ {p} \left(\left\| \mathbf {w} _ {i} ^ {*} \right\| _ {2} - \left\| \hat {\mathbf {w}} _ {i} \right\| _ {2}\right), \\ \end{array}
$$
where we used Cauchy-Schwarz inequality in the second step. Now, since  $\mathbf{D}_i$  are diagonal matrices with either 0 or 1 on the diagonals, we can immediately say that  $\left\| \epsilon^T\mathbf{D}_i\mathbf{X}\right\| _2\leq \left\| \epsilon^T\mathbf{X}\right\| _2$  for all  $i = 1,\ldots ,p$ . As a consequence, we get
$$
\begin{array}{l} \frac {1}{n} \left\| \sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \left(\hat {\mathbf {w}} _ {i} - \mathbf {w} _ {i} ^ {*}\right) \right\| _ {2} ^ {2} \leq \frac {2}{n} \left\| \epsilon^ {T} \mathbf {X} \right\| _ {2} \sum_ {i = 1} ^ {p} \| \hat {\mathbf {w}} _ {i} - \mathbf {w} _ {i} ^ {*} \| _ {2} + \lambda \sum_ {i = 1} ^ {p} \left(\left\| \mathbf {w} _ {i} ^ {*} \right\| _ {2} - \| \hat {\mathbf {w}} _ {i} \| _ {2}\right) \\ \leq \left(\frac {2}{n} \left\| \epsilon^ {T} \mathbf {X} \right\| _ {2} + \lambda\right) \sum_ {i = 1} ^ {p} \left\| \mathbf {w} _ {i} ^ {*} \right\| _ {2} + \left(\frac {2}{n} \left\| \epsilon^ {T} \mathbf {X} \right\| _ {2} - \lambda\right) \sum_ {i = 1} ^ {p} \| \hat {\mathbf {w}} _ {i} \| _ {2}, \\ \end{array}
$$
where we used the triangle inequality in the second step. Now, if  $\lambda \geq \frac{2}{n}\left\| \epsilon^T\mathbf{X}\right\|_2$ , we can get rid of the second term, namely  $\left(\frac{2}{n}\left\| \epsilon^T\mathbf{X}\right\|_2 - \lambda\right)\sum_{i=1}^{p}\|\hat{\mathbf{w}}_i\|_2$  and obtain a bound on the prediction error that does not depend on  $\hat{\mathbf{w}}$ . To this end, notice that  $\epsilon^T\mathbf{X}$  is a multivariate Gaussian with mean 0 and covariance  $\sigma^2\mathbf{X}^T\mathbf{X}$ . We now present a standard concentration result for the norms of Gaussians in the lemma below,
Lemma A.1. Let  $\mathbf{z}$  be a  $d$ -dimensional multivariate Gaussian with mean 0 and covariance matrix  $\boldsymbol{\Sigma}$ . We have
$$
\mathbb {P} \left[ \| \mathbf {z} \| _ {2} \leq z \right] \geq 1 - 2 \exp \left(\frac {- z ^ {2}}{2 \mathrm {t r} (\boldsymbol {\Sigma})}\right).
$$
Applying Lemma A.1 to  $\epsilon^T\mathbf{X}$ , we get
$$
\mathbb {P} \left[ \left\| \epsilon^ {T} \mathbf {X} \right\| _ {2} \leq z \right] \geq 1 - 2 e ^ {\frac {- z ^ {2}}{2 \sigma^ {2} \mathrm {t r} (\mathbf {X} ^ {T} \mathbf {X})}}.
$$
Plugging in  $z = \frac{n}{2}\lambda$  where  $\lambda = \frac{t\sigma\sqrt{\operatorname{tr}(\mathbf{X}^T\mathbf{X})}}{n}$  as in the statement of theorem, we get
$$
\mathbb {P} \left[ \frac {2}{n} \| \epsilon^ {T} \mathbf {X} \| _ {2} \leq \lambda \right] \geq 1 - 2 e ^ {\frac {- t ^ {2}}{8}}.
$$
Therefore, by choosing  $\lambda = \frac{t\sigma\sqrt{\mathrm{tr}(\mathbf{X}^T\mathbf{X})}}{n}$ , we have that  $\frac{2}{n}\left\| \epsilon^T\mathbf{X}\right\|_2 \leq \lambda$  with probability at least  $1 - 2e^{-t^2 /8}$ . Consequently, we have
$$
\frac {1}{n} \left\| \sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \left(\hat {\mathbf {w}} _ {i} - \mathbf {w} _ {i} ^ {*}\right) \right\| _ {2} ^ {2} \leq 2 \lambda \sum_ {i = 1} ^ {p} \| \mathbf {w} _ {i} ^ {*} \| _ {2}
$$
with probability at least  $1 - 2e^{-t^2 /8}$ , which completes the proof.
A.7.1 Proof of Lemma A.1
Since the norm of Gaussian vector does not change by applying a rotation, we know that  $\left\| \Lambda^{1 / 2}\mathbf{u}\right\| _2$  and  $\| \mathbf{z}\| _2$  have the same distribution where  $\Lambda$  is the matrix of eigenvalues from the spectral decomposition  $\boldsymbol {\Sigma} = Q\Lambda Q^T$  and  $\mathbf{u}\sim \mathcal{N}(\mathbf{0},\mathbf{I}_d)$ . Now we apply a standard Chernoff bound argument. We have for some  $s > 0$
$$
\begin{array}{l} \mathbb {P} \left[ \left\| \Lambda^ {1 / 2} \mathbf {u} \right\| _ {2} > z \right] \leq \mathbb {P} \left[ \left\| \Lambda^ {1 / 2} \mathbf {u} \right\| _ {1} > z \right] \\ \leq \mathbb {P} \left[ e ^ {s \left\| \Lambda^ {1 / 2} \mathbf {u} \right\| _ {1}} > e ^ {s z} \right] \\ \leq e ^ {- s z} \Pi_ {i = 1} ^ {d} \mathbb {E} \left[ e ^ {s \sqrt {\lambda_ {i}} | \mathbf {u} _ {i} |} \right] \\ \leq e ^ {- s z} \Pi_ {i = 1} ^ {d} 2 e ^ {s ^ {2} \lambda_ {i} / 2}. \\ \end{array}
$$
By optimizing the bound over  $s$ , we get
$$
\mathbb {P} \left[ \left\| \Lambda^ {1 / 2} \mathbf {u} \right\| _ {2} > z \right] \leq 2 \exp \left(\frac {- z ^ {2}}{2 \sum_ {i = 1} ^ {d} \lambda_ {i}}\right) \leq 2 \exp \left(\frac {- z ^ {2}}{2 \operatorname {t r} (\boldsymbol {\Sigma})}\right).
$$
Finally, using  $\mathbb{P}\left[\| \mathbf{z}\| _2\leq z\right] = 1 - \mathbb{P}\left[\| \mathbf{z}\| _2 > z\right]$  gives the result.
Hyperplane Arrangements for 1D example. In the 1D example in Figure 1, the data matrix  $\mathbf{X}$  is in  $\mathbb{R}^{n\times 2}$ , i.e. the dimension  $d = 2$ . This is because we set the second component of each input to be exactly equal to 1 to simulate a bias in the neural network. Though the datapoints lie in  $\mathbb{R}^2$ , they have some structure which allows us to enumerate all hyperplane arrangements  $\mathcal{D}_{\mathbf{X}}$  easily. Specifically, the datapoints all lie on the line  $y = 1$  in the  $xy$  plane, so the hyperplanes that separate them always separate them into two contiguous segments. Therefore, by sorting the datapoints according to their first component, we can directly enumerate all  $2n$  hyperplane arrangements.
Nonconvex ReLU network. For the black dashed line in Figure 1, we trained a two layer ReLU network with 100 hidden neurons using the standard weight decay regularized objective with gradient descent using a learning rate of 0.01 for 200000 epochs.
Convex program solver specifications. For directly solving the group lasso problem (8), we used CVXPY with the MOSEK solver [54]. For the  $\ell_2$  regularized least squares problems (16), we used iterative methods with an efficient matrix-vector product implementation to avoid explicitly constructing the large combined masked data matrix  $\tilde{\mathbf{X}}$ . Specifically, we used the LSQR and LSMR methods as described in [41, 42]. The experiments were run locally on a MacBook Air 2020 version with the Apple M1 chip.
UCI Machine Learning Repository. Here we present details for the experiments on several real datasets from the UCI Machine Learning Repository [55]. We follow the procedure described in [43] for  $n \leq 1000$  to extract and standardize the datasets. For these experiments, we use the  $75\% - 25\%$  ratio for the training and test set splits. We use the squared loss and tune the regularization parameter  $\lambda$  for both NTK and our approach by performing a grid search over the set  $\{10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 10^{0}, 10^{1}\}$ . As shown in Table 1, our method achieves higher (or the same) test accuracy for 26 datasets out 33 datasets while standard NTK achieves higher (or the same) test accuracy for 14.
C Generic Loss Functions
Here, we prove that the analysis with squared loss presented in the main paper can be extended to arbitrary convex loss function. Let us consider the following regularized two-layer ReLU network training problem with an arbitrary convex loss function  $\mathcal{L}(\cdot ,\cdot)$
$$
\min  _ {\mathbf {W} ^ {(1)}, \mathbf {w} ^ {(2)}} \mathcal {L} \left(\sum_ {j = 1} ^ {m} \left(\mathbf {X} \mathbf {w} _ {j} ^ {(1)}\right) _ {+} w _ {j} ^ {(2)}, \mathbf {y}\right) + \lambda \sum_ {j = 1} ^ {m} \left(\left\| \mathbf {w} _ {j} ^ {(1)} \right\| _ {2} ^ {2} + | w ^ {(2)} | ^ {2}\right). \tag {22}
$$
Then, as shown in [7], the corresponding gated convex reformulation of (22) is
$$
P ^ {*} := \min  _ {\mathbf {w} _ {i}} \mathcal {L} \left(\sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \mathbf {w} _ {i} - \mathbf {y}\right) + \lambda \sum_ {i = 1} ^ {p} \| \mathbf {w} _ {i} \| _ {2}. \tag {23}
$$
In order to realize the emergence of kernel characterization in the case of arbitrary loss function, we use Lagrange duality. Particularly, since (23) is a convex optimization problem and Slater's conditions hold as proven in [7, 8], we equivalently represent (22) as the following dual problem
$$
P ^ {*} = \max  _ {\mathbf {z}} - \mathcal {L} ^ {*} (\mathbf {z}, \mathbf {y}) \quad \text {s . t .} \quad \left\| \mathbf {z} ^ {T} \mathbf {D} _ {i} \mathbf {X} \right\| _ {2} \leq \lambda , \forall i \in [ p ] \tag {24}
$$
where  $\mathcal{L}(\mathbf{z},\mathbf{y})$  is the Fenchel conjugate of  $\mathcal{L}(\cdot ,\mathbf{y})$  defined as [56]
$$
\mathcal {L} ^ {*} \left(\mathbf {z}, \mathbf {y}\right) := \max  _ {\mathbf {v}} \mathbf {v} ^ {T} \mathbf {z} - \mathcal {L} \left(\mathbf {v}, \mathbf {y}\right).
$$
We then form the Lagrangian for (24) as follows
$$
\begin{array}{l} L (\mathbf {z}, \alpha_ {i}) = - \mathcal {L} ^ {*} (\mathbf {z}, \mathbf {y}) + \sum_ {i = 1} ^ {p} \alpha_ {i} \left(\lambda - \left\| \mathbf {z} ^ {T} \mathbf {D} _ {i} \mathbf {X} \right\| _ {2}\right) \\ = - \mathcal {L} ^ {*} (\mathbf {z}, \mathbf {y}) + \sum_ {i = 1} ^ {p} \alpha_ {i} \left(\lambda - \mathbf {z} ^ {T} \mathbf {D} _ {i} \mathbf {X} \mathbf {X} ^ {T} \mathbf {D} _ {i} \mathbf {z}\right) \\ = - \mathcal {L} ^ {*} (\mathbf {z}, \mathbf {y}) + \sum_ {i = 1} ^ {p} \alpha_ {i} (\lambda - \mathbf {z} ^ {T} \mathbf {K} _ {i} \mathbf {z}), \\ \end{array}
$$
where  $\alpha_{i} \geq 0$  is the Lagrange multiplier for the inequality constraint. Based on the representation above, the dual problem in (24) can be written as
$$
P ^ {*} = \max  _ {\mathbf {z}} \min  _ {\alpha_ {i} \geq 0} - \mathcal {L} ^ {*} (\mathbf {v}, \mathbf {y}) + \sum_ {i = 1} ^ {p} \alpha_ {i} (\lambda - \mathbf {z} ^ {T} \mathbf {K} _ {i} \mathbf {z}). \tag {25}
$$
Thus, similar to (12), the problem in (25) can be globally optimized with alternating minimization [33, 35].
D Extensions to Deeper Networks
In this section, we generalize the kernel characterization of two-layer ReLU networks presented in the main paper to deeper architectures. We first note that most prior works on convex reformulation of ReLU networks studied only two-layer ReLU networks, e.g., [6-10], since the non-convex interaction of multiple nonlinear layers hinders the convex analysis of deeper networks. However, recent follow-up studies [13, 57, 58] showed that the analysis can be extended to arbitrarily deep networks. Based on this recent results, below we briefly explain how one can extend our kernel characterization to three-layer networks. By following the convex reformulations in [57, 58], we have the following gated ReLU training problem for three-layer network
$$
\min  _ {\mathbf {w} _ {i j}} \left\| \sum_ {i = 1} ^ {p _ {1}} \sum_ {j = 1} ^ {p _ {2}} \mathbf {D} _ {i} ^ {(1)} \mathbf {D} _ {j} ^ {(2)} \mathbf {X} \mathbf {w} _ {i j} - \mathbf {y} \right\| _ {2} ^ {2} + \lambda \sum_ {i = 1} ^ {p _ {1}} \sum_ {j = 1} ^ {p _ {2}} \| \mathbf {w} _ {i j} \| _ {2}, \tag {26}
$$
where the hyperplane arrangements for the first and second ReLU layers, i.e., denoted as  $\mathbf{D}_i^{(1)}\in \mathcal{D}_{\mathbf{X}}^{(1)}$  and  $\mathbf{D}_j^{(2)}\in \mathcal{D}_{\mathbf{X}}^{(2)}$ , are defined as follows
$$
\mathcal {D} _ {\mathbf {X}} ^ {(1)} := \left\{\operatorname {d i a g} \left(\mathbb {1} \left\{\mathbf {X} \mathbf {u} \geq \mathbf {0} \right\}\right): \mathbf {u} \in \mathbb {R} ^ {d} \right\} \text {a n d} p _ {1} := \left| \mathcal {D} _ {\mathbf {X}} ^ {(1)} \right| \tag {27}
$$
$$
\mathcal {D} _ {\mathbf {X}} ^ {(2)} := \left\{\operatorname {d i a g} \left(\mathbb {1} \left\{\left(\mathbf {X U} ^ {(1)}\right) _ {+} \mathbf {u} ^ {(2)} \geq \mathbf {0} \right\}\right): \mathbf {U} ^ {(1)} \in \mathbb {R} ^ {d \times m _ {1}}, \mathbf {u} ^ {(2)} \in \mathbb {R} ^ {m _ {1}} \right\} \text {a n d} p _ {2} := \left| \mathcal {D} _ {\mathbf {X}} ^ {(2)} \right|.
$$
Then, following the derivations in Section 4, we can conveniently express the corresponding feature matrices of these masking feature maps on the data  $\mathbf{X}$  in terms of fixed diagonal data masks as  $\Phi_{ij} = \mathbf{D}_i^{(1)}\mathbf{D}_j^{(2)}\mathbf{X}$ . Similarly, the corresponding masking kernel matrices take the form  $\mathbf{K}_{ij} = \mathbf{D}_i^{(1)}\mathbf{D}_j^{(2)}\mathbf{X}\mathbf{X}^T\mathbf{D}_i^{(1)}\mathbf{D}_j^{(2)}$ . The same interpretation also extends to arbitrarily deep networks and yields a more complicated feature map due to the interaction of multiple hyperplane arrangement matrices, e.g.,  $\mathbf{D}_i^{(1)}\mathbf{D}_j^{(2)}\mathbf{D}_k^{(3)}\ldots$
E Cone Decomposition