SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1: Set iteration count  $k_{\mathrm{co}} \gets 0$
2: Initialize weights  $\eta_{i}^{(0)}$
3: Set  $\Phi_i \coloneqq \mathbf{D}_i\mathbf{X}$ ,  $\forall \mathbf{D}_i \in \mathcal{D}_{\mathbf{X}}$
4: while not converged and  $k \leq$  max iteration count do
5: Solve the weighted  $\ell_2$  regularized least squares problem:
$$
\left\{\mathbf {w} _ {i} ^ {(k)} \right\} _ {i} = \underset {\{\mathbf {w} _ {i} \} _ {i}} {\operatorname {a r g m i n}} \left\| \sum_ {i = 1} ^ {p} \boldsymbol {\Phi} _ {i} \mathbf {w} _ {i} - \mathbf {y} \right\| _ {2} ^ {2} + \lambda \sum_ {i = 1} ^ {p} \frac {\| \mathbf {w} _ {i} \| _ {2} ^ {2}}{\eta_ {i} ^ {(k)}}
$$
6: Update the weights:  $\eta_{i}^{(k + 1)} = \sqrt{\left\|\mathbf{w}_{i}^{(k)}\right\|_{2} + \epsilon}$
7: Increment iteration count:  $k \gets k + 1$
8: end while
9: Optional: Convert the gated ReLU network to a ReLU network (see Section E for details)
4 Gated ReLU as MKL with Masking Kernels
Motivated by the MKL interpretation of group lasso, we return to the convex reformulation (8) of the gated ReLU network. Notice that this problem has the same structure as the MKL equivalent group lasso problem (15) with a specific set of feature maps that we define below.
Definition 4.1. The masking feature maps  $\phi_j: \mathbb{R}^d \to \mathbb{R}^d$  generated by a fixed set of gates  $\mathcal{G}$  are defined as  $\phi_j(\mathbf{x}) = \mathbb{1}\left\{\mathbf{x}^T\mathbf{g}_j \geq 0\right\} \mathbf{x}$ .
These feature maps can be interpreted as simply passing the input unchanged if it lies in the positive halfspace of the corresponding gate vector  $\mathbf{g}_j$ , i.e.,  $\mathbf{x}^T\mathbf{g}_j \geq 0$ , and returning zero if the input does not lie in this halfspace. Since  $\operatorname{diag}(\mathbb{1}\{\mathbf{X}\mathbf{g}_j \geq \mathbf{0}\}) \in \mathcal{D}_{\mathbf{X}}$ ,  $\forall \mathbf{g}_j \in \mathcal{G}$  holds for an arbitrary gate set  $\mathcal{G}$ , we can conveniently express the corresponding feature matrices of these masking feature maps on the data  $\mathbf{X}$  in terms of fixed diagonal data masks as  $\Phi_j = \mathbf{D}_j\mathbf{X}$ , where  $\mathbf{D}_j = \operatorname{diag}(\mathbb{1}\{\mathbf{X}\mathbf{g}_j \geq \mathbf{0}\})$ . Similarly, the corresponding masking kernel matrices take the form  $\mathbf{K}_j = \mathbf{D}_j\mathbf{X}\mathbf{X}^T\mathbf{D}_j$ . Note that for an arbitrary set of gates, the generated masking feature matrices on  $\mathbf{X}$  may not cover the entire set of possible masks  $\mathcal{D}_{\mathbf{X}}$ . Additionally, multiple gate vectors can result in identical masks if  $\operatorname{diag}(\mathbb{1}\{\mathbf{X}\mathbf{g}_i \geq \mathbf{0}\}) = \operatorname{diag}(\mathbb{1}\{\mathbf{X}\mathbf{g}_j \geq \mathbf{0}\})$  for  $i \neq j$  leading to degenerate feature matrices. However, if we work with minimally complete gate sets as defined in Definition 2.1, we can rectify these issues.
Lemma 4.2. For a minimally complete gate set  $\mathcal{G}$  defined in Definition 2.1, we can uniquely associate a gate vector  $\mathbf{g}_i$  to each data mask  $\mathbf{D}_i\in \mathcal{D}_{\mathbf{X}}$  such that  $\mathbf{D}_i = \mathrm{diag}\left(\mathbb{1}\left\{\mathbf{X}\mathbf{g}_i\geq \mathbf{0}\right\}\right),\forall i\in [p]$ .
Consequently, for minimally complete gate sets  $\mathcal{G}$ , the generated masking feature matrices  $\forall i\in [p]$  can be expressed as  $\Phi_{i} = \mathbf{D}_{i}\mathbf{X}$  and the masking kernel matrices take the form  $\mathbf{K}_i = \mathbf{D}_i\mathbf{X}\mathbf{X}^T\mathbf{D}_i$ . In the context of the gated ReLU problem (7), since  $\mathcal{G}$  is complete, we can replace it with a minimally complete subset of  $\mathcal{G}$  without loss of generality since [21] showed that increasing  $m$  beyond  $p$  cannot reduce the value of the regularized training objective in (7). We are now ready to combine the MKL-group lasso equivalence with the convex reformulation of the gated ReLU network to present the following characterization of the nonconvex gated ReLU learning problem.
Theorem 4.3. The non-convex gated ReLU problem (7) with a minimally complete gate set  $\mathcal{G}$  is equivalent to performing multiple kernel learning (12) with the masking feature maps generated by  $\mathcal{G}$
$$
\min  _ {\boldsymbol {\eta} \in \Delta_ {p}, \mathbf {v} _ {i} \in \mathbb {R} ^ {d}} \quad \left\| \sum_ {i = 1} ^ {p} \sqrt {\eta_ {i}} \mathbf {D} _ {i} \mathbf {X} \mathbf {v} _ {i} - \mathbf {y} \right\| _ {2} ^ {2} + \hat {\lambda} \sum_ {i = 1} ^ {p} \| \mathbf {v} _ {i} \| _ {2} ^ {2}.
$$
This theorem implies that the gated ReLU network finds the optimal combination of linear models restricted to the different masked datasets  $\mathbf{D}_i\mathbf{X}$  generated by the gates. By optimizing with all possible data maskings, we obtain the best possible gated ReLU network. From the kernel perspective, we have characterized the problem of finding an optimal finite width gated ReLU network as learning a data-dependent kernel and then performing kernel regression. This is in contrast to the NTK theory where the training of an infinite width network by gradient flow is characterized by regression with a constant kernel that is not learned from data. We further explore this connection below.
5 NTK as a Weighted Masking Kernel
We now connect the NTK of a gated ReLU network with the masking kernels generated by its gates.
Theorem 5.1. Let  $\mathbf{K}_{\mathcal{G}}(\tilde{\boldsymbol{\eta}})\in \mathbb{R}^{n\times n}$  be the weighted masking kernel obtained by taking a convex combination of the masking feature maps generated by a minimally complete gate set  $\mathcal{G}$  with weights given by  $\tilde{\eta}_i = \mathbb{P}[\mathrm{diag}(\mathbb{1}\{\mathbf{X}\mathbf{h}\geq \mathbf{0}\}) = \mathbf{D}_i]$  where  $\mathbf{h}\sim \mathcal{N}(\mathbf{0},\mathbf{I}_{\mathbf{d}})$  and let  $\mathbf{H}\in \mathbb{R}^{n\times n}$  be the infinite width NTK of the gated ReLU network (10) evaluated on the training data, i.e., the  $ij^{th}$  entry of  $\mathbf{H}$  is defined as  $\mathbf{H}_{ij}\coloneqq H(\mathbf{x}_i,\mathbf{x}_j)$ . Then,  $\mathbf{K}_{\mathcal{G}}(\tilde{\boldsymbol{\eta}}) = \mathbf{H}$ .
A rough sketch of the proof of this theorem is to express the matrix  $\mathbf{H}$  as an expectation of indicator random variables using the definition of the NTK [1]. Then, by conditioning on the event that these indicators equal the masks  $\mathbf{D}_i$ , we can express the NTK as a convex combination of the masking kernels  $\mathbf{K}_i$ . The weights end up being precisely the probabilities that are described in Theorem 5.1. A detailed proof is provided in the supplementary material.
This theorem implies that the outputs of the gated ReLU network obtained via (16) with regularization weights  $\tilde{\eta}$  on the training data is identical to that of kernel ridge regression with the NTK.
Corollary 5.2. Let  $\tilde{\mathbf{w}}$  be the solution to (16) with feature matrices  $\Phi_{i} = \mathbf{D}_{i}\mathbf{X}$  and regularization weights  $\tilde{\eta}_i = \mathbb{P}[\mathrm{diag}(\mathbb{1}\{\mathbf{X}\mathbf{h}\geq \mathbf{0}\}) = \mathbf{D}_i]$  where  $\mathbf{h}\sim \mathcal{N}(\mathbf{0},\mathbf{I_d})$  and let  $\tilde{\mathbf{y}} = \mathbf{H}(\mathbf{H} + \lambda \mathbf{I_n})^{-1}\mathbf{y}$  be the outputs of kernel ridge regression with the NTK on the training data. Then,  $\sum_{i = 1}^{p}\mathbf{D}_{i}\mathbf{X}\tilde{\mathbf{w}}_{i} = \tilde{\mathbf{y}}$
Since  $\mathcal{G}$  is minimally complete, the weights in Theorem 5.1 satisfy  $\sum_{i=1}^{p} \tilde{\eta}_i = 1$ . In other words,  $\tilde{\eta} \in \Delta_p$ . Therefore, we can interpret Theorem 5.1 as follows - the NTK evaluated on the training data lies in the convex hull of all possible masking kernels of the training data. So  $\mathbf{H}$  lies in the feasible set of kernels for MKL using these masking kernels. By Theorem 4.3, we can find the optimal kernel in this set by solving the group lasso problem (8) of the gated ReLU network. Therefore, we can interpret solving (8) as fixing the NTK by learning an improved data-dependent kernel.
Remark 5.3 (Suboptimality of NTK). Note that the weights  $\tilde{\eta}$  do depend on the training data  $\mathbf{X}$ , but do not depend on the target labels  $\mathbf{y}$ . Since MKL learns the optimal kernel using both  $\mathbf{X}$  and  $\mathbf{y}$ , the NTK still cannot perform better than the optimal MKL kernel on the training set. Thus, we fix the NTK.
6 Analysis of Prediction Error
In this section, we present an analysis of the in-sample prediction error for the gated ReLU network given in (5) along the lines of existing consistency results [36, 37] for the group lasso problem (8).
Figure 1: Plot of objective value of problem (8) which is solved using IRLS (algorithm 1) for a toy 1D dataset with  $n = 5$ . The iterates are compared to the optimal value obtained by solving (8) using CVXPY (blue). Notice that the solution to (16) with regularization weights given by the NTK weights  $\tilde{\eta}$  from Theorem 5.1 (green) is sub-optimal for problem (8), and running IRLS by initializing with these weights (red) converges to the optimal objective value. We also include plots of IRLS initialized with random weights (black). The right plot shows the corresponding learned functions. The blue curve shows the output of the solution to the group lasso problem (8) after performing a cone decomposition to obtain a ReLU network. The dashed curve shows the output of running gradient descent (GD) on a ReLU network with 100 neurons. The green curve is the result of kernel ridge regression (KRR) with the NTK. We observe that our reweighted kernel method (Group Lasso) produces an output that matches the output of the NN trained via GD. In contrast, NTK produces an erroneous smooth function due to the infinite width approximation.
We assume that the data is generated by a noisy ReLU neural network model  $\mathbf{y} = f(\mathbf{X},\boldsymbol{\theta}^{*}) + \epsilon$  where  $f$  is the ReLU network defined in (1) with true parameters  $\pmb{\theta}^{*}$  and the noise is distributed as  $\epsilon \sim \mathcal{N}(0,\sigma^2\mathbf{I}_n)$ . By the seminal universal approximation theorem of [38], this model is able to capture a broad class of ground-truth functions by using a ReLU network with enough neurons. We can transform  $\pmb{\theta}^{*}$  to the weights  $\mathbf{w}^{*}$  in the convex ReLU model and write  $f(\mathbf{X},\pmb{\theta}^{*}) = \sum_{i=1}^{p}\mathbf{D}_{i}\mathbf{X}\mathbf{w}_{i}^{*}$ . We denote by  $\hat{\mathbf{w}}$  the solution of the group lasso problem (8) for the gated ReLU network with an additional  $\frac{1}{n}$  factor on the loss to simplify derivations,
$$
\hat {\mathbf {w}} = \underset {\mathbf {w} _ {i} \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \quad \frac {1}{n} \left\| \sum_ {i = 1} ^ {p} \mathbf {D} _ {i} \mathbf {X} \mathbf {w} _ {i} - \mathbf {y} \right\| _ {2} ^ {2} + \lambda \sum_ {i = 1} ^ {p} \| \mathbf {w} _ {i} \| _ {2}. \tag {17}
$$
We now present our main theorem which bounds the prediction error of the gated ReLU network obtained from the solution  $\hat{\mathbf{w}}$  below.
Theorem 6.1 (Prediction Error of Gated ReLU). For some  $t > 0$ , let the regularization parameter in (17) be  $\lambda = t\sigma \| \mathbf{X}\| _F / n$ . Then, with probability at least  $1 - 2e^{-t^2 /8}$ , we have
$$
\frac {1}{n} \left\| f _ {\mathcal {G}} (\mathbf {X}, \hat {\boldsymbol {\theta}}) - f (\mathbf {X}, \boldsymbol {\theta} ^ {*}) \right\| _ {2} ^ {2} \leq 2 \lambda \sum_ {i = 1} ^ {p} \| \mathbf {w} _ {i} ^ {*} \| _ {2}
$$
where  $f_{\mathcal{G}}(\mathbf{X},\hat{\pmb{\theta}}) = \sum_{i = 1}^{p}\mathbf{D}_{i}\mathbf{X}\hat{\mathbf{w}}_{i}$  are the predictions of the gated ReLU network obtained from  $\hat{\mathbf{w}}$
The proof closely follows the analysis of the regular lasso problem presented in [37], but we extend it to the specific case of the group lasso problem corresponding to the gated ReLU network and leverage the masking structure of the lifted data matrix to obtain simplified bounds.
Here, we empirically corroborate our theoretical results via experiments on several datasets.2
2We provide additional details in Section B.
Figure 2: Plot of the group lasso objective in the student-teacher setting with  $d = 5$ . Training data is generated using a teacher network with width  $m = 10$ . The NTK weights  $\tilde{\eta}$  are estimated using Monte Carlo sampling, and are again sub-optimal. IRLS initialized with these weights successfully fixes the NTK and converges to the optimal weights.
1D datasets. For the 1D experiments in Figure 1, we add a second data dimension with value equal to 1 for all data points to simulate a bias term in the first layer of the gated ReLU network. Also, in this case we can enumerate all  $2n$  data masks  $\mathbf{D}_i$  directly. We use the cone decomposition procedure described in [8] to obtain a ReLU network from the gated ReLU network obtained by solving (8). We also train a 100 neuron ReLU network using gradient descent (GD) and compare the learned output functions in the right plot in Figure 1. Exact details can be found in the supplementary material.
Student-teacher setting. We generate the training data by sampling  $\mathbf{X} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_{\mathbf{n}})$  and computing the targets  $\mathbf{y}$  using a fixed, randomly initialized gated ReLU teacher network. In Figure 2,  $n = 10$ ,  $d = 5$ , and we use a teacher network with width  $m = 10$ , with gates and parameters randomly drawn from independent standard multivariate Gaussians. To solve the convex formulation (8), we estimate  $\mathcal{D}_{\mathbf{X}}$  by randomly sampling unique hyperplane arrangements.
Table 1: Test accuracies for UCI experiments with  $75\% -25\%$  training-test split. Our approach achieves either higher or the same accuracy for 26 out of 33 datasets.
Dataset	n	d	NTK	Ours (Alg 1)
acute-inflammation	120	6	1.000	1.000
acute-nephritis	120	6	1.000	1.000
balloons	16	4	0.75	0.75
blood	748	4	0.524	0.583
breast-cancer	286	9	0.417	0.625
breast-cancer-wisc-prog	699	9	0.96	0.966
breast-cancer-wisc-diag	569	30	0.965	0.915
breast-cancer-wisc-prog	198	33	0.7	0.66
congressional-voting	435	16	0.266	0.266
conn-bench-sonar-mines-rocks	208	60	0.635	0.712
credit-approval	690	15	0.838	0.844
cylinder-bands	512	35	0.773	0.82
echocardiogram	131	10	0.758	0.788
fertility	100	9	0.76	0.76
haberman-survival	306	3	0.481	0.532
heart-hungarian	294	12	0.743	0.878
hepatitis	155	19	0.923	0.897
ilpd-indian-liver	583	9	0.432	0.555
ionosphere	351	33	0.955	0.966
mammographic	961	5	0.783	0.792
molec-biol-promoter	106	57	0.63	0.815
musk-1	476	166	0.782	0.866
oocytes_trisopterus_nucleus_2f	912	25	0.781	0.759
parkinsons	195	22	0.939	0.959
pima	768	8	0.552	0.599
pittsburg-bridges-T-OR-D	102	7	0.731	0.846
planning	182	12	0.435	0.543
statlog-australian-credit	690	14	1.0	0.74
statlog-german-credit	1000	24	0.512	0.576
statlog-heart	270	13	0.779	0.765
tic-tac-toe	958	9	1.0	1.0
trains	10	29	0.667	0.667
vertebral-column-2classes	310	6	0.821	0.731
Higher (or same) accuracy 14/33 26/33
Computing the NTK weights  $\tilde{\eta}$ . The weights induced by the NTK are
given as in Theorem 5.1 by  $\tilde{\eta}_i = \mathbb{P}[\mathrm{diag}\left(\mathbb{1}\left\{\mathbf{X}\mathbf{h}\geq \mathbf{0}\right\}\right) = \mathbf{D}_i]$  where  $\mathbf{h}\sim \mathcal{N}(\mathbf{0},\mathbf{I}_{\mathbf{d}})$  is a standard multivariate Gaussian vector. These probabilities can be interpreted either as the orthant probabilities of the multivariate Gaussians given by  $(2\mathbf{D}_i - \mathbf{I}_{\mathbf{n}})\mathbf{X}\mathbf{h}$  or as the solid angle of the cones given by  $\{\mathbf{u}\in \mathbb{R}^d:(2\mathbf{D}_i - \mathbf{I}_{\mathbf{n}})\mathbf{X}\mathbf{u}\geq \mathbf{0}\}$ . Closed form expressions exist for  $d = 2,3$ , and [39, 40] present approximating schemes for higher dimensions. We calculate these weights exactly for the 1D example presented in Figure 1, and estimate them using Monte Carlo sampling for the student-teacher example in Figure 2.
Fixing the NTK weights by IRLS. We solve (16) with  $\Phi_i = \mathbf{D}_i\mathbf{X}$  and regularization weights given by the NTK weights  $\tilde{\eta}$  to obtain the solution  $\tilde{\mathbf{w}}$ . We use efficient least squares solvers from [41, 42]. By Theorem 5.1, this corresponds to choosing the weights  $\eta$  in the MKL problem (13) such that the resulting kernel matrix is equal to the NTK matrix  $\mathbf{H}$ . We find the exact solution of the group lasso problem (8) using CVXPY. Comparing the optimal value of the group lasso problem (8) with the objective value of  $\tilde{\mathbf{w}}$  (given by the green line) in Figures 1 and 2, we observe that the NTK weighted solution is sub-optimal. This means that  $\mathbf{H}$  is not the optimal kernel that would be learnt by MKL (which is expected since  $\mathbf{H}$  has no dependence on the targets  $\mathbf{y}$ ). By applying IRLS initialized with the NTK weights, we fix the NTK and find the weights of the optimal MKL kernel. In Figures 1 and 2, we observe that IRLS converges to the solution of the group lasso problem (8) and fixes the NTK.
UCI datasets. We compare the regularized NTK with our IRLS algorithm (Algorithm 1) on the UCI ML Repository datasets. We follow the procedure described in [43] for  $n \leq 1000$  to extract and standardize the datasets. We observe that our method achieves higher (or the same) test accuracy for 26 out of 33 datasets (see Table 1 for details) while the NTK achieves higher (or the same) test accuracy for 14 datasets which empirically supports our main claim that the IRLS procedure fixes the NTK. Details of the experiments can be found in Section B of the supplementary material.
8 Discussion and Limitations
In this work, we explored the connection between finite-width theories of neural networks given by convex reformulations and infinite-width theories of neural networks given by the NTK. To bridge these theories, we first interpreted the group lasso convex formulation of the gated ReLU network as a multiple kernel learning model using the masking kernels generated by its gates. Then, we linked this MKL model with the NTK of the gated ReLU network evaluated on the training data. Specifically, we showed that the NTK is equivalent to the weighted masking kernel with weights that depend only on the input data  $\mathbf{X}$  and not on the targets  $\mathbf{y}$ . We contrast this with the MKL interpretation of the gated ReLU network which learns the optimal data-dependent kernel using both  $\mathbf{X}$  and  $\mathbf{y}$ . Therefore, the NTK cannot perform better than the optimal MKL kernel. To fix the NTK, we improve the weights induced by it using the iteratively reweighted least squares (IRLS) scheme to obtain the optimal solution of the group lasso formulation of the gated ReLU network. We corroborated our theoretical results by empirically running IRLS on toy datasets.
While our theory is able to link the optimization properties of the NTK with those of finite width networks via the MKL characterization of group lasso, we do not derive explicit generalization results on the test set. Applying existing generalization theory for kernel methods [44, 45] to the MKL interpretation of the convex reformulation could be a promising direction for future work.
Finally, although we studied fully connected networks in this paper, our approach can be directly extended to various neural network architectures, e.g., threshold/binary networks [46], convolution networks [47], generative adversarial networks [48], NNs with batch normalization [49], autoregressive models [50], and Transformers [51, 52].
This work was supported in part by the National Science Foundation (NSF) CAREER Award under Grant CCF-2236829, Grant DMS-2134248 and Grant ECCS-2037304; in part by the U.S. Army Research Office Early Career Award under Grant W911NF-21-1-0242; in part by the Stanford Precourt Institute; and in part by the ACCESS—AI Chip Center for Emerging Smart Systems through InnoHK, Hong Kong, SAR.
A Proofs and Theoretical Results 14
A.1 Proof of Lemma 2.2 14
A.2 Proof of Lemma 2.3 15
A.3 Proof of Lemma 4.2 15
A.4 Proof of Theorem 4.3 16
A.5 Proof of Theorem 5.1 16
A.6 Proof of Corollary 5.2 16
A.7 Proof of Theorem 6.1 17
B Experimental Details 18
C Generic Loss Functions 19
D Extensions to Deeper Networks 20
E Cone Decomposition 20
E.1 Computational Complexity 21
A Proofs and Theoretical Results
A.1 Proof of Lemma 2.2
The gradient feature map of (9) is
$$
\nabla_ {\boldsymbol {\theta}} \tilde {f} _ {\mathcal {G}} \left(\mathbf {x}, \boldsymbol {\theta}\right) = \frac {1}{\sqrt {2 m}} \left[ \begin{array}{c} \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} _ {1} \geq 0 \right\} \mathbf {x} ^ {T} \mathbf {w} _ {1} ^ {(1)} \\ \vdots \\ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} _ {m} \geq 0 \right\} \mathbf {x} ^ {T} \mathbf {w} _ {m} ^ {(1)} \\ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} _ {1} \geq 0 \right\} w _ {1} ^ {(2)} \mathbf {x} \\ \vdots \\ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} _ {m} \geq 0 \right\} w _ {m} ^ {(2)} \mathbf {x} \end{array} \right].
$$
The finite width random NTK given by  $H_{m,\pmb{\theta},\mathcal{G}}(\mathbf{x},\mathbf{x}^{\prime})\coloneqq \nabla_{\pmb{\theta}}\tilde{f}_{\mathcal{G}}(\mathbf{x},\pmb{\theta})^{T}\nabla_{\pmb{\theta}}\tilde{f}_{\mathcal{G}}(\mathbf{x}^{\prime},\pmb{\theta})$  is
$$
H _ {m, \boldsymbol {\theta}, \mathcal {G}} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \frac {1}{2 m} \sum_ {j = 1} ^ {m} \left(\mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} _ {j} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {g} _ {j} \geq 0 \right\} \left(\mathbf {x} ^ {T} \mathbf {w} _ {j} ^ {(1)} \mathbf {x} ^ {\prime T} \mathbf {w} _ {j} ^ {(1)} + \left(w _ {j} ^ {(2)}\right) ^ {2} \mathbf {x} ^ {T} \mathbf {x} ^ {\prime}\right)\right).
$$
By the Law of Large numbers since  $\mathbf{g}_j, \mathbf{w}_j^{(1)}, w_j^{(2)}$  are iid,  $H_{m,\theta,\mathcal{G}}(\mathbf{x},\mathbf{x}')$  converges in probability to the expectation of the quantity in the sum as  $m \to \infty$ . We denote this infinite limit by  $H(\mathbf{x},\mathbf{x}')$ . Therefore,
$$
H \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \frac {1}{2} \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {g} \geq 0 \right\} \left(\mathbf {x} ^ {T} \mathbf {w} ^ {(1)} \mathbf {x} ^ {\prime T} \mathbf {w} ^ {(1)} + \left(w ^ {(2)}\right) ^ {2} \mathbf {x} ^ {T} \mathbf {x} ^ {\prime}\right) \right],
$$
where  $\mathbf{g} \sim \mathcal{N}(\mathbf{0},\mathbf{I}_{\mathbf{d}})$ ,  $\mathbf{w}^{(1)} \sim \mathcal{N}(\mathbf{0},\mathbf{I}_{\mathbf{d}})$ ,  $w^{(2)} \sim \mathcal{N}(0,1)$ . Using  $\mathbb{E}\left[\left(w^{(2)}\right)^2\right] = 1$ ,  $\mathbb{E}\left[\mathbf{x}^T\mathbf{w}^{(1)}\mathbf{x}^{\prime T}\mathbf{w}^{(1)}\right] = \mathbf{x}^T\mathbf{x}'$  and the linearity of expectation, we get
$$
H \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \mathbf {x} ^ {T} \mathbf {x} ^ {\prime} \cdot \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {g} \geq 0 \right\} \right]. \tag {18}
$$
To evaluate the expectation of the product of the indicator random variables, we condition on one of the indicators being equal to 1.
$$
\begin{array}{l} \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} \geq 0 \right\} \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {g} \geq 0 \right\} \right] = \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} \geq 0 \right\} \mid \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {g} \geq 0 \right\} = 1 \right] \cdot \mathbb {P} \left[ \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {g} \geq 0 \right\} = 1 \right] \\ = \frac {1}{2} \cdot \mathbb {E} \left[ \mathbb {1} \left\{\mathbf {x} ^ {T} \mathbf {g} \geq 0 \right\} \mid \mathbb {1} \left\{\mathbf {x} ^ {\prime T} \mathbf {g} \geq 0 \right\} = 1 \right]. \\ \end{array}
$$