SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Princeton University eshnich@princeton.edu
Princeton University ad27@princeton.edu
Princeton University jasonlee@princeton.edu
One of the central questions in the theory of deep learning is to understand how neural networks learn hierarchical features. The ability of deep networks to extract salient features is crucial to both their outstanding generalization ability and the modern deep learning paradigm of pretraining and finetuning. However, this feature learning process remains poorly understood from a theoretical perspective, with existing analyses largely restricted to two-layer networks. In this work we show that three-layer neural networks have provably richer feature learning capabilities than two-layer networks. We analyze the features learned by a three-layer network trained with layer-wise gradient descent, and present a general purpose theorem which upper bounds the sample complexity and width needed to achieve low test error when the target has specific hierarchical structure. We instantiate our framework in specific statistical learning settings – single-index models and functions of quadratic features – and show that in the latter setting three-layer networks obtain a sample complexity improvement over all existing guarantees for two-layer networks. Crucially, this sample complexity improvement relies on the ability of three-layer networks to efficiently learn nonlinear features. We then establish a concrete optimization-based depth separation by constructing a function which is efficiently learnable via gradient descent on a three-layer network, yet cannot be learned efficiently by a two-layer network. Our work makes progress towards understanding the provable benefit of three-layer neural networks over two-layer networks in the feature learning regime.
The success of modern deep learning can largely be attributed to the ability of deep neural networks to decompose the target function into a hierarchy of learned features. This feature learning process enables both improved accuracy [29] and transfer learning [21]. Despite its importance, we still have a rudimentary theoretical understanding of the feature learning process. Fundamental questions include understanding what features are learned, how they are learned, and how they affect generalization.
From a theoretical viewpoint, a fascinating question is to understand how depth can be leveraged to learn more salient features and as a consequence a richer class of hierarchical functions. The base case for this question is to understand which features (and function classes) can be learned efficiently by three-layer neural networks, but not two-layer networks. Recent work on feature learning has shown that two-layer neural networks learn features which are linear functions of the input (see Section 1.2 for further discussion). It is thus a natural question to understand if three-layer networks can learn nonlinear features, and how this can be leveraged to obtain a sample complexity improvement. Initial learning guarantees for three-layer networks [16, 5, 4], however, consider simplified model and function classes and do not discern specifically what the learned features are or whether a sample complexity improvement can be obtained over shallower networks or kernel methods.
On the other hand, the standard approach in deep learning theory to understand the benefit of depth has been to establish "depth separations" [51], i.e. functions that cannot be efficiently approximated by shallow networks, but can be via deeper networks. However, depth separations are solely concerned with the representational capability of neural networks, and ignore the optimization and generalization aspects. In fact, depth separation functions such as [51] are often not learnable via gradient descent [36]. To reconcile this, recent papers [45, 44] have established optimization-based depth separation results, which are functions which cannot be efficiently learned using gradient descent on a shallow network but can be learned with a deeper network. We thus aim to answer the following question:
What features are learned by gradient descent on a three-layer neural network, and can these features be leveraged to obtain a provable sample complexity guarantee?
We provide theoretical evidence that three-layer neural networks have provably richer feature learning capabilities than their two-layer counterparts. We specifically study the features learned by a three-layer network trained with a layer-wise variant of gradient descent (Algorithm 1). Our main contributions are as follows.
- Theorem 1 is a general purpose sample complexity guarantee for Algorithm 1 to learn an arbitrary target function  $f^{*}$ . We first show that Algorithm 1 learns a feature roughly corresponding to a low-frequency component of the target function  $f^{*}$  with respect to the random feature kernel  $\mathbb{K}$  induced by the first layer. We then derive an upper bound on the population loss in terms of the learned feature. As a consequence, we show that if  $f^{*}$  possesses a hierarchical structure where it can be written as a 1D function of the learned feature (detailed in Section 3), then the sample complexity for learning  $f^{*}$  is equal to the sample complexity of learning the feature. This demonstrates that three-layer networks indeed perform hierarchical learning.
- We next instantiate Theorem 1 in two statistical learning settings which satisfy such hierarchical structure. As a warmup, we show that Algorithm 1 learns single-index models (i.e  $f^{*}(x) = g^{*}(w \cdot x)$ ) in  $d^{2}$  samples, which is comparable to existing guarantees for two-layer networks and crucially has  $d$ -dependence not scaling with the degree of the link function  $g^{*}$ . We next show that Algorithm 1 learns the target  $f^{*}(x) = g^{*}(x^{T}Ax)$ , where  $g^{*}$  is either Lipschitz or a degree  $p = O(1)$  polynomial, up to  $o_{d}(1)$  error with  $d^{4}$  samples. This improves on all existing guarantees for learning with two-layer networks or via NTK-based approaches, which all require sample complexity  $d^{\Omega(p)}$ . A key technical step is to show that for the target  $f^{*}(x) = g^{*}(x^{T}Ax)$ , the learned feature is approximately  $x^{T}Ax$ . This argument relies on the universality principle in high-dimensional probability, and may be of independent interest.
- We conclude by establishing an explicit optimization-based depth separation: We show that the target function  $f^{*}(x) = \mathrm{ReLU}(x^{T}Ax)$  for appropriately chosen  $A$  can be learned by Algorithm 1 up to  $o_{d}(1)$  error in  $d^{4}$  samples, whereas any two layer network needs either superpolynomial width or weight norm in order to approximate  $f^{*}$  up to comparable accuracy. This implies that such an  $f^{*}$  is not efficiently learnable via two-layer networks.
The above separation hinges on the ability of three-layer networks to learn the nonlinear feature  $x^{T}Ax$  and leverage this feature learning to obtain an improved sample complexity. Altogether, our work presents a general framework demonstrating the capability of three-layer networks to learn nonlinear features, and makes progress towards a rigorous understanding of feature learning, optimization-based depth separations, and the role of depth in deep learning more generally.
Neural Networks and Kernel Methods. Early guarantees for neural networks relied on the Neural Tangent Kernel (NTK) theory [31, 50, 23, 17]. The NTK theory shows global convergence by coupling to a kernel regression problem and generalization via the application of kernel generalization bounds [7, 15, 5]. The NTK can be characterized explicitly for certain data distributions [27, 39, 38], which allows for tight sample complexity and width analyses. This connection to kernel methods has also been used to study the role of depth, by analyzing the signal propagation and evolution of
the NTK in MLPs [42, 48, 28], convolutional networks [6, 55, 56, 40], and residual networks [30]. However, the NTK theory is insufficient as neural networks outperform their NTK in practice [6, 32]. In fact, [27] shows that kernels cannot adapt to low-dimensional structure and require  $d^k$  samples to learn any degree  $k$  polynomials in  $d$  dimensions. Ultimately, the NTK theory fails to explain generalization or the role of depth in practical networks not in the kernel regime. A key distinction is that networks in the kernel regime cannot learn features [57]. A recent goal has thus been to understand the feature learning mechanism and how this leads to sample complexity improvements [53, 22, 58, 3, 25, 26, 20, 54, 33, 35]. Crucially, our analysis is not in the kernel regime, and shows an improvement of three-layer networks over two-layer networks in the feature-learning regime.
Feature Learning. Recent work has studied the provable feature learning capabilities of two-layer neural networks. [9, 1, 2, 8, 13, 10] show that for isotropic data distributions, two-layer networks learn linear features of the data, and thus efficiently learn functions of low-dimensional projections of the input (i.e. targets of the form  $f^{*}(x) = g(Ux)$  for  $U \in \mathbb{R}^{r \times d}$ ). Here,  $x \mapsto Ux$  is the "linear feature." Such target functions include low-rank polynomials [18, 2] and single-index models [8, 13] for Gaussian covariates, as well as sparse boolean functions [1] such as the  $k$ -sparse parity problem [10] for covariates uniform on the hypercube. [43] draws connections from the mechanisms in these works to feature learning in standard image classification settings. The above approaches rely on layerwise training procedures, and our Algorithm 1 is an adaptation of the algorithm in [18].
Another approach uses the quadratic Taylor expansion of the network to learn classes of polynomials [9, 41] This approach can be extended to three-layer networks. [16] replace the outermost layer with its quadratic approximation, and by viewing  $z^p$  as the hierarchical function  $(z^{p/2})^2$  show that their three-layer network can learn low rank, degree  $p$  polynomials in  $d^{p/2}$  samples. [5] similarly uses a quadratic approximation to improperly learn a class of three-layer networks via sign-randomized GD. An instantiation of their upper bound to the target  $g^*(x^T Ax)$  for degree  $p$  polynomial  $g^*$  yields a sample complexity of  $d^{p+1}$ . However, [16, 5] are proved via opaque landscape analyses, do not concretely identify the learned features, and rely on nonstandard algorithmic modifications. Our Theorem 1 directly identifies the learned features, and when applied to the quadratic feature setting in Section 4.2 obtains an improved sample complexity guarantee independent of the degree of  $g^*$ .
Depth Separations. [51] constructs a function which can be approximated by a poly-width network with large depth, but not with smaller depth. [24] is the first depth separation between depth 2 and 3 networks, with later works [46, 19, 47] constructing additional such examples. However, such functions are often not learnable via three-layer networks [34]. [36] shows that approximatability by a shallow (depth 3 network) is a necessary condition for learnability via a deeper network.
These issues have motivated the development of optimization-based, or algorithmic, depth separations, which construct functions which are learnable by a three-layer network but not by two-layer networks. [45] shows that certain ball indicator functions  $\mathbf{1}(\| x\| \geq \lambda)$  are not approximatable by two-layer networks, yet are learnable via GD on a special variant of a three-layer network with second layer width equal to 1. However, their network architecture is tailored for learning the ball indicator, and the explicit polynomial sample complexity  $(n\gtrsim d^{36})$  is weak. [44] shows that a multi-layer mean-field network with a 1D bottleneck layer can learn the target ReLU  $(1 - \| x\|)$ , which [47] previously showed was inapproximatable via two-layer networks. However, their analysis relies on the rotational invariance of the target function, and it is difficult to read off explicit sample complexity and width guarantees beyond being poly  $(d)$ . Our Section 4.2 shows that three-layer networks can learn a larger class of features  $(x^T Ax$  versus  $\| x\|)$  and functions on top of these features (any Lipschitz  $q$  versus ReLU), with explicit dependence on the width and sample complexity needed  $(n,m_1,m_2 = \tilde{O} (d^4))$ .
Data distribution. Our aim is to learn the target function  $f^{*}:\mathcal{X}_{d}\to \mathbb{R}$ , with  $\mathcal{X}_d\subset \mathbb{R}^d$  the space of covariates. We let  $\nu$  be some distribution on  $\mathcal{X}_d$ , and draw two independent datasets  $\mathcal{D}_1,\mathcal{D}_2$ , each with  $n$  samples, so that each  $x\in \mathcal{D}_1$  or  $x\in \mathcal{D}_2$  is sampled i.i.d as  $x\sim \nu$ . Without loss of generality, we normalize so  $\mathbb{E}_{x\sim \nu}[f^{*}(x)^{2}]\leq 1$ . We make the following assumptions on  $\nu$ :
Algorithm 1 Layer-wise training algorithm
Input: Initialization  $\theta^{(0)}$ ; learning rates  $\eta_{1}, \eta_{2}$ ; weight decay  $\lambda$ ; time  $T$
{Stage 1: Train  $W$  }
$$
W ^ {(1)} \leftarrow W ^ {(0)} - \eta_ {1} \nabla_ {W} L _ {1} \left(\theta^ {(0)}\right)
$$
$$
a ^ {(1)} \leftarrow a ^ {(0)}
$$
$$
\theta^ {(1)} \leftarrow \left(a ^ {(1)}, W ^ {(1)}, b ^ {(0)}, V ^ {(0)}\right)
$$
{Stage 2: Train  $a$  }
for  $\bar{t} = 2,\dots ,T$  do
$$
a ^ {(t)} \leftarrow a ^ {(t - 1)} - \eta_ {2} \left[ \nabla_ {a} L _ {2} \left(\theta^ {(t - 1)}\right) + \lambda a ^ {(t - 1)} \right]
$$
$$
\theta^ {(t)} \leftarrow \left(a ^ {(t)}, W ^ {(1)}, b ^ {(0)}, V ^ {(0)}\right)
$$
$$
\hat {\theta} \leftarrow \theta^ {(T)}
$$
Output:  $\hat{\theta}$
Definition 1 (Sub-Gaussian Vector). A mean-zero random vector  $X \in \mathbb{R}^d$  is  $\gamma$ -subGaussian if, for all unit vectors  $v \in \mathbb{R}^d$ ,  $\mathbb{E}[\exp(\lambda X \cdot v)] \leq \exp(\gamma^2 \lambda^2)$  for all  $\lambda \in \mathbb{R}$ .
Assumption 1.  $\mathbb{E}_{x\sim \nu}[x] = 0$  and  $\nu$  is  $C_\gamma$ -subGaussian for some constant  $C_\gamma$ .
Assumption 2.  $f^{*}$  has polynomially growing moments, i.e there exist constants  $(C_f, \ell)$  such that  $\mathbb{E}_{x \sim \nu}[f^{*}(x)^{q}]^{1/q} \leq C_{f} q^{\ell}$  for all  $q \geq 1$ .
We note that Assumption 2 is satisfied by a number of common distributions and functions, and we will verify that Assumption 2 holds for each example in Section 4.
Three-layer neural network. Let  $m_{1}, m_{2}$  be the two hidden layer widths, and  $\sigma_{1}, \sigma_{2}$  be two activation functions. Our learner is a three-layer neural network parameterized by  $\theta = (a, W, b, V)$ , where  $a \in \mathbb{R}^{m_1}$ ,  $W \in \mathbb{R}^{m_1 \times m_2}$ ,  $b \in \mathbb{R}^{m_1}$ , and  $V \in \mathbb{R}^{m_2 \times d}$ . The network  $f(x; \theta)$  is defined as:
$$
f (x; \theta) := \frac {1}{m _ {1}} a ^ {T} \sigma_ {1} \left(W \sigma_ {2} (V x) + b\right) = \frac {1}{m _ {1}} \sum_ {i = 1} ^ {m _ {1}} a _ {i} \sigma_ {1} \left(\langle w _ {i}, h ^ {(0)} (x) \rangle + b _ {i}\right). \tag {1}
$$
Here,  $w_{i} \in \mathbb{R}^{m_{2}}$  is the  $i$ th row of  $W$ , and  $h^{(0)}(x) \coloneqq \sigma_{2}(Vx) \in \mathbb{R}^{m_{2}}$  is the random feature embedding arising from the innermost layer. The parameter vector  $\theta^{(0)} \coloneqq (a^{(0)}, W^{(0)}, b^{(0)}, V^{(0)})$  is initialized with  $a_{i}^{(0)} \sim_{iid} \mathrm{Unif}(\{\pm 1\})$ ,  $W^{(0)} = 0$ , the biases  $b_{i}^{(0)} \sim_{iid} \mathcal{N}(0,1)$ , and the rows  $v_{i}^{(0)}$  of  $V^{(0)}$  drawn  $v_{i} \sim_{iid} \tau$ , where  $\tau$  is the uniform measure on  $\mathcal{S}^{d-1}(1)$ , the  $d$ -dimensional unit sphere. We make the following assumption on the activations, and note that the polynomial growth assumption on  $\sigma_{2}$  is satisfied by all activations used in practice.
Assumption 3.  $\sigma_{1}$  is the ReLU activation, i.e  $\sigma_{1}(z) = \max (z,0)$ , and  $\sigma_{2}$  has polynomial growth, i.e  $|\sigma_2(x)|\leq C_\sigma (1 + |x|)^{\alpha_\sigma}$  for some constants  $C_\sigma ,\alpha_\sigma >0$
Training Algorithm. Let  $L_{i}(\theta)$  denote the empirical loss on dataset  $\mathcal{D}_i$ ; that is for  $i = 1,2$ :  $L_{i}(\theta) := \frac{1}{n}\sum_{x\in \mathcal{D}_{i}}(f(x;\theta) - f^{*}(x))^{2}$ . Our network is trained via layer-wise gradient descent with sample splitting. Throughout training, the first layer weights  $V$  and second layer bias  $b$  are held constant. First, the second layer weights  $W$  are trained for  $t = 1$  timesteps. Next, the outer layer weights  $a$  are trained for  $t = T - 1$  timesteps. This two stage training process is common in prior works analyzing gradient descent on two-layer networks [18, 8, 1, 10], and as we see in Section 5, is already sufficient to establish a separation between two and three-layer networks. Pseudocode for the training procedure is presented in Algorithm 1.
2.2 Technical definitions
The activation  $\sigma_{2}$  admits a random feature kernel  $K:\mathcal{X}_d\times \mathcal{X}_d\to \mathbb{R}$  and corresponding integral operator  $\mathbb{K}:L^2 (\mathcal{X}_d,\nu)\to L^2 (\mathcal{X}_d,\nu)$ :
Definition 2 (Kernel objects).  $\sigma_{2}$  admits the random feature kernel
$$
K (x, x ^ {\prime}) := \mathbb {E} _ {v \sim \tau} [ \sigma_ {2} (x \cdot v) \sigma_ {2} \left(x ^ {\prime} \cdot v\right) ] \tag {2}
$$
and corresponding integral operator
$$
(\mathbb {K} f) (x) := \mathbb {E} _ {x ^ {\prime} \sim \nu} [ K (x, x ^ {\prime}) f (x ^ {\prime}) ]. \tag {3}
$$
We make the following assumption on  $\mathbb{K}$ , which we verify for the examples in Section 4:
Assumption 4.  $\mathbb{K}f^{*}$  has polynomially bounded moments, i.e there exist constants  $C_K, \chi$  such that, for all  $1 \leq q \leq d$ ,  $\| \mathbb{K}f^{*}\|_{L^q(\nu)} \leq C_K q^\chi \| \mathbb{K}f^{*}\|_{L^2(\nu)}$ .
We also require the definition of the Sobolev space:
Definition 3. Let  $\mathcal{W}^{2,\infty}([-1,1])$  be the Sobolev space of twice continuously differentiable functions  $q:[-1,1]\to \mathbb{R}$  equipped with the norm  $\| q\|_{k,\infty}\coloneqq \max_{s\leq k}\max_{x\in [-1,1]}\left|q^{(s)}(x)\right|$  for  $k = 1,2$
2.3 Notation
We use big  $O$  notation (i.e  $O, \Theta, \Omega$ ) to ignore absolute constants ( $C_{\sigma}, C_f$ , etc.) that do not depend on  $d, n, m_1, m_2$ . We further write  $a_d \lesssim b_d$  if  $a_d = O(b_d)$ , and  $a_d = o(b_d)$  if  $\lim_{d \to \infty} a_d / b_d = 0$ . Additionally, we use  $\tilde{O}$  notation to ignore terms that depend logarithmically on  $dnm_1m_2$ . For  $f: \mathcal{X}_d \to \mathbb{R}$ , define  $\|f\|_{L^p(\mathcal{X}_d, \nu)} = (\mathbb{E}_{x \sim \nu}[f(x)^p])^{1/p}$ . To simplify notation we also call this quantity  $\|f\|_{L^p(\nu)}$ , and  $\|g\|_{L^p(\mathcal{X}_d, \nu)}$ ,  $\|g\|_{L^p(\tau)}$  are defined analogously for functions  $g: S^{d-1}(1) \to \mathbb{R}$ . When the domain is clear from context, we write  $\|f\|_{L^p}$ ,  $\|g\|_{L^p}$ . We let  $L^p(\mathcal{X}_d, \nu)$  be the space of  $f$  with finite  $\|f\|_{L^p(\mathcal{X}_d, \nu)}$ . Finally, we write  $\mathbb{E}_x$  and  $\mathbb{E}_v$  as shorthand for  $\mathbb{E}_{x \sim \nu}$  and  $\mathbb{E}_{v \sim \tau}$  respectively.
The following is our main theorem which upper bounds the population loss of Algorithm 1:
Theorem 1. Select  $q \in \mathcal{W}^{2,\infty}([-1,1])$ . Let  $\eta_1 = \frac{m_1}{m_2}\overline{\eta}$ , and assume  $n, m_1, m_2 = \tilde{\Omega}(\| \mathbb{K}f^*\|_{L^2}^{-2})$  There exist  $\overline{\eta}, \lambda, \eta_2$  such that after  $T = \mathrm{poly}(n, m_1, m_2, d, \| q\|_{2,\infty})$  timesteps, with high probability over the initialization and datasets the output  $\hat{\theta}$  of Algorithm 1 satisfies the population  $L^2$  loss bound
$$
\begin{array}{l} \mathbb {E} _ {x} \left[ \left(f (x; \hat {\theta}) - f ^ {*} (x)\right) ^ {2} \right] \\ \leq \tilde {O} \left(\underbrace {\| q \circ (\bar {\eta} \cdot \mathbb {K} f ^ {*}) - f ^ {*} \| _ {L ^ {2}} ^ {2}} _ {\text {a c c u r a c y o f}} + \underbrace {\frac {\| q \| _ {1 , \infty} ^ {2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 2}}{\operatorname* {m i n} (n , m _ {1} , m _ {2})}} _ {\text {s a m p l e c o m p l e x i t y o f}} + \underbrace {\frac {\| q \| _ {2 , \infty} ^ {2}}{m _ {1}} + \frac {\| q \| _ {2 , \infty} ^ {2} + 1}{\sqrt {n}}} _ {\text {c o m p l e x i t y o f} q}\right) \tag {4} \\ \end{array}
$$
The full proof of this theorem is in Appendix D. The population risk upper bound (4) has three terms:
1. The first term quantifies the extent to which feature learning is useful for learning the target  $f^{*}$ , and depends on how close  $f^{*}$  is to having hierarchical structure. Concretely, if there exists  $q: \mathbb{R} \to \mathbb{R}$  such that the compositional function  $q \circ \overline{\eta} \cdot \mathbb{K}f^{*}$  is close to the target  $f^{*}$ , then this first term is small. In Section 4, we show that this is true for certain hierarchical functions. In particular, say that  $f^{*}$  satisfies the hierarchical structure  $f^{*} = g^{*} \circ h^{*}$ . If the quantity  $\mathbb{K}f^{*}$  is nearly proportional to the true feature  $h^{*}$ , then this first term is negligible. As such, we refer to the quantity  $\mathbb{K}f^{*}$  as the learned feature.
2. The second term is the sample (and width) complexity of learning the feature  $\mathbb{K}f^{*}$ . It is useful to compare the  $\| \mathbb{K}f^{*}\|_{L^2}^{-2}$  term to the standard kernel generalization bound, which requires  $n\gtrsim \left\langle f^{\star},\mathbb{K}^{-1}f^{\star}\right\rangle_{L^2}$ . Unlike in the kernel bound, the feature learning term in (4) does not require inverting the kernel  $\mathbb{K}$  as it only requires a lower bound on  $\| \mathbb{K}f^{\star}\|_{L^2}$ . This difference can be best understood by considering the alignment of  $f^{\star}$  with the eigenfunctions of  $\mathbb{K}$ . Say that  $f^{\star}$  has nontrivial alignment with eigenfunctions of  $\mathbb{K}$  with both small  $(\lambda_{min})$  and large  $(\lambda_{max})$  eigenvalues. Kernel methods require  $\Omega (\lambda_{min}^{-1})$  samples, which blows up when  $\lambda_{min}$  is small; the sample complexity of kernel methods depends on the high frequency components of  $f^{*}$ . On the other hand, the guarantee in Theorem 1 scales with
$\| \mathbb{K}f^{\star}\|_{L^2}^{-2} = O(\lambda_{max}^{-2})$ , which can be much smaller. In other words, the sample complexity of feature learning scales with the low-frequency components of  $f^{*}$ . The feature learning process can thus be viewed as extracting the low-frequency components of the target.
3. The last two terms measure the complexity of learning the univariate function  $q$ . In the examples in Section 4, the effect of these terms is benign.
Altogether, if  $f^*$  satisfies the hierarchical structure that its high-frequency components can be inferred from the low-frequency ones, then a good  $q$  for Theorem 1 exists and the dominant term in (4) is the sample complexity of feature learning term, which only depends on the low-frequency components. This is not the case for kernel methods, as small eigenvalues blow up the sample complexity. As we show in Section 4, this ability to ignore the small eigenvalue components of  $f^*$  during the feature learning process is critical for achieving good sample complexity in many problems.
At initialization,  $f(x; \theta^{(0)}) \approx 0$ . The first step of GD on the population loss for a neuron  $w_{j}$  is thus
$$
\begin{array}{l} w ^ {(1)} = - \eta_ {1} \nabla_ {w _ {j}} \mathbb {E} _ {x} \left[ \left(f (x; \theta^ {(0)}) - f ^ {*} (x)\right) ^ {2} \right] (5) \\ = \eta_ {1} \mathbb {E} _ {x} \left[ f ^ {*} (x) \nabla_ {w _ {j}} f (x; \theta^ {(0)}) \right] (6) \\ = \frac {1}{m _ {2}} \bar {\eta} \mathbf {1} _ {b _ {j} ^ {(0)} \geq 0} a _ {j} ^ {(0)} \mathbb {E} _ {x} \left[ f ^ {*} (x) h ^ {(0)} (x) \right]. (7) \\ \end{array}
$$
Therefore the network  $f(x'; \theta^{(1)})$  after the first step of GD is given by