SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arvind Mahankali*
Stanford University  
ahanka@stanford.edu
Jeff Z. HaoChen†
Stanford University jhaochen@stanford.edu
Kefan Dong
Stanford University kefandong@stanford.edu
Margalit Glasgow
Stanford University mglasgow@stanford.edu
Tengyu Ma  
Stanford University  
tengyuma@stanford.edu
Despite recent theoretical progress on the non-convex optimization of two-layer neural networks, it is still an open question whether gradient descent on neural networks without unnatural modifications can achieve better sample complexity than kernel methods. This paper provides a clean mean-field analysis of projected gradient flow on polynomial-width two-layer neural networks. Different from prior works, our analysis does not require unnatural modifications of the optimization algorithm. We prove that with sample size  $n = O(d^{3.1})$  where  $d$  is the dimension of the inputs, the network trained with projected gradient flow converges in  $\mathrm{poly}(d)$  time to a non-trivial error that is not achievable by kernel methods using  $n \ll d^4$  samples, hence demonstrating a clear separation between unmodified gradient descent and NTK. As a corollary, we show that projected gradient descent with a positive learning rate and a polynomial number of iterations converges to low error with the same sample complexity.
Training neural networks requires optimizing non-convex losses, which is often practically feasible but still not theoretically understood. The lack of understanding of non-convex optimization limits the design of new principled optimizers for training neural networks that use theoretical insights.
Early analysis on optimizing neural networks with linear or quadratic activations [29, 36, 51, 35, 65, 38, 48, 39, 60] relies on linear algebraic tools that do not extend to nonlinear and non-quadratic activations. The neural tangent kernel (NTK) approach analyzes nonconvex optimization under certain hyperparameter settings, e.g., when the initialization scale is large and the learning rate is small (see, e.g., Du et al. [30], Jacot et al. [43], Li and Liang [50], Arora et al. [10], Daniely et al. [26]). However, subsequent research shows that neural networks trained with practical hyperparameter settings typically outperform their corresponding NTK kernels [9]. Furthermore, the initialization and learning rate under the NTK regime does not yield optimal generalization guarantees [76, 22, 77, 34].
Many recent works study modified versions of stochastic gradient descent (SGD) and prove sample complexity and runtime guarantees beyond NTK [23, 2, 58, 52, 1, 7, 5, 79, 78, 25, 6, 67, 70, 20, 59]. These modified algorithms often contain multiple stages that optimize different blocks of parameters
$\dagger$  Equal Contribution
and/or use different learning rates or regularization strengths. For example, the work of Li et al. [52] uses a non-standard parameterization for two-layer neural networks and runs a two-stage algorithm with sharply changing gradient clipping strength; Damian et al. [23] use one step of gradient descent with a large learning rate and then optimize only the last layer of the neural net. Nichani et al. [59] construct non-standard regularizers based on the NTK feature covariance matrix, in order to prevent the movement of weights in certain directions which hinder generalization. Additionally, Abbe et al. [2] only train the hidden layer for  $O(1)$  time in the first stage of their algorithm, and then train the second layer to convergence in the second stage. They do not study how the population loss decreases during the first stage. However, oftentimes vanilla (stochastic) gradient descent with a constant learning rate empirically converges to a minimum with good generalization error. Thus, the modifications are arguably artifacts tailored to the analysis, and to some extent, over-using the modification may obscure the true power of gradient descent.
Another technique to analyze optimization dynamics is the mean-field approach [21, 56, 57, 61, 28, 2, 44, 62, 76], which views the collection of weight vectors as a (discrete) distribution over  $\mathbb{R}^d$  (where  $d$  is the input dimension) and approximates its evolution by an infinite-width neural network, where the weight vector distribution is continuous and evolves according to a partial differential equation. However, these works do not provide an end-to-end polynomial runtime bound on the convergence to a global minimum in the concrete setting of two-layer neural networks (without modifying gradient descent). For example, the works of Chizat and Bach [21] and Mei et al. [56] do not provide a concrete bound on the number of iterations needed for convergence. Mei et al. [57] provide a coupling between the trajectories of finite and infinite-width networks with exponential growth of coupling error but do not apply it to a concrete setting to obtain a global convergence result with sample complexity guarantees. (See more discussion below and in Section 3.) Wei et al. [76] achieve a polynomial number of iterations but require exponential width and an artificially added noise. In other words, these works, without modifying gradient descent, cannot prove convergence to a global minimizer with polynomial width and iterations. We also discuss additional related works in Appendix A.
In this paper, we provide a mean-field analysis of projected gradient flow on two-layer neural networks with polynomial width and quartic activations. Under a simple data distribution, we demonstrate that the network converges to a non-trivial error in polynomial time with polynomial samples. Notably, our results show a sample complexity that is superior to the NTK approach. As a corollary, we show that projected gradient descent with polynomially small step size and polynomially many iterations can converge to a low error, with sample complexity that is superior to NTK. Our proof is similar to the standard bound on the error of Euler's method.
Concretely, the neural network is assumed to have unit-norm weight vectors and no bias, and the second-layer weights are all  $\frac{1}{m}$  where  $m$  is the width of the neural network. The data distribution is uniform over the sphere. The target function is of the form  $y(x) = h(q_{\star}^{\top}x)$  where  $h$  is an unknown quartic link function and  $q_{\star}$  is an unknown unit vector. Our main result (Theorem 3.4) states that with  $n = O(d^{3.1})$  samples, a polynomial-width neural network with random initialization converges in polynomial time to a non-trivial error, which is statistically not achievable by any kernel method with an inner product kernel using  $n \ll d^4$  samples. To the best of our knowledge, our result is the first to demonstrate the advantage of unmodified gradient descent on neural networks over kernel methods.
The rank-one structure in the target function, also known as the single-index model, has been well-studied in the context of neural networks as a simplified case to demonstrate that neural networks can learn a latent feature direction  $q_{\star}$  better than kernel methods [58, 2, 23, 16]. Many works on single-index models study (stochastic) gradient descent in the setting where only a single vector (or "neuron") in  $\mathbb{R}^d$  is trained. This includes earlier works where the link function is monotonic, or the convergence is analyzed with quasi-convexity [46, 40, 55, 64], along with more recent work [69, 72, 11] for more general link functions. Since these works only train a single neuron, they have limited expressivity in comparison to a neural network, and can only achieve zero loss when the link function equals the activation. We stress that in our setting, the link function  $h$  is unknown, not necessarily monotonic, and does not need to be equal to the activation function. We show that in this setting, the first layer weights will converge to a distribution of neurons that are correlated with but not exactly equal to  $q_{\star}$ , so that even without bias terms, their mixture can represent the link function  $h$ . Our analysis demonstrates that gradient flow, and gradient descent with a consistent inverse-polynomial learning rate, can simultaneously learn the feature  $q_{\star}$  and the link function  $h$ , which is a key challenge that is side-stepped in previous works on neural-networks which use two-stage algorithms [58, 1, 2, 23, 14].
The main novelty of our population dynamics analysis is designing a potential function that shows that the iterate stays away from the saddle points.
Our sample complexity results leverage a coupling between the dynamics on the empirical loss for a finite-width neural network and the dynamics on the population loss for an infinite-width neural network. The main challenge stems from the fact that some exponential coupling error growth is inevitable over a certain period of time when the dynamics resemble a power method update. Heavily inspired by Li et al. [52], we address this challenge by using a direct and sharp comparison between the growth of the coupling error and the growth of the signal. In contrast, a simple exponential growth bound on the coupling error similar to the bound of Mei and Montanari [54] would result in a  $d^{O(1)}$  sample complexity, which is not sufficient to outperform NTK.
We use  $O(\cdot), \lesssim, \gtrsim$  to hide only absolute constants. Formally, every occurrence of  $O(x)$  in this paper can be simultaneously replaced by a function  $f(x)$  where  $|f(x)| \leq C|x|, \forall x \in \mathbb{R}$  for some universal constant  $C > 0$  (each occurrence can have a different universal constant  $C$  and  $f$ ). We use  $a \lesssim b$  as a shorthand for  $a \leq O(b)$ . Similarly,  $\Omega(x)$  is a placeholder for some  $g(x)$  where  $|g(x)| \geq |x| / C, \forall x \in \mathbb{R}$  for some universal constant  $C > 0$ . We use  $a \gtrsim b$  as a shorthand for  $a \geq \Omega(b)$  and  $a \asymp b$  as a shorthand to indicate that  $a \gtrsim b$  and  $a \lesssim b$  simultaneously hold.
Legendre Polynomials. We summarize the necessary facts about Legendre polynomials below, and present related background more comprehensively in Appendix C. Let  $P_{k,d}:[-1,1]\to \mathbb{R}$  be the degree- $k$  un-normalized Legendre polynomial [12], and  $\overline{P}_{k,d}(t) = \sqrt{N_{k,d}} P_{k,d}(t)$  be the normalized Legendre polynomial, where  $N_{k,d}\triangleq \binom{d+k-1}{d-1} - \binom{d+k-3}{d-1}$  is the normalizing factor. The polynomials  $\overline{P}_{k,d}(t)$  form an orthonormal basis for the set of square-integrable functions over  $[-1,1]$  with respect to the measure  $\mu_d(t)\triangleq (1-t^2)^{\frac{d-3}{2}}\frac{\Gamma(d/2)}{\Gamma((d-1)/2)}\frac{1}{\sqrt{\pi}}$ , i.e., the density of  $u_1$  when  $u=(u_1,\cdots,u_d)$  is uniformly drawn from sphere  $\mathbb{S}^{d-1}$ . Hence, for every function  $h:[-1,1]\to \mathbb{R}$  such that  $\mathbb{E}_{t\sim\mu_d}[h(t)^2]<\infty$ , we can define  $\hat{h}_{k,d}\triangleq \mathbb{E}_{t\sim\mu_d}[h(t)\overline{P}_{k,d}(t)]$  and consequently, we have  $h(t)=\sum_{k=0}^{\infty}\hat{h}_{k,d}\overline{P}_{k,d}(t)$ .
We will formally define the data distribution, neural networks, projected gradient flow, and assumptions on the problem-dependent quantities and then state our main theorems.
Target function. The ground-truth function  $y(x): \mathbb{R}^d \to \mathbb{R}$  that we aim to learn has the form  $y(x) = h(q_{\star}^{\top}x)$ , where  $h: \mathbb{R} \to \mathbb{R}$  is an unknown one-dimensional even quartic polynomial (which is called a link function), and  $q_{\star}$  is an unknown unit vector in  $\mathbb{R}^d$ . Note that  $h(s)$  has the Legendre expansion  $h(s) = \hat{h}_{0,d} + \hat{h}_{2,d}\overline{P}_{2,d}(s) + \hat{h}_{4,d}\overline{P}_{4,d}(s)$ .
Two-layer neural networks. We consider a two-layer neural network where the first-layer weights are all unit vectors and the second-layer weights are fixed and all the same. Let  $\sigma(\cdot)$  be the activation function, which can be different from  $h(\cdot)$ . Using the mean-field formulation, we describe the neural network using the distribution of first-layer weight vectors, denoted by  $\rho$ :
$$
f _ {\rho} (x) \triangleq \mathbb {E} _ {u \sim \rho} [ \sigma (u ^ {\top} x) ]. \tag {3.1}
$$
For example, when  $\rho = \mathrm{unif}(\{u_1,\ldots ,u_m\})$ , is a discrete, uniform distribution supported on  $m$  vectors  $\{u_{1},\dots ,u_{m}\}$ , then  $f_{\rho}(x) = \frac{1}{m}\sum_{i = 1}^{m}\sigma (u_i^\top x)$ , i.e.  $f_{\rho}$  corresponds to a finite-width neural network whose first-layer weights are  $u_{1},\ldots ,u_{m}$ . For a continuous distribution  $\rho$ , the function  $f_{\rho}(\cdot)$  can be viewed as an infinite-width neural network where the weight vectors are distributed according to  $\rho$  (and can be viewed as taking the limit as  $m\to \infty$  of the finite-width neural network). We assume that the weight vectors have unit norms, i.e. the support of  $\rho$  is contained in  $\mathbb{S}^{d - 1}$ . The activation  $\sigma :\mathbb{R}\rightarrow \mathbb{R}$  is assumed to be a fourth-degree polynomial with Legendre expansion  $\sigma (s) = \sum_{k = 0}^{4}\hat{\sigma}_{k,d}\overline{P}_{k,d}(s)$ .
The simplified neural network defined in Eq. (3.1), even with infinite width (corresponding to a continuous distribution  $\rho$ ), has a limited expressivity due to the lack of biases and trainable second-layer weights. We characterize the expressivity by the following lemma:
Lemma 3.1 (Expressivity). Let  $\gamma_{2} = \hat{h}_{2,d} / \hat{\sigma}_{2,d}$  and  $\gamma_{4} = \hat{h}_{4,d} / \hat{\sigma}_{4,d}$ . Suppose for some  $d$  and  $q_{\star}$ , there exists a network  $\rho$  such that  $f_{\rho}(x) = h(q_{\star}^{\top}x)$  on  $\mathbb{S}^{d - 1}$ . Then we have  $\hat{\sigma}_{0,d} = \hat{h}_{0,d}$ , and  $0 \leq \gamma_{2}^{2} \leq \gamma_{4} \leq \gamma_{2} \leq 1$ . Moreover, if this condition holds with strict inequalities, then for sufficiently large  $d$ , there exists a network  $\rho$  such that  $f_{\rho}(x) = h(q_{\star}^{\top}x)$  on  $\mathbb{S}^{d - 1}$ . (A more explicit version is stated in Appendix G.1.)
Informally, an almost sufficient and necessary condition to have  $f_{\rho}(x) = h(q_{*}^{\top}x)$  for some  $\rho$  is that there exists a random variable  $w$  supported on  $[0,1]$  such that  $\mathbb{E}[w^{2}] \approx \gamma_{2}$  and  $\mathbb{E}[w^{4}] \approx \gamma_{4}$ , which is equivalent to  $0 \leq \gamma_{2}^{2} \leq \gamma_{4} \leq \gamma_{2} \leq 1$ . In particular, assuming the existence of such a random variable  $w$ , the perfectly-fit network  $\rho$  that fits the target function has the form
$q_{\star}^{\top}u \stackrel{d}{=} w$ , and  $u - q_{\star}q_{\star}^{\top}u \mid q_{\star}^{\top}u$  is uniformly distributed in the subspace orthogonal to  $q_{\star}$ . (3.2)
Motivated by this lemma, we will assume that  $\gamma_{2},\gamma_{4}$  are universal constants that satisfy  $0\leq \gamma_2^2\leq$ $\gamma_4\leq \gamma_2\leq 1$  , and  $d$  is chosen to be sufficiently large (depending on the choice of  $\gamma_{2}$  and  $\gamma_{4}$  - In addition, we also assume that  $\gamma_4\le O(\gamma_2^2)$  , that is, the equality  $\gamma_2^2\leq \gamma_4$  is somewhat tight -- this ensures that the distribution of  $w = q_{\star}^{\top}u$  under the perfectly-fitted neural network is not too spread-out. We also assume that  $\gamma_{2}$  is smaller than a sufficiently small universal constant. This ensures that the distribution of  $q_{\star}^{\top}u$  under the perfectly-fitted network does not concentrate on 1, i.e. the distribution of  $u$  is not merely a point mass around  $q_{\star}$  . In other words, this assumption restricts our setting to the most interesting case where the landscape has bad saddle points (and thus is fundamentally more challenging to analyze). We also assume for simplicity that  $\hat{\sigma}_{0,d} = \hat{h}_{0,d}$  (because otherwise, the activation introduces a constant bias that prohibits perfect fitting), even though adding a trainable scalar to the neural network formulation can remove the assumption. If  $\hat{\sigma}_{0,d} = \hat{h}_{0,d}$  , then we can assume without loss of generality that  $\hat{\sigma}_{0,d} = \hat{h}_{0,d} = 0$  , since  $\hat{\sigma}_{0,d}$  and  $\hat{h}_{0,d}$  will cancel with each other in the population and empirical mean-squared losses (defined below). In summary, we make the following formal assumptions on the Legendre coefficients of the link function and the activation function.
Assumption 3.2. Let  $\gamma_{2} = \hat{h}_{2,d} / \hat{\sigma}_{2,d}$  and  $\gamma_{4} = \hat{h}_{4,d} / \hat{\sigma}_{4,d}$ . We first assume  $\gamma_{4} \geq 1.1 \cdot \gamma_{2}^{2}$ . For any universal constant  $c_{1} > 1$ , we assume that  $\hat{\sigma}_{2,d}^{2} / c_{1} \leq \hat{\sigma}_{4,d}^{2} \leq c_{1} \cdot \hat{\sigma}_{2,d}^{2}$ , and  $\gamma_{4} \leq c_{1} \gamma_{2}^{2}$ . For a sufficiently small universal constant  $c_{2} > 0$  (which is chosen after  $c_{1}$  is determined), we assume  $0 \leq \gamma_{2} \leq c_{2}$ . We also assume that  $d$  is larger than a sufficiently large constant  $c_{3}$  (which is chosen after  $c_{1}$  and  $c_{2}$ ). We also assume  $\hat{h}_{0,d} = \hat{\sigma}_{0,d} = 0$ , and  $\hat{h}_{1,d} = \hat{h}_{3,d} = 0$ .
Our intention is to replicate the ReLU activation as well as possible with quartic polynomials; our assumption that  $\hat{\sigma}_{2,d}^2 \asymp \hat{\sigma}_{4,d}^2$  is indeed satisfied by the quartic expansion of ReLU because  $\widehat{\mathrm{relu}}_{2,d} \asymp d^{-1/2}$  and  $\widehat{\mathrm{relu}}_{2,d} \asymp d^{-1/2}$  (see Proposition C.3). Following the convention defined in Section 2, we will simply write  $\hat{\sigma}_{4,d}^2 \asymp \hat{\sigma}_{2,d}^2$ ,  $\gamma_4 \geq 1.1 \cdot \gamma_2^2$ , and  $\gamma_4 \lesssim \gamma_2^2$ .
Our assumptions rule out the case that  $\gamma_4 = \gamma_2^2$ , for the sake of simplicity. We believe that our analysis could be extended to this case with some modifications. Our analysis also rules out the case where  $\gamma_2 = 0$  and  $\gamma_4 \neq 0$ , due to the restriction that  $\gamma_4 \leq c_1 \gamma_2^2$ . The case  $\gamma_2 = 0$  and  $\gamma_4 \neq 0$  would have a significantly different analysis, and potentially a different sample complexity, since our analysis in Section 4.2 and Section 5 makes use of the fact that the initial phase of the population and empirical dynamics behaves similarly to a power method, which follows from  $\gamma_2$  being nonzero.
Data distribution, losses, and projected gradient flow. The population data distribution is assumed to be  $\mathbb{S}^{d - 1}$ . We draw  $n$  training examples  $x_{1},\ldots ,x_{n}\stackrel {\mathrm{i.i.d}}{\sim}\mathbb{S}^{d - 1}$ . Thus, the population and empirical mean-squared losses are:
$$
L (\rho) = \frac {1}{2} \cdot \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(f _ {\rho} (x) - y (x)\right) ^ {2}, \quad \text {a n d} \quad \widehat {L} (\rho) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(f _ {\rho} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) ^ {2}. \tag {3.3}
$$
To ensure that the weight vectors remain on  $\mathbb{S}^{d - 1}$ , we perform projected gradient flow on the empirical loss. We start by defining the gradient of the population loss  $L$  with respect to a particle  $u$  at  $\rho$  and
the corresponding Riemannian gradient (which is simply the projection of the gradient to the tangent space of  $\mathbb{S}^{d - 1}$ ):
$$
\nabla_ {u} L (\rho) = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - y (x)\right) \sigma^ {\prime} \left(u ^ {\top} x\right) x \right], \quad \text {a n d} \operatorname {g r a d} _ {u} L (\rho) = \left(I - u u ^ {\top}\right) \nabla_ {u} L (\rho). \tag {3.4}
$$
Here we interpret  $L(\rho)$  as a function of a collection of particles (denoted by  $\rho$ ) and  $\nabla_{u}L(\rho)$  as the partial derivative with respect to a single particle  $u$  evaluated at  $\rho$ . Similarly, the (Riemannian) gradient of the empirical loss  $\widehat{L}$  with respect to the particle  $u$  is defined as
$$
\nabla_ {u} \widehat {L} (\rho) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f _ {\rho} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(u ^ {\top} x _ {i}\right) x _ {i}, \quad \text {a n d} \quad \operatorname {g r a d} _ {u} \widehat {L} (\rho) = \left(I - u u ^ {\top}\right) \nabla_ {u} \widehat {L} (\rho). \tag {3.5}
$$
Population, Infinite-Width Dynamics. Let the initial distribution  $\rho_0$  of the infinite-width neural network be the uniform distribution over  $\mathbb{S}^{d - 1}$ . We use  $\chi$  to denote a particle sampled uniformly at random from the initial distribution  $\rho_0$ . A particle initialized at  $\chi$  follows a deterministic trajectory afterwards — we use  $u_{t}(\chi)$  to denote the location, at time  $t$ , of the particle that was initialized at  $\chi$ . Because  $u_{t}(\cdot)$  is a deterministic function, we can use  $\chi$  to index the particles at any time based on their initialization. The projected gradient flow on an infinite-width neural network and using population loss  $L$  can be described as
$$
\forall \chi \in \mathbb {S} ^ {d - 1}, u _ {0} (\chi) = \chi , \tag {3.6}
$$
$$
\frac {d u _ {t} (\chi)}{d t} = - \operatorname {g r a d} _ {u _ {t}} L \left(\rho_ {t}\right), \tag {3.7}
$$
$$
\text {a n d} \rho_ {t} = \text {d i s t r i b u t i o n} u _ {t} (\chi) \left(\text {w h e r e} \chi \sim \rho_ {0}\right). \tag {3.8}
$$
Empirical, Finite-Width Dynamics. The training dynamics of a neural network with width  $m$  can be described in this language by setting the initial distribution to be a discrete distribution uniformly supported on  $m$  initial weight vectors. The update rule will maintain that at any time, the distribution of neurons is uniformly supported over  $m$  items and thus still corresponds to a width- $m$  neural network. Let  $\chi_1, \ldots, \chi_m \stackrel{\mathrm{i.i.d.}}{\sim} \mathbb{S}^{d-1}$  be the initial weight vectors of the width- $m$  neural network, and let  $\hat{\rho}_0 = \mathrm{unif}(\{\chi_1, \dots, \chi_m\})$  be the uniform distribution over these initial neurons. We use  $\chi \in \{\chi_1, \dots, \chi_m\}$  to index neurons and denote a single initial neuron as  $\hat{u}_0(\chi) = \chi$ . Then, we can describe the projected gradient flow on the empirical loss  $\widehat{L}$  with initialization  $\{\chi_1, \dots, \chi_m\}$  by:
$$
\frac {d \hat {u} _ {t} (\chi)}{d t} = - \operatorname {g r a d} _ {\hat {u} _ {t} (\chi)} \widehat {L} (\hat {\rho} _ {t}), \tag {3.9}
$$
$$
\text {a n d} \hat {\rho} _ {t} = \text {d i s t r i b u t i o n} \hat {u} _ {t} (\chi) \left(\text {w h e r e} \chi \sim \hat {\rho} _ {0}\right). \tag {3.10}
$$
We first state our result on the population, infinite-width dynamics.
Theorem 3.3 (Population, infinite-width dynamics). Suppose Assumption 3.2 holds, and let  $\epsilon \in (0,1)$  be the target error. Let  $\rho_{t}$  be the result of projected gradient flow on the population loss, initialized with the uniform distribution on  $\mathbb{S}^{d - 1}$ , as defined in Eq. (3.7). Let  $T_{*,\epsilon} = \inf \{t > 0 \mid L(\rho_t) \leq \frac{1}{2} (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2\}$  be the earliest time  $t$  such that a loss of at most  $\frac{1}{2} (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$  is reached. Then, we have
$$
T _ {*, \epsilon} \lesssim \frac {1}{\hat {\sigma} _ {2 , d} ^ {2} \gamma_ {2}} \log d + \frac {(\log \log d) ^ {2 0}}{\hat {\sigma} _ {2 , d} ^ {2} \epsilon \gamma_ {2} ^ {8}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right). \tag {3.11}
$$