SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Emmanuel Abbe EPFL
emmanuel.abbe@epfl.ch
Elisabetta Cornacchia EPFL,MIT
ecornacc@mit.edu
Aryo Lotfi EPFL
aryo.lotfi@epfl.ch
Experimental results have shown that curriculum learning, i.e., presenting simpler examples before more complex ones, can improve the efficiency of learning. Some recent theoretical results also showed that changing the sampling distribution can help neural networks learn parities, with formal results only for large learning rates and one-step arguments. Here we show a separation result in the number of training steps with standard (bounded) learning rates on a common sample distribution: if the data distribution is a mixture of sparse and dense inputs, there exists a regime in which a 2-layer ReLU neural network trained by a curriculum noisy-GD (or SGD) algorithm that uses sparse examples first, can learn parities of sufficiently large degree, while any fully connected neural network of possibly larger width or depth trained by noisy-GD on the unordered samples cannot learn without additional steps. We also provide experimental results supporting the qualitative separation beyond the specific regime of the theoretical results.
Starting with easy concepts is an effective way to facilitate learning for humans. When presented with simpler ideas or tasks, individuals can develop a foundation of knowledge and skills upon which more complex concepts can be built. This approach allows learners to gradually gain competence and accelerate learning [EA84, RK90,  $\mathrm{AKB}^{+}97$ , SGG14].
For machine learning algorithms, this is typically referred to as Curriculum Learning (CL) [BLCW09]. Several empirical studies have shown that presenting samples in a meaningful order can improve both the speed of training and the performance achieved at convergence, compared to the standard approach of presenting samples in random order [WCZ21, SIRS22,  $\mathrm{GBM}^{+}17$ , HW19, PSL15].
The theoretical study of CL is mostly in its infancy. While some works have analytically shown benefits of CL for certain targets [SMS22, WCA18, MKAS21, CM23] (see Section 1.1 for a discussion on these), it remained open to provide a separation result between curriculum training and standard training on a common sampling distribution. Specifically, one seeks to demonstrate how a standard neural network, trained using a gradient descent algorithm on a given dataset and horizon, can successfully learn the target when certain samples are presented first during training, whereas learning does not occur if samples are presented in a random order.
This paper provides such an instance. We focus on the case of parities, as a classical case of challenging target for differentiable learning. In the experimental part of the paper, we report results suggesting that a similar picture can hold for other large leap functions as defined in [ABAM23], although more investigations are required to understand when and how curriculum learning should be developed for more general functions.
We assume that the network is presented with samples  $(x, \chi_S(x))$ , with  $x \in \{\pm 1\}^d$  and  $\chi_S(x) := \prod_{i \in S} x_i$ , for some set  $S \subseteq \{1, \ldots, d\}$  of fixed size. We consider datasets where a small fraction  $\rho$  of
the inputs are sparse, meaning they have, on average, few negative coordinates and mostly positive coordinates, and the remaining inputs are dense, with an average of half negative and half positive coordinates. We refer to Section 2 for the definition of the input distribution. Parities seem to be promising candidates for theoretical investigations into CL, as they are hard to learn in the statistical query learning model [Kea98], or on dense data by any neural network trained with gradient-based methods having limited gradient precision [AS20], but they are efficiently learnable on sparse data by regular architectures [MKAS21] (see Section 1.1 for further discussion).
We say that a neural network is trained with curriculum if it goes through an initial phase where it is exposed to batches consisting only of sparse samples (see discussion below on the interpretation of these as 'simpler' examples). Following this phase, the network is trained using batches sampled from the entire dataset. We compare this curriculum training approach with standard training, where the network receives batches of samples selected randomly from the entire dataset at each step. Our intuition is as follows. If the fraction of sparse samples is small, without curriculum, the network is exposed to batches of mostly uniform inputs, and the contribution of sparse inputs in estimating the gradients will be limited. As a result, learning parities on the mixed distribution without curriculum is roughly as hard as learning parities on dense inputs (Theorem 2). On the contrary, if the network is initially exposed to batches consisting of sparse inputs, it can efficiently identify the support of the parity, and subsequent fitting using batches sampled from the entire dataset enables learning the target with fewer training steps compared to standard training (Theorem 1).
In particular, we show that if the fraction of sparse samples  $\rho$  is small enough, (e.g.,  $\rho < d^{-4.5}$ , where  $d$  is the input dimension), a 2-layer fully connected network of size  $\theta(d)$  trained by layerwise SGD (or noisy-SGD, see Def. 1) with curriculum can learn any target  $k$ -parity, with  $k$  bounded, in  $T = \theta(d)$  steps, while the same network (or any network of similar size) trained by noisy-SGD without curriculum cannot learn the target in less than  $\Omega(d^2)$  steps. We refer to Section 2 for a more detailed explanation of our results. While our theoretical results use simplifying assumptions (e.g., layerwise training), our experiments show separations between curriculum and standard training in a broader range of settings.
Curriculum Learning (CL) was first introduced in the machine learning context by the seminal work of [BLCW09]. This paper along with many other subsequent works presented empirical evidence that training neural networks with a curriculum can be beneficial in both the training speed and the performance achieved at convergence in certain instances in different domains such as computer vision [JMMH14, GHZ $^{+}$ 18, CG15, STD19], natural language processing (NLP) [KB17, GBM $^{+}$ 17, ZKK $^{+}$ 18, PSN $^{+}$ 19], and reinforcement learning [FHW $^{+}$ 17, NPL $^{+}$ 20]. We refer to relevant surveys [WCZ21, SIRS22] for a more comprehensive list of applications of CL in different domains. In general, defining a complexity measure on samples and consequently an ordering of training inputs is not trivial. Some works determine the easiness of samples based on predefined rules (e.g., shape as used in [BLCW09]), while others also use the model's state and performance to evaluate the samples dynamically as in self-paced learning (SPL) [KPK10]. In this work, we follow the former group as we define the hardness of samples based on the number of  $-1$  bits, i.e., the Hamming weight. Note that in parity targets,  $+1$  is the identity element. Thus, one can view the number of  $-1$  bits as the length of the sample. In this sense, our work resembles [SAJ10, ZS14, KB17, PSN $^{+}$ 19] which use the length of the text inputs as a part of their curriculum.
Despite the considerable empirical work on CL, there appears to be a scarcity of theoretical analysis. Few theoretical works focused on parity targets, as in this paper. While parities are efficiently learnable by specialized algorithms that have access to at least  $d$  samples (e.g. Gaussian elimination over the field of two elements), they are hard to learn under the uniform distribution (i.e. on dense data) by any neural network trained with gradient-based methods with limited gradient precision [AS20,  $\mathrm{AKM}^{+}21$ ]. Parities on sparse data can be learned more efficiently on an active query model, which can be emulated by a neural network [AS20]. However, such emulation networks are far from the practical and homogeneous networks, and distribution class-dependent. A natural question that has been considered is whether more standard architectures can also benefit from a simpler data distribution. In [MKAS21], the authors make a first step by showing that parities on sparse inputs are efficiently learnable on a 1-layer network with a parity module appended to it, with a one-step argument. In [DM20], it is shown that for a two-layer fully connected network, sparse inputs help but
with a variant of the distribution that allows for leaky labels. [ABLR23] obtains experimental results for a curriculum algorithm presenting sparse samples first with heuristics for regular networks and hyperparameters, but without proofs. In [CM23], the authors prove that a two-layer fully connected net can benefit from sparse inputs, with a one-step gradient argument requiring a large learning rate. This paper improves on [CM23] not only in terms of having more natural training settings and hyperparameters (i.e., we use SGD with bounded batch size and bounded learning rate), it is also establishing a separation between learning with and without curriculum on a common sampling distribution, while [CM23] does not (in fact in their work curriculum involves both sparse and dense inputs while standard training involves dense inputs only). To the best of our knowledge, our paper is the first work that establishes a rigorous separation between training on adequately ordered samples and randomly ordered samples drawn from the same distribution, and our theorem gives a condition on the mixture under which this separation holds (interestingly the separation does not take place for all mixtures parameters). Beyond parities, [SMS22] studies a teacher-student model, where the target depends on a sparse set of features and where the variance on the irrelevant features. Furthermore, [WCA18, WA20] show that on some convex models, CL provides an improvement in the speed of convergence of SGD. In contrast, our work covers an intrinsically non-convex problem.
2 Setting and Informal Contributions
We consider the problem of learning a target function  $f\colon \{\pm 1\}^d\to \{\pm 1\}$  that belongs to the class of  $k$ -parities on  $d$  bits, i.e.,
$$
f \in \left\{\chi_ {S} (x) = \prod_ {j \in S} x _ {j}: S \subseteq [ d ], | S | = k \right\}, \tag {1}
$$
where we denoted by  $[d] := \{1, \ldots, d\}$ . We assume that the network has access to a dataset  $(X, Y) = \{(x^s, y^s)\}_{s \in [m]}$  where the inputs  $x^s$  are sampled i.i.d. from a mixed distribution defined as follows:
$$
\mathcal {D} _ {\text {m i x}} = \rho \mathcal {D} _ {\mu} + (1 - \rho) \mathcal {D} _ {u}, \tag {2}
$$
where  $\mathcal{D}_{\mu} = \mathrm{Rad}(\frac{\mu + 1}{2})^{\otimes d1}$ , for some  $\mu \in [-1,1]$ ,  $\mathcal{D}_u = \mathrm{Unif}\{\pm 1\}^{d2}$  and  $\rho \in [0,1]$  is some fixed parameter, and  $y^{s} = f(x^{s})$ . We will assume  $\mu \in (0,1)$ . In other words, we assume that the dataset contains approximately  $\rho m$  samples with in average  $\frac{1 - \mu}{2} d$  negative bits (sparse samples), and  $(1 - \rho)m$  samples with in average half negative bits (dense samples). We consider training the neural network with stochastic gradient descent, with or without gradient noise (SGD or noisy-SGD, see Def. 1).
Informal Description of the Curriculum and Standard Training Algorithms. We compare two training strategies: standard training and curriculum training. In the standard training, at each step, mini-batches are drawn from the whole dataset. For curriculum training, we first need to isolate the set of sparse inputs in our dataset. Let us denote this set by  $X_{1} \coloneqq \{x^{s} \in X : H(x^{s}) < \left(\frac{1}{2} - \frac{\mu}{4}\right)d\}$ , where  $H(x) \coloneqq \sum_{j \in [d]} \mathbb{1}(x_{j} = -1)$  denotes the Hamming weight of  $x$ , which is the count of negative bits in  $x$ , and by  $Y_{1} = \{y^{s} \in Y : x^{s} \in X_{1}\}$  the corresponding set of labels. Our curriculum strategy consists of training on only sparse samples for the first  $T_{1}$  steps, where  $T_{1} \leq T$  is some well-chosen time horizon and  $T$  is the total number of training steps. After this initial phase, the network is trained on samples belonging to the whole dataset. To sum up, we compare the following two training strategies:
- (standard training): At all  $t \in [T]$ , mini-batches are sampled from  $(X, Y)$ ;
- (curriculum training): At  $t \leq T_1$ , mini-batches are sampled from  $(X_1, Y_1)$  and at  $T_1 < t \leq T$  mini-batches are sampled from  $(X, Y)$ .
Informal Description of the Positive and Negative Results. In this paper, we prove that if the fraction of sparse samples  $\rho$  is small enough, there is a separation in the number of training steps needed to learn the target parity up to a given accuracy. We say that an algorithm on a neural network
$^1$ Rad denotes the Rademacher distribution, i.e., we say that  $x \sim \operatorname{Rad}(p)$ , for  $p \in [0,1]$ , if  $\mathbb{P}(x = 1) = 1 - \mathbb{P}(x = -1) = p$ .
2Unif denotes the uniform distribution. In particular,  $\mathcal{D}_u = \mathcal{D}_0$
learns a target function  $f\colon \{\pm 1\} ^d\to \{\pm 1\}$  up to accuracy  $1 - \epsilon$  , for  $\epsilon \geq 0$  , in  $T$  steps, if it outputs a network  $\mathrm{NN}(x;\theta^T)$  such that:
$$
\mathbb {P} _ {x \sim \mathcal {D} _ {\operatorname {m i x}}} (\operatorname {s g n} (\mathrm {N N} (x; \theta^ {T})) = f (x)) \geq 1 - \epsilon , \tag {3}
$$
where  $\operatorname{sgn}(.)$  denotes the sign function. In our main positive result, we consider training with the covariance loss (Def. 2) and we refer to Remark 2 for considerations on other losses. Let us state our main positive result informally.
Theorem 1 (Theorem 3, Informal). Consider the input distribution  $\mathcal{D}_{\mathrm{mix}}$  with mixture parameters  $\rho = \tilde{O}(d^{-1/2})^3$ ,  $\mu = \theta(1)$ . Let  $\epsilon > 0$ . Then, a 2-layer ReLU neural network of size  $O(d)$  initialized with an isotropic distribution, trained with the covariance loss, standard (i.e., non-diverging) learning rate schedule and either (i) a layer-wise curriculum-noisy-SGD algorithm with gradient range  $A = \tilde{O}(\sqrt{d})$ , noise-level  $\tau = \tilde{O}(d^{-1})$  and sparse samples first, or (ii) a layer-wise curriculum-SGD algorithm with sparse samples first, can learn any  $k$ -parities,  $k = \theta(1)$ , up to accuracy at least  $1 - \epsilon$  in  $T = \tilde{O}(d)/\epsilon^2$  steps.
We further show an additional positive result valid for a layer-wise curriculum SGD with the hinge loss, which holds for large batch size and diverging learning rate (Theorem 4). On the flip side, we have the following.
Theorem 2 (Theorem 5, Informal). Consider the input distribution  $\mathcal{D}_{\mathrm{mix}}$  with mixture parameters  $\rho = \tilde{O}(d^{-3.5 - \delta})$ ,  $\delta > 0$ , and  $\mu = \theta(1)$ . Then, any neural network of size  $\tilde{O}(d)$  (any depth), with any initialization and activation, trained by a noisy-SGD algorithm (layer-wise or joint) with gradient range  $A = \tilde{O}(\sqrt{d})$ , noise-level  $\tau = \tilde{\Omega}(d^{-1})$ , batch size  $B = \tilde{\Omega}(\rho^{-2})$ , any almost surely differentiable loss function and any learning rate schedule, will need at least  $T = \tilde{\Omega}(\epsilon d^{1 + \delta})$  steps to achieve accuracy  $\frac{1}{2} + \epsilon$  for any  $k$ -parities ( $k \geq 6$ ).
Conclusion of Results. When using noisy-SGD, the curriculum learning algorithm that takes sparse samples first, for the first layer training, and then full samples, can learn in a regime of hyperparameters and training steps (Theorem 1) that is contained in the hyperparameter regime of the negative result (Theorem 2), for which it is shown that the absence of curriculum cannot learn without additional training steps. Further, it is shown in part (ii) of the first theorem, that using SGD rather than noisy-SGD still allows to learn with the curriculum training; this result does however not come with a counter-part for the lower-bound, i.e., we cannot show that SGD without curriculum could not learn such parities, as there is currently no proof technique to obtain rigorous lower-bounds for SGD algorithms (as for the case of noiseless honest-SQ algorithms); see for instance discussions about this in  $\mathrm{[AKM^{+}21}$ , ABA22]. However, it is not expected that SGD can do better than noisy-SGD on 'regular' neural networks [ABA22, ABAM22, ABAM23].
3 Positive Results for Curriculum Training
The Algorithm. Let us define the noisy-SGD algorithm, that is used in both our positive and negative results.
Definition 1 (Noisy-SGD). Consider a neural network  $\mathrm{NN}(:,\theta)$ , with initialization of the weights  $\theta^0$  and a dataset  $(X,Y) = \{(x^{s},y^{s})\}_{s\in [m]}$ . Given an almost surely differentiable loss function  $L$ , the updates of the noisy-SGD algorithm with learning rate  $\gamma_{t}$  and gradient range  $A$  are defined by
$$
\theta^ {t + 1} = \theta^ {t} - \gamma_ {t} \left(\frac {1}{B} \sum_ {s = 1} ^ {B} \left[ \nabla_ {\theta^ {t}} L \left(\mathrm {N N} \left(x ^ {s, t}; \theta^ {t}\right), y ^ {s, t}, x ^ {s, t}\right) \right] _ {A} + Z ^ {t}\right), \tag {4}
$$
where for all  $t \in \{0, \dots, T - 1\}$ ,  $Z^t$  are i.i.d. Unif  $[-\tau, \tau]$ , for some noise level  $\tau$ , and they are independent from other variables,  $B$  is the batch size, and  $[z]_A := \operatorname*{argmin}_{|y| \leq A} |y - z|$ , i.e., whenever the gradient exceeds  $A$  (resp.  $-A$ ) it is rounded to  $A$  (resp.  $-A$ ).
For brevity, we will write  $L(\theta^t,y,x) \coloneqq L(\mathrm{NN}(x;\theta^t),y,x)$ . We assume that for all  $s \in [B]$ ,  $(x^{s,t},y^{s,t})$  are chosen from  $(X_{t},Y_{t})$ , where  $(X_{t},Y_{t})$  can be either the whole dataset  $(X,Y)$ , or a subset of it. We include the noise  $Z^t$  and the gradient range  $A$  to cover the noisy-SGD algorithms,
$^3\tilde{O} (d^c) = O(d^c\mathrm{poly}(\log (d))),$  for any  $c\in \mathbb{R}$
used in SQ lower bounds e.g. in [AS20,  $\mathrm{AKM}^{+}21$ , MKAS21, ABA22, ACHM22]. Note that if we set  $\tau = 0$  and  $A$  large enough, we recover the standard SGD algorithm without noise.
We assume that our dataset is drawn from the mixed distribution defined in (2), with parameters  $\rho, \mu$ . For the purposes of this section, we assume  $\rho = \tilde{O}(d^{-1/2})$  and  $\mu \in (0,1)$ . As a preliminary step, we isolate the sparse inputs from our dataset, by mean of Hamming weight. In particular, we define  $X_1 := \{x^s \in X : H(x^s) < (\frac{1}{2} - \frac{\mu}{4})d\}$ , where  $H(x) := \sum_{j \in [d]} \mathbb{1}(x_j = -1)$  denotes the Hamming weight of  $x$ , and  $Y_1 = \{y^s \in Y : x^s \in X_1\}$ . Note that, due to the concentration of Hamming weight, for  $d$  large enough, there will be approximately  $m\rho$  samples in  $X_1$  (see Lemma 1). We assume that the training set is large enough, such that each sample is observed at most once during training.
In this section, we consider a 2-layer neural network  $\mathrm{NN}(x;\theta)$ , with  $\theta = (a,w,b)$ , defined by:  $\mathrm{NN}(x;\theta) := \sum_{i=1}^{N} a_i \sigma(w_i x + b_i)$ , where  $\sigma(x) = \mathrm{ReLU}(x) := \max\{x,0\}$ . We assume that the number of hidden units is  $N \geq k+1$ , where  $k$  is the degree of the target parity. We initialize  $w_{ij}^0 = 0$ ,  $a_i^0 = \kappa$  and  $b_i^0 = \Delta d + 1$ , for some  $\kappa, \Delta > 0$ , for all  $i,j$ . We refer to  $\kappa$  and  $\Delta$  as the second layer initialization scale and the bias initialization scale, respectively. We adopt a layer-wise curriculum training approach. During the initial phase of training, we exclusively train the first layer of the network using samples taken from  $(X_1,Y_1)$ . Furthermore, we project the weights of the first layer, to guarantee that they stay bounded. In the second phase, we train only the second layer using the entire merged dataset. We refer to Algorithm 1 in Appendix A for the pseudo-code of the algorithm used in our proof. In the context of standard learning without curriculum, similar layer-wise training strategies are considered in e.g., [MSS20, BEG+22, ABAM23]. [CM23] uses a layer-wise curriculum training, but with a one-step argument that requires a diverging learning rate. In our analysis, we keep the bias weights fixed to specific values during the whole training. We believe that one could extend the result to include scenarios with more general initializations and where the bias weights are trained, but this would require further technical work.
The Covariance Loss. We train  $\mathrm{NN}(x;\theta)$  using noisy-SGD with a specific loss function, namely the covariance loss, that we define here. The covariance loss appears in [CM23] with a slightly different definition: in particular, the following definition does not depend on the average estimator output, as opposed to their definition.
Definition 2 (Covariance Loss). Let  $(X,Y) = \{(x^{s},y^{s})\}_{s\in [m]}$  be a dataset, where for all  $s$ ,  $x^{s}\in \mathcal{X}$  and  $y^{s}\in \{\pm 1\}$ , and let  $\bar{y} = \frac{1}{m}\sum_{s\in [m]}y^{s}$ . Assume  $|\bar{y}| < 1$ . Let  $\hat{f}\colon \mathcal{X}\to \{\pm 1\}$  be an estimator. We define the covariance loss as
$$
L _ {\operatorname {c o v}} (\hat {f}, y ^ {s}, x ^ {s}) := \left(1 - y ^ {s} \bar {y} - \hat {f} \left(x ^ {s}\right) \left(y ^ {s} - \bar {y}\right)\right) _ {+}, \tag {5}
$$
where  $(a)_{+} \coloneqq \max \{a, 0\}$ .
The covariance loss can be applied to classification datasets with non-negligible fractions of  $+1$  and  $-1$  labels. In Proposition 1, we show that a small covariance loss implies a small classification error.
Proposition 1. Let  $(X,Y) = \{(x^{s},y^{s})\}_{s\in [m]}$  be a dataset, where for all  $s$ $x^s\in \mathcal{X}$  and  $y^{s}\in \{\pm 1\}$ . Let  $\bar{y} = \frac{1}{m}\sum_{s\in [m]}y^{s}$  and assume  $|\bar{y}| < 1 - \delta$ , for some  $\delta >0$ . Let  $\hat{f} := \mathcal{X}\to \{\pm 1\}$  be an estimator. If  $\frac{1}{m}\sum_{s\in [m]}L_{\mathrm{cov}}(\hat{f},x^s,y^s) < \epsilon$ , then  $\frac{1}{m}\sum_{s\in [m]}\mathbb{1}(\operatorname {sgn}(\hat{f} (x^{s}))\neq y^{s}) < \frac{\epsilon}{\delta}$ .
The proof of Proposition 1 can be found in Appendix D.
Remark 1. We remark that for balanced datasets  $(\bar{y} = 0)$ , the covariance loss coincides with the hinge loss. On the other hand, note that the right hand side of (5) can be rewritten as  $\left((1 - y^s\bar{y})\cdot (1 - y^sf(x^s))\right)_+$ . Thus, in some sense, the samples in the under-represented class (where  $y^{s}\bar{y}\leq 0$ ) are assigned a higher loss compared to those in the over-represented class (where  $y^{s}\bar{y}\geq 0$ ).
Main Theorem. We are now ready to state our main theorem.
Theorem 3 (Main Positive Result). Let  $\mathrm{NN}(x;\theta)$  be a 2-layer ReLU neural network with  $N\geq k + 1$  hidden units, 2nd layer init. scale  $\kappa >0$  and bias init. scale  $\Delta >0$ . Let  $\mathcal{D}_{\mathrm{mix}}$  be a mixed distribution, as defined in (2), with parameters  $0 < \rho \leq \frac{\Delta}{4N}\frac{\sqrt{\log(d)}}{\sqrt{d}}$ , and  $\mu \in (0,1)$ . Assume we have access to
a dataset  $(X,Y) = \{x^{s},y^{s}\}_{s\in [m]}$ , where  $x^{s} \stackrel{iid}{\sim} \mathcal{D}_{\mathrm{mix}}$ ,  $y^{s} = \chi_{S}(x^{s})$ , with  $|S| = k$ . Assume NN is trained according to the layer-wise curriculum-SGD algorithm (Algorithm 1), with the covariance loss, batch size  $B$ , noise level  $\tau$  and gradient range  $A$ .
For all  $\epsilon > 0$ , there exist  $C, C_1, C^* > 0$  such that if  $\kappa \leq 1 / N(2\Delta d + 2)$ ,  $\tau \leq \kappa$ ,  $A = \tilde{\Omega}(\sqrt{d})$ ,  $m \geq 4C^*\log(3d)d\rho^{-1}B$ , and if
$$
T _ {1} = \frac {2 C \Delta^ {2} d \log (d) ^ {2}}{L _ {\mu} ^ {2}}, \quad \gamma_ {1} = \frac {L _ {\mu}}{C \Delta \kappa d \log (d) ^ {2}}, \tag {6}
$$
$$
T _ {2} = \frac {2 5 6 \left(C _ {1} + 2 \Delta k + \tau\right) ^ {2} N ^ {2} d}{\epsilon^ {2} \Delta^ {2} \log (d)}, \quad \gamma_ {2} = \frac {\epsilon \log (d)}{8 \left(C _ {1} + 2 \Delta k + \tau\right) ^ {2} d N}, \tag {7}
$$