SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Youssef Allouah* Rachid Guerraoui Nirupam Gupta Rafael Pinot Geovani Rizk
Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland
The theory underlying robust distributed learning algorithms, designed to resist adversarial machines, matches empirical observations when data is homogeneous. Under data heterogeneity however, which is the norm in practical scenarios, established lower bounds on the learning error are essentially vacuous and greatly mismatch empirical observations. This is because the heterogeneity model considered is too restrictive and does not cover basic learning tasks such as least-squares regression. We consider in this paper a more realistic heterogeneity model, namely  $(G,B)$ -gradient dissimilarity, and show that it covers a larger class of learning problems than existing theory. Notably, we show that the breakdown point under heterogeneity is lower than the classical fraction  $1/2$ . We also prove a new lower bound on the learning error of any distributed learning algorithm. We derive a matching upper bound for a robust variant of distributed gradient descent, and empirically show that our analysis reduces the gap between theory and practice.
Distributed machine learning algorithms involve multiple machines (or workers) collaborating with the help of a server to learn a common model over their collective datasets. These algorithms enable training large and complex machine learning models, by distributing the computational burden among several workers. They are also appealing as they allow workers to retain control over their local training data. Conventional distributed machine learning algorithms are known to be vulnerable to adversarial workers, which may behave unpredictably. Such behavior may result from software and hardware bugs, data poisoning, or malicious players controlling part of the network. In the parlance of distributed computing, such adversarial workers are referred to as Byzantine [24]. Due to the growing influence of distributed machine learning in public applications, a significant amount of work has been devoted to addressing the problem of robustness to Byzantine workers, e.g., see [34, 11, 2, 19, 13, 15].
A vast majority of prior work on robustness however assumes data homogeneity, i.e., local datasets are generated from the same distribution. This questions their applicability in realistic distributed learning scenarios with heterogeneous data, where local datasets are generated from different distributions. Under data homogeneity, Byzantine workers can only harm the system when the other workers compute stochastic gradient estimates, by exploiting the noise in gradient computations. This vulnerability can be circumvented using variance-reduction schemes [19, 13, 14]. In contrast, under data heterogeneity, variance-reduction schemes are not very helpful, as suggested by preliminary work [12, 20, 3]. In short, data heterogeneity is still poorly understood in robust distributed learning. In particular, existing robustness guarantees are extremely conservative, and often refuted by empirical observations. Indeed, the heterogeneity model generally assumed is typically violated in practice and does not even cover basic machine learning tasks such as least-squares regression.
*Correspondence to: Youssef Allouah <youssef.allouah@epfl.ch>.
Our work addresses the aforementioned shortcomings of existing theory, by considering a more realistic heterogeneity model, called  $(G,B)$ -gradient dissimilarity [18]. This criterion characterizes data heterogeneity for a larger class of machine learning problems compared to prior works [12, 20, 3], and enables us to reduce the gap between theoretical guarantees and empirical observations. Before summarizing our contributions in Section 1.2, we briefly recall below the essentials of robust distributed learning and highlight the challenges of data heterogeneity.
1.1 Robust distributed learning under heterogeneity
Consider a system comprising  $n$  workers  $w_{1},\ldots ,w_{n}$  and a central server, where  $f$  workers of a priori unknown identity may be Byzantine. Each worker  $w_{i}$  holds a dataset  $\mathcal{D}_i$  composed of  $m$  data points from an input space  $\mathcal{X}$ , i.e.,  $\mathcal{D}_i\coloneqq \{x_1^{(i)},\dots,x_m^{(i)}\} \in \mathcal{X}^m$ . Given a model parameterized by  $\theta \in \mathbb{R}^d$ , each data point  $x$  incurs a loss  $\ell (\theta ;x)$  where  $\ell :\mathbb{R}^d\times \mathcal{X}\to \mathbb{R}$ . Thus, each worker  $w_{i}$  has a local empirical loss function defined as  $\mathcal{L}_i(\theta)\coloneqq \frac{1}{m}\sum_{x\in \mathcal{D}_i}\ell (\theta ;x)$ . Ideally, when all the workers are assumed honest (i.e., non-Byzantine), the server can compute a model minimizing the global average loss function given by  $\frac{1}{n}\sum_{i = 1}^{n}\mathcal{L}_i(\theta)$ , without requiring the workers to share their raw data points. However, this goal is rendered vacuous in the presence of Byzantine workers. A more reasonable goal for the server is to compute a model minimizing the global honest loss [16], i.e., the average loss of the honest workers. Specifically, denoting by  $\mathcal{H}\subseteq [n]$  where  $|\mathcal{H}| = n - f$ , the indices of honest workers, the goal in robust distributed learning is to solve the following optimization problem:2
$$
\min  _ {\theta \in \mathbb {R} ^ {d}} \mathcal {L} _ {\mathcal {H}} (\theta) := \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \mathcal {L} _ {i} (\theta). \tag {1}
$$
Because Byzantine workers may send bogus information and are unknown to the server, solving (even approximately) the optimization problem (1) is known to be impossible in general [26, 20]. The key reason for this impossibility is precisely data heterogeneity. Indeed, we cannot obtain meaningful robustness guarantees unless data heterogeneity is bounded across honest workers.
Modeling heterogeneity. Prior work on robustness primarily focuses on a restrictive heterogeneity bound we call  $G$ -gradient dissimilarity [12, 20, 3]. Specifically, denoting  $\| \cdot \|$  to be the Euclidean norm, the honest workers are said to satisfy  $G$ -gradient dissimilarity if for all  $\theta \in \mathbb{R}^d$ , we have
$$
\frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {\mathcal {H}} (\theta) \| ^ {2} \leq G ^ {2}. \tag {2}
$$
However, the above uniform bound on the inter-worker variance of local gradients may not hold in common machine learning problems such as least-squares regression, as we discuss in Section 3. In our work, we consider the more general notion of  $(G,B)$ -gradient dissimilarity, which is a prominent data heterogeneity model in the classical (Byzantine-free) distributed machine learning literature (i.e., when  $f = 0$ ) [18, 23, 27, 29]. Recent works have also adopted this definition in the context of Byzantine robust learning [20, 14], but did not provide tight analyses, as we discuss in Section 5. Formally,  $(G,B)$ -gradient dissimilarity is defined as follows.
Assumption 1  $((G,B)$ -gradient dissimilarity). The local loss functions of honest workers (represented by set  $\mathcal{H}$ ) are said to satisfy  $(G,B)$ -gradient dissimilarity if, for all  $\theta \in \mathbb{R}^d$ , we have<sup>3</sup>
$$
\frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {\mathcal {H}} (\theta) \| ^ {2} \leq G ^ {2} + B ^ {2} \| \nabla \mathcal {L} _ {\mathcal {H}} (\theta) \| ^ {2}.
$$
Under  $(G,B)$ -gradient dissimilarity, the inter-worker variance of gradients need not be bounded, and can grow with the norm of the global loss function's gradient at a rate bounded by  $B$ . Furthermore, this notion also generalizes  $G$ -gradient dissimilarity, which corresponds to the special case of  $B = 0$ .
We provide the first tight analysis on robustness to Byzantine workers in distributed learning under a realistic data heterogeneity model, specifically  $(G,B)$ -gradient dissimilarity. Our key contributions are summarized as follows.
2We denote by  $[n]$  the set  $\{1,\dots ,n\}$
3The dissimilarity inequality is equivalent to  $\frac{1}{|\mathcal{H}|}\sum_{i\in \mathcal{H}}\| \nabla \mathcal{L}_i(\theta)\| ^2\leq G^2 +\left(1 + B^2\right)\| \nabla \mathcal{L}_{\mathcal{H}}(\theta)\| ^2.$
Breakdown point. We establish a novel breakdown point for distributed learning under heterogeneity. Prior to our work, the upper bound on the breakdown point was simply  $\frac{1}{2}$  [26], i.e., when half (or more) of the workers are Byzantine, no algorithm can provide meaningful guarantees for solving (1). We prove that, under  $(G,B)$ -gradient dissimilarity, the breakdown point is actually  $\frac{1}{2 + B^2}$ . That is, the breakdown point of distributed learning is lower than  $\frac{1}{2}$  under heterogeneity due to non-zero growth rate  $B$  of gradient dissimilarity. We also confirm empirically that the breakdown point under heterogeneity can be much lower than  $\frac{1}{2}$ , which could not be explained prior to our work.
Tight error bounds. We show that, under the necessary condition  $\frac{f}{n} < \frac{1}{2 + B^2}$ , any robust distributed learning algorithm must incur an optimization error in
$$
\Omega \left(\frac {f}{n - (2 + B ^ {2}) f} \cdot G ^ {2}\right) \tag {3}
$$
on the class of smooth strongly convex loss functions. We also show that the above lower bound is tight. Specifically, we prove a matching upper bound for the class of smooth non-convex loss functions, by analyzing a robust variant of distributed gradient descent.
Proof techniques. To prove our new breakdown point and lower bound, we construct an instance of quadratic loss functions parameterized by their scaling coefficients and minima. While the existing lower bound under  $G$ -gradient dissimilarity can easily be obtained by considering quadratic functions with different minima and identical scaling coefficients (see proof of Theorem III in [20]), this simple proof technique fails to capture the impact of non-zero growth rate  $B$  in gradient dissimilarity. In fact, the main challenge we had to overcome is to devise a coupling between the parameters of the considered quadratic losses (scaling coefficients and minima) under the  $(G, B)$ -gradient dissimilarity constraint. Using this coupling, we show that when  $\frac{f}{n} \geq \frac{1}{2 + B^2}$ , the distance between the minima of the quadratic losses can be made arbitrarily large by carefully choosing the scaling coefficients, hence yielding an arbitrarily large error. We similarly prove the lower bound (3) when  $\frac{f}{n} < \frac{1}{2 + B^2}$ .
1.3 Paper outline
The remainder of this paper is organized as follows. Section 2 presents our formal robustness definition and recalls standard assumptions. Section 3 discusses some key limitations of previous works on heterogeneity under  $G$ -gradient dissimilarity. Section 4 presents the impossibility and lower bound results under  $(G, B)$ -gradient dissimilarity, along with a sketch of proof. Section 5 presents tight upper bounds obtained by analyzing robust distributed gradient descent under  $(G, B)$ -gradient dissimilarity. Full proofs are deferred to appendices A, B and C. Details on the setups of our experimental results are deferred to Appendix D.
2 Formal definitions
In this section, we state our formal definition of robustness and standard optimization assumptions. Recall that an algorithm is deemed robust to adversarial workers if it enables the server to approximate a minimum of the global honest loss, despite the presence of  $f$  Byzantine workers whose identity is a priori unknown to the server. In Definition 1, we state the formal definition of robustness.
Definition 1  $((f, \varepsilon)$ -resilience). A distributed algorithm is said to be  $(f, \varepsilon)$ -resilient if it can output a parameter  $\hat{\theta}$  such that
$$
\mathcal {L} _ {\mathcal {H}} (\hat {\theta}) - \mathcal {L} _ {*}, \mathcal {H} \leq \varepsilon ,
$$
where  $\mathcal{L}_{*,\mathcal{H}}\coloneqq \min_{\theta \in \mathbb{R}^d}\mathcal{L}_{\mathcal{H}}(\theta)$
Accordingly, an  $(f,\varepsilon)$ -resilient distributed algorithm can output an  $\varepsilon$ -approximate minimizer of the global honest loss function, despite the presence of  $f$  adversarial workers. Throughout the paper, we assume that  $\frac{f}{n} < \frac{1}{2}$ , as otherwise  $(f,\varepsilon)$ -resilience is in general impossible [26]. Note also that, for general smooth non-convex loss functions, we aim to find an approximate stationary point of the global honest loss instead of a minimizer, which is standard in non-convex optimization [8].
Standard assumptions. To derive our lower bounds, we consider the class of smooth strongly convex loss functions. We derive our upper bounds for smooth non-convex functions, and for functions satisfying the Polyak-Lojasiewicz (PL) inequality. This property relaxes strong convexity, i.e., strong
convexity implies PL, and covers learning problems which may be non-strongly convex such as least-squares regression [17]. We recall these properties in definitions 2 and 3 below.
Definition 2 (L-smoothness). A function  $\mathcal{L}\colon \mathbb{R}^d\to \mathbb{R}$  is  $L$  -smooth if, for all  $\theta ,\theta^{\prime}\in \mathbb{R}^{d}$  , we have
$$
\left\| \nabla \mathcal {L} \left(\theta^ {\prime}\right) - \nabla \mathcal {L} (\theta) \right\| \leq L \left\| \theta^ {\prime} - \theta \right\|.
$$
This is equivalent [28] to, for all  $\theta, \theta'$ , having  $|\mathcal{L}(\theta') - \mathcal{L}(\theta) - \langle \nabla \mathcal{L}(\theta), \theta' - \theta \rangle| \leq \frac{L}{2} \| \theta' - \theta \|^2$ .
Definition 3 ( $\mu$ -Polyak-Łojasiewicz (PL), strong convexity). A function  $\mathcal{L}\colon \mathbb{R}^d\to \mathbb{R}$  is  $\mu$ -PL if, for all  $\theta \in \mathbb{R}^d$ , we have
$$
2 \mu (\mathcal {L} (\theta) - \mathcal {L} _ {*}) \leq \| \nabla \mathcal {L} (\theta) \| ^ {2},
$$
where  $\mathcal{L}_* \coloneqq \min_{\theta \in \mathbb{R}^d} \mathcal{L}(\theta)$ . Function  $\mathcal{L}$  is  $\mu$ -strongly convex if, for all  $\theta, \theta' \in \mathbb{R}^d$ , we have
$$
\mathcal {L} \left(\theta^ {\prime}\right) - \mathcal {L} (\theta) - \langle \nabla \mathcal {L} (\theta), \theta^ {\prime} - \theta \rangle \geq \frac {\mu}{2} \| \theta^ {\prime} - \theta \| ^ {2}.
$$
Note that a function satisfies  $L$ -smoothness and  $\mu$ -PL inequality simultaneously only if  $\mu \leq L$ . Lastly, although not needed for our results to hold, when the global loss function  $\mathcal{L}_{\mathcal{H}}$  is  $\mu$ -PL, we will assume that it admits a unique minimizer, denoted  $\theta_{*}$ , for clarity.
3 Brittleness of previous approaches on heterogeneity
Under the  $G$ -gradient dissimilarity condition, presented in (2), prior work has established lower bounds [20] and matching upper bounds [3] for robust distributed learning. However,  $G$ -gradient dissimilarity is arguably unrealistic, since it requires a uniform bound  $G^2$  on the variance of workers' gradients over the parameter space. As a matter of fact,  $G$ -gradient dissimilarity does not hold in general for simple learning tasks such as least-squares regression, as shown in Observation 1 below.
Observation 1. In general,  $G$ -gradient dissimilarity (2) does not hold in least-squares regression.
Proof. Consider the setting given by  $\mathcal{X} = \mathbb{R}^2$ ,  $n = 2$ ,  $f = 0$ , and  $d = 1$ . For any data point  $(x_1, x_2) \in \mathcal{X}$ , consider the squared error loss  $\ell(\theta; (x_1, x_2)) = \frac{1}{2} (\theta \cdot x_1 - x_2)^2$ . Let the local datasets be  $\mathcal{D}_1 = \{(1,0)\}$ ,  $\mathcal{D}_2 = \{(0,1)\}$ . Note that for all  $\theta \in \mathbb{R}$ , we have  $\nabla \mathcal{L}_1(\theta) = \theta$ , and  $\nabla \mathcal{L}_2(\theta) = 0$ . This implies that  $\frac{1}{|\mathcal{H}|} \sum_{i \in \mathcal{H}} \| \nabla \mathcal{L}_i(\theta) - \nabla \mathcal{L}_{\mathcal{H}}(\theta) \|^2 = \theta^2/4$ , where  $\mathcal{H} = \{1,2\}$ , which is unbounded over  $\mathbb{R}$ . Hence, the condition of  $G$ -gradient dissimilarity cannot be satisfied for any  $G \in \mathbb{R}$ .
In contrast, the  $(G,B)$ -gradient dissimilarity condition, presented in Assumption 1, is more realistic, since it allows the variance across the local gradients to grow with the norm of the global gradient. This condition is common in the (non-robust) distributed learning literature [23, 18, 25, 29], and is also well-known in the (non-distributed) optimization community [8, 9, 32]. While  $G$ -gradient dissimilarity corresponds to the special case of  $(G,0)$ -gradient dissimilarity, we show in Proposition 1 below that a non-zero growth rate  $B$  of gradient dissimilarity allows us to characterize heterogeneity in a much larger class of distributed learning problems.
Proposition 1. Assume that the global loss  $\mathcal{L}_{\mathcal{H}}$  is  $\mu$ -PL and  $L$ -smooth, and that for each  $i \in \mathcal{H}$  local loss  $\mathcal{L}_i$  is convex and  $L_i$ -smooth. Denote  $L_{\max} \coloneqq \max_{i \in \mathcal{H}} L_i$ . Then, Assumption 1 is satisfied, i.e., the local loss functions satisfy  $(G, B)$ -gradient dissimilarity, with
$$
G ^ {2} = \frac {2}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} \left(\theta_ {*}\right) \| ^ {2} \quad a n d \quad B ^ {2} = \frac {2 L _ {\max }}{\mu} - 1. \tag {4}
$$
We present the proof of Proposition 1 in Appendix A for completeness. However, note that it can also be proved following existing results [32, 21] derived in other contexts. The  $(G,B)$ -gradient dissimilarity condition shown in Proposition 1 is arguably tight (up to multiplicative factor 2), in the sense that  $G^2$  cannot be improved in general. Indeed, as the  $(G,B)$ -gradient dissimilarity inequality should be satisfied for  $\theta = \theta_{*}$ ,  $G^2$  should be at least the variance of honest gradients at the minimum, i.e.,  $\frac{1}{|\mathcal{H}|}\sum_{i\in \mathcal{H}}\| \nabla \mathcal{L}_i(\theta_*)\|^2$ .
Gap between existing theory and practice. The theoretical limitation of  $G$ -gradient dissimilarity is exacerbated by the following empirical observation. We train a linear least-squares regression
model on the mg LIBSVM dataset [10]. The system comprises 7 honest and 3 Byzantine workers. We simulate extreme heterogeneity by having each honest worker hold one distinct point. We implement four well-studied Byzantine attacks: sign flipping (SF) [2], fall of empires (FOE) [33], a little is enough (ALIE) [5] and mimic [20]. More details on the experimental setup can be found in Appendix D. We consider the state-of-the-art robust variant of distributed gradient descent (detailed later in Section 5.1) that uses the NNM robustness scheme [3] composed with coordinate-wise trimmed mean. The empirical success on this learning task, which could not be explained by existing theory under  $G$ -gradient dissimilarity following Observation 1, is covered under  $(G,B)$ -gradient dissimilarity, as per Proposition 1. We present formal robustness guarantees under  $(G,B)$ -gradient dissimilarity later in Section 5. Additionally, through experimental evaluations in Section 5, we observe that even if  $G$ -gradient dissimilarity were assumed to be true, the bound  $G^2$  may be extremely large in practice, thereby inducing a non-informative error bound  $\mathcal{O}(f / n \cdot G^2)$  [20]. On the other hand, under  $(G,B)$ -gradient dissimilarity, we obtain tighter bounds matching empirical observations.
Figure 1: Evolution of the training losses (left) and the trajectories (right) of robust D-GD algorithm with NNM and coordinate-wise trimmed mean (see Section 5.1), on the  $mg$  LIBSVM least-squares regression task described in Section 3.  $f = 0$  corresponds to the case where the algorithm is run without Byzantine workers.
4 Fundamental limits on robustness under  $(G,B)$ -Gradient Dissimilarity
Theorem 1 below shows the fundamental limits on robustness in distributed learning under  $(G, B)$ -gradient dissimilarity. The result has twofold implications. On the one hand, we show that the breakdown point of any robust distributed learning algorithm reduces with the growth rate  $B$  of gradient dissimilarity. On the other hand, when the fraction  $f / n$  is smaller than the breakdown point, both  $G$  and  $B$  induce a lower bound on the learning error.
Theorem 1. Let  $0 < f < n/2$ . Assume that the global loss  $\mathcal{L}_{\mathcal{H}}$  is  $L$ -smooth and  $\mu$ -strongly convex with  $0 < \mu < L$ . Assume also that the honest local losses satisfy  $(G, B)$ -gradient dissimilarity (Assumption 1) with  $G > 0$ . Then, a distributed algorithm can be  $(f, \varepsilon)$ -resilient only if
$$
\frac {f}{n} <   \frac {1}{2 + B ^ {2}} \qquad a n d \qquad \varepsilon \geq \frac {1}{8 \mu} \cdot \frac {f}{n - (2 + B ^ {2}) f} \cdot G ^ {2}.
$$
Sketch of proof. The full proof is deferred to Appendix B. In the proof, we construct hard instances for  $(f,\varepsilon)$ -resilience, using a set of quadratic functions of the following form:
$$
\mathcal {L} _ {i} (\theta) = \frac {\alpha}{2} \left\| \theta - z \right\| ^ {2}, \quad \forall i \in \{1, \dots , f \},
$$
$$
\mathcal {L} _ {i} (\theta) = \frac {1}{2 K} \| \theta \| ^ {2}, \quad \forall i \in \{f + 1, \dots , n - f \},
$$
$$
\mathcal {L} _ {i} (\theta) = \frac {\alpha}{2} \| \theta \| ^ {2}, \quad \forall i \in \{n - f + 1, \dots , n \}.
$$
To prove the theorem, we consider two plausible scenarios corresponding to two different identities of honest workers, which are unknown to the algorithm. Specifically, in scenarios I and II, we assume the indices of honest workers to be  $S_{1} \coloneqq \{1, \dots, n - f\}$  and  $S_{2} \coloneqq \{f + 1, \dots, n\}$ , respectively.
We show that for all  $\theta \in \mathbb{R}^d$
$$
\max  \left\{\mathcal {L} _ {S _ {1}} (\theta) - \mathcal {L} _ {*, S _ {1}}, \mathcal {L} _ {S _ {2}} (\theta) - \mathcal {L} _ {*, S _ {2}} \right\} \geq \frac {\left(\frac {f}{n - f}\right) ^ {2} \alpha^ {2}}{8 \left(\frac {n - 2 f}{n - f} \frac {1}{K} + \frac {f}{n - f} \alpha\right)} \| z \| ^ {2}. \tag {5}
$$
That is, every model in the parameter space incurs an error that grows with  $\|z\|^2$ , in at least one of the two scenarios. Hence, an  $(f, \varepsilon)$ -resilient algorithm, by Definition 1, must guarantee optimization error  $\varepsilon$  in both scenarios I and II, which together with (5) implies that
$$
\varepsilon \geq \frac {\left(\frac {f}{n - f}\right) ^ {2} \alpha^ {2}}{8 \left(\frac {n - 2 f}{n - f} \frac {1}{K} + \frac {f}{n - f} \alpha\right)} \| z \| ^ {2}. \tag {6}
$$
At this point, to obtain the largest lower bounds possible, our goal is to maximize the right-hand side of 6, under the constraint that the loss functions induced by the triplet  $(\alpha, K, z)$  satisfy  $(G, B)$ -gradient dissimilarity,  $L$ -smoothness and  $\mu$ -strong convexity (simultaneously in both scenarios). We separately analyze this error in two cases: (i)  $\frac{f}{n} \geq \frac{1}{2 + B^2}$  and (ii)  $\frac{f}{n} < \frac{1}{2 + B^2}$ . In both cases, we construct a coupling between the values of  $z$  and  $K$  by having the norm  $\|z\|^2$  proportional to  $K$ . Specifically, in case (i), we show that the condition  $\frac{f}{n} \geq \frac{1}{2 + B^2}$  allows us to choose  $K$  arbitrarily large while satisfying  $(G, B)$ -dissimilarity. Thus,  $\|z\|^2$  being proportional to  $K$  means that  $\varepsilon$  is arbitrarily large as per (6). Similarly, in case (ii) where  $\frac{f}{n} < \frac{1}{2 + B^2}$ ,  $K$  cannot be arbitrarily large and carefully choosing a large possible value yields
$$
\varepsilon \geq \frac {1}{8 \mu} \cdot \frac {f}{n - (2 + B ^ {2}) f} \cdot G ^ {2}.
$$