SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kaiqi Jiang
kaiqij@princeton.edu
Dhruv Malik
dhruvm@andrew.cmu.edu
Yuanzhi Li
yuanzhil@andrew.cmu.edu
Adaptive optimization methods are well known to achieve superior convergence relative to vanilla gradient methods. The traditional viewpoint in optimization, particularly in convex optimization, explains this improved performance by arguing that, unlike vanilla gradient schemes, adaptive algorithms mimic the behavior of a second-order method by adapting to the global geometry of the loss function. We argue that in the context of neural network optimization, this traditional viewpoint is insufficient. Instead, we advocate for a local trajectory analysis. For iterate trajectories produced by running a generic optimization algorithm OPT, we introduce  $R_{\mathrm{med}}^{\mathrm{OPT}}$ , a statistic that is analogous to the condition number of the loss Hessian evaluated at the iterates. Through extensive experiments on language models where adaptive algorithms converge faster than vanilla gradient methods like SGD, we show that adaptive methods such as Adam bias the trajectories towards regions where  $R_{\mathrm{med}}^{\mathrm{Adam}}$  is small, where one might expect faster optimization. By contrast, SGD (with momentum) biases the trajectories towards regions where  $R_{\mathrm{med}}^{\mathrm{SGD}}$  is comparatively large. We complement these empirical observations with a theoretical result that provably demonstrates this phenomenon in the simplified setting of a two-layer linear network. We view our findings as evidence for the need of a new explanation of the success of adaptive methods, one that is different than the conventional wisdom.
The efficient minimization of a parameterized loss function is a core primitive in statistics, optimization and machine learning. Gradient descent (GD), which iteratively updates a parameter vector with a step along the gradient of the loss function evaluated at that vector, is a simple yet canonical algorithm which has been applied to efficiently solve such minimization problems with enormous success. However, in modern machine learning, and especially deep learning, one frequently encounters problems where the loss functions are high dimensional, non-convex and non-smooth. The optimization landscape of such problems is thus extremely challenging, and in these settings gradient descent often suffers from prohibitively high iteration complexity.
To deal with these difficulties and improve optimization efficiency, practitioners in recent years have developed many variants of GD. One prominent class of these GD variants is the family of adaptive algorithms [DHS11, TH $^{+}$ 12, KB15]. At a high level, adaptive methods scale the gradient with an adpatively selected preconditioning matrix, which is constructed via a moving average of past gradients. These methods are reminiscent of second order gradient descent, since they construct approximations to the Hessian of the loss functions, while remaining computationally feasible since they eschew full computation of the Hessian. A vast line of empirical work has demonstrated the superiority of adaptive methods over GD to optimize deep neural networks, especially on Natural Language Processing (NLP) tasks with transformers  $\left[\mathrm{VSP}^{+}17\right.$ , DCLT19].
From a theoretical perspective, adaptive methods are well understood in the traditional context of convex optimization. For instance, Duchi et al. [DHS11] show that when the loss function is convex, then the Adagrad algorithm yields regret guarantees that are provably as good as those obtained by using the best (diagonal) preconditioner in hindsight. The key mechanism that underlies this improved performance, is that the loss function has some global geometric property (such as sparsity or a coordinate wise bounded Lipschitz constant), and the algorithm adapts to this global geometry by adaptively selecting learning rates for features that are more informative.
However, in non-convex optimization, and deep learning in particular, it is highly unclear whether this simple characterization is sufficient to explain the superiority of adaptive methods over GD. Indeed, for large scale neural networks, global guarantees on the geometric properties of the loss are typically vacuous. For instance, for a 20-layer feedforward neural network, if we scale up the weights in each layer by a factor of 1.5, then the global Lipschitz constant of the network is scaled up by a factor of at least  $e^{10}$ . Hence it only makes sense to study convergence by looking at the local geometry of the loss along the trajectory of the optimization algorithm [ACH18].
Figure 1: (left) Training losses of  $\mathrm{SGD + M}$  starting from  $x_{\mathrm{SGD}}$  and  $x_{\mathrm{Adam}}$ . (right) The 10th largest value over median in the diagonal of loss Hessian (which can be viewed as a variant of  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$  defined in eq. (1)) for Adam and  $\mathrm{SGD + M}$ . Since the full Hessian is too big, here we selected several layers and randomly sampled 200 coordinates per layer to compute.
Figure 2: Training losses of Adam and  $\mathrm{SGD + M}$  on the sentence classification task described in Section 4.1.
Moreover, the interaction between an optimization algorithm and neural network geometry is highly complex — recent work has shown that geometric characteristics of iterates encountered during optimization is highly dependent on the choice of optimization algorithm and associated hyperparameters  $\mathrm{[LBD^{+}20,CKL^{+}21]}$ . For instance, Cohen et al.  $\mathrm{[CKL^{+}21]}$  demonstrate that while training neural networks with GD, the maximum eigenvalue of the Hessian evaluated at the GD iterates first increases and then plateaus at a level that is inversely proportional to the step size. The viewpoint from convex optimization, where a loss function has some (potentially) non-uniform but fixed underlying geometry that we must adapt to, is thus insufficient for neural networks, since the choice of optimization algorithm can actually interact with and influence the observed geometry significantly.
To provide another example of this interactive phenomenon, we consider the following experiment. On the same network training loss function  $f$ , we run stochastic gradient descent with momentum (SGD+M) and Adam to obtain two different trajectories. We select an iterate  $x_{\mathrm{Adam}}$  from the Adam trajectory and an iterate  $x_{\mathrm{SGD}}$  from the SGD trajectory, such that  $f(x_{\mathrm{Adam}}) = f(x_{\mathrm{SGD}})$ . We then run SGD+M with the same configuration twice, once from  $x_{\mathrm{Adam}}$  and once from  $x_{\mathrm{SGD}}$ . If the underlying geometry of the loss function  $f$  was truly fixed, then we would not expect a significant difference in the performance of running SGD+M from either of the two iterates. However, as shown in Figure 1, there is a noticeable difference in performance, and running SGD+M from  $x_{\mathrm{Adam}}$  achieves lower
loss than running  $\mathrm{SGD + M}$  from  $x_{\mathrm{SGD}}$ . This suggests that Adam may bias the optimization trajectory towards a region which is more favorable for rapid training. This motivates the following question.
How does adaptive optimization impact the observed geometry of a neural network loss function, relative to SGD (with momentum)?
The remainder of this paper is dedicated to answering the above question. To this end, for each iterate in a trajectory produced by running an optimization algorithm OPT, where the Hessian of the  $t$ th iterate is given by  $H^{(t)} \in \mathbb{R}^{d \times d}$ , we define the second order statistic  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$  in the following fashion. For the  $t$ th iterate in the trajectory, let  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$  be the ratio of maximum of the absolute entries of the diagonal of  $H^{(t)}$ , to the median of the absolute entries of the diagonal of  $H^{(t)}$ . Concretely, we define
$$
R _ {\text {m e d}} ^ {\mathrm {O P T}} (t) = \frac {\operatorname* {m a x} \left\{\left| H _ {i i} ^ {(t)} \right| \right\} _ {i = 1} ^ {d}}{\text {m e d i a n} \left\{\left| H _ {i i} ^ {(t)} \right| \right\} _ {i = 1} ^ {d}}. \tag {1}
$$
This statistic thus measures the uniformity of the diagonal of Hessian, where a smaller value of  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$  implies that the Hessian has a more uniform diagonal. It can also be viewed as a stable variant of the condition number. Instead of singular values, we choose diagonal entries because adaptive methods used in practice are coordinate-wise, which can be viewed as the diagonal scaling approaches. In Appendix A.9 we discuss this intuition in detail and compare  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$  with singular value-based metrics. As a supplementary result, in Appendix F, we demonstrate that the loss Hessian approaches diagonal during training for Adam and SGD+M. There has been prior theoretical work on overparameterized neural networks showing that a smaller condition number of Hessian, Neural Tangent Kernel [JGH18] etc. could yield to faster convergence rate for (S)GD [LZB22]. As for (diagonal) adaptive methods (e.g. Adagrad), they were original designed to adapt to the nonuniform diagonal geometry. Intuitively, a smaller  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$ , which implies more uniform diagonal geometry, could lead to faster convergence.
Armed with this statistic, we make the following contributions:
1. We focus on language models as on NLP tasks, adaptive algorithms show significantly faster convergence than SGD (with momentum). On a wide variety of neural network transformer architectures and language modeling datasets, we conduct experiments to compare how  $R_{\mathrm{med}}^{\mathrm{Adam}}(t)$  and  $R_{\mathrm{med}}^{\mathrm{SGDM}}(t)$  evolve over time, when Adam and SGD+M are run from the same initialization and with their optimal (initial) learning rates respectively. In each case, we demonstrate that the Adam trajectory attains  $R_{\mathrm{med}}^{\mathrm{Adam}}(t)$  values that are significantly smaller than the  $R_{\mathrm{med}}^{\mathrm{SGDM}}(t)$  values found by SGD+M. We show a simple example of this phenomenon in Figure 1(right). This suggests that relative to SGD+M, Adam biases the optimization trajectory to a region where the Hessian diagonal is more uniform. We call this phenomenon the uniformity of diagonal geometry for adaptive methods. Moreover, we demonstrate that a more uniform Hessian diagonal, characterized by smaller  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$ , is a contributing factor to faster optimization (see Section 4.3 for discussion). This suggests that a region where the Hessian diagonal is more uniform is also a region that is more amenable to rapid optimization.
2. We complement our empirical results with a theoretical analysis of this phenomenon in the simplified setting of large batch Adam and  $\mathrm{SGD + M}$ , on a two-layer linear network with  $d$ -dimensional input and hidden layer, and one dimensional output. We show that for a wide range of  $t$ ,  $R_{\mathrm{med}}^{\mathrm{Adam}}(t) = 1 \pm o(1)$  but  $R_{\mathrm{med}}^{\mathrm{SGDM}}(t) = \Omega (\log d)$ . Our proof reveals that Adam induces the weight matrices to have low rank whose leading singular vectors have certain type of uniformity (see Section 6 for discussion), a fact that we also observe empirically in large scale neural networks, suggesting that this may be a mechanism by which adaptive methods bias trajectories to have uniformity of diagonal geometry.
Existing analyses of adaptive methods. The vast majority of prior theoretical work on adaptive methods has focused on the blackbox setting [DHS11, KB15, CZT+20, RKK18, WWB20, DBBU20,
<sup>1</sup>Consider the case where one parameter has little impact on the loss, then the second derivative w.r.t. this parameter is almost zero, making  $\frac{\max\{|H_{ii}^{(t)}|\}_{i = 1}^d}{\min\{|H_{ii}^{(t)}|\}_{i = 1}^d}$  infinity. So we consider median which is more stable.
2Recall that the main theoretical bound in the original Adagrad paper [DHS11] is in terms of the diagonal scaling.
ENV21]. These works make minimal assumptions about the structure of the loss function, beyond (possibly) some global properties such as convexity or smoothness. These global properties (governed by parameters such as the smoothness parameter) are assumed to hold over the entire domain. Hence this style of analysis is worst case, since the resulting convergence bounds depend on polynomially on these global parameters. However, as we show in Section 3.1, in neural networks these parameters are prohibitively large. This worst case analysis is hence unlikely to explain the success of adaptive methods on neural networks. By contrast, our focus is on analyzing the local trajectory that is induced by running the optimization method.
Existing analyses of (S)GD on neural networks. There is an extensive literature on the analysis of GD/SGD in the non-blackbox setting, e.g. overparameterized neural networks, [DZPS18, JT20, AZLL19, AZLS19,  $\mathrm{ADH^{+}19}$ , LZB22]. However, it is unclear how to translate these analyses of GD/SGD, to an analysis that explains the gap between GD/SGD and adaptive methods.
Influence of algorithms on the loss geometry. In many simple convex settings, e.g. linear or logistic regression and the Neural Tangent Kernel [JGH18], the loss geometry is usually fixed and not influenced by learning algorithms. However, in neural networks the interaction between algorithms and loss landscapes is more complicated. Lewkowycz et al.  $\mathrm{[LBD^{+}20]}$  find a so-called catapult effect of initial learning rate on the training trajectory of SGD and related loss curvature. Cohen et al.  $\mathrm{[CKL^{+}21]}$  demonstrate that while training neural networks with GD, the maximum eigenvalue of the Hessian evaluated at the GD iterates first increases and then plateaus at a level that is inversely proportional to the step size. However, Cohen et al.  $\mathrm{[CKL^{+}21]}$  leave open the problem of whether similar interactive phenomena occur in algorithms that are not GD, including adaptive methods.
3 Overview of main results
3.1 Issues of prior analyses on adaptive methods
As is mentioned in Section 2, existing work on adaptive algorithms has mainly focused on black-box analysis assuming some global worst-case parameters. However, these global bounds can be extremely bad in complicated deep learning models, as is discussed in Section 1. To see this, we initialized a transformer model<sup>3</sup> with default initialization in Pytorch but chose a large gain<sup>4</sup>, and computed the smoothness parameter (denoted as  $l$ ) and the condition number (denoted as  $\kappa$ ) of loss Hessian on one layer. We observed that setting the gain as a large constant (e.g. 800) results in extremely large  $l$  and  $\kappa$  ( $l \geq 10^{7}$  and  $\kappa \geq 10^{10}$ ), which makes the convergence rates in prior black-box analysis vacuous.
The failure of global worst-case analysis implies that we need to focus on the local trajectory of algorithms. However, it is unclear whether when two optimization algorithms are used, they will have the same geometry in local trajectory or not. In particular, although in theory, adaptive algorithms can yield to a convergence rate with better dependency on certain local geometry of the function comparing to SGD (with momentum), it could still be the case that the local geometry along the trajectory of adaptive algorithm can be much worse than that of SGD (with momentum).
That motivates us to study the local geometry, especially that obtained by adaptive methods comparing to SGD (with momentum) in the paper. Motivated by the diagonal scaling of Adagrad and Adam for neural network training, we ask the follow main question in our paper:
How does the local diagonal geometry (diagonal of the loss Hessian) along the local trajectory of adaptive algorithms compare to that of SGD (with momentum)?
3.2 Overview of the experiments
As is discussed in Section 1, we consider  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$  defined in eq. (1) as a measurement of the uniformity of the loss Hessian diagonal and conduct experiments on different NLP tasks where adaptive methods converge faster. The detailed experimental setup will be stated in Section 4. To explore potential different patterns of different layers, we do the computation layer by layer. On a
<sup>3</sup>https://pytorch.org/tutorials/beginner/transformer_tutorial.html
4This refers to the gain parameter in some commonly used initialization functions of Pytorch, e.g. torch(nn.init.xavier.uniform_().
wide variety of transformer architectures and language modeling datasets from the same initialization, we observe that for the vast majority of layers:
When we train the neural network using Adam, the uniformity of diagonal geometry, measured by  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$  is smaller than that when we train using SGD+M from the same initialization.
Table 1 shows a typical example of  $R_{\mathrm{med}}^{\mathrm{Adam}}(t)$  compared to  $R_{\mathrm{med}}^{\mathrm{SGDM}}(t)$  on a sentence classification task using BERT-small [TCLT19, BDR21] (See Section 4.1 for details). We repeated the experiments for 12 times starting from the same initialization. Table 1 shows the averaged  $R_{\mathrm{med}}^{\mathrm{Adam}}(t)$  and  $R_{\mathrm{med}}^{\mathrm{SGDM}}(t)$  in some randomly selected layers. We also report the averaged  $\frac{R_{\mathrm{med}}^{\mathrm{SGDM}}(t)}{R_{\mathrm{med}}^{\mathrm{Adam}}(t)}$  and their standard deviations in the brackets. Figure 2 shows the corresponding training losses of one in these 12 experiments.
Table 1:  ${R}_{\text{med }}^{\text{Adam }}\left( t\right)$  and  ${R}_{\text{med }}^{\text{SGDM }}\left( t\right)$  in some layers,on the sentence classification task (see Section 4.1).
Layer#	Iteration 0	Iteration 750	Iteration 1250
RGDMmed(t)	RAdammed(t)	RGDMmed(t)	RAdammed(t)	RGDMmed(t)	RAdammed(t)	RGDMmed(t)
RAdammed(t)	RAdammed(t)	RAdammed(t)
9	15.7	15.7	12.76	9.65	1.45 (0.65)	11.43	14.24
12	22.63	22.63	13.17	7.41	1.92 (0.67)	10.62	9.67
15	9.35	9.35	80.57	53.52	1.65 (0.65)	100.65	61.80
17	82.37	82.37	405.02	223.56	1.91 (0.53)	423.28	337.32
18	31.32	31.32	17.07	13.24	1.43 (0.58)	18.15	15.63
22	47.13	47.13	233.72	72.67	3.54 (1.21)	158.38	93.13
24	31.17	31.17	17.52	17.34	1.13 (0.40)	13.51	14.23
To understand this phenomenon in a more principled point of view, in Section 5 we provide a formal proof of the statement in a simplified setting: large batch Adam and  $\mathrm{SGD + M}$  on a 2-layer linear network with 1-dimensional output.
4 The uniformity of diagonal geometry
As is mentioned in Section 3.2, we computed  $R_{\mathrm{med}}^{\mathrm{OPT}}(t)$  defined in eq. (1) on different language models. In this section, we present the results of SGD+M and Adam on different architectures and datasets. In Appendix A, we present the results of other adaptive algorithms.
During training we started from the same initial weights and used the same learning rate schedule (constant or decreasing) for  $\mathrm{SGD + M}$  and Adam. We tuned and chose the best (initial) learning rate of  $\mathrm{SGD + M}$ . The (initial) learning rate of Adam was set as a value under which Adam converged faster than  $\mathrm{SGD + M}$  with its best learning rate. The concrete values will be stated in later parts of this section. We used large batch sizes to make the training procedure stable. When computing Hessian, we also used large batch sizes. Due to the extremely large dimension, we did the computation on some uniformly selected coordinates, more precisely, 200 coordinates per layer.
4.1 Experiments on real datasets
Sentence classification task on BERT-small. We fine-tuned BERT-small [TCLT19, BDR21] on the IMDB dataset  $\left[\mathrm{MDP}^{+}11\right]$ : the task is to classify whether movie reviews are positive or negative. $^6$  The momentum parameter  $\beta$  in SGD was set as 0.9. The two momentum parameters  $(\beta_{1},\beta_{2})$  of Adam were set as (0.9, 0.999). We trained the model using linearly decreasing learning rates for 10 epochs (2500 iterations). The initial learning rates of SGD+M and Adam were 0.001 and 5e-5, respectively. As mentioned in Section 3.2, Figure 2 and Table 1 show the training losses and the comparison between  $R_{\mathrm{med}}^{\mathrm{Adam}}(t)$  and  $R_{\mathrm{med}}^{\mathrm{SGDM}}(t)$ , respectively.
Translation task. We trained a Seq2Seq network that uses Transformer to solve a machine translation task on Multi30k [EFSS16]: this task is to train a German to English translation model. The
$R_{\mathrm{med}}^{\mathrm{SGDM}}(t)$  values in Table 1 for most layers are roughly 1.4 to 2 times  $R_{\mathrm{med}}^{\mathrm{Adam}}(t)$  in corresponding layers. In practice, it can be considered significant because it might imply 1.4 to 2 times faster convergence.
<sup>6</sup>https://huggingface.co/docs/transformers/v4.16.2/en/training
<sup>7</sup>https://pytorch.org/tutorials/beginner/translation_transformer.html
momentum parameter  $\beta$  in SGD was set as 0.9. The two momentum parameters  $(\beta_{1},\beta_{2})$  of Adam were set as (0.9, 0.98). We trained the model using constant learning rates (0.03 for SGD+M and 1e-4 for Adam) for 60 epochs (1800 iterations). The experiments were repeated for 8 times starting from the same initialization. Figure 3(left) shows the training losses for one among them. Table 2a shows the averaged  $R_{\mathrm{med}}^{\mathrm{Adam}}(t)$ ,  $R_{\mathrm{med}}^{\mathrm{SGDM}}(t)$  and  $\frac{R_{\mathrm{med}}^{\mathrm{SGDM}}(t)}{R_{\mathrm{med}}^{\mathrm{Adam}}(t)}$  (with standard deviation in the brackets) in some randomly selected layers.
Figure 3: Training losses of Adam and  $\mathrm{SGD + M}$  for the translation task on (left) Multi30k (see Section 4.1) and (right) Multi30k but with randomly generated targets (see Section 4.2).
Figure 4: Singular values and  $R_{u}$  of the weight matrix in the 27-th layer on the translation task described in Section 4.1.
Table 2:  $R_{\mathrm{med}}^{\mathrm{Adam}}(t)$  and  $R_{\mathrm{med}}^{\mathrm{SGDM}}(t)$  in some layers for the translation task. (a) on Multi30k (see Section 4.1) and (b) on Multi30k but with randomly generated targets (see Section 4.2).
(a)
Layer#	Epoch 0	Epoch 30	Epoch 55
R6GDM (t)	R4Adm (t)	R6GDM (t)	R6GDM (t)	R6GDM (t)	R6GDM (t)	R6GDM (t)	R6GDM (t)	R6GDM (t)
mean	mean	mean	mean	mean	mean	mean	mean	mean
3	4.27	4.27	5.14	2.41	2.16 (0.75)	3.14	2	1.58 (0.41)
5	7.09	7.09	36.11	18.33	2.00 (0.42)	52.12	16.59	3.16 (0.64)
7	5.79	5.79	5.91	3.87	1.55 (0.32)	7.52	3.08	2.45 (0.56)
9	18.11	18.11	28.93	20.74	1.43 (0.28)	36.67	18	2.05 (0.18)
12	11.1	11.1	6.64	7.25	0.95 (0.21)	9.27	5.06	1.88 (0.54)
15	83.15	83.15	52.41	7.5	7.15 (1.63)	46.27	5.69	8.6 (3.06)
18	14.99	14.99	4.19	4.22	1.17 (0.45)	3.09	2.72	1.2 (0.46)
21	93.5	93.5	30.29	5.36	5.72 (1.05)	19.27	4.8	4.09 (0.86)
24	36.63	36.63	6.14	4.66	1.35 (0.31)	5.02	3.2	1.6 (0.36)
28	18.47	18.47	3.07	1.95	1.58 (0.16)	2.9	1.59	1.83 (0.14)
(b)
Layer#	Epoch 0	Epoch 30	Epoch 55
\( {R}_{\text{med }}^{\text{SGDM }}\left( t\right) \)	\( {R}_{\text{med }}^{\text{Adam }}\left( t\right) \)	\( {R}_{\text{med }}^{\text{SGDM }}\left( t\right) \)	\( {R}_{\text{med }}^{\text{Adam }}\left( t\right) \)	\( {R}_{\text{med }}^{\text{SGDM }}\left( t\right) \)	\( {R}_{\text{med }}^{\text{SGDM }}\left( t\right) \)	\( {R}_{\text{med }}^{\text{Adam }}\left( t\right) \)	\( {R}_{\text{med }}^{\text{SGDM }}\left( t\right) \)	\( {R}_{\text{med }}^{\mathrm{{SG}}\mathrm{{DM}}}\left( t\right) \)
3	4.82	4.82	3.98	1.8	2.23 (0.36)	3.79	1.61	2.36 (0.32)
5	8.04	8.04	46.06	45.84	1.01 (0.17)	47.83	34.18	1.41 (0.30)
7	5.69	5.69	44.77	3.92	11.79 (2.37)	46.5	2.74	17.4 (2.99)
9	11.89	11.89	317.34	55.61	5.81 (0.70)	351.85	46.54	7.61 (0.87)
12	19.73	19.73	133.39	3.91	34.17 (4.51)	145.09	2.97	49.49 (13.40)
15	32.12	32.12	462.74	51.53	9.03 (0.91)	492.73	50.57	9.84 (1.03)
18	19.79	19.79	74.6	6.59	11.8 (3.33)	79.02	3.58	22.75 (6.01)
21	26.94	26.94	767.31	48.89	16.4 (3.38)	797.49	36.88	21.98 (3.40)
24	34.72	34.72	467.75	9.15	52.57 (11.16)	602.03	3.51	172.65 (18.85)
28	13.13	13.13	19.8	2.22	8.99 (1.74)	19	1.63	11.7 (1.48)
4.2 Experiments on random datasets