SlowGuess/NeurIPS_2023_OCR huggingface dataset:
All the models are trained with the average loss over the batch  $\mathcal{D}_B = \{(x_\mu, y_\mu)\}_{\mu=1}^B$ , i.e.,  $L(x, y_{\mathcal{D}_B}) = 1 / B \sum_{\mu=1}^B \ell(x_\mu, y_\mu)$ , where  $\ell(.)$  is the loss function. This normalization, along with initialization, ensures that the loss is  $\mathcal{O}(1)$  at initialization.
Bias: Throughout this work, we have primarily focused on models without any bias for simplicity. In Appendix K, we demonstrate that bias does not have an appreciable impact on the results.
Batch size: We use a batch size of 512 and scale the learning rate as  $\eta = c / \lambda_0^H$  in all our experiments, unless specified. Appendix J shows results for a smaller batch size  $B = 32$ .
Learning rate: We scale the learning rate constant as  $c = 2^x$ , with  $x \in \{-1.0, \dots, x_{max}\}$  in steps of 0.1. Here,  $x_{max}$  is related to the maximum learning rate constant as  $c_{max} = 2^{x_{max}}$ .
Sharpness measurement: We measure sharpness using the power iteration method with 20 iterations. We found that 20 iterations suffice both for MSE and cross-entropy loss. For MSE loss, we use  $m = 2048$  randomly selected training examples for evaluating sharpness at each step. In comparison, we found that cross-entropy requires a large number of training examples to obtain a good approximation
of sharpness. Given the computational constraints, we use 4096 training examples to approximate sharpness for cross-entropy loss.
Averages over initialization and SGD runs: All the critical constants depend on both the random initializations and the SGD runs. In our experiments, we found that the fluctuations from initialization at large  $\frac{d}{w}$  outweigh the randomness coming from different SGD runs. Thus, we focus on initialization averages in all our experiments.
A.1 Compute usage
We utilized different computational resources depending on the task complexity. For less demanding tasks, we performed computation for a total of 2800 hours, utilizing a seventh of an NVIDIA A100 GPU. For more computationally intensive tasks, we utilized a full NVIDIA A100 GPU for a total 300 hours.
The main results of this paper can be reproduced using the associated GitHub repository:https://github.com/dayal-kalra/early-training.
A.3 Details of Figures in the main text:
Figure 1: A shallow CNN ( $d = 5$ ,  $w = 128$ ) in SP trained on the CIFAR-10 dataset with MSE loss for 1000 epochs using SGD with learning rates  $\eta = c / \lambda_0^H$  and batch size  $B = 512$ . We measure sharpness at every step for the first epoch, every epoch between 10 and 100 epochs, and every 10 epochs beyond 100.
Figure 2: (top panel) A wide  $(d = 5, w = 512)$  and (bottom panel) a deep CNN  $(d = 10, w = 128)$  in SP trained on the CIFAR-10 dataset with MSE loss for  $t = 10$  steps using vanilla SGD with learning rates  $\eta = c / \lambda_0^H$  and batch size  $B = 512$ .
Figure 3: Phase diagrams of early training of neural networks trained with MSE loss using SGD. Panels (a-c) show phase diagrams with width: (a) FCNs  $(d = 8)$  trained on the MNIST dataset, (b) CNNs  $(d = 7)$  trained on the Fashion-MNIST dataset, (c) ResNet  $(d = 18)$  trained on the CIFAR-10 (without batch normalization). Panels (d-f) show phase diagrams with depth: FCNs trained on the Fashion-MNIST dataset for different widths. Each data point in the figure represents an average of ten distinct initializations, and the solid lines represent a two-degree polynomial  $y = a + bx + cx^2$  fitted to the raw data points. Here, where  $x = 1/w$ , and  $y$  can take on one of three values:  $c_{loss}$ ,  $c_{\text{sharp}}$  and  $c_{max}$ .
Figure 4: (a) FCNs in SP with  $d \in \{4,8,16\}$  and  $w \in \{256,512,1024,2048\}$  trained on the MNIST dataset, (b) CNNs in SP with  $d \in \{5,7,10\}$  and  $w \in \{64,128,256,512\}$  trained on the Fashion-MNIST dataset, (c) ResNet in SP with  $d \in \{18,34\}$  and  $w \in \{32,64,128\}$  trained on the CIFAR-10 dataset (without batch normalization).
Figure 6: Phase diagrams of  $d = 8$  layer FCNs trained on the CIFAR-10 dataset using MSE, demonstrating the effect of output scale at initialization: (a) vanilla network, (b) centered network, and (c) network initialized with the last layer set to zero. The values of widths are the same as in Figure 3.
Figure 5: Normalized sharpness measured at  $c\tau = 200$  against the learning rate constant for 7-layer CNNs in SP trained on the CIFAR-10 dataset, with  $w \in \{128, 256, 512\}$ . Each data point is an average over five random initializations. Smoothing details are provided in Appendix I.2.
Figure 7: The phase diagram of the  $uv$  model trained with MSE loss using gradient descent with (a) the top eigenvalue of Hessian  $\lambda_t^H$ , (b) the trace of Hessian  $\operatorname{tr}(H_t)$  and (c) the square of the Frobenius norm  $\operatorname{tr}(H_t^T H_t)$  used as a measure of sharpness. In (a), the learning rate is scaled as  $\eta = c / \lambda_0^H$ , while in (b) and (c), the learning rate is scaled as  $\eta = k / \operatorname{tr}(H_0)$ . The vertical dashed line shows  $c = 2$  ( $k = 2$ ) for reference. Each data point is an average over 500 random initializations.
Figure 8: Training trajectories of the  $uv$  model with (a, b) large ( $w = 512$ ) and (c, d) small ( $w = 2$ ) width, trained for  $t = 10$  training steps on a single example ( $x, y = (1,0)$ ) with MSE loss using vanilla gradient descent with learning rates (a, c)  $c = 0.5$  and (b, d)  $c = 2.50$ .
Figure 9: Training trajectories of the  $uv$  model with (a) large ( $w = 512$ ) and (v) a small ( $w = 2$ ) widths trained for  $t = 10$  training steps on a single example ( $x, y = (1, 0)$ ). For the wide network,  $c_{loss} = 2.1$ ,  $c_{sharp} = 2.6$ ,  $c_{max} = 4.0$ , and for the narrow network,  $c_{loss} = 3.74$ ,  $c_{sharp} = 4.63$ ,  $c_{max} = 4.93$ .
B Additional results for the  $uv$  model
B.1 Details of the model
Consider a two-layer linear network in (NTP) with unit input-output dimensions
$$
f (x) = \frac {1}{\sqrt {w}} v ^ {T} u x, \tag {4}
$$
where  $x, f(x) \in \mathbb{R}$ . Here,  $u, v \in \mathbb{R}^w$  are trainable parameters, with each element initialized using the normal distribution,  $u_i, v_i \sim \mathcal{N}(0,1)$  for  $i \in \{1, \dots, w\}$ . The model is trained using MSE loss on a single training example  $(x,y) = (1,0)$ , which simplifies the loss to
$$
\mathcal {L} (u, v) = \frac {1}{2} f ^ {2}. \tag {5}
$$
The trace of the Hessian  $\operatorname{tr}(H)$  has a simple expression in terms of the norms of the weight vectors
$$
\operatorname {t r} (H) = \frac {x ^ {2}}{w} \left(\| u \| ^ {2} + \| v \| ^ {2}\right), \tag {6}
$$
which is equivalent to the NTK for this model. The Frobenius norm of the Hessian  $\|H\|_F$  can be written in terms of the loss  $\mathcal{L}$  and  $\operatorname{tr}(H)$
$$
\| H \| _ {F} ^ {2} = \operatorname {t r} (H) ^ {2} + 2 f ^ {2} \left(1 + \frac {2}{w}\right) = \operatorname {t r} (H) ^ {2} + 4 \mathcal {L} \left(1 + \frac {2}{w}\right) \tag {7}
$$
The gradient descent updates of the model trained using MSE loss on a single training example  $(x,y) = (1,0)$  are given by
(b)
Figure 10: (a) Normalized sharpness measured at  $\tau = 100$  steps against the learning rate constant for the  $uv$  model trained on  $(x,y) = (1,0)$ , with varying widths. Each data point is an average of over 500 initializations, where the shaded region depicts the standard deviation around the mean trend. (b, c) Smooth estimations of the first two derivatives,  $\chi_{\tau}$  and  $\chi_{\tau}^{\prime}$ , of the, averaged normalized sharpness wrt the learning rate constant. The vertical dashed lines denote  $c_{crit}$  estimated for each width, using the maximum of  $\chi_{\tau}^{\prime}$ . Here, we have removed the points beyond  $c = 3.5$  for the calculation of derivatives to avoid large fluctuations near the divergent phase. Smoothing details are described in Appendix I.2.
$$
v _ {t + 1} = v _ {t} - \eta f _ {t} \frac {1}{\sqrt {w}} u _ {t} x \tag {8}
$$
$$
u _ {t + 1} = u _ {t} - \eta f _ {t} \frac {1}{\sqrt {w}} v _ {t} x \tag {9}
$$
The update equations in function space can be written in terms of the trace of the Hessian  $\operatorname{tr}(H)$ .
$$
f _ {t + 1} = f _ {t} \left(1 - \eta \operatorname {t r} \left(H _ {t}\right) + \frac {\eta^ {2} f _ {t} ^ {2}}{w}\right) \tag {10}
$$
$$
\operatorname {t r} \left(H _ {t + 1}\right) = \operatorname {t r} \left(H _ {t}\right) + \frac {\eta f _ {t} ^ {2}}{w} (\eta \operatorname {t r} \left(H _ {t}\right) - 4).
$$
Figure 9 shows the training trajectories of the  $uv$  model trained on  $(x, y) = (1, 0)$  using MSE loss for 10 training steps. The model shows similar dynamics to those presented in Section 2. It is worth mentioning that the above equations have been analyzed in [49] at large width. In the following subsections, we extend their analysis by incorporating the higher-order terms to analyze the effect of finite width.
B.2 The intermediate saturation regime
The  $uv$  model trained on  $(x, y) = (1, 0)$  does not show the progressive sharpening and late-time regimes (iii) and (iv) described in Section 1. Hence, we can measure sharpness at the end of training to analyze how it is reduced upon increasing the learning rate and to compare it with the intermediate saturation regime results in Section 3.
Figure 10(a) shows the normalized sharpness measured at  $\tau = 100$  steps for various widths. This behavior reproduces the results observed in the intermediate saturation regime in Section 3. In particular, we can see stages (1) and (2), where  $\lambda_{\tau}^{H} / \lambda_{0}^{H}$  starts off fairly independent of learning rate constant  $c$ , and then dramatically reduces when  $c > 2$ ; stage (3), where  $\lambda_{\tau}^{H} / \lambda_{0}^{H}$  plateaus at a small value as a function of  $c$  is too close to the divergent phase in this model to be clearly observed. The corresponding derivatives of the averaged normalized sharpness,  $\chi_{\tau}$ , and  $\chi_{\tau}'$ , are shown in Figure 10(b, c). The vertical dashed lines denote  $c_{crit}$  estimated for each width, using the maximum of  $\chi_{\tau}'$ . We observe that  $c_{crit} = 2$  for all widths.
(c)
Figure 11: (a, b) The averaged loss at the first step  $\langle \mathcal{L}_1 / \mathcal{L}_0\rangle$  against the learning rate constant  $k$  for varying widths obtained from (a) inequality 16 and (b) numerical experiments. The intersection of  $\langle \mathcal{L}_1 / \mathcal{L}_0\rangle$  with the horizontal line  $y = 1$  depicts  $k_{loss}$ . The two vertical lines  $k = 2$  and  $k = 4$  mark the endpoints of  $k_{loss}$  at small and large widths. The shaded region in (b) shows the standard deviation around the mean trend. (c) The scaling of  $\lambda_0^H$  and  $\operatorname{tr}(H_0)$  with width.
B.3 Opening of the sharpness reduction phase in the  $uv$  model
This section shows that  $\mathcal{O}(1 / w)$  terms in Equation (10) effectively lead to the opening of the sharpness reduction phase with  $1 / w$  in the  $uv$  model. In Appendix B.2, we demonstrated that for the  $uv$  model,  $c_{\text{crit}} = 2$  for all values of widths. Hence, it suffices to show that  $c_{\text{loss}}$  increases from the value 2 as  $1 / w$  increases. We do so by finding the smallest  $k$  such that the averaged loss over initializations increases during early training.
It follows from Equation 10 that the averaged loss increases in the first training step if the following holds
$$
\left\langle \frac {\mathcal {L} _ {1}}{\mathcal {L} _ {0}} \right\rangle = \left\langle \left(1 - \eta \operatorname {t r} \left(H _ {0}\right) + \frac {\eta^ {2} f _ {0} ^ {2}}{w}\right) ^ {2} \right\rangle > 1, \tag {11}
$$
where  $\langle .\rangle$  denotes the average over initializations. On scaling the learning rate with trace as  $\eta = k / \mathrm{tr}(H_0)$ , we have
$$
\left\langle \frac {\mathcal {L} _ {1}}{\mathcal {L} _ {0}} \right\rangle = \left\langle \left(1 - k + \frac {k ^ {2}}{w} \frac {f _ {0} ^ {2}}{\operatorname {t r} \left(H _ {0}\right) ^ {2}}\right) \right\rangle > 1 \tag {12}
$$
$$
\left\langle \frac {\mathcal {L} _ {1}}{\mathcal {L} _ {0}} \right\rangle = \left((1 - k) ^ {2} + 2 (1 - k) \frac {k ^ {2}}{w} \left\langle \frac {f _ {0} ^ {2}}{\operatorname {t r} \left(H _ {0}\right) ^ {2}} \right\rangle + \frac {k ^ {4}}{w ^ {2}} \left\langle \frac {f _ {0} ^ {4}}{\operatorname {t r} \left(H _ {0}\right) ^ {4}} \right\rangle\right) > 1. \tag {13}
$$
The required two averages have the following expressions as shown in Appendix B.8.
$$
\left\langle \frac {f _ {0} ^ {2}}{T r \left(H _ {0}\right) ^ {2}} \right\rangle = \frac {w}{4 (w + 1)} \tag {14}
$$
$$
\left\langle \frac {f _ {0} ^ {4}}{T r (H _ {0}) ^ {4}} \right\rangle = \frac {3 (w + 2) w ^ {3}}{1 6} \frac {\Gamma (w)}{\Gamma (w + 4)}. \tag {15}
$$
Inserting the above expressions in Equation 13, on average the loss increases in the very first step if the following inequality holds
$$
\left\langle \frac {\mathcal {L} _ {1}}{\mathcal {L} _ {0}} \right\rangle = \left((1 - k) ^ {2} + \frac {k ^ {2} (1 - k)}{2 (w + 1)} + \frac {3 k ^ {4}}{1 6 (w + 3) (w + 1)}\right) > 1 \tag {16}
$$
The graphical representation of the above inequality shown in Figure 11(a) is in excellent agreement with the experimental results presented in Figure 11(b).
Let us denote  $k_{loss}^{\prime}$  as the minimum learning rate constant such that the average loss increases in the first step. Similarly, let  $k_{loss}$  denote the learning rate constant if the loss increases in the first 10 steps. Then,  $k_{loss}^{\prime}$  increases from the value 2 as  $1 / w$  increases as shown in Figure 11(a). By comparison, the trace reduces at any step if  $\eta \operatorname{tr}(H_t) < 4$ . At initialization, this condition becomes  $k < 4$ . Hence, for  $k < k_{loss}^{\prime}$ , both the loss and trace monotonically decrease in the first training step. These arguments can be extended to later training steps, revealing that the loss and trace will continue to decrease for  $k < k_{loss}^{\prime}$ .
Next, let  $\eta_{loss}$  denote the learning rate corresponding to  $c_{loss}$ . Then, we have  $\eta_{loss} = \frac{c_{loss}}{\lambda_0^H} = \frac{k_{loss}}{\operatorname{tr}(H_0)}$ , implying
$$
c _ {l o s s} = k _ {l o s s} \frac {\lambda_ {0} ^ {H}}{\operatorname {t r} \left(H _ {0}\right)}. \tag {17}
$$
Figure 11(c) shows that  $\lambda_0^H \geq \mathrm{tr}(H_0)$  for all widths, implying  $c_{loss} \geq k_{loss}$ . Hence,  $c_{loss}$  increases with  $1/w$  as observed in Figure 7(a). In Appendix B.2, we demonstrated that for the  $uv$  model,  $c_{crit} = 2$  for all values of widths. Incorporating this with  $c_{loss}$  increases with  $1/w$ , we have sharpness reduction phase opening up as  $1/w$  increases.
B.4 Opening of the loss catapult phase at finite width
In this section, we use the Frobenius norm of the Hessian  $\|H\|_F$  as a proxy for the sharpness and demonstrate the emergence of the loss-sharpsness catapult phase at finite width. In particular, We analyze the expectation value  $\langle \mathrm{tr}(H^T H) \rangle$  after the first training step near  $k = k_{loss}$  and show that  $k_{loss} \leq k_{frob}$ , with the difference increasing with  $1/w$ . First, we write  $\mathrm{tr}(H_t^T H_t)$  in terms of  $\mathcal{L}_t$  and  $\mathrm{tr}(H_t)$
$$
\operatorname {t r} \left(H _ {t} ^ {T} H _ {t}\right) = \operatorname {t r} \left(H _ {t}\right) ^ {2} + 4 \left(1 + \frac {2}{w}\right) \mathcal {L} _ {t}. \tag {18}
$$
Next, using Equations 1, we write down the change in  $\mathrm{tr}(H_t^T H_t)$  after the first training step in terms of  $\mathrm{tr}(H_0)$  and  $\mathcal{L}_0$
$$
\Delta \operatorname {t r} \left(H _ {1} ^ {T} H _ {1}\right) = \operatorname {t r} \left(H _ {1} ^ {T} H _ {1} ^ {T}\right) - \operatorname {t r} \left(H _ {0} ^ {T} H _ {0}\right) = \operatorname {t r} \left(H _ {1}\right) ^ {2} - \operatorname {t r} \left(H _ {0}\right) ^ {2} + 4 \left(1 + \frac {2}{w}\right) \left(\mathcal {L} _ {1} - \mathcal {L} _ {0}\right)
$$
$$
\Delta \operatorname {t r} \left(H _ {1} ^ {T} H _ {1}\right) = \frac {\eta f _ {0} ^ {2}}{w} (\eta \operatorname {t r} \left(H _ {0}\right) - 4) \left[ \frac {\eta f _ {0} ^ {2}}{w} (\eta \operatorname {t r} \left(H _ {0}\right) - 4) + 2 \operatorname {t r} \left(H _ {0}\right) \right] + 4 \left(1 + \frac {2}{w}\right) \left(\mathcal {L} _ {1} - \mathcal {L} _ {0}\right) \tag {19}
$$
Next, we substitute  $\eta = k / \operatorname{tr}(H_0)$  to obtain the above equation as a function of  $k$ :
$$
\Delta \operatorname {t r} \left(H _ {1} ^ {T} H _ {1}\right) = \frac {k (k - 4)}{w} \left[ \frac {k (k - 4)}{w} \frac {f _ {0} ^ {4}}{\operatorname {t r} \left(H _ {0}\right) ^ {2}} + 2 f _ {0} ^ {2} \right] + 4 \left(1 + \frac {2}{w}\right) \left(\mathcal {L} _ {1} - \mathcal {L} _ {0}\right) \tag {20}
$$
Finally, we calculate the expectation value of  $\langle \Delta \operatorname{tr}(H_1^T H_1) \rangle$
$$
\left\langle \Delta \operatorname {t r} \left(H _ {1} ^ {T} H _ {1}\right) \right\rangle = \frac {k (k - 4)}{w} \left[ \frac {k (k - 4)}{w} \left\langle \frac {f _ {0} ^ {4}}{\operatorname {t r} \left(H _ {0}\right) ^ {2}} \right\rangle + 2 \left\langle f _ {0} ^ {2} \right\rangle \right] + 4 \left(1 + \frac {2}{w}\right) \left\langle \mathcal {L} _ {1} - \mathcal {L} _ {0} \right\rangle , \tag {21}
$$
Figure 12: The relationship between the critical constants for the  $uv$  model trained on a single training examples  $(x,y) = (1,0)$  with MSE loss using gradient descent. Each data point corresponds to a random initialization
by estimating  $\left\langle \frac{f_0^4}{\mathrm{tr}(H_0)^2}\right\rangle$  using the approach demonstrated in the previous section.
$$
\left\langle \frac {f _ {0} ^ {4}}{\operatorname {t r} \left(H _ {0}\right) ^ {2}} \right\rangle = \frac {3 w}{4 (w + 3)}. \tag {22}
$$
Inserting  $\left\langle \frac{f_0^4}{\mathrm{tr}(H_0)^2}\right\rangle$  in Equation 21 along with  $\langle f_0^2\rangle = 1$ , we have
$$
\left\langle \Delta \operatorname {t r} \left(H _ {1} ^ {T} H _ {1}\right) \right\rangle = \underbrace {\frac {k (k - 4)}{w} \left[ \frac {3 k (k - 4)}{4 (w + 3)} + 2 \right]} _ {I (k, w)} + 4 \left(1 + \frac {2}{w}\right) \left\langle \mathcal {L} _ {1} - \mathcal {L} _ {0} \right\rangle \tag {23}
$$
At infinite width, the above equation reduces to  $\left\langle \Delta \operatorname{tr}(H_1^T H_1)\right\rangle = 4\left\langle \mathcal{L}_1 - \mathcal{L}_0\right\rangle$ , and hence,  $k_{frob} = k_{loss}$ . For any finite width,  $I(k,w) < 0$  for  $0 < k < 4$ . At  $k \leq k_{loss}$ ,  $\mathcal{L}_1 - \mathcal{L}_0 \leq 0$ , and therefore  $\left\langle \Delta \operatorname{tr}(H_1^T H_1)\right\rangle < 0$ . In order for the sharpness to catapult, we require  $\left\langle \Delta \operatorname{tr}(H_1^T H_1)\right\rangle > 0$  and therefore  $k_{frob} > k_{loss}$ . As  $1/w$  increases  $|I(k,w)|$  also increases, which means a higher value of  $\mathcal{L}_1 - \mathcal{L}_0$  is required to reach a point where  $\left\langle \Delta \operatorname{tr}(H_1^T H_1)\right\rangle \geq 0$ . Thus  $k_{frob} - k_{loss}$  increases with  $1/w$ .
B.5 The early training trajectories
Figure 9 shows the early training trajectories of the  $uv$  model with large  $(w = 512)$  and small  $(w = 2)$  widths. The dynamics depicted show several similarities with early training dynamics of real-world models shown in Figure 2. At small widths, the loss catapults at relatively higher learning rates (specifically, at  $c_{loss} = 3.74$ , which is significantly higher than the critical value of  $c_{crit} = 2$ ).
B.6 Relationship between critical constants
Figure 12 shows the relationship between various critical constants for the  $uv$  model. The data show that the inequality  $c_{loss} \leq c_{sharp} \leq c_{max}$  holds for every random initialization of the  $uv$  model.
B.7 Phase diagrams with error bars
This section shows the variation in the phase diagram boundaries of the  $uv$  model shown in Figure 7(a, b). Figure 13 shows these phase diagrams. Each data point is an average of over 500 initializations. The horizontal bars around each data point indicate the region between  $25\%$  and  $75\%$  quantile.
B.8 Derivation of the expectation values
Here, we provide the detailed derivation of the averages  $\left\langle \frac{f_0^2}{Tr(H_0)^2} \right\rangle$  and  $\left\langle \frac{f_0^4}{Tr(H_0)^4} \right\rangle$ . We begin by finding the average  $\left\langle \frac{f_0^2}{Tr(H_0)^2} \right\rangle$
(b)
Figure 13: The phase diagram of the  $uv$  model trained with MSE loss using gradient descent with (a) sharpness  $\lambda_t^H$  (b) trace of Hessian  $\mathrm{tr}_0^H$  and (c) the square of the Frobenius norm  $\mathrm{tr}(H_t^T H_t)$  used as a measure of sharpness. In (a), the learning rate is scaled as  $\eta = c / \lambda_0^H$ , while in (b) and (c), the learning rate is scaled as  $\eta = k / \mathrm{tr}(H_0)$ . Each data point denotes an average of over 500 initialization, and the smooth curve represents a 2-degree polynomial fitted to the raw data. The horizontal bars around the average data point indicate the region between  $25\%$  and  $75\%$  quantile.
$$
\left\langle \frac {f _ {0} ^ {2}}{T r (H _ {0}) ^ {2}} \right\rangle = w \int_ {- \infty} ^ {\infty} \prod_ {i = 1} ^ {w} \left(\frac {d v _ {i} d u _ {i}}{2 \pi}\right) \exp \left(- \frac {\| u \| ^ {2} + \| v \| ^ {2}}{2}\right) \frac {\sum_ {j , k = 1} ^ {w} u _ {j} v _ {j} u _ {k} v _ {k}}{\left(\| u \| ^ {2} + \| v \| ^ {2}\right) ^ {2}}, \tag {24}
$$
where  $\|\cdot\|$  denotes the norm of the vectors.
The above integral is non-zero only if  $j = k$ . Hence, it is a sum of  $w$  identical integrals. Without any loss of generality, we solve this integral for  $j = 1$  and multiply by  $w$  to obtain the final result, i.e.,
$$
\left\langle \frac {f _ {0} ^ {2}}{T r (H _ {0}) ^ {2}} \right\rangle = w ^ {2} \int_ {- \infty} ^ {\infty} \prod_ {i = 1} ^ {w} \left(\frac {d v _ {i} d u _ {i}}{2 \pi}\right) \exp \left(- \frac {\| u \| ^ {2} + \| v \| ^ {2}}{2}\right) \frac {u _ {1} ^ {2} v _ {1} ^ {2}}{\left(\| u \| ^ {2} + \| v \| ^ {2}\right) ^ {2}} \tag {25}
$$
Consider a transformation of  $u, v \in \mathbb{R}^w$  into  $w$  dimensional spherical coordinates such that
$$
u _ {1} = r _ {u} \cos \varphi_ {u _ {1}}, \quad v _ {1} = r _ {v} \cos \varphi_ {v _ {1}}, \tag {26}
$$
which yields,
$$
\left\langle \frac {f _ {0} ^ {2}}{T r (H _ {0}) ^ {2}} \right\rangle = \frac {w ^ {2}}{(2 \pi) ^ {w}} \int d r _ {u} d r _ {v} d \Omega_ {u, w} d \Omega_ {v, w} r _ {u} ^ {w - 1} r _ {v} ^ {w - 1} \exp \left(- \frac {r _ {u} ^ {2} + r _ {v} ^ {2}}{2}\right) \frac {r _ {u} ^ {2} \cos^ {2} \varphi_ {u 1} r _ {v} ^ {2} \cos^ {2} \varphi_ {u 1}}{(r _ {u} ^ {2} + r _ {v} ^ {2}) ^ {2}}
$$
$$
\left\langle \frac {f _ {0} ^ {2}}{T r (H _ {0}) ^ {2}} \right\rangle = \frac {w ^ {2}}{(2 \pi) ^ {w}} \int d r _ {u} d r _ {v} \exp \left(- \frac {r _ {u} ^ {2} + r _ {v} ^ {2}}{2}\right) \frac {r _ {u} ^ {2} r _ {v} ^ {2}}{\left(r _ {u} ^ {w + 1} + r _ {v} ^ {w + 1}\right) ^ {2}} \int d \Omega_ {u, w} d \Omega_ {v, w} \cos^ {2} \varphi_ {u _ {1}} \cos^ {2} \varphi_ {v _ {1}} \tag {28}
$$
$$
\left\langle \frac {f _ {0} ^ {2}}{T r (H _ {0}) ^ {2}} \right\rangle = \frac {w ^ {2}}{(2 \pi) ^ {w}} \underbrace {\int d r _ {u} d r _ {v} \exp \left(- \frac {r _ {u} ^ {2} + r _ {v} ^ {2}}{2}\right) \frac {r _ {u} ^ {2} r _ {v} ^ {2}}{\left(r _ {u} ^ {w + 1} + r _ {v} ^ {w + 1}\right) ^ {2}}} _ {I _ {r}} \left(\underbrace {\int d \Omega_ {w} \cos^ {2} \varphi_ {1}} _ {I _ {\varphi}}\right) ^ {2}, \tag {29}
$$
where  $d\Omega$  denotes the  $w$  dimensional solid angle element. Here, we denote the radial and angular integrals by  $I_{r}$  and  $I_{\varphi}$  respectively. The radial integral  $I_{r}$  is
$$
I _ {r} = \int_ {0} ^ {\infty} d r _ {u} d r _ {v} \frac {r _ {u} ^ {w + 1} r _ {v} ^ {w + 1}}{\left(r _ {u} ^ {2} + r _ {v} ^ {2}\right) ^ {2}} \exp \left(- \frac {r _ {u} ^ {2} + r _ {v} ^ {2}}{2}\right). \tag {30}
$$
Let  $r_u = R\cos \theta$  and  $r_v = R\sin \theta$  with  $R\in [0,\infty)$  and  $\theta \in [-\frac{\pi}{2},\frac{\pi}{2} ]$ , then we have