SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 6. A standard variance is used here for convenience. In general, if  $\text{var} = o(\mu)$  as  $\mu \to +\infty$  (e.g. a bounded var), similar analysis and results are supposed to hold. However, this is different when  $\text{var} = \Theta(\mu)$  as  $\mu \to +\infty$ , since the modes are not separated in this case, and we can not characterize modes shift by simply varying  $\mu$ .
Remark 7. Simply scaling down inputs seems not to resolve the adverse effect of modes shift, since the ground truth  $\mu$  is unknown. One can use the input scale to approximate  $\mu$  on toy datasets, but it is not that trivial in practice, particularly for real-world applications with multiple high-dimensional modes in varied scales.
The model-target inconsistency. Informally, there is inconsistency between the score network model and target score function. In fact, given the target Gaussian mixture  $p_0(x) = q_1 \mathcal{N}(x; -\mu, 1) + q_2 \mathcal{N}(x; \mu, 1)$ , the target score function is
$$
s _ {0} (x) = - x + \frac {q _ {2} \mathcal {N} (x ; \mu , 1) - q _ {1} \mathcal {N} (x ; - \mu , 1)}{q _ {2} \mathcal {N} (x ; \mu , 1) + q _ {1} \mathcal {N} (x ; - \mu , 1)} \mu ,
$$
which gives  $s_0(x) \approx -x + \mu, x \geq 0$  and  $s_0(x) \approx -x - \mu, x \leq 0$ . While the score network model is
$$
s _ {0, \theta} (x) \approx \left(\frac {1}{m} \sum_ {i: w _ {i} > 0} a _ {i} w _ {i}\right) x + \left(\frac {1}{m} \sum_ {i: w _ {i} > 0} a _ {i} \pmb {u} _ {i} ^ {\top} \pmb {e} (0)\right),
$$
where " $\approx$ " holds since  $|x| = \Theta(\mu)$  by (40) ( $\mu \gg 1$ ). Both  $s_0$  and  $s_{0,\theta}$  are linear functions, but they have unmatched scales in slopes and intercepts:  $O(1)$  and  $O(\mu)$  for  $s_0$ , but both  $O(a)$ 's for  $s_{0,\theta}$ . That is, modeling Gaussian mixtures with large modes distances as random feature models is inconsistent.
Remark 8. Recall that the goal of this work is to estimate  $D_{\mathrm{KL}}\left(p_0\| p_{0,\hat{\theta}_n(\tau)}\right)$ , which aims at the density estimation characterization. This is different from [32], where the equivalence of learned and target data manifolds (support sets) is studied.
B.1 Early-Stopping Generalization
We illustrate the early-stopping generalization gap established in Theorem 1 using the Adam optimizer. All the configurations remain the same as Section 4.1 except that the learning rate is now  $10^{-3}$ .
From Figure 8, one can observe that the KL divergence achieves its minimum at the 1000th training epoch, and it starts to increase after this turning point. The plot is aligned with Theorem 1, which states that there exists an optimal early-stopping time when the model can generalize well, indicating the effectiveness of the upper bound. Further, the KL divergence begins to oscillate after the minimum point (1000th training epoch), which may suggest a phase transition in the KL divergence dynamics, and the transition point is around the optimal early-stopping time. The finding is aligned with SGD setting.
Figure 8: The KL divergence dynamics under the Adam optimizer.
B.2 Modes Shift Effect
We further test the relationship between the modes' distance and the generalization performance using the Adam optimizer under the same configurations.
In Figure 9 and Figure 10, it is shown that the training of modeled distributions exhibits the same two-stage dynamics as the SGD setting, indicating that the modes shift effect holds not particularly for a certain optimizer.
Figure 9: The Adam training dynamics when the distance between two modes is  $6(\mu = 3)$ .
Figure 10: The Adam training dynamics when the distance between two modes is  $30(\mu = 15)$ .
B.3 Model Capacity Dependency
We also numerically study the dependency of generalization on the model capacity. Following the same configurations in Section 4.1.1, we conduct experiments for different hidden dimensions varying from  $2^{1}$  to  $2^{11}$ .
In Figure 11, the left plot shows the KL divergence from the trained distribution at the 1000th training epoch to the target distribution, and the right plot shows the time duration of the modeled distribution to generalize. Here, the generalization criterion we select is  $D_{\mathrm{KL}} \leq 10^{-1}$ , and we stop training at epoch = 10000. Both the two plots in Figure 11 indicate that increasing model capacity benefits the generalization, which also verifies the corresponding theoretical results (m-dependency in Theorem 1 and Theorem 2).
Figure 11: Left: The KL divergence from the model trained after 1000 epochs to the target distribution under different hidden dimensions. Right: The earliest training epoch when  $D_{\mathrm{KL}} \leq 10^{-1}$ . Here, "×" means that the model does not generalize when the training is stopped at the maximum training epoch 10000.