SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The KL divergence The KL divergence is a standard asymmetrical measure for dissimilarity between probability distributions. For two probability distributions  $P$  and  $Q$ , it is given by  $\mathcal{D}_{KL}(P\parallel Q) = \int_{\mathcal{X}}P(x)\log (P(x) / Q(x))dx$ . For Gaussian variables, it is computed as
$$
K L \left(z _ {1} \| z _ {2}\right) = \log \frac {\sigma_ {1}}{\sigma_ {1}} + \frac {\sigma_ {1} ^ {2} + \left(\mu_ {1} - \mu_ {1}\right) ^ {2}}{\sigma_ {1} ^ {2}} - \frac {1}{2} \tag {18}
$$
The KL divergence mainly prioritizes same order-of-magnitude variances, and will initially query the same location multiple times to assess noise levels. Thereafter, it tends to query in an axis-aligned fashion, close to previous queries, to attain information regarding the length scales, but places a low priority on global exploration.