SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[37] L. Ellam, H. Strathmann, M.A. Girolami, and I. Murray. A determinant-free method to simulate the parameters of large gaussian fields. Stat, 6:271-281, 2017.
[38] P. Virtanen, R. Gommers, T.E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S.J. van der Walt, M. Brett, J. Wilson, J.K. Millman, N. Mayorov, A.R.J. Nelson, E. Jones, R. Kern, E. Larson, C.J. Carey, I. Polat, Y. Feng, E.W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cirmrnan, I. Henriksen, E.A. Quintero, C.R. Harris, A.M. Archibald, A.H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17: 261-272, 2020.
[39] T. Kiwaki. Variational optimization of annealing schedules. ArXiv, 2015. doi: 10.48550/arXiv.1502.05313.
[40] R. M. Neal. Estimating ratios of normalizing constants using linked importance sampling. ArXiv, 2005. doi: 10.48550/arXiv.math/0511216.
[41] R. Brekelmans, V. Masrani, F.D. Wood, G. Ver Steeg, and A.G. Galstyan. All in the exponential family: Bregman duality in thermodynamic variational inference. In International Conference on Machine Learning (ICML), 2020.
[42] S. Goshtasbpour, V. Cohen, and F. Perez-Cruz. Adaptive annealed importance sampling with constant rate progress. In International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 11642-11658. PMLR, 2023.
[43] N. Chopin and O. Papaspiliopoulos. An Introduction to Sequential Monte Carlo. Springer Series in Statistics. Springer International Publishing, 2020. ISBN 9783030478452.
[44] P. Del Moral and A. Doucet. Sequential monte carlo samplers. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68, 2002.
[45] C. Dai, J. Heng, P.E. Jacob, and N. Whiteley. An invitation to sequential monte carlo samplers. Journal of the American Statistical Association, 117:1587-1600, 2020.
[46] S. Syed, V. Romaniello, T. Campbell, and A. Bouchard-Cote. Parallel tempering on optimized paths. In International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 10033–10042. PMLR, 2021.
[47] G. Behrens, N. Friel, and M. Hurn. Tuning tempered transitions. Statistics and Computing, 22: 65-78, 2010.
[48] M. Pihlaja, M. Gutmann, and A. Hyvarinen. A family of computationally efficient and simple estimators for unnormalized statistical models. In Uncertainty in Artificial Intelligence (UAI), 2010.
[49] L. Wang, D.E. Jones, and X.-L. Meng. Warp bridge sampling: The next generation. Journal of the American Statistical Association, 117(538):835-851, 2022.
[50] H. Xing. Improving bridge estimators via f-GAN. Statistics and Computing, 32, 2022.
[51] Y. Polyanskiy and Y. Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2022.
[52] S.-I. Amari and H. Nagaoka. Methods of information geometry. AMS, 2000.
In the following, we will study the estimation error of annealed Bregman estimation (ABE) in two important setups: the log-normalization is computed using two distributions ( $K = 1$ ), the proposal and the target, or else using a path of distributions ( $K \to \infty$ ).
The code used for the experiments is available at https://github.com/l-omar-chehab/annealing-normalizing-constants.
A No annealing,  $K = 1$
We use [22, Eq.21] for the estimation error of any suitably parameterized  $^3$  classifier  $F(\pmb {x};\pmb {\beta})$  between two distributions  $p_1$  and  $p_0$ . The estimation error is measured by the asymptotic Mean-Squared Error (MSE)
$$
\mathrm {M S E} _ {\hat {\boldsymbol {\beta}}} \left(p _ {n}, \nu , \phi , N\right) = \frac {\nu + 1}{N} \operatorname {t r} (\boldsymbol {\Sigma}) + o \left(\frac {1}{N}\right) \tag {20}
$$
which depends on the sample sizes  $N = N_{1} + N_{0}$ , their ratio  $\nu = N_{1} / N_{0}$ , the Bregman classification loss indexed by the convex function  $\phi (x)$ , and the asymptotic variance matrix
$$
\boldsymbol {\Sigma} = \boldsymbol {I} _ {w} ^ {- 1} \left(\boldsymbol {I} _ {v} - \left(1 + \frac {1}{\nu}\right) \boldsymbol {m} _ {w} \boldsymbol {m} _ {w} ^ {\top}\right) \boldsymbol {I} _ {w} ^ {- 1}. \tag {21}
$$
We suppose the standard technical conditions of van der Vaart [24, Th. 5.23] apply so that the remainder term  $\|\hat{\boldsymbol{\beta}} - \boldsymbol{\beta}^{*}\|^2$  can indeed be written independently of the parameterization, as  $o(N^{-1})$ . Here,  $\pmb{m}_w(\pmb{\beta}^*)$ ,  $\pmb{I}_w(\pmb{\beta}^*)$  and  $\pmb{I}_v(\pmb{\beta}^*)$  are the reweighted mean and covariances of the parameter-gradient of the classifier, also known as the "relative" Fisher score  $\nabla_{\beta}F(\pmb{x};\beta^{*})$ ,
$$
\boldsymbol {m} _ {w} \left(\boldsymbol {\beta} ^ {*}\right) = \mathbb {E} _ {\boldsymbol {x} \sim p _ {d}} \left[ w (\boldsymbol {x}) \nabla_ {\boldsymbol {\beta}} F (\boldsymbol {x}; \boldsymbol {\beta} ^ {*}) \right] \tag {22}
$$
$$
\boldsymbol {I} _ {w} \left(\boldsymbol {\beta} ^ {*}\right) = \mathbb {E} _ {\boldsymbol {x} \sim p _ {d}} \left[ w (\boldsymbol {x}) \nabla_ {\boldsymbol {\beta}} F \left(\boldsymbol {x}; \beta^ {*}\right) \nabla_ {\boldsymbol {\beta}} F \left(\boldsymbol {x}; \beta^ {*}\right) ^ {\top} \right] \tag {23}
$$
$$
\boldsymbol {I} _ {v} \left(\boldsymbol {\beta} ^ {*}\right) = \mathbb {E} _ {\boldsymbol {x} \sim p _ {d}} \left[ v (\boldsymbol {x}) \nabla_ {\boldsymbol {\beta}} F \left(\boldsymbol {x}; \boldsymbol {\beta} ^ {*}\right) \nabla_ {\boldsymbol {\beta}} F \left(\boldsymbol {x}; \boldsymbol {\beta} ^ {*}\right) ^ {\top} \right] \tag {24}
$$
where the reweighting of data points is by  $w(\pmb{x}) \coloneqq \frac{p_1}{\nu p_0}(\pmb{x}) \phi''\left(\frac{p_1}{\nu p_0}(\pmb{x})\right)$  and by  $v(\pmb{x}) = w(\pmb{x})^2 \frac{\nu p_0(\pmb{x}) + p_1(\pmb{x})}{\nu p_0(\pmb{x})}$ , which are all evaluated at the true parameter value  $\beta^*$ .
Scalar parameterization We now consider a specific parameterization of the classifier:
$$
F (\boldsymbol {x}; \beta) = \log \left(\frac {f _ {1} (\boldsymbol {x})}{\nu f _ {0} (\boldsymbol {x})}\right) - \beta \tag {25}
$$
where the optimal parameter is the log-ratio of normalizations  $\beta^{*} = \log (Z_{1} / Z_{0})$ . Consequently, we have  $\nabla_{\beta}F(\pmb {x};\beta^{*}) = -1$  and plugging this into the above quantities yields
$$
\mathrm {M S E} = \frac {1 + \nu}{T} \left(\frac {\mathbb {E} _ {\boldsymbol {x} \sim p _ {1}} \left[ w ^ {2} (\boldsymbol {x}) \frac {\nu p _ {0} (\boldsymbol {x}) + p _ {1} (\boldsymbol {x})}{\nu p _ {0} (\boldsymbol {x})} \right]}{\mathbb {E} _ {\boldsymbol {x} \sim p _ {1}} [ w (\boldsymbol {x}) ] ^ {2}} - \left(1 + \frac {1}{\nu}\right)\right) + o \left(\frac {1}{N}\right)
$$
which matches the formula found in [20, Eq 3.2]. For different choices of the Bregman classification loss, the estimation error is written using a divergence between the two distributions
Name	Loss identified by φ(x)	Estimator	MSE, up to o(N-1)
IS	x log x	log Ep0f1f0	1+ν/νN Dχ2(p1,p0)
RevIS	- log x	- log Ep1f0f1	1+ν/N Dχ2(p0,p1))
NCE	x log x - (1+x) log(1+x/2)	implicit	(1+ν)2/νN D_HM(p1,p0)/1-D_HM(p1,p0)
IS-RevIS	(1-√x)2	log Ep0f1f0 - log Ep1f0f1	(1+ν)2/νN 1-(1-DH2(pd,pn))2/(1-DH2(pd,pn))2
${}^{3}$  technically,the formula was derived in [48, 3] assuming the classifier was parameterized as  $F\left( {\mathbf{x};\mathbf{\beta }}\right)  =$ $\log {p}_{1}\left( {\mathbf{x};\mathbf{\beta }}\right) /{\nu }_{{p}_{0}}\left( \mathbf{x}\right)$  but the proof seems to generalize to any well-defined parameterization  $F\left( {\mathbf{x};\mathbf{\beta }}\right)$  .
$\mathcal{D}_{\chi^2}(p_1,p_0)\coloneqq \left(\int \frac{p_1^2}{p_0}\right) - 1$  is the chi-squared divergence
$D_{H^2}(p_1,p_0)\coloneqq 1 - \left(\int \sqrt{p_1p_0}\right)\in [0,1]$  is the squared Hellinger distance
$$
\mathcal {D} _ {\mathrm {H M}} (p _ {1}, p _ {0}) := 1 - \int (\pi p _ {1} ^ {- 1} + (1 - \pi) p _ {0} ^ {- 1}) ^ {- 1} = 1 - \frac {1}{\pi} \mathbb {E} _ {p _ {1}} \frac {\pi p _ {0}}{(1 - \pi) p _ {1} + \pi p _ {0}} \in [ 0, 1 ]
$$
is the harmonic divergence with weight  $\pi \in [0,1]$ .
Here, the weight  $\pi = P(Y = 0) = \frac{T_n}{T} = \frac{\nu}{1 + \nu}$ .
Proof of Theorem 2 Exponential error of binary NCE
In the following, we will drop the remainder term in  $o(N^{-1})$  given that no other limits are taken and that we will study the dominant term only. The estimation error of binary NCE is expressed in terms of the harmonic divergence
$$
\mathrm {M S E} = \frac {4}{N} \frac {\mathcal {D} _ {\mathrm {H M}} \left(p _ {1} , p _ {0}\right)}{1 - \mathcal {D} _ {\mathrm {H M}} \left(p _ {1} , p _ {0}\right)} \tag {26}
$$
which is intractable for general exponential families. Instead, we can lower-bound the estimation error. To do so, we lower-bound the harmonic divergence using the inequality of means (harmonic vs. geometric)
$$
\mathcal {D} _ {\mathrm {H M}} \left(p _ {1}, p _ {0}\right) = 1 - \int \frac {2 p _ {0} p _ {1}}{p _ {0} + p _ {1}} \geq 1 - \int \sqrt {p _ {0} p _ {1}} = \mathcal {D} _ {H ^ {2}} \left(p _ {0}, p _ {1}\right) \tag {27}
$$
$$
\mathrm {M S E} _ {\mathrm {L B}} = \frac {4}{N} \frac {\mathcal {D} _ {H ^ {2}} \left(p _ {1} , p _ {0}\right)}{1 - \mathcal {D} _ {H ^ {2}} \left(p _ {1} , p _ {0}\right)}. \tag {28}
$$
This lower bound is expressed in terms of the squared Hellinger distance, that is tractable for exponential families:
$$
\begin{array}{l} \mathcal {D} _ {H ^ {2}} \left(p _ {1}, p _ {0}\right) := 1 - \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \sqrt {p _ {1} p _ {0}} d \boldsymbol {x} (29) \\ = 1 - \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \frac {1}{Z \left(\boldsymbol {\theta} _ {1}\right) ^ {\frac {1}{2}} Z \left(\boldsymbol {\theta} _ {0}\right) ^ {\frac {1}{2}}} \exp \left(\frac {1}{2} \left(\boldsymbol {\theta} _ {1} + \boldsymbol {\theta} _ {0}\right) ^ {\top} \boldsymbol {t} (\boldsymbol {x})\right) d \boldsymbol {x} (30) \\ = 1 - \frac {Z \left(\frac {1}{2} \boldsymbol {\theta} _ {1} + \frac {1}{2} \boldsymbol {\theta} _ {0}\right)}{Z \left(\boldsymbol {\theta} _ {1}\right) ^ {\frac {1}{2}} Z \left(\boldsymbol {\theta} _ {0}\right) ^ {\frac {1}{2}}} (31) \\ = 1 - \exp \left(\log Z \left(\frac {1}{2} \boldsymbol {\theta} _ {1} + \frac {1}{2} \boldsymbol {\theta} _ {0}\right) - \frac {1}{2} \log Z (\boldsymbol {\theta} _ {1}) - \frac {1}{2} \log Z (\boldsymbol {\theta} _ {0})\right). (32) \\ \end{array}
$$
We now wish to lower bound  $\mathrm{MSE}_{\mathrm{LB}}$ , and therefore  $\mathcal{D}_{H^2}(p_1,p_0)$ , by an expression which is exponential in the parameter distance  $\| \pmb{\theta}_1 - \pmb{\theta}_0\|$ . To do so, we note that for exponential families, the log-normalization is convex in the parameters. Here, we further assume strong convexity, so that
$$
\log Z \left(\frac {1}{2} \boldsymbol {\theta} _ {1} + \frac {1}{2} \boldsymbol {\theta} _ {0}\right) \leq \frac {1}{2} \log Z (\boldsymbol {\theta} _ {1}) + \frac {1}{2} \log Z (\boldsymbol {\theta} _ {0}) - \frac {1}{8} M \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \| ^ {2} \tag {33}
$$
where  $M$  is the strong convexity constant. Plugging this back into the squared Hellinger distance, we obtain
$$
\mathcal {D} _ {H ^ {2}} \left(p _ {1}, p _ {0}\right) \geq 1 - \exp \left(- \frac {1}{8} M \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \| ^ {2}\right) \tag {34}
$$
so that the MSE
$$
\mathrm {M S E} \geq \frac {4}{N} \frac {\mathcal {D} _ {H ^ {2}} \left(p _ {1} , p _ {0}\right)}{1 - \mathcal {D} _ {H ^ {2}} \left(p _ {1} , p _ {0}\right)} \geq \frac {4}{N} \exp \left(\frac {1}{8} M \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \| ^ {2}\right) - 1 \tag {35}
$$
grows exponentially with the euclidean distance between the parameters.
B Annealing limit,  $K\to \infty$
We now consider annealing paths  $(p_t)_{t\in [0,1]}$  that interpolate between between the proposal  $p_0$  and the target  $p_1$ .
B.1 Estimation error
We first show the optimality of the NCE loss within the family of Annealed Bregman Estimators, in the sense that it produces the smallest estimation error. We then study the estimation error of different annealed Bregman estimators in the annealing limit of a continuous path  $(K\to \infty)$ .
Proof of Theorem 1 Optimality of the NCE loss and the estimation error in the annealing limit  $K\to \infty$
- Optimality of the NCE loss
Because the annealed Bregman estimator is built by adding independent estimators
$$
\widehat {\log Z _ {1}} = \sum_ {k = 0} ^ {K - 1} \log \widehat {\left(\frac {Z _ {(k + 1) / K}}{Z _ {k / K}}\right)} + \log Z _ {0}. \tag {36}
$$
the total Mean Squared Error (MSE) is the sum of each MSEs for each estimator (indexed by  $k \in [[0, K - 1]]$ )
$$
\operatorname {M S E} \left(\left(\phi_ {k}\right) _ {k \in \llbracket 0, K \rrbracket}\right) = \sum_ {k = 0} ^ {K - 1} \operatorname {M S E} _ {k} \left(\phi_ {k}\right) \tag {37}
$$
where we highlighted the dependency on the classification losses identified by  $(\phi_k)_{k\in [0,K]}$ . The MSEs follow Eq. 26. It was shown by Meng and Wong [20] that for any of these MSEs, the optimal loss is identified by  $\phi_k(x) = x\log x - (1 + x)\log (\frac{1 + x}{2})$  and is in fact the NCE loss [22]. Thus the sum of MSEs is minimized for the same loss.
- Annealed Noise-Contrastive Estimation (NCE)
We are interested in the estimation error (asymptotic MSE) obtained for the NCE loss. Based off table 1, it is written as
$$
\begin{array}{l} \mathrm {M S E} = \sum_ {k = 0} ^ {K - 1} \left(\frac {4 K}{N} \frac {\mathcal {D} _ {\mathrm {H M}} \left(p _ {k / K} , p _ {(k + 1) / K}\right)}{1 - \mathcal {D} _ {\mathrm {H M}} \left(p _ {k / K} , p _ {(k + 1) / K}\right)} + o \left(\frac {K}{N}\right)\right) (38) \\ = \frac {4 K}{N} \sum_ {k = 0} ^ {K - 1} \frac {\mathcal {D} _ {\mathrm {H M}} \left(p _ {k / K} , p _ {(k + 1) / K}\right)}{1 - \mathcal {D} _ {\mathrm {H M}} \left(p _ {k / K} , p _ {(k + 1) / K}\right)} + o \left(\frac {K ^ {2}}{N}\right). (39) \\ \end{array}
$$
where a sample budget of  $N / K$  is used for each estimator that is summed. The estimation error of balanced  $(\nu = 1)$  NCE-JS between two "close" distributions  $p_t$  and  $p_{t + h}$ , is
$$
\operatorname {M S E} \left(p _ {t}, p _ {t + h}\right) \propto \frac {\mathcal {D} _ {\mathrm {H M}} \left(p _ {t} , p _ {t + h}\right)}{1 - \mathcal {D} _ {\mathrm {H M}} \left(p _ {t} , p _ {t + h}\right)} \tag {40}
$$
up to the remainder term. The estimation error can be simplified using a Taylor expansion. To do so, we recall that  $D_{\mathrm{HM}}$  is an f-divergence generated by  $\phi(x) = 1 - \frac{x}{\pi + (1 - \pi)x}$  [49, 50]  $(\pi = \frac{1}{2}$  here) and its expansion is therefore [51, Eq.7.64]
$$
\begin{array}{l} \mathcal {D} _ {\mathrm {H M}} \left(p _ {t}, p _ {t + h}\right) = \frac {1}{2} h ^ {2} \nabla_ {t} ^ {2} \mathcal {D} _ {\mathrm {H M}} \left(p _ {t}, p _ {t + h}\right) + o \left(h ^ {2}\right) (41) \\ = \frac {1}{2} \phi^ {\prime \prime} (1) h ^ {2} I (t) + o \left(h ^ {2}\right) = \frac {1}{4} h ^ {2} I (t) + o \left(h ^ {2}\right). (42) \\ \end{array}
$$
$$
\frac {\mathcal {D} _ {\mathrm {H M}} \left(p _ {t} , p _ {t + h}\right)}{1 - \mathcal {D} _ {\mathrm {H M}} \left(p _ {t} , p _ {t + h}\right)} = \frac {1}{4} I (t) h ^ {2} + o \left(h ^ {2}\right). \tag {43}
$$
Summing these estimation errors along the path of distributions with  $h = 1 / K$
$$
\begin{array}{l} \mathrm {M S E} = \frac {4 K}{N} \sum_ {k = 0} ^ {K - 1} \left(\frac {1}{4} I (t) \frac {1}{K ^ {2}} + o \left(\frac {1}{K ^ {2}}\right)\right) + o \left(\frac {K ^ {2}}{N}\right) (44) \\ = \left(\frac {1}{N K} \sum_ {k = 0} ^ {K - 1} I (t)\right) + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) (45) \\ = \frac {1}{N} \int_ {0} ^ {1} I (t) d t + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right). (46) \\ \end{array}
$$
In the case of a parametric path  $p(\pmb{x}|\pmb{\theta}(t))_{t\in [0,1]}$ , the proof is the same. Simply, the second-order term in the Taylor expansion of Eq. 42 is computed using the chain rule
$$
\begin{array}{l} \nabla_ {t} ^ {2} \operatorname {M S E} \left(p _ {\boldsymbol {\theta} (t)}, p _ {\boldsymbol {\theta} (t + h)}\right) (47) \\ = \dot {\boldsymbol {\theta}} (t) ^ {\top} \nabla_ {\boldsymbol {\theta}} ^ {2} \mathrm {M S E} \left(p _ {\boldsymbol {\theta} (t)}, p _ {\boldsymbol {\theta} (t + h)}\right) \dot {\boldsymbol {\theta}} (t) + \ddot {\boldsymbol {\theta}} (t) ^ {\top} \nabla_ {\boldsymbol {\theta}} \mathrm {M S E} \left(p _ {\boldsymbol {\theta} (t)}, p _ {\boldsymbol {\theta} (t + h)}\right) (48) \\ = \dot {\boldsymbol {\theta}} (t) ^ {\top} \nabla_ {\boldsymbol {\theta}} ^ {2} \operatorname {M S E} \left(p _ {\boldsymbol {\theta} (t)}, p _ {\boldsymbol {\theta} (t + h)}\right) \dot {\boldsymbol {\theta}} (t) ^ {\top} + 0 (49) \\ = \dot {\boldsymbol {\theta}} (t) ^ {\top} \boldsymbol {I} (\theta (t)) \dot {\boldsymbol {\theta}} (t) (50) \\ \end{array}
$$
- Annealed importance sampling (IS)
Similarly, for the choice of the importance sampling base estimator,
$$
\begin{array}{l} \mathrm {M S E} = \sum_ {k = 0} ^ {K - 1} \left(\frac {2 K}{N} \mathcal {D} _ {\chi^ {2}} \left(p _ {(k + 1) / K}, p _ {k / K}\right) + o \left(\frac {K}{N}\right)\right) (51) \\ = \frac {2 K}{N} \sum_ {k = 0} ^ {K - 1} \mathcal {D} _ {\chi^ {2}} \left(p _ {(k + 1) / K}, p _ {k / K}\right) + o \left(\frac {K ^ {2}}{N}\right) (52) \\ = \frac {2 K}{N} \sum_ {k = 0} ^ {K - 1} \mathcal {D} _ {\operatorname {r e v} \chi^ {2}} \left(p _ {k / K}, p _ {(k + 1) / K}\right) + o \left(\frac {K ^ {2}}{N}\right) (53) \\ = \frac {2 K}{N} \sum_ {k = 0} ^ {K - 1} \left(\frac {1}{2} \phi^ {\prime \prime} (1) I (t) \frac {1}{K ^ {2}} + o \left(\frac {1}{K ^ {2}}\right)\right) + o \left(\frac {K ^ {2}}{N}\right) (54) \\ = \frac {1}{N K} \sum_ {k = 0} ^ {K - 1} \phi^ {\prime \prime} (1) I (t) + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) (55) \\ = \frac {1}{N} \int_ {0} ^ {1} I (t) d t + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right). (56) \\ \end{array}
$$
given that  $\phi (x) = -\log (x)$  and therefore  $\phi ''(1) = 1$  for the reverse  $\chi^2$  divergence.
- Annealed reverse importance sampling (RevIS)
Similarly, for the choice of the reverse importance sampling base estimator,
$$
\begin{array}{l} \mathrm {M S E} = \sum_ {k = 0} ^ {K - 1} \left(\frac {2 K}{N} \mathcal {D} _ {\chi^ {2}} \left(p _ {k / K}, p _ {(k + 1) / K}\right) + o \left(\frac {K}{N}\right)\right) (57) \\ = \frac {2 K}{N} \sum_ {k = 0} ^ {K - 1} \left(\frac {1}{2} \phi^ {\prime \prime} (1) I (t) \frac {1}{K ^ {2}} + o \left(\frac {1}{K ^ {2}}\right)\right) + o \left(\frac {K ^ {2}}{N}\right) (58) \\ = \frac {1}{N K} \sum_ {k = 0} ^ {K - 1} I (t) + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) = \frac {1}{N} \int_ {0} ^ {1} I (t) d t + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right). (59) \\ \end{array}
$$
given that  $\phi (x) = x\log (x)$  and therefore  $\phi ''(1) = 1$  for the  $\chi^2$  divergence.
B.2 Examples of paths
Geometric path The geometric path is defined in the space of unnormalized densities by
$$
f _ {t} (\boldsymbol {x}) := p _ {0} (\boldsymbol {x}) ^ {1 - t} f _ {1} (\boldsymbol {x}) ^ {t} = p _ {0} (\boldsymbol {x}) ^ {1 - t} p _ {1} (\boldsymbol {x}) ^ {t} Z _ {1} ^ {t} \propto p _ {0} (\boldsymbol {x}) ^ {1 - t} p _ {1} (\boldsymbol {x}) ^ {t} \tag {60}
$$
so in the space of normalized densities, the path is
$$
p _ {t} := \frac {p _ {0} (\boldsymbol {x}) ^ {1 - t} p _ {1} (\boldsymbol {x}) ^ {t}}{Z _ {t}} \tag {61}
$$
where the normalization is
$$
Z _ {t} := \int_ {\boldsymbol {x} \in \mathbb {R} ^ {d}} p _ {0} (\boldsymbol {x}) ^ {1 - t} p _ {1} (\boldsymbol {x}) ^ {t} d \boldsymbol {x} = \mathbb {E} _ {\boldsymbol {x} \sim p _ {1}} \left[ \left(\frac {p _ {0} (\boldsymbol {x})}{p _ {1} (\boldsymbol {x})}\right) ^ {t} \right] = \mathbb {E} _ {\boldsymbol {x} \sim p _ {0}} \left[ \left(\frac {p _ {1} (\boldsymbol {x})}{p _ {0} (\boldsymbol {x})}\right) ^ {1 - t} \right]. \tag {62}
$$
Arithmetic path The arithmetic path is defined in the space of unnormalized densities by
$$
f _ {t} (\boldsymbol {x}) := (1 - t) p _ {0} (\boldsymbol {x}) + t f _ {1} (\boldsymbol {x}) = (1 - t) p _ {0} + t Z _ {1} p _ {1} \tag {63}
$$
$$
\propto \frac {(1 - t)}{(1 - t) + t Z _ {1}} p _ {0} + \frac {t Z _ {1}}{(1 - t) + t Z _ {1}} p _ {1} \tag {64}
$$
so in the space of normalized densities, the path is actually a mixture between the target and the proposal, where the weight of the mixture is a nonlinear function of the target normalization
$$
p _ {t} := \left(1 - \tilde {w} _ {t}\right) p _ {0} + \tilde {w} _ {t} p _ {1}, \quad \tilde {w} _ {t} = \frac {t Z _ {1}}{(1 - t) + t Z _ {1}}. \tag {65}
$$
Optimal path We know (e.g. from Gelman and Meng [17, Eq. 49] that the optimal path is
$$
p _ {t} (\boldsymbol {x}) = \left(a (t) \sqrt {p _ {0} (\boldsymbol {x})} + b (t) \sqrt {p _ {1} (\boldsymbol {x})}\right) ^ {2} \tag {66}
$$
where the coefficients  $a(t)$  and  $b(t)$
$$
a (t) = \frac {\cos ((2 t - 1) \alpha_ {H})}{2 \cos (\alpha_ {H})} - \frac {\sin ((2 t - 1) \alpha_ {H})}{2 \sin (\alpha_ {H})} \tag {67}
$$
$$
b (t) = \frac {\cos ((2 t - 1) \alpha_ {H})}{2 \cos (\alpha_ {H})} + \frac {\sin ((2 t - 1) \alpha_ {H})}{2 \sin (\alpha_ {H})} \tag {68}
$$
are simple functions of the squared Hellinger distance  $\mathcal{D}_{H^2}(p_0,p_1)$  between the proposal and the target
$$
\alpha_ {H} = \arctan \left(\sqrt {\frac {\mathcal {D} _ {H ^ {2}} \left(p _ {0} , p _ {1}\right)}{2 - \mathcal {D} _ {H ^ {2}} \left(p _ {0} , p _ {1}\right)}}\right) \in [ 0, \frac {\pi}{4} ]. \tag {69}
$$
The estimation error produced by that optimal path is [17, Eq. 48]
$$
\mathrm {M S E} = \frac {1}{N} \int_ {0} ^ {1} I (t) d t = \frac {1}{N} 1 6 \alpha_ {H} ^ {2}. \tag {70}
$$
For two Gaussians
$$
p _ {0} := \mathcal {N} \left(\boldsymbol {\mu} _ {0}, \boldsymbol {\Sigma} _ {0}\right) \tag {71}
$$
$$
p _ {1} := \mathcal {N} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right) \tag {72}
$$
the squared Hellinger distance can be written in closed-form
$$
\mathcal {D} _ {H ^ {2}} \left(p _ {0}, p _ {1}\right) = 1 - \frac {\left| \boldsymbol {\Sigma} _ {0} \right| ^ {\frac {1}{4}} \left| \boldsymbol {\Sigma} _ {1} \right| ^ {\frac {1}{4}}}{\left| \frac {1}{2} \boldsymbol {\Sigma} _ {0} + \frac {1}{2} \boldsymbol {\Sigma} _ {1} \right| ^ {\frac {1}{2}}} \exp \left(- \frac {1}{8} \left(\boldsymbol {\mu} _ {1} - \boldsymbol {\mu} _ {0}\right) ^ {\top} \left(\frac {1}{2} \boldsymbol {\Sigma} _ {0} + \frac {1}{2} \boldsymbol {\Sigma} _ {1}\right) ^ {- 1} \left(\boldsymbol {\mu} _ {1} - \boldsymbol {\mu} _ {0}\right)\right) \tag {73}
$$
and plugs into the optimal path formula, which is also obtained in closed-form.
4In Gelman and Meng [17, Eq. 49], the Hellinger distance is defined such that it is in  $[0,\sqrt{2} ]$ . We here instead use the conventional definition of the squared Hellinger distance which is normalized so that it is in [0, 1].
B.3 Estimation error from taking different paths
Proof of Theorem 3 Polynomial error of annealed NCE with the geometric path
We next study the estimation error produced by the geometric path (Figure 1). In the annealing limit  $K \to \infty$ , the MSE is written as
$$
\mathrm {M S E} = \frac {1}{N} \int_ {0} ^ {1} I (t) d t + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right). \tag {74}
$$
We recall from Grosse et al. [18] that the geometric path is closed for distributions in the exponential family: all distributions along the path remain in the exponential family. Furthermore, their Fisher information can be written in terms of the terms parameters [18, Eq. 17]; this is based off a result of exponential families from [52, Section 3.3]
$$
I (t) = \dot {\boldsymbol {\theta}} (t) ^ {\top} \boldsymbol {I} (\boldsymbol {\theta} (t)) \dot {\boldsymbol {\theta}} (t) = \dot {\boldsymbol {\theta}} (t) ^ {\top} \dot {\boldsymbol {\mu}} (t) \tag {75}
$$
where  $\pmb{\mu}(t)$  are the generalized moments, defined as  $\pmb{\mu}(t) = \mathbb{E}_{\pmb{x} \sim p_t(\pmb{x})}[\pmb{t}(\pmb{x})] = \nabla_\pmb{\theta} \log Z_t(\pmb{\theta})$ . It follows,
$$
\frac {1}{N} \int_ {0} ^ {1} I (t) d t = \frac {1}{N} \int_ {0} ^ {1} \dot {\boldsymbol {\theta}} (t) ^ {\top} \dot {\boldsymbol {\mu}} (t) d t. \tag {76}
$$