SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The geometric path is defined in parameter space by  $\pmb{\theta}_t = t\pmb{\theta}_1 + (1 - t)\pmb{\theta}_0$ , therefore
$$
\frac {1}{N} \int_ {0} ^ {1} I (t) d t = \frac {1}{N} \left(\boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0}\right) ^ {\top} \int_ {0} ^ {1} \dot {\boldsymbol {\mu}} (t) d t = \frac {1}{N} \left(\boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0}\right) ^ {\top} \left(\boldsymbol {\mu} _ {1} - \boldsymbol {\mu} _ {0}\right) \tag {77}
$$
as in [18, Eq. 17]. For exponential families,  $\log Z(\pmb{\theta})$  is convex in  $\pmb{\theta}$ . Here, we further assume strong convexity (with constant  $M$ ) and smoothness (with constant  $L$ ) so that
$$
\begin{array}{l} \left(\boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0}\right) ^ {\top} \left(\boldsymbol {\mu} _ {1} - \boldsymbol {\mu} _ {0}\right) = \left(\boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0}\right) ^ {\top} \left(\nabla_ {\boldsymbol {\theta}} \log Z _ {t} \left(\boldsymbol {\theta} _ {1}\right) - \nabla_ {\boldsymbol {\theta}} \log Z _ {t} \left(\boldsymbol {\theta} _ {0}\right)\right) (78) \\ \leq \frac {1}{M} \| \nabla_ {\boldsymbol {\theta}} \log Z _ {t} \left(\boldsymbol {\theta} _ {1}\right) - \nabla_ {\boldsymbol {\theta}} \log Z _ {t} \left(\boldsymbol {\theta} _ {0}\right) \| ^ {2} \leq \frac {L ^ {2}}{M} \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \| ^ {2} (79) \\ \end{array}
$$
$$
\mathrm {M S E} \leq \frac {L ^ {2}}{M N} \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \| ^ {2} + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) \tag {80}
$$
is polynomial in the euclidean distance between the parameters.
Proof of Theorem 4 Exponential error of annealed NCE with the arithmetic path and "vanilla" schedule
We now study the estimation error produced by the arithmetic path with "vanilla" schedule (table 2, line 3). Similarly, we start with the formula of the estimation error of NCE in the limit of a continuous path
$$
\mathrm {M S E} = \frac {1}{N} \int_ {0} ^ {1} I (t) d t + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) \tag {81}
$$
where  $I(t) = \mathbb{E}_{\boldsymbol{x} \sim p(\boldsymbol{x}, t)}[(\frac{d}{dt} \log p(\boldsymbol{x}, t))^2]$  is the Fisher information over the path, using time  $t$  as the parameter. The arithmetic path is a Gaussian mixture (see table 2) so we will conveniently use the parametric form of the path to compute the Fisher information
$$
I (t) = \dot {\tilde {w}} _ {t} ^ {\top} I (\tilde {w} _ {t}) \dot {\tilde {w}} _ {t} \tag {82}
$$
where the parameter here is the weight of the Gaussian mixture  $\tilde{w}_t = tZ_1 / (tZ_1 + 1 - t)$ . We will need to compute two quantities: the Fisher information to that mixture parameter (not the time), and the parameter speed  $\tilde{w}_b$ .
$$
\begin{array}{l} I \left(\tilde {w} _ {t}\right) := \mathbb {E} _ {\boldsymbol {x} \sim p _ {\tilde {w} _ {t}}} \left[ \left(\frac {\partial \log p _ {\tilde {w} _ {t}}}{\partial \tilde {w} _ {t}} (\boldsymbol {x})\right) ^ {2} \right] = \mathbb {E} _ {\boldsymbol {x} \sim p _ {\tilde {w} _ {t}}} \left[ \left(\frac {1}{p _ {\tilde {w} _ {t}} (\boldsymbol {x})} \frac {\partial p _ {\tilde {w} _ {t}}}{\partial \tilde {w} _ {t}} (\boldsymbol {x})\right) ^ {2} \right] (83) \\ = \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \frac {\left(p _ {1} (\boldsymbol {x}) - p _ {0} (\boldsymbol {x})\right) ^ {2}}{p _ {\tilde {w} _ {t}} (\boldsymbol {x})} d \boldsymbol {x} = \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \frac {\left(p _ {1} (\boldsymbol {x}) - p _ {0} (\boldsymbol {x})\right) ^ {2}}{\left(1 - \tilde {w} _ {t}\right) p _ {0} (\boldsymbol {x}) + \tilde {w} _ {t} p _ {1} (\boldsymbol {x})} d \boldsymbol {x} (84) \\ \geq \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \frac {\left(p _ {1} (\boldsymbol {x}) - p _ {0} (\boldsymbol {x})\right) ^ {2}}{p _ {0} (\boldsymbol {x}) + p _ {1} (\boldsymbol {x})} d \boldsymbol {x} = \int p _ {0} (\boldsymbol {x}) \frac {\left(1 - \frac {p _ {1} (\boldsymbol {x})}{p _ {0} (\boldsymbol {x})}\right) ^ {2}}{1 + \frac {p _ {1} (\boldsymbol {x})}{p _ {0} (\boldsymbol {x})}} = \mathcal {D} _ {\phi} \left(p _ {1}, p _ {0}\right) (85) \\ \end{array}
$$
which is an f-divergence with generator  $\phi(x) = (1 - x)^2 / (1 + x)$  that provides a  $t$ -independent lower bound. This will allow us to factor this quantity out of the integral defining the MSE, and simplify computations. We also have
$$
\begin{array}{l} \ddot {u} _ {t} := \frac {\partial}{\partial t} \tilde {w} _ {t} = \frac {1}{t (1 - t)} \times \sigma \left(\log \frac {t Z _ {1}}{1 - t}\right) \times \left(1 - \sigma \left(\log \frac {t Z _ {1}}{1 - t}\right)\right) (86) \\ = \frac {1}{t (1 - t)} \times \frac {t Z _ {1}}{(1 - t) + t Z _ {1}} \times \frac {(1 - t)}{(1 - t) + t Z _ {1}} = \frac {Z _ {1}}{((1 - t) + t Z _ {1}) ^ {2}}. (87) \\ \end{array}
$$
where we choose to keep the dependency on  $t$ . The intuition is that integrating this quantity will yield a function of  $Z_{1}$ , which will drive the MSE toward high values. We next show this rigorously and finally compute the estimation error.
$$
\begin{array}{l} \frac {1}{N} \int_ {0} ^ {1} I (t) d t = \frac {1}{N} \int_ {0} ^ {1} \dot {\tilde {w}} (t) I (\tilde {w} (t)) \dot {\tilde {w}} (t) d t (88) \\ \geq \frac {1}{N} \times \mathcal {D} _ {\phi} \left(p _ {1}, p _ {0}\right) \times \int_ {0} ^ {1} \dot {\tilde {w}} (t) ^ {2} d t (89) \\ = \frac {1}{N} \times \mathcal {D} _ {\phi} \left(p _ {1}, p _ {0}\right) \times Z _ {1} ^ {2} \times \int_ {0} ^ {1} \frac {1}{\left(t \left(Z _ {1} - 1\right) + 1\right) ^ {4}} d t (90) \\ = \frac {1}{N} \times \mathcal {D} _ {\phi} \left(p _ {1}, p _ {0}\right) \times Z _ {1} ^ {2} \times \frac {Z _ {1} ^ {2} + Z _ {1} + 1}{3 Z _ {1} ^ {3}} (91) \\ = \frac {1}{3 N} \times \mathcal {D} _ {\phi} \left(p _ {1}, p _ {0}\right) \times \left(Z _ {1} ^ {- 1} + 1 + Z _ {1}\right). (92) \\ \end{array}
$$
We would like to write  $Z_{1}$  in terms of the parameters. To do so, we now suppose the unnormalized target is in a simply unnormalized exponential family. Consequently,
$$
\begin{array}{l} Z _ {1} := \exp (\log Z (\boldsymbol {\theta} _ {1}) - \log Z (\boldsymbol {\theta} _ {0}) + \log Z (\boldsymbol {\theta} _ {0})) (93) \\ \geq \exp \left(\nabla \log Z (\boldsymbol {\theta} _ {0}) (\boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0}) + \frac {M}{2} \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \| ^ {2} + \log Z (\boldsymbol {\theta} _ {0})\right). (94) \\ \end{array}
$$
using the strong convexity of the log-partition function. For large enough  $\| \pmb{\theta}_1 - \pmb{\theta}_0\| > 0$ , the quadratic term in the exponential is larger than the linear term, and the divergence  $\mathcal{D}_{\phi}(p_1,p_0)$  is larger than a constant. It follows that for large enough  $\| \pmb{\theta}_1 - \pmb{\theta}_0\| > 0$ , there exists a constant  $C > 0$  such that the MSE grows (at least) exponentially with the parameter-distance
$$
\operatorname {M S E} > \frac {C}{3 N} \times \exp \left(\frac {M}{2} \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \| ^ {2}\right) + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right). \tag {95}
$$
Proof of Theorem 5 Polynomial error of annealed NCE with the arithmetic path and "oracle" schedule
We now study the estimation error produced by the arithmetic path with "oracle" schedule (table 2, line 4). Similarly, we start with the formula of the estimation error of NCE annealed over a continuous path
$$
\mathrm {M S E} = \frac {1}{N} \int_ {0} ^ {1} I (t) d t + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) \tag {96}
$$
where  $I(t) = \mathbb{E}_{\pmb{x}\sim p(\pmb{x},t)}[(\frac{d}{dt}\log p(\pmb{x},t))^2 ]$  is the Fisher information over the path, using time  $t$  as the parameter. The arithmetic path is the Gaussian mixture  $p_t(\pmb {x}) = tp_1(\pmb {x}) + (1 - t)p_0(\pmb {x})$  (see table 2). The Fisher information is therefore
$$
\begin{array}{l} I (t) := \mathbb {E} _ {\boldsymbol {x} \sim p _ {t}} \left[ \left(\frac {\partial \log p _ {t}}{\partial t} (\boldsymbol {x})\right) ^ {2} \right] = \mathbb {E} _ {\boldsymbol {x} \sim p _ {t}} \left[ \left(\frac {1}{p _ {t} (\boldsymbol {x})} \frac {\partial p _ {t}}{\partial t} (\boldsymbol {x})\right) ^ {2} \right] (97) \\ = \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \frac {\left(p _ {1} (\boldsymbol {x}) - p _ {0} (\boldsymbol {x})\right) ^ {2}}{p _ {t} (\boldsymbol {x})} d \boldsymbol {x} = \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \frac {\left(p _ {1} (\boldsymbol {x}) - p _ {0} (\boldsymbol {x})\right) ^ {2}}{(1 - t) p _ {0} (\boldsymbol {x}) + t p _ {1} (\boldsymbol {x})} d \boldsymbol {x} (98) \\ \leq \int_ {\boldsymbol {x} \in \mathbb {R} ^ {D}} \frac {p _ {1} (\boldsymbol {x}) ^ {2} + p _ {0} (\boldsymbol {x}) ^ {2}}{(1 - t) p _ {0} (\boldsymbol {x}) + t p _ {1} (\boldsymbol {x})} d \boldsymbol {x} (99) \\ \end{array}
$$
where we choose to keep the dependency on  $t$  in the bound.
We briefly justify this choice. We had first tried a  $t$ -independent bound, which led to an upper bound of the MSE that was too loose. We share insight as to why: first, recognize that the fraction can be broken in two terms, each of them a chi-square divergence between an endpoint of the path  $(p_0$  or  $p_1)$  and the mixture  $p_t$ . Each of them admits a  $t$ -independent upper bound given by the chi-square divergence between the endpoints  $p_0$  and  $p_1$ , using lemma 1. However, the chi-square divergence between two Gaussians, for example, is exponential (not polynomial) in the natural parameters [51, eq 7.41]. In fact, plotting  $I(t)$  for a univariate Gaussian model revealed that it took high values at the endpoints  $t = 0$  and  $t = 1$ , and was near zero almost everywhere else in the interval  $t \in [0, 1]$ , which again suggested that dropping the dependency on  $t$  was unreasonable.
Now we can compute the estimation error, as
$$
\frac {1}{N} \int_ {0} ^ {1} I (t) d t \leq \frac {1}{N} \int_ {\mathbb {R} ^ {d}} \int_ {0} ^ {1} \frac {p _ {1} (\boldsymbol {x}) ^ {2} + p _ {0} (\boldsymbol {x}) ^ {2}}{(1 - t) p _ {0} (\boldsymbol {x}) + t p _ {1} (\boldsymbol {x})} d t d \boldsymbol {x} = \frac {1}{N} \left(J _ {1} + J _ {2}\right). \tag {100}
$$
Let us try to solve one of these integrals, say  $J_{1}$ .
$$
\begin{array}{l} J _ {1} = \int_ {\mathbb {R} ^ {d}} \int_ {0} ^ {1} \frac {p _ {1} (\boldsymbol {x}) ^ {2}}{(1 - t) p _ {0} (\boldsymbol {x}) + t p _ {1} (\boldsymbol {x})} d t d \boldsymbol {x} = \int_ {\mathbb {R} ^ {d}} \frac {p _ {1} (\boldsymbol {x}) ^ {2}}{p _ {0} (\boldsymbol {x})} \left(\int_ {0} ^ {1} \frac {1}{1 + t \left(\frac {p _ {1} (\boldsymbol {x})}{p _ {0} (\boldsymbol {x})} - 1\right)} d t\right) d \boldsymbol {x} (101) \\ = \int_ {\mathbb {R} ^ {d}} \frac {p _ {1} (\boldsymbol {x}) ^ {2}}{p _ {0} (\boldsymbol {x})} \left(\frac {1}{\frac {p _ {1}}{p _ {0}} - 1} \log \frac {p _ {1}}{p _ {0}}\right) d \boldsymbol {x} = 1 + \mathbb {E} _ {p _ {1}} \left[ \frac {1}{1 - \frac {p _ {0}}{p _ {1}}} \log \frac {p _ {1}}{p _ {0}} - 1 \right] = 1 + \mathcal {D} _ {\phi} \left(p _ {0}, p _ {1}\right). (102) \\ \end{array}
$$
which we rewrite using an f-divergence defined by  $\phi(x) = \frac{-\log(x)}{1 - x} - 1$ . Similarly, we obtain
$$
\begin{array}{l} J _ {2} = \int_ {\mathbb {R} ^ {d}} \int_ {0} ^ {1} \frac {p _ {0} (\boldsymbol {x}) ^ {2}}{(1 - t) p _ {0} (\boldsymbol {x}) + t p _ {1} (\boldsymbol {x})} d t d \boldsymbol {x} = \int_ {\mathbb {R} ^ {d}} \frac {p _ {0} (\boldsymbol {x}) ^ {2}}{p _ {1} (\boldsymbol {x})} \left(\int_ {0} ^ {1} \frac {1}{\frac {p _ {0} (\boldsymbol {x})}{p _ {1} (\boldsymbol {x})} + t \left(1 - \frac {p _ {0} (\boldsymbol {x})}{p _ {1} (\boldsymbol {x})}\right)} d t\right) d \boldsymbol {x} (103) \\ = \int_ {\mathbb {R} ^ {d}} \frac {p _ {0} (\boldsymbol {x}) ^ {2}}{p _ {1} (\boldsymbol {x})} \left(\frac {1}{\frac {p _ {0}}{p _ {1}} - 1} \log \frac {p _ {0}}{p _ {1}}\right) d \boldsymbol {x} = 1 + \mathbb {E} _ {p _ {0}} \left[ \frac {1}{1 - \frac {p _ {1}}{p _ {0}}} \log \frac {p _ {0}}{p _ {1}} - 1 \right] = 1 + \mathcal {D} _ {\phi} \left(p _ {1}, p _ {0}\right). (104) \\ \end{array}
$$
Putting this together, we get
$$
\frac {1}{N} \int_ {0} ^ {1} I (t) d t \leq \frac {1}{N} \left(2 + \mathcal {D} _ {\phi} \left(p _ {0}, p _ {1}\right) + \mathcal {D} _ {\phi} \left(p _ {1}, p _ {0}\right)\right). \tag {105}
$$
How does this divergence depend on the parameter-distance  $\| \theta_1 - \theta_0\|$ ? Does it bring down the dependency from exponential to something lower? We next analyze this:
$$
\mathcal {D} _ {\phi} (p _ {0}, p _ {1}) + 1 = \mathbb {E} _ {p _ {1}} \frac {1}{1 - \frac {p _ {0}}{p _ {1}}} \log \frac {p _ {1}}{p _ {0}}
$$
which looks like a Kullback-Leibler divergence, where the integrand is reweighted by  $\frac{1}{1 - \frac{p_0(\mathbf{x})}{p_1(\mathbf{x})}}$ . Note that
$$
\left\{ \begin{array}{l l} \frac {1}{1 - \frac {p _ {0} (\boldsymbol {x})}{p _ {1} (\boldsymbol {x})}} \geq 1 & p _ {0} (\boldsymbol {x}) \leq p _ {1} (\boldsymbol {x}) \\ \frac {1}{1 - \frac {p _ {0} (\boldsymbol {x})}{p _ {1} (\boldsymbol {x})}} <   1 & p _ {0} (\boldsymbol {x}) > p _ {1} (\boldsymbol {x}) \end{array} \right. \tag {106}
$$
which motivates separating the integral over both domains
$$
\begin{array}{l} 1 + \mathcal {D} _ {\phi} \left(p _ {0}, p _ {1}\right) = \int_ {\left\{\boldsymbol {x} \in \mathbb {R} ^ {D} \mid p _ {0} (\boldsymbol {x}) \leq p _ {1} (\boldsymbol {x}) \right\}} p _ {1} (\boldsymbol {x}) \log \frac {p _ {1} (\boldsymbol {x})}{p _ {0} (\boldsymbol {x})} \frac {1}{1 - \frac {p _ {0} (\boldsymbol {x})}{p _ {1} (\boldsymbol {x})}} (107) \\ + \int_ {\left\{\boldsymbol {x} \in \mathbb {R} ^ {D} \mid p _ {0} (\boldsymbol {x}) > p _ {1} (\boldsymbol {x}) \right\}} p _ {1} (\boldsymbol {x}) \log \frac {p _ {1} (\boldsymbol {x})}{p _ {0} (\boldsymbol {x})} \frac {1}{1 - \frac {p _ {0} (\boldsymbol {x})}{p _ {1} (\boldsymbol {x})}} (108) \\ \leq \int_ {\left\{\boldsymbol {x} \in \mathbb {R} ^ {D} \mid p _ {0} (\boldsymbol {x}) \leq p _ {1} (\boldsymbol {x}) \right\}} p _ {1} (\boldsymbol {x}) + \int_ {\left\{\boldsymbol {x} \in \mathbb {R} ^ {D} \mid p _ {0} (\boldsymbol {x}) > p _ {1} (\boldsymbol {x}) \right\}} p _ {1} (\boldsymbol {x}) \log \frac {p _ {1} (\boldsymbol {x})}{p _ {0} (\boldsymbol {x})} (109) \\ \leq 1 + D _ {\mathrm {K L}} \left(p _ {1}, p _ {0}\right) (110) \\ \end{array}
$$
Hence we get
$$
\frac {1}{N} \int_ {0} ^ {1} I (t) d t \leq \frac {1}{N} \times \left(2 + \mathcal {D} _ {\mathrm {K L}} \left(p _ {0}, p _ {1}\right) + \mathcal {D} _ {\mathrm {K L}} \left(p _ {1}, p _ {0}\right)\right). \tag {111}
$$
We now suppose the proposal and target are distributions in an exponential family. The KL divergence between exponential distributions with parameters  $\theta_0$  and  $\theta_{1}$ , is given by the Bregman divergence of the log-partition on the swapped parameters [34, Eq. 29]
$$
\begin{array}{l} \mathcal {D} _ {\mathrm {K L}} \left(p _ {0}, p _ {1}\right) = \mathcal {D} _ {\log Z} ^ {\text {B r e g m a n}} \left(\boldsymbol {\theta} _ {1}, \boldsymbol {\theta} _ {0}\right) := \log Z \left(\boldsymbol {\theta} _ {1}\right) - \log Z \left(\boldsymbol {\theta} _ {0}\right) - \nabla \log Z \left(\boldsymbol {\theta} _ {0}\right) \left(\boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0}\right) (112) \\ \leq \frac {L}{2} \left\| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \right\| ^ {2} (113) \\ \end{array}
$$
$$
\mathrm {M S E} \leq \frac {1}{N} \times (2 + L \| \left\| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \right\| ^ {2}) + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) \tag {114}
$$
using the  $L$ -smoothness of the log-partition function  $\log Z(\pmb{\theta})$
Discussion on the assumptions for theorems 2, 3, 4, 5 For these theorems, we have supposed that the target and proposal distributions are in an exponential family with a log partition that verifies
$$
M \mathbf {I d} \preccurlyeq \nabla_ {\boldsymbol {\theta}} ^ {2} \log Z (\boldsymbol {\theta}) \preccurlyeq L \mathbf {I d}. \tag {115}
$$
We now look at the validity of this assumption for a simple example: the univariate Gaussian, which is in an exponential family. The canonical parameters are its mean and variance  $(\mu, v)$ . Written as an exponential family,
$$
p (\boldsymbol {x}) := \exp \left(\left\langle \boldsymbol {\theta}, \boldsymbol {t} (\boldsymbol {x}) \right\rangle - \log Z (\boldsymbol {\theta})\right) \tag {116}
$$
the natural parameters are  $\pmb{\theta} = (\mu / v, -1/(2v))$ , associated with the sufficient statistics  $\pmb{t}(x) = (x, x^2)$  [34]. The log-partition function and its derivatives are
$$
\log Z (\boldsymbol {\theta}) = - \frac {\theta_ {1} ^ {2}}{4 \theta_ {2}} - \frac {1}{2} \log (- 2 \theta_ {2}) \tag {117}
$$
$$
\nabla \log Z (\boldsymbol {\theta}) = \mathbb {E} _ {x \sim p} [ t (x) ] = \left(- \frac {\theta_ {1}}{2 \theta_ {2}}, - \frac {1}{2 \theta_ {2}} + \frac {\theta_ {1} ^ {2}}{4 \theta_ {2} ^ {2}}\right) \tag {118}
$$
$$
\nabla^ {2} \log Z (\boldsymbol {\theta}) = \operatorname {V a r} _ {x \sim p} [ t (x) ] = \frac {1}{2 \theta_ {2}} \left( \begin{array}{c c} - 1 & \frac {\theta_ {1}}{\theta_ {2}} \\ \frac {\theta_ {1}}{\theta_ {2}} & \frac {1}{\theta_ {2}} - \frac {1}{2} \frac {\theta_ {1} ^ {2}}{\theta_ {2}} \end{array} \right) = \left( \begin{array}{c c} v & 2 \mu v \\ 2 \mu v & 2 v ^ {2} - \mu^ {2} \end{array} \right) \tag {119}
$$
When the mean is zero, the eigenvalues of the hessian are in fact the diagonal values  $(v, 2v^2)$ , and they are bounded if and only if the variance  $v$  is bounded.
Proof of Theorem 6 Constant error of annealed NCE with the arithmetic path and "oracle-trig" schedule
We now study the estimation error produced by the arithmetic path with "oracle-trig" schedule (table 2, line 5). We write the optimal path of Eq. 66 in the limit when the distributions have little overlap, pointwise (with error  $\epsilon$ ) and on average (with error  $\epsilon$ ):
$$
\sqrt {p _ {0} (\boldsymbol {x}) p _ {1} (\boldsymbol {x})} = \epsilon^ {\prime} (\boldsymbol {x}) \tag {120}
$$
$$
\int \sqrt {p _ {0} (\boldsymbol {x}) p _ {1} (\boldsymbol {x})} d \boldsymbol {x} = \epsilon \tag {121}
$$
We briefly note that there exist certain conditions, given by the dominated convergence theorem, where the first error (pointwise) going to zero implies the second (on average) going to zero as well, but this is outside the scope of this proof. Now, many relevant quantities involved in the optimal distribution and its estimation error simplify as  $\epsilon^{\prime}(\pmb {x})\to 0$  pointwise and  $\epsilon \rightarrow 0$ . The notation  $O(\cdot)$  will hide dependencies on absolute constants only.
$$
\mathcal {D} _ {H ^ {2}} \left(p _ {0}, p _ {1}\right) := 1 - \int \sqrt {p _ {0} p _ {1}} = 1 - \epsilon \tag {122}
$$
$$
\alpha_ {H} := \arctan \left(\sqrt {\frac {\mathcal {D} _ {H ^ {2}} \left(p _ {0} , p _ {1}\right)}{2 - \mathcal {D} _ {H ^ {2}} \left(p _ {0} , p _ {1}\right)}}\right) = \frac {\pi}{4} - \frac {\epsilon}{2} + o (\epsilon) \tag {123}
$$
$$
a _ {t} := \frac {\cos ((2 t - 1) \alpha_ {H})}{2 \cos (\alpha_ {H})} - \frac {\sin ((2 t - 1) \alpha_ {H})}{2 \sin (\alpha_ {H})} = \cos \left(\frac {\pi t}{2}\right) + \epsilon (t - 1) \sin \left(\frac {\pi t}{2}\right) + o (\epsilon) \tag {124}
$$
$$
b _ {t} = \frac {\cos ((2 t - 1) \alpha_ {H})}{2 \cos (\alpha_ {H})} + \frac {\sin ((2 t - 1) \alpha_ {H})}{2 \sin (\alpha_ {H})} = \sin \left(\frac {\pi t}{2}\right) - \epsilon t \cos \left(\frac {\pi t}{2}\right) + o (\epsilon) \tag {125}
$$
$$
\begin{array}{l} \partial_ {t} a _ {t} := - \alpha_ {H} \left(\frac {\sin \left((2 t - 1) \alpha_ {H}\right)}{\cos \left(\alpha_ {H}\right)} + \frac {\cos \left((2 t - 1) \alpha_ {H}\right)}{\sin \left(\alpha_ {H}\right)}\right) (126) \\ = - \frac {\pi}{2} \sin \left(\frac {\pi t}{2}\right) + \epsilon \left(\sin \left(\frac {\pi t}{2}\right) + \frac {\pi}{2} (t - 1) \cos \left(\frac {\pi t}{2}\right)\right) + o (\epsilon) (127) \\ \end{array}
$$
$$
\begin{array}{l} \partial_ {t} b _ {t} := - \alpha_ {H} \left(\frac {\sin ((2 t - 1) \alpha_ {H})}{\cos (\alpha_ {H})} - \frac {\cos ((2 t - 1) \alpha_ {H})}{\sin (\alpha_ {H})}\right) (128) \\ = \frac {\pi}{2} \cos \left(\frac {\pi t}{2}\right) + \epsilon \left(\frac {\pi}{2} t \sin \left(\frac {\pi t}{2}\right) - \cos \left(\frac {\pi t}{2}\right)\right) + o (\epsilon) (129) \\ \end{array}
$$
$$
a _ {t} \times \partial_ {t} a _ {t} := - \frac {\pi}{4} \sin (\pi t) + \epsilon \left(\frac {1}{2} \sin (\pi t) + \frac {\pi}{2} (t - 1) \cos (\pi t)\right) + o (\epsilon) \tag {130}
$$
$$
a _ {t} \times \partial_ {t} b _ {t} := \frac {\pi}{2} \cos^ {2} \left(\frac {\pi t}{2}\right) + \epsilon \left(\frac {\pi}{2} (1 - 2 t) \sin (\pi t) + \cos (\pi t) + 1\right) + o (\epsilon) \tag {131}
$$
$$
b _ {t} \times \partial_ {t} a _ {t} := - \frac {\pi}{2} \sin^ {2} \left(\frac {\pi t}{2}\right) + \epsilon \left(\frac {1}{2} - \frac {1}{2} \cos (\pi t) + \frac {1}{2} \frac {\pi}{2} \sin (\pi t) (2 t - 1)\right) + o (\epsilon) \tag {132}
$$
$$
b _ {t} \times \partial_ {t} b _ {t} := \frac {\pi}{4} \sin (\pi t) + \epsilon \left(- \frac {1}{2} \sin (\pi t) - \frac {\pi}{2} t \cos (\pi t)\right) + o (\epsilon) \tag {133}
$$
This leads to the following simplification of the optimal path
$$
\begin{array}{l} p _ {t} ^ {\text {o p t}} (\boldsymbol {x}) := \left(a _ {t} \sqrt {p _ {0} (\boldsymbol {x})} + b _ {t} \sqrt {p _ {1} (\boldsymbol {x})}\right) ^ {2} = a (t) ^ {2} p _ {0} (\boldsymbol {x}) + b (t) ^ {2} p _ {1} (\boldsymbol {x}) + 2 a (t) b (t) \epsilon^ {\prime} (\boldsymbol {x}) (134) \\ = a (t) ^ {2} p _ {0} (\boldsymbol {x}) + b (t) ^ {2} p _ {1} (\boldsymbol {x}) + 2 a (t) b (t) \epsilon^ {\prime} (\boldsymbol {x}) (135) \\ = \cos^ {2} \left(\frac {\pi t}{2}\right) p _ {0} (\boldsymbol {x}) + \sin^ {2} \left(\frac {\pi t}{2}\right) p _ {1} (\boldsymbol {x}) + \epsilon^ {\prime} (\boldsymbol {x}) \sin (\pi t) (136) \\ + \epsilon \sin (\pi t) \left(p _ {0} (\boldsymbol {x}) (t - 1) - p _ {1} (\boldsymbol {x}) t\right) + \epsilon \epsilon^ {\prime} (\boldsymbol {x}) ((1 - 2 t) \cos (\pi t) - 1) + o (\epsilon) (137) \\ = p _ {t} ^ {\operatorname {a r i t h - t r i g}} (\boldsymbol {x}) + O \left(\epsilon^ {\prime} (\boldsymbol {x})\right) + O \left(\epsilon g _ {1} (\boldsymbol {x})\right) (138) \\ \end{array}
$$
$$
p _ {t} ^ {\text {a r i t h - t r i g}} (\boldsymbol {x}) := \cos^ {2} \left(\frac {\pi t}{2}\right) p _ {0} (\boldsymbol {x}) + \sin^ {2} \left(\frac {\pi t}{2}\right) p _ {1} (\boldsymbol {x}) \tag {139}
$$
the arithmetic path with "oracle-trig" schedule defined in table 2 (line 5); the trigonometric weights in evolve slowly at the end points  $t = 0$  and  $t = 1$ . We also define  $g_{1}(\pmb{x}) = \sin(\pi t)(p_{0}(\pmb{x})(t - 1) - p_{1}(\pmb{x})t)$  which is an integrable function. This proves the first part of this theorem, which is that the optimal path  $p^{\mathrm{opt}}$  is close to a certain arithmetic path (with trigonometric weights)  $p^{\mathrm{arith-trig}}$ , and that closeness is controlled by how little overlap there is between the endpoint distributions  $p_{0}$  and  $p_{1}$ , on average and pointwise.
Similarly, we can control how close these two paths are in terms of estimation errors. The estimation error in Eq. 70 produced by a path is
$$
\begin{array}{l} \mathrm {M S E} := \frac {1}{N} \int_ {0} ^ {1} I (t) d t + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) (140) \\ = \frac {1}{N} \int_ {\mathbb {R} ^ {D}} \int_ {0} ^ {1} \left(\partial_ {t} \log p _ {t} (\boldsymbol {x})\right) ^ {2} p _ {t} (\boldsymbol {x}) d t d \boldsymbol {x} + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) (141) \\ = \frac {1}{N} \int_ {\mathbb {R} ^ {D}} \int_ {0} ^ {1} \left(\frac {\partial_ {t} p _ {t} (\boldsymbol {x})}{p _ {t} (\boldsymbol {x})}\right) ^ {2} p _ {t} (\boldsymbol {x}) d t d \boldsymbol {x} + o \left(\frac {1}{N}\right) + o \left(\frac {K ^ {2}}{N}\right) (142) \\ \end{array}
$$