SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It should be noted that the term  $\mathbb{E}_{T_f}[\int_0^1 ||f(X_t,t)||^2 dt]$  is not multiplied by  $\frac{1}{2\epsilon}$  in the algorithm since this does not affect the optimal  $T_{f*}$  solving the inner optimization problem, see Appendix D.
Algorithm 1: Entropic Neural OT (ENOT)
samples from distributions  $\mathbb{P}_0,\mathbb{P}_1$    
Wiener prior noise variance  $\epsilon \geq 0$    
drift network  $f_{\theta}:\mathbb{R}^{D}\times [0,1]\to \mathbb{R}^{D}$    
beta network  $\beta_{\phi}:\mathbb{R}^{D}\rightarrow \mathbb{R}$    
number of steps  $N$  for Eul-Mar (App C) number of inner iterations  $K_{f}$
Output: drift  $f_{\theta}^{*}$  of  $T_{f_{\theta}^{*}}$  solving DSB (11).
Sample batches  $X_0\sim \mathbb{P}_0,Y\sim \mathbb{P}_1$ $\{X_{n},f_{n}\}_{n = 0}^{N}\gets \mathrm{Eul - Mar}(X_{0},T_{f_{\theta}});$ $\mathcal{L}_{\beta}\gets \frac{1}{|X_N|}\sum_{x\in X_N}\beta_{\phi}(x) - \frac{1}{|Y|}\sum_{y\in Y}\beta_{\phi}(y);$
Update  $\phi$  by using  $\frac{\partial L_{\beta}}{\partial\phi}$
for  $k = 1$  to  $K_{f}$  do
Sample batches  $X_0\sim \mathbb{P}_0,Y\sim \mathbb{P}_1$ $\{X_{n},f_{n}\}_{n = 0}^{N}\gets \mathrm{Eul - Mar}(X_{0},T_{f_{\theta}});$ $\widehat{\mathrm{KL}}\leftarrow \frac{1}{N}\sum_{n = 0}^{N - 1}\frac{1}{|f_n|}\sum_{m = 1}^{|f_n|}||f_{n,m}||^2;$ $\mathcal{L}_f\gets \widehat{\mathrm{KL}} -\frac{1}{|X_N|}\sum_{x\in X_N}\beta_\phi (x);$  Update  $\theta$  by using  $\frac{\partial L_\theta}{\partial\theta}$
until converged;
Relation to GANs. At the first glance, our method might look like a typical GAN as it solves a maximin problem with the "discriminator"  $\beta_{\phi}$  and SDE "generator" with the drift  $f_{\theta}$ . Unlike GANs, in our saddle point objective (12), optimization of "generator"  $T_{f}$  and "discriminator"  $\beta$  are swapped, i.e., "generator" is adversarial to "discriminator", not vise versa, as in GANs. For further discussion of differences between saddle point objectives of neural OT/GANs, see [31, §4.3], [42, §4.3], [16].
4.3 Error Bounds via Duality Gaps
Our algorithm solves a maximin optimization problem and recovers some approximate solution  $(\hat{\beta}, T_{\hat{f}})$ . Given such a pair, it is natural to wonder how close is the recovered  $T_{\hat{f}}$  to the optimal  $T_{f^*}$ . Our next result sheds light on this question via bounding the error with the duality gaps.
Theorem 4.3 (Error analysis via duality gaps). Consider a pair  $(\hat{\beta}, T_{\hat{f}})$ . Define the duality gaps, i.e., errors of solving inner and outer optimization problems by:
$$
\epsilon_ {1} \stackrel {\text {d e f}} {=} \mathcal {L} (\hat {\beta}, T _ {\hat {f}}) - \inf  _ {T _ {f}} \mathcal {L} (\hat {\beta}, T _ {f}), \quad \text {a n d} \quad \epsilon_ {2} \stackrel {\text {d e f}} {=} \sup  _ {\beta} \inf  _ {T _ {f} \in \mathcal {D} (\mathbb {P} _ {0})} \mathcal {L} (\beta , T _ {f}) - \inf  _ {T _ {f}} \mathcal {L} (\hat {\beta}, T _ {f}). \tag {15}
$$
Then it holds that
$$
\rho_ {T V} \left(T _ {\hat {f}}, T _ {f ^ {*}}\right) \leq \sqrt {\epsilon_ {1} + \epsilon_ {2}}, \quad a n d \quad \rho_ {T V} \left(\pi^ {T _ {\hat {f}}}, \pi^ {T _ {f ^ {*}}}\right) \leq \sqrt {\epsilon_ {1} + \epsilon_ {2}}, \tag {16}
$$
where we use  $\rho_{TV}(\cdot ,\cdot)$  to denote the total variation norm (between the processes or plans).
4 In practice,  $\beta_{\phi} \in \mathcal{C}_{b,2}(\mathcal{V})$  since we can choose  $u = 0$ ,  $v = \min(\text{float32})$ ,  $w = \max(\text{float32})$ .
(a)  $x\sim \mathbb{P}_0,y\sim \mathbb{P}_1$
(b) ENOT (ours),  $\epsilon = 0$
(c) ENOT (ours),  $\epsilon = 0.01$
Figure 2: Gaussian  $\rightarrow$  Mix of 8 Gaussians. The process learned with ENOT (ours) for  $\epsilon = 0, 0.01, 0.1$ .
(d) ENOT (ours),  $\epsilon = 0.1$
Relation to prior works. There exist deceptively similar results, see [38, Theorem 3.6], [42, Theorem 4.3], [16, Theorem 4], [6, Theorem 3]. None of them are relevant to our EOT/DSB case.
In [38], [42], [16], the authors consider maximin reformulations of unregularized OT (1), i.e., non-entropy. Their result requires the potential  $\beta (f,\psi$  in their notation) to be a convex function which in practice means that one has to employ ICNNs [4] which have poor expressiveness [29, 17, 26]. Our result is free from such assumptions on  $\beta$ . In [6], the authors consider general OT problem [39] and require the general cost functional to be strongly convex (in some norm). Their results also do not apply to our case as the (negative) entropy which we consider is not strongly convex.
5 Experimental Illustrations
In this section, we qualitatively and quantitatively illustrate the performance of our algorithm in several entropic OT tasks. Our proofs apply only to EOT  $(\epsilon > 0)$ , but for completeness, we also present results  $\epsilon = 0$ , i.e., unregularized case (1). Furthermore, we test our algorithm with  $\epsilon = 0$  on the Wasserstein-2 Benchmark [28], see Appendix J. We also demonstrate the extension of our algorithm to costs other than the squared Euclidean distance in Appendix I. The implementation details are given in Appendices E, F and G. The code is written in PyTorch and is publicly available at
5.1 Toy 2D experiments
Here we give qualitative examples of our algorithm's performance on toy 2D pairs of distributions. We consider two pairs  $\mathbb{P}_0, \mathbb{P}_1$ : Gaussian  $\rightarrow$  Swiss Roll, Gaussian  $\rightarrow$  Mixture of 8 Gaussians. We provide qualitative results in Figure 2 and Figure 5 (Appendix E), respectively. In both cases, we provide solutions of the problem for  $\epsilon = 0, 0.01, 0.1$  and sample trajectories. For  $\epsilon = 0$ , all the trajectories are straight lines as they represent solutions for non-regularized OT (1), see [43, §5.4]. For bigger  $\epsilon$ , trajectories, as expected, become more noisy and less straight.
5.2 High-dimensional Gaussians
For general continuous distributions  $\mathbb{P}_0, \mathbb{P}_1$ , the ground truth solution of entropic OT (2) and DSB (11) is unknown. This makes it challenging to assess how well does our algorithm solve these problems. Fortunately, when  $\mathbb{P}_0$  and  $\mathbb{P}_1$  are Gaussians, there exist closed form solutions of these related problems, see [25] and [9]. Thus, to quantify the performance of our algorithm, we consider entropic OT problems in dimensions  $D \in \{2, 16, 64, 128\}$  with  $\epsilon = 1$  for Gaussian  $\mathbb{P}_0 = \mathcal{N}(0, \Sigma_0)$ ,  $\mathbb{P}_1 = \mathcal{N}(0, \Sigma_1)$ . We pick  $\Sigma_0, \Sigma_1$  at random: their eigenvectors are uniformly distributed on the unit sphere and eigenvalues are sampled from the loguniform distribution on  $[- \log 2, \log 2]$ .
Dim	2	16	64	128	Dim	2	16	64	128
ENOT (ours)	0.01(±0.006)	0.09(±0.02)	0.23(±0.03)	0.50(±0.08)	ENOT (ours)	0.012(±0.003)	0.05(±0.01)	0.13(±0.014)	0.29(±0.04)
LSOT [45]	1.82	6.42	32.18	64.32	LSOT [45]	6.77	14.56	25.56	47.11
SCONES [14]	1.74	1.87	6.27	6.88	SCONES [14]	0.92	1.36	4.62	5.33
MLE-SB [48]	0.41	0.50	1.16	2.13	MLE-SB [48]	0.3	0.9	1.34	1.8
DiffSB [15]	0.7	1.11	1.98	2.20	DiffSB [15]	0.88	1.7	2.32	2.43
FB-SDE-A [10]	0.87	0.94	1.85	1.95	FB-SDE-A [10]	0.75	1.36	2.45	2.64
FB-SDE-J [10]	0.03	0.05	0.19	0.39	FB-SDE-J [10]	0.07	0.22	0.34	0.58
Table 1: Comparisons of  $\mathrm{BW}_2^2$ -UVP  $\downarrow$  (\%) between the target  $\mathbb{P}_1$  and learned marginal  $\pi_1$ .
Table 2: Comparisons of  $\mathrm{BW}_2^2$ -UVP  $\downarrow$  (\%) between the EOT plan  $\pi^{*}$  and learned plan  $\pi$ .
t, time	0	0.2	0.4	0.6	0.8	1
ENOT (ours)	0	0.01(±0.001)	0.023(±0.005)	0.042(±0.007)	0.067(±0.015)	0.096(±0.019)
LSOT [45]	0	N/A	N/A	N/A	N/A	6.42
SCONES [14]	0	N/A	N/A	N/A	N/A	6.88
MLE-SB [48]	0	0.10	0.23	0.30	0.36	0.50
DiffSB [15]	0	0.19	0.48	0.68	0.91	1.11
FB-SDE-A [10]	0	0.17	0.45	0.61	0.77	0.94
FB-SDE-J [10]	0	0.18	0.32	0.31	0.17	0.05
Table 3: Comparisons of  $\mathrm{BW}_2^2$ -UVP  $\downarrow$  (\%) between the learned marginal distributions and the ground truth marginal distributions at the intermediate time moments  $t = 0, \frac{2}{10}, \ldots, 1$  in dimension  $D = 16$ .
Metrics. We evaluate (a) how precisely our algorithm fits the target distribution  $\mathbb{P}_1$  on  $\mathbb{R}^D$ ; (b) how well it recovers the entropic OT plan  $\pi^*$  which is a Gaussian distribution on  $\mathbb{R}^D \times \mathbb{R}^D$ ; (c) how accurate are the learned marginal distributions at intermediate times  $t = 0, \frac{1}{10}, \ldots, 1$ .
In each of the above-mentioned cases, we compute the  $\mathrm{BW}_2^2$ -UVP [29, §5] between the learned and the ground truth distributions. For two distributions  $\widehat{\chi}$  and  $\chi$ , it is the Wasserstein-2 distance between their Gaussian approximations which is further normalized by the variance of the distribution  $\chi$ :
$$
\mathrm {B} \mathbb {W} _ {2} ^ {2} - \operatorname {UVP} (\widehat {\chi}, \chi) = \frac {100 \%}{\frac {1}{2} \operatorname {Var} (\chi)} \mathbb {W} _ {2} ^ {2} \left(\mathcal {N} \left(\mu_ {\widehat {\chi}}, \Sigma_ {\widehat {\chi}}\right), \mathcal {N} \left(\mu_ {\chi}, \Sigma_ {\chi}\right)\right). \tag{17}
$$
We estimate the metric by using  $10^{5}$  samples.
Baselines. We compare our method ENOT with LSOT [45], SCONES [14], MLE-SB [48], DiffSB[15], FB-SDE [10] (two algorithms, A and J). Results are given in Tables 1, 2 and 3. LSOT and SCONES solve EOT without solving SB, hence there are no results in Table 3 for these methods. Our method achieves low  $\mathrm{BW}_2^2$ -UVP values indicating that it recovers the ground truth process and entropic plan fairly well. The competitive methods have good results in low dimensions, however they perform worse in high dimensions. Importantly, our methods score the best results in recovering the OT plan (Table 2) and the marginal distributions of the Schrodinger bridge (Table 3). To illustrate the stable convergence of ENOT, we provide the plot of  $\mathrm{BW}_2^2$ -UVP between the learned plan and the ground truth plan for ENOT during training in Figure 6 (Appendix E).
5.3 Colored MNIST
In this section, we test how the entropy parameter  $\epsilon$  affects the stochasticity of the learned plan in higher dimensions. For this, we consider the entropic OT problem between colorized MNIST digits of classes "2"  $(\mathbb{P}_0)$  and "3"  $(\mathbb{P}_1)$ .
Effect of parameter  $\epsilon$ . For  $\epsilon = 0,1,10$ , we learn our Algorithm 1 on the train sets of digits "2" and "3". We show the translated test images in Figures 3a, 3b and 3c, respectively. When  $\epsilon = 0$ , there is no diversity in generated "3" samples (Figure 3a), the color remains since the map tried to minimally change the image in the RGB pixel space. When  $\epsilon = 1$ , some slight diversity in the shape of "3" appears but the color of the input "2" is still roughly preserved (Figure 3c). For higher  $\epsilon$ , the diversity of generated samples becomes clear (Figure 3c). In particular, the color of "3" starts to slightly deviate from the input "2". That is, increasing the value  $\epsilon$  of the entropy term in (2) expectedly leads to bigger stochasticity in the plan. We add the conditional LPIPS variance [24, Table 1] of generated samples for test datasets by ENOT to show how diversity changes for different  $\epsilon$  (Table 4). We provide examples of trajectories learned by ENOT in Figure 7 (Appendix F).
Metrics and baselines. We compare our method ENOT with SCONES [14], and DiffSB [15] as these are the only methods which the respective authors applied for data  $\rightarrow$  data tasks. To evaluate the results, we use the FID metric [23] which is the Bures-Wasserstein (Freschet) distance between the
(a) ENOT (ours) samples, (b) ENOT (ours) samples, (c) ENOT (ours) samples,  $\epsilon = 0$ , FID:6.0  $\epsilon = 1$ , FID:6.28  $\epsilon = 10$ , FID:6.9
(f) DOT samples,  $\epsilon = 0$  FID:N/A
(g) DOT samples,  $\epsilon = 1$  FID:N/A
(h) DOT samples,  $\epsilon = 10$  FID:N/A
Figure 3: Samples of colored MNIST obtained by ENOT (ours) and DOT for different  $e$ .
(d) DiffSB [15] samples, (e) DiffSB [15] samples,  $\epsilon = 1$  FID:93  $\epsilon = 10$  FID:105
(i) SCONES [14] samples (j) SCONES [14] samples  $\epsilon = 25$ , FID:14.73  $\epsilon = 100$ , FID:14.22
distributions after extracting features using the InceptionV3 model [47]. We measure test FID for every method and present the results and qualitative examples in Figure 3. There are no results for SCONES with  $\epsilon = 0,1,10$ , since it is not applicable for such reasonably small  $\epsilon$  due to computational instabilities, see [14, §5.1]. DiffSB [15] can be applied for small regularization  $\epsilon$ , so we test  $\epsilon = 1,10$ . By the construction, this algorithm is not suitable for  $\epsilon = 0$ .
Our ENOT method outperforms the baselines in FID. DiffSB [15] yield very high FID. This is presumably due to instabilities of DiffSB which the authors report in their sequel paper [46]. SCONES yields reasonable quality but due to high  $\epsilon = 25$ , 100 the shape and color of the generated images "3" starts to deviate from those of their respective inputs "2".
For completeness, we provide the results of the stochastic matching of the test parts of the datasets by the discrete OT for  $\epsilon = 0$  and EOT [12] for  $\epsilon = 1, 10$  (Figures 3f, 3g, 3h). This is not the out-of-sample estimation, obtained samples "3" are just test samples of "3" (this setup is unfair). Discrete OT is not a competitor here as it does not generate new samples and uses target test samples. Still it gives a rough estimate what to expect from the learned plans for increasing  $\epsilon$ .
5.4 Unpaired Super-resolution of Celeba Faces
For the large-scale evaluation, we adopt the experimental setup of SCONES [14]. We consider the problem of unpaired image super-resolution for the  $64\times 64$  aligned faces of CelebA dataset [35].
ε	0	1	10
Colored MNIST	0	5.3 · 10-3	2.0 · 10-2
Celeba	0	3.4 · 10-2	5.1 · 10-2
Table 4: LPIPS variability of ENOT samples.
We do the unpaired train-test split as follows: we split the dataset into 3 parts: 90k (train A1), 90k (train B1), 20k (test C1) samples. For each part we do  $2 \times$  bilinear downsample and then  $2 \times$  bilinear upsample to degrade images but keep the original size. As a result, we obtain degraded parts A0, B0, C0. For training in the unpaired setup, we use parts A0 (degraded faces,  $\mathbb{P}_0$ ) and B1 (clean faces,  $\mathbb{P}_1$ ). For testing, we use the hold-out part C0 (unseen samples) with C1 considered as the reference.
We train our model with  $\epsilon = 0,1,10$  to and test how it restores C1 (Figure 4a) from C0 images (Figure 4b) and present the qualitative results in Figures 4f, 4g, 4h. We provide examples of trajectories learned by ENOT in Figure 1.
Metrics and baselines. To quantify the results, as in [14], we compute the FID score [23] between the sets of mapped C0 images and C1 images (Table 5). The FID of ENOT is better than FID values of the other methods, but increases with  $\epsilon$  probably due to the increasing variance of gradients during training. As in §5.1 and §5.3, the diversity of samples increases with  $\epsilon$ . Our method works with small values of  $\epsilon$  and provides reasonable amount of diversity in the mapped samples which grows with  $\epsilon$  (Table 4). As the baseline among other methods for EOT we consider only SCONES, as it is the only EOT/DSB algorithm which has been applied to data→data task at  $64 \times 64$  resolution. At the same time, we emphasize that SCONES is not applicable for small  $\epsilon$  due to instabilities, see [14, §5.1]. This makes it impractical, as due to high  $\epsilon$ , its produces up-scaled images (Figures 4c) are
(c) SCONES,  $\epsilon = 100$
(d) ICNN
(f) ENOT (ours),  $\epsilon = 0$
Figure 4a shows test degraded images (C0), 4b - their original high-resolution counterparts (C1).
(g) ENOT (ours),  $\epsilon = 1$
(e) AugCycleGAN
(h) ENOT (ours),  $\epsilon = 10$
Figure 4: Faces produced by ENOT (ours) and SCONES for various  $\epsilon$ .
Method	\( \mathbf{{ENOT}},\epsilon = 0 \)	\( \mathbf{{ENOT}},\epsilon = 1 \)	\( \mathbf{{ENOT}},\epsilon = {10} \)	SCONES [14], \( \epsilon = {100} \)	AugCycleGAN [2]	ICNN [38]
FID	3.78	7.63	14.8	18.88	15.2	22.2
Table 5: Test FID values of various methods in unpaired super-resolution of faces experiment.
nearly random and do not reflect the attributes of the input images (Figure 4a). We do not provide results for DiffSB [15] since it already performs bad on Colored MNIST ( $\S 5.3$ ) and the authors also did not consider any image-to-image apart of grayscale  $28 \times 28$  images.
For completeness, we present results on this setup for other methods, which do not solve EOT: ICNN-based OT [38] and AugCycleGAN [2]. ICNN (4d) learns a deterministic map. AugCycleGAN (4e) learns a stochastic map, but the generated samples differ only by brightness.
Potential impact. There is a lack of scalable algorithms for learning continuous entropic OT plans which may be used in data  $\rightarrow$  data practical tasks requiring control of the diversity of generated samples. We hope that our results provide a new direction for research towards establishing scalable and efficient methods for entropic OT by using its connection with SB.
Potential social impact. Like other popular methods for generating images, our method can be used to simplify the work of designers with digital images and create new products based on it. At the same time, our method may be used for creating fake images just like the other generative models.
Limitations. To simulate the trajectories following SDE (9), we use the Euler-Maruyama scheme. It is straightforward but may be imprecise when the number of steps is small or the noise variance  $\epsilon$  is high. As a result, for large  $\epsilon$ , our Algorithm 1 may be computationally heavy due to the necessity to backpropagate through a large computational graph obtained via the simulation. Employing time and memory efficient SDE integration schemes is a promising avenue for the future work.
ACKNOWLEDGEMENTS. This work was partially supported by Skoltech NGP program (Skoltech-MIT joint project).
[40] Michele Pavon and Anton Wakolbinger. On free energy, stochastic control, and schrödinger processes. In Modeling, Estimation and Control of Systems with Uncertainty: Proceedings of a Conference held in Sopron, Hungary, September 1990, pages 334-348. Springer, 1991.
[41] Gabriel Peyre, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends® in Machine Learning, 11(5-6):355–607, 2019.
[42] Litu Rout, Alexander Korotin, and Evgeny Burnaev. Generative modeling with optimal transport maps. In International Conference on Learning Representations, 2022.
[43] Filippo Santambrogio. Optimal transport for applied mathematicians. *Birkhäuser*, NY, 55(58-63):94, 2015.
[44] Erwin Schrödinger. Über die umkehrung der natürgesetze. Verlag der Akademie der Wissenschaften in Kommission bei Walter De Gruyter u. Company., 1931.
[45] Vivien Seguy, Bharath Bhushan Damodaran, Remi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large scale optimal transport and mapping estimation. In International Conference on Learning Representations, 2018.
[46] Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schrödinger bridge matching. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[47] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[48] Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence. Solving schrödinger bridges via maximum likelihood. Entropy, 23(9):1134, 2021.
[49] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
[50] Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning via schrödinger bridge. In International Conference on Machine Learning, pages 10794-10804. PMLR, 2021.
[51] Yujia Xie, Minshuo Chen, Haoming Jiang, Tuo Zhao, and Hongyuan Zha. On scalable and efficient computation of large scale optimal transport. volume 97 of Proceedings of Machine Learning Research, pages 6882-6892, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
A Extended background: KL divergence with the Wiener process plan
This section illustrates that (7) holds. Consider a process  $T \in \mathcal{F}(\mathbb{P}_0)$ , i.e.,  $T$  is a probability distribution on  $\Omega$  with the marginal  $\mathbb{P}_0$  at  $t = 0$ .
Let  $W^{\epsilon}$  be the Wiener process with variance  $\epsilon$  starting at  $\mathbb{P}_0$ , i.e., it satisfies  $dX_{t} = \sqrt{\epsilon} dW_{t}$  with  $X_0\sim \mathbb{P}_0$ . Hence,  $\pi^{W^{\epsilon}}(y|x)$  is the normal distribution  $\frac{d\pi^{W^{\epsilon}}(y|x)}{dy} = \mathcal{N}(y|x,\epsilon I)$ . Then  $\mathrm{KL}(\pi^T ||\pi^{W^\epsilon})$  between joint distributions at times  $t = 0$  and  $t = 1$  of these processes is given by:
$$
\mathrm {K L} \left(\pi^ {T} \mid \mid \pi^ {W ^ {\epsilon}}\right) = - \int_ {\mathcal {X} \times \mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (x , y)}{d [ x , y ]} d \pi^ {T} (x, y) \underbrace {+ \int_ {\mathcal {X} \times \mathcal {Y}} \log \frac {d \pi^ {T} (x , y)}{d [ x , y ]} d \pi^ {T} (x , y)} _ {= - H \left(\pi^ {T}\right)}, \tag {18}
$$
where  $\frac{d\pi(x,y)}{d[x,y]}$  denotes the joint density of distribution  $\pi$ . We derive
$$
\begin{array}{l} - \int_ {\mathcal {X} \times \mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (x , y)}{d [ x , y ]} d \pi^ {T} (x, y) = - \int_ {\mathcal {X} \times \mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (y | x)}{d y} \frac {d \pi^ {W ^ {\epsilon}} (x)}{d x} d \pi^ {T} (x, y) = \\ - \int_ {\mathcal {X} \times \mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (y | x)}{d y} d \pi^ {T} (x, y) - \int_ {\mathcal {X}} \int_ {\mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (x)}{d x} d \pi^ {T} (y | x) \overbrace {d \mathbb {P} _ {0} (x)} ^ {d \pi_ {0} ^ {T} (x)} = \\ - \int_ {\mathcal {X} \times \mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (y | x)}{d y} d \pi^ {T} (x, y) - \int_ {\mathcal {X}} \log \frac {d \pi^ {W ^ {\epsilon}} (x)}{d x} \left[ \int_ {\mathcal {Y}} 1 d \pi^ {T} (y | x) \right] d \mathbb {P} _ {0} (x) = \\ - \int_ {\mathcal {X}} \int_ {\mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (y | x)}{d y} d \pi^ {T} (x, y) - \int_ {\mathcal {X}} \log \frac {d \pi^ {W ^ {\epsilon}} (x)}{d x} d \mathbb {P} _ {0} (x) = \\ - \int_ {\mathcal {X}} \int_ {\mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (y | x)}{d y} d \pi^ {T} (x, y) - \int_ {\mathcal {X}} \log \frac {d \mathbb {P} _ {0} (x)}{d x} d \mathbb {P} _ {0} (x) = \\ - \int_ {\mathcal {X} \times \mathcal {Y}} \log \frac {d \pi^ {W ^ {\epsilon}} (y | x)}{d y} d \pi^ {T} (x, y) + H (\mathbb {P} _ {0}) = \\ - \int_ {\mathcal {X} \times \mathcal {Y}} \log \left((2 \pi \epsilon) ^ {\frac {- D}{2}} \exp \left(- \frac {| | x - y | | ^ {2}}{2 \epsilon}\right)\right) d \pi^ {T} (x, y) + H (\mathbb {P} _ {0}) = \\ + \frac {D}{2} \log (2 \pi \epsilon) + \int_ {\mathcal {X} \times \mathcal {Y}} \frac {| | x - y | | ^ {2}}{2 \epsilon} d \pi^ {T} (x, y) + H (\mathbb {P} _ {0}). \\ \end{array}
$$
After substituting this result into (18), one obtains:
$$
\operatorname {K L} \left(\pi^ {T} \mid \mid \pi^ {W ^ {\epsilon}}\right) = \int_ {\mathcal {X} \times \mathcal {Y}} \frac {\left| \left| x - y \right| \right| ^ {2}}{2 \epsilon} d \pi^ {T} (x, y) - H \left(\pi^ {T}\right) + \underbrace {\frac {D}{2} \log \left(2 \pi \epsilon\right) + H \left(\mathbb {P} _ {0}\right)} _ {= C \text {i n} (7)}. \tag {19}
$$
In this section, we provide the proof for our main theoretical results (Theorems 4.1 and 4.3). The proofs require several auxiliary results which we formulate and prove in §B.1 and §B.2.