SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jaemoo Choi*
Seoul National University toony42@snu.ac.kr
Jaewoong Choi *
Korea Institute for Advanced Study chwj1475@kias.re.kr
Myungjoo Kang
Seoul National University mkang@snu.ac.kr
Optimal Transport (OT) problem investigates a transport map that bridges two distributions while minimizing a given cost function. In this regard, OT between tractable prior distribution and data has been utilized for generative modeling tasks. However, OT-based methods are susceptible to outliers and face optimization challenges during training. In this paper, we propose a novel generative model based on the semi-dual formulation of Unbalanced Optimal Transport (UOT). Unlike OT, UOT relaxes the hard constraint on distribution matching. This approach provides better robustness against outliers, stability during training, and faster convergence. We validate these properties empirically through experiments. Moreover, we study the theoretical upper-bound of divergence between distributions in UOT. Our model outperforms existing OT-based generative models, achieving FID scores of 2.97 on CIFAR-10 and 6.36 on CelebA-HQ-256. The code is available at https://github.com/Jae-Moo/UOTM.
Optimal Transport theory [57, 77] explores the cost-optimal transport to transform one probability distribution into another. Since WGAN [6], OT theory has attracted significant attention in the field of generative modeling as a framework for addressing important challenges in this field. In particular, WGAN introduced the Wasserstein distance, an OT-based probability distance, as a loss function for optimizing generative models. WGAN measures the Wasserstein distance between the data distribution and the generated distribution, and minimizes this distance during training. The introduction of OT-based distance has improved the diversity [6, 27], convergence [63], and stability [36, 52] of generative models, such as WGAN [6] and its variants [27, 45, 56]. However, several works showed that minimizing the Wasserstein distance still faces computational challenges, and the models tend to diverge without a strong regularization term, such as gradient penalty [51, 63].
Recently, there has been a surge of research on directly modeling the optimal transport map between the input prior distribution and the real data distribution [2, 3, 49, 61, 82]. In other words, the optimal transport serves as the generative model itself. These approaches showed promising results that are comparable to WGAN models. However, the classical optimal transport-based approaches are known to be highly sensitive to outlier-like samples [8]. The existence of a few outliers can have a significant impact on the overall OT-based distance and the corresponding transport map. This sensitivity can be problematic when dealing with large-scale real-world datasets where outliers and noises are inevitable.
To overcome these challenges, we suggest a new generative algorithm utilizing the semi-dual formulation of the Unbalanced Optimal Transport (UOT) problem [12, 44]. In this regard, we refer to our model as the UOT-based generative model (UOTM). The UOT framework relaxes the hard constraints
*Equal contribution. Correspondence to: Myungjoo Kang <mkang@snu.ac.kr>.
Figure 1: Generated Samples from UOTM trained on Left: CIFAR-10 and Right: CelebA-HQ.
of marginal distribution in the OT framework by introducing soft entropic penalties. This soft constraint provides additional robustness against outliers [8, 66]. Our experimental results demonstrate that UOTM exhibits such outlier robustness, as well as faster and more stable convergence compared to existing OT-based models. Particularly, this better convergence property leads to a tighter matching of data distribution than the OT-based framework despite the soft constraint of UOT. Our UOTM achieves FID scores of 2.97 on CIFAR-10 and 6.36 on CelebA-HQ, outperforming existing OT-based adversarial methods by a significant margin and approaching state-of-the-art performance. Furthermore, this decent performance is maintained across various objective designs. Our contributions can be summarized as follows:
- UOTM is the first generative model that utilizes the semi-dual form of UOT.
- We analyze the theoretical upper-bound of divergence between marginal distributions in UOT, and validate our findings through empirical experiments.
- We demonstrate that UOTM presents outlier robustness and fast and stable convergence.
- To the best of our knowledge, UOTM is the first OT-based generative model that achieves near state-of-the-art performance on real-world image datasets.
Notations Let  $\mathcal{X},\mathcal{Y}$  be two compact complete metric spaces,  $\mu$  and  $\nu$  be probability distributions on  $\mathcal{X}$  and  $\mathcal{Y}$ , respectively. We regard  $\mu$  and  $\nu$  as the source and target distributions. In generative modeling tasks,  $\mu$  and  $\nu$  correspond to tractable noise and data distributions. For a measurable map  $T$ ,  $T_{\#}\mu$  represents the pushforward distribution of  $\mu$ .  $\Pi (\mu ,\nu)$  denote the set of joint probability distributions on  $\mathcal{X}\times \mathcal{Y}$  whose marginals are  $\mu$  and  $\nu$ , respectively.  $\mathcal{M}_{+}(\mathcal{X}\times \mathcal{Y})$  denote the set of joint positive measures defined on  $\mathcal{X}\times \mathcal{Y}$ . For convenience, for  $\pi \in \mathcal{M}_{+}(\mathcal{X}\times \mathcal{Y})$ , let  $\pi_0(x)$  and  $\pi_1(y)$  be the marginals with respect to  $\mathcal{X}$  and  $\mathcal{Y}$ .  $c(x,y)$  refers to the transport cost function defined on  $\mathcal{X}\times \mathcal{Y}$ . Throughout this paper, we consider  $\mathcal{X} = \mathcal{Y}\subset \mathbb{R}^d$  with the quadratic cost,  $c(x,y) = \tau \| x - y\| _2^2$ , where  $d$  indicates the dimension of data. Here,  $\tau$  is a given positive constant. For the precise notations and assumptions, see Appendix A.
Optimal Transport (OT) OT addresses the problem of searching for the most cost-minimizing way to transport source distribution  $\mu$  to target  $\nu$ , based on a given cost function  $c(\cdot, \cdot)$ . In the beginning, Monge [53] formulated this problem with a deterministic transport map. However, this formulation is non-convex and can be ill-posed depending on the choices of  $\mu$  and  $\nu$ . To alleviate these problems, the Kantorovich OT problem [33] was introduced, which is a convex relaxation of the Monge problem. Formally, the OT cost of the relaxed problem is given by as follows:
$$
C (\mu , \nu) := \inf  _ {\pi \in \Pi (\mu , \nu)} \left[ \int_ {\mathcal {X} \times \mathcal {Y}} c (x, y) d \pi (x, y) \right], \tag {1}
$$
where  $c$  is a cost function, and  $\pi$  is a coupling of  $\mu$  and  $\nu$ . Unlike Monge problem, the minimizer  $\pi^{\star}$  of Eq 1 always exists under some mild assumptions on  $(\mathcal{X},\mu)$ ,  $(\mathcal{Y},\nu)$  and the cost  $c$  ([77], Chapter 5). Then, the dual form of Eq 1 is given as:
$$
C (\mu , \nu) = \sup  _ {u (x) + v (y) \leq c (x, y)} \left[ \int_ {\mathcal {X}} u (x) d \mu (x) + \int_ {\mathcal {Y}} v (y) d \nu (y) \right], \tag {2}
$$
where  $u$  and  $v$  are Lebesgue integrable with respect to measure  $\mu$  and  $\nu$ , i.e.,  $u \in L^{1}(\mu)$  and  $v \in L^{1}(\nu)$ . For a particular case where  $c(x,y)$  is equal to the distance function between  $x$  and  $y$ , then  $u = -v$  and  $u$  is 1-Lipschitz [77]. In such case, we call Eq 2 a Kantorovich-Rubinstein duality. For the general cost  $c(\cdot ,\cdot)$ , the Eq 2 can be reformulated as follows ([77], Chapter 5):
$$
C (\mu , \nu) = \sup  _ {v \in L ^ {1} (\nu)} \left[ \int_ {\mathcal {X}} v ^ {c} (x) d \mu (x) + \int_ {\mathcal {Y}} v (y) d \nu (y) \right], \tag {3}
$$
where the  $c$ -transform of  $v$  is defined as  $v^{c}(x) = \inf_{y\in \mathcal{V}}(c(x,y) - v(y))$ . We call this formulation (Eq 3) a semi-dual formulation of OT.
Unbalanced Optimal Transport (UOT) Recently, a new type of optimal transport problem has emerged, which is called Unbalanced Optimal Transport (UOT) [12, 44]. The Csiszar divergence  $D_{f}(\mu |\nu)$  associated with  $f$  is a generalization of  $f$ -divergence for the case where  $\mu$  is not absolutely continuous with respect to  $\nu$  (See Appendix A for the precise definition). Here, the entropy function  $f:[0,\infty)\to [0,\infty]$  is assumed to be convex, lower semi-continuous, and non-negative. Note that the  $f$ -divergence families include a wide variety of divergences, such as Kullback-Leibler (KL) divergence and  $\chi^2$  divergence. (See [71] for more examples of  $f$ -divergence and its corresponding generator  $f$ .) Formally, the UOT problem is formulated as follows:
$$
C _ {u b} (\mu , \nu) := \inf  _ {\pi \in \mathcal {M} + (\mathcal {X} \times \mathcal {Y})} \left[ \int_ {\mathcal {X} \times \mathcal {Y}} c (x, y) d \pi (x, y) + D _ {\Psi_ {1}} \left(\pi_ {0} | \mu\right) + D _ {\Psi_ {2}} \left(\pi_ {1} | \nu\right) \right]. \tag {4}
$$
The UOT formulation (Eq 4) has two key properties. First, UOT can handle the transportation of any positive measures by relaxing the marginal constraint [12, 23, 44, 57], allowing for greater flexibility in the transport problem. Second, UOT can address the sensitivity to outliers, which is a major limitation of OT. In standard OT, the marginal constraints require that even outlier samples are transported to the target distribution. This makes the OT objective (Eq 1) to be significantly affected by a few outliers. This sensitivity of OT to outliers implies that the OT distance between two distributions can be dominated by these outliers [8, 66]. On the other hand, UOT can exclude outliers from the consideration by flexibly shifting the marginals. Both properties are crucial characteristics of UOT. However, in this work, we investigate a generative model that transports probability measures. Because the total mass of probability measure is equal to 1, we focus on the latter property.
3.1 Dual and Semi-dual Formulation of UOT
Similar to OT, UOT also provides a dual formulation [12, 21, 72]:
$$
C _ {u b} (\mu , \nu) = \sup  _ {u (x) + v (y) \leq c (x, y)} \left[ \int_ {\mathcal {X}} - \Psi_ {1} ^ {*} (- u (x)) d \mu (x) + \int_ {\mathcal {Y}} - \Psi_ {2} ^ {*} (- v (y)) d \nu (y) \right], \tag {5}
$$
with  $u \in \mathcal{C}(\mathcal{X})$ ,  $v \in \mathcal{C}(\mathcal{Y})$  where  $\mathcal{C}$  denotes a set of continuous functions over its domain. Here,  $f^*$  denotes the convex conjugate of  $f$ , i.e.,  $f^*(y) = \sup_{x \in \mathbb{R}} \{\langle x, y \rangle - f(x)\}$  for  $f: \mathbb{R} \to [-\infty, \infty]$ .
Remark 3.1 (UOT as a Generalization of OT). Suppose  $\Psi_{1}$  and  $\Psi_{2}$  are the convex indicator function of  $\{1\}$ , i.e. have zero values for  $x = 1$  and otherwise  $\infty$ . Then, the objective in Eq 4 becomes infinity if  $\pi_0 \neq \mu$  or  $\pi_1 \neq \nu$ , which means that UOT reduces into classical OT. Moreover,  $\Psi_{1}^{*}(x) = \Psi_{2}^{*}(x) = x$ . If we replace  $\Psi_{1}^{*}$  and  $\Psi_{2}^{*}$  with the identity function, Eq 5 precisely recovers the dual form of OT (Eq 2). Note that the hard constraint corresponds to  $\Psi^{*}(x) = x$ .
Now, we introduce the semi-dual formulation of UOT [72]. For this formulation, we assume that  $\Psi_1^*$  and  $\Psi_2^*$  are non-decreasing and differentiable functions ( $f^*$  is non-decreasing for the non-negative entropy function  $f$  [65]):
$$
C _ {u b} (\mu , \nu) = \sup  _ {v \in \mathcal {C}} \left[ \int_ {\mathcal {X}} - \Psi_ {1} ^ {*} (- v ^ {c} (x))) d \mu (x) + \int_ {\mathcal {Y}} - \Psi_ {2} ^ {*} (- v (y)) d \nu (y) \right], \tag {6}
$$
where  $v^{c}(x)$  denotes the  $c$ -transform of  $v$  as in Eq 3.
Remark 3.2 ( $\Psi^{*}$  Candidate). Here, we clarify the feasible set of  $\Psi^{*}$  in the semi-dual form of UOT. As aforementioned, the assumption for deriving Eq 6 is that  $\Psi^{*}$  is a non-decreasing, differentiable function. Recall that for any function  $f$ , its convex conjugate  $f^{*}$  is convex and lower semi-continuous [9]. Also, for any convex and lower semi-continuous  $f$ ,  $f$  is a convex conjugate of  $f^{*}$  [7]. Combining these two results, any non-decreasing, convex, and differentiable function can be a candidate of  $\Psi^{*}$ .
3.2 Generative Modeling with the Semi-dual Form of UOT
In this section, we describe how we implement a generative model based on the semi-dual form (Eq 6) of UOT. Following [18, 28, 41, 54, 61], we introduce  $T_{v}$  to approximate  $v^{c}$  as follows:
$$
T _ {v} (x) \in \underset {y \in \mathcal {Y}} {\operatorname {a r g i n f}} [ c (x, y) - v (y) ] \quad \Leftrightarrow \quad v ^ {c} (x) = c \left(x, T _ {v} (x)\right) - v \left(T _ {v} (x)\right), \tag {7}
$$
Note that  $T_{v}$  is measurable ([10], Prop 7.33). Then, the following objective  $J(v)$  can be derived from the equation inside supremum in Eq (6) and the right-hand side of Eq 7:
$$
J (v) := \int_ {\mathcal {X}} \Psi_ {1} ^ {*} \left( \right.- \left[ \right.\left( \right.c (x, T _ {v} (x)) - v \left(T _ {v} (x)\right)\left. \right]\left. \right) d \mu (x) + \int_ {\mathcal {Y}} \Psi_ {2} ^ {*} (- v (y)) d \nu (y). \tag {8}
$$
In practice, there is no closed-form expression of the optimal  $T_{v}$  for each  $v$ . Hence, the optimization  $T_{v}$  for each  $v$  is required as in the generator-discriminator training of GAN [25]. In this work, we parametrize  $v = v_{\phi}$  and  $T_{v} = T_{\theta}$  with neural networks with parameter  $\phi$  and  $\theta$ . Then, our learning objective  $\mathcal{L}_{v_{\phi},T_{\theta}}$  can be formulated as follows:
$$
\mathcal {L} _ {v _ {\phi}, T _ {\theta}} = \inf  _ {v _ {\phi}} \left[ \int_ {\mathcal {X}} \Psi_ {1} ^ {*} \left(- \inf  _ {T _ {\theta}} [ c (x, T _ {\theta} (x)) - v _ {\phi} (T _ {\theta} (x)) ]\right) d \mu (x) + \int_ {\mathcal {Y}} \Psi_ {2} ^ {*} (- v _ {\phi} (y)) d \nu (y) \right]. \tag {9}
$$
Finally, based on Eq 9, we propose a new training algorithm (Algorithm 1), called the UOT-based generative model (UOTM). Similar to the training procedure of GANs, we alternately update the potential  $v_{\phi}$  (lines 2-4) and generator  $T_{\theta}$  (lines 5-7).
Algorithm 1 Training algorithm of UOTM
Require: The source distribution  $\mu$  and the target distribution  $\nu$ . Non-decreasing, differentiable, convex function pair  $(\Psi_1^*, \Psi_2^*)$ . Generator network  $T_{\theta}$  and the discriminator network  $v_{\phi}$ . Total iteration number  $K$ .
1: for  $k = 0,1,2,\ldots ,K$  do
2: Sample a batch  $X\sim \mu ,Y\sim \nu ,z\sim \mathcal{N}(\mathbf{0},\mathbf{I})$
3:  $\mathcal{L}_v = \frac{1}{|X|}\sum_{x\in X}\Psi_1^* (-c(x,T_\theta (x,z)) + v_\phi (T_\theta (x,z))) + \frac{1}{|Y|}\sum_{y\in Y}\Psi_2^* (-v_\phi (y))$
4: Update  $\phi$  by using the loss  $\mathcal{L}_v$ .
5: Sample a batch  $X\sim \mu ,z\sim \mathcal{N}(\mathbf{0},\mathbf{I})$
6:  $\mathcal{L}_T = \frac{1}{|X|}\sum_{x\in X}\left(c\left(x,T_\theta (x,z)\right) - v_\phi (T_\theta (x,z))\right).$
7: Update  $\theta$  by using the loss  $\mathcal{L}_T$ .
8: end for
Following [26, 42], we add stochasticity in our generator  $T_{\theta}$  by putting an auxiliary variable  $z$  as an additional input. This allows us to obtain the stochastic transport plan  $\pi(\cdot|x)$  for given input  $x \in \mathcal{X}$ , motivated by the Kantorovich relaxation [33]. The role of the auxiliary variable has been extensively discussed in the literature [13, 26, 42, 80], and it has been shown to be useful in generative modeling. Moreover, we incorporate  $R_{1}$  regularization [60],  $\mathcal{L}_{reg} = \lambda \| \nabla_{x} v_{\phi}(y) \|_2^2$  for real data  $y \in \mathcal{Y}$ , into the objective (Eq 9), which is a popular regularization method employed in various studies [13, 51, 80].
3.3 Some Properties of UOTM
Divergence Upper-Bound of Marginals UOT relaxes the hard constraint of marginal distributions in OT into the soft constraint with the Csiszar divergence regularizer. Therefore, a natural question is how much divergence is incurred in the marginal distributions because of this soft constraint in UOT. The following theorem proves that the upper-bound of divergences between the marginals in UOT is linearly proportional to  $\tau$  in the cost function  $c(x,y)$ . This result follows our intuition because down-scaling  $\tau$  is equivalent to the relative up-scaling of divergences  $D_{\Psi_1}, D_{\Psi_2}$  in Eq 4. (Notably, in our experiments, the optimization benefit outweighed the effect of soft constraint (Sec 5.2).)
Theorem 3.3. Suppose that  $\mu$  and  $\nu$  are probability densities defined on  $\mathcal{X}$  and  $\mathcal{Y}$ . Given the assumptions in Appendix A, suppose that  $\mu, \nu$  are absolutely continuous with respect to Lebesgue measure and  $\Psi^*$  is continuously differentiable. Assuming that the optimal potential  $v^{\star} = \inf_{v\in \mathcal{C}}J(v)$  exists,  $v^{\star}$  is a solution of the following objective
$$
\tilde {J} (v) = \int_ {\mathcal {X}} - v ^ {c} (x) d \tilde {\mu} (x) + \int_ {\mathcal {Y}} - v (y) d \tilde {\nu} (y), \tag {10}
$$
where  $\tilde{\mu}(x) = \Psi_1^{*'}(-v^{\star c}(x))\mu(x)$  and  $\tilde{\nu}(y) = \Psi_2^{*'}(-v^{\star}(y))\nu(y)$ . Note that the assumptions guarantee the existence of optimal transport map  $T^{\star}$  between  $\tilde{\mu}$  and  $\tilde{\nu}$ . Furthermore,  $T^{\star}$  satisfies
$$
T ^ {\star} (x) \in \operatorname {a r g i n f} _ {y \in \mathcal {Y}} [ c (x, y) - v ^ {\star} (y) ], \tag {11}
$$
$\mu$ -almost surely. In particular,  $D_{\Psi_1}(\tilde{\mu}|\mu) + D_{\Psi_2}(\tilde{\nu}|\nu) \leq \tau \mathcal{W}_2^2(\mu,\nu)$  where  $\mathcal{W}_2(\mu,\nu)$  is a Wasserstein-2 distance between  $\mu$  and  $\nu$ .
Theorem 3.3 shows that  $T_{v^{\star}}$  (Eq 7) is a valid parametrization of the optimal transport map  $T^{\star}$  that transports  $\tilde{\mu}$  to  $\tilde{\nu}$ . Moreover, if  $\tau$  is sufficiently small, then  $\tilde{\mu}$  and  $\tilde{\nu}$  are close to  $\mu$  and  $\nu$ , respectively. Therefore, we can infer that  $T_{v^{\star}}$  will transport  $\mu$  to a distribution that is similar to  $\nu$ .
Stable Convergence We discuss convergence properties of UOT objective  $J$  and the corresponding OT objective  $\tilde{J}$  for the potential  $v$  through the theoretical findings of Gallouet et al. [21].
Theorem 3.4 ([21]). Under some mild assumptions in Appendix A, the following holds:
$$
J (v) - J \left(v ^ {\star}\right) \geq \frac {1}{2 \lambda} \mathbb {E} _ {\tilde {\mu}} \left[ \| \nabla \left(v ^ {c} - v ^ {\star c}\right) \| _ {2} ^ {2} \right] + C _ {1} \mathbb {E} _ {\mu} \left[ \left(v ^ {c} - v ^ {\star c}\right) ^ {2} \right] + C _ {2} \mathbb {E} _ {\nu} \left[ \left(v - v ^ {\star}\right) ^ {2} \right], \tag {12}
$$
for some positive constant  $C_1$  and  $C_2$ . Furthermore,  $\tilde{J}(v) - \tilde{J}(v^{\star}) \geq \frac{1}{2\lambda} \mathbb{E}_{\tilde{\mu}}\left[\|\nabla(v^{c} - v^{\star c})\|_{2}^{2}\right]$ .
Theorem 3.4 suggests that the UOT objective  $J$  gains stability over the OT objective  $\tilde{J}$  in two aspects. First, while  $\tilde{J}$  only bounds the gradient error  $\| \nabla (v^c - v^{\star c}) \|_2^2$ ,  $J$  also bounds the function error  $(v^c - v^{\star c})^2$ . Second,  $J$  provides control over both the original function  $v$  and its  $c$ -transform  $v^c$  in  $L^2$  sense. We hypothesize this stable convergence property conveys practical benefits in neural network optimization. In particular, UOTM attains better distribution matching despite its soft constraint (Sec 5.2) and faster convergence during training (Sec 5.3).
Optimal Transport Optimal Transport (OT) problem addresses a transport map between two distributions that minimizes a specified cost function. This OT map has been extensively utilized in various applications, such as generative modeling [2, 45, 61], point cloud approximation [50], and domain adaptation [20]. The significant interest in OT literature has resulted in the development of diverse algorithms based on different formulations of OT problem, e.g., primary (Eq 1), dual (Eq 2), and semi-dual forms (Eq 3). First, several works were proposed based on the primary form [47, 81]. These approaches typically involved multiple adversarial regularizers, resulting in a complex and challenging training process. Hence, these methods often exhibited sensitivity to hyperparameters. Second, the relationship between the OT map and the gradient of dual potential led to various dual form based methods. In specific, when the cost function is quadratic, the OT map can be represented as the gradient of the dual potential [77]. This correspondence motivated a new methodology that parameterized the dual potentials to recover the OT map [2, 64]. Seguy et al. [64] introduced the entropic regularization to obtain the optimal dual potentials  $u$  and  $v$ , and extracted the OT map from them via the barycentric projection. Some methods [40, 49] explicitly utilized the convexity of potential by employing input-convex neural networks [1].
Recently, Korotin et al. [39] demonstrated that the semi-dual approaches [18, 61] are the ones that best approximate the OT map among existing methods. These approaches [18, 61] properly recovered OT maps and provided high performance in image generation and image translation tasks for large-scale datasets. Because our UOTM is also based on the semi-dual form, these methods show some connections to our work. If we let  $\Psi_1^*$  and  $\Psi_2^*$  in Eq 9 as identity functions, our Algorithm 1 reduces to the training procedure of Fan et al. [18] (See Remark 3.2). For convenience, we denote Fan et al. [18] as a OT-based generative model (OTM). Moreover, note that Rout et al. [61] can be considered as a minor modification of parametrization from OTM. In this paper, we denote Rout et al. [61] as Optimal Transport Modeling (OTM*), and regard it as one of the main counterparts for comparison.
(a) OTM
(b) UOTM
Figure 2: Outlier Robustness Test on Toy dataset with  $1\%$  outlier. For each subfigure, Left: Comparison of target density  $\nu$  and generated density  $T_{\#}\mu$  and Right: Transport map of the trained model  $(x,T(x))$ . While attempting to fit the outlier distribution, OTM generates undesired samples outside  $\nu$  and learns the non-optimal transport map. In contrast, UTOM mainly generates indistribution samples and achieves the optimal transport map. (For better visualization, the y-scale of the density plot is manually adjusted.)
Figure 3: Outlier Robustness Test on Image dataset (CIFAR-10 + 1% MNIST). Left: OTM exhibits artifacts on both in-distribution and outlier samples. Right: UOTM attains higher-fidelity samples while generating MNIST-like samples more sparingly, around 0.2%. FID scores of CIFAR-10-like samples are 13.82 for OTM and 4.56 for UOTM, proving that UOTM is more robust to outliers.
Unbalanced Optimal Transport The primal problem of UOT (Eq 4) relaxes hard marginal constraints of Kantorovich's problem (Eq 1) through soft entropic penalties [12]. Most of the recent UOT approaches [11, 19, 48, 58] estimate UOT potentials on discrete space by using the dual formulation of the problem. For example, Pham et al. [58] extended the Sinkhorn method to UOT, and Lubeck et al. [48] learned dual potentials through cyclic properties. To generalize UOT into the continuous case, Yang and Uhler [82] suggested the GAN framework for the primal formulation of unbalanced Monge OT. This method employed three neural networks, one for a transport map, another for a discriminator, and the third for a scaling factor. Balaji et al. [8] claimed that implementing a GAN-like procedure using the dual form of UOT is hard to optimize and unstable. Instead, they proposed an alternative dual form of UOT, which resembles the dual form of OT with a Lagrangian regularizer. Nevertheless, Balaji et al. [8] still requires three different neural networks as in [82]. To the best of our knowledge, our work is the first generative model that leverages the semi-dual formulation of UOT. This formulation allows us to develop a simple but novel adversarial training procedure that does not require the challenging optimization of three neural networks.
In this section, we evaluate our model on the various datasets to answer the following questions: