Analyzing and Improving Optimal-Transport-based Adversarial Networks
Authors: Jaemoo Choi
Jaewoong Choi
Myungjoo Kang
Venue: ICLR 2024 poster (2024)
Keywords: Optimal Transport
Generative Adversarial Networks
Primary area: generative models
TL;DR: We analyze and improve Optimal-Transport-based adversarial networks.

Abstract: Optimal Transport (OT) problem aims to find a transport plan that bridges two distributions while minimizing a given cost function. OT theory has been widely utilized in generative modeling. In the beginning, OT distance has been used as a measure for assessing the distance between data and generated distributions. Recently, OT transport map between data and prior distributions has been utilized as a generative model. These OT-based generative models share a similar adversarial training objective. In this paper, we begin by unifying these OT-based adversarial methods within a single framework. Then, we elucidate the role of each component in training dynamics through a comprehensive analysis of this unified framework. Moreover, we suggest a simple but novel method that improves the previously best-performing OT-based model. Intuitively, our approach conducts a gradual refinement of the generated distribution, progressively aligning it with the data distribution. Our approach achieves a FID score of 2.51 on CIFAR-10 and 5.99 on CelebA-HQ-256, outperforming unified OT-based adversarial approaches.

Published as a conference paper at ICLR 2024
ANALYZING AND IMPROVING OPTIMAL-TRANSPONL TRANSPORT- BASED ADVERSARIAL NETWORKS
Jaemoo Choi"
Jaewoong Choi +
Myungjoo Kang Seoul National University mka mkang@snu .r
Seoul National University toony428smu.aco
Korea Institute for Advanced Study chw]1475@K
4750k as. re. kr
ABSTRACT
Optimal Transport (OT) problem aims to find a transport plan that bridges two distributions while minimizing given cost function. OT theory has been widely utilized in generative modeling. In the beginning, OT distance has been used measure for assessing the distance between data and generated distributions. Recently, OT transport map between data and prior distributions has been utilized as generative model. These OT-based generative models share similar adversarial training objective. In this paper, we begin by unifying these OT-based adversarial methods within a single framework. Then, we elucidate the role of each component in training dynamics through a comprehensive analysis of this unified framework. Moreover, we suggest a simple but novel method that improves the previously best best-performing OTT-based model. Intuitively, our approach conducts a gradual refinement the generated distribution, progressively aligning it with the data distribution. Our approach achieves a FID score of 2.51 on CIFAR-10 and 5.99 on CelebA-HQ-256, outperforming unified OT-based adversarial approaches.
INTRODUCTION
Optimal Transport (OT) theory addresses the most cost-efficient way to transport one probability distribution to another (Villani et al. |009)| Peyré et al., 2017). OT theory has been widely exploited
in various machine learning applications, such generative modeling (Arjovsky et al. 2017 Rout et al. 2022), domain adaptation (Guan et al.] 2021 Shen et al., 2018). unpaired image-to image translation (Korotin et al. 2023 Xie al. 2019), point cloud approximation (Mérigot et al. 2021)
and data augmentation (A (Alvarez-Melis & Fusi [2020) Flamary et al.) 2016). In this work, we focus on OT-based generative modeling. During its early stages, WGAN (Arjovsky al. 2017) and its variants (Gulrajani et al. 2017 Petzka et al., 22018] Liu et al. 2019, Miyato et al. 2018) introduced OT theory to define loss functions in GANs (Goodfellow al.) (0200 (OT Loss). More precisely, -T-based Wasserstein distance is introduced for measuring a distance between data and generated distributions. These approaches have shown relative stability and improved performance compared to the vanilla GAN (Gulrajani et al.) 2017). However, these models still face challenges. such as an unstable training process and limited expressivity (Sanjabi et al. 2018 Mescheder et al., 2018).
Recently, an alternative approach has been introduced in OT-based generative modeling. These works consider OT problems between noise prior and data distributions, aiming to learn the transport map between them (An et al. 2020a bj Fan et al., (202) (OT Map). This transport map serves generative model because moves a noise sample into a data sample. this context, two noteworthy methods have been proposed: OTM (Rout et al. 2022) and UOTM (Choi et al. 2023a) Interestingly, these two algorithms present similar adversarial training algorithms as previous OT Loss approaches, like WGAN, but with additional cost function and composition with convex functions (Eq. and X). These models, especially UOTM, demonstrated promising outcomes, particularly in terms of stability in convergence and performance. Nevertheless, despite the success of OT Map approaches, there is lack of understanding about why they achieve such high performance and what their limitations are.
(i) In this paper, we propose a unified framework that integrates previous OT Loss and OT Map approaches. Since both of these approaches utilize GAN-like adversarial training, we collectively
"Equal contribution. Correspondence to: Myungjoo Kang <mkang@snu. ac kr>.

---

Published as a conference paper at ICLR 2024
refer to them OT-based GANs. (ii) Utilizing this framework, we conduct comprehensive analysis of previous OT-based GANs for an in-depth analysis of each constituent factor of OT Map. Our analysis reveals that the cost function mitigates the mode collapse problem, and the incorporation of a strictly convex function into discriminator loss is beneficial for the stability of the algorithm. (iii) Moreover, we propose straightforward but novel method for improving the previous best-performing OT-based GANs, i.c., UOTM. Our method involves a gradual up-weighting of divergence terms in the Unbalanced Optimal Transport problem. In this respect, we refer to our model UOTM with Scheduled Divergence (UOTM-SD). This gradual up-weighting of divergence terms in UOTM-SD leads to the convergence of the optimal transport plan from the UOT problem toward that of the OT problem. Our UOTM-SD outperforms UOTM and significantly improves the sensitivity of UOTM to the cost-intensity hyperparameter. Our contributions can summarized as follows:
We introduce an integrated framework that encompasses previous OT-based GANs.
We present comparative analysis of these OT-based GANs to elucidate the role of each component.
We propose simple and well-motie well-motivated modification to UOTM that improves both generation results and r-robustness of UOTM for the cost function c(z, y) = و|| v1l..
Notations Let X, y be compact Polish spaces and Jı and be probability distributions X' and V, respectively, Assume that these probability spaces satisfy some regularity conditions described in Appendix A We denote the prior (source) distribution as ş and data (target) distributions as v. M+ x Y) represents the set of positive measures on X' x V. For π E M (X x V), we denote the marginals with respect to X' and y as To and T1. [1,, 1/) denote the set of joint probability distributions on X' x Y whose marginals are and respectively. For a measurable map T X - ) Tap denotes the associated pushforward distribution of p. c(x,y) y) refers to the transport cost function defined on X x y. this paper, we assume X, y C I2d and the quadratic cost c(ar, y) = -|| y 1/ where is given positive constant. For a detailed explanation of assumptions, see Appendix
2 BACKGROUND AND RELATED WORKS
In this section, we introduce several OT problems and their equivalent forms. Then, we provide an overview various OT-based GANs, which will be the subject of our analysis.
Kantorovich OT Kantorovich (1948) formulated the OT problem through the cost-minimizing coupling II(,,)) between the source distribution and the target distribution v as follows:
U c(x, u) dr v)
C(
inf rEll(a.))
Under mild assumptions, this Kantorovich problem can be reformulated into several equivalent forms, such as the dual (Eq. [ and semi-dual (Eq. 3 formulation (Villani al., 2009):
[S u(x) u(x) dp(x) + S u(v) du(u)
C(u,D)=
sup (())+u(()))(a.n))
(2)
I (x) + / v(y) du(s) we
sup
where and v E (v), and the c-transform (2) := inf (c(2. y) u(y)). For a particular case where the and u is 1-Lipschitz cost c(..)) -) is a distance function, i.e.. the Wasserstein-1 distance, then duality.
Lebesgue integrable with respect measure μ and v, i.e., (²) and
(Villani et al. 2009). In such case, we call Eq. 2 Kamtorovich-Rubiintee
OT Loss in GANs WGAN (Arjovsky et al. 2017) introduced the Wasserstein- distance define a loss function in GAN. This Wasserstein distance serves as distance measure between generated distribution and data distribution. From Santoroich-Rubbinteinn duality, the optimization problem for WGAN is given as follows:
Lug.To sup inf [-/ up (To(x)) du(x)) + / (v) du(w)
(4)
2

---

Published as a conference paper at ICLR 2024
where the potential (critic) Us is 1-Lipschitz, i.e., | I 1, and T3 is generator. WGAN-GP (Gulrajani et al. 2017) suggested a gradient penalty regularizer to enhance the stability of WGAN training. For optimal coupling 'T,, the optimal potential vg satisfies (
-r''-"aa- -almost surely,
x². WGAN-GP exploits this optimality condition by introducing R(x, y) (1l V 0 (1)ll2 - 1)² as the penalty term.
where y (x) + (1 t)y for some 0 S f < 1 with (Ta()),
OT Map as Generative model on directly modeling the optimal transport map between the input prior distribution and the real data distribution (Rout et al., [022]] An et al. 2020a b) Makkuva et al. 2020 Yang & Uhler 2019
Parallel to OT Loss approaches, there has been surge of research
Choi et al. (023a). In this case, the optimal transport map serves as the generator itself. In particular, Rout et al. 2022) and Fan et al. (2022) leverage the semi-dual formulation (Eq. 3 of
the Kantorovich problem for generative modeling. Specifically, these models parametrize =
og in Eq. [3] and represent its c-transform ef through the transport mar To X y,
arg infyey [c(2,)) y) - v (v)]. Then, we obtain the following optimization problem:
U inf je (z, (x())) - us (To(z))) du(x)) + I, vo(u) dr(y) volu)dr(v))
Lu = sup Vig To
(5)
Intuitively, To and Ui serve as the generator and the discriminator a GAN. For convenience, we denote the optimization problem of Eq. as OT-based generative model (OTM) (Fan et al. 2022). Note that, if set c = 0 and introduce a 1 I-Lipschitz constraint on Ug. this objective has the same form as WGAN (Eq. 4 In OT map models, the quadratic cost is usually employed.
Recently, Choi et al. 2023a) extended OTM by leveraging the semi-dual form of the Unbalanced Optimal Transport (UOT) problem (Liero et al., 2018). UOT extends the OT problem by relaxing the strict marginal constraints using the Csiszár divergences Dw, (See the Appendix [A precise
definition). Formally, the UOT problem (Eq. (6) and its semi-dual form (Eq. 7 are defined follows:
Ux c(x, 4) dr 4) + Dv, (TolA) + Di (TI(M))
inf Cub(u,v)= = (xM+(
(6)
U/x eEC())
- -ii (-v du(x) -vi(-u(u)) du(u)
sup
where C()) denotes a set continuous functions over Here, the entropy function V, IR 00 co] is convex, lower semi-continuous. and non-negative function, and (x(x) = oc for x < 0. V
non-decreasinig convex function. For simplicity, we assume V (0) 0, (VT) (0) = 1. The reason for this assumption will be clarified in Sec
denotes its convex conjugate. Note that for non-negative U,, V
4 By using the same parametrization as in Eq. [5) we arrive at the following:
J vi
(-inff inf (c To()) us (To(x))) du(x)) + / Vi (- dv(v)
LUC,T iin "
(8)
We call such an optimization problem a UOT-based generative model (UOTM) (Choi et al. 2023a).
ANALYZING OT-BASED ADVERSARIAL APPROACHES
In this section, we suggest a unified framework for OT-based GANs (Sec 3.1). Using this unified framework, we compare the dynamics of each algorithm through various experimental results (Sec 3.2p. This comparative ablation study delves into the impact of employing strictly convex 91, 92 within discriminator loss, and the influence of cost function c(2,, y) = T| vilă. Furthermore, we
present an additional explanation for the success of UOTM (Sec 3.2.3
3.1 A UNIFIED FRAMEWORK
We present an integrated framework, Algorithm I that includes various OT-based adversarial net- works. These models are derived by directly parameterizing the potential and generator, utilizing the
Note that this parametrization does not precisely characterize the optimal transport map (Rout et al. 2022 The optimal transport map satisfies this relationship, but not all functions satisfying this condition transport maps. However, investigating hetter parametrization of the optimal transport heyond the scope of this work.
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Unified training algorithm
Require: Functions 91, 92- 93- Generator network Ta and the discriminator network Eg.- The number of iterations per network Kú, Kr. Total iteration number K. Regularizer R. with regularization hyperparameter X.
1: for k == 0,,,2........ K do
2:
for k: 1 to K2 do
3:
Sample a batch ~
4:
ū= TT
5:
C. X L (-c () (0)) + 1 Eyer 92(-u6 + (R 0)
6:
Update by using the loss fev.
7:
end for
8:
for k == t Kr do
9:
Sample a batch ~ u, N(0,1.
10:
LT= |x] Evex (s((c(a, To(x, 2)) v (To 2))))
Update by using the loss Cr.
12: end for
end for
dual or semi-dual formulations of OT UOT problems. Specifically, Algorithm I can represent the following models, depending on the choice of the cost function c(2, 3). the convex functions 91, 92, 93. and the regularization term R. (Note that 91: 92 correspond V in Eq
denote two convex functions, Identity and Softplus, as Id(x) = x and SP(x) = log + e") 2 log 2 Also, the Gaussian noise represents the auxiliary variable and is different from the input prior noise x ~ μ.. This auxiliary variable = introduced to represent the stochastic transport map To in the OT map models, such UOTM.
(8) Here,
WGAN (Arjovsky et al. 2017 if = and ni = 92 = 03 = Id
WGAN-GP (Gulrajani al. 2017) if 9I 92 = -a Id, and R gradient penalty.
OTM (Rout et al. 2022) if 7 > and
92 = = ld.
UOTM (Choi et al. 2023a if 7 > g1 92 SP and ga ld.
UOTM w/o cost if 7 = 0, 91 = 92 = SP and 93 = Id.
In this work, we conduct a comprehensive comparative analy- sis of OT-based GANs: WGAN, OTM, UOTM w/o cost, and UOTM (Table This comparative analysis serves as an abla- tion study of two building blocks of OT-based GANs. Thus, we focus on investigating the influence of cost c(.,,) and g 92.
Table 1: Unified Framework for OT-based GANs (ga = Id).
0L 92 ld
@t Su SP
WGAN
UOTM wlo
OTM
UOTM
3.2 COMPARATIVE ANALYSIS OF OT-BASED GANs
In this section, we present qualitative and quantitative generation results of OT-based GANs on both toy and CIFAR-10 (Krizhessky et al., 2009) datasets. We particularly discuss how the algorithms differ with respect to functions 91 & and the cost c(.--). This analysis is conducted in terms of the well-known challenges associated with adversarial training procedures, namely, Unstable training and Mode collapse/mixture. (See Appendix C for the introduction of these challenges.) Moreover, we provide an in-depth analysis of the underlying reasons behind these observed phenomena.
Experimental Settings For visual analysis of the training dynamics, we evaluated these models on 2D multivariate Gaussian distribution, where the source distribution is standard Gaussian. The network architecture fixed for fair comparison. Note that we impose R₁ regularization (Roth) et al. 2017) to WGAN and OTM because they diverge without any regularizations. Moreover, to investigate the scalability of the algorithms, we assessed these models on CIFAR-10 with various network architectures and hyperparameters. See Appendix B for detailed experiment settings.
"The softplus function is scaled and translated to satisfy SP(0) = and SP' (0)
=
For the vanilla WGAN, employed weight clipping strategy following Arjovsky et al. 2017
4

---

Published as a conference paper at ICLR 2024
200000
50009
WGAN
100999
WGAN-GP
UOTM
Volu
w/o cost
Value
OTM
UOTM
Denation
Figure 1: Comparison of Training Dynamics between OT-b GANs. Left: Visualization of generated samples (blue) and data samples (red) for every 6K iterations. Right: Training loss of the generator (To Loss) and discriminator (Ug Loss) for each algorithm.
Table Quantitative Evaluation of OT- based GANs on CIFAR-10.
50
25
CIM
30
UOTH
Model
PID (J)
Precision (1)
Recall (f)
20
10
WGAN
48.8
0.45
0.02
10
WGAN-GP
OIM
4.3
0.71
0.49
10
20
DOTM wio cost
19.7
0.80
0.13
UOTM (SP)
Figure 2: Ablation Study Figure 3: Ablation Study
LOTM (KL.
2.9
on Regularizer Intensity X. Cost Intensity
3.2.1 EFFECT OF STRICTLY CONVEX 9I AND 02
Experimental Results Fig illustrates how each model evolves during training for each 6K iterations. To investigate the effect of and g2, we compare the models with 92 Id (WGAN, OTM) and 91 92 Sp (UOTM, UOTM w/o cost). When ø1 922 1d, WGAN and OTM initially appear to converge in the early stages of training. However, as training progresses, the loss highly fluctuates, leading divergent results. Interestingly, adding a gradient penalty regularizer WGAN (WGAN-GP) is helpful in addressing this loss fluctuation. Conversely, when 91 = 92 Sp. UOTM and UOTM w/o cost consistently perform well, with the loss steadily converging during training. From these observations, we interpret that setting 91, f22 to Sp functions, which are strietly convex, contribute to the stable convergence of OT-based GANs.
Moreover, presents CIFAR-10 generation results of OT-based GANs with NCSN+ Song et al.
2021b) backbone architecture (See the Appendix D.2 for DCGAN (Radford et al. 2015) backbone results). Here, additionally compared UOTM (KL) following Choi et al. (2023a) UOTM (KL) serves as another example of strictly convex 91, 92. where 91 = 92 = -
As in the toy dataset, UOTM w/o cost and UOTM achieve better FID scores than their algorithmic counterparts with respect to 91,22, i.e., WGAN and OTM, respectively. The precision and recall metric (Kynkäänniemi et al.) 2019) results will be examined regarding the cost function in Sec The additional stability of UOTM can be observed in an ablation study on the regularizer intensity X (Fig UOTM model provides more robust FID results compared OTM.
'Here, UOTM (SP) outperformed the original UOTM (KI.). Since UOTM defines 91, 92 any non- decreasing and convex functions, adopt UOTM (SP) as the default UOTM model throughout this paper.
5

---

Published as a conference paper at ICLR 2024
(a) Small
(b) Optimal т
(c) Large
Figure 4: Qualitative Comparison of Generated Samples from UOTM. Left: When 7 is too small (T = 0.0002). Middle: When 7 is optimal (r = 0.001). Right: When r is too large (T = 0.005). On Left, we reordered randomly generated samples to gather similar-looking samples. When t is large, the samples appear noisy, and when r is small, the generated samples show a mode collapse problem.
Effect of 9i and y/2 in Optimization We observed that the introduction of strictly convex functions, such as SP or g(x) = ex - 1, into 92 contributes to more stable training of OT-basee GANs. We explain this enhanced stability in terms of the adaptive optimization the potential network ug-. From the potential loss function C, in line 5 of Algorithm I we can express the gradient descent update for the potential function Us with a learning rate as follows:
-Voo. =
Σ d(((())) V (OU =1)(.)
2 Y
Σ DEY =:w(y)
L/ (-vo(1)) V Vous(s),
where ((x) = c(x, 9) (0) Note that the generator loss CT = [X] ExEx l(z), since we assume
93 = Id. Here, w and v in Eq. serve as sample-wise weights for the potential gradient VoUg.
We interpret the role of 91,92 92 as mediating the balance between T and Suppose the generator dominates the potential for certain x. i.e., i(x:) is small. In this case, because gi is strictly increasing function, the weight (xx) becomes large for this sample z, counterbalancing the dominant generator. Similarly, consider the weight of the true data sample w(yy) Note that the goal of potential is to assign a high value to real data y and low value to generated samples ÿ. Assume that the potential is not good at discriminating certain y, which means that v(y) is small. Then, the weight w(y) becomes large for this sample y as above. We hypothesize that this failure-aware adaptive optimization of the potential Us stabilizes the training procedure, regardless of the regularizer.
3.2.2 EFFECT OF COST FUNCTION
Experimental Results To examine the effect of the cost function c(z,, y) = -| y/ we compare the models with 0 (WGAN, UOTM w/o cost) and r > 0 (OTM, UOTM) in Fig When 7 = 0. both WGAN and UOTM w/o cost exhibit a mode collapse problem. These models fail to fit all modes of the data distribution. On the other hand, WGAN-GP shows mode mixture problem. WGAN-GP generates inaccurate samples that lie between the modes data distribution. In contrast, when 7 > 0, both OTM and UOTM avoid model collapse and mixture problems. In the initial stages of training, OTM succeeds in capturing all modes of data distribution, until training instability occurs due to loss fluctuation. UOTM achieves the best distribution fitting by exploiting the stability of 91, 92 well. Moreover, Table Sable@proo provides a quantitative assessment of the mode collapse problem on CIFAR- 10. The results are consistent with our analysis on the Toy datasets (Fig D.. The recall metric assesses the mode coverage for each model. In this regard, the introduction of the cost function improves the recall metric for each model: from WGAN (0.02) OTM (0.49) and from UOTM w/o cost (0. 13) to UOTM (0.62). The precision metric evaluates the faithfulness of generated images for each model. UOTM w/o cost achieves the best precision score, but the recall metric significantly lower than UOTM. This result shows that UOTM w/o cost exhibited the mode collapse problem. From these results, we interpret that the cost function c(x, y) plays a crucial role in preventing mode collapse by guiding the generator towards cost-minimizing pairs.
Furthermore, we analyze the influence of the cost function intensity T by performing an ablation study on on CIFAR-10 (Fig 3). Interestingly, the results are quite different between OTM and UOTM. When we compare the best-performing т, UOTM achieves much better FID scores than OTM (T 10 x 10 10-4). However, when 7 is excessively small large, the performance of UOTM deteriorates severely. On the contrary, OTM maintains relatively stable results across a wide range of The deterioration of UOTM be understood intuitively by examining the generated results in

---

Published as a conference paper at ICLR 2024
(a) WGAN
(b) UOTM w/o cost
(c) OTM
(d) UOTM
Figure 5: Visualization of Generator Tg. The gray lines illustrate the generated pairs, i.e., the connecting lines between (green) and (x) (blue). The red dots represent the training data samples.
43200
30000
25200
100
20000
18000
0.65
5000
0.00
20
20 25 30
2D 25 25
30
1D
15 20
30
30
20
25
35
(a) WGAN
(b) UOTM w/o cost
OTM
(d) UOTM
Figure 6: Distribution of the absolute value of Average Rate of Change (ARC) of potential Me (v) - Ua (a]] for every 5K iterations. Due to the equi-Lipschitz property, |ARC] of UOTM potential
M-2
stable during training. This stability contributes to the stable training of UOTM.
4ig When r is too large, UOTM tends to produce noise-like samples because the cost function dominates the other divergence terms Dv, within the UOT objective (Eq. 6). When 7 is too small, UOTM shows a mode collapse problem because the negligible cost function Tails to prevent the mode collapse. Conversely, as inferred from OT objective (Eq, D. the optimal pair (v* (T) of OTM remains consistent regardless of variations in т. Hence, OTM presents relatively consistent performance across changes in r 3). In Sec we propose method that enhances the r-robustness of UOTM while also improving its best-case performance.
Effect of Cost in Mode Collapse/Mixture In Fig the OT-based GANs with the cost term (OTM, UOTM) exhibited significantly lower occurrence of mode collapse/mixture, compared to the models without the cost term. This observation proves that the cost function plays regularization role in OT-based GANs, helping to cover all modes within the data distribution. This cost function encourages the generator T to minimize the quadratic error between input x and output T(x). In other words, the generator T is indirectly guided to transport each input x to a point, that is within the data distribution support and close to x. Fig 5 visualizes the transported pair (v. 7 (xx)) by the gray line that connects and T(x). Fig demonstrates that this cost term induces OTM and UOTM to spread the generated samples (in an optimal way). (See Appendix [.I for a comprehensive comparison of generators, including WGAN-GP, UOTM-SD, and GT transport map.)
3.2.3 ADDITIONAL ADVANTAGE OF UOTM
Lipshitz Continuity of UOTM Potential We offer an additional explanation for the stable con- vergence observed in UOTM. In OT-based GANs, we approximate the generator and potential with neural networks. However, since neural networks can only represent continuous functions, is crucial to verify the regularity of these target functions, such as Lipschitz continuity. If these target functions are not continuous, the neural network approximation may exhibit highly irregular behavior. Theorem 3.1 proves that under minor assumptions 9I and g2 in UOTM (Choi et al. 2023a). there exists unique optimal potential v² and satisfies Lipschitz continuity. (See Appendix A for the proof.)
Theorem 3.1. Let and ga be real-valued functions that are non-decreasing, bounded below, differentiable, and strictly convex. Assuming the regularity assumptions in Appendix there exists unique Lipschitz continuous optimal potential v² for Eq. Moreover, for the semi-dual maximization objective -Ly (Eq. 7.
r T:= ( C. S 0, 00 =])
(10)
is equi-bounded and equi-Lipschitz.
7

---

Published as a conference paper at ICLR 2024
Table Image Generation on CIEAR-10. it indicates Table 4: Image Generation on CelebA-HQ.
the results conducted by ourselves.
Class
Model
FID (U)
Class
Model
FID (4)
OTM
13,56
UOTM (KL)
6.36
SNGAN+DGflow Ausari 2020
9.62
OT-haase
UOTM (SP)
6.31
GAN
StyleGAN2 w/o ADA
2020
8.32
StyleGAN2 w/ ADA Karras et al.
2020
2.92
UOTM-SD
5,99
DDGAN (Xiao et al
2021
3.75
RGM Choi et al.
20236
2.47
DDPM (Ho et al. 2020
3.21
OTM
Score SDE (VE)
Song cl al. 202
2.20
20
UOTM-SD (cosine)
UOTM
Diffusion
Score SDE (VP) (ong al.
1b
2.41
DDIM (50 steps) Song et
2121
4.67
15
UOTM SD (stop)
CLD (Dockhorn
2.25
LSGM Vahdat et
2021
2,10
WGAN Arjovsky al.. 2017
55.20
Gulrajan al 2017
39.40
OTM
Rouu et al
2022
4.15
5
OTT-based
COTM
(Chor et al.
20233
2.97
UOTM-SI (Cosine)
2.57
5 10
20
Txx10-)
50
UOTM-SD (Linear)?
2,51
Table 5: Comparison of T-obuustness.
UOTM-SD (Step))
2.78
Note that the semi-dual objective C. can be derived by assuming the optimality of Ta for given v, ie.. (x) E arg inf ynf [c(æ, y) v (v)]. Also, Theorem 3.1 shows that the set of v potential candidates is equi-Lipschitz, i.e., there exists a Lipshitz constant Lr that all v E I are L Lipsschhi This equi-Lipschitz continuity also explains the stable training of UOTM over OTM. The condition L, < in I is not a tough condition for the neural network Uo to satisfy during training, since L. = when u = 0 Therefore, during training, the potential network Us would remain within the domain of L-p-iissitz functions. In other words, 1. would not express any drastic changes for all input y. Furthermore, the target of training, v", also stays within this set of functions. Hence, we can expect stable convergence of the potential network as training progresses. Note that Theorem 3.] is fundamentally different from the I-Lipschitz constraint of WGAN. WGAN involves constrained optimization over a 1 I-Lipschitz potential. In contrast, Theorem 3.1 states that, under unconstrained optimization, the potential networks Us with only minor conditions satisfy equi-Lipschirznes..
Experimental Validation We tested whether this equi-Lipschitz continuity of UOTM potential Uo
observed during training in practice. In particular, we randomly choose data E y and 2D experiment and visualize the Average Rate of Change (ARC) of potential In ( Fig
on
[6 [b-a]
shows boxplots of ARC of ten thousand pairs of (a, b) for every 10K iterations. As shown in Fig 6
only UOTM shows a bounded ARC, and others, especially WGAN and OT', diverge as the training
progresses. This result indirectly shows the potential network in UOTM mostly remains within the equi-Lipschip set during training. Furthermore, we can conjecture that the highly irregular behavior
of potential networks in other models could potentially disrupt stable training processes.
4 TOWARDS THE STABLE OT MAP
In this section, we suggest a straightforward yet novel method to enhance the 7-robustness of UOTM, while improving the best-case performance. Intuitively, our idea is gradually adjust the transport map in the UOT problem towards the transport map in the OT problem. Note that the OT problem of OTM assumes hard constraint on marginal matching.
Motivation
The analysis in Sec 3 showed that the semi-dual form of the UOT problem, i.e., UOTM, provides several advantages over other OT-based GANs. However, Fig 4 showed that UOTM is
'The optimal potential satisfies the c-concavity condition S v. For the quadratic cost, this is equivalent to the condition that + Elyl2 - o(y) is convex and lower semi-continuous (Santambrogio 2015).
'In practice, the potential loss C. is always L. < after only 100 iterations.
8

---

Published as a conference paper at ICLR 2024
7-sensitive. In this respect, Choi et al. (2023a) proved that the upper bound of marginal discrepancies for the optimal It in the UOT problem (Eq. 6 is linearly proportional to T:
Dy, (TōlP) + Dv. (TilP) I (W) for c(z, y) = yll..
(11)
When the divergence term is minor (Large 7), the cost term prevents the mode collapse problem (Sec 3.2.2). but the model fails to match the target distribution, generating noisy samples (Eq. [ Conversely, when the divergence term is dominant (Small T), the model should theoretically exhibit improved target distribution matching (Eq. [I] Theorem 4. However, the mode collapse problem disturbs the optimization process in practice (Sec(3.2.2). In this regard, we introduce a method that can leverage the advantages of both regimes: preventing mode collapse with minor divergence and improving distribution matching with dominant divergence. Intuitively, we start training with smaller divergence term to mitigate mode collapse. Subsequently, as training progresses. we gradually increase the influence of the divergence term to achieve better data distribution matching.
Method Formally, we consider the following a-scaled UOT problem (a-UOT) Coo (H. v) for 2 (Eq. D.4.. Note that this a-UOT problem recovers the OT problem C(u, v) when + xi if μr, have equal mass (Fatras et al. 2021).
U cxx, v) dr (z, + aDv, (1 (Tā aDv, (Tile))
inf Ci TMEM (2xx)
(12)
Motivated by this fact, we suggest a monotone-increasing scheduling scheme during training for a to achieve the stable convergence of the UOT transport map x" towards the OT transport map. Because aDi = Dovi and (a'i)) "
(I) = (2/a), the learning objective of a-scaled UOTM are given as
follows:
inf 0o [/ avi
(- inf [olx To(x)) - (To(o))) du(x) + a To
J aV3 (-2
C CHO.T
(13) Note that, given our assumption that , C², (a uniformly converges to Id for every compact domain, since (0) = 0, (() = 1. Therefore, this a-scheduling can be intuitively understood as a gradual process of straightening the strictly convex V; function towards the identity function Id, so that CT converges to OTM (Tab Db. We refer to this UOTM with a-scheduling UOTM with Scheduled Divergence (UOTM-SD).
Convergence Theorem 4.1 proves that the optimal transport plan of the a-scaled UOT problem converges that of the OT problem as a + 0c. However, one limitation of this theorem is that shows the convergence of transport plan n, but does not address the convergence of transport map T.
Theorem 4.1. Assume the entropy functions 1 22 are strictly convex and finite on (0, oo). Then, the optimal transport plan T of the a-scaled UOT problem Cia (1, v) (Eq. () weakly converges to the optimal transport plan n² of the OT problem C(p, v)) (Eq. 1 as a goes to infinity.
a-schedule Settings
We evaluated three scheduling schemes for a. For the schedule parameters Omma 2 Omin > 0, the assessed scheduling schemes are as follows:
* Cosine Apply Cosine scheduling from Ormnn to amazz
.
Linear: Apply Linear scheduling from Ormin to O'maz-
Step : At each tetr iterations, multiply by 2 until Oman-
Note that the standard cosine scheduling technique (Loshchilov & Hutter 2017 typically works by decreasing the target parameters. In this case, we multiplied the scheduling term by (-1).
Generation Results We tested our UOTM-SI model on CIFAR-10 (32 x 32) and CelebA-HQ (256 256) datasets. For quantitative evaluation, we adopted FID (Heusel et al.) 2017) score. Tab Tab shows that our UOTM-SD improves UOTM (Choi et al., 2023a) across all three scheduling schemes. Our UOTM-SD achieves a FID of 2.51 under the best setting of linear scheduling with (amin 0mmaz)=== = (1/5,5) and T = 1 x 10 -3, surpassing all other OT-based methods. (See Appendix Dor the qualitative comparison of generated samples.) We tested UOTM-SD with linear scheduling, which performed best on CIFAR-10, on CelebA-HQ. Our UOTM-SD outperformed the previous best-performing OT-baaed model (UOTM) (Tab 4). We added a more extensive comparison with

---

Published as a conference paper at ICLR 2024
other generative models in Appendix D.2 (Due to page constraints, included the ablation study regarding schedule intensity, i.e., (amin Aman), and the schedule itself, i.c., amann amazs in Appendix D.4
Robustness We assessed the robustness of our model regarding the intensity parameter T of the cost function c(x, y). Specifically, we tested whether our UOTM-SD resolves the r-sensitivity of UOTM, observed in 3 displass FID scores of UOTM-SD, UOTM, and OTM for various values of 7. Note that we employed harsh conditions for r-robustness, where Tmox/ / Tmmn 25. We adopted Otmin 1/5 and O'mar: = 5 for each UOTM-SD. All three versions of UOTM-SD outperform UOTM and OTM under the same т. While UOTM shows large variation of FIID scores depending 7, ranging from 2.71 to 218.02, UOTM-SD provides much more stable results, (See Appendix D.2 for table results.)
CONCLUSION
In this paper, we integrated and analyzed various OT-based GANs. Our analysis unveiled that establishing Si and 92 as lower-bounded, non-decreasing, and strictly convex functions significantly enhances training stability. Moreover, the cost function contributes to alleviating mode collapse and mixture problems. Nevertheless, UOTM, which leverages these two factors, exhibits r-sensitivity, In this regard. we suggested novel approach that addresses this r-sensitivity UOTM while achieving improved best-case results. However, there are some limitations during our analysis. Also, our convergence theorem for a-scaled UOT guarantees the convergence of the transport plan, but not the transport map. Exploring these issues would be promising future research.
work. Firstly, we fixed Is Id
ACKNOWLEDGEMENTS
This work was supported by KIAS Individual Grant [AP087501] via the Center for AI and Natural Sciences at Korea Institute for Advanced Study, the NRF NTPPNN0.0021-0001343 Artificial Intelligence Graduate School Program(SNU)J.
and MSIT/
REPRODUCIBILITY
To ensure the reproducibility of this work, we submitted the source code the supplementary materials. The implementation details of all experiments clarified in Appendix B Moreover, the assumptions and complete proofs for Theorem 3.1 and 4.1 are included in Appendix
REFERENCES
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428-211 214399, 2020.
Dongsheng An, Yang Guo, Na Lei, Zhongxuan Luo, Shing-Tunn Yau, and Xianfeng Gu. Ae-ot: A new generative model based on extended semi-discrete optimal transport. ICLR, 2020a.
Dongsheng An, Yang Guo, Min Zhang, Xin Qi, Na Lei, and Xianfang Gu. Ae-ot-gan: Training gans from data specific latent distribution. In Computer Ciron-CCVV-A 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XXVI 16, pp. 548-564. Springer, 2020b.
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480-493, 2021.
Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via discrimi- nator gradient flow. arXiv preprint arXiv:20/2.00780, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks.
In International conference machine learning, pp. 213-223. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2024
Yogesh Balaji. Rama Chellappa, and Soheil Feizi. Robust optimal transport with applications in generative modeling and domain adaptation. Advances Neural Information Processing Systems, 33:12934-12944, 2020.
Jaemoo Choi, Jaewoong Choi, and Myungjoo Kang. Generative modeling through the semi-dual formulation of unbalanced optimal transport. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
Jaemoo Choi, Yesom Park, and Myungjoo Kang. Restoration based generative models. In Proceedings of the 40th International Conference on Machine Learning, volume 202. PMLR, 2023b.
Imre Csiszár. A class of measures of informativity of observation channels. Periodica Mathematica Hungarica, 2(1-4):191-23, 1972.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. The International Conference Learning Reprexentations, 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Jiaojiao Fan, Shu Liu, Shaojun Ma, Yongxin Chen, and Hao-Min Zhou. Scalable computation of monge maps with general costs. In ICLR Workshop Deep Generative Models for Highly Structured Data, 2022.
Kilian Fatras, Thibault Séjourné, Rémi Flamary, and Nicolas Courty, Unbalanced minibatch optimal transport; applications domain adaptation. In International Conference Machine Learning. pp. 3186-3197. PMLR, 2021.
R Flamary, N Courty, D Tuia, and A Rakotomamonjy. Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell, 2016.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. Advances in neural information processing systems, 2021.
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3224-3234 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):139-144, 2020.
Hao Guan, Li Wang, and Mingxia Liu. Multi-source domain adaptation via optimal transport for brain dementia identification. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1514-1517. IEEF, 2021.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6883 6851, 2020.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make strong gan. arXiv preprint vrXiv:2102.07074 1(3), 2021.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. arXiv preprint wrXiv:2205.01490 2022.

---

Published as a conference paper at ICLR 2024
Leonid Vitalevich Kantorovich. On problem of monge. Uspekhi Mat. Nauk, pp. 225-226, 1948.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104-121 2020.
Mahyar Khayatkhoei, Maneesh K Singh, and Ahmed Elgammal. Disconnected manifold learning for generative adversarial networks. Advances in Neural Information Processing Systems, 31, 2018.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Score matching model for unbounded data score. arXiv preprint arXiv:2106.0552) 2021.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Neural optimal transport. In The Eleventh International Conference on Learning Representations, 2023.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Matthias Liero, Alexander Mielke, and Giuseppe Savaré. Optimal entropy-transport problems and a new fietinger-kaatrovvin distance between positive measures. Inventiones mathematicae, 211(3): 969-1117, 2018.
Huidong Liu, Xianfeng Gu, and Dimitris Samaras. Wasserstein gan with quadratic transport cost. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4832 4841, 2019.
Iyy Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL htt.:ttt://oppeeeeiiie
mett/Eorum2iddsk4888000
Ashok Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason Lee. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning, pp. 6672-6681. PMLR, 2020.
Quentin Mérigot, Filippo Santambrogio, and Clément Sarrazin. Non-asymptotic convergence bounds for wasserstein approximation using point clouds. Advances in Neural Information Processing Systems, 34: 10-00002111111 2021.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference machine learning, pp. 3481-3490. PMLR, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018 8.
Vaishnavh Nagarajan and J Zico Kolter. Gradient descent gan optimization is locally stable. Advances in neural information processing systems, 30, 2017.
Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow. Is generator conditioning causally related to gan performance? In International conference on machine learning, pp. 3849-3858. PMLR, 2018.
Martin J. Osborne and Ariel Rubinstein. Course in Game Theory. The MIT Press, 1994. ISBN 0262150417.
12

---

Published as a conference paper at ICLR 2024
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 823 832, 2021.
Henning Petzka, Asja Fischer, and Denis Lukovnikov. On the regularization of wasserstein gans, In International Conference on Learning Representations, 2018.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Center for Research in Economics and Statistics Working Papers, (2017-86), 2017.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 14104-14113, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1rXiv:/5/1.066 2015.
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. Advances in neural information processing systems, 30, 2017.
Litu Rout, Alexander Korotin, and Evgeny Burnaev. Generative modeling with optimal transport maps. In International Conference on Learning Representations, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 29, 2016.
Antoine Salmona, Valentin De Bortoli, Julie Delon, and Agnès Desolneux. Can push- forw-forward generative models fit multimodal distributions? Advances in Neural Information Processing Systems, 15:10766-17779 2022.
Maziar Sanjabi, Jimmy Ba, Meisam Razaviyayn, and Jason D Lee. On the convergence and robustness of training gans with regularized optimal transport. Advances in Neural Information Processing Systems, 31, 2018.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58663): 2015.
Igal Sason and Sergio Verdú. f-divergence inequalities. IEEE Transactions on Information Theory, 62(11):5973-6066 2016.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. Advances in Neural Information Processing Systems, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. The International Conference Learning Representations, 2021b.
Arash Vahdat and Jan Kautz. Nvac: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33: 1366779979677, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 1111-188111000 1302, 2021.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference machine learning, pp. 1757-1756. PMLR, 2016.
13

---

Published as a conference paper at ICLR 2024
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. arXiv preprint arXiv:2010.00654 2020.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans, arXiv preprint arXiv:21 (...iv:211..07800// 2021.
Yujia Xie, Minshuo Chen, Haoming Jiang, Tuo Zhao, and Hongyuan Zha. scalable and efficient computation of large scale optimal transport. volume 97 of proceedings of machine learning research. Long Beach, California, USA, pp. 09-15, 2019.
KD Yang and C Uhler. Scalable unbalanced optimal transport using generative adversarial networks. In International Conference on Learning Representations, 2019.
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer based gan for high- reshuueool image generation, In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattrn Recognition, pp. 11304-11311 2022.
14

---

Published as a conference paper at ICLR 2024
A PROOFS
Notations and Assumptions Let X' and ye be compact complete metric spaces which are convex subsets of Rdd and μ., be positive Radon measures of the mass 1. For measurable map T A - y, T#/ denotes the associated pushforward distribution of c(z, y) refers to the transport cost function defined on X' x V. We assume X, Y. C R and the quadratic cost c(2, y) = =rl[z y11.. where is given positive constant. Let 1 and 2/ be entropy function, i.e. V; IR + [0, x2] convex, lower-semi continuous, non-negative function such that (1 = O, and V (x) = co for x < O. Let 91 := V and g2 = / be a convex, differentiable, non-decreasing function defined on R. We assume that 91 (0) 92 (0) = 0 and gi (0) = g/ (0) = 1.
Csiszär Divergence Let be an entropy function. The Csiszår divergence induced by (or V-divergence) between y and is defined as follows:
Du (plv) = / v
dg dv
du + v (2o))
(14)
where μ = duv + (=) is Radon-Nikodym decomposition of μ with respect to v.
du
Theorem A.1. Let 91 and 92 be real-valued functions that non-decreasing, bounded below, differentiable, and strictly convex. Assuming the regularity assumptions in Appendix there exists a unique Lipschitz continuous optimal potential v² for Eq. 7 Moreover, for the maximization objective
-L, of Eq. 7
r T= {o E C()) C. S 0, v ==F=U u)
(15)
is equi-bounded and equi-Lipschitz.
Proof. Let
[(v) = 1 (x))) + /,, (0((-(()))) du(y).
(16)
Since C(0) = 0, the infimum of L(v) is non-positive. Thus, I' is nonempty. We would like to prove that the set I' is equi-bounded and equi-Lipschitz i.c., there exists constant L > 0 such that for
(appppt)) every= 2 E TT of (upp(n) and u/ (upp((e) are L-Lipschitz. Let be the lower bound of functions JI and
g2, i.e. A S g1 (2) and S g2 (v) for x E X and y E V, respectively. Furthermore, since and )' are compact, there exists M 0 such that c(x, y) S M for all (I,y) E X x V. Then, since 91 (x) 2 Id(r),
02 [(v) 2 / - inf (c(æ, v) - v()) du(x)+ + / ga(-u(y)) du(y),
(17)
yey
2 - + sup (v(4)) + / 92( ve(-u())) 2 M A
(18)
,Ey
which indicates that v3)) < M - A for all E V. Note that M nor A is dependent on the choice of u. Thus, u E equi-bounded above. Moreover, by using similar logic with respect to vº, we can easily prove that is also equibounded below. Consequently, by symmetricity, v and of are couidbo.
We now prove that there exists a uniform constant L such that for every r, v is Lipschitz continuous with constant L. Since v is bounded and o v, there exists a point æ(y) such that
vyy) c(x(v)) v) - v
(19)
and for every ã E ),
v(ū) S c(a())) Ø) - v (x(x)).
(20)
Subtracting the two previous inequalities gives,
v(ğ) - uyy) s c(x(v), 5) c(x(y),))
(21)
Since e Lipschitz continuous on the compact domain X' x V. there exists a Lipshitz constant L that satisfies (c(x(y), Ø) c(x(v), v)1 < L - v112. Thus,
(o(ū) v(v) Alli vll2.
(22)
15

---

Published as a conference paper at ICLR 2024
To sum up, I is nonempty, equibounded, and equi-Lipschitz. Moreover, L(v) 2A, thus C(T) is Nower-bounder
Now, would like to prove the compactness of T. Take any sequence {tn, JnE C T. Then, since I is nonempty, equibounded, and equi-Lipschitz, we can obtain a uniformly convergent subsequence (Un, heeN - e. by Arzelä-Ascoli theorem. Because vn (g) -1ll11 is concave for each th T from v00 = u (Santambrogio 2015). v is also continuous and v(y) Tlllll is concave. Thus, is Now, prove I, we only need prove L. S Since JeeN + uniformly, it is easy to show that { JeE + of uniformly. Moreover, note that { MeNN is equibounded. By applying the dominated convergence theorem (DCT), we can easily prove that L, S O. Thus, for any sequence of,, there exists subsequence that converges to point of (Bolzano- Weierstrass property), which implies that P is compact. Finally, since L(T) is lower- howeerbounded there exists minimizer v² E T. i.e. L(v*) S ((v) for all E T by compactness of r.
c-concave, i.e. v
Now, we prove the uniqueness of the minimizer. Let Jeo| < K for every E P. There exists such K by the equiboundedness. Now, let Cx denote the collection continuous functions which are bounded by K. Since gi and 92 are strictly convex on [- K, K]. the following dual minimization problem becomes strictly convex:
0 be a real value that |u| S K and
/ (-u(x))) + J 9e((-(()) dv(y).
inf (2) (D)
(23)
Thus, there exists at most one solution. Because there exists a solution (u*o, v*), the unique solution.
Theorem A.2. Assume the entropy functions 1, 22 are strictly convex and finite on (0, oo). Then,
the optimal transport plan n of the o-scaled UOT problem Co ( v) (Eq. ( weakly converges to the optimal transport plan x² of the OT problem C(p, v) (Eq. as a goes to infinity.
the
Proof. Note that the a-scaled UOT problem C (H, v) equivalent setting the cost intensity within the cost function c(x, y) = T|aæ - y12 of the standard UOT problem Cus(H,
U
n a arg inf (XXM)
+ T| غااا dn (x, + aDv, (mflm) aD (Dvs(Til)))
(24)
U
=|1 v1l (x, + Dv, (molM) + De (mile)
= = inf aex,
(25)
a
Choi et al. (023a) proved that, in the standard UOT problem, the marginal discrepancies for the optimal nt are linearly proportional the cost intensity. This relationship can be interpreted as follows for the above ~.*
Dv, (T ai* (p) + Dv, (Ti
(v) I (2, a
(26)
Therefore, as a goes to infinity, the marginal distributions of a.* converge in the Csiszår divergences to the source y and target v distributions:
lim De, (1) lim Des, (mT 0.
(27)
The convergence in Csiszar divergence Dwv for a strictly convex V implies the convergence of measures in Total Variation distance (Sason & Verdu 2016)] Csiszár (172)). Then, this convergence Total Variation distance implies the weak convergence of measures. This can be easily shown as follows: For any continuous and bounded / E Co(d)), we have
f fddp - fdp = fd(mu- - u1) 11 /C////Iaaamm mn).
(28)
|1 I| ullry-
(29)
Therefore, a, and π..."* weakly converges to μ and v. respectively. Choi et al. 2023a) showed that
the optimal n of Eq. 25 becomes the optimal transport plan for the OT problem C( hr m =f*") for
16

---

Published as a conference paper at ICLR 2024
the same cost function c(z, v) = +||x y112 (The optimal transport plan is invariant to the constant
scaling of the cost function). Moreover, since 2, are compact, c(z, y) is bound on X' x y. Thus,
lim inf / c(x, dr dr < <0. 00000
(30)
Consequently, Theorem 5.20 from Villani et al. (2009) proves that x weakly converges to the optimal transport plan n' of the OT problem C( v) as a goes to infinity.
B IMPLEMENTATION DETAILS
For every implementation, the prior (source) distribution is a standard Gaussian distribution with the same dimension as the data (target) distribution.
2D Experiments For m, = 12 (cos 1.,, sin {x) for i 0, l,.... 7 and 0.4, we set mixture of (m,, o²) a target distribution. For all synthetic experiments, discriminator network architectures. The auxiliary variable has a dimension two. For a generator, we passed through two fully connected (FC) layers with hidden dimension of 128, resulting in 128-dimensional embedding. We also embedded data x into the 128-dimensional vector by passing it through three-layered ResidualBlock (Song & Ermon 2019). Then, we summed up the two vectors and fed them to the final output module. The output module consisted of two FC layers. For the discriminator, we used three layers of ResidualBlock and two FC layers (for the output module). The hidden dimension is 1 128. Note that the SiLU activation function is used. We used a batch size of 128, and a learning rate of 2 x 10 and 10 for the generator and discriminator, respectively. We trained for 30K iterations. For OTM and UOTM, we chose the best results between settings of r = 0.01, 0.05. OTM has shown the best performance with T 0.05 and UOTM has shown the best performance with 0.01. For WGANs and OTM. since they do not converge without any regularization, we set the regularization parameter À = We used a gradient clip of 0.1 for WGAN.
used the same generator and
CIFAR-10 For the DCGAN model, we employed the architecture of BBlajj et al. (0020), which uses convolutional layers with residual connection. Note that this is the same model architecture as in Rout et al. (2022): Choi et al. (2023a). We set batch size of 128, 50K iterations, a learning rate of 2 x 10 and 10 for the generator and discriminator, respectively. In the DCGAN backbone, we adopt a simple practical scheme suggested in OTM Rout al. (002)) for accommodating smaller dimension for the input latent space X. This practical scheme involves introducing a deterministic bicubic upsampling Q from X' to y. Then, we consider the OT map between QuIt and v. In practice, we sample in Algorithm from 192-dimensional standard Gaussian distribution. Then, is directly used as an input for the DCGAN generator Tg. The random variable = is not employed the DCGAN implementation. Meanwhile, (z) is obtained by reshaping x into a 3 x 8 x 8 dimensional tensor, and then bicubically upsampling it to match the shape of the image. The generator loss is defined as e (Q(x), (Q(x),To(<)) vs (To(z))-
For the NCSN++ model, we followed the implementation of Choi et al. 2023b) unless otherwise stated. Specifically, we set X y' and use c(x, y) = T|| yll² without introducing upsampling Q. Here, the auxiliary variable = is employed. We sample from 256-dimensional Gaussian distribution and put an additional stochastic input the generator. The input prior sample is fed into the NCSN+ network like UNet input. The auxiliary z passes through embedding layers and is incorporated into the intermediate feature maps of the NCSN++ through an attention module. We trained for 200K for OTM and 120K for other models because OTM converges slower than other models. Moreover, we used R2 regularization of X 0.2 for all methods and architectures. WGANs are known to show better performance with the optimizers without momentum term, thus, we use Adam optimizer with 3 0, for WGANs. Furthermore, since OTM has a similar algorithm to WGAN, we also use Adam optimizer with 31 O. Lastly, following Choi et al. (2023a). we use Adam optimizer with 31 = 0.5 for UOTM. Note that for all experiments, we use 32 = 0.9 for the optimizer. We used gradient clip of 0.1 for WGAN. Furthermore, the implementation of UOTM-SD follows the UOTM hyperparameter unless otherwise stated. We trained UOTM-SD for 200K iterations. For UOTM-SD (Cosine) and (Linear), initiated the scheduling strategy from the
17

---

Published as a conference paper at ICLR 2024
start and finished the scheduling at 150K iterations. For UOTM-SD (Step), halved ā for every 30K iterations until it reaches Omazz*
For the evaluation of image datasets, we used 50,000 generated samples to measure FID (Karras et al. 2018) scores. For every model, we evaluate the FID score for every 10K iterations and report the best score among them.
Evaluation Metric
C PROBLEMS OF GAN-BASED GENERATIVE MODELS
Unstable training Training adversarial networks involves finding a Nash equilibrium (Osboree & Rubinstein 1994) in two-player non-cooperative game, where each player aims to minimize their own objective function. However. discovering Nash equilibrium is an exceedingly challenging task (Salimans et al. [2016] Mescheder et al. 2018). The prevailing approach for adversarial training adopt alternating gradient descent updates for the generator and discriminator. Unfortunately, the gradient descent algorithm often struggles to converge for many GANs (Salimans et al. 2016 Mescheder et al. 2018). Notably, Mescheder et al. (2018) showed that neither WGANs nor WGANs with Gradient Penalty (WGAN-GP)) offer stable convergence.
Mode collapse/mixture Another primary challenge in adversarial training is mode collapse and mixture phenomena. Mode collapse means that generative model fails encompass all modes of the data distribution. Conversely, mode mixture represents that generative model fails to separate two modes of data distribution while attempting to cover all modes. This results in the generation of spurious ambiguous samples. Many tfate-of-the-arr GANs enforce regularization on the spectral norm (Miyato et al. 2018) to mitigate training instability. Odena et al. (2018) showed that enforcing the magnitude of the spectral norm of the networks reduces instability in training. However, recent
works (Nagarajan & Kolter 2017 Khayatkhoei et al. 2018; An et al. 2020a; Salmona et al. 2022) have revealed that such Lipschitz constraints can lead the generator to concentrate solely of the modes or lead to mode mixtures in the generated samples.
D ADDITIONAL RESULTS
D.I ADDITIONAL QUALITATIVE RESULTS ON TOY DATASETS
18

---

Published as a conference paper at ICLR 2024
barget samole
Target sample
gener strd samble
generated sarpple
scurce sample
souice sampl
(a) WGAN
(b) WGAN-GP
Larget gere atod sample Source sample
sumple ganndedd sample
50 rae sample
(c) UOTM w/o cost
(d) OTM
tarqet gener sample Source sample
sample
generated sample
sample
(e) UOTM
(I) UOTM-SD
(g) Convex OT Solver
Figure 7: Visualization of Generator Tg. The gray lines illustrate the generated pairs, ie.. the connecting lines between (green) and Tz(z) (blue). The red dots represent the training data samples.
19

---

Published as a conference paper at ICLR 2024
target semole gene nted sample cource sample
target sample
generaced sarpple
source sample
(a) WGAN
(b) WGAN-GP
vemple generated
sample
generated
sample
tampple
(c) UOTM w/o cost
(d) OTM
target nample generated
sample
generated sample
sample
(e) UOTM
(I UOTM-SD
(g) Convex OT Solver
Figure 8: Visualization of Generator Tg. The gray lines illustrate the generated pairs, ie.. the connecting lines between (green) and Tz(z) (blue). The red dots represent the training data samples.
20

---

Published as a conference paper at ICLR 2024
barget semple generated symple courte sample
target sample generated serpp 50 Sturce cample
(a) WGAN
(b) WGAN-GP
get tample gene ated sample cource sample
target sample generated sample source sample
(c) UOTM w/o cost
(d) OTM
Larget sample
nample generated
(e) UOTM
O UOTM-SD
target sample Source Sample
(g) Convex OT Solver
Figure 9; Visualization of Generator Tg. The gray lines illustrate the generated pairs, i.e., the connecting lines between x (green) and (z) (blue). The red dots represent the training data samples.
21

---

Published as a conference paper at ICLR 2024
D.2 FULL TABLE RESULT FOR CIFAR-10 10 GENERATION
Table 6: Quantitative Evaluation of OT-based GANs on CIFAR-10.
Backbone Architecture
Model
NCSN+
DCGAN
FID (L)
Precision (1)
Recall (t)
FID (L)
WGAN WGAN-GP
48.8
0.45
0,02
52.3
4.5
0.71
0.55
50.8
OТM
4.3
0.71
0.49
19.8
UOTM w/o cost
19.7
0.80
0.13
15.4
UOTM (SP)
2.7
0.78
0.62
15.8
UOTM (KL)
2.9
12.2
Table 7: Comparison of 7-robustness. We use Omin
1/5 and Qmar for each UOTM-SD.
2e-4
5e-4
le-3
2e-3
5e-3
UOTM SD (Cosine)
3.60
2.99
2.57
2.95
5.42
UOTM-SD (Linear)
4.18
3.01
2.51
3.39
4.62
UOTM-SD (Step)
3.92
2.81
2.78
2.89
5.34
UOTM
15.19
22.02
2.71
6.30
218.02
OTM
4.34
4.15
4.38
5.13
7.43
22

---

Published as a conference paper at ICLR 2024
Table 8: Image Generation on CIEAR-10. i indicates the results conducted by ourselves.
Class
Model
FID (L)
SNGAN+DGflow (Anari et al. 2020)
9.62
AutoGAN (Gong et al.
2019
12.4
TransGAN (Jang et al.
2021
9.26
StyleGAN2 w/o ADA Karras et
2020
8.32
GAN
StyleGAN2 w/ ADA Karras et al.
2020
2.92
DDGAN (T=1) Xiao et al. 2021
16.68
DDGAN (Xiao al. (02TT)
3.75
RGM (Choi al. 20236
2.47
NCSN (Song & Ermon 2019)
25.3
DDPM (Ho et al. 2020)
3.21
Score SDE (VE)
(Song et al.
202
2.20
Score SDE (VP) (Song al.
2021b
2.41
Diffusion
DDIM (50 steps) (Song ct al 2021a
4.67
CLD (Dockhorn
2022)
2.25
Subspace Diffusion (Jing et al.
2022)
2.17
LSGM (Vahdat al. 2021
2.10
NVAE (Vahdat & Kautz
20200
23.5
Glow (Kingma & Dhariwal
2018
48.9
VAE&EBM
PixeCCNN Vaa Oord et al.
2016)
65.9
VAEBM Xiao
2020
12.2
Recovery EBM (Gao et al.
2021
9.58
WGAN (Arjovsky et al. 2017)
55.20
WGAN-GP(Gura et al. 2017
39.40
Robust-OT (Balaji al 2020
21.57
OT-based
AE-OT-GAN (An et al. 20206)
17.10
OTи (Rout et al. 2022
4.15
UOTM (Chor et al. 2023a
2.97
UOTM-SI (Cosine)
2.57
UOTM-SD (Linear) UOTM-SD (Step)*
2.51
2.78
D.3 ADDITIONAL QUANTITATIVE RESULTS FOR LIPSCHITZNES OF POTENTIAL
23

---

Published as a conference paper at ICLR 2024
Table 9: Image Generation on CelebA-HQ. it indicates the results conducted by ourselves.
Class
Model
FID (1)
Score SDE (VP) Song et al.
(222
7.23
Probability Flow Song al.
b021b
128.13
LSGM Vahdat et al (2021)
7.22
Diffusion
UDM Kim et al. (2211)
7.16
DDGAN Xiao et al.
(2021
7.64
RGM Choi et al.
20233)
7.15
PGGAN Karras et al. (2017)
8.03
Adv. LAE Pidhorskyi et al.
(2020)
19.2
GAN
VQ-GAN Esser et al.
(221
10.2
DC-AE Parmar
2021
15.8
StyleSwin (Zhagg et al.
2022
3.25
NVAE Vahdat & Kautz
(2020)
29.7
VAE
NCP- VAF Aneja et al.
202T
24.8
VAEBM Xιao et al.
(2200)
20.4
OTM Rout et
2022)
13.56
UOTM (KL) (Chor al. 2023a
6.36
OT-based
UOTM (SP)
6.31
UOTM-SD
5.99
8
7
6
5
value
3
2
1
0
10
20
30 40
50
60
70
80
90
100
10
20
30
40
50
60
70
80
90 100
Iteration Num (K)
Iteration Num (K)
(a) OTM
(b) UOTM
Figure 10: Distribution of the norm of the potential gradient |/ VvVil,,, II (o))(()) random real data y and a randomly generated data ý for every 10K iterations on CIFAR-10. Due to the equi-Lipschitz property, the gradient norm of UOTM potential is stable during training. This stability contributes to the stable training of UOTM. In the Toy dataset, we measured the Average Rate of
Change (ARC) of potential (v)())-r,(m) between randomly selected training data and another randomly chosen point y within the data space (Fig 6). However. unlike the Toy dataset, the image dataset is extremely sparse in its ambient space (pixel space). Hence, randomly selecting point within the pixel space can yield undesirable results. Therefore, instead of measuring the Average Rate of Change (ARC) of potential, we measured the norm of the potential gradient.
||--
D.4 ADDITIONAL DISCUSSIONS ON SCHEDULING
Schedule Intensity Ablation To analyze the effect of schedule intensity further, we evaluated our UOTM-SD model for four different scheduling intensities. For simplicity, we focused on symmetric ones i.e., Osaaz = k, amin 1/k for some k > 1, while fixing 0.001. Overall, the Linear Scheduling scheme provided the best result, achieving FID scores below for all scheduling intensities. Nevertheless, the other scheduling schemes also demonstrated robust performance.
24

---

Published as a conference paper at ICLR 2024
40000
1:75
35000
30000
1:25
1.00
20000
15000
0.75
10000
0.50
5000
Iteration
Iteratonn
Ittraation
(a) WGAN
(b) WGAN GP
(c) UOTM w/o cost
160
0.20
80
60
15 Iteratioo Num (K)
15
Itrration Num IKI
OTM
(e) UOTM
Figure 11: Distribution of the absolute value of Average Rate of Change (ARC) of potential (v) (v)- (x]] for every 5K iterations. Due to the quii-hipschizz property, |ARC] of UOTM potential y-2
stable during training. This stability contributes to the stable training of UOTM.
Table 10: Ablation Study on Schedule Intensity (Omina O
Schedule Type
(1/2, 2)
(1/3, 3)
(1/5, 5)
(1/10, 10)
(3, 3)
(5,5)
Cosine
3.20
2.94
2.57
2.78
Linear
2.70
2.97
2.51
2.77
3.73
3.99
Step
3.29
2.85
2.78
2.70
Moreover, we tested UOTM-SD without scheduling (a-UOTM). Specifically, we tested the setting of Omna Ommn Oconst 1. In this case, the divergence weight in Eq. is constant throughout training. When we set Aconst = 3, 5, a-UOTM showed FID scores of 3.73 and 3.99. This result demonstrates that a-scheduling provides a method for harnessing the advantages of both the large regime and the small r regime. Therefore, UOTM-SD outperforms a-UOTM.
D.5 ADDITIONAL QUALITATIVE RESULTS
Figure 12: Generated samples from UOTM with Small 7(= 0.0002) CIFAR- 10 (32 32).
25

---

Published as a conference paper at ICLR 2024
Figure 13: Generated samples from UOTM with Optimal 7(= 0.001) CIFAR-10 (32 32)
Figure 14: Generated samples from UOTM with Large r(= 0.005) on CIFAR- 10 (32 x 32).
Figure 15: Generated samples from UOTM-SD (Cosine) CIFAR-10 (32 32).
26

---

Published as a conference paper at ICLR 2024
Figure 16: Generated samples from UOTM-SD (Linear) on CIFAR-10 (32 x 32).
Figure 17: Generated samples from UOTM-SD (Step) on CIFAR-10 (32 x 32).
27


Reviews:
[{"review_id": "CdHVFCkdpu", "reviewer": "Reviewer_oAtf", "summary": "OT has been widely explored in generative modeling from diverse perspectives (e.g., OT loss and OT map). However, interpretation and understanding of the pros and cons of OT are underexplored.  In this paper, the authors are proposing a framework generalizing the existing OT-based generative models and additionally propose a scheduling method to mitigate the drawback of the cost function derived from OT. Both quantitative and qualitative analyses and experiments are reported.", "questions": "1. Is the analysis valid in relatively higher-dimensional data (e.g., 128^2 or 256^2) such as  CelebA or FFHQ, CUB, or ImageNet?\n2. (Alg. 1) What is $X$ in line 3? I would assume $Y$ is from real data and $z$ is from the prior known distribution, e.g., Gaussian. Similarly, $T_\\theta$ is consistently taking $x$ as input throughout Equations in the paper while $x$ and $z$ are taken as input in the algorithm (line 4). For example, in Eq. 5, I can see that $x$ is a prior distribution $y$ is a real distribution and the OT term $c$ is applied in between the Gaussian and the generated samples, which I believe is different from the Algorithm.\n3. (page 6, “Effect of Cost in Mode Collapse”) It is not straightforward how the cost function actually helps the mode coverage. \n4. Why the results in Fig. 1 and Fig. 5 are different?\n5. Performance of UOTM-SD in the toy dataset? and Qualitative comparison results in Cifar10 (UOTM-SD v.s. UOTM)?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Background knowledge is contained well in the paper.\n- Existing OT-based methods are analyzed well under the proposed generalizing framework.\n- An additional scheduling method is proposed for mitigating a drawback of the cost function while boosting the benefit.", "weaknesses": "- It is not easy to understand without expertise in OT. It would be much easier to read if one or two lines of descriptions comparing each term and notation with those of the regular GAN setup were provided.\n- The analysis is good, but the benefit of the proposed method (UOTM-SD) is not clearly shown.\n- Experiments are limited to low-dimensional datasets which is not practical enough.\n- Some parts are not clear which are described in the Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gxaizngDuW", "reviewer": "Reviewer_12Gz", "summary": "This paper frames together two similar adversarial generative models: GANs involving a generator that learns to minimize an OT distance, and models whose objective is to directly learn the OT map from a prior distribution to the data distribution. The authors evaluate the stability and mode collapse of these models and conclude on the advantage of OT map models. In this category, one (based on unbalanced OT) is more performant than the other (based on standard OT), but less robust to hyperparameter choices. To alleviate this issue, the authors propose an interpolation strategy between both models to be scheduled during training.", "questions": "Cf. the *Weaknesses* part of the review for questions related to paper improvements.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "By studying the considered adversarial models through a unified framework, this paper provides **interesting comparative insights** on their experimental performance. These insights might be valuable for future research in this area, highlighting the value of OT map models. These insights are **well illustrated** thanks to toy experiments, and the paper is overall **clear and easy to read**. The resulting proposed model refinements provide **improvements in generative performance and robustness to hyperparameter choices**, which is a substantial contribution in this domain.", "weaknesses": "The paper suffers from three main weaknesses that, together, make me believe that it remains under the acceptance threshold. I look forward to discussing with the authors and other reviewers on this topic.\n\n### Significance of the Proposed Framework\n\nAs it is described in Algorithm 1, the proposed framework is a useful framework for exposition and experimental design, but the significance of this contribution is limited.\n- It is a straightforward generalization of the already established lookalike adversarial objectives of Equations 4 and 5. This kind of framework is common in the GAN literature, e.g. in Nagarajan et al. (2017).\n- The unified algorithm is not by itself the source of novel insights, or novel models. The proposed UOTM-SD does not necessitate this unified algorithm as it only interpolates between two OT map models using Equation (12).\n\nNagarajan et al. Gradient descent GAN optimization is locally stable. NIPS 2017.\n\n### Weak Experiments\n\nThe experiments only weakly support the claims of the paper.\n- The convergence and mode collapse properties are mainly studied in Section 3.2 on a toy dataset. Yet, the difference between low and high dimensions can be large when dealing with neural networks. Considering a higher-dimensional structured dataset could strengthen the experimental conclusions. Additionally, I would also suggest including another low-dimensional dataset to avoid any bias linked to having a data distribution evenly distributed around the prior distribution.\n- Still on Section 3.2, the experiments lack a vanilla GAN baseline, especially to conclude on advantage of the SP function. Similarly, experiments of Figures 5 and 6 miss the WGAN-GP baseline.\n- As a standalone model and with the available information, the comparative advantage of UOTM-SD is not significant enough. The gain in FID is minor and would thus require confidence intervals to be validated. The FID being computed on the training set, there is also a risk of overfitting to be taken into account. Furthermore, other datasets might be considered to test the robustness of the methods to other modalities and data dimensions.\n\n### Possible Bias against OT Loss Models (GANs)\n\nGenerators in GANs do not require their latent space to be of the same dimension as their output. Yet, it seems to be the case for OT map models, given that they learn a transport map between two distributions living in the same space. I would suggest the authors to explicitly explain how this affects their experiments and their results. Are the provided comparisons fair between the two types of models, in terms of dimensionality and neural network architectures? Both operation modes seem hard to articulate with each other, as Algorithm 1 features the sampling of both a random variable from the same space as the data and another latent variable to accommodate the two types of models.\n\nMoreover, the authors should further the comment why the Lipschitzness result of Theorem 3.1 is an advantage over GANs. WGAN(-GP) also requires Lipschitz solutions, and the Lipschitzness constraint is even applied to other GAN models nowadays.\n\n### Remarks on the Form\n\n- The references of Fan et al. and Rout et al. miss a year.\n- Some notations are not defined, like $\\Pi(\\mu, \\nu)$ and $D_{\\psi_i}$ in Section 2.\n- Differentials $d$ in integrals should be upright for better readability.\n- Abbreviations of \"Equation\" should end with a point: \"Eq.\".\n- Equations 4 and 5 are not learning objectives but optima of learning objective. The correct way to present them is in Algorithm 1.\n- The color scheme of Figure 5 should be adjusted for a better readability in grayscale.\n- Some space should be added between images and captions in Figure 4.\n\n### Post-Rebuttal\n\nI would like to thank the authors for our discussion. **Their responses addressed a number of my concerns**, especially by strengthening the experiments and ensuring the soundness of their conclusions. Two main weaknesses remain in my opinion: the lack of strong connection between the two main contributions of the paper (Sections 3 and 4), and the Lipschitzness results which are not conclusive enough. Still, the paper is interesting and I think it might inspire new research on the topic. Hence, **I raise my \"soundness\" score from 2 to 3 and my main recommendation from \"marginally below the acceptance threshold\" to \"marginally above the acceptance threshold\"**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t4tGVyYuBz", "reviewer": "Reviewer_7jfY", "summary": "In this paper, the authors propose novel theoretical results on varies aspects of the recently proposed Unbalanced Optimal Transport Model (UOTM), which is an optimal transport (OT) based generative model. In particular, in section 3, the authors provide an insight of how different choice of g_1 and g_2 and cost function could help stabilize training. In theorem 3.1, the authors prove the existence and uniqueness of the UOTM model. Moreover in section 4, the authors propose a novel alpha-scheduling method to stabilize training as well as mitigating the mode collapse/mixture problem. Theorem 4.1 shows that under this schema, the solutions of the UOT problems converge to the OT solution when alpha goes to infinity. This provides a new approach of solving the OT problem in the context of generative models.", "questions": "1. For WGAN explanation in section 3.1, the authors categorize this case as c = 0. I don't think this is the case because the in the original paper, this cost is the L1 Euclidean distance, i.e. c(x, y) = |x-y|. Also, it seems the Lipschitz constraint is missing in this case. It would be great if this part is further clarified.\n2. Same comment for the italic sentence after Eq. 5. \n3. More of a suggestion: in the experiment part (E.g. Fig. 1, Fig. 5 etc.), it would be great if the authors also include non-DNN based OT maps like the one proposed in An et. al 2019, as their solution is unique and can be found by a convex optimization.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The originality of the paper mainly comes from theorems 3.1 and 4.1, which consolidate the recently proposed UOTM method. These theorems pave the way of the proposed method that addresses the tau-sensitivity problem. Clarity of the paper looks good to me overall, but there are some places I'm confused about.", "weaknesses": "The experiment results look promising as well. It would be great if the proposed method is applied on higher solution image dataset to showcase the image generation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfdIQcTK1L", "reviewer": "Reviewer_1bYC", "summary": "In this paper, the authors trying to unify various descriptions of OT-based adversarial networks, and  compare known frameworks under the proposed unified framework. In their unified framework, i.e. Algorithm 1, there are some degrees of freedom:\n- functions $g_{1,2,3}$, that measure each term of the adversarial loss:\n    - $g_1$ is for discriminator's loss from c-transform,\n    - $g_2$ is for discriminator's loss from dual potential in OT,\n    - $g_3$ is for generator's loss from c-transform,\n- cost function $c(x,y)$ in the context of OT:\n    - $\\tau$: coefficient of squared transport cost $c(x,y) = \\tau \\|x-y\\|^2_2$,\n- regularization term $\\mathcal{R}$.\n\nThe choice of each component corresponds a certain generative model training protocol.\n\nThey prove some theoretical guarantees on the training protocols (Thm 3.1 and Thm 4.1) and verify the assumptions for the theorems, i.e. strictly convexity and finiteness of $\\Psi$s do improve the training procedure.\n\nIn addition, they clarify the limitations on their method in 5 CONCLUSION.", "questions": "Question\n- Do $g_{1,2}$ in Algorithm 1 correspond $\\Psi_{1,2}$ in eq.(8)? If so, why the authors change its notation?\n> It has been answered by authors\n- The author achieved best performance of UOTM-SD on CIFAR-10, it would be good. But how about different image data? Does the scheduling strategy also improve stability using other image data?\n> It has been answered by authors", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written, and there are some theoretical results (Thm 3.1 and Thm 4.1). \n- They conducted various experiments and show the results on UOTM(-SD), which is the proposed unified framework, can achieve the best performance on CIFAR-10 generation except for SOTA diffusion model (Table 3). In particular, they proposed a concrete method for preventing mode collapse problem in Section 4, by considering scheduled scaling of the objectives, i.e. UOTM-SD.", "weaknesses": "- I think it is slightly misleading that using $\\Psi$s in eq.(8) that would correspond to $g$s in Algorithm 1.\n> After discussion to the authors, I concluded that this issue will appear to be less of a problem, and I raised Soundness a little.\n- On the experiments for Lipschitz continuity, Fig 6, the experiments seem to be conducted only with 2d data, and I prefer counterparts of them in training for image generations also.\n- UOTM-SD look working very good in CIFAR-10, but it is not evident that is also good for other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O5YZKwFX6T", "reviewer": "Reviewer_h5iC", "summary": "This paper introduced a unified framework that encompasses previous OT-based GANs, which is derived from the semi-dual form of unbalanced OT.  The authors presented a comprehensive analysis of different OT-based frameworks, as well as other well-researched generative models such as diffusion and VAEs.  In the end, the authors demonstrated a tradeoff between perception quality and mode collapsing, which was affected by the effect of the cost term in the loss function, and proposed a new training scheme by controlling this term during training.", "questions": "I'm surprised that WGAN didn't even work at all for the first toy problem.  How many hyperparameter settings did you try?  What would be the reason you think that it just won't work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is an extremely well-written paper.  The presentation is easy to follow and technical details are clear and sound.  \n\nThe findings in the paper are interesting.  Although the newly proposed scheduling scheme isn't the most impressive, but I trust the insight of this unified view will deepen the understanding on this topic.  \n\nThe experiments are well set up to prove the hypothesis.  In general, I enjoyed reading the paper very much.", "weaknesses": "The main weakness of this paper is the proposed scheduling scheme is a very simple idea which proved to be useful for a basic experiment in an extremely low dimensional setting.  I'm not sure whether this method can be scaled to high dimensional setting easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)