Evolution of Discriminator and Generator Gradients in GAN Training: From Fitting to Collapse
Authors: Weiguo Gao
Ming Li
Venue: Accepted by TMLR ()

Abstract: Generative Adversarial Networks (GANs) are powerful generative models but often suffer from mode mixture and mode collapse. We propose a perspective that views GAN training as a two-phase progression from fitting to collapse, where mode mixture and mode collapse are treated as inter-connected. Inspired by the particle model interpretation of GANs, we leverage the discriminator gradient to analyze particle movement and the generator gradient, specifically "steepness," to quantify the severity of mode mixture by measuring the generator's sensitivity to changes in the latent space. Using these theoretical insights into evolution of gradients, we design a specialized metric that integrates both gradients to detect the transition from fitting to collapse. This metric forms the basis of an early stopping algorithm, which stops training at a point that retains sample quality and diversity. Experiments on synthetic and real-world datasets, including MNIST, Fashion MNIST, and CIFAR-10, validate our theoretical findings and demonstrate the effectiveness of the proposed algorithm.

Published in Transactions on Machine Learning Research (03/2025)
Evolution of Discriminator and Generator Gradients in GAN Training: From Fitting to Collapse
Weiguo Gao
mggaa@ffudnn.co
School of Mathematical Sciences & School of Data Science, Fudan University Shanghai Key Laboratory of Contempororg Applied Mathematics
Ming Li
mingli2230.fuudar edu.cm
School Mathematical Sciences, Pudon University
Reviewed on OpenReview: https: openreview. net/ form?? id- 58ggkcVV
Abstract
Generative Adversarial Networks (GANs) are powerful generative models but often suffer from mode mixture and mode collapse. We propose perspective that views GAN training as two-phase progression from fitting to collapse, where mode mixture and mode col- lapse are treated as inter-connected. Inspired by the particle model interpretation of GANs, we leverage the discriminator gradient to analyze particle movement and the generator gradient, specifically "steepness," to quantify the severity mode mixture by mcasuring the generator's sensilivity to changes in the latent space. Using these theoretical insights into evolution of gradients, we design specialized metric that integrates both gradients to detect the transition from fitting to collapse. This metric forms the basis of carly stopping algorithm, which stops training at a point that retains sample quality and diver- sity. Experiments on synthetic and real-world datasets, including MNTST, Fashion MNTST, and CIFAR-10, validate our theoretical findings and demonstrate the effectiveness of the proposed algorithm.
1 Introduction
Generative Adversarial Networks (GANs) serve as popular technique for unsupervisedly learning gener- ative modols of structured and complicated data (Goodfellow et al., 2014; Nowozin et al., 2016; Arjovsky et al., 2017; Goodfellow, 2017; et al., 2017; Nguyen et al., 2017; Ghosh et al., 2018; Luo & Yang, 2024). GANs typically involve a generator that generates samples resembling real samples, and a discriminator that differentiates between real and generated samples. Through adversarial training, the generator learns to produce increasingly realistic samples, while the discriminator enhances its ability to distinguish them, resulting in refined models.
One of the primary challenges in training GANs is fine-tuning the interactive dynamics between the generator and the discriminator. If these dynamics are not well-aligned, several problematic behaviors can arise. Among the most common issues are mode collapse (Goodfellow, 2017) and mode mixture (An et al., 2020). Mode collapsc occurs when the generator produces limited varicties of samples, collapsing to very few modes, while mode mixture involves blending distinct modes, resulting in unrealistie or ambiguous outputs. Numerous GAN variants have been proposed to address mode collapse (Nowozin et al., 2016; Arjovsky et al., 2017; Li et al., 2017; Nguyen et al., 2017; Ghosh et al., 2018; Luo & Yang, 2024), alongside theoretical insights (Sun et al., 2020; Becker et al., 2022). For mode mixture, research has focused on mitigation strategies, particularly within the framework of optimal transport (Lei et al., 2019; An et al., 2020; Gu et. al., 2021) and rejection sampling (Tanielian et al., 2020).
Despite extensive rescarch on GAN training, current studies share two common limitations: (i) mode col- lapse (Goodfellow, 2017) and mode mixture (An et al., 2020; Tanielian et al., 2020) are typically treated

---

Published in Transactions on Machine Learning Research (03/2025)
separate, independent issues, and (ii) mode collapse is frequently viewed as an indicator of training fail- ure (Arjovsky et al., 2017; Luo & Yang, 2024), and the current literature primarily focuses on techniques to prevent or mitigate this phenomenon (Nowozin et al., 2016; Arjovsky et al., 2017; Li et al., 2017; Nguyen et al., 2017; Ghosh et al., 2018; Luo & Yang, 2024). In contrast, we propose a perspective that views GAN training as a two-phase progression from filting lo collapse. In this view, mode mixture an integral part of the fitting phase, where the generator increasingly aligns with the real data distribution. During this alignment, some generated samples may fall outside the modes, reflecting the presence of mode mixture. Mode collapse does not signify outright failure. Instead, it occurs when the model tries to alleviate mode mixture. In this process, generated samples are pushed away from the modes, leading to loss of diversity. By identifying the transition from fitting to collapse, early stopping can prevent the loss of diversity while preserving sample qualily.
To illustrate this progression, we train the Non-Saturating GAN (NSGAN) (Goodfellow et al., 2014) on a 3-dimensional Gaussian mixture dataset and MNIST (LeCun et al., 1998), recording generated samples as shown in fig. 1. In the first row, the orange dots represent real samples drawn from the Gaussian mixture, while the blue dots show generated samples. Initially, the generated samples cluster near the origin (subfigure 1). As training progresses, these samples spread out and align with the modes of the real distribution (subfigure 2-3). This marks the fitting phase, where the generator increasingly captures the structure of the real distribution. Nevertheless, during this phase, some generated samples may fall outside the modes, indicating the presence of mode mixture. As training continues, the generated samples start to collapse into fewer modes (subfigure 3-6), eventually leading severe loss of diversity. This collapse phase is evident in the Gaussian mixture dataset, highlighting the need for early stopping to preserve diversity. Tn the sccond row, map both real MNIST images and generated images into common 3-dimensional space using UMAP (Uniform Manifold Approximation and Projection) (Mclnnes et al., 2018), where we observe similar progression.
Figure 1: The real and generated samples by training NSGAN on a 3-dimensional Gaussian mixture dataset and MNIST. First row: Gaussian mixture dataset. Orange: Real samples. Blue: Generated samples, Epochs from left to right: 0, 15, 60, 450, 850, 980. Initially, the generated samples cluster near the origin, then spread out and align with the real modes. However, instead of becoming more refined, they eventually collapse to part of the modes. Second row: MNIST embedded in 3-dimensional space. Colored: Real samples. Black: Generated samples. Epochs (Batches) from left to right: 0(0), 0(8), 0(32), 0(64), 32(0), (0). Similar progression has been obscrved.
Generalizing these observations, we propose a perspective that views GAN training as a two-phase progression from fitting to collapse. Our main tool for analyzing the two phases the study of gradient dynamics, as gradients of the discriminator and generator functions with respect to their inputs provide insight into how generated samples evolve. Table provides overview of each phase. Identifying the transition from fitting to collapse is crucial, as stopping training at this point can retain sample quality and diversity: stopping too carly yields unrcalistic samples, while stopping too late leads to reduced diversity.
2

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: An overview of the two phases: fitting, and collapse, which includes brief description and the roles of discriminator and generator gradients.
Fitting
Collapse
Description
Particles move toward and converge around the modes, reducing their spread and miti- gating mode mixture.
Particles near mode boundaries are pushed away, eventually leading to mode collapse.
Discriminator gradient
Guides particles from the initial noise prior toward regions close to the modes.
Pushes particles near mode boundaries away with significant force and magnitude.
Generator gradient
Measures how the generator maps nearby points in the latent space to distant points in the output space, quantitatively measure the severity of mode mixture.
Drops in maguitude as particles near mode boundaries are pushed away and concen- trate around fewer modes.
Our contributions can be summarized as follows:
We propose a perspective that views GAN training as a two-phase progression from fitting (section 3) to collapse (section 4), where mode mixture and mode collapse are treated as interconnected. Notably, we highlight the underexplored idea that mode collapse (i.c., the collapse phase) may emerge in the later stages of converging GAN (i.e., the fitting phase).
We employ gradient-based tools to analyze each phase, using the discriminator gradient which guides particle movement and the generator gradient, termed "steepness," " to quantify mode mixture severity. These tools are detailed in section 2.
We develop an early stopping algorithm to optimize GAN training by detecting the transition from fitting to collapse. The early stopping algorithm, outlined in section 4.3, uses a metric based on discriminator and generator gradients. By intrinsically capturing GAN training dynamics without direct dependence on generated real images, it identifies a stopping point where both sample quality and diversity are retained, as empirically demonstrated in section 6.
2 Technical Preliminaries and Basic Assumptions
In this section, we provide the technical preliminarics and basic assumptions. We begin with overview of the gradient dynamics in section 2.1, focusing how the generator and discriminator gradients shape the behavior of generated samples across the two phases, as summarized in table 1. In section 2.2, we present an interpretation of GANS as particle models, where the discriminator gradient guides the movement section 2.3, we introduce the concept of steepness, derived from the generator gradient, to quantily the severity of mode mixture. Finally, in section 2.4, we outline the assumptions we make regarding the real data distribution and the noise prior.
of generated samples as particles.
2.1 An Overview of Gradient Dynamics in GANs
In this work, the main tools we use to analyze the proposed two phases of GAN training are the generator and discriminator gradients. To clarify, we consider gradients as derivatives of the generator and discriminator functions with respect to their inputs, rather than with respect to network parameters. In section 2.2, we interpret the divergence GANs particle models, where generated samples are viewed as particles, each moving based on function of the discriminator's gradient. During the fitting phase, this gradient guides particles from the initial noise prior lowards regions near the modes. In section 2.3, we define sleepness based on the generator's gradient. Steepness quantifies how the generator maps nearby points in the latent space to potentially distant points the output space, providing a measure of mode mixture severity that enables a quantitative analysis of this phenomenon in the fitting phase. The collapse phase is characterized by two distinctive behaviors. From particle perspective, certain particles near the mode boundaries start
3

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm Interpretation of Non-Saturating GAN Particle Model (c.f. (Yi et al., 2023))
Require:
The discriminator dw (with w denoting the discriminator's parameters) and the generator go (with
0 denoting the generator's parameters), the noise prior Pz. batch size m > O, step size s > 0
1: for number of training iterations do
Train the discriminator du as in (Goodfellow et al., 2014).
Sample z 's from the noise prior pz(z).
Generate particles Z, go (z)), (1 (isism) m)
Update the particles Z, Z (d)(Zi) i m).
Apply the stop gradient operator to 2g and update de by descending V Σ (190(2)) 2.112 end for
to "escape" from these modes, phenomenon we analyze using the discriminator's gradient. From global perspective, generated particles begin to concentrate around only few modes. We use the generator's gradient to characterize this concentration effect, giving comprehensive view of the collapse phase.
2.2 Discriminator Gradient: Guiding Particle Movement
Divergence GANs such as Vanilla GAN (Goodfellow et al., 2014), NSGAN (Goodfellow al., 2014) and f-GAN (Nowozin et al., 2016) can be interpreted as particle models (Gao et al., 2019; Johnson & Zhang, 2019; Franceschi et al., 2023; Huang & Zhang, 2023; Yi et al., 2023). This paper focuses on the NSGAN for its practicality and conciseness. And we outline the methodology for other Divergence GANs in appendix H. The pseudocode of NSGAN as particle model is presented in algorithm 1, which is fundamentally grounded in the work of Yi et al. (2023). This interpretation is essentially equivalent to the original NSGAN (Yi et al., 2023, theorem 3.2). Accordingly, we refer to the generated samples as particles throughout this paper. Unless otherwise stated, we assume the discriminator is optimal', i.e,, de (x) = Pdata( (Paaaa (x) + Pv (x))) as established by Goodfellow et al. (2014) (we omit the subscript w for brevity hcrcafter). Consequently, the vector field Vd((=)/d(ar) that guides particle movement can be reformulated in terms of the density ratio r(x) Pdata ())pp(())
(d) -r(x) d,
(1)
(1=)(1+)(z))
2.3 Generator Gradient: Measuring Mode Mixture Severity
In addition to the discriminator's role guiding particle movement, the generator's gradient provides a mcasure of mode mixture severity. As described in algorithm 1, particle æ near a mode is updated in the direction of (d) typically pointing towards the nearest mode (see section 3). However, between adjacent modes, critical points exist where nearby particles are pushed apart in opposite directions. During training, two close latent points and 22 may map to outputs g=(21) and (x2) that are far apart. This behavior reflects high sensitivity in the generator's mapping, indicated by large spectral norm of the Jacobian of ge. This motivates the concept of steepness (see definition 2.1), which quantifies the generator's sensitivity across the latent space. Higher steepnoss corrosponds to regions where small differences in latent points produce large separations the output space, mitigating the severity of mode mixture. Importantly, this definition is invariant under orthogonal coordinate transformations, ensuring that steepness S, captures intrinsic properties of the generator's 's mapping. This notion of steepness is related to several works that study the conditioning and Lipschitz constants of the generator Jacobian, which we discuss in scction 5.
Definition 2.1. Let g: Th + RR" be continuously differentiable. The sleepness of g al point x, denoted by S,(),, is defined as the spectral norm of the Jacobian of g at w:
=
(2)
For the sake of completeness, we also provide an analysis of class of suboptimal discriminators appendix D.
4

---

Published in Transactions on Machine Learning Research (03/2025)
2.4 Assumptions Real Data and Noise Prior
Gaussian smoothing of the data, applied in assumption 2.1, a commonly used approach in machine learning to transform discrete datasets into contimuous probability distributions. This method enables math- ematical analysis, aligns with standard data preprocessing practices, and is widely adopted in generative modeling (Goldfeld et al., 2020; Ho et al., 2020; Song et al., 2021; Karras et al., 2022). By employing kernel density estimation with Gaussian kernel .:) and covariance matrix In we can provide a smooth and representative approximation of the underlying data distribution.
Assumption 2.1. Let 11, .2....... IN IR, where the æ; 's are in ascending order if = 1. We assume that the real data distribution has the following probability density function
1 x 11
Pdata (2)
Σ (x, N
(2mo2)n/22
exp
20²
(3)
Optionally, we may assume a seporation condition parameter > 0, such that minigisjsN -0;- 12 2 A.
We make the following assumption the noise prior (z()) for reasons in appendix B.
A Assumption 2.2. Let be the dimension of real samples. We assume that the noise prior Pz N I.)
is on n-dimensional standard Gaussian distribution.
3 The Fitting Phase: Gradient Dynamics
The fitting phase of GAN training refers to the process where generated particles align with the modes of the real data distribution. This alignment is driven by both discrininator gradients and generator gradients, each playing a distinct yet. complementary role: (i) discriminator gradients guide the generated particles by providing directions toward the nearest modes, and (ii) generator gradients, specifically steepness, determines the generator's ability to separate modes and reduce mode mixturc. Together, these gradients ensure that generated particles converge around real modes, even though some particles may still remain outside or within the inter-moda. regions, reflecting mode mixture. In this section, we analyze the role of discriminator gradients in section 3.1 and the impact of generator steepness in section 3.2.
3.1 Evolution of Discriminator Gradients
In this subsection, analyze and visualize how discriminator gradients guide generated particles towards modes during the fitting phase. Importantly, we assume an optimal discriminator, which ensures exact update directions for particle movement. This assumption applics specifically to the fitting phase and does not hold in the collapse phase, where the discriminator may lose its optimality (see section 4.1). While prior work has established a solid theoretical foundation by showing that an optimal discriminator induces a Wasserstein gradient flow to align Py with Pdata (Franceschi et al., 2023; Yi et al., 2023), we examine four illustrative scenarios in table 2 investigate how this process depends on the configurations of Pg and Paata? 2.
Table 2: Descriptions and configurations of Pdata and Pg in the four cases.
Description
Pdata
Ps
Case 1
Initialization with concentrated particles
[[ 0.142)
NV(0,0).0211
Case 2
Particles covering all modes
M[ 0.12)
u([-2,2) x 2l)
Case
Particles concentrated single mode
M( 0.
W(L1, I2)
Case
Globally separated modes
N([+3, 0.
N(3,3) 31,,3),34 312)
For simplicity, the 3-dimensional Gaussian mixture dataset in fig. is projected the ay- plane, and the covariance of
each Gaussian component set. to O.I.T
5

---

Published in Transactions on Machine Learning Research (03/2025)
2.0
1.0
15
0.80
1.0
1.
0.64
0.5
0.6
0.5
0.0
-05
044
-0.5
0.33
-10
-1.05
022
-1.0
-1.3
0.16
-15
22.0
-2
-1.3
-0.33
-5.00
0.0
-2.0
-2
1.25
1.00
2.85
0.00
2,0
-140
1.0
1.05
1.0
0.75
0.5
-100
+3.00
0.4
0.45
100
-.15
3.25
0.30
15
0.15
20 -1
-2
0
Figure 2: The vector field (z) with zoomed-in views around the bottom left mode. Colorbars in the left subfigures show discriminator values (i.e., (æ)). Top left: Case 1. Top right: Case 2. Bottom left: Case 3. Bottom right: 4, These subfigures highlight how discriminator gradients guide generated particles under different configurations.
Case (top left subfigure of fig. 2) shows that, the update vector field (z) pulls particles toward the direction of the nearest modes, with vector lengths proportional to their distances from the modes. This ensures rapid initial movement of particles towards the modes at initialization. For case 2 (top right subfigure of fig. 2), particles near the mode centers exhibit minimal movement due to weak gradient forces, while those farther away are guided towards the nearest mode by stronger discriminator gradients. This results in a progressive sharpening of Pg. as particles accumulate near the modes. For case (bottom left subfigure of fig. 2), the discriminator values (indicated by the colorbar) differ significantly across modes: the covered mode (which centers at [1, 1]) has the lowest value, while the farthest mode (which centers at -1) has the highest. The vector field's intensity peaks near unoccupied modes and diminishes near crowded ones, This dynamic ensures redistribution of particles to balance coverage across modes. For case 4 (bottom right subfigure in fig. 2), the gradient intensity weakens near all modes, making particle movement less effective. Proper initialization and balanced training are critical in this setting to avoid stagnation3.
The role of discriminator gradients during the fitting phase. Discriminator gradients play a crucial role in guiding generated particles toward the nearest modes during the fitting phase. This explains why particles, initially clustered (the first subfigure of fig. 1), spread out and move toward modes. For locally clustered modes (Cases 1-3), the gradients effectively align particles with the modes. For globally separated modes (Case 4), the gradients weaken, and successful convergence depends on careful initialization and balanced training.
3.2 Evolution of Steepness
While discriminator gradients guide particles toward modes, mode mixture is still commonly observed in practice. To understand this phenomenon, we analyze the role of generator gradients, characterized by the steepness of the generator function. Steepness measures how sharply the generator transforms adjacent points in the latent space into distinct points in the data space. In this subsection, we first establish that the optimal generator must exhibit significant steepness to transform a standard Gaussian distribution into Pdeta (both in I-dimensional and n-dimensional cases). Conversely, when the steepness is insufficient, the
3 Please refer appendix for the corresponding theoretical results.

---

Published in Transactions on Machine Learning Research (03/2025)
transformation of the standard Gaussian distribution via the generator does not fully match Pdata- This discrepancy causes mode mixture, where generated samples fail to align perfectly with the real modes.
Steepness of measurg maps. We begin by analyzing the steepness of the optimal generator function that satisfies P# = Pdata- In the I-dimensional case, any mneasure-preservnn map g can be expressed as g = o 9 where & and v are the cumulative distribution functions (CDFs) of W and Pdata; respectively, and h a measure-preserving map of the uniform distribution 24 (). Among the infinitely many possible choices of h, the identity map holds particular significance. In this case, the corresponding I represents the optimal transport map from pz to Pdata under the Wasserstein distance with strictly convex cost functions (Santambrogio, 2015), including the widely-used 2- 2-Wassesstcin distance as a specific example*.
Theorem 3.1 (Steepness of 1-dimensional peeessur--presevviin maps). Assume that the real data distribution
Pdata()) satisfies assumption 2.1 with n = 1 and seperation condition parameter A = 60. Let 4(x)) and (x) denote the cumulative distribution functions (CDFs) of 1(0,1) and (data(z), respectively. Define g(x) == (+x)))) Then, there erists point N. E IR such that the sleepness of g at u, salisfies:
(ai+1 ((
xx)3 exp(-xx
(a.)222
min a.ee exp 15i5N-1
(4)
802
where q is the (1 - t/N)th quantile of the standard Gaussian distribution.
As indicated by theorem 3.1, the steepness S, grows exponentially with the square of the distance between neighboring modes. This means that for distributions where the modes are further apart, the generator function must exhibit a much larger steepness to accurately map samples between modes. Conversely, S, is inversely related to the variance 0² of each mode. A smaller variance requires the generator to transition more sharply between modes, resulting in a steeper function. This properly extends to higher dimensions, as demonstrated in theorem 3.2, which provides an explicit lower bound for Sg-
Theorem 3.2 (Steepness of n-dimensional measure preserving maps). Assume that the real data distribution Pdota( (z) satisfies assumption 2. and that the noise prior pz(z) is the truncated Gaussian on the n-dimensional ball (0). Without loss of generality, suppose x; 0 for all 1 S I N. Let 9: (0) + Rn be continuously differentiable, piecewise injective function satisfying 9#P= Pdota Then, there crists point x, E R² such that the steepness Sy(a.) satisfies Sy 2 M, where
defined
| x; -
M - o. V2T max min exp ISiSN
(5)
2n
Here, x= z= EN 2,/N is the mean of the mode centers, and 5 = cxp(-r2/22) /2/22//22T accounts for the truncation
of the Gaussian distribution.
Similar to the 1-dimensional case, the bound exhibits exponential growth with increasing distances ||λ Mhā-20 2x:122 and with decreasing variance o². Consequently, when the modes are widely separated or when the standard deviation is small, Sg becomes significantly large. This observation may provide insight into the common practice of normalizing or rescaling image data during preprocessing, these steps can reduce the steepness required for the generator to map the noise distribution to the real data distribution, which potentially influence the training dynamics.
Evolution of steepness. We have established that in order to map a standard Gaussian distribution to Pdatas the generator function must exhibit large steepness. A natural question arises: How does the steepness evolve during training? In the next theorem 3.3, we analyze the setting where Pdata is symmetric mixture of Gaussians and derive evolution equation in continuous time for the steepness of the generator function
9: at z 0, i.e., (0) (which follows from the observation in fig. 3 that the generator function reaches its maximum steepness at. z 0). This equation provides two key implications: (i) when the third-order (3) derivative of g: at 0, i.c., ge (0), is small compared (0) (for example, when Se can be well-approximated
4 For a visua lization of generator functions g corresponding to different Paatas please refer to appendix T.
7

---

Published in Transactions on Machine Learning Research (03/2025)
by linear function near 0, fig. 3 depicts), the steepness g((0) monotonically increases; and (ii) early stage of training, the dominant terms in the numerator and denominator of the right-hand side of the evolution equation are (n² -² (0)( and (g((0))², respectively. As a result, g((0) initially grows exponentially with rate (n²- 0²)/o'. This suggests that when the modes are well separated (μ > σ), the generalor rapidly increases its steepness to match the target distribution. Theorem 3.3 uses the gradient-flow framework, which common idcalization in thcoretical analyses, as it allows us to capture the continuous- time dynamiis underlying the update process. In practice, this corresponds to settings where the generator learning rate (s in algorithm 1) is sufficiently small and the generator is smooth, that its evolution can be well-approximated by continuous-time trajectory. Although this gradient-flow assumption is an abstraction from the discrete and stochastic updates used in practical implementations, it provides clear insights into the generator's behavior, most notably the rapid increase in steepness during early training, without imposing overly restrictive conditions on the generator. Our analysis indicates that similar trends are observed in experiments even when the dynamics deviate from the idcalizod gradient-flow limit. In particular, while strict monolonicity may not always hold in practice due to the complexities of real-workd settings, our experiments (see section 6) confirm that steepness increases sharply during early training, enabling the generator to better capture different modes and reducing the severity of mode collapse.
Theorem 3.3 (Evolution of steepness under a symmetric mixture of Gaussians). Suppose Pdata is sym- metric misture of Gaussians
Pdata ~ 0,5N(-H,,0²) + 0.5N(u,
(6)
where μ >> a, i.e., the modes are well seporated, and that the discriminator is optimal, i.e., the discriminator consistently provides the precise moving direction for the particle. Let gr TR - R be a one-dimensional generator evolving in continuous time 2 0 according to the gradient- limit of the update
9(+A((2)= = =2((2)+ At. d)()(())
(7)
de(g)(z)))
as in algorithm 1, where
Pdata(x))
de(x) Pdaza (xx - Pa. (x)
(8)
and Pg. is the forh-forward of the standard normal NO, 1) under gt- Assume that gr is continuously differ- entiable for all t 2 O with go being an odd function and (0) > 0. Then the steepness of at 0, namely (0), satisfies the ODE
02-02)/0* (o)) 3 + (() (3) (0) J. 06(0) (xp(-n2/(202))) (,)( (0)2
9i
(9)
We remark that while theorem 3.3 focuses on the two-Ganssian case, the underlying principle theoretically exlends to distributions with more modes, though the corresponding evolution equations become signifi- cantly more complex and warrant a separate discussion. In contrast, for the simpler case of single-mode distribution, the steepness admits closed-form analytical expression. We provide detailed analysis of this setting in appendix C.
Quantitative relationship between steepness and mode mixture severity. Next, we present quan- titative results that illustrate how the steepness of the generator impacts the severity of mode mixture, as detailed in theorem 3.4. The theorem implies the following: (i) mode mixture inevitable when steepness is insufficient: The probability that particles fall into the regions between adjacent modes (i.c., the mode mixture regions) depends inversely on the steepness k of the generator. Specifically, for small k; these prob- abilities increase significantly, indicating that a less steep generator leads to more severe mode mixture. (ii) sufficient steepness minimizes mode mixture: As k increases, the probability of particles falling into mode mixture regions decreases. This reflects the ability of a steep generator to tightly map latent samples to real modes, thereby aligning generated particles with the real data distribution and reducing overlap wetw modes.
Figure 3 visually supports these implications. In the left subfigure, different generator functions with varying steepness are depicted, showcasing their ability to align latent and real distributions. The right subfigure
8

---

Published in Transactions on Machine Learning Research (03/2025)
0.40
-
5/0)=5
8d00
0.35
+10)=10
5,00 100
410)-100
g00)
0.30
MG.11
5
Opt mel
0.25
gbx)
0
Density
0.20
-5
0.15
-10
0.10
-15
0.05
-10.0-1.5-..-2.50 0.0 2.5 5.0 7.5
10.0
0.00
-15
15
-10
-5
5
10
15
x
X
Figure 3: Left: Generator functions 9 with varying steepness at æ 0. Right: The density plot of
Pg = 9#P2: with Pdata 0.5W( +).N 1). The shaded arcas represent the severity of mode mixture. Generator functions with larger sleepness exhibit less severe mode mixture. Quantitative results are delailed in theorem 3.4.
highlights the severity of mode mixture for the case N with #1 = -2 = 8, where shaded areas represent the mode mixture regions. These visualizations confirm that larger steepness reduces the severity of mode mixture, consistent with the thcoretical findings in thcorem 3.4. Practically, this underscores the importance of designing and training generators with sufficient steepness to alleviate mode mixture.
Theorem 3.4 (Relationship between steepness and mode mixture severity). Assume that the real data distribution (d)to(x)) satisfies assumption 2.1 with Furthermore, assume that the generator function g is increasing and satisfies supner Sy(x) k. Additionally, assume that
and separation condition parameter A 60.
D3 2i+1 9
+ I1+1
(10)
2
where (xx) denotes the cumulative distribution function (CDF) of the standard normal distribution N 1), and V (x) is the CDF of the distribulion (r) Then, the probubility that the particles fall inlo the interval
N-1
U #=1
[2] + 21-1 -30),
(11)
which indicates mode mixture, is at least
T ④ 1
(T/
+ Ii+1 )) Ti+1
30
$ x (
+ 1i+1
Fi+1
30 ))
(12)
2
2k
2
2k:
The role of steepness during the fitting phase. To push forward standard Gaussian to Pdata: the generator's steepness exhibit exponential growth with incrcasing mode scparation and inverse proportion- ality to mode variance. Under certain assumptions, steepness increases during training, aligning generated particles with real modes. Furthermore, mode mixture severity inversely correlates with steepness, such that higher steepness reduces overlap between modes. These findings collectively explain why mode mixture severity diminishes during the fitting phase, though often cannot be entirely eliminated.
4 The Collapse Phase: Gradient Dynamics and Detection
In this section, we examine the collapse phase, where the diversity of generated samples deteriorates as they concentrate around fewer modes. This phase emerges at the end of the fitting phase, when generated samples closely approximate the real data. We investigate the underlying mechanisms of collapse, highlighting the role of discriminator gradients in driving particle dynamics in section 4.1 and its relationship to generator

---

Published in Transactions on Machine Learning Research (03/2025)
steepness in section 4.2. Building on these insights, we introduce practical early stopping algorithm in section 4.3 to stop GAN training at the critical transition from fitting to collapse, thereby preserving diversity.
4.1 Collapse Induced by Discriminator Gradients
In this subsection, we analyze the role of the discriminator gradient |Vdd(z)/d(zz)l in the collapse phase, Collapse occurs at the end of the fitting phase, where generated samples closely approximate real data. Unlike the optimal discriminator in Vanilla GAN, which assigns uniform values of 0.5 to both real and generated samples at convergence, the optimal discriminator NSGAN exhibits a more nuanced behavior. It assigns values near 0.5 at the central regions of the modes, values close to in regions with scarce real data, and gradually transitions between these extremes.
A locally linear approximation of the discriminator. This behavior arises from two key mechanisms. First, recall that the optimal discriminator in NSGAN can be expressed as dxxx) Pdata /pa///aaaaaaa (æ) + P,(x)) (Goodfellow et al., 2014). At the central regions of mode, as discussed in section 3.1, empirical observations suggest the following dynamics: particles near the mode are attracted to it (Case 2). Conversely, overaccumulation of particles at a mode triggers redistribution mechanisms (Case 3), which drive particles toward alternative modes to ensure balanced coverage of the data distribution. These dynamics collectively lead to the assumption that (x)) R Paata (x²), which in lurn implies d. (ar) & 0.5. In regions far from the modes, (a)ta(() x 0. Due to the smoothing effect of (x)),, which spreads probability mass beyond the support of Pdata; Pz(2) remains finite. Consoquently, d. approaches 0. Sccond, the smoothing effect of Pa(2)) plays a critical role in shaping the transition between these extremes. Unlike pdata, which sharply concentrated within the modes, (a( spreads probability mass more broadly, partly due to the mode mixture effects. Please refer to appendix G for empirical evidence characterize the behavior of particle movement, we adopt a locally linear approximation of the discriminator, detailed in assumption 4.1.
toy example and real datasets. To better
Assumption 4.1 (A locally linear approximation of the discriminator). Assume that the real data distribu- tion (z)ta( satisfies assumption 2.1 with separation condition parameter A = 80. We assume that at the end of the fitting phase where generated samples closely resemble real samples, the discriminator d(xx) is of the form
22112, E Ban( 1|
d(x)=
2
8o
(13)
otherwise.
Discriminator gradients near mode boundaries. We compute |\V(())/d(x))l to analyze gradient behavior near the mode boundaries. For a point 전 located r away from H;, i.e., || x:1|2 = r, we derive [VVd())/d())l2 = 1/(4g - r). As approaches 4a, the sharp increase particles near these regions experience disproportionately large updates. This disrupts the equilibrium established during the fitting phase, pushing particles away from the boundaries. Please refer to appendix G for visualization of the discriminator gradient field in this scenario. As result, generated samples begin to concentrate around fewer modes, reducing diversity and triggering mode collapse. Monitoring |/Vd(x)/d((r) provides a clear signal of this transition, motivating the early stopping algorithm proposed in section 4.3 to prevent mode collapse and preserve sample diversity.
(Vd(())d(ax))) indicates that
The role of discriminator gradients during the collapse phase. Near mode boundaries, discrimina- for gradients [/Vd(x)/d(())) increase sharply, causing large particle updates that disrupt balance and push particles away from the boundaries. This behavior results in generated samples concentrating around fewer modes, thereby reducing diversity and triggering mode collapse.
4.2
A Local Analysis of Steepness
The steepness of the generator is a crucial metric for understanding its mapping behavior, particularly during the collapse phase of GAN training. A significant drop in steepness often signals mode collapse,
10

---

Published in Transactions on Machine Learning Research (03/2025)
where generated samples lose diversity and concentrate around fewer modes. By studying steepness, gain insights into how updates to the generator affect its ability to maintain diverse outpuis. This motivates the need for a rigorous analysis of steepness dynamics and its connection to discriminator gradients, which is the focus of this subsection.
The discriminator gradients and the generator gradients are intrinsically connected through the particle update rule described in algorithm This relationship, which governs how updates to particles propagate through the generator, is formally captured in theorem 4.1.
Theorem 4.1 (Relationship between discriminator gradients and generator gradients). Following the nota- tions in algorithm 1, assume. that after the update step, the generator optimal in the sense that go go Z, Further assume there are infinitely many particles and that the step size 0 is sufficiently small. Then, the Jucobian Jg (z) of the updaled generalor go sulisfies
Vdd
(z) = (z) + (2) Vx
(go(z)) Jgo (z), dw
(14)
where V (() is the Jacobian of the vector field /dw/dw evaluated at x.
Building on this relationship, we focus on local analysis of the generator's steepness near the collapsing mode. Specifically, we analyze how the steepness evolves after a single update during the collapse phase in theorem 4.2.
Theorem 4.2 (Steepness drops in the collapse phase). Assume that the real data distribution Pdata (x) satisfies assumption 2.1 with seporation condition parameter A = 80. Suppose the current generator is go, and the current discriminalor d(x)) salisfies the linear model assumplion 4.1 near a cerlain mode wi; specifically d(x) = 1/2 - || x.ll2/(85 (8l)) for all BAo (2x). Under the same conditions as in theorem 4.1, the steepness of the updated generator go satisfies
Sgw (2) S
(40 (40- (rp) (g),
(15)
for all latent vectors such that go(z) (xo() where [1)0(z) 12, provided that the step size < (4o - r]2 is sufficiently small.
Theorem 4.2 reveal that steepness decreases significantly within the mode's surrounding neighborhood, reducing the generator's ability to separate latent points and maintain output diversity. This provides a clear signal for detecting the onset of mode collapse. Since detecting the distance to the mode center is often impractical, the proportional drop in steepness offers a viable alternative for identifying mode collapse. By focusing on the relative change in steepness, we simplify detection and reduce dependence on precise spatial measurements, as proposed section 4.3.
The role of steepness during the collapse phase. Steepness decreases near collapsing modes, signaling reduced capacity to separate latent points. Monitoring proportional drops in steepness provides a practical method for detecting mode collapse.
4.3
The Early Stopping Algorithm
Based on the thcoretical results, we propose an carly stopping algorithm stop GAN training before collapse occurs. This algorithu monitors two critical metrics: the discriminator gradient [[Vd(=)/d(r) ll,)) which signals large updates near mode boundaries, and the generator steepness Sy(x), which reflects the generator's 's ability to map latent vectors diversely. Training is terminated if either the discriminator gradient exceeds a predefined threshold or the generator steepness exhibits a significant proportional drop compared to its previous value. Please refer to algorithm 2 for the pseudocode.
The algorithm involves three key ingrodients. (i) Two thresholds are introduced: ka/(2o) for the discriminator gradient and kg for the generator steepness. The value of [|Vd())/d(<))b at æ located 2a away from certain mode accounts for the 1/(20),, while ha is set proportional to the distance between adjacent modes. The
11

---

Published in Transactions on Machine Learning Research (03/2025)
underlying rationale is that when ||V](z)/d(z]]z is small relative to inter-mode distances, generated samples deviating from the modes can be re-attracted. However, as |/vd(a)/d(())ll> approaches inter-mode distances, particles gravitate toward alternate modes, risking collapse. The other threshold kg detects proportional drops in generator steepness, defined as AS, = (Sturrent / (ii) The (1 - 1/m)th quantile of [Vd(()/d(az)) is computed for each batch, where m represents the number of modes. This choice presumes that once a specific mode begins to collapse, it signifies the start of the GAN transitioning into the collapse phase. (iii) A warm-uu period of No iterations prevents premature stopping during the fitting phase by ignoring initial metric fluctuations.
Early Stopping of GANs (with Discriminator Gradient and Generator Steepness) Require: A GAN model including a generator Je and a discriminator dw, thresholds ka >> 0 and kg < O, the number of modes 2 the number of warm-up iterations Nw.
Algorithm 2
1: for each training ileration do
2: Train the discriminator du and the generator as in algorithm 1.
3: Compute the (1 - 1/ 1/m quantile of | /Vdd//dulz for the current batch. Let this value be Qd-
4: Compute the mean steepness Seurrent across the batch and calculate the proportional drop compared to the previous iteration as AS, = (Sgurent - /
if (g.- > k20 or AS,- < kg) and current iteration > No then
break
7: end if
8:
Update Sprev = Sq Seurrent
9:
end for
10: return The best-performing model from earlier checkpoints.
We also discuss the fcasibility of the number of modes m and estimation of the generator's steepness. In practice, many datasets used in GAN training (e-g. MNIST, Fashion MNIST, CIFAR-10) have well-defined modes corresponding to distinct classes or clusters in the data distribution. For such datasets, the number of modes is typically known or can be rcasonably estimated. In cases where the number of modes is unknown, clustering techniques (c.g., k-means or Gaussian Mixture Models) can provide a practical approximation of the mode count. While these methods may not always yield perfect accuracy, they offer reasonable baseline for implementing algorithm 2 in more general scenarios. As for the steepness, it can be computed using automatic differentiation tools in modern deep learning frameworks. Although the computation may be resource-intensive, is feasible for low-dimensional noise spaces or in settings with moderate computational budgets. For more efficient cstimation, one could subsample the noise space or use stochastic approximation techniques to estimate steepness over a representative subset of noise vectors.
5 Related Work
In this section, we highlight two key aspects of related work: (i) the phenomenon of final-stage mode collapse in GAN training (Brock et al., 2019), and (ii) related concepts with generator steepness (Odena et al., 2018; Taniclian et al., 2020; Salmona et al., 2022). We position our work within these contexts and emphasize its contributions. For a more comprehensive review, please refer to appendix A.
Final-stage mode collapse. The phenomenon of mode collapse the final training stages has been observed when scaling up GAN architectures, documented in (Brock et al., 2019). Their work, which fo- cuses large-scale experiments using BigGAN architectures high-resolution datasets, notes that "settings which were stable in previous works become unstable when applied at scale." Our study complements these findings by demonstrating that similar phenomena can occur in NSGAN at relatively smaller scales. Addi- tionally, our analysis adopts a different perspective by focusing on the generator's 's overall steepness and the Lg-norm of the discriminator gradients Vd(x)/d(z)) providing a complementary angle to the layer-specific singular value analysis employed in BigGAN. Importantly, while Brock al. (2019) primarily focuses on stabilizing large-scale GAN training to improve performance, our work emphasizes detecting mode collapse through quantitative metrics.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Related concepts with steepness. As for the notion of steepness, several related concepts have been examined, namely, the condition number of the generator's Jacobian (Odena et al., 2018) (i.e., the ratio of its largest to smallest singular value), and the global Lipschitz constant (Tanielian et al., 2020; Salmona et al., 2022) (which may be seen as the global supremum of steepness). Among these, Odena et al. (2018) introduced the condition number to assess the generator's conditioning and proposed Jacobian Clamping to stabilize training. While their work focuses on regularization techniques and overall performance, does not explicitly address mode mixture or collapse. Conversely, our work uses steepness as theoretical tool to quantify mode mixture and identify its decline an indicator of mode collapse. Regarding the global Lipschitz constant, Tanielian et al. (2020) and Salmona et al. (2022) have provided valuable insights into its role in capturing multimodal distributions. Tanielian et al. (2020) derived an upper bound on precision for learning disconeected manifolds and proposed a Jacobian-based truncation method to reject off-manifold samples. Salmona et al. (2022) demonstrated that a high Lipschitz constant is necessary to capture well- soparated modes and provided bounds for two-modal distributions. Our work complements these studies by deriving a lower bound for multi-modal distributions. In terms of technical details, our work differs in three key aspects: (i) we employ the density transformation formula as central tool, directly linking the generator's push-forward density to the noise distribution, whereas Tanielian et al. (2020); Salmona et al. (2022) rely on the Gaussian isoperimetric incquality to analyze divergence measures; (ii) we analyze the evolution of steepness during training (theorem 3.3), combined with the quantitative relationship wet steepness and mode mixture severity (theorem 3.4), we establish that mode mixture severity decreases over time. In contrast, Tanielian et al. (2020); Salmona et al. (2022) assume a static generator with a fixed Lipschitz constant, focusing on theoretical guarantees under this assumption; and (iii) we adopt a localized approach by leveraging steepness to analyze mode dynamics in specific regions of the data space. While Tanielian et al. (2020) follow a similar spirit in their rejection method for identifying and filtering out potentially mode mixture samples, our perspective allows us to observe how steepness declines near individual modes during collapse, offering complementary insights into the dynamics of mode collapse.
6 Experiments
In this section, we present the experimental results, All codes are provided in the supplementary material.
6.1 Verifying Fitting
We empirically verify the existence of the fitting phase in real-world datascts. Our experiments focus on MNIST and Fashion MNIST due to the clear separability of their modes. Detailed results, including those for Fashion MNIST, are provided in appendix G, with experimental settings and rationale detailed in appendix F.
Methodology. We train NSGAN on MNIST and analyze the generated images using a classification network gz)). Here, is an image tensor, and q(x) outputs a 10-dimensional vector (Po. Pt....,pp)) Pp), where Pe 1] represents the likelihood of being classified digit i. For each batch, we count the pairs (i, where both p. and Pj exceed 10-2 Such occurrences, visualized in heatmaps in fig. 4, indicate that the corresponding image exhibits characteristics of both modes i and j, which we interpret as mode mixture.
Results. At the beginning of training, the heatmap shows few nonzero entries, primarily due to the initial noise prior, which generates similar outputs across samples. As training progresses, more entries appear, reflecting the fitting phase, where generated samples spread to cover the space containing the modes. Off- diagonal entries, which indicate mode mixture, initially increase but then decrease in magnitude as the generator reduces overlap between modes. However, mode mixture persists even at the end of the fitting phase. These observations align with the theoretical analysis in section 3. This trend is closely linked to the behavior of the discriminator gradient and steepness, as shown in fig. 5. At the stage when more entries appear in the heatmap, the discriminator gradient's 's magnitude ude remains relatively small, while its steepness increases rapidly. This corresponds to the beginning of training, where the generator increases steepness to distribute samples across the space and capture more modes. As training continues and off-diagonal entries begin to decrease in magnitude, the steepness stabilizes, and the discriminator gradient starts to oscillate. The stabilization reflects the generator's ability to better separate modes, leading to a reduction in mode
13

---

Published in Transactions on Machine Learning Research (03/2025)
Figure 4: The logarithm of the occurrence of pairings (i.j) plus in a batch of size 256. Epochs from left to right: 0, 1, 2, 4, 8. At initialization, the noise prior results in few nonzero entries. As training progresses, more entries appear, indicating that generated samples spread across the mode space. Off-diagonal entries signal mode mixture, which decreases over time, validating the fitting. However, mode mixture persists even after fitting. Annotated heatmaps can be found in appendix G.
mixture. At the same time, the discriminator becomes increasingly effective at distinguishing samples and providing gradicnts that align with the assumptions in assumption 4.1.
6.2 Early Stopping
We present the results of applying early stopping (algorithm 2) to 3-dimensional Gaussian mixture, MNIST, Fashion MNIST, and CIFAR-10. Detailed experimental settings are provided in appendix F.
Early stopping. We train NSGAN on each dataset and record [V(<)//(<)]] at each epoch until reach- ing the maximun specified epochs. The thresholds for early stopping are determined by ka/(2o) for the discriminator gradient and ky = - for the generator steepness, where ka represents the estimated distance between two modes and is the estimated standard deviation of the data distribution. To evaluate the ef- fectiveness of early stopping, both before and after this threshold is crossed. The experimental results are shown in fig. 5, which are shown at intervals of a few epochs before and after early stopping. These intervals reflect the common practice of periodically saving model checkpoints. However, the intervals were determined post-hoc by visually identify- ing the collapse point, so they may not be eveuly spaced. Before the stopping point, the generated samples remain diverse and realistic. In the cases of MNIST and CIFAR-10, our algorithm effectively detects the epochs immediately preceding collapse. While the threshold may not always pinpoint the exact moment of collapse in other scenarios, it consistently ensures high-quality samples retained before collapse occurs. Beyond this stopping point, however, the samples frequently collapse into limited number of modes or oscillate between modes, leading to a significant loss in diversity and quality.
continue training beyond the stopping point to observe the sample quality
Comparison with FID score and duality gaps. In the evaluation of GAN performance, the metrics used can generally be classified into two categories: domain-specific and domain-agnostic. To comprehen- sively assess our proposed approach, we selected the FID score (Heusel et al., 2017) as representative of domain-specific metrics, focusing on the quality of the generated images, and duality gaps (Grnarova et al., 2019; Sidheekh et al., 2021) to represent donaannagnosticc metrics that evaluate the optimization process. We compare our proposed metric with the FID score across three key aspects. (i) In terms of applicability during training, the FID score is frequently used for retrospective evaluation, where generator checkpoints are saved periodically, and the FID score is calculated post hoc. This approach typically involves generating large number of samples (e.g., 10k or more) and feeding them through pretrained Inception network, making it computationally intensive. In contrast, training process, and is relatively computationally efficient (requiring batch-level gradient computations for both the generator and the discriminator, which is comparable to standard GAN training). Moreover, the use of early stopping with our metrics can reduce the number of checkpoint saving, providing a practical advantage resourco-intensive training scenarios. (ii) In terms of sensitivity mode collapse, in fig. 6, we show that our metric, steepness, closely aligned with FID in detecting mode collapse. Specifically, steep- ness exhibits sharp decline during collapse phases, corresponding to rapid increase in FID scores. Both
algorithm is designed to integrate directly into the
14

---

Published in Transactions on Machine Learning Research (03/2025)
#7 5% partentas of (nstunsonts
10% Parcentee of PRaseesoill Step spert 16: Sherpeess din Stasancc
50% Percent of (iddl Stop roach 90 Threns exrreded
10% harand (f
2.5
Strp reech 131 Thwereis receeded Stugrics
The mor 122: Stegprees smp
Stoupnssa
0.
2.
2.0-8
Spootss
17
6
1
2 4
9
4
4
7
Q
8
2
30
1DU
9
6
6 4
3 4
/
3
270
130
4
8
5
U
6
7
2
9
6
100 430
13
19
29
39
49
45
0
25
51
76
102
128
24
73 97 122 121
Figure 5: Experimental results of carly stopping. Each column, from left to right, represents results for the following datasets: Gaussian Mixture, MNIST, Fashion MNIST, and CIFAR-10. In the first row, the blue circled lines indicate /Vd((r//d(<=)]l) while the light. blue diamond-shaped lines represent the steepness. The stopping epoch is indicated by the black vertical line. In the second row, generated images are shown at intervals of a few epochs before and after early stopping, reflecting the common practice of saving model checkpoints periodically. Images highlighted with red frames correspond to the most realistic samples among those saved before the stopping point. Additionally, the final samples generated just prior to the early stopping trigger are included: 130th epoch for Gaussian Mixture, 45th epoch for MNIST, 98th epoch for Fashion MNIST, and 121th epoch for CIFAR-10. For Gaussian Mixture and MNIST (white dots appear along the edges of the images), these generated samples show turbulence. For Fashion MNIST and CIFAR- 10, the generated samples remain diverse and realistic before early stopping, without noticeable quality deterioration. Notably, in the MNIST and CIFAR-10 cases, our algorithm accurately identifies the epoch just before collapse. In other cases, while the threshold may not always precisely pinpoint the exact collapse point, it reliably ensures that samples remain of high quality prior to collapse. Beyond the stopping point, however, the samples often collapse to few modes or oscillate between modes, signilicantly reducing diversily and quality.
metrics effectively signal this transition. (iii) In terms of training dynamics, our metric provides insights that complement the FID score. Notably, steepness increases during the early epochs of training, corresponding to the phase where FID decreases most rapidly. This behavior may reflect the transition from prior noise to the modes of the real data distribution. Steepness then stabilizes and oscillates, which corresponds to the FID score slowly decreasing. This reflects the particles moving closer to the modes and improving the sample quality gradually. Toward the later stages of training, steepness begins to decline sharply, signaling mode collapse, which coincides with a rapid increase in the FID score. In summary, while FID remains a widely used and effective domain-specific measure for assessing GAN performance, our metric provides a reliable and complementary perspective. It is computationally efficient, sensitive to collapse phases, and offers interpretability during the training process, making it practical tool for real-time training monitoring and intervention. Additionally, in appendix G, compare [|Vd(z)/d(<))l> with the FID score, both metrics with the duality gaps and present the GAN training losses.
Validating the early stopping metric. As by-product, our discussion in section 4 supports the es- tablished practice of adding noise to the discriminator to stabilize GAN training (Wieluch & Schwenker, 2019). This stabilization mitigates disproportionately large [|Vd(z)/d(zx)1z values near mode boundaries which contributes to mode collapse. Conversely, observing //dd())/d(<>)) in this noised setting validates the effectiveness of our metric. Specifically, prior to the 54th epoch, the noise-free model generally exhibits larger values compared to the noised model. At the end of the 54th epoch, the noisc-free model collapses, with the value tending toward zero. Meanwhile, the noised model maintains stable values, as shown in fig. 7. As remark, the apparent concentration of density on the left side of the histogram at the 54th epoch is due
15

---

Published in Transactions on Machine Learning Research (03/2025)
x102
x102
x10P
x10
X10
x10
Steepniss
Steepness
2.0
Steepness
Stop epoch Steepness drop
Stop spoch 99: Threshols exceeded
Stop epach 122: Strepness drop
FID score
FID score
FID score
score
Stospnods
0.6
Steepnes
F.D score
1.0
0.4-
0.5
20
40
25
75
125
50 75
100
Epochs
Epochs
Epochs
Figure 6: The lendency of steepness and FID score for MNIST, Fashion MNIST and CIFAR-10, from left to right. Light blue diamond-shaped for the steepness and red square-shaped for the FID score. A consistent pattern is observed: the steepness initially increases and stabilizes. Subsequently, whenever the steepness decreases significantly, the FID score nearly concurrently escalate to high values, signifying a notable deterioration in sample quality. Please refer to appendix G for comparison between ||Vd())/d(<))l and the FID score.
the lime scale of the x-axis, which compresses the right tail where outliers are located. Although these outliers visually negligible, they correspond large [[Vd(z))d(()) values and account for 10% of the total area beyond the 90th percentile. See appendix G for additional results.
44
Figure 7: Histograms of the values of Vd(=)/d(<)]l and their 90th percentile across epochs. Red for the model with noise and blue for the model without noise. The noise-free GAN collapses at the 54th epoch. Preceding that, the noised model nearly always exhibits lower [Vd(<)/d(<)) values compared to its noise- free counterpart. Post that, this relationship reverses. Notably, in the noise-free model, tends towards zero, contributing to this observed divergence. See appendix G for additional results.
7 Conclusion
In this work, we propose a perspective that views GAN training as a two-phase progression from fitting collapse, where mode mixture and mode collapse are treated as interconnected phenomena. We demonstrated that mode collapse can emerge in the later stages of converging GAN and emphasized the importance of early stopping to retain sample diversity and quality. Using gradient dynamics, we analyzed how the discrim- inator gradient guides the movement, of particles (generated samples) towards modes, while the generator gradient quantifics the severity of mode mixture by measuring how closely the generator maps ncarby points in the latent space to distinct points in the output space. These insights allowed us to track the evolution of generated samples across the two phases. Our findings, validated through synthetic and real-world datasets, challenge conventional views mode collapse and lay the groundwork for future research into improving GAN training stability and performance. For additional discussions, please refer to appendix J.
Acknowledgements
We thank Luo Luo for his valuable suggestions. We also appreciate the anonymous reviewers for their constructive feedback, which significantly contributed to the revision process. We are grateful Fernando Perez- Cruz and Michael U. Gutmann for handling our submissions and for the editorial guidance. This research was supported by the National Key R&D Program of China under grant 2020YFA0711940 1902..
16

---

Published in Transactions on Machine Learning Research (03/2025)
References
Dongsheng An, Yang Guo, Na Lei, Zhongxuan Luo, Shing-Tung Yau, and Xianfeng Gu. AE-OT: A new
generative model based on extended semi-discrete optimal transport. In Internaiional Conference Learning Representations, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. International Conference on Machine Learning, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. ar preprint arXiv:1607.064 2016.
Evan Becker, Parthe Pandit, Sundeep Rangan, and Alyson Fletcher. Instability and local minima in GAN training with kernel discriminators. In Advances in Newral Information Processing Systems, 2022.
Giulio Biroli, Tony Bonnaire, Valentin de Bortoli, and Mare Mézard. Dynamical regimes of diffusion models. arXiv preprint Xix: 18491, 2024.
Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, 2019.
Xi Chen, Diederik Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Nya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In International Conference on Learning Representa- tions, 2017.
Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
Laurent Dinh, Jascha Sohl- Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference on Learning Representations, 2017.
Rick Durrett. Probubility: Theory and Eramples, volume 49. Cambridge University Press, 2019.
Jean-Yves Yves Francoschi, Emmanuel De Bézenac, Ibrahim Ayed, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. A neural tangent kernel perspective of GANs. In International Conference on Machine Learning, 2022.
Yean-Yves Franceschi, Mike Gartrell, Ludovic Dos Santos, Thibaut Issenhuth, Emmanuel de Bezenac, Mickael Chen, and Alain Rakotomamonjy. Unifying GANs and secore-based diffusion generative partiele models. In Advances in Neural Information Processing Systems, 2023.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations, 2021.
Yuan Gao, Yuling Jiao, Yang Wang, Yao Wang, Can Yang, and Shunkang Zhang, Deep generative learning via variational gradient flow. In International Conference on Machine Learning, 2019.
Arnab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi-agent diverse generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Ziv Goldfeld, Kristjan Greenewald, and Kengo Kato. Asymptotic guarantees for generative modeling based on the smooth Wasserstein distance. Advances in Neural Information Processing Systems, 2020.
Ian Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXio preprint arxiv: 701.001.001000 2017.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference Learning Representations, 2019.
Paulina Grnarova, Kfir Y Levy, Aurelien Lucchi, Nathanael Perraudin, Ian Goodfellow, Thomas Hofmann, and Andreas Krause. A domain agnostic measure for monitoring and evaluating GANs. In Advances in Newral Information Processing Systems, 2019.
Xianfeng Gu, Na Lei, and Shing-Tung Yau. Optimal transport for generative models. Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging: Mathematical Imaging and Vision, pp. 48. Springer, 2021.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Infor- mation Processing Systems, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Aduances in Neural Information Processing Systems, 2020.
Yu-Jui Huang and Yuchong Zhang. GANs as gradient flows that converge. Journal of Machine Learning Research, 24(217):1-40, 2023.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Rie Johnson and Tong Zhang. A framework of composite functional gradient methods for generative adver- sarial models. IEFE Transactions on Pattern Analysis and Machine Intelligence, 43(1) ((11)277-2, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advonces in Neural Information Processing Systems, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basod generative models. Advances in Newral Information Processing Systems, 2022.
Diederik P Kingma and Max Welling. Auto-cncoding variational Bayes In International Conference on Learning Representations, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible limesl convolutions. In Advances in Newral Information Processing Systems, 2018.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. In International Conference on Machine Learning, 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2272-2344, 1998.
Na Lei, Yang Guo, Dongsheng An, Xin Qi, Zhongxuan Luo, Shing- Tung Yau, and Xianfeng Gu. Mode collapse and regularity of optimal transportation maps. arXiv preprint 19Xiv:1902. orXio:1902.02934, 2019.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. MMD GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Sys- tems, 2017.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Marvin Li and Sitan Chen. Critical windows: Non-asymptotic theory for feature emergence in diffusion models. aixiv preprint or arXin:2033.066: 2024.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. PACGAN:
The power of two samples in generative adversarial networks. Tn Advances in Neural Information Processing Systems, 2018.
Yixin Luo and Zhouwang Yang. DynGAN: Solving mode collapse in GANs with dynamic clustering. 1EEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
Leland McInnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprinl arxiv:1802 arXiv:1802.00346, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gener- alive adversarial networks. In International Conference on Learning Represenlations, 2018.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets. In Aduances in Newral Information Processing Systems, 2017.
Albert No, TaeHo Yoon, Kwon Sehyun, and Ernest K Ryu. WGAN with an infinitely wide generator has no spurious stationary points. In International Conference on Machine Learning, 2021.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems, 2016.
Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, and lan Goodfellow. Is generator conditioning causally related to GAN performance? In International Conference on Machine Learning, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. In International Conference Learning Representations, 2016.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In Advances in Neural Information Processing Systems, 2019.
Antoine Salmona, Valentin De Bortoli, Julie Delon, and Agnes Desolneux. Can fush-forward generative models fit multimodal distributions? In Advances in Neural Information Processing Systems, 2022.
Filippo Santambrogio. Optimal transport for applied mathematicians, Birkäuser, NY, 55 63)994, 2015.
Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. arXiu preprint 0021:1140.16691 2024.
Sahil Sidheekh, Aroof Aimen, Vineet Madan, and Narayanan Krishnan. On duality gap as a measure for monitoring GAN training. In International Joint Conference on Neural Networks, 2021.
Vaidotas Simkus and Michael U. Gutmann. Improving variational autoencoder estimation from incomplete data with mixture variational families. Transactions on Machine Learning Research, 2024, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modcling through stochastic differential equations. In International Conference on Learning Represendations, 2021.
Akash Srivastava, Kai Xu, Michael U. Gutmann, and Charles Sutton. Generative ratio matching networks.
In International Conference on Learning Represenlations, 2018.
Jesse Sun, Dihong Jiang, and Yaoliang Yu. Conditional generative quantile networks via optimal transport. Tn TCLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
19

---

Published in Transactions on Machine Learning Research (03/2025)
Ruoyu Sun, Tiantian Fang, and Alexander Schwing. Towards better global loss landscape of GANs, In Advances in Newral Information Processing Systems, 2020.
Uoo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jeremie Mary. Learning disconnected manifolds: A no GAN's land. In International Conference on Machine Learning. 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with PixeICNN decoders. In Advances in Neural Information Processing Systems, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. In
International Conference on Machine Learning, 2016.
Sabine Wieluch and Friedhelm Schwenker. Dropout induced noise for co-creative GAN systems. In Proceed- ings of the IEEE/CVE International Conference Computer Vision Workshops, 2019.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Jong-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE Trunsactions on Pullern Analysis and Machine Intelligncce 42(1):27-44 45, 2018.
Yunfei Yang, Zhen Li, and Yang Wang. On the capacity of deep generative networks for approximating distributions. Newral Networks, 145:144 154, 2022.
Mingxuan Yi, Zhanxing Zhu, and Song Liu. MonoFlow: Rethinking divergence GANs via the perspective of Wasserstein gradient flows. In International Conference on Machine Learning, 2023.
20

---

Published in Transactions on Machine Learning Research (03/2025)
Roadmap.
The appendix is organized as follows:
Appendix A presents comprehensive review of the literature, covering generative models, practical considerations and theoretical understandings of GANs, the relationship ween GANs and particle models, and phased processes in diffusion models.
Appendix explains the rationale behind the choice of the latent dimension in assumption 2.2.
Appendix C provides proofs for all the theorems, propositions, and additional theoretical results not included in the main text, which include
Equivalence of NSGAN with its particle model interpretation (appendix C.1)
Properties of particle update dynamics the general result (appendix C.2)
Properties of particle update dynamics the data-dependent results (appendix C.3)
Characterization of measuring preservin maps (appendix C.4)
Steepness of measure prepuurripeeeviinggg map I-dimension (appendix C.5)
Steepness of measure presuuring maps in higher dimensions (appendix C.6)
Evolution of steepness (appendix C.7)
Quantitative results how steepness impacts the severity of mode mixture (appendix C.8)
Local analysis of steepness at collapse (appendix C.9)
Appendix D explores a class of suboptimal discriminators, complementing the theory of their optimal counterparts.
Appendix visualizes the distances between modes in datasets such as MNIST, Fashion MNIST, and CIFAR-10.
Appendix F outlines the detailed settings for the experiments described in section 6.
Appendix G presents additional experimental results, including the behavior of the discriminator at the collapse phase, verification of the fitting phase, a comparison with duality gaps, GAN training losses, and an evaluation of the effectiveness of the early stopping metric after applying techniques to mitigate mode collapse.
Appendix H discusses how the analyses in this work can be extended to other divergence GANs.
Appendix I provides visualizations of generator functions under different settings to offer intuition for section 3.2.
Appendix J shares additional intuitions and implications.
A Additional Literature Review
In this section, we provide a detailed literature review.
Generative models. Learning the generative model based on large amounts of data is a fundamental task in machine learning and statistics. Popular techniques include Variational Autoencoders (Kingma Welling, 2014; Chen et al., 2017; Razavi et al., 2019; Child, 2021; Simkus & Gutmann, 2024), Generative Adversarial Networks (Goodfellow et al., 2014; Radford et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017; Nguyen et al., 2017; Ghosh et al., 2018; Lin et al., 2018; Brock et al., 2019; Karras et al., 2020). flow-based generative models (Dinh et al., 2017; Kingma & Dhariwal, 2018; Chen et al., 2019; Grathwohl et al., 2019), autoregressive models (Van den Oord et al., 2016; Van Den Oord et al., 2016), energy-based models (Xie et al., 2018; Gao et al., 2021), diffusion models (Ho et al., 2020; Song et al., 2021; Karras et al., 2022), and other variants (Srivastava et al., 2018; Sun et al., 2022). Among these modcls, GANs' ability for rapid sampling, unsupervised feature learning and broad applicability makes them the primary focus of this study.
21

---

Published in Transactions on Machine Learning Research (03/2025)
Practical considerations of GANs. In the realm of GANs, mode collapse (Goodfellow, 2017) is ar- guably one of the major challenges which has received a lot of attention. It refers to the situation where the generator produces samples on only a few modes instead of the entire data distribution. The issue of mode collapse has been addressed mainly from three perspectives: modifying the network architecture, designing new objective functions and using normalization techniques. Regarding the network architecture, existing approaches involve increasing the number of generator (Ghosh et al., 2018) or discriminator (Nguyen et al., 2017), using joint architectures (Larsen et al., 2016). From the objective function side, various metrics such as the Wasserstein distance (Arjovsky et al., 2017), f-divergence (Nowozin et al., 2016), least squares distance (Mao et al., 2017), maximum mean discrepancy (Li et al., 2017) are employed. Normalization tech- niques such as batch normalization (loffe & Szegedy, 2015), layer normalization (Ba et al., 2016) and spectral normalization (Miyato et al., 2018) have also achieved superb empirical performance. Mode mixture (Lei et al., 2019) is another troublesome phenomenon in which the generated samples fall outside the real dis- tribution and are thus unrcalistic. Existing approaches include picking generated samples using a rejection sampling method (Tanielian et al., 2020), or generating samples with discontinuous optimal transport rather than deep neural networks (Lei et al., 2019; An et al., 2020; Gu et al., 2021).
Theoretical understandings of GANs. Another line of research approaches mode collapse and mode mixture by developing theoretical understandings for better analyzing and optimizing GAN training. These researches fall into two calegories: landscape analysis and dynamic analysis. Landscape analysis is static because it examines the results of GAN training; it ignores the interaction between the discriminator and generator during training. For instance, Sun et al. (2020) analyzed the landscape of a family of GANs called reparable-GAN They proved that the landscape separable- has exponentially many bad basins, all of which are deemed as mode-collapse. No et al. (2021) demonstrated that Wasserstein GAN with an infinitely broad generator has no spurious stationary points by modeling both the generator and the discriminator using random feature theory. Lei et al. (2019) used results from optimal transport theory to account for mode mixture. Dynamic analysis, on the other hand, considers how the discriminator and generator interact. Franceschi et al. (2022) considered GANs from the perspective of Neural Tangent Kernel (NTK). Becker et al. (2022) suggested the "Isolated Points Model" to explain the causes of GANs' instability. Another dynamical way of modeling GANs is to regard it as a particle model (Huang & Zhang, 2023; Franceschi et al., 2023). This kind of modeling is used in conjunction with Fokker-Planck equation theories by Huang & Zhang (2023) to demonstrate the convergence of GANs to the global stationary point.
Relationship between GANs and particle models. There has been an emerging trend in recent years to conceptualize GANs particle models. We present the interpretation of NSGAN as particle model in algorithm 1, which is fundamentally grounded in the work of Yi et al. (2023). Their framework rethinks Divergence GANs from the perspective of differential equations, interpreting the evolution of generated samples as particle flows guided by vector fields derived from the discriminator's gradients. Huang & Zhang (2023) examined a similar interpretation of vamilla GAN, but did not specifically discuss NSGAN. Gao et al. (2019) used a variational gradient flow approach to analyze GANs, without placing much emphasis on the connection to particle models. Franceschi et al. (2023) unified GANs within the context of particle models and interpreted GANs as "interactive particle models."
Phased processes in diffusion models. Recently, analogous phase transition phenomena, akin to those elucidated in our paper, have been uncovered in diffusion models. For example, Biroli et al. (2024) showed that the generative process in diffusion models undergoes a "speciation" transition, revealing data structure from noise, followed by a "collapse" transition, converging dynamics to memorized data points, akin to condensation in a glass phase. Sclocchi et al. (2024) found that the backward diffusion process acting after a time t is governed by a phase transition at some threshold time, where the probability of reconstructing high-level features suddenly drops and the reconstruction of low-level fcatures evolves smoothly across the whole diffusion process. Li & Chen (2024) studied properties of critical windows that are are narrow time intervals in sampling during which particular features of the final image emerge.
22

---

Published in Transactions on Machine Learning Research (03/2025)
B Choice of Latent Dimension
In this section, we provide the rationale behind our choice of the latent dimension in assumption 2.2. At the population level, Yi et al. (2023) demonstrated that NSGAN minimizes the f-divergence D (lp,) with
f(u) -u + log
u u-1 + u(1 2log 1.
(16)
Let and y be mutually singular measures on R", Yang et al. (2022) proved that
Dy(plv)) f(0) (0) > 0.
(17)
where f stands for the Fenchel conjugate of f. the latent dimension is less than n, then 90 is supported on a low-dimensional manifold, so that gowPz and v will be mutually singular. Thus there is always a positive gap in f-divergence between 9e#P; and v. In other words, ge=Ps cannot approximate V well even if the GAN model has been trained perfectly. To prevent such inherent misalignment, we assume that the latent dimension always equals Combined with the continuous data augmentation of real-world datasets, we assume that the noise prior (=) is an n-dimensional standard Gaussian distribution, denoted as (0, In), where n is the dimension of real samples.
C Proofs to Theorems
Here, we aggregate all the theorems presented in the paper and furnish proofs for some of them.
C.1 Equivalence of NSGAN with Its Particle Model Interpretation
Corollary C.1 ((Yi et al., 2023)). The update of go via applying the stop gradient operator to 2, and descending the gradient
V8 Σ [g0(2)) 2:11
(18)
in algorithm 1 is equivalent to descending the gradient
log (ga(z)))
(19)
in the original formalation of NSGAN.
Proof. We prove by directly computing the gradient using the chain rule. In fact, we have
190(2.) - 2.11²=
2 m
(ega(z,) T. ((0(2z)) - 2x)
m
Vd (Z.)
V Tago (2;)
(20)
m
du (Z:)
log (di (go (z:)))
Note that in the first equation, we implicitly use the fact that Voln 0 due to the assumption that the stop gradient operator is applied to 2,
C.2 Properties of Particle Update Dynamics The General Result
Theorem C.I. Assume that the discriminator is optimal, i.c., dx(x)) (atat((:)) (p + Py(x))) De- note r(x) = (d)ooo()))) (x) (x). At point x where r(x) R 0, x is updated following approzimately V log (r(x)). Conversely, when r(x)) > 1, is updated following approzimately V 1/r(x:)).
23

---

Published in Transactions on Machine Learning Research (03/2025)
Proof. We rewrite vd(z)/d(z) in terms of r(æ):
Vd(a))
-Pdatalz)VPp + Po ( (a)ata())
d(x:)
(Pdata(2) + Pg Pdata (2z)
<v (Prraso((
Pg (x)
(21)
Po(x))
Pdata( (Pdata (x) + Pg (x))
Vr(x)) r(a:)(1 r(a)))
When r(z) O, we have
r (x:) + +r())) ~ r(a:)
(22)
As a result,
Vd(x)) d(x)
logr(z).
(23)
When r(z) > 1, we have
ra:)(1 r(x:)) ~ r(x))²
(24)
Consequently,
Vd(x) d(x)
2
(25)
r(x)
We hereby outline the implications of this theorem. The value of log (r(x)) changes dramatically as æ decreases from to O, leading to correspondingly large magnitudes of ||V log (r(z)) ll when r(æ) 0. This indicates that in the regions where Py(x)) significantly exceeds Pdata (x), particles are propelled towards distant points. Conversely, ((11/r(z)) changes more gradually with increasing æ, resulting in smaller magnitudes of I/V( 1/rr())) when r(æ) > 1. In such regions where (x) lower than Pdaca(x), particles tend remain relatively stationary. These align with observations in section 3.
C.3 Properties of Particle Update Dynamics - The Data-Dependent Results
Proposition C.1. Assume that
1 Pdota I [(LL + 1 N(1, ([LL,, ( (1(11,,
(26)
and that Py ~ ([00,0],0.24) Let x = [2]. Then the vector field that governs particles' update is given
by
Vr(m). r(x) r(x))
(27)
where
r(x)= =
Σ exp 2.5((z) - -2a)² - (x2 - -26)2) + 5a -562)
(28)
2
((=1,,
and
Vr(x) =
5
Σ
exp (1-25((x 2a)² + (x2 26)2) 5a + 5b2)
[1 2a
(29)
I22 26
Proof. For each Gaussian distribution, the density function
(- - (1) 1- -41).
N(n [x) 2 /
exp
(30)
24

---

Published in Transactions on Machine Learning Research (03/2025)
Here, E -1], 11,1-11111, 11,,,,,,, and Σ = 0.1Ig. Therefore,
2T 0.1 exp (- 1
N((a, b). 0.112)(x))
2 0.1 ((x) - aa2 + (I2 b)9)
(31)
1
( (- 5 (2) a)² (x2 - b)²))
exp
0.2m
Thus,
Pdaca (z)
exp ( (-55 5((21 a)² + (x2 b) Σ
(32)
0.8
(c.b)
For Pz( which is normally distributed with mean [o, 0] and covariance 0.212, we have
1
Pg(x)
exp ( 1-2.(zz] + xz)).
(33)
0.4m
Combining the above results, have
r(x) -
E
exp ( 1-2,5((x) - 2a)² + (x2 26)²) 5a² + 5b²).
(34)
((t,))((+
Next, compute Vr(w:):
Vr(x) = ((.1))((
Vexp 2-2pp-55(((r) - -2a)² + (x2 - 2b)²) + 5a + 5b2).
(35)
For each term exp ( ( 1-2.(( -2a)² + (x2 - 2)) + 5a + +bb), its gradient is:
V expp ( ( (-2.5 (0) 2a)² (x2 2b) + 5a + 56²)
5exp ( 2.5((25 2a)² (x2 2b) + 5a +5²) [x) 2n
(36)
22 2h
Thus,
Vr(z) = 2
Σ exp (-2.(((1 -2a)2 + (x2 26)²) + 5a + 562)
21 2a.
(37)
42 2b
((1,))
Putting the expressions of r(æ:) and Vr(æ) together, we will have
Vr(x) r(x)(1+r(e))
(38)
When we take a closer look at the numerator Vr(æ), we observe that it is a weighted sum of the vectors originating from æ and pointing towards two times the centers of the four modes, which are (2, (2, 2)). (-2,2), and (-2,-2) Due to the exponential decay property of the exponential function, the influence of these vectors diminishes rapidly with distance. Consequently, the vector field is predominantly influenced Specifically, if we assume without loss of generality that lies in
by the mode in the same quadrant the first quadrant, the vector field will be approximately [2 x1, - r2]", up scaling factor.
Proposition C.2. Assume that
1 1 4 []((1,1),0111, + N(L (1.1,-1),0.11.)) 1 1(11111,0011 N(--1,
Pdota
(39)
and that Pg ~ 21- x 2,). Let x = [21, Then the vector field that governs particles' updale is given by
Vr(x). r(x) r(x))
(40)
25

---

Published in Transactions on Machine Learning Research (03/2025)
where
r(x) 20
Σ
exp ( [-(((221 - aa2 + (x2 - 6)9)) 1
(41)
((
and
Vr(x)
200
Σ
exp ( - -a)² 2 + (a2 - -])))
21 -a -a)
(42)
122 b)
(o.)))((=1
Proof. For each Gaussian distribution, the density function is
M(n (x) 2T del
exp (-z(0- u) 1) 1x -
1))
(43)
Here, E -1, and Σ = 0.112. Therefore,
N(ia, b). (.)1))()=))) = 2x 0.1 exp 2-0.1 ((01 - aa2 + (x2 b)()
(44)
1
exp
(x) - a)² (x2 - -)))).
0.2m
Thus,
Pdata (x)
E
exp ( 5(((((() - a)² + (x2 - -))))
(45)
0.8m
(.,,))(((+1
For Pz( which is uniformly distributed, we have
1
(z(x) =
1 222
(46)
16
Combining the above results,
r(x)
20
Σ
exp ( 5((2) - a)² + (x2 - -b))) 1mE[-22(x-222)
(47)
Now, compute Vr(x):
Vr(x)= 20
Σ
V Vexp ( - 5((1 - aa) + (x2 - -)))) 2
(48)
( (,))
For each term exp ( ( -(((1) - a)²a + ( - b) its gradient is:
[ a
V eepp - 5(() a²²² + (2:2 )) 100exp 5((2) a)2+ (x2 b)))
(49)
x2 bj
Thus,
Vr(x)
200
Σ
exp ( 5((2) - a)2 + (xz - b)?))
x1 a x2 b
[ 1=E[-22211-2221
(50)
((=),,)))
Putting the expressions of r(æ) and Vr(æ) logether, we will have
Vr(x)- ((x)(1+((()
(51)
When we take closer look at the numerator Vr(z), we observe that it is a weighted sum of the vectors originating from æ and pointing towards the centers of the four modes, which are (1, 1), (1, (-1, 1), and -1). Due to the exponential decay property the exponential function, the influence of these vectors diminishes rapidly with distance. Consequently, the voctor field is prodominantly influenced by the mode in the same quadrant as æ. Specifically, if we assume without loss of generality that lies in the first quadrant, the vector field will be approximately x1, x2] up scaling factor.
26

---

Published in Transactions on Machine Learning Research (03/2025)
Proposition C.3. Assume that
( Pdata 4' [( + 1 N(, -1).011) 1 ((((11,1,0011))) + -
1
(52)
and that Pg ~ [(11 1), I2). Let a = [2]. Then the vector field that governs particles' update is given by
Vr(wc) r1 (111+((m)) +r(x))
(53)
where
5
Σ exp (-2 -2(())
10a 1 1
10b - )
2)
5 (a
5 1)2 +
(b- 1)²) (54)
r(æ))
2
J/2
9
(+),){((+1.+1))
and
Vr(x) = 2
10a
2 2(10- 10a - 10--1- 1000-11)
9
)) + 122
10b
2 )))
5 g(a - 1)2 + (b 1)3)
21
Σ
exp
9
9
+
9
10b 1
((+,,)(())
22
(55)
Proof. For each Gaussian distribution, the density function
N(n, (x) 2 /et(() exp (- 1
D (1) -1))
(56)
Here, /a E [[1, -1,, |-1, 1], 1-1,--1-- -1]; and E = 00.1gg. Therefore,
Niia, 6). (.) = 2T 0.1 exp
2 0.1 ((01 -a)² + (I2 6)²)
(57)
exp ( (- 5 (xx - a)² + (02 - b)²)).
0.2m
Thus,
Pdata (x)
Σ
exp 5((x1 a)² + (x2 b)2))
(58)
0.8w
(111)E((+1,21))
For Py(x) which is normally distributed with mean [1, 1] and covariance I2, we have
pa(x)= 2n
exp (- 0.5 1)² + (x2 -))
(59)
Combining the above results, we have
r(x) 5
Σ
exp
9 ((m1
10a - ) +
2
22
10b -
1)
1 2
5
(a 1)²
(6 1)3) (60)
2
9
9
(
Next, we compute Vr(x):
Vr(z)
5 2
Σ
V exp ( (-2( [ 10a - 1 1 ) +
9
(x2
106 - 1 ?
1) + (a 1)2
+ 5
(b (b-1))) 132) (61)
9
g
For each term on the right-hand hand side, its gradient is:
V eexp
9
F1
10a
1
x22
10b
1
2
(a 1)2
+ g9 (b (b-1)2 1)2)
5
2
9
g
9
[11 ((m)
10a
1 2
1)
10b
1
1
2
5 (a 1)a (b 172)
[r] (10a 1)/9
-xp
+ D2
+
2
9
9
9
42 (106 1)/9
(22)
27

---

Published in Transactions on Machine Learning Research (03/2025)
Thus,
Vr(x) 45
10a
Σ exp (-9(6) ((m) 10a - 1)2
106 1 2
5
21
9 + 122
g(a
1)2
(b 1)2)
106
((1,,L)
3/2
(63)
Putting the expressions of r(z) and Vr(x) together, we will have
Vr(æ): r(a:)(1 r(x)))
(64)
When we take a closer look at the mumerator Vr(x), we observe that it is a weighted sum of the vectors originating from and pointing towards (1,1), (111/9,11 1), (1,-11/9), and (-11/9, -1199), respectively. Due to the exponential docay property the exponential function, the influence of these vectors diminishes rapidly with distance. Consequently, the vector field predominantly influenced by the mode in the same quadrant
Specifically, if we assume without loss of generality that lies in the first quadrant, the vector field will be approximately [1 æ1, -x2]] up to scaling factor.
Proposition C.4. Assume that
1 Pdota 4' 31. + M[ -3).0.1))) 1 3(113333,00133)
1
1 (11-3,-3..0.13
(65)
and that Pg 3I.) Let æ [21. Then the vector field that governs particles' update is given by
Vr(x) ([))(1+r(()
(66)
where
r(x:) = 15
Σ exp (-Ø
29 ((1
30a 3
T2
30b
3)2) 2)
)
+
(a 3)²
5
(b
(b-3)²
3)
(67)
4
6
29
29
29
29
(3.33)
and
Vr(x) = 145
30a 3
Σ
exp
21
30a 3
+
022
306
29
3
2)
+
29
(a
3)²
29
3)
305
29
3
22
29
(68)
Proof. For each Gaussian distribution, the density function is
N(μ, (x) 2 det(E)
exp (-
2 11) D I-1(xx 1
1
-1)1)
(69)
Here, /a E (3, -3] 3), (33.), -3]] and == 0.12.. Therefore,
2T 0.1 exp 2.1.1 ((2) - -a)² + (0z b)() N(ia, 6) (z)
(70)
0.2m exp ( 5((1) a)² (x2 - b)²)).
Thus,
Pdata (x)
Σ
exp
5 (œ) a)² + (xz b)2)).
(71)
0.8w
((13,333)
28

---

Published in Transactions on Machine Learning Research (03/2025)
For Pz(z) which is normally distributed with mean [3,3] and covariance 312, we have
Pa(x)) 1
exp (- 6
(11 - 3)² + (a2 -3)3)
(72)
6π
Combining the above results, we have
r(x:) 4
15
Σ exp (-6
29 ((m
30a 3.2
29 )
2/2
30b - 3 2
5
(a
+)
5
(b 3)² (73)
29
29
29
(a3))/(33,13)
Next, compute Vr(æ)):
15
29 ((0)
30a - 3)2 +
306
3)²
2)
5
Σ
V eex
(-6
(x22
+
(a 3)2
(b 3)² (74)
29
29
29
29
((t,)E(())))
For each term the right-hand side, its gradient is:
exp
29
30a
3
2
22
306 - 3
5
(a
3)2
(b 3)2
29
29
29
29
29
29
30a
3
2
306
3
2
5
(30a 3)/29]
3
exp
Ø1
29
2'2
29
29
(a
3)
29
(b 3)
222 (30b 3)/29
(75)
Thus,
145
Vr(æ)=
4
30a
Σ
exp
29
((1)
30a
29
3
122
30b
29
3
2
29
(a
3)²
29
(b
3)
31 29
300
3
((33,))))
22
29
(76)
Putting the expressions r(æ) and Vr(x) together, we will have
Vr(x) r(x)(1+r(x))
(77)
When we take a closer look at the numerator Vr(x), we observe that is a weighted sum of the vectors originating from æ and pointing towards (27/29, 27/29), (-33/29, 27/2,,,7220 27/29), 0/7/29..33/221
and (-33/29, 33/229), respectively. Due to the exponential decay property of the exponential function, the influence of these vectors diminishes rapidly with distance. Consequently, the vector field is predomi- nantly influenced by the mode in the same quadrant as z. Specifically, if we assume without loss of generality that æ lies in the first quadrant, the vector field will be approximately [27/29 - 21. 21. 27/29 x2]T, up to a scaling factor. Regarding the term 1 r(x) in the denominator, we observe that its magnitude is large when æ far from the coordinates I = 0, 42 = 0, and the centers of the modes. This increased magnit compared to the scenario in proposition C.3 explains the overall weakening of the attraction intensity near all the modes.
C.4 Characterization of Measuring- Prreriiggg-rreeviingggg Maps
Lemma C.1 ((Dureett, 2019)). Let X be a random variable taking values on R and let Fx(x) be its CDF. Then
Px 1)) X
(78)
and
(x U(0, 1),
(79)
where 11)01) denotes the uniform distribution on (0, 1).
29

---

Published in Transactions on Machine Learning Research (03/2025)
Theorem C.2. Let (()) denotes the cumalative distribution function (CDF) of (N0, and let (x)) be that of (d). If 9 salisfies 9#P> = Pdata; then g 1p h o Φ where h is a mensure-prsservnng map of [(,, 1), i..., the uniform distribution on 1),
Proof. We only need to show that V Vogogt + 1 is measure- map of ((, 1). In fact, by lemma C.1, wc have
( = (V #p = PPlaaa = (1 1).
(80)
C.5 Steepness of Measure-Preserving Map in 1-Dimension
Theorem C.3. Assume that the real data distribution pdata(() satisfies assumption 2.1 with and separation condition = 60. Let (xx) and (xx) denote the cumulative distribution functions (CDFs) of (0, and (d),a(r), respectively. Define g(x) = v (D(x)). Then, there ewists point Ia E I such that
the steepness of g at x, satisfies:
(Iit1 - x1)2 80
Sg(a.)2
min 1SKSN-1
exp
exp(-=9?
(81)
where is the (1 - t/ quantile of the standard Gaussian distribution.
Proof. Instead of computing the derivative of 9, compute that of g g By the formula for the derivative of inverse functions, we have that for any y E R,
V'(
4 (rw)
(V(())
Σ exp (- xx:3
N
(
= No 1=1
20²
.exp
exp
2
(82)
N exp ( (ri+1 - nr)2
{ ( (V-1( + xz+1) 1)2)))
max
exp
18IS
No
2
(_ - x;)²
(Ii+1
max 18i
exp
exp(g²).
80
where q is the (1- 1/ quantile of the standard Gaussian distribution. Again, by the formula for the derivative of inverse functions, there exists v. E IR such that
(I++1 - x.)²
Sy
min exep 1915-1
exp(-1²).
(83)
802
C.6 Steepness of Measure-Prest PeesuurePreeerving Maps in Higher Dimensions
The standard result in (Durrett, 2019) specifically addresses the case of lemma C.2 where K = And it can be straightforwardly extended to encompass any K.
Lemma C.2 (Durrett, 2019)). Let p(æ)de a n-dimensioonl random vector. isfy P(X D) = 1. Assume that the map
Let R
sal-
D=
+ km1
D. + R
(84)
satisfies the following requirements: for each 1 5 K, 8 L/D,, is injective and its inverse function is continuously differentiable. Then the probability density function Y xx(X) is
K Py (v) Σ (ox(x=1(1)) det (Jo, (()) (1)
(85)
k=
30

---

Published in Transactions on Machine Learning Research (03/2025)
Equivalently, for any D,
K
(r(((x)) Σ px(())-|det[[p(ee k=1
(o(os)(((())
(86)
Theorem C.4. Assume that the real data distribution p satisfies assumption 2, and that the noise prior pz(z) the truncoted Gaussion N, I,) defined on the n-dimensional ball (0) Without loss of
generalily, suppose æ, + 0 for all S S N. Let g: (,) Rn be continuously differentiable, piecewise injective function satisfying 9#P= Pdata Then, there erists point w. = such that the steepness Sz( satisfies (x.))) M, where
| 2;112
M =. V
max min exp xe ISIN
(87)
2no²
Here, - N is the mean of the mode centers, and ô exp((r2/12) V2T accounts for the truncation of the Gaussian distribution.
Proof. Let D& (1 k K) be partition of (0) such that for each S k K, glv, is injective. We regard g as the composition of two functions = 92 91- Here, 91: B,) - 1) satisfies
gi (a) gi (2.,),......, = (0, (21), (12). Φ, (xn));
(88)
where (-) is the cumulative density function of the 1-dimensional standard Gaussian distribution truncated in (-r, It is straightforward to show that the derivative of 4, has a positive lower bound, say,
p
8
exp
(89)
2m
Thus det Jy. (x)) 2 8th for any x E (0).
By lemma C.1, Pp= = n, where our focus 92: 1) + RR, which satisfies g2 -Piaaa (z). Because g2 g go and g is injective on D; (1 $ is N): conclude that 92 is injective on (Dx) (1 < K). By applying lemma C.2 to 92 and (Pxx (1 S le K), deduce that for y E (0, 1)",
the uniform distribution on 10, In the rest of the proof, we direct
K
1
Pdtta (92(y))
=
det(J (y))
sate (D.)) (92())
det(J gz (3))
(o)())) = dee( (3)) (90)
1=1
Let (0) be the n--dimensional open ball centered at the origin with radius R max |2 We consider the point yo satisfying
(2(D)) arg min || 1===2;ll2- =EBr(0) IS:5N
(91)
If there are many of them, we randomly pick one. Let x EN /./N. For this you we have
N
exp (- |A x;
Pdtta (22a0))(())) - f(xx)
(92)
N
(2ma2
202
for any À [0.2].
Hence
| w
13
(1)2)))))) 2 Paata (02(16))
22m022n/2
min ISiSN
exp
(93)
20
1
x2
Recall that we have dee((J,, (92(3)))) 2 8", where =
exp
22T
2
31

---

Published in Transactions on Machine Learning Research (03/2025)
Combine the above results and we have
[| xi
[dct(J)(8))) = (ct(( det(J,, (92(36))) 2 (V2T05)" min exp ISIN
(94)
20
If Sy(00) M, then by the property that the determinant of matrix is bounded above by the nth power of its spectral norm, we have
(3 < M
(95)
However, substituting the expression for M into this inequality leads to a contradiction with the previously derived bounds. Therefore, we conclude that the assumption (y(W) < M invalid. Let w. = yo: which completes the proof.
We remark that by choosing A 1, the lower bound becomes
| xz 113
M = 6. V2T min exp I5IS
(96)
2n
which provides useful baseline it directly relates the bound to the distance between the mean of all modes, σ, and individual modes, offering an interpretable measure of steepness.
C.7 Evolution of Steepness
In thcorem C.5, first analyze simplified sctting where Pdata composed of single mode with large variance, and derive recurrence relation for the steepness hi of the generator g at æ 0. This relation provides two key implications: (i) the steepness ke increases monotonically over time, ensuring that the generator progressively adapts to map the latent distribution Pz to the real data distribution Pastai and (ii) as + co, ke converges to k. 5 the steepness the optimal generator, indicating that the generator ually reaches the steepness required for effective mode alignment.
Theorem C.5. Assume that Pdota ~ NO, III and that the discriminator optimal, i.e., the discriminator consistently provides the precise moving direction for the particle. Then ke, the steepness of g at = at discrete time step satisfies
1
ket1 ki +
(97)
1+
Sapp(m))
where 0 < L T, and T is the marimum time. Here, yp is the probability densily function of N Ia)-
Proof. Let ((()) be the probability density function of the n-dimensional standard Gaussian distribution
(xx)= = (2 exp (- (-
2 x x).
(98)
Then the probability density function of N' 1m) is ((æ/k)/k. Let x, = k,w denotes the position of the particle at time (. Here, bi represents the steepness of the generalor function al = 0. We investigate the evolution of the particle subject to the vector field given by Vd(x)/d(x)) Assuming the discriminator is optimal, this process is governed by the following explicit formula (Yi et al., 2023):
Vr(a:))
0++1 = Et +
t= 1,2,,..., T.
(99)
((x1)(((x)) + 1)'
Here, s denotes the step size, T is the maximum time, and
r(x) pp(z/k))/k (pxx//// //:)///
(100)
This result he shown by taking the limit + +00 both sides of the recurrence relation.
32

---

Published in Transactions on Machine Learning Research (03/2025)
is the ratio of the probability density function of Pdata and Pg. By the formula of p(xx) we deduce that Vy((x) =.. Below we compute Vr(x)) by the chain rule:
ht Vy(ar/k.) (/ kt) Vpla/k))
Vr(w))
lia
4 ( ki))
(101)
ki
8 (z/ k.)
(
Using 2t = ky we derive the following recurrent formula for {hi PT=o
1 1.2
k++1 = ke + 8
(102)
( K. ((())
In the next theorem C.6, analyze the setting where Pdata is symmetric mixture of Gaussians and derive evolution equation in continuous time for the steepness of the generator function J: at 0, i.e. g
This equation provides two key implications: (i) when the third-order derivative of 9r at 0, i.., 9 (0). is small compared to (0) (for example, when 9t can be well-approximated by lincar function near x 0. as fig. depicts), the steepness 16(0) monotonically increases; and (ii) at an early stage of training, the dominant terms in the numerator and denominator of the hight-hand side of the evolution equation are (n² /0 (0)( and (0)),)), respectively. As result, (0) initially grows exponentially with rate - This suggests that when the modes are well separated (μ >> σ), the generator rapidly increases its steepness to match the target distribution.
Theorem C.6. Suppose Pdato is a symmetric mirture of Gaussians
Pdata O,5N(-H,0))+ + 0.5N(
(103)
where μ >> o, i.e., the modes are well seporated, and that the discriminator is optimal, i.e., the discriminator consistently provides the precise moving direction for the particle. Let 9t: + R be a one-dimensional generator evolving in continuous time 2 0 according to the gradient-flow limät of the update
9(-Ao(=) =9((2) + AL d((o((2))
(104)
de()
as algorithm 1, where
Ddatalxx)
de(x) (data(I) Pye (x)'
(105)
and Pg. the push-forward of the standard normal (0, 1) under gr- Assume that 9t is continuously differ- entiable for all i 2 0 with gn being an odd function and (6(0) 0. Then the sleepness of g al 0, namely (0), satisfies the ODE
di 00 -22-02)/0- (o) + (0(0) 91 (3) (0) exp(=n?((2o2))/ (g((0))3 (1)
(106)
Proof. We split the argument into the following steps.
(i) Deriving the continuous-time limit and PDE. As At + 0, the discrete update
dy (x) de (g(
(z) 9x(=) + At
(107)
yields, in the limit, the gradient-flow PDE
i d((g(())) at gx)= de(g)
(108)
Pdata (x) where dx)x Pdata (x) + Pg. (x)
and Ps. the push-forward of the standard normal under gt-
33

---

Published in Transactions on Machine Learning Research (03/2025)
(ii) Proving 9r remains an odd function. By assumption, 9o is an odd function. Because the update rule involves only derivatives and ratios that preserve the symmetry, it follows by symmetry argument that 9r remains odd for all &. In particular, this implies 9:(0) = 0 and g (0) O.
(iii) Differentiating with respect to 2. Differentiate both sides with respect to z. By the chain rule, ( d'' (g(()) (d)()
0)(()) - Oz do(g(())) (g) (d+()(=)))) s((=).
(109)
Evaluating at gives
d dt (0) = (0)
(0)3
(d. (0)) (0)
(110)
(0)
(iv) Computing Pa. (0), py. (0), P.g. (0) and likewise for Pdata- By the density transformation formula,
Ps. (x) Ps (r (x))
where p2 = NO,,
(111)
6' (x)))
(x)]]²
Setting x 0 and noting that 9r (0) (since gt is odd) yields
Ps. (0) (2)
(112)
Differentiating again and using (0) = 0 along with (0) = obtain
P/. (0) = 0,
1
g( + 92 (0)
(3)
(113)
Fgg (0)
2'
(o)(0)))
For the mixture distribution (data()), we have
1
112
Pdata (0)
exp
2
-0
Palata (0) 0,
(114)
112
0²
112
Palna (0) 0 22T
exp
20
Then, by definition,
4((0) Plaasa (2) Po (x) Pdlata (2) Dp (I)
0,
(115)
(Pdata (x) + Ps. (x))
(0) Palata (0)pg, (0) Pdtata (0)p", (0)
(116)
(pdata (0) + Ps. (0))
(v) Assembling the ODE for g(( Substitute the above expressions into
d (0) (d(()))²
dt (0 (1( (d)( (0(
(117)
to get
dt g. (0) = Plinta (0)p,, (0) Pdata (0)p", (0)
(penta (0) + Ps. (0) Pdata (0) g0(
(118)
Plugging in the explicit forms, we obtain
Jl g(()) = (42- 02) / (g. +
ol (0)
(119)
exp(-112/o1) ( (
This completes the proof.
34

---

Published in Transactions on Machine Learning Research (03/2025)
C.8 Quantitative Results How Steepness Impacts the Severity of Mode Mixture
Theorem C.7. Assume that the real dala distribulion (d)ta( sulisfies assumplion 2.1 with and separation condition A = G0. Purthermore, assume that the generator function 9 is increasing and satisfies suPDER (²)x) S k. Additionally, ossume that
T4 2it1
+ F(+1
9
(120)
2
where ((x) denotes the cumulative distribution function (CDF) of the stondard normal distribution N 1), and v (x) is the CDP of the distribulion Pdota(z)). Then, the probability that the porticles fall into the interval
N-1
U
c + 30-1 30),
(121)
which indicates mode misture, is least
N-1
④ (v
24 + 2
2141
Tj
30
1 (0'(v
24+1
))
2141 I2
33
)) (122)
2
2k
2
2k
Proof.
Given x ~ NNO,1), 1), need to calculate the probability that
x
U 9 (x; 30),9 (x141 30)].
(123)
Since 9 -(x, + xx-1)/2) is identical to its optimal counterpart, it suffices to analyze how g 'x; + 30) and g (I:41 30) deviate from this value. In other words, we only need to compute the maximum value of g (x; + 30) and the minimum value g ( 3d), as the probability that a standard Gaussian variable falls within an interval decreases with respect its left endpoint and increases with respect to its right endpoint. Using the property that supzeR S,(x) k; we have:
1 (z. + 300 S 5 9-
Ti+1
30
(124)
2
2k:
and
[i + 3i-1) g (x141- - -30 2 ((
2 Ti+1 - T2
30
(125)
2k
By summing over all intervals, we derive that the probability that particles fall into
N-1
[2; + 30, T++1 3c]
(126)
2=1
is at least
+
x1+1
I++1 I4
30
(--1(2))
24 + 2141)
I++1 24 30
))
+
2k
2h
N
+b
क -
(T)
Di + 1i+1 ))
T4- - 24
30
--1 *1V
Di + Iot1 ))
Ti+1
24
30
2
2k
2
2k:
(127)
Note that the case that = and -F = x2 = 2. this probability simplifies to
2x
30
2x
30
(128)
23:
2k
35

---

Published in Transactions on Machine Learning Research (03/2025)
C.9 A Local Analysis of Steepness at Collapse
Theorem C.8. Following the nolations in algorillom 1, assume that after the updule slep, the generalor is optimal in the sense that ger(z,) Zi Further assume there are infinitely many particles and that the step size & > 0 is sufficiently small. Then, the Jacobian Jor (2) of the updated generator go satisfies
Vdw ) (0o(z)) gg0(z))
=z) + Va
(129)
dw
where V (zV is the Jacobian of the vector field Vdy/ evaluated at x.
Proof. By the algorithm, particles are updated as
z= (Z) (Z) where Z, - =
(130)
Assuming the generator optimal after the update, the new gencrator satisfics
Vdd(go(=))
go (z) 2 = =00(==)=+ + 8 du (z)
(131)
Differentiating both sides with respect to 2, obtain
Jgw (z) = (z) - s. V. (dol
(132)
du (z))
Applying the chain rule to compute the Jacobian of the velocity field:
Vz ((doloo()))) = V Vdl. du
(90(z)) ( Jg (z).
(133)
du (g0
where Vn (vd(v///d))()) is the Jacobian of the vector field evaluated at æ.
Theorem C.9. Assume that the real data distribution (zata(z) satisfies assumption 2.1 with seporation condition A 80. Suppose the current generator is go, and the current discriminator d(x:) satisfies the linear model ussumplion 4.1 near cerlain mode H,, specifically d(æ:) = 1/2 - æ æx/l2/(80) for all x E (x.)) Under the same conditions as in theorem 4-1, the steepness of the updated generator go satisfies
Sgor (z) (40 (10 )1-))))))) (z),
(134)
for all latent. veclors z such that go(z) E (2o( where r = (1)0(z) x; ei,1,, provided that the step size s < (4o r) is sufficiently small.
Proof. Given the discriminator's 's linear behavior near the mode H; expressed as
1 x x;112,
d(x: 2 8o
(135)
for x E (2s( the gradient and Hessian (whore the Hcssian spocifically refers to 12 ((x))(x)) of d(x:) are:
vd(())
y
Vd(2z)
yy r(4o r) 3 r)
yy 2 - r)
(136)
Sor d(x:)
where == || x; 122 and y Ct Denoting the Hessian by H, the updated generator's Jacobian satisfies
(z) = (I + sH) Jo (z).
(137)
By the submultiplicative property of the spectral norm, we have
Sg (z) || (z)ll2 - + sH 11 (=)112 I sH 12 Sy (2).
(138)
36

---

Published in Transactions on Machine Learning Research (03/2025)
Next, analyze | +H l2- Since yy" T is rank-1 with the only nonzero eigenvalue r², the eigenvalues of H are
(multippicityy n 1), r(A0 -r) + 40 2r
1
(nultiplicity 1).
(139)
r(4a - r)
r(40 - r)2
The eigenvalues of I sH are therefore
+ s(Ao 2r) 1 r(4o (multiplicity n - 1), r(Ao r) r(4o r)²
8
(multiplicity 1).
(140)
Since + sH is a symmetric matrix, the spectral norm of sH the largest eigenvalue
+ s = r(40- r) r(40 r)² = - 8
s(40 2r)
(141)
(40 r))²
where we implicitly use the condition that the step size < (4x - -r)?, , so that + s sH is positive definite. Substituting back, we obtain
Sgw (z) S
(40 - rrp) (1)
(z)(
(142)
required.
D Analysis of Class of Suboptimal Discriminators
D.1 The Class of Suboptimal Discriminators
In this section, we analyze a class of suboptimal discriminators that can be expressed as
Pdata (x)
d (x) (a)aa(()) + fr(()) (o)())'
(143)
where r(æ) - (2) represents the density ratio, and f is a scalar function. The optimal discrim- inaoor, established by Goodfellow et al. (2014), corresponds to the case where f = This formulation naturally arises from the training process of the discriminator, which effectively functions as binary clas- sifier distinguishing between real and generated data. During training, gradient descent approximates the density ratio r(æ),, and deviations from the true value are captured by the function f(r(x)))
The term f(r(=)) quantifies the error the estimation the density ratio. Specifically, the suboptimal discriminator can be rewritten as:
(z) 1 f(((x))
(144)
1+ (r(a)//((())))
This highlights the role of f(r(#)) as measure of deviation from the optimal case. When f(r(x)) 1, the discriminator achieves optimality, perfectly distinguishing between real and generated samples. However, deviations from [r(()) = 1 reflect imperfections classification process. Such a modeling allows us to analyze and understand the behavior of suboptimal discriminators and their impact on the overall performance of GANs.
the discriminator, introducing bias or error into the
D.2 The Influence of the Suboptimal Discriminator to the Vector Field
In this subsection, we investigate the influence of the suboptimal discriminator on the vector field that governs the movement of particles. This analysis complements the discussion in section 3.
Proposition D.1. Assume that f E C² +xo). Then, at a point where (x)ta(x)) > 0, the cosine of the angle A between the suboptimal vector (z) and the optimal vector (z) is given by
cos 0 (x), Vd) sign [[((())) -f f+((())) (x) Vd.
(145)
r(x)
Consequently, there crists 8 > that depends on such that whenever r(x) < 6, the two vectors are in the same direction.
37

---

Published in Transactions on Machine Learning Research (03/2025)
Proof. To calculate the angle between two vectors, ignore their scalar coefficients. Therefore, only need to determine the angle between (dr) and (d. Using the results derived in theorem C.1, this calculation reduces to finding the angle between
-Pdata (2: (x))) Po V Pitaaa
(146)
and
Paata (a (z)VV(((2)) Py (a)) + (a(x:) Py P2(2:)) V Pdaat (x)
(147)
=(x2 Pdata x (py(2) (z) Vddata Pdata (
where a<x) = (r(x))) We use the same technique and divide both vectors by the scalar Paata (z) By applying the chain rule, we only need to compute the angle between
-V (x)) + V log Pelata (x) = V log r(x)
(148)
and
V log (ag)(()) - V log cx(x)).
(149)
We proceed with the final calculations:
V (ar), V [oggr(()
cos 0
(150)
V logr( Vlog(r(a))/((((()]]
For the numerator, we have Vlor((>) Vr(z)//(2)). and
[r((())) Vr(x),
V flog((r(x))) = f(r(x))
(151)
implying that V logr(z) and V Vlog(r(at//((((z))) are both parallel to Vr(æ). Therefore,
cos 0 sign 1 f(((z)))
r(x) [(r(x))
(152)
[((((x))
sign
r(x) f(((x)))
By the continuity of f"; there exists e >> such that for x E (0,e), we have [f((z)) M. As a result, for æ such that r(w) ő min (e, 22f(0)/M), we have
cos = sign sign(f(r(x)) ((a)//(((((()))))(((((()))))))))
(())) =
(153)
for some & (0, (2:)), where we use Taylor's expansion with the Lagrange remainder.
We now briefly discuss the implications of proposition D.1. Firstly, this proposition considers special case, which cos 0 = 1 for any choice of æ. Secondly, although the proposition seems to hold only for where r(x) is small, this sufficient for our purposes. In this subsection, we are focusing on the fitting phase, where r(æ:) is typically small for (x)). Finally, it may field of the suboptimal discriminator aligns perfectly with that of the optimal discriminator. However, it is important to note that while the directions of these two vector fields may be the same, their magnitudes can differ. We choose not to delve further into this topic because the magnitudes can be adjusted by varying the step sizes.
as a
counter-intuitive that the vector
D.3
The Influence of the Suboptimal Discriminator to the Evolution of Steepness
In this subsection, we investigate the influence of the suboptimal discriminator on the evolution of steepness. This analysis complements the discussion in section 3.2.
38

---

Published in Transactions on Machine Learning Research (03/2025)
Proposition D.2. Assume that Pdasa (.,,, and that the discriminator is suboptimal and takes the form
du (2)
Pd)to(
(154)
Paasa + f(r(z)) pp(x))
where r(æ: = (dataa()) PPsga())//p(aa)) and f is a function measuring the deviation of du(x:) from the optimal dis- criminator. Then hi, the steepness of g at x at discrete time step satisfies
[(r(k)0))) - r(ka (r((k)o))) r(kaao) + f(r(k(z)))
k4t1 = ke + k2 -1a)
(155)
where S LTST, and T is the mazimum lime. Here, is the probability density function of N I,) and
((0(kk
r(k,00) li,)
(156)
Proof. Let p(æ) be the probability density function of the n-dimensional standard Gaussian distribution
1 x: x) (xx) (2T)mm/3 exp (-
(157)
Then the probability density function of NO k2 In))) is p(z/k)/k. Let æ; = k,æy denotes the position of the particle at time t. Here, ke represents the steepness of the generator function. We investigate the evolution of the particle subject to the vector field given by (x) which can be written terms of r(x) as
Vr(z) t= 1.2.....,
x++1 = x + s.
(158)
(rxx))((x;)) f(r(xx)))
By the formula of ((x),, we deduce that Vip(x) = -(x)) Below we compute Vr(x:) by the chain rule
he Vv(x/kk.) f/xx//////// k2) plax/k))Vpaa//h
Vr(x)
4 <<x/kk,²²
(159)
ki R2
(xx//k:))
k2
(/x kt)
Using ki we derive the following recurrent formula for {Ke TT)
f(r(krao))) - r(k()o) (r(aao)))
kt+1 = +
(160)
ki?
1:2
r(k,zo) + ((((k())))
where
kep(kkoo//kk))
r(k)2))
(161)
Note that this proposition considers = 1 as special case, leading to the same conclusion as in theorem 3.3.
E
Disparity Among Modes Across Different Datasets
E.1 MNIST
Preprocessing. We first transform the images in MNIST by sequentially resizing the images to 64 x 64 pixels, converting them to PyTorch tensors, and normalizing the tensor values to the range of [11,
Computation. We calculate the average image tensor for cach label based on a set of 10 image tensors sharing the same label. Next, we compute the pairwise distances between these average tensors using the Frobenius norm. The resulting distances visualized heatmap in fig. 8.
39

---

Published in Transactions on Machine Learning Research (03/2025)
33.7
24.6
27.0
26.2
25.2
26.1
27.4
-00
31.7
5.0
22.5
22.4
25.0
21.3
25.1
23.5
20.3
23.2
22.5
21.1
26.1
16.3
23.8
21.8
22.7
25.6
15
25.1
17:3
25.6
18.0
19.2
26.1
20
20
10
20.2
22.5
22.0
22.7
27.9
23.0
25.1
0.0
20.9
13.1
24.4
23.4
16.6
15.6
16.2
17.6
20.4
20.4
68
16.7
10.0
16.5
20.7
11.1
-0
Figure 8: Frobenius distances between different modes MNIST. The tensor the modes are approximated by taking the average of image tensors that share the same label.
E.2 Fashion MNIST
Preprocessing. We first transform the images in Fashion MNIST by first resizing the images to 64 x 64 pixels, converting them to Tororc lensors, and normalizing the tensor values to the range of [-1,1]-
We calculate the average image tensor for each label based on a set of 10 image tensors sharing the same label. Next, we compute the pairwise distances between these average tensors using the Frobenius norm. The resulting distances are visualized a heatmap in fig.
Computation.
Tshrtttoo
28.2
43.2
Trouser
28.2
0.0
37.5
17.4
36.9
40.7
30.8
46.2
43.5
48.1
-40
24.7
37.5
Dress
20.0
17.4
32.3
0.0
30.7
40.1
23.77
43.0
37.1
44.0
25.5
Sandal
43.0
40.7
43.2
40.1
46.8
0.0
37.6
17.2
34.1
33.8
-20
Shirt
16.3
30.8
37.6
36.8
Sreaber
46.3
44.2
45.0
43.0
47.7
17.2
40.6
0.0
33.6
32.5
34.0
43.5
27.8
Ankle boot
43.2
48.1
38.7
44.0
41.1
31.08
36.8
32.5
27.08
0.0
-0
T-hhrrhopp
Tousrr
Polover
Dress
sandal
Shrt.
Snesler
Anble
Figure 9: Frobenius distances between different modes in Fashion MNIST. The tensor of the modes are approximated by taking the average of image tensors that share the same label.
E.3 CIFAR-10
Preprocessing. We first transform the images in CIFAR-10 by sequentially resizing the images 64 x 64 pixels, converting them to Torch tensors, and normalizing the tensor values to the range of [-1,1].
40

---

Published in Transactions on Machine Learning Research (03/2025)
Computation. We calculate the average image tensor for each label based set of 10 image tensors sharing the same label. Next, compute the pairwise distances between these average tensors using the Frobenius norm.
The resulting distances are visualized as a heatmap in fig, 10.
airplane
23.5
33.1
30.7
22.5
automobile
25.9
0.0
13.8
16.0
17.8
19.0
18.7
14.9
20.5
14.8
9:2
22.2
C3t
29.5
16.0
8.1
0.0
7.3
6.3
10.7
9.3
29.0
26.3
0.0
20
dng
30.7
19.0
11.1
6.3
9.5
0.0
12.8
13.2
29.9
29.6
15
horse
25.9
14.9
6.5
9.3
10.3
13.2
14.7
0.0
26.3
21.3
32.2
16.8
truck
22.5
14.8
22.2
26.3
28.5
25.6
30.0
21.3
16.8
0.0
sirplane
aucomob
cak
daer
deg
fmg
Inik
Figure 10: Frobenius distances between different modes in CIAFR-10. The tensor of the modes are approx-
imated by laking the average of image tensors that share the same label.
F Detailed Experimental Settings
All codes are provided in the supplementary material.
F.1
Verifying Fitting
Methodology. We demonstrate that the fitting phase exist in real-world datasets. To do this, we use a classification network q(x:) that takes an image tensor æ as an input and outputs a 10-dimensional vector,
(po. (po.PP....,pp))
(162)
where each E [0, 1] denotes the likelihood of æ corresponding to the ith category (e.g., the 1st category in MNIST corresponds to the handwritten digit and the 2nd category in Fashion MNIST represents pullovers). Our focus gravitates towards those p, 's that exhibit significant magnitudes. For discornibility, a threshold 7 is set to 10-2 In other words, if ps 10-2, then there is notable probability that æ belongs to the ith category. Empirical observations suggest that seldom do more than three pt's surpass the designated threshold. Hence, for any æ, may pair (i,j) when both p; and Pj exceed т. By pairing, the intuition is that such æ potentially resides between modes and j. In scenarios where only a single p; surpasses T, i is paired with itself, implying that the æ predominantly belongs to the ith category. We count the occurrences of the pairings (i,j) (0 < i.j < 9) in a batch of size 256 and visualize them with heatmaps in fig. 4, fig. 14 and fig. 15. In these figures, the value the entry (i.j) represents the logarithmically transformed occurrence frequency of pair (i,j) within a batch, adjusted by one, thereby mitigating the impact of dominant diagonal values on the colorbar.
Classification networks. We use the MNIST classification network in MNIST classification network and the Fashion MNIST classification work in Fashion MNIST classification network.
41

---

Published in Transactions on Machine Learning Research (03/2025)
Number of training runs. We conducted our experiments at least 50 times and consistently observed similar patterns across all trials. Therefore, we randomly selected two of these experiments to present in this paper.
F.2 Early Stopping
Early stopping on 3-dimensional Gaussian mixture. In this part, our codes borrow heavily from NSGAN. Both the generator and the discriminator are implemented as full-connected neural networks with SGD optimizers. Now we elaborate how to calculate the thresholds defined in algorithm 2. The dis- criminator threshold is given by (2o() We set ka = 2, the distance between two ncarest modes in the 3-dimensional Gaussian mixlure dataset. For it equals v0.0125 in our setting. Therefore the threshold is
ka/(2o) 22((2x v0.0125) x 8.9.
(163)
We set the generator threshold kg -05.. As for the warm-up training iteration parameter Nws we set it to 50.
Early stopping on MNIST. In this part, our generator and discriminator architectures borrow heavily from NSGAN MNIST. Both the generator and the discriminator are implemented as convolutional neural networks with Adam optimizers. Now we claborate on how to calculate the threshold defined in algorithm 2. The discriminator threshold is given by k2/() We set ky 33.7, the distance between two farthest modes in MNIST (please refer to appendix E). For a, we first compute the population variance of the images from each label, arriving at 10 values. Then we compute their average value, and divide this value by 64 x 64 x 1, i.c., the total number of dimensions. Thercfore the threshold
(2o) = 33.7/(2 x 0.333/44P) v 1877.
(164)
We set the generator threshold hn -0.5. As for the warm-up training iteration parameter Nw, we set it to 20.
Early stopping on Fashion MNIST. In this part, our generator and discriminator architectures borrow heavily from NSGAN on Fashion MNIST. Both the generalor and the discriminator are implemented as convolutional neural networks with Adam optimizers. Now we elaborate on how to calculate the threshold defined algorithm 2. The discriminator threshold is given by ka/(2o). We set ka = 48.1, the distance between two farthest modes in Fashion MNIST (please refer to appendix E). For o, we first compute the population variance of the images from each label, arriving at 10 values. Then we compute their average value, and divide this value by 64 x 64 x i.., the total number of dimensions. Therefore the threshold is
ka/(20) = 48.1/(2 x /0.33/642) 2679.
(165)
We set the generator threshold hy = -055. As for the warm-up training iteration parameter No, we set it. to 50.
Early stopping on CIFAR-10. In this part, our generator and discriminator architectures borrow heav- ily from NSGAN on CIFAR-10. Both the generator and the discriminator are implemented as convolutional neural networks with Adam optimizers. Now we elaborate on how calculate the threshold defined in al- gorithm 2. The discriminator threshold is given by ka/ (2r). We set ka 38.0, the distance between two farthest modes in CIFAR-10 (please refer to appendix E). For or, we first compute the population variance of the images from each label, arriving at 10 values. Then we compute their average value, and divide this value by 64 x 64 x 3, i.e., the total number of dimensions. Therefore the threshold is
ka/(2o) 38.0/(2 x
/ (0.23/(642 3)) 4391.
(166)
We the generator threshold kg -0.5. As for the warm-up training iteration parameter No, we set it to 50.
42

---

Published in Transactions on Machine Learning Research (03/2025)
Number of training runs. On all of the datasets mentioned above, conducted our experiments least 100 times. We observed similar patterns across all trials, although the point at which the GANs collapsed varied. Therefore, we choose to present those that collapsed before a certain threshold to ensure consistency in our reported results. It is important to note that the generated samples eventually collapsed in our experiments, either sooner or later, without contradicting the lindings in our paper.
G Additional Experimental Results
G.1 The Behavior of the Discriminator at the Collapse Phase
In this subsection, elaborate on the optimal discriminator's behavior outlined in section 4.1.
Verification toy example. We first consider the following synthetic dataset
Pdata 1 (1111,,00011255 + (
1 N( + [((11,,1,0001115))) 1,0 + 1 ( [((11,,11,,001223
(167)
and train the discriminator until optimal. We plot the values of the optimal discriminator in fig. 11. We observe that discriminator values are close 0.5 (red) the central regions of the modes and approach zero (dark blue) in regions far from the modes. The contours between these regions are approximately equally spaced, indicating linear decline in discriminator values. The distance from the mode centers to where discriminator values vanish is approximately 0.4, aligning with the predicted 4o 4 v0.0125 v 0.45 in assumption 4.1.
0.42
0.36
0.50
0.00
Figure 11: The values of the optimal discriminator. The discriminator values are close 0.5 the central regions of the modes (i.e., #1]) and vanish in the regions far from the modes. Between them, the discriminator values smoothly change from 0.5 to O.
Verification in real datascts. We justify assumption on MNIST and Fashion MNIST. We use saved checkpoints prior to mode collapse as the basis for our analysis, which are the 43rd epoch and the 124th epoch, respectively, for MNIST and Fashion MNIST, under the same setting as in figs. 5, 6 and 17 to 19. Specifically, randomly sampled 10k points from MNIST and Fashion MNIST, and applied the k-means algorithm to cluster these points into 100 clusters, treating the cluster centers as representative modes of the data. We then computed the discriminator values at these cluster centers and took their average as the baseline measurement. To evaluate the impact perturbations, we introduced random standard Gaussian noise with varying scales to these cluster centers. The maximum perturbation scale was determined according the 4a 4a/din /im criterion described in assumption 4.1 where a represents the average population variance of the labels in the dataset, and dim is the dimension of the standard Gaussian noise. This ensures that
43

---

Published in Transactions on Machine Learning Research (03/2025)
the largest expected distance to the cluster centers equals 40. Finally, plotted the mean discriminator values as function of the perturbation scale in fig. 12, providing visualization of how the discriminator responds to deviations from the original mode centers. The results demonstrate that the discriminator values approximately follow a linear trend near the mode centers, decreasing gradually from 0.5 to as the distance from the mode centers increases.
0.6
discriminator values
Mean dis discriminator values
0.4
0.4
0.3
0.3
0.2
0.1
0.0
0.025 0.030 0.035 Perturbation scale
0.015
0.035
Perturbation scale
Figure 12: Mean discriminator values as a function of distance from the mode centers for MNIST (left) and Fashion MNIST (right). These results support assumption 4.1 by illustrating how the discriminator values vary with increasing distance from the mode centers. The mode centers are estimated using the k-means algorithm applied to each dataset. To introduce perturbations, random standard Gaussian noise with varying scales is added to these cluster centers. The maximum perturbation scale is set 40 /dimm, where o is the average population variance across dataset labels, and dim 64 64 represents the dimensionality of the perturbation. The plots reveal that the discriminator values approximately follow linear trend near the mode centers, decreasing from 0.5 to 0 as the perturbation scale increases.
Visualization of the discriminator gradient field. We plot the discriminator gradient field Vd(z)/d(a under assumption 4.1 in fig. 13. In the left pancl, the unscaled vector field demonstrates the existence of particles that experience large-magnitudee gradients, which could propel them away from one mode to another. In the right panel, the zoomed-in view shows the vector lengths reduced to one-tenth of their original size, yet some vectors still pass through the centers of the modes. This serves to verify the discussions in section 4.1.
G.2 Verifying Fitting
Annotated heatmaps for MNIST. We verify the existence of fitting on MNIST. Annotated heatmaps are employed to track the evolution of pairings (i,j) occurrence within batches of size 256. The values depicted in these heatmaps represent the logarithm of occurrence counts plus 1, with darker colors indicating higher values. Each heatmap includes epoch numbers ranging from 0 to 38 displayed at the bottom. Initially, the heatmap has few nonzero entries, indicating limited sample diversity at the beginning of the fitting phase. As training advances, more entries became nonzero, reflecting a broader distribution of generated samples across the mode space. Notably, the values of off-diagonal entries signifies the severity of mode mixture, which gradually decrease over the course of training, validating the fitting phase. However, the issue of mode mixture persists even the end of fitting. By the 36th epoch, the heatmap only has two nonzero entries, suggesting the collapse phase, where the generated samples become less diverse and concentrate around few modes. These observations provide empirical evidence for our proposed two phases of GAN training.
We verify the existence of fitting in Fashion MNIST using an- notated heatmaps. The heatmap values are the logarithm of pairings (i,j) occurrence plus 1 in batches of size 256, with darker colors indicating higher values. Each epoch is divided into 5 collections of batches, denoted as e, b where e is the epoch and b the batch collection within the epoch. Initially, there are only two nonzero entries, which suggests limited sample diversity. As training progresses, more entries become nonzero, indicating broader sample distribution across the mode space during the fitting phase. Notably,
Verifying fitting in Fashion MNIST.
44

---

Published in Transactions on Machine Learning Research (03/2025)
2.0
0.90
1.5
0.450
1.0
0.375
0.5
0.300
0.0
0.225
00.
0.150
1.0
-1.05
0.075
1.5
2.0 2
-1.10
0.000
-1
1
2
Figure 13: The discriminator gradient field Vd(x)/d(z) under assumption 4.1 on the toy example. The left panel shows the unscaled vector field, where particles experience large-magnitude gradients, with vectors passing through the mode centers and pointing outward. This could trigger propulsion away to another mode, The right panel provides a zoomed-in view around the bottom-left mode, with the vector lengths redueed to one-tenth of their original size. Even with this scaling, some vectors are seen passing through the centers of the modes.
unlike MNIST, the phase of fitting in Fashion MNIST occur quickly, evidenced by the rapid stabilization of off-diagonal values. It is important to note that the large values in off-diagonal entries do not neces- sarily imply severe mode mixture. For example, "T-shirt", "Pullover", and "Shirt" are frequently confused in Fashion MNIST classification tasks.
45

---

Published in Transactions on Machine Learning Research (03/2025)
Figure 14: Annotated heatmaps for verifying fitting in MNTST. The values the logarithm of the occurrence of pairings (i.5) plus 1 in batch of size 256. Darker colors indicate higher values. The epochs, ranging from 0 to 38, are displayed at the bottom of cach heatmap. Initially, there are few nonzero entries, suggesting limited sample diversity. As training progresses, more entries become nonzero, indicating wider sample distribution across mode space, which corresponds to the fitting phase. Off-diagonal entries reflect mode mixture, which diminishes over training, confirming the fitting phase. Remarkably, mode mixture persists even at the closure of the fitting phase. Note that by the 36th epoch, only two entries remain nonzero, indicating the collapse phase.
46

---

Published in Transactions on Machine Learning Research (03/2025)
Figure 15: Annotated heatmaps for verifying fitting in Fashion MNIST. The labels 0 to 9 mcan "T-shirt/top" "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", and "Ankle boot", respectively. The values are the logarithm of the occurrence of pairings (i.j) plus 1 in batch of size 256. Darker colors indicate higher values. Each epoch is equally divided into 5 collection of batches. The label "e, 6" at the bottom of each heatmap denotes the bth collection within the eth epoch. Therefore, the heatmaps displayed are for the first epochs only. Initially, the few nonzero entries indicate limited sample diversity. As training progresses, more entrics became nonzero, reflecting a broader sample distribution across the mode space, which corresponds to the fitting phase. Unlike MNIST, the phase of fitting in Fashion MNIST take place rapidly because the off-diagonal values stabilize quickly. It is important to note that the large values of some off-diagonal entrics do not necessarily imply severe mode mixture; for instance, "T-shirt", "Pullover", and "Shirt" are often confused in Fashion MNIST classification tasks.
47

---

Published in Transactions on Machine Learning Research (03/2025)
G.3 Illustration of Particle Dynamics in Real Dataset
In this subsection, use the CIFAR-10 dataset to illustrate particle dynamics in real datasel, particularly focusing on their relationship with mode collapse.
Methods. To investigate particle dynamics, we set fixed noise vectors and track their corresponding gener- ated images (referred to as particles) across different epochs. For each particle, we compute two key metrics: (i) the discriminator gradient norm, (Vd())/d(x)ls and (ii) the steepness of the generator, [1))()) We visualize the generated images across epochs and analyze how these metrics evolve during training.
Results. From the visualizations, we observe that particles initially appear vague and gradually align with the modes, which supports our discussion in section 3.1. Over time, they move closer to the modes, described in section 3.2, and eventually collapse. For the steepness metric, Il (x)]l2; in the early stages of training, likely due to the generator aligning the noise prior with the modes, consistent with the analysis in theorem 3.3. Later, it stabilizes but drops sharply to near zero at collapse theorem 4.2. In contrast, the discriminator gradient norm, [Vd(z)/d(z)]l> shows oscillatory behavior throughout training but spikes sharply at the point of collapse, aligning with our findings in section 4.1. These results provide particle-lvee evidence that our theoretical findings approximately hold real datasets, further validating our analysis.
find that it increases
10a
[[vo(x//bb01>>
10a
10a
Steepness
10
100
10a
10-1
10a
0
20
40
60
80
100
0
20
40
60
80
100
Epoch
Epoch
Figure 16: Visualization of particle dynamics the CIFAR-10 dataset. The top row shows the evolution of particles (generated images) across training epochs, specifically at epochs 0, 4, 9, 14, ..., and 99. The bottom left plot illustrates the discriminator gradient norm, [Vd(r)/d(2)]l for cach particle over training epochs (indicated by different colors), while the bottom right plot shows the steepness of the generator, [|,g(x) These visualizations highlight the transition of particles from initial noise to alignment. with modes, and eventually to collapse, validating our theoretical findings.
48

---

Published in Transactions on Machine Learning Research (03/2025)
G.4 Comparison with the FID score
We present comparison between [V][))/((=]]> and the FID score in fig. 17, complementing the analysis in section 6, where steepness was compared with the FID score. A connection emerges between the behavior of |/Vd(=)/d(<))l>> and the FID score: During the carly training epochs, [Vd(2)/d((x)) remains small, corresponding to a rapid decline in the FID score, As training progresses, [Vd(a)/d(()) begins to oscillate while gradually increasing, with the FTD score decreasing at a slower rate. This trend likely reflects the discriminator's behavior approaching the condition implied by assumption 4.1. For MNIST and CIFAR-10, the stopping point is triggered by proportional drop in steepness, without concurrent rapid increase in both //V(())/ /d(2)) and the FID score. In the case of Fashion MNIST, our early stopping algorithm halts training when \Vd())/d(<))l> surpassos a predefined threshold, effectively preventing a deterioration in sample quality.
Xx00
Parcentile of [
90% Parcentlle of ||aalxolii|
90% Parcentile of |[abx|/tx
epoch 46: Steepriess drop
Stop epech 99: Threshold excesdeddexeeedd
2.5
Stop epach 122: Steepness
FID score
[[rote kllls
12.0
score
RID score
2.5
1.0
1
0.5
10
20
30
40
30
25
30
75
100
125
0.0
25
30
73
100
125
Epachs
Epochs
Epochs
Figure 17: The tendency of [Vd(2)/d(<)]l and FID score for MNIST, Fashion MNIST, and CIFAR-10, presented from left to right. Blue circled for [Vd(()/d(()]l and red square-shaped for the FID score. For MNIST and CIFAR-10, since the stopping point is primarily triggered by steepness dropping below the threshold, we do not observe a concurrent rapid increase in ||Vd(=))d((=)]z and the FID score. In contrast, for Fashion MNIST, when || [V(())/((x)]> surges past its threshold, we halt training at a point where deterioration in sample quality is prevented.
G.5 Comparison with the Duality Gaps
We compare our metrics, [V((=)/((<))) and steepness, with the duality gap (Grnarova et al., 2019), along with its improved counterpart, the perturbed duality gap (Sidheokh et al., 2021). We first briefly introduce the two metrics, and then show the results in fig. 18.
Duality gap. The duality gap is an optimization concept that measures the difference between the pri- mal and dual forms of a problem. In GANs, quantifies the suboptimality of the current generator and discriminator. For parameters (0g, 02) a given iteration, the duality gap is defined as:
DG 0a) max F(Bg. 0')) min F(B,, 0a), o,
(168)
where O₂ and O, are the parameter spaces for the discriminator and generator, respectively, and F the objective function of the Vanilla GAN: F(ey. 0a) Exrppara [log d(x)) + Ez<p, [log(1 d(g(z)))). In practice, Grnarova al. (2019) proposed estimate the duality gap through the following steps:
1. Train the GAN to itcration f, obtaining parameters (0), (.).
2. Find the worst-case discriminator and generator by optimizing one while keeping the other fixed:
g
gworst arg max F(B), 82), o
gwort arg min F(O,, 0%) e c
(169)
g
3. Estimate the duality gap DG 03) Rs F(O gworst) F(O 0%)
49

---

Published in Transactions on Machine Learning Research (03/2025)
Perturbed duality gap. The perturbed duality gap; introduced by Sidheekh al. (2021), improves upon the standard duality gap by more effectively distinguishing between Nash and non-Nash critical points. This metric performs local perturbations to the parameters (0y, 0%) with Gaussian noise before the second optimization step, helping the model escape from saddle points. This ensures the subsequent optimization does not get stuck in suboptimal regions.
Results. We compare /d(x)//(<>)l and the steepness with the vanilla and perturbed duality gaps across three datasets: MNIST, Fashion MNIST, and CIFAR-10, as shown fig. 18. In the first row, [|Vd(=)) /d(x)l2 is plotted alongside the duality gaps. In the second row, the steepness is compared against the duality gaps. Prior to the collapse, both duality gaps decrease with oscillations, while || [Vd(())/d(<r)]lb increase with oscillations. After collapse, the vanilla duality gap drops to zero, mirroring the behavior of steepness. In contrast, the perturbed duality gap oscillates, making it difficult to pinpoint the beginning of collapse. These results demonstrate the robustness of our metrics, which consistently and clearly detect the collapse phase.
x10²
x10
x100
X10
x10
10
90% Parcantlle of | Rdixxidtxxi
90% Percentlic of [Rdox
90% Percentile of [Rdxx/ddiiill
3.0
4
Stop epach Steepness drop
Vanilla duality gap
Vanilla duality gap
epoch Threshod exceaded
3.0
Vanilla duality gap
epoch Steepness drop
2.
Perturbed duality gap
alllly gap
Perturbed duality gap
Quality gap
Perturbed dualiyy gap
2.0
3-
1.5-
2-
1.0
1.0
1.0
0.5
0.5-
0.5
0
30
50
75
125
0.0
0.0
100
125
0.0
Epochs
Epochs
Epochs
Steepmess
1.0
Steepness
3.5
2.0
Steepness
Vanilla duality gap
Stop epoch 46; Steepness drop
0.8
Stop epoch 99: Thresheld exceeded
Vanilla duality gap
3.0
Vanilla duality gap
Stop epach 122: Steepness drop
2.5
Perturbed duality gap
Duality gap
Stespnos
Perturbed duality gap
2.5
1.5
Perturbed duality gop
2.0
-1.5
-1.0
0.2
0.5
0.5
0
10
20
30
40
50
0.0
0
25
50
75
100
125
0.0
0.0
25
50
75
100
125
0.0
Epochs
Epochs
Epochs
Figure 18: The first row compares [Vd(=)/((<> with the duality gaps for MNIST, Fashion MNIST, and CIFAR-10, from left to right. The second row compares the steepness with the duality gaps for the same datasets. Blue circled represents |/V((=)/d((=)])> light blue square-shaped represents steepness, while orange triangle-shaped and orange square-shaped represent the vanilla. and perturbed duality gaps, respectively, Prior to the collapse, //V(()/d((z))) exhibits a similar trend with the perturbed duality gap. After collapse, the vanilla duality gap often drops to zero, mirroring the behavior of steepness, while the perturbed duality gap oscillates, making it difficult to determine whether collapse has occurred.
G.6 GAN Training Losses
We present the loss curves under the same experimental settings as in figs. 5, 6 and 17 to 19. The loss functions are defined
Ca. = -Eg-pinn
Ez-p, Ez-g,.log(1 Lyp EEzop, (Iog(d
1 - (g)
(170)
and are computed using torch. nn. functional. for discriminator losses, while the red squarc-shaped markers represent generator losses. Notably, after the collapse, the discriminator and generator losses either escalate to large values or drop to zero. While this divergence behavior is consistent across experiments, it poses challenges when used as early stopping
The blue circled markers stand
50

---

Published in Transactions on Machine Learning Research (03/2025)
criterion, particularly in encoding practical termination condition. The simplest approach is perhaps monitoring both losses and terminating when they increase or decrease beyond certain threshold. However, this strategy comes with several limitations: (i) the interpretability of GAN losses is limited. GAN losses do not directly reflect the quality of generated samples. For instance, an increase in discriminator loss might indicate overfitting rather than a collapse of generator performance, while an increase in generator loss could stem from the discriminator becoming overly sensitive, leading loss fluctuations without corresponding drop sample quality. (ii) threshold values may be difficult to generalize across experiments. The dynamics of GAN training are highly sensitive hyperparameters such as learning rates, batch sizes, and architecture choices. Consequently, the patterns of loss escalation can vary significantly, making it challenging to define a universal threshold for early stopping. Nevertheless, it is interesting to see if GAN loss functions can also be used for designing alternative early stopping criteria that leverage their dynamic behaviors to predict mode collapse.
Discriminator loss
Discriminator loss
Discriminator loss
10:
Generator loss
Collapse spoch 46
Collapse epach 128
10
Collapse spoch 122
10a
Losses
20'
10-4
101
10-
10a
109
20
30
40
50
20
40
60
80
100
120
140
0
20
40
60
1000
120
Epochs
Epochs
Epochs
Figure 19: Loss curves of the discriminator and generalor in experiments on MNIST, Fashion MNIST, and CIFAR-10 (from left to right). Blue circled represent discriminator losses, while red square-shaped represent gencrator losses. The black vertical lines indicate the stopping epochs determined by our carly stopping algorithm. The exact collapse epochs are 46, 128 and 122, for the respective datasets. Notably, after the collapse, both losses either escalate to large values or drops to
G.7 Impact on the Early Stopping Metric after Applying Techniques to Mitigate Mode Collapse
In this subsection, we validate our early stopping metric's effectiveness by demonstrating that injecting noise into the intermediate layers of the discriminator combats mode collapse and pushes back the metric.
Figure 20: The generated images from the noise-free GAN and the noised GAN. Upper: Noise-free GAN. Lower: Noised GAN. The noise-free GAN collapses the 54th epoch, whereas the noised GAN consistently produces high-quality images.
51

---

Published in Transactions on Machine Learning Research (03/2025)
Experimental setup. We devise two generator models of identical architecture and implement two dis- criminators, one adhering to the original design (which we will refer to as "noise-free") and the other modified to incorporate Gaussian noise with a standard deviation of 0.1 before forwarding the input to the subsequent layer (which we will refer to as "noised"). Both generators and discriminators are initialized using the same random seed. During training, the four networks are concurrently trained, with each generator paired with discriminator. We present the generated images of the two models on Fashion MNIST fig. 20 and histograms of [Vd())/d(())l l in fig. 21.
16
20
22
24
2B
30
32
34
36
38
40
42
44
46
48
50
52
54
56
58
Figure 21: Histograms of the values of [Vd(x))d((x) lla and their 90th percentile across epochs. The epochs are displayed at the bottom of each histogram. The r-axis represents [Vd(<)//(<>){b values on a logarithmic scale, while the y-axis denotes density. Results are differentiated by color: red for the model with noise and blue for the model without noise. Preceding the 54th epoch where the noise-free GAN collapses, the noised model nearly always exhibits lower [[Vd(=))/(()) values compared its noise-free counterpart. Post 54th epoch, this relationship reverses. Notably, in the noisc-free model, [[Vd(z))d((<))) tends towards zero, contributing to this observed divergence.
The noisc-frec GAN collapses the 54th epoch, while the noised GAN consistently generates high-quality images. The introduction of noise results in an overall decrease in the [Vd(()) compared to its noise-free counterpart before the 54th epoch. After the 54th epoch, the opposite trend is observed,
Results.
52

---

Published in Transactions on Machine Learning Research (03/2025)
attributed to the vanishing of |d( in the noise-free GAN. This indicates that the strategy of in- jecting noise to mitigate mode collapse leads to overall decrease in our proposed metric, thereby validating the effectiveness of the metric.
Furthermore, from our experiments, we observe that collapse is avoided altogether the noised GAN. Specifically, we trained the noised GAN for 1000 epochs and did not detect any signs of collapse throughout the training process. This supports the understanding that adding noise to the discriminator stabilizes GAN training by mitigating extreme gradient magnitudes near mode boundaries, which are primary contributor to collapse. Regarding the early stopping algorithm, it remains effective in the noised GAN setting. The algorithm identifies a point in the training process where sample quality and diversity are well- retained. However, we do not observe significant sample deterioration or signs of subtle collapse even after this identified point. This suggests that the early stopping algorithm's role in the noised setting may focus more on preserving computational efficiency rather than strictly avoiding collapse. A possible explanation for this behavior is that the added noise reduces the discriminator's capacity to provide overly precise gradients, which prevents the discriminator from overfitting specific modes. This stabilization mechanism ensures that the generator continues to explore and cover the data distribution effectively, even in later training stages. As a result, even subtle collapses do not scem to occur.
H Extension to Other Divergence GANs
In this section, we outline how to extended to other Divergence GANs. We focus on the S-GAN proposed in (Nowozin et al., 2016) with the f-divergence defined as
( (2:) (( Q0(Ø) )dx.
Dy(QollPdata) = Pduta (x))
(171)
The variational lower bound of D Dy(Qollpdass is used as the training objective:
F(O:w) Eerp [9s (V)()) Eu~q, (fr()_((((((((((a)))
(172)
Here, f the Fenchel conjugate of f. gs is analogous to the generator and Vn is similar to the discriminator. We consider its variant where the objective function of the generator is modified to
-o (9s (V)()))),
(173)
while the objective function of the discriminator remains unchanged.
General methodology. The key to analyzing Divergence GANs is their interpretation as particle models. The update of the generator Qa can be recasted as: (i) generate particles Z, = Qo(z;;; and (ii) update the particles 2, = Z, + 9 (V.( to the Mean Square Error (MSE) loss betweeen 2,'s and g(z:)'s.
and (iii) update 0 by descending its stochastic gradient with respect
Fitting phase. We may plot the vector field of (Z instead of the original vd(x)/d(z)) to visualize the updating process of particles, which promotes the fitting of the modes. Only theorems 3.3 and 3.4 in section 3.2 needs to be modified to accommodate the desired Divergence GAN.
Collapse phase. In section 4.1, apart from modifying the update formula for particles, a more appropriate model for the discriminator needs to be established and a new threshold may be developed on the basis of it.
Visualizing Generator Functions
This section visualizes generator functions g that satisfy 9#P: Pasta: where Pa NO, 1) and Paata is Gaussian mixture, as shown in fig. 22. For qualitative effects of the parameters in Pdacas please refer to table 3. We then discuss about how to plot fig. 22. While & can be computed in MATLAB using the
53

---

Published in Transactions on Machine Learning Research (03/2025)
built-in function normedf, v- typically necessitates solving non-linear equation each evaluation point. To mitigate computational expenses, we choose to calculate the inverse of g. which is g = 4 o V. In this context, v can be computed by employing gmdistribution to construct a Gaussian mixture model, followed by utilizing cdf to assess the cumulative distribution function (CDF) the model at a specific point. To generate a plot of 9; a mere interchange of the and g (x)) the plot function suffices.
Figure 22: The functions g that satisfy I#Ps Pdatas where Pdata a Gaussian mixture. First: Varying
the mcan μ.. Second: Varying the variance o². Third: Varying the mixing coefficients {ai]i- Fourth: Varying the number of Gaussians Please refer to table 3 for a detailed description.
Table 3: Qualitative effects of the parameters Pdata ~ a)N(H,, + ...
....... + anN((nn on
Parameters
Qualitative Effects on g
Means {4:),-1
Larger || 14441122 increases the magnitude of g between the two modes.
Variances o²
Larger o² increases the asymplotic slope of g as
Mixing coefficients {a:)=
Different combinations of a; incline towards specific modes.
Number of Gaussians n
Larger n increases the number of segments in g.
J Discussions
In this section, we provide additional intuitions and implications.
In terms of applicability scope, our theoretical findings are primarily derived from Divergence GANs, specifi- leverage their particle model interpretations. While Divergence GANs represent significant category within GANs, they do not encompass some prominent GAN models, such Wasser- stein GAN with gradient penalty and MMD GAN. Exploring how our theoretical findings can be extended to incorporate these Integral Probability Metric (IPM) based GAN variants presents an intriguing avenue for future research.
cally NSGAN, where
Regarding the proposed two phases, it is important note that not all Divergence GANs may fit neatly into the this characterization. While we often observe such empirical patterns, acknowledge the possibility that when networks are not well-initialized or when advanced techniques are used, GAN training may deviate from the fitting phase entirely. However, these inquiries may spark independent interests and are beyond the scope of our study.
In our numerical experiments, we used relatively small-scale real-world datasets compared to modern datasets. This choice was deliberate as we aimed assess the effectiveness of our early stopping algorithm in detecting the transition from fitting to collapse phases. Modern datasets often comprise exponentially more modes, which could potentially limit the efficacy of our algorithm, particularly considering that a algorithm takes the number of modes as an input parameter.
54


Reviews:
[{"review_id": "wOuN1mRClv", "reviewer": "Reviewer_JU7e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n- This paper presents some compelling and interesting findings about the training dynamics of GANs (in particular, around the phenomena of mode collapse and mode mixture) which are to my understanding, novel. The gradient based tools described in this work appear to be well suited to detect mode collapse, and have already proved fruitful in identifying successful periods of GAN training prior to collapse. \n- It also contains a large quantity of theoretical and experimental results, some of which I believe could be of interest to the broad machine learning community. \n- The methods described here could certainly lead to many interesting follow up works to better understand the dynamics of training GAN model. \n\n## Weaknesses\nMy main concerns for the paper are regarding support for the main claims of the paper. Many of these concerns are related to weaknesses in paper organization and clarity. I have categorized my concerns below. \n\n### Support for claims.\n- One of the main claims of this paper is that GAN training can be understood as three sequential phases of training, designated as fitting, refining, and collapsing. I found that there is very compelling evidence to support the claim that mode collapse often follows training that we would otherwise deem successful (as verified by the early stopping algorithm), but very little evidence to support fitting and refining as independent and sequential phases of training. \n\t- The empirical evidence to support the existence of a fitting and refining phase is presented in Figures 1, 4, 12 and 13. I do not find strong evidence in these figures to suggest that fitting and refining must occur sequentially, as opposed to being concurrent features of what we might call \"successful GAN training\", as the idea that these states are sequential is often suggested based on qualitative descriptions of the data. \n\t- Furthermore, it is not clear to me what quantitative confirmation of the fitting and refining stages would even consist of. Sections 3 and 4 are titled as fitting and refining, but describe the discriminator and generator gradient tools respectively, which are then applied to all stages of GAN training. Are there clear features of fitting vs. refining which can be detected with the gradient based tools described here? If not, how can we evaluate the claim that these stages exist in real data? I have provided further detail for these concerns in the paper organization section below. \n- It is not clear how to understand the performance of the early stopping algorithm in comparison with other evaluation metrics for GANs, including those presented here (i.e. FID score and duality gap). Based on the presented results, the early stopping algorithm presented appears to align closely with FID score, but I do not find evidence that \"The combined use of\nour two proposed metrics offers enhanced sensitivity and reliability in detecting mode collapse compared to\nexisting approaches\", as stated on page 12 of the main text. More careful quantitative results or explanation are necessary in order to support this claim. \n- Many of the theoretical results in this paper are concerned with detailed, mechanistic descriptions of particle (e.g. generated sample) evolution over the course of GAN training. For example, the description of mode collapse in section 5 suggests that during collapse individual data modes are gradually vacated of samples, as 1) samples around the boundary of single mode are ejected and attracted towards other modes and 2) sample closer to the center of a data mode become more similar. While these claims may be difficult to justify in real world datasets, the empirical results (while broadly consistent with the theoretical/toy models), do not offer conclusive evidence in support of many aspects of the mechanistic theory. \n\n## Paper Organization. \nI believe there should be a correspondence between the section titles describing the three putative phases of GAN training (3-5), and the corresponding results presented.\n- Section 3 is labeled as fitting, which I understood to be the first phase when particles are putatively pushed from initialization across the entire support of the true data. The analysis presented within this section offers an analysis which features discriminator gradients and does not seem scoped to this first fitting phase. What features of the fitting phase specifically are we meant to understand from this section? Are there features of discriminator gradients which would identify this fitting phase, as distinct from a refining phase in real data? \n- Section 4 is labeled as refining, which I understand to be the second phase of training when the primary focus of GAN training is to reduce mode mixture. While steepness appears to be well suited as a technique to identify refining, it is not clear to me which aspects of steepness would help us to identify the transition from the fitting to the refining stage of training. \n\nThere are also several theoretical results which are presented on toy datasets, without a great deal discussion of how they generalize to real data. It would be useful to understand if and how theoretical results presented inform the particular early stopping algorithm described, or other potential applications/avenues of study in the future.  \n\n## Clarity. \nI think this paper could also be greatly improved with greater clarity in the description of results and figures. I have made my specific suggestions in the requested changes section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper proposes to study the training dynamics of GAN models using a set of gradient based tools. \n- They first describe these tools: 1) the discriminator gradient, i.e. the gradient of the discriminator with respect to samples, as well as 2) steepness; i.e. the spectral norm of the Jacobian of the generator's gradient with respect to the samples. \n- The suggest that these tools can help us to better identify different regimes of GAN training, which they describe as fitting, refining, and collapse. \n- The authors next present a series of theoretical results and simplified data analysis settings which are nominally specific to each phase of GAN training: \n\t1. For the first fitting phase, they describe how discriminator gradients would encourage movement of data samples under different configurations of the true and simulated data distributions. The novelty of these results lies in the two toy configurations of true/generated data distributions that they present, as well as a qualitative analysis of discriminator gradient vector fields under these conditions. \n\t2. For the second refining phase, the authors:\n\t\t1. provide a theoretical analysis of steepness, which measures the ability of a generator to map similar points in the prior to different samples in the data space. \n\t\t2. They next provide a lower bound on steepness under simplifying assumptions on the true data distribution, as well as analyses of how generator steepness evolves with time.\n\t\t3. Finally, they empirically correlate simple generators having higher steepness with less severe mode mixture on a gaussian mixture dataset, and support this empirical evidence with theoretical results in one dimension. \n\t3. In the third collapsing phase, the authors propose two characteristics of discriminator gradients and steepness which they suggest are indicative of mode collapse. \n\t\t1. They show that under simplified conditions and discriminators of a certain form, samples which are close to boundaries of data modes will experience large discriminator gradients which can trigger the onset of mode collapse. They suggest that monitoring the norm of discriminator gradients is therefore a useful metric to detect the onset of mode collapse. \n\t\t2. They show that under these same simplified conditions and discriminators, the generator will map samples which are close to a true data sample to a smaller neighborhood of that sample. \n\t\t3. Finally, the authors suggest an early stopping metric, montoring discriminator gradients and steepness evaluated on generated samples to monitor for the onset of mode collapse. \n- Finally, the authors present experimental results to corroborate their findings, as follows: \n\t- The authors evaluate image generation using a GAN model trained on MNIST (as well as Fashion MNIST in the appendix), and consider the probability of sample classification (as evaluated by a separate network $q(x)$ throughout the course of network training. They suggest that the dynamics of how generated samples are classified reflects the time course of the fitting and refining organization. \n\t- The authors then present empirical results for their early stopping algorithm, showing that the combination of discriminator gradients and steepness appears to identify a threshold before which generated samples are diverse and realistic, whereas samples generated after the threshold appear to be generated from collapsed modes. They also present the FID score (as well as the duality gap/perturbed duality gap in the appendix) and show that the metrics that they suggest to detect mode collapse appear to correlate well with these established metrics. \n- Finally, the authors study the effect of adding noise to the discriminator, showing that doing so appears to mitigate fluctuations in the norm of the discriminator gradient. \n- In the appendix, the authors provide an extensive review of the literature around GAN models, as well as proofs for theoretical results and additional experimental results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have organized this section to mirror the weaknesses described above. \n## Support for claims\n- I believe that more evidence is required to support the claim that fitting and refining as described in the paper currently are sequential phases of GAN training. In particular, understanding how we might identify these phases quantitatively and measure the transition between them would be critical. Addressing this concern would be crucial for me to recommend this paper for acceptance in its current form. \n-  If the authors will make the claim that their early stopping metric is more reliable and sensitive than existing approaches, I think this claim requires a more thorough explanation, especially in comparison to the FID score as is presented in the main text. What aspects of selecting an early stopping criterion based on FID score would be less reliable/sensitive than the algorithm presented? Addressing this concern would be crucial for me to recommend this paper for acceptance in its current form. \n\t-  Additionally, given the results presented Figure 7, it would be useful to understand if the early stopping algorithm still works well on noised GANs. Is it the case that collapse is avoided altogether? Or rather, is there a more subtle collapse that can still be detected. Addressing this concern would strengthen this work. \n-  It would be useful to have more illustrative examples to study mode collapse in real datasets. For example, is it possible to track a small number of particles through training, and measure steepness/discriminator gradients on it in order to verify that the mechanistic dynamics described on toy datasets do indeed generalize to the behavior of samples in real data? Addressing this concern would strengthen the paper. \n## Paper organization\n- I believe this paper is significant for its study of mode collapse, without necessarily describing separate fitting and refining stages of GAN training. If the authors were to reorganize sections 3 and 4 to be studies of discriminator gradients and steepness, and focus their study on the analysis of the transition to mode collapse, I believe it would be much easier to support the claims relating to mode collapse with the existing results. Addressing this concern (which is the same as the first \"support for claims\" concern) would be critical for me to recommend this paper for acceptance. \n- A discussion of each theoretical result presented, as well as descriptions of how they do/could inform practical algorithms or tools to study GAN training would greatly strengthen sections 3-5 in my opinion. \n## Clarity \nI believe that clarification on the points below are all critical for me to recommend this paper for acceptance, unless otherwise indicated. \n- What do the colors represent in Figure 2? I initially thought this was the density of $p_{data}$, but it should be uniform across all four modes in all presented cases. I then thought it was the norm of the vectors (which is very difficult to see- please consider subsampling) but the colors would then be inversely correlated to the scale provided. \n- Regarding the early stopping algorithm: In Figure 5, I assume the epoch indicated by the horizontal line in this Figure (also Figure 15) is the epoch where the early stopping algorithm would halt training. Are the samples indicated in red boxes those which are selected to optimize the GAN cost? If not, can you show these samples instead, as well as the samples generated immediately prior to the epoch where your early stopping algorithm is triggered? \n- In Figure 7, epoch 54, it appears that the histogram of discriminator gradient norms for noised GAN is entirely to the left of the 90% threshold. Where is the mass of the histogram for the noised GAN? I am also unable to read the axis labels for this figure. \n  - I also have concerns about the interpretation of this figure. My understanding was that mode collapse is indicated by **increases** in the norm of the discriminator gradient, whereas here it seems that mode collapse is indicated by a small gradient norm. Please elaborate on these results.", "broader_impact_concerns": "I do not have concerns about the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YUmMjMkXqD", "reviewer": "Reviewer_AFrb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper highlights **interesting insights into GAN training**. To my knowledge, this is the first paper decomposing GAN training into phases. This allows the paper to present mode collapse in a global picture as the outcome of the optimization process trying to alleviate mode mixture.\n\n**The proposed early stopping algorithm is interesting and novel**, grounded in the theoretical and empirical observations using the paper's framework. It may serve as a useful tool for practitioners to easily detect the collapse of GAN training.\n\nI appreciate the balanced mix of theoretical results and empirical observations in toy and real-world examples made in the paper that **grounds the conclusions into GAN practice**.\n\n## Weaknesses\n\nThe paper suffers from three important issues, described below.\n\n### Writing and Clarity\n\nWhile the paper is easy to read, the message it conveys is not clearly conveyed in the analysis of the three phases. **Most results, while potentially interesting, lack contextualization and an explicit conclusion** that would help the reader interpret them in the overall context of the paper. This is particularly the case in Section 3 where I do not know what to conclude from the empirical observations, and in Section 4.2 where there is no interpretation of the theoretical results.\n\n### Insufficiently Supported Claims\n\nPartly due to this writing problem, several of the paper's central claims are not sufficiently supported.\n\n**(1.)** First of all, in contradiction with the abstract, introduction and conclusion, the paper **does not characterize GAN training into three phases**. Such a characterization would necessitate a clear separation between phases and a precise definition of each phase, unlike the high-level description of Table 1. While collapsing is clearly identified in Section 5 and by the early stopping criterion, **the other two phases and not properly identifiable**. Fitting is especially blurry as explained above, and its separation with refining is never described. In its current state, the paper is more of an analysis of the training dynamics of GANs before and after collapse.\n\n**(2.)** The paper **overclaims that Algorithm 2** \"balances sample quality and diversity\" (abstract) when it actually only stops training when collapse occurs as seen in Figures 5 and 6, and that it \"offers enhanced sensitivity and reliability in detecting mode collapse compared to existing approaches\" (Section 6.2). For the last assertion to hold, the paper should at least explain why monitoring the FID is not sufficient (as it aligns with the proposed criterion) and show whether collapse can already be detected in the GAN losses.\n\n**(3.)** **Section 5.1's assumptions are insufficiently grounded**, making its conclusion questionable.\n- There is no explanation why $p_g \\approx p_{\\mathrm{data}}$ close to modes.\n- Assumption 5.1 artificially makes the discriminator ill-conditioned without proper justification. The referenced Appendix G does not show such a sharp transition in-between modes for the discriminator.\n\n**(4.)** **The steepness and the identification of mode collapse as the last stage of GAN training** are presented as novel in page 1, but the paper **lacks sufficient discussion of the literature** for this claim to hold. Details are given below.\n\n### Prior Work Discussion\n\nTo my understanding, **the notion of generator steepness, including its link with GAN trainability and minimal steepness requirements, has already been studied in the literature**, with a few variations, by, notably, Odena et al. (2018), Tanielian et al. (2020) and Salmona et al. (2022). Given the potentially strong links with the paper's contributions, a discussion of the paper's novelty w.r.t. the existing literature should be included.\n\nFurthermore, the paper **undermines pre-existing knowledge of GAN training and collapse in the literature**, for example via the assertion \"[the paper] challenges the conventional view that mode collapse signifies GAN training failure\" (page 2) -- which should be motivated by one or several references. I would like to point out in particular the seminal paper of Brock et al. (2019) documenting and analyzing collapse in the final training stages.\n\nBrock et al. Large scale GAN training for high fidelity natural image synthesis. ICLR 2019.\\\nOdena et al. Is generator conditioning causally related to GAN performance? ICML 2018.\\\nSalmona et al. Can push-forward generative models fit multimodal distributions? NeurIPS 2022.\\\nTanielian et al. Learning disconnected manifolds: a no GAN’s land. ICML 2020.\n\n### Other Issues\n\n- Algorithm 2 necessitates to know the number of modes of the data distribution and to estimate the generator's steepness. The paper would benefit from a discussion on the feasibility of these choices.\n- It is not clear how Assumption 2.1 (Gaussian smoothing of the data) is related to minibatch training. I believe this assumption can be justified more directly by noticing that data smoothing has now become a standard even beyond diffusion models.\n- I assume that $\\omega$ denotes the discriminator's parameters, but this is not specified in the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper breaks down non-saturating GAN training into three phases (fitting, refining, collapsing) and analyses each of them using the spatial gradients of the generator and the discriminator with the following conclusions. In the first phase (fitting), discriminator gradients encourage the generator to cover all modes of the data distribution. In the second phase (refining), the generator's steepness (related to its Lipschitz constant) provably needs to increase in order to fit a multimodal data distribution. In the third phase (collapsing), mode collapse occurs, attributed to an exploding discriminator gradient; together with decreased steepness of the generator, the discriminator's gradient magnitude constitutes a criterion for detecting mode collapse in a proposed early stopping strategy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Most of the aforementioned issues are crucial for acceptance at TMLR. I would strongly recommend the authors to:\n- improve the writing to better highlight conclusions and provide interpretations in the overall context of the paper;\n- clarify or amend the unsupported claims, especially on the phases' characterization;\n- discuss prior work on generator steepness and training collapse.\n\nDetails are provided above.\n\n---\n\n## Post-rebuttal update\n\nAs highlighted in the discussion with the authors below, the vast majority of my concerns have been successfully addressed in the revisions of the paper. Writing is significantly improved, the (modified) claims are now correctly supported, and close related work has been discussed.\n\n**Disclaimer.** Before providing a recommendation, I would like to state that I did not have the time to assess the proofs of theoretical results in the appendix.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uEbG03eFFL", "reviewer": "Reviewer_dg58", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper demonstrates solid theoretical foundations and empirical rigor in analyzing GAN training dynamics, particularly through its well-structured verification experiments. I commend the authors' improvements in Section 2, where they've clearly laid out their assumptions and maintained consistency throughout the paper. The presentation has notably improved by focusing on the early stopping methodology and its empirical validation.\n\nHowever, I have two main concerns. First, the experimental validation would be more convincing if extended beyond CIFAR-10, where the reported FID scores exceed 100 – significantly higher than current standards. I strongly encourage the authors to evaluate their proposed criterion on either modern GAN architectures that achieve FID scores of 10-30 on CIFAR-10, or on larger-scale datasets such as LSUN or ImageNet.\n\nMy second point concerns the paper's title. While the revisions have substantially improved the manuscript, the current title \"Magnifying Three Phases of GAN Training\" seems disconnected from the paper's core contributions. The work more accurately reflects an understanding or characterization of GAN training phases, with particular emphasis on addressing collapse issues. A title revision would better serve potential readers and more accurately represent the paper's contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I was a reviewer of the previous version of this manuscript and I appreciate the significant changes that the authors have made to the new manuscript. In particular, it addressed the original concerns that were discussed in previous reviews, particularly regarding citing related works and clarifying the novelty points. \n\nThe current version is an in-depth analysis of the training dynamics of GANs from the gradient perspective, and the proposed analysis leads to a practical algorithm that helps detect model collapse during training. I think this contribution makes a valuable contribution to the practice of GAN training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a resubmission of a previously submitted work that has undergone significant improvements. Additionally, this version has been further refined based on discussions with the three reviewers.

This paper analyzes GAN training dynamics, introducing a novel approach to identifying key transitions during training using gradient-based tools. The authors successfully demonstrate how these methods can detect different epochs within GAN training, leading to meaningful practical implications.

Initially, the submission required improvement in several areas, including missing discussions on relevant related work, unclear claims, and insufficient empirical support. However, following a thorough revision process, the paper has significantly improved, effectively addressing all major concerns. The updated version now clearly articulates the transition of GAN training from a fitting phase to collapse, supported by well-established theoretical foundations. The revised manuscript is well-structured and presents a compelling argument for its contributions. The authors have refined the writing, enhanced empirical evaluations, and made the paper’s claims more transparent and well-supported. 

The changes made during the revision were substantial and have ultimately strengthened the paper’s contributions. Given these improvements, I recommend its acceptance, as it meets the TMLR acceptance criteria and makes a meaningful contribution to the field.
Decision: Accept (poster)