BrGANs: Stabilizing GANs' Training Process with Brownian Motion Control
Authors: Tianjiao Luo
Ziyu Zhu
Gabriele Oliaro
Jun Zhu
Zhidong Deng
Venue: Submitted to ICLR 2023 (2023)
Keywords: GAN
stability
control theory
Brownian motion

Abstract: The training process of generative adversarial networks (GANs) is unstable and does not converge globally. In this paper, we propose a universal higher order noise based control called Brownian Motion Control (BMC) that is invariant to GANs frameworks so that the training process of GANs is exponential stable almost surely. Specifically, starting with the prototypical case of Dirac-GANs, we design a BMC and propose Dirac-BrGANs that retrieve exactly the same but reachable optimal equilibrium regardless of GANs' framework. The optimal equilibrium of our Dirac-BrGANs' training system is globally unique and always exists. Furthermore, the training process of Dirac-BrGANs achieve exponentially stability almost surely for any arbitrary initial value. Then we extend our BMC to normal GANs' settings and propose BrGANs. We provide numerical experiments showing that our BrGANs effectively stabilizes GANs's training process and obtains state-of-the art performance compared to other stabilizing methods. 

Under review as a conference paper at ICLR 2023
BRGANS: STABILIZING GANs' TRAINING PROCESS WITH BROWNIAN MOTION CONTROL
Anonymous authors Paper under double-blind review
ABSTRACT
The training process of generative adversarial networks (GANs) is unstable and does not converge globally. In this paper, we propose a universal higher-order noise-based controller called Brownian Motion Controller (BMC) that is invari- ant to GANs' frameworks so that the training process of GANs is stabilized. Specificallyy, starting with the prototypical case of DANas we design a BMC and propose Dirac-BrGANs, which retrieve exactly the same but reachable op- timal equilibrium regardless of GANs' framework. The optimal equilibrium of our Dirac-BrGANe training system is globally unique and always exists. Fur- thermore, we give theoretical proof that the training process of Dirac-BrGANs achieves exponential stability almost surely for any arbitrary initial value and de- rive bounds for the rate of convergence. Then we extend our BMC to normal GANs and propose BrGANs. We provide numerical experiments showing that our BrGANs effectively stabilize GANs' training process and obtain state-of-the- art performance in terms of FID and inception score compared to other stabilizing methods.
INTRODUCTION
Generative Adversarial Networks (GANs) Goodfellow al. 2014) are popular deep learning based generative architecture. Given tain an estimated Ponodet and produce new entries that are as close to indistinguishable as possible from the input entries. For example, GANs can be used to generate images that look real to the human eyes (Wang et al. (017). A GAN's architecture consists of two neural networks: a generator and a discriminator. The generator creates new elements that resemble the entries from the input dataset as closely as possible. The discriminator, on the other hand, aims to distinguish the (counter- feit) entries produced by the generator from the original members of the input dataset. The GAN's two networks can be modeled as a minimax problem; they compete against one another while striv- ing reach Nash-equilibrium, an optimal solution where the generator can produce fake entries that are, from the point of view of the discriminator, in all respects indistinguishable from real ones.
input dataset with unknown Preat. GANs can ob-
Unfortunately, training GANs often suffers from instabilities. Previously, theoretical analysis has been conducted on GAN's training process. Fedus et al. 2018) argue that the traditional view of considering training GANs as minimizing the divergence of real distribution and model distribution is too restrictive and thus leads to instability. Arora et al. (2018) show that GANs training process does not lead generator to the desired distribution. Farnia & Ozdaglar (2020) suggest that current training methods of GANs do not always have Nash equilibrium, and Heusel et al. (2017a) is able to push GANs to converge to local Nash equilibrium using two-time scale update rule (TTUR).
Many previous methods (Mescheder et al. 2018: Arjovsky & Bottou 2017 Nagarajan & Kolter 2017 Kodali et al. 2017) have investigated the causes of such instabilities and attempted to reduce them by introducing various changes to the GAN's architecture. However, Mescheder et al. (2018) show in their study, where they analyze the convergence behavior of several GANs models, despite bringing significant improvements, GANs and its variations are still far from achieving stability in the general case.
To accomplish this goal, in our work, we design Brownian Motion Control (BMC) using control theory and propose a universal model, BrGANs, to stabilize GANs' training process. We start with the prototypical Dirac- MrA (Mescheder et al. 2018) and analyze its system of training dynamics.

---

Under review as a conference paper at ICLR 2023
We then design Brownian motion controller (BMC) the training dynamic of Dirac-GAN in order to stabilize this system over time domain We generalize our BMC to normal GANs' setting and propose BrGANs.
0.1
24
W
1.5
Figure The gradient map and convergence behavior Dirac-WGANS (first row) and Dirac- BrWGANs (second row), where the Nash equilibrium of both model should be at (0.0)".
1.1 SUMMARY OUR CONTRIBUTION
Compared with previous methods, have the following contributions:
We design Brownian Motion Controller (BMC), a universal higher order noise-based con- troller for GANs' training dynamics, which compatible with all GANs' frameworks, and we give both theoretical and empirical analysis showing BMC effectively stabilizes GANs' training process.
Under Dirac-GANs setting, we propose Dirac-BrGANs and conduct theoretical stability analysis to derive bounds on the rate of convergence. Our proposed Dirac-BrGANG are able to converge globally with exponential stability.
We extend BMC to normal GANs' settings and propose BrGANs. In experiments, our BrGANs converge faster and perform better in terms of inception scores and FID scores on CIFAR-I0 and CelebA datasets than previous baselines various GANs models.
1.2
RELATED WORK
To stabilize GANs training process, lot of work has been done modifying its training archi- tecture. Karras et al. (2018) train the generator and the discriminator progressively to stabilize the training process. Wang et al. (202T) observe that during training, the discriminator converges faster and dominates the dynamics. They produce attention map from the discriminator and use it to improve the spatial awareness of the generator. this way, they push GANs' solution closer to the equilibrium.
On the other hand, many work stabilizes GANs' training process with modified objective functions. Kodali et al. (0117) add gradient penalty to their objective function avoid local equilibrium with their model called DRAGAN. This method has fewer mode collapses and can be applied to a lot of GANs' frameworks. Other work, such Generative Multi- Multi-Adversariall Network (GMAN) (Du- rugkar et al. 2017), packing GANs (PacGAN) (Lin et al.) 2017) and energy-basee based GANs Zhao al. 2016), modifies the discriminator achieve better stability.
2

---

Under review as a conference paper at ICLR 2023
Xu et al. 2019) formulate GANs system of differential equations and add closed-loop control (CLC) on few variations the GANs to enforce stability. However, the design of their controller depends on the objective function of the GANs models and does not work for all variations of the GANs models. Motivated by them, we analyze GANs' training process from control theory's perspective and design invariant Brownian Motion Controller (BMC) stabilize GANs training process. Compared with Xu et al. (2019), our proposed BrGANs converge faster, perform better, and do not rely on any specific GANs' architecture.
2 CONTROLLING DIRAC-GAN WITH BROWNIAN MOTION
In this section, we come up with the BMC, higher order noise-based controller, a universal control function that is invariant to objective functions of various GANs models. In addition, we prove that Dirac-GAN with BMC exponentially stable and we derive bounds on its converge rate.
2.1 DYNAMIC SYSTEM OF DIRAC-GANS
In Dirac-GANs' settings, the distribution of generator G follows pa(x) 8(x 0) and the dis- criminator is linear Dø(r) Φz. The true data distribution given by PD(x) 8(x c) with constant c. (Notice that for Dirac distribution, ö can be considered as impulse the origin such that 6(x) = 1 at the origin and otherwise.)
The objective functions of Dirac-GANs can be written as:
max LDD, (d: 8) h1 (D)( h2())(0) {
(I)
max La. (0; (a)),
where hn and h-)-) are increasing functions and hz2-) is a decreasing function around zero Xu al. 2019)
The training process of Dirac-GAN can be modelled as a system of differential equations. Follow- ing Mescheder et al. (2018) and Xu al. (2019), the training dynamical system of Dirac-GAN is formulated as:
dø(!) (((ooo)))))c++++(()))) df
dlll) ()(((())())((()))
(2)
This system has constant nontrivial solution (0, o) = (c, 0).
Let ((t) = ((t) c, and convert the original system to:
do(() ( 000
dt
(3)
d0(t) (()()(t)((t+++))))))
dt
At this time, the equilibrium of system (0,
Define X(1) = (d(1), ((0) ((X)) = (1) (o(())) + h ((()()()( + c)) +1(((1)
(bc Then system (3) can be rewritten as:
dx dt
(4)
2.2 DESIGNING BROWNIAN MOTION CONTROLLER FOR DIRAC-GAN
Brownian motion is a natural phenomenon that captures the random displacements of particles in d-dimensional space. At each time step. the displacement B, is independent, identical random variable ranging in ddd The distribution of B, is normally characterized by a multivariate Gaussian distribution.
3

---

Under review as a conference paper at ICLR 2023
Denote the position of particle at initial time 0 x().. Then time T, this particle's position is given as
X(T) - X(0) + L.' B, dt.
(5)
In control theory, noise- basse controllers like our Brownian motion controller (BMC) are useful tool to stabilize dynamical systems and push the solution towards the optimal value over time domain t (Mao et al. 2002). In this section, we design BMC on Dirac-GAN's training dynamic to improve stability.
To stabilize system
we propose the following higher order noise-based controller:
u(t) = 01 (t) + (x))
(6)
where (t) and (t( are independent one-dimensional Brownian motions, B > 1, 01 and @2 are non-negative constants. Incorporating BMC (6), the controlled system is given as
dx (t) (t) + u(t).
2.3 DIRAC-BRGAN WITH EXPONENTIAL STABILITY In this section, we derive the existence of unique global solution and stability of system The equilibrium point of system is
(Ó(L.), ((te)) (0,
Without the BMC, the training of regular GAN or WGAN (Arjovsky al. 2017) unstable and
it oscillates around the equilibrium point (0, 0) Figure iillustatte the gradient maps of 0 against and convergence behavior over time domain We can see that the gradients of both the generator and the discriminator are oscillating around the equilibrium point, but they never converge to it.
For the stability analysis, we impose the following assumption on the smoothness of functions h1, h2, ha in system (
Assumption 1. There exist positive constants a1 Q2 0g such that for any x, R",
(x)()) (1) I an || yll, 1 (x) - hi(()) S az || vll
(11(x)) - lig(v)) - 03 | - yll
In what follows, we first prove that the BMC from equation (6) yields a unique global solution in Theorem 1. That is, no matter which initial point X we are starting from, system 7 a.s. will have a unique solution X X()) as l goes to infinity. Then, in Theorem 2, we show that this unique global solution exponentially converges to the equilibrium point a.s. with bounds on the hyper- parameters @1 l2 and 3 which in turn affect the rate of convergence. Combining Theorem I and Theorem 2, we claim that system is stable and thus the Birac-BrGAN stable and converges to the optimal solution required.
Theorem 1. (Proof in Appendix A) Under Assumption 7 for any initial value X(0) & E IR, if @2 * 0 and B >> 1, then there a.s. exists unique global solution X to system 7 on I. [0, 2o).
Theorem 2. (Proof in Appendix B) Let Assumption hold. Assume that @2 + 0 and B 1. !f
-6>0
2
where ip takes the value of
2 2
20
1 afje²² 2c 1 2 2 + (a} + go)) + 1(1 (+=gaa)
1
(9)
then for any (00 & with sufficiently small constant c E (0, e2/2 sp), the global solution X(t) of system (7) has the property that
log (X(1)
S-
c - +e, a.s.
lim sup 1-000
that the solution of system
is a.s. exponentially stable.
4

---

Under review as a conference paper at ICLR 2023
Here since 2 -p > and e a sufficiently small constant, then when Eq. 9 is satisfied, we have
log (x)1) E ->, a.s.
lim sup (122
(10)
for positive constant X. Rearranging get
lim sup (X(t)) S Ar- t-+00
(11)
which implies
lim sup ((1) (0,
(12)
++000
required. Notice that the rate of convergence depends only constant \, which in turn depends on Ø1 and yp. means that the convergence rate is decided by the choice of tune these three variables as desired, long they satisfy the constraint
@1, 02.
and B. In practice, from equation 9
Notice that our Birac-BrGAMe works for any h1, h2 and ha long they satisfy the smoothness condition under assumption []] In other words, we have proven that the BraacBBrAANs are stable regardless of the GANs' architecture, and have given theoretical bounds the rate of conver- gence. In Figure | we present visual proof that the Dirac-BrGANs are stable and converge to the optimal equilibrum as required.
GENERALIZATION OF BMC TO GANs
In section 2 we designed a universal BMC for Dirac-GANs and proved that the Dirac-BrGANs are globally exponentially stable. this section, we are going to generalize the BMC for normal GANs (i.e., GANs other than the Dirac-GAN). We consider any GANs where the Generator (G) and the Discriminator (D) are neural networks in their respective function spaces.
3.1 MODELLING DYNAMICS OF GANS
Analogously the Dirac-GAN, the training dynamics normal GANs can be formulated as system of differential equations. Instead of 0 and ø, we directly start with G(z, t) and Dix, t) to represent, respectively, the generator and the discriminator. The objective functions of GANs can be written as:
max (D: (4 (D(x)) + Epto) (a((Dx)))))
(13)
max LoG = Ep) (ha((D((())))
Following Xu (2019)'s ' notation, the training dynamic for the generator and discriminator over the time domain & can be transformed to:
dD(x, !) dt
p(x) dhy (D(x))
dD)(D(x))
dD ( dD(x)
Vr
(14)
dG(z.t) = p2(2) dD(G(2)) dG(z) Vz
dhg (D(G()))) dD(G(z))
df
Now we define (t) (D(x t), T and
f(x(t)) = (p(x) (D(()
aDla.) pα( dhy (D()))
diat D(G(()) dD(G()
dD(x)
(15)
(z(2) dDIG()) dG(c)
Now we convert system (4)
dx S(X
(16)
5

---

Under review as a conference paper at ICLR 2023
3.2 BRGAN: STABILIZED GANS WITH BMC
The optimal solution of a normal GAN is achieved when D(x) = 0 for the discriminator and (G(x) p(a) for the generator. In control theory, when we design controller for dynamical system, we need to know the optimal solution so that we can use the controller to push this dynamical system its equilibrium point without changing With normal GANs, we only have information on the optimal solution for the discriminator, so we are going to impose the BMC only on the discriminator this time.
Notice that in Dirac-GAN, we impose the BMC u(t) = e (t) + ((t) on (t), which is for both the generator and discriminator. Since now we are going to impose BMC only on the discriminator, without losing information from the generator, here we slightly modify Eq. (6) so that
ut( = Di + + D(GGE))
(17)
where (t) and B2(t) are independent one-dimensional Brownian motions, and @2 are non- negative constants.
Since we are going to implement BrGANs through gradient descent, our BMC can be reflected on the discriminator's objective function with the derivative being Eq. (17). We thus take integration of
Eq. 17 and modify the objective function of the discriminator in (13 to:
1 = Lo je Jen - (x)
2 (1) +
max D
(
(18)
We implement our designed objective functions in section Our numerical experiments show that BrGANs successfully stabilize GANs models and are able to generate images with promising quality.
4 EVALUATION
In this section, show the effectiveness of BMC by providing both quantitive and qualititive results.
4.1 EXEPERIMENTAL SETTING
Dataset: We evaluate our proposed BrGANs on well-established CIFARIO (Krizhessky et al. 2009) and CelebA datasets (Liu al. 2015). The CIFAR-10 10 dataset consists of 60000 x2x x22 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. This dataset can be used for both conditional image generation and unconditional image generation. In order to compare our training method fairly with the solutions from the related works, we use batch size of 64, the same generator, and discriminator architecture under the same codebase. CelebA contains 202,599 face images of size 64 x 64, which has diverse facial features.
Implementation details: Both our generator and discriminator are composed of convolutional lay- ers, batch normalization, and activation layers. The generator uses four layers of transposed con- 100 latent vector to 3 x 32 x image. Batch normalization and ReLU activation are followed by each layer. For the discriminator, first, we use three layers of convolutional layer to obtain 1024 x 4 x 4 image and then feed this image to an MLP to get a single value.
volutional layers to convert
Each of our models is trained on an Nvidia 2080TT GPU. The batch size 64, while the generator trained for 50000 iterations, and the discriminator is trained for 250000 iterations. We use Adam with le-4 learning rate as an optimizer to train our model.
Evaluation Metric:
For the Dirac-GAN problem, the optimal solution is known to us we can measure the convergence speed and draw the gradient map for different training algorithms. For CIFAR 10,
the FID score (Heusel et al. (017b)) and the inception (core(Barratt & Sharma 2018) to measure the quality of the generated images. We also compare the FID and inception scores across different timestamps to show the convergence speed.

---

Under review as a conference paper at ICLR 2023
4.2 CONVERGENCE OF DIRAC-BRGAN
The gradient map and the convergence curve are presented in Figure I These results show that our Dirac-BrGANs have better convergence patterns and speed than Dirac-GANs. Without adding BMC to the training objective, Dirac-GANs cannot reach equilibrium. The parameters of the generator and the discriminator are oscillating circle as shown figure [ However, the parameters of the Dirac-BrGANs only oscillate in the first 500 iterations and soon converge in 800 iterations.
and / 2. As shown in table and table 2, BraacBBrAANs converge better when we set @1 = 0.1 and Ø2 0.01. Generally, larger º will lead faster convergence rate but when & is large enough, the effect of increasing 2 will become saturated. On the other hand, when is too small, BrracBBsAANss will take more than 100000 iterations to converge.
We also study different combinations of @1 and P2 under B
Table Convergence iters for / 2 under Dirac-BrGANs
B 2
P2 0.0001
@2 0.001
P 0.01
@1 0.1
1500
750
700
0.01
>100000
9000
8500
@1 0.001
>100000
>100000
100000
Table Converge iters for 3 = under Dirac-BrGANs
B
@ 0.0001
@2 0.001
@ 0.01
Q1 0.1
600
400
400
2 0.01
25000
15000
10000
@1 0.001
> >00000
>> >100000
40000
4.3 STABILIZED BRGANS: CONVERGE FASTER AND PERFORM BETTER
We demonstrate that BrGANs converge fast and generate higher fidelity image than Wasserstein GANs (WGAN) (Arjovsky et al. (017)). WGAN with weight clipping (WGAN-CP), WGAN with gredient penlty (WGAN-GP) (Gulrajani et al. 2017). GAN with closed loop control (WGAN- CLC) (Xu et al. 2019) and their combinations. From the results in Table 3 we can observe that our proposed WGAN-BR-CG achieves 22.10 FID and 5.42 inception score on CIFAR1O dataset, which is the best result among all other GANs. Specifically, WGAN-BR out performs WGAN, WGAN-BR-CP outperforms WGAN-CP and WGAN-CLC-CP, and WGAN-BR-GE outperforms WGAN-GP and WGAN-CLC-GP. From Table 4 can observe similar trends
Table 3: Results on 10.AR10.
Table 4: Results on CelebA
Method
FID
Inception
Method
FID
Inception
WGAN
94.77
3.29
WGAN
366.12
2.06
WGAN-CLC
52.39
4.25
WGAN-CLC
14.87
3.28
WGAN-BR
36.50
4.80
WGAN-BR
23.91
3.34
WGAN-CP
37.81
4.69
WGAN-CP
13,60
3.14
WGAN-CLC-CP
35.59
4.64
WGAN-CLC-CP
13.80
3.12
WGAN-BR-CP
34.18
5.03
WGAN-BR-CP
14.74
3.19
WGAN-GP
30.81
5,03
WGAN-GP
8.30
3,09
WGAN-CLC-GP 53.47
4.24
WGAN-CLC-GP
61.94
3,42
WGAN-BR-GP 22.10
5.42
WGAN-BR-GP
6.20
2.91
Convergence iterations of GANs is presented in Fig. 2 and Fig. measured by FID score and inception score, respectively. It is readily observed that our proposed BrGANs achieves better FID and inception scores given the same training iteration. Xu et al. (2019) add 1.2 regularization

---

Under review as a conference paper at ICLR 2023
(CLC) the objective function discriminator. In our BrGANs, incorporate both information from the generator and discriminator to our controller, so that the discriminator and generator to make sure the discriminator does not dominate the training process. Our BrGANs also compute faster than Xu et al. (2019) since we do not need to keep a buffer and update accordingly during training process.
VID WIGH eigina
303
wggn
200
ogan 22
10000
SE060
10000
IC6C0
4C0C0
Number Ceneratzr Haratoo
Numier of Cenerater baraton
Figure 2: FID score on CIFAR1O.
Figure 3: Inception score on CIFARIO.
4.4 QUALITATIVE RESULTS
We provide qualitative results on CIFAR and CelebA datasets. In Fig. and Fig. 4 these images are generated by WGAN, WGAN-GP, WGAN-GP-BA and WGAN-GP-CLC, from top left to bottom right respectively. It can observed that our WGAN-GP-BR can generate images with higher fidelity and more reasonable in visual perception.
Figure 4: CIFAR
Figure 5: CelebA
5 CONCLUSION AND DISCUSSION
In this paper, we revisit GANs instability problem from the perspective of control theory. Our work novelty incorporates a higher order non-linear controller and modify the objective function of the discriminator to stabilize GANs models. We innovatively design universal noise-based control method called Brownian Motion Control (BMC) and propose BrGANs to achieve exponential sta- BMC is compatible with all GANs variations. Experimental results demonstrate
bility. Notably, that our BrGANs converge faster and and perform better in terms of FID and inception scores on CIFAR-18 10 and CelebA.
8

---

Under review as a conference paper at ICLR 2023
In paper, theoretical analysis has been done under Dirae-GANt setting and are able to stabilize both generator and discriminator simultaneously. However, under normal GANs' settings, only design BMC for the discriminator and stabilize the discriminator, then force the stability of the generator. Additionally, our BMC is derived under continuous setting, but GANs' training process is considered discrete time steps. To resolve these two problems, further work can be done on estimating the generator's 's equilibrium at each time step and imposing controller on both generators and discriminators simultancously.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. International Conference Learning Representations (ICLR), 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. International conference machine learning, pp. 214-223, PMLR, 2017.
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do gans learn the distribution? some theory and empirics. International Conference on Learning Representations (ICLR), 2018.
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint arXiv:1801.0197 2018.
Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi nulti-adversarial networks. Inter- national Conference on Learning Representations (ICLR), 2017.
Farzan Farnia and Asuman Ozdaglar. Do gans always have nash equilibria? conference on machine learning. PMLR, 2020.
In International
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew Dai, Shakir Mohamed, and Ian Goodfellow, Many paths to equilibrium: Gans not need to decrease divergence at every step. International Conference Learning Representations (ICLR), 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- tion Processing Systems, pp. 2672-2680, 2014.
Ishaan Gulrajani. Faruk Ahmed, Martin Arjovsky. Vincent Dumoulinl, and Aaron Courville. Im- proved training of wasserstein gans. arXiv preprint arXiv:1704 00008, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, and Bernhard Nessler. Gans trained by a two time-scale update rule converge local nash equilibrium. arXiv preprint arXiv: (rXxv:/706.08500
2017a.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017b
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. International Conference on Learning Representations (ICLR), 2018.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint urXiv:1705.0721550 70, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power two samples in generative adversarial networks. arXiv preprint MiXv:1712..4008. 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.

---

Under review as a conference paper at ICLR 2023
Xuerong Mao, Glenn Marion, and Eric Renshaw. Environmental brownian noise suppresses explo- sions in population dynamic. Stochastic Processes and their Applications, 97, 2002.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference machine learning, pp. 3481-3490. PMLR, 2018.
Vaishnavh Nagarajan and J. Zico Kolter. Gradient descent gan optimization is locally stable. 31st Conference on Neural Information Processing Systems, 2017.
Jianyuan Wang, Ceyuan Yang, Yinghao Xu, Yujun Shen, Hongdong Li, and Bolei Zhou. Improving gan equilibrium by raising spatial awareness. arXiv preprint arXiv:21 2021.
Ting-Cue Cing-Chun Wang, Mingyu I. Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. arXiv preprint arXiv:1711.1155.. 2017.
Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Understanding and stabilizing GANs' training dynamics with control theory. arXiv preprint arXiv: 2019.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXhv:1609.03126. 2016.
APPENDIX A: PROOF OF THEOREM
Under Assumption I for any initial value X(0) { R?, if 02 o 0 and B > 1, then there a.s. exists a unique global solution (t) to system 7 on t [0).
Proof. Under Assumption 1. then, can calculate that (x(t))(X(tt
(1000000)0000(00000000000++)) +))+)) (()))(((())((+))))))
((0))(()()))((+++++))))))))))))) + c)))((1
1 <I(1 + 1 ag))² + 2c / ||I| + (a} + a x
2
For any bounded initial value (0) E R", there exists unique maximal local strong solution X (t) of system 7 on Te). where Te the explosion time. To show that the solution is actually global. we only need to prove that Te = co a.s. Let ho be sufficiently large positive number such that (X) < ki- For each integer > ko, define the stopping time
= inf E (0, [X(t) 2 k}
with the traditional setting inf @ 0c, where @ denotes the empty set. Clearly, тыe is increasing as + 0o and Tk + Too S Te a.s. If we can show that Txx = co, then T. = oo a.s., which implics the desired result. This is also equivalent to prove that, for any t 0, PP(Tk prove this statement, for any E (0, 1), define Cu-function
0 as k - co. To
V(x) (X(P)
One can obtain that X(t) # 0 for all 0 s S Te a.s. Thus, one can apply the Itô formula show that for any t E [0, Te),
dV(((()) -LV per 1o0
+ +
where LV is defined as
x X (x(((( p(p o) |
p(p 1)
+
X 294D
2
10

---

Under review as a conference paper at ICLR 2023
By Assumption
|
therefore have
LV(X) < p(p - 1)02 [
| + ((1 + 2 zaire² + 2c }
platate + (p-1)ee
(a² + 2 2 xxp.
2
Noting that p E (0, and B > and * 0. by the boundedness of polynomial functions, there exists positive constant H such that (V(x) H. We therefore have
AATE
(V(A((ATT))
- Elgl
(s)
- E| H =: Hh,
where He is independent of k. By the definition of Th (((Tx)] = k, so
P(rk S t)h²
S P(Tx S ())((((T)) I E ((X(EA s (V(((tATT))
s He.
which implies that
H. ko
lim suppP(r t) S lim
0,
k+000
k0000
required.
APPENDIX B: PROOF OF THEOREM 2
Let Assumption I hold. Assume that @2 F 0 and 8 > 1. If
4 0,
2
where
8 max
0 23 + (a +
23
ag)a² + +[(1+ ag)e + 2c +
1
1
(19)
2
2
2
then for any X(0 = { with sufficiently small constant E e0.e 2 - sp), the global solution X(t) of system 7 has the property that
log IX(1)
of -2) 1-))+ te, a.s.
lim sup t-+00
that is, the solution of system
is a.s. exponentially stable.
Proof. Applying Itô formula to log [t) yields
/ (x))(()
log [x(t) = log [X(0)+ +
ei
ds + [' eidBi(t)
X(s)
2
2
+ 22 1' (X(s))
11

---

Under review as a conference paper at ICLR 2023
Letting (t) 02 So (dss)) clearly M(t is continuous local martingale with the quadratic variation
((t) > 22 (sss)
For any & E (0, 1), choose martingale inequality gives
> 0 such that 0e > 1. Then for each integer m >> 0, the exponential
=g 2
J (x ds 2 Os log
P
sup 15t5m
M(I)
m
Since 2m=1 m 0= < co, by the well-known Borel-Cantelli lemma, there exists an Qo _ with P(R)) = such that for any w E Sg there exists an integer m(w),, when m m(w), and m s 1 S
M(1) S Eg 2
/ [x(s)
ds + De log(( + 1).
This, together with Assumption ! yields
e3(1 =c)
log [((t) < log 151 +
{(s)
2
(1 + aj)e² +
(s{(s))
2
1
(a² + 0
ds
2
L. 2id (t) + 0c log(t + 1).
+
Letting be sufficiently small, by the definition of in 9 for sufficiently small E 20, have
- s).
- b+ L: (Id
gi - v) - -
log X()) S log |sl
2
Os log +1.
Applying the strong law of large number, therefore have
log [X(t)
-( et
lim sup sup t-+00
p) + a.s.
12


Reviews:
[{"review_id": "yPuFUNfNb94", "reviewer": "Reviewer_sHwP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a higher order Brownian Motion Controller (BMC) for BrGANs to stabilize GANs' training process. Starting with the prototypical case of Dirac-GANs, the authors design a BMC and propose Dirac-BrGANs that retrieve exactly the same but reachable optimal equilibrium regardless of GANs' framework. The authors also provide corresponding proof and experiments, although the experimental results show poor performance.\n", "summary_of_the_review": "This work propose a new way to stabilize the training process of GAN, using Brownian Motion Controller. Although the motivation is interesting, the results are less satisfactory and the quality of the work is lower than the ICLR's bar.", "strength_and_weaknesses": "Strengths:\n- The idea of taking GAN training as a dynamic system and applying Brownian Motion Controller on it sounds interesting.\n\nWeaknesses:\n- My biggest concern is about the experimental results. Only CIFAR-10 (32x32 resolution) and CelebA (64x64 resolution) are evaluated and the FID score is not state-of-the-art (actually there is a very big gap between the provided results and SOTA, eg. IS 5.42 on CIFAR-10 v.s. 9.18 in [1]). I truly understand the architecture may be different, but it would be better to see whether the proposed method can continually improve the performance on top of the best architecture. Given there is plenty of theory works studying GAN training stability issues, it is hard to trust this work can indeed help improve the GAN training process since this one considering its poor performance.\n- Meanwhile, only two datasets are evaluated, and both of them are low-resolution. It would be better to show more results on LSUN, FFHQ, with their high-resolution version.\n- The claim of \"the training process of Dirac-BrGANs achieves exponential stability almost surely\" sounds like an overclaim to me. To verify its correctness, I think some additional experiments are needed: e.g., using various hyperparameters (training G after training D every $N$ times), optimizers, batch size, and demonstrating that the proposed methods can help GAN training robust to different settings.\nReferences:\n[1] Differentiable Augmentation for Data-Efficient GAN Training", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gDUFJqIhFK3", "reviewer": "Reviewer_Vy2E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends the CLC-GAN framework of Xu et al 2019 https://proceedings.mlr.press/v119/xu20d.html with a gaussian noise regularisation derived from Brownian Model Control theory, showing that this additionally improves over the CLC controller in terms of FID on CIAR10 and celeba", "summary_of_the_review": "As noted, if the empirical gains survive an improved evaluation, I think this paper is a fine incremental improvement over previous work, using not-so widely known BMC theory", "strength_and_weaknesses": "Strengths\n\nIf the main weaknesses below are fixed and the papers empirical improvements persists, the paper becomes an incremental improvement over CLC gan which will spread awareness of BMC theory in the community, which I think is absolutely enough. I'd also be interested in possible links to diffusion processes. \n\nWeaknesses\n\n1. the FID of WGAN-CLC-GP is much higher than in the Xu et al paper, but the IS is much closer (still higher). To me this indicates FID was calculated with a different batch size than in the Xu et al paper *but* also that the improvement might be due to architecture/training differences. An ablation study might alleviate this...\n2. ....As would reporting multiple seeds, reporting mean and std and performing a statistical significance test\n3. the novelty relies on adding gaussian noise to the control process + an analysis, which I think is just barely enough to warrant a paper, but is sabotaged by limited evaluation quality. If the evaluation is fixed and benefits persist, this weakness becomes irrelevant.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Wz0ZHY1C4NB", "reviewer": "Reviewer_y1pm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors leverage ideas from control theory to propose a regularizer, called the Brownian motion controller (BMC), which aims to stabilize GAN training. In practice, this amounts to an additive regularization term which is applied to the discriminator loss.", "summary_of_the_review": "Overall, I found the direction of the work promising, but its execution wanting. It was unclear why the authors chose DiracGANs as a motivating example, and why the control theory perspective was the most compelling direction to alleviate the training instabilities. Moreover, experimental comparisons were weak, and limited to one competing regularizer (WGAN-CLC), applied to three GAN losses (WGAN, WGAN-GP, WGAN-CP). ", "strength_and_weaknesses": "Strengths:\n- Interesting perspective on GAN training.\n\nWeaknesses:\n- Theory is limited. Convergence results of the proposed method are restricted to the DiracGAN setting in a two-dimensional parameter space.\n- Experiments could be more compelling. The authors claim that their method is an improvement over a previous work in a very similar direction that also applies control theory to GAN training -- WGAN-CLC [1]. However, the results reported in Table 3 deviate significantly from the results in [1]. Namely, [1] report much better FID scores than reported here. Can the authors comment on this discrepancy?\n- Introduces three new hyperparameters $\\rho_1$, $\\rho_2$, and $\\beta$. It is unclear how sensitive they are to perturbations, when applied to models trained on more realistic data (e.g. CIFAR-10, CelebA).\n\n[1] Xu, K., Li, C., Zhu, J. and Zhang, B., 2020, November. Understanding and stabilizing GANs’ training dynamics using control theory. In International Conference on Machine Learning (pp. 10566-10575). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-EjR8qJTeEG", "reviewer": "Reviewer_YEms", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a stochastic controller to stabilize the training process of GANs. It shows in a simple Dirac-GANs case, the proposed method converges theoretically to a unique optimal equilibrium. For general GANs, a modification of the controller is proposed and numerical results are given to show the improved stability of the training process in terms of common benchmarks for image generations. ", "summary_of_the_review": "I tend to stay with my current score, as I find that the current version could still be improved for a future submission. ", "strength_and_weaknesses": "The strength of the paper is that the proposed methodology is very interesting and it provides a new way to deal with the circling issue in bilinear games. However, I am not fully convinced of the stability result, as well the convergence of the modified version of the controller for general GANs. Here are a few questions:\n- What is the behavior of the modified version (in eq. 17) on Dirac-GANs? Is there still any convergence as in Theorem 2?\n- When one talks about the stability of the training process, would it make more sense to check whether the losses of the discriminator and generator converge, compared to the use of FID or inception score? Somehow it is not clear why the training can be stable in terms of these scores. \n- The discussion at the beginning of Section 3.2 regarding the optimal solution is too ideal. In practice, a normal GAN only has access to finite samples of p(x), so it is not clear why one can have p_G(x) = p(x), and D(x)=0 as an equilibrium. More seriously, the numerical results are based on the use of WGAN-GP, where the GP penalty does not make D(x)=0 (as it encourages that the gradient of D has a unit norm). Thus the idea of BRGAN seems not compatible with the GP, nevertheless wgan_gp_br seems to work best in Fig 2 and 3. I think more discussions are needed regarding these aspects to explain how wgan_gp_br works, and why it works well. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject