Understanding and Stabilizing GANs' Training Dynamics with Control Theory
Authors: Kun Xu
Chongxuan Li
Huanshu Wei
Jun Zhu
Bo Zhang
Venue:  (2020)
Keywords: Generative Adversarial Nets
Stability Analysis
Control Theory

Abstract: Generative adversarial networks~(GANs) have made significant progress on realistic image generation but often suffer from instability during the training process. Most previous analyses mainly focus on the equilibrium that GANs achieve, whereas a gap exists between such theoretical analyses and practical implementations, where it is the training dynamics that plays a vital role in the convergence and stability of GANs. In this paper, we directly model the dynamics of GANs and adopt the control theory to understand and stabilize it. Specifically, we interpret the training process of various GANs as certain types of dynamics in a unified perspective of control theory which enables us to model the stability and convergence easily. Borrowed from control theory, we adopt the widely-used negative feedback control to stabilize the training dynamics, which can be considered as an $L2$ regularization on the output of the discriminator. We empirically verify our method on both synthetic data and natural image datasets. The results demonstrate that our method can stabilize the training dynamics as well as converge better than baselines.

Under review as a conference paper at ICLR 2020
UNDERSTANDING AND STABILIZING GANS' TRAIN- ING DYNAMICS WITH CONTROL THEORY
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) have made significant progress on realistic image generation but often suffer from instability during the training process. Most previous analyses mainly focus on the equilibrium that GANs achieve, whereas a gap exists between such theoretical analyses and practical implementations, where it is the training dynamics that play a vital role in the convergence and stability of GANs. In this paper, directly model the dynamics of GANs and adopt the control theory to understand and stabilize it. Specifically, we interpret the training process of various GANs as certain types of dynamics in a unified perspective of control theory, which enables us to model the stability and convergence easily. Borrowed from control theory, we adopt the widely-used negative feedback control to stabilize the training dynamics, which can be considered as an L2 regularization on the output of the discriminator. We empirically verify our method on both synthetic data and natural image datasets. The results demonstrate that our method can efficiently stabilize the dynamics and converge to better results.
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have shown promise in generating realistic natural images (Brock et al., 2018) and facilitating unsupervised and semi-supervised learning (Chen al., 2016; Li et al., 2017; Donahue & Simonyan, 2019). In GANs, generator (G) is defined by mapping noise distribution to the data space. Since no density function defined for the implicit generator, a discriminator (D) introduced estimate the density ratio between the data distribution Pp and the generating distribution pG by telling the real samples from fake ones. G aims to recover the data distribution by maximizing this ratio. This framework is formulated as minimax optimization problem, which can be solved by optimizing G and D alternately. In practice, however, GANs can be extremely sensitive to hyperparameters (Lucic et al., 2018; Radford al., 2015), and oscillations are often observed (Liang al., 2018; Chavdarova & Fleuret, 2018), thereby suffering from the instability of training (Goodfellow, 2016).
implicit
There some theoretical analyses aiming understand and stabilize the training process of GANs, such as designing different objective functions using other statistical divergences (Nowozin et al., 2016; Nguyen al., 2017; Du et al., 2018; Mao et al., 2017), and introducing auxiliary regularization terms (Gulrajani et al., 2017; Arjovsky et al., 2017). However, there is a gap between the theoretical analyses and practical implementations: most of previous work assumes that (1) D achieves optimal when training G (Goodfellow et al., 2014; Arjovsky et al., 2017; Guirajani et al., 2017; Nowozin et al., 2016); and (2) the optimal D is smooth function of G (Metz et al., 2016). These two assumptions are violated in most practical cases. First, GANs are optimized by alternating between i steps of optimizing D and one step of optimizing G, where i generally takes 1 or 5. It results in a sub-optimal D, especially in the early stage of the training process. Second, D is constant w.r.t. G when G is optimized, which results in missing gradients and an unstable training process (Metz et al., 2016).
Instead, to ensure practical convergence, it is the training dynamics of both G and D, i.e., how G (or D) changes given current G and D during the training process, that should be considered, where the previous two assumptions are more required. There are some methods that focus on the dynamics of the parameters. Mescheder et al. (2017) and Nagarajan & Kolter (2017) propose to model the dynamics of the parameters using the vector field defined by its gradients. In such cases, the local convergence and stability are fully determined by the eigenvalues of the Jacobian matrix. Mescheder et al. (2018) further propose regularization term to stabilize GANs by adjusting the eigenvalues

---

Under review as a conference paper at ICLR 2020
of the Jacobian matrix. Gidel et al. (2018) analyze the effect of momentum and propose to stabilize GANs using negative momentum. In these methods, the Jacobian matrix is the key to understand and stabilize the dynamics, whose calculation is, however, computationally expensive. It largely impedes us to generalize this analysis to complex models such as neural networks (LeCun et al., 2015).
In this paper, we argue that modeling the dynamics in the function space is more convenient for convergence analysis. Specifically, we directly model G and D as two dynamics whose output is considered as a signal of time i.e., representing the output of G and D G(t, z) and ((1, respectively. The control theory (Kailath, 1980) provides a powerful tool to understand them and analyze their stability. Specifically, differential equations can be used to denote the dynamics, e.g.. the dynamics of G' can be denoted as dG't.2) = f(G, D), which represents how G changes given current G' and D. By applying Laplacian transformation (Widder, 2015) to the differential equations, the dynamics can be represented as a transfer function, and then the stability and convergence can be easily modeled (Kailath, 1980) introduced in Sec. 2.
de
Under the above perspective of control theory, we unify the dynamics of G and D as certain dynamics that are well-studied in control theory for various GANs, including Standard GAN (SGAN), WGAN and LSGAN (Mao et al., 2017), as illustrated in Sec. 3 and Appendix A. Specifically, the training process of GANs can be considered as dynamic whose output is determined by its intrinsic properties (i.e., the objective function of G and D) and its input (i.e., the data distribution). Within the proposed framework, a variety of existing methods (Mao al., 2017; Mescheder et al., 2018; Gidel et al., 2018) 8) can be considered as certain controller which is widely used in control theory as discussed in Sec. 1.1 and Appendix C. It worth noting that through control theory, the stability of GANs can be easily inferred from the transfer function, instead of analyzing the complicated Jacobian matrix of the dynamics. The proposed framework provides a promising direction that can further benefit the training dynamics of GANs using advanced control methods. To verify it, we propose to use the widely adopted controlling method, the negative feedback (NF) control (Aström & Hägglund, 1995), to stabilize GANs. What's 's more, when applying NF to stable models, the performance can still be improved. The proposed NF acts as a regularization term which penalizes the L2 norm of the output of D as we described in Sec. 4. NF is verified on the toy data such as Dirac GAN as well as the natural images such as CIFAR1O (Krizhevsky al., 2009). The results demonstrate that our method can successfully stabilize the dynamics of GANs and outperform the baseline significantly.
2 PRELIMINARY
In this section, we provide the background and preliminary about Laplacian transformation (LT) and control theory. LT a powerful tool to present an ordinary differential equation (ODE) as a rational fraction. Since most dynamics can be represented as an ODE, LT can largely simplify the analysis of stability by merely analyzing the properties of the rational fraction.
2.1 LAPLACIAN TRANSFORMATION
Laplacian transformation (LT) (Widder, 2015) can be considered as an extension of Fourier transfor- mation (FT), which transforms signal, i.e., function of time, to function of complex variables. Formally, the transformation and its inversion are given by:
=()()= le [(0)e= FF) = 2 c-oci
1 ctoii
e (())ds
(1)
where f is a signal that is represented function over time t,, c is constant that ensures F(s) < co. E C is a complex number, i.e., = a + wi with real numbers a and w and for each E C, F(s) denotes the components of certain frequency in the original signal f similar to FT.
With LTT the integration and derivation can be presented as a single operator, which is given by:
-())) FLf. ((w)d))= = -
di
With this property, an ODE can be converted to an algebraic equation by substituting derivation as an operator s. simple example illustrated as follows:
d2x x
- x
x
(3)
di
2

---

Under review as a conference paper at ICLR 2020
U(s)
Hs(s)
Hy (s)
X(s)
1 (5) 52 +1
U(s)
x())
Hy(s)
Figure I: The dingram represcntations the inputs and dynamics after LT. Left: the original dingram of the spring dynamic in Eqn. (3), Right: the diagram of classical controlled dynamic with feedback H. and controllr H.
where u and x: are the input and output signal correspondingly and U and X are their Laplacian transformation. This ODE can represent spring dynamic, whose acceleration dex depends on the de input u and the current position x according to Hooke's ' law. In this case, the ODE can be solved by simply applying the inverse Laplacian transformation to X - (s) Here (s)()) = क is the transfer function of this dynamic which can also be represented as a diagram illustrated in Fig. Since the input signal can be considered as summation of infinity Dirac signal at different positions, i.e., [(t) f [(u (u))(t u)du where i denotes the Dirac delta function (Dirac, 1981), we only need to analyze the output the dynamics for Dirac signal whose LT is constant, i.e., F(6) 1. Therefore, the transfer function represents the intrinsic properties of the dynamics.
2.2 CONTROL THEORY AND NEGATIVE FEEDBACK
Given dynamic, equivalently a transfer function, the stability can be analyzed according to the roots of the denominator the transfer function, e.g., 8² + the poles of the dynamics. Specifically, if all poles' real parts negative, the dynamic is stable; if there are poles which are pure imaginary numbers, the dynamic will oscillate; if there is at least one pole with positive real parts, the dynamic will diverge (Kailath, 1980). In the above example, the poles of the dynamic are () i, which indicates that the object will oscillate around the original.
for Eqn. (3). The roots are called
For those unstable dynamics, of the most commonly adopted methods here is introducing negative feedback with an auxiliary controller (Kailath, 1980). The diagram illustrated in Fig. (right). The Hs(s) denotes the transfer function of feedback and He (s) denotes the transfer function of controller.
need to stabilize it by adjusting the poles of the dynamics. One
In this setring, the dynamic takes the error (i.e., F((s) (s) Hs(s)X(s)) rather than U(s) as input, resulting the fact that the transfer function of the whole controlled dynamic is given by:
Ho(s) + + 1 (c)
U, (4) which provides approach to adjust the poles of unstable dynamic. For most cases, simply adopting the negative feedback, i.e., let H₁ = H. = 1. is enough to stabilize a dynamic. In the following, we provide the dynamics of GANs in the perspective of control theory and introduce the negative feedback control to stabilize the training process of GANs.
E(s) - U(s) Hs(s))))))))))))) Hs(s) X(s) H.(s)
3 UNIFYING THE DYNAMICS OF GANS IN CONTROL THEORY
In this section, we provide novel perspective on the dynamic of GANs. Unlike previous methods that mainly focus on the dynamics of the parameters, we directly model the output of two dynamics G' and D in the function space. In this part, we mainly focus on the Dirac GAN as in Mescheder et al. (2018) and unregularized WGAN, This method can be extended to other GANs easily, as we illustrated in Appendix A.
To model the dynamics of GANs, we first denote the output G and D as a function with respect to time t, which is also known as signal, i.c., D D(t,z) and G G(t, 2). Then the training process can be represented as a dynamic using an ODE, and the equilibrium is equivalent to the final value of the dynamics (Oppenheim al., 1998), i.e., the output of the dynamics with t
3.1 THE DYNAMICS OF DIRAC GAN
In Dirac GAN, the data distribution is defined (x) 6.(x)) 8(x - c) which is the density of an idealized point mass at point c. The generator distribution is defined as Po(x) = 6ạ(x,, where is the learnable parameter of G. In this case, G(t) G(t, 2) for simplicity as its output does
3

---

Under review as a conference paper at ICLR 2020
1 Hp:
(x()
Px(
Hp 8
Px())
(x)
1 Ho! = $
a(x) - Pc(x)
Hg* 1 S
Ox
Figure The diagram block Dirac GAN and unregularized WGAN Left: Dirac GAN, Right: unregularized WGAN
depend on 2. The discriminator simple linear function (tt, 2)) <(t) x æ. Therefore, the minimax optimization problem of Dirac GAN is formulated as:
min max Epp [ox] Epa = cΦ fΦ.
(5)
G
D
For the discriminator, the objective function is given by: max UDD cΦ 0b. which indicates that the dynamics D can be formulated as:
(U) cc dø(t) c @D x)
(c (c-0), Vz.
as
Ot
By considering as the error signal, i.e., e = c- 0, the dynamics of the discriminator can be modeled integral part (tt, x) x So c- G(t)dt. By applying LT to the dynamic, we have:
F(D(t,z)) x:F(e) E(s)
where consider the dynamics of D each For the generator, the objective function is given by maxg (G) = 0Φ. Therefore the dynamics of G can be formulated as:
(U(G) = - d0(1)) = aG(1) 00
@D(1,)) Ot
(8)
dt
Or
Since Dtt,x)= x x G(t)dtt, we have aD(L.z) fex c G(t)dt. We therefore combine the x part in D's dynamic and the the partial derivation in G into identity mapping. Then D can be considered as an integral dynamic and its transfer function is given by H x. Further, the generator also integrate the output of D, which is also
8g
1
Hence, the block diagram of the Dirac GAN is illustrated in the left panel of Fig. 2. As we can see, D takes the difference between the real sample and fake sample, i.e., A input, and output the integration of the input signal. The generator takes D's output as input and output the generator distribution which is the integration of D's output.
Remark 1. When treating G and D as dynamics, the both the inputs and outputs of the dynamics are functions over t. For example, the input of D is (t, pa (t, (f, x)) and the output is (tt,
3.2 THE DYNAMICS WGAN
In this subsection, further model the dynamics of WGAN (Arjovsky et al., 2017) without Lipschitz continuity constraints in the function space. this case, we consider the nonparametric setting, where D is piece-wiss linear function, G is mapping from the noise space to the data space, and both of them are of infinite capacity. This analysis can be extended to other variants of GANs easily.
In the general case, the objective function of D is given by:
max U(D) = Ep, [(x)] - Epc [(x)) - (PD(x)) D
(9)
According to the calculus of variations (Gelfand et al., 2000), the gradient of UDD) with respect to the function D, and the dynamic of D) are given by:
@U(D) aD
dD(t,)) pn(x) - po(x).
PG
(10)
dt
Since the input signal is Pn Pc. the dynamics of WGAN is similar to that the Dirae GAN, whose transfer function can be denoted as (s(s)
4

---

Under review as a conference paper at ICLR 2020
For the generator, the objective function and its dynamics are given by:
max U(G) = (x) = p()D aU(G)
@D(G(z)) DG(z)
8G p=(2))
Vz. (11)
is worth noting that compared to Dirac GAN, this integral is taken in the function space, i.e., G(t
0, G(t, + (z)
aG( Besides, the generator distribution PG is not Dirac distribution, "
resulting in that another operation, which converts the generated samples to the distribution, should be added after G to make the model consistent. The diagram blocks of WGAN illustrated in Fig. 2 (right). Compared to Dirac-GAN another two operations further provide the dynamics of other GANs, including standard GAN proposed by Goodfellow et al. (2014) and Least-Square GAN proposed by Mao et al. (2017) in Appendix A.
introduced to the framework. We
Remark 2. We only consider the unregularized WGAN to simplify the analysis. Note that the dynamics of D follows Eqn. (10) least locally around the equilibrium where updating D according to Eqn. (10) will not violate the Lipschitz constraints. Formally proof is given in Appendix D.
Remark 3. Here we model the dynamics of G and D in the functional space, which is more convenient for the following stability analysis and designing controllers. In practice, we update the models in the parameter space approximate the dynamics in Eqn. (10) and Eqn. (11).
4 UNDERSTANDING AND STABILIZING GANS WITH NEGATIVE FEEDBACK
In this section, we first provide the stability analysis of GANs from the perspective of control theory. Besides, many previous methods can be interpreted this perspective, and we use the momentum as example. Other further examples are given in Appendix C. We further propose to stabilize the dynamics of GANs using the negative feedback controlling methods, and an example of another controller is given in Appendix B.
4.1 STABILITY
In the dynamics GANs, we argue that it is dominated by the two integral dynamics G and D. Therefore, ignore the two non-linear operations in GANs in the following analysis. Since both G and D can be considered as integral parts, we can formulate the transfer function of and as He HD 1. According Eqn. (4), the transfer function of WGAN is given by:
HD H(s) 1+ HDHG
1/8 1/s2
8
(12)
+
is worth noting that the poles of the transfer function of the whole dynamic are 0 + i, which indicates that the output will oscillate around the equilibrium rather than converge to O. This result is also consistent with the analysis in (Mescheder et al., 2018). This perspective can also partially explain why larger learning rates or adopting multiple update steps for D, where the transfer function of D can be approximated by k with & > 1, cannot provide principle solution for stabilizing GANs. In this case, the poles of the dynamics are adjusted to 0 + Vki which still oscillation dynamics.
4.1.1 THE AFFECT OF MOMENTUM
Our method can also provide stability analysis for other practical techniques used in training GANs. In this part, we use the momentum as an example. Gidel et al. (2018) provide some theoretical analysis of momentum in training GANS, which is formulated in the parameter space. In the following, we re-analyze the momentum in the function space under the perspective of control theory.
The momentum method (Qian, 1999) achieves great success in training deep neural networks. Previous methods such as SGD with momentum can help neural networks escape from local optima. Its theoretical formulation is given by:
θ = 30, + (1 Ve, 0c+1 = + T/.+1-
(13)
The 3 is the coefficient for the exponential decay of the gradients, which generally takes 0.9. However, things changed when it comes to training GANs. DCGAN (Radford al., 2015) recommends smaller momentum with 3 = 0.5. Furthermore, recent state-of-the-a models (Mescheder et al., 2018; Brock al., 2018; Guirajani al., 2017) just remove the momentum.
5

---

Under review as a conference paper at ICLR 2020
Pb(x) +
Hp: -
Pa(x)
HND: A
HG:
(a) The diagram of GANs with negative foodback.
(b) The dynamics of GANs with nogative feedback.
Figure 3: The diagram its corresponding dynamics of GANs using nogative foodback..
Algorithm 1 Negative Feedback GAN
1: Input: Buffer size No, feedback coefficient X, batch size N, initialized 0 and Φ, learning rate 1).
2: Initialize B, and B, for real samples and fake samples respectively.
3: repeat
4: Sample a batch of { ~ >DD { ~ PG of N samples.
5: Update B, with {.. Update By with {
6: Sample a batch of uf B,, a'g ~ By of N samples respectively.
7:
Estimate the objective D:
U(D) = 2 D(z) - Ex((,,) D(x)] - 인े ExEe))) D²(x) + Ene(ot) D²(x)).
8:
Update á to maximize (D) with learning rate n).
9: Estimate the objective of G: (G) = N Ene(ay) ((x).
10: Update 0 to maximize (G) with learning rate 17.
11:
until Convergence
In control theory, the momentum is equivalent to adding an exponential decay to the input of the dynamics (An et al., 2018), and the LT of an exponential decay dynamic is 1 x+r. T > 0 denotes the decay coefficient which depends on 3. Therefore, the transfer function of the D with momentum is HsD(s) a(str)+
1 and the transfer function of the WGAN with momentum is given by:
HDD
1/(s(s +T)
H(s) 1+ HmD HG 1 1 (+1((((s+7))
(14)
33 + 782 +1
By letting (s + (s + (s) there are at least one pole of this dynamic whose real part is larger than 0, indicating the instability of the dynamics for GANs with momentum.
+)) 8 78 1, we have that b-bbb-=- O, which indicates that
4.2 STABILIZE GANS
According to the above analysis, for both the unregularized WGAN and the GAN with momentum, the equilibrium point (i.e., PG = PD) is not a stable point. the following, we propose to stabilize GANs using negative feedback control, which is the most widely-used method in control theory.
In this setting, we set Hxr = À which is the transfer function of the negative feedback for D. After applying the negative feedback to D, The diagram of the controlled dynamics is given in Fig. 3 (a). According to Eqn. (4), the transfer function D and the whole dynamic can be represented as:
1/s (b) 1+8
(')ss
H(s) = 1+ + (+)
(15)
8 8+
32 + +
Therefore, for positive and small X, the real part of the two poles are < O. which indicates that the dynamic stable. The dynamics of the Dirac GAN using different X is illustrated in Fig. (b). As we can see, when no feedback is applied (i.c., 0), the dynamic will oscillate around the cquilibrium. Applying negative feedback can significant improve the stability of the dynamics.
4.2.1 PRACTICAL IMPLEMENTATION
The most direct way to add negative feed to introduce a regularization term RD = 1/2 (x) to the objective function of D. However, directly integrating over the input space is computationally infeasible. Instead, we give an approximation to Rp. We assume that the area where [D(x) is large mainly concentrates its training samples, i.e., previous samples used train D. Therefore, keep two buffers B, and B, of size N₂, to store the old real samples and fake samples, respectively. Then Rp is evaluated on these buffers regularize D, and the two buffers replacement at each step. The training procedure is illustrated in Alg. 1. In the following, we use NF-GAN (-) denote our proposed method with the hyperparameters X/2 denoted in the parentheses.
updated with

---

Under review as a conference paper at ICLR 2020
80
WCAN
ViGAN
Inception Scare
60
2CAN
6GAM
40
Reg W-SM
Vice
20
Time/tours
Time/hours
Figure 4: Thee learning curve buselines and our proposed methods. Left: The Inception Score of CIFARIO. Right: The HID score for CelebA. We plat the with respect time for better representation the compulational cost and convergence speed.
RELATED WORK
GANs are known for its unstable training process. Radford al. (2015) provide several practical techniques to stabilize the training, such as designing suitable architecture of neural networks and the hyperparameters selection. The Unrolled GAN (Metz al., 2016) proposes to make D function of G by unrolling the training process of the discriminator where the gap between the theoretical analysis and implementation can be reduced. Arjovsky al. (2017) provide theoretical analysis of the gradient vanishing problem and further propose to optimize GANs using Wasserstein distance. Gulrajani et al. (2017) use the gradient penalty to ensure the Lipschitz continuity in discriminator, which is required by WGAN. However, GANs still suffer from instability in the training process.
Besides, there some work that directly focuses on the stability of the training process. Except for the works mentioned in Sec. 1, Peng et al. (2019) also directly model the dynamics of the parameters and adjust the update rules to stabilize GANS. Gidel et al. (2016) propose find the saddle point of the optimization problem using Frank-Wolfe algorithms. Feizi et al. (2017) also use the control theory to model the stability of GANs, while the dynamics GANs are not explicitly captured. Balduzzi et al. (2018) decompose dynamics into the oscillating and non-oscillating behaviour and model training dynamics as finding stable fixed points. However, none of them only can generalize their methods to complex datasets such as natural images.
EXPERIMENTS
In this section, we empirically verify our proposed method on the widely-adopted CI- FARIO (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015) datasets. We denote the regularization proposed in Mescheder al. (2018) Reg-GAN, which is the direct baseline of our method. Besides, we also apply NE-GAN to SN-GAN (Miyato et al., 2018) and provide a significant improvement to implementation mainly follows the officially
the ffate-ofthhe-ee art method. For fair comparisons, released code for Reg-GAN and SN-GAN, respectively. More details about the experimental setting and further results synthetic dataset can be found in Appendix B. The code is provided here.
6.1 COMPARISON TO REG-GAN
Table I: The FID Score (top) and Inception Score (bottom) on CIFARIO. The results reported here the best results the training process. The results of IS are averages over 3 runs.
We first compare our proposed method with Reg- GAN. Since for large datasets, the non-linearity cannot be ignored, which results in the gap between the simulation of Dirac GAN and our methods. A small X/2, such as 0.01 which is suitable for Dirac GAN is not enough for stabilize the large mod- els. We therefore select the coefficient X/2 among 2,, 10, 15, 20). In this part, we only report representative results. Though our analysis mainly focuses the WGAN framework, NF can also cas- ily generalize to SGAN whose stability can also be improved and NF-SGANs with different coefficient X/2 are evaluated.
Regularization
WGAN
SGAN
No regularization
105.21
28.51
Reg-GAN
30.43
28.39
Gradient Penalty
28.20
NF-GAN(2) (ours)
21.90
22.35
NF-GAN(5) (ours)
22.31
21.61
NF-GAN(10) (ours)
21.19
23.34
Objective
SN-GAN
NF-SN-GAN
WGAN
3.29
8.28
.09
Hinge
8.22 + .05
8.45 + .11
We use the Inception Score (IS) (Salimans al.. 2016) evaluate the image quality CIFAR 10 and FID score (Gulrajani et al., 2017) on both CIFARIO and CelebA. The quantitative results are given in Table and the learning curve is reported in Fig. 4. Our method gives improvement to the FID
7

---

Under review as a conference paper at ICLR 2020
Figure The generated results of CIFAR dataser. From left right: WGAN-GP, Reg. WGAN, WG SFANIS
OO
Figure 6: The generated results of CelebA dalaset. From left to right: WGAN-GP, Reg WGAN, NF-WGAN(15), NG-SG))(
and IS for both WGAN and SGAN. Specifically, for unregularized methods, the models demonstrate the unstable training process and finally diverge away from the data distribution. Compared with Reg-GAN, our method maintains stability and provides better convergence results, as also shown in Table 1 (top). Besides, our method is more computationally efficient, since regularizing the output of D needs less computation compared to regularizing || [| || and this gap is significant for large dataset. Specifically, on Geforce 1080Ti, our method can conduct approximate 12/8 iterations per second of training for CIFAR 10/Celeb,, whereas Reg-GAN can only conduct 10/4 iterations per second. What's 's more, our method is robust to the choice of the hyperparameter A. We present the generated images in Fig. 5 and Fig. 6.
6.2 COMPARISON TO SN-GAN
We further apply NF-GAN the SN-GAN'S architecture. With stability guaranteed, our proposed method can further help SN-GAN to address the potentially unstable issue and converge to better final results, as illustrated in Table (bottom). We manually select A/2 E {0.01, 0.1, 1} and finally adopt X/2 = 0.1. Though SN-GAN provides an elegant normalization method to ensure the Lipschitz continuity, SN-GAN with WGAN loss still cannot converge to the data distribution and provide unreasonable results, which verifies with our stability analysis. Instead, the hinge loss, as introduced Eqn. (16)&(17) in Miyato et al. (2018), is used to stabilize SN-GAN but no theoretical guarantee is provided. In contrast, NF-GAN can successfully regularize SN-GAN with WGAN loss and provide better results to previous state-of-the-an results. What's 's more, NE-GAN can also benefit SN-GAN with Hinge loss and provide significant improvement the IS of CIFAR-10.
7 CONCLUSION
In this paper, we propose a novel perspective to analyze the dynamics of GANs. Instead of focusing on the equilibrium, we directly model the dynamics of the discriminator and the generator during the training process. Using Laplacian transformation and control theory, the stability can be easily understood according to the poles of the dynamics. Various methods can be motivated from this perspective, and we verify it with the widely used negative feedback control. The empirical results demonstrate that our method can successfully stabilize GANs and provide better convergence results.
Negative feedback shows promise results, but further analysis is required for better results. For one thing, our analysis mainly focuses on the continuous cases, where the practical implementation optimizes both G' and D in discrete time steps, where the Z-transformation provides better tools for discrete-time dynamics. For another, two approximations are made: using the update in the parameter space to approximate the dynamics and ignore two non-linear operations in GANs. Recent analyses of GANs on the functional spaces (Johnson & Zhang, 2018) provides a promising solution to solve the first approximation and modern control theory and non-linear control methods (Khalil, 2002) can be adopted to solve the second one. Indeed, our method does not provide a perfect solution to GANs' dynamics. It does provide novel perspective understand and stabilize GANs.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Wangpeng An, Haoqian Wang, Qingyun Sun, Jun Xu, Qionghai Dai, and Lei Zhang. A pid controller the approach for stochastic optimization of deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8522-8531, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pp. 213-223, 2017
Karl Johan Áström and Tore Hägglund. PID controllers: theory, design, and tuning, volume 2. Instrument society of America Research Triangle Park, NC, 1995.
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanics of playa differentiable games. arXiv preprint arXiv:/802.05642, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Tatjana Chavdarova and François Fleuret. Sgan: An alternative training of generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9407-9415, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Paul Adrien Maurice Dirac. The principles of quantum mechanics. Number 27. Oxford university press, 1981.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv preprint 2019.
Chao Du, Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Learning implicit generative models by teaching explicit ones. arXiv preprint arXiv:1807.00 100.03800000870 2018
Soheil Feizi, Farzan Farnia, Tony Ginart, and David Tse. Understanding gans: the lgg setting, arXiv preprint arXiv: 10793, 2017.
Izrail Moiseevitch Gelfand, Richard A Silverman, et al. Calculus of variations. Courier Corporation, 2000.
Gauthier Gidel, Tony Jebara, and Simon Lacoste-Julien. Frank-wolfe algorithms for saddle point problems. arXiv preprint arXiv:1610.07797, 2016.
Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Remi Lepriol, Gabriel Huang, Simon Lacoste-Julien, and loannis Mitliagkas. Negative momentum for improved game dynamics. arXiv preprint arXiv: 1807.0::000.444 007400444040.000 2018.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323, 2011.
lan Goodfellow. Nips 2016 tutorial: Generative adversarial networks.
arXiv preprint
arXiv:1701.00160. 2016.
Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14:8, 2012.
Rie Johnson and Tong Zhang. Composite functional gradient learning of generative adversarial models. arXiv preprint arXiv:1801.06309 2018.
Thomas Kailath. Linear systems, volume 156. Prentice-Hall Englewood Cliffs, NJ, 1980.
Hassan K Khalil, Nonlinear systems. Upper Saddle River, 2002.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(7553):436, 2015.
Chongxuan Li, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In Advances in neural information processing systems, pp. 4088 4098. 2017.
Kevin Liang, Chunyuan Li, Guoyin Wang. and Lawrence Carin. Generative adversarial network training continual learning problem. arXiv preprint arXiv:/811 I. 1083, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in neural information processing systems, pp. 700-709, 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802, 2017.
Lars Mescheder, Sehastian Nowozin, and Andreas Geiger. The numerics of gans, In Advances in Neural Information Processing Systems, pp. 1825-1835, 2017.
Lars Mescheder, Andreas Geiger, and Sehastian Nowozin. Which training methods for gans do actually converge? arXiv preprint (.0iv:/801.0440 2018.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. ArXiv, abs/1611.022 1.0163,, 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for 2018.
generative adversarial networks. arXiv preprint
Vaishnavh Nagarajan and J Zico Kolter. Gradient descent gan optimization is locally stable. In Advances in Neural Information Processing Systems, 5585-5595, 2017.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2670-2680, 2017.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Alan Oppenheim, Alan S Willsky, and S Hamid Nawab. Señales sistemas. Pearson Educación, 1998.
Wei Peng, Yuhong Dai, Hui Zhang, and Lizhi Cheng. Training gans with centripetal acceleration. arXiv preprint arXiv: 0902.9002200 0092999949 2019.
10

---

Under review as a conference paper at ICLR 2020
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1): 145-151, 1999.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.064533 2015.
Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp- 2234 2242, 2016.
David Vernon Widder. Laplace transform (PMS-6). Princcton university press, 2015.

---

Under review as a conference paper at ICLR 2020
A
GANS AND THEIR CORRESPONDING DYNAMICS
In this section, we provide an illustration for different kinds of GANs and their corresponding
dynamics.
A.1 VANILLA GAN AND - GAN
The GANs proposed by (Goodfellow et al., 2014) are formulated as follows:
min max Ep,, (log a (x))) Epg - =(D(z)))
G DDD
Therefore, the objective function and the corresponding dynamics of D are given by:
The objective function:
max D
U(D) =Epp llog ( -
= po(x) log
1 + exp(-D(x)) + (c) log exp(- [(()) de
1+exp(- D(a))
= =- pD( log(1 exp(-DD(t)) + PG (z) log(1 + exp(D(x)))da
Given current G, the gradient of U(D) with respect to D at certain æ given by: AU(D)
exp(D(x))
dD(x) 1 1+ exp(-D(x)) PG 1 1+ exx((((())) PD
PD (D(x))) - -PoO (D(z))
dD.(x) (U(D(x))
dD pD -
di
The objective function and the corresponding dynamics of G are given by:
The objective function:
min U(G) = =Epa [log(1 - oo(((=))) = E,, llogl- -
G
/ p=(2)) log exp(-D(G(z)) 1+ L+ex(-DD(G(2)) dz
=-1x /
cxxp
Since G is optimized using reparameterization, we rewrite G(z) as x; for simplicity. Given current D, the gradient of U (G) w.r.t. G at certain is given by: exp(D(<x2) @D(x.) - x p(z)o(D(z..)
aU Ox
aD(x.)
p(2) (z) = (D))o aD(xz) dl
1 exp(((xx))) dx,
Or:
Ix:
For well-trained D, which can successfully tell real samples from generated samples with high confidence, i.e., Dxx2) x 0 for all the gradient for G will tend to zero almost everywhere, which will result in gradient vanishing. This analysis is consistent with previous literature (Johnson & Zhang, 2018; Arjovsky et al., 2017).
For non-saturation GAN, the objective function and dynamics of G are given by:
The objective is given by: max U(G) = =Bpa [log (D())) Epg
/ po()) log exp(D(G(2))) dz I+ex)(D(G(z))
/ (p(())(((C()) log( se(1+ead +xx((D(G())))
12

---

Under review as a conference paper at ICLR 2020
We rewrite G(2) as 2; and provide the gradient of (G) w.r.t 2; follows:
(U(G)
p=(2)(1 - exp(D(x=)) @D(2))
Oz.
1+ (sz()()=)) dr:
- pp=(2) - o(D(x,))) ( (D(zz)
00,
dGr(2) = -1 -(()(L-((((z))) (D(z.)
dr,
A.2 LEAST- GAN
(Mao et al., 2017) propose to use the least square loss to train both the discriminator and the generator. The objective for the D and G selected differently according to its theoretical analysis. Specifically, the objective function of D and G are given by:
min (D) E. (D(x) - 1) + 1 Pr (D()))
2
min (G) 2
Ep (D(((())) - -)
Therefore, the dynamics of the D and G are given by:
The dynamic of D:
(U(D) 1 a 0D(z) 20D(z) (Dz( 1)² (x)()²)
PD(D(z) 1) + p(zx)..
dDv(r)
dt PD (pD + (z)
(16)
The dynamic of G:
OU(G) (W(xs) -1) (z(zz)
Or
dG((z) ps (=) - D(xx)) (D(xz) dt
It is worth noting that if the G is fixed, the dynamics of D can be considered decay which is the same as the D with negative feedback control. However, since G is changing during the training process, the dynamics of D is actually coupling with the dynamics of G. which generally results in the unstable dynamics (Kailath, 1980). The experimental results on CIFARIO also verify this argument. It also explains why we use two buffers of old samples to approximate the regularization term instead of directly regularizing the output of D using current samples.
exponential
B FURTHER EXPERIMENTAL RESULTS
B. DETAILED EXPERIMENTAL SETTINGS
In this paper, we mainly focus on the stability of the training dynamics of GANs, and therefore directly compare our proposed method NF-GAN with the vanilla SGAN/WGAN and the most direct baseline Reg-GAN. We use the resnet (He et al., 2016) for both the generator and the discriminator following Mescheder et al. (2018) and adopt the ReLU activation (Glorot al., 201 1). The batch size 64, and the buffer size is set to 100 times of the batch size for all settings. We use rmsprop (Hinton et al., 2012) optimizer with learning rate 0.0001 and 0.99 for all models.
Note that we make the following two modifications to the original implementation of Reg-GAN which results in more challenging setting the stability. First, in the original Reg-GAN,
13

---

Under review as a conference paper at ICLR 2020
WGAN
SGAN
Reg-WGAN
NF-GAN(0.1)
NF-GAN(0.01)
The generated samples for mixture of gaussian distribution. The red points demonstrate the location of data distribution and bluc points are generated samples. Each distribution plotted using kernel donsity estimation with 50,000 samples.
coefficient 0.1 is introduced the residual connection of the ResNet for both and Specifically, h h2 0.1 (h)) where fL is the transformation defined in the L-th layer and h is the output of L - 1-th layer as well as the input the L-th layer. We instead remove the coefficient and just use the regular ResNet, i.e., h hz, + fr. (hr.). Second, for both G and D, we use the ReLU activation, rather than leaky ReLU. The above two modifications make the training of GANs more difficult, which explains why the reported results paper. In this more challenging setting, our method still outperforms the Reg-GAN. Specifically. the inception of Reg-GAN is below 6.0 in this difficult setting and about 6.5 in the original setting. In contrast, both NE-WGAN and NF-SGAN achieve more than 6.7 in this difficult setting.
not consistent with the scores in the original
B.2 SYNTHETIC DATA
In this section, we evaluate our proposed method on a mixture of Gaussian on the two dimensions. The data distribution consists of 2D isotropic Gaussian distributions arranged in ring, where the radius of the ring is 1, and the deviation of each component Gaussian distribution is 0.05. For the coefficient X/2, we follow the theoretical analysis and set it among A/2 E (0.01, 0.05, 0.13. We adopt two-layer MLPs for both the generator and the discriminator whose hidden units are 128 and 512 respectively. The batch size 512 and other settings just follow the Reg-GAN.
The generated results are illustrated in Fig. 7 and we further provide the dynamics of the generator distribution in Fig. 8 in Fig. As we can see, the unregularized WGAN and SGAN suffer from severe model collapse problem and cannot cover the whole data distribution. Besides, the oscillation can be observed during the training process of WGAN: the generator distribution oscillates among the modes of data distribution. Our method can successfully cover all modes compared to the WGAN and SGAN and is comparable to Reg-GAN,
B.3 FURTHER CONTROL METHOOD: PD CONTROLLER
Besides negative feedback, the most commonly used controller is the PID controller, which consists of three parts: the partial (P), integration (I) and derivation (D). Specifically, the PID controller takes the error signal as input, and output the summation of the above three parts, whose formal definition is defined as follows:
c(t) = Kpe(t) K2 J" e(u)du + Ka de(t) + Hs(s) Kps + K; + Kas?
(17)
dt
u=0
Kp. Kü, Ka denotes the corresponding coefficients for PID controller. e(t) denotes the error signal and c is the output of the controller, which also the actual input the dynamic. Generally, the I part can reduce the static error, and the D part can make the dynamic stable and accelerate the convergence.
The PID controller is introduced to adjust the poles of the dynamic. Since the static error, i.c.. the difference between the stable point and targeted point, is zero, the integral part is not necessary, and only employ the PD controller in the following. By letting (s)s) = p + ds, where p and d is the corresponding coefficient for the and D part, the transfer function the whole dynamic is given by:
(((s)
s(p + ds)
((s) = 1 H((S))))))
(18)
$2 + ds +
As we can see, given positive p and d, the PD controller can successfully adjust the real parts poles of the dynamic negative. In the following analysis and implementation, fix p for simplicity.
14

---

Under review as a conference paper at ICLR 2020
WGAN
SGAN
WGAN-GP
Reg-WGAN
NF-GAN(O.1)
NF-GAN(O.05)
NF-GAN(0.01)
Figure 8: The training dynamics various GANs on synthetic data.
Po(x)
p + ds
Hp s
Pa(x)
Hg:
(a) diagam
dynamics Controller.
Figure diagram corresponding dynamics of GANs using controller.
15

---

Under review as a conference paper at ICLR 2020
3000
2000
5000
10000
20000
50000
100000
Figure 10: The generated results of PD controller with
1
Po(x)
HD: S
p + ds
(x)
1 Hg:
(t.0
Figure 11: The diagram previous methods, Left: diagram Reg-GAN which takes the 11 regularization. Right: diagram negative momentum logether with weight decay.
The dynamics of Dirac GANs using PD controller is given in Fig. 9 (b). With PD controller, the dynamic can be successfully stabilized. For general GAN, the objective of D time is given by:
U(D,t) (t
[D(z) [ ppe(t..)
(t.) (t,p(t..))D(t) Epp
[x))] (t (D()))
(19)
The regularization is actually minimizing the loss of D in the current step well maximizing the loss of D for previous step.
B.3.1 EXPERIMENTAL RESULTS
The results the synthetic data are illustrated in Fig. 10. WGAN with PD controller can successfully cover the whole data distribution and be stable at the equilibrium.
However, we fail to generalize PD controller to the natural image dataset such as CIFARIO. We argue that the main reason is the overshooting for the dynamies of D, illustrated in Fig. 9 (b), where the D gives a large derivation from the equilibrium at the beginning. Because of the highly-nonlinearity in neural networks, this overshooting will result in unexpected behavior of neural networks. In contrast, for the negative feedback, the overshooting is much smaller, which enables it to generalize the non-linear neural networks.
C UNIFYING PREVIOUS METHODS IN CONTROL THEORY
In this section, we also provide some analysis that interprets previous methods in the perspective of control theory.
C.I REG-GAN
Mescheder et al. (2018) propose regularization which is added on D's gradient of x, i.c., Ra(z) (aD()) I13. Therefore, it can considered an other form of negative feedback, which is illustrated in Fig. 11.
Compare to our analysis, Reg-GAN provides more accurate modeling of the dynamics and directly regularizes the input of the dynamic of G. Therefore, Reg-GAN can also successfully stabilize the dynamic of GANs, which is consistent with our experimental results. Compared to Reg-GAN, our method is more computationally efficient.
C.2 WEIGHT DECAY AND NEGATIVE MOMENTUM
Gidel et al. (2018) analyze the effect of momentum in the training of GANs, and we already interpret this analysis in the context of control theory. Besides, the authors also propose the negative
16

---

Under review as a conference paper at ICLR 2020
momentum to stabilize GANs, which is illustrated as follows:
W++1 = W₁ + VV + >(W, - W2-1)-
(20)
Together with weight decay, the controlled dynamics of W is given:
W4+1 W; v/W ( Wi-1) W2
(21)
In this setting, Eqn. (21) is equivalent to the PD controller with negative feedback the parameter space whose diagram can be illustrated in Fig. 11. The X1 (W, Wi-1) denotes the D part and X₂ W denotes the P part.
D LOCAL APPROXIMATION TO THE REGULARIZED DYNAMICS
In this section, we prove that around the equilibrium, the dynamics of regularized D with Lipschitz constraint is equivalent to the unregularized D as in Eqn. (10). Since the updating direction of D the gradient of in the functional space, we only need to prove that updating / according to Eqn. (10) will not violate the Lipschitz constraints, at least locally around the equilibrium. Here make the following assumptions:
1. Both (D()) and 2a((,)) Cm-smooth: Po 2), VL.
exists and is continuous for pn and dpíz) dx
2. p(x) and
dp(w) when x - for pD and po(l, dz
3. There exists an M such that dp(2]] de l M for po and (t,
The above assumptions satisfied for most probability density functions.
The distance in the function space is defined as d(p1. P2) supaeR= (pi(x) - p2(x)] which always
exists because of the 2-nd conditions above. We define ft {p(x))p(z) C² do(z) l2 < L Vr.}
and B(e) {p(z))p(z) E C², sup, |p(z)] < e). Then we have the follow theorem:
Theorem 1. There exists 5 >> 0, such that vD(x) E Do.s, we have D(x) + ((pp(x) po S2
Proof. By denoting (' = D(x) + 6(p((x) - pa(z)), We have:
d(D(x) + ((pD(x) pa()))) dD(x) + +5( PD(()) pa(x))
dr
dr
dx
dr
Therefore, have
d(D(x) + 8(pp(x) - pc(x))) dr.
dD(x) dx
PD(
Pa (z)
5(
)
(22)
dx dr
< 0.5 + 8(M + M).
(23)
By letting 5
TM7 we have
d(LD') di 12 S 0.75. Therefore we have D'(x) E Ω.
The above theorem indicates that when (xx) sufficient close the equilibrium, then the dynamics of D still follows Eqn. (10).
E CONNECTION TO REGULARIZATION ON JACOBIAN MATRIX
In this paper, mainly analyze our proposed method in the functional space, including stability analysis and controller designing. Instead, our proposed method can also interpreted as certain regularization terms on the Jacobian matrix of the training dynamics. Below we provide a formal demonstration.
First, we denote the equilibrium of G and D in the functional space (0", g²), where Pa (ar; 0*) (z(x) and D(x; o²) = 0 for all z. Therefore, we have that o* is also a global minimum point the regularization term L(D) D2 dd.. Then have 8²L(D) x 0.
17

---

Under review as a conference paper at ICLR 2020
We denote GDD as the objective function of the minimax optimization problem in WGAN without NF regularization. Then the Jacobian matrix of the training dynamic can be denoted as:
(U(DD.)) 88
('U(D.G)
J
B03e B.DD).))
(24)
(UD,)))
8006
002
Because of the linearity of the derivation, the training dynamics of the WGAN with NF regularization denoted as:
L(D) 0
0
J' J JL =
(25)
0
where we abuse the to denote the zero matrix with certain size to match the size of J. Since L(L
862 = 0, we have Jz 3 Therefore, the NF regularization introduces a negative semi-definite
matrix to the original Jacobian matrix, which helpful to stabilize the training dynamics of GANs.
18


Reviews:
[{"review_id": "rJxoXLbCFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors use control theory to analyze and stabilize GAN's training. Their method, effectively, adds an L2 regularization to the output of the discriminator.\n\nI have some concerns regarding the novelty, analysis and also the experiments.\n\n- The analysis has focused on a very simple case of having a linear discriminator which for example in WGAN, forces the first moments to match. How does the analysis extend to more realistic cases?\n\n- In eq 9 in the dynamics of WGAN section, the discriminator should be restricted to Lip functions. This has not been considered in the analysis.\n\n- There are a few work in the literature that analyze local stability of GANs (e.g. https://arxiv.org/abs/1706.04156) as well as using some control theory for analyzing global stability of GANs (e.g. https://arxiv.org/abs/1710.10793). The connections of the proposed approach with existing literature should be better explained.\n\n- In the empirical results, the performance of the proposed method and Reg-GAN (from the numerics of GAN paper) are quite similar. Are there instances that the proposed approach significantly improves the stability of practical GAN architectures?\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJx4v04YKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel view for stabilising GANs from the perspective of control theory. This view provides new insights into GAN training and may inspire future research along this direction. This paper is overall well written, with a smooth introduction of background material that might be less familiar for machine learning researchers. There are places that need further clarification, but I think the proposed direction is promising.\n\nQuestions about the method:\n\n- Since the proposed method starts from Laplace transform, it would be helpful to further discuss the connection between other methods that regularises the eigenvalues of the Jacobian (such as spectral-normalisation), which work in the frequency domain from a different perspective. For example, could the proposed regulariser be interpreted as imposing certain constraint on the spectrum of Jacobian?\n\n- Does section 2.2 depend on the assumption of linear dynamics?\n\n- Does the E in eq.7 come from eq. 4?\n\n- Could you give some intuition for the paragraph above section 3.4, about the different form of inputs when treating D and G as dynamics? For consistency, it is perhaps better to keep the dependency of p_D and p_G on x explicit (same for eq. 10), unless this is intended?\n\n- My main concern about the analysis is that it shows why several methods (e.g., momentum, multiple update steps) are *not* helpful for stabilising GANs, but does not tell why training with these methods, as well as others such as gradient penalty, *do converge* in practice with properly chosen hyper-parameters?\n\nExperiments:\n\n- About setup: the paper reports using ResNets for natural images as in Mescheder et al. (2018). However, Mescheder et al. (2018) uses DCGAN for CIFAR10, which raises further questions about the scores on this dataset:\n\n- The baseline scores of Reg-SGAN and Reg-WGAN seem to be worse than those reported in Mescheder et al. (2018), which have inception scores above 6 according to Figure 6 of their paper. In Figure 5 of this paper, they are clearly below 6. What’s the reason for this discrepancy?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJel4kxAKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper gives a perspective of GANs from control theory, where it is well established in which cases a dynamical system (which can be described analytically as a function of time) is stable or not (by looking at the roots of the denominator of the so-called transfer function in control theory). It is interesting that the analysis using this framework on simple examples is in line with known results in the GAN literature (Dirac GAN). \n\nAlthough I personally enjoyed reading the results that from control theory perspective are inline with GAN literature, the paper does not provide novel surprising results. For e.g. the results on the oscillating behavior of Dirac-GAN are described in related works (e.g. Mescheder et al. 2018), and in practice, WGAN with no regularization is not used (as well as GAN with momentum, as normally beta1=0 in practice). In my understanding, the authors present these results to justify the validity of the approach. However, this limits the novelty of the results relative to existing literature. The authors do not focus (in the main paper) on GAN variants used currently, and it is not clear if the proposed approach provides improvement relative to the current state of the art implementations (see next paragraph). Moreover, if I understand correctly the WGAN analysis does not take into account that G and D are non-linear, and it is unclear if these can be done.\n\nI am also wondering if the comparison with the baselines is fair. In other words, although in the present results, the proposed NF-SGAN/WGAN outperforms the baseline, the reported performance of the baselines is worse than in related works on CIFAR10. In particular, FID of ~30 on CIFAR10 for the baselines is notably higher then current reported results on this dataset (e.g. Miyato et al. 2018; Chavdarova et al. 2019). In my opinion, the authors could start from the existing state of the art implementations on this dataset, and report if negative feedback (NF) improves upon.\n\nAs the approach uses NF is derived specifically for unstable dynamics, it is not clear to me how adding it would affect the training if the dynamics *is stable*. In this context, I consider that for example, the work by Balduzzi et al. 2018 may be relevant as it describes that the dynamics of games (the Jacobian) has two components, one of which describes the oscillating behavior (Hamiltonian game); whereas most games are a mix of oscillating and non-oscillating dynamics. It is not clear to me if NF would improve stability/performances in general games. As the authors’ main claim is improved stability I am curious to see more detailed analysis on real-world datasets (e.g. multiple seed runs, 2nd-moment estimates over iterations as in Chavdarova et al. 2019).\n\nIn summary, although the proposed perspective seems promising given the presented results and it is interesting, in my opinion, it does not provide novel insights nor obtains current state-of-the-art results on CIFAR10, or guarantee that if pursuing it would allow for solving current issues of GAN training. \n\n--- Minor ---\n- Abstract: ‘converge better’ it is not clear to me in what sense (faster/better final performances)\n- Page 3, Eq. 5: as D and G are functions of time here, eq 5 should maybe be written in more detail to include t\n-  Page 3, Sec. 3: I think citing works that also focus on Dirac-GAN would motivate better why you focus on Dirac-GAN in this paper (e.g. writing `as in Mescheder et al. 2018`)\n- Page 4: infinity - infinite\n- Page 5: can also.. explains ->  explain\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper suggests stabilizing the training of GANs using ideas from control theory. The reviewers all noted that the approach was well-motivated and seemed convinced that that the problem was a worthwhile one. However, there were universal concerns about the comparisons with baselines and performance over previous works on Stabilizing GAN training and the authors were not able to properly address them.
Decision: Reject