Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Authors use control theory to analyze and stabilize GAN's training. Their method, effectively, adds an L2 regularization to the output of the discriminator.

I have some concerns regarding the novelty, analysis and also the experiments.

- The analysis has focused on a very simple case of having a linear discriminator which for example in WGAN, forces the first moments to match. How does the analysis extend to more realistic cases?

- In eq 9 in the dynamics of WGAN section, the discriminator should be restricted to Lip functions. This has not been considered in the analysis.

- There are a few work in the literature that analyze local stability of GANs (e.g. https://arxiv.org/abs/1706.04156) as well as using some control theory for analyzing global stability of GANs (e.g. https://arxiv.org/abs/1710.10793). The connections of the proposed approach with existing literature should be better explained.

- In the empirical results, the performance of the proposed method and Reg-GAN (from the numerics of GAN paper) are quite similar. Are there instances that the proposed approach significantly improves the stability of practical GAN architectures?


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper gives a perspective of GANs from control theory, where it is well established in which cases a dynamical system (which can be described analytically as a function of time) is stable or not (by looking at the roots of the denominator of the so-called transfer function in control theory). It is interesting that the analysis using this framework on simple examples is in line with known results in the GAN literature (Dirac GAN). 

Although I personally enjoyed reading the results that from control theory perspective are inline with GAN literature, the paper does not provide novel surprising results. For e.g. the results on the oscillating behavior of Dirac-GAN are described in related works (e.g. Mescheder et al. 2018), and in practice, WGAN with no regularization is not used (as well as GAN with momentum, as normally beta1=0 in practice). In my understanding, the authors present these results to justify the validity of the approach. However, this limits the novelty of the results relative to existing literature. The authors do not focus (in the main paper) on GAN variants used currently, and it is not clear if the proposed approach provides improvement relative to the current state of the art implementations (see next paragraph). Moreover, if I understand correctly the WGAN analysis does not take into account that G and D are non-linear, and it is unclear if these can be done.

I am also wondering if the comparison with the baselines is fair. In other words, although in the present results, the proposed NF-SGAN/WGAN outperforms the baseline, the reported performance of the baselines is worse than in related works on CIFAR10. In particular, FID of ~30 on CIFAR10 for the baselines is notably higher then current reported results on this dataset (e.g. Miyato et al. 2018; Chavdarova et al. 2019). In my opinion, the authors could start from the existing state of the art implementations on this dataset, and report if negative feedback (NF) improves upon.

As the approach uses NF is derived specifically for unstable dynamics, it is not clear to me how adding it would affect the training if the dynamics *is stable*. In this context, I consider that for example, the work by Balduzzi et al. 2018 may be relevant as it describes that the dynamics of games (the Jacobian) has two components, one of which describes the oscillating behavior (Hamiltonian game); whereas most games are a mix of oscillating and non-oscillating dynamics. It is not clear to me if NF would improve stability/performances in general games. As the authors’ main claim is improved stability I am curious to see more detailed analysis on real-world datasets (e.g. multiple seed runs, 2nd-moment estimates over iterations as in Chavdarova et al. 2019).

In summary, although the proposed perspective seems promising given the presented results and it is interesting, in my opinion, it does not provide novel insights nor obtains current state-of-the-art results on CIFAR10, or guarantee that if pursuing it would allow for solving current issues of GAN training. 

--- Minor ---
- Abstract: ‘converge better’ it is not clear to me in what sense (faster/better final performances)
- Page 3, Eq. 5: as D and G are functions of time here, eq 5 should maybe be written in more detail to include t
-  Page 3, Sec. 3: I think citing works that also focus on Dirac-GAN would motivate better why you focus on Dirac-GAN in this paper (e.g. writing `as in Mescheder et al. 2018`)
- Page 4: infinity - infinite
- Page 5: can also.. explains ->  explain

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel view for stabilising GANs from the perspective of control theory. This view provides new insights into GAN training and may inspire future research along this direction. This paper is overall well written, with a smooth introduction of background material that might be less familiar for machine learning researchers. There are places that need further clarification, but I think the proposed direction is promising.

Questions about the method:

- Since the proposed method starts from Laplace transform, it would be helpful to further discuss the connection between other methods that regularises the eigenvalues of the Jacobian (such as spectral-normalisation), which work in the frequency domain from a different perspective. For example, could the proposed regulariser be interpreted as imposing certain constraint on the spectrum of Jacobian?

- Does section 2.2 depend on the assumption of linear dynamics?

- Does the E in eq.7 come from eq. 4?

- Could you give some intuition for the paragraph above section 3.4, about the different form of inputs when treating D and G as dynamics? For consistency, it is perhaps better to keep the dependency of p_D and p_G on x explicit (same for eq. 10), unless this is intended?

- My main concern about the analysis is that it shows why several methods (e.g., momentum, multiple update steps) are *not* helpful for stabilising GANs, but does not tell why training with these methods, as well as others such as gradient penalty, *do converge* in practice with properly chosen hyper-parameters?

Experiments:

- About setup: the paper reports using ResNets for natural images as in Mescheder et al. (2018). However, Mescheder et al. (2018) uses DCGAN for CIFAR10, which raises further questions about the scores on this dataset:

- The baseline scores of Reg-SGAN and Reg-WGAN seem to be worse than those reported in Mescheder et al. (2018), which have inception scores above 6 according to Figure 6 of their paper. In Figure 5 of this paper, they are clearly below 6. What’s the reason for this discrepancy?

review_title: Official Blind Review #2

Meta-review: This paper suggests stabilizing the training of GANs using ideas from control theory. The reviewers all noted that the approach was well-motivated and seemed convinced that that the problem was a worthwhile one. However, there were universal concerns about the comparisons with baselines and performance over previous works on Stabilizing GAN training and the authors were not able to properly address them.
Decision: Reject