Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a higher order Brownian Motion Controller (BMC) for BrGANs to stabilize GANs' training process. Starting with the prototypical case of Dirac-GANs, the authors design a BMC and propose Dirac-BrGANs that retrieve exactly the same but reachable optimal equilibrium regardless of GANs' framework. The authors also provide corresponding proof and experiments, although the experimental results show poor performance.

strength_and_weaknesses: Strengths:
- The idea of taking GAN training as a dynamic system and applying Brownian Motion Controller on it sounds interesting.

Weaknesses:
- My biggest concern is about the experimental results. Only CIFAR-10 (32x32 resolution) and CelebA (64x64 resolution) are evaluated and the FID score is not state-of-the-art (actually there is a very big gap between the provided results and SOTA, eg. IS 5.42 on CIFAR-10 v.s. 9.18 in [1]). I truly understand the architecture may be different, but it would be better to see whether the proposed method can continually improve the performance on top of the best architecture. Given there is plenty of theory works studying GAN training stability issues, it is hard to trust this work can indeed help improve the GAN training process since this one considering its poor performance.
- Meanwhile, only two datasets are evaluated, and both of them are low-resolution. It would be better to show more results on LSUN, FFHQ, with their high-resolution version.
- The claim of "the training process of Dirac-BrGANs achieves exponential stability almost surely" sounds like an overclaim to me. To verify its correctness, I think some additional experiments are needed: e.g., using various hyperparameters (training G after training D every $N$ times), optimizers, batch size, and demonstrating that the proposed methods can help GAN training robust to different settings.
References:
[1] Differentiable Augmentation for Data-Efficient GAN Training
clarity,_quality,_novelty_and_reproducibility: The experimental results are less satisfying and the analysis is not enough.
summary_of_the_review: This work propose a new way to stabilize the training process of GAN, using Brownian Motion Controller. Although the motivation is interesting, the results are less satisfactory and the quality of the work is lower than the ICLR's bar.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors leverage ideas from control theory to propose a regularizer, called the Brownian motion controller (BMC), which aims to stabilize GAN training. In practice, this amounts to an additive regularization term which is applied to the discriminator loss.
strength_and_weaknesses: Strengths:
- Interesting perspective on GAN training.

Weaknesses:
- Theory is limited. Convergence results of the proposed method are restricted to the DiracGAN setting in a two-dimensional parameter space.
- Experiments could be more compelling. The authors claim that their method is an improvement over a previous work in a very similar direction that also applies control theory to GAN training -- WGAN-CLC [1]. However, the results reported in Table 3 deviate significantly from the results in [1]. Namely, [1] report much better FID scores than reported here. Can the authors comment on this discrepancy?
- Introduces three new hyperparameters $\rho_1$, $\rho_2$, and $\beta$. It is unclear how sensitive they are to perturbations, when applied to models trained on more realistic data (e.g. CIFAR-10, CelebA).

[1] Xu, K., Li, C., Zhu, J. and Zhang, B., 2020, November. Understanding and stabilizing GANs’ training dynamics using control theory. In International Conference on Machine Learning (pp. 10566-10575). PMLR.
clarity,_quality,_novelty_and_reproducibility: Writing could be improved, as space is not used wisely. For example, authors provide limited motivation and exposition on DiracGANs (Section 2.1), but spend space defining FID and Inception Score (Section 4.1), which are well known quantities. As a result, Section 2 required some outside reading to understand --- I had to refer to [2] for more information on DiracGAN. Even so, it is not clear to me why improving stability on the DiracGANs will improve general GAN stability.

Questions:

Tables 3 and 4 and Figures 4 and 5 — Why is WGAN and WGAN-BR performance so poor? The model does not appear to be converged to me.

A key contribution of this work is its connection to control theory. However, there is limited discussion on the relationship of this work to that of [1], and the fairness of the empirical comparisons is unclear. Therefore it is somewhat difficult to gauge novelty and quality. Can the authors elaborate on this?

Why is BrGAN only compared to CLC-GAN? Surely there are other appropriate regularizers / training schemes that aim to stabilize GAN training? If the authors choose to focus on only the control theory perspective, then the suitability of applying control theory to GAN training should be further justified.

[2] Mescheder, L., Geiger, A. and Nowozin, S., 2018, July. Which training methods for GANs do actually converge?. In International conference on machine learning (pp. 3481-3490). PMLR.
summary_of_the_review: Overall, I found the direction of the work promising, but its execution wanting. It was unclear why the authors chose DiracGANs as a motivating example, and why the control theory perspective was the most compelling direction to alleviate the training instabilities. Moreover, experimental comparisons were weak, and limited to one competing regularizer (WGAN-CLC), applied to three GAN losses (WGAN, WGAN-GP, WGAN-CP).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a stochastic controller to stabilize the training process of GANs. It shows in a simple Dirac-GANs case, the proposed method converges theoretically to a unique optimal equilibrium. For general GANs, a modification of the controller is proposed and numerical results are given to show the improved stability of the training process in terms of common benchmarks for image generations. 
strength_and_weaknesses: The strength of the paper is that the proposed methodology is very interesting and it provides a new way to deal with the circling issue in bilinear games. However, I am not fully convinced of the stability result, as well the convergence of the modified version of the controller for general GANs. Here are a few questions:
- What is the behavior of the modified version (in eq. 17) on Dirac-GANs? Is there still any convergence as in Theorem 2?
- When one talks about the stability of the training process, would it make more sense to check whether the losses of the discriminator and generator converge, compared to the use of FID or inception score? Somehow it is not clear why the training can be stable in terms of these scores. 
- The discussion at the beginning of Section 3.2 regarding the optimal solution is too ideal. In practice, a normal GAN only has access to finite samples of p(x), so it is not clear why one can have p_G(x) = p(x), and D(x)=0 as an equilibrium. More seriously, the numerical results are based on the use of WGAN-GP, where the GP penalty does not make D(x)=0 (as it encourages that the gradient of D has a unit norm). Thus the idea of BRGAN seems not compatible with the GP, nevertheless wgan_gp_br seems to work best in Fig 2 and 3. I think more discussions are needed regarding these aspects to explain how wgan_gp_br works, and why it works well. 
clarity,_quality,_novelty_and_reproducibility: I think more intuition could be given to eq (6). Also what does it mean dot B_1, is it dB_1? What is p(g) in eq (13), does that include the GP penalty of WGAN-GP? In section 4.3, which basic algorithm is the proposed method compared to, e.g. gradient-descent-ascent or its alternative version, or Adam, etc? What is the discrete version of the proposed method?

summary_of_the_review: I tend to stay with my current score, as I find that the current version could still be improved for a future submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper extends the CLC-GAN framework of Xu et al 2019 https://proceedings.mlr.press/v119/xu20d.html with a gaussian noise regularisation derived from Brownian Model Control theory, showing that this additionally improves over the CLC controller in terms of FID on CIAR10 and celeba
strength_and_weaknesses: Strengths

If the main weaknesses below are fixed and the papers empirical improvements persists, the paper becomes an incremental improvement over CLC gan which will spread awareness of BMC theory in the community, which I think is absolutely enough. I'd also be interested in possible links to diffusion processes. 

Weaknesses

1. the FID of WGAN-CLC-GP is much higher than in the Xu et al paper, but the IS is much closer (still higher). To me this indicates FID was calculated with a different batch size than in the Xu et al paper *but* also that the improvement might be due to architecture/training differences. An ablation study might alleviate this...
2. ....As would reporting multiple seeds, reporting mean and std and performing a statistical significance test
3. the novelty relies on adding gaussian noise to the control process + an analysis, which I think is just barely enough to warrant a paper, but is sabotaged by limited evaluation quality. If the evaluation is fixed and benefits persist, this weakness becomes irrelevant.

clarity,_quality,_novelty_and_reproducibility: Everything except evaluation quality is fine,  see weaknesses above
summary_of_the_review: As noted, if the empirical gains survive an improved evaluation, I think this paper is a fine incremental improvement over previous work, using not-so widely known BMC theory
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to use a Brownian Motion Controller (BMC) to stabilize the GANs' training process. The stability of GANs is an important problem in the field so this is a relevant problem. On the plus side, the reviewers also find the idea to be interesting so I would encourage the authors to further pursue it and improve the execution of the paper. For now, the paper is still not at the level of an ICLR publication and I'm not able to recommend acceptance.

The most important comments from the reviewers seem to point to the following avenues for improvement:
- possibly strengthen the theory of the paper
- or strengthen the empirical evaluation
- better justify the choice of hyper-parameters, run experiments with multiple seeds, better describe the experimental setting in the text


justification_for_why_not_higher_score: Reviews are clearly negative overall.
Decision: Reject