Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the gradient descent-ascent optimization problem of GANs in the following setting:
 1. without generator, the GAN manipulating a finite set of elements;
 2. under an MMD GAN model with a fixed kernel, the kernel discriminator optimizing the corresponding IPM;
 3. in a novel framework called the Isolated Points Model where the kernel value between true data points approaches zero.

The latter assumptions enable the derivation of new convergence results. The paper particularly characterizes the existence, position and nature of equilibrium points in the gradient descent-ascent optimization around the true samples tanks to the introduction of a probability mass difference criterion. This notably allows the authors to rule out the possibility of strict mode collapse on the one hand and to identify the existence of bad local minima leading to an approximate mode collapse phenomenon. The theoretical results are then assessed experimentally on a toy dataset.
strengths_and_weaknesses: ### Strengths

The study of the optimization procedure of GANs is a highly challenging task, as understanding the outcome of its non-convex non-concave behavior remains an open question in the literature. In this regard, **the tackled problem is relevant** to the NeurIPS community.

This **well-written paper** presents a clear proposition to advance our understanding of this difficult problem. It properly defines its analysis framework composed of assumptions (1, 2, 3) as described in the above summary. While **conditions (1, 2) are reasonable, well-motivated choices** to make the optimization problem amenable to theoretical analysis, the Isolated Point Model (3) necessarily limits the applicability of the presented results. Nonetheless, such a strong assumption is understandable given the challenge of the tackled issues and **pushes back the limits of previous convergence analyses** whose assumptions were even more restrictive.

To my knowledge, the obtained theoretical results are **clearly presented**, **novel** and **may contribute to a better understanding of GAN optimization**. The identification of precise criteria for the existence of stable and unstable equilibrium points as well as the discovery of an approximate mode collapse is very welcome and participate in deepening our knowledge of local convergence effects in GANs.

### Weaknesses

My main interrogation on the paper deals with **the significance of the presented results**. Given the restrictions imposed by the Isolated Point Model, their impact on the domain remains unclear and difficult to gauge. Indeed, the proposed Isolated Point Model is **unrealistic** in the context of GANs. While kernel discriminators may be considered via the theory of NTKs [8], these kernels are non-standard and typically do not resemble the RBF-like kernels considered in this paper. Even in the case of the RBF, **a small kernel width is not a typical choice**, as explained by the empirical results of Section 7. This lack of grounding in GAN practice is highlighted by the divergence result of Section 5, which to my knowledge do not correspond to a standard GAN behavior. I am also unable to judge whether the introduced framework might also serve as a basis for future, less limited analyses, as this is a difficult task.

### Overall Sentiment

Given the difficulty of the tackled problem, the state of the literature on this matter, and the interesting phenomena unveiled by the authors, I think that this paper is worth publishing at NeurIPS.

Even though the chosen setting might be seen as unrealistic, I believe that the presented advances have the potential to be useful to the community, hence my positive recommendation. I would be glad to discuss this matter with the authors and the other reviewers during the discussion phase.

### Typos

 - Figure 1: I think the description of $V_4$ in the caption actually corresponds to $V_2$ in the figure.
 - Line 172: "excactly" should be "exactly".
 - Line 268: "singl" should be "single".
 - Lines 189, 214, 265 and 295: please fix the quote typesetting.
questions:  - Could the authors further evaluate and discuss the realism of the introduced Isolated Point Model? In particular, how might it relate to the theory of NTKs as mentioned in the conclusion?
 - Would it be possible to highlight how the obtained results and/or the used proof techniques might serve as a basis for future work?
 - Do the authors see any relaxation of the Isolated Point Model and Assumption 1 beyond considering exponentially vanishing kernels like the RBF? For example, by constraining only the gradients of the kernel instead of its values, since the movement of the generated points only depend on its gradients?
limitations: The authors adequately present the assumptions and limitations of their study. While many of them are standard and reasonable, further discussion of the realism of the main and novel assumptions would be beneficial for the paper as it could help the reader gauge the impact of the obtained results (cf. the above sections).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper analyzes the dynamics of training GANs under the case that the true and generated samples are discrete, finite sets, and the discriminator is kernel-based. Specifically, the paper assumes that the true samples are at least one kernel width away from each other, so called Isolated Points Model. This paper analyzes the necessary and sufficient conditions for local stability and instability, points out the existence of approximate mode collapse and divergence, and provides insights into the role of the kernel width of the kernel discriminator. Numerical experiments are also conducted to show the effect of kernel width on GAN training.
strengths_and_weaknesses: Strengths
- This paper is well organized and well written.
- This paper is theoretically grounded. It provides a theoretical understanding of the dynamics of GAN training.


Weaknesses
- Assumptions are strong. For example, in line 110, supposing that N_i^k is constant over time is unreasonable. It is not clear to me at this time how this assumption affects the theoretical results. I would like to see the authors analyze the results when this assumption does not hold.
- Experiments are weak. Only numerical experiments with simple experimental settings are provided in this work. 
questions: - What new principles does this study bring to GAN design? In other words, how do we avoid instability and mode collapse in training GANs?

typos:
- In line 104, "generated samples" -> "true samples"
limitations: The authors should discuss the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This article propose a simple theoretical model, called the Isolated Points Model, which is analytically tractable and rigorously study the stability and convergence properties of training a GAN. The article analyze the GAN's training problems from local stability and instability, approximate mode collapse and divergence three perspectives.  Finally, this article argues a kernel based discriminator to explain the above three problems.
strengths_and_weaknesses: Strengths:

1. This article proposed originality insight for the training of GAN using Isolated Points Model. 

Weaknesses:
1. This article proposed originality insight for the training of GAN, but without a solution for this problems.
2. There are some problems with the misuse of symbols in the derivation section
questions: 1. Experiments are too simple to convince the theoritic prove.

2. This article does not supply a useful solution to solve the problems mentioned in the article.

3. Role of the Kernel Width in the section 6 might be a very important factor for the article, but the analysis for this factor are too simple.

4.  In the caption of Figure 1, article states that "In V4, the excess point mass the $\triangle_4$ < 0". But in the V4 of figure 1, the $\triangle_4=0.25$. Both seetings are contradictory.

5. If the RBF is set as discriminator kernel, how the RBF kernel satisfies the assumption in the equation 10. 

6. Substituting equation 6 into equation 5 does not yield equation 4. Because in equation 4 is a(x_i), but the kernel is (a^T)a(x_i).

7. The article migth explain the meaning of $\delta$ in equation 1.

limitations: yes.
review_title: null

Meta-review: metareview: All reviewers agree this paper presents interesting analysis and results on GAN training dynamics. However they also note several limitations of the proposed setting, namely the assumptions of kernel width and the isolated points model, restrict directly applying results to practical settings. Authors in the response have done a good job of explaining how one can use the observations from the analysis to improve stability of real world training of GANs. Overall I think this work has some promising directions towards improving our understanding of training GANs, hence I suggest acceptance.
confidence: Less certain
Decision: Accept