Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In a standard GAN, a fixed-capacity discriminator is used during training. In contrast, this paper proposes DynamicD, which changes the network capacity of the discriminator dynamically during training by varying the network width. In particular, the authors introduce two strategies. The first strategy is to increase the capacity progressively. This strategy is proper when given sufficient data. The second strategy is to decrease the capacity gradually. This strategy is useful when given limited data. The effectiveness of the proposed method was demonstrated in various settings, including 2D and 3D-aware image synthesis. Furthermore, the compatibility with other techniques, such as data augmentation, regularizers, and pre-training, was shown.
strengths_and_weaknesses: **Strengths**
- The proposed ideas (i.e., increasing the capacity progressively and decreasing the capacity gradually) are simple. It will be easy to implement them.
- The effectiveness of the proposed method was demonstrated in various settings, including 2D and 3D-aware image synthesis. In the experiments, the proposed model was compared with multiple state-of-the-art baselines.
- The compatibility with other techniques, such as data augmentation, regularizers, and pre-training, was shown. The proposed method has high versatility.

**Weaknesses**
- The effectiveness of the proposed method was mainly demonstrated through an empirical analysis, and theoretical analysis was not sufficiently conducted. This paper would become more appealing if a theoretical analysis could be performed.
- The explanation about the implementation of a decreasing strategy is a bit rough. It is unclear what is the difference between the proposed method and the dropout where the dropout rate increases gradually. In Table 2, the results using a variant of dropout (adaptive dropout [30]) was presented. However, in the method, multiplicative Gaussian dropout was used. It would be more important to compare the proposed model with the model that uses a standard dropout where zero is inserted and the dropout rate increases gradually.
- The primary focus of this paper is to improve the training dynamics. Therefore, it would be better to analyze the time-varying performance in more detail. However, such an analysis is only conducted for the baselines in Figure 1. It would be better to perform the same analysis for the proposed method.
- There are many blanks in Table 2. This makes it difficult to judge whether the proposed model constantly outperforms or underperforms the baselines.
- Although the authors have already admitted, the model capacity must be adjusted manually. This is not an appealing property.
questions: - What is the theoretical background of the proposed method?
- Does the proposed model outperform the model that uses a standard dropout where zero is inserted and the dropout rate increases gradually?
- Could you provide the time-varying performance analysis? I would like to know whether the proposed method outperforms the baselines constantly through training.
- Is it possible to provide the results for the blanks in Table 2?
limitations: In the paper, the exploration of the influence of different factors, such as network depth, applications to other architectures, such as a transformer-based discriminator, and development of the method for controlling the network capacity automatically, are described as future work.

In addition to the above, broader impacts (e.g., the risk of making it easy to create fake content, such as deepfake) are addressed in the supplementary material.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a dynamic discriminator scheme for training generative adversarial networks. This mitigates the overfitting issues in limited data scenarios and improve GAN training in large data settings by ensuring a balance b/w the generator and discriminator. The dynamic discriminator is achieved by increasing/decreasing the number of channels during training. The experiments conducted show the benefit of the dynamic discriminator across different settings. 
strengths_and_weaknesses: Strengths:
1. The proposed method of dynamic discriminator to improve GAN training is nice, intuitive and the results show its benefit in different settings. 
2. The paper is well written and is easy to understand. 

Weakness:
1. Comparison of the method “full -> half” with random half channel selection at each training step from a full channel configuration is not shown. It’s not clear if the benefit from the method is due to random selection of channels which will also help in preventing overfitting issues in limited data scenario. 
2. Comparison of the method with StyleGAN2+ADA (or StyleGAN2 + DiffAugment) in Table 2 and Table 3 instead of only having it as an ablation experiment. As mentioned in the supplementary, the code is built upon StyleGAN2-ADA, thus I am not sure why this was not chosen as the baseline. Its one of the leading state-of-the-art methods and if the benefit of combining it with the proposed method is significant in all settings it will strengthen the paper further. 
3. For the extremely limited setting for e.g., 0.1K FFHQ experiments, nearest neighbor test as done in [60] and latent interpolation visualization results should be shown as FID is not reliable in those settings. There might be overfitting and memorization of real images which is not evident from only quantitative evaluation. 
questions: 1. In line 294 the paper mentions improved training efficiency. It would be great if quantitative numbers can be reported in terms of training time and convergence curves. 
2. Table 5 (b) MetFaces KID w/ FreezeD setting is reported as 4.33. But [30] (StyleGAN2-ADA https://arxiv.org/pdf/2006.06676.pdf) Fig 11 (a) reports similar setting KID as 2.05. Any comments on the difference? As I understood both settings are training w/ FreezeD on MetFaces from a pretrained model on FFHQ. 
3. The schedule of increasing alpha or decreasing beta is not mentioned in the paper. Is it linear and does having different schedules affects training? Also, decreasing channels schedule is only applied to higher level layers. What are the specific layer numbers and how does this choice affect the results?
4. In line 262 it is mentioned that “we extract the numbers of adaptive dropout from ADA”. Not sure what is meant by this. StyleGAN2-ADA doesn’t involve any dropout in discriminator layers.  
5. Minor comment: line 28 “line fundions”

limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In the training stage of GANs, the synthesized data distribution varies with the evolving generator. Thus the discriminator has a distribution shift issue. To tackle this issue, the paper proposes adjusting the discriminator's capacity on the fly (named DynamicD). The paper presents two capacity adjusting schemes and confirms that different training data regimes have different favored schemes. The paper gradually increases the network width by including newly initialized filters to increase discriminator capacity, and progressively decreases
the network width by randomly dropping a subset of filters to decrease capacity. Extensive experiments substantiate the generalizability of the DynamicD. 
strengths_and_weaknesses: Strengths:

1. The paper is well-written, and the experiments are comprehensive.

2. The problem studied in the paper is important and would be a good contribution to the community.

3. The method is simple and easy to follow.

4. The proposed method shows a large improvement in the image generation task, especially when the training dataset is limited


Weaknesses:

1. In Table 4, "we find that, for 3D-aware image generation, even the full set of FFHQ and Carla is insufficient ... all experiments adopt the decreasing capacity scheme." It seems a little counter-intuitive. FFHQ has as many as 70,000 face images, which is enough for the face manifold. Why use a decreasing capacity scheme instead of an increasing capacity scheme? If possible, the authors can show the FID curves under both schemes. 

2. The increasing/decreasing coefficient is an important hyperparameter for the dynamic discriminator. The author should do ablation experiments on this coefficient in detail. 

3. For generative domain adaptation experiments (L316), it is a commonly used and effective method for training GANs with limited data. In order to clearly compare the different methods, I suggest the author provide FID curves for different settings such as w/ DynamicD, w/o Dynamic, w/ DiffAug. In addition, if possible, the logits of the discriminator (i.e., D(x), D(G(z))) also help us understand the training dynamics, thus which also need to be provided. 

4. L273, "DynamicD slightly outperforms DiffAugment [60] which uses a fixed data augmentation and performs worse than the adaptive ones (i.e., ADA [30] and APA [25])." In fact, ADA has the risk of leaking rotating augmentation onto the generated images under the limited data setting. Therefore, a lower FID does not mean that ADA is better than DiffAug. If possible, the author should compare more uncurated generated images of different methods to get a visual impression of the quality of the methods. 

questions: Please see the weaknesses. 
limitations: Yes, they did.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a simple method for avoiding overfitting and underfitting in the discriminator network of a GAN. This work proposes linearly adjusting the capacity of the discriminator network during training in order to deal with these problems. To deal with underfitting, the authors propose gradually increasing the number of feature maps of each layer. In cases where data is insufficient and the discriminator is prone to overfitting the authors propose gradually subsampling feature maps from each layer, effectively reducing its capacity. Results are presented on image synthesis and 3D-aware image synthesis and compared to StyleGAN2 and StyleNerf baselines. Furthermore, the proposed approach is compared and also used in combination with other methods that avoid discriminator overfitting.
strengths_and_weaknesses: ## Strengths:
- The paper is well motivated. The problem of overfitting discriminators is known to affect the ability of GANs to produce realistic samples with limited data.
- The proposed method is simple and can be applied to most network architectures.

## Weaknesses:

- A citation to DropoutGAN[1*] should be added to the text since it is one of the first methods to propose using dynamically changing discriminators during training.  
- The novelty of the method is limited. The problem of overfitting discriminators is well known and several approaches have already suggested adaptively limiting the expressive power of the discriminator to solve it [30, 1*].
- The proposed method interpolates between 2 network capacities but does not explain how these are selected and how important their choice is. In the paper, the authors start with a "standard" full discriminator capacity but for new tasks, this is not known in advance. Furthermore, for a novel task, it is not immediately clear when to select a decreasing or increasing strategy because in practice it is hard to know in advance what constitutes "sufficient" training data. Choosing the correct strategy and capacity seem to be important hyperparameters that require additional experiments to determine. 
- Compared to existing approaches in Table 2 the proposed method is only better than other approaches when dealing with tiny datasets (100 images). It seems to perform worse than nearly all other approaches when 2K images are used for training and comparably when 140K images are used.
- Line 25 states that improvements of the discriminator are "much less explored" than the generator. This is not accurate. It is well known that the discriminator drives the generator to learn the correct distribution and many different discriminator approaches have been proposed in order to improve results [24, 1*, 2*, 3*]. This statement should be revised.
- Certain details are not mentioned in the paper (see Questions section)

[1*] Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators, Mordido et.al. ACM KDD 2018
[2*] Conditional Image Synthesis with Auxiliary Classifier GANs, Odena et. al. ICML 2017
[3*] cGANs with Projection Discriminator, Miyato and Koyama, ICLR 2018
questions: - Although the starting and ending values α and β are reported in the paper the number of iterations n after which their values are adjusted is not stated.
- The supplementary material mentions that "Notably, the total number of seen images for sufficient data training is 25 million regardless of datasets". How did the authors come up with this number and why is it independent of the dataset?
- I am unsure what to make of the attention maps obtained using Grad-CAM presented in the supplementary material. The authors should provide further insights as to why the discriminator focusing on the entire image is beneficial. Also to some extent, it is expected that the focus of the discriminator will change during training even at a fixed capacity. Hence averaging 10 attention maps throughout training even for the fixed discriminator should also result in a larger attention map.
- The paper mentions that the proposed approach is more efficient. It is clear that the adaptive capacity reduces the memory required at certain stages of training (at the start or end depending on strategy). However, in practical terms, what difference does this make during training and how do the authors take advantage of this? Does training take less time than other approaches?
limitations: The authors have sufficiently discussed the limitations of the work and possible future improvements. I do not believe that there are significant ethical concerns with regard to this work (beyond those associated with generative modelling in general) hence I do not believe the paper requires an additional discussion about societal impact.
review_title: null

Meta-review: metareview: This method uses a dynamic-capacity discriminator to improve GAN training, improving perforation eg in limited data settings. The method interested the reviewers, though there were common concerns about how the work was presented, both in what it is doing precisely as well as how it presents itself relative to prior works. This concern I share as well reading the reviews and glancing at the paper, yet this seems to have satisfied the reviewers, so I will go with their consensus though lower my confidence:

I therefore recommend (with uncertainty) that this paper is accepted to NeurIPS.

Reviewer  h4W2 participated the most in the discussion, but unfortunately discussion was overall light.
confidence: Less certain
Decision: Accept