Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new concept of pre-conditioning layers. The authors claim that this idea is novel and they introduce this idea in the context of GANs. I cannot verify or falsify if the idea is really novel, but assuming the idea is novel, I definitely believe it is worthy to be explored and eventually published somewhere. The idea is creative and technically interesting.

The authors argue that not only large singular values matter, but also the condition number.
Based on their experiments, the author find that a bad condition number might be a reason for poor results using spectral normalization in a GAN.

The technical realization of the idea seems reasonable.

The evaluation is a bit difficult to follow. The baseline method is a GAN that is no-longer state of the art. It is not even clear to me what this GAN architecture is exactly that is being used. The description is a bit vague. I am not sure if the baseline architecture can be considered reasonable. Certainly, while spectral normalization might cause certain problem with one type of architecture, it is not that clear that this problem will persist in a state-of-the-art GAN and how many state-of-the-art GANs really rely on spectral normalization. For example, the StyleGANv2 paper writes
"It should be noted that spectral normalization [31] of the generator [46] only constrains the largest singular value, posing no constraints on the others and hence not necessarily leading to better conditioning. We find that enabling spectral normalization in addition to our contributions — or instead of them — invariably compromises FID, as detailed in Appendix E"

Further, just relying on FID is risky. If there is no demonstration of visual quality improvement I would be quite skeptical if an improvement really has been made. It is ultimately up to the authors to demonstrate an improvement. With this submission, the authors shift a massive amount of experimental validation of the work to the reader. Who is going to try out your idea on a state-of-the-art GAN and evaluate visual quality?
review_title: New idea, no demonstration of improved visual quality

final_score: rating: 5: Marginally below acceptance threshold
review_content: *Summary:

This paper mainly solves the instability issue on the spectral normalization for generative adversarial networks (SN-GANs) when training with high dimensional data. To address this, the authors present a preconditioning layer (PC-layer) with two different ways (i.e., FPC and APC) to perform a low-degree polynomial preconditioning. Experiments on LSUN 256x256 training data demonstrate that FPC and APC are able to control the strength of preconditioning. My detailed comments are as follows.

*Positive points:

1. The instability of training GANs is an important research problem. This paper relieves this issue by introducing a preconditioning layer. 

2. This paper provides an empirical study on the training instability of GANs. The authors found that the instabilities are accompanied by large condition numbers of the discriminator weight matrices.

*Negative points:

1. Some theoretical evidence should be provided. In Section 1, the authors suspect that an ‘adequate’ weight matrix spectrum is also important for GAN training. Could you please provide sufficient theoretical analysis for this?

2. This paper mainly addresses the instability issue on SN-GANs when training with high dimensional data. However, it is not clear why SN-GANs have poor performance on higher-dimensional data. It would be better to provide theoretical analysis to support this.

3. Some technical details are not clear. The use of notations in this paper is confusing since they are used without clear explanations, which makes the paper hard to follow. For example, is $\| \|$ L2-norm by default? Should $F(\theta; x)$ and $F(\theta; X)$ be the same? Is the division in $W/ \tidle{W}$ an element-wise division? In addition, it would be better to provide intuitive understanding for the proposed theorems and claims.

4. In Section 2, this paper only considers a deep linear neural network. However, it is impractical for some real-world case. Could you please extend the analysis to more general case on deep non-linear neural network? In addition, the optimization problem mainly addresses the weighted least-square problem. Can it be extended to other losses, e.g., cross entropy loss?

5. In the experiment, the authors improve the training instability on LSUN 256x256 data. Could you please conduct more experiments on higher-dimensional dataset, e.g., CelebA 512x512 data.

review_title: Comments

final_score: rating: 4: Ok but not good enough - rejection
review_content: The current work introduces a pre-conditioning (PC) layer using low-degree polynomials as a way to implement spectral normalization (SN) of the weight matrix W in the D-net. The primary application is to improve the training of GANs by SN techniques, which has been previously established in [1] and later works like [2]. 

The empirical results measured by FID are good, and the experiments on higher resolution datasets like LSUN 256x256 is appreciated. However, the reviewer would like to point out some possible issues as below.

- The first issue is the conceptual relation between matrix condition number and the GAN training instability. The condition number is the ratio of the largest and smallest singular value, and thus only depends on the extreme values (the "edge") of the spectrum. Meanwhile, previous works have already shown that controlling the whole spectrum is important and beneficial for GAN training, e.g. Fig 1 in [2]. Thus it is a question whether condition number is the right object to focus on, which motivates the PC approach.
The authors show that SN-GAN can have large condition numbers, and the PC-GAN improves it (Fig. 1), which accompanies improved FID score. However, this does not mean that it is the smallest singular value, not the whole spectrum, that brings the benefit. For example, when all the singular values are 1 except for the smallest one which is 1e-8, then the matrix is almost full rank and well-conditioned except for 1 direction, but the condition number is 1e8. Suppose the spectrum of W is like this, will the training instability remain? 
The theory in Section 2 is about a simplified model (linear network) under another setting, and the reviewer is not convinced that the result there (convergence is upper bounded by a quantity related to the condition number) can resolve the issue and justify the importance of condition number. 

- Another possible issue is computational cost. First, the non-adaptive PC proposed is p(AA^T)A, where p is a polynomial. Even p is low-degree, this involves large matrix multiplication. Note that in the case of ill-conditioning, AA^T squares and worsen the condition number.
Second, the best empirical improvement comes from the adaptive PC, and less evident with the fixed PC. The APC method needs to compute the value of the modified condition number, and that seems to involve eigen decomposition (e.g. SVD or approximate SVD of the matrix A), which is usually expensive. 
So how will the PC approach compare with the SVD like approach such as in [2] which directly controls the spectrum? It is not clear if the extra computational cost, especially eigen computation, can be justified in the practice of GAN training. 

Other comments:

 Inception score in the experimental results should also be reported in the main text.

 Thm 1, Eqn (2), is it \mu/\beta or \beta/\mu? 

[1] Miyato et al. "Spectral normalization for generative adversarial networks." In ICLR, 2018.

[2]  Jiang et al. "On computation and generalization of generative adversarial networks under spectrum control." In ICLR, 2019.
review_title: good empirical result but with possible conceptual and computational issue

final_score: rating: 7: Good paper, accept
review_content: Update:  
Thank the authors for the detailed feedback. I decide to keep the score.
---
---
The paper shows that the failure mode of spectral normalization (SN) is often accompanied by large condition numbers in the discriminator layers. Motivated from this observation, the paper proposes to control the condition numbers of the discriminator layers, by adding preconditioning to the weights. The results show that the proposed approach makes the training more stable and achieves better sample quality on several datasets.

Overall, I enjoy reading this paper. The motivation is clearly explained, and the approach is simple yet effective. I would recommend an accept. I do find that the writing needs to be improved (e.g. there are many typos in both the main paper and the appendix), and more experiments can be explored. However, I think these are relatively minor issues that do not diminish the overall quality of the paper.

The minor issues and suggestions are listed below.

Missing details:
* Section 3.4 mentioned that "the best preconditioner varies for different datasets". It is fine, but you should show the FPC results with all different degrees, so that it is clearer how sensitive FPC is to the degree. I only see the results with degree 3 and 7 in the main text and the appendix.
* APC needs to use the singular values of the weights. How do you compute the singular values, by exact computation or estimation? If you do the exact computation, then I have a question about the scalability of the approach. For convolutional layers (which are the dominant components in the network architectures you experimented on), the computation is cheap because the kernels are small. But it might computationally expensive for fully connected layers.
* How do you compute the actual true spectral norm in Appendix E.1? By exact computation or estimation?

Suggested experiments/discussions/related work:
* From the appendix, I understand that for convolution layers you conduct the preconditioning on the reshaped kernels. However, the condition numbers/singular values of this reshaped kernel are NOT the same as those of the convolution layer (see the discussions in [1]). I wonder how your theorems and results would change considering this difference (e.g. how the theorems would change if you are controlling the condition numbers of the reshaped kernels instead of the layers, and how the results would be if you strictly controlling the condition number of convolutional layers)?
* The spectral normalization paper [2] shows that spectral normalization is robust across different learning rates and betas in the Adam optimizer. You only tried one learning rate and one beta for each experiment. How sensitive are FPC and APC to the learning rates and betas?
* I realize that this paper is very relevant to [1] (which is online recently so I understand that you are not required to know it). That paper also found out the instability issue in different variants of spectral normalization, and proposed an improved version (which are different from yours but could be relevant). Their results on CIFAR-10 and STL-10 seem to be better than FPC and APC you proposed in Table 4. You might want to add the discussions and/or experimental comparisons to it.
* Another missing related work is [3], which discussed the importance of condition numbers in the generators of GANs. They proposed Jacobian Clamping for controlling the condition numbers by regularization. Although they focused on generators, nothing stops it from applying Jacobian Clamping on discriminators. You might want to add the discussions and/or experimental comparisons to this approach.

Writing:
* It is weird to discuss the scaling trick twice in Section 3.1 and in the "Choice of desirable range" paragraph of Section 3.3. 
* Before the "Choice of target function" paragraph of Section 3.3, the scaling trick is already discussed, and the previous paragraph ends with [\gamma_L,\gamma_U]=[0,1.1]. It is weird to go back to [\gamma_L,\gamma_U]=[\lambda_1,\lambda_m] again in this paragraph. It is better to stick to the scaled version of singular values in this paragraph.
* It is better the discuss briefly how SVD works in the main text as you are comparing with it.

Typos:
* 6th line in "Choice of desirable range" paragraph in Section 3.3: \sigma_{min}(A) -> \sigma_{min}(A)/\sigma_{max}(A)
* The end of "Search space of preconditioning polynomial" paragraph in Section 3.3: wrong latex code for the section reference?
* Last line before Section 4: compputation -> computation
* Paragraph "Failure mode: large condition numbers." in Section 4: missing a period at the end
* Header of table 1: there should be a bar on top of each W_l
* Caption of table 1: there should be a bar on top of the W_l
* Header of table 2: there should be a bar on top of the W_l
* The last two paragraphs on page 18 & the first paragraph on page 19 (appendix): Table -> Algorithm
* The first paragraph in Appendix F.1: deg-9 -> deg-7
* The last paragraph on page 28 (appendix): mojority -> majority

[1] Lin, Zinan, Vyas Sekar, and Giulia Fanti. "Why Spectral Normalization Stabilizes GANs: Analysis and Improvements." arXiv e-prints (2020).

[2] Miyato, Takeru, et al. "Spectral normalization for generative adversarial networks." arXiv preprint arXiv:1802.05957 (2018).

[3] Odena, Augustus, et al. "Is generator conditioning causally related to gan performance?." arXiv preprint arXiv:1802.08768 (2018).
review_title: Good paper with clear motivation, nice approach, and good results

Meta-review: This paper presents a new method for training GAN by adding a precondition Layer. All reviewers are positive about the empirical results. However, some concerns were raised about the justification: (1) Only linear networks are considered, which is a bit impractical; (2) Existing work has shown the importance of controlling the whole spectrum instead of the condition number. There should be some connection missing between the proposed result and existing results; (3) The computational cost is a bit high. The paper would be much stronger if these concerns could be addressed.
Decision: Reject