Precondition Layer and Its Use for GANs
Authors: Tiantian Fang
Alex Schwing
Ruoyu Sun
Venue:  (2021)
Keywords: GAN
Preconditioning
Condition Number

Abstract: One of the major challenges when training generative adversarial nets (GANs) is instability.  To address this instability spectral normalization (SN) is remarkably successful.  However, SN-GAN still suffers from training instabilities, especially when working with higher-dimensional data.  We find that those instabilities are accompanied by large condition numbers of the discriminator weight matrices. To improve training stability we study common linear-algebra practice and employ preconditioning.   Specifically,  we introduce a  preconditioning layer  (PC-layer)that performs a  low-degree polynomial preconditioning.   We use this  PC-layer in two ways:  1)  fixed preconditioning  (FPC)  adds a  fixed  PC-layer to all layers, and 2)  adaptive preconditioning  (APC)  adaptively controls the strength of preconditioning. Empirically, we show that FPC and APC stabilize the training of un-conditional GANs using classical architectures.  On LSUN256×256 data, APC improves FID scores by around 5 points over baselines.

Under review as a conference paper at ICLR 2021
PRECONDITION LAYER AND ITS USE FOR GANS
Anonymous authors
Paper under double-blind review
ABSTRACT
One of the major challenges when training generative adversarial nets (GANs) is instability. To address this instability spectral normalization (SN) is remarkably successful. However, SN-GAN still suffers from training instabilities, especially when working with higher-dimensional data. We find that those instabilities are accompanied by large condition numbers of the discriminator weight matrices. To improve training stability we study common linear-algebra practice and employ preconditioning. Specifically,, we introduce a preconditioning layer (PC-layer) that performs low-degree polynomial preconditioning. We use this PC-layer in two ways: 1) fixed preconditioning (FPC) adds fixed PC-layer to all lay- ers; and 2) adaptive preconditioning (APC) adaptively controls the strength of preconditioning. Empirically, we show that FPC and APC stabilize training un- conditional GANs using classical architectures. On LSUN 256 x 256 data, APC improves FID scores by around 5 points over baselines.
INTRODUCTION
Generative Adversarial Nets (GANs) GGodffello et al. 2014) one distribution to another. Nevertheless, training GANs is known be challenging, and its perfor- mance is often sensitive to hyper-parameters and datasets. Understanding the training difficulties of GAN is thus an important problem.
successfully transform samples from
Recent studies in neural network theory (Pennington et al. 2017 Xiao et al. 2018 2020) suggest that the spectrum of the input-output Jacobian or neural tangent kernel (NTK) an important metric for understanding training performance. While directly manipulating the spectrum of the Jacobian or NTK is not easy, a practical approach to manipulate the spectrum of weight matrices, such orthogonal initialization (Xiao et al. 2018). For special neural net, Hu al. (2020) showed that orthogonal initialization leads to better convergence result than Gaussian initialization, which provides early theoretical evidence for the importance of manipulating the weight matrix spectrum.
Motivated by these studies, we suspect that an 'adequate' weight matrix spectrum is also important for GAN training. Indeed, one of the most popular techniques for GAN training, spectral normal- ization (SN) (Myyato et al., 2018), manipulates the spectrum by scaling all singular values by a is upper bounded. However,
constant. This ensures the spectral parameters and for high-resolution datasets, SN-GAN fails to generate good images. In study we find the condition numbers of weight matrices become very large and the majority of the singular values are close to 0 during training. See Fig. I a) and Fig. 2 a). This can happen as SN does not promote small condition number.
find that for some hyper-
This finding motivates to reduce the condition number of weights during GAN training. Recall that controlling the condition number is also a central problem in numerical linear algebra, known as preconditioning (see Chen (2055) ) We hence seek to develop "plug-in" preconditioner for weights. This requires the preconditioner to be differentiable. Out of various preconditioners, we find the polynomial preconditioner to be suitable choice due to the simple differentiation and strong theoretical support from approximation theory. Further, we suggest to adaptively adjust the strength of the preconditioner during training the efficacy of preconditioning on CIFAR1O (32 x 32), STL (48 x 48) and LSUN bedroom, tower and living room (256 256).
not overly restrict the expressivity. We show
Summary of contributions. For a deep linear network studied in (uu et al,, 2020). we prove that if all weight matrices have bounded spectrum, then gradient descent converges to global min-

---

Under review as a conference paper at ICLR 2021
(a)
(b)
Figure 1: Evolution of the smallest singular values (a) SN-GAN, FID 147.9 and (b) APC-GAN, FID 34.08 when generating STL-10 images with ResNet trained with Du 1 for 2006 iterations. The max singular value around due to SN, thus shown here.
Conditin Number
200-
3.0
Condionn
10 16 10 20 X10000 Beration
10 12 Herrton
(a)
(b)
Figure 2: Evolution of condition number of (a) SN-GAN, FID 147.9 and (b) APC-GAN, FID 34.08 when generating STL-10 images with ResNet trained with Dit = for 200k iterations.
imum at a geometric rate. We then introduce a PC-layer (preconditioning layer) that consists of low-degree polynomial preconditioner. We further study adaptive preconditioning (APC) which adaptively controls the strength of PC on different layers in different iterations. Applying PC and APC to unconditional GAN training LSUN data (256 x 256), permits generate high-quality images when SN-GAN fails. We also show that APC achieves better FID scores on CIFAR10, 10, STL, and LSUN than a recently proposed method of Jiang et al. 2019
1.1 RELATED WORK
Related to the proposed method is work by Jiang et al. (019), which also controls the spectrum in GAN training. They re-parameterize a weight matrix W via W = USVT, add orthogonal regu- larization of U, V and certain regularizer of entries of the diagonal matrix S. This approach differs from ours in a few aspects. First, Jiang et al. (209)) essentially solves constrained optimization using penalty method (Bertsekas)
problem with constraints UTU 7, VVV trast, our approach solves an unconstrained problem since we add one layer into the neural net, similar batch normalization (BN) (loffe & Szegedy 2015) and SN (Miyato et al. 2018). Second, our PC layer is a direct generalization of SN as it includes SN-layer as a special case. In contrast, the method of Jiang al. (019) differs from SN-layer any case. Our proposed method thus offers smoother transition for existing users of SN.
1997). In con-
In a broader context, number of approaches have been proposed to stabilize and improve GAN training. such as modifying the loss function (Arjovsky et al. 2017 Arjovsky & Bottou 2017 Mao
al.
et al., 2017b; Deshpande et al.) 2018), normalization and regularization (Gulrajani 2017 Miyato et al. 2018), progressive growing techniques (Karras et al., 2018 01 Huang et al.)
2017
et al. 2017). ) changing architecture (Zhang et al. 2019; Karnewar& & Wang 2019). utilizing side informa-
tion like class labels (Mirza & Osindero 2014 Odena et al. 2017| Miyato & Koyama 2018). Using this taxonomy, our approach fits the "normalization and regularization" category (even though our method is not exactly normalization, the essence of "embedded control" is similar). Note that these directions are relatively orthogonal, and our approach can be potentially combined with other tech- niques such as progressive growing. However, due to limited computational resources, we focus on unconditional GANs using classical architectures, the setting studied by Miyato el al. 20188
1.2 NOTATION AND DEFINITION
We use eig(A) to denote the multiset (i.e., allow repetition) of all eigenvalues of A. afl eigenvalues of A are neg-tege real numbers, say A is positive semidefinite (PSD) matrix. The singular values of a matrix A E R=xm are the square root of the eigenvalues of A A E Rm Let mmax (A) and mmin (A) denote the maximum and minimum singular values of A. Let ||A]|2 denote
2

---

Under review as a conference paper at ICLR 2021
the spectral norm of A, i.e., the largest singular value. The condition number of square matrix A is traditionally defined ((A) ||A|||/|| 112
Gmax (A) We extend this definition rectangular
amin (A)
Let deg(p) denote the degree of a polynomial p and let Pk- = {p deg(p) < k} be the set of polynomials with degree no more than k.
matrix A E gmxm where n 2 17 via ((A) Imax
umin ())
2
WHY CONTROLLING THE SPECTRUM?
To understand why controlling the spectrum is helpful we leverage recent tools in neural network theory to prove the following result: if weight matrices have small condition numbers, then gradient descent for deep pyramid linear networks converges to the global-min fast. This is inspired by Hu
et al.
2020p analyzing a deep linear network to justify orthogonal initialization.
Similar to Hu et al. 2020), we consider a linear network that takes an input x' E IR4xx1 x1 and outputs
F(0:z)= = W.WL-1--. Wix E R
(1)
where 0 = (W1, Wt) represents the collection of all parameters and W, is matrix of di- mension d, x dj-1- I,...,,, L. Here we define dg = d₂ and d2 = dy. Assume there exists dg S d1 S
E {1..... L}, such that dy dp dz-1 S d,, and the network a pyramid network, which generalizes the equal-width network of Hu et al. 2020).
d,. This means
Suppose y (yii... :()) E Rd the labels, and the predictions are F(0:X) (f); F (0; E xR x1, We consider a quadratic loss L(() = 2 by - F(O; X)
Starting from 0(0), we generate ((k) (W1 (R), ... WL (L)), & 1,2,... via gradient descent:
O(k + 1) 0(k) (V(()(
Denote the residual e(k) = F(O(k); X) y. For given n 2 1, 2 0,1=1,.... I...., L. define
R 2 (0 = (W,...,,Wy T max (W)) 2 amin (Wi) 2 μl: Vl}. .2)....))22
B A L -.. T1 (I|c(0)] ||X
P = (#I.... ... 22) o (X)
min
The following result states that if ((k) stay within region R (i.e., weight matrices have bounded spectrum) for k- = 0, 1,..., K, then the loss decreases at geometric rate until iteration K. The rate (1 ) depends on ( which is related the condition numbers of all weights. (M
Theorem Suppose 7 = }.4 Assume 0(k) E R, = K. Then we have
|ee(k: + 1)1F² = ( ( k == =.0, K.
(2)
See Appendix D.3.I for the proof and detailed discussions.
For proper initial point ((0) where (0)'s are full-rank, we can always pick Ti oi so that 0(0) E R. The trajectory {0(k) either stays in R forever (in which case K = oo), leaves R at some finite iteration K. the former case, the loss converges to zero at geometrical rate; the latter case, the loss decreases to below (1- 3)33** [|e(0)l However,
theorem does not specify how large K is for a given situation. Previous works on convergence (c.g., Hu et al. 2020 Du et al. 2018
Allen-Zhu et al. 2019 Zou al. 2018 bound the movement of the weights with extra assumptions,
so that the trajectory stays in a certain nice regime (related to R). We do not attempt to prove the trajectory stays in R. Instead, we use this as motivation for algorithm design: if can improve the condition numbers of weights during training, then the trajectory may stay in R for longer time, and thus lead smaller loss values. Next, we present the preconditioning layer such a method.
PRECONDITIONING LAYER
In the following, we first introduce classical polynomial preconditioners in Sec. 3.1 the preconditioning layer for deep nets in Sec. 3.2 We explain how to compute a preconditioning polynomial afterwards in Sec. 3.3 and finally present adaptive preconditioning in Sec. 3.4
We then present
3

---

Under review as a conference paper at ICLR 2021
3.1 PRELIMINARY: POLYNOMIAL PRECONDITIONER
Preconditioning considers the following classical question: for a symmetric matrix Q, how to find an operator 9 such that k()(Q)) is small? Due to the importance of this question and the wide applicability there is huge literature on preconditioning. See, e.g., [hhen (2005) for an overview, and Appendix B for a short introduction. In this work, we focus on polynomial preconditioners (Johnson et al. (983). The goal is to find polynomial P such that PQQ)Q has small condition number. The matrix p(Q) is often called preconditioner, and ((Q) & p{Q)Q is the precondtioned matrix. We call g the preconditioning polynomial. Polynomial preconditioning has a special merit: the difficult problem of manipulating eigenvalues can be transformed to manipulating I-d function, based on the following fact (proof in Appendix E.
Claim 3.1 Suppose I is any polynomial, and Q E R is a real symmetric matrix with eigen- values X1 S <... S Am- Then the eigenvalues of the matrix o(Q) are o(A:), -1...,m. As corollary, if (()( C [] - e, 11, then eig()(Q)) S [1,,1
m.
To find a matrix g(Q) = Q that is well-conditioned, we need to find a polynomial ý such that ((x) p(z)x maps [Az, Am. to [1 c, 1]. This can be formulated as a function approximation problem: find a polynomial ((x) of the form xf(2) that approximates a function (A) in E (X1, Am Under some criterion, the optimal polynomial is variant of the Chebychev polynomial, and the solutions to more general criteria are also well understood. See Appendix B.T for more.
A scaling trick is commonly used in practice. It reduces the problem of preconditioning Q to the problem of preconditioning a scaled matrix Qsea Q/ Q//m- Scaling employs two steps: first, we find polynomial, g that approximates [xx 1 in /Am: 1]; second, set (A) = (A/Am. and use g(Q) g(Q/Ami = g(Quca) as the final preconditioned matrix. It is easy to verify 9 approximates f(A) = in [}1, Amal. Thus this approach is essentially identical to solving the approximation problem in Am Johnson et al. (188)) use this trick mainly to simplify notation since they can assume Am without loss of generality. We will use this scaling trick for a different purpose (see Section
3.2 PRECONDITIONINS LAYER IN DEEP NETS
Suppose D(W WL) is deep net parameterized by weights Wi...., WL for layers e {I...., L}. To control the spectrum of weight Wi, we want to embed a preconditioner 9 into the neural net. Among various preconditioners, polynomial ones are appealing since their gradient simple and permits natural integration with backpropagation. For this we present preconditioning layer (PC-layer) as follows: a PC-layer g(W) g(NN W )) is the concatenation of a precondition- ing polynomial g and an SN operation of (Myytoo et al. 2018) (see Appendix app sub: details of FPC and APC for details of S.. The SN operator is used as a scaling operator (reason explained later). We describe an efficient implementation of PC-layer in Appendix C.3
In our case, we use A = SN(W) to indicate the scaled matrix. Prior work on polynomial pre- conditioners (Jhhnson et al., 1983 Chen 2005) often study square matrices. To handle rectangular matrices, some modifications are needed.
A naive solution is to apply preconditioner to the symmetrized matrix ATA, leading matrix g(4) = P(AT A)ATA. A. This solution works for linear models (see Appendix B.2 for details), but is not appropriate for deep nets since the shape of p(AA A" A E Rm differs from A. To maintain the shape of size n X m, we propose to transform A to g A) p(AA²') A E Rexm, This transformation works for general parameterized models including linear models and neural nets. For a detailed comparison of these two approaches, see Appendix B.2 The following claim relates the )
spectrum of A and ((AAT)) A; see the proof in Appendix F.2.2
Claim 3.2 Suppose E paxm has singular values 0 S ... < Tm. Suppose g(x) = p(x²)x where p is polynomial. Then the singular values of ((A) p(AA") are (g(0r)).... ..... !g(am)]
To find polynomial p such that ((A) p(AAT) is well-conditioned.
need to find a poly- nomial p such that g(z) = p(2²) )r maps [0), om.] to [1 - e, 1] for some e. This can be formulated as function approximation problem: find a polynomial g(x) in Gk that approximates function
4

---

Under review as a conference paper at ICLR 2021
f(o) = i E [m1) where G₁ (g(x) xp(x²) p e Ps]. We will describe the algorithm
for finding the preconditioning polynomial g in Sec. 3.3
In principle, the PC-layer can be added to any deep net including supervised learning and GANs. Here, we focus on GANs for the following reason. Current algorithms for supervised learning already work quite well, diminishing the effect of preconditioning. In contrast, for GANs, there is a lot of room to improve training. Following SN-GAN which applies SN to the discriminator of GANs, in the experiments we apply PC the discriminator.
3.3 FINDING PRECONDITIONING POLYNOMIALS
In this subsection, we discuss how to generate preconditioning polynomials. This generation is done off-line and independent of training. We will present the optimization formulation and discuss the choice of a few hyperparameters such as the desirable range and the target function f.
Optimization formulation. Suppose we are given a range Tu]. target function S, and an integer k; the specific choices are discussed later. Suppose we want to solve the following approx- imation problem: find the best polynomial of the form g(x) = x(ao + 012² + .... + 2x that approximates f(x) in domain [YL, yu]. i.c., solve
min dm f(x), gEGr
where G2 {g(z) xp(x²) p Pkt, dm. distance metric on the function space (f..)))(f.g))) (f.g) max nol (f(t)) g(t)). We consider
Cul, such the loo distance dpr weighted least-square problem suggested by Johnson et al. (983):
min gEGi
[g(x)
TL
where w(z) x" weight function used in (Johnson al. 1983 We discretize the objective and solve the finite-sample version of Eq.
44 as follows:
min
Σ
S (xx) w(x;);
cm O4 CEREE1
where 2; e Yu], Vi (c.g., drawn from uniform distribution on ln., JuD. This is weighted least squares problem (thus convex) that can be easily solved by standard solver.
Choice of desirable range [TL, 7uJ. The range [7L, vu] within which we want to approximate the target function is often chosen to be the convex hull of the singular values of the matrix to be preconditioned. For the original matrix W, the desirable range (2L, = [o (W), Fmax (m))] However, this range varies across different layers and different iterations. For this reason we scale each W by 1/|[W]]a to obtain so that its singular values lie in a fixed range (0. 1). Note that more precise range is [m ( max (A), 1J. but we can relax it to [0, 1J. We follow Miyato et al. (0188) to use one power iteration to estimate the spectral norm W ~ [ and denote SN(W) W/W.
Since W is not exactly ||W]|,, the range of singular values of A SN(W) may not be exactly in [0, 1]. We have checked the empirical estimation and found that the estimated spectral norm during the training of SN-GAN is often less than 1.1 times the true spectral norm (see Fig. in Appendix E.I. thus we pick In. yu] = [0, 1.1] in our implementation.
Choice of target function fA). Previously, we discuss the ideal situation that [7,, Tul [Λi Am]- thus the target function is 1. In the previous paragraph, we have relaxed the desirable range to [0. 70]. then we cannot set f(x)
in [0, yc]. because any polynomial g()) E G must satisfy g(0) = O, causing large approximation eτror at À = O. We shall set f(0) 0. A candidate target
[x/6, I < b
function is PL)(z), where PL)(x) 1 J > b
is defined as piece-wise linear function with
cutoff point If the cutoff point 6 Amin (A), then PLa(A) maps all singular values of to 1.
While setting all singular values to 1 is ideal for fast training, this may reduce the expressiveness for deep nets. More specifically, the set of functions (D(WI,.... Wr. | eig(W? Wi) S {1)} is smaller than (D(Wi,.... Wr) eig(W?W,) C (Ao, 1)), thus forcing all singular values be
5

---

Under review as a conference paper at ICLR 2021
precondiioviigg
0.0
0.7
14
Figure 3: Left: Different piecewise linear functionn; Right: the corresponding fitted preconditioning polyno- mials. The smaller the cutoff, the more aggressive the preconditioner. For instance, when the cutoff is 0.3, the preconditioner pushes all singular values above 0.3 We show the fitting polynomials of degree 3, 5, 7,9 for PLo.s, Pl... e, Pl..., 4, PL... respectively. More details are in Sec.
may hurt the representation power. Therefore, do not want the target function to have value 1 in [Amin (4), Ntl- In practice, the value of Amin (A) varies for different problems, therefore we permit flexible target function f. be chosen by a user.
In our implementation, we restrict target functions to a family of piece-wise linear functions. We (I:) with relatively large cutoff point b, such 0.8 and 0.3. We plot our candidate target functions PLo.3 PLo.A, PLo.6 and PLo.s in Figure 3
As the cutoff point b changes from to O, pushes more singular values 1. As a result, the
the function PL, becomes more aggressive optimization will likely become easier, while the representation power becomes weaker. The exact choice of the target function is likely problem-dependen and we discuss two strategies select them in Section 34
Search space of preconditioning polynomial. As mentioned earlier, the default search space G5 [9(x) xp(x²) p E Ps. for a pre-fixed k. The degree of g(A) is an important hyperpa- rameter. On the one hand, the higher the degree, the better the polynomial can fit the target function f. On the other hand, higher degree leads more computation. In our implementation, we consider
1, 2,3, 4, i.c., polynomials of degree 3, 5, and 9. The extra time is relatively small; see Section for details.
3.4 FIXED PRECONDITIONING AND ADAPTIVE PRECONDITIONING
The preconditioning polynomial be determined by the target function and the degree k. Which polynomial shall we use during training?
Candidate preconditioners. At first sight, there are two hyper-parameters b and k. Nevertheless, if 6 is small (steep slope), then it is hard to approximate PL, by low-order polynomials. For each degree & {3,5, 5, 7, 9). there is a certain by such that b < by leads large approximation error. We find that by R 0.8, by R 0.6, by Rs 0.4, by R 0.3. After fitting PLo.3. PL..4. PLo.6 and PL..s. the resulting polynomials are shown in Figure
A natural approach is to add the PC-layer to all layers of the neural net, resulting in preconditioned net Drc(0) (0) D(g (SN(W))) (S)(WW))) We call this method fixed preconditioning (FPC). Just like other hyperparameters, in practice, we can try various preconditioners and pick the one with the best performance. Not surprisingly, the best preconditioner varies for different datasets.
Adaptive preconditioning (APC). Motivated by adaptive learning rate schemes like Adam (Kingma & Bal Ba|2001 2014) and LARS (You et al., (2077), we propose an adaptive preconditioning scheme. In APC, we apply the preconditioner in a epoch-adaptive and layer-adaptive manner: at each epoch and for each layer the algorithm will automatically pick a proper preconditioner based on the current condition number.
The standard condition number x(A) =
aman (4)
not necessarily a good indicator for the opti-
amin (A)
mization performance. In APC, we use a modified condition number R(A) = (Σ:"" (.(A))/mp
amax (m)
where A has m columns and mg = [m] We prepare r preconditioning polynomials 91..... 9r with different strength (e.g.. the four polynomials 91, 92 Ia. 94 shown in Figure B). We set number of ranges [0, 71 [n, T2l..... [T,, oo] and let To = 0, T 0o. If the modified condition number of A falls into the range [T,, T241 for (0. 1, . will use di in the PC-layer. In our im-

---

Under review as a conference paper at ICLR 2021
plementation, we set 4. To save computation, only compute the modified condition number and update PC strength at a fixed interval (e.g., every 1000 iterations). The summary of APC is presented in Table B in Appendix
Computation time. We use a few implementation tricks; see Appendix In our implementation of FPC with degree 3,5,7 of the original training time of SN-GAN. Fig. (b) shows that the extra time of APC over SN is often less than 10%. See Appendix C.4 for more on the computation time.
polynomial, the actual added time is around 20 30% (Fig. 4 (a))
EXPERIMENTAL RESULTS
We will demonstrate the following two findings. First, SN-GAN still suffers from training instabili- ties, and the failure case is accompanied by large condition numbers. Second, PC layers can reduce the condition number, and improve the final performance, especially for high resolution data (LSUN 256 x 256).
We conduct a set of experiments for unconditional image generation on CIFAR-IO (32 x 32), STL- 10 (48 x 48), LSUN-bedree (128 x 128 and 256 x 256). LSUN-tower (256 x 256) and LSUN- living-room (256 x 256). We also compare the condition numbers the discriminator layers for different normalization methods to demonstrate the connection between the condition number and the performance. The following methods are used in our experiments: standard SN; SVD with D- Optimal Reg. (Tiang al. 2019;; FPC with degree 3 or 7 preconditioners; APC. Following Miyato et al. 2018). we use the log loss GAN on the CNN structure and the hinge loss GAN on the ResNet structure.
CIFAR and STL: Training failure of (1.1)-update. Tuning GAN is notoriously difficult and sensitivity to eyperparaameers Even for low-resolution images, without prior knowledge of good hyper-parameters such as Da, Gtt, training GAN is often not trivial. On CIFARIO, SN-GAN uses Dr. = 5, G1 = for ResNet; for simplicity, we call a (5, 1uppddtt. However, using (1, 1)- update, i.e., changing Da 5 to Di. = 1 while keeping Ga. 1, will lead to an SN-GAN training failure: dramatic decrease of final performance and an FID score above 77. SN-GAN with (1, update also fails STL data, yielding
FID score above 147. We are interested in stabilizing the (I, 1upuddtte for two reasons: first, trainability for both (1, 1))udate and (5, 1)-update means improved training stability; second, the (1, 1) upddte requires only about 1/3 of the time compared to the (5, 1) uppdaee. Therefore, in the first experiment, we explore GAN-training with (1, 1) uppdaee on CIFAR-10 10 and STL-10.
Failure mode: large condition numbers. Understanding the failure mode of training is often very useful for designing algorithms (e.g.. Glorot & Bengio, 2010). We suspect that a large condition number is a failure mode for GAN training. As Table I shows, the high FID scores (bad case) of SN-GAN are accompanied by large condition numbers.
PC reduces condition numbers and rectifies failures. Table shows that FPC and APC can both greatly improve the training performance: they reduce FID from 77 to less than 20 for CIFAR-IO, and reduce FID from 147 to less than 34 for STL in 200k iterations. The evolution of the 5 smallest singular values of the adaptive preconditioned matrices and the condition numbers are showed in Fig. (b) and Fig. 1b) for STL-10 training on ResNet with Di 1. This shows that PC-GAN successfully improves the spectrum of weight matrices in this setting.
Experiments on "good" case of SN-GAN. We report the results for the (5, 1) 1)-update on CIFAR- and STL-10 with ResNet the Appendix. For those FPC and APC achieve similar or slightly better FID scores. We also report IS scores there. We also list the results of PC and multiple baselines on the CNN structure in the Appendix.
High resolution images LSUN. Using high-resolution data is more challenging. We present numer- ical results on LSUN bedroom (128 x 128, and 256 x 256). LSUN tower (256 x 256) and LSUN living room (256 x 256) data in Table The training time for one instance is 30 hours on a single RTX 2080 Ti (200k iterations).
Note, SN-GAN is unstable and results in FID 80 for LSUN-bedroom 256 256. The SVD method, our FPC and APC generate reasonable FID scores on all three data sets. Importantly, our FPC is comparable or better than SVD, and our APC consistently outperforms the SVD method by
7

---

Under review as a conference paper at ICLR 2021
Setting
Method
FID score
max R(Wi)
2nd max R(Wi)
Avg R(W))
CIFAR-10
SN-GAN
77.82
113.55
19.46
17.77
ResNet
FPC, deg-3
20.09
22.65
1.88
3.16
Dit
FPC, deg-7
19.31
9.88
1.32
1.85
APC
19.53
12.06
2.81
3.19
STL-10 ResNet
SN-GAN
147.90
217.06
167.97
53.37
FPC, deg-3
33.99
23.75
22.35
4.36
Dj =1
FPC, deg-7
34.28
2.55
2.72
1.53
APC
34.08
6.79
6.47
3.90
Table Comparison SN-GAN PC-GAN, using ResNet with Da. Here W; preconditioned weighted matrix (i.e., after applying preconditioning). "2nd maxf-1 R(W))" indicates the second largest con- dition number of all layers. "Avg R(Wi)" indicates the average of all layer condition numbers.
Setting
Method
FID score
max R(Wi)
2nd max K(Wi)
Avg R(Wi)
SN
53.48
5.00
2.20
2.14
LSUN
GaTech
50.92
1.75
1.55
1.46
Bedroom 128
PC, deg-3
51.05
1.23
1.15
1.09
APC
45.32
2.75
2.40
1.82
SN
81.09
26.19
6.02
6,45
LSUN Bedroom 256
GaTech
36.89
2.76
2.18
1.82
PC, deg-3
35.61
3.04
2.03
1. 1.64
APC
31.17
4.35
2.80
2.18
SN
193.75
66.87
9.86
13.54
LSUN
GaTech
32.79
2.00
1.95
1.60
Living Room 256
PC, deg-3
28.20
2.95
2.91
1.64
APC
28.29
3.99
3.63
2.23
SN
33.10
12.75
3.48
3.50
LSUN Tower 256
GaTech
35.05
2.83
2.36
1.85
PC, deg-3
30.81
2.43
1.45
1.33
APC
29.58
3.38
2.53
1.91
Table 2: Results on LSUN data.
4-6 FID scores in most cases. Also note, the condition numbers of the failure case of SN-GAN much higher than the two normal cases of SN-GAN. In all cases, FPC and APC achieve significantly lower condition numbers than SN-GAN. APC achieves higher condition numbers than FPC, and also better FID scores. We suspect that FPC over-controls the condition numbers which leads to lower representation power. In contrast, APC strikes better balance between representation and optimization than FPC. The generated image samples are presented in Appendix F.5
5 CONCLUSION
We prove that for a deep pyramid linear networks, if all weight matrices have bounded singular values throughout training, then the algorithm converges to global minimal value at a geometric rate. This result indicates that small weight matrix condition numbers are helpful for training. We propose preconditioning (PC) layer to improve weight matrix condition numbers during training, by leveraging tools from polynomial preconditioning literature. It is differentiable, and thus can be plugged into any neural net. We propose two methods to utilize the PC-layer: in FPC (fixed preconditioning). we add fixed PC-layer to all layers; in APC (adaptive preconditioning) we add PC-layers with different preconditioning power depending we show that applying FPC and APC to GAN training. we can generate good images in few cases that SN-GAN perform badly, such LSUN-hheroom x56 256 image generation.
the condition number. Empirically,
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jonas Adler and Sebastian Lunz. Banach wasserstein gan. In NeurlPS, 2018.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In ICML, 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In ICLR,, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. In ICML, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In ICML, 2017.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint Xr 1955, 2019.
David Balduzzi, Sebastien Racamiere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Grae- pel. The mechanics of n-player differentiable games. arXiv preprint arXtv:1802.05642. 2018.
Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards AI. In Large Scale Kernel Machines. MIT Press, 2007.
Hugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent, and Simon Lacoste-Julien. A closer look the optimization landscapes of generative adversarial networks. arXiv preprint 1906.0/906.04484 2019.
David Berthelot, Tom Schumm, and Luke Metz. Began: Boundary equilibrium generative adversar- ial networks. arXiv preprint arXiv: 1703. 2017.
Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3): 334-334, 1997.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.07093, 2016.
Ke Chen. Matrix preconditioning techniques and applications, volume 19, Cambridge University Press, 2005.
Ruohan Wang Antoine Cully, Hyung Jin Chang, and Yiannis Demiris. Magan: Margin adaptation for generative adversarial networks. arXiv preprint arXiv:1704.03817, 2017.
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. In ICLR, 2018.
Ishan Deshpande, Ziyu Zhang, and Alexander Schwing. Generative modeling using the sliced wasserstein distance. In CVPR, 2018.
Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint arXiv: 1811::111..000 1.3804, 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal machine learning research, 12(7), 2011.
Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. A variational inequality perspective on generative adversarial networks. arXiv preprint arXiv: 100. 2018.
Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Remi Lepriol, Gabriel Huang, Si- mon Lacoste-Julien, and Ioannis Mitliagkas. Negative momentum for improved game dynamics. In AISTATS, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.

---

Under review as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In NeurlPS, 2017.
Roger A Horn, Roger A Horn, and Charles R Johnson. Topics in matrix analysis. Cambridge university press, 1994.
Wei Hu, Lechao Xiao, and Jeftrey Pennington. Provable benefit of orthogonal initialization in opti- mizing deep linear networks. arXiv preprint arXi::2001.0592) 2020.
Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, and Serge Belongie. Stacked generative adversarial networks. In CVPR, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1rXiv:1502.03 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In NeurlPS, 2018.
Haoming Jiang, Zhehui Chen, Minshuo Chen, Feng Liu, Dingding Wang, and Tuo Zhao. On com- putation and generalization of generative adversarial networks under spectrum control. In Inter- national Conference on Learning Representations, 2019.
Olin G Johnson, Charles A Micchelli, and George Paul. Polynomial preconditioners for conjugate gradient calculations. SIAM Journal on Numerical Analysis, 20(2):362-377 1983.
Animesh Karnewar and Oliver Wang. Msg-gan: multi-scale gradient gan for stable image synthesis. arXiv preprint arXiv: 1rXxvx1903000045 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In ICLR, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
Soheil Kolouri, Charles E Martin, and Gustavo K Rohde. Slised-wasserstein autoencoder: An embarrassingly simple generative model. arXiv preprint 1 2018.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. In Advances in neural information processing systems, pp. 8572-8583, 2019.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. In NeurIPS, 2017a.
Chunyuan Li, Changyou Chen, David Carlson, and Lawrence Carin. Preconditioned stochastic gradient langevin dynamics for deep neural networks. arXiv preprint arXiv:1512.07666, 2015.
Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. Towards understanding the dy-
namics of generative adversarial networks. In ICML, 2018
Li, A. G. Schwing, K.-C. Wang, and R. Zemel. Dualing GANs. In NeurlPS, 2017b.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. In NeurlPS, 2018.
Zinan Lin, Vyas Sekar, and Giulia Fanti. Why Spectral Normalization Stabilizes GANs: Analysis and Improvements. In arXiv e-prints, 2020.
Xudong Mao, Qing Li. Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In ICCV, 2017.
10

---

Under review as a conference paper at ICLR 2021
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin, Which training methods for gans do actually converge? In ICML, 2018.
Luke Metz, Ben Poole. David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks, ICLR, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.1784, 2014.
arXiv preprint
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. urXiv:1802.05637 2018.
arXiv preprint
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Youssef Mroueh and Tom Sereu. Fisher gan. NeurIPS, 2017.
Youssef Mrouch, Tom Sercu, and Vaibhava Goel. Mcgan: Mean and covariance feature matching gan. arXiv preprint weXiv::702.083.. 2017.
Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In Pmo ceedings of the 34th International Conference Machine Learning- Volume 70, pp. 2603-2612.
JMLR. org, 2017.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In NeurlPS, 2016.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In International conference on machine learning, pp. 2642-2651, 2017.
Augustus Odena, Jacob Buckman, Catherine Olsson, Tom B. Brown, Christopher Olah, Colin Raf- fel, and Ian Goodfellow. Is Generator Conditioning Causally Related to GAN Performance? In ICMI., 2018
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Advances in neural information processing systems, pp. 4785-4795, 2017.
Ben Poole, Alexander A Alemi, Jascha Sohl-Dickstein, and Anelia Angelova. Improved generator objectives for gans. arXiv preprint arXiv: 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization to accelerate training of deep neural networks. In Advances in neural information processing systems, pp. 901- 909, 2016.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), NeurIPS, pp. 2234- 2242. Curran Associates, Inc., 2016. URL htttp://papers wwt..m...pppppppe....eeppppppppe
6125 penrroovdd-eechnee sowtees--oo-traaee for-training gans.
Jiging Wu, Zhiwu Huang, Wen Li, Janine Thoma, and Luc Van Gool. Sliced wasserstein generative models. In CVPR, 2019.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. arXiv preprint 0tt::1886.053933 2018.
Lechao Xiao, Jeffrey Pennington, and Samuel S Schoenholz. Disentangling trainability and gener- alization in deep neural networks. 2020.

---

Under review as a conference paper at ICLR 2021
Yasin Yazııı, Chuan-Shenge Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, and Vijay Chan- drasekhar. The unusual effectiveness of averaging in gan training. In ICLR, 2019.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv: 708.03888, 6, 2017.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning, pp. 7354-7363. PMLR, 2019.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. arXiv preprint ar ArXiv:/81 2018.
12


Reviews:
[{"review_id": "nemx5xWqYwm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary:\n\nThis paper mainly solves the instability issue on the spectral normalization for generative adversarial networks (SN-GANs) when training with high dimensional data. To address this, the authors present a preconditioning layer (PC-layer) with two different ways (i.e., FPC and APC) to perform a low-degree polynomial preconditioning. Experiments on LSUN 256x256 training data demonstrate that FPC and APC are able to control the strength of preconditioning. My detailed comments are as follows.\n\n*Positive points:\n\n1. The instability of training GANs is an important research problem. This paper relieves this issue by introducing a preconditioning layer. \n\n2. This paper provides an empirical study on the training instability of GANs. The authors found that the instabilities are accompanied by large condition numbers of the discriminator weight matrices.\n\n*Negative points:\n\n1. Some theoretical evidence should be provided. In Section 1, the authors suspect that an ‘adequate’ weight matrix spectrum is also important for GAN training. Could you please provide sufficient theoretical analysis for this?\n\n2. This paper mainly addresses the instability issue on SN-GANs when training with high dimensional data. However, it is not clear why SN-GANs have poor performance on higher-dimensional data. It would be better to provide theoretical analysis to support this.\n\n3. Some technical details are not clear. The use of notations in this paper is confusing since they are used without clear explanations, which makes the paper hard to follow. For example, is $\\| \\|$ L2-norm by default? Should $F(\\theta; x)$ and $F(\\theta; X)$ be the same? Is the division in $W/ \\tidle{W}$ an element-wise division? In addition, it would be better to provide intuitive understanding for the proposed theorems and claims.\n\n4. In Section 2, this paper only considers a deep linear neural network. However, it is impractical for some real-world case. Could you please extend the analysis to more general case on deep non-linear neural network? In addition, the optimization problem mainly addresses the weighted least-square problem. Can it be extended to other losses, e.g., cross entropy loss?\n\n5. In the experiment, the authors improve the training instability on LSUN 256x256 data. Could you please conduct more experiments on higher-dimensional dataset, e.g., CelebA 512x512 data.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "R5YnsewX608", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new concept of pre-conditioning layers. The authors claim that this idea is novel and they introduce this idea in the context of GANs. I cannot verify or falsify if the idea is really novel, but assuming the idea is novel, I definitely believe it is worthy to be explored and eventually published somewhere. The idea is creative and technically interesting.\n\nThe authors argue that not only large singular values matter, but also the condition number.\nBased on their experiments, the author find that a bad condition number might be a reason for poor results using spectral normalization in a GAN.\n\nThe technical realization of the idea seems reasonable.\n\nThe evaluation is a bit difficult to follow. The baseline method is a GAN that is no-longer state of the art. It is not even clear to me what this GAN architecture is exactly that is being used. The description is a bit vague. I am not sure if the baseline architecture can be considered reasonable. Certainly, while spectral normalization might cause certain problem with one type of architecture, it is not that clear that this problem will persist in a state-of-the-art GAN and how many state-of-the-art GANs really rely on spectral normalization. For example, the StyleGANv2 paper writes\n\"It should be noted that spectral normalization [31] of the generator [46] only constrains the largest singular value, posing no constraints on the others and hence not necessarily leading to better conditioning. We find that enabling spectral normalization in addition to our contributions — or instead of them — invariably compromises FID, as detailed in Appendix E\"\n\nFurther, just relying on FID is risky. If there is no demonstration of visual quality improvement I would be quite skeptical if an improvement really has been made. It is ultimately up to the authors to demonstrate an improvement. With this submission, the authors shift a massive amount of experimental validation of the work to the reader. Who is going to try out your idea on a state-of-the-art GAN and evaluate visual quality?", "extra_scores": {}, "extra_text": {}}, {"review_id": "EdiKydx5iVM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update:  \nThank the authors for the detailed feedback. I decide to keep the score.\n---\n---\nThe paper shows that the failure mode of spectral normalization (SN) is often accompanied by large condition numbers in the discriminator layers. Motivated from this observation, the paper proposes to control the condition numbers of the discriminator layers, by adding preconditioning to the weights. The results show that the proposed approach makes the training more stable and achieves better sample quality on several datasets.\n\nOverall, I enjoy reading this paper. The motivation is clearly explained, and the approach is simple yet effective. I would recommend an accept. I do find that the writing needs to be improved (e.g. there are many typos in both the main paper and the appendix), and more experiments can be explored. However, I think these are relatively minor issues that do not diminish the overall quality of the paper.\n\nThe minor issues and suggestions are listed below.\n\nMissing details:\n* Section 3.4 mentioned that \"the best preconditioner varies for different datasets\". It is fine, but you should show the FPC results with all different degrees, so that it is clearer how sensitive FPC is to the degree. I only see the results with degree 3 and 7 in the main text and the appendix.\n* APC needs to use the singular values of the weights. How do you compute the singular values, by exact computation or estimation? If you do the exact computation, then I have a question about the scalability of the approach. For convolutional layers (which are the dominant components in the network architectures you experimented on), the computation is cheap because the kernels are small. But it might computationally expensive for fully connected layers.\n* How do you compute the actual true spectral norm in Appendix E.1? By exact computation or estimation?\n\nSuggested experiments/discussions/related work:\n* From the appendix, I understand that for convolution layers you conduct the preconditioning on the reshaped kernels. However, the condition numbers/singular values of this reshaped kernel are NOT the same as those of the convolution layer (see the discussions in [1]). I wonder how your theorems and results would change considering this difference (e.g. how the theorems would change if you are controlling the condition numbers of the reshaped kernels instead of the layers, and how the results would be if you strictly controlling the condition number of convolutional layers)?\n* The spectral normalization paper [2] shows that spectral normalization is robust across different learning rates and betas in the Adam optimizer. You only tried one learning rate and one beta for each experiment. How sensitive are FPC and APC to the learning rates and betas?\n* I realize that this paper is very relevant to [1] (which is online recently so I understand that you are not required to know it). That paper also found out the instability issue in different variants of spectral normalization, and proposed an improved version (which are different from yours but could be relevant). Their results on CIFAR-10 and STL-10 seem to be better than FPC and APC you proposed in Table 4. You might want to add the discussions and/or experimental comparisons to it.\n* Another missing related work is [3], which discussed the importance of condition numbers in the generators of GANs. They proposed Jacobian Clamping for controlling the condition numbers by regularization. Although they focused on generators, nothing stops it from applying Jacobian Clamping on discriminators. You might want to add the discussions and/or experimental comparisons to this approach.\n\nWriting:\n* It is weird to discuss the scaling trick twice in Section 3.1 and in the \"Choice of desirable range\" paragraph of Section 3.3. \n* Before the \"Choice of target function\" paragraph of Section 3.3, the scaling trick is already discussed, and the previous paragraph ends with [\\gamma_L,\\gamma_U]=[0,1.1]. It is weird to go back to [\\gamma_L,\\gamma_U]=[\\lambda_1,\\lambda_m] again in this paragraph. It is better to stick to the scaled version of singular values in this paragraph.\n* It is better the discuss briefly how SVD works in the main text as you are comparing with it.\n\nTypos:\n* 6th line in \"Choice of desirable range\" paragraph in Section 3.3: \\sigma_{min}(A) -> \\sigma_{min}(A)/\\sigma_{max}(A)\n* The end of \"Search space of preconditioning polynomial\" paragraph in Section 3.3: wrong latex code for the section reference?\n* Last line before Section 4: compputation -> computation\n* Paragraph \"Failure mode: large condition numbers.\" in Section 4: missing a period at the end\n* Header of table 1: there should be a bar on top of each W_l\n* Caption of table 1: there should be a bar on top of the W_l\n* Header of table 2: there should be a bar on top of the W_l\n* The last two paragraphs on page 18 & the first paragraph on page 19 (appendix): Table -> Algorithm\n* The first paragraph in Appendix F.1: deg-9 -> deg-7\n* The last paragraph on page 28 (appendix): mojority -> majority\n\n[1] Lin, Zinan, Vyas Sekar, and Giulia Fanti. \"Why Spectral Normalization Stabilizes GANs: Analysis and Improvements.\" arXiv e-prints (2020).\n\n[2] Miyato, Takeru, et al. \"Spectral normalization for generative adversarial networks.\" arXiv preprint arXiv:1802.05957 (2018).\n\n[3] Odena, Augustus, et al. \"Is generator conditioning causally related to gan performance?.\" arXiv preprint arXiv:1802.08768 (2018).", "extra_scores": {}, "extra_text": {}}, {"review_id": "-o9swdIngKP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The current work introduces a pre-conditioning (PC) layer using low-degree polynomials as a way to implement spectral normalization (SN) of the weight matrix W in the D-net. The primary application is to improve the training of GANs by SN techniques, which has been previously established in [1] and later works like [2]. \n\nThe empirical results measured by FID are good, and the experiments on higher resolution datasets like LSUN 256x256 is appreciated. However, the reviewer would like to point out some possible issues as below.\n\n- The first issue is the conceptual relation between matrix condition number and the GAN training instability. The condition number is the ratio of the largest and smallest singular value, and thus only depends on the extreme values (the \"edge\") of the spectrum. Meanwhile, previous works have already shown that controlling the whole spectrum is important and beneficial for GAN training, e.g. Fig 1 in [2]. Thus it is a question whether condition number is the right object to focus on, which motivates the PC approach.\nThe authors show that SN-GAN can have large condition numbers, and the PC-GAN improves it (Fig. 1), which accompanies improved FID score. However, this does not mean that it is the smallest singular value, not the whole spectrum, that brings the benefit. For example, when all the singular values are 1 except for the smallest one which is 1e-8, then the matrix is almost full rank and well-conditioned except for 1 direction, but the condition number is 1e8. Suppose the spectrum of W is like this, will the training instability remain? \nThe theory in Section 2 is about a simplified model (linear network) under another setting, and the reviewer is not convinced that the result there (convergence is upper bounded by a quantity related to the condition number) can resolve the issue and justify the importance of condition number. \n\n- Another possible issue is computational cost. First, the non-adaptive PC proposed is p(AA^T)A, where p is a polynomial. Even p is low-degree, this involves large matrix multiplication. Note that in the case of ill-conditioning, AA^T squares and worsen the condition number.\nSecond, the best empirical improvement comes from the adaptive PC, and less evident with the fixed PC. The APC method needs to compute the value of the modified condition number, and that seems to involve eigen decomposition (e.g. SVD or approximate SVD of the matrix A), which is usually expensive. \nSo how will the PC approach compare with the SVD like approach such as in [2] which directly controls the spectrum? It is not clear if the extra computational cost, especially eigen computation, can be justified in the practice of GAN training. \n\nOther comments:\n\n Inception score in the experimental results should also be reported in the main text.\n\n Thm 1, Eqn (2), is it \\mu/\\beta or \\beta/\\mu? \n\n[1] Miyato et al. \"Spectral normalization for generative adversarial networks.\" In ICLR, 2018.\n\n[2]  Jiang et al. \"On computation and generalization of generative adversarial networks under spectrum control.\" In ICLR, 2019.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new method for training GAN by adding a precondition Layer. All reviewers are positive about the empirical results. However, some concerns were raised about the justification: (1) Only linear networks are considered, which is a bit impractical; (2) Existing work has shown the importance of controlling the whole spectrum instead of the condition number. There should be some connection missing between the proposed result and existing results; (3) The computational cost is a bit high. The paper would be much stronger if these concerns could be addressed.
Decision: Reject