Single-level Adversarial Data Synthesis based on Neural Tangent Kernels
Authors: Yu-Rong Zhang
Reddy Su
Sheng-Yen Chou
Shan-Hung Wu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Adversarial
Data Synthesis
Neural Tangent Kernels

Abstract: Generative adversarial networks (GANs) have achieved impressive performance in data synthesis and have driven the development of many applications. How- ever, GANs are known to be hard to train due to their bilevel objective, which leads to the problems of convergence, mode collapse, and gradient vanishing. In this paper, we propose a new generative model called the generative adversarial NTK (GA-NTK) that has a single-level objective. The GA-NTK keeps the spirit of adversarial learning (which helps generate plausible data) while avoiding the training difficulties of GANs. This is done by modeling the discriminator as a Gaussian process with a neural tangent kernel (NTK-GP) whose training dynam- ics can be completely described by a closed-form formula. We analyze the conver- gence behavior of GA-NTK trained by gradient descent and give some sufficient conditions for convergence. We also conduct extensive experiments to study the advantages and limitations of GA-NTK and propose some techniques that make GA-NTK more practical.

Under review as a conference paper at ICLR 2023
SINGLE-LEVEL ADVERSARIAL DATA SYNTHESIS BASED ON NEURAL TANGENT KERNELS
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) have achieved impressive performance in data synthesis and have driven the development of many upplications. How- ever, GANs are known to be hard to train due to their bilevel objective, which leads to the problems of convergence, mode collapse, and gradient vanishing. this paper, propose new generative model called the generative adversarial NTK (GA-NTK) that has a single-level objective. The GA-NTK keeps the spirit of adversarial learning (which helps generate plausible data) while avoiding the training difficulties of GANs. This is done by modeling the discriminator as a Gaussian process with neural tangent kernel (NTK-GP) whose training dynam- ics can be completely described by a closed-form formula. We analyze the conver- gence behavior of GA-NTK trained by gradient descent and give some sufficient conditions for convergence. We also conduct extensive experiments to study the advantages and limitations of GA-NTK and propose some techniques that make GA-NTK more practical
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al. 2014]] Radford et al. (016). a branch of deep generative models based on adversarial learning, have received much attention due to their novel problem formulation and impressive performance in data synthesis. Variants of GANs have also driven recent developments of many applications, such as super-resolution (Ledig et al. 2017).
image inpainting (Xu et al.) 2014), and video generation (Vondrick et al. 2016).
A GANs framework consists of a discriminator network D and a generator network & parametrized by OD and 0g. respectively. Given d-dimensional data distribution Paat and c-dimensional noise distribution Pooise. the generator G maps a random noise z E R to a point g(z) E Rd in the data space, while the discriminator D takes point x' Rd as the input and tells whether x' is real or fake, i.e., D(x') = 1 if x' ~ Pasta and D(x:') = 0 if x' ~ Pgen where Pgen is the distribution of G(2) and P The objective of GANs is typically formulated problem:
bilevel optimization
arg min max EarPa (log D(=)] + Eg [log(1 D(G())))). 0g On
(I)
The discriminator and generator & aim to break each other through the inner max and outer min objectives, respectively. The studies by Goodfellow al. (2014); Radford et al. (2016) show that this adversarial formulation can lead to better generator that produces plausible data points/images.
However, GANs known to be hard to train due to the following issues (Goodfellow) 2016).
Failure to converge. In practice, Eq. ( is usually only approximately solved by an alternating first-order method such the alternating stochastic gradient descent (SGD). The alternating up- dates for On and Og may cancel each other's progress. During each alternating training step, it is also tricky balance the number of SGD updates for Op and that for Og, too small large number for Op leads to low-quality gradients for 0g. Mode collapse. The alternating SGD is at- tracted by stationary points and therefore is not good distinguishing between a ming, maNe, problem and a maXp, ming, problem. When the solution to the latter is returned, the generator tends to always produce the points at modes that best deceive the discriminator, making Pgmm of low
Our code is available GitHub at

---

Under review as a conference paper at ICLR 2023
diversity F Vanishing gradients. At the beginning of training process, the finite real and fake train- ing data may not overlap with each other in the data space, and thus the discriminator may be able to perfectly separate the real from fake data. Given the cross-entropy loss (or more generally, any f-divergence measure (Rényi et al., 1961) between Paata and P Pgen), the value of the discriminator becomes saturated both sides of the decision boundary, resulting in zero gradients for 0g.
In this paper, we argue that the above issues are rooted in the modeling of D. In most existing vari- ants of GANs, the discriminator is deep neural network with explicit weights Op. Under gradient descent, the gradients of 0g in Eq. ① cannot be back-propagater through the inner maxep problem because otherwise it requires the computation of high-order derivatives of 0p. This motivates the use of alternating SGD, which in turn causes the convergence issues and mode collapse. Further- more, the DD is a single network whose particularity may cause a catastrophic effect, such as the vanishing gradients, during training.
We instead model the discriminator D as Gaussian process whose mean and covariance are gov- erned by kernel function called the neural tangent kernel (NTK-GP) (Jacot et al. [018 [ee al.) 2019 Chizat et al., 2019). The 'D approximates an infinite ensemble of infinitely wide neural net- works in nonparametric manner and has no explicit weights. In particular, its training dynamics can be completely described by a closed-form formula. This allows us to simplify adversarial data synthesis into single-level optimization problem, which we call the generative adversarial NTK (GA-NTK). Moreover, since 'D is an infinite ensemble of networks, the particularity of a single ele- ment network does not drastically change the training process. This makes GA-NTK less prone to vanishing gradients and stabilizes training even when an f f-divergence measure between Paata and Pgen is used as the loss of D. The following summarizes our contributions:
We propose single-level optimization method, named GA-NTK, for adversarial data syn- thesis. It can be solved by ordinary gradient descent, avoiding the difficulties of bi-level optimization in GANs.
We prove the convergence of GA-NTK training under mild conditions. We also show that D being an infinite ensemble of networks can provide smooth gradients for G. which stabilizes GA-NTK training and helps fight vanishing gradients.
We propose some practical techniques to reduce the memory consumption of GA-NTK during training and improve the quality of images synthesized by GA-NTK.
We conduct extensive experiments real- worldw datasets study the advantages and limi- tations of GA-NTK. In particular, we find that GA-NTK has much lower sample complexity compared GANs, and the presence of generator is not necessary generate images under the adversarial setting.
Note that the goal of this paper is not to replace existing GANs nor advance the state-of-the-ar per- formance, but to show that adversarial data synthesis can be done via a single-level modeling. Our work has implications for future research. In particular, the low sample complexity makes GA-NTK suitable for applications, such as medical imaging, where data are personalized or not easily col- lectible. In addition, GA-NTK bridges the gap between kernel methods and adversarial data/image synthesis and thus enables future studies on the relationship between kernels and generated data.
2 RELATED WORK
2.1 GANS AND IMPROVEMENTS
Goodfellow et al. 2014) proposes GANs and gives a theoretical convergence guarantee in the func- tion space. However, in practice, one can only optimize the generator and discriminator in Eq. (T) in the parameter/weight space. Many techniques have been proposed to make the bilevel optimiza- tion easier. Failure to convergence. To solve this problem. studies devise new training algorithms for GANs (Nagarajan & Kolter| [2017] Daskalakis et al. 2018)) or more general minimax problems Thekumparampi et al. 2019 Mokhtari et al.) (020). But recent works by MMecchedrr et al. (2018); Farnia & Ozdaglar (2020) show that there may not be a Nash equilibrium solution in GANs. Mode
Mode collapse can he caused by other reasons, such as the structure of G. This paper only solves the problem due to alternating SGD.
2

---

Under review as a conference paper at ICLR 2023
collapse. Metz et al. (0017) alleviates this issue by back- propaagtggg the computation of 0g through the discriminators traindd with several steps to strengthen the ming, maxe, property. Other works mitigate mode collapse by diversifying the modes of D through regularization (Che et al. 2017 Mao
et al.
2019), modeling 'D as an ensemble of multiple neural networks (Durugkar et
Ghosh
al. 2018), using additional auxiliary networks(Srvvastava et al. 2017; Bang & Shim [021]
Li et al., (021). Vanishing gradients. Mao et al. (2017) tries to solve this problem by using the
Pearson xidiieegeence between Paann and Peen as the loss penalize data points that are far away
from the decision boundary. However, it still suffers from vanishing gradients as any f-divergence measure, including the cross-entropy loss and Pearson x² diveeggee,,, cannot measure the differ- ence between disjoint distributions (Sajjadi et al., 2018). Later studies replace the loss with either the Wasserstein distance (Arjovsky et al. 2017] Gulrajani et al. (017) or maximum mean discrep- ancy (Gretton et al. 2012; et al. 2015 (017) that can measure the divergence of disjoint Pama and Pgen In addition, the works by Miyato et al. (2018); Qi (2020) aim to constrain the Lipschitz continuity of the discriminator to prevent its value from being saturated.
Despite that many efforts have been made to improve the training of GANs, most existing ap- proaches address only one two issues at time with different assumptions, and in the meanwhile, they introduce new hyperparameters or side effects. For example, in the Wasserstein GANs (Ar-) jovsky et al.] 2017 Gulrajani et al., 2017) mentioned above, efficient computation of Wasserstein distance requires the discriminator to be Lipschitz continuous. However, realizing Lipschitz conti- nuity introduces new hyperparameters and could limit the expressiveness of the discriminator (Anil et al. (2019). Until now, training GANs is still not an easy task because one has to 1) tune many hy- perparameters and 2) strike a balance between the benefits and costs of different training techniques to generate satisfactory data points/images.
2.2 GAUSSIAN PROCESSES AND NEURAL TANGENT KERNELS
Consider an infinite ensemble of infinitely wide networks that use the mean square error (MSE) as the loss and are trained by gradient descent. Recent developments in deep learning theory show that the prediction of the ensemble can be approximated by special instance of Gaussian process called NTK-GP (Jacot et al., 2018 Lee al. 2019 Chizat al. 2019). The NTK-GP a Bayesian method, so it outputs distribution of possible values for an input point. The mean and covariance of the NTK-GP prediction are governed by kernel function ((::)) called the neural tangent kernel (NTK). Given two data points x' and x², the k(x', x²) represents the similarity score of the two points in kernel space, which is fixed once the hyperparameters of the initial weights, activation function, and architecture of the networks in the target ensemble are determined.
Here, we focus on the mean prediction of NTK-GP as is relevant to our study. Consider a super- vised learning task given D (X" E Raxd Y E R2xc) as the training set, where there are examples and each example consists of a pair of d-dimensional input and o-dimensional output. Let K E Rexn be the kernel matrix for X", i.e., K2 &(X,, X²)). Then, time step during gradient descent, the mean prediction of NTK-GP for X evolve
( - e " "K y E R
(2)
where I" E Rax= is an identity matrix and r) is sufficiently small learning rate (Jacot et al.
2018
Lee et al. 2019
The NTK used in Eq. can be extended to support different network architectures, including convolutional neural networks (CNNs) (Arora et al., 2019 Novak al.) 2019b). recurrent neural networks (RNNs) (Alemohammad et al. 2021 Yang 2019b). networks with the attention mecha- nism (Hron et al. (020), and other architectures Yang) 2019b Arora et al.] 2009)) Furthermore, studies (Novak et al. [011a; Lee et al. 2020 Arora et al.) 2020 Geifman et al. (020)) show that NTK-GPs perform similarly to their finite-width counterparts (neural networks) in many situations
and sometimes even better on small-data tasks.
A recent study by Franceschi et al. (2021) analyzes the behavior of GANs from the NTK perspective by taking into account the alternating optimization. It shows that, in theory, the discriminator can provide a well-defined gradient flow for the generator, which is opposite to previous theoretical interpretations (Arjovsky & Bottou 2017). Our work, on the other hand, focuses on adversarial data
3

---

Under review as a conference paper at ICLR 2023
synthesis without alternating optimization We make contributions in this direction by (1) formally proving the convergence of the proposed single-level optimization, (2) showing that a generator network is not necessary to generate plausible images (although it might be desirable), and (3) proposing the batch-wise and multi-resolutiinnl extensions that respectively improve the memory efficiency of training and global coherency of generated image patterns.
3 GA-NTK
We present a new adversarial data synthesis method, called the generative adversarial NTK (GA-
NTK), based on the NTK theory (Jacot et al. 2018 Lee et al. 2019 Chizat et al. 2019). For simplicity of presentation, we let G(z) = E R' and focus on the discriminator for now. We will discuss the case where G(-) is generator network in Section 3.2 Given an unlabeled, d-dimensional
dataset X" E Raxd of points, we first augment X" to obtain labeled training set D
(X" @ Z 2nxd 1" @ 0" E IR²",, where Z" E nxdd contains n generated points, 1 IR" and 0 E R are label vectors of ones and zeros, respectively, and @ is the vertical stack operator. Then, we model discriminator trained on D22m as an NTK-GP. Let K2nn.2n E R22xx2n be the kernel matrix for X @ Z" where the value of cach element K 222 be computed once we decide the initialization, activation function, and architecture of the element networks in the target infinite ensemble, i.e., the diseriminator. By Eq. (2) and let À n) the mean predictions of the discriminator can be written as
k((X"@ (e )4.: (X" *rz* )"),,))
));)) can
D(X". Z"; X) (I e 2 )11 @ 0*) E 2
(3)
where 2 E 222x2n is an identity matrix. We formulate the objective of GA-NTK as follows:
arg min C(Z") where C(Z") 2n
1 D(X" Z" A)lI-
((.) is the loss function and 1 2m the Pearson xi-divergence [Jeffreys 1946). ). a case of f-divergence, hetween Pasta + Pgen and 2P gen- where Pgon the distribution of generated points, Please see Sectionsoiin Appendix for more details.
E R 2m is vector of ones. Statistically, Eq.. (4 aims to minimize
GA-NTK formulates an adversarial data synthesis task as a single-level optimization problem. On one hand, GA-NTK aims find points 2" that best deceive the discriminator such that it outputs wrong labels 1 2 for these points. On the other hand, the discriminator is trained on 2 with the correct labels 1" @ 0" and therefore has the opposite goal of distinguishing between the real and generated points. Such an adversarial setting can be made single-level because the training dynamics of the discriminator
by gradient descent can be completely described by a closed-form formula in Eq. (3) any change of z" causes 'D to be "retrained" instantly. Therefore, one can easily solve Eq. by ordinary SGD.
Training. Before running SGD, one needs to tune the hyperparameter X. We show in the next section that the value of À should be large enough but finite. Therefore, the complete training process of GA-NTK is to 1) find the minimal À that allows the discriminator separate real data from pure noises in an auxiliary task, and 2) solve zn in Eq. by ordinary SGD with the fixed X. Please see Section 7.3 in Appendix for more details.
3.1 MERITS
As compared to GANs, GA-NTK offers the following advantages: Convergence. The GA-NTK can be trained by ordinary gradient descent. This gives much nicer convergence properties:
Theorem 3.1 Let s be the number of the gradient descent iterations solving Eq. (4) and let zm.(s)
be the solution at the s-th iteration. Suppose the following values are bounded: (a) X and Z,,(()) Vi, j. (b) I and n, and (c) and L. Also, assume that (d) X contains finite, non-identical, normal- ized rows. Then, for a sufficiently large I, we have
min V 2 L(0 (1) )1 < 00( 8 1 - )
From GAN perspective, our work can be regarded as special case of the framework proposed by Franceschi et al. 2021). where the discriminator neglects the effect of historical generator updates and only distinguish between the true and currently generated data each alternating step.
4

---

Under review as a conference paper at ICLR 2023
We prove the above theorem by showing that, with large enough X. Vz- Z(Z is smooth enough to lead to the convergence of gradient descent. For more details, please see Section 6 in Appendix. Diversity. GA-NTK avoids mode collapse due to the confusion between the min-max and max-min problems in alternating SGD. Given different initial values, the generated points in Z" can be very different from each other. No vanishing gradients,
side effects. The hyperparameter À controls how much D should learn from the true and fake data during each iteration. Figure 5 shows the gradients of D with finite A, which do not saturate. This avoids the necessity of using a loss that imposes side effects, such as the Wasserstein distance (Arjovsky et al. 2017 al. 2017) whose efficient evaluation requires Lipschitz continuity of D.
Gulrajani
3.2 GA-NTK IN PRACTICE
Scalability. To generate a large number of points, we can parallelly solve multiple z in Eq. on different machines. On a single machine, the gradients of z" need to be back-propagated through the computation of K22n,2n which has O(n²) space complexity. This may incur scalability issues for large datasets. Although recent efforts by Ayroro et al. (2019): Bietti Mairal (2019): Hann et al. 2021): Zandieh et al. (2021) have been made to reduce the time and space complexity of the evaluation of NTK and its variants, they are still at an early stage of development and the consumed space in practice may still be too large. To alleviate this problem, we propose the batch-win GA- NTK with the objective
arg min 2"
-1' 2b/2; A)]l,
(5)
that can be solved using mini-batches, during each gradient descent iteration, we 1) randomly sam- ple a batch rows in X @ z" and their corresponding labels, and 2) update 2 based on Kb. Although the base GA-NTK is cosmetically similar to the original GA-NTK, it solves dif- ferent problem. In the original GA-NTK, the Z" aims to fool a single discriminator D trained on 2n examples, while in the batch-wie wise GA-NTK, the Z goal is to deceive many discriminators, each trained on b examples only. Fortunately, Shankar et al. (2020) ; Arora et al. (2020) have shown that NTK-based methods perform well on small datasets. We will conduct experiments verify this
later.
Generator Network. So far, we let G(z) = and show that a generator is not necessary in ad- versarial data synthesis Nevertheless, the presence of a generator network may be favorable in some applications to save time and memory at inference time. This can be done by extending the batch-wise GA-NTK as follows:
arg min Exwaxx 2 | D(X 922 k:)
0g
where G(-: is generator network parametrized by gg. and Z E Ie where I S d. Note that this still a single-level objective, and 0g can be solved by gradient descent. We denote this variant GA-NTKg.
Image Quality. To generate images, one can pair up GA-NTK with a convolutional neural tangent kernel (CNTK) (Arora et al. 2019 Novak et al. 2019b; Garriga-Alon Alonso et al.) 2019 Yang. 2019a) that approximates a CNN with infinite channels. This allows the NTK-GP (discriminator) to distin- guish between real and fake points based on local patterns in the pixel space. However, the images synthesized by this GA-NTK variant may lack global coherency, just like the images generated by the CNN-based GANs (Radford et al. 2016 Salimans et al. 2016). Many efforts have been made to improve the image quality of CNN-based GANS, and this paper opens up opportunities for them to be adapted the kernel regime. In particular, we propose the
GA-CNTK based
on the work by Wang et al. (2018), whose objective is formulated as:
arg min 2
Σ11 D" (pool" (X"), "oo!"
where 2d is an NTK-GP taking input a particular pixel resolution and p.) is a downsample operation (average pooling) applied to each row of X and Z". The generated points in 2" aim to simultaneously fool multiple NTK-GP- -Pss (discriminators), each classifying real and fake images at distinct pixel resolution. The NTK-GPs working at low and high resolutions encourage global coherency and details, respectively, and together they lead to more plausible points in z
'In GANs, solving Z directly against finite-width discriminator is infeasible hecause it amounts finding adversarial examples (Goodfellow et al. 2015) whose gradients known be very noisy (Iyas et al. 2019
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
We conduct experiments to study how GA-NTK works in image generation.
Datasets. We consider the unsppervised/uncoodttoonll image synthesis tasks over real-world datasets, including MNIST (LeCun et al. 2010), CIFAR-10 10 (Krizhevsky 2009), CelebA (Liu et al.
2013). CelebA-HQ (Liu et al. 2013), and ImageNet (Deng al.) 2009). To improve training efni ciency, we resize CelebA images to 64 x 64 and ImageNet images to 128x 128 pixels, respectively. We also create 2D toy dataset consisting of 25-modal Gaussian mixtures of points visualize the behavior of different image synthesis methods. GA-NTK implementations. GA-NTK works with different NTK-GPs. For the image synthesis tasks, we consider the NTK-GPs that model the ensembles of fully-connected networks (Jacot et al., 2018: Lee et al., (10]1209 [Ciizt et al., 2019) and convolutional networks (Arora et al., [2199] NNvak et al. [019b] Garriga-Alonss et al. 2019 Yang, 20T9a), respectively. We implement GA-NTK using the Neural Tangents library (Novak) et al., 2019a) and call the variants based on the former and latter NTK-GPs the GA-FNTK and GA-CNTK, respectively. In GA-FNTK, an element network of the discriminator has 3 infinitely wide, fully-connected layers with ReLU non-linearity, while in GA-CNTK, an element network follows the architecture of InfoGAN (Chen et al.] 2016) except for having infinite filters at each layer. We tune the hyperparameters of GA-FNTK and GA-CNTK following the method proposed in Poole et al. (SchoenScco et (201);; |Rahu al. 2017 We also implement their batch- wise, generator, and multi-resolutional variants described in Section 3.2 See Section 7 in Appendix for more details. Baselines. We compare GA-NTK with some popular variants of GANs, includ- ing vanilla GANs (Goodfellow et al. 2014), DCGAN (Radford et al.) 2016). LSGAN (Mao et al., 2017.. WGAN (Arjovsky et al. 2017] WGAN-GI (Gulrajani et al. 2017). SNGAN (Miyato al., 2018 and StyleGAN2 (Karras et al. 2020). To give a fair comparison, we let the discriminator of each baseline follow the architecture of InfoGAN (Chen et al. 2016) and tune the hyperparameters using grid search. Metrics. We evaluate the quality of a set of generated images using the Fréchet Inception Distance (FID) (Heusel et al., 2017). The lower the FID score the better. We find that an image synthesis method may produce downgrade images that look almost identical to some images in the training set. Therefore, we also use a metric called the average max-SSIM (AM-SSIM) that calculates the average of the maximum SSIM score (Wang et al. 2004) between Pgen and Paaaa
(Schoenholz
AM-SSIM gen Paatn) Brg' maax SSIM x)]]
A generated image set will have a higher AM-SSIM score if it contains downgrade images. Envi- ronment and limitations. We conduct all experiments on cluster of machines having 80 NVIDIA Tesla V100 GPUs As discussed in GA-NTK consumes a significant amount of memory on each machine due to the computations invol ved in the kernel matrix K2,,2n, With the current version of Neural Tangents library (Novak et al., 2019a) and a V100 GPU of 32GB RAM, the maximum sizes of the training set from MNIST, CIFAR-IO, CelebA, and ImageNet are 1024, 512, 256, and 128, respectively (where the computation graph and backprop operations of K 2n consume about 27.5 GB RAM excluding other necessary operations). Since our goal is not to achieve ofth-offthhe-at performance but to compare different image synthesis methods, we train all the methods using up to 256 images randomly sampled from all classes of MNIST, the "horse" class CIFAR-10, 10, the "male with straight hair" class of CelebA, and the "daisy" class of ImageNet, respectively. We will conduct larger-scale experiments in Section 9.2 For more details about our experiment settings, please Section 7 in Appendix.
4.1 IMAGE QUALITY
We first study the quality of the images synthesized by different methods. Table [ summarizes the FID and AM-SSIM scores of the generated images. LSGAN and DCGAN using f-divergence as the loss function give high FID and fail to generate recognizable images on CIFAR-10 and CelebA datasets due to the various training issues mentioned previously. StyleGAN, although being able to generate impressive images with sufficient training data, gives high FID here due to the high sample complexity of the style-based generator. Other baselines, including WGAN, WGAN-GP and SN-GAN, can successfully generate recognizable images on all datasets, as shown in Figure I In particular, WGAN-GP performs the best among the GAN variants. However, WGAN-GP limits the Lipschitz continuity of the discriminator and gives higher FID scores than GA-CNTK. Also, gives higher AM-SSIM values as the size of the training set decreases, implying there are many

---

Under review as a conference paper at ICLR 2023
Table 1: The FID and AM-SSIM scores of the images generated by different methods.
WGANGP
StyleGAN
GACNTK
GACNTKg
MNIST
AMSSIM
31.89
38.52
AMSSIM
0.85
0.80
0.74
0.76
0.67
0.66
0.52
0.72
FID
69.76
35.33
50.33
24.37
29.49
84.7
14.96
51.21
AMSSIM
0.69
0.78
0.72
0.73
0.70
0.65
0.54
0.73
FID
312.21
258.41
117.85
49.29
118.16
406.02
55.54
106.44
CIFAR-IA 10
AMSSIM
0.22
0.25
0.29
0.74
0.28
0.64
0.41
0.44
229.94
339.27
101.90
68.53
128.65
484.36
39.98
61.19
128
AMSSIM
0.36
0.10
0.26
0.60
0.21
0.39
0.41
0.44
181.15
255.19
85.34
426.58
256
489.82
83.71
122.36
83.71
169.04
323.37
30.83
95.91
ColebA
AMSSIM
0.02
0.05
0.29
0.56
0.29
0.23
0.60
0.21
FID
55.01
450.81
125.82
92.73
168.11
337.58
33.51
$8.39
AMSSIM
0.03
0.11
0.28
0.54
0.28
0.21
0.51
0.38
FID
461.95
403.79
108.07
79.36
161.20
333.16
63.15
78.46
256
AMSSIM
0.04
0.09
0.31
0.39
0.27
0.30
0.38
0.40
S
2
WGAN
(a)
8
(b) WGANGP
1
2
4
/
2
3
/
3
6
8
8
0
3
(C) SNGAN
6
0
3
1
(d) GACNTK
0
3
4
7
?
7
0
4
8
9
GACNTKg
7
9
/
0
8
(C)
6
3
3
3
Figure 1: The images generated by different methods MNIST, CIFAR-10. 10, and CelebA datasets given only 256 training images.
7

---

Under review as a conference paper at ICLR 2023
(a) GA-CNTK
(b) GA-CNTKg
Figure 2: The images generated by GA-CNTK (a) without and (b) with a generator given 256 CelebA-HQ training images.
928
0.9-
0.76
0.70
20
1008 120
3440
ter attions
Rerations
(a) MNIST
(b) CIFAR-10
CelebA
Figure 3: The learning curve and image quality at different stages of training process.
downgrade images that look identical to some training images. This is because the Wasserstein distance, which also called the earth mover's distance, allows fewer ways of moving when there are less available "earth" (i.e., the density values of Pdata and Pgen) due to a small and thus the Pgan needs to be exactly the same as Pdata to minimize the distance The GA-NTK variants, including GA-CNTK and GA-CNTKg ("gg" means "with generator"), pertorm relatively well due to their lower sample complexity, which aligns with the previous observations (Shankar et al.) 2020
Arora et 2020) in different context.
Next, we compare the images generated by the multi-resolutional GA-CNTK and GA-CNTKg (see Section B.2h on the CelebA-HQ dataset. The multi-resolutional GA-CNTK employs 3 discrimina- tors working at 256x256, 64x64, and 16x 16 pixel resolutions, respectively. Figure 2 shows the results. We can see that the multi-resolutional GA-CNTK (without a generator) gives better-looking images than GA-CNTKg (with generator) because learning generator, which maps two spaces, essentially harder problem than finding set of plausible z's. Although synthesizing data faster at inference time, generator may not be necessary to generate high-quality images under the ad- versarial setting.
4.2 TRAINING STABILITY
Convergence. Figure 3 shows the learning curve and the relationship between the image quality and the number of gradient descent iterations during a training process of GA-CNTK. We find that
The problem Lipschitz continuity may alleviated when becomes larger.
(a) Truth (b) Vanilla GAN
(c) LSGAN (d) WGAN (c) WGAN-GP (f) SN-GAN (g) GA-FNTK
Figure 4: Visualization distribution alignment and mode collapse on a 2D toy dataset.
8

---

Under review as a conference paper at ICLR 2023
GA-CNTK easily converges under various conditions, which is supported by Theorem 3.1 Further- more, we can see a correlation between the image quality and the loss value as the loss becomes smaller, the quality of the synthesized images improves. This correlation can save human labor from monitoring the training processes, which is common when training GANs. Note that the im- ages generated the latter stage of training contain recognizable patterns that change over training time. This is a major source of GA-CNTK creativity. Please see Section 9.4 for more discussions. Mode collapse. To study how different methods align Pgen with Paaaa. we train them using 2D toy training set where Pauta is a 25- modal Gaussian mixture. We use two 3-layer fulne-connected neu-
norm of gradients
1.0 10-3
ral networks as the generator and discriminator for each baseline and an ensemble of 3-layer, infinitely wide counterpart as the dis- criminator in GA-FNTK. For GANs, stop the alternating SGD training when the generator receives 1000 updates, and for GA- FNTK, terminate the GD training after 1000 iterations. Figure shows the resultant Pgen of different methods. GA-ENTK avoids mode collapse due to the use of alternating SGD. Gradient van- ishing, To verify that GA-NTK gives no vanishing gradients with finite X, conduct an experiment using another toy dataset con- sisting of 256 MNIST images and 256 random noises. We replace the discriminator of GA-CNTK with a single parametric network of the same architecture but finite width. We train the finite-width wiith network on the toy dataset by minimizing the MSE loss using gradient descent. We set the training iteration to large value (65536) to simulate the situation where the network value becomes satu- rated on both sides of the decision boundary. Figure 5 compares Comparess the gradients of a generated image 4 in Eq. 4 obtained from I) the finite-width network and 2) the corresponding GA-CNTK with large t. As Z evolves through gradient descent iterations, the norm of its gradients obtained from the finite-wide discriminator quickly shrinks to zero. On the other hand, the gradient norm obtained from the discriminator of GA-CNTK is always positive thanks to the infinite ensembling.
CNN
0.0
2500
5000 7500 10000 iterations
Figure 5: Comparison be- tween the gradients of Z,.: in Eq. (4) obtained from dif- ferent types of D.
4.3 SCALABILITY
Unlike GA-CNTK, the GA-CNTKg is batch-wise and thus can be trained by more examples. Here, we scale up WGAN-GP and GA-CNTKg by training them on CelebA dataset consisting of 2048 images. The batch size 256. T the FID and AM-SSIM scores of the generated images. On MNIST, WGAN-GP slightly out- performs GA-CNTKg. The training of WGAN-GP on MNIST easy, so GA-CNTKg does not offer much ad- vantage. However, in more complex task like CIFAR- 10 or CelebA, GA-CNTKg outperforms WGAN-GP, sug- gesting that our single-level modeling is indeed benefi- cial.
Table 2: The FID and AM-SSIM scores of the images output by WGAN-GP and GA GA-CNTKg trained on 2048 CelebA images with batch size 256.
2024448
Metric
WGANGP
GACNTKe
FID
23.47
56.73
MNIST
ASSIM
0.786
0.787
FID
110.70
78.85
CIFAR-10
ASSIM
0.404
0.432
FID
67.29
59.91
We have conducted more experiments. Please see Ap- pendix for their results.
CelehA
ASSIM
0.337
0.411
CONCLUSION
We proposed GA-NTK and showed that adversarial data synthesis can be done via single-level modeling. can be solved by ordinary gradient descent, avoiding the difficulties of livel training of GANS. We analyzed the convergence behavior of GA-NTK and gave sufficient conditions for convergence. Extensive experiments were conducted to study the advantages and limitations of GA- NTK. We proposed the batch-wise and multi-resolutiinal variants to improve memory efficiency and image quality, and showed that GA-NTK works either with or without a generator network. GA-NTK works well with small data, making it suitable for applications where data are hard to collect. GA-NTK also opens up opportunities for one to adapt various GAN enhancements into the kernel regime. These matters of our future inquiry.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard G. Baraniuk. The recurrent neural tangent kernel. In Proc. of ICLR, 2021.
Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In Proc. of ICML, 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In Proc. of ICLR, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proc. of ICMI,, 2017.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Proc. of NeurlPS, 2019.
Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets on small-data tasks. In Proc. ICLR, 2020.
Duhyeon Bang and Hyunjung Shim. Mggan: Solving mode collapse using manifold-guided training. In Proc. of CVPR, 2021.
Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and Carsten Steger. Improving supervised defect segmentation by applying structural similarity to autoencoders. In VISIGRAPP, 2019.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Proc, of NeurlPS, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. In Proc. of ICLR, 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proc. of NeurlPS, 2016.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming.
In Proc. of NeurIPS, 2019.
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. In Proc. of ICLR, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In Proc. of CVPR, 2009.
Ishan P. Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks. In Proc. of ICLR, 2017.
Farzan Farnia and Asuman E. Ozdaglar. Do gans always have nash equilibria? In Proc. of ICML. 2020.
Lean-Yves Franceschi, Emmanuel de Bézenac, Ibrahim Ayed, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. A neural tangent kernel perspective of gans. CoRR, abs/2106.05566 2021.
Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional net- works shallow gaussian processes. In Proc. of ICLR, 2019.
Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. On the similarity between the laplace and neural tangent kernels. In Proc. of NeurlPS, 2020.
Arnab Ghosh, Viveka Kulharia, Vinay P. Namboodiri, Philip H. S. Torr, and Puneet Kumar Dokania. Multi-agent diverse generative adversarial networks. In Proc. of CVPR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Ian Goodfellow. Nips 2016 tutorial:
Generative adversarial networks.
arXiv preprint
arXiv:1701.00160, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. of NeurlPS, 2014.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Proc. of ICLR, 2015.
Robert M Gower.
Convergence theorems for gradient descent, May 2022.
https:
//gowerrobert.githie wtithub.io//eo.. io/p /Mt_//M2tttistsee que_ossttuuueeipueeeeeoo_e sat on grad
conv. pdf
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. A kernel two-sample test. J. Mach. Learn. Res., 2012.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Im- proved training of wasserstein gans. In Proc. of NeurlPS, 2017.
Insu Han, Haim Avron, Neta Shoham, Chaewon Kim, and Jinwoo Shin. Random features for the neural tangent kernel. CoRR, abs/2104.01351, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Proc. of NeurlPS, 2017.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: NNGP and NTK for deep attention networks. In Proc. of ICML, 2020.
Andrew lyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Proc. of NeurIPS, 2019.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Proc. NeurlPS, 2018.
Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proc. of the Royal Society of Londonn. Series A. Mathematical and Physical Sciences, 186(1007)44334611, 1946.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proc. of CVPR, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. 2010.
Available: http://yann.lecum.coom wtt..//www....ee///etiiiiie
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo- realistic single image ruper-resolution using a generative adversarial network. In Proc. of CVPR, 2017.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In Proc. of ICLR, 2018
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. In Proc. of NeurlPS, 2019.
Jachoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstedi Finite versus infinite neural networks: empirical study, In Proc. of NewrlPS, 2020.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. MMD GAN: towards deeper understanding of moment matching network. In Proc. of NeurlPS, 2017.
11

---

Under review as a conference paper at ICLR 2023
Wei Li, Li Fan, Zhenyu Wang, Chao Ma, and Xiaohui Cui. Tackling mode collapse in multi- generator gans with orthogonal vectors. Pattern Recognition, 2021.
Yujia Li, Kevin Swersky, and Richard S. Zemel. Generative moment matching networks. In Proc. of ICML, 2015.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. of ICCV, December 2015.
Mario Lucic, Karol Kurach, Marein Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. In Proc. of NewrlPS, 2018.
Qi Mao, Hsin-Ying Lee, Hung-Yi Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. In Proc. of CVPR, 2019.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proc. of ICCV, 2017.
Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In Proc. of ICLR, 2018.
Lars M. Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In Proc. of ICML, 2018.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In Proc. of ICLR, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proc. of ICLR, 2018.
Aryan Mokhtari, Asuman E. Ozdaglar, and Sarath Pattathil. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In Proc. of AISTAYS, 2020.
Vaishnavh Nagarajan and Zico Kolter. Gradient descent GAN optimization is locally stable. In Proc. of NewrIPS, 2017.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A Alemi, Jascha Sohl-Dickstein and Samuel Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In Proc. of ICLR, 2019a.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abo- lafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In Proc. of ICLR, 2019b.
Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- Ben Poole.
tial expressivity in deep neural networks through transient chaos. In Proc. of NeurlPS, 2016.
Guo-Jun Qi. Loss-sensitive generative adversarial networks on lipschitz densities. Int. J. Comput. Vis., 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proc. of ICLR, 2016.
Maithra Raghu, Ben Poole, Jon M. Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In Proc. of ICML, 2017.
the 4th Berkeley symposium Alfréd Rényi et al. On measures of entropy and information. In Proc. the mathematical statistics and probability, volume I, 1961.
Mchdi S. M. Sajjadi, Giambattista Parascandolo, Arash Mchrjou, and Bernhard Schölkopf. Tem- pered adversarial networks. In Proc. of ICML, 2018.
Tim Salimans, Ian J. Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proc. of NewrlPS, 2016.
12

---

Under review as a conference paper at ICLR 2023
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In Proc. of ICLR, 2017.
Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich- Keil, Jonathan Ragan-Kelley, Ludwig Schmidt, and Benjamin Recht. Neural kernels without tangents. In Proc. of ICML. 2020.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. VEE- GAN: reducing mode collapse gans using implicit variational learning. In Proc. of NeurIPS, 2017.
Kiran Koshy Thekumparampil, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh. Efficient algo- rithms for smooth minimax optimization. In Proc. of NeurIPS, 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Proc. of NeurIPS, 2016.
Ting- Baug--hhn Wang, Ming- Yu Liu, Lun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 8798-8807, 2018.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 2004.
Li Xu, Jimmy S. Ren, Ce Liu, and Jiaya Jia. Deep convolutional neural network for image deconvolution. In Proc. of NeurlPS, 2014.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. CoRR, abs/1902.04760, 2019a.
Greg Yang. Tensor programs wide feedforward recurrent neural networks any architecture are gaussian processes. CoRR, abs/1910. 12478, 2019b.
Amir Zandieh, Insu Han, Haim Avron, Neta Shoham, Chaewon Kim, and Jinwoo Shin. Scaling neural tangent kernels via sketching and random features. In Proc. NeurIPS, 2021.
STATISTICAL INTERPRETATION OF GA-NTK
Statistically. minimizing Eq. (4) or (6) amounts to minimizing the Pearson x2 -iveergeee (Jeffreys 1946), a case of f-divergence (Rényi et al. 1961). between Paata + +Pgen and 2P gen, where P is the distribution of real data and P gen the distribution of generated points. To this, first rewrite the loss of our discrimonator D, denoted by L(D), in expectation:
arg min L(D) arg min E En [(D(x) 1)²] EmPP [((x))
0)
(8)
Here, Pgen can represent either Z in Eq. (4p or the output of the generator G in Eq. 66 Similarly,
the loss function for our Pgem denoted by I[P gen; D), can be written follows:
arg min C(P gen; D) = arg min En [(((x) - 1>²] + En~ ac. [D( Pazs
1)
Pan
GA-NTK, in the form of Eqs. ( and (9. is special case of LSGAN (ao et al.
2017 Let D be
the minimizer of Eq. (8). We can see that Eqs. (4) and (6) effectively solve the problem:
arg min L(P gen' Dr) arg min Ea.
[DD (x) 1)²] + En- (D) 1)²].
(10)
Pice
Pasi
Mao et al. (017)) show that, under mild relaxation, minimizing Eq. TT)) yields minimizing the Pearson x -iverrgence between Paita + Pgen and 2P 2Ppcn
arg min L(Pgeni D*) = arg min X (Paata + Pgen 12Pgen)
Pys
Ppn
/ (Panta((r) + Ron (x))
2P gon
arg min Pam
+ Pgen (z) 1) da.
Panta (x))
The loss becomes zero when Panna( (2) = P Pgen(x) for all æ. Therefore, minimizing Eq. or brings gen closer Paua-
13

---

Under review as a conference paper at ICLR 2023
7 PROOF OF THEOREM 3.1
In this section, we prove the convergence of GA-NTK whose discriminator 'D approximates an infinite ensemble of infinitelywwide, fully-connected, feedforward neural networks. The proof can be easily extended to other network architectures such as convolutional neural networks.
7.1 BACKGROUND AND NOTATION
Consider fully-connected, feedforward neural network / Rd IR.
Ow w/ ow
dL-33 W 1y Wh 1 (...)) Tw Vd
W' x + tobb) ...) + torbi-1) + oab",
f( 0) Vdl-I
(11)
where ((-) is the activation function (applied element-wisely), L the number of hidden lay- ers, {dl .... ,dl-1} are the dimensions (widths) of hidden layers, O - 0' - (W' Rdd xd= b' E I²d') are trainable weights and biases whose initial values are i... Gaussian random variables (O, 1), and o and of are scaling factors that control the variances of weights and biases, respectively. Suppose f is trained by minimizing the MSE loss using tesceett-descent iterations with the learning rate 17. Let ( and g(()) be the initial and trained parameters, respectively. As d'..... d' 0o, we can approximate
labeled dataset D (X" + Z RRaxxd 1" 0 ?
the distribution of fæ; (())) as Gaussian process (NTK-GP) (Jacot et al. 2018 Lee et al. 2019
Chizat et al. 2019) whose behavior is controlled by kernel matrix
K 2n.2n Vof(X" Z"; (()) T @ z"; 0() E 2nx2n
(12)
where f(X" @ z"; ((0) E 2 is the vector of in-sample predictions made by the initial f. The value of each element K 2n = ((X" $ ).:: (X" @ )".).) presents the similarity score of two rows (points) of X + Z" in a kernel space, and it can be expressed by a kernel function kL Rdd T + R, called the neural tangent kernel (NTK). The NTK is deterministic as it depends only (b)). dw оb, and L rather than the specific values in (("), Furthermore, it can be evaluated liyer- Let h4 (ax) Rd be the pre-activation of the j-th neuron at the I-th layer of fx; (()). The distribution of ht (zx) is still an NTK-GP, and its associated NTK is defined as k Rd x Rd - R,
k( V vesth (x)
V Vosth/('';
where 0 = = 0² Note that all h2 (2)'s, Vj. are i.i.d. and thus share the same kernel. It can be shown that
(z) (V)(=)]V[lll (x)) + h)) (/)=-1)((z"))
= xxx + oigh 1
(x, x x') (n (ng- (1 Ri-1)
(t-)) (x'))
(13)
and
02 x
1s (z,
x'' o%
(14)
R is the NNGP kernel Lee et al. controls the behavior of another Gaussian process, called NNGP. approximating the distribution of f(x; (()), and
where 70' Rd x .dd
2018
Matthews et al. 2018 that
1 w) 7--1 (x,x') x!)
1 x') 1-'1 x²)
K 1-1
E R
7.2 CONVERGENCE
The GA-NTK employs the above NTK-GP as the discriminator D. So, the in-sample mean predic- tions of D can be written closed-form formula:
D(X" z²) 2I 2m.2
2 E R2n
(15)
14

---

Under review as a conference paper at ICLR 2023
where 2 is identity matrix and y 1" 0" R is the "correct" label vector for training
D. We formulate the objective of GA-NTK as:
arg min C(Z") arg min 2 | 4
D(X", 2²)
(16)
2 2
where 1 2n E R2n in the loss L(-) is the "wrong" label vector that guides find the points (Z")
that best deceive the discriminator. We show that
Theorem 7.1 Let s be the number of the gradient descent iterations solving Eq. (6). and let Zm.(s)
be the solution at the s-th iteration. Suppose the following values are bounded: (a) X and Z vi, (b) t and T, and (c) a and L. Also, assume that (d) X" contains finite, non-identical, normal- ized rows. Then, for a sufficiently large we have
min V Vz ((" 50 1S
7.3 PROOF
To prove Theorem 7.1
we first introduce the notion of B smoothness:
Definition 7.1 A continuously differentiable function g R2 + IR is 8-smooth if there exits B E R such that
[[ag(()) Vbg(b)] s $.31 Blaa bll
for any a. Rd
be shown that gradient descent finds stationary point of f-smooth function efficiently Gower 2022).
Lemma 7.1 Let al)) be the input of a function Rdd - TK after applying 8 gradient descent iterations to initial input a (0) Ifg is 3-smooth, then g(al))) converges stationary point at rate
min 3
8 [
00
So, our goal is to show that the loss L(Z") in Eq. 16 is 3-smooth w.r.t. any generated point Rd
Corollary 7.1 all the conditions (a)-(d) in Theorem 7.1 hold, there exits constant ei RR+ such that V L((")) S e for each row z E Rdd of z". This makes L(Z") 3-smooth.
To prove Corollary 7.1 consider ( zn) and V [(2"), the i-th and j-th elements of D(X", Zd) E R2n and V ) E Rd respectively. We have
V ((Z")
V v 1 D(X", 1") Σ 2n
(17)
i=1 (Di 2 -)) D,(" Z")
Given sufficiently large t. the ( Z can be arbitrarily close to E {0, 1} because K 2n,2n is positive definite (Jacot et al. 2018) and therefore (I2n 2I2n-e-1 e 15 There exists e E IR such that
I2n as t co in Eq.
V., ((z")
Ei X 2") +1 D 2n
Etent1 V., ln)l
< (1+ 2 -i
2n
V (X )
(1+c) E 2n
V プp= (I?2n
2n
2n.2n.22 }y?
c1+ Exp.g= ci,
nt
aniin
((X" Z
Jm. (X" Z") )p.")"""
KtKKkmm 2n
E IR+ can be arbitrarily close to 0 with sufficiently large t. Hence, Corollary 7.1 holds long V. ((X" @ "d) g.:: (X" n Z )p:) is bounded.
Note that 6,
15

---

Under review as a conference paper at ICLR 2023
Corollary 7.2 If the conditions (a)-(d) in Theorem 7.1 hold, there exits constant 0o R such that V k2 (a, b) I for any two rows a and of X Z
It clear that V 1,,""(a, b) 0f a, + z. So, without loss of generality, we consider Vz,k"(a, 2) only. From Eq. 13 we have
kk (a, z)
0k 2) 0k 0k--1 (a,
(a,
Ōzj
0h 2) Oh. 1- (a,
Øzj
For each 1 2, :.. L. we can bound 0k' x) 2)/0kt-\(aa by
0k' 0k'(a. (a.z) 0k-1(aa (a,z)
o (n (a) (f (2))) K 1) (=))
(o max; ø' ((h))²
provided that the maximum slope of @ is limited, which is true for many popular activation functions including ReLU and erf. Also, by Eq. (14), the value
Ok (a, 2) 0zj
d aj
bounded. Therefore, Corollary 7.2 holds, which in turn makes L(Z") 3-smooth via Corollary By Lemma 7.I we obtain the proof of Theorem 7.1
EXPERIMENT SETTINGS
This section provides more details about the settings of our experiments.
8.1 MODEL SETTINGS
The network architectures of the baseline GANs used in our experiments are based on InfoGAN (Chen al. 2016). We set the latent dimensions, training iterations, and batch size according to the study (Lucic et al. 2018). The latent dimensions for the generator are all 64. The batch size for all baselines is and CIFAR-10 10 datasets, respectively. For the optimizers, we follow the setting from the respective original papers. Below we list the network architecture the baselines for each dataset as well as the optimizer settings.
64. The training iterations are 80K, 100K, and 400K for MNIST, CelebA,
Table 3: The architectures of the discriminator and generator in the baseline GANs for the MNIST dataset.
Discriminator
Generator
Input 28 2 Gray image
Inpute R64
N(0, I)
4x4 conv; 64 leaky ReLU; stride 2
Fully Connected 1024 ReLU; batchnorm
4x4 conv; 128 leaky ReLU; stride 2. batchnorm
Fully Connected 7 7 x 128 ReLU; batchnorm
Fully Connected 1024 leaky ReLU; batchnorm
4x4 deconv; 64 ReLU. stride 2; batchnorm
Fully Connected output
4 4 deconv; sigmoid
16

---

Under review as a conference paper at ICLR 2023
Table 4: The architectures of the discriminator and generator in the baseline GANs for the CIFAR-10
dataset.
discriminator
generator
Input 32x32x3 Image
Inpute R64 I)
4x4 conv; 64 leaky ReLU; stride 2
Fully Connected x 2 x 448 ReLU; batchnorm
4x4 conv; 128 leaky ReLU; stride 2; batchnorm
4 4 deconv; 256 ReLU; stride 2; batchnorm
4x4 conv; 256 leaky ReLU; stride 2; batchnorm
4x4 deconv; 128 ReLU; stride 2
Fully Connected output
4x4 deconv; ReLU; stride 2
4x deconv; Tanh; stride
Table 5: The architectures of the discriminator and generator in the baseline GANs for the CelebA dataset.
discriminator
generator
Input 64 Image
Inpute R NO I)
4x4 conv; 64 leaky ReLU; stride 2
Fully Connected x 2 x 448 ReLU; batchnorm
4x4 conv; 128 leaky ReLU; stride 2; batchnorm
4 4x4 deconv; 256 ReLU; stride 2; batchnorm
4x4 conv; 256 leaky ReLU; stride 2; batchnorm
4x4 deconv; 128 ReLU; stride 2
4x4 conv; 256 leaky ReLU; stride 2; batchnorm
4x4 decony; 64 ReLU; stride 2
Fully Connected output
4x4 deconv; 32 ReLU; stride 2
4 deconv; Tanh; stride
Table 6: The optimizer settings for each GAN baseline. Rddis denotes the training steps for discrimi- nators in the alternative training process.
Optimizer type
Learning Rate
p1
32
Rdis
DCGAN
Adam
0.0002
0.5
0,999
LSGAN
Adam
0.0002
0.5
0.999
WGAN
RMSProp
0.00005
None
None
WGAN-GP
Adam
0.0001
0.5
0.9
SN-GAN
Adam
0.0001
0.9
0.999
Note that we remove all the batchnorm layers for the discriminators in WGAN-GP, We architect the element network of the discriminator in our GA-NTK following InfoGAN (Chen et al. 2016). except that the width (or the number of filters) of the network is infinite at each layer and has no batchnorm layers.
The generator of GA-NCTKg consumes memory. To reduce memory consumption, we let D dis- criminates true and fake images the code space of pre-trained autoencoder A (Bergmann al., 2019). After training, code output by & is fed into the decoder of A to obtain an image. The architectures of the pre-trained A for different datasets are summarized as follows:
17

---

Under review as a conference paper at ICLR 2023
Table 7: The architectures of A for different datasets.
MNIST
CIFAR-10
Input 2 Image
Input x2x32x3 Image
3x3 conv: 16 ScLU; stride
3x3 conv; 32 ScLU; stride
3x3 conv; 32 SeL.U; stride
3x3 conv; 64 SeLUU stride
3 3x3 conv; 64 SeLU; stride
3x3 conv; 128 SeLU; stride
Fully Connected; 128 tanh
Fully Connected; 1024 tanh
3x3 transposeconv; 64 SeLU; stride 2
3x3 transposeconv; 128 SeLU; stride
3x3 transposeconv; 32 SeLU; stride 2
3x3 transposeconv; 64 SeLU; stride 2
3x3 transposeconv; 16 SeLU; stride 2
3x3 transposeconv; 32 SeLU; stride 2
output
output
CelebA
CelebA-HQ
Input 64x64x3 Image
Input 256x256x3 Image
3x33 conv; 32 SeLU; stride 1)
3x(3x3 conv; 64 SeLU; stride I)
3x3 conv; 32 SeLU; stride 2
3x3 conv; 64 SeLU; stride 2
x33 conv; 64 SeLU; stride
(3x3 conv; 128 SeLU; stride 1)
3x3 conv; 64 SeLU; stride
3x3 conv; 128 SeLU; stride 2
3x conv; 128 SeLU, stride 1)
3x(3x3 conv; 256 SeLU; stride 1)
3x3 conv; 128 ScLU; stride 2
3x3 conv; 256 ScLU; stride
Fully Connected; 2048 tanh
3xx3 conv; 512 SeLU; stride 1)
3 3xx3 transposeconv; 128 SeL.U; stride 1)
3x3 conv; 512 SeLU; stride 2
3x3 transposeconv; 128 SeLU; stride 2
Fully Connected; 2048 tanh
333 transposeconv; 64 SeLU; stride
3x33x3 transposeconv; 512 ScLU; stride
3x3 transposeconv; 64 SeLU; stride
3x3 transposeconv; 512 SeLU; stride
3x x3 transposeconv; 32 SeLU; stride 1)
3x x3 transposeconv; 256 SeLU; stride I)
3x3 transposeconv; 32 SeLU; stride 2
3x3 transposeconv; 256 SeLU; stride
output
3x33x3 transposeconv; 128 SeLU; stride 1)
3x3 transposeconv; 128 SeLU; stride 2
3x3(3x3 transposeconv; 64 SeL.U; stride 1)
3x3 transposeconv; SeLU; stride 2
output
8.2 METRICS
The FID scores computed using the code from the original paper (Heusel et al. 2017). We sample 2048 images to compute the FID scores. We calculate the AM-SSIM scores using the SSIM settings: filter size 4, filter sigma 1.5, kn 0.01, and l2 0.03 (Wang al. 2004)
18

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Unidirectional search for the hyperparameter À of GA-NTK.
Input: Data X", kernel k, and separation tolerance
Output: À for GA-NTK
Randomly initiate z" E Rn
while D(X", 2" k, A) (1" + 00 v do
A A-2
end return A
8.3 HYPERPARAMETER TUNING
For each data synthesis method, we tune its hyperparameter using grid search. GA-NTK. The computation of K 2n. requires one to determine the initialization and architecture of the element networks in the ensemble discriminator. |Poole et al. (2016); |Schoenholz et al.] (2017); Raghu et al. (2017) have proposed a principled method to tune the hyperparameters for the initialization. From empirical results, we also find that the quality of the images generated by GA-NTK is not significantly impacted by the choice of the architecture-4 a fully connected network with rectified linear unit (RcLU) activation suffices to generate recognizable image patterns. Once K2n,2n is decided, there is only one hyperparameter À = nyt to tune in Eq. (16). The X controls how well the discriminator is trained on ID, either too small large value can lead to poor gradients for Z" and final generated points. But since there is no alternating updates as in GANs, we can decide an appropriate value of À without worrying about canceling the learning progress of Z". We propose a simple, unidirectional search algorithm for tuning A. as shown in Algorithm| [ Basically, we search, from small to large, for a value that makes the discriminator nearly separate the real data from pure noises in an auxiliary learning task, and then use this value to solve Eq. (16). In practice, a small positive e ranging from 10-3 to 10-2 suffices to give an appropriate X. Multi-resolutional GA-NTK. We use 3 NTK-GP's as the discriminators, whose architectures are listed in Table 8
Table 8: The architectures of the discriminators for multi-resolution GA-NTK.
Discriminator large
Discriminator medium
Input 256x256xX Image
Discriminator small
Input 64 x 64 Image
4x4 conv; ReLU; stride 2
Input 16x 16x3 Image
4x4 conv; ReLU; stride
4 4x4 conv; ReLU; stride 2
4x cony; ReLU; stride 2
4 4x conv; ReLU; stride 2
4x4 conv; ReLU; stride 2
4x4 conv; ReLU; stride 2
4 4x4 conv; ReLU; stride 2
4x4 conv: ReLU; stride 2
Fully Connected 1 output
4x4 conv; ReLU; stride 2
4x4 conv; ReLU; stride 2
Fully Connected output
4x4 conv; ReI.U;; stride 2
Fully Connected output
MORE EXPERIMENTS
9.1 GA-ENTK vs. GA-CNTK
Next, we compare the images generated by GA-FNTK, GA-CNTK, and the multi-resolltiionll GA- CNTK described in Section 3.2 on the CelebA and CelebA-HQ datasets. The multi-resolutional GA-CNTK employs discriminators working at 256 x 256, 64x64, and 16x 16 pixel resolutions, respectively. Figure 6 shows the results. To our surprise, GA-NTK (which models the discriminator as an ensemble of fully connected networks) suffices to generate recognizable faces. The images synthesized by GA-FNTK and GA-CNTK lack details and global coherence, respectively, due to
19

---

Under review as a conference paper at ICLR 2023
(a) GA-ENTK CelehA
(b) GA-CNTK CelebA
Figure 6: The images generated by (a) GA-FNTK and (b) GA-CNTK given 256 CelebA training images.
Figure 7: When ò 1, GA-NTK tends to generate blurry mean image.
the characteristics of FNNs and CNNs. On the other hand, the multi-resolutional GA-CNTK gives both the details and global coherence thanks the multiple discriminators working different pixel resolutions. The results also demonstrate the potential of GA-NTK variants to generate high-quality data as there are many other techniques for GANs that could be adapted into GA-NTK.
9.2 BATCH-WISE GA-NTK
To work with larger training set, modify GA-CNTK by following the instructions in Section 3.2 to obtain the batch-wise GA-CNTK, which computes the gradients of Z" in Eq. (4) from 256 randomly sampled training images during each gradient descent iteration. We train the batch-wise
GA-CNTK on two larger datasets consisting of 2048 images from CelebA and 1300 images from ImageNet, respectively. Figure shows the results, and the batch-wise GA-CNTK can successfully generate the "daisy" images on ImageNet.
Note that the batch-wise GA-CNTK solves different problem than the original GA-CNTK-the former finds n that deceives multiple discriminators, each trained on 256 examples, while the latter searches for z" that fools single discriminator trained on 256 examples. We found that, when the batch size is small (b 1), GA-NTK tends to generate a blurry mean image regardless of model architectures and initializations of model weights and Z", as shown Figure 7 This is because the mean image is the best for simultancously fooling many NTK discriminators, each trained on a single example. However, in practice this setting is less common as one usually aims to use the largest b possible (Brock et al. 2018). Figure 8 shows that batch size of 256 suffices give plausible results on the CelebA and ImageNet datasets. Comparing the images in Figure [f)) with those in Figure aa), we can see that the batch-wise GA-CNTK gives little more blurry images but the patterns in each synthesized image are more globally coherent, both due to the effect of multiple discriminators.
9.3 SENSITIVITY TO HYPERPARAMETERS
Here, study how sensitive is the performance of WGAN, WGAN-GP, and GA-FNTK their hy- perparameters. We adjust the hyperparameters of different approaches using the grid search under a time budget of 3 hours, and then evaluate the quality of 2048 generated data points by the Wasser-
20

---

Under review as a conference paper at ICLR 2023
(b)
Figure 8: The images generated by batch-wise GA-CNTK on (a) CelebA dataset of 2048 randomly sampled images and (b) ImageNet dataset 1300 randomly sampled images.
10
10
2
10
10-3
GA-NTK
WGAN
WGAN-GP
GA-NTK
WGAN
WGAN-GP
(a)
(b)
Figure 9: The distribution of Wasserstein distance between P and Pdata (used to measure the quality of the generated points) over the searched hyper-parameters on training sets of (a) 8- and (b) 25-modal Gaussian mixtures.
stein distance between Pgen and Posa- We train different methods on two toy datasets consisting of 8- and 25-modal Gaussian mixtures following the settings described in Section 4.2 [guueeIshowe the results, and we can see that GA-FNTK achieves the lowest average Wasserstein distance both cases. Morcover, its variances smaller than the two other baselines, too. This shows that the performance of GA-FNTK is less sensitive to the hyperparameters and could easier to tune in practice.
Note that, with 3-hour time budget, the hyperparameters we obtained through the grid search are good enough for reproducing the experiments conducted by Mao et al. (201)) mode collapse. In the experiments, the Pgen of different methods aim to align 2D 8-modal Gaussian mixtures in the ground truth. Our results shown in Figure T0
21

---

Under review as a conference paper at ICLR 2023
3
680c
824
037
521
036
10000
13909
Intradora
Intrraiiona
Intortions
(a) MNIST
(b) CIFAR-II
(c) CelebA
Figure 11: The learning curve of 9 in GA-CNTKg and the generated images (2)) at different stages of training given the same input
(a) Ground truth
(b) WGAN
(c) WGAN-GP
(d) GA-FNTK
Figure 10: Visualization of distribution alignment and mode collapse 2D 8-modal Gaussian
mixtures dataset.
9.4 EVOLUTION OF IMAGES DURING TRAINING
Figure II shows the learning curve of the generator in GA-CNTKg and the relationship between the quality of images output by G and the number of gradient descent iterations. The results show that the loss can be minimized even if it is an f-divergence. and a lower loss score implies higher image quality. This is consistent with the results of GA-CNTK (without a generator) shown in Figure
Source of creativity. The diversity of our generated data not only comes from the randomness of an optimization algorithm (e.g., initialization of or splitting of X into batches, as discussed in Section 3.2) but also from the objective in Eq. 44) itself. To see this, observe in Figure 3 that the images generated at the later stage training contain recognizable patterns that change constantly training time, despite little change in the loss score. The reason is that, in Eq. (4), the Z" is optimized for a moving target-an any change of Z" causes D to be "retrained" instantly. The training of the generator in Eq. (6) also shares this nice property. In Figure II] the patterns of generated image g(z) change over training time even when the input z is fixed. However, getting diverse artificial data through this property requires prolonged training time. In practice, we simply initialize Z differently to achieve diversity faster.
10
MORE IMAGES GENERATED BY GA-CNTK AND GA-CNTKG
Figures 12 16 show more sample images synthesized by GA-CNTK and GA-CNTKg, All these images are obtained using the settings described in the main paper and the above.
We can see that the quality of the images synthesized by GA-CNTKg worse than that of the images synthesized by GA-CNTK, as discussed in Section 4.1 Furthermore, recall from Table that, without a generator network. the GA-NTK performs better when the date size increases. However, this is not the case for GA-NTKg having generator network. We have resampled training data and rerun the experiments times with different initial values of Z" but obtained similar results. Therefore, believe the instability is due to the sample complexity of the generator network-22 examples or less are insufficient to train a stable, high-quality generator. This is evident Figures 12(b)-15(b) where the generator outputs unrecognizable images more often.
22

---

Under review as a conference paper at ICLR 2023
4 7 8 7
0
8
8
93
0
1 7 69
0 3 4
4
3
1 6 6
3 O 7
2
5
4 9
9
/ 4 9
6 60 8 3
2 0
0
8
9 3 6
7 2
4 444466666633777
23432025/095113
O 059381092/60433
/22SE82321922044
2 5 65937334163300
5
2 2 0 7
3
6 7
2 3 3
8
8 S 6 8
8
8 9
6
O 0
5 8
3
3
8
2 3
3
5
2
9
5
4
7 O
R
6
7
7
O
5
(a)
9
4
A
A
&
9
5
9 0
5
/
7
7
3
4
4
4
/
0
0
3
3
5
3
5
5
9
0
4
2
5
0
41
7
2
9
4
0
3
9
5
3
8
7
9
8
2
9
3
5 7 2 O
3
0
4'
5
;
3
1
5
9
7
3
0
9
3
5
5
0
3
3
9
0
/
C
9
9
5
4
0
6
6
5
0
9
2
2
O
3
9
4
9
8
"
8
6
3
3
9
3
6
/
7
0
0
7
8
2
3
3
3
4
0
2
0
9
2
0
1
7
4
2
8
0
6
3
4
4
3
1
3
K
5
7
4
3
2
R
3
4
4
4
1
4
4
4
3
3
4
(b)
3
6
4
8
7
5
3
.3
Figure 12: Sample images generated by GA-CNTK (a) without and (b) with generator on the MNIST dataset of 256 randomly sampled images.
23

---

Under review as a conference paper at ICLR 2023
(b)
Figure 13: Sample images generated by GA-CNTK (a)without generator generator on the
CIFAR- 10 dataset of 256 randomly sampled images.
24

---

Under review as a conference paper at ICLR 2023
b
Figure 14: Sample images generated by GA-CNTK (a) without and (b) with generator on the CclebA dataset of 256 randomly sampled images.
25

---

Under review as a conference paper at ICLR 2023
M
OO
O
OR
TIO
Figure 15: Sample images generated by multi-resollttinal GA-CNTK on the CelebA-HQ dataset of 256 randomly sampled images.
Figure 16: Sample images generated by GA-CNTKg the CelebA-HQ dataset of 256 randomly sampled images.
26

---

Under review as a conference paper at ICLR 2023
11 DOWNGRADE IMAGES
As discussed in the main paper, find that, when the size of training set is small, image synthesis method may produce downgrade images that look almost identical to some images in the training set. This problem is less studied in the literature but important applications with limited training data. We investigate this problem by showing the images from the training set that are the nearest to generated image. We the SSIM (Wang et al., 2004) as the distance measure. Figures 17, 18 and
19 show the results for some randomly sampled synthesized images. As compared to GANs, S, both GA-CNTK and batch- wise GA-CNTK can generate images that look less similar the ground-truth
images.
Figure 17: Comparison between the images generated by WGAN-GP trained on 256 images and the nearest neighbors (measured by SSIM) from the training set. Images with red bounding boxes are generated images.
27

---

Under review as a conference paper at ICLR 2023
Figure 18: Comparison between the images generated by GA-CNTK trained on 256 images and the nearest neighbors (measured by SSIM) from the training set. Images with red bounding boxes are generated images.
Comparison between the images generated by GA-CNTKg trained on 256 images and the nearest neighbors (measured by SSIM) from the training set. Images with red bounding boxes are generated images.
Figure 19:
28

---

Under review as a conference paper at ICLR 2023
Table 9: The convergence speed and training time of different methods on a machine with a single NVIDIA Tesla V100 GPU given different datasets of 256 randomly sampled images. The GA- CNTK and GA-CNTKg are base,-wise, and the batch size b is set to 64 for all methods.
Metric
DCGAN
LSGAN
WGAN
WGANGP
SNGAN
GACNTK
GACNTKg
MNIST
Iterations
7400
5100
7000
3400
12800
500
1600
sec.
20
19
14
Seconds
370
368
189
711
35
177
CIFAR-10
Iterations
N/A
N/A
14000
11100
N/A
600
6200
Iter. sec.
17
17
16
15
14
13
8
Seconds
N/A
N/A
875
740
N/A
46
775
CelebA
Iterations
N/A
N/A
18800
11200
N/A
1200
5900
Iter. / sec.
13
12
12
10
6
5
Seconds
N/A
N/A
1566
1120
N/A
20
1180
12 SEMANTICS LEARNED BY GA-CNTKG
Here, we investigate whether the features learned by GA-NTK can encode high-level semantics. We plot "interpolated" images output by the generator G of GA-CNTKg taking equidistantly spaced e's along a segment
space as the input. For ease of presentation, we consider 2-dimensional space and train G on MNIST and CelebA datasets of 256 examples. Figure 20 shows the re- sults, where the generated patterns transit smoothly across the 2D z space, and neighboring images share similar looks. These similar-looking images generated from adjacent but meaningless z's. suggesting that the learned features encode high-level semantics.
13 CONVERGENCE SPEED AND TRAINING TIME
In this section, we study the time usage for training GA-NTK variants and compare with training of GANs. We conduct experiments to investigate the number of iterations and the wall-clock time required to train different methods on different datasets 256 randomly sampled images. We the batch-wise GA-CNTK and GA-CNTKg and set the batch size b 64 for all methods. We run the experiments on a machine with a single NVIDIA Tesla V100 GPU. For DCGAN and LSGAN whose loss scores do not reflect image quality, we monitor the training process manually and stop it as long as the generated images contain recognizable patterns, But these methods do not seem to converge. For other methods, of 0.05 to determine convergence. The results are shown in Table 9 As we can see, the number of iterations required by either batch-wise GA-CNTK or GA-CNTKg significantly smaller than that used by GANs. This justifies our claims in Section 1. However, the batch-wise GA-CNTK and GA-CNTKg run fewer iterations per second than GANs because of the higher computation cost involved in back- propagatpggg through K.b. In terms of wall-clock time, the batch-wise GA-CNTK the fastest while the GA-CNTKg runs as fast as WGAN- We expect that, with the continuous optimization of the Neural Tangents library (Novak et al.) 2019a) which our code is based on, the training speed of GA-NTK variants can be further improved.
the early-stopping with the patience of 10000 steps and delta
29

---

Under review as a conference paper at ICLR 2023
3
2
3
3
3
7
3
3
3
3
3
3
5
5
0
2
5
5
9
4
6
7
7
7
7
9
4
4
5
5
7
7
9
4
4
4
6 9 7 7 7 2 2 2
7
9
9
7
7
7
9
9
9
7
7 7
f
f
1
7
5
9
9
4
{
{
4
8
8
0
9
/
4
4
4
4
4
3
3
0
2 2
y
4
4
_
3
2
4
4
3
2
3
2
Figure 20: Interpolated images generated by GA-CNTKg, which is trained on (a) MNIST and (b) CelebA datasets of 256 randomly sampled examples from all classes. The & takes 2-dimensional z's as input. For each dataset, we feed equidistantly spaced z's along a segment in z space to G to get the interpolated images.
30


Reviews:
[{"review_id": "SW1AGo-WHHN", "reviewer": "Reviewer_LpjQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose generative adversarial neural tangent kernel (GA-NTK) and its variants.\nThey model the discriminator as a Gaussian process whose mean (and covariance) is governed by NTK.\nAlso, they can be performed via a single-level training process as a whole, because the training dynamics of their discriminators can be evaluated in closed form.\nThe authors claim that this property allows them to avoid the training difficulties often encountered with GANs, and experimentally confirmed this claim and their effectiveness for data synthesis.", "summary_of_the_review": "I give this study a positive score now, on the basis of the novelty and interest of the idea [1*].\nHowever, the description of the proposed method is insufficient and not clear [3]--[6*].\nThese points could be improved!\nI will change the score up or down, depending on the responses to [3]--[6*].\n\n---\n\nI changed the score from 6 to 8.", "strength_and_weaknesses": "I place special emphasis on comments with the mark *.\n\n---\n\nStrength:\n\n[1*] As the authors also wrote at the end of Section 1, the main significance of this study is not performance improvement but novelty of the research direction.\nThis study is an interesting attempt that uses an interesting idea to solve the problems of GANs, which are the motivation for the study.\n\n---\n\nWeakness:\n\n[2] There are writing misses.\n\n[2-1] You should use \\citep as well as \\citet.\nFor example, you can write \"theoretical interpretations Arjovsky & Bottou (2017).\" in p.3 as \"theoretical interpretations (Arjovsky & Bottou, 2017).\" by using \\citep.\nAlso, \"Franceschi et al. Franceschi et al. (2021)\" in p.3 is strange for me; you can write \"Franceschi et al. (2021)\" by using \\citet only.\nPlease look up the use of \\citep and \\citet and rewrite the relevant parts.\n\n[2-2] In l.12 in p.4, value each element -> value of each element.\nCheck your writing (including other parts) again.\n\n[2-3] In eq.(4), $\\arg\\min\\_{Z^n} L(Z^n)=\\arg\\min\\_{Z^n} ||\\cdot||_2$ -> $\\arg\\min\\_{Z^n} L(Z^n)$, where $L(Z^n)=||\\cdot||^2$.\nYou should define $L(Z^n)$ explicitly, and unify $||\\cdot||\\_2$ and $||\\cdot||$ into one if they are the seme (see also norm in other parts).\n\n[2-4] The $f$-divergence appears in places, but what function was used for $f$?\n\n[2-5] In Figure 3, title is partially obscured.\n\n[2-6] In Figure 5, what is \"real\" in the legend?\n\n[2-7] In the last equation in p.14, absolute value symbol is missing:\nfor example, $\\nabla_{z_j} L(Z^n)\\le$ -> $|\\nabla_{z_j} L(Z^n)|\\le$.\n\n[2-8] You should centralize Figure 14.\n\n[2-9] In Section 10, ?? -> 18.\n\n---\n\nQuestion:\n\nRegarding the following points, it may be that the authors' description is appropriate and there is no problem, just because I have not understood it correctly.\nI do not reflect these points in my current recommendation score.\nDepending on the authors' response, I may change my recommendation score.\n\n[3] Is it necessary to solve an optimization problem like eq.(4) every time to generate pseudo examples?\nIf so, that is a demerit compared with GANs, which can generate an infinite number of pseudo examples without additional training from a trained model.\nThis disadvantage should be written more clearly.\n(Even so, I will not lower the score.)\n\n[4*] From the experimental results (see especially Figure 3) and my understanding of the formulas, it seems to me that GA-NTK that is sufficiently trained with a large $t$ would simply return an training example (or mixture of training examples) as a pseudo example.\nIs this question correct?\nIf correct, such data synthesis would have low practical value.\nIt does not seem to me that the use of a generator model adequately solves such a problem.\nAlso, the authors should explain when (i.e., at what $t$) to stop learning.\n\n[5] This question relates to [4*].\nI am interested in the relationship between the batch size $b$ and the novelty of the pseudo example (dissimilarity to the training examples $X^{b/2}$).\nI suspect that if $b$ is small, each generated pseudo example will be fairly close to one of the training examples $X^{b/2}$.\nIs this question correct?\nIf correct, I think that the authors should mention this issue.\n\n[6*] It is difficult for me to understand the whole algorithm.\nI could not understand the relation between the index $t$ in $\\lambda=\\eta \\cdot t$ in eq.(3), the index $j$ in Theorem 3.1, \"Epochs\" in Figure 3, and \"Iterations\" in Figure 5, and the evolution of $Z^n$ and $Z^{n,(j)}$.\nPlease write a pseudo code (or modify the text) so that the relation of these objects become clear.\n\nAlso, how does $t$ change when $Z^n$ changes?\nI think this question is relevant to the validity of applying NTK theory.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "SAgdxxDphZ", "reviewer": "Reviewer_qHF8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel generative model GA-NTK that mitigates the drawbacks of traditional GANs trained with the alternative SGD method. This is achieved by using a closed-form discriminator based on a neural tangent kernel (NTK) instead of a neural network. The author proved the convergence of the proposed model during the training phase. Also, numerical experiments are conducted to show the efficacy of the proposed method.", "summary_of_the_review": "Overall the paper provides a new angle of view for the GAN models. But there are still some points the paper needs to address to meet the bar of acceptance. ", "strength_and_weaknesses": "*Strengths*\n- The motivation of the proposed method is clearly stated. The mathematical settings of the proposed method are clearly stated. The overall writing is fluent and easy to understand.\n- Numerical experiments of generating new samples with the method are conducted. Also, the result is consistently evaluated against state-of-the-art methods.\n\n*Weaknesses*\n- The statement of the method could be made better if the training process of the proposed method is more clearly stated. In particular, I'm confused how the proposed discriminator is trained (or not trained) during the overall training process, e.g. what are the variables to be updated during training? \n- Since the main advantage of the proposed method is to substitute traditional neural network-based discriminator with kernel methods, it would be plausible to include the comparison of the discriminator performance between the two methods. \n- The scalability of the proposed method is one of the drawbacks of the proposed method. In particular, the proposed discriminator has a space complexity of O(n^2) where n is the number of data points. \n- The paper makes the assumption that mode collapse is solely caused by min-max training regime in traditional GAN methods. But mode collapse can be caused by other factors as well, such as the structure of generators. A more thorough study of the causal relationship between min-max training and mode collapse could be plausible to provide.\n- In the experiment part, generated images and their most similar images from the dataset are provided. It looks to me that the GA-CNTK method tries to memorize the training samples and the pixel level and not able to learn high-level semantics from the images.\n- typo on page 4: \"my be desirable\" -> \"might be desirable\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HLlECLgvv3v", "reviewer": "Reviewer_rAgg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel NTK-based GAN named GA-NTK. Specifically, the authors use an infinite-wide linear neural network (i.e., NTK model) as the discriminator for GAN, which thus enables one to directly obtain a closed-form solution for the inner maximization problem in GAN training. Compared with vanilla GAN, GA-NTK enjoys better convergence and less mode collapse and can avoid gradient vanishing. Experiments are conducted on both synthesized and real-world datasets to justify the effectiveness of the proposed method.", "summary_of_the_review": "The proposed GA-NTK is novel and the experiment results are strong enough to justify the effectiveness of GA-NTK. Besides, the paper also extends the application domain of NTK and opens up new research direction of GAN training. Therefore, I suggest to accept this paper.", "strength_and_weaknesses": "In general, the paper is well-written and the proposed GA-NTK method is very interesting and novel. Other strengths are listed below:\n\n1. GA-NTK converts the original bilevel optimization problem of training GAN to a single-level problem, which significantly reduces the hardness of training GAN.\n\n2. Experiment results on real-world datasets show that GA-NTK can generate images that align well with human perception.\n\n3. Experiment results on synthesized data are strong enough to justify that GA-NTK can avoid mode collapse.\n\n\nOther comments / questions:\n\n1. I wonder the relationship between the batch size and the generative performance of the proposed method. For example, based on the FID results in Table 1, for GACNTKg, one can find a positive correlation on MNIST, a negative correlation on CIFAR-10, and no correlation on CIFAR-100. Any insight about that?\n\n2. What is the time usage for training GA-NTK and its variants?\n\n3. In the second paragraph of page 2, I disagree with the claim that gradients of $\\theta_g$ can not be back-propagated through the inner maximization problem. Actually, once you find the solution of the inner maxization problem in which we denoting it as $\\theta_D^*$, then the gradient for the current $\\theta_g$ can be written in an analytical form with respect to $\\theta_D^*$.\n\n4. The comparison between GA-NTK and existing approaches may be a little unfair, as GA-NTK does not use the same model architectures as that in existing approaches. A more fair comparison for me may be comparing existing approaches with its GA-NTK version in which the finite-wide discrimators are replaced with their corresponded infinite-wide linear conterparts.\n\n5. It would be interesing to compare the convergence speed of GA-NTK with previous GAN training algorithms.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0ABMJSIcS5T", "reviewer": "Reviewer_ihk5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework, GA-NTK, of Generative adversarial networks (GANs) by constructing the discriminator based on the neural tangent kernel (NTK). The NTK allows to describe the discriminator output in a compact and simple function of the given data X and the auxiliary variables Z, and hence many computational issues involved in the GANs training, such as convergence problem, mode collapse, and gradient vanishing, are greatly suppressed. The proposed method's experimental results succeeded in producing artificial images of comparable qualities with other standard methods such as WGAN and SNGAN.\n", "summary_of_the_review": "The core idea of the paper is novel and interesting but its statistical meaning is not clear as mentioned above. Taken as best one can, it can be said that the authors proposed an optimization framework of generative models rather than the statistical one. However, it is not possible for me to take it so favorably, and hence I vote for a rejection side about this paper.\n", "strength_and_weaknesses": "Strengths:\n- The core part of the idea is novel. \n\n- The computational cost of the proposed method is significantly smaller than other comparable methods.  \n\nWeaknesses:\n- From a statistical viewpoint, there is no sufficient theoretical background for the proposed method, I think. The details are described in the Quality section below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject