Curriculum Learning for Deep Generative Models with Clustering
Authors: Deli Zhao
Jiapeng Zhu
Zhenfang Guo
Bo Zhang
Venue:  (2020)
Keywords: curriculum learning
generative adversarial network

Abstract: Training generative models like Generative Adversarial Network (GAN)  is challenging for noisy data. A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper. The curriculum construction is based on the centrality of underlying clusters in data points.  The data points of high centrality takes priority of being fed into generative models during training. To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality. Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models. The experiments on cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g. ProGAN) with respect to specified quality metrics for noisy data. An interesting finding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper.

Under review as a conference paper at ICLR 2020
CURRICULUM LEARNING FOR DEEP GENERATIVE MODELS WITH CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
Training generative models like Generative Adversarial Networks (GANs) is chal- lenging for noisy data. A novel curriculum learning algorithm pertaining to cluster- ing is proposed to address this issue this paper. The curriculum construction based on the centrality of underlying clusters in data points. The data points of high centrality takes the priority being fed into generative models during training. To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality. Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models. The experiments data validate that our algorithm is able to learn the optimal generative models (e.g. ProGAN) with respect specified quality metrics for noisy data. An interest- ing finding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper. Besides, the centrality-accune accuracy curve in our algorithm provides an effective way of visualizing the model state with curriculum training.
the cat and human-face
INTRODUCTION
Deep generative models have piqued researchers' interest in the past decade. The fruitful progress has been achieved on this topic, such as auto-encoder (Hinton & Salakhutdinoy (006) and varia- tional auto-encoder (VAE) (Kingma & Welling 2013 Rezende et 2014), generative adversarial Arjovsky et al. (017)), normalizing flow (Rezende & Mohamed 2015 Dinh al. 2015 (015|[2117 Kingma Dhariwal (018), and auto- regressive models (van den Oord et al. 601 a 2017). However, it is non-trivia to train deep generative model that can converge to proper minimum of the associated optimization. For example, GAN suffers non-stability, mode collapse, and generative distortion during training. Many insightful algorithms have been proposed to circumvent those issues, including feature engineering (Salimans) al. 2016). various discrimination metrics (Mao et al. 2016; Arjovsky et al. 2017 Berthelot et al. 2017), distinctive gradient penalties (Gulrajani et al. 2017 Mescheder et al.) 20188), spectral nor- malization to discriminator (Miyato al.) 2018), and orthogonal regularization generator (Brock) jet al., (019). What particularly of interest is that the breakthrough for GANs has been made with simple technique of progressively growing neural networks of generators and discriminators from low-resolution images to high-resolution counterparts (Karras et al., (01 This kind of progressive growing also helps push the state of the arts new level by enabling StyleGAN to produce photo-realistic and detail-sharp results (Karras et al. 2018b), shedding new light on wide applications of GANs in solving real problems. This idea of progressive learning actually general manner of cognition process (Elman 1993, Oudeyer et al. (007), which has been formally named curriculum learning machine learning (Bengio al.) (009). The central topic of this paper is to explore new curriculum for training deep generative models.
network (GAN) (Goodfellow et al. 2014 Radford et al. [2066] |rrjoo//rr
To facilitate robust training deep generative models with noisy data, we propose curriculum learning with clustering. The key contributions are listed as follows:
We first summarize four representative curricula for generative models, i.e. architecture (generation capacity), semantics (data content), dimension (data space), and cluster (data structure). Among these curricula, cluster curriculum newly proposed in this paper.

---

Under review as a conference paper at ICLR 2020
Cluster curriculum is to treat data according to the centrality of each data point, which is pictorially illustrated and explained in detail. To foster large-scale learning, we devise the active set algorithm that only needs an active data subset of small fixed size for training.
The geometric principle is formulated to analyze hardness of noisy data and advantage of cluster curriculum. The geometry pertains to counting a small sphere packed in an ellipsoid, on which is based the percolation theory we use.
The research on curriculum learning is diverse. Our work focuses on curricula that are closely related data attributes, beyond which is not the scope we concern in this paper.
CURRICULUM LEARNING
Curriculum learning has been basic learning approach to promoting performance of algorithms in machine learning. We quote the original words from the seminal paper (Bengio et al. 2009) as its definition:
Curriculum learning, "The basic idea is to start small, learn easier aspects of the task or easier sub-tasks, and then gradually increase the difficulty level" according to pre-defined or self-learned curricula.
From cognitive perspective, curriculum learning is common for human and animal learning when they interact with environments (Elman 1993), which the reason why it is natural as a learning rule for machine intelligence. The learning process of cognitive development is gradual and progres- sive (Oudeyer et al., 2007). In practice, the design of curricula lask-dependent and data-dependent. Here we summarize the representative curricula that are developed for generative models.
Architecture curriculum. The deep neural architecture itself can be viewed as a curriculum from the viewpoint of learning concepts (Hinton & Salakhutdinov 2006 Bengio et al. (2066) or disentangling representations (Lee et al. 2011). For example, the different layers decompose distinctive features of objects for recognition (Lee et al. 2011 Zeiler & Fergus 2014 Zhou et al. 2016) and generation (Bau
ct al. 2018). Besides, Progressive growing of neural architectures is successfully exploited in
GANs Karras et al.) 2018a Heljakka et al., 2018; Korkinof al., 2018 Karras et al. 2018b).
Semantics curriculum. The most intuitive content for each datum the semantic information that the datum conveys. The hardness of semantics determines the difficulty of learning knowledge from data. Therefore, the semantics can be a common curriculum. For instance, the environment for game in deep reinforcement learning (Justesen al., 2018) and the number sense of learning cognitive concepts with neural networks (Zou & McClelland 2013) can be such curricula.
Dimension curriculum. The high dimension usually poses the difficulty of machine learning due to the curse of dimensionality (Donoho 2000). in the sense that the amount of data points for learning grows exponentially with dimension of variables (Vershynin) 2018). Therefore, the algorithms expected to be beneficial from growing dimensions. The effectiveness of dimension curriculum is evident from recent progress on deep generative models, such as ProGANs (Karras et al. [0188a b) by gradually enlarging image resolution and language generation from short sequences to long sequences of more complexity (Rajeswar al. 2017 Press et al.) 2017).
CLUSTER CURRICULUM
For fitting distributions, dense data points are generally easier to handle than sparse data or outliers. To train generative models robustly, therefore, it is plausible to raise cluster curriculum, meaning that generative algorithms first learn from data points close cluster centers and then with more data progressively approaching cluster boundaries. Thus the stream of feeding data points to models for curriculum learning the process of clustering data points according to cluster centrality that will be The toy example in Figure TnguureTillsseen illluttrae how to form cluster curriculum.
explained in section] 3.2
3.1
WHY CLUSTERS MATTER
The importance clusters for data points is actually obvious from geometric point of view. The data sparsity in high-dimensional spaces causes the difficulty of fitting the underlying distribution of
2

---

Under review as a conference paper at ICLR 2020
100
Tu 200
71 300
n 400
Figure I: Cluster Curriculum. From magenta color to black color, the centrality of data points reduces. The value n the number of data points taken with centrality order.
data points [Vershynin 2018). So generative algorithms may be beneficial when proceeding from the local spaces where data points are relatively dense. Such data points form clusters that are generally informative subsets with respect to the entire dataset. addition, clusters contain common regular patterns of data points, where generative models are easier to converge. What is most important is that noisy data points deteriorate performance of algorithms. For classification, the effectiveness of curriculum learning is theoretically proven to circumvent the negative influence of noisy data (Gong) cct al. 2016). We will analyze this aspect for generative models with geometric facts.
3.2 GENERATIVE MODELS WITH CLUSTER CURRICULUM
With cluster currieulum, are allowed to gradually learn generative models from dense clusters to cluster boundaries and finally to all data points. In this way, generative algorithms are capable of avoiding the direct harm of noise or outliers. To this end, we first need measure called centrality that is the terminology in graph-based clustering. It quantifies the compactness of cluster in data points or a community in complex networks (Newman 2010). A large centrality implies that the associated data point is close to one of cluster centers. For easy reference, we provide the algorithm of the centrality we use in Appendix. For experiments in this paper, all the cluster curricula are constructed by the centrality of stationary probability distribution, i.e. the eigenvector corresponding to the largest eigenvalue of the transition probability matrix drawn from the data.
To be specific, let E RR denote the centrality vector of m data points. Namely, the i-th entry of c is the centrality of data point z;. Sorting c in descending order and adjusting the order of original data points accordingly give data points arranged by cluster centrality. Let X = { u. X , X
signify the set of centrality sorted data points, where X, the base set that contains sufficient data to attain convergent generative models, and the rest of X' is evenly divided into subsets according to centrality order. In general, the number of data points in Xo is much less than m and determined according to X. Such division of X, serves to efficiency of training, because we do not need to train models from very small dataset. The cluster curriculum learning is carried out by incrementally feeding subsets in X into generative algorithms. In other words, algorithms are successively trained on X, X U X,, after X o. meaning that the curriculum for each round of training is accumulated with X,,
In order to determine the optimal curriculum 2 X U... we need the aid of quality metric of generative models, such as Fréchet inception distance (FID) or sliced Wasserstein distance (SWD) (Bori 2018). For generative models trained with each curriculum, we calculate the associated score s; via the specified quality metric. The optimal curriculum for effective training can be identified by the minimal value for all s;, where = 1..... . I+ 1. The interesting phenomenon of this score curve will be illustrated in the experiment. The minimum of score is apparently metric-dependent. One can refer to (Borji 2018) for the review of evaluation metrics. In practice, we can opt one of reliable metrics use or multiple metrics for decision-making of the optimal model.
There are two ways of using the incremental subset X2 during training. One is that the parameters of models are re-randomized when the new data are used, the procedure of which is given in Algorithm 1 in Appendix. The other is that the parameters are fine-tuned based on pre-training of the previous model, which will be presented with fast learning algorithm in the following section.
3

---

Under review as a conference paper at ICLR 2020
3.3 ACTIVE SET FOR SCALABLE TRAINING
To obtain the precise minimum of the cardinality of X; needs to be set much smaller than m, meaning that 1 will be large even for a dataset of moderate scale. The training of many loops will be time-consuming. Here we propose the active set to address the issue, in the sense that for each loop of cluster curriculum, the generative models are always trained with a sub- set of a small fixed size instead of Zo + Xo U X, whose size becomes incrementally large. To form the active set A, the subset Ao of data points are randomly sampled from Lo to combine with X; for the next loop, where |ool = |A4| - (, For easy understanding, we illustrate the active set with toy example in Figure 2 In this scenario, progressive pre-training must be applied, meaning that the update of model parameters for the current train- ing is based on parameters of previous loop. The procedure of cluster curricu- lum with active set is detailed in Algo- rithm 2 in Appendix.
(a) x (1,
(b) A= Ao U
Figure 2: Schematic illustration of active set for cluster curriculum. Here [R = | = [x=1= 100. The cardinal- ity |A| of the active set A is 200. When X, is taken for training. we need to randomly sample another 100 (i.e. |A| (7aa) data points from the history data X. u E, to form Au- Then the complete active set is composed by Ao R.. We see that data points in X. X, become less dense after sampling.
The active set allows us to train genera- tive models with small dataset that is actively adapted, thereby significantly re- ducing the training time for large-scale data.
GEOMETRIC VIEW OF CLUSTER CURRICULUM
Cluster curriculum bears the interesting relation to high-dimensional geometry, which can provide geometric understanding of our algorithm. Without loss of generality, we work on a cluster obeying the normal distribution. The characteristic of the cluster can be extended into other clusters of the same distribution. For easy analysis, let us begin with a toy example. As Figure (a shows, the confidence ellipse Eg fitted from the subset of centrality-ranked data points is nearly conformal to E1 of all data points, which allows us to put the relation of these two ellipses by virtue of the confidence-level equation. Let (0, Σ) signify the center and covariance matrix of the cluster C of interest, where = {a. xx, E Rdd n.). To make formal, we can write the equation by
- Xd.or x
(1)
where x2.a can be the chi-squared distribu- tion or Mahalanobis distance square, d is the degree of freedom, and (1 a1) jo, 1] is the confidence level. For conciseness, write Xa as Xa in the following con-
(b)
Figure 3: Illustration of growing cluster in cluster curriculum. (a) Data points taken with large centrality. (b) The annulus formed by of removing the inner ellipse from the outer one.
text. Then the ellipses Ea, and Exg corre-
spond to X2, and X2 respectively, where 01 I 0:2.
To analyze the hardness of training generative models, a fundamental aspect is to examine the number m(F)) of given data points falling geometric entity F and the number N N(E) of lattice points in it. The less n [E) is compared to N(E),, the harder the problem will be. However, the enumeration of lattice points is computationally prohibitive for high dimensions. Inspired by the information theory of encoding data of normal distributions (Roman 1996 Ma et al. 2007). we count the number of small spheres S₂ of radius s packed in the ellipsoid instead. Thus we can use this number to replace the role of (EE) as long as the radius of the sphere Sg is set properly. With little abuse of
For cluster curriculum, it is annulus explained shortly.
4

---

Under review as a conference paper at ICLR 2020
Data and sphere numbers in annulus
noral
d=2
d=20
5
d-50
d=100
d=500
4
d=1000
lattice
3
d-2
d-20
2
d-50
d-100
1
d-500
d-1000
0
0.01
0.8
Chi quantile X=
Figure 4: Comparison between the number rıA) of data points sampled from the isotropic normal distributions and ((A) of spheres (lattice) packed in the annulus A with respect to the Chi quantile Xag- d the dimension of data points. For each dimension, sample 70,000 data points from OO I). The scales of y-axis and r-axis are normalized by 10,000 and Xa. respectively.
notation, we still use (E) to denote the packing number in the following context. Theorem I gives the exact form of (EE).
Theorem 1. For set C = {æ E }Rd} of n data points drawn from normal distribution (0, E). the ellipsoid Ea of confidence
a is defined as x -E-xx S Xa. where Σ has no zero eigenvalues and a E [0, 1]. Let N(Ea) be the number of spheres of radius packed in the ellipsoid Ea. Then we can establish
N(E.) - Xaa) )
/det (E).
We see that (E.) admits tidy form with Mahalanobis distance Xo. dimension d, and sphere radius as variables. The proof is provided in Appendix.
The geometrie region of interest for cluster curriculum is the annulus A formed by removing the ellipsoid E.,, from the ellipsoid Ea,, as Figure 3)b) displays. We investigate the varying law between ni A) and (A) in the annulus when the inner ellipse Faz grows with cluster eurriculum. For this purpose, we need the following two corollaries that immediately follows from Theorem
Corollary 1.
Let (()) be the number of spheres of radius packed in the annulus A that is formed by removing the ellipsoid Eag from the ellipsoid Eay, where on 02. Then we have
N(A)=
(X==)
det (2).
Corollary 2. (EA))N(E...) =1 (Xaz/Xan
obvious that (A) goes infinite when d oc under the conditions that Xa, Xax and is bounded. Besides, when Ea, (cluster) grows, ((A) reduces with exponent if Ex, is fixed.
In light of Corollary [ we can now demonstrate the functional law between n(A) and N(A). First, we determine Xa, as follows
Xar max ||a 2; C,
which means that Ea, is the ellipsoid of minimal Mahalanobis distance to the center that contains all the data points in the cluster. addition, we need to estimate a suitable sphere radius E, such that n( (Fa,) and N N(F.,) have comparable scales in order make n(A) and ((A) comparable in scale. To achieve this, we define an oracle ellipse where ni( E)
N(E). . For simplicity, we let Ea, be
the oracle ellipse. Thus we can determine with Corollary
Corollary 3. we let Emn be the oracle ellipse such that n E.,) N N(Ea,) ), then the free parameter E can be computed with =x, (nnn(()))))))
To make the demonstration amenable to handle, data points we use for simulation are assumed to obey the isotropic normal distribution, meaning that data points generated with nearly equal
Thhe ellipse refers the surface and the ellipsoid refers the elliptic hall.

---

Under review as a conference paper at ICLR 2020
Figure 5: Examples of LSUN cat dataset and CelebA face dataset. The samples in the first row are of high centrality and the samples of low centrality in the second row are noisy data or outliers that we call in the context.
variance along each dimension. Figure [ shows that n (AA) gradually exhibits the critical phenomena of percolation processes when the dimension d goes large, implying that the data points in the annulus A are significantly reduced when E,, grows a little bigger near the critical point. In contrast, the number (AA) of lattice points is still large and varies negligibly until Eag approaches the boundary. This discrepancy indicates clearly that fitting data points in the annulus is pretty hard and guaranteeing the precision is nearly impossible when crossing the critical point of no (() even for a moderate dimension (e.g. 500). Therefore, the plausibility of cluster curriculum can be drawn naturally from this geometric fact.
5 EXPERIMENT
The generative model that we use for experiments are Progressive growing of GAN (ProGAN) (Karras et al. 2018a). This algorithm is chosen because ProGAN is the state-of-the-ard algorithm of GANs with official open sources available. According to convention, we opt the Fréchet inception distance (FID) Borji 2018) for ProGAN as the quality metric.
5.1 DATASET AND EXPERIMENTAL SETTING
We randomly sample the 200,000 cat images from the LSUN dataset Yu et al., 2015). These cat images are captured in the wild. So their styles vary significantly. Figure 5 shows the cat examples of high and low centralities. We can that the noisy cat images differ much from the clean ones. There actually contain the images of very few informative cat features, which are the outliers we refer
to. The curriculum parameters set as | 20, 000 and [Xys 10, 000, which means that the algorithms are trained with 20,000 images first and after the initial training, another 10,000 images according centrality order are merged into the current training data for further re-training. For active set, its size is fixed to be 30, 000.
The CelebA dataset is large-scale face attribute dataset (Liu et al. 2015). We use the cropped and well- well-aligned faces with a bit of image backgrounds preserved for generation task. For cluster- curriculum learning, we randomly sample 70,000 faces as the training set. The face examples of
different centralities are shown in Figure 5 The curriculum parameters are set as | ol = 10, 000 and | || 5, 000. We bypass the experiment of the active set faces because it is used for the large-scale data.
Each image in two databases resized to be 64 64. To form cluster curricula, we exploit ResNet34 (He et al.) 2016) pre-trained on ImageNet (Russakovsky et al.] 2015) to extract 512- dimensional Teatures for each face and cat images. The directed graphs are built with these feature vectors. We determine the parameter a of edge weights by enforcing the geometric mean of weights to be 0.8. The robustness of varying the value was validated in (Zhao & Tang (0088) for clustering. The number of nearest neighbors is set to be K = 4 log m. The centrality is the stationary probability distribution. All codes are written with TensorFlow.
Percolation theory is a fundamental tool of studying the structure of complex systems in statistical physics and mathematics. The critical point is the percolation threshold where the transition takes place. One can refer (Staufer & Aharony 1994) if interested.

---

Under review as a conference paper at ICLR 2020
0.15
centraliy
45
centrality
35
normal training active set
training FFHQ
normal training
30
0.14
0.1
Centraliy
0.12
Centrality
25
0.1
FDD
0.08
20
0.05
0.06
15
0.02
25
10
0
6
8
10
12
14
16
18
20
5.5 6
7
Order of data points
O Order of data points
(a) Cat
(b) Face
Figure 6: FID curves of cluster-currculum learning for ProGAN on the cat dataset and CelebA face dataset. The centrality and the FID share the z-axis due to that they have the same order of data points. The same colors of the y-axis labels and the curves denote the figurative correspondence. The network parameters for "normal training" are randomly re-initialized for each re-training. The active set based on progressive pre-training of the fixed small dataset. The scale of the r-axis is normalized by 10,000.
5.2 EXPERIMENTAL RESULT
From Figure Ga, minima exist amid the training process. This is the clear evidence that the noisy data and outliers deteriorate the quality of generative models during training. From the optimal curricula found by two algorithms (i.e. curricula at 1 110,000 and 100,000), we can see that the curriculum of the active set differs from that of normal training only by one-step data increment, implying that the active set is reliable for fast cluster-currcuuum learning. The performance of the active set measured by FID is much worse than that of normal training, especially when more noisy data are fed into generative models. However, this does not change the whole V-shape of the accuracy curve. Namely, it is applicable as long as the active set admits the metric minimum corresponding to the appropriate curriculum.
that the FID curves are all nearly V-shaped, indicating that the global
The V-shape the centrality-FID curve on the cat data is due to that the noisy data of low centrality contains little effective information characterize the cats, already displayed in Figure However, is different for the CelebA face dataset where the face images of low centrality also convey the part of face features. As evident by Figure (b. ProGAN keeps being optimized by the majority of the data until the curriculum of size 55, 000. To highlight the meaning of this nearly negligible minimum, we also conduct the exactly same experiment of high-quality (Karras et al. 201 8b). For FFHQ data, the noisy face data can be ignored. The gray curve of normal training Figure 6p indicates that the FID of ProGAN monotonically decreased for all curricula. This gentle difference of the FID curves at the ends between CelebA and FFHQ clearly demonstrates the difficulty of noisy data generative algorithms.
the FFHQ face dataset containing 70, 000 face images
5.3 GEOMETRIC INVESTIGATION
To understand cluster curriculum deeply, we employ the geometric method formulated in section F to analyze the cat and face data. The percolation processes are both conducted with 512-dimensional features from ResNet34. Figure [ displays the curve of n(A) that is the variable of interest in this scenario. As expected, the critical point in the percolation process occurs for both cases, as shown by blue curves. An obvious fact is that the optimal curricula (red strips) both fall into the (feasible) domains of percolation processes after the critical points, as indicated by gray color. This is desirable property because data become rather sparse in the annuli when crossing the critical points. Then noisy data play the non-negligible role on tuning the parameters of generative models. Therefore, fast learning strategy can be derived from the percolation process. Training may begin from the curriculum specified by the critical point, thus significantly accelerating gluster-curriclum learning.
7

---

Under review as a conference paper at ICLR 2020
20
critical point
critical point
Data mumber in annuls
Data number annulus
6
15
4
1g
5
2
1.5
0
0.01
0.2
0.4
0.6
0.8
0.01
0.2
0.6
0.8
quantile Xa
Chi quantile
(a) cat data
(b) face data
Figure 7: Geometric phenomenon of cluster curriculum on the LSUN cat and CelebA face datasets. The pink strips are intervals of optimal curricula derived by generative models. For example, the value 9 of the pink interval in (a) is obtained by 9 20 11, where 11 is one of the minima (i.e. 1 110,000) in Figure [o. The others are derived in the same way. The subtraction transforms the data number the cluster to be the one in the annulus. The critical points are determined by searching the maxima of the absolute discrete difference of the associated curves. The scales of y-axes are normalized by 10,000.
Another intriguing phenomenon is that the more noisy the data, the closer the optimal interval (red strip) is the critical point. We can see that the optimal interval of the cat data is much closer to the critical point than that of the face data. What surprises us here is that the optimal interval of cluster curricula associated with the cat data nearly coincides with the critical point the percolation process in the annulus! This means that the optimal curriculum may be found at the intervals close to the critical point of ((A)) percolation for heavily noisy data, thus affording great convenience to learning appropriate generative model for such datasets.
ANALYSIS AND CONCLUSION
Cluster curriculum is proposed for robust training of generative models. The active set of cluster curriculum is devised to facilitate scalable learning. The geometric principle behind cluster curriculum analyzed detail as well. The experimental results on the LSUN cat dataset and CelebA face dataset demonstrate that the generative models trained with cluster curriculum is capable of learning the optimal parameters with respect to the specified quality metric such as Fréchet inception distance and sliced Wasserstein distance. Geometric analysis indicates that the optimal curricula obtained from generative models are closely related to the critical points of the associated percolation processes established in this paper. This intriguing geometric phenomenon is worth being explored deeply in terms of the theoretical connection between generative models and high-dimensiones geometry.
worth emphasizing that the meaning of model optimality refers the global minimum of the centrality-FID curve. As we already noted, the optimality is metrie-dependent. We are able to obtain the optimal model with cluster curriculum, which does not mean that the algorithm only serves to this purpose. We know that more informative data can help learn more powerful model covering the large data diversity. Here trade-off arises, i.c. the robustness against noise and the capacity of fitting more data. The centrality-FID curve provides visual tool to monitor the state of model training, thus aiding us in understanding the learning process and selecting suitable models according to noisy degree of given data. For instance, we can pick the trained model close to the optimal curriculum for heavily noisy data or the one near the end of the centrality-Fil curve for datasets of little noise. In fact, this may be the most common way of using cluster curriculum.
In this paper, we do not investigate the cuusstclcurr learning for the multi-class case, e.g. the ImageNet dataset with BigGAN (Brock et al.] (2019). The cluster-curriullm learning of multiple classes is more complex than that we have already analyzed on the face and cat data. We leave this study for future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. In arXiv: 1701.0/77007777 2017.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. arXiv: 1811. 2018.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems (NeurIPS), 2006.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning (ICML), pp. 41-48, 2009.
David Berthelot, Thomas Schumm, and Luke Metz. BEGAN: boundary equilibrium generative
adversarial networks. arXiv:1703. 2017.
Ali Borji. Pros and cons of GAN evaluation measures. arXiv: (oXi::/802.03446, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations (ICLR), 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. In International Conference Learning Representations (ICLR), 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In
International Conference Learning Representations (ICLR), 2017.
David Donoho. nigh-dimeesional data analysis: The curses and blessings of dimensionality. In AMS Math Challenges Lecture, 2000.
Jeffrey Elman. Learning and development in neural networks: The importance starting small. Cognition, 48:781-799, 1993.
Tieliang Gong, Qian Zhao, Deyu Meng, and Zongben Xu. Why curriculum learning & self-paced learning work in big/noisy data: A theoretical perspective. Big Data & Information Analytics, 1 (1):111-127, 2016.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS). 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In arxiv:1704.00022, 2017.
Kaiming He, Xiangyu Zhang, Shaojing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 770-778, 2016.
Ari Heljakka, Arno Solin, and Juho Kannala. Pioneer networks: Progressively growing generative autoencoder. In arXivv:8077 03026, 2018.
Geoffrey E. Hinton and Ruslan R. Salakhutdinov. Reducing the dimensionality data with neural networks. Science, 3136578))5504-507, 2006.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv:/800 10729, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. Proceedings of the 6th International Conference on Learning Representations (ICIR), 2018a.

---

Under review as a conference paper at ICLR 2020
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 0xxiv:1812.04948, 2018b.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In arXiv:8807 1807. 0007003039, 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In Proceedings of the 2th International Conference on Learning Representations (ICLR), 2013.
Dimitrios Korkinof, Tobias Rijken, Michael O' O'Neill, Joseph Yearsley, Hugh Harvey, and Ben Glocker. High-resolution mammogram synthesis using progressive generative adversarial networks. In arXiv:1807.03401, 2018.
Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng. Unsupervised learning of hierarchical representations with convolutional deep belief networks. Proceedings of the 28th International Conference Machine Learning (ICML), 2011.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), 2015.
Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. Transactions on Pattern Analysis and Machine Intelligence, 29(9)114461162 2007.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. arXiv: 16fv:6611.000 1.100006, 2016.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? SEIV:18001.04006 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations (ICLR), 2018.
Mark Newman. Networks: An Introduction. Oxford, 1st edition, 2010.
Pierre- Yves Oudeyer, Frederic Kaplan, and Verena V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2):265- 286, 2007.
Ofir Press, Amir Bar, Ben Bogin, Jonathan Berant, and Lior Wolf. Language generation with recurrent generative adversarial networks without pre-training. arXiv:1706.01330 399,, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, In Proceedings of the 4th International Conference on Learning Representations (ICLR), 2016.
Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, and Aaron Courville. Adversarial generation natural language. urXiv:1705: 10929, 2017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML.) pp. 1530-1538, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML). pp. 1278-1286, 2014.
Steven Roman. Introduction Coding and Information Theory. Springer, 1st edition, 1996.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bemnsten,, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (ICCV,, 115(3): 21 1--22,, 2015.
10

---

Under review as a conference paper at ICLR 2020
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. (0l.::6600.04998, 2016.
Dietrich Stauffer and Ammon Aharony. Introduction To Percolation Theory. CRC Press, 2st edition, 1994.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv: (e0..000099999 2016a.
Aaron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In 1rXX.0/601.06770 2016b.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast high-fidelity speech synthesis. In arXiv: 10433, 2017.
Roman Vershynin. Biggh-Dimessioall Probability: An Introduction with Applications in Data Science. Cambridge University Press, 1st edition, 2018.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN:
Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv: 03365, 2015.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), 2014.
Deli Zhao and Xiaoou Tang. Cyclizing clusters via Zeta function of graph. In Advances in Neural Information Processing Systems (NeurIPS), 2008.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 2921-2929, 2016.
Will Zou and James McClelland. Progressive development of the number sense in deep neural network. In Anmual Conference of the Cognitive Science Society, 2013.
A APPENDIX
A.1 CENTRALITY MEASURE
The centrality or clustering coefficient pertaining to a cluster in data points or community in complex network a well-studied traditional topic in machine learning and complex systems. Here we introduce the graph-theoretic centrality for the utilization of cluster curriculum. Firstly, we construct directed graph (digraph) with K nearest neighbors by the method (Zhao & Tang 2008). The weighted adjacency matrix W of the digraph can be formed in this way: Wi, exp( (xp(-dffp / æ is one of the nearest neighbors x; and 0 otherwise, where de is the distance between æ; and 2 and a is free parameter.
The density of data points can be quantified with the stationary probability distribution of a Markov chain. For digraph built from data, the transition probability matrix can be derived by row normalization, say, P.j = W E, Way- Then the stationary probability u can be obtained by solving an eigenvalue problem
(5) where T denotes the matrix transpose. It is straightforward to know that u is the eigen-vector of PT corresponding to the largest eigenvalue (i.e. 1). is also defined kind of PageRank in many scenarios.
P Tu= u,
For density- based cluster currieulum, the centrality e coincides with the stationary probability Figure in the main context shows the plausibility of using the stationary probability distribution to quantify the data density.
11

---

Under review as a conference paper at ICLR 2020
A.2 THEOREM AND PROOF
Theorem 1. For a set C = {xilax E { n data points drawn from normal distribution (0, E). the ellipsoid Eas of confidence a is defined as x: E--x S Xa. where E has no zero eigenvalues and a E (0, I.. Let N(Ea) be the number of spheres of radius & packed in the ellipsoid Ex- Then we can establish
((E.)
Xo (
/det det (E)
Proof. As explained the main context, the ellipse equation with respect to the confidence a can be expressed by the following equation
Σーー
Σ
Suppose that At,..., Aa are the eigenvalues of Σ. Then equation
can be written as
21
Xa
(8)
Ad
Further, eliminating xa on the right side gives
2
1.
(9)
Xa Ad
Then we derive the length of semi-axis with respect to Xa. i..
Ti = Xa Az.
(10)
For d-dimensional ellipsoid E, the volume of E is
Vol(Ea) 2 d/2
2Td/22 2122--.0
(11)
where y; the leng of semi-axis of E and (-))) is the Gamma function. Substituting 110 into the above equation, we obtain the final formula of volume
Vol(E.) 2 4/2
(dd/2))
Xa
A
(12)
2 4/2
aT(/2) Xax
det (E).
(13)
Using the volume formula in (1I. is straightforward to get the volume of the packing sphere S
4d/2
Vol(S.) = (d/2) det (s2T).
(14)
By the definition of ((a),, we can write
((Ea) = # of spheres = Vol(Ea)/Vol(55
(15)
Xa
det (E).
(16)
We conclude the proof of the theorem.
12

---

Under review as a conference paper at ICLR 2020
A.3 PROCEDURES OF ALGORITHM AND ALGORITHM 2
Algorithm Cluster Curriculum for Generative Models
Require:
1: X, dataset containing m data points
2: Generat veeoodeeell 10. generative models
3: QualityScore(, metric for generative results
4: L, number of subsets
Ensure:
5: // Solve centralities
6: Cent ra ity(())
b section A.1
7: // Cluster curriculum
8: Get sorted data X according to descending order of
9: Divide X' to be {X ,,...,, X.
// Train generative models
11: for X,E (X K..,.,,,, X1) do
Initialize model parameters 0 randomly
13: 0 (GenerativeModelll. 0)
b P.g.g. GAN
14: Xo RROUX,
15: end for
16: // Search the optimal set
17: for 0, E . 01+1) do
18: Generate data X with model of parameter 0
19: 8; Qual i tyScore(A) X'
e.e.g FID
20: end for
21: i"
arg min 8;, I...., , ++11
22: Return the optimal model parameter 0,-
13

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Cluster Curriculum with Active Set
Require:
X. GenerativeMode10 QualityScore0 in Algorithm
!
2: [A], cardinality of active set A
Ensure:
3: // Solve contralities
4: c Cent ra lity(.))
section A.I
5: // Cluster curriculum
6: Get sorted data X' according to descending order of
7: Divide X be {X'o, Ri....,74 X1)
8: // Train generative models
9: Initialize model parameters 0 randomly
10: O, SeeneratieModellde
0)
e.g. GAN
for X,E {X 1X,,..,, X)) do
12:
Derive Ag by randomly sampling X'
Ao + UAUUR,
13:
14:
// Use pre-training model
0. Generati ieeeooaaiveMode veModel (Ao. 0;-1)
15:
Xo - (
16:
17: end for
18: // Search the optimal set
19: for 0, E {01 ..... 01+1} do
20: Generate data X with model of parameter O, by sampling prior
c.g. Gaussian b e.g. FID
21: 5; Qualuttttto. QualityScore( X)
22: end for
23: i"
arg min s;, = 1.... ,
24: Return the optimal model parameter 0.-
14


Reviews:
[{"review_id": "rkeN7KLoKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This well written paper presents an effective way to remove outliers for deep generative models, provided examples are ranked along their centrality. One question I have is how exactly this centrality is measured\n\nThe word ‘curriculum’ has become a terminology used to describe a wide variety of totally different algorithms. While the authors provide an excellent introduction to this diversity and clearly differentiate their own flavor of ‘cluster curriculum’, I am wondering if they would not have been better off by describing the proposed approach in terms of outlier removal, especially has it has very little in common with the original idea of curriculum, which is a learning progression designed by the teacher.\n\nNevertheless, the paper is very clearly written and reads well in the present form. I am especially impressed about the clarity of the rather technical part on percolation.\n\nIn terms of outlier removal, this could be interpreted as the following constructive algorithm:\n•\tExamples are ranked along their centrality measure \n•\tOne identify the critical point in percolation for a starting point where too many examples are removed\n•\tMore examples are added until a minimum is found on validation data\nThis process can be made faster as:\n•\tThe optimum can happen very soon after the critical point\n•\tOtherwise, an active set algorithm can be used to control the size of the training set.\n\nWhile my recent expertise has been more NLP than Vision, I think this algorithm is original, and could have a significant impact as it can be extended beyond GANs. The technical presentation is excellent.\n\nOne puzzling issue is the computation of the centrality measure: all I could find in Appendix A1. and A.3 is that it is directly measured on the raw image (RGB pixels?) by taking some distance, which I assume is Euclidean? My experience in image classification suggests such a distance is meaningless, so I may be missing something. I looked for a Github pointer, but none was provided.\n\nThe English is OK, but there are missing words and strange constructions:\n•\tPage 1:  “Deep generative models HAVE piqueD researchers’ interest in the past decade”\n•\tPage 4: “The training of many loops will lead to time-consuming (MISSING WORD)”\n\nIn particular, the usage of “the”, for instance in pages:\n•\t7 “Therefore, a fast learning strategy can be derived from THE percolation process.”\n•\t7 “Training may begin from the curriculum”\n•\t8 “Cluster curriculum is proposed for robust training of generative models.”\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1xYpE3w9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper states that training generative models is challenging when there are noisy data points in your training set. To address this, the authors propose to training methodology (or curricula)  where \"easier\" or more \"relevant\" points are presented first followed to the model followed by the less relevant ones. The relevance is determined used by calculating centrality on a graph constructed out of the data points. \n\nI lean towards rejecting the paper, primarily because 1) The authors have not provided evidence to claim the noisy data points makes training challenging or characterized anything about under how much noise the training breaks down. 2) Experimental evidence is not convincing  3) There are not of imprecise statements in the paper. \n\nThe authors claim (without citation) that training generative models in the presence of noisy data is challenging. How much noise are we talking about? If the entire training set is very noisy, maybe we don't have any hope to learn to generate clean samples, but if it's just a little bit of noise, maybe it's fine. I also understand that when the authors use the word \"noise\" they don't really mean noise,  but changes in view point etc. Some convincing demonstration that such characteristics of dataset adversely affects the training of generative models will be helpful (in addition to one passing sentence in the experiment section about it).\n\nI am not convinced by any of the experiments. More importantly, it seems like the proposed training curricula is in general valid for a lot of generative models. The experimental evidence is really not convincing. Very limited experiments is done on two datasets only using one particular GAN. To convincingly demonstrate that your training methodology is doing something non-trivial, you should show that this works on multiple generative models on multiple datasets and compare your performance (and visualize some generated samples) against just training blindly on the entire dataset. In addition to this, I don't understand the relevance of some of the results in the paper. For ex, what does Corollary 3 signify?\n\nSome other points:\n1. First para. \"non-trial' -> non-trivial\n2. First para: \"model collapse\" -> mode collapse\n3. 3.2 para 2. \"base set that guarantees a proper converge of generative models\". What do you mean by \"proper convergence\"?\n4. 3.2 para 2. \"moderate compared to m\". Again, what is moderate? \n5. Why do you use ResNet features (5.1) for distance? Why is this a reasonable or the best metric while computing your graph?\n6. \"We determine the parameter \\alpha of the edge weight by enforcing geometric mean of the weights to be 0.8\". It seems very arbitrary to me. Can you justify this choice?\n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a curriculum learning approach to training generative models like GANs. The reviewers had a number of questions and concerns related to specific details in the paper and experimental results. While the authors were able to address some of these concerns, the reviewers believe that further refinement is necessary before the paper is ready for publication.
Decision: Reject