Curriculum Learning for Deep Generative Models with Clustering

1 INTRODUCTION . Deep generative models have piqued researchers ’ interest in the past decade . The fruitful progress has been achieved on this topic , such as auto-encoder ( Hinton & Salakhutdinov , 2006 ) and variational auto-encoder ( VAE ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ; Arjovsky et al. , 2017 ) , normalizing flow ( Rezende & Mohamed , 2015 ; Dinh et al. , 2015 ; 2017 ; Kingma & Dhariwal , 2018 ) , and autoregressive models ( van den Oord et al. , 2016b ; a ; 2017 ) . However , it is non-trivial to train a deep generative model that can converge to a proper minimum of the associated optimization . For example , GAN suffers non-stability , mode collapse , and generative distortion during training . Many insightful algorithms have been proposed to circumvent those issues , including feature engineering ( Salimans et al. , 2016 ) , various discrimination metrics ( Mao et al. , 2016 ; Arjovsky et al. , 2017 ; Berthelot et al. , 2017 ) , distinctive gradient penalties ( Gulrajani et al. , 2017 ; Mescheder et al. , 2018 ) , spectral normalization to discriminator ( Miyato et al. , 2018 ) , and orthogonal regularization to generator ( Brock et al. , 2019 ) . What is particularly of interest is that the breakthrough for GANs has been made with a simple technique of progressively growing neural networks of generators and discriminators from low-resolution images to high-resolution counterparts ( Karras et al. , 2018a ) . This kind of progressive growing also helps push the state of the arts to a new level by enabling StyleGAN to produce photo-realistic and detail-sharp results ( Karras et al. , 2018b ) , shedding new light on wide applications of GANs in solving real problems . This idea of progressive learning is actually a general manner of cognition process ( Elman , 1993 ; Oudeyer et al. , 2007 ) , which has been formally named curriculum learning in machine learning ( Bengio et al. , 2009 ) . The central topic of this paper is to explore a new curriculum for training deep generative models . To facilitate robust training of deep generative models with noisy data , we propose curriculum learning with clustering . The key contributions are listed as follows : • We first summarize four representative curricula for generative models , i.e . architecture ( generation capacity ) , semantics ( data content ) , dimension ( data space ) , and cluster ( data structure ) . Among these curricula , cluster curriculum is newly proposed in this paper . • Cluster curriculum is to treat data according to the centrality of each data point , which is pictorially illustrated and explained in detail . To foster large-scale learning , we devise the active set algorithm that only needs an active data subset of small fixed size for training . • The geometric principle is formulated to analyze hardness of noisy data and advantage of cluster curriculum . The geometry pertains to counting a small sphere packed in an ellipsoid , on which is based the percolation theory we use . The research on curriculum learning is diverse . Our work focuses on curricula that are closely related to data attributes , beyond which is not the scope we concern in this paper . 2 CURRICULUM LEARNING . Curriculum learning has been a basic learning approach to promoting performance of algorithms in machine learning . We quote the original words from the seminal paper ( Bengio et al. , 2009 ) as its definition : Curriculum learning . “ The basic idea is to start small , learn easier aspects of the task or easier sub-tasks , and then gradually increase the difficulty level ” according to pre-defined or self-learned curricula . From cognitive perspective , curriculum learning is common for human and animal learning when they interact with environments ( Elman , 1993 ) , which is the reason why it is natural as a learning rule for machine intelligence . The learning process of cognitive development is gradual and progressive ( Oudeyer et al. , 2007 ) . In practice , the design of curricula is task-dependent and data-dependent . Here we summarize the representative curricula that are developed for generative models . Architecture curriculum . The deep neural architecture itself can be viewed as a curriculum from the viewpoint of learning concepts ( Hinton & Salakhutdinov , 2006 ; Bengio et al. , 2006 ) or disentangling representations ( Lee et al. , 2011 ) . For example , the different layers decompose distinctive features of objects for recognition ( Lee et al. , 2011 ; Zeiler & Fergus , 2014 ; Zhou et al. , 2016 ) and generation ( Bau et al. , 2018 ) . Besides , Progressive growing of neural architectures is successfully exploited in GANs ( Karras et al. , 2018a ; Heljakka et al. , 2018 ; Korkinof et al. , 2018 ; Karras et al. , 2018b ) . Semantics curriculum . The most intuitive content for each datum is the semantic information that the datum conveys . The hardness of semantics determines the difficulty of learning knowledge from data . Therefore , the semantics can be a common curriculum . For instance , the environment for a game in deep reinforcement learning ( Justesen et al. , 2018 ) and the number sense of learning cognitive concepts with neural networks ( Zou & McClelland , 2013 ) can be such curricula . Dimension curriculum . The high dimension usually poses the difficulty of machine learning due to the curse of dimensionality ( Donoho , 2000 ) , in the sense that the amount of data points for learning grows exponentially with dimension of variables ( Vershynin , 2018 ) . Therefore , the algorithms are expected to be beneficial from growing dimensions . The effectiveness of dimension curriculum is evident from recent progress on deep generative models , such as ProGANs ( Karras et al. , 2018a ; b ) by gradually enlarging image resolution and language generation from short sequences to long sequences of more complexity ( Rajeswar et al. , 2017 ; Press et al. , 2017 ) . 3 CLUSTER CURRICULUM . For fitting distributions , dense data points are generally easier to handle than sparse data or outliers . To train generative models robustly , therefore , it is plausible to raise cluster curriculum , meaning that generative algorithms first learn from data points close to cluster centers and then with more data progressively approaching cluster boundaries . Thus the stream of feeding data points to models for curriculum learning is the process of clustering data points according to cluster centrality that will be explained in section 3.2 . The toy example in Figure 1 illustrates how to form cluster curriculum . 3.1 WHY CLUSTERS MATTER . The importance of clusters for data points is actually obvious from geometric point of view . The data sparsity in high-dimensional spaces causes the difficulty of fitting the underlying distribution of data points ( Vershynin , 2018 ) . So generative algorithms may be beneficial when proceeding from the local spaces where data points are relatively dense . Such data points form clusters that are generally informative subsets with respect to the entire dataset . In addition , clusters contain common regular patterns of data points , where generative models are easier to converge . What is most important is that noisy data points deteriorate performance of algorithms . For classification , the effectiveness of curriculum learning is theoretically proven to circumvent the negative influence of noisy data ( Gong et al. , 2016 ) . We will analyze this aspect for generative models with geometric facts . 3.2 GENERATIVE MODELS WITH CLUSTER CURRICULUM . With cluster curriculum , we are allowed to gradually learn generative models from dense clusters to cluster boundaries and finally to all data points . In this way , generative algorithms are capable of avoiding the direct harm of noise or outliers . To this end , we first need a measure called centrality that is the terminology in graph-based clustering . It quantifies the compactness of a cluster in data points or a community in complex networks ( Newman , 2010 ) . A large centrality implies that the associated data point is close to one of cluster centers . For easy reference , we provide the algorithm of the centrality we use in Appendix . For experiments in this paper , all the cluster curricula are constructed by the centrality of stationary probability distribution , i.e . the eigenvector corresponding to the largest eigenvalue of the transition probability matrix drawn from the data . To be specific , let c ∈ Rm denote the centrality vector of m data points . Namely , the i-th entry ci of c is the centrality of data point xi . Sorting c in descending order and adjusting the order of original data points accordingly give data points arranged by cluster centrality . Let −→ X = { −→ X 0 , −→ X 1 , . . . , −→ X l } signify the set of centrality-sorted data points , where −→ X 0 is the base set that contains sufficient data to attain convergent generative models , and the rest of −→ X is evenly divided into l subsets according to centrality order . In general , the number of data points in −→ X 0 is much less than m and determined according to X . Such division of −→ X 0 serves to efficiency of training , because we do not need to train models from a very small dataset . The cluster curriculum learning is carried out by incrementally feeding subsets in −→ X into generative algorithms . In other words , algorithms are successively trained on −→ X 0 ← −→ X 0 ∪ −→ X i after −→ X 0 , meaning that the curriculum for each round of training is accumulated with −→ X i . In order to determine the optimal curriculum −→ X 0 ∪ −→ X 1 ∪ · · · ∪ −→ X i , we need the aid of quality metric of generative models , such as Fréchet inception distance ( FID ) or sliced Wasserstein distance ( SWD ) ( Borji , 2018 ) . For generative models trained with each curriculum , we calculate the associated score si via the specified quality metric . The optimal curriculum for effective training can be identified by the minimal value for all si , where i = 1 , . . . , l + 1 . The interesting phenomenon of this score curve will be illustrated in the experiment . The minimum of score s is apparently metric-dependent . One can refer to ( Borji , 2018 ) for the review of evaluation metrics . In practice , we can opt one of reliable metrics to use or multiple metrics for decision-making of the optimal model . There are two ways of using the incremental subset −→ X i during training . One is that the parameters of models are re-randomized when the new data are used , the procedure of which is given in Algorithm 1 in Appendix . The other is that the parameters are fine-tuned based on pre-training of the previous model , which will be presented with a fast learning algorithm in the following section . 3.3 ACTIVE SET FOR SCALABLE TRAINING . To obtain the precise minimum of s , the cardinality of −→ X i needs to be set much smaller than m , meaning that l will be large even for a dataset of moderate scale . The training of many loops will be time-consuming . Here we propose the active set to address the issue , in the sense that for each loop of cluster curriculum , the generative models are always trained with a subset of a small fixed size instead of −→ X 0 ← −→ X 0 ∪ −→ X i whose size becomes incrementally large . To form the active set A , the subset−→ A0 of data points are randomly sampled from −→ X 0 to combine with −→ X i for the next loop , where | −→ A0| = |A| − | −→ X i| . For easy understanding , we illustrate the active set with toy example in Figure 2 . In this scenario , progressive pre-training must be applied , meaning that the update of model parameters for the current training is based on parameters of previous loop . The procedure of cluster curriculum with active set is detailed in Algorithm 2 in Appendix . The active set allows us to train generative models with a small dataset that is actively adapted , thereby significantly reducing the training time for large-scale data .

Summary: The paper states that training generative models is challenging when there are noisy data points in your training set. To address this, the authors propose to training methodology (or curricula)  where "easier" or more "relevant" points are presented first followed to the model followed by the less relevant ones. The relevance is determined used by calculating centrality on a graph constructed out of the data points.