SEGEN: SAMPLE-ENSEMBLE GENETIC EVOLUTIONARY NETWORK MODEL

1 INTRODUCTION . In recent years , deep learning , a rebranding of deep neural network research works , has achieved a remarkable success . The essence of deep learning is to compute the hierarchical feature representations of the observational data Goodfellow et al . ( 2016 ) ; LeCun et al . ( 2015 ) . With multiple hidden layers , the deep learning models have the capacity to capture very good projections from the input data space to the objective output space , whose outstanding performance has been widely illustrated in various applications , including speech and audio processing Deng et al . ( 2013 ) ; Hinton et al . ( 2012 ) , language modeling and processing Arisoy et al . ( 2012 ) ; Mnih & Hinton ( 2009 ) , information retrieval Hill ( 2012 ) ; Salakhutdinov & Hinton ( 2009 ) , objective recognition and computer vision LeCun et al . ( 2015 ) , as well as multimodal and multi-task learning Weston et al . ( 2010 ; 2011 ) . By this context so far , various kinds of deep learning models have been proposed already , including deep belief network Hinton et al . ( 2006 ) , deep Boltzmann machine Salakhutdinov & Hinton ( 2009 ) , deep neural network Jaeger ( 2002 ) ; Krizhevsky et al . ( 2012 ) and deep autoencoder model Vincent et al . ( 2010 ) . Meanwhile , deep learning models also suffer from several serious criticism due to their several severe disadvantages Zhou & Feng ( 2017a ) . Generally , learning and training deep learning models usually demands ( 1 ) a large amount of training data , ( 2 ) large and powerful computational facilities , ( 3 ) heavy parameter tuning costs , but lacks ( 4 ) theoretic explanation of the learning process and results . These disadvantages greatly hinder the application of deep learning models in many areas which can not meet the requirements or requests a clear interpretability of the learning performance . Due to these reasons , by this context so far , deep learning research and application works are mostly carried out within/via the collaboration with several big technical companies , but the models proposed by them ( involving hundreds of hidden layers , billions of parameters , and using a large cluster with thousands of server nodes Dean et al . ( 2012 ) ) can hardly be applied in other real-world applications . In this paper , we propose a brand new model , namely SEGEN ( Sample-Ensemble Genetic Evolutionary Network ) , which can work as an alternative approach to the deep learning models . Instead of building one single model with a deep architecture , SEGEN adopts a genetic-evolutionary learning strategy to train a group of unit models generations by generations . Here , the unit models can be either traditional machine learning models or deep learning models with a much “ narrower ” and “ shallower ” structure . Each unit model will be trained with a batch of training instances sampled form the dataset . By selecting the good unit models from each generation ( according to their performance on a validation set ) , SEGEN will evolve itself and create the next generation of unit modes with probabilistic genetic crossover and mutation , where the selection and crossover probabilities are highly dependent on their performance fitness evaluation . Finally , the learning results of the data instances will be effectively combined from each unit model via diffusive propagation and ensemble learning strategies . These terms and techniques mentioned here will be explained in great detail in Section 4 . Compared with the existing deep learning models , SEGEN have several great advantages , and we will illustrate them from both the bionics perspective and the computational perspective as follows . From the bionics perspective , SEGEN effectively models the evolution of creatures from generations to generations , where the creatures suitable for the environment will have a larger chance to survive and generate the offsprings . Meanwhile , the offsprings inheriting good genes from its parents will be likely to adapt to the environment as well . In the SEGEN model , each unit network model in generations can be treated as an independent creature , which will receive a different subsets of training instances and learn its own model variables . For the unit models suitable for the environment ( i.e. , achieving a good performance on a validation set ) , they will have a larger chance to generate their child models . The parent model achieving better performance will also have a greater chance to pass their variables to the child model . From the computational perspective , SEGEN requires far less data and resources , and also has a sound theoretic explanation of the learning process and results . The unit models in each generation of SEGEN are of a much simpler architecture , learning of which can be accomplished with much less training data , less computational resources and less hyper-parameter tuning efforts . In addition , the training dataset pool , model hyper-parameters are shared by the unit models , and the increase of generation size ( i.e. , unit model number in each generation ) or generation number ( i.e. , how many generation rounds will be needed ) will not increase the learning resources consumption . The relatively “ narrower ” and “ shallower ” structure of unit models will also significantly enhance the interpretability of the unit models training process as well as the learning results , especially if the unit models are the traditional non-deep learning models . Furthermore , the sound theoretical foundations of genetic algorithm and ensemble learning will also help explain the information inheritance through generations and result ensemble in SEGEN . In this paper , we will use network embedding problem Wang et al . ( 2016 ) ; Chang et al . ( 2015 ) ; Perozzi et al . ( 2014 ) ( applying autoencoder as the unit model ) as an example to illustrate the SEGEN model . Meanwhile , applications of SEGEN on other data categories ( e.g. , images and raw feature inputs ) with CNN and MLP as the unit model will also be provided in Section 5.3 . The following parts of this paper are organized as follows . The problem formulation is provided in Section 3 . Model SEGEN will be introduced in Section 4 , whose performance will be evaluated in Section 5 . Finally , Section 2 introduces the related works and we conclude this paper in Section 6 . 2 RELATED WORK . Deep Learning Research and Applications : The essence of deep learning is to compute hierarchical features or representations of the observational data Goodfellow et al . ( 2016 ) ; LeCun et al . ( 2015 ) . With the surge of deep learning research and applications in recent years , lots of research works have appeared to apply the deep learning methods , like deep belief network Hinton et al . ( 2006 ) , deep Boltzmann machine Salakhutdinov & Hinton ( 2009 ) , Deep neural network Jaeger ( 2002 ) ; Krizhevsky et al . ( 2012 ) and Deep autoencoder model Vincent et al . ( 2010 ) , in various applications , like speech and audio processing Deng et al . ( 2013 ) ; Hinton et al . ( 2012 ) , language modeling and processing Arisoy et al . ( 2012 ) ; Mnih & Hinton ( 2009 ) , information retrieval Hill ( 2012 ) ; Salakhutdinov & Hinton ( 2009 ) , objective recognition and computer vision LeCun et al . ( 2015 ) , as well as multimodal and multi-task learning Weston et al . ( 2010 ; 2011 ) . Network Embedding : Network embedding has become a very hot research problem recently , which can project a graphstructured data to the feature vector representations . In graphs , the relation can be treated as a translation of the entities , and many translation based embedding models have been proposed , like TransE Bordes et al . ( 2013 ) , TransH Wang et al . ( 2014 ) and TransR Lin et al . ( 2015 ) . In recent years , many network embedding works based on random walk model and deep learning models have been introduced , like Deepwalk Perozzi et al . ( 2014 ) , LINE Tang et al . ( 2015 ) , node2vec Grover & Leskovec ( 2016 ) , HNE Chang et al . ( 2015 ) and DNE Wang et al . ( 2016 ) . Perozzi et al . extends the word2vec model Mikolov et al . ( 2013 ) to the network scenario and introduce the Deepwalk algorithm Perozzi et al . ( 2014 ) . Tang et al . Tang et al . ( 2015 ) propose to embed the networks with LINE algorithm , which can preserve both the local and global network structures . Grover et al . Grover & Leskovec ( 2016 ) introduce a flexible notion of a node ’ s network neighborhood and design a biased random walk procedure to sample the neighbors . Chang et al . Chang et al . ( 2015 ) learn the embedding of networks involving text and image information . Chen et al . Chen & Sun ( 2016 ) introduce a task guided embedding model to learn the representations for the author identification problem . 3 PROBLEM FORMULATION . In this section , we will provide the definitions of several important terminologies , based on which we will define the network representation learning problem . 3.1 TERMINOLOGY DEFINITION . The SEGEN model will be illustrated based on the network representation learning problem in this paper , where the input is usually a large-sized network structured dataset . DEFINITION 1 ( Network Data ) : Formally , a network structured dataset can be represented as a graphG = ( V , E ) , where V denotes the node set and E contains the set of links among the nodes . In the real-world applications , lots of data can be modeled as networks . For instance , online social media can be represented as a network involving users as the nodes and social connections as the links ; e-commerce website can be denoted as a network with customer and products as the nodes , and purchase relation as the links ; academic bibliographical data can be modeled as a network containing papers , authors as the nodes , and write/cite relationships as the links . Given a large-sized input network data G = ( V , E ) , a group of sub-networks can be extracted from it , which can be formally represented as a sub-network set of G. DEFINITION 2 ( Sub-network Set ) : Based on a certain sampling strategy , we can represent the set of sampled subnetworks from network G as set G = { g1 , g2 , · · · , gm } of size m. Here , gi ∈ G denotes a sub-network of G , and it can be represented as gi = ( Vgi , Egi ) , where Vgi ⊆ V , Egi ⊆ E and G 6= gi . In Section 4 , we will introduce several different sampling strategies , which will be applied to obtained several different sub-network pools for unit model building and validation . 3.2 PROBLEM FORMULATION . Problem Statement : Based on the input network data G = ( V , E ) , the network representation learning problem aims at learning a mapping f : V → Rd to project each node from the network to a low-dimensional feature space . There usually exist some requirements on mapping f ( · ) , which should preserve the original network structure , i.e. , closer nodes should have close representations ; while disconnected nodes have different representations on the other hand .

Summary: This manuscript introduces SEGEN, a model based on Evolutionary Computation for building deep models. Interestingly, the authors define deep models in a different way. Instead of stacking several hidden layers one after the other (as in traditional deep learning models), SEGEN uses the idea of generations in evolutionary models (Genetic Algorithms or GA) and puts the unit models in the successive generations into layers, i.e., “evolutionary layer”. Each layer then performs the validation, selection, crossover, and mutation operations, as in GA. Another interesting point of the proposed method is that the choice of unit models in SEGEN can be traditional machine learning or recent deep learning models.