Learning concise representations for regression by evolving networks of trees

1 INTRODUCTION . The performance of a machine learning ( ML ) model depends primarily on the data representation used in training ( Bengio et al. , 2013 ) , and for this reason the representational capacity of neural networks ( NN ) is considered a central factor in their success in many applications ( Goodfellow et al. , 2016 ) . To date , there does not seem to be a consensus on how the architecture should be designed . As problems grow in complexity , the networks proposed to tackle them grow as well , leading to an intractable design space . One design approach is to tune network architectures through network hyperparameters using grid search or randomized search ( Bergstra & Bengio , 2012 ) with cross validation . Often some combination of hyperparameter tuning and manual design by expertise/intuition is done ( Goodfellow et al. , 2016 ) . Many approaches to network architecture search exist , including weight sharing ( Pham et al. , 2018 ) and reinforcement learning ( Zoph & Le , 2016 ) . Another potential solution explored in this work ( and others ) is to use population-based stochastic optimization ( SO ) methods , also known as metaheuristics ( Luke , 2013 ) . In SO , several candidate solutions are evaluated and varied over several iterations , and heuristics are used to select and update the candidate networks until the population produces a desirable architecture . The general approach has been studied at least since the late 80s in various forms ( Miller et al. , 1989 ; Yao , 1999 ; Stanley & Miikkulainen , 2002 ) for NN design , with several recent applications ( Real et al. , 2017 ; Jaderberg et al. , 2017 ; Conti et al. , 2017 ; Real , 2018 ) . In practice , the adequacy of the architecture is often dependent on conflicting objectives . For example , interpretability may be a central concern , because many researchers in the scientific community rely on ML models not only to provide predictions that match data from various processes , but to provide insight into the nature of the processes themselves . Approaches to interpretability can be roughly grouped into semantic and syntactic approaches . Semantic approaches encompass methods that attempt to elucidate the behavior of a model under various input conditions as a way of explanation ( e.g . ( Ribeiro et al. , 2016 ) ) . Syntactic methods instead focus on the development of concise models ∗Send correspondence to lacava @ upenn.edu that offer insight by virtue of their simplicity , in a similar vein to models built from first-principles ( e.g . ( Tibshirani , 1996 ; Schmidt & Lipson , 2009 ) ) . Akin to the latter group , our goal is to discover the simplest description of a process whose predictions generalize as well as possible . Good representations should also disentangle the factors of variation ( Bengio et al. , 2013 ) in the data , in order to ease model interpretation . Disentanglement implies functional modularity ; i.e. , sub-components of the network should encapsulate behaviors that model a sub-process of the task . In this sense , stochastic methods such as evolutionary computation ( EC ) appear well-motivated , as they are premised on the identification and propagation of building blocks of solutions ( Holland , 1975 ) . Experiments with EC applied to networks suggest it pressures networks to be modular ( Huizinga et al. , 2014 ; Kashtan & Alon , 2005 ) . Although the identification functional building blocks of solutions sounds ideal , we have no way of knowing a priori whether a given problem will admit the identification of building blocks of solutions via heuristic search ( Oppacher , 2014 ) . Our goal in this paper is thus to empirically assess the performance of several SO approaches in a system designed to produce intelligible representations from NN building blocks for regression . In Section 2 , we introduce a new method for optimizing representations that we call the feature engineering automation tool ( FEAT ) 1 . The purpose of this method is to optimize an archive of representations that characterize the trade-off between conciseness and accuracy among representations . Algorithmically , two aspects of the method distinguish FEAT from previous work . First , it represents the internal structure of each NN as a set of syntax trees , with the goal of improving the transparency of the resultant representations . Second , it uses weights learned via gradient descent to provide feedback to the variation process at a more granular level . We compare several multi-objective variants of this approach using EC and non-EC methods with different sets of objectives . We discuss related work in more detail in Section 3 . In section 4 and 5 , we describe and conduct an experiment that benchmarks FEAT against state-of-the-art ML methods on 100 open-source regression problems . Future work based on this analysis is discussed in Section 6 , and additional detailed results are provided in the Appendix . 2 METHODS . We are interested in the task of regression , for which the goal is to build a predictive model ŷ ( x ) using N paired examples T = { ( xi , yi ) } Ni=1 . The regression model ŷ ( x ) associates the inputs x ∈ Rd with a real-valued output y ∈ R . The goal of feature engineering / representation learning is to find a new representation of x via a m-dimensional feature mapping φ ( x ) : Rd → Rm , such that a model ŷ ( φ ( x ) ) outperforms the model ŷ ( x ) . We will assume that each predictor in x is scaled to zero mean , unit-variance . When applying a NN to a traditional ML task like regression or classification , a fixed NN architecture φ ( x , θ ) , parameterized by θ , is chosen and used to fit a model ŷ = φ ( x , θ ) T β̂ ( 1 ) In this case φ = [ φ1 . . . φm ] T is a NN representation with m nodes in the final hidden layer and a linear output layer with estimated coefficients β̂ = [ β̂1 . . . β̂m ] T . Typically the problem is then cast as a parameter optimization problem that minimizes a loss function via gradient descent . In order to tune the structure of the representation , we instead wish to solve the joint optimization problem φ∗ ( x , θ∗ ) = arg min φ∈S , θ N∑ i L ( yi , ŷi , φ , θ , β̂ ) ( 2 ) where φ̂ ( x , θ̂ ) is chosen to minimize a cost function L , with global optimum φ∗ ( x , θ∗ ) . ( L may depend on θ and β in the case of regularization . ) S is the space of possible representations realizable by the search procedure , and φ∗ is the true structure of the process underlying the data . The assumption of SO approaches such as evolutionary computation ( EC ) and simulated annealing ( SA ) 1http : //github.com/lacava/feat is that candidate solutions in S that are similar to each other , i.e . reachable in few mutations , are more likely to have similar costs than candidate solutions that are far apart . In these cases , despite Eqn . 2 being non-convex , S can be effectively searched by maintaining and updating a population of candidate representations that perform well . Multi-objective SO methods extend Eqn . 2 to minimizing additional cost functions ( Deb et al. , 2000 ) . 2.1 FEAT . FEAT uses a typical µ + λ evolutionary updating scheme , where µ = λ = P . The method optimizes a population of potential representations , N = { n1 . . . nP } , where n is an “ individual '' in the population , iterating through these steps : 1 . Fit a linear model ŷ = xT β̂ . Create an initial population N consisting of this initial representation , φ = x , along with P − 1 randomly generated representations that sample x proportionally to β̂ . 2 . While the stop criterion is not met : ( a ) Select parents P ⊆ N using a selection algorithm . ( b ) Apply variation operators to parents to generate P offspring O ; N = N ∪O ( c ) Reduce N to P individuals using a survival algorithm . 3 . Select and return n ∈ N with the lowest error on a hold-out validation set . Individuals are evaluated using an initial forward pass , after which each representation is used to fit a linear model ( Eqn . 1 ) using ridge regression ( Hoerl & Kennard , 1970 ) . The weights of the differentiable features in the representation are then updated using stochastic gradient descent . 2.1.1 REPRESENTATION . for modelling nonlinear relationships . 2.1.2 VARIATION . During variation , the representations are perturbed using a set of mutation and crossover methods . FEAT chooses among 6 variation operators that are as follows . Point mutation changes a node type to a random one with matching output type and arity . Insert mutation replaces a node with a randomly generated depth 1 subtree . Delete mutation removes a feature or replaces a sub-program with an input node , with equal probability . Insert/Delete dimension adds or removes a new feature . Sub-tree crossover replaces a sub-tree from one parent with the sub-tree of another parent . Dimension crossover swaps two features between parents . The exact probabilities of each variation operator will affect the performance of the algorithm , and others have proposed methods for adjusting these probabilities , e.g . Igel & Kreutz ( 2003 ) . For the purposes of our study , we use each operator with uniform probability . Feedback The use of an ML model to assess the fitness of each representation can be used to provide information about the elements of the representation that should be changed . In particular , we assume that programs in the representation with small coefficients are the best candidates for mutation and crossover . With this in mind , let n be an m-dimensional candidate representation with associated coefficients β ( n ) ∈ Rm . Let β̃i ( n ) = |βi|/ ∑m i |βi| . The probability of mutation for tree i in n is denoted PMi ( n ) , and defined as follows : si ( n ) = exp ( 1− β̃i ) / m∑ i exp ( 1− β̃i ) PMi ( n ) = fsi ( n ) + ( 1− f ) 1 m ( 3 ) The normalized coefficient magnitudes β̃ ∈ [ 0 , 1 ] are used to define softmax-normalized probabilities , s in Eqn . 3 . The smaller the coefficient , the higher the probability of mutation . The parameter f is used to control the amount of feedback used to weight the probabilities ; 1m in this case represents uniform probability . Among nodes in tree m , mutation occurs with uniform probability . This weighting could be extended for differentiable nodes by weighting the within-tree probabilities by the magnitude of the weights associated with each node . However we expect this would yield diminishing returns . 2.1.3 SELECTION AND SURVIVAL . The selection step selects among P parents those representations that will be used to generate offspring . Following variation , the population consists of 2P representations of parents and offspring . The survival step is used to reduce the population back to size P , at which point the generation is finished . In our initial study , we empirically compared five algorithms for selection and survival : 1 ) -lexicase selection ( Lex ) ( La Cava et al. , 2016b ) , 2 ) non-dominated sorting genetic algorithm ( NSGA2 ) ( Deb et al. , 2000 ) , 3 ) a novel hybrid algorithm using Lex for selection and NSGA2 for survival , 4 ) simulated annealing ( Kirkpatrick et al. , 1983 ) , and 5 ) random search . These comparisons are described in Appendix Section A.2 . We found that the hybrid algorithm ( 3 ) performed the best ; it is described below . Parents are selected using Lex . Lex was proposed for regression problems ( La Cava et al. , 2016b ; 2018a ) as an adaption of lexicase selection ( Spector , 2012 ) for continuous domains . Under -lexicase selection , parents are chosen by filtering the population according to randomized orderings of training samples with the threshold defined relative to the sample loss among the selection pool . This filtering strategy scales probability of selection for an individual based on the difficulty of the training cases the individual performs well on . Lex has shown strong performance among SR methods in recent tests , motivating our interest in studying it ( Orzechowski et al. , 2018 ) . The survival step for Lex just preserves offspring plus the best individual in the population . Survival is conducted using the survival sub-routine of NSGA2 , a popular strategy for multi-objective optimization ( Deb et al. , 2000 ) . NSGA2 applies preference for survival using Pareto dominance relations . An individual ( ni ) is said to dominate another ( nj ) if , for all objectives , ni performs at least as well as nj , and for at least one objective , ni strictly outperforms nj . The Pareto front is the set of individuals in N that are non-dominated in the population and thus represent optimal trade-offs between objectives found during search . Individuals are assigned a Pareto ranking that specifies the number of individuals that dominate them , thereby determining their proximity to the front . The survival step of NSGA2 begins by sorting the population according to their Pareto front ranking and choosing the lowest ranked individuals for survival . To break rank ties , NSGA2 assigns each individual a crowding distance measure , which quantifies an individual ’ s distance to its two adjacent neighbors in objective space . If a rank level does not completely fit in the survivor pool , individuals of that rank are sorted by highest crowding distance and added in order until P individuals are chosen .

Summary: This paper introduces a genetic algorithm that maintains an archive of representations that are iteratively evolved and selected by comparing validation error. Each representation is constructed as a syntax tree consists of elements that are common in neural network architectures. The experimental results showed that their algorithm is competitive to the state-of-the-art while achieving much smaller model size.