Building the Building Blocks: From Simplification to Winning Trees in Genetic Programming

1 INTRODUCTION . Genetic programming ( GP ) is an evolutionary algorithm ( EA ) used to automatically generate computer programs to solve specific tasks ( Koza , 1992 ) . Up to now , GP has been used in diverse domains like image analysis ( Varniab et al. , 2020 ) , cybersecurity ( Picek et al. , 2018 ) , or scheduling ( Nguyen et al. , 2017 ) . What differentiates GP from other types of evolutionary algorithms is the fact that the solutions are commonly represented as expression trees . Naturally , due to a specific solution representation , appropriate variation operators need to be used . Unfortunately , while being very successful in diverse domains , GP also faces certain representation-specific problems . Indeed , the GP process will introduce redundancy and functionally useless sections of programs , commonly denoted as program bloat ( Koza , 1992 ; Blickle et al. , 1994 ) . Program bloat can cause various issues to the GP process . As a consequence of bloat , the evolution process can prematurely terminate and explore large parts of the search space that are not promising . Since the solutions with program bloat will be more complex than needed , the interpretability of solutions will suffer . Additionally , program bloat can be connected with some forms of building blocks , i.e. , parts of the solutions that can be combined to form even more fit solutions . As there are no universal ways to deal with program bloat , this problem is difficult . Commonly used methods are mostly based on applying various forms of simplification to reduce the complexity of a solution and eliminate any useless details . While such techniques can reduce program bloat , there are further questions to consider . For instance , simplification can be done at the end of the evolution process ( Hooper & Flann , 1996 ) but also during the evolution process ( Wong & Zhang , 2006 ; 2007 ) . As simplification will reduce redundancy , it can also have negative effects on the diversity of the obtained solutions and decrease the protection of useful building blocks within solutions from the destructive nature of the crossover operator ( Blickle et al. , 1994 ) . Numerous works have explored the issues of program bloat or how to simplify the GP solutions as discussed in Section 2 . What is more , multiple works have discussed the notion of building blocks for GP ( Langdon & Poli , 2002 ) , but a widely accepted definition for building blocks is still not agreed upon . A somewhat similar concept to building blocks has been recently discussed in the deep learning community . There , the Lottery Ticket Hypothesis for artificial neural networks discussed how “ ... a randomly initialized dense neural network contains a sub-network that is initialized such that , when trained in isolation , can match the test accuracy of the original network after training for at most the same number of iterations. ” , which surmises the existence of a “ winning ticket ” for neural networks ( Frankle & Carbin , 2019 ) . This opens up the question of the existence of something like winning tickets for GP . This work considers simplifications ( both exact and heuristic ones ) in tree-based GP applied for symbolic regression , and we show that building blocks play an important role . More precisely , our contributions are : • We present a new technique to conduct heuristic simplification for GP . We combine the heuristic simplification with the exact one , allowing us to reduce the tree sizes significantly . • We discuss a new concept in GP that we denote as winning trees . While there are similarities between winning trees and building blocks , the main differences stem from the fact that the winning trees have constraints on fitness and tree depth . We introduce the notion of winning trees to help us understand better how to guide the evolutionary process to highly-fit solutions . • We provide extensive experimental analysis to showcase that the evolutionary process when using winning trees can result in 1 ) more fit solutions , 2 ) faster convergence , and 3 ) a more robust evolutionary process . Additionally , we discuss the depth constraints for winning trees and show they work well only if of sufficient size , coupled with the appropriate solution depth . 2 RELATED WORK . Several studies have already dealt with the problem of simplifying the expression trees evolved by GP . These methods can roughly be divided into two groups , methods that simplify the expression without changing the behavior of the original expression and those which , by simplifying the expression , change its behavior compared to the original one . The former group consists mostly of methods that perform algebraic simplification . These methods apply predefined algebraic rules to replace a part of an expression with a simpler but equivalent one . In the literature , such methods are usually denoted as algebraic simplification or editing . Such a method was applied by ( Wong & Zhang , 2006 ) , and ( Zhang et al. , 2006 ) online during the evolution ; the authors defined 30 algebraic rules which were applied to the evolved trees in the population . The algebraic simplification method was combined with hashing , which helped to determine equivalent expressions ( Wong & Zhang , 2006 ; 2007 ) . These methods hash sub-expressions and determine whether certain expressions are equivalent or not based on the hash value . This helps the algebraic procedure to detect situations that would otherwise go undetected . The second group of simplification methods introduces modifications that can change the expression ’ s behavior . The idea is to detect parts of the expression that are not meaningful and remove them , although this might affect the numerical outcome . The main question is how to determine which parts of the expression should be eliminated . ( Kinzett et al. , 2008 ; 2010 ) proposed the application of a numerical simplification based on node contribution to reduce the complexity of the evolved trees . ( Song et al. , 2009 ) pruned trees by removing a child of an operator with two arguments if , when removed , the removal does not change the output of the node beyond a given threshold . ( Garcia-Almanza & Tsang , 2006 ) proposed a method for tree pruning applied to decision trees used to generate classification rules for financial stock markets . The method consists of extracting rules from a decision tree , evaluating them , and then pruning the rules that did not achieve the expected pruning threshold in the evaluation phase . A constant sub-tree pruning method was proposed by ( Rockett , 2020 ) . This pruning scheme replaces a subtree with a constant value . The constant is calculated as the expectation of the subtree , which is replaced on the entire dataset . Building blocks were first discussed in the context of genetic algorithms as short , low-order , and highly fit schemata that are sampled , recombined , and resampled to provide solutions of higher fitness ( Goldberg , 1989 ) . Intuitively , building blocks are small parts of a solution that can be formed into larger , more fit components through genetic operators . Defining a building block for GP is more difficult , where some common ( traditional ) interpretations are a subtree to a solution tree ( Altenberg , 1994 ) , a rooted subtree ( Rosca , 1997 ) , and a block of code ( Angeline & Kinnear , 1996 ) . In this work , we will follow the work from Langdon and Poli that define building blocks as parts of the solution that can be combined to form more fit solutions but without any constraints on fitness , length , or order of building blocks ( Langdon & Poli , 2002 ) . We note a recent work by O ’ Neill et al . where the authors discuss transfer learning for GP and how to create valuable material as building blocks dynamically ( O ’ Neill et al. , 2017 ) . Still , we consider this work orthogonal to ours as we do not aim at transferability while imposing other constraints . Since its definition , several studies which focus on the lottery ticket hypothesis have been made . For instance , ( Chen et al. , 2020 ) used observations from the work on the lottery ticket hypothesis ( Frankle & Carbin , 2019 ) to find trainable and transferable subnetworks in pre-trained BERT models , which are commonly used in natural language processing . ( Frankle et al. , 2020 ) used iterative magnitude pruning ( IMP ) to determine “ whether a neural network optimizes to the same , linearly connected minimum under different samples of SGD noise ” . 3 METHODOLOGY . To evaluate the winning trees hypothesis for GP , we need to consider the expressions that GP obtains . There is no guarantee that the expressions obtained by GP are exact ( i.e. , they represent the exact solution ) , and at the same time , they can contain many unnecessary subexpressions . This is often true when GP is executed for longer periods , and bloat starts occurring ( Luke & Panait , 2006 ) . To mitigate this problem , we introduce two methods for reducing the complexity of the expression trees . We denote these methods as simplification , for the exact procedure and pruning , for the heuristic procedure . 3.1 SIMPLIFICATION . The goal of the simplification approach is to simplify the expression without changing its behavior . Therefore , this approach replaces expressions with equivalent but simpler ones . One such example would be to replace the expression x+ 0 with x since the first one includes two redundant symbols . The rules for simplification in our experiments are based on common algebraic axioms and aim to reduce the size of the expression , such as the example above , and the complete list is given in Appendix A , Table 7 . The applied rules are not exhaustive , and more rules can be defined and used . However , the given set of rules has shown to be enough to simplify the obtained expressions . The simplification is performed using the Compare-Match algorithm proposed in ( Steyaert & Flajolet , 1983 ) . The algorithm includes two recursive functions , Match and Compare . The Match function iterates through the expression tree from the root node through all of its subtrees and determines whether it can find a pattern P in the given expression T . If a match is found , then P is replaced by a simpler expression defined for that pattern . Finding patterns is performed by the Compare function , which traverses through the expression tree and returns true if it could locate pattern P in the expression tree , otherwise it returns false . The Compare function applies an additional Map function to retrieve the information about the located pattern . The outline of the method is given in Appendix A , Algorithm 1 . 3.2 PRUNING . Unlike simplification , which is exact and does not change the behavior of the expression , the pruning method allows the expression to be simplified by removing parts of it , which will directly influence its output . This means that the expression does not necessarily need to have the same output value for a given input before and after pruning . This technique is performed so that certain elements in the expression are simply replaced by a neutral element for a given operator . For example , in the expression +XY , where X and Y represent arbitrary subexpressions , the expression can be pruned by removing either operand and replacing it with a neutral element for summation , which is 0 . Thus , it is possible to obtain either +X0 or +0Y . In that way , one part of the expression can be removed while keeping the rest of the expression fixed . Each binary operator has its neutral element , which for the summation and subtraction are 0 , whereas , for multiplication and division , they are equal to 1 . Unary operators do not have a neutral element as they represent a single expression . Thus , they are not considered in the pruning process . The removal of certain subexpressions can change the behavior of the entire expression . To observe how the modifications influence the expression ’ s behavior ( the semantics ) , we monitor the change in fitness between the original expression and the pruned one . If the fitness improves or stays the same , the change is accepted . Our approach also allows pruning to be performed even if it reduces the fitness compared to the original expression ; however , a limit is imposed on how much the simplified expression can be worse than the original one . The procedure that performs the pruning starts from the root node of the expression tree and traverses the tree . At each node , it replaces the current node with a corresponding neutral element . Then , the new expression is evaluated , and its fitness is compared to the fitness of the original expression . The new expression is accepted if it does not degrade the performance of the original expression beyond a certain degree . In our experiments , we set that the solution should not be degraded beyond 15 % of its original quality . Naturally , this parameter can be arbitrarily chosen to balance the complexity and quality of the expressions . If the change is not accepted , then the procedure continues the traversal through the expression tree . Otherwise , if the change was accepted , the procedure is restarted at the root node and traverses the tree again to check whether it is possible to perform additional simplifications . The procedure stops when both the entire expression tree is traversed , and no modification is performed . To the best of our knowledge , this kind of pruning approach for GP has not been applied before .

Summary: On a technical level, the paper proposes two methods for Genetic Programming (GP) tree pruning, as well as a method for a population _restart_ that relies on both pruning methods. The aim of such approach is that in the second, or subsequent, GP runs, the GP algorithm starts from with an already-somewhat-good initial population of individuals that would allow GP to reach better solutions. On a scientific level, authors try to draw analogies between their approach, and recent advances in the deep learning (DL) community, regarding the Lottery Ticket Hypothesis (LTH). The authors complement their draft with a set of experimental evaluations on different standard datasets.