Evolutionary perspective on model fine-tuning

1 INTRODUCTION . Evolution-inspired algorithms are all but new in machine learning . Introduced in 1966 , Simulated Evolution took the core components of the evolutionary processes as understood at the time - mutation and selection - and attempted simulate them in order to generate an artificial intelligence ( Fogel et al. , 1966 ) . Appearing a mere 15 years after Robbins and Monro introduced the Stochastic Gradient Descent ( SGD ) ( Robbins & Monro , 1951 ) , Evolutionary Algorithms ( EA ) captured the imagination of the AI and ML communities and were refined to more closely follow the understanding of natural evolution , such as chromosomes and crossover ( Schwefel , 1981 ) , resulting in perhaps the most well-known evolutionary optimization and search algorithm of the family - the Genetic Algorithm ( Goldberg , 1989 ) . However , this fascination slowly came to an end after in the late 70s and early 80 , as several groups discovered independently that in most cases , SGD on the artificial neural networks ( AANs ) could achieve same , if not better , results as EA but with significantly lower computational expenses ( Werbos , 1974 ; Parker , 1985 ; LeCun , 1985 ; Rumelhart et al. , 1986 ) . Shortly after , theoretical investigations in ANNs suggested that even simple architectures , such as multilayer feedforwards perceptrons , could work as universal approximators and be trained to approximate any measurable function , provided they had a sufficient number of layers , nodes per layer and non-linear activation functions ( Hornik et al. , 1989 ) . A rapid flurry of innovations showing that ANNs armed with SGD could be scaled up to perform better and could efficiently deal with a number of problems considered hard until then ( LeCun et al. , 2015 ) , switched the attention away from EA , until the superhuman performance of ImageNet on image classification tasks ( Krizhevsky et al. , 2012 ) made the ANN/SGD approach ubiquitous for computer vision , whereas the Transformer and its derivatives ( Vaswani et al. , 2017 ; Brown et al. , 2020 ; Fedus et al. , 2021 ) made ANN and SGD with inertia ubiquitous in Natural Language Processing . However , despite its impressive success , the SGD has a flaw . By it ’ s nature , it requires the manifold on which learning occurs to be differentiable . A limitation to which the Evolutionary Algorithms are not subject to , and one that is frequently encountered in conditions where multiple agents need to interact within a real or simulated environment ( as for instance described by Majumdar et al . ( 2020 ) ) . This advantage , combined with the advent of massively parallel computing that made the simultaneous evaluation of entire populations of models viable and led to a steady progress in EA application to ANNs , with the introduction and refinement of methods such as ESP ( Gomez & Miikkulainen , 1997 ) , CMA-ES ( Hansen & Ostermeier , 2001 ) , CoSynNE ( Gomez et al. , 2008 ) , NES ( Wierstra et al. , 2008 ) , reviewed in ( Hansen et al. , 2015 ) . In addition to that , neuroevolution has been used to optimize the topology and hyperparametrs of ANNs before training them with SGD , as introduced by Floreano et al . ( 2008 ) , or in combination with deep reinforcement learning ( RL ) ( Mnih et al. , 2015 ) - another approach to optimization in non-differentiable manifolds , as presented for instance by Fernando et al . ( 2017 ) . Overall , the intersection of approaches trying to combine EA with ANN is a rapidly expanding area , reviewed more in depth by Galván & Mooney ( 2020 ) . However , compared to SGD , our understanding of how well EA are able to perform optimization and explore the parameter space of the machine learning model based on the data is limited . Specifically , it is unclear how prone they are to being trapped in local minima ( Nguyen et al. , 2021 ; Nguyen , 2019 ; Jacot et al. , 2018 ) , finding flat minima ( Hochreiter & Schmidhuber , 1994 ; 1997 ; Goodfellow & Vinyals , 2015 ; Li et al. , 2018 ) , generalizing ( Dinh et al. , 2017 ; Zhang et al. , 2020 ) , memorizing ( Arpit et al. , 2017 ; Yun et al. , 2019 ; Zhang et al. , 2017 ; 2021 ) or interacting with ANNs architectures ( Balduzzi et al. , 2017 ; Li et al. , 2018 ) . While the question in itself would likely be harder to approach than in the case of SGD , just because of the sheer diversity of EAs , we approach the topic specifically in the context of model fine-tuning , or transfer learning ( Yosinski et al. , 2014 ; Oquab et al. , 2014 ; Bengio , 2012 ; Bengio et al. , 2011 ; Caruana , 1995 ) . The fine-tuning and transfer learning have grown in the importance over the last five years , given the exponential growth in the ANN models size , ( Sanh et al. , 2019 ; Brock et al. , 2019 ; Fedus et al. , 2021 ) , going hand in hand with the training dataset sizes and computing power required to train them Brown et al . ( 2020 ) . In this context , the from-scratch model training is prohibitively expensive for the vast majority of users , leading to the proliferation of model fine-tunes or transfers to closely related domains . As of the time of submission of this article , HuggingFace NLP model repository ( HuggingFace , 2020 ) counts over 1300 model fine-tunes , of which around 700 of BERT model alone ( Devlin et al. , 2019 ) . Within the context of fine-tuning models or Tranfer Learning ( Bozinovski & Fulgosi , 1976 ; Pratt , 1992 ; Pan & Yang , 2010 ) , we show that thanks to very general results on the limit distributions derived by Gnedenko et al . ( 1968 ) and more generally Fréchet ( 1927 ) ; Fisher & Tippett ( 1928 ) ; Mises ( 1936 ) ; Gnedenko ( 1943 ) , we can establish a direct equivalence between the SGD and the Fisher Geometric Model of Evolution ( FGM ) ( Fisher , 1930 ) in the Orr-Gillespie formalization - a fairly large family of EA commonly used in theory of evolution ( Tenaillon , 2014 ; Joyce et al. , 2008 ; Orr , 2005 ) . By building upon this direct equivalence as well as some heuristics with regards to how much the FGM in the Orr-Gillespie formalization can be generalized outside its formal description ( Joyce et al. , 2008 ; Rokyta et al. , 2008 ) , we suggest additional insights on some features of ANNs trained with SGD , notably with regards to generalization and minima flatness , as well as implications for the EA family with which we drew the equivalence . We then offer a number of hypotheses applicable to ANNs trained with SGD , based on behaviors observed in biological system undergoing natural evolution . 1.1 FISHER GEOMETRIC MODEL IN THE ORR-GILLESPIE FORMALIZATION . The Fisher Geometric Model was introduced by sir Ronald Fisher as a cornerstone in his effort to unify the theory of evolution and reconcile Biometricists , convinced of the gradual evolution as presented by Darwin , and Geneticists , convinced of combinatorial genetic inheritance presented by Mendel ( for a detailed review see Orr ( 2005 ) ) . The elegance of the model consisted in suggesting that the fitness , rather than being absolute , was a result of a mach between the organism and its environment , mediated by a set of traits that cold be adjusted to enable such an adaptation - aka a phenotypic space . Within this space , environment allowed the organism to achieve a maximal fitness in a single point , with the fitness gradually decaying as the organism moved further from the optimum . Gene alleles could either have a drastic effect on the phenotypic space , in which case their inheritance was Mendelian , or small , in which mixing of alleles from parents provided an illusion of gradual change ( Fisher , 1930 ) . The resulting visual aid , presenting a stochastic walk towards the optimum on the fitness landscape , driven by the successive mutations and selections , is perhaps the most iconic aspect of the Fisher Geometric model and was declined for their own needs by fields with their own version of greedy exploration of fitness ( or loss ) landscapes . However simplicity did not seem to bode well with the biological reality . The slow and gradual adaptation suggested by the model was at odds with fossil records ( Gould & Lewontin , 1979 ) and couldn ’ t be easily mapped to the recently discovered biological reality of the almost-binary-tape DNA sequence controlling the biological machinery of organisms and radically differences in phenotypes , back then almost always tracked to modifications in a single gene . Most importantly , the general and statistical nature of the FGM provided little insight into which genes were specifically under selection in organisms for different phenotypes These shortcomings of the classical Fisher model , associated with a general change of perspective on the evolutionary genetics , led to a transition towards sequence substitution - oriented evolution . One of the first post-FGM models was the purely neutral drift model , where all the genetic and phenotype variation was a product of randomness and a number of successive bottleneck effects ( Kimura & Crow , 1964 ; Kimura , 1968 ) . With hard experimental evidence that some mutations were indeed advantageous and a large number was indeed deleterious , the model was refined to the nearneutral theory of evolution , where the vast majority of mutations had no effect on the fitness of the organism , a significant part had a deleterious effect , of which only a small fraction had a significant impact , and only a vanishingly small number of mutations were beneficial ( Ohta , 1992 ) . However , the real breakthrough that allowed the FGM to be resurrected occurred through a paradigm shift in theoretical biology . First , a realization that fitness is environment-dependent . In other terms , organism that are highly adapted to their environment are subject to no evolutionary pressure and can remain unchangend for hundrends of millions of years , like the customary example of the horseshoe crab . An evolution will start only if a shift in fitness peak occurs , usually accompagniated by the initial population contraction allowing a rapid adaptive burst to occur ( Lande , 1986 ) , sometimes by leveraging pre-existing diversity in the genetic code space accumulated by neutral drift ( Kauffman , 1969 ; Kauffman & Johnsen , 1991 ) . Second , by leveraging recent results from limit distribution - specifically the Fisher-Tippet-Gnedenko ( Fréchet , 1927 ; Fisher & Tippett , 1928 ; Mises , 1936 ; Gnedenko , 1943 ) , closely realed to the generalization of the Central limit theorem by Gnedenko et al . ( 1968 ) , Gillespie ( 1983 ; 1984 ) established that if the adaptative burst occurred under high selection/low mutation condition , starting from one already fit organism , the iterative steps in the genetic code space towards the most adapted genetic code would increase the fitness according to the Gumbel limit distribution . Consequently , Orr ( 2002 ; 2006 ) showed that the distribution of fitness of steps in the phenotypic space in the FGM model belonged to the same bassin of attraction as the distribution of fitness of steps in the genetic code space . In turn , it meant that the FGM could be used as a convenient visual model of evolutionary processes occurring in the genetic codes space , with concepts such as phenotypic dimension or fitness peak flatness becoming directly interpretable . Follow-up work by Joyce et al . ( 2008 ) showed that the conclusions reached for the distributions in the Gumbel limit distribution bassin of attraction still mostly held in the vicinity of it , in the bassins of attraction of Weibull and Fréchnet distributions . Remarkably , the Gillespie-Fisher model of evolution is in no way linked to biological genetic codes specifically . It is applicable to any finite codes for which a fitness ( or conversely loss ) is defined and that are being iterative modified by a greedy search algorithm , starting from a code with already a high fitness value . Fine-tuning real-world ANNs through SGDs is an instance of such code space search .

Summary: The paper presents the foundation for a new class of Evolution Strategies inspired from the work in biology of Gillspie and Orr published in the 1980s. Particularly intriguing are an equivalence with SGD (but limited to problems with the same assumptions and requirements of SGD) and heuristics to accelerate convergence using Transfer Learning. The paper however lacks any experimental validation of the theoretical results proposed.