Evolutionary perspective on model fine-tuning
Authors: Andrei Kucharavy
Ljiljana Dolamic
Rachid Guerraoui
Venue: ICLR 2022 Submitted (2022)
Keywords: Evolutionary algorithms
stochastic gradient descent
fine-tuning

Abstract: Be it in natural language generation or in the image generation, massive performances gains have been achieved in the last years. While a substantial part of these advances can be attributed to improvement in machine learning architectures, an important role has also been played by the ever-increasing parameter number of machine learning models, which made from-scratch retraining of the models prohibitively expensive for a large number of users.
In response to that, model fine-tuning - starting with an already good model and further training it on the data relevant to a new, related problem, gained in popularity. This fine-tuning is formally similar to the natural evolution of genetic codes in response to shifting environment. 
Here, we formalize this similarity in the framework of Fisher Geometric model and extreme value theory and present a set of tricks used by naturally evolving organisms to accelerate their adaptation, applicable to model fine-tuning.

Under review as a conference paper at ICLR 2022
EVOLUTIONARY PERSPECTIVE ON MODEL FINE - TUNING
Anonymous authors Paper under double-bling review
ABSTRACT
Be it in natural language generation or in the image generation, massive perfor- mances gains have been achieved in the last years. While substantial part of these advances can be attributed to improvement in machine learning architectures, an important role has also been played by the ever-increasing parameter number of machine learning models, which made from-scratch retraining of the models pro- hibitively expensive for a large number of users. In response to that, Transfer Learning (TL) starting with an already good model and further training it the data relevant to a new, related problem, gained in popularity. TL is formally sim- ilar to the natural evolution of genetic codes in response to shifting environment. Our core contribution, presented in this paper, is to define a class evolutionary algorithms- Gillespie-Orr EA (GO-EA) and prove that they equívalent in the limit to stochastic gradient descent (SGD). Based on this equivalence we present a number of tricks used y naturally evolving organisms to accelerate their adapta- tion, applicable to TL, as well as set of hypotheses as to properties of artificial neural networks trained with SGD and GO-EA, resulting from such equivalence.
INTRODUCTION
Exolution-inspired algorithms are all but new in machine learning. Introduced in 1966, Simulated Evolution took the core components of the evolutionary processes as understood at the time muta- tion and selection and attempted simulate them order to generate an artificial intelligence (Fogel et al. 1966). Appearing mere 15 years after Robbins and Monro introduced the Stochastic Gradient Descent (SGD) (Robbins & Monro, 195T), Evolutionary Algorithms (EA) captured the imagination of the Al and MI. communities and were refined to more closely follow the understanding of natu- ral evolution, such as chromosomes and crossover (Schwefel 1981), resulting in perhaps the most well-known evolutionary optimization and search algorithm of the family the Genetic Algorithm Goldberg, 1989).
However, this fascination slowly came to end after in the late 70s and early 80, several groups discovered independently that in most cases, SGD on the artificial neural networks (AANs) could achieve same, if not better, results as EA but with significantly lower computational expenses (Wer- bos 1974] Parker 1985] LeCun 1985) Rumelhart et al. 1986). Shortly after, theoretical inves- tigations in ANNs suggested that even simple architectures, such multilayer feedforwards per- ceptrons, could work as universal approximators and be trained to approximate any measurable function, provided they had a sufficient number of layers, nodes per layer and non-linear activa- tion functions (Hornik et al. 1989). A rapid flurry of innovations showing that ANNs armed with SGD could be scaled up to perform better and could efficiently deal with a number of problems considered hard until then (LeCCnn et al., 2015). switched the attention away from EA, until the su- perhuman performance of ImageNet on image classification tasks (Krizhevsky et al., (012) made the ANN/SGD approach ubiquitous for computer vision, whereas the Transformer and its deriva- tives Vaswwan et al. 2017: Brown et al. 22200 Fedus et al. (202I) made ANN and SGD with inertia ubiquitous in Natural Language Processing.
However, despite its impressive success, the SGD has a flaw. By it's nature, it requires the manifold on which learning occurs to be differentiable. A limitation to which the Evolutionary Algorithms are not subject to, and one that is frequently encountered in conditions where multiple agents need to interact within real simulated environment (as for instance described by Majumdar et al.

---

Under review as a conference paper at ICLR 2022
2020)). ) This advantage, combined with the advent of massively parallel computing that made the simultaneous evaluation of entire populations of models viable and led to a steady progress in EA application to ANNs, with the introduction and refinement of methods such as ESP (Gomez & Mikkulainen 1997), CMA-ES (Hansen & Ostermeier (001), CoSynNE (Giomez et al. 2008), NES (Wierstra al. 2008), reviewed in (Hansen al.] 2015). In addition to that, neuroevolution has been used to optimize the topology and hyperparametrs of ANNS before training them with SGD, as introduced by Floreann et al. (2008), or in combination with deep reinforcement learning (RL) (Mnih et al., 2015) another approach to optimization in oondiffeerentablee manifolds, as presented for instance by Fernando et al. (2017). Overall, the intersection of approaches trying to combine EA with ANN is rapidly expanding area, reviewed more in depth by Galván & Mooney (2020).
However, compared to SGD, our understanding of how well EA are able to perform optimization and explore the parameter space of the machine learning model based on the data is limited. Specifically, unclear how prone they are to being trapped local minima (Nguyen et [ guyenn
2019 Jacot et al. 2018). finding flat minima (Hochreiter & Schmidhuber] 1994, 1997 Goodfellow
& Vinyals 2015: Li et al. 2018), generalizing (Dinh et al. [017] Zhang et al., 2020), memorizing
Arpit | 2017: Yun et al. [2019] Zhang et al. 2017 2021 or interacting with ANNs architectures
Balduzzi et al. [017]] Li et al. (0118).
While the question in itself would likely be harder approach than in the case of SGD, because of the sheer diversity of EAs, we approach the topic specifically in the context of model fine-tuning,
or transfer learning (Yosinski et al. 2014 Oquab al.] 2014] [Bengio| 2012 |Bengio et al.) 2011
Caruana 1995). The fine-tuning and transfer learning have grown in the importance over the last five years, given the exponential growth in the ANN models size, (Sanh al. 2019 Brock et al. 2019
Fedus et al. (02T), going hand in hand with the training dataset sizes and computing power required to train them Brown et al. (2020). In this context, the serathh model training is prohibitively expensive for the vast majority of users, leading to the proliferation of model fine-tumes or transfers to closely related domains. As of the time of submission of this article, HuggingFace NL.P model repository (HuggingFace 2020) counts over 1300 model fine-tunes, of which around 700 of BERT model alone (Devlin et al. 2019).
Within the context of fine-tuning models or Tranfer Learning (Bozinovski & Fulgosi 1976 Pratt 1992 Pan & Yang 2010), we show that thanks to very general results on the limit distributions derived by Gnedenko et al. (1968) and more generally |Frcchet (927);; Fisher & Tippett 19228;
Mises 1936); Gnnedenko (1943), we can establish a direct equivalence between the SGD and the
Fisher Geometric Model of Evolution (FGM) (Fisher) 1930) in the Orr-Gillespie formalization
fairly large family of EA commonly used in theory of evolution (Tenaillon 2014 Joyce et al. 2008
Orr 2005).
By building upon this direct equivalence as well as some heuristics with regards to how much the FGM in the Ort-Gillespie formalization can be generalized outside its formal description (Joyce) et al. 2008 Rokyta el al., 2008), we suggest additional insights on some features of ANNs trained with SGD, notably with regards to generalization and minima flatness, as well as implications for the EA family with which we drew the equivalence. We then offer a number of hypotheses applicable to ANNs trained with SGD, based on behaviors observed in biological system undergoing natural evolution.
1.1 FISHER GEOMETRIC MODEL IN THE ORR- GRR-GILLESPIE FORMALIZATION
The Fisher Geometric Model was introduced by sir Ronald Fisher as a cornerstone in his effort to unify the theory of evolution and reconcile Biometricists, convinced of the gradual evolution as presented by Darwin, and Geneticists, convinced of combinatorial genetic inheritance presented by Mendel (for a detailed review see Orr (005)). The elegance of the model consisted suggesting that the fitness, rather than being absolute, was a result of mach between the organism and its environment, mediated by a set of traits that cold be adjusted to enable such an adaptation aka phenotypic space. Within this space, environment allowed the organism to achieve maximal fitness in single point, with the fitness gradually decaying as the organism moved further from the optimum. Gene alleles could either have drastic effect on the phenotypic space, in which case their inheritance was Mendelian, or small, in which mixing of alleles from parents provided an illusion of gradual change (Fisher 1930
2

---

Under review as a conference paper at ICLR 2022
The resulting visual aid, presenting stochastic walk towards the optimum the fitness landscape, driven by the successive mutations and selections, is perhaps the most iconic aspect of the Fisher Geometric model and was declined for their needs by fields with their own version of greedy exploration of fitness (or loss) landscapes.
However simplicity did not seem bode well with the biological reality. The slow and gradual adaptation suggested by the model was at odds with fossil records (Gould & Lewontin (979) and couldn't be easily mapped to the recently discovered biological reality of the almost-binary-tape DNA sequence controlling the biological machinery of organisms and radically differences in phe- notypes. back then almost always tracked to modifications in single gene. Most importantly, the general and statistical nature of the FGM provided little insight into which genes were specifically under selection in organisms for different phenotypes
These shortcomings of the classical Fisher model, associated with a general change of perspective on the evolutionary genetics, led to a transition towards sequence substitution oriented evolution. One of the first post-FGM models was the purely neutral drift model, where all the genetic and phenotype variation was a product of randomness and a number of successive bottleneck effects (Kimura& & Crow 1964; Kimura 1968). With hard experimental evidence that some mutations were indeed advantageous and a large number was indeed deleterious, the model was refined to the near- neutral theory of evolution, where the vast majority of mutations had no effect the fitness of the organism, a significant part had a deleterious effect, of which only a small fraction had significant impact, and only vanishingly small number of mutations were beneficial (Ohta 1992).
However, the real breakthrough that allowed the FGM to be resurrected occurred through a paradigm shift in theoretical biology. First, a realization that fitness is environmenntdeeeeddent. In other terms, organism that are highly adapted to their environment are subject to no evolutionary pressure and can remain unchangend for hundrends of millions of years, like the customary example of the horseshoe crab. An evolution will start only if a shift in fitness peak occurs, usually accompagniated by the initial population contraction allowing rapid adaptive burst to occur (Lande) 1986), sometimes by leveraging pre-existing diversity in the genetic code space accumulated by neutral drift (Kauffman) 1969 Kauffman & Johnsen 199T). Second, by leveraging recent results from limit distribution specifically the Finteee (Fréchet, 1927 Fisher & Tippetl 1928; Mises] [936] Gnedenko 1943). closely realed to the generalization of the Central limit theorem by Gnedenkoo it al. 1968), Gillespie (1983) 1984) established that if the adaptative burst occurred under high selection/low mutation condition, starting from one already fit organism, the iterative steps in the genetic code space towards the most adapted genetic code would increase the fitness according to the Gumbel limit distribution. Consequently, Orr (2002) 2006) showed that the distribution of fitness of steps in the phenotypic space in the FGM model belonged to the same bassin of attraction as the distribution of fitness of steps in the genetic code space. In turn, it meant that the FGM could be used as a convenient visual model of evolutionary processes occurring in the genetic codes space, with concepts such as phenotypic dimension or fitness peak flatness becoming directly interpretable. Follow-up work by Joyce al.) (2008) showed that the conclusions reached for the distributions in the Gumbel limit distribution bassin of attraction still mostly held in the vicinity of it, in the bassins of attraction of Weibull and Fréchnet distributions.
Remarkably, the Gillespie- Fisher model of evolution is in no way linked to biological genetic codes specifically. It is applicable to any finite codes for which a fitness (or conversely loss) is defined and that are being iterative modified by greedy search algorithm, starting from a code with already high fitness value.
Fine-tuning real-world ANNs through SGDs is an instance of such code space search.
CENTRAL RESULTS
2.1
GILLESPIE- -OR EVOLUTIONARY ALGORITHM
In their formulation of the evolutionary process, Gillespie and Orr make three fundamental assump- tions to make their model analytically tractable. Specifically:
Haploid populations (single code evaluated for fitness)
3

---

Under review as a conference paper at ICLR 2022
Under high selection (Ns > 1)
In the low mutation limit (Ny <
Where N is the population size, is typical selection coefficient and μ is the per-site mutation rate.
The main purpose of those assumptions to ensure that a new advantageous mutation swipes [ through the population entirely upon appearance, before a next advantagcous mutation can emerge and ensuring that a deleterious mutation never appears at the same time as an advantageous one.
While covering an important class of biological questions, such drug resistance in cancers and bacteria Tenaillon 2014), this formulation is restrictive from the population genetics point of view. It is, however, perfectly adapted for the ANN neuroevolution, whenever after mutation round only the highest fitness (conversely lowest loss) parameter is retained and no parameter mixing occurs. We will hence refer to such an Evolutionary Algorithm Gillespie-Ore Orr Evolutionary Algorithm (GO-EA)
2.2 SGD EQUIVALENCE TO THE GILLESPIE-ORR EA FOR SUFFICIENTLY LOW LEARNING RATE
Let (-(-) be a neural network parametrized by 0, that maps inputs X of the form {xi 1M e Zn xdx xM to outputs Y of the form [y:]M, Z XdyxM where Z₂ {0, 1), d2 and dy respectively the dimensionalitity of x and y: nl and My respectively the binary code length required to describe a single component of the vectors of and y and M the maximum number of inputs the network can encounter, with potentially M = inf.
Let Le be the fitness function associated fe on the X and Y. A priory, L is inaccessible, given requires an evaluation on all the possible input-output pairs. However, it can estimated with a finite sample of imputs and outputs Xsamp. Y samp. giving us an Lelx.amp.YY. kamp''
Let O be a greedy optimization process, such that (0)) 0', with a rewrite capacity d. such that d, where p E N and Lo (xumpr samp Lav X xampp, Y Jamy. for any 0" such that
10' 0 pp 10" 0 < d.
Greedy optimization processes SGD and GO-EA are almost surely equivalent in the limit of SGD learning rate 1 0 and GO-EA neighbourhood sample population N + inf with GO-EA rewrite capacity d = I, up to a saddle point.
can perform the Taylor expansion in the neighbourhood of 0 of L Y xamp' As d + O, only the first order terms of the expansion remain, meaning that gradient descent of the loss function ( (- with step of I will lead to the optimum within the d ball around O. Given GO-EA has an infinite population, it will find the same optimum within the ball d. In case all the first order terms are nill, either the greedy optimization algorithm acheived a local minimum, in which case both SGD and GO-EA will stay put, or it is located in local saddle point, in which case the SGD will acheive no movement and GO-EA will move to the hightest fitness point within of O. In real conditions, the noise level provided by the sampling on the fitness function would lead the probability of the saddle point vanish, if the samples on which 7 evaluated are different.
Since the SGD is applicable, Xxamp.
In order to achieve this result, we had introduce the rewrite distance on the model parameters 0, which operates on norm and seemingly is incompatible with the single-mutation edits with which GO-EA operates in the context of genetic codes biological organism. This not the case. The parametrization 0 combined with the ANN architecture is just one of the many possible ways to encode the model f-(-), and certain that more compact and efficient codings exist, where the transition to local minimum would correspond to single code character change.
In population genetics and theory of evolution, an allele is said to swipe through the population when it prevalence increases until every single individual in the population has it. At this point, it is said to have been fixated in the population
4

---

Under review as a conference paper at ICLR 2022
2.3 PROBABILITY OF FINDING BETTER PARAMETERS DURING FINE-TUNING WITH GILLESPIE- ORR EA
In the context of fine tuning, we expect to start off a with a model fe, (.) parametrized so that already performs well on all the sample tests drawn from the distribution it used to train with
aka v(X Y samp) C X Y, P(Co, X.amp, Xampp maxe Lo X.amp. Yoomp)
Formally, fine-tuning consists in finding a new transfer parametrization Or. so that V(X Y (amp)) C X U x U Y., P(Corlx,,mm samp maxg Lo X Campe Yoamy) ~ 1, where the X' and Y' new domains application the model.
Assuming | >> |X| and |Y| >> YY (otherwise fine-tuning would be equivalent to model re-training), the model is already performing well the fine-tuned model and the vast major- ity of the parameters within rewrite capacity of Og would be deleterious or neutral, meaning that the parametrizations offering improvement would be distributed according to the general- ized Pareto distribution (Pickands, 1975| Joyce et al.) 22088), which in the case of Gumbel do- main of attraction would result in an exponential distribution of fitnesses & ($1:-.., 8{-1) where Sy = Le, Ixoop.. Y kamp' the jth best parametrization of of better parametrizations and a probability to reach the better parametrization Oy of rank j in the neighbourhood from parametrization 0, of the rank of Pa.j(8)
In other terms, with finite populations, GO-EA sampling the parametrization neighborhood of the current optimum 0, will find advantageous model code rewrites with the probability that's 's in reverse exponential probability of the difference between the loss associated to 0, and smallest possible loss within the edit distance budget.
While formally proven for the distributions the Gumbel domain of attraction, this results has been shown to hold as well in the adjacent domains of attraction of the Weibull and Frechnet limit distributions, although the behavior further away from the Gumbel domain might differ radically, leading to all ranks for fitness being equally likely to be picked up in the limit of Weibull bassin of attraction and only the lowest rank distribution being reachable by neighborhood sampling in the limit case of Frechnet bassin of attraction (reviewed in depth by Poyce al. (2008).
Unfortunately, the specific size of the sampling population needed to sample at least one advan- tageous parameter within the rewrite capacity is directly connected the effective latent dimension of the model by analogy with the phenotypic space of the FGM. While we will discuss potential strategies estimate it in the section 3, it is not directly accessible.
3 IMPLICATIONS OF CENTRAL RESULTS
3.1 DIRECT IMPLICATIONS
3.1.1 ROBUSTNESS OF THE GILLESPIE- -OR EVOLUTIONARY ALGORITHM
A substantial amount of research has been invested to better understand how SGD interacts with ANNs architecture, managing to robustly find parametrizations for the ANNs that avoid local min- ima and provide reasonable noise-resistance and generalization capabilities.
To our knowledge, such results were entirely absent for the Evolutionary Algorithms until now. Thanks to the results in the 2.2, we can claim that in case of differentiable loss landscape, Gillespie-Orr Evolutionary Algorithm is equivalent to SGD in the limit of low learning rate and high sampling population size. This means that all the results previously shown for SGD are valid for GO-EA whenever this approximation holds.
3.1.2 COMPUTATIONAL ADVANTAGE OF SGD OVER EAS
Given these small learning rate and large sampling population approximation needed for the proof of 2.2, we can also fairly confidently say that whenever applicable. SGD is also more computationally efficient than GO-EA, given that the derivation and back prok-prroagttion are not computationally more expensive than new parameters sampling by more than the expected latent dimension of the model.
5

---

Under review as a conference paper at ICLR 2022
Given the simplicity of the GO-EA, expect this computational efficiency relation to hold for other evolutionary algorithms, given that GO-EA one of the possibly simplest ones.
3.1.3 MOST GENER MANNERALIZEALE MODEL SELECTION AND EFFECTIVE EMBEDDING DIMENSION DETECTION
Prior work on diverse populations of biological systems evolving in a manner compatible with the Gillespie-Ort Oir formalization has shown that it possible both estimate the effective phenotypic dimension, which for ANNs correspond to the latent dimension of the model, as well as to select within the population the sub-population that the best general performance (Kucharavy al., 2018).
Specifically, in order to perform the latent dimension extraction, this work leveraged the Gnedenko- Kolmogorov formulation of the Central limit theorem, Gnedenko et al. (1968) that has proved that a complex system can be altered in large number of random ways, the resulting deviation from the base state converges to a Gaussian. Specifically, such a deviation would be visible along each axis relevant to adaptation to the environment, with the total deviation being characterized by the Chi-n function, where n. is the effective phenotypic dimension of the organism/environmeed match, up to a renormalization. By subjecting this population to a number diverse environments, it was possible to leverage the relationship between the mean and standard deviation of the fitnesses of the heterogeneous population among different population to both calculate the effective phenotypic dimension in which the population could move to match the environment, well as the population that were the best at dealing with all the environments.
In the context of model fine-tume fine-tuning this could mean that in presence of sufficiently diverse set of validation datasets and heterogeneous models, differing either by their architecture, initialization, hyperparameters or training dataset, it would be possible to both evaluate the effective latent dimen- sion of the model family and the problem and to determine the model that is most likely to be a good starting point for fine-tunes that could cover all the datasets. Alternatively, the model ability to retain generality across validation datasets could also be used to identify models within the family least prone catastrophic forgetting during the transfer learning.
3.2 HYPOTHESES
Whereas the previous subsection was dedicated to the direct implications of our central results, here present several hypotheses based the conceptual framework of evolutionary algorithms
3.2.
MINIMA FLATNESS AS ERROR CORRECTION REDUNDANCY
SGD converging to flat minima is one of the conditions on the architecture of the ANN models for their training be stable (Li al. 2018).
Minima flatness was considered to the generalization abilities of the model through its presumed relationship the minimal coding length of the model (Hochreiter & Schmidhuber, 1997 Goodfel- low & Vinyals 2015). although recently evidence to the contrary emerged (Dinh et al. 2017 Zhang al. 2020 Mulayoff Michaeli 2020).
Within the theory of evolution, the flatness of the fitness peak is commonly associated to the toler- ance the neutral drift- aka error correction capabilities. By using this analogy, we suggest that just like in the context of the evolution. the flatness of the loss function minimum in ANNs opti- mized through SGD determined by the redundnancy of the features used by the trained ANN to recognize patterns in the target data.
This intuition seems be consistent with empirical observations about the loss function minima flatness. Architectures that provide the model with means to encode redundant features, such as with extremely large hidden layers or with skip-forwards connections in deep Convolutional Feed- Forwards ANNs, contribute making the loss landscape minima more flat, as demonstrated by Li et al. (2018). Similarly, drop-out regularization Srivastava et al. (2014), forcing the ANNs to learn redundant, coroe-ccrre codings seem to flatten minima well, along with the smaller batches, which can contain a large proportion of samples that defy the heuristics that the ANN has learnt until now (Goodfellow & Vinyals 2015)

---

Under review as a conference paper at ICLR 2022
From this perspective, do not expect flatter minima to lead better generalization, but rather to allow for more robust and less noise- sensitive models, which seems to be confirmed by the numerical experiments showing that ANNs with architectures, regularizations and training modes known to lead to flatter minima also tend to memorize less (Arpit et al. 2017).
3.2.2 FLAT MINIMA AND TRANSFER LEARNING
Building on top of the hypothesis presented above, if the minima flatness is indeed related to the clas- sification robustness and eтror correction, we expect models that learnt variety of correcttng representations of training data to not be able to transfer those representations without training onto new data presenting similar features.
Intuitively, they rely on a simultaneous redundant subpaths through their ANN layers detecting redundant relevant features present the training dataset. With only some those features present in the dataset on which the transfer task is performed, their error correction property is likely to interfere with the the output of corresponding output without an expected degree of redundant detection.
If this hypothesis is correct, a particular attention need to be payed when training ANNs that need to be both robustly map inputs to outputs in noisetresistant manner and can be exposed to rare inputs presenting only some of the features on which the action need to be taken. We expect this problem to be separate from the adversarial examples and more closely related to the generalization one, given that its goal is to recognize partial features.
4 HEURISTICS FROM THEORY OF EVOLUTION TO ACCELERATE THE FINE-TUNING PROCESS
Here we present a number of heuristics that are thought to be critical for the acceleration of the natural evolution, that we expect to be transferable to the training of well-formed ANNs through SGD.
4.1 MODEL MIXING IS NOT NECESSARY, ALTIIOUGHI CAN BE BENEFICIAL.
One of the prominent features of the later Genetic Programming compared to the early evolutionary algorithms was the emphasis on mixing the models through "chromosome" "recombination" Directly mapping to the importance the sexual reproduction in the context of natural evolution, is neither necessary nor applicable in the context of GO-EA SGD. Unlike in biological systems, there is no spurious mutation accumulation, so there is no need for the purifying selection to prevent Muller's ratchet from eliminating the population of ANNs (Lynch et al. 1993). Similarly, studies of model parameter interpolation between two good solutions indicate that the intermediate parameters tend to perform uniformly poorly without dedicated regularization (Goodfellow & Vinyals 2015).
In case the model mixability is desireable however, for instance for rapid model aggregation at- tempts, in the case the models independently drift away from the optimum, is possible to develop regularization schemes that would preserve mixability (Livnat et al., 2008). Another rea- son such a mixability might be desirable, is that might promote the diversity and redundance of feauture extraction subnets in the ANNs, further improving training stability and resilience to memorization.
4.2 RAPID EXPLORATION OF THE LATENT FEATURE SPACE
One of biological systems that are most consistent with the assumptions of the Gillespie-On model of natural evolution are pathogens developing resistance to treatments. While the means to achieve such resistance differ between the pathogens and treatments, they nonetheless use a common trick to accelerate their adaptation before going extinct. Specifically, rapidly generate random variation in the phenotypic space, in hopes that at least one of such variant populations would be more fit than the original one Kucharavy al. (2018). This trick seems to be highly efficient, allowing the pathogens to adapt to new stressful conditions in a matter of a dozen of generations as opposed to thousands that would be expected in case of gradual evolution. Given that the underfying model

---

Under review as a conference paper at ICLR 2022
for theoretical work of Kucharavy al. (2018) is FGM with asexual reproduction, the results fully compatible with GO-EA and hence with models fine-tuned with SGD.
To summarize the results from (Kucharavy al., 2018), the dimensionality detection algorithms relies on a family of related models (for instance members of population with random perturba- reference model), each evaluated on heterogeneous benchmark.
tion parameters compared Based on the correlation of the average performance of models on each test task in the benchmark compared to the standard deviation of the models performance on each test, Kucharavy et al. (2018) shows that there is an expected correlation between the two and that a direct regression with two or- thogonal parameters possible to evaluate the underlying dimension of the problem. It is important to note that the dimension of the problem is not intrinsic to the problem at hand but also involves the architecture and parameter values of the ANN that has been trained to solve it, corresponding to the amount of independent "axes" along which the ANN can move to better adapt to task.
The mechanism we expect to limit the catastrophic forgetting requires a cross- evosseevalutiion of family of models on heterogeneous benchmark. Unlike the problem dimension evaluation, it evaluates the average performance of model, as well as how uniformly performs relative to other models on different tasks in the heterogeneous benchmark. Specifically, by calculating the Gini index of the model performance. Kucharavy et al. (2018) predicts that it will be inversely correlated with model performance and that the model with the lowest inequality of performance across tasks would also have reasonable performance. Given that it have a good average performance and perform well across most tasks in the benchmark, we expect that this model has not undergone catastrophic forgetting. It can be seen as a regularization that leverages intrinsic parallelizability of the GO-EA class of algorithms and ensures that as transfer learning is performed, each exploration-sclectio cycle the loss of performance on other tasks in the benchmark is minimal while the new task is learnt. This regularization can also be applied in case of parallel model-fine tuning with SGD, resulting in a family of related models, from which the least "forgetfull" model is selected.
By analogy between the GO-EA and SGD adapted to fine-tuning the models, we expect that if a random variation was injected into the ANNs before starting the fine-tuning process, at the level where they would be getting close leaving the flat minimum in their original loss space, their fine-tuning could be significantly accelerated, even if the least performing models are eliminated after the first couple fine-tuning training epochs performed parallel.
DISCUSSION
In this paper we establish a formal equivalence between the Gillespie-Or model of evolution and SGD applied to ANNs. Build on top of strong limit distribution convergence results established by the Fisher- Teetee-Tpppp----eeeee Gnedenko theorem, GO-EA leverages the representation ANNs as learnt codes that are greedily improved through neighbourhood exploration to provide insight into how SGD might work and how it can be improved, as well as to what can be expected from parameter space search with GO-EA.
While we expect GO-EA algorithms to be more computationally expensive than SGD on differ- entiable manifolds, does have two important application domains. First, in the cases where the evaluation of gradient and backpropagation of gradients are significantly more computationally ex- pensive than the evaluation of the model performance (on the order of magnitude of the number of model parameters). In this cireumstances, parallel computational capabilities and tricks allowing a more efficient communication of model updates, such as introduced in |Such et al. (2017) can allow faster and more computationally efficient model training. Second, in cases where the loss surface can be assumed to be smooth, but cannot be directly differentiated, such as in behavior strategies learning (as for instance described by Majumdar et al. 22020))
While on the surface that last case seems to be rather limited, research in the context of SGD Li et al. 2018 8) have shown that the smoothness of the loss landscape is dependent on the ANN archi- tecture and parameter values rather than the problem alone. This suggests that there are classes of problems that area currently assumed to be difficult due to non-smooth loss landscapes that can be made more approachable by new ANN architectures and hence be efficiently explored by GO-EA algorithms.
8

---

Under review as a conference paper at ICLR 2022
Based on that formal equivalence we offer new perspective on the minima flatness, which we link to the redundancy of the compressed codes representing the learnt model, rather than the minimal code length, suggested previously, showing that is consistent with the experimentally observed results linking loss landscape flatness with skip connections, hidden layers width and the use of dropout regularization.
We further build on this insight in order to hypothesize that is possible to identify the number of latent dimensions used by model family to learn to map a training dataset inputs to corresponding outputs, as well as the most generalizeable model in population, provided a sufficiently diverse set of secondary validation datasets available.
Finally, we suggest a number of heuristics we expect would accelerate model fine-tuning or EA application to ANNs in general.
While the paper could greatly benefit from the experimental validation of hypotheses and heuristics presented here, the multiple model training restarts collect statistics, hyperparameter space explo- ration and model population sizes needed for the EA methods mean that it is task better suited for an entity with large computational capabilities and could represent work in its own right.
In fact, ResNet-56 on CIFAR-10 used in (Li et al. 2018) to evaluate minima flatness requires 18 hours of training time on top-of-the line consumer hardware. An initial training run combined with fine-tuning is likely to multiply this time by a factor of magnitued, whereas orthogonal filter space search on grid with filter normalization on the pre-fine-tuned model and fine-tuned models with intermediate steps would require similar similar of magnitude of compute time. Combined with multiple restarts from different random seeds in order to obtain statistics on results, this means single experiment run times in the 70-80 days range of GPU time, assuming other bottlenecks. While those experiments are trivially parallelizable, they require access to a cluster with sufficient compute budget to them.
Moreover, our paper is consistent with prior experimental results, such as presented in Such et al. (2117). There, authors define a "genetic algorithm" that is fully elitist and does not proceed any recombination and is hence an algorithm in the GO-EA class. By using this algorithm, they observe a number of features initially discovered and theoretically explored in the context of SGD, such as for instance local density of good solutions near random initialized vector in case of sufficient model over-parameetriztion (Jacot et al.] 2018). However, even a single training run of their model to train their 4 million parameter ANN for single Atari game required 720 CPU core-hours for a single run or 4-8 days of wall time on pusumerr-grade CPUs with 4-8 cores. Multiple restarts would be required to collect representative and comparable results, leading to simulation run times in the 20-40 days on consumer hardware.
We hope that our work provides novel perspective on the SGD convergence in the context of ANNs, as well as Evolutionary Algorithm application more generally.
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio. Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedingy of Machine Learning Research, pp. 233-242. PMLR, 2017. URL arpit17a. html
mir. wttps//sssmmm....//oooooo//o
David Balduzzi, Marcus Frean, Lennox Leary, Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning. ICML 2017. Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 342-350. PMLR, 2017. URL ml www...rrrs../...m......mmmmm...e / vvv0 duzz i 7b. html
http:://rrocedi
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. Unsuper- vised and Transfer Learning Challenges in Machine Learning, 7:19, 2012.

---

Under review as a conference paper at ICLR 2022
Yoshua Bengio, Frédéric Bastien, Arnaud Bergeron, Nicolas Boulanger- LowaadoorrLLwwddwwkkk.k.. Thomas M. Breuel, Youssouf Chherawala, Moustapha Cissé, Myriam Côté, Dumitru Erhan, Jeremy Eustache, Xavier Glorot, Xavier Muller, Sylvain Pannetier Lebeuf, Razvan Pascanu, Salah Rifai, François Savard, and Guillaume Sicard. Deep learners benefit more from out-of-distribution examples. In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík (eds.), AISTATS, volume 15 of JMLR Proceedings, pp. 164-172. JogR.org, 2011. URL httpp:////dllp uni meiii-rreerdde do/ /o/ ja//oouraals//// jmir/ 1immpp15. html BBBenggSOBBBBCCEEEELLLSSL
S Bozinovski and A Fulgosi. The influence of pattern similarity and transfer learning upon training of a base perceptron b2. In Proceedings of Symposium Informatica, pp. 3-121, 1976.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fi- delity natural image synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:
//openzeview.o www.neet/foorree umm B X99
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemnrr, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL cite arxiv:2005.44165ooo 14165cmmmet 40+32 pages.
http://arxiv..t wwwp://ggg/abaeeeeeo abs/ /205 14165
Rich Caruana. Learning many related tasks at the same time with backpropagation. In G. Tesauro, D. Touretzky, and T. Leen (eds.). Advances in Neural Information Processing Systems, volume 7, pp. 657-664. The MIT Press, 1995.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19 1423. URL 1ttps://oo..cog/1..116///
1423
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6- I1 August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pp. 1019-1028. PMLR, 2017. URL html
http://proceeddings mll. prwse.ressseeee wwt.m//p..../////iini....
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.03961 2021.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/ 08734, 2017. URL http::/ttp://b wttp:/.......irirr.....r.e Dournals/corr/coril//
uni-trier wwt.iiierrrdddb.. de/db/
htm har.rroddbtttteeereradddeezz
R. A. Fisher. The genetical theory of natural selection. Oxford University Press, Ox-
ford, 1930. ISBN 0-19-850440-3. URL http:/ http://wwwww........ owt.:/iiillo//tttass//sssss
6mbb
Ronald Aylmer Fisher and Leonard Henry Caleb Tippett. Limiting forms of the frequency distribu- tion of the largest smallest member of a sample. In Mathematical proceedings of the Cambridge philosophical society, volume 24, pp. 180-190. Cambridge University Press, 1928.
Dario Floreano, Peter Dürr, and Claudio Mattiussi. Neuroevolution: from architectures leamn ing. Evol. Intell, 1(1):47-62 62, 2008 htthtt:: /tt::///bll /blp.. uni -trier.... de/db/ db/ /tbp//ounass/////
evi /evi html #Floreanno00
10

---

Under review as a conference paper at ICLR 2022
L.J. Fogel, A.J. Owens, and M.J. Walsh. Artificial intelligence through simulated evolution. Wiley, Chichester, WS, UK, 1966.
Maurice Fréchet. Sur loi de probabilité de l'écart maximum. Ann. Soc. Math. Polon., 6:93-116, 1927.
Edgar Galván and Peter Mooney. Neuroevolution in deep neural networks: Current trends and future challenges. CoRR, abs/2006.05415, 2020. cootnals/coorrceeete
uni-trier. /b/ html abs 2006 05415
John Gillespie. A simple stochastic gene substitution model. Theoretical population biology, 23 (2):202-215, 1983.
John H Gillespie. Molecular evolution over the mutational landscape. Avolution, pp. 1116-11111 129, 1984.
Boris Gnedenko. Sur la distribution limite du terme maximum d' d'une serie aleatoire. mathematics, pp. 423-453, 1943.
Annals of B.V. Gnedenko, A.N. Kolmogorov, and K.I. Chung. Limit Distributions for Sums of Independent Wesley Mathematical Series. Addison-Wesley, 1968. URL https:
Random Variables. Addison Adeleyn-Wesley
gbooks..gogle..oo //booksspoogleecch/bbokk////io rYYAAAAAAA
David E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-
Wesley, Reading, MA, 1989.
Faustino J. Gomez and Risto Mikkkulainen. Incremental evolution of complex general behavior.
Adapt. Behav,, 5(3-4):317-342, 1997. doi: 101117710097223700000333 URL https://
Faustino J. Gomez, Jürgen Schmidhuber, and Risto Mikkkulainen. Accelerated neural evolution
through cooperatively coevolved synapses. J. Mach. Learn. Res., 9:937-965, 2008. URL https:
acm. org htt.mh/rrccaaaat........ocaaaaa.
lan J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network optimization prob- lems. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Rep- resentations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
org/abs 1412. 6544
Stephen Jay Gould and Richard C Lewontin. The spandrels of san marco and the panglossian paradigm: a critique of the adaptationist programme. Proceedings of the royal society of London. Series B. Biological Sciences, 205( 161 ):581-599 1979.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolu- tion strategies. Evol. Comput., 9()): 159-199, 2001. doi: 041166/10066556111001000000 URL
Nikolaus Hansen, Dirk V. Arnold, and Anne Auger. Evolution strategies. In Janusz Kacprzyk and Witold Pedrycz (eds.). Springer Handbook of Computational Intelligence, Springer Handbooks, pp. 871 Springer, 2015. doi: 10.
URL https::/doi.
0rg/10.1007/2
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat min- ima. In Gerald Tesauro, David S. Touretzky, and Todd K. Leen (eds.), Advances in Neural Information Processing Systems 7. INIPS Conference, Denver, Colorado, USA, 1994], pp. 529-536. MIT Press, 1994. URL http::/papers nips cc/paper/
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Comput, 9(1):1422 1997. doi: (116///cco.1999....... UtLhhtss:: /tt.t:pp:///oio..rg///// wwt.:/...ro.....///eeeeo...o
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural Networks, 2(5):359 366, 1989. ISSN 0893-6080. doi: http:
URL attp::/ http://wwww.ceeeeeeeee.e.m com/

---

Under review as a conference paper at ICLR 2022
HuggingFace. HuggingFace Model repository. https///augiingaaee./o/oo/m 2020.
[Online; accessed 05- 20201.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Biance and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 8580 8589, 2018. URL https:s:t///proctetee iggs neurips 2ct/paper/2018/hasee Abstract html
Paul Joyce, Darin R Rokyta, Craig J Beisel, and Allen Orr. general extreme value theory model for the adaptation of dna sequences under strong selection and weak mutation. Genetics, 180(3): 1627-1643, 2008.
Stuart A Kauffman. Metabolic stability and epigenesis randomly constructed genetic nets. Journal of theoretical biology, 22(3):437-467. 1969.
Stuart A Kauffman and Sonke Johnsen. Coevolution to the edge of chaos: coupled fitness land- scapes, poised states, and coevolutionary avalanches. Journal of theoretical biology, 149(4): 467-505, 1991.
Motoo Kimura. Genetic variability maintained in a finite population due to mutational production of neutral and nearly neutral isoalleles. Genetics research, 11(3):247-770, 1968.
Motoo Kimura and James F Crow. The number of alleles that can be maintained in finite popula- tion. Genetics, 49(4):725, 1964.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. ing systems, pp. 1097-1105, 2012. URL http::/
Advances in neural information procesy- cttpps.rc/ppper/// won-wwtthtntthhhhh-eeccmhh htt.://wwwott..ooooooooooooaaaaa
Andrei Kucharavy, Boris Rubinstein, Jin Zhu, and Rong Li. Robustness and evolvability of hetero- geneous cell populations. Molecular biology of the cell, 19(11))1400-144 2018.
Russell Lande. The dynamies of peak shifts and the pattern of morphological evolution. Paleobiol- ogy, 12(4):343-354 1986.
Yann LeCun. Une procedure d' d'apprentissage ponr reseau seuil asymetrique. Cognitiva 85, pp. 599-604, 1985.
Proceedings of
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):336444, 2015.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 6391-6401 2018. URL
neurips /cc/paper//21B/hast// html
Adi Livnat, Christos Papadimitriou, Jonathan Dushoff, and Marcus W Feldman. A mixability theory for the role of sex in evolution. Proceedings of the National Academy of Sciences, 105(50):19803- 19808, 2008.
Jonaaaan
Michael Lynch, Reinhard Bürger, D Butcher, and Wilfried Gabriel. The mutational meltdown in asexual populations. Journal of Heredity, 84(5):339-344 1993.
Somdeb Majumdar, Shauharda Khadka, Santiago Miret, Stephen Mealeer, and Kagan Tumer. Evo- lutionary reinforcement learning for sample-efficient multiagent coordination. In International Conference Machine Learning, pp. 6651 6651-6660. PMLR, 2020.
12

---

Under review as a conference paper at ICLR 2022
R von Mises. La distribution de la plus grande de valeurs. Rev. Math. Union Interbalcanique, 1: 141-160, 1936.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-lever control through deep reinforcement learning. Nat., 518(7540):529 5333333, 2015. doi: 0011088/ature142230 URL https: httpa:://doo.oorg/////
1038/nature 4236
Rotem Mulayoff and Tomer Michacli. Unique properties of flat minima in deep networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7108-7118. PMLR, 2020. URL
htt.htttpp//rrrceddngg.. /
mir. prwss/ /vl.vv11//9 httss///11mmmmaaof2202e... html
Quynh Nguyen. On connected sublevel sets in deep learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Ma- chine Learning Research, pp. 4790 PMLR, 2019. URL htll:ttppppprcceeeiiie mmlr.
press/v91 www.//ngunnnnnn 9a. html
Quynh Nguyen, Pierre Bréchet, and Marco Mondelli. On connectivity of solutions in deep learn- ing: The role of parmetaraatttiinttioonn and feature quality. CoRR, abs/2102.09671, 2021. URL
https:///aaat http:///aaxiv..oraaas/s//soos/// wttp://////////////////////////a 0967
Tomoko Ohta. The nearly neutral theory of molecular evolution. Annual review of ecology and
systematics, 23(1):263-288) 1992.
Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level im- age representations using convolutional neural networks. In CVPR, pp. 1717-1724. IEEE Com- puter Society, 2014. ISBN 978-14499--518855 URL http://ablp
de db/
Eon.//eppx/cypr20444 http:/22000..tttaaoooooaaaaa.aa.
H Allen Orr The population genetics of adaptation: the adaptation of dna sequences. Evolution, 56 (7):1317-1330, 2002.
H Allen Orr. The genetic theory of adaptation: brief history. Nature Reviews Genetics, 6(2): 119-127, 2005.
H Allen Orr. The distribution of fitness effects among beneficial mutations in fisher's geometric model of adaptation. Journal of theoretical biology, 238(2):279-288 2006.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Trans, Knowl, Data Eng., 22 (10):1345-139,, 2010. doi: 10.1109/TKDE.2009.199 URL https:://co.iiii wtt.oiiorg/1...1. 1109/
2009..00..1..
D. B. Parker. Learning-logice Technical Report TR-47, Center for Comp. Research in Economics and Management Sci., MIT, 1985.
James III Pickands. Statistical inference using extreme order statistics. The Annals of Statisties, 3(1): 119-131, 1975. ISSN 00905364. doi: 10. 10.1214//0//1773333003 URL /wwp http://wwww..ow...o/o.. jstor. org/ htt/stabbee/258005
Lorien Y. Pratt. Discriminability-aasee transfer between neural networks. In Stephen Jose Hanson, Jack D. Cowan, and C. Lee Giles (eds.), Advances in Neural Information Process- ing Systems 5, (NIPS Conference, Denver, Colorado, USA, November 30 December 3, 1992), pp. 204-211. Morgan Kaufmann, 1992. URL 641 di scrirriin i mi nabi lity transeddtraasaeerree wewerraneeeeebeeeeeeeeeeeeeeeeee -tw.:-uuuuuaa-eeeoeee-------- -ntworks
http://ssipps.ccpppppppepapp cc/paper/
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Math- ematical Statistics, 22(3):400-444 407, September 1951. doi: 00.1214/ams/11177999 URL
https://doo www./oo..... 1t1./11.111111 Fww.Faammmmm11711mmm
13

---

Under review as a conference paper at ICLR 2022
Darin R Rokyta, Craig J Beisel, Paul Joyce, Martin T Ferris, Christina L Burch, and Holly A Wich- man. Beneficial fitness effects are not exponential for two viruses. Journal of molecular evolution, 67(4):368, 2008
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning represseteeesst by back- propagating errors. Nature, 323(6088) October 1986. /
10. 20.1038/322 32353330
Victor Sanh. Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108 2019. URL http://
arxiv. arxiv.org/abe. httpg//bb/1911..0110
HP. Schwefel. Numerical optimization of computer models. Wiley, Chichester, WS, UK, 1981.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting J. Mach. Learn. Res., 15(1):
1929-1958, 2014. htttt:::/d...mm.oo../ooooo tttp:://..........o......aa.aaa. 2et.:taaaa.....m.....2211111111.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. CoRR, abs/17 0tns/1722.06667. 2017. URL http:
Olivier Tenaillon. The utility of fisher's geometric model in evolutionary genetics. Annual review of ecology, evolution, and systematics, 45:179-201, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL at ion-tiioniiie http://ioo-attttttn-------aaaaaa
https: http::://ppaprss nips. cct wwwpe//appe/aaeeee
P.J. Werbos. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University, 1974.
Daan Wierstra, Tom Schaul, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. In Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2008, June 1-6, 2008, doi: 10.1009/CEC.200844415555 URL
Hong Kong, China, pp. 3381-3387. IEEE, 2008
https: //w.i///iii//// www.oogg11... /00 1 100099/0000 CEC 2008 4 4631255
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec. Canada, pp. 3320-3328, 2014. URL http:::::///pcceedeeeoeo wewp:rocccciiioooeeii........... neurips.cc.. 2tt.a//0....2222ec22222000000000 (tt.:/eddccaa220000000000000
caper/2014/hashi 2tw.000//..aa.11111112222220e..0
.tml
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small relu networks are powerful memoriz- ers: a tight analysis of memorization capacity. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 15532-15543, 2019. URL https::/ppoceeedin/////////////t .ew.eeeedddinnn.n.n............. newrips. wwtpuuuiss.cccpppeeeeee cc/paper//011/ http://a33022a111111022222232333
html
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
net /aet/foo feree//ooummddde SynddByyyyBBBBBB 9xx
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer, and Yoram Singer. Identity cri- sis: Memorization and generalization under extreme overparameterization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL. hwtp:tttess//oogeeeeieeee..e
1y0VFF
14

---

Under review as a conference paper at ICLR 2022
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107-115, 2021.
doi: 10.1145/344677. URL BtRhhttps: / orrg/1..114/3344777
15


Reviews:
[{"review_id": "r858839Dze6", "reviewer": "Reviewer_DWHG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The comparison of the SGD and the evolutionary algorithm is quite interesting but not completely new from the conceptual point of view. The paper however introduces a very specific - weaker type - of evolutionary algorithm that is easier to be formalized. The main conclusions of the paper are that the flatness of the SGD minima is an equivalent statement for the stability learning in ANN. Another hypothesis proposed is that the transfer learning of a model with SGD minima flatness can only be transferred to a data that would result in a similar minima flatness. \n\nWhile the hypothesis seems to be plausible there is a general lack of formalism which prevents to clearly discussing the validity of the model.  The generality of the proposed hypotheses would require at least experimental verification in order to provide supporting evidence from the data stand point of view. \n\nFor instance, the notion of the SGD minima flatness is not well enough defined to make decision what kind data is required so that the transfer learning is successful. Similarly the hypothesis of the error correcting ability of ANN, while seems empirically correct needs to be verified. ", "summary_of_the_paper": "This paper introduces a conceptual equivalence between a certain class of evolutionary algorithms and the stochastic gradient descent. The paper is relatively well written but often introduces notation without proper definitions. ", "summary_of_the_review": "The paper in general lacks supporting formulation that would allow clear understanding and clear conclusions about the validity of the proposed statements. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y_pztwVgd0i", "reviewer": "Reviewer_UoCs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- This paper reviews a long history of evolutionary and stochastic gradient descent algorithms. \n- It provides a new perspective that the fine-tuning process is similar to the adaptation in environmental change. \n\nWeaknesses\n- Their central algorithm is from previous milestone works. \n- The perspective of evolving the parameter itself is not new.\n  - Recent works apply evolutionary algorithms directly into the parameters of ANN. Here I reference some works in the reinforcement learning field as an example [1-5].\n- The central results, hypothesis, and heuristics are not supported by the experimental results. \n  - It is better to show some results compared with SGD. \n  - Further analysis is necessary to provide the validity of claimed heuristics. \n- Several typos and not organized sentences. \n  - Some repeated words, like \"model model\", \"steps steps\" and more.\n  - The sentence is not finished, \"... in reverse exponential probability of the\"\n  - Some paragraphs are extremely long that makes it hard to read, and some are very short, seems no reason to separate \n\nReferences  \n1.  Salimans, Tim, et al. \"Evolution strategies as a scalable alternative to reinforcement learning.\" arXiv preprint arXiv:1703.03864 (2017).\n2.  Such, Felipe Petroski, et al. \"Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning.\" arXiv preprint arXiv:1712.06567 (2017).\n3.  Khadka, Shauharda, and Kagan Tumer. \"Evolution-guided policy gradient in reinforcement learning.\" Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018.\n4. Pourchot, Aloïs, and Olivier Sigaud. \"CEM-RL: Combining evolutionary and gradient-based methods for policy search.\" arXiv preprint arXiv:1810.01222 (2018).\n5. Lee, Kyunghyun, et al. \"An Efficient Asynchronous Method for Integrating Evolutionary and Gradient-based Policy Search.\" arXiv preprint arXiv:2012.05417 (2020).", "summary_of_the_paper": "This work introduces Gillespie-Orr Evolutionary Algorithm (GO-EA) and applies it to the fine-tuning of parameters. It shows the perspective of evolution on model fine-tuning by comparing with SGD and analyzing the probability of finding better parameters. Finally, it provides some heuristic techniques to accelerate the fine-tuning process.\n", "summary_of_the_review": "Although it provides a new perspective in fine-tuning process, the core idea seems not novel and is not supported by robust experiments. \nAlso, the writing seems not thoroughly reviewed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vy8p9fFyBf1", "reviewer": "Reviewer_DYJE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe authors provided a significant amount of generalization in their methodology using the Gillespie-Orr genetic algorithm to be useful outside of the specific application they described. \nIn proving the equivalence between the genetic algorithm and SGD, they provided specific details on the proof and limitations that existed therein. It provides a pathway for the expansion of performance approximation of GO-EA given the described equivalence holds.\nWithin the description of the minima flatness as related to a genetic algorithm’s error correction, they provided a well detailed analysis on how robust, noise-resistant models might fail to transfer successfully as the redundant features will interfere with the capability of the model to train under the new dataset. In their apt analogy, this would be the population of an EA suppressing beneficial changes due to strong error correction in the original population. \nFinally, the two heuristics proposed for the improvement of the model fine-tuning process provided a great description of the problem space each feature of the GA would be able to tackle, and how it was related.\nOverall, the detailed descriptions of the evolutionary processes and their relations to model fine-tuning processes provided a very convincing argument for the usefulness of their ideas.\n\nWeaknesses\n\nThis paper mentions multiple shortcomings on the usefulness of genetic algorithms in the field of fine-tuning and on their specific equivalency between the Gillespie-Orr GA and SGD. Overall, the authors provided some details but failed to provide an in-depth analysis on the equivalency given their assumptions failed. \nThe section on dimensionality detection was too brief to discuss how they calculated the effective phenotypic dimension and leveraged it to find the best population. While including this information is still useful, the paper in general would benefit greatly to discuss how some of the discussed facets of using a genetic algorithm would look like. For instance, discussing the specific relationship between the newly found best population would be able to reduce catastrophic forgetting would help improve the credibility of this statement. \nIn general, as mentioned at the end of the paper as a flaw, this paper would benefit greatly from empirical evidence supporting the claims made. Without it, many of the claims lack credibility as the proposed gain in accuracy or generality in models might amount to very little in practice.\n\nThis paper contains a number of minor spelling and syntax errors that would benefit from an extra review.\nAlso, although grabbing empirical data is likely outside the scope of this project, I feel that the addition of more information into some of the secondary points brought up in the paper would provide a more comprehensive view on the outcome of the original goal of the paper.\n\nOverall, the lack of empirical results and more in-depth theoretical analysis severely limits the significance of the paper.", "summary_of_the_paper": "The authors of the paper sought to describe the relationship between the Gillespie-Orr model of evolution and Stochastic Gradient Descent in an attempt to describe the usefulness of evolutionary processes from a model fine-tuning perspective. After first describing the general use and limitations of the two methods, they proved an equivalency between the two and the improved ability of Gillespie-Orr EA to find better model parameters during fine tuning. In addition, they constructed a set of two potential cases for the use of evolutionary algorithms in the general methodology of model fine-tuning as a potential performance improvement.\nThis paper addresses a novel concept within the field of model fine-tuning and provides information on its practicality and usefulness.", "summary_of_the_review": "A nice investigation with limited significance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "PvZIrtvEvqR", "reviewer": "Reviewer_5E1f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well argumented and potentially very interesting. I am myself a strong advocate of Black-Box Optimization and very excited for new work in the area. Writing aside however, the authors do not provide any semblance of experimentation or results sustaining their theoretical results, which is typically by itself unacceptable in major machine learning conferences. With still 1.5 pages from the page limits, I can see no problem for the authors to extend and complete this work.\nThe following points would need to be addressed in order for this work to be considered for publication:\n- Either a theoretical or practical experiment, possibly both. For comparison of EC algorithms, particularly for continuous optimization, I would propose the standard BBOB/COCO. For the practical results, since neuroevolution is mentioned in depth, I would expect at least some standard RL control problem as found in the OpenAI Gym -- for simplicity, though any other environment or application from the literature would be acceptable, as long as the paper includes a comparison with previously published results.\n- Many of the discussions proposed regarding EAs seem to utilize a traditional implementation, rather than the more modern implementations like CMA-ES (industry standard since over 15 years) or NES (itself over 10 years old), which utilize fitness ranking to sidestep entirely most of the problems addressed in this paper. I propose the experiments above should utilize standard CMA-ES as the baseline to showcase the advantages of this alternative approach.\n- The key argument of the paper is the equivalence between SGD and Gillespie-Fischer EA / Gillespie-Orr EA. I think even a simple \n- The paper does not provide a performance analysis of the newly proposed EAs. Given the comparison with SGD, and the unfortunately low performance of most ES implementations, I believe it is important to disclose whether the method could scale to the high dimensions and sample efficiency that we expect nowadays when discussing SGD.\n- The otherwise extensive literature review first skips over the 20 years 1990-2010 in which Neuroevolution was the prolific state of the art for RL control, presenting it later in an incomplete manner. I would expect to see its discussion extended fairly in comparison to the other topics. Some of the main names include Hansen, Miikkulainen/Stanley, and for the current state of the art certainly Glasmachers.\n- Neuroevolution is mostly applied to RL problem, given the compatibility of its fitness scoring system directly with a RL paradigm reinforcement (direct policy search). When equating the proposed ESs and SGD however, the authors hypothesize a differentiability of the model (though earlier stating that it is not a requirement in principle in BBO), and the availability of the error gradient itself. This introduces another big requirement, which is a supervised learning setup, with availability of labeled data. I find it necessary for improving exposition to split the cases explicitly, by (i) showing the equivalence between the proposed ES and SGD when subject to the same requirements of SGD, then (ii) motivating why would a user choose an ES when SGD is viable (as all studies so far point in the other direction), and most importantly (iii) what value is this equivalence in applications where SGD is not applicable (such as in problems with non-differentiable models or lack of labeled data), which is arguably the greatest strength of BBO methods such as ES.\n- The literature review should also correctly attribute credit for Transfer Learning, called \"fine-tuning\" in the first part of the paper, which is a crucial component for some of the claims.\n- The claim that the computational resources required for experimentation would be prohibitive should be supported by actual budgeting. As a counter-argument, I know I can run my own ES experiments on my own laptop; and a quick search online can find plenty of hobbyists solving complex problems without access to industry-grade computational resources. I suggest for example checking the [leaderboard for the OpenAI Gym](https://github.com/openai/gym/wiki/Leaderboard): problems such as the BipedalWalkerHardcore are extremely complex to solve, and yet there are evolved solutions proposed on the leaderboard that do not require high-performance computing. I point out as an example the results from `hardmaru` (David Ha), as they are commonly recognized as a high-quality approach. He used (2017) custom neural network code, a network with two layers of 40 neurons each, and the published python implementation implementation of CMA-ES: all the code is easily obtainable online and could run on a moderately recent laptop in a matter of days at worst.\n\nI look forward to an extended version of this work as it certainly looks promising, and the research direction proposed is of high interest in the field.", "summary_of_the_paper": "The paper presents the foundation for a new class of Evolution Strategies inspired from the work in biology of Gillspie and Orr published in the 1980s. Particularly intriguing are an equivalence with SGD (but limited to problems with the same assumptions and requirements of SGD) and heuristics to accelerate convergence using Transfer Learning. The paper however lacks any experimental validation of the theoretical results proposed.\n", "summary_of_the_review": "Very interesting approach, incomplete comparison with the current state of the art (e.g. fitness ranking), and complete lack of any experimental results or any graphics. While I would not recommend the current state as ready for publication, I look forward to future more complete implementation of this work as it seems to be very promising.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this is an interesting treatise on some relationships between SGD fine tuning and evolutionary algorithms. All reviewers have requested some experimental validation or demonstration of the theory developed in this paper, which is not currently included. Whilst the computational requirements (and time required) may be long, this will significantly assist the many readers of the paper and save them from having to run such an experiment many times themselves.  The reviewers provided a number of suggestions of how this might be done. The reviewers also highlighted a number of specific improvements that can be made to the writing of the paper.
Decision: Reject