Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a conceptual equivalence between a certain class of evolutionary algorithms and the stochastic gradient descent. The paper is relatively well written but often introduces notation without proper definitions. 
main_review: The comparison of the SGD and the evolutionary algorithm is quite interesting but not completely new from the conceptual point of view. The paper however introduces a very specific - weaker type - of evolutionary algorithm that is easier to be formalized. The main conclusions of the paper are that the flatness of the SGD minima is an equivalent statement for the stability learning in ANN. Another hypothesis proposed is that the transfer learning of a model with SGD minima flatness can only be transferred to a data that would result in a similar minima flatness. 

While the hypothesis seems to be plausible there is a general lack of formalism which prevents to clearly discussing the validity of the model.  The generality of the proposed hypotheses would require at least experimental verification in order to provide supporting evidence from the data stand point of view. 

For instance, the notion of the SGD minima flatness is not well enough defined to make decision what kind data is required so that the transfer learning is successful. Similarly the hypothesis of the error correcting ability of ANN, while seems empirically correct needs to be verified. 
summary_of_the_review: The paper in general lacks supporting formulation that would allow clear understanding and clear conclusions about the validity of the proposed statements.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work introduces Gillespie-Orr Evolutionary Algorithm (GO-EA) and applies it to the fine-tuning of parameters. It shows the perspective of evolution on model fine-tuning by comparing with SGD and analyzing the probability of finding better parameters. Finally, it provides some heuristic techniques to accelerate the fine-tuning process.

main_review: Strengths
- This paper reviews a long history of evolutionary and stochastic gradient descent algorithms. 
- It provides a new perspective that the fine-tuning process is similar to the adaptation in environmental change. 

Weaknesses
- Their central algorithm is from previous milestone works. 
- The perspective of evolving the parameter itself is not new.
  - Recent works apply evolutionary algorithms directly into the parameters of ANN. Here I reference some works in the reinforcement learning field as an example [1-5].
- The central results, hypothesis, and heuristics are not supported by the experimental results. 
  - It is better to show some results compared with SGD. 
  - Further analysis is necessary to provide the validity of claimed heuristics. 
- Several typos and not organized sentences. 
  - Some repeated words, like "model model", "steps steps" and more.
  - The sentence is not finished, "... in reverse exponential probability of the"
  - Some paragraphs are extremely long that makes it hard to read, and some are very short, seems no reason to separate 

References  
1.  Salimans, Tim, et al. "Evolution strategies as a scalable alternative to reinforcement learning." arXiv preprint arXiv:1703.03864 (2017).
2.  Such, Felipe Petroski, et al. "Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning." arXiv preprint arXiv:1712.06567 (2017).
3.  Khadka, Shauharda, and Kagan Tumer. "Evolution-guided policy gradient in reinforcement learning." Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018.
4. Pourchot, Aloïs, and Olivier Sigaud. "CEM-RL: Combining evolutionary and gradient-based methods for policy search." arXiv preprint arXiv:1810.01222 (2018).
5. Lee, Kyunghyun, et al. "An Efficient Asynchronous Method for Integrating Evolutionary and Gradient-based Policy Search." arXiv preprint arXiv:2012.05417 (2020).
summary_of_the_review: Although it provides a new perspective in fine-tuning process, the core idea seems not novel and is not supported by robust experiments. 
Also, the writing seems not thoroughly reviewed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors of the paper sought to describe the relationship between the Gillespie-Orr model of evolution and Stochastic Gradient Descent in an attempt to describe the usefulness of evolutionary processes from a model fine-tuning perspective. After first describing the general use and limitations of the two methods, they proved an equivalency between the two and the improved ability of Gillespie-Orr EA to find better model parameters during fine tuning. In addition, they constructed a set of two potential cases for the use of evolutionary algorithms in the general methodology of model fine-tuning as a potential performance improvement.
This paper addresses a novel concept within the field of model fine-tuning and provides information on its practicality and usefulness.
main_review: Strengths

The authors provided a significant amount of generalization in their methodology using the Gillespie-Orr genetic algorithm to be useful outside of the specific application they described. 
In proving the equivalence between the genetic algorithm and SGD, they provided specific details on the proof and limitations that existed therein. It provides a pathway for the expansion of performance approximation of GO-EA given the described equivalence holds.
Within the description of the minima flatness as related to a genetic algorithm’s error correction, they provided a well detailed analysis on how robust, noise-resistant models might fail to transfer successfully as the redundant features will interfere with the capability of the model to train under the new dataset. In their apt analogy, this would be the population of an EA suppressing beneficial changes due to strong error correction in the original population. 
Finally, the two heuristics proposed for the improvement of the model fine-tuning process provided a great description of the problem space each feature of the GA would be able to tackle, and how it was related.
Overall, the detailed descriptions of the evolutionary processes and their relations to model fine-tuning processes provided a very convincing argument for the usefulness of their ideas.

Weaknesses

This paper mentions multiple shortcomings on the usefulness of genetic algorithms in the field of fine-tuning and on their specific equivalency between the Gillespie-Orr GA and SGD. Overall, the authors provided some details but failed to provide an in-depth analysis on the equivalency given their assumptions failed. 
The section on dimensionality detection was too brief to discuss how they calculated the effective phenotypic dimension and leveraged it to find the best population. While including this information is still useful, the paper in general would benefit greatly to discuss how some of the discussed facets of using a genetic algorithm would look like. For instance, discussing the specific relationship between the newly found best population would be able to reduce catastrophic forgetting would help improve the credibility of this statement. 
In general, as mentioned at the end of the paper as a flaw, this paper would benefit greatly from empirical evidence supporting the claims made. Without it, many of the claims lack credibility as the proposed gain in accuracy or generality in models might amount to very little in practice.

This paper contains a number of minor spelling and syntax errors that would benefit from an extra review.
Also, although grabbing empirical data is likely outside the scope of this project, I feel that the addition of more information into some of the secondary points brought up in the paper would provide a more comprehensive view on the outcome of the original goal of the paper.

Overall, the lack of empirical results and more in-depth theoretical analysis severely limits the significance of the paper.
summary_of_the_review: A nice investigation with limited significance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents the foundation for a new class of Evolution Strategies inspired from the work in biology of Gillspie and Orr published in the 1980s. Particularly intriguing are an equivalence with SGD (but limited to problems with the same assumptions and requirements of SGD) and heuristics to accelerate convergence using Transfer Learning. The paper however lacks any experimental validation of the theoretical results proposed.

main_review: The paper is well argumented and potentially very interesting. I am myself a strong advocate of Black-Box Optimization and very excited for new work in the area. Writing aside however, the authors do not provide any semblance of experimentation or results sustaining their theoretical results, which is typically by itself unacceptable in major machine learning conferences. With still 1.5 pages from the page limits, I can see no problem for the authors to extend and complete this work.
The following points would need to be addressed in order for this work to be considered for publication:
- Either a theoretical or practical experiment, possibly both. For comparison of EC algorithms, particularly for continuous optimization, I would propose the standard BBOB/COCO. For the practical results, since neuroevolution is mentioned in depth, I would expect at least some standard RL control problem as found in the OpenAI Gym -- for simplicity, though any other environment or application from the literature would be acceptable, as long as the paper includes a comparison with previously published results.
- Many of the discussions proposed regarding EAs seem to utilize a traditional implementation, rather than the more modern implementations like CMA-ES (industry standard since over 15 years) or NES (itself over 10 years old), which utilize fitness ranking to sidestep entirely most of the problems addressed in this paper. I propose the experiments above should utilize standard CMA-ES as the baseline to showcase the advantages of this alternative approach.
- The key argument of the paper is the equivalence between SGD and Gillespie-Fischer EA / Gillespie-Orr EA. I think even a simple 
- The paper does not provide a performance analysis of the newly proposed EAs. Given the comparison with SGD, and the unfortunately low performance of most ES implementations, I believe it is important to disclose whether the method could scale to the high dimensions and sample efficiency that we expect nowadays when discussing SGD.
- The otherwise extensive literature review first skips over the 20 years 1990-2010 in which Neuroevolution was the prolific state of the art for RL control, presenting it later in an incomplete manner. I would expect to see its discussion extended fairly in comparison to the other topics. Some of the main names include Hansen, Miikkulainen/Stanley, and for the current state of the art certainly Glasmachers.
- Neuroevolution is mostly applied to RL problem, given the compatibility of its fitness scoring system directly with a RL paradigm reinforcement (direct policy search). When equating the proposed ESs and SGD however, the authors hypothesize a differentiability of the model (though earlier stating that it is not a requirement in principle in BBO), and the availability of the error gradient itself. This introduces another big requirement, which is a supervised learning setup, with availability of labeled data. I find it necessary for improving exposition to split the cases explicitly, by (i) showing the equivalence between the proposed ES and SGD when subject to the same requirements of SGD, then (ii) motivating why would a user choose an ES when SGD is viable (as all studies so far point in the other direction), and most importantly (iii) what value is this equivalence in applications where SGD is not applicable (such as in problems with non-differentiable models or lack of labeled data), which is arguably the greatest strength of BBO methods such as ES.
- The literature review should also correctly attribute credit for Transfer Learning, called "fine-tuning" in the first part of the paper, which is a crucial component for some of the claims.
- The claim that the computational resources required for experimentation would be prohibitive should be supported by actual budgeting. As a counter-argument, I know I can run my own ES experiments on my own laptop; and a quick search online can find plenty of hobbyists solving complex problems without access to industry-grade computational resources. I suggest for example checking the [leaderboard for the OpenAI Gym](https://github.com/openai/gym/wiki/Leaderboard): problems such as the BipedalWalkerHardcore are extremely complex to solve, and yet there are evolved solutions proposed on the leaderboard that do not require high-performance computing. I point out as an example the results from `hardmaru` (David Ha), as they are commonly recognized as a high-quality approach. He used (2017) custom neural network code, a network with two layers of 40 neurons each, and the published python implementation implementation of CMA-ES: all the code is easily obtainable online and could run on a moderately recent laptop in a matter of days at worst.

I look forward to an extended version of this work as it certainly looks promising, and the research direction proposed is of high interest in the field.
summary_of_the_review: Very interesting approach, incomplete comparison with the current state of the art (e.g. fitness ranking), and complete lack of any experimental results or any graphics. While I would not recommend the current state as ready for publication, I look forward to future more complete implementation of this work as it seems to be very promising.
review_title: null

Meta-review: The reviewers agree that this is an interesting treatise on some relationships between SGD fine tuning and evolutionary algorithms. All reviewers have requested some experimental validation or demonstration of the theory developed in this paper, which is not currently included. Whilst the computational requirements (and time required) may be long, this will significantly assist the many readers of the paper and save them from having to run such an experiment many times themselves.  The reviewers provided a number of suggestions of how this might be done. The reviewers also highlighted a number of specific improvements that can be made to the writing of the paper.
Decision: Reject