SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Summary. We introduce NeuroEvoBench - a benchmark targeting the rigorous evaluation of Evolutionary Optimization algorithms in the context of neural network training. It allows for easy, programmatic, and automatic investigation into a wide range of different BBO/neuroevolution tasks, types of EO, and budgets. Our initial investigation answers several core scientific questions regarding resource allocation, fitness-shaping, regularization, and scalability of EO.
Limitations. We aim to support a representative selection of tasks and problems. Nonetheless, we acknowledge that the actual real-world workload can represent a much more comprehensive range of scenarios. Going forward and as EO methods increase in capabilities, we envision a gradual enhancement and update of the task sweet and evaluation protocols.
Ethical Considerations. As a benchmark paper, we do not identify particular issues per se. Since we expect our work to be used by the community, we call for a range of considerations, including those of potential implicit bias in the task selection, evaluation protocols and result interpretation.
Future Work. We acknowledge that benchmarks have to be 'living projects' requiring constant refinement and updating with respect to the community's needs. We plan to continuously add more results for EO supported by evosax (Lange, 2022a) and to provide a modular and easy to extend benchmark protocol. Further investigations may include strategy restarts, asynchronous evaluation methods or the influence of shared randomness.
Conference, EvoMUSART 2022, Held as Part of EvoStar 2022, Madrid, Spain, April 20-22, 2022, Proceedings, Springer, 275-291.
TOKLU, N. E., P. LISKOWSKI, AND R. K. SRIVASTAVA (2020): "Clipup: a simple and powerful optimizer for distribution-based policy evolution," in International Conference on Parallel Problem Solving from Nature, Springer, 515-527.
WASKOM, M. L. (2021): "Seaborn: statistical data visualization," Journal of Open Source Software, 6, 3021.
XIE, X., P. ZHOU, H. LI, Z. LIN, AND S. YAN (2022): "Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models," arXiv preprint arXiv:2208.06677.
YOUNG, K. AND T. TIAN (2019): "MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments," arXiv preprint arXiv:1903.03176.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section 4 and open-sourced benchmark code.
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See https://github.com/neuroevobench/neuroevobench for the benchmark code and https://github.com/neuroevobench/neuroevobench-analysis for reproduction configurations and visualizations.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See https://github.com/neuroevobench/neuroevobench-analysis and supplementary information.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See Figure captions.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See https://github.com/neuroevobench/neuroevobench-analysis and supplementary information.
(a) If your work uses existing assets, did you cite the creators? [Yes] See supplementary information.
(b) Did you mention the license of the assets? [Yes] See supplementary information.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] https://github.com/neuroevobench/neuroevobench for the benchmark code and https://github.com/neuroevobench/neuroevobench-analysis for reproduction configurations.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]