Improved Training Speed, Accuracy, and Data Utilization via Loss Function Optimization
Authors: Santiago Gonzalez
Risto Miikkulainen
Venue:  (2020)
Keywords: metalearning
evolutionary computation
loss functions
optimization
genetic programming

Abstract: As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions can be optimized with metalearning as well, and result in similar improvements. The method, Genetic Loss-function Optimization (GLO), discovers loss functions de novo, and optimizes them for a target task. Leveraging techniques from genetic programming, GLO builds loss functions hierarchically from a set of operators and leaf nodes. These functions are repeatedly recombined and mutated to find an optimal structure, and then a covariance-matrix adaptation evolutionary strategy (CMA-ES) is used to find optimal coefficients. Networks trained with GLO loss functions are found to outperform the standard cross-entropy loss on standard image classification tasks. Training with these new loss functions requires fewer steps, results in lower test error, and allows for smaller datasets to be used. Loss function optimization thus provides a new dimension of metalearning, and constitutes an important step towards AutoML.

Under review as a conference paper at ICLR 2020
IMPROVED TRAINING SPEED, ACCURACY, AND DATA UTILIZATION VIA Loss FUNCTION OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
As the complexity of neural network models has grown, it has become increas- ingly important optimize their design automatically through metaleaming. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss fune- tions can be optimized with metalearning as well, and result in similar improve- ments. The method, Genetic Loss-function Optimization (GLO), discovers loss functions de novo, and optimizes them for target task. Leveraging techniques from genetic programming. GLO builds loss functions hierarchically from a set operators and leaf nodes. These functions are repeatedly recombined and mutated to find an optimal structure, and then a covariance- -atunc adaptation evolutionary strategy (CMA-ES) is used find optimal coefficients. Networks trained with GLO loss functions are found to outperform the standard cross-entropy loss on standard image classification tasks. Training with these new loss functions re- quires fewer steps, results in lower test error, and allows for smaller datasets to be used. Loss function optimization thus provides a new dimension of metalearning. and constitutes an important step towards AutoML.
INTRODUCTION
Much the power modern neural networks originates from their complexity, i.e., number of parameters, hyperparameters, and topology. This complexity is often beyond human ability to op- timize, and automated methods are needed. An entire field metalearning has emerged recently to address this issue, based various methods such as gradient descent, simulated annealing, rein- forcement learning, Bayesian optimization, and evolutionary computation (FC) (Elsken et al. 2018).
While wide repertoire of work now exists for optimizing many aspects of neural networks, the dynamics of training are still usually set manually without concrete, scientific methods. Training schedules, loss functions, and learning rates all affect the training and final functionality of neural network. Perhaps they could also be optimized through metalcarning?
The goal of this paper is verify this hypothesis, focusing optimization of loss functions. A general framework for loss function metalcarning, covering both novel loss function discovery and optimization, is developed and evaluated experimentally. This framework. Genetic Loss-function Optimization (GLO), leverages Genetic Programming to build loss functions represented as trees, and subsequently Covariance-Mann Matrix Adaptation Evolution Strategy (CMA-ES) to optimize their coefficients.
EC methods were chosen because EC arguably the most versatile of the metalearning approaches. FCC, being a type of nopulation-based search method, allows for extensive exploration, which often results in creative, novel solutions (Lehman et al.] 2018). EC has been successful in hyperparameter optimization and architecture design in particular (Mikkulainen al. 2019 Stanley et al.) 2019 Real et al.] 2019) Loshchilov & Hutter (0166). has also been used to discover mathematical formulas to explain experimental data (Schmidt & Lipson 2009). It is, therefore, likely to find creative solutions in the loss-function optimization domain as well.
Indeed, the MNIST image classification benchmark, GLO discovered surprising loss func- tion, named Baikal for its shape. This function performs very well, presumably by establishing an implicit regularization effect. Baikal outperforms the standard cross-entropy loss in terms of training

---

Under review as a conference paper at ICLR 2020
speed, final accuracy, and data requirements. Furthermore, Baikal was found to transfer to more complicated classification task, CIFAR-10, 10, while carrying over its benefits.
At first glance. Baikal behaves rather unintuitively; loss does not decrease monotonically work's predictions become more correct. Upon further analysis, Baikal was found to perform im- plicit regularization, which caused this effect. Specifically, by preventing the network from being too confident its predictions, training was able to produce a more robust model. This finding was surprising and encouraging, since it means that GLO is able to discover loss functions that train networks that are more generalizable and overfit less.
net-
The next section reviews related work in metalearning and EC, to help motivate the need for GLO. Following this review, GLO is described in detail, along with the domains upon which it has been evaluated. The subsequent sections present the experimental results, including an analysis of the loss functions that GLO discovers.
RELATED WORK
In addition to hyperparameter optimization and neural architecture search, new opportunities for metalearning have recently emerged. In particular, learning rate scheduling and adaptation can have significant impact on model's performance. Learning rate schedules determine how the learning rate changes as training progresses. This functionality tends by different gradient-descent optimizers, such as AdaGrad (Duchi et al. (01I]) and Adam (Kingma Bal 2014). While the general consensus has been that monotonically decreasing learning rates yield good results, new ideas, such cyclical learning rates (Smith 2017). have shown promise in learning better models in fewer epochs.
encapsulated away in practice
Metalearning methods have also been recently developed for data augmentation, such as AutoAug- ment (Cubuk et al. 2018), a reinforcement learning based approach to find new data augmentation policies. In reinforcement learning tasks, EC has proven successful approach. For instance, in evolving policy gradients (Houthooft et al., 2018), the policy loss is not represented symbolically, but rather neural network that convolves over a temporal sequence of context vectors. In ward function search (Niekum et al. 2010)), the task is framed as genetic programming problem, leveraging PushGP (Spector et al. 2001).
In terms of loss functions, a generalization of the L loss was proposed with an adaptive loss param- eter (Barron 2017). This loss function is shown be effective domains with multivariate output spaces, where robustness might vary across between dimensions. Specifically, the authors found im- provements Variational Autoencoder (VAE) models, unsupervised monocular depth estimation, geometric registration, and clustering.
Additionally. work has found promise in moving beyond the standard cross-entropy loss for clas- sification (Janocha & Czarnecki [017]]. I and 7.2 losses were found to have useful probabilistic properties. The authors found certain loss functions to more resilient to noise than the cross- entropy loss.
Notably, no existing work the metalearning literature automatically optimizes loss functions for neural networks. As shown in this paper, evolutionary computation can be used in this role to improve neural network performance, gain a better understanding of the processes behind learning, and help reach the ultimate goal of fully automated learning.
3 THE GLO APPROACH
The task of finding and optimizing loss functions can be framed functional regression problem. GLO accomplishes this through the following high-level steps (shown in Figure (1) loss function discovery: using approaches from genetic programming, a genetic algorithm builds candidate loss functions, and (2) coefficient optimization: to further optimize a specific loss function, a covariance-matris adaptation evolutionary strategy (CMA-ES) leveraged to optimize coefficients.
2

---

Under review as a conference paper at ICLR 2020
(1) Loss function discovery genetic algorithm.
(2) Coefficient optimization via CMA-ES.
Vi
yi
y logG;)
G605 eg log(egy)))
Figure 1: Genetic Loss-function Optimization (GLO) overview. A genetic algorithm constructs candidate loss functions trees. The best loss functions from this set then has its coefficients optimized using CMA-ES. GLO loss functions are able to train models more quickly and more accurately.
3.1 LOss FUNCTION DISCOVERY
GLO uses a population-bast search approach, inspired by genetic programming, to discover new optimized loss function candidates. Under this framework, loss functions are represented as trees within a genetic algorithm. Trees are logical choice represent functions due their hierarchical nature. The loss function search space is defined by the following tree nodes:
Unary Operators: log(o), v
Binary Operators: +, Leaf Nodes: x, y, 1, 1,, where x represents true label, and y represents predicted label.
The search space is further refined by automatically assigning a fitness of 0 to trees that do not contain both at least one x and one y. Generally, a loss function's 's fitness within the genetic algorithm is the validation performance of network trained with that loss function. To expedite the discovery process, and encourage the invention of loss functions that make learning faster, training does not proceed to convergence. Unstable training sessions that result in NaN values assigned fitness of O. Fitness values are cached to avoid needing to retrain the same network twice. These cached values are each associated with canonicalized version of their corresponding tree, resulting in fewer required evaluations.
The initial population is composed of randomly generated cursively starting from the root, nodes are randomly chosen from the allowable operator and leaf nodes using weighting (where log(o), 2r y are three times likely and / is two times likely as +, *, - +, 1, 1).. This weighting can impart a bias and prevent, for example, the integer from occurring too frequently. The genetic algorithm has a population size of 80, incorporates elitism with six elites per generation, and uses roulette sampling.
with a maximum depth of 2. Re-
Recombination is accomplished by randomly splicing two trees together. For given pair of parent trees, a random element is chosen in each as crossover point. The two subtrees, whose roots are the two crossover points, are then swapped with each other. Figure presents an example this method of recombination. Both resultant trees become part of the next generation. Recombination occurs with a probability of 80%.
To introduce variation into the population, the genetic algorithm has the following mutations. applied
bottom-up fashion:
Integer scalar nodes are incremented decremented with a 5% probability.
Nodes are replaced with a weighted-rande random node with the same number of children with a 5% probability.
Nodes (and their children) are deleted and replaced with a weighted- random leaf node with 5% 50% 2.5% probability.
3

---

Under review as a conference paper at ICLR 2020
Leaf nodes deleted and replaced with weighted-randon element (and weighted- random leaf children if necessary) with a 5% 50% = 2.5% probability.
Combined, the iterative sampling, recombination, and mutation of trees within the population leads to the discovery of new loss functions which maximize fitness.
3.2 COFFFICIENT OPTIMIZATION
Loss functions found by the above genetic algorithm can all be thought of having unit coefficients for each node in the tree. This set of coefficients be represented equal to the number of nodes in a loss function's tree. The number of coefficients can be reduced by pruning away coefficients that can be absorbed by others (e.g.. (5x + 2y) 15x + 6y). The coefficient vector is optimized independently and iteratively using a covariance-matriy adaptation evolutionary strategy (CMA-ES) (Hansen & Ostermeier 1996). The specific variant of CMA-ES that GLO uses is (p/u, A)-CMA-ES (Hansen & Ostermeier (001), which incorporates weighted rank-ju updates (Hansen Kern 2004) to reduce the number of objective function evaluations that are needed. The implementation of GLO presented in this paper uses an initial step size = 1.5. As in the discovery phase, the objective function the network's performance on a validation dataset after shortened training period.
vector with dimensionality
EXPERIMENTAL EVALUATION
This section provides an experimental evaluation of GLO, on the MNIST and CIFAR-10 image classification tasks. Baikal, GLO loss function found on MNIST, is presented and evaluated in terms of resulting testing accuracy, training speed, training data requirements, and transferability to CIFAR-10. Implementation details presented in the appendix in Section A.I
4.1 TARGET TASKS
Experiments on GLO are performed using two popular image classification datasets, MNIST Hand- written Digits (LeCun et al.) 1998) and CIFAR-10 (Krizhessky & Hinton 2009). Both datasets, with MNIST in particular, are well understood, and relatively quick to train. The choice of these datasets allowed rapid iteration in the development of GLO and allowed time for more thorough experimentation. The selected model architectures are simple, since achieving state-of-the-art accu- racy on MNIST and CIFAR-10 10 is not the focus of this paper, rather the improvements brought about by using a GLO loss function are. More information on the datasets, along with their corresponding architectures and experimental setup is provided in the appendix, under Section A.2)
Both of these tasks, being classification problems, are traditionally framed with the standard cross- entropy loss (sometimes referred as the log loss): Loog - Σr-1 x; log(y;), where x is sampled from the true distribution, y is from the predicted distribution, and n is the number of classes. The cross-entropy loss is used as a baseline in this paper's experiments.
4.2
THE BAIKAL LOSS FUNCTION
The most notable loss function that GLO discovered against the MNIST dataset (with 2,000-step training for candidate evaluation) is the Baikal loss, named due to its similarity the bathymetry of Lake Baikal when its binary variant is plotted in 3D (Section
X;
Laakal
log(us)
(1)
y
=
where x is from the true distribution, y is from the predicted distribution, and is the number of classes. Additionally, after coefficient optimization, GLO arrived at the following version of the Baikal loss:
1 2.7279 0.9863 log(1.5352 * y:) 1.8158 I₁
LBakkaCMA =
(2)
72. 1=0
3/1
4

---

Under review as a conference paper at ICLR 2020
1.0000
Testing Accuracy
0,9625
Log Loss
0.9250
Baikal Loss
BaikalCMA Loss
0.8875
0.8500
Training
Figure 3: Training curves for different loss functions MNIST. Baikal and BaikalCMA result in faster and smoother training compared to the cross-entropy loss.
This loss function, BaikalCMA, was selected for having the highest validation accuracy out of the population. The Baikal and BaikalCMA loss functions had validation accuracies at 2,000 steps equal to 0.9838 and 0.9902, respectively. For comparison, the cross-entropy loss had a validation accuracy at 2,000 steps of 0.9700. Models trained with the Baikal loss MNIST and CIFAR-10 (to test transfer) are the primary vehicle for validating GLO's efficacy, as detailed subsequent sections.
4.3 TESTING ACCURACY
1.0000
4YV
Figure 2 shows the increase in testing accuracy that Baikal and BaikalCMA provide on MNIST over models trained with cross-entropy loss. Over trained models each, the mean testing accuracies for cross-entropy loss, Baikal, and BaikalCMA were 0.9899, 0.9933, and 0.9947, respectively.
Mean Test Accuracy
0.9950
0.9900
0.9850
This increase in accuracy from Baikal over cross-entropy loss found to be statistically significant, with a p-value of 2.4 x 10-11 in a heteroscedastie, two-tailed T-test, with 10 samples from each distribution. With the same significance test, the increase in accuracy from BaikalCMA over Baikal found to be statistically significant, with a p-value of 8.5045 x 10-6
0.9800
Log Loss Baikal BaikalCMA
Figure 2: Mean testing accuracy on MNIST, n = 10. Both Baikal and BaikalCMA provide statisti- cally significant improvements to testing accuracy over the cross- entropy loss.
4.4 TRAINING SPEED
Training curves for networks trained with the cross-entropy loss, Baikal, and BaikalCMA are shown in Figure 3 Each
curve represents 80 testing dataset evaluations spread evenly (i.e.. every 250 steps) throughout 20,000 steps training on MNIST. Networks trained with Baikal and BaikalCMA both learn signif- icantly faster than the cross-entropy loss. These phenomena make Baikal compelling loss function for fixed time-budget training, where the improvement in resultant accuracy over the cross-entropy loss becomes evident.
4.5 TRAINING DATA REQUIREMENTS
Figure provides an overview of the effects of dataset size on networks trained with cross-entropy loss. Baikal, and BaikalCMA. For each training dataset portion size, five individual networks were trained for each loss function.
The degree by which Baikal and BaikalCMA outperform cross-entropy loss increases the train- ing dataset becomes smaller. This provides evidence of less overfitting when training a network with Baikal BaikalCMA. As expected, BaikalCMA outperforms Baikal all tested dataset sizes. The size of this improvement in accuracy does not grow as significantly as the improvement over cross-entropy loss, leading the belief that the overfitting characteristies of Baikal and BaikalCMA

---

Under review as a conference paper at ICLR 2020
1.00
0.98
Testing Accuracy
0.96
0.94
Log Loss
0.92
Baikal Loss
0.90
BaikalCMA Loss
0.88
1.0
0.2
0.1
0.02
0.005
Training Dataset Portion
Figure 4: Sensitivity to different dataset sizes for different loss functions MNIST. For each size, n 5. Baikal and BaikalCMA increasingly outperform the cross-entropy loss on small datasets, providing evidence of reduced overfitting.
Log Loss
Baikal Loss
Testing Accurapy
0.8095
0.8343
0.8438 0.8453
0.7474 0.7606
0.7284
0.8205
0.8321 0.8379
0.7
0.6645
0.7729
0.7053
0.5788
0.6704
0.6450
0.55
0.5918
0.4937
0.4
500
1000
1500
2000
2500
5000
10000
15000
20000
Training Steps
Figure 5: Testing accuracy across varying training steps on CIFAR-10. The Baikal loss, which has been transferred from MNIST, outperforms the cross-entropy loss all training durations.
very similar. Ostensibly, one could run the optimization phase of GLO on reduced dataset specifically to yield a loss function with better performance than BaikalCMA on small datasets.
4.6 LOss FUNCTION TRANSFER CIFAR-10
Figure presents a collection of 18 separate tests of the cross-entropy loss and Baikal applied to CIFAR- 10. Baikal is found to outperform cross-entropy across all training durations, with the differ- ence becoming more prominent for shorter training periods. These results present an interesting use case for GLO, where loss function that is found on simpler dataset can be transferred complex dataset while still maintaining performance improvements. This faster training provides particularly persuasive argument for using GLO loss functions in fixed time-budget scenarios.
more
5 WHAT MAKES BAIKAL WORK?
This section presents a symbolic analysis of the Baikal loss function, followed by experiments that attempt elucidate why Baikal works better than the cross-entropy loss. likely explanation is that Baikal results in implicit regularization, reducing overfitting.
5.1 BINARY CLASSIFICATION
Loss functions used on the MNIST dataset, a 10-dimensional classification problem, are difficult plot and visualize graphically. To simplify, loss functions analyzed in the context of binary

---

Under review as a conference paper at ICLR 2020
800
800
Quantty
600
Quantiyy
600
400
400
200
200
8.0
8.0
0.8
1.0
0.8
1.0
Log Loss Network Output Probabilities
BaikaiCMA Network Output Probabilitees
Figure 7: Output probabilities of networks trained with cross-entropy loss and BaikalCMA. With BaikalCMA, the peaks shifted away from extreme values and more spread out, indicating im- plicit regularization. The BaikalCMA histogram matches that from a network trained with a confi- dence regularizer (Pereyra al. 2017).
classification, with 2, the Baikal loss expands to
LBaiall2
log(10)
To
+ log(y1)
(3)
y/
y1
Since vectors x and y sum by consequence of being passed through a softmax function, for binary classification x = (.o, 1 zo) and y = (yo. 1 - 9o). This constraint simplifies the binary Baikal loss the following function of two variables (zy and yu):
log( yo) + 1 - 20
LBakkllD log(vo)
(4)
yo
yo
This same methodology can be applied to the enoss-yn loss and BaikalCMA.
In practice, true labels are assumed be correct with certainty, thus, 2o is equal to either 0 or 1. is plotted in
30
The specific case where No Figure 6 for the cross-entropy loss, Baikal, and
Lees Kpe1
20
BaikalCMA. The cross-entropy loss shown to be monotonically decreasing, while Baikal and BaikalCMA counterintuitively show an in- crease in the loss value the predicted label, yo. approaches the true label Tg. This unex- pected increase allows the loss functions to pre- vent the model from becoming too confident in its output predictions, thus providing form of
Log Loss
10
Baikal
BalkalCMA
0.2
0.1
0.6
0.8
1.0
Prodicted Labol
Figure 6: Binary classification loss functions
To = 1. Correct predictions lic on the right side of the graph, and incorrect ones the left. The log loss decreases monotonically, while Baikal and BaikalCMA present counterintuitive, sharp increases in loss as predictions, approach the true label. This phenomenon provides regularization by preventing the model from being too confident in its predictions.
regularization. Section 5.2 provides reasoning for this unexplained result.
As also seen in Figure 6. the minimum for the Baikal loss where Fo while the minimum for the BaikalCMA loss where Zo
lies near 0.71,
lies near 0.77. This minimum, along with the more pronounced slope around 2o = 0.5 is likely a reason why BaikalCMA performs better than Baikal.
5.2 IMPLICIT REGULARIZATION
The Baikal and BaikalCMA loss functions are surprising in that they incur high loss when the output is very close to the correct value (as illustrated in Figure 6). Although at first glance this behavior is counterintuitive, may provide important advantage. The outputs of trained network will not exactly correct, although they are close, and therefore the network is less likely to overfit. Thus, these loss functions provide implicit form of regularization, enabling better generalization.

---

Under review as a conference paper at ICLR 2020
This effect is similar to that of the confidence regularizer (Pereyra al. 2017 which penalizes low-entropy prediction distributions. The bimodal distribution of output probabilities that results from confidence regularization is nearly identical to that of a network trained with BaikalCMA. His- tograms of these distributions on the test dataset for cross-entropy and BaikalCMA networks, after 15,000 steps of training on MNIST, are shown in Figure 7 The abscissae in Figures [ match, making it clear how the distribution for BaikalCMA has shifted away from the extreme values. The improved behavior under small-dataset conditions described in Section 4.5 further supports implicit regularization; less overfitting was observed when using Baikal and BaikaCMMA compared to the cross-entropy loss.
Notably, the implicit regularization provided by Baikal and BaikalCMA complements the different types of regularization already present in the trained networks. As detailed in Section (A.2 MNIST networks are trained with dropout (Hinton et al.] 2012), and CIFAR-10 networks are trained with I,a weight decay and local response normalization (Krizhevsky al. 2012), yet Baikal able to improve performance further.
DISCUSSION AND FUTURE WORK
This paper proposes loss function discovery and optimization as a new form of metalearning, and introduces an evolutionary computation approach to it. GL.O was evaluated experimentally in the image classification domain, and discovered a surprising new loss function, Baikal. Experiments showed substantial improvements accuracy, convergence speed, and data requirements. Further analysis suggested that these improvements result from implicit regularization that reduces overfit- ting to the data. This regularization complements the existing regularization in trained networks.
In the future, GLO can be applied to other machine learning datasets and tasks. The approach is general, and could result in discovery of customized loss functions for different domains, or even specific datasets. One particularly interesting domain is generative adversarial networks (GANs). Significant manual tuning is necessary in GANs to ensure that the generator and discriminator net- works learn harmoniously. GLO could find co-optimal loss functions for the generator and discrim- inator networks in tandem, thus making GANs more powerful, robust, and easier to implement.
GAN optimization is an example of co-evolution, where multiple interacting solutions devel- oped simultaneously. GLO could leverage co-evolution more generally: for instance, it could be combined with techniques like CoDeepNEAT (Mikkulainen et al., 2019) to learn jointly-optimal network structures, hyperparameters, learning rate schedules, data augmentation, and loss functions simultaneously. Such an approach requires significant computing power, but may also discover and utilize interactions between the design elements that result in higher complexity and better perfor- mance than is currently possible.
7 CONCLUSION
This paper proposes Genetic Loss-function Optimization (GLO) as a general framework for dis- covering and optimizing loss functions for given task. A surprising new loss function, Baikal, was discovered in the experiments, and shown to outperform the cross-entropy loss on MNIST and CIFAR- 10 in terms of accuracy, training speed, and data requirements. Further analysis suggested that Baikal's improvements result from implicit regularization that reduces overfitting to the data. GLO can be combined with other aspects of metalearning in the future, paving the way to robust and powerful AutoML.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Leven- berg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Va- sudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A sys- tem for lcrge machine learning. In 12th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI 16), pp. 265-283, Savannah, GA, 2016. USENIX Association.
8

---

Under review as a conference paper at ICLR 2020
ISBN 978-1-931971-33-1. URL https://www
/ttpt//////////oooooooo//o/ooooa Technical- Rechalccl-sassooon//reesstttttrn abadi
Jonathan Barron. A general and adaptive robust loss function. arXiv preprint arXiv:170.03377,,
2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint vrXiv:1805.09501 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(ul))2121-2155) 2011.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, arXiv
preprint arXiv: 05377, 2018.
Santiago Gonzalez, Joshua Landgraf, and Risto Miikkulainen. Faster training by selecting samples using embeddings. In 2019 International Joint Conference on Neural Networks (IJCNN), 2019.
Nikolaus Hansen and Stefan Kemn Evaluating the CMA evolution strategy on multimodal test functions. International Conference on Purallel Problem Solving from Nature, pp. 282-291. Springer, 2004.
Nikolaus Hansen and Andreas Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEEE international conference on evolutionary computation, pp. 312-317. IEEE, 1996.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2)119--155, 2001.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Advances in Neural Information Processing Systems, pp. 5400-5409, 2018.
Katarzyna Janocha and Wojciech Marian Czarnecki. On loss functions for deep neural networks in classification. arXiv preprint arXiv:1702.05657 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25. pp. 1097- 1105. Curran Associates, Inc.. 2012. URL http::/papers: nips cc/paper/ 1182-imagenet-on www.eenttt-lasssicaaitiioaoo with oaw.th-eee--coooooootiooaaaeeeee
pdf
Yann LeCun, Corinna Cortes, and CJC Burges. The MNIST dataset of handwritten digits, 1998.
Joel Lehman et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communitics. arXiv preprint arXiv:1803.03453 2018.
llya Loshchilov and Frank Hutter. CMA-ES for hyperparameter optimization of deep neural net- works. arXiv preprint 160.0766,. 2016.
Risto Mükkullainnn, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Nenworks and Brain Computing, pp- 293-312. Elsevier, 2019.

---

Under review as a conference paper at ICLR 2020
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Scott Niekum, Andrew G Barto, and Lee Spector. Genetic programming for reward function search. IEEE Transactions Autonomous Mental Development, 2(2):83-90, 2010.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4780-4789, 2019.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science,
324(5923):81-85, 2009. ISSN 0036-8075. doi: 10. 126/science. 165893. URL http:// science www.sciieeaaa..... org/ httg//oootent/32211
Leslie N Smith. Cyclical learning rates for training neural networks. 2017 IEEE Winter Confer- ence on Applications of Computer Vision (WACV), pp. 464-472. IEEE, 2017.
Lee Spector, Erik Goodman, Wu, W B. Langdon, Hm. Voigt, M Gen, S Sen, M Dorigo, Pezeshk, M Garzon, E Burke, and Morgan Kaufmann Publishers. Autoconstructive evolution: Push, PushGP, and Pushpop. Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), 05 2001.
Kenneth O. Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulaieen. Designing neural networks through neuroevolution. Nature Machine Intelligence. 1(1):24-35, 2019. ISSN
2522-5839. doi: 10.1088/4225-0088-0066 642256-018-00000
URL https:://ei..rr/....111.... 1038/
Douglas Thain, Todd Tannenbaum, and Miron Livny. Distributed computing in practice: the Condor experience. Concurrency and computation: practice and experience, 17(2-))323-336, 2005.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Due to the large number of partial training sessions that needed for both the discovery and optimization phases, training distributed across the network to a cluster dedicated machines that Condor TThin al. 2005) for scheduling. Each machine in this cluster has NVIDIA GeForce GTX Titan Black GPU and two Intel Xeon E5-2603 (4 core) CPUs running at 1.80GHz with 8GB of memory. Training itself implemented with TensorFlow (Abadi et al., (0166) in Python. The primary components of GLO (i.e., the genetic algorithm and CMA-ES) are implemented in Swift. These components run centrally on one machine and asynchronously dispatch work to the Condor cluster over SSH.
A.2
EXPERIMENTAL SETUP
The following two sections detail the experimental setup that was used for the evaluation presented this paper.
A.2.1 MNIST
The first target task used for evaluation was the MNIST Handwritten Digits dataset (LeCun et al., 1998), widely used dataset where the goal is classify 28 28 pixel images of ten dig-
its. The MNIST dataset has 55,000 training samples, 5,000 validation samples, and 10,000 testing
samples.
10

---

Under review as a conference paper at ICLR 2020
088
10
10
05
00
Log Loss
(b) Baikal Loss
Figure 8: Loss function surface plots for binary classification. is the prediction for one class, and is the true label for that class. Baikal Seannel-shapeee while the log loss has a saddle-like shape.
A simple CNN architecture with the following layers is used: (1) x 5 convolution with 32 filters, (2) 2 x2 stride-2 max max--nooling (3) 5 x convolution with 64 filters, (4) 2 x 2 stride-2 max-pooling (5) 1024-unit fully-connected layer, (6) dropout layer (Hinton al. 2012) with 40% dropout probability, and (7) a softmax layer. ReLU (Nair & Hinton (010) activations are used. Training uses stochastic gradient descent (SGD) with batch size of 100, learning rate of 0.01, and, unless otherwise specified, occurred over 20,000 steps.
Several experiments tested various learning rate values across handful of orders-of- to arrive at the step size used in the paper. For the baseline, this step size provided the highest accuracy.
A.2.2 CIFAR-10
To further validate GLO, the more challenging CIFAR-10 dataset (Krizhevsky & Hinton 2009), popular dataset of small, color photographs in classes, used medium to test the transferability of loss functions found on a different domain. CIFAR-10 consists of 50,000 training samples, and 10,000 testing samples.
A simple CNN architecture, taken from (Gonzalez et al., 2019) (and itself inspired by AlexNet
(Krizhevsky al., (012 with the following layers is used: (1) 5 convolution with 64 filters
and ReLU activations. (2) 3 x 3 max max-pooling with a stride of 2. (3) local response normalization
Krizhevsky et al. 2012) with & 1, 0.001/9, 8 0.75, (4) 5 x 5 convolution with 64 filters and ReLU activations, (5) local response normalization with A = 1, a = 0.001 4,001/9,2 B 0.75, (6) x max-pooling with stride of 2, (7) 384-unit fully-connected layer with ReLU activations, (8) 192-unit fully-connected, linear layer, and (9) softmax layer.
Inputs to the network are sized 24 x these smaller sized inputs enable more sophisticated data augmentation. To force the network to learn better spatial invariance, random 24 x 24 croppings are selected from each full-size image, randomly flipped longitudinally, randomly lightened or darkened, and their contrast is randomly perturbed. Furthermore, to attain quicker convergence, an image's 's mean pixel value and variance are subtracted and divided, respectively, from the whole image during training and evaluation. CIFAR- 10 networks were trained with SGD, L regularization with weight decay of 0.004, a batch size of 1024, and initial learning rate of 0.05 that decays by factor of 0.1 every 350 epochs.
3, rather than 32 32 x 3 as provided in the dataset;
Several experiments tested various initial learning rate values across a handful of orders-of- magnitude arrive the step size used in the paper. For the baseline, this initial learning rate provided the highest accuracy.
A.3 BINARY CLASSIFICATIONS SURFACE PLOTS
When plotted in three-dimensions, as in Figure X the binary cross-entropy and Baikal loss functions can be observed to have characteristic surfaces. The shape of Baikal's surface, and its similarity to the bathymetry of Lake Baikal, is where it gets its name. Note that the case plotted in Figure is equivalent to the front "slice" of the surface plots in Figure
11


Reviews:
[{"review_id": "rkgPrHM4tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a framework to perform meta-learning on the loss used for\ntraining. They introduce the Baikal loss, obtained using the MNIST dataset, and\nBaikalCMA where the coefficients have been tuned. The evaluation of these loss\nfunctions is performed on the MNIST and CIFAR-10, and according to the results\nthey converge faster, towards lower test error and need fewer samples to obtain\nresults similar to the cross-entropy loss.\n\nThe claims are clearly stated and the framework is detailed, the experiments\ncover all the potential benefits of the Baikal loss. However it seems that some\npotentially critical points have been omitted. The cross-entropy loss is well\nknown to be beneficial in dataset with severe class imbalance. The two datasets\nused for evaluation are perfectly balanced, it might beneficial to see how it\nperforms in the unbalanced case.\n\nI have a couple of concerns about the method. First about step \"(1) loss\nfunction discovery\": The initial population starts with trees of depth at most\n2, and the final solution(Baikal) has either 2 or 3 (depending on which\ndefinition of depth is chosen). It is unclear that the genetic optimization is\nsuperior to simply choosing random loss functions. I think it would be relevant\nto add a figure that shows how the fitness of the leader of each generation\nevolves over time.\n\nThe second step \"(2) coefficient optimization\", while objectively generating a\nloss function that was superior on the metrics evaluated, raised some\nquestions. In equation (2) the factor \"1.5352\" seems to be equivalent to adding\na constant to the loss, which should not impact optimization. Also the factor\n\"2.7279\" seems to be equivalent to a change in learning rate. This may be an\nindication that the learning rate search was not done thoroughly. It would be\nbeneficial to clarify when it is happening: a) For each individual of the\npopulation during step (1), b) before performing CMA, c) after CMA. Also: Was\nlearning rate search was performed on the network trained with Cross-Entropy? It\nwas not entirely clear from the experiment details in Appendix A.2.1.\n\nAbout the Baikal loss itself, I fear that it could produce models that have very\npoor calibration, it might be nice to evaluate that (even if it is only in\nthe appendix).\n\n\nWhile the paper does a great job at presenting the problem and its applications\nand propose a framework that generated a loss that can transfer to other\ndatasets without any tuning required. I think it lacks a more thorough\nevaluation and description of the dynamics observed during the genetic\nevolution, and the performance of the Baikal loss on other datasets (my quick\nexperients with it on ImageNet diverged I did not have the time necessary to\ntune the hyper-parameters).\n\nMinor remarks:\n\nThere might be a slight omission in section 3.1: according to Figure 1, exp(x)\nis one of the potential unary operators explored by the GLO framework. However\nit is not present it the list of operators. Could you clarify this?\n\nTo the best of my knowledge, in the machine learning literature, it seems that\nthe letter x is used to denote the prediction and y for the ground truth. The\nfact that this paper used the opposite convention confused me the first time I\nread it.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJg3SzKCFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\nThe authors propose using evolutionary computation (EC) to perform meta learning over the set of symbolic expressions for loss functions. It's a compelling idea that is well-motivated. They find that applying their EC method to mnist yields an interesting loss function that they name the 'Baikal loss.' Much of the paper is devoted to analyzing the properties and performance of the Baikal loss. \n\n*Overall Assessment*\nThe paper's idea is very interesting. However, there are some important drawbacks of this work. These should be fixed and the paper should be resubmitted to a different conference soon.\n1) The experiments focus almost entirely on the Baikal loss (a particular loss function found once when running EC on mnist), and do not analyze the overall behavior of EC for loss functions. Does EC consistently converge to the same loss, or do different ones emerge different times you run it? What happens if you optimize convergence speed vs. generalization accuracy with EC? How do these loss functions differ?\n2) The experiments are largely on mnist, with a small study showing that the Baikal loss can be applied to cifar-10. It would be good to show that loss functions meta-learned on mnist generalize to larger-scale problems than cifar. \n\n*Comments*\nI was surprised when you optimized in fig 3 for convergence speed, rather than final accuracy of something that runs for a while. Why should our goal be to find loss functions that lead to fast optimization, instead of loss functions that lead to models that generalize best? If these are two different goals, then you should have two sets of experiments analyzing how GLO can find interesting (and perhaps different) loss functions for each.\n\nMnist is possible to get basically 100% accuracy. This means that the loss will only be evaluated in certain regimes of its inputs. What happens when you transfer this to problems where the best achievable accuracy is something like 60% for binary classification?\n\nYou should cite the Focal loss as another alternative to the cross entropy loss. Is the focal loss achievable in your particular grammar over loss functions? You should also cite label smoothing as an additional way to achieve a very similar implicit regularization effect as the Baikal loss.\n\nYou only analyze one loss function that came from your EC. What if you run it multiple times? Do you find different formulas? How do these perform? The beginning of the paper is very focused on EC, but then you transition suddenly to only discussing the Baikal loss. Can you present experiments demonstrating, for example, how the EC performance varies with the number of steps, with different ways to define the search space, etc?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJenQ5wnYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a very interesting idea of loss function optimization. At first sight, loss function is the goal of optimization and can not be optimized directly. However, the true goal of optimization is the final accuracy (for classification). So lots of loss functions can be designed and combined to form a large search space. In this paper, the authors adopt genetic programming to design loss functions hierarchically.  And experiments show that GLO (Genetic Loss-function Optimization) based loss function can achieve better results than cross entropy. \n\nThe paper is well written and easy to understand. I like the idea. Baikal loss is a form searched by GLO. Interestingly and counter intuitively , it is not a monotonically decreasing function. The authors explain it as a regularizer which can prevent the model to be too confident. \n\nExperiments on MNIST and Cifar10 are conducted to show the effectiveness of the proposed method. This part is very weak since MNIST and Cifar10 are very small datasets and the provided results are far from state-of-the-art results. Experiments on larger datasets such as ImageNet and more analysis about the optimization details are suggested to make this work more promising. Since the optimization is rather complex, it's better to show if it is stable enough to generalize to various datasets and models.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a GA-based method for optimizing the loss function a model is trained on to produce better models (in terms of final performance). The general consensus from the reviewers is that the paper, while interesting, dedicates too much of its content to analyzing one such discovered loss (the Baikal loss), and that the experimental setting (MNIST and Cifar10) is too basic to be conclusive. It seems this paper can be so significantly improved with some further and larger scale experiments that it would be wrong to prematurely recommend acceptance. My recommendation is that the authors consider the reviewer feedback, run the suggested further experiments, and are hopefully in the position to submit a significantly stronger version of this paper to a future conference.
Decision: Reject