Optimizing Loss Functions Through Multivariate Taylor Polynomial Parameterization
Authors: Santiago Gonzalez
Risto Miikkulainen
Venue:  (2021)
Keywords: taylorglo
loss function
metalearning
evolution
deep networks
evolutionary strategies
taylor polynomials
glo

Abstract: Metalearning of deep neural network (DNN) architectures and hyperparameters has become an increasingly important area of research. Loss functions are a type of metaknowledge that is crucial to effective training of DNNs, however, their potential role in metalearning has not yet been fully explored. Whereas early work focused on genetic programming (GP) on tree representations, this paper proposes continuous CMA-ES optimization of multivariate Taylor polynomial parameterizations. This approach, TaylorGLO, makes it possible to represent and search useful loss functions more effectively. In MNIST, CIFAR-10, and SVHN benchmark tasks, TaylorGLO finds new loss functions that outperform functions previously discovered through GP, as well as the standard cross-entropy loss, in fewer generations. These functions serve to regularize the learning task by discouraging overfitting to the labels, which is particularly useful in tasks where limited training data is available. The results thus demonstrate that loss function optimization is a productive new avenue for metalearning.

Under review as a conference paper at ICLR 2021
OPTIMIZING Loss FUNCTIONS THROUGH MULTI- VARIATE TAYLOR POLYNOMIAL PARAMETERIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Metalearning of deep neural network (DNN) architectures and hyperparameters has become an increasingly important area of research. Loss functions are type of metaknowledge that is crucial to effective training of DNNs, however, their potential role in metalearning has not yet been fully explored. Whereas early work focused genetic programming (GP) on tree representations, this paper proposes continuous CMA-ES optimization of multivariate Taylor polynomial parameterizations.. This approach, TaylorGLO, makes it possible represent and search useful loss functions more effectively. In MNIST, CIFAR- 10, and SVHN benchmark tasks, TaylorGLO finds new loss functions that outperform the standard cross-entropy loss as well as novel loss functions previously discovered through GP, in fewer generations. These functions serve regularize the learning task by discouraging overfitting to the labels, which is particularly useful in tasks where limited training data available. The results thus demonstrate that loss function optimization is a productive new avenue for metalcarning.
INTRODUCTION
As deep learning systems have become more complex, their architectures and hyperparameters have become increasingly difficult and time-consuming optimize by hand. In fact, many good designs may be overlooked by humans with prior biases. Therefore, automating this process, known as metalearning. has become an essential part of the modern machine learning toolbox. Metalcarning aims to solve this problem through variety of approaches, including optimizing different aspects of the architecture from hyperparameters to topologies, and by using different methods from Bayesian
optimization to evolutionary computation (Schmidhuber 1987 Elsken et al. 2019 Miikkulainen
ctt al. 2019 Lemke et al. 2015).
Recently, loss-function discovery and optimization has emerged as new type of metalearning. Focusing on neural network's 's root training goal it aims to discover better ways to define what being optimized. However, loss functions can be challenging to optimize because they have discrete nested structure as well as continuous coefficients. The first system to do so, Genetic Loss Optimization (GLOO; Gonzalez & Miikkulainen 2020) tackled this problem by discovering and optimizing loss functions in two separate steps: (1) representing the structure as trees, and evolving them with Genetic Programming (GP: Banzhaf al. [998]; and (2) optimizing the coefficients using Covariance-Matrix Adaptation Evolutionary Strategy (CMA-ES; Hansen & Ostermeier, 1996). While the approach was successful, such separate processes make it challenging find mutually optimal structure and coefficients. Furthermore, small changes in the tree-based search space do not always result in small changes in the phenotype, and can easily make a function invalid, making the search process ineffective.
In an ideal case, loss functions would be mapped into fixed-length vectors in a Hilbert space. This mapping should be smooth, bell-behaved, well-defined, incorporate both function's 's structure and coefficients, and should by its very nature exclude large classes of infeasible loss functions. This paper introduces such an approach: Multivariate Taylor expansion-based genetic loss-function optimization (TaylorGLO). With a novel parameterization for loss functions, the key pieces of information that affect loss function's behavior are compactly represented in a vector. Such vectors are then optimized for a specific task using CMA-ES. Special techniques can be developed to narrow down the search space and speed up evolution.

---

Under review as a conference paper at ICLR 2021
Loss functions discovered by TaylorGLO outperform the standard cross-entropy loss (or log loss) on the MNIST, CIFAR-10, 10, CIFAR-100, and SVHN datasets with several different network architectures. They also outperform the Baikal loss, discovered by the original GLO technique, and do it with significantly fewer function evaluations. The reason for the improved performance is that evolved functions discourage overfitting to the class labels, thereby resulting in automatic regularization. These improvements are particularly pronounced with reduced datasets where such regularization matters the most. TaylorGLO thus further establishes loss-function optimization as a promising new direction for metalearning.
RELATED WORK
Applying deep neural networks to new tasks often involves significant manual tuning of the network design. The field of metalearning has recently emerged to tackle this issue algorithmically (Schmid) huber 1987 Lemke et al. 2015 Elsken et al. 2019 Mikkullinnnn et al. 2019). While much of the work has focused on hyperparameter optimization and architecture search, recently other aspects, such activation functions and learning algorithms, have been found useful targets for optimization (Bingham et al. 2020 Real et al. 2020). Since loss functions are at the core of machine learning, it is compelling to apply metalearning to their design as well.
Deep neural networks are trained iteratively, by updating model parameters (i.e., weights and biases) using gradients propagated backward through the network (Rumelhart et al. 1985). The process starts from error given by a loss function, which represents the primary training objective of the network. In many tasks, such as classification and language modeling, the cross-entropy loss (also known as the log loss) has been used almost exclusively. While in some approaches a regularization term (e.g. 2 weight regularization; Tikhonov 1963) is added to the the loss function definition, the core component is still the cross- entropy loss. This loss function motivated by information theory: It aims to minimize the number of bits needed to identify a message from the true distribution, using a code from the predicted distribution.
In other types of tasks that do not fit neatly into a single-label classification framework different loss functions have been used successfully (Gonzalez al. 2019 Gao & Grauman 2019 Kingma & Welling, 2014; Zhou et al.] 2016 Dong et 2017). 7) Indeed, different functions have different properties; for instance the Huber Loss (Huber 1964) is more resilient to outliers than other loss functions. Still, most of the time one of the standard loss functions is used without justification; therefore, there is an opportunity to improve through metalearning.
Genetic Loss Optimization (GLO; Gonzalez & Miikkulainen/ 2020) provided initial approach into metalearning of loss functions. As described above, GLO is based on tree-based representations with coefficients. Such representations have been dominant in genetic programming because they flexible and can be applied to a variety of function evolution domains. GLO was able to discover Baikal, new loss function that outperformed the cross-entropy loss in image classification tasks. However, because the structure and coefficients are optimized separately in GLO, it cannot easily optimize their interactions. Many of the functions created through tree-based search are not useful because they have discontinuities, and mutations can have disproportionate effects on the functions. GLO's search is thus inefficient, requiring large populations that are evolved for many generations. Thus, GLO does not scale to the large models and datasets that are typical in modern deep learning.
The technique presented in this paper, TaylorGLO, aims to solve these problems through novel loss function parameterization based on multivariate Taylor expansions. Furthermore, since such representations are continuous, the approach can take advantage of CMA-ES (Hansen & Ostermeier) 1996) as the search method, resulting in faster search.
3 LOss FUNCTIONS AS MULTIVARIATE TAYLOR EXPANSIONS
Taylor expansions (Taylor 1715) are a well-known function approximator that can represent dif- ferentiable functions within the neighborhood of point using a polynomial series. Below, the common univariate Taylor expansion formulation is presented, followed by a natural extension to aubirarily-multiivriate functions.
2

---

Under review as a conference paper at ICLR 2021
Given C. smooth (i.e., first through kmax derivatives continuous), real-valued function, f(x) - R, kth-order Taylor approximation at point E R. fx a), where be constructed as
kmaxs can
(x 0) Σ n.
(() a)"
Conventional, univariate Taylor expansions have natural extension to arbitrarily high-dimensional inputs of f. Given a Ainu+++ smooth, real-valued function, f(x) R + R, a kth-order Taylor approximation at point E R", (x. a), where 0 < k S kmax. can be constructed. The stricter smoothness constraint compared the univariate case allows for the application Schwarz's theorem equality of mixed partials, obviating the need to take the order of partial differentiation into account.
Let us define an nth-ddgrre multi-index, (a1, 02...... an), where as E No. |a a! = =I= ag!. x" I ot and x E R". Multivariate partial derivatives can be concisely written using a amulti-indes
> Giml ag.
Ola
00 8 0 ...0 0.
Ou 21 0x Ou
Thus, discounting the remainder term, the multivariate Taylor expansion for f(x) at is
1 (x. = Σ f(a) a!
(3)
Vo.lo/sk
The unique partial derivatives in fee and are parameters for kth order Taylor expansion. Thus, kth order Taylor expansion of a function in n variables requires n parameters to define the center, a, and one parameter for each unique multi-index a, where |a| k. That is: # parameters (n,,)) + ("+th)
(n++)) R
The multivariate Taylor expansion be leveraged for novel luns-funttion parameterization. Let an n-class classification loss function be defined as Logg = - f(x,, 3a). The function f(I,, vi) be replaced by its kthhorder, bivariate Taylor expansion, f& (Ir, y, az ay)- More sophisticated loss functions can be supported by having more input variables beyond and V/. such as time variable or unscaled logits. This approach can be useful, for example, evolve loss functions that change as training progresses.
For example, a loss function and y has the following third-order parameterization with parameters 0 (where (9o. 01)):
C(x,y)
02 0z(y: - 0,) 104(v: - 0,)3 1 - 01)3 + 0o(1; - 0n)
1=1
+07(x; 00) 01) 10 00) 01)² + f00 - 0o)² (4) + +1000(1; -00)² 01) 10, 0n)3)
Notably, the reciprocal- factooria coefficients can be integrated to be part of the parameter set by direct multiplication desired.
As will be shown in this paper, the technique makes it possible to train neural networks that are more accurate and learn faster than those with tree-based loss function representations, Representing loss functions in this manner confers several useful properties:
It guarantees smooth functions;
Functions do not have poles (i.c., discontinuities going to infinity or negative infinity) within
their relevant domain;
They can be implemented purely as compositions of addition and multiplication operations;
They can be trivially differentiated;
Nearby points in the search space yield similar results (i.e., the search space is locally
smooth), making the fitness landscape easier to search;
be found in fewer generations and with higher frequency; Loss function discovery is consistent and not dependent on a specific initial population; and
Valid loss functions
The search space has tunable complexity parameter (i.e., the order of the expansion).
3

---

Under review as a conference paper at ICLR 2021
These properties not necessarily held by alternative function approximators. For instance:
Fourier series are well suited for approximating periodic functions [Fourier 1829). Consequently, they are not well suited for loss functions, whose local behavior within narrow domain is important. Being a composition of waves, Fourier series tend to have many critical points within the domain of interest. Gradients fluctuate around such points, making gradient descent infeasible. Additionally, close approximations require a large number of terms, which itself can be injurious, causing large. seeerrreenncyy fluctuations known as "ringing", due to Gibb's s phenomenon (Wilbraham) 1848).
Padé approximants can be more accurate approximations than Taylor expansions; indeed, Taylor expansions are a special case of Padé approximants where M = 0 (Graves-Morris 1979). However, unfortunately Padé approximants can model functions with one or more poles, which valid loss functions typically should not have. These problems still exist, and are exacerbated, for Chisholm approximants (a bivariate extension; Chisholm, 1973) and Canterbury approximants (a multivariate generalization; Graves- & Roberts 1975). Laurent polynomials can represent functions with discontinuities, the simplest being x x= -While Laurent polynomials provide generalization of Taylor expansions into negative exponents, the extension is not useful because it results in the same issues as Padé approximants. Polyharmonic splines can represent continuous functions within finite domain, however, the number of parameters is prohibitive in multivariate cases.
The multivariate Taylor expansion therefore a better choice than the alternatives. It makes possible to optimize loss functions efficiently in TaylorGLO, as will be described next.
4 THE TAYLORGLO METHOD
Initial Solution Vector
Candidate Evaluation
Build TeylorGLO Loss Function
TaylorGLO (Figure 1 aims to find the optimal param- eters for a loss function represented as a multivariate Taylor expansion. The parameters for Taylor ap- proximation (i.e., the center point and partial deriva- tives) referred to as fy: 8 θ, θ IR paiaeters TaylorGLO strives to find the vector 8 that parameter-
CMA-ES
Vactor
Partial Model Training (Fo Epochs)
Covarfance Matnx
Sampler
Solution
Validtion Set Evaluation
izes the optimal loss function for a task. Because the values continuous, opposed to discrete graphs of the original GL.O, is possible to use continuous optimization methods.
Build TeylorLLO Loss Function
Σ (l n
1
In particular, Covariance Matrix Adaptation Evolution- ary Strategy HOMM-EESaansen & Ostermeier (996) is a popular population-based, black-box optimization technique for rugged, continuous spaces. CMA-ES functions by maintaining covariance matrix around a mean point that represents a distribution of solutions. At each generation, CMA-ES adapts the distribution to better evaluated objective values from sampled individuals. In this manner, the area in the search space that is being sampled at each step grows, shrinks, and moves dynamically as needed to maximize sampled candidates' fitnesses. TaylorGLO uses the (1/p. X) variant of CMA-ES (Hansen & Ostermeier 2001). which incorporates weighted rank-p updates (Hansen & Kern 2004) to reduce the number of objective function evaluations needed.
The TaylorGLO method. Start- ing with a population of initially unbiased loss functions, CMA-ES optimizes their Tay- lor expansion parameters order to maxi- mize validation accuracy after partial train- ing. The candidate with the highest accuracy is chosen as the final, best solution.
Figure 1:
In order to find 07. each generation CMA-ES samples points in O. Their fitness is determined by training a model with the corresponding loss function and evaluating the model on a validation dataset. Fitness evaluations may be distributed across multiple machines in parallel and retried limited number of times upon failure. An initial vector of 0 O is chosen as a starting point in the search space to avoid bias.
Fully training model can be prohibitively expensive in many problems. However, performance the beginning of training is usually correlated with performance at the end of training, and therefore it is enough train the models only partially to identify the most promising candidates. This type of
4

---

Under review as a conference paper at ICLR 2021
approximate evaluation is common in metalearning (Grefenstette & Fitzpatrick 1983 Jin 201 An additional positive effect is that evaluation then favors loss functions that learn more quickly.
For loss function to be useful. it must have derivative that depends on the prediction. Therefore, internal terms that do not contribute to By , can be trimmed away. This step implies that any
term within f(x; y4) with 2t can be replaced with O. For example, this refinement simplifies ay. Equation 4 providing a reduction in the number of parameters from twelve eight:
((x,y) 1
O2(yi 01) 200 - 01)2 1 01)² 3 0z(x; 00) 01) +200(x: (o)(v: 01)² 20r(2; 0o) (y: 81)
(5)
EXPERIMENTAL SETUP
This section presents the experimental setup that was used to evaluate the TaylorGLO technique.
Domains: MNIST LeCun et al., 1998) included simple domain to illustrate the method and to provide a backward comparison with GLO;; CIFAR-10 10 (Krizhevsky & Hinton (009). CIFAR-100 (Krizhevsky & Hinton 2009), and SVHN (Netzer al.,) 2011) were included as more modern benchmarks. Improvements were measured in comparison to the standard cross-entropy loss function Lieg = - - 2 log(y;), where is sampled from the true distribution, y from the predicted distribution, and n is the number of classes.
Evaluated architectures:
variety of architectures CNN architecture evaluated in the GLO study (Gonzalez& & Mikkullanenn 2020), AlexNet (Krizhessyy
used evaluate TaylorGLOO: the basic
al. 2012). ALICNN-C (Springenberg al., 2013), Preactivation ResNet-20 (He al.) 2016a)). which is an improved variant of the ubiquitous ResNet architecture (He et al. 6b). and Wide
ResNets of different morphologies (Zagoruyko Komodakis (016)) Networks with Cutout Vries & Taylor 2017) and CutMix (Yun al. 2019) were also evaluated, to show that TaylorGLO provides a different, complementary approach to regularization.
TaylorGLO setup: CMA-ES was instantiated with population size À 28 on MNIST and À 20 on all other datasets, and an initial step size a 1.2. These values were found to work well in preliminary experiments. The candidates were third-order (i.e., k 3) TaylorGLO loss functions (Equation Such functions were found experimentally to have a better trade-off between evolution time and performance compared second- and fourth-order TaylorGI.O loss functions, although the differences were relatively small.
Further experimental setup and implementation details provided in Appendix
6 RESULTS
2k-0hpp Valcaatin Accomney
0.965
This section illustrates the TaylorGLO process and demonstrates how the evolved loss func- tions can improve performance over the standard cross-entropy loss function, especially on reduced datasets. A summary of results three datasets across a variety of models are shown in Table
0.99
0.8
MW
New Best
0.8
AllTime Best
Generation Best
0.4
Generation Average
10
20
30
40
50
60
Generation
6.1 THE TAYLORGLO DISCOVERY PROCESS
Figure 2: The process of discovering loss func- tions in MNIST. Red dots mark generations where new improved loss functions were found. TaylorGLO discovers good functions in very few generations. The best had a 2000-step vali- dation accuracy 0.9948, compared 0.9903 with the cross-entropy loss, averaged over ten runs. This difference translates a similar im- provement on the test set, as shown in Table [T.
Figure illustrates the evolution process over 60 generations, which is sufficient to reach conver- gence on the MNIST dataset. TaylorGLO is able to discover highly-pertee pirfoy-performing loss functions quickly, ie. within 20 generations. Generations' average validation accuracy approaches generations' best accuracy as evolution progresses, indicating that population whole is improving. Whereas
5

---

Under review as a conference paper at ICLR 2021
Table 1: Test-set accuracy of loss functions discovered by TaylorGI.O compared with that of the cross-entropy loss. The TaylorGLO results are based on the loss function with the highest validation accuracy during evolution. All averages are from ten separately trained models and p-values are from one-tailed Welch's 's t- t-Tssss. Standard deviations are shown in parentheses. TaylorGLO discovers loss functions that perform significantly better than the cross-entropy loss in almost all cases, including those that include Cutout, suggesting that it provides different form of regularization.
Task and Model
Avg. TaylorGLO Acc.
Avg. Baseline Acc.
p-value
MNIST
Basic CNN
0.9951 (0.0005)
0.9899 (0.0003)
2.95x10-15
CIFAR-10
AlexNet 2
0.7901 (0.0026)
0.7638 (0.0046)
1.76x10
AlexNet Cutout
0.7786 (0.0022)
0.7741 (0.0040)
0.0049
AlexNet CutMix
0.7928 (0.0027)
0,7856 (0.0026)
8.13x10-1
PreResNet-20 +
0.9169 (0.0014)
0,9153 (0.0021)
0.0400
AIICNN-C2
0,9271 (0.0013)
0.8965 (0.0021)
0,42x10
AICCNN-C + Cutout 6
0.9329 (0.0022)
0.8911 (0.0037)
1.60110
ALICNN-C + CutMix
0,9327 (0.0014)
0.8749 (0.0042)
1.89
Wide ResNet 16-8*
0,9558 (0.0011)
0.9528 (0.0012)
1.77x10
Wide ResNet 16-8 + Cutout
0,9618 (0.0010)
0,9582 (0.0011)
2.55100-1
Wide ResNet 28-5
0.9548 (0.0015)
0.9556 (0.0011)
0.0984
Wide ResNet 28-5 + Cutout
0,9621 (0.0013)
0.9616 (0.0011)
0.1882
CIFAR-100
PyramidNet 110a48
0.7409 (0.0040)
0.7523 (0.0037)
3.87x10-6
PyramidNet 110a48 + Cutout
0.7708 (0.0029)
0.7674 (0.0036)
0.0189
SVHN
Wide ResNet 16-8
0.9658 (0.0007)
0.9597 (0.0006)
1.94x10
Wide ResNet 16-8 + Cutout
0.9714 (0.0010)
0,9673 (0.0008)
9.10x10-1
Wide ResNet 28-5
0.9657 (0.0009)
0.9634 (0.0006)
6.62x 10
Wide ResNet 28-5 + Cutout
0.9727 (0.0006)
0.9709 (0.0006)
2.96x 10-6
Network architecture references: Gonzalez & Milkkulainne
Krizhevsky et al.
2012
Springenberg at 2015
He et al
2016a
Zagoruyko & Komodakis
2016
DeViies & Taylor 2017
Yun Ytnetta al
2019
Han et al. 2017)
GLO's unbounded search space often results in pathological functions, every TaylorGLO train- ing session completed successfully without any instabilities.
Figure shows the shapes and parameters of each generation's highest-scoring loss function. In Figure Bh the functions are plotted as if they were being used for binary classification, i.e. the loss
for an incorrect label on the left and for correct one on the right (Gonzalez & Miikkulainen) 2020). The functions have distinct pattern through the evolution process. Early generations include wider variety of shapes, but they later converge towards curves with a shallow minimum around No = 0.8. In other words, the loss increases near the correct output-which is counterintuitive. This shape is also strikingly different from the cross-entropy loss, which decreases monotonically from left to right, one might expect all loss functions to do. The evolved shape is effective most likely because can provide an implicit regularization effect: discourages the model from outputting unnecessarily extreme values for the correct class, and therefore makes overfitting less likely (Gonzalez & Mikkkulainen) 2020). This is a surprising finding, and demonstrates the power of machine learning create innovations beyond human design.
6.2 PERFORMANCE COMPARISONS
Over 10 fully-trained models, the best TaylorGLO loss function achieved a mean testing accuracy of 0.9951 (stddev 0.0005) in MNIST. In comparison, the cross-entropy loss only reached 0.9899 (siddev 0.0003), and the "BaikalCMA" loss function discovered by GLO, 0.9947 (stddev 0.0003) Gonzalez & Miikkulainen) 2020): both differences are statistically significant (Figure Notably, TaylorGLo achieved this result with significantly fewer generations. GLO required 11,120 partial evaluations (i.e., 100 individuals over 100 GP generations plus 32 individuals over 35 CMA-ES generations),

---

Under review as a conference paper at ICLR 2021
Relatov Loss atx1=1
15
Punumate Value
40
-6
-10
0.8
0.8
40
Predicted Label (Po)
Generation
Ganesation
(a) Best discovered functions over time
(a) Best function parameters over time
Figure 3: The best loss functions (a) and their respective parameters (b) from each generation of T on MNIST. The functions are plotted in binary classification modality, showing loss for different values of the network output (yo the horizontal axis) when the correct label 1.0. The functions are colored according to their generation from blue to red, and vertically shifted such that their loss at yo is zero (the raw value of a loss function is not relevant; the derivative, however, is). TaylorGLO explores varying shapes of solutions before narrowing down on functions in the red band; this process can also be seen in (b), where parameters become more consistent over time, and in the population plot of Appendix B The final functions decrease from left to right, but have significant increase in the end. This shape is likely to prevent overfitting during learning, which leads to the observed improved accuracy.
Perfurbed Weights
Perturbed Weights
Cross-entropy loss test-set accuracy
TaylorGa.O loss teet- accuracy
Figure 4: Accuracy basins for AIICNN-C models trained with both cross-entropy and TaylorGLO loss functions. The TaylorGLO basins are both flatter and lower, indicating that they more robust and generalize better (Keskar et al. 2017), which results in higher accuracy.
while the top TaylorGLO loss function only required 448 partial evaluations, i.e. 4.03% as many. Thus, TaylorGLO achieves improved results with significantly fewer evaluations than GLO.
Due to the very large number evaluations required by GLO, TaylorGLO is only compared to GLO on MNIST. GLO is not practically applicable deeper models with longer training times. For example, even a relatively small deep network. PreResNet-201 He et al. (2016a), would require over 171 GPU days of computation, assuming the same number of evaluations as above on MNIST.
The large reduction in evaluations during evolution compared to GLO allows TaylorGLO to tackle harder problems, including models that have millions of parameters. On CIFAR-IÓ, CIFAR-100, and SVHN, TaylorGLO was able to outperform cross-entropy baselines consistently on variety models, as shown in Table [ These increases in accuracy are greater than what is possible through implicit learning rate adjustment alone (detailed in Appendix [E. TaylorGLO also provides further improve- ment on architectures that use Cutout Vries & Taylor (017). suggesting that mechanism of avoiding overfitting is different from other regularization techniques.
In addition, TaylorGLO loss functions result in more robust trained models. In Figure accuracy basins for two AIICNN-C models, one trained with the TaylorGLO loss function and another with the cross-entropy loss, are plotted along a -wo-dimensional slice [-1, 1] of the weight space (a technique due to Li et al., 2018). The TaylorGLO loss function results in flatter, lower basin. This result suggests that the model is more robust, i.e. its performance is less sensitive to small perturbations in the weight space, and it also generalizes better (Keskar et al. 2017).
7

---

Under review as a conference paper at ICLR 2021
1,0000
12000
Accuracy
Portial Evaluntons
0.9960
5000
Tecing Acouracy
0.86
0.9900
6000
0.84
Cross-Eniropy
0.9850
3000
TeylorGLO
0.80
Log Lone BakalCVA ThyorrL
Trylecio
(a) Accuracy
(b) Evaluations
1,0
0.5
0.2
0.02
0.005
Training Delseet Portion
Figure 5: (a) Mean test accuracy across ten runs on MNIST. The TaylorGLO loss func- tion with the highest validation score signifi- cantly outperforms the cross-entropy loss (p 2.95 x 10-15 in a one-tailed Welch's is t-test) and the BaikalCMA loss (Gonzalez& Müikkulainen 2020 (y 0.0313). (b) Required partial train- ing evaluations for GLO and TaylorGLO on MNIST. The TaylorGLO loss function was dis- covered with 4% of the evaluations that GLO required to discover BaikalCMA.
Figure 6: Accuracy with reduced portions of the MNIST dataset. Progressively smaller portions of the dataset were used to train the models (averag- ing ten runs). The TaylorGLO loss function provides significantly better performance than the cross-entropy loss all training dataset sizes, and particularly on the smaller datasets. Thus, its abil- ity discourage overfitting particularly useful in applications where only limited data is available.
6.3 PERFORMANCE ON REDUCED DATASETS
The performance improvements that TaylorGLO provides especially pronounced with reduced datasets. For example, Figure comppres accuracies of models trained for 20,000 steps on different portions of the MNIST dataset (similar results were obtained with other datasets and architectures). Overall, TaylorGLO significantly outperforms the cross-entropy loss. When evolving a TaylorGLO loss function and training against 10% of the training dataset, with 225 epoch evaluations, TaylorGLO reached an average accuracy across ten models of 0.7595 (stddev 0.0062). In contrast, only four out of ten cross-entropy loss models trained successfully, with those reaching a lower average accuracy of 0.6521. Thus, customized loss functions can be especially useful in applications where only limited data is available to train the models, presumably because they are less likely to overfit to the small number of examples.
7 DISCUSSION AND FUTURE WORK
TaylorGLO was applied to the benchmark tasks using various standard architectures with standard hyperparameters. These setups have been heavily engineered and manually tuned by the research community, yet TaylorGLO was able to improve them. Interestingly, the improvements were more substantial with wide architectures and smaller with narrow and deep architectures such as the Preactivation ResNet. While may be possible further improve upon this result, is also possible that loss function optimization is more effective with architectures where the gradient information travels through fewer connections, or is otherwise better preserved throughout the network. important direction of future work is therefore to evolve both loss functions and architectures together, taking advantage possible synergies between them.
As illustrated in Figure B., the most significant effect of evolved loss functions is to discourage extreme output values, thereby avoiding overfitting. is interesting that this mechanism apparently different from other regularization techniques such as dropout (as shown by Gonzalez & Miikkulaing 2020) and data augmentation with Cutout (as seen in Table M Dropout and Cutout improve performance over the bascline, and loss function optimization improves it further. This result suggests that regularization a multifaceted process, and further work is necessary to understand how to best take advantage of it.
Another important direction is to incorporate state information into TaylorGLO loss functions, such the percentage of training steps completed. TaylorGLO may then find loss functions that best suited for different points in training, where, for example, different kinds regularization work best (Golatkar et al. 2019) Unintuitive changes to the training process, such cycling learning rates
8

---

Under review as a conference paper at ICLR 2021
(Smith 2017). have been found to improve performance; evolution could be used to find other such opportumities automatically. Batch statistics could help evolve loss functions that are more well-tuned to each batch; intermediate network activations could expose information that may help tune the function for deeper networks like ResNet. Deeper information about the characteristics of a model's weights and gradients, such as that from spectral decomposition of the Hessian matrix (Sagun et al. 2017), could assist the evolution of loss functions that adapt to the current fitness landscape. The technique could also be adapted to models with auxiliary classifiers (Szegedy et al. 2013) as means to touch deeper parts of the network.
8 CONCLUSION
This paper proposes TaylorGLO as a promising new technique for loss-function metalearning. TaylorGLO leverages novel parameterization for loss functions, allowing the use of continuous optimization rather than genetic programming for the search, thus making it more efficient and more reliable. TaylorGI. loss functions serve to regularize the learning task, outperforming the standard cross-entropy loss significantly on MNIST, CIFAR-10, 10, CIFAR-100, and SVHN benchmark tasks with variety of network architectures. They also outperform previously loss functions discovered in prior work, while requiring many fewer candidates to be evaluated during search. Thus, TaylorGLO results in higher testing accuracies, better data utilization, and more robust models, and is a promising new avenue for metalearning.
REFERENCES
United States Environmental Protection Agency eGRID2018. http::/ww..e....o//eeeeee.. 2020.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irv- ing, M. Isard, M. Kudlur, Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: A sys- tem for large-scale machine learning. In 12th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI 16), pp. 265-283, Savannah, GA, 2016. USENIX Association. ISBN 978-1-931197-33--1 URL https: wwww. useni x. cor..r/////o.oeeeee////aiiii Lechnical www.allsesssinno wttpssssisssopeeeetatttiiiioo abadi
W. Banzhaf, P. Nordin, R. E. Keller, and F. D. Francone. Genetic programming: An introduction, volume 1. Morgan Kaufmann San Francisco, 1998.
G. Bingham, W. Macke, and R. Müikkulainen. Evolutionary optimization of deep learning activation functions. In Proceedings of the Genetic and Evolutionary Computation Conference, 2020.
Chisholm. Rational approximants defined from double power series. Mathematics of Computation, 27(124):841-855 1973.
T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 700.70080444 045522 2017.
H. Dong, S. Yu, C. Wu, and Y. Guo. Semantic image synthesis via adversarial learning. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 5706-5714, 2017.
T. Elsken, H. Metzen, and F. Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55): 2019.
J. B. Fourier. La théorie analytique de la chaleur. Mémoires de l'Académie Royale des Sciences de I'Institut de France, 8:581-622, 1829.
R. Gao and Grauman. 2.5D visual sound. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 324-333, 2019.
A. S. Golatkar, A. Achille, and S. Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances in Neural Information Processing Systems 32, pp. 10677-10687, 2019.

---

Under review as a conference paper at ICLR 2021
S. Gonzalez and R. Mikkkuainenn Improved training speed, aceuracy, and data utilization through loss function optimization. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC), 2020.
S. Gonzalez, J. Landgraf, and R. Miikkulainen. Faster training by selecting samples using embeddings. In 2019 International Joint Conference on Neural Networks (IJCNN, 2019.
Graves-Morris. The numerical calculation of Padé approximants. In Padé approximation and its applications, pp. 231-245. Springer, 1979.
Graves-Morris and D. Roberts. Calculation of Canterbury approximants. Computer Physics Communications, 10(4):234-244 1975.
J. J. Grefenstette and J. M. Fitzpatrick. Genetic search with approximate function evaluations. In Proceedings of an International Conference on Genetic Algorithms and Their Applications, pp. 112-120, 1985.
D. Han, J. Kim, and J. Kim. Deep pyramidal residual networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). pp. 5927-5935, 2017.
N. Hansen and S. Kern. Evaluating the CMA evolution strategy on multimodal test functions. In International Conference on Parallel Problem Solving from Nature, pp- 282-291. Springer, 2004.
N. Hansen and A. Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEFE international conference on evolutionary computation, pp. 312-317. IEEE, 1996.
N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, (12::159-195, 2001.
K. He, X. Zhang, Ren, and Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645 Springer, 2016a.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 201
G. E. Hinton, N. Srivastava, A. Krizhevsky, Sutskever, and R. R. Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.0580 2012.
Huber. Robust estimation of location parameter. The Annals of Mathematical Statistics, pp. 73-101, 1964.
Y. Jin. Surrogate-assisted evolutionary computation: Recent advances and future challenges. Swarm and Evolutionary Computation, 1:61-70, 06 2011. doi: 10.1016/j.swevo.22 105.001..
N. Keskar, D. Mudigere, Nocedal, M. Smelyanskiy, and P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In Proceedings of the Fifth International Conference on Learning Representations (ICLR), 2017.
D. Kingma and M. Welling. Auto-encoding encoding variational Bayes. In Proceedings of the Second International Conference Learning Reprexentations (ICLR), 12 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep con- volutional neural networks. In Pereira, C. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097- 1105. Curran Associates, Inc., 2012. URL htttp::/papprs.niii mips www.niipsspcppppppeppppppppppppe
4824
pdf
A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres. Quantifying the carbon emissions machine learning. arXiv preprint 1910::1100 2019.
Y. LeCun, C. Cortes, and C. Burges. The MNIST dataset of handwritten digits, 1998.
10

---

Under review as a conference paper at ICLR 2021
C. Lemke, M. Budka, and B. Gabrys. Metalearning: survey of trends and technologies. Artificial Intelligence Review, 44(1):117-130, 2015.
H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 6389-6399, Curran Associates, Inc., 2018. URL
nips. cttpp///pprppp.c.ppppp.e..
7875-visua
I. v. d. Maaten and G. Hinton. Visualizing data using t-SNE, Journal of Machine Learning Research, (Nov))2557--660) 2008.
R. Miikkulainen, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, H. Shahrzad, A. Navruzyan, N. Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Compuring, pp. 293-312. Elsevier, 2019.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. Neural Information Processing Systems, Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
E. Real, C. Liang. So, and Q. V. Le. Automl-zero: Evolving machine learning algorithms from scratch. 1xXiv:2003.033844 2020.
D. E. Rumelhart, G. Hinton, and R. J. Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
G. D. Ruxton. The unequal variance t-test is underused alternative to Student's t-test and the Mann-Whitney U test. Behavioral Ecology, 17(4))688-69) 2006.
L. Sagun, U. Evei, V. U. Guney, Y. Dauphin, and L. Bottou. Empirical analysis of the Hessian of over-parametrized neural networks. arXiv preprint 0rXiv:1706.044535 2017.
Schmidhuber. Evolutionary principles in reff-referlntial learning, or on learning how to learn: the melu-mela-.. hook. PhD thesis, Technische Universität München, 1987.
L. N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464-472. IEEE, 2017.
J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, abs/1442.6880, 2015.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
B. Taylor. Methodus incrementorum directa & inversa. Auctore Brook Taylor, LL. D. & Regiae Societatis Secretario. typis Pearsonianis: prostant apud Gul. Innys ad Insignia Principis, 1715.
A. N. Tikhonov. Solution of incorrectly formulated problems and the regularization method. In Proceedings of the USSR Academy of Sciences, volume 4, pp. 1035-1038, 1963.
B. L. Welch. The generalization of Student's problem when several different population variances are involved. Biometrika, 34(1/2):28-35, 1947.
H. Wilbraham, On a certain periodic function. The Cambridge and Dublin Mathematical Journal, 3: 198-201, 1848.
S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 6023-6032, 2019.
Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXivv1605.07144 2016.
Y. Zhou, C. Liu, and Y. Pan. Modelling sentence pairs with tree-structured attentive encoder. In Proceedings of the 26th International Conference on Computational Linguistics (COLING), Technical Papers, pp. 2912-2922, 2016.
11

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL SETUP
The following subsections cover specific experimental setup details. The three evaluated datasets detailed in how they were used, along with implementation details.
A.1
MNIST
The first domain was MNIST Handwritten Digits, a widely used dataset where the goal is to classify 28 28 pixel images as one of ten digits. MNIST has 55,000 training samples, 5,000 validation samples, and 10,000 testing samples. The dataset is well understood and relatively quick to train, and forms good foundation for understanding how TaylorGLO evolves loss functions.
The basic CNN architecture evaluated in the GLO study (Gonzalez & Mikkkulainen (020)) can also be used to provide a direct point of comparison with prior work on MNIST. Importantly, this architecture includes dropout layer (Hinton et al. (2022) for explicit regularization. As in GL.O, training is based on stochastic gradient descent (SGD) with a batch size of 100, a learning rate of 0.01, and, unless otherwise specified, occurred over 20,000 steps.
A.2 CIFAR-10 AND CIFAR-100
To validate TaylorGI.O in more challenging context, the CIFAR-10 (Krizhesky & Hinton 2009) dataset was used. consists of small 32 x 32 pixel color photographs of objects in ten cl classes. CIFAR-I0 traditionally consists of 50,000 training samples, and 10,000 testing samples; however 5,000 samples from the training dataset were used for validation of candidates, resulting in 45,000 training samples.
Models were trained with their respective hyperparameters from the literature. Inputs were normalized by subtracting their mean pixel value and dividing by their pixel standard deviation. Standard data augmentation techniques consisting of random horizontal flips and croppings with two pixel padding were applied during training.
CIFAR-100 is similar, though significantly more challenging, dataset where a different set of 60,000 images is divided into 100 classes, instead 10. The same splits for training, validation, and testing were used for CIFAR- 100 as for CIFAR- 10, and evaluate TaylorGLO further.
A.3 SVHN
The Street View House Numbers (SVHN; Netzer et al. 2011 dataset is another image classification domain that was used to evaluate TaylorGLO, consisting of 32 x 32 pixel images of numerical digits from Google Street View. SVHN consists of 73,257 training samples, 26,032 testing samples, and 131,131 supplementary, easier training samples. To reduce computation costs, supplementary examples were not used during training; this fact explains why presented baselines may be lower than other SVHN baselines in the literature. Since a validation set is not in the standard splits, 26,032 samples from the training dataset were used for validation of candidates, resulting in 47,225 training samples.
As with CIFAR-10, models were trained with their respective hyperparameters from the literature and with the same data augmentation pipeline.
A.4 CANDIDATE EVALUATION DETAILS
During candidate evaluation, models were trained for 10% of full training run MNIST, equal to 2,000 steps (i.e., four epochs). An in-depth analysis on the technique's sensitivity to training steps during candidate evaluation is provided in Appendix overall, few training steps. However, on more complex models with abrupt learning rate decay schedules, greater numbers of steps provide better fitness estimates.
-vverall,, the technique is robust even with
12

---

Under review as a conference paper at ICLR 2021
PC2
64
PC1
24
20
Generation
Figure 7: A visualization of all TaylorGLO loss function candidates using t-SNE (Maaten & Hinton 2008) on MNIST. Colors map to each candidate's 's generation. Loss function populations show evolutionary path and focus over time towards functions that perform well, consistent with the convergence and settling in Figure
A.5 STATISTICAL TESTING
Statistical significance tests define null hypothesis and reject if p-value is below a predefined significance level, typically 0.05. A p-value is the probability of obtaining extreme results at the same level or greater than the results observed given that the null hypothesis is true.
When comparing results in this paper, one-tailed null hypothesis is typically used:
Ho - (p1 < p2),
(6)
where Pl and P2 are mean values from two separate sets of training sessions. The rejection of this null hypothesis implies that #i is statistically significantly larger than μ41 Thus, the change between the two sets of training sessions robust to training stochasticity, such as from varying weight initializations.
Throughout this paper, Welch's 's t- Test Welch 1947) is used to determine statistical significance when comparing sets of results which may not have equal variances. It is also better fit than Student's t-Test due its higher robustness and statistical power Ruxton (2006).
A.6 IMPLAMENTATION DETAILS
Due to the number of partial training sessions that are needed evaluate TaylorGLO loss function candidates, training was distributed across the network to a cluster, composed of dedicated machines with NVIDIA GeForce GTX 1080Ti GPUs. Training itself was implemented with TensorFlow (Abadi et al. 2016) in Python. The primary components of TaylorGLO (i.e., the genetic algorithm and CMA- ES) were implemented in the Swift programming language which allows for easy parallelization. These components run centrally on one machine and asynchronously dispatch work to the cluster.
Training for each candidate was aborted and retried up to two additional times if validation accuracy was below 0.15 at the tenth epoch. This method helped reduce computation costs.
B ILLUSTRATING THE EVOLUTIONARY PROCESS
The TaylorGLO search process can be illustrated with t-SNE dimensionality reduction (Maaten & Hinton 2008) on every candidate loss function within run (Figure The initial points (i.e. loss functions) are initially widespread the left side, but quickly migrate and spread to the right as CMA-ES explores the parameter space, and eventually concentrate in a smaller region dark red points. This pattern is consistent with the convergence and settling in Figure 3
C TOP MNIST LOSS FUNCTION
The best loss function obtained from rumning TaylorGLO MNIST was found in generation 74.
This function, with parameters 0 = (11.9039, -4.0240, 6.9796, 8.5834, -166677, 11.6064. 12.6684. 3..6674) (rounded four decimal-places), achieved 2k-step validation accuracy of 0.9950 on its
13

---

Under review as a conference paper at ICLR 2021
single evaluation, higher than 0.9903 for the cross entropy loss. This loss function was modest improvement over the previous best loss function from generation 16, which had a validation accuracy of 0.9958.
D MNIST EVALUATION LENGTH SENSITIVITY
200-step TaylorGLO is surprisingly resilient when evaluations during evolution shortened to 200 steps (i.e., 0.4 epochs) of training. With so little training, returned accuracies are noisy and dependent each individual network's particular random initialization. On 60-generation run with 200-step evaluations, the best evolved loss function had a mean testing accuracy of 0.9946 across ten samples, with standard deviation of 0,0016. While slightly lower, and significantly more variable. than the accuracy for the best loss function that was found on the main 2,000-step run, the accuracy still significantly higher than that of the cross-entropy baseline, with a p-value of 6.3 10 This loss function was discovered in generation 31, requiring 1,388.8 8 2, 2,000-step- 2.000-see-eqquiaaeen evaluations. That is, evolution with 200-step partial evaluations is over three-times less sample efficient than evolution with 2,000-step partial evaluations.
equivalent partial
20,000-step On the other extreme, where evaluations consist of the same number of steps full training session, one would expect better loss functions to be discovered, and more reliably, because the fitness estimates are less noisy. Surprisingly. that is not the case: The best loss function had mean testing accuracy of 0.9945 across ten samples, with a standard deviation of 0.0015. While also slightly lower, and also significantly more variable, than the accuracy for the best loss function that was found the main 2,000-step run, the accuracy is significantly higher than the cross-entropy baseline, with a p-value 5.1 x 66 This loss function was discovered in generation 45, requiring 12,600 2, -000-step-equivalent partial evaluations. That is, evolution with 20,000-step full evaluations over 28-times less sample efficient than evolution with 2,000-step partial evaluations.
These results thus suggest that there is an optimal way to evaluate candidates during evolution, resulting in lower computational cost and better loss functions. Notably, the best evolved loss functions from all three runs (i.e., 200-, 2,000-, and 20,000-step) have similar shapes, reinforcing the idea that partial-evaluations can provide useful performance estimates.
E LEARNING RATE SENSITIVITY
Loss functions can embody different learning rates implicitly. This section shows that TaylorGI.O loss functions' benefits come from more than just metalearning such learning rates. Increases in performance that result from altering the base learning rate with cross-entropy loss are significantly smaller than those that TaylorGLO provides.
More specifically, Figure g quantifies the effect of varying learning rates on the final testing accuracy of AlICNN-C models trained on CIFAR- ALICNN-C was chosen for this analysis since exhibits the largest variations in performance. making this effect more clear. While learning rates larger than 0.01 (the standard learning rate for ALICNN-C) reach slightly higher accuracies, this effect comes at the cost of less stable training. The majority of models trained with these higher learning rates failed train. Thus, the standard choice of learning rate for ALICNN-C is appropriate for the cross-entropy loss, and TaylorGLO loss functions are able to improve upon it.
F TAYLOR APPROXIMATIONS OF THE CROSS-ENTROPE LOSS
While TaylorGLO's performance originates primarily from discovering better loss functions, it is informative to analyze what role the accuracy of the Taylor approximation plays in it. One way to characterize this effect is to analyze the performance of various Taylor approximations of the cross-entropy loss.
Table provides results from such a study. Bivariate approximations the cross-entropy loss, centered at a = (0.5, 0.5), with different orders & were used to train AIICNN-C models on CIFAR- 10. Third-order approximations and above trainable. Approximations' performance is within few
14

---

Under review as a conference paper at ICLR 2021
CIFAR-10 Testing Accuracy
0.90
0.85
0.80
0.75
0.001
0.005
0.01
0.05
Initial Learning Rate
Figure 8: Effect of varying learning rates in ALICNN-C when trained with the cross-entropy loss on CIFAR-10, 10. For each learning rate, ten models
trained, with up to ten retries if training failed. The majority of training attempts failed for learning rates larger than 0.01. The 0.01 learning rate used in the experiments this paper results in best stable performance. Overall, the small performance differences that can result from adjusting the learning rate, regardless of stability, are much smaller than those that result from training with TaylorGLO. Thus, TaylorGLO provides mechanism for improvement beyond implicit adjustments of the learning rate.
Table 2: Performance Taylor approximations of the cross-entropy loss function on AICNN--C with CIFAR-I0. Approximations of different orders, with (0.5, 0.5), presented. Presented accuracies are the mean from ten runs. The baseline is the standard cross-entropy loss. Higher-order approximations are better, suggesting potential (although computationally expensive) opportunity for improvement in the future.
Loss Function
Mean Accuracy (stddev)
h 2
0.1034 (0.0101)
=
0,8451 (0.0043)
k 4
0,8592 (0.0032)
= Cross-Entrope
0.8649 (0.0042)
0.8965 (0.0021)
percentage points of the cross-entropy loss, with higher-order approximations yielding progressively better accuracies, expected.
The results thus show that third-order TaylorGLO loss functions cannot represent the cross-entropy baseline loss accurately. One possibility for improving TaylorGLO is thus to utilize higher order approximations. However, it remarkable that TaylorGLO can still find loss functions that outperform the cross-entropy loss. Also, the increase in the number of parameters- increase in computational requirements-nnn representation. This effect was seen in preliminary experiments, and the third-order approximations (used in this paper) deemed to strike good balance.
the corresponding in practice outweigh the benefits from finer-grained
G TAYLORGLO EXPERIMENT DURATIONS AND ENVIRONMENTAL IMPACT
The infrastructure that ran the experiments in this paper located in California, which is estimated to have had an estimated carbon dioxide equivalent total output emission rate of 226.21 kgCO,eq/khh in 2018 (epa) 2020). This quantity can be used to calculate the climate impact of inaputeeintensive experiments.
Table provides estimates of durations and total emissions for various TaylorGLO experiments. Emissions were calculated using the Machine Learning Impact calculator (Lacoste et al. (2019). assuming that no candidates failed evaluation (which would result in slightly lower estimates). Presented values can thus be thought of being upper bound.
15

---

Under review as a conference paper at ICLR 2021
Table 3: Estimated TaylorGI. experiment durations and total emissions. The estimates assume populations of 20 concurrent candidates and 50 generation runs. Emission values are upper bounds reported in equivalent kilograms of carbon dioxide, thus accounting for other gases of interest. Overall, experiments are short enough that they can each be run over few days.
TaylorGLO Experiment
Duration (hours)
Total Emissions (kgCOzeq)
AlexNet on CIFAR-10
3.60
ResNet-20 on CIFAR-10
10.24
11,58
Pre ResNet-20 on CIFAR-10
9.26
10.48
ALICNN-C CIFAR-10
17.06
19.30
PyramidNet 10a48 on CIFAR-10
73.86
83.53
Wide ResNet 28-5 on CIFAR-10
42,90
48.52
Wide ResNet 16-8 on CIFAR-10
36.08
40.80
Wide ResNet 28 10 CIFAR-10 10
105.30
119.10
Overall, experiment durations are short enough that TaylorGLOO can be practically applied different tasks to find customized loss functions.
16


Reviews:
[{"review_id": "cLgkTS0IIf-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the TaylorGLO method for learning loss functions for classification.\n\n## Pros\n1. The paper is well written and quite easy to follow.\n2. It proposes the novel idea of parameterising learned loss functions as Taylor polynomials, which overcomes the downside of previous work that relies on a slow two-stage optimisation process that first infers the structure of the loss function.\n3. Networks trained with the learned loss functions typically outperform those trained with cross entropy.\n4. Analysis is undertaken to determine why the learned loss functions work better than cross entropy (they penalise overly confident outputs), and also characterise when the loss functions discovered with TaylorGLO are most effective (better sample efficiency).\n\n## Cons\n1. The main performance comparison is with cross entropy, but there exist other methods for learning loss functions (e.g., GLO and [1]). The performance comparison with these methods should be more comprehensive: currently there is only a single comparison with GLO that uses a simple network trained on MNIST, which is not enough to make general conclusions about the relative performance of the two methods.\n2. The potential for impact is limited. The main improvements in performance over cross entropy are seen when using older network architectures, but when using state of the art networks the improvements are much smaller. If it was shown that the learned loss functions could be transferred to new tasks, hence resulting in a modest performance increase without any additional computational overhead, then I would be inclined to revisit this point.\n3. It is unclear what hypothesis is being tested when applying t-tests: given that the same data is used for each performance measurement it seems the hypothesis is that performance gains are robust to the choice of random seed. Hypothesis tests are typically used in machine learning to determine whether the experimental results will generalise to new tasks [2]. As they are currently presented, the results are likely to mislead some readers into inferring a false sense of generality of the results.\n\n## Questions\n1. Why use CMA-ES over gradient-based bi-level optimisation methods commonly used in meta-learning? Gradient-based optimisation is typically faster than evolutionary methods, and it seems like it would be possible to apply it in this situation.\n\n[1] Sarah Bechtle et al. Meta-Learning via Learned Loss. arXiv:1906.05374, 2019.\n[2] Janez Demšar. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 7(1):1−30, 2006.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XtAUky9cDNe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method, called TaylorGLO, to learn the loss functions, for training deep neural network, by meta-learning. Specifically, the authors proposed to parameterize the loss function with multivariate Taylor polynomial, and then learn the parameters in the polynomial using evolutionary algorithm within the meta-learning framework. The experiments showed improved performance of the TaylorGLO over cross-entropy baseline on several datasets and with different network architectures.\n\nAlthough the performance seems promising, there are several issues should be addressed:\n\n1. My major concern is about the parameterization of the loss function. As is well known, a qualified loss function in machine learning generally contains only one valley concerning its geometric shape. However, with Taylor polynomial parameterization, a function might have multiple valleys, or even have no minimum.\n\n2. The authors set the order of Taylor polynomial to 3, and simply claimed this is a better trade-off compared with other orders. However, this claim should be supported by empirical evaluations, i.e., use other orders to see the differences regarding the performance and evolution time.\n\n3. Since the expensive evolutionary process is required to learn the loss function before really train the model, the computational cost should be shown and discussed in detail, especially that the performance improvement is not that much in some cases as shown in Table 1.\n\n4. The comparison with GLO was only performed on MNIST dataset, where the accuracies between the two methods are very close.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NRs3f0TXoNx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Warning: I'm not an expert in loss function metalearning, so I've attempted to provide my perspective as someone interested in the topic but unfamiliar with prior work.\n\n# Summary\n\nThis paper tackles the problem of loss function metalearning by proposing a novel parameterization of loss functions based on multivariate polynomial expressions.\n\nThe intuitive motivation is that any continuous and full differentiable loss function can be written as a $k$-th order multivariate Taylor expansion (i.e. a multivariate polynomial expression). So for high enough $k$, the space of such polynomials will be a reasonable approximation of the space of all loss functions we might care about.\n\nThe claimed advantages of the method are smoothness, lack of poles, compositionality, meaningful metric (closeness in parameter space ==> similar loss function), and tunable complexity (via $k$) of the search space.\n\nThe paper then proposes a loss function optimizer named TaylorGLO, which proposes to optimize loss functions in parameter space (i.e. coefficient space) using CMA-ES, which requires a continuous optimization space.\n\n# Strengths\n\nThis is a neat idea, although I think calling it TaylorGLO is a bit of a misnomer as we're not starting with arbitrary functions that we're then approximating via Taylor expansion. Rather, the paper optimizes directly in polynomial space. Maybe \"PolyGLO\" is a more accurate name? The point can be made within the paper text that for sufficiently high $k$, the space of polynomials becomes an arbitrarily good approximation of any cont/diffble loss function thanks to Taylor's thm.\n\nRegardless, I really like this approach! It definitely seems like a more promising approach than the referenced prior approach, which first optimizes the structure of the loss fn in discrete (tree) space and then optimizes the coefficients via continuous methods. The fact that the experiments found a function that outperforms \"BaikalCMA\" is welcome confirmation that we don't lose much by staying in continuous space.\n\n# Weaknesses / unanswered questions\n\nI'm worried that to some extent, the improvements from the new loss functions are thanks to an implicit learning rate tuning effect. The derivative of the loss function w.r.t. the model output directly scales the parameter updates for backpropogation-based SGD. So, maybe what's happening is that the learning rate is being tuned implicitly by TaylorGLO choosing a steeper or more compressed loss function. Maybe there's even a learning rate *scheduling* effect that's coming from TaylorGLO choosing loss functions with higher curvatures.\n\nI think it's quite important to disentangle the effects of the loss function optimization from an implicit learning rate tuning, so I would ideally like to see experiments comparing TaylorGLO's best-accuracy loss function with plain old CE sweeped across learning rates and LR scheduling approaches. The difference in accuracy between TaylorGLO and a comprehensive CE sweep will be a more accurate indication of the effects of the loss optimization itself.\n\nI'm also curious to know the extent of prior art in continuous loss function optimization. This approach seems like the first thing one would do if trying to design a loss function metalearning approach in continuous space, so I'd like to hear about what (if anything) has been done before.\n\n# Overall Rating\n\nI rate this paper a 6. I would be happy to see it in ICLR as is, but I think for its results to be exciting to the ICLR audience at large, the paper needs to convincingly isolate the effects of the function optimization itself.\n\n# Comments\n\n- p.1 \"arbitrarily long, fixed-length vectors in a Hilbert space\" seems self-contradicting. Is this meant to say \"arbitrary norm\"?\n- Formatting of some of the \\citep's can be improved, e.g. CMA-ES.\n- I'd love to see a comparison of classification accuracy between standard cross-entropy and $k$-th order approximations of CE to get a sense of how much the fidelity loss actually matters.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-i14pOGorf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors studied the optimization problem of loss functions, where the loss function is not referred to as the error criterion but the empirical error term in empirical risk minimization. The main approach proposed in this paper is the polynomial parametrization of the loss function via multivariate Taylor expansion, namely, equation (5), which should also be regarded as the main contribution of this paper.   \n\nPros: The paper is well written and is easy to follow. The main approach and its rationality are well articulated. It is good to see that different approximates of loss functions are compared. The effectiveness of the proposed approach is also well illustrated numerically.\n\nCons: While I believe that the proposed approach works, it is not clear to me what is the relation between the newly proposed approach and the original approach where loss functions are used directly instead of their approximates. In my opinion, the new approach works not because of the approximate of the loss function but because of the new error criterion which is essentially a polynomial. This gives my main concern. I'm expecting more comments in this regard.       ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Pros: Reviewers generally agreed the paper was well written and is easy to follow. The goal of learning loss functions also seems quite promising.

Cons: There were concerns about whether credit for experimental performance was attributable to the core algorithm+functional form presented in the paper. There was also some skepticism about the specific form of the learned loss. Of greatest concern, no reviewer argued for acceptance during discussion, and one reviewer lowered their score during discussion.
Decision: Reject