Effective Regularization Through Loss-Function Metalearning
Authors: Santiago Gonzalez
Risto Miikkulainen
Venue:  (2021)
Keywords: regularization
loss
loss function
metalearning
meta-learning
optimization
theory
robustness
adversarial attacks

Abstract: Loss-function metalearning can be used to discover novel, customized loss functions for deep neural networks, resulting in improved performance, faster training, and improved data utilization.  A likely explanation is that such functions discourage overfitting, leading to effective regularization. This paper theoretically demonstrates that this is indeed the case: decomposition of learning rules makes it possible to characterize the training dynamics and show that loss functions evolved through TaylorGLO regularize both in the beginning and end of learning, and maintain an invariant in between. The invariant can be utilized to make the metalearning process more efficient in practice, and the regularization can train networks that are robust against adversarial attacks. Loss-function optimization can thus be seen as a well-founded new aspect of metalearning in neural networks.

Under review as a conference paper at ICLR 2021
EFFECTIVE REGULARIZATION THROUGH LOsS-FUNCTIN METALEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Loss-function metalearning can be used to discover novel, customized loss func- tions for deep neural networks, resulting in improved performance, faster training, and improved data utilization. A likely explanation is that such functions dis- courage overfitting, leading effective regularization. This paper demonstrates theoretically that this is indeed the case for the TaylorGLO method: Decomposition of learning rules makes it possible to characterize the training dynamies and show that the loss functions evolved by TaylorGLO balance the pull to zero error, and a push away from it to avoid overfitting. This observation leads to an invariant that can be utilized to make the metalearning process more efficient in practice, and result in networks that are robust against adversarial attacks. Loss-function optimization can thus be seen as well-founded new aspect of metalearning in neural networks.
INTRODUCTION
Regularization a key concept in deep learning: it guides learning towards configurations that are likely to perform robustly on unseen data. Different regularization approaches originate from intuitive understanding the learning process and have been shown to be effective empirically, However, the understanding of the underlying mechanisms, the different types of regularization, and their interactions, is limited.
Recently, loss function optimization has emerged new area of metalearning, and shown great potential in training better models. Experiments suggest that metalearned loss functions serve as regularizers in a surprising but transparent way: they prevent the network from learning too confident predictions (e.g. Baikal loss; Gonzalez & Mükkulainen 2020a). While may be too early to develop comprehensive theory of regularization, given the relatively nascent state of this area, it may be possible to make progress understanding regularization this specific type. That is the goal of this paper.
Since metalcarned loss functions are customized to a given architecture-tase task pair, there needs to be shared framework under which loss functions be analyzed and compared. The TaylorGIOO (Gonzalez Mikkulainen 2020b) technique for loss function metalearning lends itself well to such analysis: It represents loss functions multivariate Taylor polynomials, and leverages evolution to optimize a fixed number of parameters in this representation. In this framework, the SGD learning rule is decomposed coefficient expressions that can be defined for wide range of loss functions. These expressions provide an intuitive understanding of the training dynamics in specific contexts.
Using this framework, mean squared error (MSE), cross-entropy. Baikal, and TaylorGLO loss functions are analyzed at the null epoch, when network weights are similarly distributed (Appendix[C) and in zero training error regime, where the training samples' labels have been perfectly memorized. For any intermediate point in the training process, the strength of the zero training error regime as an attractor is analyzed and a constraint on this property is derived on TaylorGLO parameters by characterizing how the output distribution's 's entropy changes. In concrete TaylorGLO loss function that has been metalearned, these attraction dynamics are calculated for individual samples at every epoch in real training run, and contrasted with those for the cross-entropy loss. This comparison provides clarity on how TaylorGLO avoids becoming overly confident its predictions. Further, the analysis shows (in Appendix (.2) how label smoothing (Szegedy et al. 2016), traditional type

---

Under review as a conference paper at ICLR 2021
of regularization, can be implicitly encoded by TaylorGLO loss functions: Any representable loss function has label-smoothed variants that are also representable by the parameterization.
From these analyses, practical opportunities arise. First, at the null epoch, where the desired behavior can be characterized clearly, an invariant can be derived on a TaylorGLO loss functionn' parameters that must hold true for networks to be trainable. This constraint is then applied within the TaylorGLO algorithm to guide the search process towards good loss functions more efficiently. Second, loss- (unction-based regularization results in robustness that should e.g. make them more resilient to adversarial attacks. This property is demonstrated experimentally by incorporating adversarial robustness as an objective within the TaylorGLO search process. Thus, foss-funtiion metalearning can be seen as
and practical approach to effective regularization in deep learning.
BACKGROUND
Regularization traditionally refers to methods for encouraging smoother mappings by adding regularizing term to the objective function, i.e., to the loss function in neural networks. It can be defined more broadly, however, e.g. as "any modification we make to learning algorithm that is intended to reduce its generalization error but not its training error" (Goodfellow et al. (015). To that end, many regularization techniques have been developed that aim to improve the training process in neural networks. These techniques be architectural in nature, such Dropout (Srivastaval 2014) and Batch Normalization (loffe & Szegedy, 2013). or they can alter some aspect of the
et al. training process, such as label smoothing (Szegedy al. (016) the minimization of a weight norm (Hanson & Pratt 1989). These techniques are briefly reviewed in this section, providing context for loss-function metalearning.
2.1 IMPLICIT BIASES IN OPTIMIZERS
may seem surprising that overparameterized neural networks are able to generalize at all, given that they have the capacity to memorize training set perfectly, and in fact sometimes do (i.e., zero training error is reached). Different optimizers have different implicit biases that determine which solutions are ultimately found. These biases are helpful in providing implicit regularization the optimization process (Neyshabur et al., 2015). Such implicit regularization is the result of a network norm-a measure of complexity- that minimized as optimization progresses. This is why models continue to improve even after training set has been memorized (i.c., the training error global optima reached) (Neyshabur et al. 2017).
For example, the process of stochastic gradient descent (SGD) itself has been found to provide regular- ization implicitly when learning on data with noisy labels (Blanc al., 2020). overparameterized networks, adaptive optimizers find very different solutions than basic SGD. These solutions tend to have worse generalization properties, even though they tend to have lower training errors (Wilson
et al. 2017).
2.2 REGULARIZATION APPROACHES
While optimizers may minimize network norm implicitly, regularization approaches supplement this process and make it explicit. For example, a common way to restrict the parameter norm explicitly is through weight decay. This approach discourages network complexity by placing a cost on weights (Hanson & Pratt 1989.
Generalization and regularization are often characterized the end of training, i.e. behavior that results from the optimization process. Various findings have influenced work in regularization. For example, flat landscapes have better generalization properties (Keskar et al. 2017 Li et al. 2018) Chaudhari et al., 2019). In overparameterized cases, the solutions at the center of these landscapes may have zero training error (i.e., perfect memorization), and under certain conditions, zero training eтror empirically leads to lower generalization error (Belkin et al.] 2019 Nakkiran et al.) 2019). However, when training loss of zero is reached, generalization suffers (shida et al. 2020). This behavior can be thought of overtraining, and techniques have been developed to reduce at the end of the training process, such as early stopping (Morgan & Bourlard 1990) and flooding (Shia al. 2020
2

---

Under review as a conference paper at ICLR 2021
Both flooding and early stopping assume that overfitting happens at the end of training, which is not always true (Golakkar et al. 2019). In fact, the order in which teayy-tt-geerrlizz and hard-to- generalize concepts learned important for the network's 's ultimate generalization. For instance, larger learning rates early in the training process often lead to better generalization in the final model et 2019). Similarly, low-error solutions found by SGD in relatively quick manner such as through high learning rates-often have good generalization properties (Yao et al. 2007).
Other techniques tackle overfitting by making it more difficult. Dropout (Srivastava et al. 2014
makes some connections disappear. Cutout (DeVries & Taylor 2017). Mixup (Zhang et al. 2018). and their composition, CutMix (Yun al. 2019). augment training data with broader variation of examples.
Notably, regularization is not one-dimensional continuum. Different techniques regularize in different ways that often interact. For example, flooding invalidates performance gains from early stopping (Ishida al. 2020). However, ultimately all regularization techniques alter the gradients that result from the training loss. This observation suggests loss-function optimization might be an effective way to regularize the training process.
2.3 LOSS-FUNCTION METALEARNING
The idea of metalearning functinnss has deep history with many recent developments that have shown promise in practical settings.
Prior work in reinforcement learning showed that metalearning various types of objectives is useful. For instance, in evolving policy gradients (Houthooft et al. 2018), the policy loss is not represented symbolically, but rather neural network that convolves over temporal sequence of context vectors. In reward function search (Nickum et al.) 2010). the task is framed as a genetic programming problem, leveraging PushGP (Spector al.) (2001). Various actor-critic reinforcement learning approaches have tackled learning a meta-critic neural network that can generate losses (Sung et al. 2017] al. 2020). Metalearned critic network techniques have also been applied outside of reinforcement learning to train better few-shot classifiers (Antoniou & Storkey 2019).
In unsupervised representation learning, weight update rules for semi-supervised learning have themselves been metalearned successfully (Metz et al., 2018). The update rules were constrained to fit a biological neuron model and transferred well between tasks.
Concrete loss-function metalearning for deep networks was first introduced by Gonzalez & Mi. ikkulainen (2020a) as an automatic way to find customized loss functions that aim to optimize a performance metric for a model. The technique, genetic programming approach, named GLO, discovered one particular loss function, Baikal, that improves classification accuracy, training speed, and data utilization. Baikal appeared achieve these properties through form of regularization that ensured the model would not become overly confident in its predictions. That is, instead of monotonically decreasing the loss when the output gets closer the correct value, Baikal loss increases rapidly when the output is almost correct, thus discouraging extreme accuracy. This paper shows how training dynamics are specifically impacted in this manner when training with the Baikal loss.
Overall, GLO demonstrated that while learned loss functions' generalization effects transfer across datasets and models to some extent, they are most powerful when they are customized to individual tasks and architectures. Diffeerntt loss functions can take advantage of the different characteristies of each such setting. Other techniques have advanced this new field further, for example by metalearning state-dependent loss functions for inverse dynamics models (Morse al. 2020), and using a trained network that is itself metalearned loss function (Bechtle al. 2019).
One particular technique, TaylorGLO (Gonzalez & Miikkulainen 2020b), lends itself well to an- alyzing what makes loss-function metalearning effective. TaylorGLO represents loss functions as parameterizations of multivariate Taylor polynomials. This parameterization allows it to scale to models with millions of trainable parameters, including variety of deep learning architectures in image classification tasks. TaylorGLO loss functions have a tunable complexity based on the order of the polynomial; third-order functions were identified work best in practical settings.
3

---

Under review as a conference paper at ICLR 2021
The third-order TaylorGLO loss function parameterization provides fixed representation that can be used to analyze a large family of loss functions through a unified methodology. In prior work, TaylorGLO loss functions were shown to improve generalization empirically (Gonzalez & Mikkulainen (020b). This paper complements that work, aiming to explain why that is the case by analyzing the dynamics of training theoretically.
3 LEARNING RULE DECOMPOSITION
This section develops the framework for the analysis in this paper. By decomposing the learning rules under different loss functions, comparisons can be drawn different stages of the training process. Consider the standard SGD update rule:
+ 0 Vo (C(x,, y: 0))
where yj is the learning rate, L( Vi, 0) is the loss function applied to the network h(:;; 0). æ; is an input data sample, yi is the ith sample's 's corresponding label, and is the set of trainable parameters in the model. The update for single weight 0g is
8 =, 0, 7 aas C( ye + sÿ)
0 Dn (C(x,, y. 0)) =
(2)
6-+0
where j is basis vector for the jth weight. The following text illustrates decompositions of this general learning rule in classification context for a variety of loss functions: mean squared error (MSE), the cross-entropy loss function, the general third-order TaylorGLO loss function, and the Baikal loss function. The TaylorGLO and Baikal loss functions are referenced Section 2.3 Each decomposition results in learning rule of the form
8 + 0, +
Inn Us. 0)Dj (ha(2,, 0))]
(3)
where (2, ye, 0) an expression that specific each loss function. Substituting the Mean squared error (MSE) loss into Equation
2
2 ha (x;, 0 + sÿ)
0, -
2 (ha( 0 =5) yak)
(4)
Is
and breaking up the coefficient expressions into nM ( yi, 0) results the weight update step 7Ak (2; yi, 0) 2yj 2hs (æ,, 0).
(5)
Substituting the Cross-entropy loss into Equation 2
0 8 8g
Vik hg (x,, 0 + sj) as
hu ha 0 + sj)
(6)
s-+0
and breaking up the coefficient expressions into y (x;, y: 0) results in the weight update step yik
mAk (x,; yı, 0) hg (z; 0)
(7)
Substituting the Baikal loss into Equation 2
0 0, + 1 Σ
ha, 0 + sj) (
Vik
2
hy (a:; + sg)
(8)
(2, 0 + sj)
Ôs
50+0
and breaking up the coefficient expressions into (2;; yi, 0) results in the weight update step
1
Yik
(x y: 0) (a,, 0)
h 0)2
Substituting the Third-order TaylorGLO loss with parameters into Equation
i
0, 8, +
ha (x,, 0 + + (hx 0 + sj) >1) hy h, 0+
Øs
8 h4 (ar, 0 + sÿ) + > >o) 0s
+ X43 (14 cr, +j) x1)² as
hu (2,, 0 + +g) (10)
+ +Moole - 2h) 0 + sÿ) A1) + XT( -203 0s
(0,, 0 + sj)
8-20
4

---

Under review as a conference paper at ICLR 2021
and breaking up the coefficient expressions into (2, yt, 0) results in the weight update step 7h (2t, yss 0) (2; 0) 23 2Aghu (2,, 0) 2A 20 0)
~2i1d6BiR 2A Ao + X2 + AsyAik Asd + ATy 207
(11)
+ + ( 0)2 AA (z,, 0) + 3A4
To simplify analysis, T) (zi, y. 0) can be decomposed into a linear combination of [1, he (a; 0), Mue(2;, 0)² ha(ani 0) )ik yik, Vie with respective coefficients (c1, (h,,h, Chuy Chys Cys Cyvl whose values are implicitly functions X:
"/k (mi, Vs. 0) + ca 0) + Chhhi 0)² + Chylis (z;, Q)ua + Colik + Coylhe- (12)
4 CHARACTERIZING TRAINING DYNAMICS
Using the decomposition framework above, it is possible to characterize and compare training dynamics under different loss functions. In this section, the decompositions are first analyzed under zero training error regime to identify optimization biases that lead to implicit regularization. Second, generalizing to the entire training process, theoretical constraint is derived on the entropy of a network's outputs. Combined with experimental data, this constraint characterizes the data fitting and regularization processes that result from the TaylorGLO training process. This characterization shows how individual training samples are treated by different loss functions.
4.1 OPTIMIZATION BIASES THE ZERO TRAINING ERROR REGIME
Certain biases in optimization imposed by a loss function can be best observed in the case where there nothing new learn from the training data. Consider the case where there zero training error, that is. h4. 0) Vik O. In this case, all ha, 0) can be substituted with yik in Ye (Ti, Vi. 0), as is done below for the different loss functions.
Mean squared error (MSE): In this case,
The ( yı, 0) 2yi 2h (z,, 0) = O.
(13)
Thus, there are no changes to the weights the model once error reaches zero. This observation contrasts with the findings in Blanc et al. (2020), who discovered implicit regularization effect when training with MSE loss and label noise. Notably, this null behavior is representable a non-degenerate TaylorGLO parameterization since MSE is itself representable by TaylorGLO with = [0, 0,0, 1, 0, 2, 0, 0). Thus, this behavior can be leveraged in evolved loss functions.
Cross-entropy loss: Since ha, 0) O for non-target logits in a training error regime, (2,, y:, 0) g. i.e. an indeterminate form. Thus, an citttarilly-lose-too-----
training error regime analyzed instead, such that (2;, 0) for non-target logits for an arbitrarily small e. Since all scaled logits sum to 1, (2;; 0) = (n - I)e for the target logit. Let us analyze the learning rule
as tends towards 0:
Vik D3 (a. 0))
Vik 0
0g - 03 + lim 17
-0
yak
Dj (hu (2x, 0))
Diu 1 (14)
k=1
(n 1)e
Yik 0 ya = 1.
02 + 1
(15)
Dj (1,, 0))
Intuitively, this learning rule aims to increase the value of the target scaled logits. Since logits are scaled by a softmax function, increasing the value of one logit decreases the values of other logits. Thus, the fixed point of this bias will be to force non-target scaled logits to zero, and target scaled logits to one. In other words, this behavior aims to minimize the divergence between the predicted distribution and the training data's 's distribution.
TaylorGLO can represent this behavior, and can thus be leveraged in evolved loss functions, through any where 0 and b c+c2 0. Any À where A 2λ + As 221 X² represents such a satisfying family of cases. Additionally, TaylorGLO allows for the strength of this bias to be tuned independently from by adjusting the magnitude of l
3
5

---

Under review as a conference paper at ICLR 2021
Notably, the Baikal loss function results in infinite gradients at zero training error, rendering it unstable, even if using it to fine-tune from a previously trained network that already reached zero training error. However, the zero-error regime is irrelevant with Baikal because it cannot be reached in practice:
Baikal loss:
Theorem I: Zero training error regions of the weight space are not attractors for the Baikal loss function.
The reason is that if a network reaches reaches training enror that is arbitrarily close to zero, there is a repulsive effect that biases the model's weights away from zero training error. Proof of this theorem in Appendix D.T
Third-order TaylorGLO loss: can be written (2, yi, 0) = + byik + cyjk
According to Equation 2 in the zero-error regime (a. y/ 0) linear combination of [1, Vik: Via and a, b, c] (as defined in Appendix Bp. i.c.
Notably, in the basic classification case, Vw Ni yik Vik- since you {0, 1]. This observa- tion provides an intuition for why higher-order TaylorGLO loss functions are not able to provide fundamentally different behavior (beyond a more overparameterized search space), and thus no improvements in performance over third-order loss functions. The learning rule thus becomes
0, 02 + 1 Σ
aDz (h+ (x; 0)) (a+b+ c)D2 (hx (z 0))
Vik: = 0
(16)
Dik = 1,
where a, b, are functions of À defined in Appendix B
As a concrete example, consider the loss function TaylorGLO discovered for the AIICN--C model on CIFAR-10 (Gonzalez& & Miikkulainen 2020b). It had
-373.917, b -12..929 Notably, all three coefficients are negative, i.e. all changes to 8, are a negatively scaled values of Dj (h& (2,, 0)), as be seen from Equation 16 Thus, there are two competing processes in this learning rule: one that aims to minimize all non-target scaled logits (increasing the scaled logit distribution's is entropy), and one that aims to minimize the target scaled logit (decreasing the scaled logit distribution's 's entropy). The processes conflict with each other since logits are scaled through softmax function. These processes can shift weights in a particular way while maintaining zero training error, which results in implicit regularization. If, however, such shifts in this zero training error regime do lead to misclassifications on the training data, h4 (I,, O) would no longer equal yik- and non-zero error regime's 's learning rule would come into effect. It would strive to get back to zero training error with different 0.
11.3145.
Similarly to Baikal loss, training еrror of exactly zero is not attractor for some third-order T loss functions (this property can be seen through an analysis similar to that in Section D.I].. The zero-error case would occur in practice only if this loss function were to be used to fine tune network that truly has zero training error. is, however, a useful step in characterizing the behavior of TaylorGLO, will be seen later.
4.2 DATA FITTING AND REGULARIZATION PROCESSES
Under what gradient conditions does a network's softmax function transition from increasing the entropy in the output distribution to decreasing it? Characterizing this behavior can yield insights on how specific training samples affect a network's 's trainable parameters. Let us analyze the case where all non-target logits have the same value, カ- and the target logit has the value That is, all non-target classes have equal probabilities.
Theorem The strength of entropy reduction is proportional to
Drrió D fofn 3)
<(c ()-1) 1) (ele-1)(1----))
(17)
(te-1)yp(n-
where Y-2 is the value of "/ for non-target logits, and y for the target logit. Thus, values less than zero imply that entropy is increased, values greater than zero imply that is decreased, and values equal zero imply that there is change. The proof is in Appendix D33

---

Under review as a conference paper at ICLR 2021
Attradion Towards Training Emorr
1.0
Alraccion Towards Zero Training Error
0.0
0.0
-0.5
-1.0
-1.0
150
300
350
Epoch
Epoch
(a) Cross-Entropy Loss
(a) TaylorGLO Loss
Figure 1: Attraction towards zero training error with cross-entropy and TaylorGLO loss functions on CIFAR-10 ALICNN-C models. Each point represents individual training sample (500 randomly sampled per epoch); its x-location indicates the training epoch, and y-location the strength with which the loss functions pulls the output towards the correct label, or pushes it away from it. With the cross-entropy loss, these values are always positive, indicating a constant pull towards the correct label for every single training sample. Interestingly, the TaylorGLO values span both the positives and the negatives; at the beginning of training there is a strong pull towards the correct label (seen as the dark area on top left), which then changes more prominent push away from later epochs. This plot shows how TaylorGIO regularizes by preventing overconfidence and biasing solutions towards different parts of the weight space.
The strength of the entropy reduction in Theorem 2 can also be thought of as measure of the strength of the attraction towards zero training errоr regions of the parameter space (i.e., shrinking non-target logits and growing target logits imply reduced training error). This strength can be calculated for individual training samples during any part of the training process, leading the insight that the process results from competing "push" and "pull" forces. This theoretical insight, combined with empirical data from actual training sessions, explains how different loss functions balance data fitting and regularization.
Figure T provides one such example on ALICNN-C (Springenberg et al. 2015) models trained on CIFAR-10 (Krizhevsky & Hinton 2009) with cross-entropy and custom TaylorGLO loss functions. Scaled target and non-target logit values were logged for every sample at every epoch and used to calculate respective "rr and "y-r values. These values then substituted into Equation 17 to get the strength of bias towards zero training error. Specific instances of Equation 17 for four loss functions are plotted in Appendix E
The cross-entropy loss exhibits a tendency towards zero training error for every single sample, as expected. The TaylorGIO loss, however, has much different behavior: initially, there is much stronger pull towards zero training error for all samples-mine which leads to better generalization (Yao) al., (2007; [i al., 2019) which a stratification occurs, where the majority samples are repelled, and thus biased towards a different region of the weight space that happens to have better performance characteristies empirically. A similar behavior is identified for the Baikal loss in Appendix
5 INVARIANT ON TAYLORGLO PARAMETERS
There are many different instances of À for which models are untrainable. One such case, albeit degenerate one, is X 0 (i.e., function with zero gradients everywhere). Given the training dynamics at the null epoch (characterized in Appendix C). more general constraints on X can be derived (in Appendix D.4], resulting in the following theorem:
Theorem third-order TaylorGLO loss function not trainable if the following constraints on X satisfied: C1 + + Cyn +
Ch Chy
Chh
(n (n-1)
Ch
Chub
(18)
n²
n
n²
Chy
Ch
Chb
Cy + Cuy
(n 2)
(19)
n
The inverse of these constraints may be used as an invariant during loss function evolution. That is, they can be used to identify entire families of loss function parameters that usable, rule
7

---

Under review as a conference paper at ICLR 2021
Table I: Test-set accuracy of loss functions discovered by TaylorGLO with and without an invariant constraint on A. Models were trained on the loss function that had the highest validation accuracy during the TaylorGLO evolution. All averages are from ten separately trained models and p-values are from one-tailed Welch's s t-Tests. Standard deviations are shown in parentheses. The invariant allows focusing metalearning to viable areas of the search space, resulting in better loss functions.
Task and Model
Avg. TaylorGI.O Acc.
Invariant
p-value
CIFAR-10 on AlexNet
0.7901 (0.0026)
0.7933 (0.0026)
0.0092
CIFAR-I0 PreResNet-20"
0.9169 (0.0014)
0.9164 (0.0019)
0.2827
CIFAR-IE on AIICNN-C*
0,9271 (0.0013)
0.9290 (0.0014)
0.0004
CIFAR- 10 AlICNN-C* Cutout
0.9329 (0.0022)
0.9350 (0.0014)
0.0124
Krizhevsky at.
2012) He el 2016
Springenherg et al. 2015 DeVries Taylor
them out during search, and thereby make the search more effective. More specifically, before each candidate À is evaluated, it is checked for conformance the invariant. If the invariant is violated, the algorithm can skip that candidate's validation training and simply assign a fitness of zero. However, due to the added complexity that the invariant imposes the fitness landscape, larger population size is needed for evolution within TaylorGLO to be more stable. Practically, a doubling of the population size from 20 to 40 works well.
Table presents presents results from TaylorGLO runs with and without the invariant on the CIFAR-10 image classification benchmark dataset (Krizhevsky & Hinton 2009) with various architectures. Networks with Cutout (De & Taylor, 2017) were also evaluated to show that TaylorGLO provides different approach to regularization. Standard training hyperparameters from the references were used for each architecture. Notably, the invariant allows TaylorGLO to discover loss functions that have statistically significantly better performance many cases and never a detrimental effect. These result demonstrate that the thcoretical invariant is useful in practice, and should become a standard in TaylorGLO applications.
ADVERSARIAL ROBUSTNESS
TaylorGI.O loss functions discourage overconfidence, i.e. their activations are less extreme and vary more smoothly with input. Such encodings are likely to be more robust against noise, damage, and other imperfections in the data and in the network execution. In the extreme case, they may also be more robust against adversarial attacks. This hypothesis will be tested experimentally in this section.
Adversarial attacks elicit incorrect predictions from trained model by changing input samples in small ways that can even be imperceptible. They are generally classified as "white-box" or "black-oox" attacks, depending on whether the attacker has access the underlying model or not, respectively. Naturally white-box attacks are more powerful at overwhelming a model. One such white-box attack is the Fixed Gradient Sign Method (FGSM: Goodfellow et al., 2015): following evaluation of dataset, input gradients are taken from the network following a backward pass. Each individual gradient has its sign calculated and scaled by an e scaling factor that determines the attack strength. These values are added to future network inputs with an e scaling factor, causing misclassifications
Figure 2 shows how robust networks with different loss functions are FGSM attacks of various strengths. In this experiment, ALICNN-C and Wide ResNet 28-5 (Zagoruyko & Komodakis 2016) networks were trained on CIFAR-10 10 with TaylorGLO and cross-entropy loss; indeed T outperforms the cross-entropy loss models significantly at all attack strengths. Note that in this case, loss functions were evolved simply to perform well, and adversarial robustness emerged as side benefit. However, it is also possible to take adversarial attacks into account as an explicit objective in loss function evolution. Since TaylorGLO can uses non-differentiable metrics as objectives in its search process, the traditional validation accuracy objective can be replaced with validation accuracy at a particular FGSM attack strength. Remarkably, loss functions found with this objective outperform both the previous TaylorGLO loss functions and the cross-entropy loss. These results demonstrate that the TaylorGLO regularization leads to robust encoding (detailed in Appendix [F). and such robustness optimization.
can be further improved by making it explicit goal in
8

---

Under review as a conference paper at ICLR 2021
1,00
Cress Breacy Loos
Tayoo3LO
TLD Robusticss
TaylerLO Loso with Robustrose Objoctie
0.80
Accurscy
0.80
0.6
0.40
Testing
0.4
0.20
0.2
0.00
0
0.05
0.1
0.15
0.2
0.25
0.3
0
0.05
0.1
0.15
0.2
0.25
0.3
FGSM Epsilon (Atrack Strength)
FGSM Epsilon (Attsck Strengtth)
(a) AIICNN-C
(b) ALICNN-C C with Cutout
Croses Entropy Loss lap Loon
1,00
Cross- Entr--
ledertaa Losa
0.8
0.80
NepertiL Loss Roobustnes Oojectiv
Teating Accuracy
Testing Accuracy
0.6
0.60
0.4
0.40
0.2
0.20
0.00
0.05
0.15
0.05
0.1
0.15
0.2
0.25
0.3
FGSM Epsilnn (Attack Strengthh
FGSM (Attack Strength)
(c) Wide ResNet 16-8
(d) Wide ResNet 28-5
Figure 2: Robustness of TaylorGLO loss functions against FGSM adversarial attacks on CIFAR- 10. For each architecture, the blue bars represent accuracy achieved through training with the cross- entropy loss, green bars with a TaylorGI.O loss, and gray bars with a TaylorGI.O loss specifically evolved in the adversarial attack environment. The leftmost points on each plot represent evaluations without adversarial attacks. TaylorGLO regularization makes the networks more robust against adversarial attacks, and this property can be further enhanced by making it an explicit goal in evolution.
Future work can naturally extend these analyses to black-box adversarial attacks and adversarial training. Moreover, just custom loss functions improve intrinsic robustness against adversarial attacks, custom loss function may make adversarial training more effective.
CONCLUSION
Regularization has long been a crucial aspect of training deep neural networks, and exists in many different flavors. This paper contributed an understanding of one recent compelling family of regularization techniques: loss-function metalearning. A theoretical framework for representing different loss functions was first developed in order to analyze their training dynamics various contexts. The results demonstrate that TaylorGLO loss functions implement guard against overfitting, resulting in automatic regularization. Two practical opportunities emerged from this analysis: filtering based on invariant to make the networks more robust against adversarial attacks. The results thus extend the scope of metalearning, focusing it not just finding optimal model configurations, but also improving regularization, learning efficiency, and robustness directly.
shown to improve the search process, and the robustness against overfitting
REFERENCES
Antreas Antoniou and Amos Storkey. Learning to learn by self-critique. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32, pp. 9940-9950. Curran Asso-

---

Under review as a conference paper at ICLR 2021
ciates, Inc., 2019. URL http::///ppoeeedii....eeeooeooep 6018df1842 111.LBBB227701222866611166-6---- pdf
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta-learning via learned loss. arXiv preprint arXiv: 2019.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical variinaiannee trade-off. Proceedings of the National Academy of Sciences, 116(3))15849-11854, 2019.
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an orsteen-uuleeneck like process. In Conference Learning Theory, pp. 483-513, 2020.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124018, 2019.
Terrance Vriei and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /00. 04552, 2017.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances Neural Information Processing Systems 32, pp. 10677-10687, 2019.
Santiago Gonzalez and Risto Miikkulainen. Improved training speed, accuracy, and data utiliza- tion through loss function optimization. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC), 2020a.
Santiago Gonzalez and Risto Miikkulainen. Evolving loss functions with multivariate taylor polyno- mial parameterizations. arXiv preprint arXiv:2002.00055 2020b.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Third International Conference on Learning Representations (ICLR), 2015.
Stephen José Hanson and Lorien Y Pratt. Comparing biases for minimal network construction with pack-propagation. Advances in Neural Information Processing Systems, pp. 177-185, 1989.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. Advances in Neural Information Processing Systems 31, pp. 5400-5409, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 448-456, 2015.
Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Do we need zero training loss after achieving zero training error? arXiv preprint arXiv:2002.08809, 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In Proceedings of the Fifth International Conference on Learning Representations (ICLR), 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012.
10

---

Under review as a conference paper at ICLR 2021
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape neural nets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gareett (eds.), Advances in Neural Information Processing Systems 31, pp. 6389-6399. Curran Associates, Inc., 2018.
Yuanzhi Li. Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bue, Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 11674-11685. Curran Associates, Inc., 2019.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. In International Conference on Learning Representations, 2018.
Nelson Morgan and Hervé Bourlard. Generalization and parameter estimation in feedforward nets: Some experiments. In Advances in Neural Information Processing Systems, pp. 630-633-- 000-667775 1990.
Kristen Morse, Neha Das, Yixin Lin, Austin Wang, Akshara Rai, and Franziska Meier. Learning state-dependent losses for inverse dynamics learning. arXiv preprint (0Xx::0003.44447 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. International Conference on Learning Representations (ICLR), 2019.
Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In C. Cortes, N. D. Lawrence, D. D. Lec, M. Sugiyama, and R. Garett (eds.), Advances in Neural Information Processing Systems 28, pp. 2422-2430. Curran Associates, Inc., 2015.
Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Geometry of optimization and implicit regularization in deep learning. arXiv preprint arXiv:1705.03077, 2017.
Scott Niekum, Andrew G Barto, and Lee Spector. Genetic programming for reward function search. IEEE Transactions on Autonomous Mental Development, 2(2):83-90, 2010.
Lee Spector, Erik Goodman, Wu, W B. Langdon, m. Voigt, M Gen, S Sen, M Dorigo, S Pezeshk, M Garzon, E Burke, and Morgan Kaufmann Publishers. Autoconstructive evolution: Push, pushgp, and pushpop. Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001). 05 2001.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, abs/1412.68800 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research JMLR, 15(1)1192-1158, 2014.
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning to learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv: 1706.0990, 2017.
C. Szegedy, Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp- 4148-4158 Curran Associates, Inc., 2017.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-315 2007.
11

---

Under review as a conference paper at ICLR 2021
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 6023 6032, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 105.011005.07146 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018.
Wei Zhou, Yiying Li, Yongxin Yang, Huaimin Wang, and Timothy M Hospedales. Online meta-critic learning for off-policy actor-critic methods. arXiv preprint arXiv:2003.05334 2020.
A NOTATION OVERVIEW
Symbol
Description
h(a,, 0)
The model, with softmax The model's kth scaled logit
hy 0)
Dj (f)
The directional derivative of f along j
Paaa
Probability distribution of original data An input data sample, where æ; Paau
A label that corresponds to the æ; sample Learning rate
y
n
Number of classes
0
model's trainable parameters
À
The loss function's 's parameters
C(a,, Yt. 0)
The loss function
B THIRD-ORDER TAYLORGLO LOSS FUNCTION IN THE ZERO TRAINING ERROR REGIME
(1, yı. y: 0) = a + byjk + cykke
(20)
where
a = A2 - 2-22143- - Asdo + +211 + A7 3Ma27
(21)
b = 2dg 2N 21176 As 2\7du 6d4
(22)
c = 2 + A7 + +3A4
(23)
C BEHAVIOR AT THE NULL EPOCH
Consider the first epoch of training. Assume all weights are randomly initialized:
Vk [1, n), where 2: E (a; 0)]
(24)
That is. logits are distributed with high entropy. Behavior at the null epoch can then be defined piecewise for target vs. non-target logits for each loss function.
In the case of Mean squared error (MSE),
2n 2n
Vik 0
The x Dr, 0)
(25)
2n 2n -
Vik 1.
Since n 2 2, the yıu case will always be negative, while the You case will always be positive. Thus, target scaled logits will be maximized and non-target scaled logits minimized.
In the case Cross-entropy loss
0
= 0 Vak 1.
7h a y:s 0)
(26)
n
12

---

Under review as a conference paper at ICLR 2021
Target scaled logits are maximized and, consequently, non-target scaled logits minimized as result of the softmax function.
Similarly in the case of Baikal loss,
71 n Vik 1.
Vik
"ik (x;, ys 0)
(27)
Target scaled logits are minimized and, consequently, non-tarn con-ttrget scaled logits minimized as result of the softmax function (since the yik = 1 case dominates).
In the case of Third-order TaylorGLO loss, since behavior is highly dependent X, consider the concrete loss function used above:
373.917 - 130.264 h4 (z; 8) - 11.2188 hg (z, 0)2 372.470735 131.47 hg (Z,, 0) 11.2188 hg (z,, 0)²
Yik = 0
7/k ( Ve. 0)
(28)
Yak
Note that Equation 16 is special case of this behavior where hy: (2;; 0) Yik- Let substitute hy (x;, 0) (i.e., the expected value of a logit at the null epoch):
- -373.917 130.264 n 11.2188 *Ak (Zi, yi, 0) 377.470735 131.47 n- 11.2188 n Yik = 1.
= 0
(29)
Since this loss function was found on CIFAR-10, 10-class image classification task, n 10:
n 7 (z,, yi: 0)
386.9546188
Vik =
(30)
-385.729923
Yiu: 1.
Since both cases of (2,, yi, 0) are negative, this behavior implies that all scaled logits will be minimized. However, since the scaled logits are the output of softmax function, and the Yik = 0 case is more strongly negative, the non-target scaled logits will be minimized more than the target scaled logits, resulting in maximization of the target scaled logits.
D PROOFS AND DERIVATIONS
Proofs and derivations for theorems in the paper are presented below:
D.1 ZERO TRAINING ERROR IS NOT AN ATTRACTOR OF BAIKAL
Given that Baikal does tend to minimize training error to large degree-otherwise otherwise it would be useless as loss function since we are effectively assuming that the training data is in-distribution- we can observe what happens as we approach a point parameter space that arbitrarily-close to zero training error. Assume, without loss of generality, that all non-target scaled logits have the same value.
22
lim be (m,
7k (a;; yr, O)D (a:; 0))
Vik 0
8 + 83 +
(31)
72
lim 7/ (0 yr. Dy (z, 0)) (e
Vik 1
1
0
n
by(mn, 0) - -
lim
hu (an; 0)
hi (m; 0)2 1
Dj (hx (2:;; 0))
Vik
0, +1
(32)
11
lim hy (m;
hi (2n;, A)
1
(mii 0)2
Dj (hx(x:: 0))
yeu =
Dj (2, 0))
Vik = 0
03 11
(33)
Dy (hyla,, 0))
yok = 1
(1 e)²
1
n
Di (mx 0))
Vik
0, +
(34)
71
Dj (1x 0))
ViA
2 +
13

---

Under review as a conference paper at ICLR 2021
The behavior in the You 0 case will dominate for small values of c. Both cases have positive range for small values of e, ultimately resulting in non-target scaled logits becoming maximized, and subsequently the non-target logit becoming minimized. This is equivalent, in expectation, to saying that e will become larger after applying the learning rule. A larger clearly implies a move away from training error area of the parameter space. Thus, zero training error is not attractor for the Baikal loss function.
D.2 LABEL SMOOTHING IN TAYLORGLO
Consider a basic setup with standard label smoothing, controlled by hyperparameter a E (0, 1), such that the target value in any y: is a==1, rather than 1, and non-target values are rather than O. The learning rule changes the general case as follows:
c1 chhx 0) + chaha( 0)
a²
tchgha 0) a
cy
+
Cyv
ya
n
7
n2
(2: Vi, 0)
n
c1 + cx,4( 0) + chnhi (2; 0)2 Chyhhx2; 0)
n
+Gy 1 1-- + Cyg
1
f1
(35)
Let ê1, En. Cab. Chyy cy, Cuy represent settings for Ch,ch, Chiss Chyy Cys Cyy in the non-label-mmoothed case that implicitly apply label smoothing within the TaylorGLO parameterization. Given the two cases in the label-smoothed and non-label-smoother definitions of 7 (2 ys 0), there two equations that must be satisfiable by settings of è constants for any constants, with shared terms highlighted in blue and red:
+ cnhi (x; 0) + caa 0)2 + Chy (m; 0) + cy n
a
+ Cyu n
(36)
21 + cn (wii,
i)
ènalie (xi
0)
1
C + cnhha 0) + cna
0)2 + chyle (x) 0)
1
n -=11 + Cuy
n - 2
(37)
+Cg 1
1 - a
= é + Ca 0) + hn (2,, 0)² + Cnyhu (2),, 0) + Cy + Cyy
Let us then factor the left-hand side of Equation 36 in terms of different powers of (2;, 0):
a² + Ch + Chy
C1 + Cy
+ Cyn n²
he (ai, 0) + Chu h4 (x, 0)
n
(38)
Resulting in definitions for èι, cn. Cah- Let us then add the following form of zero to the left-hand side of Equation 37
Chty Chglu (2; 0) Cy C
Cyv n²
Chyltx (z; 0) + Cy
Cy
7
(39)
This allows to substitute the definitions for é₁ cn. End from Equation 38
into Equation 37
2
é + Epiieaa 0) + Cnn (2;;
Chy (2:i, A) a
+ Cy a
+ Cyu n
tehyhy(aa,, 0) - a n _n-1) + +cy 1 - a n1 1n-1) + Cyp (1 1- a
n 2
(40)
n
E + Cnhi (2;, 0) + Cha (2; 0)2 + Cony (2;; 0) + C, + Eyy
14

---

Under review as a conference paper at ICLR 2021
Simplifying into:
( 0) (1- n a
n
n1
2
+ cy
a
+ Cyv
1 - a
n
Chylla (20;, 0 a + Cyj
2
(41)
+ Cap
n
= Cnyhh(lss 0) + E + Cyp
Finally, factor the left-hand side of Equation
41
in terms of, 11 (vi, 0), 1, and 12:
Cby
a
72
1
Chay
hy (2;; f)
n
(42)
11
at
71
O
+
Cy
1
Cy
+
Cyz
-
a
Cyy
nt
Euv
Thus, the in- parameterization constants with implicit label smoothing can be defined for any desired, label-smoothed constants follows:
2
é₁ = 61 + Cy
Cyy
(43)
n
a
En. = on + Chy
(44)
m
Chh
(45)
71
0
Čhe = Clyy
ak
Chyy
(46)
n
1
Cy
1
a
Cy
(47)
Eyg
Cyg
a
Cyn n2
(48)
71
So for any À and any (0, 1). there exists Å such that the behavior imposed by Å without explicit label smoothing is identical to the behavior imposed by À with explicit label smoothing. That is, any degree of label smoothing can be implicitly represented for any TaylorGLO loss function. Thus, TaylorGLO may discover and utilize label smoothing as part of discovering loss functions, increasing their ability regularize further.
D.3 SOFTMAX ENTROPY CRITICALITY
Let us analyze the case where all non-target logits have the same value, مंب and the target logit has the value c. That is, non non-target classes have equal probabilities.
A model's scaled logit for an input æ; can be represented
e
(x ( 0) = ( 0))
(49)
プ (m..0)
where Sa(a 0) is raw output logit from the model.
The (k, j)th entry of the Jacobian matrix for h(x,, 0) can be easily derived through application of the chain rule:
Ja, 0) (hg 0) hy (2,, 0) (1 (x,, 0)) fx, 0) 0f5(2; a., 0)
k ==
(50)
(a, 0) (x: 0) (a 0)
k: k
Consider SGD learning rule of the form:
n 0, - 0 + n Σ
(x, vi. D) (2, 0))
(51)
15

---

Under review as a conference paper at ICLR 2021
Let us freeze network at any specific point during the training process for any specifie sample. Now, treating all (Zz J).j J. E [1, n| as free parameters with unit derivatives, rather than as functions. That is, 0g = (Z,, 0). We observe that updates are as follows:
Af
Σ Arm
{
hy 0) (ai, 0)) (2s, 0) (2i, 0)
k == j k *
(52)
For downstream analysis, we can consider, substitutions for " above, 7-T be the value for non-target logits, and "/ for the target logit.
This can be expanded and conceptually simplified by considering j indices and -j indices. -/ indices, of which there are n - 1, are either all non-target logits, or one is the target logit in the case where j not the target logit. Let us consider both cases, while substituting the scaled logit values defined above:
Afs x { 7-T Jk= 0) + (n - 2) T J Juzzh 0) + yr Jk 0) TT J Jk=z/h Jk=zh( (r; 0) + (n 1yy J Jkzph(a; O)
non-target j target j
(53)
7- 1i (1 h-T (æ; 0)
((n 2 (- h- Di 0) h-T (; 0)
+7 (-h-r (x 0) hr ( 0))
non-target
Aff
(54)
Yphr 0) (1 hr ( 0))
+(n 1)- (-h . (1; 0 lu ( 0))
target j
where (x 0) = c, h (zi, 0)
(55)
n - 1
2
7-T
1 T-T + 7- - 2) n2= 2n +
+ 7T(E 1)
-1)--
e
non-target
Afj x
n1
6
PPC - rr + 7-r(n 1)(c - 1)
target j
n
(56)
At this point, we have closed-form solutions for the changes to softmax inputs. To characterize entropy, we must now derive solutions for the changes to softmax outputs given such changes to the inputs. That is:
(f(
Lo,(f(v;,,) 0))
(57)
Er=1 efa +)
Due to the two cases in Af,, Ao, (f(x; 0)) is thus also split into two cases for target and non-target logits:
()
non-target
(n
Ao,(f(z;,0)
(58)
target
(n 1) efr 02
Now, we can see that scaled logits have a lower entropy distribution when (f 0)) > 0 and Ao Ar-r( (f 0)) < O. Essentially, the target and non-target scaled logits are being repelled from each other. We can ignore either of these inequalities, if one is satisfied then both are satisfied, in part because (x 0))l 1. The target-case constraint (i.e., the target scaled logit must grow) can be represented as:
+)
(n 1)ef (m.0))AA + cfr(m,))+ 1 -
(59)
Consider the target logit case prior to changes:
())
(n- - ()-)))e-r(;;0)) ctr(2,00) =1
(60)
Let us solve for efr( 0
(n - 1)ef-r(m;;0) + ( - c(n - ()0f-r((;00) - (0;
(61)
12 n-1 + ef-r(m;;))
=
(62)
16

---

Under review as a conference paper at ICLR 2021
Substituting this definition into Equation
59:
exr
n
n2 + ef-r(=;;0)
e
>1-6
(63)
(n () 0)) AS-T + cods
- n (" cl-rlon.)) -n1))crrro
Coalescing exponents:
n- -
+
(64)
11 (n-1
(n 1) catr+f-r(an00)
-n +
Substituting in definitions for ASr and AS- and greatly simplifying in a CAS is able to remove instances f-r:
c(c f(6-1) 1)r((n - (7-(((((( - +
<e ()-1) 0ole-119-7-70)-
(n 1)2
(65)
e(c - 1)rr(n 1) + 87 ( +
(e- -)ee----)-----mmm 1r) ee
(n 1)2
D.4 TAYLORGLO PARAMETER INVARIANT AT THE NULL EPOCH
At the null epoch, valid loss function aims to, in expectation, minimize non non-target scaled logits while maximizing target scaled logits. Thus, we attempt to find cases of X for which these behaviors occur. Considering the representation for 7A (m yı, 0) in Equation 12
(ar + cahs(a, 0) + Chahs(as 0)2 Dj (h, 0))
Vik =
8 0, +
(a chhn (x; 0) + ch 0)² +Chyhs(a, 0) + cy + Cau) Dj (h4 (x 0))
(66)
=
Let us substitute (1, 0) (i.e., the expected value of logit at the null epoch):
(c +
Ch
Chu
Dj 0))
Ysk 0
0, - 0, +
n
n²
(67)
C1 + Cy + Cyp
Ch
+ Chy
Chub
D3 (hy (1,, 0))
Vik =
n
n²
For the desired degenerate behavior to appear, the directional derivative's coefficient in the y/z case must be less than zero:
1
Ch +
Chy
Cbu
+ Cy + Cwy +
(68)
n
This finding be made more general, by asserting that the directional derivative's coefficient in the Vik case be less than (n - 1) times the coefficient in the yik = O case. Thus arriving at the following constraint on A:
+ Cha
Cha
Chh
C1 + Cy
Cyp
(n 1)
C1
+
(69)
n
n²
n
n
Cy +
Cyp
Chu
(n 2)
c1
C₁
Chh
(70)
n
n
The inverse of these constraints may be used as invariants during loss function evolution.
E EXAMPLES OF ZERO TRAINING ERROR ATTRACTOR DYNAMICS
The strength of the attraction towards zero training error regions of the parameter space (described in Theorem 2) be plotted for any given number of classes at different values using the
17

---

Under review as a conference paper at ICLR 2021
Attraction Towards Zero Training
1.0
Attraction Towards Training
1.0
0.5
0.5
0.0
0.0
n=
n 8
-0.5
n
n
10
-0.5
-18.0
-18.0
0.2
0.4
0.8
0.4
Deviation from Memorization
Deviation from Memorization
(a) Cross-Entropp Loss
(b) TaylorGLO Loss
Attracion Towards Zero Training
Attraction Towards Zero Training
1.0
0.5
0.5
0.0
0.0
2
n
-0.5
n 4
n 10
-0.5
n 6
-1.0
-1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Deviation from Memorization
Deviation from Memorization (E)
(c) MSE Loss
(d) Baikal Loss
Figure 3: Attraction towards zero training error with different loss functions. Each loss function has a characteristic curve plotted using Equation [7 177-thtt describes zero training error attraction dynamics for individual samples given their current deviation from perfect memorization, Plots (a) and (b) only have the n = 10 case plotted, ie. the 10-class classification case for which they were evolved. Cross-entropy (a) and MSE (c) loss functions have positive attraction for all values of In contrast, the TaylorGI.O loss function for CIFAR-10 on ALICNN-C (b) and the Baikal loss function (d) both have very strong attraction for weakly learned samples (on the right side), and repulsion for highly confidently learned samples (on the left side). Thus, this illustration provides graphical intuition for the regularization that TaylorGLO and Baikal loss functions establish.
corresponding "rr and 7 values from particular loss function. These characteristic curves for four specific loss functions are plotted in Figure 3
Both Baikal and TaylorGLO loss functions have significantly different attraction curves than the cross-entropy and mean squared error loss functions. Cross-entropy and mean squared error always exhibit positive attraction to zero training error. Conversely, TaylorGLO and Baikal only exhibit this positive attraction behavior for samples that are weakly memorized; well memorized samples produce a repulsive effect instead. This difference what contributes to both metalearned loss functions' regularizing effects, where overconfidence is avoided.
F COMPARING MINIMA OF TRAINED NETWORKS
TaylorGI.O loss functions are known result in trained networks with flatter, lower minima (Gonzalez & Miikkulainen, 2020b). This result suggests that models trained with TaylorGLO loss function are more robust, i.c. their performance is less sensitive to small perturbations in the weight space, and that they also generalize better (Keskar et al. 2017). Since TaylorGLO loss functions that were discovered against an adversarial performance objective were even more robust (Section (6), what do their minima look like?
Model performance can be plotted along a random slice [-1, 1] of the weight space using a prior loss surface visualization technique (Li et al. 22018) The random slice vector normalized in filter-wise
18

---

Under review as a conference paper at ICLR 2021
80
Test Accuracy
60
Cross-Entropy Loss
40
TaylorGLO Loss
Adversarially Robust TaylorGLO Loss
20
-1.0
-0.5
1.0
Perturbation Along Random Basis Vector
Figure 4: Comparing accuracy basins of AIICNN-C with cross-entropy, TaylorGLO, and adversarially robust TaylorGLO loss functions on CIFAR-10. Basins are plotted along only one perturbation direction for clarity, using prior loss surface visualization technique (Li et al. 2018). While the adversarially robust TaylorGLO loss function leads to the same accuracy as the standard one, it has a wider, flatter minima. This result suggests that the TaylorGLO loss function that has been evolved to be robust against adversarial attacks is more robust in general, even when adversarial attacks are of concern.
manner accommodate network weights' scale invariance, thus ensuring that visualizations for two separate models can compared. As a result of the randomness, this parameter space slice unbiased and should take all parameters into account, to degree. It can therefore be used to perturb trainable parameters systematically.
When ALICNN-C is trained with adversarially robust versus standard TaylorGLO loss function, its absolute accuracy is the same. However, the minimum is wider and flatter (Figure 4 This result suggests that it may be advantageous to evaluate TaylorGLO against an adversarial performance metric, even when the target application does not include adversarial attacks.
19


Reviews:
[{"review_id": "aCSPPnp4I4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes a learned loss function called TaylorGLO based on third-order Taylor expansion and its regularization properties. This approach is novel and interesting in that the loss function is also learned on data. The analysis of the TaylorGLO loss and another learned loss function Baikal loss near zero error reveals interesting properties of preventing overconfident predictions. \n\nHowever, I have some reservations on the idea of learning loss functions in general. \n1. The improvements in classification accuracy using a learned loss function is relatively small, from previous works such as Gonzalez & Miikkulainen 2020b. \n2. It is more difficult to interpret a learned loss function. What does a=-373.917, b=-129.928, c=-11.3145 mean in the Taylor expansion of the loss function? \n3. Since the loss function is learned from data, would it become degenerate in the small sample regime? \n4. Traditional loss functions like cross-entropy and hinge loss have classification rules that are Bayes consistent. It is not clear whether this is the case for learned loss functions. \n\nApart from these questions, there is also the issue of clarity in presentation. The authors should include some discussions or illustration of the Baikal or TaylorGLO loss in this paper to make it more self-contained. \n\nAs a question for Section 5, does the condition in Theorem 3 guarantee the trainability of the TaylorGLO loss (sufficient condition)? Or there are potentially other constraints needed? \n\nOverall I believe this approach has merits in discovering new interesting functions for learning, followed by study of the loss function's properties by humans. But I am not convinced if we should directly use a learned loss function in training. I think there are more work needed for this paper before it could be accepted for ICLR. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GPDHkroU7fh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nTaylor polynomial based loss function metalearning acts as a regularizer that improves the networks adversarial attack robustness, performance, training time, and data utilization. The authors evolve weights, and so add arbitrary other factors to the loss, including adversarial robustness to learn a loss function parameterization which is more robust. They provide analysis of the attractor states under the optimization of a suite of loss functions.\n\nQuality:\n\nWriting Quality:\nThe authors have paid attention to detail. The writing is succinct and precise throughout, and I was only able to find one typo though the first 8 pages of the manuscript. The progression between concepts is well motivated.\n\nEvaluation Quality:\nThe evaluation methodology is novel and its results address the dynamics of training rather than static outcomes. The choice of adversarial attack robustness to demonstrate the value of optimizing the loss function for an alternate metric is a sound one. Validation acuuracy is used elsewhere to evaluate the generalization ability of the loss. Transfer of the learned loss across datsets and models is not evaluated.\n\nResult Quality:\nThe improvements in adversarial attack robustness are large when optimizing TaylorGLO directly for that objective. The differences in attractor dynamics are dramatic (but also are less surprising). \n\nClarity:\n\nGraphs & Tables:\nFigure 1, the attraction dynamics graph, is readable and quite clear. Unfortunately the entropy reduction definition of attraction in equation 17 can’t be efficiently described in the caption in the same way that it is described in the text.\n\nTable 1’s invariance results are clearly presented. The Welch’s t-Test results generating P-value scores and the corresponding bolding is good. I would like to have seen a measure of meta-training stability or consistency in addition to or instead of accuracy, backing the claim about improved stability moving with the evolution population size. \n\nFigure 2’s attack strength against testing accuracy plot is clean. Each axis’ meaning is clear, as and the interpretation of the result is natural. \n\nThe paper’s writing clarity is very good. The background is comprehensible. The decompositions in section 3 are well factored. The disagreement with Blanc et al. (2020) in section 4.1 can be fleshed out in more detail, but the writing in the rest of the section is precise and succinct.\n\n\nOriginality:\n\nOne challenge with addressing the originality in the paper is understanding what novelty should be attributed here and what should be attributed to the original TaylorGLO paper.\n\nNovel evaluation methodology (attractor dynamics) are underemphasized relative to novel regularization results, and depends on the insight that the upside of the zero training error regime is that much of the continued update is all about the optimizer’s bias and not about the training data. \n\n\nSignificance:\n\nOne major question in this work concerns the generality of its findings. Are these attractor dynamics specific to TaylorGLO? What are ther implications for other regularizers? \n\nThe method’s added complexity makes the method unlikely to be used unless it can clearly differentiate itself from other regularizers. For example, label smoothing is likely to create very similar attractor dynamics to the dynamics seen in TaylorGLO. The regularization effect (output entropy penalty) is also very similar. One differentiating feature is the ability to add other objectives (like adversarial robustness) to the learned loss. \n\n\nPros:\n\nThe novel evaluation methodlogy which relies on the insight that all loss function changes will have a downstream impact on the gradients is a nice addition to the loss function metalearning suite. Attractor dynamics, optimizing for an alternate objective like adversarial robustness and the demonstrated flexibility of TaylorGLO to cover label smoothing, MSE, Cross-Entropy, and more are welcome contributions.\n\nCons:\n\nThe appreciation of the existing loss function metalearning literature is poor in this paper. Loss functions are commonly learned with Neural Networks! These losses are often easier to optimize and are more flexible than standard losses. They can also make non-differentiable feedback differentiable. See this metalearning survey for plenty of references. https://arxiv.org/pdf/2004.05439.pdf\n\nComparisons between taylor approximation paremeterized loss functions and neural network parameterized loss functions would have been an important comparison to see, but this side of loss function metaleanring isn’t referenced. What are the attractor dynamics for those neural network learned losses? Is this different / improved? While many of these papers focus on reinforcement learning or unsupervised learning, they point to very similar improvements coming out of loss function metalearning. Ex, any of the following:\n\nEvolved Policy Gradient\nhttps://arxiv.org/abs/1802.04821\nLearning to Learn: Meta-Critic Networks for Sample Efficient Learning\nhttps://www.researchgate.net/publication/318029457_Learning_to_Learn_Meta-Critic_Networks_for_Sample_Efficient_Learning\nOnline Meta-Critic Learning for Off-Policy Actor-Critic Methods\nhttps://arxiv.org/abs/2003.05334\nOnline-Within-Online Meta-Learning (learned regularization)\nhttp://papers.nips.cc/paper/9468-online-within-online-meta-learning.pdf\nMeta-Learning Update Rules for Unsupervised Representation Learning\nhttps://arxiv.org/abs/1804.00222\nLearning to Learn by Self-Critique\nhttps://papers.nips.cc/paper/9185-learning-to-learn-by-self-critique\n\n\nThis paper doesn’t focus on task generality. Many of the other metalearning loss functions papers do. Why? How general are their learned losses? Can they generalize from task to task, or does it have to be retrained every time? Why don’t they discuss these issues?\n\nThey don’t release code for reproducibility.\n\n\nNotes:\n\nIdeally Figure 1’s information would be shown for Bikal, MSE, and Label Smoothing as well (perhaps in the appendix) to assess whether TaylorGLO’s training dynamics add anything on top of Label Smoothing (which one would expect to have the same transition to push away from the correct label in later epochs). But there the definition of zero training error itself is modified, and so their metric may not capture very similar optimization dynamics.\n\nThe claim that TaylorGLO lowers confidence could be evaluated on calibration, rather than or in addition to entropy. \n\nTaylorGLO may be doing much more than regularization in practice, and the evaluiton criteria don’t seem sufficient to know that more isn’t happening to the model optimzed for this loss.\n\nIt would be good to see the attractor dynamic graphs for label smoothing (presumably it is very similar to TaylorGLO).\n\nIt appears that Theorem 4.2 basically describes label smoothing, though they don’t say this.\n\nA typo on page 6! “Thus, values less than zero imply that entropy is increased, values greater than zero that it is decreased, and values equal to zero imply that there is no change.”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "D_uKnsrZcz", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper mainly deals with the theoretical support for the loss function meta-learning and focuses on illustrating the generalization superiority of the TaylorGLO method [1]. Although the generalization performance is the core aspect of machine learning, I have several concerns as follows.\n\n1. Since the TaylorGLO method [1] is also under review in ICLR 2021, I can not judge the value of this paper.\n2. This paper claims that it theoretically analyzes the generalization superiority of the TaylorGLO method. But I cannot get this point from this paper. In my opinion, when we consider the generalization performance, the generalization error bound is preferred to answer this. Here the authors try to analyze the training dynamics of SGD for deep models since previous work has shown the implicit regularization effect of the gradient-based optimization algorithms. But, what's the connection between the training dynamics and generalization, or what's the hypothesis between these two in this paper? Do I miss something? \n3. what do the theorems provided in this paper want to tell? I don't get the points that the authors want to tell. More intuitive explanations should be given followed by the theorem.\n\n\n[1] OPTIMIZING LOSS FUNCTIONS THROUGH MULTI-VARIATE TAYLOR POLYNOMIAL PARAMETERIZATION, https://openreview.net/pdf?id=bJLHjvYV1Cu", "extra_scores": {}, "extra_text": {}}, {"review_id": "7XjGztR6OX0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE:\n\nAfter the reviewers clarifications and some further explanations of the implications of Theorem 2 (in Appendix E) I think now that the paper tells an interesting story and thus I will vote to accept.\n\n\n\n========================\n\nSummary:\n\nThe paper addresses the setting of meta-learning loss functions and in particular analyses the effect of the loss function on the entropy of the resulting learned function. In particular it shows that TaylorGLO learned functions tend to lead to higher entropic, and thus more regularized, neural networks, than when they are trained with the cross entropy loss.  The paper also discusses that the property of high entropy predictions can lead to better robustness against adversarial attacks.\n\n========================\n\nPros:\n\n- Well written and structured, and thus easy to follow. (With a few exceptions, but I think with a bit effort that can be fixed. See additional feedback)\n\n- Fairly unexplored but interesting setting.\n\n========================\n\nCons:\n\n- Some things are bit too informal, or not defined, see additional feedback.\n\n- In my opinion the results are not very strong. In particular the one shown in Table 1. The result from Theorem 2 is to me weak in the sense that by itself it does not give any intuition in what is important for a loss function to reduce entropy, and what is important for the magnitude of it.  (See also additional feedback). \n\n========================\n \nScoring:\n\nFor now I will vote for a weak accept, under the assumption that some of the smaller problems would be fixed for a final submission, see additional feedback. There you can also find what is missing for me for a stronger accept. I think the paper addresses an interesting and not much explored topic, and adds sufficient new insights to warrant a publication.\n\n========================\n\n========================\n\nAdditional feedback (along some questions.):\n\nRecommendation for smaller adjustments:\n\n- Third page first paragraph: ..'is important [for] the network's...'\n\n- Introduce somewhere the Baikal and TaylorGLO loss, those are not that known.\n\n- After Cross-Entropy analysis you refer to TaylorGLO's parameters a,b,c, which is at that point not introduced yet. Should somehow change the order.\n\n- Below Theorem 1, there is a bracket that never opens.\n\n- Theorem 2 is too informal. But in Theorem 2 you miss to introduce the gamma_T notation. For me both Theorem 1 and 2 rely too much on intuition, or are 'not attractors' and 'strength of entropy reduction' well-defined terms? (If so, then it should go to the appendix. As the intuition was at least clear to me, there is for me no strict need to change that though.)\n\n- Under Table 1, you say that you use Theorem 2 to calculate the strength of the bias, but Theorem 2 only holds for the case in where all non-targets receive the same probability or not?\n\n- Under Theorem 3, I don't understand why you need the inverse of the contraints to avoid non-usable loss functions.\n\n\nAdjustments that would increase my score:\n\n- Theorem 2 by itself is pretty void for me. I am missing that you draw some conclusions from it, at least for the loss functions you analyzed, in particular to analyze the magnitude of the bias.\n\n- The results in Table 1 are essentially the same after adding the invariant, the experiment is not convincing to me. I think you should create maybe even a toy example where you really can highlight the potential benefit of the invariant.\n\n- I do like the adversarial part. I would have found it very interesting to see how it compares to actually adversarially trained models. (I understand that this is not the point of the paper, but to me that would be still an interesting comparison)\n\n=====================\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for meta-learning the loss function. The analysis mainly concerns the recently proposed TaylorGLO method on the (slightly less recent) Baikal loss. There was no consensus on this paper, but no reviewer was willing to fight for acceptance either. I found the paper not self-contained, with important non-standard elements undefined, starting with the Baikal loss, notations that are not defined in the main text, and a nomenclature that is also unusual with important terms such as "attractor" or "invariant" used in meanings that are non-standard in optimization or machine learning.

Regarding content, most of the analyses refer to properties of the Baikal loss (not presented in the main text) that are deemed to be positive, without any theoretical support (Theorems 1 and 2). The inability to overfit is here posed as an obvious quality of a training loss. Then, a way to prevent the failure of the meta-training algorithm is presented in Theorem 3. Finally, an experiment is provided, showing that the proposed meta-training algorithm performs better than "vanilla" training with respect to adversarial attacks with FGSM. There is no comparison with other defense mechanisms and no analysis explains the results. Overall, although some interesting aspects may be developedin this paper, they are currently not well served by writing or the experimental evidences, so I recommend rejection.

Decision: Reject