Meta-Learning Adaptive Loss Functions
Authors: Christian Raymond
Qi Chen
Bing XUE
Mengjie Zhang
Venue: Accepted by TMLR ()

Abstract: Loss function learning is a new meta-learning paradigm that aims to automate the essential task of designing a loss function for a machine learning model. Existing techniques for loss function learning have shown promising results, often improving a model's training dynamics and final inference performance. However, a significant limitation of these techniques is that the loss functions are meta-learned in an offline fashion, where the meta-objective only considers the very first few steps of training, which is a significantly shorter time horizon than the one typically used for training deep neural networks. This causes significant bias towards loss functions that perform well at the very start of training but perform poorly at the end of training. To address this issue we propose a new loss function learning technique for adaptively updating the loss function online after each update to the base model parameters. The experimental results show that our proposed method consistently outperforms the cross-entropy loss and offline loss function learning techniques on a diverse range of neural network architectures and datasets.

Published in Transactions on Machine Learning Research (10/2025)
Meta-Learning Adaptive Loss Functions
Christian Raymond
christion.rrummonnnii www.............................
Victoria University of Wellington
Qi Chen
Victoria University of Wellington
Bing Xue Victoria University of Wellington
bing. Becs. www. ac.nz
Mengjie Zhang
mengjie.hhangSecs. www. ac.n
Victoria University of Wellington
Reviewed on OpenReview: https: // openreview. net/ forum? id= 0002
Abstract
Loss function learning is a new meta-learning paradigm that aims to automate the essential task of designing a loss function for machine learning model. Existing techniques for loss function learning have shown promising results, often improving model's training dynamics and final inference performance. However, significant limitation of these techniques is that the loss functions are mela-learned in an offline fashion, where the mela-objective only considers the very first few steps of training, which is a significantly shorter time horizon than the one typically used for training deop neural networks. This causes significant bias towards loss functions that perform well at the very start training but perform poorly at the end of training. To address this issue we propose new loss function learning technique for adaptively updating the loss function online after each update to the base model parameters. The experimental results show that proposed method consistently outperforms the cross- entropy loss and offline loss function learning techniques diverse range of neural network architectures and datascts.
1 Introduction
When applying deep neural networks to a given learning task, a significant amount of time is typically allocated lowards performing manual luning of the hyper-parameters to achieve competitive learning perfor- mances (Bengio, 2012). Selection of the appropriate inductive biases into the lcarning algorithm (Gordon & Desjardins, 1995). The inductive biases control both the set of searchable models and the learning rules used to find the final model parameters. Therefore, the field of meta-learning (Schmidhuber, 1987; Vanschoren, 2018; Peng, 2020; Hospedales et al., 2022), as well as the closely related field of hyper-parameter optimization (Bergstra et al., 2011; Feurer & Hutter, 2019), aim to automate the design and selection of a suitable set of inductive biases (or a subset of them) and have been oong-standing areas of interest the machine learning community.
parameers critical for embedding the relevant
One component that has only very recently been receiving attention in the meta-learning context the loss function. The loss function (Wang et al., 2022) is one of the most central components of any gradient- based supervised learning system, it determines the base learning algorithm's learning path and the selection of the final model (Reed & MarksII, 1999). Furthermore, in deep learning, neural networks are typically trained through the backpropagation of gradients that originate from the loss function (Rumelhart el al., 1986; Goodfellow et al., 2016). Given this importance, a new and emerging subfield of meta-learning referred as Loss Function Learning (Gonzalez Mikkkulainen, 2020b; Bechtle et al., 2021; Raymond et al., 2023; Collet al., 2022) aims attempt the difficult task of inferring a highly performant loss function directly from the given data.

---

Published in Transactions on Machine Learning Research (10/2025)
60
40-
Learned Loss
Learned Loss
100
-20-
0.6
0.6
Predicted Probability (y 1)
Predicted Probability (y
200
Lecrmed Loss
Learned Loss
100-
140-
-20-
120-
00
02
0.4
0.6
0.8
3.0
0.0
0.3
04
9.6
0.8
18
Predicted Probability
Predicted Probability
2000
4000
6000
8000
10000
Figure 1: Example adaptive meta-learned loss functions generated by AdaLFL on the CIFAR-10 dataset, where each row represents a classification loss function, and the color represents the current gradient step.
Loss function learning aims to meta-learn a task-specific loss function, which yields improved performance capabilitics when utilized in training compared to handcrafted loss functions. Initial approaches to loss function learning have shown promise at enhancing various aspects of deep neural network training, such as improving the convergence and sample efficiency (Gonzalez & Mikkkulaien, 2020b; Bechtle et al., 2021), as well as the generalization (Gonzalez Miikkulainen, 2021; Liu et al., 2020; Li et al., 2022; Leng et al., 2022), and model robustness (Gao et al., 2021; 2022). However, one prevailing limitation of the existing approaches to loss function learning is that they have thus far exclusively focused on learning loss function in the offline meta-learning settings.
In offline loss function learning, training is prototypically partitioned into two phases. In the first phase, the base loss function is meta-learned via iteratively updating the loss function by performing one or a few base training steps to approximate the performance. Second, the base model is trained using the learned loss function, which is now fized and is used in place of the conventional handcrafted loss function. Unfortunately, this methodology prone to severe short-horizon bias (Wu et al., 2018) towards loss functions which are performant in the early stages of training but often have poor performance in the later stages.
To address the limitation of offline loss function learning, we propose a new technique for online loss function learning called Adaptive Loss Function Learning (AdaLFL). In the proposed technique, the learned loss function is represented as a small food-forward neural network trained simultancously with the base learning model. Unlike prior methods, AdaLFL can adaptively transform both the shape and scale of the loss function throughout the learning process to adapt to what is required at each stage of the learning process, as shown in Figure 1. In offlin loss function learning, the central goal is to improve the performance of a model by specializing the loss function to a small set of related tasks. Online loss function learning naturally extends this general philosophy, specializing the loss function to each individual gradient step single task.
2

---

Published in Transactions on Machine Learning Research (10/2025)
1.1 Contributions
We introduce method for efficiently learning general-purposs adaptive loss functions using online meta- learning, where the loss function is updated after each base model update via one-step unrolled difler- entiation algorithm.
We identify and address shortcomings in the design of neural network-based loss function parameteriva- tions, which previously caused learned loss functions to be biased toward overly flat shapes resulting in poor training dynamics.
Empirically we show that models trained with our method exhibit faster convergence and improved inference performance compared to those trained with handcrafted or ofllne-learned loss functions.
Finally, we analyze the learned loss functions and uncover key insights, such as implicit meta-learning, which reveals why meta-learned loss functions consistently outperform traditional handcrafted losses.
2 Online Loss Function Learning
In this work, aim to automate the design and selection of the loss function and improve upon the performance of supervised machine learning systems. This is achieved via meta-learning an adaptive loss function that transforms both its shape and scale throughout the learning process. To achieve this, we propose Adaptive Loss Function Learning (AdaLFL), an efficient task and model-agnostic approach for online adaptation of the basc loss function.
2.1 Problem Setup
In prototypical supervised learning sctup, we are given a set of N independently and identically distributed (i.i.d.) examples of form = ((x1,))),... ..., (xN,,NN), yN)), where x, E is the ith instance's feature vector and yi E is its corresponding class label. We want to learn a mapping between X and Y using some base learning model, e.g., a classifier regressor, fo X + y, where 0 is the base model parameters. In this paper, similar to others (Finn et al., 2017; Bechtle et al., 2021), constrain the selection of the base models to those amenable to stochastic gradient descent (SGD) style training procedures such that optimization of the model parameters 0 occurs via optimizing some task-specific loss function LT with learning rule
0.11 = 0, aV, (V,
(1)
where Ly is a handcrafted loss function, typically the cross-entropy between the predicted label and the ground truth label in classification the squared error in regression. The principal goal of AdaLFL is to reduce the reliance of models on manually designed handcrafted loss functions such as LT, by instead using meta-learned adaptive loss function MMs where the meta-parameters
are learned simultaneously with the base parameters 0, allowing for online adaptation of the loss function. We formulate the task of learning & and & as non-stationary bi-level optimization problem, where t is the current time step
Φ++1 = arg min L (y, fere1 (x))
s.t. ( = min Ma. (v. fo, (x)).
The outer optimization aims to meta-learn a performant loss function Ma that minimizes the error on the given task. The inner optimization directly minimizes the learned loss value produced by MM to learn the base model parameters 0.
2.2 Loss Function Representation
In AdaLFL, the choice of loss function parameterization a small feedforward neural network, which is chosen due its high expressiveness and design flexibility. Our meta-learned loss function parameterization inspired by (Bechtle et al., 2021; Psaros et al., 2022) is small feedforward neural network denoted by to
3

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm Loss Function Initialization (Offline)
Input: Lr + Task loss function (meta-objective)
Mas Initialize parameters of meta learner
2: for &E (O...., Sinit 1} do
3: 0o + Reset parameters of base learner
4: for i (0, (0,...,Snnner - }} do
5: X, t Sample from Dtrain
Miearned t fo. (X))
0i41 + 0; - ao Mterradd end for
X, f Sample from Duatid
10: Ceask + CT fe..1 (X))
11: @1+1 t Oe - Vo, frask
12:
end for
with two hidden layers and 40 hidden units each, which is applied output-wise across outputs (making it invariant to the number of outputs).
& Lolvn fe(x),
(3)
Crucially, key design decisions are made regarding the activation functions used in ls to enforce desirable behavior. In Bechtle et al. (2021), ReLU activations are used in the hidden layers, and the smooth Softplus activation is used in the output layer to constrain the loss to be non-negative, é.e., to R2 - R Un- fortunately, this network architecture is prone to unintentionally encouraging overly flat loss functions, see Appendix B.1. Generally, flat regions in the loss function are very detrimental to training as uniform loss is given to non-uniform errors. Removal of the Softplus activation in the output can partially resolve this flatness issue; however, without it, the learned loss functions would violate the typical constraint that loss function should be at least CL, i.e., continuous in the zcroth and first derivatives.
Allernative smooth activations, such as Sigmoid, TanH, ELU, elc., can be used instead; however, due to their sange-bounded limits, they are also prone to encouraging loss functions that have large flat regions when their activations saturate. Thercfore, to inhibit this behavior, the unbounded Icaky ReLU (Maas al., 2013) is combined with the smooth ReLU, i.e., SoftPlus (Dugas et al., 2000)
Phidden (z) log(e +1) (1 - yz.
(4)
This smooth leaky ReLU activation function with leak parameter y and smoothness parameter 3 has desirable characteristics for representing a loss function. It is smooth and has linear asymptotic behavior necessary for tasks such regression, where extrapolation of the learned loss function can often oceur. Furthermore, as its output is not bounded when 0, it does not encourage flatness in the learned loss function. See Appendix B.2 and D.2 for more details.
2.3 Loss Function Initialization
One challenge for online loss function learning is achieving stable and performant initial set of parameters for the learned loss function. II Φ is initialized poorly, too much time is spent on fixing in the early stages of the learning process, resulting in poor base convergence, or in the worst case, fe to diverge. To address this, offline loss function learning using Meta-learning via Learned Loss (ML3) (Bechtle et al., 2021) is utilized to fine-tune the initial loss function to the base model prior to online loss function learning. The Algorithm 1, where Sint = 2500. In AdaLFL''s initialization process
initialization process is summarized one step on 0 is taken for each step in d, i.e., inner gradient steps Sinner 1. However, if Sinner 1, implicit differentiation (Lorraine et al., 2020: Gao et al., 2022) can instead be utilized to reduce the initialization process's 's memory and computational overhead.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 2 Loss Function Adaptation (Online)
Input: Ma f- Learned loss function (base-objective)
Input: LT f- Task loss function (mela-objective)
1: Po < Initialize parameters of base learner
2: for LE (0, - do
3: X, y t- Sample from Dtrain
4: Miearned t Mo, (v. fe, (X))
01-1 t- 8.- a ote.rnerreed
X, c Sample from Duatid
Lrask CT, f0r.1 (X))
8: #1+1 + Øc - 1/ di Frask
9: end for
2.4 Online Oetimiptininttion
To optimize o, unrolled differentiation is utilized in the outer loop to update the learned loss function after each update to the base model parameters the inner loop, which occurs via vanilla backpropagation, This is conceptually the simplest way to optimize as all the intermediate iterates generated by the optimizer in the inner loop can be stored and then backpropagate through in the outer loop (Maclaurin et al., 2015). The full iterative learning process is summarized in Algorithm 2 and proceeds follows: perform forward pass fo, (z) to obtain an initial set of predictions. The learned loss function Ma. is then used to produce a base loss value
Miearned= = (v. fx))
(5)
Using Miearned; the current weights 0, are updated by taking step in the opposite direction of the gradient of the loss with respect to the base model paramcters A,,
0++1 01 a a Vo. (v, (z))
(6)
0, a V2,Ex. [M. (y. fo, (z))].
where a the base learning rate. This can be further decomposed via the chain rule,
0.+1 0 - a y.Mo (v, fo.
(7)
Importantly, all intermodiate iterates generated by the base optimizer the 4th time-step when updating 0 are stored in memory. The mera-paaameters Φt can now be updated to Φ1-1 based on the learning progression made by 0. Using 0.+1 as function of Φt, compute forward pass using the updated base weights fe... (x) to obtain new set of predictions. The instances can either be sampled from the training set or held-out validation set. The new set of predictions is used to calculate the task loss Ly to optimize di through 0c11:
Liask CT,, fooz (x)).
(8)
where Ly is selected based on the respective application. For example, the squared error loss for the task of regression or the cross-entropy loss for classification. The task loss is a crucial component for embedding the end goal task into the learned loss function. Optimization of the current meta-loss network loss weights Φi now occurs by taking the gradient of the task loss,
dc+1 = Φ: - 1) V de LT (y, for. (x))
(9)
= Φ: - 7 V s. Ex CT fen (x))),
where 17 is the meta learning rate. The gradient computation is decomposed by applying the chain rule as shown in Equation (11) where the gradient with respect to the meta-loss network weights Φi requires the updated model parameters 6,11 from Equation (6).
d++1 = ti - ,,V, LT Soena V0,0441(án
(10)
or V LT,LVV Vo, Forn Vo. 10, ao,
(11)
5

---

Published in Transactions on Machine Learning Research (10/2025)
Outer Optimization
y
Inner Optimization
X
f&
Mo
fe..
LT
Meta Update
Rose Update
Figure 2: Computational graph of AdaLFL, where 0 is updated using Ma. in the inner loop (Base Update). The optimization path tracked in the computational graph and then used to update & based on the meta-objective in the outer loop (Meta Update). The dashed lines show the gradients for and & with respect to their given objectives.
This process is repeated for fixed number of gradient steps Strain, which is identical to what would typically be used for training fa. An overview and summary of the full associated data flow between the inner and outer optimization of and Φ, respectively, is given in Figure 2.
3 Related Work
The method that we propose in this paper addresses the general problem of meta-learning a (base) loss func- tion, i.e. loss function learning. Existing loss function learning methods can be categorized along two kcy axes, loss function representation, and meta-oplimization. Frequently used representations learning include parametric (Gonzalez & Miikkulainen, 2020b; Raymond al., 2023) and nonparametric (Liu al., 2020; Li al., 2022) genetic programming expression trees. In addition this, alternative reprosentations such truncated Taylor polynomials (Gonzalez & Miikkulainen, 2021; Gao et al., 2021; 2022) and small feed-forward neural networks (Bechtle et al., 2021) has also been recently explored. Re- garding meta-optimization loss function learning methods have heavily utilized computationally expensive evolution-based methods such as evolutionary algorithus (Koza, 1994) and evolutionary strategies (Hansen & Ostermeier, 2001). While more recent approaches have made use of gradient based approaches unrolled differentiation (Maclaurin al., 2015), and implicit differentiation (Lorraine et al., 2020).
loss function
A common trait among these methods is that, contrast to AdaLFL, they perform offlie loss function learning, resulting in severe short-horizon bias towards loss functions which are performant in the early stages of training but often have sub-optimal performance at the end of training. This short-horizon bias arises from how the various approaches compute their respective meta-objectives. In offline evolution-based approaches, the fitness, i..., meta-objective, partial training session, e.g., I 1000 gradient steps (Gonzalez & Mikkulainen, 2021; Raymond et al., 2023). A truncated number of gradient steps are required to be used as evolution- based methods evaluate the performance of large number of candidate solutions, typically L. loss function over K iterations where 25 L, K < 100. Therefore, performing full training sessions, which can be hundreds, thousands, or even millions of gradient steps for each candidate solution, is infeasible.
calculated by computing the performance at the end of a
Regarding the existing gradient- based approaches, offline unrolled optimization requires the whole optimiza- tion path to be stored in memory; in practice, this significantly restricts the number of inner gradient stops before computing the meta-objective to only a small number of steps. Methods such as implicit differentia- tion can obviate these memory issues; however, it would still require a full training session in the inner loop, which is a prohibitive number of forward passes to perform in tractable time. Furthermore, the dependence of the model parameters the meta-pane portaparrseters increasingly shrinks and eventually vanishes the number of steps increases (Rajeswaran et al., 2019).

---

Published in Transactions on Machine Learning Research (10/2025)
3.1 Online vs Offline Loss Function Learning
The key algorithmic difference of AdaLFL from prior oflline gradient-based methods (Bechtle et al., 2021; Gao et al., 2022) is that & is updated after each update to 0 in lockstep in a single phase as opposed to learning f and ø in separate phases. This is achieved by not resetting 0 after cach update to ø (Algorithm 1, line 3), and consequently, & has to adapt to each newly updated timestep such that ø o= (@u. 01...., S In offline loss function learning, & is learned separately at meta-training time and then fixed for the full duration of the meta-testing phase where 0 is learned and ø (do). Another crucial difference is that in online loss function learning, there is an implicit meta-learning of the learning rate schedule and built in early stopping mechanism, further discussed in Section 5.5.
4 Experimental Evaluation
In this section, the experimental setup for evaluating AdaLFL is presented. In summary experiments are conducted across seven open-access datasets and multiple well-cstablished network architectures. The per- formance of AdaLFL is assessed against three benchmark methods. All experiments are implemented in PyTorch (Paszke et al., 2017) and Higher (Grefenstette et al., 2019), and the code is available at 1 Further experimental details and results, including ablations on the loss function representation, are provided in Appendix C and D, respectively.
4.1 Benchmark Methods
AdaLFL is compared to three benchmark methods. The first a baseline, which uses conventional hand- crafted losses, namely the mean squared error and cross-entropy loss. Next, is MI. (Bechtle et al., 2021) the offline counterpart of AdaLFL which meta-learns the loss function offline and does not adapt it during train- ing. Finally, MetaLR, an equivalent algorithm for mela-learning a single scalar adaptive learning rate, which adjusts the learning rate online during training identical to AdaLFL, see Appendix C.1 for more details. between MetaLR, ML3 and AdaLFL has been standardizod to
Importantly, the choice of enable as fair of a comparison as possible.
4.2 Benchmark Tasks
Following the established literature on loss function learning, the regression dalasets Communities and Crime (Redmond, 2009), Diabetes (Efron et al., 2004), and California Housing (Pace & Barry, 1997) are used as a simple domain to illustrate the capabilities of the proposed method. Following this classification datascts MNIST (LeCun et al., 1998), CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009), and SVHN (Netzer et al., 2011), are employed to assess the performance of AdaLFI. to determine whether the results can generalize larger, more challenging tasks. The original training-testing partitioning is used for all datasets, with 10% of the training instances allocated for validation. In addition, standard data augmentation techniques consisting of normalization, random horizontal flips, and cropping are applied to the training data of CIFAR- 10, CIFAR-100, and SVHN during meta and base training.
4.3 Benchmark Models
A diverse set of well-established benchmark architectures utilized to evaluate the performance of AdaLFL. For Communities and Crime, Diabctes, and California Housing two hidden layer multi-layer perceptron (MLP) taken from (Baydin et al., 2018) is used. For MNIST, logistic regression (McCullagh et al., 1989), the previously mentioned MI.P and the LcNet-5 (LeCun et al., 1998) architecture is used. Following this experiments are conducted on CIFAR-10, VGG-16 (Simonyan & Zisserman, 2015), ALICNN-C (Springenberg et al., 2014), ResNet-18 (He et al., 2016), and SqueezeNet (landola et al., 2016) are used. For the remaining datasets, CIFAR-100 and SVHN, WideResNet 28-10 and NiteResNet 16-8 (Zagoruyko & Komodakis, 2016) are employed.
GitHub Repository:
7

---

Published in Transactions on Machine Learning Research (10/2025)
16"
10
Log Error Rate
30
Log Error Rate
10"
107
Log
10"
10
10
10
2000
5000
10000
4000
6000
10000
2000
5000
10000
(a) Crime + MLP
(b) Diabetes + MLP
California + MLP
Log Error Rate
Log Error Rate
10"
107
Log Error
107-
10'"
20"
10"
(d) MNIST + Logistic
MNIST + MLP
(f) MNIST LeNet-5
Log Squared Brror
10
10.
Log Squared Error
10²
Log Eiror Rate
10"
3077
10"d
30
10
30
10"
(g) CIFAR-10 VGG-16
(h) CIFAR-10 AHICNN-C
(i) CIFAR-10 RReNNet-18
Log Squared Brror
10
16.
Log Eiror Rate
Log Error Rate
10"
10"
200
10"
CIFAR-10 SqueezeNet
(k) CIFAR-100 WRN 28-10
(I) SVHN WRN 16-8
Bodix
ML. (O.lnne)
MeaLR (Ohline)
AdaLFL (Online)
Figure 3: Mean learning curves across 10 independent executions of each algorithm on each task + model pair, showing the log of the training mcan squared error error rate (y-axis) against gradient steps (x-axis). Best viewed in color.
5 Results and Analysis
The results in Figure 3 show the average (log) training learning curves AdaLFL compared with the bascline, MctaLR, and ML3, across 10 exccutions of each datasct-model pair. Overall, AdaLFL demonstrates consistent improvements in convergence speed compared to the baseline and ML,3 These gains evident across wide range of tasks and architectures, with the exception of the regression datasets, which is due to regularization bchavior as shown in Appendix D.4. Importantly, the errors obtained by AdaLFL at the end of training are typically lower than those of all compared methods, indicating that AdaLFL delivers both faster convergence and stronger overall performance.
On the more challenging tasks of CIFAR-10, CIFAR-100, and SVHN, AdaLFL improves noticeably over the baseline and 3L Improved performance on these datasets is achieved through AdaLFL's adaptive updating of the learned loss function throughout the learning process, which adapts to changes in the training dynamics. This is in contrast to ML3, where the loss function remains static, resulting in poor performance on tasks where the training dynamics at the beginning of training vary significantly from those at the end of training.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Results reporting the mean + standard deviation final inference testing mean squared error or error rate across 10 independent executions of each algorithm on each task + model pair (using no base learning rate scheduler).
Task
Model
Bascline
ML (Offline)
Meta-LR, (Online)
AdaLFL (Online)
Crime
MLP
0.0274#0.0017
0.0270=0000055
0.0274=0.0018
0.0263+0.0023
Diabetes
MLP
0.0432++0.003
0.0430+000012
0.0463+0.0013
0.0420+0.0014
California
MLP
0.0157:100001 0.0276:100058
0.015420.0004
0.015110.0007
MNIST
Logistic 2
0.0766+000009
0.0710=0000100
0.0756=0.0008
0.0697=0.0010
MLP
0.02031000006
0.0185+0.0004
0.0192=0.0007
0.0184=0.0006
LeNet-5
0.0125=0.0007
0.0094+0.0005
0.0097=0.0013
0.0091#0.0004
CIFAR-10
VGG-16
0.1036+000009
0.1024+000055
0.0966=0.0087
0.0903+0.0033
AIICN-C 5
0.1030+0.0062
0.1015+
0.0672+0.0068
0.0835+0.0000
ResNet-18
0.0007
0.0883:L0.0041
0.0866
SqueezeNet
0.1226::00000
0.1162::0.0022
0.1173=0.0065
0.1083=0.0049
CIFAR-100
WRN 28-10
0.3046+000087
0.3108:5000055
0.2288=0.0019
0.266810.0283
SVHN
WRN 16-8
0.0512+0.0043
0.0500#0.0034
0.0367=0.0007
0.0441#0.0014
(Baydin et al., 2018) 2 (McCullagh al., 1989)
3 (LeCun et al., 1908)
(Simonyan & Zisserman, 2015)
5 (Springenberg et al., 2014) 6 (He et al., 2016)
7 (Tandola et al., 2016)
(Zagoruyko & Komodakis, 2016)
Compared to MetaLR, AdaLFL generally achieves superior performance across tasks. This suggests that meta-learned loss functions capture nuanced iosta-information about the training process that cannot be fully represented by meta-learning only the base learning rate, consistent. with findings in (Raymond et al., 2023). Notably, on few specific datasets: CIFAR-10 AIICNN-C, CIFAR-100 WRN28-10. 10, and SVHN WRN16- 8, MetaLR achieves slightly better performance. We hypothesize that this is due to the fixed learning rate settings adopted from prior works (Gonzalez & Mikkullinen, 2021; Raymond et al., 2023) being sub-optimal in these cases, making explicit meta-learning of the learning rate more effective.
5.1 Final Inference Testing Performance
Table 1 reporting the average mean squared error or eтror rate across 10 independent executions of each method. Across all tested problems, AdaLFL consistently outperform the baseline, these improvements are both substantial and stable, as reflected by the relatively small standard deviations across runs, indicating lower variability. Even though the models were originally designed and optimized around bascline loss functions, AdaLFL demonstrates clear and consistent gains across tasks. This result highlights the gains of using an adaptive meta-learned loss function compared to the typically used handcrafted loss functions.
Compared to MI,3, AdaLFL achieves better performance on all datascts. The improvements on MNIST are relatively modest; this suggests that the training dynamics on MNIST at the beginning of training are similar to those at the end; hence there are minimal gains from adapting the loss function online. While on more challenging datasets such as CIFAR-10, CIFAR-100, and SVHN, the gains are considerably larger, demonstrating the scalability and robustness of the approach. Regarding MctaLR, AdaLFI achieves better performance on most. lasks, with the exception of CIFAR-10 ALICNN-C, CIFAR-100, and SVHN. This is likely due to the sub-optimal learning rates used these tasks. Notably, MetaLR is also shown to outperform MLL, this further highlights the importance of adaptation on large complex learning tasks.
The results attained by AdaLFL are promising given that the models tested were designed and optimized around the baseline loss functions. Larger performance gains may be attained using models designed specifi- cally around meta-learned loss function (Kim et al., 2018; Elsken et al., 2020; Ding et al., 2022). Thus future work will explore learning the loss function in tandem with the network architecture
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Average run-time of the entire learning process (end-to-end)) for each beuchmark method. Each algorithm is run on single Nvidia RTX A5000, and results are reported in hours.
Task and Model
Baseline
ML
MetaLR
AdaLFL
Crime ML.
0.01
0.03
0.05
0.05
Diabetes MLP
0.01
0.03
0.05
0.05
California + MLP
0.01
0.03
0.05
0.05
MNIST + Logistic
0.06
0.31
0.53
0.55
MNIST + MLP
0.06
0.32
0.53
0.56
MNIST + LeNet-5
0.10
0.38
0.64
0.67
CIFAR-10 VGG-16
1.50
1.85
5.46
5.56
CIFAR-10 ALCCNN-C
1.41
1.72
5.39
5.53
CIFAR-10 + ResNet-18
1.81
2.18
8.21
8.38
CIFAR-10 + SqueozeNet
1.72
2.02
7.84
7.88
CIFAR-100 + WRN 28-10
8.81
10.3
48.71
50.49
SVHN + WRN 16-8
7.32
7.61
23.33
24.75
5.2 Run-time Analysis
Table 2 reports the average run-time of all benchmark methods on all tasks, including the time to initialize the learning rate and loss function for MctaLR and AdaLFL, respectively. Notably, there are three key reasons why the computational overhead of AdaLFL is not as bad as may at first seem. First, the baseline time excludes the implicit cost of manual and learning rate schodule required to achieve reasonable performance (Goodfellow most the computational overhead of AdaLFL (and MetaLR) comes from storing the one-step optimization trajectory and backpropogating through it to update Φ. Importantly, this is identical computation to those used in other meta-learning paradigms (Andrychowicz et al., 2016; Finn et al., 2017). Consequently, when combined with optimization-bant based meta-learning methods the computational overhead would be amortized, since the intermediate iterates generated by those algorithms can be reused (Li el al., 2017; Park Oliva, 2019; Flennerhag et al., 2020; Baik et al., 2020; 2021). Finally, we take one meta-gradient step on & per base step on 0, but meta steps could instead occur periodically, e.g,, every 100 1000 steps, reduce the computational and memory overhead. As shown in Figures 9-18, the meta-learned loss functions interpolate very smoothly between their initial and final states, and in many cases converges well before the base model has finished training.
tuning for the loss function, initial learning rate,
al., 2016). Socond,
5.3 Inner Gradient Steps
Table 3: Results reporting the mean + standard deviation testing error rates when using an incrcasing number of inner gradient steps with 3L
In 3 (Bechtle et al., 2021) suggested taking only one inner step, i.e., setting Sinner = =1 in Algorithm 1. A reasonable question to ask is whether increas- ing the number of inner steps to extend the hori- zon of the mela-objective past the first step will re- duce the disparity in performance between 3L and AdaLFL. To answer this question, experiments are performed on CIFAR-10 ALICNN-C with 3 set- ting Sinner (1,5, 10, 15, 20}. The results reported in Table 3 show that increasing the number of inner steps in 3L up to the limit of what feasible in memory on a consumer GPU does not resolve the
Method
CIFAR- 10 + AIICNN-C
ML, (Sinner = 1)
0.1015+0.0055
ML (Sinner = 5)
0.0978+0.0052
ML (Smnne 10)
0.0985+0.0050
3L (Sinner = 15)
0.098910.0049
3L (Smnnr = 20)
0.0974+0.0061
AdaLFL (Online)
0.0835+0.0050
10

---

Published in Transactions on Machine Learning Research (10/2025)
17.5
00-
15.0
-2.
Leamed Loss
Leamed Loss
10.6
7.5
5.-
12.
2.5
0.0
0.0
64
0.5
0.0
Predicted Probability (y 1)
Predicted Probubility (y 0)
16
20-
20-
10
Learned
-10-
1.0
Predicted Probability (y 1)
Predicted Probability (y - 0)
30-
Learned Loss
06 Predicted Probability (y
Predicted Probability (y 1)
20000
40000
60000
80000
100000
Figure 4: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where each row represents a loss function and the color represents the current gradient step.
short horizon bias present in offline loss function learning. Furthermore, the results show that increasing the number of inner steps only results in marginal improvements the performance over Sinner = Honce, oflline learning methods that seek to obviate the memory issues of unrolled differentiation allow for an increased number of inner steps, such as (Gao al., 2022), which uses implicit differentiation, are still prone to kind of short-horizon bias.
5.4 Visualizing Learned Loss Functions
To better understand why the meta-learned loss functions producod by AdaLFL are so performant, two of the learned loss functions are highlighted in Figure 4, where the learned loss function is plotted equispaced intervals throughout the training. See Appendix D.6 for further examples of the diverse and creative loss function meta-learned by AdaLFL.
Analyzing the learned loss functions, it can be observed that the loss functions change significantly in their shape throughout the learning process. both cases, the learned loss functions attributed strong penalties for severe misclassification at the start of the learning process, and then gradually pivoted more moderate or minor penalty as learning progressed. This behavior enables fast and efficient learning early on and reduces the sensitivity of the base model to outliers in the later stages of the learning process.
11

---

Published in Transactions on Machine Learning Research (10/2025)
5.5 Implicit Tuning of Learning Rate Schedule
In oflline loss function learning, it is known from (Gonzalez & Miikkulainen, 2021; Raymond et al., 2023) that there is implicit initial learning rate tuning of a when meta-learning a loss function since 3a3ø V-aVVeC x 0 VoMg. Consequently, an emergent behavior, unique to online loss function learning, is that the adaptive loss function generated by AdaLFL implicitly embodies multiple different learning rates throughout the learning process hence often causing a fine-tuning of the fixed learning rate or of predetermined learning rate schedule. Analyzing the learned loss functions in Figure 4, it can be observed that the scale of the learned loss function changes, confirming that implicit learning rate schedulligg is occurring over time. In Appendix C.1 additional experiments are given comparing the performance of AdaLFF, to a method for meta-learning the base learning rate.
5.6 Implicit Early Stopping Regularization
A unique property observed in the loss functions generated by AdaLFL is that often once base convergence is achieved the learned loss function will intentionally start to flatten or take on a parabolic form, as shown in Figure 4 (second row). This is implicitly type of early stopping, also observed in related paradigms such as in hypergradient descent (Baydin et al., 2018), which meta-lcarns base learning rates. In hypergradient descent the learned learning rate has been observed to oscillate around 0 near the end of training, at times becoming negative, essentially terminating training. Implicit early stopping is beneficial as it known have regularizing effect model training (Yao et al., 2007); however, if not performed carefully it also be detrimental to training due to terminating training prematurely. Therefore, in future work, we aim to further investigate regulating this behavior, a potential avenue for further improving performance.
5.7 Implicit Label Smoothing Regularization
Another novel observation make is that in many of the classification loss functions, such as in Figure 4 (third row), the target loss initially decreases the model becomes more confident in its predictions. However, counterintuitivel, as the predicted probability approaches 1 (i.e., the model becomes very confi- dent), the loss begins to increase. This behavior has been previously observed and theorelically studied in (Gonzalez & Mikkkulainen, 2020a; Raymond et al., 2023), where it was shown to resemble a form of label smoothing regularization. This later inspired a new regularization method called sparse label smoothing regularization (Raymond, 2024). In such cases, the loss function penalizes overconfident predictions, thereby encouraging better generalization. Importantly, this is the first time this phenomenon has been observed in neural network parameterizations of the loss functions. Moreover, because AdaLFL learns the loss function adaptively, unlike prior methods that learn static loss functions, it implicitly meta-learns an adaptive form of label smoothing regularization which is dynamically adjusted throughout the learning process.
6 Conclusion
In this work, the first fully online approach to loss function learning has been proposed. The proposed technique, Adaptive Loss Punction Learning (AdaLFL), infers the base loss function directly from the data and adaptively trains it with the base model parameters simultaneously using unrolled differentiation. The results showed that models trained with our method have enhanced convergence capabilities and inference performance compared with the de facto standard mean squared error and cross-entropy loss, and offline loss function learning method ML3. Further analysis of the learned loss functions identified common patterns in the shape of the learned loss function, as well as revealed unique emergent behavior present only in adaptively learned loss functions. Namely, implicit tuning of the learning rate schedule, carly stopping regularization, and an adaplive form of label smoothing regulization. While this work has solely set focus on meta-learning the loss function in isolation to better understand and analyze its properties, we believe that further benefits can be realized upon being combined with existing optimization-ul based meta- learning techniques.
12

---

Published in Transactions on Machine Learning Research (10/2025)
References
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to Learn by Gradient Descent by Gradient Descent. In Advances in Newral Information Processing Systems (NeurIPS), 2016.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to Train Your MAML. International Con- ference on Learning Representations (ICLR), 2019.
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta-Tearning with Adaptive Hyperparameters. Advonces in Neural Information Processing Systems (NewrIPS), 2020.
Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee. with Task-Adaptive Loss Function for Few-Shot Learning. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Jonathan T Barron. A General and Adaptive Robust Loss Function. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Atilim Güneş Baydin, Robert Cornish, David Martínez Rubio, Mark Schmidt, and Frank Wood. Online Learning Rate Adaptation with Hypergradient Descent. In International Conference Learning Repre- sentations (ICLR), 2018.
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta-Learning via Learned Loss. In International Conference on Pattern Recognition (ICPR). IEEE, 2021.
Yoshua Bengio. Practical Recommendations for Gradient-Based Training of Deep Architectures. In Neural Networks: Tricks of the Trade. Springer, 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for mization. Advances in Neural Information Processing Systems (NeurIPS), 2011.
Opti-
Alan Collet, Antonio Bazco- NoguueN, Albert Banchs, and Marco Fiore. Loss Meta-Learning for Forecasting, 2022.
Yadong Ding, Yu Wu, Chengyue Huang, Siliang Tang, Yi Yang, Longhui Wei, Yueting Zhuang, and Qi Tian. Learning to Learn by Jointly Optimizing Neural Architecture and Weights, In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Charles Dugas, Yoshma Bengio, François Bélisle, Clande Nadeau, and René Garcia. Incorporating Second- Order Functional Knowledge for Better Option Pricing. Advances Neural Information Processing Sys- tems (NeurIPS), 2000.
Bradley Efron, Trevor Hastie, lain Johnstone, and Robert Tibshirani. Least Angle Regression. 2004.
Thomas Elsken, Benedikt Stafler,, Jan Hendrik Metzen, and Frank Hutter. Meta-Learning of Neural Archi- tectures for Few-Shot Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Matthias Feurer and Frank Hutter. Hyperparameter Optimization. In Automated Machine Learning. Springer, 2019.
Chelsca Finn, Picter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In International Conference on Machine Leurning (ICML), 2017.
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-Leerning with Warped Gradient Descent. International Conference on Learning Representations (ICLR), 2020.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Boyan Gao, Henry Gouk, and Timothy M Hospedales. Searching for Robustness: Loss Learning for Noisy Classification Tasks. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Boyan Gao, Henry Gouk, Yongxin Yang, and Timothy Hospedales. Loss Function Learning for Domain Generalization by Implicit Gradient. In International Conference on Machine Learning (TCMT.), 2022.
Santiago Gonzalez and Risto Mikkullanen. Effective Regularization through Loss Function Meta-Learning, arXiv preprint xrXi::2010000788, 2020a.
Santiago Gonzalez and Risto Miikkulainen. Improved Training Speed, Accuracy, and Data Utllization Through Loss Function Optimization. In IEEE Congress on Evolutionary Computation (CEC), 2020b.
Santiago Gonzalez and Risto Mikkullaieen. Optimizing Loss Functions through Multi- Variate Taylor Poly- nomial Parameterization. In ACM Genetic and Evolutionary Computation Conference (GECCO), 2021.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016
Diana F' Gordon and Marie Desjardins. Evaluation and Selection of Biases in Machine Learning. Machine Learnång, 1995.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized Inner Loop Meta-Learning. arXiv preprint ArX:::1910.0I1 017277 2019.
Nikolaus Hansen and Andreas Ostermeier. Completely Derandomized Self-Adaptation in Evolution Strate- gies. Evolutionary Computation, 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-Learning in Neural Net- works: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
Forrest N landola, Song Han, Matthew W Moskewiez, Khalid Ashraf, William J Dally, and Kurt Keutzer, SqueezeNet: AlexNel-Level Accuracy with 50x Fewer Parameters and S 0.5MB Model Size. arXiv preprinl arXiv:1602.0733 07360, 2016.
Jaehong Kim, Sangyeul Lee, Sungwan Kim, Moonsu Cha, Jung Kwon Lee, Youngduck Choi, Yongseok Choi, Dong- Yoog-Yeon Cho, and Jiwon Kim. Auto-Meta: Automated Gradient-Based Meta Learner Search. arXiv preprint ar Xix: 06927, 2018
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. International Conference on Learning Represenlations (ICLR), 2015.
John R Koza. Genetic Programming II. MIT press Cambridge, 1994.
Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, Universily of Toronto, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 1998.
Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Xiaojie Shi, Shuyang Cheng, and Dragomir Anguelov. PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions. International Conference on Learning Representations (ICLR), 2022.
Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. AM-LFS: AutoMI. for Loss Function Search. In IEEE/CVF International Conference Computer Vision (ICCV), 2019.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu. AutoLoss- Zero: Searching Loss Functions from Scratch for Generic Tasks. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning. arXiv preprint Xi0:1107709888 2017.
Peidong Liu, Gengwei Zhang, Bochao Wang, Hang Xu, Xiaodan Liang, Yong Jiang, and Zhenguo Li. Loss Function Discovery for Object Detection via Convergence-Simulln Simulation Driven Search. In International Conference Learning Represendations (ICLR), 2020.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing Millions of Hyperparameters by Implicit Differentiation. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Andrew I. Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier Nonlinearities Improve Neural Network Acoustic Models. In International Conference on Machine Learning (ICML), 2013.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-Based Hyperparameter Optimization through Reversible Learning. In International Conference Machine Learning (ICML), 2015.
Peter McCullagh, John A Nelder, et al. Generalized Linear Models. Routledge, 1989.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. Advances in Neural Information Processing Systems (NeurIPS), 2011.
R. Kelley Pace and Ronald Barry. Sparse Spatial Antoregrossions. Statistics and Probability Letters, 1997.
Eunbyung Park and Junier B Oliva. Meta-Curvature. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic Differentiation in Pytorch. 2017.
Huimin Peng. A Comprehensive Overview and Survey of Recent Advances in Meta-Learning, arXiv preprinl 0rXiv:2004.11149 2020.
Apostolos F Psaros, Kenji Kawaguchi, and George Em Karniadakis. Meta-Learning PINN Loss Functions. Journal of Computational Physics, 2022.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-Learning with Implicit Gradients. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Christian Raymond. Meta-Learning Loss Functions for Deep Neural Networks. arXiv preprint
arXiv:2406.09775 097113, 2024.
Christian Raymond, Qi Chen, Bing Xue, and Mengjie Zhang. Learning Symbolic Model-Agnostic Loss Functions via Mela-Learning, IEEE Transactions Pallern Analysis und Machine Intelligence (TPAMI), 2023.
Michael Redmond. Communtiies and Crime, 2009.
Russell Reed and Robert J MarkslI. Neural Smithing: Supervised Leurning in Feedforward Arlificial Neural Networks. Mit Press, 1999.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning Representations by Back- propagating Errors. Nature, 1986.
Jürgen Schmidhuber. Evolutionary Principles in Reff-Refeeential Learning. PhD thesis, Technical University Munich, 1987.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recog- nition. International Conference on Learning Represenlations (TCLR), 2015.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for Simplicity:
The All Convolutional Net. International Conference on Learning Representations (TCLRR, 2014.
Joaquin Vanschoren. Meta-Learning: Survey. ar preprint 1xxx::810.00544 2018.
Qi Wang, Yue Ma, Kun Zhao, and Yingjie Tian. A Comprehensive Survey of Loss Functions in Machine Learning. Annals of Data Science, 2022.
Xiaobo Wang, Shuo Wang, Cheng Chi, Shifeng Zhang, and Tao Mei. Loss Function Search for Face Recog- nition. In International Conference on Machine Learning (ICML), 2020.
Yuhuai WI, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding Short-Horizon Bias in Stochastic Meta-Optimization. International Conference on Learning Representations (ICLR), 2018.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On Early Stopping in Gradient Descent Learning. Constructive Approzimation, 2007.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. arviv preprint orXiv:6605.0714, 2016.
16

---

Published in Transactions on Machine Learning Research (10/2025)
A Extended Background
300
Learned Loss
Learned Loss
200
0.6
04
0.6
Predicted Probability (y 1)
Predicted Probability (y
Learned Loss
Learned Loss
400
200
100
0.0
02
04
0.6
0.8
1.0
00
02
044
06
0.
14
Predicted Probahility (y
Prodicted Probability (y
20000
40000
60000
80000
100000
Figure Four example loss functions generated by AdaLFL using the network architecture proposed in (Bechtle et al., 2021), which uses softplus activation in the output layer, cansing flattening behavior degrading learning performance.
A.1 Online vs Offline Loss Function Learning
The key algorithmic difference of AdaLFL from prior offline gradient-based methods (Bechtle et al., 2021; Gao et al., 2022) that & updated after each update to lockstep in a single phase as opposed learning 0 and é in separate phases. This achieved by not resetting 0 after each update to & (Algorithm 1, line 3), and consequently, & has to adapt to each newly updated timestep such that ø (Øo, $1,..., & ) In ofllne loss function learning, o is learned separately at mela-training time and then is fixed for the full duration of the meta-testing phase where 0 is learned and ó = (do). Another crucial difference is that in online loss function learning, there is an implicit tuning of the learning rate schedule and implicit early stopping regularization, as discussed in Section 5.5 and 5.6, respectively.
A.2 Alternative Paradigms
Although online loss function learning has not been explored in the meta-learning context, some existing research outside the subfield has previously explored the possibility of adaptive loss functions (Barron, 2019; Li et al., 2019; Wang et al., 2020). However, we emphasize that these approaches are categorically different in that they do not learn the loss function from scratch; instead, they interpolate between small subset of handcrafted loss functions, updating the loss function after each epoch. Furthermore, in contrast. to loss function learning which is both task and model-agnostic, these techniques restricted to being task-specific, e.g., face recognition only. Finally, this class of approaches does not implicitly tune the base learning rate as is the case in loss function learning.
17

---

Published in Transactions on Machine Learning Research (10/2025)
B Loss Function Representation
The reprosentation of the Icarned loss function under considcration in AdaLFL is a simple foed-forward neural network. We consider the general case of a feed-forward neural network with one input layer, L hidden layers, and one output layer. A hidden layer refers to a feed-forward mapping between two adjacent layers hale such that
hacs = (o))))))))))
(12)
where v(.)()) refers to the element-wise activation function of the 1sh layer, and x00) is the matrix of inter- connecting weights between hat--) and hg(0) For the input layer, the mapping is defined as hgo (yt, fo)))) and for the output layer as Rigooes (hoces). Subsequently, the meta-lcarned loss function to parameterized by the set of meta- meta-parametent such that
fou..... Φ1, Dout} can be defined composition of feed-forward mappings
(o = hasowe) (Mores) (... (wi, (v.
(13)
which is applied output-wise across the C output channels of the ground truth and predicted labels, e.g.. applied to each index of the one-hot encoded class vector in classification, to each continuous output in regression. The loss value produced by to is then summed across the output channel to reduce the loss vector into its final scalar form
Moly.f(x))
2 E (o. fo(z)))
(14)
==0
B.1 Network Architecture
The learned loss function used in our experiments has == 2 hidden layers and 40 hidden units in each layer, inspired by the network configuration utilized in Mela-Tearning via Learned Loss (MI.3 Supervised) (Bechtle et al., 2021). We found no consistent improvement in performance across our experiments by incrcasing or decreasing the number of hidden layers or nodes. However, it was found that the choice of non-linear activations used in ML3, was highly prone to encouraging poor- poor-performing loss functions with large flat regions, as shown Figure
In ML rectified linear units, (x)) max(0, x)), are used in the hidden layers and the smooth SoftPlus Psoftplus = log(e + 1) is used in the output layer to enforce the optional constraint that Miearned should be non-negative, i.e., VyNfo(=)M (v. fa(x)) 2 O. An adverse side-cffect of using the softplus activation in the output is that all negative imputs to the output layer go to O, resulting in flat regions in the learned loss. Furthermore, removal of the output activation does not resolve this issue, as ReL.U, as well as other common activations such as Sigmoid, TanH, and ELU, are also bounded and are prone to causing flatness when their activations saturate, a common occurrence when taking gradients through long unrolled optimization paths (Antoniou al., 2019).
B.2 Smooth Leaky ReLU
To inhibit the flattening behavior of learned loss functions, a range unbounded activation function should be used. A popular activation function that is unbounded (when the leak parameter y < 0) is the Leaky ReLU (Maas et al., 2013)
(1e)kky()) max(r =.,a) max(0, (1 - 7) +
(15)
(16)
However, it is typically assumed that loss function should be at least Cl i.e., continuous in the zeroth and first derivatives. Fortunately, there is smooth approximation to the ReLU, commonly referred to as the SoftPlus activation function (Dugas et al., 2000), where 3 (typically set to 1) controls the smoothness.
Ssmooth (I) = llogee +11 1)
(17)
18

---

Published in Transactions on Machine Learning Research (10/2025)
1.0
0.8
0.6
y=
0-0.0
y= 0.25
0.4
y- 0.25
0.5
2
0.75
V-10
0.0
(a)
(b)
1.0
8 1
8
2
0.9
8
8 3
8
8 4
0.8
5
8
0.6
-2
0.5
(c)
(d)
ReLU
Leaky ReLU
Smooth ReLU
2
Smooth Leaky ReL.U
-2
(c)
Figure 6: The proposed activation function and its corresponding derivatives when shifting are shown in (a) and (b), respectively. In (c) and (d) the activation function and its derivatives when shifting 3 are shown. Finally, (c), the smooth leaky ReLU contrasted with the original smooth and leaky variants ReLU.
19

---

Published in Transactions on Machine Learning Research (10/2025)
The leaky ReLU is combined with the smooth ReLU by taking the term maz(0,2) from Equation (16) and
substituting it with the smooth SoftPlus defined in Equation (17) to construct a smooth approximation to the leaky ReLU,
Phidden (z) = 1 log(e + +1) (1 - 7) + ye,
(18)
B
where the derivative of the smooth leaky ReI.U with respect to the input a is
Phidden (x) d [log(e8r
8r + 1) (1 y) 70
(19)
dr d de
B
llog(e0x
+ -1]] (1- -1)
+7
(20)
- 1| (1 -1
+7
(21)
B e +1
e0z, 3. (1 y)
+ +
(22)
B.e8z +1
e (1 7)
+7
(23)
00v +1
caa (1 7)
He +1)
(24)
@ +1
+1
an +7
(25)
e +1
The smooth leaky ReLU and its corresponding derivatives are shown in Figure 6. Early iterations of AdaLFI, learned y and 3 simultancously with the network weights 6, however; empirically, we found that setting = 0.01 and 3 = 10 gave adequate inference performance across our experiments.
C Experimental Setup
For all datasets, we use the original training- split, with 10% of the training data held out as a validation set. The validation set is used to optimize the training setup and tune hyper-parameters for the baseline models, ensuring they represent the strongest possible comparisons. Standard data augmentation caaniques--nrmalizattee random horizontal flips, and random cropping-are are applied to the training data of CIFAR-10, CIFAR-100, and SVHN during both meta-training and base training.
In the inner loop, all regression models are trained using stochastic gradient descent (SGD) with base learning rate of a = 0.001. Classification models are trained with SGD using base learning rate of a = 0.01, and on CIFAR-10. 10, CIFAR-100, and SVHN Nesterov momentum 0.9 and weight decay 0.0005 are applied. These choices reflect the scttings that gave the best performance on the validation set, while the remaining base-model hyper-parameters follow the standard configurations reported in the literature, consistent with the setup in (Gonzalez & Mikkulainen,, 2021).
To initialize Ma, Smit 2500 steps are taken in offinne mode with a meta learning rate of 1 le 3. In contrast, in online mode, meta learning rate of y le 5 is used (note, a high meta learning rate in online mode can cause a jittering effect in the loss function, which can cause training instability). For meta-optimization, the Adam optimizer (Kingma & Ba, 2015) is used in the outer loop for both initialization and online adaptation.
All experimental results reported show the average across 10 independent executions with different seeds to analyze algorithmic consistency. To ensure fairness, all methods are initialized with identical parameters under the same random seed, such that any differences in variance can be attributed to the respective algorithms and their loss functions. Furthermore, the choice of hyper-parameters between MetaL.R, ML.3. and A has been standardized to allow for fair comparison.
20

---

Published in Transactions on Machine Learning Research (10/2025)
C.1 Meta-Learned Learning Rate
A notable finding presented in Section 5.5 of the main manuscript is the intimate relationship wee learning a loss function and learning a learning rate/learning rate schedule, which is due to learned loss functions not just learning shape, but also learning scale. Given this relationship, it is intcresting to compare and contrast the performance of AdaLFL to a method for meta-learning a base learning rate online. To construct a fair comparison, we use an identical learning setup that used in AdaLFI, which allows us control for the hyper-parameter settings. The algorithm, which we further refer as Meta-LB presented in Algorithms 3 and uses an offiine initialization process to find the best initial base learning rate, following which learning subsequently progresses to an online adaptation process.
Algorithm 3 Learning Rate Initialization (Offline)
Input: CT Task loss function (meta-objective)
1: On < Initialize the base learning rate
2: for E (O, - 1} do
Do +- Reset parameters base learner for E {0, ..., Sinner 1} do
X, < Sample from Dtrain
0,41 +- 0, Vo, LT (X))
7: end for
8:
X, y + Sample from D Dvalid
9: Ltask + fou, (X))
10: 01++1 +- at - V,, Ltask
11: end for
Algorithm 4 Learning Rate Adaptation (Online)
Input: Learned learning rate
Input: L Task loss function
1: 0p < Initialize parameters of base learner
for E
1} do
X, y f- Sample from Dtroin
fr-1 t- 0, Vo, LT fo, (X))
5: X, f.y < Sample from Deatid
6: Liask + (T (X))
7: 04+1 t at - V, Lrask
8: end for
21

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Experimental results exploring alternative loss function representations based on Taylor polynomial parameterizations reporting the mean standard deviation of final inference testing mean squared error or error rate across 10 independent executions of each algorithm on each task + model pair.
Task
Model
Quadratic- TP (Online) Cubic-TP (Online)
AdaLFL (Online)
Crime
MLP
0.0254=0.0015
0.0263=0.0023
Diabetes
MLP
0.0418=0.0041
0.0420=0.0014
California
MLP
0.0783=0.0167
0.0151=0.0007
MNIST
Logistic
0.0810=0.0281
0.0707=0.0009
0.0697=0.0010
MLP
0.0205=0.0008
0.0185=0.0007
0.0184=0.0006
LeNet-5
0.1357=0.0728
0.0096=0.0006
0.0091=0.0004
CIFAR-10
VGG-16
0.1442=0.0025
0.1439-0.0027
0.0903+0.0053
AIICNN-C
0.1086=000100
0.0908=0.0020
0.0835=0.0050
ResNet-18
0.1133-000033
0.13091100070
SqueezeNet
0.1506=0.0092
0.1367#0.0041
0.1089=0.0049
CIFAR-100
WRN 28-10
0.2952-0.0220
0.2934=0.0621
0.2668=0.0283
SVHN
WRN 16-8
0.0494=0.0000
0.0431=0.0000
0.0441=0.0014
Further Experiments
D.1 Loss Function Representations
In AdaLFT. a two-hidden-layee feedforward neural network is used for the loss function representation, this was inspired by its use in prior studics (Bechtle et al., 2021; Psaros et al., 2022). We chose this represcntation it has more expressive power than both quadratic and eubic Taylor polynomials, which were used in (Gonzalez & Mikkkulaneen 2021) and (Gao et al., 2021; 2022), respectively. Although the best representation for learned loss functions is not under investigation; it is important to note that the proposed method of online meta-optimization discussed in Section 2.4 makes no assumptions about the underlying representation used for the learned loss function. Therefore, alternative representations can be used in AdaLFT.
In Table 4, results comparing and contrasting the performance between different learned loss function rep- resentations are presented. Specifically, we contrast the performance of AdaLFL which uses ferd-forward neural network (NN) with smooth leaky ReLU activations against the aforementioned quadratic and eu- bic Taylor polynomials (TP) representation. The results show that the NN representation has average the best performance and consistency in contrast to quadratic and cubic Taylor polynomials, with better performance and very little variance between independent executions on all datasets except the two small regression datasots Crime and Diabetes. These results demonstrate the superiority of the NN representation for learned loss functions, especially when dealing with relatively large learning tasks where expressive be- havior is important. Note, that on the regression datasets, we found that the majority of the quadratic TP experiments diverged, even with hyper-parameter tuning.
D.2 Loss Network Activation Function
An important difference between AdaLFL's neural network representation and prior neural network-basod learned loss function representation such the one used in ML3, is the use of smooth leaky ReLU activation functions presented in Section 2.2 of the main manuscript. This new activation function resolves many issues with the prior network design; however, it remains to be seen how much of the performance improvement be attributed to the newly proposed smooth leaky ReLU activation function vs the newly proposed online optimization algorithhm.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Table 5: Experimental results ablating the newly proposed smooth leaky ReLU activation function, reporting the mean + standard deviation of final inference testing mean squared error or error rate across 10 independent executions of each algorithm on each task + model pair.
Task
Model
ML ReLU (Offline)
ML SLReLU (Offline)
AdaLFL (Online)
Crime
MLP
0.0270=0.0025
0.0274#0.0029
0.0263+0.0023
Diabetes
MLP
0.0481+0.0020
0.0430+0.0012
0.0420+0.0014
California
MLP
0.0346:10.0087
0.0276:000058
0.015111.0007
MNIST
Logistic
0.0782:0.0111
0.0710=0.0015
0.0697+0.0010
MLP
0.0167=000021
0.0185+:00004
0.0184+0.0006
LeNet-5
0.0095+000006
0.0094+0.0005
0.009140.0004
CIFAR-10
VGG-16
0.1034+0.0058
0.1024+000055
0.0903+0.0032
AlICNN-C
0.1087
0.1015++0.005
0.0835+0.0055
ResNet-18
0.0972+4..255
0.0883++0.001
0.0788:10.0335
SqueezeNet.
0.1282:10.0086
0.1162:10.0052
0.1083:10.0049
CIFAR-100
WRN 28-10
0.311440.0063
0.3108:0.0075
0.2668-100283
SVHN
WRN 16-8
0.0500++0.004
0.0502+:0.0022
0.0441+0.0014
In Table 5, results are presented comparing and contrasting the performance between offline loss function learning (i.e. MI.3) with the standard ReL.U + SoftPlus network architecture, and the new smooth leaky ReLU network architecture. The results show that on most tasks the new activation function improves per- formance compared to the conventional architecture used in ML However, this performance improvement is not significant. contributing factor compared to the change in optimization algorithm, i.e. going from offline online meta-lcarning.
D.3 Second-Order Hyperparameter Sensitivity
We further investigated the sensitivity of AdaLFL to second-order hyperparameters, with particular fo- cus on the meta-level learning rate n, which was the only parameter found to significantly affect perfor- mance, a behavior consistent with observations in other widely used meta-learning algorithms such as MAML (Finn al., 2017). In the online adap- tation sctting, setting 1) too high can cause the learned loss function to change ter each update, leading to unstable or oscillatory training dynamics. To evaluate this effect, we per- formed an ablation study on MNIST using LeNet- with varying values for the meta learning rate 7 E
Table 6: Ablating the socond-order hyperparameter sensitivity to the mela-level learning rale 1). Results report the mean + standard deviation of testing error rate across 10 independent exccutions of cach m
Task
Model
AdaLFI
MNIST
LeNet-5 (1 = 10
0.0191 + 0.0070
abruptly af-
LeNet-5 (n = 10-2)
0.0159 + 0.0017
LeNet-5 (n) = 10-3)
0.0104 + 0.0011
LeNet-5 (n 10 10-4)
0.0094 + 0.0007
LeNet-5 (11 10 10-5)
0.0091 + 0.0004
The results presented in Ta- ble 6, demonstrate that AdaLFL maintains stable and consistent performance for all but the largest value of n, where there is slight degradation in accuracy. Overall, these findings indicate that AdaLFL is robust to the choice of within a relatively broad range, providing evidence of its stability with respect to sedond-order hyperparameter sensitivity.
LeNet-5 (17 = 10 10-6)
0.0088 + 0.0004
23

---

Published in Transactions on Machine Learning Research (10/2025)
Log Squared Error
10
Log Squared Brror
10
10
10a
10
10
10
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
(a) Crime MLP
(b) Diabetes MLP
Adal.FI.(Trrinnnng (Training)
Fd (Validatto)))))
Figure 7: Mean learning curves across 10 independent executions of AdaLFL showing the training mean squared error (y-axis) against gradient steps (x-axis) when taking meta gradient steps on the meta-training (blue) vs meta-validation (red) set.
D.4 Regularization and Meta- Beea-Objess
When computing the meta-objective of AdaLFL (Equation 8 in the main manuscript) Ltask = CT (x)), the instances can either be sampled from Drrain or vallid- Consequently, the learned loss function can either optimize for in-sample performance or out-of-sample gener- alization, respectively. This behavior is shown the Communities and Crime and Diabetes dataset, where as shown in Figure 7, optimiz- ing the meta-objective using training samples results in the training error quickly approaching 0. In contrast, when using the validation samples the train- ing error does not converge as quickly and to as low of a training error value. This behavior a form of regularization since as shown in Table the final inference testing error is superior when using validation samples on both the Communities and Crime and Diabetes datasets. This an important discovery as it suggests that loss function learning can induce form of regularization, similar to the findings in (Gonzalez & Müikkulainen, 2021; 2020a; Raymond et al., 2023).
Table 7: Results reporting the mean + standard deviation
testing mean squared error across 10 independent exccutions of each algorithm on each task + model pair.
Task
AdaLFL (Training)
AdaLFI. (Validation)
Crime
0.0267++0.002
0.0265++00002
Diabetes
0.0468:1000016
0.042010.0014
D.5 Generalization Dynamics During Training
To further assess the generalization perfor- mance of AdaLFL during training, con- ducted experiments on MNIST using the LeNet- 5 base model. The dataset was partitioned into four subsets: training, validation, evalua- tion, and testing. The validation set was used learn the loss function parameters é, while the evaluation set, a distinct held-out subset, employed to monitor oft-of-sample per- formance throughout training. As shown in Figure 8, the proposed method achieves faster convergence and improved generalization com- pared to the baseline. These results indicate that AdaLFL not only accolorates optimization but also enhances robustness against overfit- ting, leading to more stable learning dynamics.
10"
Log Error Rate
10
5000
10000
15000
20000
25000
Reseline
ML (Odflinn)
Metal. (Chline)
AdaL.FL. (Online)
Figure 8: Mean leaning curves across 10 independent executions of cach algorithm LoNST+ showing the log of the ian error rale (y-axis) against gradient steps (x-axis).
24

---

Published in Transactions on Machine Learning Research (10/2025)
D.6 Learned Loss Functions (Extended)
7.5
Learned Loss
7.0
Learned Loss
6.0
20
5.5
4.5
-044
-022
0.0
0.2
0.4
-04
-02
0.0
0.2
0.
Error
Error
21
Learned Loss
Learned Loss
13
19
12
11
17
10
Error
Error
22
21
Leamed Loss
Leamed Loss
21-
20
19
19
-0.2
-44
-6.2
0.0
0.2
Error
Error
Learned Loss
Learned Loss
25
24-
19
18
22
-0.4
-0.2
-044
0.0
0.2
Error
Error
8000
Figure 9: Loss functions generated by AdaLFF. the Communities and Crime dataset, where each plot represents loss function, and the color represents the current gradient step.
25

---

Published in Transactions on Machine Learning Research (10/2025)
120
Learned Loss
Learned Loss
100
100
100
70
-0.4
0.0
0.2
-0.4
-02
0.0
0.2
Error
Error
110
Learned Loss
100
Learned Loss
80
90
70
60-
0.2
Error
Error
110
Learned Loss
Learned Loss
-044
0.2
Error
Error
100
80-
Learned Loss
Learned Loss
70-
70
-0.4
-0.2
0.0
0.2
0.4
-0.4
-02
0.0
0.2
0.4
Error
Error
2000
4000
6000
8000
10000
Figure 10: Loss functions generated by AdaLFL on the Diabetes dataset, where each plot represents loss function, and the color represents the current gradient step.
26

---

Published in Transactions on Machine Learning Research (10/2025)
Learned Loss
Learndd Loss
0.4
-022
0.0
0.2
0.4
-0.4
0.0 Error
0.2
Brror
2.6
2.4
Learne Loss
Leamed Loss
2.2
2.0
4.0
1.6
3.5
1.4
3.0
0.2
0.9
0.4
Error
Error
3.25
4.0
3.00
Learned Loss
Leamnd Loss
3.5
2.50
3.0
2,25
2.00
2.5
2.0
-0.4
-0.2
0.0
0.2
0.1
0.4
0.2
0.0
0.2
0.4
Error
Error
2.5
2.5
Learned Loss
Learned Loss
1.5
1.0
1.0
0.5
0.2
0.0
Error
Error
2000
4000
6000
8000
10000
Figure 11: Loss functions generated by AdaLFL on the California Housing dataset, where each plot represents loss function, and the color represents the current gradient step.
27

---

Published in Transactions on Machine Learning Research (10/2025)
17.5
Learned Loss
Learned
0.0
0.8
1.0
0.0
0.2
0.4
06
6.
16
Predicted Probability (y 1)
Predicted Probability (y 0)
16-
15.
Leamnd Loss
Learned Loss
12-
66-
10-
0.8
Predicted Probability (y 1)
Predicted Probability (y 00)
16-
Leamnd Loss
Leaned Loss
12
5-
00
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
06
0.8
1.9
Predicted Probubility
Predicted Probability
0)
25.0
22.5
Leared Loss
26.0
20
Learned
15
12.5
10.0
Predicted Probability (y - 1)
Predicted Probability (y
5000
10000
15000
20000
25000
Figure 12: Loss functions generated by AdaLFL on the MNIST dataset, where each row represents loss function, and the color represents the current gradient step.
28

---

Published in Transactions on Machine Learning Research (10/2025)
235
-50
Learned Loss
Learned Loss
5+
-100
-150
-175
0.0
04
0.5
1.0
0.0
0.2
04
0.5
6.
Predicted Probubility (y
Predicted Probubibity (y
10-
Leamedd Loss
Learned
10-
0.
06
Predicted Probability (y - 1)
Predicted Probability (y 0)
201
14-
Learned Loss
12
Learned Loss
10
0.2
0.4
66
0.8
1.0
0.0
0.2
0.4
066
6.6
Predicted Probability 1)
Predicted Probebility (y 00
201
20-
15
Leamed Loss
Leannd Loss
16-
14-
10
0.4
0.6
0.8
06 Predicted Probability (y 0)
Predicted Probuhility (y
5000
10000
15000
20000
25000
Figure 13: Loss functions generated by AdaLFL on the MNIST dataset, where each row represents a loss function, and the color represents the current gradient step.
29

---

Published in Transactions on Machine Learning Research (10/2025)
80-
40-
Learned Loss
50
60-
20-
40-
10-
63
04
56
0.
1.0
0.0
64
0.5
6.
Predicted Probubility (y
Predicted Probubility (y
50
60-
Leamnd Loss
46-
Learned Loss
20
30-
20-
20
-60
-40
0.8
Predicted Probability (y 1)
Predicted Probability (y - 0)
00
126
30
100-
Learned Loss
80-
Learned
40-
-10
26-
0.2
0.4
06
0.8
1.0
64
0.8
Predicted Probubility (y
Predicted Probubility (y 00
Learne Loss
60
-40
-50
-60
70-
Prodicted Probability (y 1)
Predicted Probability (y -y-00 0)
5000
10000
15000
20000
25000
Figure 14: Loss functions generated by AdaLFL on the MNIST dataset, where each row represents a loss function, and the color represents the current gradient step.
30

---

Published in Transactions on Machine Learning Research (10/2025)
20-
100
75
Leamed Loss
Learned Loss
-20
-40
25
60-
-50
0.6
0.2
64
0.6
6%
1.0
6.0
0.3
6.6
Predicted Probubility (y
Predicted Probability (y-
150
26
Learned Loss
Learned
75
-20
50
Predicted Probability (y 1)
Predicted Probability (y 0)
40
80-
Learned Loss
40
20
-40
-60
0.0
0,4
0.3
1.0
Predicted Probubility (y
Predicted Probability (y=
Learned Loss
Learned
80
-100
30
0.5
04
06
0.
1.0
0.6
6.2
0.4
0.6
Predicted Probability (y
Predicted Probability
20000
40000
60000
80000
100000
Figure 15: Loss functions generated by AdaLFL on the CIFAR-10 datasot, where each row represents a loss function, and the color represents the current gradient step.
31

---

Published in Transactions on Machine Learning Research (10/2025)
80
Learned Loss
60
06
Predicted Probability (y 1)
Predicted Probability (y -0-0)) 0)
100
60-
Learned Logs
20-
40-
Learned
26-
-40
60
-50-
0.0
63
04
06
0.
1.0
0.0
Predicted Probability (y
Predicted Probability (y
60-
120
Learned Loss
100
86-
Learned
60
26-
Predicted Probability (y
Predicted Probability (y
100
25
Leamed Loss
50
-35
Learne
-75
40-
30
Predicted Probability (y 1)
Predicted Prohability (y 0)
20000
40000
60000
80000
100000
Figure 16: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where each row represents a loss function, and the color represents the current gradient step.
32

---

Published in Transactions on Machine Learning Research (10/2025)
20
10
Leurted Loss
Learned Loss
-20
-30-
-10
0.0
04
36
0.8
1.0
0.0
0.2
0.4
066
688
Predicted Probability (y 1)
Predicted Probability (y= 0)
20-
20-
Learned Loss
Leamnd
10-
-10
.0
-20-
Predicted Probubility (y 1)
Predicted Probubility (y- 0)
50
20
26-
Learned Loss
10-
Learned
-10
-0-
0.0
0.8
1.0
Predicted Probubility (y
Predicted Probability (y=
60-
40
50
Leareed Loss
Leaned Loss
40
-0
20-
-10-
-20
10
-10
94
0.6
0.8
Predicted Probubility (y
Predicted Probability (y-
20000
40000
60000
80000
100000
Figure 17: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where each row represents a loss function, and the color represents the current gradient step.
33

---

Published in Transactions on Machine Learning Research (10/2025)
50
20-
Learned Loss
Learned
10
20-
10-
-00
Predicted Probability (y 1)
Predicted Probability (y 0)
40
Learned Loss
Learned
-20
-90
-10-
-00
0,4
0.3
1.0
Predicted Probubility (y - 1)
Predicted Probability (y- 0)
30
10-
40
Learned Loss
Learned
10-
-10-
-20
5.2
0.4
06
0.8
1.0
02
0.8
Predicted Probubility (y
Predicted Probability
50-
10
40-
Learned Loss
36
Learned
-10
-50
-10-
25-
Predicted Probability (y 1)
Predicted Probability (y
20000
40000
60000
80000
100000
Figure 18: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where each row represents a loss function, and the color represents the current gradient step.
34


Reviews:
[{"review_id": "KaZ8UZzXzQ", "reviewer": "Reviewer_NvPj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Adaptive Loss Function Learning (AdaLFL), an online meta-learning method that updates a learned loss function after each base model update to address the short-horizon bias in prior offline approaches (e.g., ML3). A small feed-forward network parameterizes the loss, and smooth leaky ReLU activations are used to avoid flatness. Experiments on several datasets show improved convergence and final performance over cross-entropy and ML3.\n\nStrengths:\n\n1. Addresses a well-motivated limitation of offline loss function learning.\n2. Experimental evaluation covers multiple datasets and architectures.\n3. Some qualitative analysis of learned loss functions.\n\nWeaknesses:\n\n1. Method novelty is incremental; largely an adaptation of existing offline approaches to an online setting.\n2. Computational and memory cost of unrolled differentiation not addressed; scalability to large models unclear.\n3. Initialization stability issues (e.g., CIFAR-100) remain unresolved.\n4. Experiments are on relatively small-scale benchmarks; unclear real-world applicability.", "claims_and_evidence_yn": "No", "claims_explanation": "1. While the reported numbers show some improvement, the gains are often small, inconsistent across datasets, and not convincingly tied to the proposed online adaptation mechanism. \n\n2. The paper does not sufficiently analyze why AdaLFL outperforms baselines in certain settings but fails to offer stable benefits in others. \n\n3. What's more, the introduction of the hyperparameter of Eq 4 make the algorithm more sensitive and I doubt the generation of the algorithm.", "audience_yn": "Yes", "audience_explanation": "Loss function learning is a niche area, and the presented method is an incremental variation without compelling theoretical insight or large-scale validation.", "requested_changes": "1. Eq 4 list the formulation of loss function, I doubt the sensitivity of the hyperparapmeter. It is already a meta learning algorithm to learn hyper parameter, you add second order hyper parameters, which make the algorithm hard to tune. Is it correct? I believe a fix form of \\phi can be better.\n\n2. Provide a thorough computational/memory cost analysis for large models and larger datasets.\n\n3. Analyze failure cases and stability issues in more depth.", "broader_impact_concerns": "No major ethical concerns, but the lack of scalability and limited evaluation on larger datasets.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L7WX8nf0so", "reviewer": "Reviewer_E7po", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work addresses the challenge of handcrafting loss functions or learning them in an offline manner, both of which tend to perform well in early stages of training but not in later stages. Instead, the authors lift the assumption of a fixed loss function throughout training and propose online loss function learning through AdaLFL (Adaptive Loss Function Learning), a bilevel optimization algorithm that jointly optimizes task parameters and a loss network. The loss network is updated for each training iteration of the base task network, in addition to some \"warm-start-type\" initialization of the loss network. The paper empirically shows that models trained with AdaLFL exhibit faster convergence and improved inference performance compared to those trained with handcrafted or offline-learned loss functions, for several choices of datasets and model architectures. It also attempts to analyze the learned loss functions to interpret why they perform well. The overall ideas of the paper are clearly presented, although with a few typos and some ambiguous figure captions (listed in the Requested Changes).", "claims_and_evidence_yn": "No", "claims_explanation": "While the paper offers an interesting and intuitive idea, i.e. of online loss function learning, the experimental evidence lacks soundness in a few ways:\n- It is unclear whether the better training curves of AdaLFL shown in Figure 3 are due to improved learning or more overfitting. Showing validation curves instead would improve this piece of evidence.\n- The test performance gains are small in several cases, relative to the variance of the results. Hence, strong improvement due to AdaLFL is difficult to claim.\n- The qualitative interpretation of the learned loss curves is definitely interesting, but there isn't much formalization or quantitative evidence to support claims about implicit early stopping or implicit label smoothing. That said, the implicit learning rate scheduling claim is well-supported by the change in the scale of the learned loss.", "audience_yn": "Yes", "audience_explanation": "Handcrafting loss functions manually is a time-consuming and challenging dark art that a lot of researchers have to deal with currently. I believe TMLR's audience, especially ML practitioners, would be interested in alternatives that perform well and do not have prohibitive computational overheads. A method like the online loss learning paradigm proposed by this work has the potential to improve researchers' efficiency as well their task performance.", "requested_changes": "- The takeaway of Figure 1 is a bit unclear at the point in the paper at which it appears. In general, authors might consider stating each figure's takeaway in its caption. Is there any difference between Figure 1 and Figure 4, except that they show different examples?\n- Please clarify whether the \"gradient steps\" referred to in the caption of Figure 1 refer to training iterations. What does the \"predicted probability\" refer to? Is it of the true class? The (y=0) and (y=1) labels are confusing since CIFAR is not a binary classification task. Also, does every \"row\" in the caption of Figure 1 refer to each curve? \n- The authors state that the \"principal goal of AdaLFL is to replace this conventional handcrafted loss function with a meta-learned adaptive loss function\". However, given that AdaLFL depends on the task loss function to train the learned loss network, it doesn't quite replace it, but perhaps learns from it. I would recommend improving the wording sos as to avoid overstating claims.\n- Does \"Appendix 2.1\" in section 2.2 refer to B.1? Authors might consider using `cref` to refer to different section of the paper\n- The citation for Bechtle et al., 2021 after equation 3 seems to use `citep` but should probably use `citet`.\n- Do authors mean \"If $S_{inner} > 1$\" towards the end of section 2.3?\n- Can authors show validation curves instead of, or in addition to, the training curves in Figure 3, so that learning can be disambiguated from overfitting during training? \n- Since there is space in the paper, the figures that are supposed to provide evidence for the implicit early stopping regularization and implicit label smoothing claims should appear in the main paper, as opposed to the appendix. Is it possible to quantify this effect and investigate whether it appears systematically across settings?\n- What is the additional computational cost of using AdaLFL during training, including the \"initialization\" training it requires?\n\nAll the above requested changes would be critical to making the paper well-written and supporting its claims.", "broader_impact_concerns": "The paper does not contain a Broader Impacts Statement. Authors should consider discussing the computational overhead that their method has.", "additional_comments": "Overall, I find the online loss learning paradigm a useful one and the authors' proposed algorithm interesting. However, the clarity of writing and soundness of its evaluation could be improved to help communicate its benefits.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UsmWTWhI3E", "reviewer": "Reviewer_4VXf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new meta-learning approach called Adaptive Loss Function Learning (AdaLFL), which automates the design of loss functions for machine learning models. Unlike previous \"offline\" methods that learn a static loss function before training the main model, which causes \"short-horizon bias\", AdaLFL learns and updates the loss function simultaneously and adaptively with the base model's parameters, adapting the shape and scale of the loss function throughout the training process. The learned loss function is represented as a small feed-forward neural network. The method consistently outperforms both traditional handcrafted loss function baselines and previous offline loss function learning techniques in experiments on a breadth of tabular regression and image classification tasks. It shows convergence over fewer training steps and better final inference performance. The paper addresses a flaw in previous neural network-based loss function designs that caused learned functions to become too flat and proposes a new smooth leaky ReLU activation function to prevent this. The adaptive nature of AdaLFL allows the learned loss function's scale to change over time, effectively tuning the learning rate schedule. The learned loss function can flatten or become parabolic at the end of training, which acts as a form of early stopping. In classification tasks, the loss function penalizes overconfident predictions, a behavior that resembles and dynamically adjusts an adaptive form of label smoothing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper presents experimental evaluation comparing AdaLFL against baselines and a leading offline loss function learning method (ML3). The experiments are conducted on a diverse range of tasks and models, including regression and classification datasets and various neural network architectures. The key claims summarized above are supported with quantitative results and qualitative discussion and analysis. The evidence presented is convincing and the narrative is clear overall. The authors have included ablation studies and comparisons to alternative methods (like meta-learning a learning rate) to strengthen their arguments, although as specified below, some of these should be moved from appendices to the main paper.", "audience_yn": "Yes", "audience_explanation": "The findings of this paper would be of significant interest to a broad segment of the TMLR audience, particularly those involved in meta-learning, hyperparameter optimization, and the fundamental theory of deep learning. This paper would interest an even wider audience if the methods were demonstrated on more modern architectures. For example, BERT-Small  $^1$ has fewer parameters than WRN-28-10, and SmolLM2-135M $^2$ also has an accessible size.\n\n1. Turc, I., Chang, M. W., Lee, K., & Toutanova, K. (2019). Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962.\n\n2. Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., ... & Wolf, T. (2025). SmolLM2: When Smol Goes Big--Data-Centric Training of a Small Language Model. arXiv preprint arXiv:2502.02737.", "requested_changes": "The following requested changes are roughly in order of importance.\n\nFigure 3 is rather misleading, as the computational overhead of the offline initialization optimization and online bilevel optimization is not represented. Please at least factor in the steps for AdaLFL initialization, and even better use \"FLOPs consumed\" for the horizontal axes. The paper only discusses efficiency in the appendix, but this topic is important to discuss in the main paper so please move it to Section 5. Further, Figure 3 would be better be shown on a completely held-out set (disjunct from the newly created \"validation\" set), rather than the training data for each dataset. The subplots would be easier to parse with logarithmic vertical axes, as well as lowering the upper $y$-limit of most plots: cropping away some of the early high-error steps is acceptable in order to view the end of the curve more clearly. Finally, please include standard deviation as shaded bands in these plots, as Table 1 suggests that the spread of the results are rather overlapping.\n\nI believe you use the newly created \"validation\" set for the outer optimization step. Since the \"baseline\" algorithm presumably does not have such a step, you should not hold-out this set from the training loop of baseline experiments. This is not specified in the paper, so I suggest to rerun the baseline experiments if necessary and explicitly state this dataset distinction in the text once it's added. Further, please explicitly describe the baseline algorithm(s) in Section 4, as I haven't found where it is specified if it is.\n\nThe meta-learned learning rate experiments are currently only discussed in the appendix, but should be moved to the main paper by merging Tables 1 and 3.\n\nAdding experiments on transformer architectures would strengthen this work, but are not necessary for my recommendation for acceptance.\n\nPlease use textual citations (`\\citet{}`) whenever the citation is a subject or object of the sentence. For example, \"In (Bechtle et al., 2021), ReLU activations are...\" should rather be \"In Bechtle et al. (2021), ReLU activations are...\".", "broader_impact_concerns": "I have no broader impact concerns regarding this work. The paper focuses on a fundamental and general-purpose machine learning technique. The research advances the science of training models and does not present any obvious ethical issues that would necessitate a Broader Impact Statement beyond what is standard for foundational AI research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)