Loss meta-learning for forecasting
Authors: Alan Collet
Antonio Bazco-Nogueras
Albert Banchs
Marco Fiore
Venue: ICLR 2022 Submitted (2022)
Keywords: meta-learning
loss function
forecasting
learning to learn

Abstract: Meta-learning of loss functions for supervised learning has been used to date for classification tasks, or as a way to enable few-shot learning. In this paper, we show how a fairly simple loss meta-learning approach can substantially improve regression results. Specifically, we target forecasting of time series and explore case studies grounded on real-world data, and show that meta-learned losses can benefit the quality of the prediction both in cases that are apparently naive and in practical scenarios where the performance metric is complex, time-correlated, non-differentiable, or not known a-priori.



Under review as a conference paper at ICLR 2022
Loss META-LEARNING FOR FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning of loss functions for supervised learning has been used to date for classification tasks, or as a way to enable few-shot learning. In this paper, we show how fairly simple loss meta-learning approach can substantially improve regression results. Specifically, we target forecasting of time series and explore case studies grounded on real-world data, and show that meta-learned losses can benefit the quality of the prediction both in cases that are apparently naive, and in practical scenarios where the performance metric is complex, time-correlated, non-differentiable, or not known priori.
INTRODUCTION
Loss functions drive the training process of supervised machine learning models. In the vast majority of cases, loss functions are designed to be generic enough to work well with a wide range of application scenarios. In regression problems, including forecasting tasks, Mean Absolute Error (MAE), Mean Square Error (MSE), or Mean squared logarithmic error (MSLE) are common choices for expressing the loss.
In this paper, we question the assumption that a fixed and generic loss function is in fact the best choice in regression -and more specifically forecasting- problems, and investigate how losses learned from experience benefit the model performance in this type of task. Therefore, our work contributes to recent efforts in meta-learning and specifically to those aimed at so-called learning teach (Wu et al.) 2018). In this context, several proposals have been set forth for automatically learning the parameters, components, or shape of loss functions for neural network training. However, as extensively discussed in Section prior studies have focused on parametrizable losses for classification, and little attention has been paid to the learning to teach paradigm in the context of regression tasks. Part of the reason comes from the fact that loss meta-learning has been generally considered inefficient approach for single-task regression (Sung et al., 2017), under the assumption that losses such as MAE or MSE can already optimally drive regressor training in that case.
Our study challenges such an assumption and shows that, in practical cases, a simple yet aptly designed meta-learning model can in fact improve the training of regression models with respect to legacy and presumably optimal losses. As detailed in Section 3 our proposed model, named Met Meos,, builds on joint co-training of the main regressor network and of the loss-learning network, and takes advantage of controlled noise during training to implement the exploration of the correct loss function behavior for rarely observed samples. We apply MetaLoss the specific problem of time series forecasting in Sections 4 and and demonstrate how meta-learning of loss functions can help forecasting tasks under both apparently naive and complex performance metrics. In the case of naive metrics such as MAE or MSF, we disclose how the automated tailoring of the loss to different magnitudes (and not only errors) of the prediction can provide substantial prove that MetaLoss successfully
gains on the accuracy. In the case of complex metrics, learn differentiable approximations of time-correlated and non-differentiable performance measures observed in two real- world applications, driving the regressor network towards forecasts that improve solutions currently considered in practice.
Overall, by unveiling for the first time the advantages of meta-learning of loss functions for forecasting tasks, our study paves the road for the adoption of this paradigm in a previously unexplored machine learning domain.

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
Meta-learning, also referred to as learning-to-lean, overcomes the limitations of fixed learning-based models, and allows automatically tuning different aspects of the learning algorithm to the target task (Hospedales et al. 2021). Meta-learning has been successfully applied to, e.g. distillation [Wang] et al. 2020), augmentation (Cubuk et al.) 2019) or batching (Fan et al. 2018 of training data, initialization (Finn et al. 2017) or optimization (Andrychowicz et 2016)) of the model parameters, tuning (Micaelli & Storkey (020) of its typer-parameteed
and discovery (Liu et al., 2019) of the actual architecture, possibly as a composition of modules (Alet et al., 2019). Our focus on meta- learning of loss functions, which aims at learning the loss to be used to train the actual model. The problem can be seen as an instance of a hicrarchical optimization. where a meta-model is optimized under constraint represented by the main model optimization (Franceschi et al. 2018). We stress that this is semantically different from meta-learning optimization schedules in iterative and alternate optimization processes (Xu et al. 2019). Specifically, distinguish two main approaches to loss meta-lcarning, discussed next.
The first approach consists in having teacher network infer the most suitable configuration of a predefined, parametrizable loss function. In this direction, several studies have investigated the use of decision networks to select among set of predefined (family of) loss functions (Liu & Lai 2020: Denevi et al., 2018). while others have focused on multi-part loss functions, where the goal setting (Huang et al. 2019 Zhao et al. 2019) and possibly dynamically updating (Heydari et al. 2019) the function weights based on (live) performance metrics. In similar way. models have been proposed compose a loss function from primitive mathematical operators (Li et al. al.] 2021). or to express the performance metric function of (reduced) set of simple surrogates (Grabocka et al. 2iang et al. (020). Also related to the same concept are strategies such as training a network to correct the optimization trajectory produced by a fixed loss (Huang et al., (021), or introducing general loss functions that contain hyper-pare parameters to be learned during training along with the neural network parameters (Barron 2019). In all these cases, the loss- independently of whether it is expressed as a tunable function, set of primitives, or surrogates - must be designed or selected manually, which is very challenging or even not possible when the performance metric of interest is especially complex, non-differentiable or even not known a priori. With respect to these works, we seek instead a solution that can learn a clean-slate loss.
The second approach to loss meta-learning is more appropriate for clean-slate losses, and thus the one we also adopt in our work. The basic concept is representing the loss function itself via a (typically lightweight) neural network, which is fed with relevant input (e.g., main model output, labels, or features) and produces a fit loss to be used to train the main model. In an early work, it was proposed the use of teacher network to dynamically train parameters of loss function that adapts to the learning stage of the main model (Wu et al.] 2018); yet, this approach still relies on generic known loss function to be parametrized by the teacher, hence suffers from the same limitations of the studies listed above. Closer to our methodology, the seminal idea of a trainable introduced for reinforeement and supervised learning by the meta-critic model, where an action-value function neural network learns to criticise the actions in a specified task (Sung al. 2017 Zhou et al., 2020). However, the meta-critic model is only applied to supervised learning problems as a tool for pre-training that allows for the few-shot learning of new tasks (e.g., by generalizing to unseen value ranges in the same domain). Indeed, the authors make explicit that, in the case of a single task, modeling the loss via dedicated neural network creates an indirection that makes learning more inefficient (Sung et al.) 2017). We show instead that loss meta-learning can in fact improve over presumably optimal losses also in simple single forecasting tasks. As such, our model in fact complementary to the meta-critic one, as aetaLoss could be integrated with the meta-critic helper block to automatically model more tasks.
musk-parmmetrized loss generator
is worth mentioning that in the vast majority of previous works, and in other studies (Shu et al., 2020, Jiang et al. 2020 Gao et al.) 2021] Jiang et al. 2020 Marchetti et al. (021) that also include recent proposals to employ genetic programming tools to learn loss functions (Gonzalez & Mikkullinnnn 2020: 20,1), the aim of the loss meta-learning strategy is on discrete-space classification tasks or ranking problems. Little attention has been instead paid until now to loss meta-learning for regression. By investigating meta-leaming solutions that target loss functions for forecasting tasks, our work shed new light on the advantages that this emerging paradigm can bring to a class of machine learning problems where loss meta-learning has been overlooked to date.
2

---

Under review as a conference paper at ICLR 2022
3 A MODEL FOR REGRESSION LOSS META-LEARNING
The proposed approach, named Metaloss, consists of forms twofold learning. First, it must learn to forecast the value of action that minimizes a certain loss function. Second, it must also learn which is the said loss function according to the measures obtained from the environment. This concept has obvious applications in many practical regression problems where but the expression of the objective cannot be characterized manually because of its complexity, or is directly not known priori; in these scenarios, we cannot apply particular or well-defined loss function during the training phase. However, we will show that the approach can also benefit simpler cases, where the goal is pure prediction of time series, and where MAE, MSE, or variants of the same are generally considered to be very effective losses.
fluncionnaggootiio regressor that per-
measure the performance resulting from the actions taken by the system,
In the following, first provide formal definition of the problem, and then present the high-level concept of Met KetaLoss as well as the detailed design of its architecture and operation.
3.1 PROBLEM FORMULATION
Let us denote the space of system state variables as S. We also denote the input space of the predictor as X and the output space as Y. such that the predictor can be modeled as fw X + Y. and the decision for time + taken at time t by the predictor can be written a 0t-1 fw (zz)), where 2, E X includes the past observations of the system state, and W represents the parameters of the predictor.
Let us denote the performance cost of the predictor's decision taken at time t, measured at time + 1, as Mr+1 (M++1 , Ut+1, 1, 1)+1), where f/-11 denotes the current observations (at the time of the measurement / + 1) of the system variables that may impact M Ut+11 denotes the actual value that Ø641 attempts to predict, and f-((-) represents the a priori unknown expression of the objective. Note that, even if this expression is not known, the performance is assumed to be measurable, and samples of (-) can be obtained by observing the outcome of the predictor's 's output on the system.
Since the relation f-(()) is uncharted at first, the model also has to learn it. For that, we define second optimization function to describe the loss-leaming task. Such loss-learning task takes as inputs (i) the predictor's decision 11+1 and (ii) the current observations + 1, and it casts performance cost estimate M fw (V++1, Dc+1, 1141), where W represents the parameters of the loss-learning process.
Consequently, MetaLoss is composed of two optimization problems. First, the loss-learning task aims at correctly characterizing (M It+1, Ue+1) through the estimated fw (Je+1, It+11 0e+1). 1). This is done by computing legacy loss function (e.g., MAE or MSE). For example, if we consider the MSE 1.2 T.2 b) = ||a - bll², it follows that the loss-learning optimizer objective is
min W
I,2 (Jwr (ye+1, De+1, v(41), [M De+1, u(+1))-
In turn, the predictor's objective is to minimize the performance cost of its decisions based on the predicted performance fw (Ye+1, Be+1, U2+1), i.e., to solve the following optimization:
min W
fw (xx)) v(41).
3.2 MODEL CONCEPT AND TRAINING
The MetaLoss model aims at solving the problem above in an effective manner. To this end, it represents both the predictor and the loss function through Deep Neural Networks (DNN), one for each of the two blocks, as illustrated in Figure Note that MetaLoss is a conceptual model that can accommodate diverse implementations: hence, the exact architecture the neural networks, including their layering and activations, can differ depending on the considered task and the complexity of the
'Hereinafter, for the sake of simplicity and clarity, we use the scalar notation to describe the predictor's output. Nevertheless, the proposed approach is not limited to uni-dimensional fields and can be applied to more generic multidimensional cases.
3

---

Under review as a conference paper at ICLR 2022
2y (0)
V
24 (1)
Ut+1
91+1 +
(2)
Predictor
Swe
91+1
Loes: Learning Black Jw
(1))
(1+1(0)
M
U1+1(m)
VMa+1(yt-1 00-111101111110 + 6, 0t+1)
The unknown relation between the objective and the outputs of the predictor is learnt and encoded into a loss-learning block. Then, this block acts as the loss function to train the predictor, such that the predictor directly provides an output fitting the desired metric. The whole process is automated.
Figure I:
involved data. Therefore, the detailed implementation of the main blocks will be reported in Sections and separately for each application use case.
The algorithm adopted train the MetaLoss model is outlined Algorithm There, we make use of the Cyclic Learning Rate (CLR) method (Smith, 2017), which lets the learming rate oscillate within range, with the extreme values of this range updated each batch iteration. This method reduces the sensitivity of the system to the initial configuration, and it prevents that a wrongly selected initial value triggers a poor performance by becoming entrenched in some local minimum. In general, CLR accelerates the automatic convergence, which proved especially useful in contexts where the loss function must be inferred during training. We employ the simple triangular version of CLR, with eycles across the full training phase.
Algorithm 1: Training procedure of MetaLoss
Initialize predictor nn, fwe (2
Initialize loss nn fw (ye+1, St+1, V/41) Initialize predictor's learning rate of for t = 21,2.... T......Trnunee Trannngg do Randomly choose é
Predict the output 0t+1 = (wr E)
w < W a. V T. (sw (Ue+1, det1 + e, fm(H+,, 041 + +.,04+1)
W t w af Vgv= fwr (1t+1, Det1 + e, 0.41)
Update of
end
Three important remarks are in order, concerning the Metaloss concept and training.
First, the meta-learning model outlined above able to approximate a non-differentiable objective M by differentiable alternative fw (-), which is implemented by the loss-learning DNN upon training. In turn, this allows optimizing the predictor DNN under metrics that could not be directly used losses by producing suitable approximation of the same.
The second remark concerns the fact that the input of the loss-learning DNN is implemented by different expression than that indicated in the formal problem definition of Section 3.1 Specifically,
4

---

Under review as a conference paper at ICLR 2022
portrayed in Figure instead of providing the computed action Dt+1 to the loss- loss-learning DNN, we feed it with a disturbed version of 9t+1 by adding random noise e that also imput to the predictor DNN. As discussed in detail in Section 3.3 this is a methodological novelty that allows loss exploration during training.
The third observation is that the Metaloss model design creates the possibility of co-training the predictor and loss-learning blocks during same gradient descent iteration, which allows each DNN to be informed of (and learn from) the improvements of the other: this makes the learned loss fw (.) adapted to the inherent forecasting limits of the predictor, and we will expound this aspect in Section below.
3.3 LOSs EXPLORATION
During training, the loss-learning block does not receive the exact value output by the predictor, 2c+11 rather is input disturbed version of it, Ut-1 + where is random variable with mean. This reflected in Figure Therefore, the weights update computed by the gradient descent are shown in Algorithm[1 [ and are given by the following expressions for a particular weight of the predictor's DNN (w[+1) and a particular weight of the loss-learning DNN (wf41):
2 f.: (ye+1, Dc-1 + v(+1)
wf41 =w² - at"
(3)
Ouy
0T? OT. (Sw (Ue+1, det1 + veti), fmnUe+1, dr + v(+1)
wf+1 wf -a" a
Ouf
where af and af are the learning rates of the predictor and the loss-learning DNNs, respectively. The dependency of af on i is due to the use of CLR mentioned in Section 3.2
The goal of the random variable e is to allow for further exploration of the input values, supplying the loss-learning block with a broader observation of the input domain beyond that provided by the training samples. This enlargement of the input space improves the reliability of the characterization of the loss function over the continuous domain by the loss-learning block.
Note that the noise e is only needed during training, and it is set to once the expression the loss fw (.) learnt, i.e., during model testing. In this regard, critical design feature of Met aLoss is that e is also input to the regressor DNN: during training, this lets the prediction block learn the correlation between such input and the added disturbance its output. Then, during inference, setting c to allows producing forecasts D1+1 that are not biased by the loss exploration used in training.
3.4 CO-TRAINING OF PREDICTOR AND LOSS
MetaLoss is implemented block is fed by the current observations and the forecast output by the predictor. This allows us to jointly optimize the two blocks through the same backpropagation process. Specifically, and as also summarized in Algorithm | the weights of the two DNNs are optimized during training as follows.
cascaded DNNs, illustrated in Figure I where the learnnng
First, during the forward pass, the predictor is fed with of past observations of the system state from N previous time instants (as well as other possibly relevant inputs), and it outputs a prediction for + 1 (or, more generally, for the next M next time instants) at time fw" (x; e). At time the current observations are measured and passed to the loss-lcarning system, which computes the estimated performance function M = fw (ye+1:fw- (wr c) c, U(+1) At the same time, the actual performance of the taken decision M = fM (M++1, fwr (x,, e) + Bt+1) is measured.
Then, the mismatch between estimated and true performance is evaluated via legacy or standard loss function, and backpropagated first to the loss-learning DNN. Here, the loss-learning DNN updates its weights wf+1 to better capture the relation between M and the combined values of the prediction Ut+1 and the system state U++1 and 91+1- Within the same iteration, the updated loss is sequentially backpropagated to predictor DNN, which allows improving the aligmment the forecast with the optimal decision that minimizes M
5

---

Under review as a conference paper at ICLR 2022
This design increases the efficiency of the training phase with respect the case where each block is optimized independently, e.g., by feeding the loss-learning block with random predictions and, once the loss has been learned, using it to train the predictor. Indeed, co- co-training allows learning loss fw (+) that is adapted to the intrinsically limited accuracy of the predictor; as an example, co-training may lead to learning diverse shapes of the loss depending on the magnitude of the target variable 9t+1 if the quality of the prediction is found to be affected by the absolute value of yr+1- We will observe practical situations where this type of adaptation occurs in Sections and
It is worth noting that such a co-training represents a major novelty of our model with respect to previous related proposals (Sung et al. 2017; Wu et al. (2018). Indeed, the end-to-end backpropagation training was not possible in prior models, and the two elements (e.g., the learning to-act block and the learning-to-come to-coreect block) were trained either iteratively or nested manner only.
4 EXPERIMENTS: TIME SERIES FORECASTING
We first investigate the performance of MetaLoss in plain time series forecasting tasks. In these tests, we compare the one-step forecast returned by a same predictor DNN, which is trained under the MetaLoss model (i.e., via co-training with a loss-learning DNN), and with MAE loss. The predictor DNN identical the two cases, as all of the hyperparameters are kept the same between the classical MAE and the Met Meta approach; also, the architecture (number and type of layers, number of neurons per layer) are the same, and the same holds for the predictor learning rate, the number of epochs, and the activation functions. The starting seed is also fixed the start of both trainings to be the exact same in order to reduce randomness as most as possible. The results are presented on MinMaxsealed datasets and averaged with 5 different runs for each experiment.
We remark that the design of the predictor can vary across the forecasting tasks on different datasets later listed in Section 4.1 and we do not detail those here for the sake of brevity. What matters is that the architecture of the predictor DNN is identical under Meta loss and MAE, which allows juxtaposing the results obtained under the two approaches. Concerning the loss-lcarning block employed by MetaLose we use for all considered datasets simple Multi-Layer Perceptron (MLP). The size of the MIP depends on the number of input, i.e., on the size of v.
Importantly, we assess the quality of the prediction in terms of MAE itself. Therefore, static MAE loss that we use benchmark for MetaLoss represents the apparently optimal -and very commonly adopted- choice to drive the optimization of the predictor.
4.1 DATASETS
Experiments using different real-world datasets. The first dataset describes the mobile data traffic demand generated by four video streaming services (Facebook Live, Netflix, Twitch, and Youtube) in large metropolitan area during several consecutive months (Marquez al. 2017). The data was collected and aggregated by the mobile network operator using passive measurement probes, resulting in traffic levels (in bytes) every five minutes, for total of more than 22,000 samples for each of the four services.
The second dataset describes the hourly energy consumption in part of the Eastern Interconnection grid in the United States of America between 2002 and 2018. The data comes from a regional transmission organization (RTO) (Mulla 2018), and it incorporates information from different power providers, each one them managing a different geographical area. Overall, the time series consists of more than 145,000 samples.
A third dataset describes the energy consumption evolution within single household over time, using outdoor and indoor characteristies such as temperature, humidity or wind speed as inputs. These data are provided every 10 minutes for more than 4 months, and include over 20,000 data points (Candanedo et al. 2017).
4.2 RESULTS
The results for the application of Met MetaLoss to standard loss functions are shown in Table and Figure For each one of the analyzed time series, Table T shows the MAE performance (mean and

---

Under review as a conference paper at ICLR 2022
Table MAE measured in the one-step forecasting of different time series, when same predictor DNN is trained with static MAE loss function and with loss learned via Metaloss. The rightmost column reports the percent gain in MAE scored by the MetaLoss approach over the static MAE loss.
Dataset
MAE (x10-²)
MetaLoss (x10-²)
Gain (%)
Facebook Traffic
2.44:10.05
2.33 0.03
4.51
Netflix Traffic
4.38++0008
4.35 0.05
0.68
Twitch Traffic
3.81=00077
3.75
1.57
Youtube Traffic
3.71
3.68:10.03
0.81
Power Grid
2.68+10.04
0.05
4.85
House Energy
2.33++0.03
2.27=10.02
2.58
OHA 1010 MOL
Los po
sic. enc
(a) Facebook, MAE
(b) Facebook, MetaLoss
(c) Power, MAE
(d) Power, MetaLoss
Figure 2: Loss function and error distributions for the top and bottom 10% of predicted values, for the Facebook Live traffic under (a) MAE loss and (b) MetaLoss, and for the energy demand under ca) MAE loos and (d) MetaLoss.
standard deviation) obtained from (i) training the predictor block with standard MAE loss function (leftmost column), and (ti) using the approach Metaloss presented above, where the loss function is also learned (central column), as well as the percentage gain of MetaLoss over standard learning, Table T shows how Metal oss reduces costs for standard loss function as MAE up to 5% the best cases observed. This represents a significant gain, considering that the predictor architecture is the same for a MAE loss and for our model, especially we take into account that Met aLoss trained to learn the MAE loss function. Thus, MetaLoss succeeds in learning more than the teacher knows.
This excellent performance mostly comes from the fact that MetaLoss adapts its shape to the input data and, therefore, it optimizes the training phase almost independently for different input values. We can clearly see this phenomenon happening in plots (c) and (d) of Figure 2 where the loss learned by MetaLoss has learned and evolved during training in order to minimize the loss for both sets represented (the 10% highest and 10% lowest value samples), whereas with the standard model one of these sets is clearly biased the end of the training phase. This also be from different perspective within the case of Facebook Traffic (plots (a) and (b)): Although the bias correction only visible lesser extent, we can clearly see how the loss learning function has actually different shape for each one of the two sets considered.
EXPERIMENTS: APPLICATION USE CASES
Having proven the advantage that MetaLoss can yield in the context of plain time series forecasting tasks that aim at minimizing the MAE of the one-step prediction, we consider the more convoluted case where the relation between the decision space output by the predictor and the objective perfor- mance a complex, non-differentiable. and possibly not (fullyy known function of the prediction. We analyze two different use cases with strong practical applications, i.e., (i) anticipating the resources to be allocated in a mobile network to serve real-world traffic demands, and (it) managing power grid settings to serve the energy demand in nationwide scenario. These cases employ the first two datasets outlined in Section|4.] and are characterized by entangled and diverse relations between the prediction and the performance metric, detailed next.
7

---

Under review as a conference paper at ICLR 2022
Table 2: Operator cost of an anticipatory allocation of network resources measured for the mobile
data traffic generated by four different services, when a same predictor DNN is trained with a static a-OMC loss function (Bega et al. 2019) and with loss learned via Metaloss. The rightmost column reports the cost reduction achieved by the Metaloss approach over the static a-OMC loss.
Dataset
a-OMC (x10-))
MetaLoss (x10
Gain ( (%) )
Facebook Traffic
1.37+0.06
8.03
Netflix Traffic
1,57=0.09
0.07
3.29
Twitch Traffic
1.53=0007
1.47
3.92
Youtube Traffic
1.36=t0.05
1.31
3.68
6%
(a) Facebook, Q-OMC
(h) Facebook, Metaloss
(c) Power, MetaLoss (d) Power, Iearned 3D loss
Figure 3: Loss function and error distributions for the top and bottom 10% of predicted values, for the network resource allocation the Facebook Live service under (a) a-OMC loss and (b) Met loss, and for the power grid management with a lloss, with (c) two-dimensional and (d) three-dimensional representation of the learned loss.
5.1 USE CASE I: ANTICIPATORY NETWORK RESOURCE ALLOCATION
Metric. The goal of the operator at time is forecasting the required network resources (e.g., data transport capacity) needed to serve the demand for each of four streaming services (Facebook Live, Netflix, Twitch, and YouTube) in the following 5-minute time step. A plain prediction is insufficient in this case, as the cost asymmetric: specifically, the operator seeks to (i) avoid an expensive monetary fee 3 owed to the service provider in case insufficient resources are allocated and the future traffic y cannot be served, and (ii) prevent unnecessary overdimensioning beyond yt+11 in case of prediction errors with positive sign. Formally, the cost incurred by the operator is
M = 3 1 + (Je+1 - Bt+1) 19e4 20
(5)
This expression is clearly not differentiable due to the presence of the indicator functions 1c. which takes value 1 if the condition C in the subscript is met, and takes the value 0 otherwise. Recent studies in the computer networking literature have proposed expert-designed loss function, termed a- a-OMC, which is a manually devised differentiable approximation of Equation (5) above (Bega al. 2019). We employ a-OMC as a state-of-the-arr fixed loss benchmark be used term of comparison for Met aLess.
Network architecture. In this experiment, we employ as the predictor fw a multi-layer RNN regressor. The loss-learning neural network fw is an MLP with the simplest possible case for MetaLoss, using only yt+1 and Ột+1 as input. The predictor RNN is also trained using the a-OMC loss function for comparison purposes. Results are provided in the Table 2 for the four services.
Results. We can see in Table how Metaless yields cost reductions over a-MMC that range from 3% to 8%, which correspond to significant operating expense cuts in mobile network infrastructure management. This made possible by the fact that Met MotaLoss learns a loss that better tailored to the performance metric in Equation 5 than numan-designe the metric itself. In sense, the result proves how meta-learning losses for regression via Metaloss allows apprehending more than the teacher knows. Indeed, co- co-training the predictor and loss- leasning block as done in our proposed model enables discovering loss that is optimized for different absolute values of the predicted variable.
version developed with full knowledge of
8

---

Under review as a conference paper at ICLR 2022
This adaptation is illustrated in the first two plots in Figure where the losses and error distributions are told apart for the 10% of cases with lowest and highest traffic demands, under (a) the a a-OMC loss and (b) Metaloss. While the learned loss in Metaloss always captures the general behavior of Equation [] we can observe slight shifts in the error Ø++1 y++1 that minimizes the cost, depending on the absolute value of yt+11- In other words, Met loss learns a loss that naturally compensates for the different accuracy of the predictor in anticipating traffic values of diverse magnitude. Such an adaptation impossible to ascertain by just looking at the performance metric (i.e., the "teacher"), as it inherently depends on the prediction quality; yet, it is successfully learned by our model.
5.2 USE CASE II: POWER GRID MANAGEMENT
Metric. The complex field of power grid management and the study of smart grids are ruled by significant number of diverse Key Performance Indicators (KPI) (Pow) 2012 Personal et al. 2014] Harder 2017). One of the fundamental dimensions that define the performance in such scenarios is the reliability of the network, Le., how often the network fails to provide the required power.
Interestingly, the reliability in power management is not only measured by the frequency of power cuts due to the under-proee provisioovii but also by the duration of these cuts [Pow] (012). The service ender--stinatiins,
provider is especially interested in preventing network resource allocation; however, the metric applied to capacity forecasting cannot be considered this scenario because it does not take into account the duration of the under-provisioning. Now, require a cumulative metric that incorporates some memory. Hence, the metric considered for the Power Grid Management use case is as follows:
in the previous
case of
M,+1 = (B + M, 1g. 1 1 + (0e+1 Vc+1) 1904 200-1"
(6)
We can that, in the case of overdimensioning, the cost scales linearly with the unnecessary estimated power as for the previous use case. Yet, the cost is considerably different in case of underestimation: if the previous forecast was also underestimated, the current cost is added to the precedent cost in a recursive manner. Thus, this metric depends on the previous state of the network.
Network architecture. In this experiment the predictor fw- used is also multi-layer RNN regressor. The loss neural network fw is a MLP using inputs, i.e., Yt+11 and Dr+1 plus the number of previous successive anver-provisioned samples (maxed 5), which is time-dependent variable. Designing a loss function matching this metric is nothing simple and would not even make sense. Metaloss allows to use such metric which would not be possible otherwise. As 3 values are used as input of the loss neural network fwe. the resulting loss function is a dimensional shape in 4-dimension space. This resulting loss after training is presented in plot (d) of Figure 3 where the color describes the dimension and where only discretized values are presented corresponding to the number of successive past underprovisioned samples.
Loss-learning results. For this use case, we are particularly interested in showing how MetaLoss can adapt to complex cost relations. This can be observed plots (c) and (d) of Figure Metaloos precisely learns the complex and recursive cost function, which depends on previous samples. The result shows the potential of the proposed approach to characterize unknown and non-trivial loss functions for generic regression problems.
6 CONCLUSIONS
We have proposed Metaloss, meta-learning model for regression losses, and unveiled how this previously overlooked approach can in fact yield significant gains in forecasting tasks. Experiments with heterogeneous datasets prove how MetaLoss can improve the performance of plain one-step predictors in minimizing the MAE of the forecast, with respect to the common practice of employing a fixed MAE loss. Similarly, our tests demonstrate how Metaloss can successfully learn losses that capture complex, tmme-correlated and non- dinddifferentiable metrics. Finally, we highlight for the first time how MetaLoss co-training of prediction and loss allows learning more than the teacher knows, since the loss can be tailored to the inherent (in)accuracy of the predictor, e.g., for different values of the target variable.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
In our work, we employ de-personalized datasets that report information on mobile service usage and energy consumption aggregated over millions of individuals. This makes re-identification of data subjects from the data impossible, hence our research do involve risks for the mobile subscribers or power grid customers.
REPRODUCIBILITY STATEMENT
We commit to make the source code Metallos publicly available upon publication of the paper, ensure the reproducibility of results, Also, the power grid and house energy datasets are openly accessible at (Mulla 2018) and (Candanedo et al. (2017). respectively, which will allow regenerating the results plots (c) and (d) Figures and
REFERENCES
IEEE guide for electric power distribution reliability indices. IEEE Std 1366-2012 (Revision of IEEE Std 1366-2003), pp. 1-43, 2012. doi: 10.1109/EEEETD SWI.HEEEEESDDD222222999333 2212.620938...
Ferran Alet, Erica Weng, Tomás Lozano Pérez, and Leslie Pack Kaelbling. Neural Relational Inferecce with Fast Modular Meta-Learning Curran Associates Inc., Red Hook, NY, USA, 2019.
Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Advances Neural Information Processing Systems (NIPS 2016), volume 29, pp. 3981-3989. NIPS Proceedings, 2016.
Jonathan T. Barron. A general and adaptive robust loss function, 2019.
Dario Bega, Marco Gramaglia, Marco Fiore, Albert Banchs, and Xavier Costa-Perez. Deepcog: Cognitive network management in sliced 5g networks with deep learning. In Proceedings of the IEEE Conference Computer Communications (INFOCOM), pp. 280-288, 2019.
Luis M. Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in a low-energy house. Energy and Buildings, 140:81-97, 2017. ISSN 0378-7788.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan. and Quoe V. Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning learn around a common mean. In Advances in Neural Information Processing Systems (NIPS 2018), volume 31. NIPS Proceedings. 2018
Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and Tie-Ya Ren Liu. Learning to teach. In International Conference on Learning Representations, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning. volume 70 of ICML'I7, pp. 1126-112 1666-1355 135. JMLR.org, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riecardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1563-1572. PMLR, July 2018
Boyan Gao, Henry Gouk, and Timothy M. Hospedales. Searching for robustness: Loss learning for noisy classification tasks, 2021.
Santiago Gonzalez and Risto Mikkulainen. Improved training speed, accuracy, and data utilization through loss function optimization. In 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1-8, 2020. doi: 10.1 109/CCC48006 2099CEC488000220099199 200.991877.......7....
10

---

Under review as a conference paper at ICLR 2022
Santiago Gonzalez and Risto Mikkulainen. Optimizing loss functions through multi-variate taylor polynomial parameterization. Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '21, pp. 305-313. Association for Computing Machinery, 2021.
Josif Grabocka, Randolf Scholz, and Lars Schmidt-Thieme Learning surrogate losses, 2019.
W.J. Harder. Key performance indicators for smart grids. Master's thesis, University of Twente, 2017.
A. Ali Heydari, Craig A. Thompson, and Asif Mehmood. Softadapt: Techniques for adaptive loss weighting neural networks with multi-part loss functions, 2019.
M. Hospedales, A. Antoniou, Micaelli, and J. Storkey. Meta-learning in neural networks: A survey, IEEE Transactions Pattern Analysis & Machine Intelligence, (01):1-1, may 2021. ISSN 1939-3539, doi: 10. 1000TTAAM.2222.3079999920
Chen Huang. Shuangfei Zhai, Walter Talbott, Miguel Ángel Bautista, Shih- Yu Sun, Carlos Guestrin, and Joshua M. Susskind. Addressing the loss-metric mismatch with adaptive loss alignment. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of International Conference Machine Learning. ICML, volume 97 of Proceedings of Machine Learning Research, pp.
2891-2900. PMLR, June 2019.
Chen Huang, Shuangfei Zhai, Pengsheng Guo, and Josh M. Susskind. Metricopt: Learning to optimize black-box evaluation metries. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 174-183. Computer Vision Foundation / IEEE, 2021.
Qijia Jiang, Olaoluwa Adigun, Harikrishna Narasimhan, Mahdi Milani Fard, and Maya Gupta. Optimizing black-box metrics with adaptive surrogates, 2020.
Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang. and Xizhou Zhu. Autoloss-zero: Searching loss functions from scratch for generic tasks, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.
Qingliang Liu and Jinmei Lai. Stochastic loss function. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04))48844881 Apr. 2020.
Francesco Marchetti, Sabrina Guastavino, Michele Piana, and Cristina Campi. Score-oriented loss (sol) functions, 2021
Cristina Marquez, Marco Gramaglia, Marco Fiore, Albert Banchs, Cezary Ziemlicki, and Zbigniew Smoreda. Not all apps created equal: Analysis of spatiotemporal heterogeneity in nationwide mobile service usage. In Proceedings of the 13th International Conference on Emerging Networking EXperiments and Techmologies, CONEXT 17, pp. 180-186. Association for Computing Machinery, 2017. |Rt
Paul Micaelli and Amos Storkey. Non-greedy gradient- based hyperparameter optimization over long horizons, 2020.
Rob Mulla. Pjm hourly energy consumption data (version 3). 2018. URL hwt.ttttts//wwwkkktttt com/ robi wtabbikscbe//hhurryyyyrre.e www.oooollleeeeee--oooooooeoooee www.rrayyccooouu...ooomm
Enrique Personal, Juan Ignacio Guerrero, Antonio Garcia, Manuel Pena, and Carlos Leon. Key
performance indicators: A useful tool assess smart grid goals. Energy, 76:976-988, 2014, ISSN 0360-5442.
Jun Shu, Qian Zhao, Keyu Chen, Zongben Xu, and Deyu Meng. Learning adaptive loss for robust learning with noisy labels. arXiv preprint arXiv:2002.06482. 2020.
Leslie N Smith. Cyclical learning rates for training neural networks. In 20/7 IEEE winter conference on applications of computer vision (WACV), pp. 464-472. IEEE, 2017.
Flood Sung, Li Zhang, Tao Xiang, Timothy M. Hospedales, and Yongxin Yang. Learning to learn: Meta-critic networks for sample efficient learning. arxiv, abs/1706.000 706.09529, 2017.
11

---

Under review as a conference paper at ICLR 2022
Tongzhou Wang, Jun- Jun Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2020.
Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Jianhuang Lai, and Hie-aan Liu. Learning to teach with dynamic loss functions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 6467-6478, Red Hook, NY, USA, 2018. Curran Associates Inc.
Haowen Xu, Hao Zhang, Zhiting Hu, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Autoloss: Learning discrete schedule for alternate optimization. In International Conference on Learning Representations, 2019.
Sen Zhao, Mahdi Milani Fard, Harikrishna Narasimhan, and Maya Gupta. Metric-optimized example weights. In International Conference on Machine Learning, pp. 7533-7542. PMLR, 2019.
Wei Zhou, Yiying Li, Yongxin Yang, Huaimin Wang, and Timothy M. Hospedales. Online meta-critic learning for off-policy actor-critic methods, 2020.
12


Reviews:
[{"review_id": "a10ALsndQGL", "reviewer": "Reviewer_5PNy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\nConversion of the non-differentiable loss function to a differentiable loss function has been a tough challenge. The authors have proposed a meta-learning of loss approach for this task. Their proposal has been clearly explained and well supported through experiments and discussion. \n\n# Weaknesses \n\n(i) The paper could have been written well for readability. I find that the wording was quite confusing at times, thereby breaking the flow.\n\n(ii) Since “loss exploration” and “co-training” is claimed as a novelty I think there need to be some experiments displaying their effect for when they are or aren’t used. It might be that these ideas haven’t been explored in the “meta-learning of loss” community. However, it still warrants more discussion around what are the benefits these proposals give. E.g. \n\n- How does $\\epsilon$ help the learning task? Perhaps, further experiments can be conducted to help understand how $\\epsilon$ is helping achieve “loss exploration”. What happens when there is no “loss exploration”? How does MAE compare to MetaLoss in the presence or absence of this noise? Does $\\epsilon$ depend on your dataset?\n- Co-training is more understandable, in my opinion. However, it might be useful to show the effects of CLR or no-CLR on the performance of MetaLoss. For example, how does the metric performance vary with different schedules of CLR (or no-CLR)? \n\n(iii) *MetaLoss succeeds in learning more than the teacher knows* - I find this sentence/claim to be misleading/confusing. Consider, for example,  (A) MAE loss and (B) loss-learned NN. (A) knows nothing about data, while (B) adapts itself to the data and the predictor. Thus, \"teacher knows\" only applies to (B) while, if I understand correctly, it is meant to refer to (A), thereby causing the whole confusion. (B) uses (A) as a meta-teacher and adapts itself to help the predictor perform well on (A). \n\n(iv) Are there examples of plain regression tasks (not time-series forecasting) on which MetaLoss's performance can be demonstrated? The authors have presented this model as generically applicable to regression, however, the experiments are conducted on time-series forecasting tasks. It might be useful for the readers to understand the effect of MetaLoss on vanilla regression tasks. \n\n*Clarifications* - \n- How are the points for Loss Perceived in figures obtained for MetaLoss? How are y_t and \\hat{y}_{t} decided for the loss-learning DNN?\n \n\n*Minor Comments*\n- Figure 2: typo - “loos” \n- “where the loss learned by MetaLoss has learned and evolved during training “ - confusing phrasing of the sentence\n- Figure 2 is quite confusing. It talks about evolution, but there is no notion of time. Also, the y-axis is not well explained, e.g., “Perceived loss”. \n- If “Overdimensioning” is the same as overestimation, it might be helpful to use this term as it's more familiar to the readers. \n- Figure 3(d) labels are not clear\n\n", "summary_of_the_paper": "This work proposes meta-learning of loss functions for regression tasks. This approach has powerful applications for cases when the performance metrics are non-differentiable. Such cases often arise in tasks where the cost of misprediction is asymmetric and ripe with conditional clauses. The authors have proposed a simple yet powerful two-step process, wherein two neural networks are involved such that (a) a loss-learning NN learns an approximate differentiable loss function that mimics the non-differentiable performance metric, and (b) predictor NN that aims to perform well on the loss function defined by this loss-learning NN. The authors specify two crucial approaches to make this proposal work - (a) loss exploration, and (b) co-training of two neural networks. Their proposal is well supported by the experiments on time-series prediction tasks. ", "summary_of_the_review": "The model proposed by the authors seems to have powerful use cases. Their proposal has been well explained and supported by experiments. I would love to see some more experiments/discussions around some of their claims. I am happy to reconsider my scores thereafter. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JRmpTiVAFEn", "reviewer": "Reviewer_m1fA", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, this paper has the following three major problems.\n\n1. Unlike the vast majority of meta-learning approaches where the outer-learning objective is set to the validation or test loss, In this paper the outer learning objective simply aim to minimize the gap from the existing loss function. This seems very weird to me. The goal of loss learning is to outperform the existing loss functions such as MAE, but they propose to minimize the gap between MAE and their loss learning module. How does it make sense? Did the authors justify this choice in the paper rigorously? I think this is why they introduce injecting noise during the inner- and outer-learning process to make the loss learning non-trivial, but it also seems very ad-hoc and no intuition has been provided as to why doing so. This is very a significant methodological concern which requires thorough explanations.\n\n2. In section 3.4, they argue that their method of co-training predictor and loss is novel, but this is completely wrong. As far as I understood, this \"co-training\" is almost identical to one-step lookahead approach (i.e. T1-T2) or online first-order approximation widely used in many meta-learning and gradient-based hyperparameter optimization literatures. There exists tones of papers doing so, but they fail to recognize it. The authors need to cite them and clearly specify what is different from those existing literatures.\n\n3. Lastly, there is just too little baselines. At least, the authors should have compared against MSE, MSLE, and other existing pre-defined loss functions. The authors also should justify other regularizers such as dropout or smoothing techniques in order to justify that the performance improvement is not simply due to such randomness. Most importantly, the performance improvement is very marginal. I guess this is why they also report what they call \"Gain(%)\", which also seems very weird to me. Is it a standard way of reporting performance in other literatures?", "summary_of_the_paper": "This paper proposes a loss learning framework for the timeseries forecasting regression problem. They introduce two learning blocks, one for producing the next step prediction and the other for learning the loss function. The experimental results show that the proposed method slightly increase the performance compared to MAE.", "summary_of_the_review": "In my view, this paper is well below the acceptance threshold and more literature survey seems required.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "BgZDwfI2wl", "reviewer": "Reviewer_qQ83", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The meta-learning problem studied in this work is to learn the loss function used in a time-series forecasting method. An intuitive overview figure that succinctly illustrates the problem would be extremely useful for the reader in Section 3.1 of the problem formulation. The symbol and terms denoted in the first sentence of the problem formulation are never used in the actual problem, “Let us denote the space of system state variables as S.”\n\nThe evaluation of the method in Section 4 is not convincing. To demonstrate the effectiveness of the approach, it is required to compare to the state-of-the-art methods across all datasets. However, the results provided in Table 1 of this paper uses only a single very simple baseline for comparison. To be able to understand the advantage of this method, MetaLoss should be compared more fairly with state-of-the-art methods. There are also other issues. For instance, the training runtime is also not shown, which would obviously be much larger for MetaLoss.\n\nResults are also for a single selection of hyperparameters, which includes number and type of layers, number of neurons per layer, predictor learning rate, number of epochs, and activation functions, and so on. It would have been useful to see an ablation study where many of these are varied and the performance is shown. Currently, I didn’t even notice where the actual values used in the one experiment are given, so it seems impossible to even reproduce the simple result shown in Table 1. Nevertheless, these need to be stated, and an ablation study showing performances for a variety of settings need to be provided, otherwise, it is impossible to know whether the method works for only one specific selection of hyperparameters, or if it works better for other possible settings, and when does it work well and when does it fail. Currently, we only see results for one specific setting from this enormous space of hyperparameters (which are also seemingly missing from the paper). This is hardly proof that the method works well, or is even useful. It is probably easy to find a selection of hyperparameters where it gives worse results, given the huge space of possibilities. This appears to be a major flaw in the evaluation.\nRuntime performance is also crucial. In many applications, having a faster approach may be more important than performance, and thus may be willing to trade-off a minor decrease in performance for a significantly faster approach. \n\nThe writing of the paper also requires significant improvement, as various parts are unclear. The problem is not well motivated, and the approach can also be explained significantly better with more concrete and succinct examples.\n\nTypo in first paragraph of Section 4, “… to reduce randomness as most as possible.”\nAlso, in the caption of Figure 2, “MAE loos”. Some parts of Figure 2 are also not explained.\n\nThe paper does a poor job on reproducibility, as mentioned above. The hyperparameters used in the one experiment are not provided, nor is it explained why these are chosen, etc. Source code is also not available, and the majority of the data (the 4 traffic datasets) appear to also be private (as far as I can tell). Hence, the first sentence of the reproducibility statement itself seems incorrect, “… so as to ensure the reproducibility of our results”\n\nThere are some terms used in the paper incorrectly or in a misleading way, e.g., “we prove” on page 1. Many of these sentences also are somewhat overclaiming due to the above issue of showing results for a single hyperparameter selection, which may have been difficult to find and cherry picked. It is unclear if the approach actually works better for other selections, even against this simple baseline. \n\n", "summary_of_the_paper": "This paper focuses on meta-learning of loss functions for time-series forecasting, where the loss function to be used to train the actual model is learned. For this problem, they propose MetaLoss, which performs joint co-training of a regressor network and of the loss-learning networks. They show under one specific set of hyperparameters and for a single simple baseline method, that it can work well. They also apply it for a few different applications. The idea seems interesting. However, the actual contribution of this work, and effectiveness of the approach is unclear due to the issues discussed in detail below.", "summary_of_the_review": "While the idea of MetaLoss seems interesting, the evaluation and comparison of it is not convincing and needs significant work. For instance, instead of comparing a single simple baseline to MetaLoss, the approach needs to be compared to the state-of-the-art forecasting methods (e.g., NBEATS, Deep Factors, Graph Deep Factors, DeepAR, and so on…). The authors should also include experiments showing the runtime performance across all these baseline methods and datasets. Such results are omitted in this work, though are crucial. \n\nThe exact hyperparameters used are also seemingly missing from the paper. But most importantly, the results shown in the paper are for a single set of hyperparameters, and therefore, it is unclear if the model works better for the specific hyperparameters chosen, or if it routinely outperforms the other across a wide variety of hyperparameter selections. In other words, the results shown are not significant due to this issue. It is likely easy to find a selection of hyperparameters where the proposed approach performs poorly compared to even the simple baseline shown. \n\nFurthermore, the writing can also be significantly improved including the introduction, problem formulation, and motivation and importance of the problem. Overall, this paper needs significant work before it can be accepted. See above for other important issues that need to be addressed as well.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "5KVGQ8ycUmh", "reviewer": "Reviewer_9TL4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- The motivation of this work is clear and the authors have covered the relevant literature adequately.\n\nWeaknesses \n\n-  Authors claim that the proposed MetaLoss is a loss-function agnostic framework (Page 3), however in the experiments, they have only used MAE as the loss function in the loss-learning block to evaluate the model. To claim MetaLoss is loss-function agnostic, authors must evaluate this framework using multiple loss functions such as MSE, Huber loss, Tweedie loss (can evaluated using intermittent time series data). \n\n- Moreover, authors have used only one prediction architecture (DNN) to train with the MetaLoss. This raises doubts over the extensibility of the proposed framework, whether MetaLoss can be used to train with other prediction architectures such as RNN, CNN, and tree based regressors such as LightGBM. This because these architectures are currently being heavily used in the time series forecasting literature; therefore, to assess the aptness of this method in the time series forecasting context, it is important to evaluate the proposed Metaloss method with these architectures. If these evaluations can’t be done, authors should clearly explain the limitations of this study (why these experiments can’t be done). Moreover, authors have not mentioned about the hyper-parameter ranges used to train the DNN and nor the procedure used to fine tune the hyper-parameters (at least using an appendix)\n\n- In the experiments, authors mention that they generate one-step forecasts for each time series. Does that mean you generate one step ahead forecasts, i.e., forecast horizon is equivalent to 1, What is the reason for that? Generally, in time series forecasting, we generate multi-step ahead forecasts, i.e., for multiple horizons. Also, the benchmark datasets used in the experiments are also not commonly used datasets in the forecasting literature (check [1] for more commonly used datasets) \n\n-  Another concern is that the study uses MAE as the primary loss function in the experiments and then evaluate the accuracy of the generated forecasts using MAE error measure again. What is the reason for this? This is not the general approach used in time series forecasting, because once the machine learning models are trained using the loss functions such as MAE, MSE, we use proper forecasting error measures such as sMAPE, MASE to evaluate the forecasts. Moreover, authors also do not show whether the result differences are statistically significant (showing the percentage of improvement is not adequate)\n\n-  My biggest concern is about the computational cost of the proposed framework. This study does not report the overall running time of the MetaLoss and compare it against the classical fixed learning-based models. Without the running times, it is quite difficult to conclude the usability of this framework for more data extensive applications. \n\n[1] https://forecastingdata.org/ \n", "summary_of_the_paper": "The study proposes MetaLoss, a loss meta-learning approach that is capable of autonomously optimising the properties of traditional loss functions (e.g., MAE, MSE) to a target task. The proposed framework has improved the regression accuracy in multiple time series datasets, outperforming classical fixed learning-based models, which do not employ any learning procedure to train a target loss function. The motivation of this work is clear and sound; nevertheless, I have multiple concerns about the experimental setup of this work.", "summary_of_the_review": "Even though the motivation of this work is clear, I have multiple concerns about the scientific rigour of the experimental setup (see the review section). Based on the current status of the manuscript, I am recommending to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The meta-learning framework based on learning the loss function for time series forecasting is an interesting and important topic. However, the reviewers think the literature, baselines, and experimental results need significant improvement.
Decision: Reject