It Is Likely That Your Loss Should be a Likelihood
Authors: Mark Hamilton
Evan Shelhamer
William T. Freeman
Venue:  (2021)
Keywords: Adaptive Losses
Outlier Detection
Adaptive Regularization
Recalibration
Robust Modelling

Abstract: Many common loss functions such as mean-squared-error, cross-entropy, and reconstruction loss are unnecessarily rigid. Under a probabilistic interpretation, these common losses correspond to distributions with fixed shapes and scales. We instead argue for optimizing full likelihoods that include parameters like the normal variance and softmax temperature. Joint optimization of these ``likelihood parameters'' with model parameters can adaptively tune the scales and shapes of losses in addition to the strength of regularization. We explore and systematically evaluate how to parameterize and apply likelihood parameters for robust modeling, outlier-detection, and re-calibration. Additionally, we propose adaptively tuning $L_2$ and $L_1$ weights by fitting the scale parameters of normal and Laplace priors and introduce more flexible element-wise regularizers.

Under review as a conference paper at ICLR 2021
IT Is LIKELY THAT YOUR Loss SHOULD BE A LIKELIHOOD
Mark Hamilton MIT. Microsoft
Evan Shelhamer MIT, Adobe Research she whwaaarerrr adobe. con
William Freeman MIT, Google billfemit.cm edu
markthêmit edu
ABSTRACT
Many common loss functions such as sean-squared-error, cross-entropy, and re- construction loss unnecessarily rigid. Under probabilistic interpretation. these common losses correspond to distributions with fixed shapes and scales. We instead argue for optimizing full likelihoods that include parameters like the normal variance and softmax temperature. Joint optimization of these "likelihood parameters" with model parameters adaptively tune the scales and shapes of losses in addition to the strength of regularization. We explore and systematically evaluate how parameterize and apply likelihood parameters for robust mod- eling, outlier-detection tuning L2 and L1 weights by fitting the scale parameters of normal and Laplace priors and introduce more flexible element-wise regularizers.
and re-calibration. Additionally, we propose adaptively
INTRODUCTION
Choosing the right loss matters. Many common losses arise from likelihoods, such as the squared emror loss from the normal distribution absolute error from the Laplace distribution, and the cross entropy loss from the softmax distribution. The same true of regularizers, where L2 arises from normal prior and L1 from a Laplace prior.
Deriving losses from likelihoods recasts the problem as a choice of distribution which allows data- dependent adaptation. Standard losses and regularizers implicitly fix key distribution parameters, limiting flexibility. For instance, the squared error corresponds to fixing the normal variance at constant. The full normal likelihood retains its scale parameter and allows optimization a parametrized set of distributions. This work examines how to jointly optimize distribution and model parameters to select losses and regularizers that encourage generalization, calibration, and robustness to outliers. We explore three key likelihoods: the normal, softmax, and the robust re- gression likelihood p of Barron (2019). Additionally, we adaptive priors in the same light and introduce adaptive regularizers. Our contributions:
1. We systematically survey and evaluate global, data, and predicted likelihood parameters and introduce a new self-tuning variant of the robust adaptive loss
2. We apply likelihood parameters to create new classes of robust models, outlier detectors, and re-calibrators.
3. We propose adaptive versions of L1 and L2 regularization using parameterized normal and Laplace priors on
model parameters.
2 BACKGROUND
Notation We consider a dataset DD of points and targets y; indexed by i E (1....,N] Targets for regression are real numbers and targets for classification are one-hot vectors. The model f with parameters / makes predictions given the target. To learn model parameters we solve the following loss optimization:
fz(x). A loss L(ū, y) measures the quality of the prediction
min E (n
L(ğ = (x()),
(1)

---

Under review as a conference paper at ICLR 2021
0.6
8
dit-1)
0.4
d1 4]
Dattein is belter
0.3
100 21
600 41
0.0
0.0 -10
-5
-10
10
(a) The Normal PDF and NLL
(b) The Softmax CDF and NLL
Figure 1: Optimizing likelihood parameters adapts the loss without manual hyperparameter tuning
to balance accuracy and certainty.
A likelihood C(ply, measures the quality of the prediction as distribution over 9 given the target and likelihood parameters ø. We use the negative oog-likelihood & (NLL), and the likelihood interchangeably since both have the same optima. We define the full likelihood optimization:
min E llU fo())ly. 0,0 (a.y)~
to jointly learn model and likelihood parameters. "Full" indicates the inclusion of Φ, which controls the distribution and induced NLL loss. We focus on full likelihood optimization this work. We note that the target, y. the only supervision needed to optimize model and likelihood parameters, 0 and & respectively. Additionally, though the shape and scale varies with Φ, reducing the error U always reduces the NLL for our distributions.
Distributions Under Investigation This work considers the normal likelihood with variance (Bishop et al., 2006; Hastie et al., 2009), the softmax likelihood with temperature (Hinton al., 2015), and the robust likelihood p (Barron, 2019) with shape and scale that control the scale and shape of the likelihood. The first two are among the most common losses in machine learning, and the last loss provides an important illustration of "scale". We note that changing the scale and shape of the likelihood distribution is not "cheating" as there is a trade-off between uncertainty and credit. Figure shows how this trade-off affects the Normal and softmax distributions and their NLL.s.
likelihood parameter that affects "shape" instead
The normal likelihood has terms for the residual ÿ and the variance as
N(Wl,, o) (2 ) exp (- (ú (0- a²
with a E (0, oo) scaling the distribution. The normal NLL can be written fxr (0-) log a,, after simplifying and omitting constants that do not affect minimization. We recover the squared error by substituting a=
The softmax defines categorical distribution defined by scores 2 for each class
6O0
softmax(j = vl,,7)
(4)
E. g
with the temperature, E (0, oo), adjusting the entropy of the distribution. We recover the classifi- cation cross cross-entropy loss, - log plū = y). by substituting T = the respective NLL. We state the gradients of these likelihoods with respect their and in Section A of the supplement.
The robust loss and its likelihood are
%/2
2a-
(#/o)² a
p (1 a
+
- 1
and
(5)
- 2
p (0),,0,0)=
oZ (a) exp (-p(ū- a. 0)),
(6)
2

---

Under review as a conference paper at ICLR 2021
with shape E [0, 00), scale E (0,o0), and normalization function Z (a). This robust loss, Ps
has the interesting property that it generalizes several different loss functions commonly used in robust learning such as the L.2 loss (a 2). pseudo- loss (Charbonnier et al., 1997) = 1), Cauchy loss (Li et al., 2018) (a = 0). Geman- McClure loss (Ganan & McClure, 1985), (a = 2),
and Welsch (Dennis Jr & Welsch, 1978) loss (alpha = -oo). Learning the shape parameter allows models to adapt the shape of their noise distribution.
3 RELATED WORK
Likelihood optimization follows from maximum likelihood estimation (Hastic et al., 2009; Bishop et al., 2006), yet is uncommon in practice for fitting deep regressors and classifiers for discriminative tasks. However Kendall & Gal (2017); Kendall et al. (2018); Barron (2019); Saxena et al. (2019) optimize likelihood parameters to their advantage yet differ in their tasks, likelihoods, and param- eterizations. In this work we aim to systematically experiment, clarify usage, and encourage their wider adoption.
Early work on regressing means and variances (Nix & Weigend, 1994) had the key insight that optimizing the full likelihood can fit these parameters and adapt the loss. Some recent works use likelihoods for loss adaptation, and interpret their parameters as the uncertainty (Kendall & Gal, 2017: Kendall et al., 2018), robustness (Kendall Gal, 2017: Barron, 2019; Saxena et al., 2019), and curricula (Saxena et al., 2019) of losses. MacKay & Mac Kay (2003) uses Bayesian evidence to select hyper-parameters and losses based on proper likelihood normalization. Barron (2019) define generalized robust regression loss, P. to jointly optimize the type and degree of robustness with global. data-independent, parameters. Kendall & Gal (2017) predict variances for regression and classification to handle data-dependent uncertainty. Kendall et al. (2018) balance multi-task loss weights by optimizing variances for regression and temperatures for classification. These global parameters depend on the task but not the data, and are interpreted Saxena et al. (2019) define a differentiable curriculum for classification by assigning each training point its own temperature. These data parameters depend on the index of the data but not its value. We compare these different likelihood parameterizations across tasks and distributions.
inherent task uncertainty.
In the calibration literature, Guo al. (2017) have found that deep networks often miscalibrated, but they can be re-calibrated by cross-validating the temperature of the softmax. In this work we explore several generalizations of this concept. Alternatively, Platt scaling (Platt, 1999) fits a sig- moid regressor to model predictions to calibrate probabilities. Kuleshov et al. (2018) re- re-calibrate regressors by fitting an Isotonic regressor the empirical cumulative distribution function.
LIKELIHOOD PARAMETER TYPES
We explore the space of likelihood parameter representations for model optimization and inference. Though we note that some losses, like adversarial losses, are difficult to represent as likelihoods, many different losses in the community have natural probabilistic interpretation. Often, these probabilistic interpretations can be parametrized in variety of ways. We explore two key axes of generality when building these loss functions: conditioning and dimensionality.
Conditioning We represent the likelihood parameters by three functional classes: global, data, and predicted. Global parameters, á c, are independent of the data and model and define the same likelihood distribution for all points. Data parameters, di are conditioned on the index, i, of the data, 24. but not its value. Every training point is assigned an independent likelihood parameter, Φ; that define different likelihoods for each training point. Predicted parameters, ((x) 9,, are determined by a model, 9. with parameters 7) (not to be confused with the task model parameters 8). Global and predicted parameters can be used during training and testing, but data parameters only assigned to each training point and are undefined for testing. We show a simple example of predicted temperature in Figure 4, and an illustration of the parameter types in Figure 2.
We note that for certain global parameters like a learned Normal scale, changing the scale does not affect the optima, but does change the probabilistic interpretation. This invariance has led many authors to drop the scale from their formulations. However, when models can predict these scale parameters they naturally remain calibrated in the presence of heteroskedasticity and outliers.
3

---

Under review as a conference paper at ICLR 2021
ddtaa (1)
preed h
y
1V,P,o.
sof foftmax(2,0 1)
hxw cxhXw
Z
Fglobal
Figure 2: Illustration of an image classifier with three different types of likelihood temperature conditioning: global. predicted, and data. Each represents different way parametrize the model's temperature.
Figure 3: An image loss function with three different likelihood parameter dimensionalitiiss Each represents possible way to parametrize the additional scale parameter added to the loss.
Classification without Temperature
Classification with Temperature
Temperature Regression
0.96
0.92
0.82
0.84
40
0.72
0.72
35
0.62
30
0.52
0.48
0.42
20
0.36
0.32
15
0.22
0.12
0.02
0.00
X1
X1
X1
Figure 4: A synthetic logistic regression experiment. Regressing softmax temperature reduces the influence of outliers (blue, bottom-left), by locally raising temperature. The jointly optimized model (center and right panel) achieves more accurate classification that model trained without adaptive lemperature (left panel).
Additionally we note that for the shape parameter of the robust likelihood, P. changing global pa- rameters does affect model fitting. Previous works have adapted global softmax temperature for model distillation (Hinton et al., 2015), and recalibration (Guo et al., 2017). Barron (2019) also experiments with global values of loss function shape and scale parameters. The main work on Data parameters is that of Saxena et al. (2019) who use these to learn curriculum. Model-based param- eters appear in earlier work on regressing variance (Nix & Weigend, 1994), and more recent work by Kendall & Gal (2017).
The dimensionality, (ø), of likelihood parameters can vary with the dimension of the task prediction, ÿ. For example, image regressors can use single likelihood parameter for Dimensionality
each image |ő| = 1, RGB image channel |+] C. or even every pixel (ő) = W H x C' as in Figure 3. These choices correspond to different likelihood distribution classes. Dimensionality and Conditioning
likelihood parameters can interact. For example, data parameters with |o] W x H x C' would result in N x W x H. x C' additional parameters, where N is the size of the dataset. This can complicate implementations and slow down optimization due to disk I/O when their size exceeds memory. Table 5 in the appendix contrasts the computational requirements of different likelihood parameter types. The work Barron (2019) explores both scalar and pixel-wiss -iice dimensionalities for his robust loss.
APPLICATIONS
4

---

Under review as a conference paper at ICLR 2021
Table 1: MSE, Time, and Memory increase (compared standard normal likelihood) for reconstruc- tion by variational auto-encoders with different parameterizations of the robust loss, p. Predicted likelihood parameters yield more accurate reconstruction models.
Param.
Dim
MSE
Time
Mem
Global
1x1x1
225.8
1.04x
1KB
Data
1xlx
244.2
2.70x
0.6GB
Pred.
Lxlx1
228.5
1.04x
< 1MB
Global
HxWxC
231.1
1.08x
< <IB
Data
HxWxC
252.6
9.42x
4.4GB
Pred.
HxWxC
222.3
1.08x
< <MB
5.1 ROBUSTNESS AND OUTLIER DETECTION
Data in the wild is noisy, and machine learning methods should be robust to noise, heteroskedastic- ity, and corruption. Unfortunately, models trained with the standard mean squared error (MSE) loss are highly susceptible outliers, and cannot naturally handle heteroskedasticity due to this loss' fixed variance (Huber, 2004). Allowing models to predict and optimize their likelihood parameters allows models to generalize to these more complex settings. More specifically, likelihood parame- ters naturally transform standard methods such as regressors, classifiers, and manifold learners into robust variants without expensive outer-loop of model fitting such as RANSAC (Fischler & Bolles, 1981) and Theil-Sen (Theil, 1992). Figure 4 demonstrates this effect with a simple classification dataset, and we point readers to Figures of the Supplement for similar examples for regression and manifold learning.
In certain datasets, even the assumption of Gaussianity is too restrictive and one must consider more robust and long-tailed distributions. This has led many to investigate broader classes of likelihoods such as Generalized Linear Models (GLMs) (Nelder & Wedderburn, 1972) or the more recent gen- eral robust loss, p. of (Barron, 2019). To systematically explore how likelihood parameter dimen- sion and conditioning affect model robustness and quality, we reproduce Barron (2019)'s variational auto-encoding (Kingma & Ba, 2015) (VAE) experiments
faces from the CelebA dataset (Liu et al., 2015) in Table 1. We explore learned data (Saxena et al., 2019) and model parameters in addi- tion Barron's 's learned global parameters. We also include two natural parameter dimensionalities: a single set of parameters for the whole image, and set of parameters for each pixel and chan- nel. We find that predicted parameters achieve the best performance while maintaining fast training time and small memory footprint. We also find that pixel-wise learned parameters correlate with challenging areas of images and we visualize these parameters in Section of the Appendix.
This experiment uses X convolution on the last hidden layer of the decoder parameter model and has the same resolution as the output. The low and high dimensional losses use the same convolutional regressor, but the 1 dimensional case averages over pixels. In the high di- mensional case, the output has three channels (for RGB), with six channels total for shape and scale regression. We use the same con-lineasssies constrain the shape and scale outputs reasonable ranges as in (Barron, 2019). More specifically, we use an affine sigmoid to keep the shape a E [o. 3] and the softplus to keep scale E [10- 8, oo). Table 1 gives the results of evaluating each method by MSE on the validation set, while training each method with their respective loss parameters. Data parameter optimization uses Tensorflow's implementation of sparse RMSProp (Tieleman & Hinton, 2012). We also inherit weight decay || gradient clipping V / V 12g and learning rate scaling m for learning rate a and multiplier m from Barron (2019).
likelihood
The robustness we see in our VAE experiments stems from the fact that likelihood parameter predic- tion gives models a direct channel to express their "uncertainty" for each data-point with respect to the task. This allows models to naturally down-weight and clean outliers from the dataset which can improve model robustness. Consequently, one harness this effect to create outlier detectors from any underlying model architecture by using learned scales or temperatures as an outlier score func- tion. Furthermore, predicted likelihood parameters allow these methods to detect outliers in unseen data. In Figure 5 we show how auditing temperature or noise parameters can help practitioners spot erroneous labels and poor quality examples. In particular, the model-parametee
temperatures of
5

---

Under review as a conference paper at ICLR 2021
0.011
0.012
t=0.012
t=0.013
t=0.013
to 0.0.313
t.0.014
0.04
t=0.014
t- 0.05
88
12622
62(16155
15
2
7
5
6
3276.0
582.0
t- 380.0
376.0
t- 344.0
t- 300.0
t- 248.0
t 165.0
149.0
139.0
57
332
50
Figure 5:
The data with the lowest (top) and highest (bottom) predicted temperatures in the SVHN dataset. High temperature entries are blurry, cropped poorly, and generally difficult to classify.
Median outlier detection performance of several methods across 22 benchmark datasets from ODDS.
Table 2:
1.0
0.8
Method LOF
Median AUC
.669
AUC
0.6
FB
.702
ABOD
.727
0.4
AE
.737
VAE
.792
0.2
COPOD
.799
PCA
808
0.0
OCSVM
.814
FB
ABOD
AE
VAE
COPOD
PCA
OCSVM
MCD
KNN
HBOS
CBLOF
AE+S
PCA+S
MCD
820
KNN
.822
Model
HBOS
.822
Figure 6: Distribution of Outlier Detection AUC across the ODDS Benchmark. Our approaches, PCA+S and AE+S, are competitive with other Outlier Detection systems.
IF
.823
CBLOF
.836
AE+S (Ours)
8466
PCA+S (Ours)
.868
an image classifier (trained using the setup of 5.3) correlates strongly with blurry, dark, and diffi- cult examples on the Street View House Number (SVHN) dataset. We use this approach create simple outlier detection algorithms by considering deep (AE+S) and linear (PCA+S) auto-encoders (Kramer, 1991) with data-conditioned scale parameters outlier scores. We evaluate this approach on tabular datasets using deep and linear auto-encoue encoders with paadel-aramettrizee scales. In Table 2 we quantitatively demonstrate the quality of these simple likelihood parameter approaches across 22 datasets from the Outlier Detection Datasets (ODDS), a standard outlier detection benchmark (Rayana, 2016). The ODDS benchmark supplies ground truth outlier labels for each dataset, which allows one treat outlier detection as an unsupervised classification problem. We compare against a variety of established outlier detection approaches included in the pyOD (Zhao et al., 2019) frame- work including: One-Class SVMs (OCSVM) (Schölkopf et al., 2000), Local Outlier Fraction (LOF) (Breunig et al., 2000), Angle Based Outlier Detection (ABOD) (Kriegel et al., 2008), Feature Bag- ging (FB) (Lazarevic & Kumar, 2005), Auto Encoder Distance (AE) (Aggarwal, 2015), K-Ncarest Neighbors (KNN) (Ramaswamy et al., 2000; Angiulli& & Pizzuti, 2002), Copula Based Outlier Detec- tion (COPOD) (Li et al., 2020), Variational AutoEncoders (VAE) (Kingma & Welling, 2013), Mini- mum Covariance Determinants with Mahlanohbis Distance (MCD) (Rousseeuw & Driessen, 1999; Hardin & Rocke, 2004), Histogram-be based Outlier Scores (HBOS) (Goldstein & Dengel, 2012), Prin- cipal Component Analysis (PCA) (Shyu et al., 2003), Isolation Forests (IF) (Liu al., 2008; 2012), and the Clustering Based Local Outlier Factor (CBLOF) (He et al., 2003).
Our predicted scale auto-encoders use PyTorch's 's layers API (Paszke et al., 2019) with rectified linear unit (ReLU) activations for deep auto-encoders and Glorot uniform initialization (Dahl et al., 2013; Glorot & Bengio. 2010) for all layers. We use Adam (Kingma & Ba. 2015) with a learning rate of

---

Under review as a conference paper at ICLR 2021
W
147
m
18
137
187
12
Figure 7: Performance of L2 (left) and L1 (middle) regularized linear regression 500 dimen- sional synthetic dataset where the true parameters, w", are known. Dynamic Ridge (D-Ridge) and D-LASSO regression find the regularization strength that best estimates the true parameters. M-LASSO outperforms any single global regularization strength and does not shrink informative weights. (right) Performance of adaptive L1 regularization methods as function of true model sparsity. In all cases, Multi-LASSO outperforms other methods by orders of magnitude.
10055 for 4000 steps with 20% dropout before the code space. We follow ODDS guidelines and standard scale the data prior fitting.
Our methods (PCA+S and AE+S) use similar principle as isolation-basee approaches that deter- mine outliers based on how difficult they are to model. In existing approaches, outliers influence and skew the isolation model which causes the model exhibit less confidence the whole. This hurts a model's s ability to distinguish between inliers and outliers. In contrast, our approach allows the un- derlying model to nown-weight outliers. This yields boundary between outliers and inliers as shown in Figure 4. As a future direction of investigation we agnostic, and can be combined with domain -peaif-
consistent model with clearer decision
note that our approach is architectures to create outlier detection methods tailored to images, text, and audio.
5.2 ADAPTIVE REGULARIZATIO WITH PRIOR PARAMETERS
In addition to optimizing the shape and scale of the likelihood distribution of the model output, use the same approach to optimize the prior distribution of the model parameters. More specifically, we propose adaptive regularizers for a model's parameters, 0. This approach optimizes the distribution parameters of the prior, Oprior- to naturally tune the degree of regularization. In particular, the Normal (Ridge, L2) and Laplace (LASSO, LI) priors, with scale parameters a and regularize model parameters for small magnitude and sparsity respectively (Hastie et al., 2009). The degree of regularization, À E [0, co), is conventionally hyperparameter of the regularized loss function:
P
min Σ (=:= fo(w.) )1)²
Σ
10
We note that we cannot choose À by direct minimization because it admits a trivial minimum at O. In the linear case, one can select this weight efficiently using Least Angle Regression (Efron et al., 2004). However, in general À is usually learned through expensive cross validation methods. Instead, we retain the prior with its scale parameter, and jointly optimize over the full likelihood:
P. 20 (0r vi)² + log Σ
-(0,)
min Σ P.a
+ log
This approach, the Dynamic Lasso (D-LASSO), admits no trivial solution for the prior parameter b, and must balance the effective regularization strength, 1 with the normalization factor, log b. D-LASSO selects the degree of regularization by gradient descent, rather than expensive black-box search. In Figure (left) and (middle) we show that this approach, and its Ridge equivalent, yield ideal settings of the regularization strength suite of synthetic regression problems. Figure

---

Under review as a conference paper at ICLR 2021
(right) shows D-LASSO converges to the best LASSO regularization strength for variety of true- model sparsities. As a further extension, we replace the global a or b with a o, or by for each model parameter, Oj. to locally adapt regularization to each model weight (Multi-Lasso). This consistently outperforms any global setting of the regularization strength and shields important weights from undue shrinkage 7 (middle). For our experiments we use 500 samples of 500 dimensional normal distributions mapped through linear functions with additive gaussian noise. Linear transformations
each
use Uniform 1, 2] weights and LASSO experiments sparse transformations. We use tensorflow's s Adam optimizer with Ir .0055 for 100000 steps.
Our approach of learning regularizer scale parameters can be viewed naturally through the lens of hierarchical priors (Gelman al., 2013). More specifically this approach is implicitly performing maximum a posteriori (MAP) inference on the prior's scale with respect to a uniform prior on that parameter. We note that though these methods for hyperparameter selection are common in the Bayesian literature, they are not widely used in practice in the deep learning community. This work aams to bring these parameters back within the scope of deep learning where they can be easily expanded to more flexible forms such as our introduced Multi-Lasso.
5.3 RE-CALIBRATION
The work of (Guo et al., 2017) shows that modern networks are accurate, yet systematically over- confident, a phenomenon called mis-calibration. We investigate the role of optimizing likelihood parameters to re-calibrate models. More specifically, we can fit likelihood parameter regressors on a validation set to modify an existing model's confidence to better align with the validation set. This approach is a generalization of Guo et al. (2017)'s Temperature Scaling method, which we refer to as Global Scaling (GS) for notational consistency. Global Scaling re-calibrates classifiers with a learned global parameter, in the loss function: a (
Fitting model-conditioned likelihood parameters to a validation set defines broad class of re- calibration strategies. From these introduce three new re-calibration methods. Linear Scaling (LS) learns a linear mapping, I, to transform logits to a softmax temperature: (Z, Linear Feature Scaling (LFS) learns linear mapping, I, to transform the features prior to the logits. f. to softmax temperature: o (z ((f)). Finally, we introduce Deep Scaling (DS) for regressors which learns a nonlinear network, N, to transform features, f. into a temperature: (f)).
In Table 3 we compare our recalibration approaches to the previous state of the art: Global Scaling. We note that (Guo al., 2017) have already shown that Global Scaling outperform Bayesian Binning into Quantiles (Nacini et al., 2015). Histogram binning (Zadrozny & Elkan, 2001), and Isotonic Regression. We recalibrate both ResNet50 (He et al., 2016) and DenseNet121 (Huang et al., 2017) on a variety of vision datasets. We measure classifier miscalibration using the Expected Calibration Error (ECE) (Guo et al., 2017) to align with prior art. We additionally evaluate Isotonic recalibration, Platt Scaling (Platt, 1999). and Vector Scaling (VS) (Guo et al., 2017), which learns vector, V, to re-weight logits: o (UI, 1). LS and LFS tend to outperform other approaches like GS and VS, which demonstrates that richer likelihood parametrizations can improve calibration akin to how richer models can improve prediction.
Our experiments leverage Tensorflow's 's Dataset APIs that include the SVHN, (Netzer et al., 201 ImageNet (Deng et al., 2009), CIFAR- 100, CIFAR-10 (Krizhevsky, 2009) datasets. We use Keras implementations of DenseNet- 12 (Huang et al., 2017) and ResNet-50 (He et al., 2016) with default initializations. For optimization we use Adam with (r = 0.0001, 31 = .9, 82 = .99 (Kingma & Ba, 2015) and train for 300 epoch with a batch size of 512.
For recalibrating regressors, we compare against the previous state of the art, Kuleshov et al. (2018), who use an Isotonic regressor to correct a regressors' confidence. We use the same experimental setting as Kuleshov et al. (2018) including the UCI datasets (Dua & Graff, 2017), and regressor calibration metric (CAL.. Table shows that our approaches can outperform this baseline as well as the regression equivalent of Global Scaling. Inputs and targets are scaled to unit norm and variance prior to fitting for all regression experiments and missing values are imputed using scikit-learn's "Simplelmputer" (Pedregosa et al., 2011). Experiments utilize Keras" layers API with two hidden rectified linear unit (ReLU) layers, Glorot uniform initialization (Dahl et al., 2013; Glorot & Bengio, 2010) and Adam optimization with Ir 0.001 for 3000 steps without minibatching.
8

---

Under review as a conference paper at ICLR 2021
Table 3: Comparison of calibration methods by ECE for ResNet-50 (RN50) and DenseNet-121 (DN121) architectures on test data. Our predicted likelihood parameter methods: Linear Scaling (LS) and Linear Feature Scaling (LFS) outperform other approaches. In all cases our methods reduce miscalibration with comparable computation time as GS.
Model
Dataset
Uncalibrated
Platt
Isotonic
GS
VS
LS
LFS
RN50
CIFAR-10
.50
.034
.053
.046
.037
.018
.018
RN50
CIFAR-100
.642
.061
.072
.035
.044
.030
.173
RN50
SVHN
.072
.053
.010
.029
.022
.009
.009
RN50
ImageNet
.430
.018
.070
.019
.023
.026
.015
DN121
CIFAR-10
.253
.048
.042
.039
.034
.028
.028
DN121
CIFAR-100
.537
.049
.067
.024
.024
.014
.031
DN121
SVHN
.079
.018
.010
.022
.017
.011
.010
DN121
ImageNet
.229
.028
.095
.021
.019
.043
.019
Table 4: Comparison of regression calibration methods as evaluated by their calibration error as defined in (Kuleshov et al., 2018). Predicted likelihood parameters often outperform other methods.
Dataset
Uncalibrated Isotonic
GS
LS
DS
crime
0,3624
0.3499
0.0693 0.0125 0.0310
kinematics
0.0164
0.0103
0.0022 0.0021 0.0032
bank
0.0122
0.0056
0.0027 0.0024 0.0020
wine
0.0091
0.0108
0.0152 0.0131 0.0064
mpg
0.2153
0.2200
0. 0.19 0. 1.843 0.0233
cpu
0.0862
0.0340
0.3018 0.2078 0.1740
soil
0.3083
0.3000
0.3130 0.3175 0.3137
fried
0.0006
0.0002
0.0002 0.0002 0.0002
EXPERIMENTAL DETAILS
We run all experiments on Ubuntu 16.04 Azure Standard NV24 virtual machines (24 CPUs, 224 Gb memory, and 4x M60 GPUs) with Tensorflow 1.15 (Abadi et al., 2015) and PyTorch 1.17 (Paszke et al., 2019). Many likelihood parameters have constrained domains, such as the normal variance [0, oo). To evade the complexity of constrained optimization, we define unconstrained param- eters 6u and choose a transformation t(-) with inverse (-) to map to and from the constrained g. For positivity, exp/log parameterization is standard (Kendall & Gal, 2017; Kendall et al., 2018; Saxena et al., 2019). However, this parameterization can lead to instabilities and we use the softplus, +*) = log(1 + exp(x)), instead. Shifting the softplus, st (xx (In(1 + + + (In(2) + c). further improves stability and we explore this effect in Figure 12 of the Appendix. We use an affine softplus 8,1 and 2 respectively for adaptive scales and temperatures respectively. The one excep- tion is adaptive regularizer scales where we found exp led to faster convergence. For the constrained interval (a, b) we use affine transformations of the sigmoid s(x) = I+expi-x] (Barron, 2019). We initialize likelihood parameter biases to settings that yield MSE and Cross Entropy (o = r 1).
CONCLUSION
Optimizing the full likelihood can improve model quality by adapting losses and regularizers. Full likelihoods are agnostic the architecture, optimizer, and task, which makes them simple substitutes for standard losses. Global, data, and predicted likelihood parameters offer different degrees of expressivity and efficiency. point during training and testing without significant time and space overhead. By including these parameters in a loss function one can improve a model's robustness and generalization ability and create new classes of outlier detectors and recalibrators that outperform baselines. More generally, we hope this work encourages joint optimization of model and likelihood parameters, and argue likely that your loss should be likelihood.
particular, predicted parameters adapt the likelihood to each data

---

org/. Software available from Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. uttpt /ttp::://///ttttteeeee tensorflow.org.
Charu C Aggarwal. Outlier analysis. In Data mining, pp. 237-263. Springer, 2015.
Fabrizio Angiulli and Clara Pizzuti. Fast outlier detection in high dimensional spaces. In European conference on principles of data mining and knowledge discovery, pp. 15-27. Springer, 2002.
Jonathan Barron. general and adaptive robust loss function. In CVPR, 2019.
Christopher M Bishop. Mixture density networks. 1994.
Christopher M Bishop et al. Pattern recognition and machine learning, volume springer New York, 2006.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93-104, 2000.
Pierre Charbonnier, Laure Glanc-Féraude Gilles Aubert, and Michel Barlaud. Deterministic edge- preserving regularization in computed imaging. IEEE Transactions on image processing, 6(2): 298-311, 1997.
George E Dahl, Tara Sainath, and Geoffrey Hinton. Improving deep neural networks for Ivesr using rectified linear units and dropout. In ICASSP, pp. 8609-8613, IEEE, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. leee, 2009.
John Dennis and Roy E Welsch. Techniques for nonlinear least squares and robust regression. Communications in Statisties-simulatiie and Computation, 7(4):345-355 359,, 1978.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htttt://rrccieee.. ics. www.uui.u/.uu...
Bradley Efron, Trevor Hastie, lain Johnstone, Robert Tibshirani, et al. Least angle regression. The
Annals of statistics, 32(2):407-499 2004.
Martin Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395 1981.
Stuart Ganan and D McClure. Bayesian image analysis: An application to single photon emission tomography. Amer. Statist. Assoc, pp. 12-18, 1985.
Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. CRC press, 2013.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pp. 249-256, 2010.
Markus Goldstein and Andreas Dengel. Histogram- based outlier score (hbos): A fast unsupervised anomaly detection algorithm. K1-2012: Poster and Demo Track, pp. 59-63, 2012.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
10

---

Under review as a conference paper at ICLR 2021
Johanna Hardin and David M Rocke. Outlier detection in the multiple cluster setting using the minimum covariance determinant estimator. Computational Statistics & Data Analysis, 44(4): 625-638, 2004.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Zengyou He, Xiaofei Xu, and Shengchun Deng. Discovering cluster-based local outliers. Pattern Recognition Letters,
2003.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02533 2015.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Peter Huber. Robust statistics, volume 523. John Wiley & Sons, 2004.
Alex Kendall and Yarin Gal. What uncertainties we need bayesian deep learning for computer vision? In NeurlPS, pp. 5574-5584, 2017.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, pp. 7482-7491, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1311v:1312.6114 2013.
arXiv preprint
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233-243 1991.
Hans-Peter Kriegel, Matthias Schubert, and Arthur Zimek. Angle-based outlier detection high- dimensional data. In Proceedings of the 14th ACM SIGKDD international conference Knowl- edge discovery and data mining, pp. 444-452, 2008.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learn- ing using calibrated regression. Jennifer Dy and Andreas Krause (eds.), ICML, volume 80 of Proceedings of Machine Learning Research, pp. 2801-2809, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Aleksandar Lazarevic and Vipin Kumar. Feature bagging for outlier detection. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery data mining, pp- 157-166, 2005.
Xuelong Li, Quanmao Lu, Yongsheng Dong, and Dacheng Tao. Robust subspace clustering by cauchy loss function. IEEE transactions on neural networks and learning systems, 30(7):2067- 2078, 2018.
Zheng Li, Yue Zhao, Nicola Botta, Cezar Ionescu, and Xiyang Hu. Copod: copula- basula-based outlier detection. arXiv preprint arXiv:2009.09463. 2020.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Eighth IEEE Interna- tional Conference on Data Mining, pp. 413-422. JEEE, 2008.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation-band anomaly detection. ACM Trans- actions Knowledge Discovery from Data (TKDD), 6(1):1-39, 2012.
11

---

Under review as a conference paper at ICLR 2021
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, December 2015.
David JC MacKay and David JC Mac Kay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
John Ashworth Nelder and Robert WM Wedderburn. Generalized linear models. Journal of the Royal Statistical Sociery: Series (General), 135(3):370-385 1972.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
David A Nix and Andreas S Weigend. Estimating the mean and variance of the target probability distribution. In ICNN, volume 1, pp. 55-60. IEEE, 1994.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bir ButheeeB.... E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems 32, pp. 8024-8033 Curran Associates, Inc., 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leam: Machine learning in python. Journal of machine learning research, 12(0ct):2825-28800 2011.
John C Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 427-438, 2000.
Shebuti Rayana. Odds library. Stony Brook, 2016,, (2017), 2016.
Peter Rousseeuw and Katrien Van Driessen. A fast algorithm for the minimum covariance deter- minant estimator. Technometrics, 41(3):212-23,, 1999.
Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning a differentiable curriculum. In NeurlPS, pp. 11093-11103, 2019.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylor, and John C Platt. Support vector method for novelty detection. In Advances in neural information processing sys- tems, pp. 588-588, 2000.
Mei-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakorn, and LiWu Chang. A novel anomaly de- tection scheme based on principal component classifier. Technical report, MIAMI UNIV CORAL GABLES FL DEPT OF ELECTRICAL AND COMPUTER ENGINEERING, 2003.
Henri Theil. A Fank-invariant method of linear and polynomial regression analysis. In Henri Theil's contributions to economics and econometrics, pp. 345-366 3411 Springer, 1992.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26 31, 2012.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICML, volume I, pp. 609-616. Citeseer, 2001.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detec- tion. Journal of Machine Learning Research, 20(96):1-7, 2019. URL http: jmlr.. org// papers/v20/19 9 011 html.
12


Reviews:
[{"review_id": "tRqFrpfHLcT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper's starting point is a probabilistic interpretation of three losses: MSE, cross-entropy, and the loss introduced in reference (Barron 2019). It proposes the minimize the average loss jointly over the predictor's parameters $\\theta$, but also the loss' own parameters $\\phi$, vary, yielding an adaptive loss. It categorizes ways to define losses according to whether they depend on the data value, on the data index but not its value, or are independent of the data. The jointly optimized loss yields superior performance as re-calibrators (table 2 column CAL for regressors and table 5 for classifiers). It allows a natural definition of outlier detectors (sec  5.2)\n\nThe paper looks correct, specifically sec1 the approach of interpreting losses as NLL, sec4 the categorization, the experiment setup (though the description in the paper is cursory), the experiment design and use of CAL and ECE in sec5.4 on recalibration. The competitor methods and benchmarks for recalibration and outlier detection are quite exhaustive. (I am familiar with the references in section 3 related work)\n\nThe claims of superiority made in sec5.1, 5.2, 5.4 seem supported by the experimental evidence.\n\nDespite these strengths, I had a hard time evaluating and understanding the experiments, and partly the concrete details of the mathematical setup. For experiments, this happened because the paper glosses over almost every aspect: design, task, model, training, to only show final results. (I did not consult the supplementary material.)  For instance fig 2 and 3 are cryptic (in the caption of fig 2, I guess \"model\" should be replaced by \"predicted\"? and fig 3 relates to either predicted or data conditionings?). Table 1 is not referenced or explained in the text; one can assume it relates to the CelebA experiment from sec 5.1, but what is the form of the data-dependent loss for the \"predicted\" case? Table 2, why are we speaking of temperature (which only is a parameter for softmax loss in classification tasks) in the case of this regression experiment? Sec5.2, there is no experimental detail of the SVHN (and results are not discussed) and ODDS experiment.\n\nThe paper should make connections to maximum likelihood and maximum a posteriori optimization more apparent. I'm further curious about connections to Bayes' risk (eg cf Minka 2001 \"ERM is an incomplete inductive principle\") and KL/Bregman divergences (eg Buja, Stuetzle, Shen 2005). What is meant by \"optimizing a model's prior distribution\" sec5.3 ? Does sec5.3 amount to MAP optimization with some suitably defined prior? Other losses (eg hinge, Huber loss) than the three mentioned here can be given probabilistic interpretations, what motivates their choice (especially the choice of the recent, hence little used, robust loss $\\rho$) ?\n\nI found it difficult to settle on a rating for this paper. The work behind it definitely seems to have strenghts, for instance, it looks interesting, and I might want to point it out to colleagues, it investigates an interesting, not yet mainstream probabilistic interpretation in a relatively simple way, with few ad hoc assumptions. Yet the paper on its own is lacking in detail so much that the experiments are impossible to understand fully\nAs a conference paper, it seems to be \"bursting at the seams\"; maybe it would be better suited as a journal paper, where the important aspects can be developed. According to ICLR guidelines, it could be made up to 10 pages long; sec2 could be made terser. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "jYXmhcKOwiS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe authors propose to test experimentally training of the loss hyperparameters. Given a loss $L$ with a reconstruction part $\\ell$ and a penalty part $r$, they train the hyperparameters of $\\ell$ and of $r$.\n\n### Clarity\nThe main clarity issue is the reference formatting. Finding the cited papers in the references is tedious: hypertext links towards the *References* section should be added.\n\n### Details\nThe main contribution is the experimental part. The authors have tested loss hyperparameters learning mainly in robustness against outliers and outlier detection.\nI am not a specialist in outlier detection, so I cannot evaluate fairly most of the experimental results.\n\nTable 2, CRIME dataset with MSE: it seems that \"Base\" works better than \"Temp\".\n\n### Comments\nFrom a Bayesian point of view, there is no problem with learning hyperparameters in the reconstruction part $\\ell$ of the loss $L$. This is just equivalent to extending the family of probabilistic models. The user should only verify that $\\ell$ can be interpreted as the negative log of a probability distribution, that is, $\\exp(- \\ell)$ is integrable.\nHowever, training the hyperparameters of the penalty part of $L$ corresponds to training the parameters of the *prior* distribution, which is not acceptable (as such) from a Bayesian point of view.\n\nWeak accept: optimization of hyperparameters of $\\ell$ is theoretically well-founded from a Bayesian point of view, and should be more explored, as the authors do. However: 1) there is no consideration for this approach, while the preceding works of Barron (cited in the paper) had a word about it. This is important since we are still talking about a *likelihood*. 2) I am not sure whether the experimental results are significant enough. 3) The generalization of this approach to the penalty is not well-founded (at least, the authors do not justify it).\n\nEdit:\n### Rebuttal\nI had read the rebuttal and the other reviews. It seems that there are some clarity issues, which are independent from my knowledge of the area chosen for the experiments. Moreover, outlier detection is not necessarily the only possible application.\nHowever, I consider that the Bayesian point of view (which comes from preceding papers) has been well highlighted in the revised version. \nThese points put together, and given that I'm not sure of the significance of the experimental part, I do not change my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "h4g2A3zgEa2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary:\nThe paper proposes the use of complete parametrized likelihoods for providing supervision in place of the commonly used loss functions. The normal distribution, the categorical distribution defined by softmax and the likelihood of the robust rho-estimator are considered. The main idea is that by including the parameters of these likelihoods and optimizing over them, one can increase robustness, detect outliers and achieve re-calibration. In addition, by considering parametric priors and tuning their parameters one can obtain more flexible regularizers over the trainable parameters of a model.\n\n# Strengths:\nThe idea of the paper is quite interesting as it lifts some commonly made and often overlooked assumptions regarding the data distribution. By lifting these assumptions one can improve the performance of the trained model by considering likelihoods that better capture the data distribution. For example, data is usually affected by heteroskedasticity as well as outliers, and if the likelihood considered in its full form covers these aspects the resulting models will be better calibrated.\nThe proposed methods consider different aspects of conditioning and dimensionality of the likelihoods employed, varying from global to data-specific modeling.\n\nThe use of likelihoods instead of common loss functions leads to competitive new methods and variants for robust modeling, outlier detection, adaptive regularization and model re-calibration.\n\n\n# Weaknesses:\nAlthough the use of likelihoods instead of loss functions is not a common practice in deep learning, its advantages have been thoroughly studied in statistics, econometrics and other disciplines, as also discussed in the related work of the paper. Hence, the novelty mainly lies in the application of these ideas in deep learning and the employment of some likelihoods better suited for the respective problems (i.e. softmax and rho-estimators).\n\nThe paper is interesting however I found it somewhat difficult to read. In my view it tries to pack many different aspects and applications of the main idea (use of likelihood) into a very limited space. In fact, there are too many cross-references to the supplemental material, to the point that it seems that most of the paper is described in the supplemental material. \nOn a similar note, due to the fact that four different application domains are considered, there numerous methods, metrics and datasets involved in each one of them which are not sufficiently covered in the text. Additionally, many of the proposed methods/improvements/variants on each domain are not explained in sufficient detail (e.g. AE+S and PCA+S in Sec. 5.2). I would expect some more principled and thorough guidance on how to use the likelihood functions and, regarding the conditioning and dimensionality, strategies on how to choose among the various options.\n\nAlso some editing is required, for example the likelihood of the softmax is not provided as the respective sentence after eq. 4 is suddenly interrupted (see also the comments below).\n\n## Minor comments\n* the text in the figure is very small, making it very difficult to read in typical zoom levels (~100%)\n* Figure 3: the text does not correspond to the figure for the intermediate case\n* Figure 4, caption: include reference to left, middle and right panel\n* Table 1: there is no reference of this table in the text. Also, the three dots should be replaced with the actual setting.\n\n\n# Rating Justification:\nI think that the overall idea of the paper is interesting and provides improved data modeling which leads to important advantages of the estimated models. However, possibly due to space limitations, the paper does not explain in sufficient detail important aspect of applying the proposed idea in the domains considered.\n\n# Rating and comments after the rebuttal\nI think that in the revised version the paper has addressed many of the weaknesses pointed out in our reviews, hence I increase my rating to 6. Nevertheless, the paper still packs too much information which makes it difficult to read and appreciate.\nRegarding novelty, although I agree with other reviews that the core idea is not novel I think that it is important that the paper stresses the applicability and usefulness of considering likelihoods in deep learning models, as it appears to be not fully appreciated currently.\nOverall, I think that the paper would shine as a journal paper while it is only a borderline submission in its current form.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6Rr7kIuTiXz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studied loss functions by interpreting them from a likelihood viewpoint and by proposing to optimize \"full\" likelihoods for robust modeling, outlier-detection, and re-calibration purposes. Many loss functions stem from maximum likelihood estimation (MLE). For instance, the quadratic loss stems from MLE under Gaussianity, the absolute deviation loss stems from MLE under the Laplace noise assumption, and the check loss from MLE under the asymmetry Laplace noise assumption. What is common about these noise assumptions is that they contain scale/location parameters. Traditionally, in the machine learning community, these loss functions are used by ignoring the underlying noise assumptions and so the scale parameters are abandoned. The point of the paper is to take into account these scale parameters when solving the resulting optimization problems in learning.    \n\n\nPros:\nThe problem studied in this paper is interesting. By investigating the full likelihood when applying loss functions, it does bring back our attention to the origin of loss functions. The three applications mentioned in this paper are also typical and important.   \n\nCons: \n\n1. My main concern is about the novelty of this paper. As mentioned above, I believe that it does bring back our attention to the origin of loss functions. However, given that the main point of this paper is to optimize full likelihoods for learning problems, it is not clear to me what are the real novel contributions made by the paper. I'm questioning this as simultaneous estimation scale/location parameters and target functions seems to be traditional approaches in parametric robust statistics. I'm expecting more comments in this regard.\n\n2. The authors proposed to investigated three distributions, namely, Gaussian, softmax, and the distribution from Barron (2019). In my opinion, the three distributions are merely examples from three different scale families of distributions. I think more comments should be given when investigating the three specific ones. \n\n3. Three types of parameters are mentioned, namely, global parameters, data parameters, and predicted parameters, which are used to categorize distributions. I would suggest that the authors should at least provide several examples by pointing out which is which. \n\n4. In my opinion, the presentation of the paper could be further improved. For instance, the methodologies that the authors proposed become clear to me only after section 5.3. In addition, the title of this paper seems to be inappropriate in describing the main content of the paper. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper has some strengths to it, and some commented that the revision improved the manuscript, but the paper remained borderline with no strong champions in its favor. The reviews are encouraging and suggest that the paper is a bit tightly packed for the conference format, and perhaps because it is dense it is hard to the strength and scope of contributions, while other relationships such as to the Bayesian context could be explored more fully. Multiple reviewers find that a longer improved version would "shine" in a better suited journal. 

The decision to reject is independent of the fact that the authors seem to have violated the anonymity rules in the revised version.
Decision: Reject