It Is Likely That Your Loss Should be a Likelihood

1 INTRODUCTION . Choosing the right loss matters . Many common losses arise from likelihoods , such as the squared error loss from the normal distribution , absolute error from the Laplace distribution , and the cross entropy loss from the softmax distribution . The same is true of regularizers , where L2 arises from a normal prior and L1 from a Laplace prior . Deriving losses from likelihoods recasts the problem as a choice of distribution which allows datadependent adaptation . Standard losses and regularizers implicitly fix key distribution parameters , limiting flexibility . For instance , the squared error corresponds to fixing the normal variance at a constant . The full normal likelihood retains its scale parameter and allows optimization over a parametrized set of distributions . This work examines how to jointly optimize distribution and model parameters to select losses and regularizers that encourage generalization , calibration , and robustness to outliers . We explore three key likelihoods : the normal , softmax , and the robust regression likelihood ρ of Barron ( 2019 ) . Additionally , we cast adaptive priors in the same light and introduce adaptive regularizers . Our contributions : 1 . We systematically survey and evaluate global , data , and predicted likelihood parameters and introduce a new self-tuning variant of the robust adaptive loss ρ 2 . We apply likelihood parameters to create new classes of robust models , outlier detectors , and re-calibrators . 3 . We propose adaptive versions of L1 and L2 regularization using parameterized normal and Laplace priors on model parameters . 2 BACKGROUND . Notation We consider a dataset D of points xi and targets yi indexed by i ∈ { 1 , . . . , N } . Targets for regression are real numbers and targets for classification are one-hot vectors . The model f with parameters θ makes predictions ŷi = fθ ( x ) . A loss L ( ŷ , y ) measures the quality of the prediction given the target . To learn model parameters we solve the following loss optimization : min θ E ( x , y ) ∼D L ( ŷ = fθ ( x ) , y ) ( 1 ) A likelihood L ( ŷ|y , φ ) measures the quality of the prediction as a distribution over ŷ given the target y and likelihood parameters φ . We use the negative log-likelihood ` ( NLL ) , and the likelihood interchangeably since both have the same optima . We define the full likelihood optimization : min θ , φ E ( x , y ) ∼D ` ( ŷ = fθ ( x ) |y , φ ) ( 2 ) to jointly learn model and likelihood parameters . “ Full ” indicates the inclusion of φ , which controls the distribution and induced NLL loss . We focus on full likelihood optimization in this work . We note that the target , y , is the only supervision needed to optimize model and likelihood parameters , θ and φ respectively . Additionally , though the shape and scale varies with φ , reducing the error ŷ−y always reduces the NLL for our distributions . Distributions Under Investigation This work considers the normal likelihood with variance σ ( Bishop et al. , 2006 ; Hastie et al. , 2009 ) , the softmax likelihood with temperature τ ( Hinton et al. , 2015 ) , and the robust likelihood ρ ( Barron , 2019 ) with shape α and scale σ that control the scale and shape of the likelihood . The first two are among the most common losses in machine learning , and the last loss provides an important illustration of a likelihood parameter that affects “ shape ” instead of “ scale ” . We note that changing the scale and shape of the likelihood distribution is not “ cheating ” as there is a trade-off between uncertainty and credit . Figure 1 shows how this trade-off affects the Normal and softmax distributions and their NLLs . The normal likelihood has terms for the residual ŷ − y and the variance σ as N ( ŷ|y , σ ) = ( 2πσ2 ) − 12 exp ( −1 2 ( ŷ − y ) 2 σ2 ) , ( 3 ) with σ ∈ ( 0 , ∞ ) scaling the distribution . The normal NLL can be written ` N = 12σ2 ( ŷ−y ) 2+log σ , after simplifying and omitting constants that do not affect minimization . We recover the squared error by substituting σ = 1 . The softmax defines a categorical distribution defined by scores z for each class c as softmax ( ŷ = y|z , τ ) = e zyτ∑ c e zcτ , ( 4 ) with the temperature , τ ∈ ( 0 , ∞ ) , adjusting the entropy of the distribution . We recover the classification cross-entropy loss , − log p ( ŷ = y ) , by substituting τ = 1 in the respective NLL . We state the gradients of these likelihoods with respect to their σ and τ in Section A of the supplement . The robust loss ρ and its likelihood are ρ ( x , α , σ ) = |α− 2| α ( ( x/σ ) 2 |α− 2| + 1 ) α/2 − 1 and ( 5 ) p ( ŷ | y , α , σ ) = 1 σZ ( α ) exp ( −ρ ( ŷ − y , α , σ ) ) , ( 6 ) with shape α ∈ [ 0 , ∞ ) , scale σ ∈ ( 0 , ∞ ) , and normalization function Z ( α ) . This robust loss , ρ , has the interesting property that it generalizes several different loss functions commonly used in robust learning such as the L2 loss ( α = 2 ) , pseudo-huber loss ( Charbonnier et al. , 1997 ) ( α = 1 ) , Cauchy loss ( Li et al. , 2018 ) ( α = 0 ) , Geman-McClure loss ( Ganan & McClure , 1985 ) , ( α = −2 ) , and Welsch ( Dennis Jr & Welsch , 1978 ) loss ( alpha = −∞ ) . Learning the shape parameter allows models to adapt the shape of their noise distribution . 3 RELATED WORK . Likelihood optimization follows from maximum likelihood estimation ( Hastie et al. , 2009 ; Bishop et al. , 2006 ) , yet is uncommon in practice for fitting deep regressors and classifiers for discriminative tasks . However Kendall & Gal ( 2017 ) ; Kendall et al . ( 2018 ) ; Barron ( 2019 ) ; Saxena et al . ( 2019 ) optimize likelihood parameters to their advantage yet differ in their tasks , likelihoods , and parameterizations . In this work we aim to systematically experiment , clarify usage , and encourage their wider adoption . Early work on regressing means and variances ( Nix & Weigend , 1994 ) had the key insight that optimizing the full likelihood can fit these parameters and adapt the loss . Some recent works use likelihoods for loss adaptation , and interpret their parameters as the uncertainty ( Kendall & Gal , 2017 ; Kendall et al. , 2018 ) , robustness ( Kendall & Gal , 2017 ; Barron , 2019 ; Saxena et al. , 2019 ) , and curricula ( Saxena et al. , 2019 ) of losses . MacKay & Mac Kay ( 2003 ) uses Bayesian evidence to select hyper-parameters and losses based on proper likelihood normalization . Barron ( 2019 ) define a generalized robust regression loss , ρ , to jointly optimize the type and degree of robustness with global , data-independent , parameters . Kendall & Gal ( 2017 ) predict variances for regression and classification to handle data-dependent uncertainty . Kendall et al . ( 2018 ) balance multi-task loss weights by optimizing variances for regression and temperatures for classification . These global parameters depend on the task but not the data , and are interpreted as inherent task uncertainty . Saxena et al . ( 2019 ) define a differentiable curriculum for classification by assigning each training point its own temperature . These data parameters depend on the index of the data but not its value . We compare these different likelihood parameterizations across tasks and distributions . In the calibration literature , Guo et al . ( 2017 ) have found that deep networks are often miscalibrated , but they can be re-calibrated by cross-validating the temperature of the softmax . In this work we explore several generalizations of this concept . Alternatively , Platt scaling ( Platt , 1999 ) fits a sigmoid regressor to model predictions to calibrate probabilities . Kuleshov et al . ( 2018 ) re-calibrate regressors by fitting an Isotonic regressor to the empirical cumulative distribution function . 4 LIKELIHOOD PARAMETER TYPES . We explore the space of likelihood parameter representations for model optimization and inference . Though we note that some losses , like adversarial losses , are difficult to represent as likelihoods , many different losses in the community have a natural probabilistic interpretation . Often , these probabilistic interpretations can be parametrized in a variety of ways . We explore two key axes of generality when building these loss functions : conditioning and dimensionality . Conditioning We represent the likelihood parameters by three functional classes : global , data , and predicted . Global parameters , φ = c , are independent of the data and model and define the same likelihood distribution for all points . Data parameters , φi , are conditioned on the index , i , of the data , xi , but not its value . Every training point is assigned an independent likelihood parameter , φi that define different likelihoods for each training point . Predicted parameters , φ ( x ) = gη ( x ) , are determined by a model , g , with parameters η ( not to be confused with the task model parameters θ ) . Global and predicted parameters can be used during training and testing , but data parameters are only assigned to each training point and are undefined for testing . We show a simple example of predicted temperature in Figure 4 , and an illustration of the parameter types in Figure 2 . We note that for certain global parameters like a learned Normal scale , changing the scale does not affect the optima , but does change the probabilistic interpretation . This invariance has led many authors to drop the scale from their formulations . However , when models can predict these scale parameters they can naturally remain calibrated in the presence of heteroskedasticity and outliers . conditioning : global , predicted , and data . Each represents a different way to parametrize the model ’ s temperature . Each represents a possible way to parametrize the additional scale parameter added to the loss . Dimensionality The dimensionality , |φ| , of likelihood parameters can vary with the dimension of the task prediction , ŷ . For example , image regressors can use a single likelihood parameter for each image |φ| = 1 , RGB image channel |φ| = C , or even every pixel |φ| = W × H × C as in Figure 3 . These choices correspond to different likelihood distribution classes . Dimensionality and Conditioning of likelihood parameters can interact . For example , data parameters with |φ| = W × H × C would result in N ×W × H × C additional parameters , where N is the size of the dataset . This can complicate implementations and slow down optimization due to disk I/O when their size exceeds memory . Table 5 in the appendix contrasts the computational requirements of different likelihood parameter types . The work of Barron ( 2019 ) explores both scalar and pixel-wise dimensionalities for his robust loss . 5 APPLICATIONS

Summary: This paper's starting point is a probabilistic interpretation of three losses: MSE, cross-entropy, and the loss introduced in reference (Barron 2019). It proposes the minimize the average loss jointly over the predictor's parameters $\theta$, but also the loss' own parameters $\phi$, vary, yielding an adaptive loss. It categorizes ways to define losses according to whether they depend on the data value, on the data index but not its value, or are independent of the data. The jointly optimized loss yields superior performance as re-calibrators (table 2 column CAL for regressors and table 5 for classifiers). It allows a natural definition of outlier detectors (sec  5.2)