Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

1 Introduction . An important concern that limits the adoption of deep neural networks ( DNN ) in critical safety systems is how to assess our confidence in their predictions , i.e , quantifying their generalization capability ( Kaufman et al. , 2019 ; Willers et al. , 2020 ) . Take , for instance , a machine learning model for medical diagnosis ( Bibas et al. , 2021 ) . It may produce ( wrong ) diagnoses in the presence of test inputs that are different from the training set rather than flagging them for human intervention ( Singh et al. , 2021 ) . Detecting such unexpected inputs had been formulated as the out-of-distribution ( OOD ) detection task ( Hendrycks and Gimpel , 2017 ) , as flagging test inputs that lie outside the training classes , i.e. , are not in-distribution ( IND ) . Previous learning methods that designed to offer such generalization measures , include VCdimension ( Zhong et al. , 2017 ; Vapnik and Chervonenkis , 2015 ) and norm based bounds ( Neyshabur et al. , 2018 ; Bartlett et al. , 2017 ) . As a whole , these methods characterized the generalization ability based on the properties of the parameters . However , they do not consider the test sample that is presented to the model ( Jiang et al. , 2020 ) , which makes them useless for OOD detection . Other approaches build heuristics over the empirical risk minimization ( ERM ) learner , by post-processing 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the model output ( Sastry and Oore , 2020 ) or modifying the training process ( Papadopoulos et al. , 2021 ; Vyas et al. , 2018 ) . Regardless of the approach , these methods choose the learner that minimizes the loss over the training set . This may lead to a large generalization error because the ERM estimate may be wrong on unexpected inputs ; especially with large models such as DNN ( Belkin et al. , 2019 ) . To produce a useful generalization measure , we exploit the individual setting framework ( Merhav and Feder , 1998 ) . In the individual setting , there is no assumption about how the training and the test data are generated , nor about their probabilistic relationship . Moreover , the relationship between the labels and data can be deterministic and may therefore be determined by an adversary . The generalization error in this setting is often referred to as the regret ( Merhav and Feder , 1998 ) . This regret is defined by the log-loss difference between a learner and the genie : a learner that knows the specific test label , yet is constrained to use an explanation from a set of possible models . The individual setting is the most general framework and so the result holds for a wide range of scenarios . Specifically , the result holds to OOD detection where the distribution of the OOD inputs is unknown . The pNML learner ( Fogel and Feder , 2018 ) was proposed as the min-max solution of the regret , where the minimum is over the learner choice and the maximum is for any possible test label value . Intuitively , the pNML assigns a probability for a potential outcome as follows : Add the test sample to the training set with an arbitrary label , find the ERM solution of this new set , and take the probability it gives to the assumed label . Follow this procedure for every label and normalize to get a valid probability assignment . The pNML was developed before for linear regression ( Bibas et al. , 2019b ) and was evaluated empirically for DNN ( Fu and Levine , 2021 ; Bibas et al. , 2019a ) . We derive an analytical solution of the pNML learner and its generalization error ( the regret ) for a single layer NN . We analyze the derived regret and show it obtains low values when the test input either ( i ) lies in a subspace spanned by the eigenvectors associated with the large eigenvalues of the training data empirical correlation matrix or ( ii ) is located far from the decision boundary . Crucially , although our analysis focuses on a single layer NN , our results are applicable to the last layer of DNNs without changing the network architecture or the training process : We treat the pretrained DNN as a feature extractor with the last layer as a single layer NN classifier . We can therefore show the usage of the pNML regret as a confidence score for the OOD detection task . To summarize , we make the following contributions . 1 . Analytical derivation of the pNML regret . We derive an analytical expression of the pNML regret , which is associated with the generalization error , for a single layer NN . 2 . Analyzing the pNML regret . We explore the pNML regret characteristics as a function of the test sample data , training data , and the corresponding ERM prediction . We provide a visualization on low dimensional data and demonstrate the situations in which the pNML regret is low and the prediction can be trusted . 3 . DNN adaptation . We propose an adaptation of the derived pNML regret to any pretrained DNN that uses the softmax function with neither additional parameters nor extra data . Applying our pNML regret to a pretrained DNN does not require additional data , it is efficient and can be easily implemented . The derived regret is theoretically justified for OOD detection since it is the individual setting solution for which we do not require any knowledge on the test input distribution . Our evaluation includes 74 IND-OOD detection benchmarks using DenseNet-BC-100 , ResNet-34 , and WideResNet-40 trained with CIFAR-100 , CIFAR-10 , SVHN , and ImageNet-30 . Our approach outperforms leading methods in nearly all 74 OOD detection benchmarks up to a remarkable +15.2 % 2 Related work . OOD detection . Many recent work use additional OOD set in training ( Malinin and Gales , 2018 ; Hendrycks et al. , 2019a ; Nandy et al. , 2020 ; Mohseni et al. , 2020 ) . A well-known method for OOD detection , named ODIN ( Liang et al. , 2018 ) , manipulates the input image based on the gradient of the loss function . This input manipulation increases the margin between the maximum probability of images with known classes and images with unknown objects . However , the manipulation strength is defined by OOD data , which might be unavailable or different from the OOD samples in inference . Different approaches propose to manipulate the loss function in the training phase . Lee et al . ( 2018 ) proposed to use the Mahalanobis distance between the test input representations and the class conditional distribution in the intermediate layers to train a logistic regression classifier to determine if the input sample is OOD . The Energy method ( Liu et al. , 2020 ) adds a cost function to the training phase to shape the energy surface explicitly for OOD detection . Papadopoulos et al . ( 2021 ) suggested the outlier exposure with confidence control ( OECC ) method in which a regularization term is added to the loss function such that the model produces a uniform distribution for OOD samples . All mentioned methods require manipulating the DNN architecture or adding additional data . The Baseline method ( Hendrycks and Gimpel , 2017 ) is one of the earliest approaches designed to identify whether a test input is IND and does not require changing the pretrained model . This method uses the maximum probability of the DNN output as the OOD score . The Gram method ( Sastry and Oore , 2020 ) detects OODs based on feature representations obtained in intermediate layers . Given a test sample , the Gram matrices of the test sample features are compared with those of the training samples known to belong to the estimated class of the test sample . However , this method requires IND validation set and does not have a theoretical motivation . Other work suggested using Bayesian techniques to estimate the prediction confidence ( Gal and Ghahramani , 2016 ; Lakshminarayanan et al. , 2017 ; van Amersfoort et al. , 2020 ) . However , the Bayesian techniques add extensive compute overhead to the prediction . The pNML learner . The pNML learner is the min-max solution of the supervised batch learning in the individual setting ( Fogel and Feder , 2018 ) . For sequential prediction it is termed the conditional normalized maximum likelihood ( Rissanen and Roos , 2007 ; Roos and Rissanen , 2008 ) . Several methods deal with obtaining the pNML learner for different hypothesis sets . Bibas et al . ( 2019b ) and Bibas and Feder ( 2021 ) showed the pNML solution for linear regression . Rosas et al . ( 2020 ) proposed an NML based decision strategy for supervised classification problems and showed it attains heuristic PAC learning . Fu and Levine ( 2021 ) used the pNML for model optimization based on learning a density function by discretizing the space and fitting a distinct model for each value . For the DNN hypothesis set , Bibas et al . ( 2019a ) estimated the pNML distribution with DNN by fine-tuning the last layers of the network for every test input and label combination . This approach is computationally expensive since training is needed for every test input . Zhou and Levine ( 2020 ) suggested a way to accelerate the pNML computation in DNN by using approximate Bayesian inference techniques to produce a tractable approximation to the pNML . Pesso et al . ( 2021 ) used the pNML with adversarial target attack as a defense mechanism against adversarial perturbation . 3 Notation and preliminaries . In the supervised machine learning scenario , a training set consisting of N pairs of examples is given DN = { ( xn , yn ) } Nn=1 , xn ∈ X , yn ∈ Y , ( 1 ) where xn is the n-th data instance and yn is its corresponding label . The goal of a learner is to predict the unknown test label y given a new test data x by assigning probability distribution q ( ·|x ) to the unknown label . The performance is evaluated using the log-loss function ` ( q ; x , y ) = − log q ( y|x ) . ( 2 ) For the problem to be well-posed , we must make further assumptions on the class of possible models or hypothesis set that is used to find the relation between x and y. Denote Θ as a general index set , this class is a set of conditional probability distributions PΘ = { pθ ( y|x ) , θ ∈ Θ } . ( 3 ) The ERM is the learner from the hypothesis set that attains the minimal log-loss on the training set . The individual setting . An additional assumption required to solve the problem is related to how the data and the labels are generated . In this work we consider the individual setting ( Merhav and Feder , 1998 ; Fogel and Feder , 2018 ; Bibas et al. , 2019b , a ) , where the data and labels , both in the training and test , are specific individual quantities : We do not assume any probabilistic relationship between them , the labels may even be assigned in an adversarial manner . The genie . In the individual setting the goal is to compete with a reference learner , a genie , that has to following properties : ( i ) knows the test label value , ( ii ) is restricted to use a model from the given hypotheses set PΘ , and ( iii ) does not know which of the samples is the test . This reference learner then chooses a model that attains the minimum loss over the training set and the test sample θ̂ ( DN ; x , y ) = arg min θ∈Θ [ ` ( pθ ; x , y ) + N∑ n=1 ` ( pθ ; xn , yn ) ] . ( 4 ) The regret is the log-loss difference between a learner q and this genie : R ( q ; DN ; x , y ) = − log q ( y|x ) − [ − log pθ̂ ( DN ; x , y ) ( y|x ) ] . ( 5 ) Theorem 1 ( Fogel and Feder ( 2018 ) ) . The universal learner , denoted as the pNML , minimizes the regret for the worst case test label Γ = R∗ ( DN , x ) = min q max y∈Y R ( q ; DN ; x , y ) . ( 6 ) The pNML probability assignment and regret are qpNML ( y|x ) = pθ̂ ( DN ; x , y ) ( y|x ) ∑ y′∈Y pθ̂ ( DN ; x , y′ ) ( y ′|x ) , Γ = log ∑ y′∈Y pθ̂ ( DN ; x , y′ ) ( y ′|x ) . ( 7 ) Proof . The regret is equal for all choices of y . If we consider a different probability assignment , it should assign a smaller probability for at least one of the outcomes . If the true label is one of those outcomes it will lead to a higher regret . For more information see Fogel and Feder ( 2018 ) . The pNML regret is associated with the model complexity ( Zhang , 2012 ) . This complexity measure formalizes the intuition that a model that fits almost every data pattern very well would be much more complex than a model that provides a relatively good fit to a small set of data . Thus , the pNML incorporates a trade-off between goodness of fit and model complexity as measured by the regret . Online update of a neural network . Let XN and YN be the data and label matrices of N training points respectively XN = [ x1 x2 . . . xN ] > ∈ RN×M , YN = [ y1 y2 . . . yN ] > ∈ RN×C , ( 8 ) such that the number of input features and model outputs are M and C respectively . Denote X+N as the Moore–Penrose inverse of the data matrix X+N = { ( X > NXN ) −1X > N Rank ( X > NXN ) = M X > N ( XNX > N ) −1 otherwise , ( 9 ) f ( · ) and f−1 ( · ) as the activation and inverse activation functions , and θ ∈ RM×C as the learnable parameters . The ERM , that minimizes the training set mean squared error ( MSE ) , is θ̂N = X + Nf −1 ( YN ) = arg min θ ||YN − f ( XNθ ) ||22 . ( 10 ) Recently , Zhuang et al . ( 2020 ) suggested a recursive formulation of the DNN weights . Using their scheme only one training sample is processed at a time : Denote the projection of a sample x onto the orthogonal subspace of the training set correlation matrix as x⊥ = ( I −X+NXN ) x , ( 11 ) the update rule when receiving a new training sample with data x with label y is θ̂ ( DN ; x , y ) = θ̂N + g ( f−1 ( y ) − x > θ̂N ) , g , { 1 ‖x⊥‖2 x⊥ x⊥ 6= 0 1 1+x > X+NX + > N x X+NX + > N x , x⊥ = 0 . ( 12 ) In their paper , Zhuang et al . ( 2020 ) applied this formulation for a DNN , layer by layer .

Summary: This paper derives an explicit expression of the predictive normalized maximum likelihood (pNML) learner a single layer neural network probability assignment and regret (Eq. 7). This derived pNML regret expression is applied for the OOD detection task from pre-trained NN models. The authors also connect their pNML framework for single layer NN with soft-max activations and demonstrate that it can be applied for DNN models to obtain consistent improvements for the OOD detection performance.   ---------------------- After rebuttal:- Overall, I am satisfied with the paper. I am increasing my final score to 7 for the paper.  Few additional comments:-  "... more closely related to the domain of adversarial attacks":  I am not sure about this comment. Typically adversarial examples deal with minor perturbations. However in this case, I was talking about "any input sample that is far away from the training distributions". I think that should be part of out-of-distribution.   "synthetic dataset":- Please label Feature 1 and Feature 2 values in the plot.