Multiscale Score Matching for Out-of-Distribution Detection

1 INTRODUCTION . Modern neural networks do not tend to generalize well to out-of-distribution samples . This phenomenon has been observed in both classifier networks ( Hendrycks & Gimpel ( 2017 ) ; Nguyen et al . ( 2015 ) ; Szegedy et al . ( 2013 ) ) and deep likelihood models ( Nalisnick et al . ( 2018 ) ; Hendrycks et al . ( 2018 ) ; Ren et al . ( 2019 ) ) . This certainly has implications for AI safety ( Amodei et al . ( 2016 ) ) , as models need to be aware of uncertainty when presented with unseen examples . Moreover , an out-ofdistribution detector can be applied as an anomaly detector . Ultimately , our research is motivated by the need for a sensitive outlier detector that can be used in a medical setting . Particularly , we want to identify atypical morphometry in early brain development . This requires a method that is generalizable to highly variable , high resolution , unlabeled real-world data while being sensitive enough to detect an unspecified , heterogeneous set of atypicalities . To that end , we propose multiscale score matching to effectively detect out-of-distribution samples . Hyvärinen ( 2005 ) introduced score matching as a method to learn the parameters of a nonnormalized probability density model , where a score is defined as the gradient of the log density with respect to the data . Conceptually , a score is a vector field that points in the direction where the log density grows the most . The authors mention the possibility of matching scores via a nonparametric model but circumvent this by using gradients of the score estimate itself . However , Vincent ( 2011 ) later showed that the objective function of a denoising autoencoder ( DAE ) is equivalent to matching the score of a non-parametric Parzen density estimator of the data . Thus , DAEs provide a methodology for learning score estimates via the objective : 1 2 Ex̃∼qσ ( x̃|x ) pdata ( x ) [ ||sθ ( x̃ ) −∇x̃ log qσ ( x̃|x ) || ] ( 1 ) Here sθ ( x ) is the score network being trained to estimate the true score ∇x log pdata ( x ) , and qσ ( x̃ ) = ∫ qσ ( x̃|x ) pdata ( x ) dx . It should be noted that the score of the estimator only matches 1https : //github.com/ahsanMah/msma the true score when the noise perturbation is minimal i.e qσ ( x̃ ) ≈ pdata ( x ) . Recently , Song & Ermon ( 2019 ) employed multiple noise levels to develop a deep generative model based on score matching , called Noise Conditioned Score Network ( NCSN ) . Let { σi } Li=1 be a positive geometric sequence that satisfies σ1σ2 = ... = σL−1 σL > 1 . NCSN is a conditional network , sθ ( x , σ ) , trained to jointly estimate scores for various levels of noise σi such that ∀σ ∈ { σi } Li=1 : sθ ( x , σ ) ≈ ∇x log qσ ( x ) . In practice , the network is explicitly provided a one-hot vector denoting the noise level used to perturb the data . The network is then trained via a denoising score matching loss . They choose their noise distribution to beN ( x̃|x , σ2I ) ; therefore∇x̃ log qσ ( x̃|x ) = − ( x̃−x/σ2 ) . Thus the objective function is : 1 L L∑ i=1 λ ( σi ) [ 1 2 Ex̃∼qσi ( x̃|x ) pdata ( x ) [ ∣∣∣∣∣∣∣∣sθ ( x̃ , σi ) + ( x̃− xσ2i ) ∣∣∣∣∣∣∣∣2 2 ] ] ( 2 ) Song & Ermon ( 2019 ) set λ ( σi ) = σ2 after empirically observing that ||σsθ ( x , σ ) ||2 ∝ 1 . We similarly scaled our score norms for all our experiments . Our work directly utilizes the training objective proposed by Song & Ermon ( 2019 ) i.e . we use an NCSN as our score estimator . However , we use the score outputs for out-of-distribution ( OOD ) detection rather than for generative modeling . We demonstrate how the space of multiscale score estimates can separate in-distribution samples from outliers , outperforming state-of-the-art methods . We also apply our method on real-world medical imaging data of brain MRI scans . 2 MULTISCALE SCORE ANALYSIS . Consider taking the L2-norm of the score function : ||s ( x ) || = ||∇x log p ( x ) || = ∣∣∣∣∣∣∇x p ( x ) p ( x ) ∣∣∣∣∣∣ . Since the data density term appears in the denominator , a high likelihood will correspond to a low norm . Since out-of-distribution samples should have a low likelihood with respect to the indistribution log density ( i.e . p ( x ) is small ) , we can expect them to have high score norms . However , if these outlier points reside in “ flat ” regions with very small gradients ( e.g . in a small local mode ) , then their score norms can be low despite the point belonging to a low density region . This is our first indicator informing us that a true score norm may not be sufficient for detecting outliers . We empirically validate our intuition by considering score estimates for a relatively simple toy dataset : FashionMNIST . Following the denoising score matching objective ( Equation 2 ) , we can obtain multiple estimates of the true score by using different noise distributions qσ ( x̃|x ) . Like Song & Ermon ( 2019 ) , we choose the noise distributions to be zero-centered Gaussian scaled according to σi . Recall that the scores for samples perturbed by the lowest σ noise should be closest to the true score . Our analyses show that this alone was inadequate at separating inliers from OOD samples . We trained a score network sFM ( x , σL ) on FashionMNIST and used it to estimate scores of FashionMNIST ( x ∼ DFM ) , MNIST ( x ∼ DM ) and CIFAR-10 ( x ∼ DC ) test sets . Figure 1a shows the distribution of the score norms corresponding to the lowest noise level used . Note that CIFAR10 samples are appropriately given a high score by the model . However , the model is unable to distinguish FashionMNIST from MNIST , giving MNIST roughly the same scores as in-distribution samples . Though far from ideal , this result is still a considerable improvement on existing likelihood methods , which have been shown to assign higher likelihoods to OOD samples ( Nalisnick et al . ( 2018 ) ) . Our next line of inquiry was to utilize multiple noise levels . That is instead of simply considering sFM ( x , σL ) , we analyze the L-dimensional space [ ||sFM ( x , σ1 ) || , ... , ||sFM ( x , σL ) || ] for x ∼ { DFM , DM , DC } . Our observations showed that datasets did tend to be separable in the Ldimensional space of score norms . Figure 1b visualizes the UMAP embeddings of scores calculated via a network trained to estimate L = 10 scales of σs , with the lowest σ being the same as one in Figure 1a . 2.1 SCALES AND NEIGHBORHOODS . To our knowledge multiscale score analysis has not been explored in the context of OOD detection . In this section , we present an analysis in order to give an intuition for why multiple scales can be beneficial . Consider the toy distribution shown in Figure 2 . We have three regions of interest : an inlier region with high density centered around x = 10 , an outlier region with low density around x = 30 , and a second outlier region with a local mode centered around x = 50 . Recall that adding Gaussian noise to a distribution is equivalent to convolving it with the Gaussian distribution . This not only allows us to visualize perturbations of our toy distribution , but also analytically compute the score estimates given any σi . Initially with no perturbation , both a point in the low-density region and one very close to ( or at ) the local-mode will have small gradients . As we perturb the samples we smooth the original density , causing it to widen . The relative change in density at each point is dependent on neighboring modes . A large scale perturbation will proportionally take a larger neighborhood into account at each point of the convolution . Therefore , at a sufficiently large scale , nearby outlier points gain context from in-distribution modes . This results in an increased gradient signal in the direction of inliers . Figure 3 plots the score norms of samples generated from the original density along with markers indicating our key regions . Note how even a small scale perturbation ( σL = 0.1 ) is enough to bias the density of the Low-Density outliers towards the nearby in-distribution mode . A medium scale ( σM = 10 ) Gaussian perturbation is still not wide enough to reach the inlier region from the LocalMode outlier densities , causing them to simply smooth away into flat nothingness . It is only after we perform a large scale ( σH = 20 ) perturbation that the in-distribution mode gets taken into account , resulting in a higher gradient norm . This analysis allows us to intuit that larger noise levels account for a larger neighborhood context . We surmise that given a sufficiently large scale , we can capture gradient signals from distant outliers . It is imperative to note that one large scale is not guaranteed to work for all outliers . Consider outliers close to inlier modes such as the samples between Low-Density outliers and Inliers in Figure 2. σH results in an overlap in the score distribution of inliers and Low-Density outliers . This makes it difficult to differentiate the aforementioned “ in-between ” outliers from the in-distribution samples . However , this large scale was necessary to get a big enough neighborhood context in order to capture the more distant Local-Mode outliers . Therefore , we postulate that a range of scales is necessary for separating outliers . Admittedly , selecting this range according to the dataset is not a trivial problem . In a very recent work , Song & Ermon ( 2020 ) outlined some techniques for selecting { σi } Li=1 for NCSNs from the perspective of generative modelling . Perhaps there is a similar analog to OOD detection . We leave such analyses for future work and use the default range for NCSN in all our experiments . However , we observed that our defaults are surprisingly generalizable , evident by the fact that all our experiments in Section 5 were performed with the same scale range . In Section 5.5 , we further analyze how varying the scale range effects downstream accuracy and observe that our defaults already provide near optimal performance . 2.2 PROPOSED TRAINING SCHEME . In this work , we propound the inclusion of all noisy score estimates for the task of separating in- and out-of-distribution points , allowing for a Multiscale Score Matching Analysis ( MSMA ) . Concretely , given L noise levels , we calculate the L2-norms of per-sample scores for each level , resulting in an L-dimensional vector for each input sample . Motivated by our observations , we posit that indistribution data points occupy distinct and dense regions in the L-dimensional score space . The cluster assumption states that decision boundaries should not pass high density regions , but instead lie in low density regions . This implies that any auxiliary method trained to learn in-distribution regions should be able to identify OOD data points that reside outside the learned space . Thus , we propose a two step unsupervised training scheme . First , we train a NCSN model sIN ( x , σL ) to estimate scores for inlier samples , given { σi } Li=1 levels of noise . Once trained , we calculate all L noisy score estimates for the N training samples and take the L2-norms across the input dimensions : [ ||sIN ( X , σ1 ) ||22 , ... , ||sIN ( X , σL ) || 2 2 ] . This results in an NxL matrix . We now train an auxiliary model ( such as a Gaussian Mixture Model ) on this matrix to learn the spatial regions of in-distribution samples in the L-dimensional space .

Summary: The authors leveraged and repurposed Noise Conditioned Score Network (NCSN) that was originally introduced by Song & Ermon (2019) for generative modeling to be used for detection out-of-distribution (OOD) images. The authors unfold the intuition and rationale behind score matching followed by the equivalence of denoising autoencoder (DAE) to derive NCSN as a score estimator and provide an analysis to demonstrate the value of multiscale score analysis. In an experimental analysis on SVHN and CIFAR datasets they demonstrate superiority of their method (MSMA) over previously reported findings in the literature using state-of-the-art models (ODIN, JEM, Likelihood Ratios) on OOD task.