ReAct: Out-of-distribution Detection With Rectified Activations

1 Introduction . Neural networks deployed in real-world systems often encounter out-of-distribution ( OOD ) inputs— unknown samples that the network has not been exposed to during training . Identifying and handling these OOD inputs can be paramount in safety-critical applications such as autonomous driving [ 9 ] and health care . For example , an autonomous vehicle may fail to recognize objects on the road that do not appear in its object detection model ’ s training set , potentially leading to a crash . This can be prevented if the system identifies the unrecognized object as OOD and warns the driver in advance . A driving idea behind OOD detection is that the model should be much more uncertain about samples outside of its training distribution . However , Nguyen et al . [ 38 ] revealed that modern neural networks can produce overconfident predictions on OOD inputs . This phenomenon renders the separation of in-distribution ( ID ) and OOD data a non-trivial task . Indeed , much of the prior work on OOD detection focused on defining more suitable measures of OOD uncertainty [ 17 , 19 , 28 , 29 , 31 , 33 ] . Despite the improvement , it is arguable that continued research progress in OOD detection requires insights into the fundamental cause and mitigation of model overconfidence on OOD data . In this paper , we start by revealing an important observation that OOD data can trigger unit activation patterns that are significantly different from ID data . Figure 1 ( b ) shows the distribution of activations 1Code is available at : https : //github.com/deeplearning-wisc/react.git 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in the penultimate layer of ResNet-50 trained on ImageNet [ 7 ] . Each point on the horizontal axis corresponds to a single unit . The mean and standard deviation are shown by the solid line and shaded area , respectively . The mean activation for ID data ( blue ) is well-behaved with a near-constant mean and standard deviation . In contrast , for OOD data ( gray ) , the mean activation has significantly larger variations across units and is biased towards having sharp positive values ( i.e. , positively skewed ) . As a result , such high unit activation can undesirably manifest in model output , producing overconfident predictions on OOD data . A similar distributional property holds for other OOD datasets as well . The above observation naturally inspires a simple yet surprisingly effective method—Rectified Activations ( dubbed ReAct ) for OOD detection . In particular , the outsized activation of a few selected hidden units can be attenuated by rectifying the activations at an upper limit c > 0 . Conveniently , this can be done on a pre-trained model without any modification to training . The dashed horizontal line in Figure 1 ( b ) shows the cutoff point c , and its effect on the OOD uncertainty score is shown in Figure 1 ( c ) . After rectification , the output distributions for ID and OOD data become much more well-separated and the false positive rate ( FPR ) is significantly reduced from 55.72 % to 20.38 % . Importantly , this truncation largely preserves the activation for in-distribution data , and therefore ensures the classification accuracy on the original task is largely comparable . We provide both empirical and theoretical insights , characterizing and explaining the mechanism by which ReAct improves OOD detection . We perform extensive evaluations and establish state-ofthe-art performance on a suite of common OOD detection benchmarks , including CIFAR-10 and CIFAR-100 , as well as a large-scale ImageNet dataset [ 7 ] . ReAct outperforms the best baseline by a large margin , reducing the average FPR95 by up to 25.05 % . We further analyze our method theoretically and show that ReAct is more beneficial when OOD activations are more chaotic ( i.e. , having a larger variance ) and positively skewed compared to ID activations , a behavior that is typical of many OOD datasets ( cf . Figure 1 ) . In summary , our key results and contributions are : 1 . We introduce ReAct —a simple and effective post hoc OOD detection approach that utilizes activation truncation . We show that ReAct can generalize effectively to different network architectures and works with different OOD detection methods including MSP [ 15 ] , ODIN [ 31 ] , and energy score [ 33 ] . 2 . We extensively evaluate ReAct on a suite of OOD detection tasks and establish a stateof-the-art performance among post hoc methods . Compared to the previous best method , ReAct achieves an FPR95 reduction of 25.05 % on a large-scale ImageNet benchmark . 3 . We provide both empirical ablation and theoretical analysis , revealing important insights that abnormally high activations on OOD data can harm their detection and how ReAct effectively mitigates this issue . We hope that our insights inspire future research to further examine the internal mechanisms of neural networks for OOD detection . Code and dataset will be released for reproducible research . 2 Background . The problem of out-of-distribution detection for classification is defined using the following setup . Let X ( resp . Y ) be the input ( resp . output ) space and let P be a distribution over X ⇥ Y . Let f : X ! R|Y| be a neural network trained on samples drawn from P to output a logit vector , which is used to predict the label of an input sample . Denote by Din the marginal distribution of P for X , which represents the distribution of in-distribution data . At test time , the environment can present a distribution Dout over X of out-of-distribution data . The goal of OOD detection is to define a decision function G such that for a given test input x 2 X : G ( x ; f ) = ⇢ 0 if x ⇠ Dout , 1 if x ⇠ Din . The difficulty of OOD detection greatly depends on the separation between Din and Dout . In the extreme case , if Din = Dout then the optimal decision function is a random coin flip . In practice , Dout is often defined by a distribution that simulates unknowns encountered during deployment time , such as samples from an irrelevant distribution whose label set has no intersection with Y and therefore should not be predicted by the model . 3 Method . We introduce a simple and surprisingly effective technique , Rectified Activations ( ReAct ) , for improving OOD detection performance . Our key idea is to perform post hoc modification to the unit activation , so to bring the overall activation pattern closer to the well-behaved case . Specifically , we consider a pre-trained neural network parameterized by ✓ , which encodes an input x 2 Rd to a feature space with dimension m. We denote by h ( x ) 2 Rm the feature vector from the penultimate layer of the network . A weight matrix W 2 Rm⇥K connects the feature h ( x ) to the output f ( x ) , where K is the total number of classes in Y = { 1 , 2 , ... , K } . ReAct : Rectified Activation . We propose the ReAct operation , which is applied on the penultimate layer of a network : h̄ ( x ) = ReAct ( h ( x ) ; c ) , ( 1 ) where ReAct ( x ; c ) = min ( x , c ) and is applied element-wise to the feature vector h ( x ) . In effect , this operation truncates activations above c to limit the effect of noise . The model output after rectified activation is given by : fReAct ( x ; ✓ ) = W > h̄ ( x ) + b , ( 2 ) where b 2 RK is the bias vector . A higher c indicates a larger threshold of activation truncation . When c = 1 , the output becomes equivalent to the original output f ( x ; ✓ ) without rectification , where f ( x ; ✓ ) = W > h ( x ) + b . Ideally , the rectification parameter c should be chosen to sufficiently preserve the activations for ID data while rectifying that of OOD data . In practice , we set c based on the p-th percentile of activations estimated on the ID data . For example , when p = 90 , it indicates that 90 % percent of the ID activations are less than the threshold c. We discuss the effect of percentile in detail in Section 4 . OOD detection with rectified activation . During test time , ReAct can be leveraged by a variety of downstream OOD scoring functions relying on fReAct ( x ; ✓ ) : G ( x ; f ReAct ) = ⇢ in S ( x ; fReAct ) out S ( x ; fReAct ) < , ( 3 ) where a thresholding mechanism is exercised to distinguish between ID and OOD during test time . To align with the convention , samples with higher scores S ( x ; f ) are classified as ID and vice versa . The threshold is typically chosen so that a high fraction of ID data ( e.g. , 95 % ) is correctly classified . ReAct can be compatible with several commonly used OOD scoring functions derived from the model output f ( x ; ✓ ) , including the softmax confidence [ 15 ] , ODIN score [ 31 ] , and the energy score [ 33 ] . For readers ’ convenience , we provide an expansive description of these methods in Appendix B . In Section 4 , we default to using the energy score ( since it is hyperparameter-free and does not require fine-tuning ) , but demonstrate the benefit of using ReAct with other OOD scoring functions too . 4 Experiments . In this section , we evaluate ReAct on a suite of OOD detection tasks . We first evaluate a on large-scale OOD detection benchmark based on ImageNet [ 18 ] ( Section 4.1 ) , and then proceed in Section 4.2 with CIFAR benchmarks [ 27 ] . 4.1 Evaluation on Large-scale ImageNet Classification Networks . We first evaluate ReAct on a large-scale OOD detection benchmark developed in [ 18 ] . Compared to the CIFAR benchmarks that are routinely used in literature , the ImageNet benchmark is more challenging due to a larger label space ( K = 1 , 000 ) . Moreover , such large-scale evaluation is more relevant to real-world applications , where the deployed models often operate on images that have high resolution and contain more classes than the CIFAR benchmarks . Setup . We use a pre-trained ResNet-50 model [ 12 ] for ImageNet-1k . At test time , all images are resized to 224 ⇥ 224 . We evaluate on four test OOD datasets from ( subsets of ) Places365 [ 62 ] , Textures [ 5 ] , iNaturalist [ 16 ] , and SUN [ 55 ] with non-overlapping categories w.r.t ImageNet . Our evaluations span a diverse range of domains including fine-grained images , scene images , and textural images ( see Appendix C for details ) . We use a validation set of Gaussian noise images , which are generated by sampling from N ( 0 , 1 ) for each pixel location . To ensure validity , we further verify the activation pattern under Gaussian noise , which exhibits a similar distributional trend with positive skewness and chaoticness ; see Figure 7 in Appendix G for details . We select p from { 10 , 65 , 80 , 85 , 90 , 95 , 99 } based on the FPR95 performance . The optimal p is 90 . All experiments are based on the hardware described in Appendix D. Comparison with competitive OOD detection methods . In Table 1 , we compare ReAct with OOD detection methods that are competitive in the literature , where ReAct establishes the state-of-the-art performance . For a fair comparison , all methods use the pre-trained networks post hoc . We report performance for each OOD test dataset , as well as the average of the four . ReAct outperforms all baselines considered , including Maximum Softmax Probability [ 15 ] , ODIN [ 31 ] , Mahalanobis distance [ 29 ] , and energy score [ 33 ] . Noticeably , ReAct reduces the FPR95 by 25.05 % compared to the best baseline [ 31 ] on ResNet . Note that Mahalanobis requires training a separate binary classifier , and displays limiting performance since the increased size of label space makes the class-conditional Gaussian density estimation less viable . In contrast , ReAct is much easier to use in practice , and can be implemented through a simple post hoc activation rectification . Effect of rectification threshold c. We now characterize the effect of the rectification parameter c , which can be modulated by the percentile p described in Section 3 . In Table 2 , we summarize the OOD detection performance , where we vary p = { 10 , 65 , 80 , 85 , 90 , 95 , 99 } . This ablation confirms that over-activation does compromise the ability to detect OOD data , and ReAct can effectively alleviate this problem . Moreover , when p is sufficiently large , ReAct can improve OOD detection while maintaining a comparable ID classification accuracy . Alternatively , once a sample is detected to be ID , one can always use the original activation h ( x ) , which is guaranteed to give identical classification accuracy . When p is too small , OOD performance starts to degrade as expected . Effect on other network architectures . We show that ReAct is effective on a different architecture in Table 1 . In particular , we consider a lightweight model MobileNet-v2 [ 44 ] , which can be suitable for OOD detection in on-device mobile applications . Same as before , we apply ReAct on the output of the penultimate layer , with the rectification threshold chosen based on the 90-th percentile . Our method reduces the FPR95 by 9.18 % compared to the best baseline [ 31 ] . What about applying ReAct on other layers ? Our results suggest that applying ReAct on the penultimate layer is the most effective , since the activation patterns are most distinctive . To see this , we provide the activation and performance study for intermediate layers in Appendix E ( see Figure 6 and Table 6 ) . Interestingly , early layers display less distinctive signatures between ID and OOD data . This is expected because neural networks generally capture lower-level features in early layers ( such as Gabor filters [ 61 ] in layer 1 ) , whose activations can be very similar between ID and OOD . The semantic-level features only emerge as with deeper layers , where ReAct is the most effective .

Summary: This paper identifies that out-of-distribution (OOD) data tend to cause overconfident activations much more often than in-distribution data. Based on this observation, a simple approach called ReAct (Rectified Activations) is proposed, which essentially applies a threshold to the activations of the penultimate layer of a CNN classifier. Extensive experiments demonstrate that ReAct works with different network architectures and OOD detection algorithms, and the improvements over previous SOTA methods are significant.