ReAct: Out-of-distribution Detection With Rectified Activations

1 Introduction . Neural networks deployed in real-world systems often encounter out-of-distribution ( OOD ) inputs— unknown samples that the network has not been exposed to during training . Identifying and handling these OOD inputs can be paramount in safety-critical applications such as autonomous driving [ 9 ] and health care . For example , an autonomous vehicle may fail to recognize objects on the road that do not appear in its object detection model ’ s training set , potentially leading to a crash . This can be prevented if the system identifies the unrecognized object as OOD and warns the driver in advance . A driving idea behind OOD detection is that the model should be much more uncertain about samples outside of its training distribution . However , Nguyen et al . [ 38 ] revealed that modern neural networks can produce overconfident predictions on OOD inputs . This phenomenon renders the separation of in-distribution ( ID ) and OOD data a non-trivial task . Indeed , much of the prior work on OOD detection focused on defining more suitable measures of OOD uncertainty [ 17 , 19 , 28 , 29 , 31 , 33 ] . Despite the improvement , it is arguable that continued research progress in OOD detection requires insights into the fundamental cause and mitigation of model overconfidence on OOD data . In this paper , we start by revealing an important observation that OOD data can trigger unit activation patterns that are significantly different from ID data . Figure 1 ( b ) shows the distribution of activations 1Code is available at : https : //github.com/deeplearning-wisc/react.git 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in the penultimate layer of ResNet-50 trained on ImageNet [ 7 ] . Each point on the horizontal axis corresponds to a single unit . The mean and standard deviation are shown by the solid line and shaded area , respectively . The mean activation for ID data ( blue ) is well-behaved with a near-constant mean and standard deviation . In contrast , for OOD data ( gray ) , the mean activation has significantly larger variations across units and is biased towards having sharp positive values ( i.e. , positively skewed ) . As a result , such high unit activation can undesirably manifest in model output , producing overconfident predictions on OOD data . A similar distributional property holds for other OOD datasets as well . The above observation naturally inspires a simple yet surprisingly effective method—Rectified Activations ( dubbed ReAct ) for OOD detection . In particular , the outsized activation of a few selected hidden units can be attenuated by rectifying the activations at an upper limit c > 0 . Conveniently , this can be done on a pre-trained model without any modification to training . The dashed horizontal line in Figure 1 ( b ) shows the cutoff point c , and its effect on the OOD uncertainty score is shown in Figure 1 ( c ) . After rectification , the output distributions for ID and OOD data become much more well-separated and the false positive rate ( FPR ) is significantly reduced from 55.72 % to 20.38 % . Importantly , this truncation largely preserves the activation for in-distribution data , and therefore ensures the classification accuracy on the original task is largely comparable . We provide both empirical and theoretical insights , characterizing and explaining the mechanism by which ReAct improves OOD detection . We perform extensive evaluations and establish state-ofthe-art performance on a suite of common OOD detection benchmarks , including CIFAR-10 and CIFAR-100 , as well as a large-scale ImageNet dataset [ 7 ] . ReAct outperforms the best baseline by a large margin , reducing the average FPR95 by up to 25.05 % . We further analyze our method theoretically and show that ReAct is more beneficial when OOD activations are more chaotic ( i.e. , having a larger variance ) and positively skewed compared to ID activations , a behavior that is typical of many OOD datasets ( cf . Figure 1 ) . In summary , our key results and contributions are : 1 . We introduce ReAct —a simple and effective post hoc OOD detection approach that utilizes activation truncation . We show that ReAct can generalize effectively to different network architectures and works with different OOD detection methods including MSP [ 15 ] , ODIN [ 31 ] , and energy score [ 33 ] . 2 . We extensively evaluate ReAct on a suite of OOD detection tasks and establish a stateof-the-art performance among post hoc methods . Compared to the previous best method , ReAct achieves an FPR95 reduction of 25.05 % on a large-scale ImageNet benchmark . 3 . We provide both empirical ablation and theoretical analysis , revealing important insights that abnormally high activations on OOD data can harm their detection and how ReAct effectively mitigates this issue . We hope that our insights inspire future research to further examine the internal mechanisms of neural networks for OOD detection . Code and dataset will be released for reproducible research . 2 Background . The problem of out-of-distribution detection for classification is defined using the following setup . Let X ( resp . Y ) be the input ( resp . output ) space and let P be a distribution over X ⇥ Y . Let f : X ! R|Y| be a neural network trained on samples drawn from P to output a logit vector , which is used to predict the label of an input sample . Denote by Din the marginal distribution of P for X , which represents the distribution of in-distribution data . At test time , the environment can present a distribution Dout over X of out-of-distribution data . The goal of OOD detection is to define a decision function G such that for a given test input x 2 X : G ( x ; f ) = ⇢ 0 if x ⇠ Dout , 1 if x ⇠ Din . The difficulty of OOD detection greatly depends on the separation between Din and Dout . In the extreme case , if Din = Dout then the optimal decision function is a random coin flip . In practice , Dout is often defined by a distribution that simulates unknowns encountered during deployment time , such as samples from an irrelevant distribution whose label set has no intersection with Y and therefore should not be predicted by the model . 3 Method . We introduce a simple and surprisingly effective technique , Rectified Activations ( ReAct ) , for improving OOD detection performance . Our key idea is to perform post hoc modification to the unit activation , so to bring the overall activation pattern closer to the well-behaved case . Specifically , we consider a pre-trained neural network parameterized by ✓ , which encodes an input x 2 Rd to a feature space with dimension m. We denote by h ( x ) 2 Rm the feature vector from the penultimate layer of the network . A weight matrix W 2 Rm⇥K connects the feature h ( x ) to the output f ( x ) , where K is the total number of classes in Y = { 1 , 2 , ... , K } . ReAct : Rectified Activation . We propose the ReAct operation , which is applied on the penultimate layer of a network : h̄ ( x ) = ReAct ( h ( x ) ; c ) , ( 1 ) where ReAct ( x ; c ) = min ( x , c ) and is applied element-wise to the feature vector h ( x ) . In effect , this operation truncates activations above c to limit the effect of noise . The model output after rectified activation is given by : fReAct ( x ; ✓ ) = W > h̄ ( x ) + b , ( 2 ) where b 2 RK is the bias vector . A higher c indicates a larger threshold of activation truncation . When c = 1 , the output becomes equivalent to the original output f ( x ; ✓ ) without rectification , where f ( x ; ✓ ) = W > h ( x ) + b . Ideally , the rectification parameter c should be chosen to sufficiently preserve the activations for ID data while rectifying that of OOD data . In practice , we set c based on the p-th percentile of activations estimated on the ID data . For example , when p = 90 , it indicates that 90 % percent of the ID activations are less than the threshold c. We discuss the effect of percentile in detail in Section 4 . OOD detection with rectified activation . During test time , ReAct can be leveraged by a variety of downstream OOD scoring functions relying on fReAct ( x ; ✓ ) : G ( x ; f ReAct ) = ⇢ in S ( x ; fReAct ) out S ( x ; fReAct ) < , ( 3 ) where a thresholding mechanism is exercised to distinguish between ID and OOD during test time . To align with the convention , samples with higher scores S ( x ; f ) are classified as ID and vice versa . The threshold is typically chosen so that a high fraction of ID data ( e.g. , 95 % ) is correctly classified . ReAct can be compatible with several commonly used OOD scoring functions derived from the model output f ( x ; ✓ ) , including the softmax confidence [ 15 ] , ODIN score [ 31 ] , and the energy score [ 33 ] . For readers ’ convenience , we provide an expansive description of these methods in Appendix B . In Section 4 , we default to using the energy score ( since it is hyperparameter-free and does not require fine-tuning ) , but demonstrate the benefit of using ReAct with other OOD scoring functions too . 4 Experiments . In this section , we evaluate ReAct on a suite of OOD detection tasks . We first evaluate a on large-scale OOD detection benchmark based on ImageNet [ 18 ] ( Section 4.1 ) , and then proceed in Section 4.2 with CIFAR benchmarks [ 27 ] . 4.1 Evaluation on Large-scale ImageNet Classification Networks . We first evaluate ReAct on a large-scale OOD detection benchmark developed in [ 18 ] . Compared to the CIFAR benchmarks that are routinely used in literature , the ImageNet benchmark is more challenging due to a larger label space ( K = 1 , 000 ) . Moreover , such large-scale evaluation is more relevant to real-world applications , where the deployed models often operate on images that have high resolution and contain more classes than the CIFAR benchmarks . Setup . We use a pre-trained ResNet-50 model [ 12 ] for ImageNet-1k . At test time , all images are resized to 224 ⇥ 224 . We evaluate on four test OOD datasets from ( subsets of ) Places365 [ 62 ] , Textures [ 5 ] , iNaturalist [ 16 ] , and SUN [ 55 ] with non-overlapping categories w.r.t ImageNet . Our evaluations span a diverse range of domains including fine-grained images , scene images , and textural images ( see Appendix C for details ) . We use a validation set of Gaussian noise images , which are generated by sampling from N ( 0 , 1 ) for each pixel location . To ensure validity , we further verify the activation pattern under Gaussian noise , which exhibits a similar distributional trend with positive skewness and chaoticness ; see Figure 7 in Appendix G for details . We select p from { 10 , 65 , 80 , 85 , 90 , 95 , 99 } based on the FPR95 performance . The optimal p is 90 . All experiments are based on the hardware described in Appendix D. Comparison with competitive OOD detection methods . In Table 1 , we compare ReAct with OOD detection methods that are competitive in the literature , where ReAct establishes the state-of-the-art performance . For a fair comparison , all methods use the pre-trained networks post hoc . We report performance for each OOD test dataset , as well as the average of the four . ReAct outperforms all baselines considered , including Maximum Softmax Probability [ 15 ] , ODIN [ 31 ] , Mahalanobis distance [ 29 ] , and energy score [ 33 ] . Noticeably , ReAct reduces the FPR95 by 25.05 % compared to the best baseline [ 31 ] on ResNet . Note that Mahalanobis requires training a separate binary classifier , and displays limiting performance since the increased size of label space makes the class-conditional Gaussian density estimation less viable . In contrast , ReAct is much easier to use in practice , and can be implemented through a simple post hoc activation rectification . Effect of rectification threshold c. We now characterize the effect of the rectification parameter c , which can be modulated by the percentile p described in Section 3 . In Table 2 , we summarize the OOD detection performance , where we vary p = { 10 , 65 , 80 , 85 , 90 , 95 , 99 } . This ablation confirms that over-activation does compromise the ability to detect OOD data , and ReAct can effectively alleviate this problem . Moreover , when p is sufficiently large , ReAct can improve OOD detection while maintaining a comparable ID classification accuracy . Alternatively , once a sample is detected to be ID , one can always use the original activation h ( x ) , which is guaranteed to give identical classification accuracy . When p is too small , OOD performance starts to degrade as expected . Effect on other network architectures . We show that ReAct is effective on a different architecture in Table 1 . In particular , we consider a lightweight model MobileNet-v2 [ 44 ] , which can be suitable for OOD detection in on-device mobile applications . Same as before , we apply ReAct on the output of the penultimate layer , with the rectification threshold chosen based on the 90-th percentile . Our method reduces the FPR95 by 9.18 % compared to the best baseline [ 31 ] . What about applying ReAct on other layers ? Our results suggest that applying ReAct on the penultimate layer is the most effective , since the activation patterns are most distinctive . To see this , we provide the activation and performance study for intermediate layers in Appendix E ( see Figure 6 and Table 6 ) . Interestingly , early layers display less distinctive signatures between ID and OOD data . This is expected because neural networks generally capture lower-level features in early layers ( such as Gabor filters [ 61 ] in layer 1 ) , whose activations can be very similar between ID and OOD . The semantic-level features only emerge as with deeper layers , where ReAct is the most effective .

Summary: This proposes ReAct to reduce model overconfidence on OOD data. This paper observes the mean activation of OOD data in the penultimate layer has larger variations across units and is biased towards having sharp positive values, which produces overconfident predictions on OOD data. ReAct rectifies the activations at an upper limit for OOD data and preserves the activation for ID data. The results show that ReAct makes output distributions between ID and OOD data more separable.