SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chao Chen $^{1}$ , Zhihang Fu $^{1*}$ , Kai Liu $^{2}$ , Ze Chen $^{1}$ , Mingyuan Tao $^{1}$ , Jieping Ye $^{1}$ $^{1}$ Alibaba Cloud  $^{2}$ Zhejiang University  
{ercong.cc, zhihang.fzh, yejieping}@alibaba-inc.com
For a machine learning model deployed in real world scenarios, the ability of detecting out-of-distribution (OOD) samples is indispensable and challenging. Most existing OOD detection methods focused on exploring advanced training skills or training-free tricks to prevent the model from yielding overconfident confidence score for unknown samples. The training-based methods require expensive training cost and rely on OOD samples which are not always available, while most training-free methods can not efficiently utilize the prior information from the training data. In this work, we propose an Optimal Parameter and Neuron Pruning (OPNP) approach, which aims to identify and remove those parameters and neurons that lead to over-fitting. The main method is divided into two steps. In the first step, we evaluate the sensitivity of the model parameters and neurons by averaging gradients over all training samples. In the second step, the parameters and neurons with exceptionally large or close to zero sensitivities are removed for prediction. Our proposal is training-free, compatible with other post-hoc methods, and exploring the information from all training data. Extensive experiments are performed on multiple OOD detection tasks and model architectures, showing that our proposed OPNP consistently outperforms the existing methods by a large margin.
Over the past decade, deep neural networks have achieved dramatic performance gains in computer vision [7], natural language processing [38], and AI for Science [25]. However, when deploying those deep learning models in real world scenarios, the model will provide a false prediction result for unseen categories, which will lead to serious security issues. A promising approach to address this problem is Out-of-Distribution (OOD) detection [51], which aims to distinguish whether the given sample is from training class or unknown category. The deep learning models with good OOD detection ability know what they don't know and can be used safely in real world applications.
Recently, a large number of works have been proposed to address the OOD detection problem [22, 21, 26]. Early representative methods utilized the Maximum Softmax Probability (MSP) [21] or Mahalanobis distance [26] as score function. The main challenge is that modern overparameterized deep neural networks can easily produce overconfident predictions on OOD samples, making the in-distribution (ID) data and OOD data inseparable. To alleviate the overconfident problem, some training-based methods and post-hoc methods have been proposed. The training-based methods mitigate the overconfident problem by incorporating OOD samples in training process [22] or synthesizing virtual outliers to regularize the model's decision boundary [9]. The post-hoc methods mainly focused on optimizing score function [24, 29, 31], or rectifying activations [43, 6] to make the ID and OOD samples more separable. The training-based methods are controllable and interpretable, but they rely on expensive training cost and additional OOD samples, which are not always available.
(a) ResNet50
(b) ViT-B/16
Figure 1: Illustration of parameter sensitivity distribution for (a) ResNet50 and (b) ViT-B/16. The parameters are selected from the last fully-connected layer for both ResNet50 and ViT-B/16. The dotted line in red indicates the average sensitivity and the maximum sensitivity is normalized to 1.
In contrast, the post-hoc methods are training-free, low-cost and plug and play, however they can not effectively leverage the prior information from the training data and trained models.
It has been widely observed that overparameterized deep neural networks often suffer from redundant parameters and neurons, which consequently result in overconfident predictions. Conversely, a precisely pruned sub-network is capable of achieving comparable performance [13, 17, 5, 30, 46]. This motivates a straightforward question: Can we identify the parameters and neurons that lead to overconfident outputs by leveraging the prior information from the off-the-shelf models and training samples? To answer this question, we first demonstrate the parameter sensitivity of the last fully-connected (FC) layer for ResNet50 [18] and Vision Transformer (ViT-B/16) [7] in Fig. 1. As can be observed, the distribution of parameter sensitivity is heavily positively skewed. More specifically, a significant proportion of parameter sensitivities are close to zero, while only a few parameters exhibit exceptionally high sensitivities. The gap between the maximum and minimum sensitivity values is more than 200 times.
The above observation naturally inspires a simple yet surprisingly effective method — Optimal Parameter and Neuron Pruning (OPNP) for OOD detection. The motivations of our OPNP are in two aspects: (1) The parameters and neurons with sensitivities close to zero are redundant and can lead to overconfident predictions [23, 15, 44]. (2) The parameters with exceptionally large sensitivity result in a sharp landscape, which hurts the model generalization [11, 55, 54]. Therefore, in this study, we present empirical and theoretical evidences to show that the OOD detection performance can be significantly improved by pruning parameters and neurons with exceptionally large or close to zero sensitivities. The main contribution of this paper are as follows:
- A gradient based approach is present to estimate the sensitivity of parameters and neurons in deep model. Building upon this approach, we introduce OPNP - A simple yet effective training-free method, which significantly improves OOD detection performance by removing weights and neurons with exceptionally large or close to zero sensitivities.
- We evaluate OPNP on different OOD detection tasks and model architectures, including ResNet and ViT. Compared to the baseline model, OPNP achieves  $32.5\%$  FPR95 reduction on a large-scale ImageNet-1k benchmark, and outperforms existing state-of-the-art post-hoc OOD detection method by  $5.5\%$  in FPR95.
- Extensive ablation experiments are performed to reveal the insight and effectiveness of the proposed method. We show that OPNP is compatible with other post-hoc methods and can benefit model calibration. We believe our insights can inspire and accelerate future research in related tasks.
General OOD Detection. OOD detection is highly relevant to several early research tasks, including outlier detection (OD), anomaly detection (AD) [40] and open-set recognition (OSR) [14]. All
these tasks are aim to identifying the OOD samples in the open world scenario [51]. The most promising OOD detection methods can be divided into five categories, including density-based methods [39], distance-based methods [26, 45], outlier exposure methods [22, 28, 2], virtual OOD synthesis methods [9, 47], and post-hoc methods [31, 43]. Besides, some recent advances also pay attention to exploring large-scale pretrained vision-language model for OOD detection [35, 10], and extending OOD detection from classification to other learning tasks, such as object detection [8] and segmentation [20]. In this study, we mainly focus on post-hoc OOD detection, which is a training-free approach that does not require any OOD samples. For a more comprehensive understanding of the general OOD detection task, we suggest referring to [51] for further details.
Post-hoc OOD Detection. Recently, the post-hoc OOD detection methods have achieved promising performance and drawn increasing attention [44, 43, 57, 31, 29]. To alleviate the over-confident prediction caused by the softmax function, ODIN [29] introduces a temperature scaling strategy to make the softmax scores between ID and OOD images more separable. Liu et al. [31] replace the MSP score with energy score, which is theoretically aligned with the probability density and less susceptible to the overconfident problem. In huang et. al [24], GradNorm is presented which utilizes the magnitude of gradients as OOD score. Another line of work relieve overconfident problem by rectifying typical features in the penultimate layer [43, 6, 57]. In particular, ReAct [43] truncates features with a global threshold, which is chosen to preserve the activations for ID data while rectifying that of OOD data. ASH [6] shows that simply removing lower activations or binarizing representations leads to promising OOD detection performance. The most relevant method to our proposal is DICE [44], which ranks weights based on a measure of contribution, and selectively use the most salient weights to derive the output for OOD detection. Both DICE and our proposal alleviate over-fitting by model pruning. DICE selects the most important connections while our proposal removes the most sensitive and insensitive parameters and neurons. Besides, the method used to measure the weight importance or sensitivity is different.
Parameter and Neuron Pruning. In deep neural networks, parameter and neuron pruning is widely used for reducing over-fitting [23, 42, 15, 5] and model compression [33, 17, 27]. Dropout is the most well-known method to improve the robustness of deep models, which randomly drops some neurons [42] or connections [48] in training time. Based on dropout, Gomez et al. [15] introduce targeted dropout, which drops units and weights with low magnitude. They experimentally demonstrated that target dropout benefit to post-hoc pruning of units and weights. Besides, parameter and neuron pruning have also been widely used in model compression. Han et al. [17] indicate that the weights with low magnitude is less important and propose to remove those weights for model compression. In contrast, Li et al. [27] propose to drop the feature maps based on weight norms, the model performance is well preserved after pruning more than  $30\%$  units. In [30, 3], the authors demonstrate the effectiveness of ensembling multiple sparse networks, which are trained from scratch with dynamic sparsity constraint, for OOD detection. The aforementioned researches have demonstrated that the weights and neurons are significantly redundant in deep networks, which inspires us to prune the parameters and neurons that result in over-fitting for OOD detection. We believe optimal post-hoc parameter and neuron pruning is a promising approach for OOD detection.
3.1 Problem statement
Assume that we have an in-distribution dataset  $\mathcal{D}_{in}$  of pairs  $(\pmb{x}_{in},y_{in})$  and an out-of-distribution dataset  $\mathcal{D}_{out}$  of pairs  $(\pmb{x}_{out},y_{out})$ , where  $x_{in},x_{out}\in \mathcal{X}$  denote the input feature vector of ID and OOD samples,  $y_{in}\in \mathcal{Y}_{in}:= \{1,2,\dots ,K\}$  denotes the ID class label, and  $y_{out}\in \mathcal{Y}_{out}$  denotes the output class label,  $\mathcal{Y}_{in}\cap \mathcal{Y}_{out} = \emptyset$ . Given a classification model  $f(\pmb {x};\pmb {\theta})$  trained from in-distribution dataset  $\mathcal{D}_{in}$ . The goal of post-hoc OOD detection is to design a binary classifier  $g_{\lambda}(x)$  which is able to distinguish whether the test sample is from ID or OOD distribution. Therefore, the challenge of OOD detection is to find an optimal score function  $S(x)$  such that for a given test sample  $x\in \mathcal{X}$
$$
g _ {\lambda} (x) = \left\{ \begin{array}{l l} I D, & S (\boldsymbol {x}) \geq \lambda \\ O O D, & S (\boldsymbol {x}) <   \lambda \end{array} \right. \tag {1}
$$
where samples with higher score  $S(x)$  are classified as ID and vice versa,  $\lambda$  is the threshold which usually set to ensure 95% ID samples are correctly classified. Maximum softmax probability [21]
and energy score [31] are the most widely used score functions in post-hoc OOD detection. We follow previous post-hoc methods [43, 44] to utilize energy score as OOD detection metric, which consistently outperforms MSP score.
We denote  $h(\pmb{x}) \in \mathbb{R}^L$  the feature representation from the penultimate layer, denote  $\mathbf{W} \in \mathbb{R}^{L \times K}$  and  $\mathbf{b} \in \mathbb{R}^K$  the output weights and bias of the last FC layer. Then, the output logit can be given as
$$
f (\boldsymbol {x}; \boldsymbol {\theta}) = \mathbf {W} ^ {\top} h (\boldsymbol {x}) + \mathbf {b} \tag {2}
$$
The energy score function [31] maps the output logit to a energy value by,
$$
E (\boldsymbol {x}; \boldsymbol {\theta}) = - \log \sum_ {i = 1} ^ {K} \exp \left(f _ {i} (\boldsymbol {x})\right) \tag {3}
$$
where  $f_{i}(\pmb{x})$  denotes the logit output for class  $i$ . The energy score reduces over-fitting caused by softmax function, but the overparameterized connection weights  $\mathbf{W}$  in the last fully-connected layer may still cause overconfident logit output. Therefore, in the following section, we aim to provide an optimal parameter and neuron pruning strategy to reduce over-fitting.
3.2 Parameter sensitivity estimation
In this section, we propose to estimate parameter sensitivity by measuring how sensitive the output energy score to a small change of parameters. For a given sample  $\pmb{x}_k$  and corresponding energy output  $E(\pmb{x}_k; \pmb{\theta})$ , a small change  $\delta_{ij}$  is added to the parameter  $\pmb{\theta}_{ij}$ , which results in a change in the output energy score,
$$
E \left(\boldsymbol {x} _ {k}; \boldsymbol {\theta} + \boldsymbol {\delta}\right) - E \left(\boldsymbol {x} _ {k}; \boldsymbol {\theta}\right) \approx \sum_ {i, j} g _ {i j} \left(\boldsymbol {x} _ {k}\right) \boldsymbol {\delta} _ {i j} \tag {4}
$$
$$
g _ {i j} \left(\boldsymbol {x} _ {k}\right) = \frac {\partial E \left(\boldsymbol {x} _ {k} ; \boldsymbol {\theta}\right)}{\partial \boldsymbol {\theta} _ {i j}} \tag {5}
$$
Here,  $g_{ij}(\pmb{x}_k)$  denotes the gradient of model output to the parameter  $\theta_{ij}$  at data point  $\pmb{x}_k$ . Since  $\delta_{ij}$  is a small constant, the parameter sensitivity to model output can be measured by the magnitude of the gradient  $g_{ij}$ . In this respect, given a batch of samples  $\{\pmb{x}_k\}_{k=1}^m$ , the parameter sensitivity can be estimated by accumulating the gradients over all input samples,
$$
\mathbf {M} _ {i j} = \frac {1}{m} \sum_ {k = 1} ^ {m} \left| g _ {i j} \left(\boldsymbol {x} _ {k}\right) \right| \tag {6}
$$
where  $\mathbf{M}_{ij}$  denotes the sensitivity of parameter  $\theta_{ij}$ . It's worth noting that we utilize the change in energy score as the sensitivity measure, which is better aligned with the OOD detection metric. Practically, the parameter sensitivity can also be measured by other model output, such as the change of logit norm  $\| f(\pmb{x};\pmb{\theta})\|_2$ , which has been exploited in lifelong learning [1].
In Fig. 1, we illustrate the sensitivity of parameter  $\mathbf{W}$  for two representative deep networks ResNet50 [18] and ViT-B/16 [7], where the maximum sensitivity is normalized to 1. It can be seen that the maximum parameter sensitivity of the ResNet50 and ViT-B/16 is nearly 20 times than the average sensitivity. In addition, compared with ViT-B/16, there are more parameters with sensitivity close to zero in ResNet50, which indicates that the last FC layer of ResNet50 has more redundant parameters. Based on the intuition that the parameters and neurons with exceptionally large sensitivity or with sensitivity close to zero tend to result in overconfident prediction, an optimal parameter and neuron pruning strategy is introduced to improve the OOD detection performance.
3.3 Optimal parameter and neuron pruning
In this section, we mainly introduce how to prune the connection weights and neurons in the last fully-connected layer, which directly result in overconfident results. The pruning strategy in other layers can be achieved in the same manner.
(a) Optimal parameter and neuron pruning
(b) Sensitivity distribution
Figure 2: (a) Illustration of the last fully-connected layer before and after OPNP, the connections and neurons in grey color represent the pruned ones. (b) The first row illustrate the parameter sensitivity before and after pruning and the second row illustrate the neuron sensitivity before and after pruning.
Parameter Pruning. Given the connection weights  $\mathbf{W}$  that maps the feature representation to logit, the corresponding parameter sensitivity  $\mathbf{M}$  can be computed according to Eq. 6. As mentioned above, the parameters with exceptionally large or close to zero sensitivity tend to result in over-fitting. Therefore, a simple threshold function can be utilized to remove the risky connections,
$$
\overline {{\mathbf {W}}} _ {i j} = \left\{ \begin{array}{l l} 0, & \mathbf {M} _ {i j} <   \Omega_ {\text {m i n}} ^ {w} \\ 0, & \mathbf {M} _ {i j} > \Omega_ {\text {m a x}} ^ {w} \\ \mathbf {W} _ {i j} & \text {o t h e r} \end{array} \right. \tag {7}
$$
where  $\overline{\mathbf{W}}_{ij}$  denotes the weights after pruning,  $\Omega_{min}^{w}$  and  $\Omega_{max}^{w}$  denote the minimum and maximum thresholds. To align with previous post-hoc methods [43, 44, 6], we obtain the threshold by a percentile  $\rho$ , which indicates that the threshold is set to the  $\rho$ th-percentile of the entire sensitivity matrix. For example,  $\rho_{max}^{w} = 1\%$  represents that  $\Omega_{max}^{w}$  is set to the  $1\%$  largest sensitivity value in  $\mathbf{M}$ , and  $\rho_{min}^{w} = 10\%$  represents that  $\Omega_{min}^{w}$  is set to the  $10\%$  smallest sensitivity value in  $\mathbf{M}$ . In Fig. 2(b), the first row illustrates the sensitivity distribution of parameter  $\mathbf{W}$  in ResNet50 before and after parameter pruning. After pruning, the connection weights that larger than  $\Omega_{max}^{w}$  or smaller than  $\Omega_{min}^{w}$  are removed, and will not contribute to the output logit, which reduces the risk of over-fitting.
Neuron Pruning. In addition to parameter pruning, we also propose to prune the neurons in the pre-logit layer, which has also been shown to mitigate over-fitting [42, 15]. For the  $i$ -th neuron in the pre-logit layer, it contributes to all output neurons, therefore the sensitivity of the  $i$ -th neuron should be defined based on the sensitivity of the connection weights between the  $i$ -th hidden neuron and all output neurons. Considering that the  $\ell_1$  or  $\ell_2$  norm of the weights are usually used to measure the importance of the units in deep networks [41, 15]. We define the neuron sensitivity as the average sensitivity of weights that connected with the neuron, which is equivalent to  $\ell_1$  norm of the weight sensitivity, and reflects the average sensitivity to all classes, i.e.,
$$
\mathbf {O} _ {i} = \frac {1}{K} \sum_ {p = 1} ^ {K} \mathbf {M} _ {i p} \tag {8}
$$
where  $\mathbf{O}_i$  denotes the sensitivity of  $i$ -th neuron in pre-logit layer,  $K$  represents the number of output neurons. In this respect, we can use a similar threshold function as Eq. 7 to remove the risky neurons,
$$
\bar {h} ^ {i} (\boldsymbol {x}) = \left\{ \begin{array}{l l} 0, & \mathbf {O} _ {i} <   \Omega_ {\text {m i n}} ^ {o} \\ 0, & \mathbf {O} _ {i} > \Omega_ {\text {m a x}} ^ {o} \\ h ^ {i} (\boldsymbol {x}) & \text {o t h e r} \end{array} \right. \tag {9}
$$
where  $\overline{h}^i (\pmb {x})$  denotes the output feature from the  $i$ -th pruned neuron in the pre-logit layer,  $\Omega_{min}^{o}$  and  $\Omega_{max}^{o}$  represent the minimum and maximum sensitivity thresholds determined by the pruning percentage  $\rho_{min}^{o}$  and  $\rho_{max}^{o}$ . The second row in Fig. 2(b) illustrates the sensitivity distribution of the hidden neurons in ResNet50 before and after neuron pruning, where the maximum sensitivity is
normalized to 1. As observed, after neuron pruning, the redundant neurons (with sensitivity close to zero) and risky neurons (with sensitivity far above the average) are removed and the distribution of sensitivity across neurons becomes more uniform, which potentially reduces over-fitting.
3.4 Insight Justification
The following remarks are provided to explain why OPNP improves OOD detection performance.
Remark 1. Parameter and neuron pruning avoid overconfident predictions. The overparameterized deep neural networks tend to generate overconfident predictions even for OOD samples [19, 37, 16]. Therefore, most existing methods improve OOD performance by avoiding overconfident predictions [29, 31, 44]. For a deep network, the last fully connected layer can be regarded as a linear classifier. The most widely used technique to prevent a classifier from overfitting is to employ a  $\ell_1$  or  $\ell_2$  regularization, which can be formulated as  $\min_{\pmb{\theta}} \mathbb{E}_{(\pmb{x}, \pmb{y}) \sim D} \| \pmb{\theta}^{\top} \cdot h(\pmb{x}) - \pmb{y} \|_2^2 + \lambda \mathcal{R}(\pmb{\theta})$ , where  $\mathcal{R}(\pmb{\theta})$  represents  $\ell_1$ - or  $\ell_2$ -norm of  $\pmb{\theta}$ . As the parameter pruning is able to reduce  $\mathcal{R}(\pmb{\theta})$ , it can be regarded as an effective post regularization technique that reduces the model complexity and avoids overconfident predictions. Besides, the neuron pruning is similar to target dropout [15] which has also been demonstrated to reduce overfitting. Therefore, the proposed OPNP avoids overconfident predictions and potentially improves the OOD detection performance.
Remark 2. Pruning the least sensitive parameters and neurons improve separability between ID and OOD samples. We denote  $f_{j}(x)$  the logit output of  $j$ -th class, after pruning the least sensitive parameters, the logit reduction of the  $j$ -th class can be estimated as
$$
\Delta f _ {j} (\boldsymbol {x}) = \sum_ {\mathbf {M} _ {j k} <   \Omega_ {\text {m i n}} ^ {w}} \mathbf {M} _ {j k} \cdot | \mathbf {W} _ {j k} | \cdot h _ {k} (\boldsymbol {x}) \tag {10}
$$
It shows that the logit reduction is positively correlated with the average sensitivity of the pruned weights. As the parameter sensitivity is computed over the training ID set, the least sensitive parameters on ID distribution should be more sensitive for OOD samples on average, i.e.,
$$
\sum_ {\mathbf {M} _ {j k} <   \Omega_ {\text {m i n}} ^ {w}} \mathbf {M} _ {j k} ^ {O O D} > \sum_ {\mathbf {M} _ {j k} <   \Omega_ {\text {m i n}} ^ {w}} \mathbf {M} _ {j k} ^ {I D} \tag {11}
$$
Therefore, the logit reduction on OOD samples is larger than on ID samples  $\Delta f_{j}(\pmb{x}_{out}) > \Delta f_{j}(\pmb{x}_{in})$ , which leads to better separability between ID and OOD samples, and improves OOD detection performance. We also show the parameter sensitivity distribution (on ID and OOD sets) of the pruned weights in Fig. 9, which experimentally verifies Eq. 11.
Remark 3. Pruning the most sensitive parameters and neurons improves generalization. We follow [54] to define the first-order flatness as
$$
R _ {\rho} (\boldsymbol {\theta}) \triangleq \rho \cdot \max  _ {\boldsymbol {\theta} ^ {\prime} \in B (\boldsymbol {\theta}, \rho)} \| \Delta f \left(\boldsymbol {\theta} ^ {\prime}\right) \|, \quad \forall \boldsymbol {\theta} \in \Theta \tag {12}
$$