SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To reduce the cost for parameter sensitivity estimation, it is advisable to compute the parameter sensitivity based on a subset of the training set rather than the entire set. To demonstrate the effectiveness of using a subset for sensitivity estimation. We randomly select  $1\%$ ,  $5\%$ ,  $20\%$  and  $100\%$  training samples for sensitivity estimation, and perform parameter pruning based on the sensitivities. The experimental results are shown in Table 8, which demonstrate that using only  $1\%$  of the training set (ImageNet-1K) also achieves promising performance.
B.3 Neuron sensitivity estimation with different statistics
In the main experiments, we utilize the average sensitivity (equivalent to  $\ell_1$  norm) as the neuron sensitivity. It is worth noting that other statistics, such as  $\ell_2$  norm, Variance, Maximum (Max),
Table 8: Comparison of experimental results when estimating parameter sensitivity with different numbers of training samples. The performance of OPP in ResNet50 is reported. w/o indicates Energy baseline without parameter pruning. Numbers are percentages.
Sampling Ratio	w/o pruning	1%	5%	20%	100%
SUN	59.3/85.9	32.57/92.83	32.06/92.78	30.92/93.05	30.40/93.17
Places	64.9/82.9	42.30/90.00	41.96/90.08	41.21/90.33	40.76/90.65
Table 9: Comparison of experimental results when estimating neuron sensitivity with different statistics. The performance in ResNet50 is reported. Numbers are percentages.
Statistics	Mean	Max	Min	Median	\( \ell_2 \)Norm	Variance
SUN	26.7/94.7	32.4/93.1	26.0/95.1	26.3/94.6	31.2/92.7	46.1/88.5
Places	32.7/92.9	39.8/91.2	32.6/92.8	33.4/92.5	34.5/91.2	51.1/87.4
Minimum (Min) and Median may also feasible statistics to measure the neuron sensitivity. Therefore, we compare the performance of using different statistics as the neuron sensitivity. The results are presented in Figure 9, which show that using the Mean, Min and Median of the weights sensitivity as neuron sensitivity achieve similar performance, and using the Variance statistic performs worst. As the Mean value is more robust and less likely to be susceptible by noisy connections, we suggest the users utilize the Mean statistic of parameter sensitivity for neuron pruning.
C Sensitivity Distribution Visualization
C.1 Parameter and neuron sensitivity distribution on CIFAR benchmarks.
In Fig. 7 and Fig. 8, we illustrate the parameter and neuron sensitivity distribution for CIFAR10 and CIFAR100 classification models, which reveal several interesting insights. First, on both CIFAR10 and CIFAR100 classification models, there are a few parameters that exhibit exceptionally high sensitivities and close to zero sensitivities, which is consistent with the sensitivity distribution on the ImageNet benchmark. Second, there are more parameters with close to zero sensitivities and more neurons with abnormal sensitivities on CIFAR100 classification model, which explains why OPNP brings more performance gains on CIFAR100 benchmark.
(b) CIFAR100
Figure 7: Illustration of parameter sensitivity distribution of classification models on (a) CIFAR10 and (b) CIFAR100. The parameters are taken from the last fully-connected layer in ResNet50, which are  $10 \times 2048$  and  $100 \times 2048$  for CIFAR10 and CIFAR100 classification model respectively. The dotted line in red indicates the average sensitivity and the maximum sensitivity is normalized to 1.
C.2 Parameter Sensitivity Distribution of the Pruned Weights
The parameter sensitivity distribution (on ID and OOD set) of the pruned weights is illustrated in Fig. 9. The results show that the pruned weights contain many high sensitive connections for OOD set.
(b) CIFAR100
Figure 8: Illustration of neuron sensitivity distribution on (a) CIFAR10 and (b) CIFAR100. The neurons are from the pre-logits layer in ResNet50. The maximum sensitivity is normalized to 1.
Besides, the average sensitivity on OOD set (0.00024) is larger than the average sensitivity on ID set (0.00018), which experimentally verifies Eq. 11.
Figure 9: Sensitivity distribution on ID and OOD samples. The top  $20\%$  sensitive parameters (measure on ID set) are illustrated. The average sensitivity on OOD samples is 0.00024 and the average sensitivity on ID samples is 0.00018.
The whole OOD detection pipeline is presented in Algorithm 1. In the main experiments, we integrate parameter pruning and neuron pruning in a unified framework, which actually can be used individually. The effectiveness of parameter and neuron pruning may vary for different model architectures and ID tasks. Therefore, we recommend readers to attempt different pruning percentage combinations for different tasks. Energy score [31], ReAct [43], DICE [44] ans ASH [6] are the most relevant researches to our work. The difference between our OPNP and the baseline method - Energy Score [31] is that we remove some output connections and hidden neurons according to pre-computed parameter and neuron sensitivities. Different from ReAct [43], which clips features with abnormal magnitude, our OPNP removes the parameters and neurons with abnormal sensitivity. The differences between our method and DICE [44] are: (1) DICE obtains the weight importance by averaging the contribution of weights to logit output, while our method measures the parameter and neuron sensitivity to energy score by averaging gradient; (2) DICE only removes the weights with low importance, while our OPNP not only removes the non-sensitive connections and neurons but also remove the most sensitive ones. Besides, ASH clips or binarizes the features based on the magnitude to improve OOD performance, whereas our OPNP prunes both weights and neurons based on sensitivity. Compared to those methods that also explore sparsity to improve OOD detection performance, the
introduced sensitivity metric measured by output energy is directly coupled with OOD detection tasks, and the sensitivity guided pruning is more intuitive, comprehensive and user-friendly.
Algorithm 1 OPNP - Optimal Parameter and Neuron Pruning
Input: A trained model  $f(\pmb {x};\pmb {\theta})$  training samples  $\{\pmb {x}_i\}_{i = 1}^m$  , pruning percentage  $\rho_{max}^{w},\rho_{min}^{w},\rho_{min}^{o}$ $\rho_{max}^{o}$  , test sample  $\{\pmb {x}_k\}_{k = 1}^n$    
Output: Energy score of test sample  $\{E(\pmb {x}_k)\}_{k = 1}^n$    
1: for  $i = 1$  to m do   
2: compute gradient  $g(\pmb {x}_i)$    
3: end for   
4:  $\mathbf{M} = \frac{1}{m}\sum_{k = 1}^{m}|g(\pmb {x}_k)|$  .   
5:  $\mathbf{O}_i = \frac{1}{K}\sum_{p = 1}^{K}\mathbf{M}_{ip},\quad i = 1,\dots ,L;$    
6: Get threshold  $\Omega_{min}^{w}$ $\Omega_{max}^{w}$  by ranking M;   
7: Get threshold  $\Omega_{min}^{o}$ $\Omega_{max}^{o}$  by ranking O;   
8: W[M>  $\Omega_{max}^{w}] = 0$  and W[M<  $\Omega_{min}^{w}] = 0$  .   
9: for  $k = 1$  to n do   
10: Compute the pre-logit embedding  $h(\pmb {x}_k)$    
11:  $h(\pmb {x}_k)[\mathbf{O} > \Omega_{max}^o ] = 0$  and  $h(\pmb {x}_k)[\mathbf{O} <   \Omega_{min}^o ] = 0$  .   
12:  $f(\pmb {x}_k) = \mathbf{W}\cdot h(\pmb {x}_k) + \pmb {b};$    
13:  $E(\pmb {x}_k) = -\log \sum_{i = 1}^{K}\exp (f_i(\pmb {x}_k));$    
14: end for   
15: Return  $\{E(x_k)\}_{k = 1}^n$