SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sima Behpour
Thang Doan
Xin Li
Wenbin He
Liang Gou
Liu Ren
Bosch Research North America, Bosch Center for Artificial Intelligence (BCAI) {sima_behpour, thang.doan, xin.li9, wenbin.he2, liang.gou, liu.ren} @us.bosch.com
Detecting out-of-distribution (OOD) data is crucial for ensuring the safe deployment of machine learning models in real-world applications. However, existing OOD detection approaches primarily rely on the feature maps or the full gradient space information to derive OOD scores neglecting the role of most important parameters of the pre-trained network over in-distribution (ID) data. In this study, we propose a novel approach called GradOrth to facilitate OOD detection based on one intriguing observation that the important features to identify OOD data lie in the lower-rank subspace of ID data. In particular, we identify OOD data by computing the norm of gradient projection on the subspaces considered important for the in-distribution data. A large orthogonal projection value (i.e., a small projection value) indicates the sample as OOD as it captures a weak correlation of the ID data. This simple yet effective method exhibits outstanding performance, showcasing a notable reduction in the average false positive rate at a  $95\%$  true positive rate (FPR95) of up to  $8\%$  when compared to the current state-of-the-art methods.
The issue of identifying out-of-distribution (OOD) data, which falls outside training data distributions, has become a significant focus in deep learning. OOD data challenges real-world model deployment, as it can lead to unreliable or incorrect predictions, particularly in safety-critical applications such as healthcare, autonomous vehicles, and physical sciences [14, 55, 64, 1, 4, 2]. This problem arises because modern Deep Neural Networks (DNNs) produce overconfident predictions on OOD inputs, complicating the separation of in-distribution (ID) and OOD data [63, 41]. The main goal of OOD detection is to develop methods that can accurately detect when a model encounters OOD data, allowing the model to either reject these inputs or provide more informative responses, such as uncertainty indication or confidence measures.
Many studies have investigated approaches to detecting OOD in deep learning [5, 18, 22, 27, 29, 32-34, 38, 39]. The majority of prior work focused on calculating OOD uncertainty from the activation space of a neural network, for example, by using model output [18, 27, 32, 34] or feature representations [29]. Another line of studies like ODIN [32], GradNorm [23], and ExGrad [24] leverage the gradient information of deep neural network models to compute OOD uncertainty score and achieve performant results. GradNorm [23] investigates the richness of the gradient space and presents that gradients provide valuable information for OOD detection. In particular, GradNorm utilizes the vector norm of gradients explicitly as an OOD scoring function. GradNorm, however, considers the full gradient space information, which might be noisy and lead to sub-optimal solutions.
A recent direction of research employs network parameter sparsification to improve OOD detection performance like DICE [47] and ASH [12]. ASH removes a majority of the activation by obtaining the  $p$ th-percentile of the entire representation. However, the potential consequence may result in
diminished performance due to the partial removal of critical parameters within the pre-trained network. Also, this is an empirical approach without a principled way to sparsify models.
Based on key observations presented in gradient-based and sparcification-based OOD detection methods, an intriguing insight emerges: the crucial discriminative features for OOD data identification reside within the gradient subspace of the ID data. This suggests that by focusing on the gradient information in the subspace of the ID data, which captures the most salient information, we can enhance the accuracy and reliability of OOD data detection algorithms. However, it takes non-trivial work to identify such an efficient gradient subspace.
Inspired by recent low-rank factorization research for DNNs [59, 52, 20, 51], indicating the intrinsic model information resides in a few low-rank dimensions, we introduce a novel approach named GradOrth. More specifically, the proposed method, GradOrth, distinguishes OOD samples by employing orthogonal gradient projection in the low-rank subspaces of ID data (figure 1). These ID subspaces ( $S^L$  in figure 1) are derived through singular value decomposition (SVD) of pre-trained network activations, specifically on a small subset of randomly selected ID data. By leveraging SVD, GradOrth effectively computes and identifies the relevant subspaces associated with the ID data, enabling accurate discrimination of OOD samples through orthogonal gradient projection. A large magnitude (figure 1-a) of orthogonal projection (i.e., small projection) serves as a significant criterion for classifying a sample as OOD since it captures a weak correlation with the ID data.
Our key results and contributions are:
- We present GradOrth, a novel and efficient method for out-of-distribution (OOD) detection. Our approach leverages the most important parameter space of a pre-trained network and its gradients to accomplish this task. To the best of our knowledge, GradOrth is the pioneering endeavor to investigate and showcase the efficacy of the subspace of a DNN's gradients in OOD detection.
- We evaluate the performance of GradOrth on widely-used benchmarks, and it demonstrates competitive results compared to other post-hoc OOD detection baselines. Notably, GradOrth outperforms the strong baseline methods by consistently reducing the false positive rate at the 95th percentile (FPR95) by a margin ranging from  $2.71\%$  to  $8.05\%$ . Moreover, our experi
Figure 1: The main idea of GradOrth: Measuring the orthogonal projection of the gradient of a testing sample onto a  $k$ -dimension (e.g.  $k = 2$  here) subspace of pre-trained network on ID data, by the angle  $\alpha$  between  $\overrightarrow{g(x_i)} = \nabla_{\theta^L}\mathcal{L}(\pmb{\theta}^L)$  and  $\overrightarrow{O(x_i)} = P_S^L (\nabla_{\pmb{\theta}^L}\mathcal{L}(\pmb{\theta}^L))$ . If  $\alpha$  is large (i.e., small projection on subspace  $\mathcal{S}^L$ ), shown in (a), the sample  $x_{i}$  is weakly correlated to ID data, and therefore it is recognized as OOD. Otherwise, it is ID data, shown in (b).
ments highlight that GradOrth effectively enhances OOD detection capabilities while maintaining high accuracy in classifying in-distribution (ID) data.
- We present a comprehensive analysis, including ablation experiments and theoretical investigation, aimed at enhancing the understanding of our proposed method for OOD detection. Through these rigorous analyses, we aim to provide valuable insights and improve the overall comprehension of the intricacies and effectiveness of our OOD detection approach.
2 Background and Our Notations
We consider a neural network with  $L$  layers and a set of learning parameters  $\theta = \{\theta^l\}_{l=1}^L$  where  $\theta^l$  present the learning parameters of layer  $l$ .  $x_i^l$  denotes the representation of input  $x_i$  at layer  $l$  in the successive layers given the data input  $x_i$ . The network performs the following computation at each layer:
$$
x _ {i} ^ {l + 1} = \sigma \left(f \left(\theta^ {l}, x _ {i} ^ {l}\right)\right), \quad l = 1, \dots L, \tag {1}
$$
where,  $\sigma (.)$  is a non-linear function and  $f(.,.)$  is a linear function. We leverage matrix notation for input  $(X_{i})$  in convolutional layers and vector notation for input  $(x_{i})$  in fully connected layers. In the first layer,  $x_{i}^{1} = x_{i}$  refers to the raw input data. It is noteworthy to mention that our approach holds applicability across all layers of the network. However, our experimental investigations reveal that the last layer yields the most optimal performance, please refer to appendix, section B for our
empirical studies report. This preference is advantageous as it alleviates significant time complexity arising from gradient computations across multiple network layers.
2.1 Input and Gradient Space
Our algorithm capitalizes on the intrinsic property of stochastic gradient descent (SGD) updates lying within the span of input data points, as validated by [62, 20, 60]. The subsequent subsections present this relationship, specifically in fully connected layers. We present the details regarding convolutional layers in the appendix, section A.
Fully Connected Layer Consider a single-layer linear neural network in a supervised learning setup where each (input, label) training data pair is driven from a training dataset,  $\mathbb{D}$ . We use  $\pmb{x} \in \mathbb{R}^n$  to present the input vector,  $\pmb{y} \in \mathbb{R}^m$  to present the label vector in the dataset, and  $\pmb{\theta} \in \mathbb{R}^{m \times n}$  to express the learning parameters (weights) of the network. In general, the network is trained by minimizing a loss function (e.g. mean-squared error) as follows:
$$
\mathcal {L} = \frac {1}{2} | | \boldsymbol {\theta} \boldsymbol {x} - \boldsymbol {y} | | _ {2} ^ {2}. \tag {2}
$$
Following stochastic gradient optimization, we can present the gradient of this loss with respect to weights as:
$$
\nabla_ {\boldsymbol {\theta}} \mathcal {L} = (\boldsymbol {\theta} \boldsymbol {x} - \boldsymbol {y}) \boldsymbol {x} ^ {T} = \boldsymbol {\Omega} \boldsymbol {x} ^ {T}, \tag {3}
$$
Here,  $\Omega \in \mathbb{R}^m$  denotes the error vector. Consequently, the gradient update will reside within the input span  $(\pmb{x})$ , wherein the elements in  $\Omega$  exhibit varying magnitudes, thus influencing the scaling of  $\pmb{x}$  accordingly, please refer to section E for the proof. For simplicity, we have considered per-example loss (batch size of 1) and mean-squared loss function here. Furthermore, it is important to mention that the aforementioned relationship remains applicable even in the context of the mini-batch setting or when utilizing alternative loss functions such as cross-entropy loss, where the calculation of  $\Omega$  may differ. For more comprehensive information on this subject, please consult the appendix, specifically section D. The input-gradient relationship in equation 3 can be applied to any fully connected layer of a neural network where  $\pmb{x}$  is the input to that layer and  $\Omega$  is the error coming from the next layer. In addition, this equation also applies to the networks with non-linear units (such as ReLU) and cross-entropy losses, though  $\Omega$  will be calculated differently.
2.2 Matrix Approximation with SVD:
In our algorithm, we utilize singular value decomposition (SVD) for matrix factorization. Specifically, a rectangular matrix  $\pmb{R} = \pmb{U}\pmb{\Sigma}\pmb{V}^T \in \mathbb{R}^{m\times n}$  can be factorized using SVD into the product of three matrices. Here,  $\pmb{\Sigma}$  represents a matrix containing the singular values sorted along its main diagonal, while  $\pmb{U} \in \mathbb{R}^{m\times m}$  and  $\pmb{V} \in \mathbb{R}^{n\times n}$  denote orthogonal matrices [10]. In the case where the rank of the matrix  $\pmb{R}$  is  $r$  ( $r \leq \min(m,n)$ ), the matrix  $\pmb{R}$  can be represented as  $\pmb{R} = \sum_{i=1}^{r} \sigma_i \pmb{u}_i \pmb{v}_i^T$ , where  $\sigma_i \in \mathrm{diag}(\pmb{\Sigma})$  denotes the singular values and  $\pmb{u}_i \in \pmb{U}$  as well as  $\pmb{v}_i \in \pmb{V}$  represent the left and right singular vectors, respectively. Furthermore, we can formulate the  $k$ -rank approximation to the matrix as  $\pmb{R}_k = \sum_{i=1}^k \sigma_i \pmb{u}_i \pmb{v}_i^T$ , where  $k \leq r$ . The specific value of  $k$  can be determined as the smallest value that satisfies the condition  $||\pmb{R}_k||_F^2 \geq \epsilon_{th}||\pmb{R}||_F^2$ . In this equation,  $||\cdot||_F$  represents the Frobenius norm of the matrix, and  $\epsilon_{th}$  ( $0 < \epsilon_{th} \leq 1$ ) serves as the threshold [10]. For a comprehensive explanation of the SVD method, more explanation regarding  $k$ -rank matrix approximation, and the impact of  $\epsilon_{th}$  in method performance please refer to the appendix sections F, H, I, respectively.
2.3 Problem Statement: Out-of-distribution Detection
OOD is typically characterized by a distribution that represents unknown scenarios encountered during deployment. These scenarios involve data samples originating from an irrelevant distribution, whose label set has no intersection with the predefined set. Consider the supervised setting where a neural network is given access to a set of training data  $\mathbb{D} = \{\left(\mathbf{x}_i,y_i\right)\}_{i = 1}^N$  drawn from an unknown joint data distribution  $P$  defined on  $\mathcal{X}\times \mathcal{Y}$  in the training phase. We denote the input space and output space by  $\mathcal{X} = \mathbb{R}^n$  and  $\mathcal{Y} = \{1,2,\dots,m\}$ , respectively.
$$
z (\mathbf {x}) = \left\{ \begin{array}{l l} \mathrm {I D}, & \text {i f} O (\mathbf {x}) \geq \gamma \\ \mathrm {O O D}, & \text {i f} O (\mathbf {x}) <   \gamma . \end{array} \right. \tag {4}
$$
The parameter  $\gamma$  is typically selected to ensure a high percentage of correct classification for indistribution (ID) data, such as  $95\%$ . A major hurdle is to establish a scoring function  $O(\mathbf{x})$  that
effectively captures the uncertainty associated with OOD samples. Prior approaches have predominantly relied on various factors, including the model's output, gradients, or features, to estimate OOD uncertainty [23, 12]. In our proposed approach, we aim to compute the scoring function  $O(\mathbf{x})$  by leveraging orthogonal gradient projection on parameter subspace of a pre-trained network over ID data. The details of our methodology are described in the subsequent section.
2.4 Orthogonal Projection
In this section, we discuss the concept of orthogonal projection and our notation, which holds significant importance in our methodology. To simplify the explanation, we will present it in a 2D space, but it can be extended to higher dimensions. Orthogonal projection serves as a metric that we utilize to calculate the distance between a vector  $\vec{V}$  and a space W (represented as a matrix in this case).
The result of the orthogonal projection of vector  $\overrightarrow{V}$  onto space W consists of three essential components: (1) The orthogonal projection vector  $\overrightarrow{b}$ , (2) the projection vector  $\overrightarrow{c}$ , and (3) the angle  $\alpha$ . These three components' values can be utilized to determine the correlation between the vector  $\overrightarrow{V}$  and the space W. As depicted in the figure 2, a larger value of  $\overrightarrow{b}$  indicates a weaker correlation. On the other hand,  $\overrightarrow{c}$  exhibits the opposite pattern, where a larger
Figure 2: Orthogonal Projection
value of  $\vec{c}$  indicates a stronger correlation. Depending on the specific application requirements, any of these values can be chosen to compute the correlation. In order to align with the OOD score, where a smaller value indicates a higher degree of OODness as presented in equation 4, we incorporate the projection vector  $(\vec{c})$  into our computations.
3 Our Method: GradOrth
In this section, we describe our method GradOrth where we recognize ID Vs. OOD data from a different view of previous studies, computing the norm of gradient projection on the subspaces considered important for the in-distribution data.
We define a sample as OOD data if its orthogonal projection value is large (i.e., small projection value), indicating a weak correlation with the ID data.
GradOrth OOD detection is developed following these steps:
1. Pre-trained Network subspace Computation: Our  $L$ -layer neural network with learning parameter  $\theta$  is trained using ID data. Upon completion of the training process, the model parameters  $\theta$  are frozen, resulting in a pre-trained network specialized in ID data. It is worth mentioning that we can also leverage the existing pre-trained network over our interest ID data. To retain the most significant parameters of the pre-trained network with respect to the ID data, we compute the network's last layer  $(L)$  subspace. For this purpose, we construct a representation matrix denoted as  $\pmb{R}_{ID}^{L} = [\pmb{x}_{1}^{L},\pmb{x}_{2}^{L},\dots,\pmb{x}_{n}^{L}]$ , which concatenates  $n$  representations obtained from the network's last layer  $(L)$  through the forward pass of  $n$  randomly selected samples (a small subset,  $n\ll N$ ) of the ID data.
Next, we perform SVD on  $\pmb{R}_{ID}^{L}$ , resulting in  $\pmb{R}_{ID}^{L} = \pmb{U}_{ID}^{L}\pmb{\Sigma}_{ID}^{L}(\pmb{V}_{ID}^{L})^{T}$ . We then proceed to approximate its rank  $k$  by obtaining  $(\pmb{R}_{ID}^{L})_{k}$ , guided by the given criteria that rely on a specified threshold, denoted as  $\epsilon_{th}$ :
$$
\left| \left| \left(R _ {I D} ^ {L}\right) _ {k} \right| \right| _ {F} ^ {2} \geq \epsilon_ {t h} \| R _ {I D} ^ {L} \| _ {F} ^ {2}. \tag {5}
$$
The pre-trained network subspace, denoted as  $S^L = \text{span}\{\pmb{u}_1^L, \pmb{u}_2^L, \dots, \pmb{u}_k^L\}$ , is defined as the space of significant representation for the pre-trained network at the last layer  $L$ . This subspace is spanned by the first  $k$  vectors in  $U_{ID}^{L}$  and encompasses all directions associated with the highest singular values in the representation. We store this subspace,  $S^L$ , and leverage it in the next step. We present the algorithm to compute the ID subspace in Algorithm 1.
2. Inference with OOD Data: During the inference phase, the pre-trained model is exposed to an OOD sample  $x_{i}$ . The OOD sample is propagated through the pre-trained network, and subsequently, its gradient at layer L is computed which is presented as  $g(x_{i}) = \nabla_{\pmb{\theta}^{L}}\mathcal{L}(\pmb{\theta}^{L})$
In accordance with the GradNorm approach, we calculate the cross-entropy loss by comparing the model's predicted softmax probability to a uniform vector used as the target. Consequently, during testing, we employ an all-one vector as the ground truth, assuming a uniform distribution for the target data.
3. Detector Construction: The model is transformed into a detector by generating a score based on its output, enabling the differentiation between ID and OOD inputs. To this end, we compute the norm of sample gradient projection onto the subspace of the pre-trained network  $(S)$ . We compute projection of the gradients  $\nabla_{\theta^L}\mathcal{L}(\theta^L)$  onto the subspace  $S^L$  as follows:
$$
P _ {S ^ {L}} \left(\nabla_ {\boldsymbol {\theta} ^ {L}} \mathcal {L} \left(\boldsymbol {\theta} ^ {L}\right)\right) = \left(\nabla_ {\boldsymbol {\theta} ^ {L}} \mathcal {L} \left(\boldsymbol {\theta} ^ {L}\right)\right) \mathcal {S} ^ {L} \left(\mathcal {S} ^ {L}\right) ^ {\prime}. \tag {6}
$$
Here,  $(.)'$  presents the matrix transpose. Next, we define the OOD score for the sample as follows by computing the projection norm:
$$
O \left(x _ {i}\right) = \left\| P _ {S ^ {L}} \left(\nabla_ {\boldsymbol {\theta} ^ {L}} \mathcal {L} \left(\boldsymbol {\theta} ^ {L}\right)\right) \right\| \tag {7}
$$
This score serves as a surrogate to characterize the correlation between the sample and ID data that the pre-trained network trained on it. As presented in figure 1, it implies a weak correlation between the new sample  $x_{i}$  and ID when the gradient  $g(x_{i}) = \nabla_{\theta^{L}}\mathcal{L}(\pmb{\theta}^{L})$  has a small projection (large orthogonal projection) onto the subspace of the pre-trained network (large angle  $\alpha$ ) due to the fact that stochastic gradient descent (SGD) updates lie in the span of input data points [65], please refer to the appendix, section E for the proof. Algorithm 2 presents OOD score computation.
Algorithm 1 ID Subspace Computation	Algorithm 2 OOD Score Computation
1: function Compute subspace (fθ, ID, εth)	1: function OOD Detector (fθ, IDOD, SL)
2: //Initialization	2: //Initialization
3: SL← []	3: O← []
4: Bn← Sample a mini-batch of size n from ID data (ID)	4: for xi ∈ DOOD do
5: θ← Pre-trained network f learning parameters	5: // Gradient computation
6: // subspace Computation	6: ∇θ(L(θ)) ← SGD(xi, fθ)
7: RLID← forward(Bn, f(θ)),	7: //Gradient projection computation
8:ˆLID← SVD(ˆRLD)	8: PSL(∇θL L(θL)) ← (∇θL L(θL))SL(SL)′
9: k← criteria(ˆRLD, RLD, εth) // Refer to equation 5	9: O(xi) ← ||PSL(∇θL L(θL))||
10: SL←ˆLID[0:k]	10: end for
11: return SL	11: return O
In this section, we evaluate the performance of our method GradOrth running extensive experiments considering different ID/OOD datasets and network architectures. We follow the experiment setting in general OOD baselines and explain the experimental setup in section 4.1. These empirical studies demonstrate the superior performance of GradOrth over existing state-of-the-art baselines that are reported in section 4.1. We report extensive ablations and analyses that provide a deeper understanding of our methodology, please refer to the appendix, section G.
Dataset We leverage 2 benchmarks proposed by [22] and [12] for detecting OOD images that are based on the large-scale ImageNet dataset and CIFAR dataset. To provide a fair comparison, we adopt an average results-over-5-run approach. In each run, distinct random seeds are employed to select random samples from each class, generating small subsets of in-distribution data. Subsequently, we compute the subspace of the pre-trained network based on these subsets. The OOD scores of the test data are then calculated, and FPR95 and AUROC scores are derived. This process is repeated five times, and the average of these five runs is reported as the final score.
ImageNet Benchmark: This benchmark is more challenging than others because it has higher-resolution images and a larger label space of 1,000 categories. To test our approach, we evaluate four OOD test datasets, including subsets of iNaturalist [53], SUN [56], Places [63], and Textures
[8]. These datasets have non-overlapping categories compared to the ImageNet-1k dataset and cover a diverse range of domains including fine-grained, scene, and textural images. We follow the experimental setting reported in [12] and use the Resnet-50 model [16] pre-trained on ImageNet-1k. For a fair comparison, all the methods use the same pre-trained backbone, without regularizing with auxiliary outlier data. Details and hyperparameters of baseline methods can be found in appendix J.1. The outcomes of this study present results obtained by applying GradOrth after the last fully connected layer in all the experiments. In this configuration, the feature size is 2048 for ResNet-50 and 1280 for MobileNetV2. For subspace computation, we choose 10 random samples per class and set the SVD threshold to 0.97.
CIFAR Benchmark: We evaluate our approach on the commonly used CIFAR-10 [26], and CIFAR-100 [26] benchmarks as in-distribution data following the experimental setting in [12, 50]. We employ the standard split with 50,000 training images and 10,000 test images. For subspace computation, we choose 5 random samples per class and set the SVD threshold to 0.97. We assess the model on six widely used OOD benchmark datasets: Textures [8], SVHN [40], Places365 [63], LSUN-Crop [61], LSUN-Resize [61], and iSUN [58]. Regarding pre-trained network architecture, we use DenseNet-101 architecture [21]. We leverage pre-trained networks over ID datasets. Please refer to section J.1 in the appendix for more details regarding the experiment setting. It is important to note that no modifications were made to the network parameters during the OOD detection phase.
Evaluation Metrics We assess the effectiveness of our proposed method by utilizing threshold-free metrics that are commonly used for evaluating OOD detection, as standardized in [18]. These metrics include (i) AUROC, which stands for the Area Under the Receiver Operating Characteristic curve; and (ii) FPR95, which is the false positive rate. FPR95 represents the probability that a negative (i.e., OOD) example is misclassified as positive (i.e., ID) when the true positive rate is as high as 95 [31].