SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Considering equation 16, the gradient of this loss with respect to the weights can be presented as:
$$
\nabla_ {\boldsymbol {\theta}} L _ {\text {b a t c h}} = \rho_ {1} \boldsymbol {x} _ {1} ^ {T} + \rho_ {2} \boldsymbol {x} _ {2} ^ {T} + \dots + \rho_ {n} \boldsymbol {x} _ {n} ^ {T}. \tag {22}
$$
It is important to note that the gradient update is constrained within the subspace spanned by the  $n$  input examples.
F Singular Value Decomposition (SVD) Explanation
Consider an  $m \times n$  matrix  $\mathbf{R}$ , where  $m$  is the number of rows and  $n$  is the number of columns. The goal of SVD is to factorize matrix  $\mathbf{R}$  into three separate matrices:  $\mathbf{U}, \mathbf{\Sigma}$ , and  $\mathbf{V}^T$  (transpose of matrix  $\mathbf{V}$ ), such that  $\mathbf{R} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^T \in \mathbb{R}^{m \times n}$ , as presented in figure 5.
$\pmb{U}$ : An  $m \times m$  orthogonal matrix, where the columns represent the left singular vectors of  $\pmb{R}$ .  $\pmb{\Sigma}$ : An  $m \times n$  diagonal matrix, where the diagonal entries are the singular values of  $\pmb{R}$  (non-negative and
sorted in descending order).  $\pmb{V}^T$ : An  $n \times n$  orthogonal matrix, where the columns represent the right singular vectors of  $\pmb{R}$ .
Along with singular values and singular vectors, eigen-values and eigen-vectors are also defined. An eigenvalue  $\lambda$  and its corresponding eigenvector  $\mathbf{v}$  of a square matrix  $\pmb{R}$  satisfy the equation  $\pmb{R} = \lambda \pmb{V}$ . Eigen-vectors represent directions in the vector space that are only scaled by the matrix  $\pmb{R}$ , while eigenvalues represent the scaling factors for those eigen-vectors.
SVD and Relationship to Eigen-values and Eigen-vectors: SVD connects eigen-values and eigenvectors with the singular values and singular vectors of a matrix. The singular values of  $\pmb{R}$  are the square roots of the eigen-values of  $\pmb{R}\pmb{R}^T$  or  $\pmb{R}^T\pmb{R}$ , and the left and right singular vectors are the eigen-vectors of  $\pmb{R}\pmb{R}^T$  and  $\pmb{R}^T\pmb{R}$ , respectively.
Rank and Matrix Approximation: The rank of a matrix  $\mathbf{R}$  is determined by the number of non-zero singular values in  $\boldsymbol{\Sigma}$ . By keeping only the largest singular values and their corresponding singular vectors, it is possible to approximate the original matrix  $\mathbf{R}$  with a lower-rank approximation, which can be useful for dimensionality reduction and noise reduction. We leverage this feature in our approach.
Figure 5: Singular Value Decomposition
Properties of SVD:
The singular values in  $\pmb{\Sigma}$  are non-negative and arranged in descending order. The columns of  $\pmb{U}$  and  $\pmb{V}$  are orthonormal, meaning they form an orthogonal basis for their respective vector spaces. The SVD decomposition is unique up to the sign of the singular values and the order of the singular vectors. SVD is a powerful matrix factorization technique that provides a compact representation of a matrix while preserving important structural properties. It finds widespread applications in various fields, such as data analysis, image processing, recommendation systems, and more [10].
G Ablation study on the Impact of SVD
In this experiment, our focus is to examine the impact of utilizing singular value decomposition (SVD) in the GradOrth method as an ablation study. To achieve this, we compute the space (as opposed to the subspace) of the pre-trained network using the ID data, without employing SVD. Consequently, the OODness score in GradOrth-NoSVD solely incorporates the orthogonal projection of the new sample onto the space of the ID pre-trained network. The experimental results obtained from both the ImageNet and CIFAR benchmarks are presented in tables 12, 13, and 14. The outcomes reported in tables 12, 13, and 14 demonstrate the significant and robust performance of GradOrth compared to the GradOrth variant without SVD (GradOrth-NoSVD). On the ImageNet benchmark, GradOrth surpasses GradOrth-NoSVD by an average of  $14.12\%$  and  $10.93\%$  in terms of FPR95 for the ResNet and MobileNet pre-trained networks, respectively. Furthermore, GradOrth exhibits exceptional performance on the CIFAR benchmark, outperforming GradOrth-NoSVD by  $6.44\%$  and  $8.06\%$  on CIFAR10 and CIFAR100, respectively. The results obtained from this ablation study emphasize the significance of employing SVD in the GradOrth method's OODness score. It underscores the core principle of our approach, which suggests that the essential discriminative features for identifying OOD data reside within the subspace of the ID data.
Model	Methods	OOD Datasets	Average
iNaturalist	SUN	Places	Textures
FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑	FPR95 ↓	AUROC ↑
ResNet	GradOrth-NoSVD	28.31	94.30	30.09	93.73	44.18	81.43	28.17	93.21	32.69	90.67
GradOrth	11.04	98.00	19.61	95.76	33.67	91.78	11.19	98.06	18.57	96.31
MobileNet	GradOrth-NoSVD	38.45	90.21	38.02	89.26	49.31	90.18	28.52	89.68	38.58	89.83
GradOrth	26.81	93.17	30.82	93.18	40.27	89.12	12.69	97.52	27.65	93.25
Table 12: Ablation study to present the importance of SVD in GradOrth. GradOrth presents outstanding performance on average and across all datasets. OOD detection results with ImageNet-1k as ID. GradOrth presents outstanding performance on average and across all datasets. The ResNet and MobileNet models are pre-trained solely with ID data from the ImageNet-1k dataset. We use  $\uparrow$  to denote that larger values are preferable, and  $\downarrow$  to denote that smaller values are preferable.
Method	OOD Datasets	Average
SVHN	LSUN-c	LSUN-r	ISUN	Textures	Places365
FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC
↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑
GradOrth-NoSVD	18.32	94.03	1.12	99.57	3.56	98.74	10.29	98.04	33.17	93.10	46.22	85.62	18.78	94.85
GradOrth	5.84	98.72	0.81	99.78	2.33	98.71	4.25	98.32	20.63	94.77	38.22	91.64	12.34	96.99
Table 13: Ablation study to recognize the importance of SVD in GradOrth. Detailed results on six common OOD benchmark datasets with CIFAR-10 as ID: Textures [8], SVHN [40], Places365 [63], LSUN-Crop [61], LSUN-Resize [61], and iSUN [58]. For each ID dataset, we use the same DenseNet pre-trained on CIFAR-10.  $\uparrow$  indicates larger values are better and  $\downarrow$  indicates smaller values are better.
Method	OOD Datasets	Average
SVHN	LSUN-c	LSUN-r	iSUN	Textures	Places365	FPR95	AUROC
FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC
↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↑	↓	↑
GradOrth-NoSVD	31.43	93.70	9.53	96.82	57.69	87.54	51.93	92.11	42.09	90.16	55.76	82.97	41.41	90.55
GradOrth	24.27	93.47	3.71	99.07	48.09	91.26	42.73	91.48	32.71	92.62	48.61	89.03	33.35	92.82
Table 14: Ablation study to recognize the importance of SVD in GradOrth. GradOrth presents outstanding performance on average and across all datasets. Detailed results on six common OOD benchmark datasets with CIFAR-100 as ID: Textures [8], SVHN [40], Places365 [63], LSUN-Crop [61], LSUN-Resize [61], and iSUN [58]. For each ID dataset, we use the same DenseNet pre-trained on CIFAR-100.  $\uparrow$  indicates larger values are better and  $\downarrow$  indicates smaller values are better.
H  $K$ -Rank Matrix Approximation
Singular Value Decomposition (SVD) can be used to factorize a rectangular matrix,  $\pmb{R} = \pmb{U}\pmb{\Sigma}\pmb{V}^T\in$ $\mathbb{R}^{m\times n}$  into the product of three matrices, where  $\pmb {U}\in \mathbb{R}^{m\times m}$  and  $\pmb {V}\in \mathbb{R}^{n\times n}$  are orthonormal matrices, and  $\boldsymbol{\Sigma}$  is a diagonal matrix that contains the sorted singular values along its main diagonal (Deisenroth et al. [10]). If the rank of the matrix is  $r$  ( $r\leq \min(m,n)$ ),  $\pmb{R}$  can be expressed as  $\pmb{R} = \sum_{i=1}^{r}\sigma_{i}\pmb{u}_{i}\pmb{v}_{i}^{T}$ , where  $\pmb{u}_i\in U$  and  $\pmb{v}_i\in V$  are left and right singular vectors and  $\sigma_{i}\in diag(\pmb{\Sigma})$  are singular values.  $k$ -rank approximation of  $\pmb{R}$  can be written as,  $\pmb{R_k} = \sum_{i=1}^k\sigma_i\pmb{u}_i\pmb{v}_i^T$ , where  $k\leq r$  and its value can be chosen by the smallest  $k$  that satisfies the norm-based criteria:  $||\pmb{R_k}||_F^2\geq \epsilon_{th}||\pmb{R}||_F^2$ . Here,  $||.||_F$  denotes the Frobenius norm of the matrix and  $\epsilon_{th}\in (0,1)$  is the threshold hyperparameter.
$$
\boldsymbol {R} _ {k} = \sum_ {i = 1} ^ {k} \sigma_ {i} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {T} \tag {23}
$$
$$
\left| \left| \boldsymbol {R} - \boldsymbol {R} _ {k} \right| \right| ^ {2} = \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n} \left| a _ {i j} - \hat {a} _ {i j} \right| ^ {2} \sum_ {j = k + 1} ^ {r} \sigma_ {j} ^ {2} \quad 0 \leq k \leq n \text {w h e r e} R _ {k} = \hat {a} _ {i j}. \tag {24}
$$
The degree to which  $R_{k}$  approximates  $R$  depends on the sum of the r-k smallest singular values squared. As  $k$  approaches  $r$ , this sum becomes progressively smaller and eventually goes to zero at  $k = r$ . To provide a convenient measure for this behavior independent of the size of  $R$ , let us consider the normalized matrix approximation ratio
$$
\epsilon_ {t h} (k) = \frac {| | R _ {k} | |}{| | R | |} = \left[ \frac {\sigma_ {1} ^ {2} + \sigma_ {2} ^ {2} + \dots + \sigma_ {k} ^ {2}}{\sigma_ {1} ^ {2} + \sigma_ {2} ^ {2} + \dots + \sigma_ {r} ^ {2}} \right] ^ {\frac {1}{2}}, 1 \leq k \leq r. \tag {25}
$$
Clearly, this normalized ratio approaches its maximum value of 1 as  $k$  approaches  $r$ . For matrices of low effective rank,  $\epsilon_{th}(k)$  is close to 1 for values of  $k$  significantly smaller than  $r$ . On the other hand, matrices for which  $m$  must take on high values (i.e.,  $k \approx r$ ) to achieve a  $\epsilon_{th}(k)$  near 1 are said to be of high effective rank [3].
I Impact of the Threshold Parameter  $(\epsilon_{th})$  on GradOrth Performance
The hyperparameter  $\epsilon_{th}$ , confined to the range  $(0,1)$ , serves as a threshold that influences the selection of the value of  $k$  in the matrix  $k$ -rank approximation. The initial  $k$  column vectors within matrix  $\mathbf{U}$
encompass the most pivotal input (representation) space for the pre-trained network. We conducted an experiment to assess the impact of  $\epsilon_{th}$  on GradOrth's performance, with results outlined in table 15. Notably, values of  $\epsilon_{th}$  near 1 exhibit substantial effectiveness, and we empirically set it to 0.97.
Model	OOD Datasets	Average
SVHN	LSUN-c	LSUN-r	ISUN	Textures	Places365	FPR95	AUROC
FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC	FPR95	AUROC
↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑	↓	↑
GradOrth-εth=0.80	24.57	93.39	3.82	98.87	48.19	91.15	42.80	91.34	32.83	92.51	48.70	88.93	33.49	92.70
GradOrth-εth=0.90	24.36	93.42	3.75	99.00	48.12	91.24	42.74	91.46	32.76	92.58	48.64	89.01	33.39	92.78
GradOrth-εth=0.97	24.27	93.47	3.71	99.07	48.09	91.26	42.73	91.48	32.71	92.62	48.61	89.03	33.35	92.82
Table 15: Detailed results on six common OOD benchmark datasets considering different values for  ${\epsilon }_{th}$  used in  $k$  -rank matrix approximation. For the pre-trained network and ID dataset, we use DenseNet pre-trained on CIFAR-100.
J Details of Experiments
J.1 Model and Hyper Parameter
In our empirical studies and experiments, we adopt an experimental setting that aligns with the state-of-the-art (SOTA) approaches, specifically ASH [12] and DICE [47] on the CIFAR dataset, as well as ReAct [48] on the ImageNet dataset. The datasets and model architectures utilized in our experiments are summarized in table 16.
For the CIFAR-10 and CIFAR-100 experiments, we employ the six OOD datasets employed in the DICE study [47]: SVHN [40], LSUN-Crop [61], LSUN-Resize [61], iSUN [58], Places365 [63], and Textures [8]. The ID dataset used in these experiments corresponds to the respective CIFAR dataset. The model architecture employed is a pre-trained DenseNet-101 [21].
For our ImageNet experiments, we adhere to the precise setup as outlined in the ReAct study [48] and [12]. The ID dataset employed in this context is ImageNet-1k, while the OOD datasets consist of iNaturalist [53], SUN [56], Places365 [63], and Textures [8]. The network architectures utilized in these experiments are ResNet50 [16] and MobileNetV2 [44]. All networks undergo pre-training using the ID data and remain unaltered post-training, with their parameters remaining unchanged during the OOD detection phase. The performance of the baselines primarily relies on ASH and VRA. When conducting experiments involving gradient-based methods such as GradNorm [23] and ExGrad [24], we re-run the experiments using the code provided by the respective authors, as there may be variations between our pre-trained network over ID data and the models employed by the authors themselves.
ID Dataset	OOD Datasets	Model architectures
CIFAR-10	SVHN, LSUN C, LSUN R, iSUN, Places365, Textures	DenseNet-101
CIFAR-100	SVHN, LSUN C, LSUN R, iSUN, Places365, Textures	DenseNet-101
ImageNet	iNaturalist, SUN, Places365, Textures	ResNet50, MobileNetV2
Table 16: The datasets and models we used in our OOD experiments range from moderate to large scale, including evaluations of up to 10 OOD datasets and three architectures.
J.2 Datasets
ImageNet Benchmark, Large-scale evaluation In this study, the ImageNet-1k dataset [11] is employed as the ID dataset. The evaluation of the proposed approach is conducted on four OOD test datasets, following the experimental setup outlined in [23]:
- iNaturalist The dataset introduced by [53], referred to as "iNaturalist", comprises a substantial collection of 859,000 images featuring various plant and animal species. These images span more than 5,000 distinct species. To facilitate efficient processing, each image within the dataset is resized to ensure that the maximum dimension does not exceed 800 pixels. For the evaluation phase, a subset of 10,000 images is randomly sampled from a set of 110 classes. Importantly, these selected classes are disjoint from the ImageNet-1k dataset, thereby ensuring the validity and independence of the evaluation process.
- SUN that is introduced by Xiao et al., encompasses a vast collection of over 130,000 images representing various scenes. These scenes are categorized into 397 distinct categories.
Notably, it is important to acknowledge that there are overlapping categories between the SUN dataset and the ImageNet-1k dataset. For the evaluation process, a subset of 10,000 images is randomly sampled from a set of 50 classes. Importantly, these selected classes are disjoint from the labels present in the ImageNet dataset. This ensures the integrity and independence of the evaluation conducted in the study.
- Places The dataset introduced by [63], commonly referred to as "Places" in the research literature, is another notable scene dataset that exhibits similar concept coverage as the SUN dataset. In this study, a carefully selected subset of 10,000 images is utilized from a total of 50 classes. Importantly, these selected classes are intentionally excluded from the ImageNet-1k dataset, ensuring that the evaluation process remains independent and free from any potential overlap with the aforementioned dataset.
- Textures [7] consisting of 5,640 real-world texture images categorized into 47 distinct categories, is utilized in this research study. For the purpose of evaluation, the entire dataset is utilized, ensuring comprehensive coverage across all available categories.
CIFAR Benchmark The CIFAR-10 and CIFAR-100 datasets, introduced by [26], are extensively employed as ID datasets in the existing literature. CIFAR-10 comprises 10 classes, while CIFAR-100 consists of 100 classes. In line with standard practices, the dataset split utilized in this study consists of 50,000 training images and 10,000 test images. To evaluate the proposed approach, four commonly employed OOD datasets are utilized. The specific OOD datasets used in this evaluation are listed below:
- SVHN [40] The Street View House Numbers (SVHN) dataset consists of color images depicting house numbers. The dataset encompasses ten distinct classes corresponding to the digits 0-9. In this research study, the entire test set comprising 26,032 images is employed for evaluation purposes.
- LSUN [61] includes a collection of 10,000 testing images featuring 10 different scenes. In this research study, image patches of size  $32 \times 32$  are randomly cropped from the LSUN dataset to facilitate analysis and experimentation.
- Places365 [63] comprises a vast collection of large-scale photographs depicting scenes across 365 distinct scene categories. Notably, the test set of this dataset consists of 900 images per category. For the evaluation phase, a random sample of 10,000 images is drawn from the test set, thereby ensuring a representative subset for rigorous analysis and assessment.
- Textures [7] includes a comprehensive collection of 5,640 real-world texture images, classified into 47 distinct categories. In this research study, the entire dataset is utilized for evaluation, enabling a thorough examination of the performance and capabilities of the proposed approach across all available texture categories.
J.3Baselines
For the reader's convenience, we summarize in detail a few common techniques for defining OOD scores that measure the degree of ID-ness on the given sample. By convention, a higher (lower) score is indicative of being in-distribution (out-of-distribution).
MSP [18] utilizes probabilities obtained from softmax distributions to distinguish between correctly classified and erroneous or out-of-distribution examples. The baseline method relies on the observation that correctly classified examples tend to have higher maximum softmax probabilities than misclassified and out-of-distribution examples.
ODIN [32] is based on the observation that using temperature scaling and adding small perturbations to the input can separate the softmax score distributions between in- and out-of-distribution images, allowing for more effective detection.
Mahalanobis [29] utilizes multivariate Gaussian distributions to effectively model the class-conditional distributions of softmax neural classifiers. Additionally, they employed Mahalanobis distance-based scores as a means of detecting out-of-distribution samples.
Energy Score [34] The concept of utilizing energy scores for estimating out-of-distribution uncertainty was initially introduced by Liu et al.. The energy function employed in their approach maps the logit outputs to a scalar value denoted as  $S_{\mathrm{Energy}}(x; f) \in \mathbb{R}$ . Notably, this scalar value tends to be relatively lower for in-distribution data. It is important to mention that the authors adopted the convention of using the negative energy score for OOD detection, ensuring that  $S(x; f)$  exhibits higher values for ID data and lower values for OOD data.
GradNorm [23] computes the norm of the gradients of a neural network with respect to an input. The norm of the gradients is a measure of how sensitive the network is to the input. Inputs with high norms are more likely to be OOD because they are more likely to cause the network to make a mistake.
Exgrad[24] calculates the expected norm of the gradients of a neural network with respect to an input. The expected norm of the gradients is a measure of how sensitive the network is to the input.
DICE [47] aims at selectively utilizing a subset of significant weights to compute the output for OOD detection. By employing the technique of sparsification, the network effectively avoids incorporating irrelevant information into the output, thereby enhancing its OOD detection capabilities.
ReAct [49] This approach is based on the observation that OOD data trigger distinctive activation patterns in neural networks. ReAct selectively rectifies and truncates the activations of specific hidden units, reducing overconfident predictions on OOD data.
Variational Rectified Activation (VRA) [57] VRA leverages the variational method to find the optimal operation for maximizing the gap between ID and OOD data. It introduces suppression and amplification operations for abnormally low, high, and intermediate activations, unlike ReAct which only focuses on high activations. VRA uses piecewise functions to simulate these operations.
ASH [12] This method removes a large portion of activations and adjusts the remaining ones. The remaining activations (e.g.,  $10\%$ ) are either simplified or lightly adjusted. The simplified activation representation is then propagated through the rest of the network to generate scores for both classification and OOD detection. The energy score, calculated from the logits, is commonly used for OOD detection, although the softmax score can also be used.
J.4 Software and Hardware
Software We run all experiments with Python 3.8.0 and PyTorch 1.12.1.
Hardware All experiments are run on NVIDIA RTX 3090.